Compare commits

...
Author SHA1 Message Date
jschoubben 5b9908bed2 Hold a joining build for a person without superseding its delivery, join only on an engine that keeps own groups, and keep a held module's settings (hq ADR 0328 §8, ADR 0334; re-review of 96e15f86)
A delivery does not send a joining build held for its library or its engine: the module waits for a person,
the plan stays open saying so and judges its other modules, and sends it once nothing holds it. A held move is
no move for the gate. A module joins only on an engine at contract 7, and a held module is composed with the
settings it last ran with. The runner's verdicts are said for their modules and judged by whether a move
changes them. mesh-host pinned at #97 (7a41e120), its bundle now examples/foundation-first-node.lock.
2026-10-13 00:55:54 +02:00
jschoubben 1de9bbc448 Own directories of a joining module carry setgid, to an engine that takes it (hq ADR 0334 v4) 2026-10-13 00:31:54 +02:00
jschoubben 96e15f8693 Hold a joining module at the build it ran, own its directories by its own group, and take a library only when its act is done (hq ADR 0328 review, ADR 0334)
A held module is composed exactly as before it joined, at the build its machine runs, so its port, resources and
contributions stay. A joining module's directories belong to its user and its own group, and its tools' directories
are refused closed to that group; the runner joins it where the engine can say so. A take records the root and is
taken only when the engine states the act done. Libraries' paths and verdicts count modules pulled in by a
requirement, and a unit takes the identity.
2026-10-13 00:03:18 +02:00
jschoubben 7ea6d36eab Pin the node-engine at its ADR 0328 merge (#95), and speak its contract: a library's verdict is said for the modules that use it 2026-10-12 23:02:31 +02:00
jschoubben f634e33792 Libraries and the modules that join them, on the controller's side (hq ADR 0328) 2026-10-12 22:29:35 +02:00
jschoubben fa0a5f1c88 Carry over the ADR 0325 controller work as the base for ADR 0328 2026-10-12 21:57:29 +02:00
mesh-admin c0c379af2b Merge pull request 'Say the class of what a change plan moves, for a fix's budget (issue 403, hq ADR 0313)' (#265) from fix/403-fix-time into main 2026-10-12 19:41:26 +00:00
jochen daf52b3aeb Merge remote-tracking branch 'origin/main' into fix/403-fix-time
mesh/merge-gate head daf52b3aeb79 merged into trunk 4dbd4066ca54 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head daf52b3aeb79 merged into trunk 4dbd4066ca54 — pass: its merge-check.sh passed
mesh/delivery failed: its walk failed, carrying its own commit: a gate on a first machine (what it carried put back), a build, a machine
mesh/delivery-group group fix/403-fix-time failed: a member failed
# Conflicts:
#	internal/link/events.go
2026-10-12 21:28:52 +02:00
mesh-admin 4dbd4066ca Merge pull request 'A seat names the paths its holder owns, and a claim or a declaration without the seat is refused (issue 412, hq ADR 0287)' (#267) from fix/412-a-seat-names-its-paths into main 2026-10-12 19:22:11 +00:00
mesh-admin 84deb530ee Merge pull request 'Keep the controller and node-engine at the build a machine was last sent over an older record (hq issue 201)' (#273) from fix/201-a-push-never-downgrades-the-core into main 2026-10-12 18:52:04 +00:00
jochen b3787e4d8e Merge origin/main into fix/201-a-push-never-downgrades-the-core
mesh/merge-gate head b3787e4d8eb2 merged into trunk a681227af373 — pass: builds mesh-controller → novox; no bus step; every machine composes with the chang…
mesh/repo-check head b3787e4d8eb2 merged into trunk a681227af373 — pass: its merge-check.sh passed
mesh/delivery delivered
# Conflicts:
#	testdata/store-tests
2026-10-12 20:31:19 +02:00
mesh-admin a681227af3 Merge pull request 'Add issue 401's replay, written with what the controller had before its fix (hq issue 401)' (#274) from fix/401-replay into main 2026-10-12 18:22:09 +00:00
mesh-admin d8f36a8fb7 Merge pull request 'Read an xz release in Go, so the build seat needs no xz tool (hq issue 345)' (#264) from fix/345-xz-in-go into main 2026-10-12 17:35:56 +00:00
jochen f0fd7fe886 Leave a kept core move out of what a gate judges (issue 201, review 91)
mesh/merge-gate head f0fd7fe8868e merged into trunk a681227af373 — fail: builds mesh-controller → novox; no bus step; head f0fd7fe8868e does not merge clea…
mesh/repo-check head f0fd7fe8868e merged into trunk a681227af373 — fail: head f0fd7fe8868e does not merge cleanly into trunk a681227af373: in conflict test…
mesh/delivery superseded: a newer head of the same pull request
gatedSend took its judged moves from the record, so on a machine kept at
a newer controller the gate judged the controller moving back to the
record, read its own send's kept controller as a later build there, and
was superseded with nothing judged. The kept moves are now left out
there as in ungatedIn (withoutKept). The doc says the bus step keeps
too, and the three store tests are listed.
2026-10-12 19:16:26 +02:00
jochen 41a9ff15dd Merge origin/main into fix/201-a-push-never-downgrades-the-core 2026-10-12 19:07:20 +02:00
mesh-admin ba685b17f5 Merge pull request 'Let plans wait for a plan to move and answer it as data (issue 229)' (#266) from fix/229-plans-wait into main 2026-10-12 17:06:50 +00:00
jochen d170491256 Keep the core over an older record instead of refusing the send (issue 201, review 90)
mesh/merge-gate out of date: judged against trunk f8daec01d3f0, the trunk is now ba685b17f5e1; judged again on a recheck or when it is merged
mesh/repo-check out of date: judged against trunk f8daec01d3f0, the trunk is now ba685b17f5e1; judged again on a recheck or when it is merged
mesh/delivery superseded: a newer head of the same pull request
Refusing the whole send wedged a machine whose gate put back the
controller and whose put-back send was refused: every later automatic
send there was refused, retried each tick, and held up every other walk.

A send now composes the controller and the node-engine at the build the
machine was last sent when the mesh's record is older, and carries the
rest; nothing is refused or retried, and a plan whose own build is the
older one ends with its gate superseded. A sent build that failed its
gate is never kept, so the next send completes a put-back. The condition
names the two ways out and clears on any send that carries the record,
a person's push included.
2026-10-12 18:51:17 +02:00
jochen 6bb1c9b768 Merge remote-tracking branch 'origin/main' into fix/201-a-push-never-downgrades-the-core 2026-10-12 18:40:12 +02:00
mesh-admin f8daec01d3 Merge pull request 'Read a plan by its id as a parameter, never as SQL (issue 561)' (#268) from fix/561-plan-id-parameter into main 2026-10-12 16:37:07 +00:00
jochen f3c15e00af Add issue 401's replay, written with what the controller had before its fix, so the lab can prove it
mesh/merge-gate head f3c15e00af50 merged into trunk d8f36a8fb706 — pass: the change touches no module of the mesh's graph
mesh/repo-check head f3c15e00af50 merged into trunk d8f36a8fb706 — pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group fix/401-replay delivered: every member is delivered
The fix's own test reads GivenOwnSecret, which the commit before does not have, so it cannot be laid over it. TestReplay401 asserts the same outcome with register, assign, planFor, declarationFor and the inventory's secret reads; it fails on 1e28342^1 and passes on main. Named in testdata/store-tests so a skip without the store fails the check. Replay for hq issue 401.
2026-10-12 18:36:22 +02:00
jochen 9038d4b0d3 Refuse a send that puts the controller or node-engine behind its machine's build (issue 201)
mesh/merge-gate out of date: judged against trunk f10aedac5880, the trunk is now f8daec01d3f0; judged again on a recheck or when it is merged
mesh/repo-check out of date: judged against trunk f10aedac5880, the trunk is now f8daec01d3f0; judged again on a recheck or when it is merged
mesh/delivery superseded: a newer head of the same pull request
On 2026-10-02 a plan sent the control node a controller older than the one
running there, whose seat row the newer build had widened; the older binary
refused the row and the mesh went quiet for ten minutes. ADR 0185 took the
outage out of it, and the race stayed open.

Every send but a person's push now refuses a move of mesh-controller or
mesh-host to a build asked before the one its machine was last sent (the
builds' registration order, issue 219), whatever composed it — gated or not.
A gate's own put-back is the one exception, for the modules it puts back,
and says so in its send's scope. The refusal is the send's error, which the
plan keeps as its note, and a warning condition on the machine, cleared by
the next send there that carries no such move.
2026-10-12 18:21:17 +02:00
jochen 24347ea5a0 Step back from a rendered resolver file only, and compare a declared path cleaned (issue 412, review 80)
mesh/merge-gate head 24347ea5a02b merged into trunk 84deb530ee5a — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head 24347ea5a02b merged into trunk 84deb530ee5a — pass: its merge-check.sh passed
mesh/delivery delivered
A resource on the resolver file was left out by the check and still composed by the declaration, so an
uplink declaring it as a resource beside the resolver passed with two writers; it is refused again, as
before. And /etc//hosts or /etc/../etc/hosts is /etc/hosts to the machine, so it is compared cleaned.
A seat names no root or top-level directory.
2026-10-12 17:08:55 +02:00
jochen 5ad36127ee Hold the plans schema's longest bound to the value, and say what a slow store gives (review 75)
mesh/merge-gate head 5ad36127ee8b merged into trunk f8daec01d3f0 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head 5ad36127ee8b merged into trunk f8daec01d3f0 — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-12 16:50:59 +02:00
jochen a1a6f6493d Refuse a plans argument that is not a plan's id (issue 561)
mesh/merge-gate head a1a6f6493d50 merged into trunk f10aedac5880 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head a1a6f6493d50 merged into trunk f10aedac5880 — pass: its merge-check.sh passed
mesh/delivery delivered
Every argument of plans that names a plan (id, wait, stop, close, retry,
go) is checked once where the call becomes a command line, against the
ids the mesh gives plans (plan-<digits>, release-<digits>), so a quote
from a caller never reaches a command or the store (review 75 on #266).
2026-10-12 16:46:29 +02:00
jochen e9f060e0be Read a plan by its id as a parameter, never as SQL (issue 561)
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
PlanByID pasted its id into the query, and the id comes from a caller of
the plans verb: an id holding a quote changed the query (x' or '1'='1
answered another plan). The id is now $1, and RecentPlans's limit too, so
no plan query is built from a value.
2026-10-12 16:33:23 +02:00
jochen a3bf5da9b1 Name the paths a seat's holder owns, and refuse a claim or a declaration without the seat (issue 412)
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
Which module owns /etc/hosts or /etc/resolv.conf on a machine was answered only by reading every
manifest, and a module not holding node-hostname could declare /etc/hosts where nobody held it.
hq ADR 0287 makes the paths part of the seat's definition: node-hostname names /etc/hosts and
/etc/hostname, node-uplink names /etc/resolv.conf stepping back for node-resolver, node-resolver
names /etc/resolv.conf.

CanHold refuses a holder that does not declare every path (registration and handover), registration
refuses a module declaring a seat's path while claiming none of the seats naming it, a module's own
seat names paths under the same rules (node only, literal, not named by another seat), composition
refuses a generated resource on a seat's path, the uplink's step-back is read from the seat instead
of a special case, and `seats` shows each seat's paths and who owns each on each machine.
2026-10-12 16:19:49 +02:00
jochen d20b52cb19 Let plans wait for a plan to move and answer it as data (issue 229)
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
An agent following a rollout had plans answer once, in prose, and polled the
console with a curl loop of its own to be told a plan moved. plans now takes
wait (a plan's id), until (moved, tier or done) and bound, and answers when the
plan moves or the bound passes, saying which, with the plan as data and its
line as the output.

A call's one answer has to reach the console inside its thirty seconds, so
link.AnswersWithin lets a handler that waits on purpose hold its answer up to
LongestAnswer (25s) instead of AnswerWithin; the bound is at most 20s, and a
caller following a plan further asks again.
2026-10-12 16:18:54 +02:00
jochen 7ca5a45ecc Class a new module by its own name, so adding a core module is not called leaf (issue 403, review 69)
mesh/delivery-group group fix/403-fix-time checking: composed check out of date: novox/mesh-controller@7ca5a45ecc5a was judged against trunk ?, the trunk is no…
mesh/merge-gate out of date: judged against trunk 84deb530ee5a, the trunk is now 4dbd4066ca54; judged again on a recheck or when it is merged
mesh/repo-check out of date: judged against trunk 84deb530ee5a, the trunk is now 4dbd4066ca54; judged again on a recheck or when it is merged
mesh/delivery superseded: a newer head of the same pull request
2026-10-12 16:16:02 +02:00
jochen 539a1c16cd Refuse an xz file of more than 1,024 blocks before decoding it (review 68, hq issue 345)
mesh/merge-gate head 539a1c16cd4e merged into trunk ba685b17f5e1 — pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus…
mesh/repo-check head 539a1c16cd4e merged into trunk ba685b17f5e1 — pass: its merge-check.sh passed
mesh/delivery delivered
The xz reader keeps one delta or BCJ decoder for every block of a stream and frees them only when the
stream ends, so a file of many tiny blocks grows the heap with its length: about 67 MiB live for a 6 MB
`xz --block-size=1 --delta` file, about 5.6 GiB at the 512 MiB download cap.

The file's framing is now walked first, without decompressing: stream header, each block header, the
LZMA2 chunk headers (each carries its compressed size, so a chunk is skipped unread), padding, check,
index and footer. A file of more than 1,024 blocks or 16 streams is refused, as is an index whose block
count is not the count walked. The index alone is not trusted because the reader compares it with the
blocks only after decoding them all, so a lying index would still be decoded. Refusing delta and BCJ
outright was not chosen: it needs the same walk to see each block header, and would refuse sound
releases filtered with BCJ.

1,024 blocks keep at most about 0.35 MiB of decoders. Betterbird's archive is 1 stream with 1 block;
multi-threaded xz cuts blocks of 3 times the dictionary, so 512 MiB in blocks of 512 KiB still fits.
The library stays unforked.
2026-10-12 16:08:33 +02:00
jochen 05eccbf22b Say the class of what a change plan moves, so a fix's budget follows its modules (issue 403)
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery-group group fix/403-fix-time ready: every member ready, and composed together they pass
mesh/delivery superseded: a newer head of the same pull request
The issue tracker holds a fix to the budget of the slowest-budgeted module it
moves, and gives none to a fix that moves no module (hq ADR 0313 §2). The
walk's class covers every merge a walk carries; the plan's class is the
change's own: core, leaf, or none.
2026-10-12 15:50:11 +02:00
jochen 01fc9e1d73 Read xz with a real dictionary maximum (review 65, hq issue 345)
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
ulikunitz/xz's ReaderConfig.DictCap is a minimum: a stream asking for a
1 GiB or 4 GiB dictionary was given it. github.com/therootcompany/xz, a port
of xz-embedded, takes a maximum and refuses a block whose LZMA2 dictionary is
larger before allocating it. 64 MiB, xz's largest preset.
2026-10-12 15:39:34 +02:00
jochen 10f76cafc4 Read an xz release in Go, so the build seat needs no xz tool (hq issue 345)
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
Betterbird's pinned .tar.xz failed on the build seat with "this build seat has
no xz to read it": the seat's image is the catalogue's build-agent recipe,
which installs git and docker-cli alone, and xz had been added only to
cmd/mesh-builder/Dockerfile, which the seat does not build from. The stream
is now read with github.com/ulikunitz/xz (vendored), its dictionary capped
at xz's own largest preset, and what it unpacks to bounded as before; the
xz line in the unused Dockerfile is taken back out.
2026-10-12 15:10:52 +02:00
91 changed files with 11126 additions and 173 deletions
+2 -3
View File
@@ -16,8 +16,7 @@ RUN CGO_ENABLED=0 go build -trimpath -ldflags '-s -w' -o /mesh-builder ./cmd/mes
FROM ${ALPINE_BASE}
# git to clone what it is asked to build, and the docker client to build and push it. The daemon
# is the machine's, reached through its socket — a build machine shares the runtime it was given
# rather than running one inside itself. xz to read an upstream release an archive is pinned to
# (novox/hq ADR 0316): Go's standard library reads gzip and bzip2 and not xz.
RUN apk add --no-cache git docker-cli xz
# rather than running one inside itself.
RUN apk add --no-cache git docker-cli
COPY --from=build /mesh-builder /usr/local/bin/mesh-builder
ENTRYPOINT ["/usr/local/bin/mesh-builder"]
+23 -2
View File
@@ -117,10 +117,13 @@ type asked struct {
// Proposal is a settings layer proposed through a verb (proposals.go, novox/hq ADR 0277): about no
// condition, set on Approve by the serving controller itself, and left alone by the reconciling of conditions.
Proposal *settingsProposal `json:"proposal,omitempty"`
// Take is a library the operator is asked to let the mesh take (kept_libraries.go, novox/hq ADR 0328 §7): about no
// condition, taken on Approve by the serving controller itself, asked once per library.
Take *libraryTake `json:"take,omitempty"`
}
// ofACondition says an ask is one of a condition's: not a rehearsal, not a proposal.
func (r asked) ofACondition() bool { return !r.Rehearsal && r.Proposal == nil }
func (r asked) ofACondition() bool { return !r.Rehearsal && r.Proposal == nil && r.Take == nil }
// partKey is an ask's place among what is asked: its condition and its part.
func partKey(condition, part string) string { return condition + "#" + part }
@@ -192,6 +195,11 @@ type asker struct {
call func(ctx context.Context, a conditions.Action, args map[string]string) error
// setLayer sets a proposed settings layer on the operator's warrant (novox/hq ADR 0277); nil cannot.
setLayer setOnWarrant
// folders is every library waiting to be taken (novox/hq ADR 0328 §7), and take what takes one on the operator's
// warrant; nil asks about none. machines are the mesh's machine names, allowed in an ask's words.
folders func(ctx context.Context) ([]libraryTake, error)
take takeOnWarrant
machines func(ctx context.Context) ([]string, error)
// record writes the hand-act log.
record func(ctx context.Context, act link.HandAct) error
// routerRecord reads the router's record of an ask for a warrant missed; nil reads nothing.
@@ -554,6 +562,10 @@ func (a *asker) reconcile(ctx context.Context) error {
return err
}
}
// And every library waiting to be taken, once per library (novox/hq ADR 0328 §7).
if err := a.askFolders(ctx); err != nil {
a.logf("the operator could not be asked about the libraries waiting to be taken: %v", err)
}
why := "the router refused the ask"
if len(refusedWords) > 0 {
why += ": " + refusedWords[0]
@@ -818,7 +830,7 @@ func (a *asker) Decided(ctx context.Context, body []byte) error {
if err != nil {
return err
}
stillOpen := r.Rehearsal || r.Proposal != nil // a rehearsal and a proposal are about no condition
stillOpen := r.Rehearsal || r.Proposal != nil || r.Take != nil // a rehearsal, a proposal and a take are about no condition
for _, c := range open {
stillOpen = stillOpen || c.Key == r.Condition
}
@@ -876,6 +888,15 @@ func (a *asker) Decided(ctx context.Context, body []byte) error {
if acted == nil && outcome != "" && !strings.HasPrefix(outcome, "nothing") {
outcome = "done: " + outcome
}
case r.Take != nil && act.Verb == takeVerb:
// A library taken on the operator's warrant (novox/hq ADR 0328 §7): the act is composed again from the
// record and held to what the option bound before the engine is asked.
outcome, acted = a.decideTake(ctx, *r, act, w)
if acted == nil && !strings.HasPrefix(outcome, "nothing") {
outcome = "done: " + outcome
}
case r.Take != nil:
acted = fmt.Errorf("the take %s offers no act %s", r.ID, act.Verb)
case act.Arguments["silence"] != "":
acted = a.silence(ctx, act.Arguments["silence"], conditions.MaxSilence, byWords(w), why)
default:
+21
View File
@@ -572,6 +572,27 @@ func servingAsker(open *stores, conn *nats.Conn, js jetstream.JetStream, keeper
call: callAction(conn),
// A proposed settings layer is set by this controller itself on the warrant (novox/hq ADR 0277).
setLayer: setLayerIn(open),
// A shared folder waiting to be kept is asked about once, and taken on the warrant (novox/hq ADR 0325 §8).
folders: foldersIn(open),
take: takeFolderIn(open, func(ctx context.Context, act link.AccessActAsk) (link.HandOverAnswer, error) {
ident, err := open.Identity(ctx)
if err != nil {
return link.HandOverAnswer{}, fmt.Errorf("the mesh's signing key cannot be read, so nothing was asked of "+
"%s: %w", act.Node, err)
}
return link.AskTake(ctx, conn, ident, act, link.TakeWithin)
}),
machines: func(ctx context.Context) ([]string, error) {
nodes, err := open.inventory.Nodes(ctx)
if err != nil {
return nil, err
}
names := make([]string, 0, len(nodes))
for _, n := range nodes {
names = append(names, n.Name)
}
return names, nil
},
record: func(ctx context.Context, act link.HandAct) error {
_, err := link.RecordHandAct(ctx, conn, act)
return err
+11 -1
View File
@@ -1109,6 +1109,16 @@ const resolverSeat = "mesh-dns-resolver"
// classOf is a walk's class (novox/hq ADR 0282 decision 1): core when it walks a module on the controller's own
// path or one holding the mesh's resolver, leaf otherwise.
func classOf(walk inventory.Plan, entries []inventory.Entry) string {
modules := make([]string, 0, len(walk.Modules))
for m := range walk.Modules {
modules = append(modules, m)
}
return classOfModules(modules, entries)
}
// classOfModules is the class of modules moved together: core when one is on the controller's own path or holds
// the mesh's resolver, leaf otherwise (ADR 0282 decision 1).
func classOfModules(modules []string, entries []inventory.Entry) string {
resolvers := map[string]bool{}
for _, e := range entries {
for _, c := range e.Manifest.Claims {
@@ -1117,7 +1127,7 @@ func classOf(walk inventory.Plan, entries []inventory.Entry) string {
}
}
}
for m := range walk.Modules {
for _, m := range modules {
if _, own := onTheControllersPath[m]; own || resolvers[m] {
return inventory.ClassCore
}
+24
View File
@@ -2,6 +2,7 @@ package main
import (
"fmt"
"path"
"slices"
"sort"
"strings"
@@ -105,9 +106,32 @@ func changePlanOf(repository, base, head string, r mergeReach, entries []invento
p.Machines = append(p.Machines, *machines[name])
}
p.Summary = summaryOf(p)
p.Class = changeClassOf(p, entries)
return p
}
// classNone is the class of a change that moves no module: nothing for a walk to move (novox/hq ADR 0313 §2).
const classNone = "none"
// changeClassOf is a change plan's class (novox/hq ADR 0313 §2): that of the slowest-budgeted module it moves,
// itself or as a dependent — core when one is on the controller's own path or holds the mesh's resolver, leaf
// otherwise — and none when it moves nothing. A module it adds a directory for counts by its own class: the
// catalogue names a module's directory by the module, so the directory's last element is its name. The graph holds
// no manifest of it yet, so a new module is core by its name alone (on the controller's own path), never by a
// resolver claim its manifest makes.
func changeClassOf(p link.ChangePlan, entries []inventory.Entry) string {
if len(p.Moved) == 0 && len(p.Dependents) == 0 && len(p.New) == 0 {
return classNone
}
modules := append(append([]string{}, p.Moved...), p.Dependents...)
for _, dir := range p.New {
if name := path.Base(strings.Trim(dir, "/")); name != "." && name != "/" && name != "" {
modules = append(modules, name)
}
}
return classOfModules(modules, entries)
}
// grantsStepWord names the grants step (novox/hq issue 490): the bus's user list sent alone to the machine
// holding the bus ahead of a walk's gate. Not the bus step, which replaces the bus itself.
const grantsStepWord = "grants step"
+8
View File
@@ -152,6 +152,14 @@ func moduleCheckFor(paths []string, longestMachine int, out io.Writer) error {
}
}
// **A module that uses several libraries is warned** (novox/hq ADR 0328 §4): an image that drops added groups when
// it changes user reaches only the first. Printed, never counted.
for _, name := range names {
for _, w := range catalogue.LibraryWarnings(shelf[name]) {
fmt.Fprintf(out, "warning: %s\n", w)
}
}
// **A file that asks for a setting and does not say whether it is trusted counts as trusted** (novox/hq issue
// 339): listed and counted, never refused, so an author can opt out a file nothing trusts.
unsaid := 0
+215
View File
@@ -0,0 +1,215 @@
package main
import (
"context"
"fmt"
"sort"
"time"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/lease"
)
// **No send but a person's `push <node>` and a gate's own put-back moves the mesh's own core to a build older
// than the one its machine was last sent** (novox/hq issue 201). Every send made through sendToEach keeps it — a
// plan's, a release's, a healer's, a rotation's, and the bus step too, which is a person's act but composes under
// sendKeeps like the rest. On 2026-10-02 two merges a minute apart made two plans;
// the controller's plan rolled the new controller, which widened its seat's row, and the node-engine's plan then
// sent the control node the controller build before it. The older binary read the row it could not serve and the
// mesh lost its voice for ten minutes — the thing that would have corrected it was the thing that was down. ADR
// 0185 took the outage out of it (a controller behind its row serves what it can); the race is closed here,
// without needing its trigger: whatever made the mesh's record older, the send keeps the build.
//
// **Kept, not refused.** The send composes the controller or the node-engine at the build the machine was last
// sent, as a recorded module is kept (ADR 0245), and carries everything else: no other module, secret, healer or
// walk waits on it, and nothing is retried. The kept move is no move for a gate either: ungatedIn and gatedSend
// leave it out (withoutKept), so a gate judges the other moves of its send. A plan whose own build is the older one finds its gate superseded by
// the build kept there (issue 352) and ends. The mesh's record stays older, and a warning condition says so until
// a send carries the record — a person's `push <node>` accepts the older build — or the record moves forward: a
// newer merge, or `rebuild mesh-controller`.
//
// Older is by the builds' registration order — when each was asked (novox/hq issue 219) — never by the commits'
// dates. So a retry or a rebuild of an old commit, asked anew, counts as newer: it is registered as the module's
// current build by the same order, and the guard does not stop that move. A build that failed its gate is never
// ahead: the machine was sent it and the record was put back before it (gateFailed), and if the put-back's own
// send was refused, the next send completes it rather than keeping the failed build there. A gate's own put-back
// carries its modules back whatever the order (sendScope.puttingBack); the controller's is held to the store's
// schema already (issue 352).
// kindCoreDowngrade is a core module kept at the build its machine was last sent, over an older record.
const kindCoreDowngrade = "core-downgrade-kept"
// coreDowngradeSource is the condition's source: the sends, which raise and clear it.
const coreDowngradeSource = "sends"
// downgradeGuarded is the core components a send may not move backwards: the controller, which reads the
// store and its seat's row, and the node-engine, which reads what the controller declares.
var downgradeGuarded = []string{lease.ComponentController, lease.ComponentEngine}
// askedBuild is what the order needs of one build of a module: when it was asked, and whether it failed its gate.
type askedBuild struct {
asked time.Time
failedGate bool
}
// buildAsked is the newest build of a module from a commit that worked — when it was asked (or recorded, when
// the request time is not known), and whether it failed its gate — and false when the records hold none. A
// variable, so a test reads its own order.
var buildAsked = func(ctx context.Context, inv *inventory.Inventory, module, commit string) (askedBuild, bool, error) {
builds, err := inv.Builds(ctx, module, askedLookBack)
if err != nil {
return askedBuild{}, false, err
}
for _, b := range builds {
if b.Worked() && sameCommit(b.Commit, commit) {
failed, err := inv.GateFailed(ctx, b.ID)
if err != nil {
return askedBuild{}, false, err
}
return askedBuild{asked: b.AskedOrAt(), failedGate: failed}, true, nil
}
}
return askedBuild{}, false, nil
}
// askedLookBack is how many of a module's newest builds are read for when a build was asked.
const askedLookBack = 50
// coreDowngrade is one core module a machine was last sent a build of that is newer than the mesh's record.
type coreDowngrade struct {
module, node, component string
// sent is the build the machine was last sent, record the one the mesh holds.
sent, record string
sentAsked, recordAsked time.Time
}
func (d coreDowngrade) String() string {
return fmt.Sprintf("%s was last sent %s %s, and the mesh holds the older %s (asked %s, before %s): a send there "+
"keeps %s; `push %s` sends the older build, and a newer merge or `rebuild %s` moves the record forward "+
"(novox/hq issue 201)", d.node, d.module, short(d.sent), short(d.record),
d.recordAsked.UTC().Format(time.RFC3339), d.sentAsked.UTC().Format(time.RFC3339), short(d.sent), d.node,
d.module)
}
// coreDowngrades is, for one machine, each guarded core module whose record is older than the build the machine
// was last sent, leaving out what a gate's own put-back puts back and a sent build that failed its gate. A move
// whose either build the records do not hold is not proved older and is left to the gate that judges it. Pure
// but for asked.
func coreDowngrades(node string, sent map[string]string, record map[string]string, scope sendScope,
asked func(module, commit string) (askedBuild, bool, error)) ([]coreDowngrade, error) {
var out []coreDowngrade
for module, was := range sent {
component := coreComponent(module)
now := record[module]
if !guarded(component) || was == "" || now == "" || sameCommit(was, now) {
continue
}
if scope.puttingBack && scope.modules[module] {
continue
}
from, known, err := asked(module, was)
if err != nil {
return nil, fmt.Errorf("when %s's build %s was asked cannot be read: %w", module, short(was), err)
}
if !known || from.failedGate {
continue
}
to, known, err := asked(module, now)
if err != nil {
return nil, fmt.Errorf("when %s's build %s was asked cannot be read: %w", module, short(now), err)
}
if known && to.asked.Before(from.asked) {
out = append(out, coreDowngrade{module: module, node: node, component: component, sent: was, record: now,
sentAsked: from.asked, recordAsked: to.asked})
}
}
sort.Slice(out, func(i, j int) bool { return out[i].module < out[j].module })
return out, nil
}
func guarded(component string) bool {
for _, c := range downgradeGuarded {
if c == component {
return true
}
}
return false
}
// coreKept is, for a send that keeps (sendKeeps), each guarded core module of a machine to keep at the build it
// was last sent, module → commit, each said and raised as a condition.
func coreKept(ctx context.Context, inv *inventory.Inventory, node string, sent map[string]string,
current map[string]inventory.CurrentBuild) (map[string]string, error) {
record := map[string]string{}
for m, c := range current {
record[m] = c.Commit
}
found, err := coreDowngrades(node, sent, record, scopeOf(ctx), func(module, commit string) (askedBuild, bool, error) {
return buildAsked(ctx, inv, module, commit)
})
if err != nil || len(found) == 0 {
return nil, err
}
out := map[string]string{}
for _, d := range found {
out[d.module] = d.sent
sayCoreDowngrade(ctx, d)
}
return out, nil
}
// coreDowngradeObservation is a kept core build as a condition: `core.<component>.<machine>.<kind>`.
func coreDowngradeObservation(component, node, summary string) conditions.Observation {
return conditions.Observation{Scope: conditions.ScopeCore, ID: component + "." + node, Kind: kindCoreDowngrade,
Token: kindCoreDowngrade, Machine: node, Severity: conditions.Warning, Source: coreDowngradeSource,
Summary: summary}
}
func sayCoreDowngrade(ctx context.Context, d coreDowngrade) {
fmt.Printf("KEPT: %s\n", d)
keeper := conditionsFrom
if keeper == nil {
return
}
if _, err := keeper.Observe(ctx, coreDowngradeObservation(d.component, d.node, d.String())); err != nil {
fmt.Printf("the condition for %s kept on %s could not be raised: %v\n", d.module, d.node, err)
}
}
// settleCoreDowngrades clears, after a send to a machine is away, the condition of each guarded core module the
// send carried at the mesh's record — a person's push that accepted the older build, or a send after the record
// moved forward — or did not carry at all. A send that kept the newer build leaves it open.
func settleCoreDowngrades(ctx context.Context, inv *inventory.Inventory, node string, carried map[string]string) {
keeper := conditionsFrom
if keeper == nil || carried == nil {
return
}
var open []string
for _, c := range downgradeGuarded {
if _, found, err := keeper.Get(ctx, coreDowngradeObservation(c, node, "").Key()); err == nil && found {
open = append(open, c)
}
}
if len(open) == 0 {
return
}
current, err := inv.CurrentBuilds(ctx)
if err != nil {
fmt.Printf("whether %s still keeps a core build over the mesh's record cannot be read: %v\n", node, err)
return
}
for module, now := range current {
component := coreComponent(module)
if !guarded(component) {
continue
}
if was, has := carried[module]; has && !sameCommit(was, now.Commit) {
continue
}
key := coreDowngradeObservation(component, node, "").Key()
if _, err := keeper.Clear(ctx, key, node+" was sent the "+component+" the mesh holds"); err != nil {
fmt.Printf("the condition %s could not be cleared: %v\n", key, err)
}
}
}
+54
View File
@@ -0,0 +1,54 @@
package main
import (
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// **A change plan says its class** (novox/hq ADR 0313 §2): the issue tracker holds a fix to the budget of the
// slowest-budgeted module the fix moves, and a fix that moves no module has no fix budget. The plan says core,
// leaf or none, so mesh-delivery can pass it on with the fix's delivery.
func TestAChangePlanSaysTheClassOfWhatItMoves(t *testing.T) {
const repo = "http://forge.internal:20000/novox/mesh-catalog.git"
entry := func(name, path string) inventory.Entry {
e := fromRepo(name, repo, path)
e.Source.BuiltFrom = "old"
e.On = []string{"anchor"}
return e
}
gitea := entry("gitea", "modules/gitea")
resolver := entry("dnsmasq", "modules/dnsmasq")
resolver.Manifest.Claims = []catalogue.Claim{{Name: resolverSeat}}
delivery := entry(catalogue.DeliverySeat, "modules/mesh-delivery")
webApp := entry("photos", "modules/photos")
entries := []inventory.Entry{gitea, resolver, delivery, webApp}
// photos stands on gitea: a change to gitea moves photos as its dependent.
edges := []inventory.Edge{{From: "photos", To: "gitea", Kind: inventory.EdgeStandsOn}}
policy := func(string) (inventory.Upgrade, bool) { return inventory.Upgrade{RollOut: true}, true }
plan := func(paths ...string) link.ChangePlan {
m := link.SourceMoved{Owner: "novox", Repo: "mesh-catalog", Base: "main", Commit: "head", Paths: paths}
return changePlanOf("novox/mesh-catalog", "main", "head", reachOfMerge(m, entries, nil, edges), entries, policy)
}
for _, c := range []struct {
why string
paths []string
want string
}{
{"a leaf module and its dependent", []string{"modules/gitea/index.ts"}, inventory.ClassLeaf},
{"the delivery's owner, on the controller's own path", []string{"modules/mesh-delivery/cmd/x.go"}, inventory.ClassCore},
{"the mesh's resolver", []string{"modules/dnsmasq/module.json"}, inventory.ClassCore},
{"a leaf and a core module together: the slowest-budgeted", []string{"modules/gitea/index.ts", "modules/dnsmasq/module.json"}, inventory.ClassCore},
{"nothing any module's build reads", []string{"README.md"}, "none"},
{"a new leaf module alone", []string{"modules/weather/module.json"}, inventory.ClassLeaf},
{"a new module on the controller's own path, by its own class", []string{"next/mesh-delivery/module.json"}, inventory.ClassCore},
{"a new leaf module beside a core one moved", []string{"modules/weather/module.json", "modules/dnsmasq/module.json"}, inventory.ClassCore},
} {
if got := plan(c.paths...).Class; got != c.want {
t.Errorf("%s: class %q, want %q", c.why, got, c.want)
}
}
}
+58 -2
View File
@@ -154,6 +154,10 @@ type gateFacts struct {
// groupsAdded is, per module, whether the move judged puts an account in a group its previous build did
// not (issue 318 review): the only move whose wait for a new login is excused.
groupsAdded map[string]bool
// accessesChanged is, per module, whether the move judged changes its accesses (novox/hq ADR 0325): an access the
// engine cannot judge this look (unknown), or entries it left while a disk was away, of a module whose move leaves
// its accesses as they were is no fault of the move. A module absent here is taken to change them: judged as before.
accessesChanged map[string]bool
// waits is what the controller holds to check a module's wait for the operator (novox/hq ADR 0283): the
// manifest of each module's build judged, and the secrets given on each machine.
waits operatorWaitFacts
@@ -528,6 +532,7 @@ func judgeMoves(ctx context.Context, open *stores, g *inventory.PlanGate, pairs
return "", err
}
facts.groupsAdded = movesAddingGroups(ctx, open.inventory, g, pairs, shelf)
facts.accessesChanged = movesChangingAccesses(ctx, open.inventory, g, pairs, shelf)
facts.waits = gateWaitFacts(ctx, open.inventory, g, pairs, shelf, facts.health)
facts.sent = g.Sent
facts.commits, facts.sentBuilds = judgedCommits(g, pairs), map[string]map[string]string{}
@@ -783,6 +788,57 @@ func joinSaid(a, b string) string {
return a + "; " + b
}
// movesChangingAccesses is, per module a gate judges, whether its move changes its accesses — one added, dropped,
// or given another use — read from the builds' manifests (novox/hq ADR 0325). A move not known changes them: an
// access it cannot judge then holds the gate to its bound, and never passes unseen.
func movesChangingAccesses(ctx context.Context, inv *inventory.Inventory, g *inventory.PlanGate, pairs []judged,
shelf map[string]catalogue.Manifest) map[string]bool {
out := map[string]bool{}
for _, j := range pairs {
if _, done := out[j.module]; done {
continue
}
from, to := g.From, g.To
for _, c := range g.Carried {
if c.Module == j.module {
from, to = c.From, c.To
break
}
}
target, found, err := inv.ManifestAt(ctx, j.module, to)
if err != nil || !found {
target = shelf[j.module]
}
before, had, err := inv.ManifestAt(ctx, j.module, from)
if from == "" || err != nil || !had {
out[j.module] = true
continue
}
out[j.module] = !sameAccessUse(before, target)
}
return out
}
// sameAccessUse says a move leaves a module's accesses and libraries as they were (the review of e1c2b744, novox/hq
// ADR 0328): the same accesses at the same use, the same library entries, and whether it joins — a module that starts
// joining runs as another user, and changes who uses its libraries.
func sameAccessUse(a, b catalogue.Manifest) bool {
return sameAccesses(a.Accesses, b.Accesses) && catalogue.LibraryUseShape(a) == catalogue.LibraryUseShape(b)
}
// sameAccesses says two manifests declare the same accesses, at the same use, in any order.
func sameAccesses(a, b []catalogue.Access) bool {
key := func(as []catalogue.Access) string {
var out []string
for _, x := range as {
out = append(out, x.ID+"\x00"+x.Path+"\x00"+x.At())
}
sort.Strings(out)
return strings.Join(out, "\x01")
}
return key(a) == key(b)
}
// movesAddingGroups is, per module a gate judges, whether its move puts an account in a group the build it
// moved from did not: read from the builds' manifests. A module whose move is not known adds none.
func movesAddingGroups(ctx context.Context, inv *inventory.Inventory, g *inventory.PlanGate, pairs []judged,
@@ -1039,7 +1095,7 @@ func gateFailed(ctx context.Context, open *stores, p *inventory.Plan, module str
}
return
}
sent, err := sendRollout(withScope(ctx, sendScope{modules: map[string]bool{module: true}}), open, g.Machines)
sent, err := sendRollout(withScope(ctx, sendScope{modules: map[string]bool{module: true}, puttingBack: true}), open, g.Machines)
if err != nil {
notBack(fmt.Sprintf("its registered build is back at %s, and sending it to %s was refused: %v — `push %s` "+
"sends it", short(previous.Commit), strings.Join(g.Machines, ", "), err, g.Machines[0]))
@@ -1132,7 +1188,7 @@ func sendRollbacks(ctx context.Context, open *stores, p *inventory.Plan, b *roll
}
inv := open.inventory
sort.Strings(b.machines)
sent, err := sendRollout(withScope(ctx, sendScope{modules: b.modules}), open, b.machines)
sent, err := sendRollout(withScope(ctx, sendScope{modules: b.modules, puttingBack: true}), open, b.machines)
var back []string
for _, r := range b.pending {
if err != nil {
+164
View File
@@ -0,0 +1,164 @@
package main
import (
"context"
"fmt"
"sort"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// A module that joins keeps the user it ran as until it may join (novox/hq ADR 0328 §8, ADR 0334).
//
// **Held at the build it ran before, composed exactly as then.** A module whose registered build joins — declares
// libraries or uses its identity — on a machine that was last sent another build of it, is held there while
//
// - the machine's engine is older than link.OwnGroupContract: it cannot put the tool runner in the module's own group
// nor keep its own directories' group, so the module's tools would lose their read (the re-review of 96e15f86, 2);
// - or a library it uses is not yet taken (§8).
//
// Held, it is composed at that build — the manifest the build records hold, in place of the registered one, as a
// recorded module is kept (recorded_kept.go) — and with the settings it last ran with (SettingsAsOf the machine's last
// send): its resources, its ports in the filter, its contributions and bindings are what they were, and nothing of the
// new contract is sent for it. The send records that the machine still carries that build. A delivery that brings the
// joining build does not send it while it is held: the module waits for a person, the plan says so and judges its
// other modules, and sends it once nothing holds it (release_plan.go, heldIn). A module the machine never ran has
// nothing to keep, and is composed as registered; a build the records no longer hold cannot be kept.
// heldModule is one module held on a machine: the build it keeps and why, in words with no path.
type heldModule struct {
commit string
// library is the first library it uses that is not taken, empty when none.
library string
why string
}
// holdJoining is every module held on a machine, by module; shelf, when given, has each one's manifest replaced by
// the build it keeps.
func holdJoining(ctx context.Context, inv *inventory.Inventory, node string,
shelf map[string]catalogue.Manifest) (map[string]heldModule, error) {
registered := shelf
if registered == nil {
var err error
if registered, err = inv.Catalogue(ctx); err != nil {
return nil, err
}
}
sent, known, err := inv.SentBuilds(ctx, node)
if err != nil || !known {
return nil, err
}
var joining []string
for m := range sent {
if man, has := registered[m]; has && catalogue.Joins(man) {
joining = append(joining, m)
}
}
if len(joining) == 0 {
return nil, nil
}
sort.Strings(joining)
stated, had, err := inv.HealthOf(ctx, node)
if err != nil {
return nil, err
}
engineJoins := had && stated.Contract >= link.OwnGroupContract
libs, err := inv.Libraries(ctx)
if err != nil {
return nil, err
}
taken, err := librariesTaken(ctx, inv, libs)
if err != nil {
return nil, err
}
current, err := inv.CurrentBuilds(ctx)
if err != nil {
return nil, err
}
out := map[string]heldModule{}
for _, m := range joining {
was := sent[m]
if was == "" || sameCommit(was, current[m].Commit) {
continue // the machine already runs the registered build: nothing before it to keep
}
h := heldModule{commit: was}
for _, u := range registered[m].Libraries {
if !taken[u.Library] {
h.library = u.Library
break
}
}
switch {
case !engineJoins:
h.why = fmt.Sprintf("%s keeps the build it runs on %s until that machine's node-engine keeps a module's own "+
"group (novox/hq ADR 0334)", m, node)
case h.library != "":
h.why = fmt.Sprintf("%s waits for the operator to let the mesh keep the library %s (novox/hq ADR 0328 §8)",
m, h.library)
default:
continue
}
before, found, err := inv.ManifestAt(ctx, m, was)
if err != nil {
return nil, err
}
if !found || (engineJoins && usesLibrary(before, h.library)) {
continue
}
if shelf != nil {
shelf[m] = before
}
out[m] = h
}
return out, nil
}
// heldIn is why a module is held on any of the machines given, or "": what a delivery's first send waits on.
func heldIn(ctx context.Context, inv *inventory.Inventory, module string, machines []string) (string, error) {
for _, node := range machines {
held, err := holdJoining(ctx, inv, node, nil)
if err != nil {
return "", err
}
if h, is := held[module]; is {
return h.why, nil
}
}
return "", nil
}
// heldSettings puts, for each module held on a machine, the settings it last ran with in place of today's: the layers
// as they stood at the machine's last send. Today's stand where no send is recorded.
func heldSettings(ctx context.Context, inv *inventory.Inventory, node string, held map[string]heldModule,
settings catalogue.SettingsBy) error {
if len(held) == 0 {
return nil
}
last, found, err := inv.LastSend(ctx, node)
if err != nil || !found {
return err
}
for m := range held {
layers, err := inv.SettingsAsOf(ctx, node, m, last.SentAt)
if err != nil {
return err
}
if len(layers) == 0 {
delete(settings, m)
continue
}
settings[m] = layers
}
return nil
}
func usesLibrary(m catalogue.Manifest, library string) bool {
for _, u := range m.Libraries {
if library != "" && u.Library == library {
return true
}
}
return false
}
+144
View File
@@ -0,0 +1,144 @@
package main
import (
"encoding/json"
"reflect"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
const heldImage = "x@sha256:2222222222222222222222222222222222222222222222222222222222222222"
// radarrBefore is radarr as it ran before it joined: its own port, its own data, PUID from a setting.
func radarrBefore() catalogue.Manifest {
return catalogue.Manifest{Module: "radarr", Version: "1", Capabilities: []string{"container-runtime"},
Listens: []catalogue.Listening{{Port: 7878, From: catalogue.FromMesh, Name: "web", Why: "its web UI"}},
Resources: []map[string]any{
{"id": "config", "type": "directory", "mode": "0750"},
{"id": "app", "type": "container", "name": "radarr", "image": heldImage, "ports": []any{"7878"},
"env": map[string]any{"PUID": "1000"}, "volumes": []any{"${dir:config}:/config"}},
}}
}
// radarrJoined is the same module's next build, joining the media library.
func radarrJoined() catalogue.Manifest {
m := radarrBefore()
m.Libraries = []catalogue.LibraryUse{{ID: "movies", Library: "media", Path: "movies", Use: catalogue.AccessReadWrite}}
m.Resources = []map[string]any{
{"id": "config", "type": "directory", "mode": "0750"},
{"id": "app", "type": "container", "name": "radarr", "image": heldImage, "ports": []any{"7878"},
"env": map[string]any{"PUID": "${identity:uid}"},
"volumes": []any{"${dir:config}:/config", "${library:movies}:/movies"}},
}
return m
}
// novox/hq ADR 0328 §8 (the review of 7ea6d36e, item 1): a joining module that ran on a machine before is held until
// its libraries are taken — composed EXACTLY as before it joined, at the build the machine runs: its resources, its
// port in the filter, its contributions; only the new build waits. The record says the machine still carries that build.
func TestAHeldModuleIsComposedAsBeforeItJoined(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
controller := catalogue.Manifest{Module: catalogue.LibrariesModule, Version: "1"}
register(t, open, controller)
if err := inv.SetSettings(ctx, "", catalogue.LibrariesModule, map[string]any{catalogue.LibrariesSetting: map[string]any{
"media": map[string]any{"home": "anchor", "root": "/storage/media"}}}); err != nil {
t.Fatal(err)
}
before, joined := radarrBefore(), radarrJoined()
raw, _ := json.Marshal(before)
b1 := inventory.Build{ID: "b1", Repository: "https://forge.invalid/media.git", Ref: "main", Module: "radarr",
Commit: "c1c1c1c1", On: "builder", Manifest: raw}
if err := inv.RecordBuild(ctx, b1); err != nil {
t.Fatal(err)
}
if err := inv.RegisterModule(ctx, before, inventory.Source{Repository: b1.Repository, Ref: "main", BuiltFrom: b1.Commit}); err != nil {
t.Fatal(err)
}
if _, err := assign(ctx, open, "anchor", "radarr"); err != nil {
t.Fatal(err)
}
gens, err := generators(ctx, open)
if err != nil {
t.Fatal(err)
}
// What anchor composes for radarr before it joined.
plan, settings, err := planFor(ctx, open, "anchor")
if err != nil {
t.Fatal(err)
}
was, err := declarationWith(ctx, open, "anchor", plan, settings, gens, Reading)
if err != nil {
t.Fatal(err)
}
wasRules, err := plan.Rules(catalogue.Rendering{Settings: settings})
if err != nil {
t.Fatal(err)
}
record, err := inv.NodeByName(ctx, "anchor")
if err != nil {
t.Fatal(err)
}
if err := inv.RecordSentWith(ctx, record.ID, "d1", map[string]string{"radarr": b1.Commit}, 1, 1,
inventory.Send{Sequence: 1}); err != nil {
t.Fatal(err)
}
// Anchor's engine keeps libraries; the next build joins media, which is not taken.
if _, err := inv.RecordHealth(ctx, inventory.NodeHealth{Node: "anchor", Contract: link.LibraryContract,
SaidAt: time.Now(), HeardAt: time.Now()}); err != nil {
t.Fatal(err)
}
raw, _ = json.Marshal(joined)
b2 := b1
b2.ID, b2.Commit, b2.Manifest = "b2", "c2c2c2c2", raw
if err := inv.RecordBuild(ctx, b2); err != nil {
t.Fatal(err)
}
if err := inv.RegisterModule(ctx, joined, inventory.Source{Repository: b2.Repository, Ref: "main", BuiltFrom: b2.Commit}); err != nil {
t.Fatal(err)
}
plan, settings, err = planFor(ctx, open, "anchor")
if err != nil {
t.Fatal(err)
}
held, err := declarationWith(ctx, open, "anchor", plan, settings, gens, Reading)
if err != nil {
t.Fatal(err)
}
if why, left := held.leftOutWhy["radarr"]; left {
t.Fatalf("radarr was left out while held: %s", why)
}
if !reflect.DeepEqual(radarrResources(was.Resources), radarrResources(held.Resources)) {
t.Fatalf("radarr held is composed otherwise than before it joined:\n %v\n %v", radarrResources(was.Resources),
radarrResources(held.Resources))
}
rules, err := plan.Rules(catalogue.Rendering{Settings: settings})
if err != nil {
t.Fatal(err)
}
if !reflect.DeepEqual(rules, wasRules) || !strings.Contains(asJSON(rules), "7878") {
t.Fatalf("radarr's port is not in the filter as before: %s", asJSON(rules))
}
}
func radarrResources(rs []map[string]any) []map[string]any {
var out []map[string]any
for _, r := range rs {
if strings.HasPrefix(r["id"].(string), "radarr.") {
out = append(out, r)
}
}
return out
}
func asJSON(v any) string {
raw, _ := json.Marshal(v)
return string(raw)
}
+329
View File
@@ -0,0 +1,329 @@
package main
import (
"context"
"encoding/json"
"fmt"
"slices"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/lease"
"github.com/novox/mesh-controller/internal/link"
)
// The re-review of 96e15f86, item 1 (novox/hq ADR 0328 §8): a delivery that brings a joining build of a module whose
// library is not taken does not send it — the module waits for a person, the plan stays open saying so, and the other
// modules of the plan are sent and judged as usual. Once the library is taken, the waiting build is sent and judged,
// with no new merge.
func TestADeliveryWaitsForALibraryAndJudgesTheRest(t *testing.T) {
judgeLeavesAsTheCore(t)
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
keeper, _ := withConditionsInMemory(t)
was := doctorFrom
doctorFrom = &doctor{open: open, keeper: keeper, teller: &conditions.Told{}}
t.Cleanup(func() { doctorFrom = was })
register(t, open, catalogue.Manifest{Module: catalogue.LibrariesModule, Version: "1"})
if err := inv.SetSettings(ctx, "", catalogue.LibrariesModule, map[string]any{catalogue.LibrariesSetting: map[string]any{
"media": map[string]any{"home": "anchor", "root": "/storage/media"}}}); err != nil {
t.Fatal(err)
}
app1 := catalogue.Manifest{Module: "app", Version: "c1"}
app2 := catalogue.Manifest{Module: "app", Version: "c2"}
builds := []struct {
id, module, commit string
m catalogue.Manifest
at time.Time
}{
{"build-app-1", "app", "c1", app1, time.Now().Add(-2 * time.Hour)},
{"build-radarr-1", "radarr", "r1", radarrBefore(), time.Now().Add(-2 * time.Hour)},
{"build-app-2", "app", "c2", app2, time.Now().Add(-time.Minute)},
{"build-radarr-2", "radarr", "r2", radarrJoined(), time.Now().Add(-time.Minute)},
}
for _, b := range builds {
raw, _ := json.Marshal(b.m)
if err := inv.RecordBuild(ctx, inventory.Build{ID: b.id, Module: b.module, Commit: b.commit,
Repository: "novox/mesh-media-catalog", Path: "modules/" + b.module, Ref: "main", Manifest: raw,
Asked: b.at, At: b.at}); err != nil {
t.Fatal(err)
}
}
registerAt := func(m catalogue.Manifest, commit string, asked time.Time) {
if err := inv.RegisterModule(ctx, m, inventory.Source{Repository: "novox/mesh-media-catalog", Seat: "git",
Path: "modules/" + m.Module, Ref: "main", BuiltFrom: commit, Head: commit, Asked: asked}); err != nil {
t.Fatal(err)
}
}
registerAt(app1, "c1", time.Now().Add(-2*time.Hour))
registerAt(radarrBefore(), "r1", time.Now().Add(-2*time.Hour))
if _, err := inv.Assign(ctx, "anchor", "app"); err != nil {
t.Fatal(err)
}
if _, err := inv.Assign(ctx, "anchor", "radarr"); err != nil {
t.Fatal(err)
}
if err := inv.RecordSent(ctx, nodeID(t, open, "anchor"), "d-anchor-0", map[string]string{"app": "c1", "radarr": "r1"}); err != nil {
t.Fatal(err)
}
if _, err := inv.RecordHealth(ctx, inventory.NodeHealth{Node: "anchor", Contract: link.OwnGroupContract,
SaidAt: time.Now(), HeardAt: time.Now()}); err != nil {
t.Fatal(err)
}
registerAt(app2, "c2", time.Now().Add(-time.Minute))
registerAt(radarrJoined(), "r2", time.Now().Add(-time.Minute))
// Every send records what it carries: the registered builds, a held module at the build it runs.
var sent []string
wasSend := sendRollout
sendRollout = func(ctx context.Context, open *stores, names []string) ([]string, error) {
current, err := open.inventory.CurrentBuilds(ctx)
if err != nil {
return nil, err
}
for _, node := range names {
held, err := holdJoining(ctx, open.inventory, node, nil)
if err != nil {
return nil, err
}
carried := map[string]string{"app": current["app"].Commit, "radarr": current["radarr"].Commit}
for m, h := range held {
carried[m] = h.commit
}
sent = append(sent, node+":app="+carried["app"]+",radarr="+carried["radarr"])
digest := fmt.Sprintf("d-%s-%d", node, len(sent))
if err := open.inventory.RecordSent(ctx, nodeID(t, open, node), digest, carried); err != nil {
return nil, err
}
if _, err := open.inventory.RecordDoing(ctx, nodeID(t, open, node), inventory.Doing{Node: node,
Outcome: inventory.OutcomeApplied, Declared: digest, Applied: 1, At: time.Now()}); err != nil {
return nil, err
}
}
return names, nil
}
t.Cleanup(func() { sendRollout = wasSend })
wasGather := gatherGateFacts
gatherGateFacts = func(ctx context.Context, open *stores, component string) (gateFacts, error) {
f := gateFacts{now: time.Now(), reports: map[string]inventory.Reported{}, engines: map[string]string{},
rolledBack: map[string][]lease.Rollback{}, served: map[string]served{}}
reports, err := open.inventory.LastReports(ctx)
for _, r := range reports {
f.reports[r.Node] = r
}
return f, err
}
t.Cleanup(func() { gatherGateFacts = wasGather })
wasSettle, wasEvery := gateSettle, gateEvery
gateSettle, gateEvery = 0, 0
t.Cleanup(func() { gateSettle, gateEvery = wasSettle, wasEvery })
built := time.Now().UTC()
plan := inventory.Plan{ID: "plan-held", Repository: "novox/mesh-media-catalog", Branch: "main", Commit: "c2",
Created: built, State: inventory.PlanBuilding, Tiers: [][]string{{"app", "radarr"}},
Modules: map[string]*inventory.PlanModule{
"app": {State: "built", BuiltAt: &built, Commit: "c2", Build: "build-app-2"},
"radarr": {State: "built", BuiltAt: &built, Commit: "r2", Build: "build-radarr-2"}}}
if err := inv.SavePlan(ctx, &plan); err != nil {
t.Fatal(err)
}
for i := 0; i < 4; i++ {
advancePlans(ctx, open)
}
p, err := inv.PlanByID(ctx, "plan-held")
if err != nil {
t.Fatal(err)
}
if p.State == inventory.PlanSuperseded || p.State == inventory.PlanDone || p.State == inventory.PlanFailed {
t.Fatalf("the plan is %s (%s); want it open, waiting for the library", p.State, p.Note)
}
if g := p.Modules["app"].Gate; g == nil || g.Verdict != inventory.GatePassed {
t.Fatalf("app was not judged and passed beside the held module: %+v (%s)", p.Modules["app"], p.Note)
}
if p.Modules["radarr"].FirstAt != nil || !strings.Contains(p.Note, "library media") {
t.Fatalf("radarr was sent, or the plan does not say it waits for the library: %+v, %q", p.Modules["radarr"], p.Note)
}
if slices.ContainsFunc(sent, func(s string) bool { return strings.Contains(s, "radarr=r2") }) {
t.Fatalf("radarr's joining build was sent while held: %v", sent)
}
// The library is taken: the waiting build is sent, with no new merge.
if err := inv.RecordTakeStarted(ctx, inventory.TakenLibrary{Library: "media", Home: "anchor", Root: "/storage/media",
By: "the operator", Ask: "c01", Warrant: "c01"}); err != nil {
t.Fatal(err)
}
if err := inv.TakeDone(ctx, "media", "c01"); err != nil {
t.Fatal(err)
}
for i := 0; i < 4; i++ {
advancePlans(ctx, open)
}
if !slices.ContainsFunc(sent, func(s string) bool { return strings.Contains(s, "radarr=r2") }) {
t.Fatalf("radarr's joining build was not sent once its library was taken: %v", sent)
}
}
// The re-review of 96e15f86, item 2: a module joins only on an engine stating link.OwnGroupContract (7). On an engine
// at 6 a module that ran there before is held at the build it ran, whatever its library — so its tools never lose
// their read of its directories.
func TestAJoiningModuleIsHeldOnAnEngineOlderThanOwnGroups(t *testing.T) {
open, record := heldMesh(t)
ctx := t.Context()
inv := open.inventory
if err := inv.RecordTakeStarted(ctx, inventory.TakenLibrary{Library: "media", Home: "anchor", Root: "/storage/media",
By: "the operator", Ask: "c01", Warrant: "c01"}); err != nil {
t.Fatal(err)
}
if err := inv.TakeDone(ctx, "media", "c01"); err != nil {
t.Fatal(err)
}
_ = record
if _, err := inv.RecordHealth(ctx, inventory.NodeHealth{Node: "anchor", Contract: link.LibraryContract,
SaidAt: time.Now(), HeardAt: time.Now()}); err != nil {
t.Fatal(err)
}
held, err := holdJoining(ctx, inv, "anchor", nil)
if err != nil {
t.Fatal(err)
}
if held["radarr"].commit != "c1c1c1c1" {
t.Fatalf("on an engine at contract 6 radarr is not held at the build it ran: %+v", held)
}
if _, err := inv.RecordHealth(ctx, inventory.NodeHealth{Node: "anchor", Contract: link.OwnGroupContract,
SaidAt: time.Now().Add(time.Second), HeardAt: time.Now()}); err != nil {
t.Fatal(err)
}
if held, err = holdJoining(ctx, inv, "anchor", nil); err != nil || len(held) > 0 {
t.Fatalf("on an engine at contract 7 with its library taken radarr is still held: %+v (%v)", held, err)
}
}
// heldMesh is anchor running radarr's build from before it joined, the joining build registered, media defined.
func heldMesh(t *testing.T) (*stores, inventory.Node) {
t.Helper()
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
register(t, open, catalogue.Manifest{Module: catalogue.LibrariesModule, Version: "1"})
if err := inv.SetSettings(ctx, "", catalogue.LibrariesModule, map[string]any{catalogue.LibrariesSetting: map[string]any{
"media": map[string]any{"home": "anchor", "root": "/storage/media"}}}); err != nil {
t.Fatal(err)
}
for _, b := range []struct {
id, commit string
m catalogue.Manifest
}{{"b1", "c1c1c1c1", radarrBefore()}, {"b2", "c2c2c2c2", radarrJoined()}} {
raw, _ := json.Marshal(b.m)
if err := inv.RecordBuild(ctx, inventory.Build{ID: b.id, Repository: "https://forge.invalid/media.git", Ref: "main",
Module: "radarr", Commit: b.commit, On: "builder", Manifest: raw}); err != nil {
t.Fatal(err)
}
}
if err := inv.RegisterModule(ctx, radarrBefore(), inventory.Source{Repository: "https://forge.invalid/media.git",
Ref: "main", BuiltFrom: "c1c1c1c1"}); err != nil {
t.Fatal(err)
}
if _, err := assign(ctx, open, "anchor", "radarr"); err != nil {
t.Fatal(err)
}
record, err := inv.NodeByName(ctx, "anchor")
if err != nil {
t.Fatal(err)
}
if err := inv.RecordSentWith(ctx, record.ID, "d1", map[string]string{"radarr": "c1c1c1c1"}, 1, 1,
inventory.Send{Sequence: 1}); err != nil {
t.Fatal(err)
}
if err := inv.RegisterModule(ctx, radarrJoined(), inventory.Source{Repository: "https://forge.invalid/media.git",
Ref: "main", BuiltFrom: "c2c2c2c2"}); err != nil {
t.Fatal(err)
}
return open, record
}
// The re-review of 96e15f86, item 3: a held module is composed with the settings it last ran with — a layer changed
// for the joining build (its puid removed) does not reach the build it still runs.
func TestAHeldModuleKeepsTheSettingsItRanWith(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
register(t, open, catalogue.Manifest{Module: catalogue.LibrariesModule, Version: "1"})
if err := inv.SetSettings(ctx, "", catalogue.LibrariesModule, map[string]any{catalogue.LibrariesSetting: map[string]any{
"media": map[string]any{"home": "anchor", "root": "/storage/media"}}}); err != nil {
t.Fatal(err)
}
before := radarrBefore()
before.Resources = append(before.Resources, map[string]any{"id": "identity", "type": "file",
"path": "${dir:config}/identity.env", "mode": "0644", "content": "PUID=${setting:puid}\n"})
joined := radarrJoined()
for _, b := range []struct {
id, commit string
m catalogue.Manifest
}{{"b1", "c1c1c1c1", before}, {"b2", "c2c2c2c2", joined}} {
raw, _ := json.Marshal(b.m)
if err := inv.RecordBuild(ctx, inventory.Build{ID: b.id, Repository: "https://forge.invalid/media.git", Ref: "main",
Module: "radarr", Commit: b.commit, On: "builder", Manifest: raw}); err != nil {
t.Fatal(err)
}
}
if err := inv.RegisterModule(ctx, before, inventory.Source{Repository: "https://forge.invalid/media.git",
Ref: "main", BuiltFrom: "c1c1c1c1"}); err != nil {
t.Fatal(err)
}
if err := inv.SetSettings(ctx, "anchor", "radarr", map[string]any{"puid": "1000"}); err != nil {
t.Fatal(err)
}
if _, err := assign(ctx, open, "anchor", "radarr"); err != nil {
t.Fatal(err)
}
record, err := inv.NodeByName(ctx, "anchor")
if err != nil {
t.Fatal(err)
}
if err := inv.RecordSentWith(ctx, record.ID, "d1", map[string]string{"radarr": "c1c1c1c1"}, 1, 1,
inventory.Send{Sequence: 1}); err != nil {
t.Fatal(err)
}
time.Sleep(20 * time.Millisecond)
if err := inv.RegisterModule(ctx, joined, inventory.Source{Repository: "https://forge.invalid/media.git",
Ref: "main", BuiltFrom: "c2c2c2c2"}); err != nil {
t.Fatal(err)
}
if err := inv.ClearSettings(ctx, "anchor", "radarr"); err != nil { // the joining build reads no puid
t.Fatal(err)
}
if _, err := inv.RecordHealth(ctx, inventory.NodeHealth{Node: "anchor", Contract: link.OwnGroupContract,
SaidAt: time.Now(), HeardAt: time.Now()}); err != nil {
t.Fatal(err)
}
gens, err := generators(ctx, open)
if err != nil {
t.Fatal(err)
}
plan, settings, err := planFor(ctx, open, "anchor")
if err != nil {
t.Fatal(err)
}
held, err := declarationWith(ctx, open, "anchor", plan, settings, gens, Reading)
if err != nil {
t.Fatal(err)
}
if why, left := held.leftOutWhy["radarr"]; left {
t.Fatalf("radarr held was left out: %s", why)
}
for _, r := range held.Resources {
if r["id"] == "radarr.identity" {
if r["content"] != "PUID=1000\n" {
t.Fatalf("radarr held reads %q; want the puid it ran with", r["content"])
}
return
}
}
t.Fatal("radarr held has no identity file")
}
+259
View File
@@ -0,0 +1,259 @@
package main
import (
"context"
"encoding/json"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/lease"
)
// novox/hq issue 201: on 2026-10-02 a plan sent the control node a controller older than the one running there,
// whose seat row the newer one had already widened; the older binary refused the row and the mesh went quiet for
// ten minutes. No send but a person's push and a gate's own put-back moves the controller or the node-engine back
// behind the build its machine was last sent: the send keeps it there, and carries the rest.
// askedIn answers a build from a fixed table: module@commit → when it was asked; failed names those that failed
// their gate.
func askedIn(table map[string]time.Time, failed ...string) func(module, commit string) (askedBuild, bool, error) {
return func(module, commit string) (askedBuild, bool, error) {
at, known := table[module+"@"+commit]
b := askedBuild{asked: at}
for _, f := range failed {
if f == module+"@"+commit {
b.failedGate = true
}
}
return b, known, nil
}
}
func TestACoreModuleIsKeptOverAnOlderRecord(t *testing.T) {
old, now := t0, t0.Add(time.Minute)
table := map[string]time.Time{
"mesh-controller@c1": old, "mesh-controller@c2": now,
"mesh-host@h1": old, "mesh-host@h2": now,
"app@a1": old, "app@a2": now,
}
sent := map[string]string{"mesh-controller": "c2", "mesh-host": "h2", "app": "a2"}
record := map[string]string{"mesh-controller": "c1", "mesh-host": "h1", "app": "a1"}
found, err := coreDowngrades("anchor", sent, record, sendScope{judged: map[string]bool{"anchor": true}}, askedIn(table))
if err != nil {
t.Fatal(err)
}
if len(found) != 2 || found[0].module != "mesh-controller" || found[0].component != lease.ComponentController ||
found[0].sent != "c2" || found[1].module != "mesh-host" || found[1].component != lease.ComponentEngine {
t.Fatalf("the controller and the node-engine over an older record are kept, and nothing else: %+v", found)
}
if said := found[0].String(); !strings.Contains(said, "anchor was last sent mesh-controller c2") ||
!strings.Contains(said, "`push anchor`") || !strings.Contains(said, "`rebuild mesh-controller`") {
t.Fatalf("it says the machine, the build kept and the two ways out: %s", said)
}
// The record forward, the same build, or a build the records do not hold: nothing kept.
for _, c := range [][2]string{{"c1", "c2"}, {"c2", "c2"}, {"c2", "c9"}, {"c9", "c1"}} {
found, err := coreDowngrades("anchor", map[string]string{"mesh-controller": c[0]},
map[string]string{"mesh-controller": c[1]}, sendScope{}, askedIn(table))
if err != nil || len(found) != 0 {
t.Fatalf("sent %s, record %s: no downgrade proved, nothing kept: %+v %v", c[0], c[1], found, err)
}
}
// A sent build that failed its gate is never ahead: the record put back before it is sent.
if found, _ := coreDowngrades("anchor", sent, record, sendScope{}, askedIn(table, "mesh-controller@c2")); len(found) != 1 ||
found[0].module != "mesh-host" {
t.Fatalf("a controller that failed its gate was kept over the build put back: %+v", found)
}
// A gate's own put-back carries its own modules back, and nothing else.
putBack := sendScope{modules: map[string]bool{"mesh-controller": true}, puttingBack: true}
if found, _ := coreDowngrades("anchor", sent, record, putBack, askedIn(table)); len(found) != 1 || found[0].module != "mesh-host" {
t.Fatalf("a put-back of the controller carries it back, and not the node-engine: %+v", found)
}
if found, _ := coreDowngrades("anchor", sent, record, sendScope{modules: map[string]bool{"mesh-controller": true}},
askedIn(table)); len(found) != 2 {
t.Fatalf("a send for the controller that is no put-back carries it back: %+v", found)
}
}
// core201 is a mesh whose control node, anchor, was sent the newer controller c2 while the mesh's record of the
// controller is the build before it, c1, which passed its gate when it rolled out.
func core201(t *testing.T) *stores {
t.Helper()
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
old, now := time.Now().Add(-2*time.Hour), time.Now().Add(-time.Minute)
record := func(commit string, asked time.Time) inventory.Build {
manifest, _ := json.Marshal(catalogue.Manifest{Module: "mesh-controller", Version: "1"})
b := inventory.Build{ID: "build-mesh-controller-" + commit, Module: "mesh-controller", Commit: commit,
Repository: "novox/mesh-controller", Manifest: manifest, Asked: asked, At: asked,
Made: []inventory.Artifact{{Name: "x", Kind: "bundle", Reference: "sha256:" + commit + "0000000000000000"}}}
if err := inv.RecordBuild(ctx, b); err != nil {
t.Fatal(err)
}
return b
}
before := record("c1aaaaaaaaaa", old)
if err := inv.RecordGate(ctx, inventory.GateVerdict{Build: before.ID, Module: "mesh-controller", Commit: before.Commit,
Machines: []string{"anchor"}, Verdict: inventory.GatePassed}); err != nil {
t.Fatal(err)
}
record("c2bbbbbbbbbb", now)
if err := inv.RegisterModule(ctx, catalogue.Manifest{Module: "mesh-controller", Version: "1"}, inventory.Source{
Repository: "novox/mesh-controller", BuiltFrom: "c2bbbbbbbbbb", Head: "c2bbbbbbbbbb", Asked: now}); err != nil {
t.Fatal(err)
}
if _, err := inv.Assign(ctx, "anchor", "mesh-controller"); err != nil {
t.Fatal(err)
}
if err := inv.RecordSent(ctx, nodeID(t, open, "anchor"), "d-anchor", map[string]string{"mesh-controller": "c2bbbbbbbbbb"}); err != nil {
t.Fatal(err)
}
if err := inv.RestoreModule(ctx, before); err != nil {
t.Fatal(err)
}
wasMoves := machineMoves
machineMoves = func(ctx context.Context, open *stores, f moveFacts, node string, all bool) ([]inventory.CarriedMove, error) {
modules, err := open.inventory.Assigned(ctx, node)
if err != nil {
return nil, err
}
sent, known, err := open.inventory.SentBuilds(ctx, node)
if err != nil {
return nil, err
}
return f.moves(node, modules, sent, known, all), nil
}
t.Cleanup(func() { machineMoves = wasMoves })
return open
}
// TestReplay201 replays the race on a store: the control node runs the newer controller and the mesh's record is
// the build before it. A plan's send there — the node-engine's plan of 2026-10-02 — goes, keeps the newer
// controller, and says so as a condition; the gate's own put-back carries it back; a send that carries the record,
// as a person's push does, clears the condition.
func TestReplay201(t *testing.T) {
open := core201(t)
ctx := t.Context()
k, _ := withConditionsInMemory(t)
judged := withScope(ctx, sendScope{judged: map[string]bool{"anchor": true}})
if kept, err := ungatedIn(judged, open, []string{"anchor"}, ""); err != nil || len(kept) != 1 {
t.Fatalf("a plan's send to the machine is made, the rest of it with it: %v %v", kept, err)
}
keeps, err := recordedKept(keepingRecorded(judged), open, "anchor")
if err != nil || keeps["mesh-controller"] != "c2bbbbbbbbbb" {
t.Fatalf("the send keeps the newer controller there: %v %v", keeps, err)
}
shelf := map[string]catalogue.Manifest{"mesh-controller": {Module: "mesh-controller", Version: "old"}}
if _, err := keepRecorded(keepingRecorded(judged), open, "anchor", shelf); err != nil || shelf["mesh-controller"].Version != "1" {
t.Fatalf("the send composes the kept build's manifest: %+v %v", shelf, err)
}
key := coreDowngradeObservation(lease.ComponentController, "anchor", "").Key()
if c, found, err := k.Get(ctx, key); err != nil || !found || !strings.Contains(c.Summary, "c2bbbbbb") {
t.Fatalf("the condition %s is raised, saying the build kept: %+v %v %v", key, c, found, err)
}
putBack := keepingRecorded(withScope(ctx, sendScope{modules: map[string]bool{"mesh-controller": true}, puttingBack: true}))
if keeps, err := recordedKept(putBack, open, "anchor"); err != nil || len(keeps) != 0 {
t.Fatalf("the gate's own put-back of the controller carries it back: %v %v", keeps, err)
}
settleCoreDowngrades(ctx, open.inventory, "anchor", map[string]string{"mesh-controller": "c2bbbbbbbbbb"})
if _, found, _ := k.Get(ctx, key); !found {
t.Fatal("a send that kept the newer controller cleared the condition")
}
settleCoreDowngrades(ctx, open.inventory, "anchor", map[string]string{"mesh-controller": "c1aaaaaaaaaa"})
if _, found, _ := k.Get(ctx, key); found {
t.Fatal("a send carrying the mesh's record — a person's push — leaves the condition open")
}
}
// The review of #273: a gate failed the newer controller, put the record back, and the put-back's own send was
// refused. The next automatic send — a healer's, which judges nothing — completes the put-back: a build that
// failed its gate is never kept.
func TestARollbackWhoseSendWasRefusedIsCompletedByTheNextSend(t *testing.T) {
open := core201(t)
ctx := t.Context()
k, _ := withConditionsInMemory(t)
if err := open.inventory.RecordGate(ctx, inventory.GateVerdict{Build: "build-mesh-controller-c2bbbbbbbbbb",
Module: "mesh-controller", Commit: "c2bbbbbbbbbb", Previous: "c1aaaaaaaaaa", Machines: []string{"anchor"},
Verdict: inventory.GateFailed, Rollback: inventory.NotRolledBack}); err != nil {
t.Fatal(err)
}
healer := withScope(ctx, sendScope{})
if kept, err := ungatedIn(healer, open, []string{"anchor"}, ""); err != nil || len(kept) != 1 {
t.Fatalf("the healer's send is made: %v %v", kept, err)
}
if keeps, err := recordedKept(keepingRecorded(healer), open, "anchor"); err != nil || len(keeps) != 0 {
t.Fatalf("the failed controller was kept over the build put back: %v %v", keeps, err)
}
if open, _ := k.Open(ctx); len(open) != 0 {
t.Fatalf("a completed put-back raised a condition: %+v", open)
}
}
// Review 91 of #273: a gated send to a machine whose controller is kept over an older record leaves the kept
// controller out of what its gate judges. Judged as moving back to the record, the gate would read the send's own
// kept controller as another build sent there after it (issue 352) and be superseded, judging nothing.
func TestAGateJudgesTheOtherMovesBesideAKeptCore(t *testing.T) {
open := core201(t)
ctx := t.Context()
withConditionsInMemory(t)
inv := open.inventory
old, now := time.Now().Add(-2*time.Hour), time.Now().Add(-time.Minute)
for _, b := range []struct {
commit string
at time.Time
}{{"a1aaaaaaaaaa", old}, {"a2bbbbbbbbbb", now}} {
manifest, _ := json.Marshal(catalogue.Manifest{Module: "app", Version: "1"})
if err := inv.RecordBuild(ctx, inventory.Build{ID: "build-app-" + b.commit, Module: "app", Commit: b.commit,
Repository: "novox/mesh-catalog", Path: "modules/app", Manifest: manifest, Asked: b.at, At: b.at,
Made: []inventory.Artifact{{Name: "x", Kind: "bundle", Reference: "sha256:app-" + b.commit}}}); err != nil {
t.Fatal(err)
}
if err := inv.RegisterModule(ctx, catalogue.Manifest{Module: "app", Version: "1"}, inventory.Source{
Repository: "novox/mesh-catalog", Seat: "git", Path: "modules/app", BuiltFrom: b.commit, Head: b.commit,
Asked: b.at}); err != nil {
t.Fatal(err)
}
}
if _, err := inv.Assign(ctx, "anchor", "app"); err != nil {
t.Fatal(err)
}
if err := inv.RecordSent(ctx, nodeID(t, open, "anchor"), "d-anchor-2",
map[string]string{"mesh-controller": "c2bbbbbbbbbb", "app": "a1aaaaaaaaaa"}); err != nil {
t.Fatal(err)
}
// The send as it is made: the controller kept at c2, the app moved to the record.
wasSend := sendRollout
sendRollout = func(ctx context.Context, open *stores, names []string) ([]string, error) {
return names, open.inventory.RecordSent(ctx, nodeID(t, open, "anchor"), "d-anchor-3",
map[string]string{"mesh-controller": "c2bbbbbbbbbb", "app": "a2bbbbbbbbbb"})
}
t.Cleanup(func() { sendRollout = wasSend })
moves, sent, _, err := gatedSend(ctx, open, "anchor", nil)
if err != nil || len(sent) != 1 {
t.Fatalf("the gated send is made: %v %v", sent, err)
}
if len(moves) != 1 || moves[0].Module != "app" || moves[0].To != "a2bbbbbbbbbb" {
t.Fatalf("the gate judges the app's move and not the kept controller: %+v", moves)
}
builds, _, err := inv.SentBuilds(ctx, "anchor")
if err != nil {
t.Fatal(err)
}
g := &inventory.PlanGate{Carried: moves}
f := gateFacts{commits: judgedCommits(g, nil), sentBuilds: map[string]map[string]string{"anchor": builds}}
for _, m := range []string{"app", "mesh-controller"} {
if why, superseded := f.supersededOn(m, "anchor"); superseded {
t.Fatalf("the gate is superseded by its own send: %s", why)
}
}
}
+401
View File
@@ -0,0 +1,401 @@
package main
// Existing data joins a library once, on the operator's word (novox/hq ADR 0328 §7, §8).
//
// **What waits.** A node-engine that keeps libraries states a library waiting, with an access wait of reason `keep`,
// while its root on its home holds data not yet in its group; a module that uses it waits the same way. The
// controller excuses such a wait only for a defined library that reaches the machine and is not yet taken on its home,
// on the filesystem the wait names (checkLibraryWait), and the module is said as its needs-operator condition, in words
// with no path.
//
// **One ask per library.** The serving controller's asker asks the operator once for each such library — however many
// modules and machines wait on it — at the level approve (ADR 0259), in the clause's words: "Let the mesh keep library
// <name> at <root> on <machine>: …". It is kept in the controller's asks beside a proposal (asked.Take), never a
// condition's. While it is open no second is asked; one that expired unanswered is asked again while the library
// waits; one about a library that no longer waits is taken back; one whose library moved to another filesystem is
// asked again for the one it is on now. None is asked for a library already taken.
//
// **On the warrant, and only on it**, the controller asks the library's home engine to take it (link.AskTake), naming
// "library:<name>" and the warrant — after checking the library still waits on the filesystem the operator was asked
// about. On the engine's word that it started, the library is recorded taken (kept_library), and from the next send
// the library says `taken`.
import (
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"sort"
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/asks"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
"github.com/novox/mesh-controller/internal/outward"
)
// libraryTake is one library the operator is asked to let the mesh take, as the controller's ask keeps it
// (asked.Take). Every field the ask is composed from is here, so the act on the warrant is composed from the record.
type libraryTake struct {
Library string `json:"library"`
// Node is the library's home, where it is taken; Root its root there.
Node string `json:"node"`
Root string `json:"root"`
// Modules are the modules waiting on it, sorted.
Modules []string `json:"modules"`
// Placed is the filesystem the root is on, as the home's engine names it without a path: the warrant is for it.
Placed string `json:"placed,omitempty"`
At time.Time `json:"at"`
}
// takeVerb is the verb a take's answer binds: the controller's own, performed by itself on the warrant.
const takeVerb = askerName + ".take"
// key is the library on its home, at its root, on its filesystem: what one ask is kept for. A library that moved is
// asked again.
func (t libraryTake) key() string {
return t.Library + "\x00" + t.Node + "\x00" + t.Root + "\x00" + t.Placed
}
// digest binds everything the take does and was said for.
func (t libraryTake) digest() string {
raw, _ := json.Marshal(struct {
Library, Node, Root, Placed string
Modules []string
}{t.Library, t.Node, t.Root, t.Placed, t.Modules})
sum := sha256.Sum256(raw)
return "sha256:" + hex.EncodeToString(sum[:])
}
// actions is the take's one answer, at the level approve, binding the exact act through boundAct.
func (t libraryTake) actions(id string) []conditions.Action {
return []conditions.Action{{Label: "Let the mesh keep it", Verb: takeVerb, Machine: t.Node,
Level: conditions.LevelApprove, Arguments: map[string]string{"take": id, "library": t.Library,
"node": t.Node, "access": libraryAccess + t.Library, "digest": t.digest()}}}
}
// takeWords is the clause's ask (ADR 0328 §7), with the root as shown.
func takeWords(library, shown, machine string) string {
return fmt.Sprintf("Let the mesh keep library %s at %s on %s: its group becomes %s, the group may read and write, "+
"folders keep the group; owners and everyone else's rights stay as they are", library, shown, machine,
catalogue.MeshName(library))
}
// ask is the take's ask, composed from it alone. The root is said whole where the router's content rule lets it leave
// the mesh, and otherwise as "its root".
func (t libraryTake) ask(id string, machines []string) (asks.Ask, map[string]int) {
shown := t.Root
if _, ok := outward.Check(shown, machines...); !ok {
shown = "its root"
}
headline := fmt.Sprintf("Let the mesh keep library %s?", t.Library)
if len([]rune(headline)) > asks.HeadlineLength {
headline = "Let the mesh keep a library?"
}
explanation := takeWords(t.Library, shown, t.Node) + "."
if len(t.Modules) > 0 {
explanation += " Waiting on it: " + strings.Join(t.Modules, ", ") + "."
}
if _, ok := outward.Check(explanation, machines...); !ok {
explanation = sayable(explanation, machines)
}
q := asks.Ask{ID: id, Headline: headline, Explanation: explanation,
Who: asks.Operator, Expires: t.At.Add(askApproveFor),
OnExpiry: "nothing is changed, and you are asked again while the library waits", About: "take." + t.Library}
options := map[string]int{}
for i, act := range t.actions(id) {
binds, _ := asks.ActDigest(boundAct(act))
oid := optionID(act.Label)
options[oid] = i
q.Options = append(q.Options, asks.Option{ID: oid, Label: act.Label, Level: asks.Level(act.Level), Binds: binds,
Does: "the library's data joins its group, and is kept from then on without asking"})
}
return q, options
}
// takeOnWarrant performs a take the operator approved: the engine is asked, and the library recorded taken on its
// word. It answers what the engine said.
type takeOnWarrant func(ctx context.Context, t libraryTake, askID, by string) (string, error)
// askFolders asks the operator about every library waiting to be taken that is not asked about yet, and takes back an
// open ask about one that no longer waits. Called at the end of each reconcile, after the conditions' asks, within the
// asks the router holds open at once.
func (a *asker) askFolders(ctx context.Context) error {
if a.folders == nil {
return nil
}
wanted, err := a.folders(ctx)
if err != nil {
return err
}
now := a.now()
all, err := a.store.All(ctx)
if err != nil {
return err
}
open, openNow := map[string]asked{}, 0
for _, r := range all {
if r.State != askOpen || !now.Before(r.Ask.Expires) {
continue
}
openNow++
if r.Take != nil {
open[r.Take.key()] = r
}
}
want := map[string]bool{}
for _, t := range wanted {
want[t.key()] = true
}
for key, r := range open {
if !want[key] {
if err := a.cancel(ctx, r, "the library no longer waits to be taken there"); err != nil {
return err
}
openNow--
}
}
var machines []string
if a.machines != nil {
if machines, err = a.machines(ctx); err != nil {
return err
}
}
for _, t := range wanted {
if _, asking := open[t.key()]; asking {
continue // one ask per library (§7)
}
if openNow >= askMostOpen {
return nil // asked when one of the open ones ends
}
t.At = now
id := newAskID()
q, options := t.ask(id, machines)
if err := q.Check(now); err != nil {
a.logf("the operator could not be asked to let the mesh keep the library %s: %v", t.Library, err)
continue
}
body, err := json.Marshal(q)
if err != nil {
return err
}
take := t
if err := a.store.Create(ctx, asked{ID: id, Condition: q.About, Ask: q, Actions: t.actions(id),
Options: options, State: askOpen, Opened: now, Take: &take}); err != nil {
return fmt.Errorf("the take could not be kept in the controller's asks, so it was not asked: %w", err)
}
if err := a.publish(ctx, asks.AskSubject(askerName), body, "ask."+id); err != nil {
_, _ = a.store.Change(ctx, id, func(x *asked) bool {
if x.State != askOpen || x.Acted != "" {
return false
}
x.State, x.Ended, x.Acted = askUnsent, a.now(), "nothing: it could not be published: "+err.Error()
return true
})
a.logf("the operator could not be asked to let the mesh keep the library %s: %v", t.Library, err)
continue
}
openNow++
a.logf("asked the operator to let the mesh keep the library %s on %s (%s)", t.Library, t.Node, id)
}
return nil
}
// decideTake is what the asker does with a warrant for a take (asker.Decided): the act only when it is the one the
// option bound — composed again from the record — and the library still waits on the filesystem the operator was
// asked about; then the engine is asked.
func (a *asker) decideTake(ctx context.Context, r asked, act conditions.Action, w asks.Warrant) (string, error) {
t := *r.Take
recomposed := t.actions(r.ID)
option, offered := r.Ask.Option(w.Option)
if !offered || len(recomposed) != 1 {
return "", fmt.Errorf("the take %s offers no answer %q: nothing is taken", r.ID, w.Option)
}
if err := option.Performs(boundAct(recomposed[0])); err != nil {
return "", fmt.Errorf("the take kept for %s is not the one the operator was shown: %v", r.ID, err)
}
if err := option.Performs(boundAct(act)); err != nil {
return "", fmt.Errorf("the act chosen for %s is not the one the operator was shown: %v", r.ID, err)
}
if a.take == nil {
return "", errors.New("this controller cannot take a library on a warrant")
}
// **The library must still wait where the operator was asked about it** (the review of e1c2b744): a take is about
// no condition, so nothing else rechecks it. A library that moved to another home or filesystem, or no longer
// waits, is not taken: the ask is spent, and one for where it waits now is asked.
if a.folders != nil {
waiting, err := a.folders(ctx)
if err != nil {
return "", fmt.Errorf("whether the library %s still waits could not be read, so nothing is taken: %w",
t.Library, err)
}
still := false
for _, now := range waiting {
still = still || now.key() == t.key()
}
if !still {
return "nothing: the library no longer waits to be taken where the operator was asked about it (it " +
"moved, or was taken meanwhile); asked again where it waits now", nil
}
}
by := fmt.Sprintf("approved by %s via %s at %s (ask %s)", byWords(w), viaWords(w),
w.At.Local().Format("2006-01-02 15:04"), r.ID)
return a.take(ctx, t, r.ID, by)
}
// takeFolderIn is takeOnWarrant on this controller's stores and bus: the home's engine is asked with the mesh's key,
// naming the library and the ask the warrant answered; on its word that it started, the library is recorded taken. A
// refusal records nothing.
func takeFolderIn(open *stores, ask func(ctx context.Context, act link.AccessActAsk) (link.HandOverAnswer, error)) takeOnWarrant {
return func(ctx context.Context, t libraryTake, askID, by string) (string, error) {
// Who asked, as one short line the engine takes (at most link.MaxHandOverBy): the ask names the rest.
short := by
if len(short) > 180 {
short = "the operator, on the warrant for ask " + askID
}
answer, err := ask(ctx, link.AccessActAsk{Node: t.Node, Path: t.Root, By: short,
Access: libraryAccess + t.Library, Warrant: askID})
if err != nil {
return "", err
}
if answer.Refused != "" {
return "", fmt.Errorf("%s refused to take the library %s: %s", t.Node, t.Library, answer.Refused)
}
// Started, not taken: the library is taken when the engine states the act done (stateHealth, TakeDone).
if err := open.inventory.RecordTakeStarted(ctx, inventory.TakenLibrary{Library: t.Library, Home: t.Node,
Root: t.Root, Placed: t.Placed, By: by, Ask: askID, Warrant: askID, Said: answer.Said}); err != nil {
return "", fmt.Errorf("%s started the take of %s (%s), and it could NOT be recorded: %w", t.Node, t.Library,
answer.Said, err)
}
return fmt.Sprintf("%s started taking the library %s: %s; it is taken when the engine says the act is done",
t.Node, t.Library, answer.Said), nil
}
}
// waitingFolder is one library wait excused, as foldersWaiting gathers them.
type waitingFolder struct {
module, machine string
// placed is the filesystem the library's root is on, as its home's engine names it without a path; empty from
// another machine.
placed string
}
// foldersWaiting is every library waiting to be taken across the mesh (ADR 0328 §7, §8): from each machine's newest
// statement, each wait of reason keep that is excused (checkWaiting), by library.
func foldersWaiting(ctx context.Context, inv *inventory.Inventory, healths map[string]inventory.NodeHealth) (
map[string][]waitingFolder, operatorWaitFacts) {
var f operatorWaitFacts
out := map[string][]waitingFolder{}
machines := make([]string, 0, len(healths))
for m := range healths {
machines = append(machines, m)
}
sort.Strings(machines)
for _, machine := range machines {
rs := healths[machine].Resources
var modules []string
for _, r := range rs {
for _, w := range r.Waits {
if r.State == link.StateWaiting && w.Access != "" && !w.Disk {
modules = append(modules, r.Module)
}
}
}
if len(modules) == 0 {
continue
}
readWaitFacts(ctx, inv, machine, modules, nil, &f)
for _, r := range checkWaiting(machine, rs, f) {
if r.State != link.StateWaiting {
continue
}
for _, w := range r.Waits {
if w.Access == "" || w.Disk || w.Reason == link.WaitDisk {
continue
}
lib, isLibrary := strings.CutPrefix(w.Access, libraryAccess)
if !isLibrary {
lib = catalogue.LibraryOfEntry(f.manifests[r.Module], w.Access)
}
if lib == "" {
continue
}
placed := ""
if l := f.libraries[lib]; l.Home == machine {
placed = w.Placed
}
out[lib] = append(out[lib], waitingFolder{module: r.Module, machine: machine, placed: placed})
}
}
}
return out, f
}
// foldersIn is the asker's folders on this controller's stores: every library waiting to be taken, with its home, its
// root and the modules waiting on it.
func foldersIn(open *stores) func(ctx context.Context) ([]libraryTake, error) {
return func(ctx context.Context) ([]libraryTake, error) {
healths, err := open.inventory.Healths(ctx)
if err != nil {
return nil, err
}
waiting, f := foldersWaiting(ctx, open.inventory, healths)
return withoutTakesUnderWay(takesOf(waiting, f.libraries), f.taken), nil
}
}
// withoutTakesUnderWay is the takes not already under way: a take started on the same home, root and filesystem, which
// the engine has not yet stated done or failed, is not asked about again (the review of 7ea6d36e, item 5).
func withoutTakesUnderWay(takes []libraryTake, recorded map[string]inventory.TakenLibrary) []libraryTake {
var out []libraryTake
for _, t := range takes {
r, was := recorded[t.Library]
if was && !r.Done && r.Home == t.Node && r.Root == t.Root && (t.Placed == "" || r.Placed == "" || r.Placed == t.Placed) {
continue
}
out = append(out, t)
}
return out
}
// takesOf is one take per library and filesystem from the waits gathered, sorted. A library no longer defined is
// asked about nowhere.
func takesOf(waiting map[string][]waitingFolder, libraries map[string]catalogue.Library) []libraryTake {
var out []libraryTake
for lib, ws := range waiting {
l, defined := libraries[lib]
if !defined {
continue
}
// The home's word on the filesystem stands; waits from other machines name none.
placed := map[string]bool{}
for _, w := range ws {
if w.placed != "" {
placed[w.placed] = true
}
}
if len(placed) == 0 {
placed[""] = true
}
for p := range placed {
t := libraryTake{Library: lib, Node: l.Home, Root: l.Root, Placed: p}
seen := map[string]bool{}
for _, w := range ws {
if w.module != "" && !seen[w.module] {
seen[w.module] = true
t.Modules = append(t.Modules, w.module)
}
}
sort.Strings(t.Modules)
out = append(out, t)
}
}
sort.Slice(out, func(i, j int) bool { return out[i].key() < out[j].key() })
return out
}
+585
View File
@@ -0,0 +1,585 @@
package main
import (
"context"
"crypto/ed25519"
"encoding/json"
"errors"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/asks"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// novox/hq ADR 0328 §7, §8 and §9 on the controller's side: a library's wait is excused only for a defined library
// that reaches the machine and is not yet taken, or whose disk is not there; the gate passes it as a wait for a person
// and the operator reads it as needs-operator in words with no path; the operator is asked once per library, never
// about one taken; and the library is taken only on the warrant, where it still waits.
var radarrManifest = catalogue.Manifest{Module: "radarr", Version: "1",
Libraries: []catalogue.LibraryUse{{ID: "movies", Library: "media", Path: "movies", Use: catalogue.AccessReadWrite}}}
var plexManifest = catalogue.Manifest{Module: "plex", Version: "1",
Libraries: []catalogue.LibraryUse{{ID: "films", Library: "media", Path: "movies", Use: catalogue.AccessRead}}}
var mediaLib = catalogue.Library{Name: "media", Home: "ace", Root: "/storage/media", Reach: []string{"novox"}}
func keepWait(access string) inventory.Wait {
return inventory.Wait{Part: "radarr", Access: access, Reason: link.WaitKeep,
What: "let the mesh keep the library media"}
}
func placedWait(access, placed string) inventory.Wait {
w := keepWait(access)
w.Placed = placed
return w
}
func diskWait(access string) inventory.Wait {
return inventory.Wait{Part: "radarr", Access: access, Disk: true, Reason: link.WaitDisk,
What: "mount the disk that holds media"}
}
// libraryFacts is the facts with media defined, and each library given taken on ace on fs-000000000001.
func libraryFactsWith(taken ...string) operatorWaitFacts {
t := map[string]inventory.TakenLibrary{}
for _, l := range taken {
t[l] = inventory.TakenLibrary{Library: l, Home: "ace", Root: "/storage/media", Placed: "fs-000000000001", Done: true}
}
return operatorWaitFacts{
manifests: map[string]catalogue.Manifest{"radarr": radarrManifest, "plex": plexManifest},
given: map[string]map[string]time.Time{"radarr@ace": {}, "plex@ace": {}},
libraries: map[string]catalogue.Library{"media": mediaLib},
taken: t,
librariesRead: true,
}
}
// Rows 8 and 9: excused for a defined library reaching the machine and not yet taken — by the module's entry or the
// library's own wait — and for one whose disk is not there, taken or not; judged unhealthy when undefined, not
// reaching the machine, already taken on the filesystem it is on, for an access that is not a library, or for a
// reason the controller does not know.
func TestALibraryWaitIsExcusedOnlyForALibraryNotYetTaken(t *testing.T) {
for _, c := range []struct {
name string
machine string
w inventory.Wait
f operatorWaitFacts
says string
}{
{"the module's entry, not taken", "ace", keepWait("movies"), libraryFactsWith(), ""},
{"the library's own wait, not taken", "ace", keepWait("library:media"), libraryFactsWith(), ""},
{"on a machine it reaches", "novox", keepWait("movies"), libraryFactsWith(), ""},
{"its disk not there", "ace", diskWait("movies"), libraryFactsWith(), ""},
{"its disk not there, taken before", "ace", diskWait("movies"), libraryFactsWith("media"), ""},
{"taken already", "ace", keepWait("movies"), libraryFactsWith("media"), "taken already"},
{"taken on the filesystem it is on", "ace", placedWait("library:media", "fs-000000000001"),
libraryFactsWith("media"), "taken already"},
{"taken on another filesystem: it moved", "ace", placedWait("library:media", "fs-0000000000ff"),
libraryFactsWith("media"), ""},
{"taken at another root: its root changed", "ace", keepWait("movies"), func() operatorWaitFacts {
f := libraryFactsWith("media")
f.taken["media"] = inventory.TakenLibrary{Library: "media", Home: "ace", Root: "/storage/old-media", Done: true,
Placed: "fs-000000000001"}
return f
}(), ""},
{"a machine it does not reach", "far", keepWait("movies"), libraryFactsWith(), "does not reach far"},
{"undefined", "ace", keepWait("library:books"), libraryFactsWith(), "not defined"},
{"not a library", "ace", keepWait("downloads"), libraryFactsWith(), "only a library waits"},
{"a reason not known", "ace", inventory.Wait{Part: "radarr", Access: "movies", Reason: "later", What: "w"},
libraryFactsWith(), "take it or for its disk"},
{"libraries unread", "ace", keepWait("movies"), operatorWaitFacts{manifests: libraryFactsWith().manifests},
"could not be read"},
} {
err := checkWait("radarr", c.machine, c.w, c.f)
switch {
case c.says == "" && err != nil:
t.Errorf("%s: refused: %v", c.name, err)
case c.says != "" && (err == nil || !strings.Contains(err.Error(), c.says)):
t.Errorf("%s: %v; want a refusal saying %q", c.name, err, c.says)
}
}
}
func libraryWaiting(module string, w inventory.Wait) inventory.ResourceHealth {
return inventory.ResourceHealth{Module: module, Resource: module + ".app", Kind: "container", Target: module,
State: link.StateWaiting, Waits: []inventory.Wait{w}}
}
// Rows 8 and 9 at the gate: both excused waits pass the first-node gate as a wait for a person, and the module is said
// as needs-operator in words with no path; a wait on a library taken already does not pass.
func TestALibraryWaitPassesTheGateAndAsksTheOperatorWithoutAPath(t *testing.T) {
now := time.Now()
for _, w := range []inventory.Wait{keepWait("movies"), diskWait("movies")} {
w.What += " at /storage/media" // an engine's words carrying a path never reach the operator's
f := gateFacts{now: now, waits: libraryFactsWith(), groupsAdded: map[string]bool{"radarr": false},
health: map[string]inventory.NodeHealth{"ace": {Node: "ace", HeardAt: now,
Resources: []inventory.ResourceHealth{libraryWaiting("radarr", w)}}}}
h, why := moduleHealthWord("radarr", "ace", now.Add(-time.Minute), f)
if h != healthPerson || !strings.Contains(why, "waits for the operator") || strings.Contains(why, "/") {
t.Fatalf("an excused library wait reads %v %q; want a wait for a person, with no path", h, why)
}
o := needsOperatorObservation("radarr", "ace", []inventory.Wait{w}, nil)
for _, words := range []string{o.Summary, o.Headline, o.Explanation, o.Needs} {
if strings.Contains(words, "/") {
t.Errorf("the operator reads a path: %q", words)
}
}
want := "let the mesh keep the library entry movies when you are asked"
if w.Disk {
want = "mount the disk that holds the library entry movies on ace"
}
if o.Kind != kindNeedsOperator || !strings.Contains(o.Needs, want) {
t.Errorf("needs-operator %s asks %q; want %q", o.Kind, o.Needs, want)
}
}
f := gateFacts{now: now, waits: libraryFactsWith("media"), groupsAdded: map[string]bool{"radarr": false},
health: map[string]inventory.NodeHealth{"ace": {Node: "ace", HeardAt: now,
Resources: []inventory.ResourceHealth{libraryWaiting("radarr", keepWait("movies"))}}}}
if h, why := moduleHealthWord("radarr", "ace", now.Add(-time.Minute), f); h != healthNotYet || !strings.Contains(why, "taken") {
t.Fatalf("a wait on a taken library reads %v %q", h, why)
}
}
func takeRig(t *testing.T, waiting func() map[string][]waitingFolder) (*askerRig, *[]string) {
r := newAskerRig(t)
var taken []string
libs := map[string]catalogue.Library{"media": mediaLib}
r.a.folders = func(context.Context) ([]libraryTake, error) { return takesOf(waiting(), libs), nil }
r.a.machines = func(context.Context) ([]string, error) { return []string{"ace", "novox"}, nil }
r.a.take = func(_ context.Context, lt libraryTake, askID, by string) (string, error) {
taken = append(taken, lt.Library+" "+lt.Node+" "+lt.Root+" "+lt.Placed+" "+askID+" "+by)
return "taking", nil
}
return r, &taken
}
func takeAsks(r *askerRig) []asked {
var out []asked
for _, a := range r.store {
if a.Take != nil {
out = append(out, a)
}
}
return out
}
func openTakes(r *askerRig) []asked {
var out []asked
for _, a := range takeAsks(r) {
if a.State == askOpen {
out = append(out, a)
}
}
return out
}
// Row 7: one ask per library however many modules and machines wait on it, in the clause's words, at approve; none
// again while it is open; taken back when the library no longer waits.
func TestTheOperatorIsAskedOncePerLibrary(t *testing.T) {
current := map[string][]waitingFolder{"media": {
{module: "radarr", machine: "ace", placed: "fs-000000000001"},
{module: "plex", machine: "novox"},
{module: "", machine: "ace", placed: "fs-000000000001"}}}
r, _ := takeRig(t, func() map[string][]waitingFolder { return current })
ctx := context.Background()
for i := 0; i < 3; i++ {
if err := r.a.reconcile(ctx); err != nil {
t.Fatal(err)
}
}
kept := takeAsks(r)
if len(kept) != 1 {
t.Fatalf("%d asks for one library", len(kept))
}
q := kept[0].Ask
if err := q.Check(r.now); err != nil {
t.Fatal(err)
}
if q.Highest() != asks.Approve || len(q.Options) != 1 {
t.Fatalf("the ask offers %+v; want one answer at approve", q.Options)
}
// The root may not leave the mesh (the router's content rule): it is said as "its root".
want := "Let the mesh keep library media at its root on ace: its group becomes mesh-media, the group may read " +
"and write, folders keep the group; owners and everyone else's rights stay as they are. Waiting on it: plex, radarr."
if q.Explanation != want {
t.Errorf("the ask says\n %q\nwant\n %q", q.Explanation, want)
}
if kept[0].Take.Node != "ace" || kept[0].Take.Placed != "fs-000000000001" {
t.Errorf("the take is for %+v; want the home and the filesystem its home names", kept[0].Take)
}
current = nil
if err := r.a.reconcile(ctx); err != nil {
t.Fatal(err)
}
if got := takeAsks(r); len(got) != 1 || got[0].State != askCancelled {
t.Fatalf("the ask about a library that no longer waits: %+v", got)
}
}
// Row 7: nothing is taken without the operator's warrant — a warrant for another asker, an ask that expired, or a
// record changed after the ask takes nothing; the warrant takes it once, on its home, naming library:<name>.
func TestALibraryIsTakenOnlyOnTheWarrant(t *testing.T) {
waits := func() map[string][]waitingFolder {
return map[string][]waitingFolder{"media": {{module: "radarr", machine: "ace", placed: "fs-000000000001"}}}
}
ctx := context.Background()
r, taken := takeRig(t, waits)
if err := r.a.reconcile(ctx); err != nil {
t.Fatal(err)
}
a := takeAsks(r)[0]
other := warrantOn(a, "let-the-mesh-keep-it", r.now)
other.Asker = "someone-else"
answerWith(t, r, other)
changed := r.store[a.ID]
changed.Take = &libraryTake{Library: "media", Node: "ace", Root: "/", Modules: a.Take.Modules, Placed: a.Take.Placed}
r.store[a.ID] = changed
answerWith(t, r, warrantOn(a, "let-the-mesh-keep-it", r.now))
if len(*taken) != 0 || !strings.HasPrefix(r.store[a.ID].Acted, "failed") {
t.Fatalf("a take whose record changed after the ask: %v, %q", *taken, r.store[a.ID].Acted)
}
r2, taken2 := takeRig(t, waits)
if err := r2.a.reconcile(ctx); err != nil {
t.Fatal(err)
}
b := takeAsks(r2)[0]
answerWith(t, r2, asks.Warrant{Ask: b.ID, Asker: "mesh-controller", Outcome: asks.OutcomeExpired, At: r2.now})
if len(*taken2) != 0 {
t.Fatal("taken on an ask that expired")
}
r3, taken3 := takeRig(t, waits)
if err := r3.a.reconcile(ctx); err != nil {
t.Fatal(err)
}
c := takeAsks(r3)[0]
w := warrantOn(c, "let-the-mesh-keep-it", r3.now)
answerWith(t, r3, w)
answerWith(t, r3, w)
if len(*taken3) != 1 || !strings.HasPrefix((*taken3)[0], "media ace /storage/media fs-000000000001 "+c.ID) ||
!strings.Contains((*taken3)[0], "telegram") {
t.Fatalf("the warrant took %v", *taken3)
}
if !strings.HasPrefix(r3.store[c.ID].Acted, "done") {
t.Fatalf("the take was recorded %q", r3.store[c.ID].Acted)
}
}
// A warrant for a library on one filesystem is never carried out on another: when it comes and the library waits on
// another, nothing is sent, the ask is spent, and the operator is asked again for the filesystem it is on now.
func TestAWarrantForOneFilesystemIsNotCarriedOutOnAnother(t *testing.T) {
placed := "fs-000000000001"
r, taken := takeRig(t, func() map[string][]waitingFolder {
return map[string][]waitingFolder{"media": {{module: "radarr", machine: "ace", placed: placed}}}
})
ctx := context.Background()
if err := r.a.reconcile(ctx); err != nil {
t.Fatal(err)
}
a := takeAsks(r)[0]
placed = "fs-0000000000ff"
answerWith(t, r, warrantOn(a, "let-the-mesh-keep-it", r.now))
if len(*taken) != 0 {
t.Fatalf("a warrant for %s was carried out on %s: %v", a.Take.Placed, placed, *taken)
}
if spent := r.store[a.ID]; spent.State == askOpen || !strings.HasPrefix(spent.Acted, "nothing") {
t.Fatalf("the ask after the library moved: %s, %q", spent.State, spent.Acted)
}
if err := r.a.reconcile(ctx); err != nil {
t.Fatal(err)
}
if again := openTakes(r); len(again) != 1 || again[0].Take.Placed != placed {
t.Fatalf("not asked again for the filesystem it is on now: %+v", again)
}
}
// The take act names library:<name> and the warrant, is signed over its own context, and is recorded only on the
// engine's word.
func TestATakeAskNamesTheLibraryAndItsWarrant(t *testing.T) {
_, key, _ := ed25519.GenerateKey(nil)
signer := keySigner{key}
ask := link.AccessActAsk{Node: "ace", Path: "/storage/media", By: "the operator", Access: "library:media"}
if _, err := link.SignTake(context.Background(), signer, ask); err == nil {
t.Fatal("a take with no warrant was signed")
}
ask.Warrant = "c0123456789abcdef"
body, err := link.SignTake(context.Background(), signer, ask)
if err != nil {
t.Fatal(err)
}
var signed link.SignedHandOver
if err := json.Unmarshal(body, &signed); err != nil {
t.Fatal(err)
}
if !ed25519.Verify(key.Public().(ed25519.PublicKey), append([]byte(link.AccessActContext), signed.Ask...), signed.Signature) {
t.Fatal("the take is not signed over its own context")
}
var got map[string]any
if err := json.Unmarshal(signed.Ask, &got); err != nil {
t.Fatal(err)
}
if got["act"] != "take" || got["warrant"] != "c0123456789abcdef" || got["access"] != "library:media" ||
got["path"] != "/storage/media" {
t.Fatalf("the ask carries %v", got)
}
var asked link.AccessActAsk
refusing := takeFolderIn(nil, func(_ context.Context, a link.AccessActAsk) (link.HandOverAnswer, error) {
asked = a
return link.HandOverAnswer{Refused: "the disk is not there"}, nil
})
if _, err := refusing(context.Background(), libraryTake{Library: "media", Node: "ace", Root: "/storage/media"},
"c01", "the operator"); err == nil {
t.Fatal("a refused take was answered as done")
}
if asked.Access != "library:media" || asked.Node != "ace" || asked.Path != "/storage/media" || asked.Warrant != "c01" {
t.Fatalf("the engine was asked %+v", asked)
}
unreachable := takeFolderIn(nil, func(context.Context, link.AccessActAsk) (link.HandOverAnswer, error) {
return link.HandOverAnswer{}, errors.New("nothing on ace answers")
})
if _, err := unreachable(context.Background(), libraryTake{Library: "media", Node: "ace", Root: "/storage/media"},
"c01", "the operator"); err == nil {
t.Fatal("an unanswered take was answered as done")
}
}
type keySigner struct{ key ed25519.PrivateKey }
func (s keySigner) Sign(_ context.Context, b []byte) ([]byte, error) {
return ed25519.Sign(s.key, b), nil
}
// The envelope carries groups, identities and libraries only when composed — for an engine at link.LibraryContract —
// and an older engine's body is what it was.
func TestTheDeclarationSaysItsLibrariesOnlyWhenComposed(t *testing.T) {
older, err := sendable{Resources: []map[string]any{{"id": "a"}}}.Body()
if err != nil {
t.Fatal(err)
}
for _, key := range []string{"groups", "identities", "libraries"} {
if strings.Contains(string(older), key) {
t.Fatalf("an older engine's body names %s: %s", key, older)
}
}
newer, err := sendable{Resources: []map[string]any{{"id": "a"}},
Groups: []catalogue.GroupEntry{{Name: "mesh-media", GID: 50010}},
Identities: []catalogue.IdentityEntry{{Module: "radarr", Name: "mesh-radarr", UID: 50020, GID: 50010}},
Libraries: []catalogue.LibraryEntry{{Library: "media", Root: "/storage/media", Group: "mesh-media", GID: 50010,
Home: true, Paths: []string{"movies"}, Runner: "jochen"}}}.Body()
if err != nil {
t.Fatal(err)
}
for _, want := range []string{`"groups":[{"name":"mesh-media","gid":50010}]`,
`"identities":[{"module":"radarr","name":"mesh-radarr","uid":50020,"gid":50010}]`,
`"libraries":[{"library":"media","root":"/storage/media","group":"mesh-media","gid":50010,"home":true,` +
`"taken":false,"paths":["movies"],"runner":"jochen"}]`} {
if !strings.Contains(string(newer), want) {
t.Errorf("the body lacks %s: %s", want, newer)
}
}
if link.LibraryContract != 6 {
t.Fatalf("the contract the engine states for ADR 0328 is 6, and this controller reads %d", link.LibraryContract)
}
}
// The gate: an access or library the engine cannot judge this look, or entries left while a disk was away, of a module
// whose move leaves its accesses and libraries as they were, are not counted; of one whose move changes them — a
// library added, or joining — or is not known, they hold the gate to its bound.
func TestAnUnknownLibraryIsJudgedByWhetherTheMoveChangesIt(t *testing.T) {
now := time.Now()
since := now.Add(-time.Minute)
unknown := inventory.ResourceHealth{Module: "radarr", Resource: "radarr.library-movies", Kind: link.KindAccess,
Target: "/storage/media/movies", State: link.StateUnknown, Since: now}
left := inventory.ResourceHealth{Module: "radarr", Resource: "radarr.access-gone~entries", Kind: link.KindKeptEntries,
Target: "/storage/media", State: link.StateUnhealthy, Since: since.Add(-time.Hour)}
facts := func(changes map[string]bool, rs ...inventory.ResourceHealth) gateFacts {
return gateFacts{now: now, waits: libraryFactsWith(), groupsAdded: map[string]bool{"radarr": false},
accessesChanged: changes, health: map[string]inventory.NodeHealth{"ace": {Node: "ace", HeardAt: now,
Resources: rs}}}
}
unchanged := map[string]bool{"radarr": false}
if h, why := moduleHealthWord("radarr", "ace", since, facts(unchanged, unknown)); h != healthGood {
t.Errorf("an unknown library of a move that changes nothing reads %v %q", h, why)
}
if h, why := moduleHealthWord("radarr", "ace", since, facts(unchanged, left)); h != healthPerson ||
!strings.Contains(why, "access-gone~entries") {
t.Errorf("entries left, a move that changes nothing: %v %q", h, why)
}
for name, changes := range map[string]map[string]bool{"changes them": {"radarr": true}, "not known": nil} {
for _, r := range []inventory.ResourceHealth{unknown, left} {
if h, _ := moduleHealthWord("radarr", "ace", since, facts(changes, r)); h != healthNotYet {
t.Errorf("%s %s of a move that %s reads %v; want held to the bound", r.Kind, r.State, name, h)
}
}
}
plain := catalogue.Manifest{Module: "radarr"}
if sameAccessUse(plain, radarrManifest) || !sameAccessUse(radarrManifest, radarrManifest) {
t.Fatal("a move that adds a library, and joins, reads unchanged — or an unchanged one reads changed")
}
}
// The engine states a library once, of no module; each module assigned to the machine that uses it is judged and said
// by a copy of that verdict — its wait excused and passed by the gate as a wait for a person. A library unknown or
// starting (a take walking under it) of a move that changes no library is not counted.
func TestALibrarysVerdictIsSaidForTheModulesThatUseIt(t *testing.T) {
inv := inventory.ForTest(t)
ctx := t.Context()
if _, err := inv.AddNode(ctx, "ace"); err != nil {
t.Fatal(err)
}
for _, m := range []catalogue.Manifest{radarrManifest, plexManifest, {Module: "plain", Version: "1"}} {
if err := inv.RegisterModule(ctx, m, inventory.Source{}); err != nil {
t.Fatal(err)
}
if m.Module == "plex" {
continue // pulled in by a requirement, not assigned: it is in what the machine was sent (item 4)
}
if _, err := inv.Assign(ctx, "ace", m.Module); err != nil {
t.Fatal(err)
}
}
record, err := inv.NodeByName(ctx, "ace")
if err != nil {
t.Fatal(err)
}
if err := inv.RecordSentWith(ctx, record.ID, "d1", map[string]string{"radarr": "c1", "plex": "c2", "plain": "c3"},
1, 1, inventory.Send{Sequence: 1}); err != nil {
t.Fatal(err)
}
lib := inventory.ResourceHealth{Resource: "library:media", Kind: link.KindLibrary, Target: "/storage/media",
State: link.StateWaiting, Waits: []inventory.Wait{{Part: "media", Access: "library:media", Reason: link.WaitKeep,
Placed: "fs-000000000001", What: "let the mesh keep the library media"}}}
copies := libraryVerdictsFor(ctx, inv, "ace", []inventory.ResourceHealth{lib})
var modules []string
for _, c := range copies {
modules = append(modules, c.Module)
}
if strings.Join(modules, ",") != "plex,radarr" {
t.Fatalf("the library's verdict is said for %v; want plex and radarr, the modules using it", modules)
}
paths, err := inv.LibraryPaths(ctx)
if err != nil {
t.Fatal(err)
}
if strings.Join(paths["media"], ",") != "movies" {
t.Fatalf("the paths kept of media are %v", paths["media"])
}
now := time.Now()
f := gateFacts{now: now, waits: libraryFactsWith(), groupsAdded: map[string]bool{"radarr": false},
accessesChanged: map[string]bool{"radarr": false},
health: map[string]inventory.NodeHealth{"ace": {Node: "ace", HeardAt: now, Resources: copies}}}
if h, why := moduleHealthWord("radarr", "ace", now.Add(-time.Minute), f); h != healthPerson {
t.Fatalf("radarr waiting on media reads %v %q; want a wait for a person", h, why)
}
for _, state := range []string{link.StateUnknown, link.StateStarting} {
c := copies[1]
c.State, c.Waits = state, nil
f.health["ace"] = inventory.NodeHealth{Node: "ace", HeardAt: now, Resources: []inventory.ResourceHealth{c}}
if h, why := moduleHealthWord("radarr", "ace", now.Add(-time.Minute), f); h != healthGood {
t.Errorf("media %s, a move that changes no library: %v %q", state, h, why)
}
f.accessesChanged = map[string]bool{"radarr": true}
if h, _ := moduleHealthWord("radarr", "ace", now.Add(-time.Minute), f); h != healthNotYet {
t.Errorf("media %s, a move that changes a library: %v; want held to the bound", state, h)
}
f.accessesChanged = map[string]bool{"radarr": false}
}
}
// The review of 7ea6d36e, item 5: the engine saying it started a take does not make the library taken. While the act
// runs the library still waits and the operator is not asked again; it is taken when the engine states the act done
// under that warrant.
func TestALibraryIsTakenOnlyWhenTheEngineSaysTheActIsDone(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
register(t, open, catalogue.Manifest{Module: catalogue.LibrariesModule, Version: "1"})
if err := inv.SetSettings(ctx, "", catalogue.LibrariesModule, map[string]any{catalogue.LibrariesSetting: map[string]any{
"media": map[string]any{"home": "anchor", "root": "/storage/media"}}}); err != nil {
t.Fatal(err)
}
take := takeFolderIn(open, func(context.Context, link.AccessActAsk) (link.HandOverAnswer, error) {
return link.HandOverAnswer{Said: "the take of library:media started"}, nil
})
lt := libraryTake{Library: "media", Node: "anchor", Root: "/storage/media", Placed: "fs-000000000001"}
if _, err := take(ctx, lt, "c01", "the operator"); err != nil {
t.Fatal(err)
}
libs, err := inv.Libraries(ctx)
if err != nil {
t.Fatal(err)
}
taken, err := librariesTaken(ctx, inv, libs)
if err != nil {
t.Fatal(err)
}
if taken["media"] {
t.Fatal("a take only started made the library taken")
}
// While it runs: the wait is still excused, and no new ask is made for it.
var f operatorWaitFacts
readWaitFacts(ctx, inv, "anchor", nil, nil, &f)
if err := checkWait("radarr", "anchor", placedWait("library:media", "fs-000000000001"), f); err != nil {
t.Fatalf("the wait while the take runs: %v", err)
}
if got := withoutTakesUnderWay([]libraryTake{lt}, f.taken); len(got) != 0 {
t.Fatalf("a library whose take runs is asked about again: %+v", got)
}
// The engine states the act done under that warrant.
act, _ := json.Marshal(map[string]any{"act": "take", "access": "library:media", "warrant": "c01", "state": "done"})
if err := stateHealth(ctx, inv, nil, "anchor", link.Health{Contract: link.LibraryContract, At: time.Now(),
Resources: []link.ResourceHealth{{Resource: "library:media", Kind: link.KindLibrary, Target: "/storage/media",
State: link.StateHealthy, Since: time.Now(), Act: act}}}, time.Now()); err != nil {
t.Fatal(err)
}
if taken, err = librariesTaken(ctx, inv, libs); err != nil || !taken["media"] {
t.Fatalf("the act done did not make the library taken (%v)", err)
}
}
// ADR 0334: the engine states the tool runner's membership as resources of kind runner — `runner:<module>` of that
// module, `runner:library:<name>` of no module. A library's is said for the modules that use it; neither fails the
// first-node gate of a move that leaves the module's libraries as they were, and both hold one that changes them.
func TestARunnersVerdictIsSaidAndJudgedByWhetherTheMoveChangesIt(t *testing.T) {
inv := inventory.ForTest(t)
ctx := t.Context()
if _, err := inv.AddNode(ctx, "ace"); err != nil {
t.Fatal(err)
}
for _, m := range []catalogue.Manifest{radarrManifest, plexManifest} {
if err := inv.RegisterModule(ctx, m, inventory.Source{}); err != nil {
t.Fatal(err)
}
if _, err := inv.Assign(ctx, "ace", m.Module); err != nil {
t.Fatal(err)
}
}
lib := inventory.ResourceHealth{Resource: "runner:library:media", Kind: link.KindRunner, Target: "jochen",
State: link.StateUnhealthy, Reason: "the tool runner is not in mesh-media yet", Since: time.Now()}
copies := libraryVerdictsFor(ctx, inv, "ace", []inventory.ResourceHealth{lib})
if len(copies) != 2 || copies[0].Module != "plex" || copies[1].Module != "radarr" {
t.Fatalf("the runner's library verdict is said for %+v; want plex and radarr", copies)
}
own := inventory.ResourceHealth{Module: "radarr", Resource: "runner:radarr", Kind: link.KindRunner, Target: "jochen",
State: link.StateUnhealthy, Reason: "the tool runner is not in mesh-radarr yet", Since: time.Now()}
now := time.Now()
for _, r := range []inventory.ResourceHealth{copies[1], own} {
f := gateFacts{now: now, waits: libraryFactsWith(), groupsAdded: map[string]bool{"radarr": false},
accessesChanged: map[string]bool{"radarr": false},
health: map[string]inventory.NodeHealth{"ace": {Node: "ace", HeardAt: now, Resources: []inventory.ResourceHealth{r}}}}
if h, why := moduleHealthWord("radarr", "ace", now.Add(-time.Minute), f); h != healthGood {
t.Errorf("%s, a move that changes no library: %v %q; want not counted", r.Resource, h, why)
}
f.accessesChanged = map[string]bool{"radarr": true}
if h, _ := moduleHealthWord("radarr", "ace", now.Add(-time.Minute), f); h != healthNotYet {
t.Errorf("%s, a move that changes a library: %v; want held to the bound", r.Resource, h)
}
}
}
+138 -2
View File
@@ -2,7 +2,9 @@ package main
import (
"context"
"encoding/json"
"fmt"
"slices"
"sort"
"strings"
"sync/atomic"
@@ -40,6 +42,96 @@ const (
moduleUnhealthyAfter = 2
)
// runnerLibrary prefixes the engine's verdict on the tool runner's membership of a library's group (ADR 0334).
const runnerLibrary = "runner:library:"
// libraryVerdictsFor is, for each library verdict in a statement (kind library, no module), one copy for each module
// on the machine — assigned, or pulled in by a requirement — whose manifest uses that library: what the module is judged and said by. None when the
// assignments or the catalogue cannot be read: the library's own verdict stands alone, and no module is excused by it.
func libraryVerdictsFor(ctx context.Context, inv *inventory.Inventory, node string,
rs []inventory.ResourceHealth) []inventory.ResourceHealth {
var libraries []inventory.ResourceHealth
for _, r := range rs {
if r.Module == "" && ((r.Kind == link.KindLibrary && strings.HasPrefix(r.Resource, libraryAccess)) ||
(r.Kind == link.KindRunner && strings.HasPrefix(r.Resource, runnerLibrary))) {
libraries = append(libraries, r)
}
}
if len(libraries) == 0 || inv == nil {
return nil
}
assigned, err := inv.Assigned(ctx, node)
if err != nil {
// empty-on-error: the library's own verdict is kept and judged; no module is excused by a copy it could not make
return nil
}
// And the modules a requirement pulled in: what the machine was last sent (the review of 7ea6d36e, item 4).
sent, _, err := inv.SentBuilds(ctx, node)
if err != nil {
// empty-on-error: the library's own verdict is kept and judged; no module is excused by a copy it could not make
return nil
}
for m := range sent {
if !slices.Contains(assigned, m) {
assigned = append(assigned, m)
}
}
sort.Strings(assigned)
shelf, err := inv.Catalogue(ctx)
if err != nil {
// empty-on-error: the library's own verdict is kept and judged; no module is excused by a copy it could not make
return nil
}
var out []inventory.ResourceHealth
for _, lib := range libraries {
name := strings.TrimPrefix(strings.TrimPrefix(lib.Resource, runnerLibrary), libraryAccess)
for _, module := range assigned {
for _, u := range shelf[module].Libraries {
if u.Library == name {
copied := lib
copied.Module = module
out = append(out, copied)
break
}
}
}
}
return out
}
// takesStated keeps what a statement says of each library's take act: done under a warrant, the library is taken;
// failed, the take is forgotten and the operator asked again. Running or stopped (it goes on) changes nothing.
func takesStated(ctx context.Context, inv *inventory.Inventory, rs []link.ResourceHealth) error {
if inv == nil {
return nil
}
for _, r := range rs {
lib, isLibrary := strings.CutPrefix(r.Resource, libraryAccess)
if !isLibrary || r.Kind != link.KindLibrary || len(r.Act) == 0 {
continue
}
var act struct {
Act string `json:"act"`
Warrant string `json:"warrant"`
State string `json:"state"`
}
if err := json.Unmarshal(r.Act, &act); err != nil || act.Act != link.ActTake || act.Warrant == "" {
continue
}
var err error
switch act.State {
case "done":
err = inv.TakeDone(ctx, lib, act.Warrant)
case "failed":
err = inv.TakeFailed(ctx, lib, act.Warrant)
}
if err != nil {
return err
}
}
return nil
}
// healthRefused counts the statements refused as older than the one kept, for the log and a test.
var healthRefused atomic.Int64
@@ -77,8 +169,22 @@ func stateHealth(ctx context.Context, inv *inventory.Inventory, k *conditions.Ke
for _, w := range r.Waits {
kept.Waits = append(kept.Waits, inventory.Wait{Part: w.Part, Secret: w.Secret, Setting: w.Setting, What: w.What})
}
// An access's waits (novox/hq ADR 0325 §7), kept beside the others as one more thing the operator gives: the
// access by its module's id, and whether it waits for a disk rather than for the operator's word.
for _, w := range r.AccessWaits {
kept.Waits = append(kept.Waits, inventory.Wait{Part: w.Part, Access: w.Access, Disk: w.Reason == link.WaitDisk,
Placed: w.Placed, Reason: w.Reason, What: w.What})
}
resources = append(resources, kept)
}
// **A library's take is done, or failed, when its engine says so** (novox/hq ADR 0328 §7): only then is it taken,
// or asked about again.
if err := takesStated(ctx, inv, h.Resources); err != nil {
return err
}
// **A library's verdict is said for the modules that use it** (novox/hq ADR 0328 §8): the engine states each
// library once, as `library:<name>`, of no module; a module that uses it waits with it, and is said so.
resources = append(resources, libraryVerdictsFor(ctx, inv, node, resources)...)
// **A wait for the operator is checked before it is excused** (novox/hq ADR 0283): a waiting resource whose
// wait does not check out is judged unhealthy, saying why; one that does is kept beside the unhealthy ones, so
// judgeModuleHealth can say it as needs-operator. What is stored is what the machine said.
@@ -644,7 +750,7 @@ func moduleHealthWord(module, machine string, since time.Time, f gateFacts) (hea
if waits && !f.groupsAdded[module] {
waits = false
}
var found, onWaiting []string
var found, onWaiting, leftover []string
var forOperator []inventory.Wait
for _, r := range resources {
if r.Module != module {
@@ -667,6 +773,32 @@ func moduleHealthWord(module, machine string, since time.Time, f gateFacts) (hea
found = append(found, r.Resource)
continue
}
// **The mesh's entries left on a folder whose disk was away are a condition this build found** when its move
// leaves the module's accesses and their users as they were (novox/hq ADR 0325 §8): they may name an access no
// longer declared, they wait for the disk, and a delivery that did not bring them is not held for them — said in
// the verdict, as a directory used as found is. A move that changes the accesses is held to them, whenever they
// were stated (the review of e1c2b744).
changed, known := f.accessesChanged[module]
changes := !known || changed
if r.Kind == link.KindKeptEntries && r.State == link.StateUnhealthy && !changes {
leftover = append(leftover, r.Resource)
continue
}
// **An access the engine could not judge this look** (its filesystem could not be named — a hung mount) carries
// no wait. Of a module whose move leaves its accesses as they were it is a condition the move found, not
// counted; of one whose move changes them it is not yet healthy, held to the gate's bound like anything still
// unsettled, so the gate fails safe at the bound and never waits for ever.
// The same for a library (ADR 0328): unknown this look, or starting while a take walks under it, of a move that
// leaves the module's libraries as they were, is not this move's.
if (r.Kind == link.KindAccess || r.Kind == link.KindLibrary) && !changes &&
(r.State == link.StateUnknown || (r.Kind == link.KindLibrary && r.State == link.StateStarting)) {
continue
}
// The tool runner's membership (ADR 0334 §2), of a move that changes no library of the module, is not this
// move's: it is said as the module's condition, and holds a move that changes them to the gate's bound.
if r.Kind == link.KindRunner && !changes {
continue
}
switch r.State {
case link.StateHealthy:
case link.StateStarting:
@@ -690,7 +822,7 @@ func moduleHealthWord(module, machine string, since time.Time, f gateFacts) (hea
reasonAfter(r.Reason))
}
}
if waits || len(found) > 0 || len(forOperator) > 0 || len(onWaiting) > 0 {
if waits || len(found) > 0 || len(forOperator) > 0 || len(onWaiting) > 0 || len(leftover) > 0 {
said := onWaiting
if waits {
said = append(said, wait)
@@ -702,6 +834,10 @@ func moduleHealthWord(module, machine string, since time.Time, f gateFacts) (hea
said = append(said, fmt.Sprintf("on %s, %s uses %s as found and waits for the operator to hand it over "+
"(`nox node hand-over %s <directory>` on the control-node)", machine, module, strings.Join(found, ", "), machine))
}
if len(leftover) > 0 {
said = append(said, fmt.Sprintf("on %s, the mesh's entries %s could not be taken out while their disk was "+
"away, stated before this send: they are taken out when it is back", machine, strings.Join(leftover, ", ")))
}
return healthPerson, strings.Join(said, "; ")
}
return healthGood, ""
+102 -2
View File
@@ -44,17 +44,34 @@ func needsOperatorKey(module, node string) string {
type operatorWaitFacts struct {
manifests map[string]catalogue.Manifest
given map[string]map[string]time.Time
// libraries is every library the operator defined, and taken every one taken (novox/hq ADR 0328 §7, §8);
// librariesRead says both were read — when not, no access wait is excused.
libraries map[string]catalogue.Library
taken map[string]inventory.TakenLibrary
librariesRead bool
}
// checkWait is nil when a wait is excused, and otherwise why not, in words. Pure.
func checkWait(module, machine string, w inventory.Wait, f operatorWaitFacts) error {
// A library's own wait names it, and no module's manifest (novox/hq ADR 0328 §7).
if lib, ok := strings.CutPrefix(w.Access, libraryAccess); ok && w.Secret == "" && w.Setting == "" {
return checkLibraryWait(machine, lib, w, f)
}
m, known := f.manifests[module]
if !known {
return fmt.Errorf("says it waits for %s, and the mesh holds no manifest of %s to check it against", waitNames(w), module)
}
named := 0
for _, n := range []string{w.Secret, w.Setting, w.Access} {
if n != "" {
named++
}
}
switch {
case w.Secret != "" && w.Setting != "", w.Secret == "" && w.Setting == "":
return fmt.Errorf("says it waits, naming %s, where a wait names one secret or one setting", waitNames(w))
case named != 1:
return fmt.Errorf("says it waits, naming %s, where a wait names one secret or one setting, or one access", waitNames(w))
case w.Access != "":
return checkAccessWait(module, machine, m, w, f)
case w.Setting != "":
if _, declared := m.Settings[w.Setting]; !declared {
return fmt.Errorf("says it waits for the setting %s, which %s does not declare", w.Setting, module)
@@ -80,6 +97,63 @@ func checkWait(module, machine string, w inventory.Wait, f operatorWaitFacts) er
return nil
}
// libraryAccess prefixes the access a library's own wait and take name: "library:<name>".
const libraryAccess = "library:"
// checkAccessWait is nil when a module's access wait is excused (novox/hq ADR 0328 §8, §9): the access is one of its
// library entries, and its library's wait is excused. Any other access waits for nothing the operator gives: an
// access that is not a library keeps ADR 0051's meaning.
func checkAccessWait(module, machine string, m catalogue.Manifest, w inventory.Wait, f operatorWaitFacts) error {
lib := catalogue.LibraryOfEntry(m, w.Access)
if lib == "" {
return fmt.Errorf("says it waits for %s, which is no library entry %s declares: only a library waits for "+
"the operator (novox/hq ADR 0328 §8)", w.Access, module)
}
return checkLibraryWait(machine, lib, w, f)
}
// checkLibraryWait is nil when a wait on a library is excused (novox/hq ADR 0328 §8, §9): the library is defined and
// reaches the machine, and either its disk is not there (reason disk) or it has not yet been taken on its home, on the
// filesystem the wait names (reason keep). Any other is judged unhealthy, saying why: a module that uses a library
// already taken is never asked about it.
func checkLibraryWait(machine, lib string, w inventory.Wait, f operatorWaitFacts) error {
if !f.librariesRead {
return fmt.Errorf("says it waits for the library %s, and the libraries could not be read", lib)
}
l, defined := f.libraries[lib]
switch {
case !defined:
return fmt.Errorf("says it waits for the library %s, which is not defined", lib)
case !l.Reaches(machine):
return fmt.Errorf("says it waits for the library %s, which does not reach %s", lib, machine)
case w.Reason == link.WaitDisk || (w.Reason == "" && w.Disk):
return nil
case w.Reason != link.WaitKeep:
return fmt.Errorf("says it waits for the library %s for %q, and a library waits for the operator's word to "+
"take it or for its disk", lib, w.Reason)
}
// Taken on another home or filesystem (it moved) is not taken by that warrant: excused, and asked again (§7).
// Only the home's engine names the root's filesystem; another machine's wait names its mount, and says nothing of it.
placed := w.Placed
if machine != l.Home {
placed = ""
}
if t, was := f.taken[lib]; was && t.Done && t.Home == l.Home && t.Root == l.Root &&
(placed == "" || t.Placed == "" || t.Placed == placed) {
return fmt.Errorf("says it waits for the library %s, which is taken already: a module that uses a taken "+
"library is never asked about it", lib)
}
return nil
}
// accessWords is an access a wait names, in words with no path: "the library media", or "its library entry movies".
func accessWords(access string) string {
if lib, ok := strings.CutPrefix(access, libraryAccess); ok {
return "the library " + lib
}
return "the library entry " + access
}
// waitRefusedPrefix opens every reason checkWait gives, so the words of a refused wait are told from a check's own.
const waitRefusedPrefix = "says it waits"
@@ -88,6 +162,10 @@ func waitNames(w inventory.Wait) string {
switch {
case w.Secret != "" && w.Setting != "":
return "the secret " + w.Secret + " and the setting " + w.Setting
case w.Access != "" && (w.Secret != "" || w.Setting != ""):
return "the access " + w.Access + " and a secret or a setting"
case w.Access != "":
return "the access " + w.Access
case w.Secret != "":
return "the secret " + w.Secret
case w.Setting != "":
@@ -147,6 +225,14 @@ func operatorWaitSaid(module, machine string, waits []inventory.Wait) string {
var parts []string
for _, w := range waits {
part := fmt.Sprintf("%s (%s", w.What, waitNames(w))
// An access in the controller's own words, never the engine's: words with no path (novox/hq ADR 0325 §7).
switch {
case w.Access != "" && w.Disk:
part = fmt.Sprintf("the disk that holds %s mounted (%s", accessWords(w.Access), waitNames(w))
case w.Access != "":
part = fmt.Sprintf("leave to keep %s (%s, asked of you once for the library", accessWords(w.Access),
waitNames(w))
}
if w.Secret != "" {
part += fmt.Sprintf(", given with `nox secret ask %s %s %s`", machine, module, w.Secret)
}
@@ -180,6 +266,11 @@ func needsOperatorWords(module, node string, waits []inventory.Wait) words {
act, secret = fmt.Sprintf("type %s at %s's desk prompt", w.What, node), true
case w.Setting != "":
act = fmt.Sprintf("approve %s of %s on %s when it is proposed to you", w.Setting, module, node)
// An access, in words with no path (novox/hq ADR 0325 §7): the folder by the access's name.
case w.Access != "" && w.Disk:
act = fmt.Sprintf("mount the disk that holds %s on %s", accessWords(w.Access), node)
case w.Access != "":
act = fmt.Sprintf("let the mesh keep %s when you are asked", accessWords(w.Access))
}
if act != "" && !seen[act] {
seen[act] = true
@@ -217,6 +308,7 @@ func readWaitFacts(ctx context.Context, inv *inventory.Inventory, machine string
if f.given == nil {
f.given = map[string]map[string]time.Time{}
}
if inv == nil {
return
}
@@ -245,6 +337,14 @@ func readWaitFacts(ctx context.Context, inv *inventory.Inventory, machine string
f.given[module+"@"+machine] = given
}
}
// The libraries and which are taken (ADR 0328 §7, §8): read once.
if !f.librariesRead {
libs, lerr := inv.Libraries(ctx)
taken, terr := inv.TakenLibraries(ctx)
if lerr == nil && terr == nil {
f.libraries, f.taken, f.librariesRead = libs, taken, true
}
}
}
// waitingModules is every module with a waiting resource in a statement.
+14
View File
@@ -182,6 +182,20 @@ var plainWordings = map[string]func(conditions.Observation) words{
"register it, so it was not made.", module, m),
Resolved: "Resolved: " + module + " on " + m + " is settled"}
}),
// The controller or the node-engine kept at the build a machine was last sent, over an older record (novox/hq
// issue 201).
kindCoreDowngrade: worded(func(o conditions.Observation) words {
m := machineOr(o, "a machine")
component := idPart(o, 0)
if o.Scope != conditions.ScopeCore || !guarded(component) {
component = "core component"
}
return words{Headline: m + " keeps a newer " + component + " than the mesh holds",
Needs: fmt.Sprintf("push %s to take the older build, or a newer merge or rebuild moves the record forward.", m),
Explanation: fmt.Sprintf("The mesh's record of its own %s is older than the build %s was last sent, so "+
"the mesh's sends keep the newer one there rather than put it back.", component, m),
Resolved: "Resolved: " + m + " runs the " + component + " the mesh holds"}
}),
kindBindingKept: worded(func(o conditions.Observation) words {
m := machineOr(o, "a machine")
return words{Headline: "A module's data source is held on " + m,
+160 -4
View File
@@ -9,6 +9,7 @@ import (
"fmt"
"io"
"os"
"path/filepath"
"slices"
"sort"
"strings"
@@ -133,6 +134,11 @@ func planFor(ctx context.Context, open *stores, nodeName string) (catalogue.Reso
if _, err := keepRecorded(ctx, open, nodeName, shelf); err != nil {
return catalogue.Resolution{}, nil, err
}
// And a module that may not join yet, at the build it ran before (novox/hq ADR 0328 §8, ADR 0334).
held, err := holdJoining(ctx, inv, nodeName, shelf)
if err != nil {
return catalogue.Resolution{}, nil, err
}
resolved, err := catalogue.Resolve(shelf, assigned,
catalogue.Node{Name: nodeName, Site: site, Capabilities: capabilities,
@@ -218,6 +224,10 @@ func planFor(ctx context.Context, open *stores, nodeName string) (catalogue.Reso
}
settings[m.Module] = layers
}
// A held module with the settings it last ran with (ADR 0328 §8).
if err := heldSettings(ctx, inv, nodeName, held, settings); err != nil {
return catalogue.Resolution{}, nil, err
}
return resolved, settings, nil
}
@@ -447,7 +457,8 @@ func declarationWith(ctx context.Context, open *stores, node string,
if err != nil {
return sendable{}, err
}
out := sendable{Resources: composed.Resources, Adoption: adoption,
out := sendable{Resources: composed.Resources, Adoption: adoption, Groups: composed.Groups,
Identities: composed.Identities, Libraries: composed.Libraries,
Received: composed.Received, Mesh: with.Mesh, BusUsers: with.BusUsers,
LeftOut: sortedKeysOf(composed.LeftOut), leftOutWhy: composed.LeftOut, withheld: with.Withheld,
unbound: with.Unbound, foreseen: composed.Foreseen, unplaced: composed.Unplaced,
@@ -471,11 +482,22 @@ func declarationWith(ctx context.Context, open *stores, node string,
names = append(names, m.Module)
}
out.Builds = carriedBuilds(names, composed.LeftOut, current, before)
// A recorded module kept at the build the machine runs is recorded as carrying that one (ADR 0245).
// A recorded module kept at the build the machine runs is recorded as carrying that one (ADR 0245), and so
// is a joining module held at the build it ran before (ADR 0328 §8).
kept, err := recordedKept(ctx, open, node)
if err != nil {
return sendable{}, err
}
held, err := holdJoining(ctx, open.inventory, node, nil)
if err != nil {
return sendable{}, err
}
if kept == nil && len(held) > 0 {
kept = map[string]string{}
}
for m, h := range held {
kept[m] = h.commit
}
for m, was := range kept {
if _, carried := out.Builds[m]; carried {
out.Builds[m] = was
@@ -975,9 +997,17 @@ func renderingFor(ctx context.Context, open *stores, node string,
if err != nil {
return catalogue.Rendering{}, inventory.Node{}, err
}
// The libraries, the numbers of the modules that join and of the libraries' groups, and which libraries are taken
// (novox/hq ADR 0328), to an engine that keeps libraries.
lf, err := libraryFacts(ctx, inv, node, plan, choosing)
if err != nil {
return catalogue.Rendering{}, inventory.Node{}, err
}
return catalogue.Rendering{
ReadsHealth: readsHealth,
JudgesRoot: judgesRoot,
ReadsHealth: readsHealth,
JudgesRoot: judgesRoot,
KeepsLibraries: lf.keeps, Identities: lf.identities, Libraries: lf.libraries, LibraryGroups: lf.groups,
TakenLibraries: lf.taken, LibraryPaths: lf.paths, RunnerJoinsOwnGroups: lf.ownGroups,
BusMembership: memberships[node],
Settings: settings, Generators: gens, Grants: grants, Needed: needed, Foreseen: foreseen, Ports: ports,
Certificate: certificate, Authority: authority, Mesh: private, Names: names,
@@ -1010,6 +1040,132 @@ func engineJudgesRoot(ctx context.Context, inv *inventory.Inventory, node string
return had && stated.Contract >= link.RootContract, nil
}
// engineKeepsLibraries says whether a machine's node-engine makes the mesh's users and library groups, keeps
// libraries, and keeps a joining module's own groups (novox/hq ADR 0328, ADR 0334: its statement's contract is
// link.OwnGroupContract or later), by its own newest statement. On an older one no module joins: the libraries
// alone are not enough, since its tools would lose their read of a joining module's directories.
func engineKeepsLibraries(ctx context.Context, inv *inventory.Inventory, node string) (bool, error) {
stated, had, err := inv.HealthOf(ctx, node)
if err != nil {
return false, err
}
return had && stated.Contract >= link.OwnGroupContract, nil
}
// libraryRendering is what a machine's composition needs of ADR 0328.
type libraryRendering struct {
keeps bool
identities map[string]catalogue.ModuleIdentity
libraries map[string]catalogue.Library
groups map[string]int
taken map[string]bool
paths map[string][]string
ownGroups bool
}
// libraryFacts reads what a machine's composition needs of ADR 0328: the libraries and their group numbers, the
// numbers of the modules here that join, which libraries are taken, and the sub-paths every module asks of each. **A number is allocated only on the send path** (Allocating),
// as a port is: a question about what would be sent reads what was allocated. The libraries are read for every engine,
// so a module that joins is left out, naming why, where its engine cannot be told it does.
func libraryFacts(ctx context.Context, inv *inventory.Inventory, node string, plan catalogue.Resolution,
choosing Choosing) (libraryRendering, error) {
var lf libraryRendering
joining := false
for _, m := range plan.Modules {
joining = joining || catalogue.Joins(m)
}
libs, err := inv.Libraries(ctx)
if err != nil {
return lf, err
}
reached := false
for _, l := range libs {
reached = reached || l.Reaches(node)
}
if !joining && !reached {
return lf, nil // nothing here joins, and no library reaches it: composed exactly as before (§14)
}
lf.libraries = libs
if lf.keeps, err = engineKeepsLibraries(ctx, inv, node); err != nil || !lf.keeps {
return lf, err
}
lf.ownGroups = true // a module joins only on an engine that also keeps its own groups (ADR 0334)
lf.identities = map[string]catalogue.ModuleIdentity{}
for _, m := range plan.Modules {
if !catalogue.Joins(m) {
continue
}
var id catalogue.ModuleIdentity
had := true
if choosing == Allocating {
id, err = inv.AllocateIdentity(ctx, m.Module)
} else {
id, had, err = inv.IdentityOf(ctx, m.Module)
}
if err != nil {
return lf, err
}
if had {
lf.identities[m.Module] = id
}
}
if lf.groups, err = inv.LibraryGroups(ctx); err != nil {
return lf, err
}
if choosing == Allocating {
for name, l := range libs {
if _, has := lf.groups[name]; has || !l.Reaches(node) {
continue
}
if lf.groups[name], err = inv.AllocateLibraryGroup(ctx, name); err != nil {
return lf, err
}
}
}
if lf.paths, err = inv.LibraryPaths(ctx); err != nil {
return lf, err
}
if lf.taken, err = librariesTaken(ctx, inv, libs); err != nil {
return lf, err
}
return lf, nil
}
// librariesTaken is which libraries are taken now (§7): recorded taken on the home they have, on the filesystem their
// home's newest statement names — or stated kept by their home's engine, which keeps one it found empty without asking.
func librariesTaken(ctx context.Context, inv *inventory.Inventory, libs map[string]catalogue.Library) (map[string]bool, error) {
recorded, err := inv.TakenLibraries(ctx)
if err != nil {
return nil, err
}
out := map[string]bool{}
for name, l := range libs {
t, was := recorded[name]
taken := was && t.Done && t.Home == l.Home && t.Root == l.Root
stated, had, err := inv.HealthOf(ctx, l.Home)
if err != nil {
return nil, err
}
if had {
for _, r := range stated.Resources {
kept := (r.Resource == libraryAccess+name || filepath.Clean(r.Target) == l.Root) &&
(r.Kind == link.KindAccess || r.Kind == link.KindLibrary)
if kept && r.State == link.StateHealthy {
taken = true
}
for _, w := range r.Waits {
if w.Access == libraryAccess+name && w.Reason == link.WaitKeep && w.Placed != "" && was &&
t.Placed != "" && w.Placed != t.Placed {
taken = false // it moved to another filesystem: not taken there until its new warrant
}
}
}
}
out[name] = taken
}
return out, nil
}
// zonesInTheMesh is every zone a module in the mesh declares, where the mesh placed it (novox/hq ADR
// 0199): the zone settled from that node's settings, the node's private address, the port the
// answering listen is published on there.
+26
View File
@@ -0,0 +1,26 @@
package main
import (
"fmt"
"regexp"
)
// planID is every id the mesh gives a plan: a merge's walk or batch (plan-<nanoseconds>) or a release
// (release-<nanoseconds>).
var planID = regexp.MustCompile(`^(plan|release)-[0-9]+$`)
// planIDArguments are the arguments of plans that name a plan. wait is among them for when it is declared
// (novox/hq issue 229); an argument not declared is refused before this is asked.
var planIDArguments = []string{"id", "wait", "stop", "close", "retry", "go"}
// plainPlanIDs refuses a plans call naming a plan by anything but a plan's id (novox/hq issue 561): checked
// once, where the call becomes a command line, so nothing else a caller sent reaches a command or the store.
func plainPlanIDs(given map[string]string) error {
for _, arg := range planIDArguments {
if id, named := given[arg]; named && !planID.MatchString(id) {
return fmt.Errorf("plans %s: %q is not a plan's id (plan-<digits> or release-<digits>, as `plans` "+
"lists them); nothing was done", arg, id)
}
}
return nil
}
+37
View File
@@ -0,0 +1,37 @@
package main
import (
"strings"
"testing"
)
// **A plan's id is checked once, where a call becomes a command line** (novox/hq issue 561): every argument of
// plans that names a plan is refused unless it is an id the mesh gives a plan, so a quote never reaches a
// command, nor the store behind it.
func TestAPlanIdThatIsNoPlansIdIsRefusedBeforeACommandStarts(t *testing.T) {
for _, arg := range []string{"id", "stop", "close", "retry", "go"} {
for _, bad := range []string{"x' or '1'='1", "plan-1'", "plan-", "plan-1 2", "batch-1", "PLAN-1"} {
args := map[string]any{arg: bad}
if arg == "stop" || arg == "close" || arg == "go" {
args["why"] = "a test"
}
argv, err := argvFor("plans", args)
if err == nil {
t.Errorf("plans %s %q was composed: %v", arg, bad, argv)
continue
}
if !strings.Contains(err.Error(), "is not a plan's id") {
t.Errorf("plans %s %q was refused for another reason: %v", arg, bad, err)
}
}
for _, good := range []string{"plan-1791813626034291775", "release-1791811938267404408"} {
args := map[string]any{arg: good}
if arg == "stop" || arg == "close" || arg == "go" {
args["why"] = "a test"
}
if _, err := argvFor("plans", args); err != nil {
t.Errorf("plans %s %q was refused: %v", arg, good, err)
}
}
}
}
+158
View File
@@ -0,0 +1,158 @@
package main
import (
"context"
"encoding/json"
"fmt"
"os"
"strconv"
"strings"
"time"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// Following a plan (novox/hq issue 229): `plans wait <id>` holds its answer until the plan moves, reaches its
// next tier or ends, or until its bound passes, and says which. Before it, an agent following a rollout had
// `plans` answer once, in prose, and wrote a loop of its own round the mesh's tools to be told the plan moved.
//
// **No wait is longer than a call is answered within** (link.LongestAnswer, inside the console's thirty
// seconds): a longer one would answer "still running" and leave its caller to ask `calls` — the loop again.
// So a caller following a plan to its end asks again; each call answers the moment the plan moves.
// maxPlanWait is the longest bound a wait takes, and the bound when none is given: what a call is answered
// within, less the time a verb's command takes to start and read the store.
var maxPlanWait = link.LongestAnswer - planWaitStart
// planWaitStart is what the verb's window allows beyond the bound: starting the command and opening the stores.
// A store slow enough to push the read and the answer past it degrades as any long call does: the caller is
// answered that the call is still running, and `calls` with its id says what came of it.
const planWaitStart = 5 * time.Second
// planWaitEvery is how often a wait reads the plan again.
const planWaitEvery = time.Second
// The things a wait waits until.
var planWaitUntil = []string{"moved", "tier", "done"}
// planWaitAsked checks what a wait is asked for: until (moved when empty) and a bound (a duration, or a number
// of seconds; maxPlanWait when empty).
func planWaitAsked(until, bound string) (string, time.Duration, error) {
if until == "" {
until = "moved"
}
known := false
for _, u := range planWaitUntil {
known = known || u == until
}
if !known {
return "", 0, fmt.Errorf("a wait is until %s, not %q", strings.Join(planWaitUntil, ", "), until)
}
d := maxPlanWait
if bound = strings.TrimSpace(bound); bound != "" {
var err error
if secs, isNumber := strconv.Atoi(bound); isNumber == nil {
d = time.Duration(secs) * time.Second
} else if d, err = time.ParseDuration(bound); err != nil {
return "", 0, fmt.Errorf("the bound %q is not a duration (such as 20s) or a number of seconds", bound)
}
if d <= 0 || d > maxPlanWait {
return "", 0, fmt.Errorf("the bound is at most %s, the longest a call is answered within; %s was asked "+
"— to follow a plan longer, ask again: each call answers the moment the plan moves", maxPlanWait, d)
}
}
return until, d, nil
}
// planEnded is a plan that will not move again: done, failed or superseded. A batch being assembled is not one.
func planEnded(p inventory.Plan) bool { return !p.Open() && !p.Batch() }
// waitOnPlan reads the plan until what it waits for happens or the bound passes, and answers the plan as last
// read and what ended the wait: moved (any save of it), tier (its tier changed), done (it ended — which ends
// every wait, since nothing comes after it), or bound. Moved and tier are judged against the plan as first read.
func waitOnPlan(ctx context.Context, read func(context.Context) (inventory.Plan, error), until string,
bound, every time.Duration) (inventory.Plan, string, error) {
first, err := read(ctx)
if err != nil {
return inventory.Plan{}, "", err
}
if planEnded(first) {
return first, "done", nil
}
deadline := time.Now().Add(bound)
tick := time.NewTicker(every)
defer tick.Stop()
last := first
for {
select {
case <-ctx.Done():
return last, "", ctx.Err()
case <-tick.C:
}
if !time.Now().Before(deadline) {
return last, "bound", nil
}
p, err := read(ctx)
if err != nil {
return last, "", err
}
last = p
switch {
case planEnded(p):
return p, "done", nil
case until == "moved" && p.Revision != first.Revision:
return p, "moved", nil
case until == "tier" && p.Tier != first.Tier:
return p, "tier", nil
}
}
}
// planWaited is what `plans wait` answers: the plan, what ended the wait, and the line a person reads (`text`,
// which the verb answers as its output and not again in the data).
type planWaited struct {
// Came is what ended the wait: moved, tier, done, or bound when nothing did.
Came string `json:"came"`
Until string `json:"until"`
Waited string `json:"waited"`
Plan planStatus `json:"plan"`
// Revision is the plan's as read: a caller asking again with the same plan sees whether it moved since.
Revision int64 `json:"revision"`
Text string `json:"text"`
}
func planWaitedDoc(p inventory.Plan, st planStatus, until, came string, waited time.Duration, line string) planWaited {
said := map[string]string{
"moved": "moved", "tier": "reached another tier", "done": "ended",
"bound": "did not " + map[string]string{"moved": "move", "tier": "reach another tier", "done": "end"}[until],
}[came]
text := fmt.Sprintf("%s %s in %s (waited until %s)\n%s — %s\n", p.ID, said, waited.Round(time.Second), until,
p.ID, line)
return planWaited{Came: came, Until: until, Waited: waited.Round(time.Millisecond).String(), Plan: st,
Revision: p.Revision, Text: text}
}
// planWaitCommand is `plans wait <id>`: the plan followed until it moves, or the bound passes.
func planWaitCommand(ctx context.Context, inv *inventory.Inventory, id, untilAsked, boundAsked string) error {
until, bound, err := planWaitAsked(untilAsked, boundAsked)
if err != nil {
return err
}
began := time.Now()
p, came, err := waitOnPlan(ctx, func(ctx context.Context) (inventory.Plan, error) {
return inv.PlanByID(ctx, id)
}, until, bound, planWaitEvery)
if err != nil {
return err
}
// Said as `plans <id>` says its first line, and as `status` says the plan.
now, one := time.Now(), []inventory.Plan{p}
bounds, pause := readTierBounds(ctx, inv, now), buildSeatPause(ctx, inv, one)
line := planLineSaid(p, now, pause, bounds.of(p.Repository), func(module string) []string {
on, _ := inv.Running(ctx, module)
return on
}, waitsOf(ctx, inv, one))
doc := planWaitedDoc(p, planStatuses(one, now, pause, bounds)[0], until, came, now.Sub(began), line)
return json.NewEncoder(os.Stdout).Encode(doc)
}
+196
View File
@@ -0,0 +1,196 @@
package main
import (
"context"
"encoding/json"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/inventory"
)
// A plan as the store would answer it at each read: the reads are counted, and each answers the next.
type planReads struct {
plans []inventory.Plan
n int
}
func (r *planReads) read(context.Context) (inventory.Plan, error) {
p := r.plans[min(r.n, len(r.plans)-1)]
r.n++
return p, nil
}
func followedPlan(rev int64, state string, tier int) inventory.Plan {
return inventory.Plan{ID: "plan-1", Repository: "novox/x", Commit: "abc", State: state, Tier: tier,
Tiers: [][]string{{"a"}, {"b"}}, Revision: rev}
}
// **`plans` waits for a plan, and says what ended the wait** (novox/hq issue 229): an agent following a
// rollout is told when the plan moved, reached its next tier or ended, or that the bound passed — in one
// call, without a loop of its own.
func TestAWaitForAPlanEndsWhenWhatItWaitsForHappens(t *testing.T) {
cases := []struct {
name string
until string
reads []inventory.Plan
came string
times int
}{
{"moved: any save", "moved", []inventory.Plan{
followedPlan(1, inventory.PlanBuilding, 0), followedPlan(1, inventory.PlanBuilding, 0),
followedPlan(2, inventory.PlanBuilding, 0)}, "moved", 3},
{"tier: a save inside the tier is not enough", "tier", []inventory.Plan{
followedPlan(1, inventory.PlanBuilding, 0), followedPlan(2, inventory.PlanBuilding, 0),
followedPlan(3, inventory.PlanRolling, 1)}, "tier", 3},
{"tier: a plan that ended has no next tier to wait for", "tier", []inventory.Plan{
followedPlan(1, inventory.PlanBuilding, 0), followedPlan(2, inventory.PlanFailed, 0)}, "done", 2},
{"done: a new tier is not the end", "done", []inventory.Plan{
followedPlan(1, inventory.PlanBuilding, 0), followedPlan(2, inventory.PlanRolling, 1),
followedPlan(3, inventory.PlanDone, 1)}, "done", 3},
{"done: a plan already ended answers at once", "done", []inventory.Plan{
followedPlan(4, inventory.PlanSuperseded, 1)}, "done", 1},
{"the bound passes", "moved", []inventory.Plan{followedPlan(1, inventory.PlanBuilding, 0)}, "bound", -1},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
r := &planReads{plans: c.reads}
p, came, err := waitOnPlan(context.Background(), r.read, c.until, 50*time.Millisecond, time.Millisecond)
if err != nil {
t.Fatal(err)
}
if came != c.came {
t.Fatalf("the wait ended as %q, want %q", came, c.came)
}
if c.times > 0 && r.n != c.times {
t.Errorf("read the plan %d times, want %d: it answered before or after what it waited for", r.n, c.times)
}
if want := c.reads[min(r.n, len(c.reads))-1]; p.Revision != want.Revision {
t.Errorf("answered revision %d, not the plan as it was last read (%d)", p.Revision, want.Revision)
}
})
}
}
// What a wait is asked for is checked before anything is read: what it waits until, and a bound no longer than
// a call is answered within.
func TestAWaitIsRefusedWhatItCannotDo(t *testing.T) {
for _, bad := range []struct{ until, bound string }{{"landed", ""}, {"", "1h"}, {"", "-3s"}, {"", "soon"}} {
if _, _, err := planWaitAsked(bad.until, bad.bound); err == nil {
t.Errorf("until %q, bound %q was taken", bad.until, bad.bound)
}
}
until, bound, err := planWaitAsked("", "")
if err != nil || until != "moved" || bound != maxPlanWait {
t.Fatalf("the defaults are %q %s %v, want moved and %s", until, bound, err, maxPlanWait)
}
if _, bound, _ := planWaitAsked("tier", "12"); bound != 12*time.Second {
t.Errorf("a bare number is seconds; read %s", bound)
}
}
// **The answer is data, with its prose as a field beside it** (novox/hq issue 229): the verb's `answer` carries
// the plan and what ended the wait, and its output is the line a person reads — never JSON inside a string.
func TestAWaitForAPlanAnswersAsData(t *testing.T) {
p := followedPlan(7, inventory.PlanRolling, 1)
st := planStatuses([]inventory.Plan{p}, time.Now(), pauseView{}, tierBounds{})[0]
doc := planWaitedDoc(p, st, "tier", "tier", 3*time.Second, "rolling, tier 2 of 2")
stdout, _ := json.Marshal(doc)
a := answerOf([]string{"plans", "wait", "plan-1"}, stdout, "", true)
data, ok := a.Answer.(map[string]any)
if !ok {
t.Fatalf("the answer is not data: %v", a.Answer)
}
if data["came"] != "tier" || data["revision"] != float64(7) {
t.Errorf("the data does not say what ended the wait: %v", data)
}
if plan, _ := data["plan"].(map[string]any); plan["state"] != "rolling" || plan["tier"] != float64(1) {
t.Errorf("the data does not carry the plan: %v", data["plan"])
}
if _, text := data["text"]; text {
t.Errorf("the prose is in the data too: %v", data)
}
if strings.Contains(a.Output, "{") || !strings.Contains(a.Output, "plan-1") || !strings.Contains(a.Output, "tier") {
t.Errorf("the output is not the line a person reads: %q", a.Output)
}
}
// The verb composes the wait, and asks for a window as long as its bound.
func TestThePlansVerbComposesAWait(t *testing.T) {
argv, err := argvFor("plans", map[string]any{"wait": "plan-1", "until": "done", "bound": "15s"})
if err != nil {
t.Fatal(err)
}
if got := strings.Join(argv, " "); got != "plans wait plan-1 --until done --bound 15s" {
t.Fatalf("composed %q", got)
}
if _, err := argvFor("plans", map[string]any{"until": "done"}); err == nil {
t.Error("until without wait was taken")
}
if _, err := argvFor("plans", map[string]any{"wait": "plan-1", "bound": "1h"}); err == nil {
t.Error("a bound past what a call can wait was taken")
}
}
// Live, with the store: `plans wait` run as the verb runs it answers when the plan reaches its next tier, as
// data, and says the bound passed when nothing moved (novox/hq issue 229).
func TestAPlanFollowedFromTheStoreAnswersWhenItMoves(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
now := time.Now().UTC()
walk := inventory.Plan{ID: "plan-229", Repository: "novox/mesh-catalog", Branch: "main", Commit: "c229c229",
Created: now, State: inventory.PlanBuilding, Tiers: [][]string{{"a"}, {"b"}},
Modules: map[string]*inventory.PlanModule{"a": {}, "b": {}}}
if err := open.inventory.SavePlan(ctx, &walk); err != nil {
t.Fatal(err)
}
moved := make(chan error, 1)
go func() {
time.Sleep(1500 * time.Millisecond)
p, err := open.inventory.PlanByID(ctx, walk.ID)
if err == nil {
p.Tier, p.State = 1, inventory.PlanRolling
err = open.inventory.SavePlan(ctx, &p)
}
moved <- err
}()
began := time.Now()
var got planWaited
out := printed(t, func() error {
return plansCommand(ctx, []string{"wait", walk.ID, "--until", "tier", "--bound", "15s"})
})
if err := <-moved; err != nil {
t.Fatal(err)
}
if err := json.Unmarshal([]byte(out), &got); err != nil {
t.Fatalf("plans wait did not answer data: %v: %q", err, out)
}
if got.Came != "tier" || got.Plan.Tier != 1 || got.Plan.State != inventory.PlanRolling {
t.Fatalf("the wait answered %+v", got)
}
if waited := time.Since(began); waited > 10*time.Second {
t.Errorf("it answered after %s, not when the plan moved", waited)
}
out = printed(t, func() error {
return plansCommand(ctx, []string{"wait", walk.ID, "--bound", "2"})
})
if err := json.Unmarshal([]byte(out), &got); err != nil || got.Came != "bound" || got.Until != "moved" {
t.Fatalf("a plan that did not move answered %+v (%v)", got, err)
}
}
// The verb's schema says the longest bound in words, and the words are the value (review 75 on #266): the
// catalogue cannot read maxPlanWait, so this holds them together.
func TestThePlansSchemaSaysTheLongestBoundThereIs(t *testing.T) {
v, ok := controllerVerb("plans")
if !ok {
t.Fatal("no plans verb")
}
props, _ := v.Input["properties"].(map[string]any)
bound, _ := props["bound"].(map[string]any)
said, _ := bound["description"].(string)
if !strings.Contains(said, maxPlanWait.String()+", the default, is the longest") {
t.Fatalf("the schema says %q; the longest bound is %s", said, maxPlanWait)
}
}
+3
View File
@@ -1027,6 +1027,9 @@ func (b overTheBus) declare(ctx context.Context, s readyNode, body []byte) (stri
if err != nil {
return "", err
}
// A send that carried the mesh's record of the controller and the node-engine — a person's push included —
// ends the condition of one kept over it (novox/hq issue 201).
settleCoreDowngrades(ctx, b.open.inventory, s.node, s.declared.Builds)
if len(s.declared.Bindings) > 0 {
// And where it bound each consumer of a provision that keeps its data (novox/hq ADR 0232):
// what the next resolution keeps it at. On the same outliving context as the send's record.
+17
View File
@@ -135,6 +135,17 @@ func recordedKept(ctx context.Context, open *stores, node string) (map[string]st
}
out[m] = was
}
// **And the controller and the node-engine at the build the machine was last sent, over an older record**
// (novox/hq issue 201, core_downgrade.go). Under keepingEveryBuild they are kept already.
if !every {
core, err := coreKept(ctx, inv, node, sent, current)
if err != nil {
return nil, err
}
for m, was := range core {
out[m] = was
}
}
return out, nil
}
@@ -159,6 +170,12 @@ func keepRecorded(ctx context.Context, open *stores, node string, shelf map[stri
return nil, fmt.Errorf("%s runs %s's build %s, which the build records no longer hold: the bus's user "+
"list cannot be sent there alone with it kept (novox/hq issue 490)", node, m, short(kept[m]))
}
if !found && guarded(coreComponent(m)) {
return nil, fmt.Errorf("%s was last sent %s's build %s, newer than the one the mesh holds, and the build records "+
"no longer hold it: this send cannot keep it and does not move it back — `push %s` sends the older build on "+
"a person's word, and a newer merge or `rebuild %s` moves the record forward (novox/hq issue 201)",
node, m, short(kept[m]), node, m)
}
if !found {
return nil, fmt.Errorf("%s records rather than rolls out, and %s runs its build %s, which the build "+
"records no longer hold: this send cannot keep it and does not move it — `push %s` sends the new "+
+51 -10
View File
@@ -51,6 +51,9 @@ type sendScope struct {
modules map[string]bool
// person is a person's act: it carries what it carries.
person bool
// puttingBack is a gate's own put-back of its modules: the one send that may carry a core module to a build
// older than the one its machine runs (novox/hq issue 201).
puttingBack bool
}
type sendScopeKey struct{}
@@ -237,6 +240,47 @@ var machineMoves = func(ctx context.Context, open *stores, f moveFacts, node str
return f.moves(node, modules, sent, known, all), nil
}
// withoutKept is a machine's moves less those its send keeps at the build the machine was last sent, and so does
// not make: the grants step keeps every module (novox/hq issue 490), and every send keeps the controller and the
// node-engine over an older record (issue 201). Read as the send composes it (sendKeeps, set after the checks that
// call this), so a move it could not keep is still counted.
func withoutKept(ctx context.Context, open *stores, node string, moves []inventory.CarriedMove) ([]inventory.CarriedMove, error) {
if len(moves) == 0 {
return moves, nil
}
keeping := ctx
if _, on := keptExcept(ctx); !on {
keeping = keepingRecorded(ctx)
}
keeps, err := recordedKept(keeping, open, node)
if err != nil {
return moves, err
}
// And a joining module held at the build it runs (novox/hq ADR 0328 §8): its new build is no move this send makes,
// and judging it would read the gate as superseded by its own send (the re-review of 96e15f86, item 1).
held, err := holdJoining(ctx, open.inventory, node, nil)
if err != nil {
return moves, err
}
if len(held) > 0 && keeps == nil {
keeps = map[string]string{}
}
for m, h := range held {
keeps[m] = h.commit
}
if len(keeps) == 0 {
return moves, nil
}
var out []inventory.CarriedMove
for _, mv := range moves {
if was, kept := keeps[mv.Module]; kept && sameCommit(was, mv.From) {
continue
}
out = append(out, mv)
}
return out, nil
}
// ungatedIn refuses a send whose machines would move a build no gate has seen, outside its scope: the
// machines named, and why; the machine holding the bus, added only for its user list, is left instead.
func ungatedIn(ctx context.Context, open *stores, names []string, addedHolder string) ([]string, error) {
@@ -254,19 +298,11 @@ func ungatedIn(ctx context.Context, open *stores, names []string, addedHolder st
if err != nil {
return nil, err
}
// The grants step composes every module at the build its machine was last sent (novox/hq issue 490):
// a move it keeps is not made. Read, not assumed, so a move it could not keep is still refused here.
var keeps map[string]string
if everyBuildKept(ctx, n) {
if keeps, err = recordedKept(ctx, open, n); err != nil {
return nil, err
}
if moves, err = withoutKept(ctx, open, n, moves); err != nil {
return nil, err
}
var waiting []string
for _, mv := range moves {
if was, kept := keeps[mv.Module]; kept && sameCommit(was, mv.From) {
continue
}
if !scope.judged[n] && !scope.modules[mv.Module] {
waiting = append(waiting, fmt.Sprintf("%s %s → %s", mv.Module, short(mv.From), short(mv.To)))
}
@@ -308,6 +344,11 @@ func gatedSend(ctx context.Context, open *stores, node string, owns []inventory.
if err != nil {
return nil, nil, nil, err
}
// A move the send keeps is not made, and is no move for the gate to judge (novox/hq issue 201): a core module
// kept over an older record judged as moving back to it would read the gate as superseded by its own send.
if moves, err = withoutKept(ctx, open, node, moves); err != nil {
return nil, nil, nil, err
}
own := func(module string) bool {
return slices.ContainsFunc(owns, func(o inventory.CarriedMove) bool { return o.Module == module })
}
+33 -4
View File
@@ -690,7 +690,7 @@ func advanceOnce(ctx context.Context, open *stores, p *inventory.Plan,
// being behind. Now every module of the tier whose first machine is the same is sent there in one
// send, judged there by one gate — kept on the first of them, the others gated by it — and the rest
// of the tier's machines are sent once, together, when the gates have passed.
var pending []string
var pending, waitsForAPerson []string
firstOn := map[string][]string{}
runningOf := map[string][]string{}
var rest []string
@@ -706,6 +706,20 @@ func advanceOnce(ctx context.Context, open *stores, p *inventory.Plan,
return false, err
}
runningOf[m] = running
// **A joining build held for a person is not sent** (novox/hq ADR 0328 §8, the re-review of 96e15f86): its
// machines keep the build they run until its library is taken (or their engine keeps own groups), so sending
// it would send nothing new and judge the old build. It waits for a person, said; the plan stays open, the other
// modules are sent and judged, and it is sent on the walk after nothing holds it — no new merge.
if state.FirstAt == nil {
why, err := heldIn(ctx, inv, m, running)
if err != nil {
return false, err
}
if why != "" {
waitsForAPerson = append(waitsForAPerson, why)
continue
}
}
// **A build whose source is unchanged is never a move** (novox/hq issue 280): made from the source
// of the build every machine running it was sent, it is registered with that build's artifacts —
// nothing to send, and nothing for a gate to judge.
@@ -868,9 +882,17 @@ func advanceOnce(ctx context.Context, open *stores, p *inventory.Plan,
strings.Join(sent, ", "))
return true, nil
}
if len(pending) > 0 {
note := "tier " + fmt.Sprint(p.Tier) + " built; waiting for " + strings.Join(pending, "; ") +
" to report it applied before the rest are sent"
if len(pending) > 0 || len(waitsForAPerson) > 0 {
note := "tier " + fmt.Sprint(p.Tier) + " built; "
if len(pending) > 0 {
note += "waiting for " + strings.Join(pending, "; ") + " to report it applied before the rest are sent"
}
if len(waitsForAPerson) > 0 {
if len(pending) > 0 {
note += "; and "
}
note += "waiting for a person: " + strings.Join(waitsForAPerson, "; ")
}
changed := p.State != inventory.PlanRolling || p.Note != note
p.State = inventory.PlanRolling
p.Note = note
@@ -1463,6 +1485,10 @@ func plansCommand(ctx context.Context, args []string) error {
modules := set.String("modules", "", "or the modules it would change, comma-separated")
moduleDirs := set.String("module-dirs", "", "with --paths: the directories holding a module.json at the commit, "+
"comma-separated, as the forge's announcer says them (novox/hq issue 278)")
// Following a plan (novox/hq issue 229): `plans wait <id>`.
until := set.String("until", "", "with wait: what to wait for — moved (any change), tier (its next tier) or done (its end)")
bound := set.String("bound", "", "with wait: how long to wait at most, as 15s or seconds (default and at most the "+
"longest a call is answered within)")
// Ending a plan by hand is a repair, and says why (novox/hq to-be 45 §7).
why := addHandActFlags(set)
positionals, err := parseAround(set, args)
@@ -1482,6 +1508,9 @@ func plansCommand(ctx context.Context, args []string) error {
defer open.Close()
inv := open.inventory
now := time.Now()
if len(positionals) == 2 && positionals[0] == "wait" {
return planWaitCommand(ctx, inv, positionals[1], *until, *bound)
}
if len(positionals) == 1 {
p, err := inv.PlanByID(ctx, positionals[0])
if err != nil {
+83
View File
@@ -0,0 +1,83 @@
package main
import (
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
)
// TestReplay401 replays novox/hq issue 401 (2026-10-10): the mounts module's smb-credentials, an own secret
// only an outside party issues, was given by nobody, and the send made a random value for it and placed it on
// the machine; the module read it as a credential given and wrong. The outcome asserted (ADR 0228): the send
// makes no value for an own secret issued outside the mesh that nobody gave and places none, and a value the
// mesh made for it before is not placed either; once a person gives one, that value is placed. Written with
// only what the controller had before its fix (mesh-controller #262) — register, assign, planFor,
// declarationFor and the inventory's secret reads — so it is laid over the commit before. Needs the store
// (MESH_TEST_POSTGRES), and is named in testdata/store-tests.
func TestReplay401(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
const made, outside = "/var/lib/mesh/mounts/password", "/var/lib/mesh/mounts/smb-credentials"
register(t, open, catalogue.Manifest{Module: "mounts", Version: "1",
OwnSecrets: catalogue.OwnSecrets{
"password": {Path: made, Taken: catalogue.TakenAtStart},
"smb-credentials": {Path: outside, Taken: catalogue.TakenAtStart, IssuedBy: catalogue.IssuedOutside},
},
Resources: []map[string]any{
{"id": "state", "type": "directory", "path": "/var/lib/mesh/mounts", "mode": "0700"},
}})
if _, err := assign(ctx, open, "laptop", "mounts"); err != nil {
t.Fatal(err)
}
sent := func() []map[string]any {
t.Helper()
plan, settings, err := planFor(ctx, open, "laptop")
if err != nil {
t.Fatal(err)
}
declared, err := declarationFor(ctx, open, "laptop", plan, settings)
if err != nil {
t.Fatalf("the send to laptop did not compose: %v", err)
}
return declared.Resources
}
placed := func(resources []map[string]any, path string) map[string]any {
for _, r := range resources {
if r["path"] == path && r["sealed"] != nil {
return r
}
}
return nil
}
first := sent()
if placed(first, made) == nil {
t.Fatal("the secret the mesh makes was not placed")
}
if got := placed(first, outside); got != nil {
t.Errorf("the send placed a value for smb-credentials, which nobody gave: %v", got)
}
if _, held, err := open.inventory.ModuleSecretIfIssued(ctx, "laptop", "mounts", "smb-credentials"); err != nil || held {
t.Fatalf("the send made a value for a secret issued outside the mesh (held %v, %v)", held, err)
}
// A value the mesh made for it before the rule is not one a person gave: it is not placed.
if _, err := open.inventory.SecretForModule(ctx, "laptop", "mounts", "smb-credentials"); err != nil {
t.Fatal(err)
}
if got := placed(sent(), outside); got != nil {
t.Fatalf("a value the mesh made for a secret issued outside was placed: %v", got)
}
// Given by a person, it is placed, as given.
if err := open.inventory.AcceptSecretForModule(ctx, "laptop", "mounts", "smb-credentials", "the share's credentials"); err != nil {
t.Fatal(err)
}
given, held, err := open.inventory.ModuleSecretIfIssued(ctx, "laptop", "mounts", "smb-credentials")
if err != nil || !held {
t.Fatalf("the value given is not held (%v)", err)
}
if got := placed(sent(), outside); got == nil || got["sealed"] != given {
t.Fatalf("the value given is placed as %v", got)
}
}
+41 -8
View File
@@ -34,8 +34,10 @@ type seatRow struct {
Delivers string `json:"delivers,omitempty"`
Decision string `json:"decision"`
// Replicated says the seat may be held on several machines at once (novox/hq ADR 0223).
Replicated bool `json:"replicated,omitempty"`
Holders []seatHolder `json:"holders"`
Replicated bool `json:"replicated,omitempty"`
// Paths are the paths on its machine the seat's holder owns (novox/hq ADR 0287).
Paths []catalogue.SeatPath `json:"paths,omitempty"`
Holders []seatHolder `json:"holders"`
}
// seatsHeld is every seat the mesh defines with its holders, and every claim held that names no
@@ -49,7 +51,7 @@ func seatsHeld(seats []catalogue.Seat, held []catalogue.Held) ([]seatRow, []cata
rows := make([]seatRow, 0, len(seats))
for _, s := range seats {
row := seatRow{Seat: s.Name, Scope: s.Scope, Delivers: s.Delivers, Decision: s.Decision,
Replicated: s.Replicated, Holders: []seatHolder{}}
Replicated: s.Replicated, Paths: s.Paths, Holders: []seatHolder{}}
seen := map[seatHolder]bool{}
for _, h := range held {
// Resolve the held claim to a seat rather than comparing names, so a record naming a
@@ -255,12 +257,16 @@ func seatsCommand(ctx context.Context, args []string) error {
return err
}
rows, outside := seatsHeld(catalogue.Seats(), world.Held)
// Which module owns each seat's path on each machine (novox/hq ADR 0287): the seat naming it, and
// that seat's holder there, with the stepping back applied.
owners := catalogue.PathOwners(catalogue.Seats(), world.Held)
if *asJSON {
out := struct {
Seats []seatRow `json:"seats"`
Outside []catalogue.Held `json:"outside,omitempty"`
}{rows, outside}
Seats []seatRow `json:"seats"`
Owners []catalogue.PathOwner `json:"path-owners,omitempty"`
Outside []catalogue.Held `json:"outside,omitempty"`
}{rows, owners, outside}
body, err := json.MarshalIndent(out, "", " ")
if err != nil {
return err
@@ -270,7 +276,7 @@ func seatsCommand(ctx context.Context, args []string) error {
}
w := tabwriter.NewWriter(os.Stdout, 0, 0, 2, ' ', 0)
fmt.Fprintln(w, "SEAT\tSCOPE\tDELIVERS\tHELD BY")
fmt.Fprintln(w, "SEAT\tSCOPE\tDELIVERS\tPATHS\tHELD BY")
for _, r := range rows {
delivers := r.Delivers
if delivers == "" {
@@ -284,11 +290,22 @@ func seatsCommand(ctx context.Context, args []string) error {
}
holders = strings.Join(parts, ", ")
}
fmt.Fprintf(w, "%s\t%s\t%s\t%s\n", r.Seat, r.Scope, delivers, holders)
fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\n", r.Seat, r.Scope, delivers, seatPathsText(r.Paths), holders)
}
if err := w.Flush(); err != nil {
return err
}
if len(owners) > 0 {
fmt.Println("\nwho owns each seat's path, by machine (novox/hq ADR 0287):")
w = tabwriter.NewWriter(os.Stdout, 0, 0, 2, ' ', 0)
fmt.Fprintln(w, "PATH\tMACHINE\tSEAT\tMODULE")
for _, o := range owners {
fmt.Fprintf(w, "%s\t%s\t%s\t%s\n", o.Path, o.Node, o.Seat, o.Module)
}
if err := w.Flush(); err != nil {
return err
}
}
if len(outside) > 0 {
fmt.Println("\nheld, and not a seat this mesh defines (registered before the set closed — novox/hq ADR 0110):")
for _, h := range outside {
@@ -297,3 +314,19 @@ func seatsCommand(ctx context.Context, args []string) error {
}
return nil
}
// seatPathsText is a seat's paths in one column: each path, and the seat it steps back for, if any.
func seatPathsText(paths []catalogue.SeatPath) string {
if len(paths) == 0 {
return "—"
}
parts := make([]string, 0, len(paths))
for _, p := range paths {
if p.StepsBackFor != "" {
parts = append(parts, p.Path+" (steps back for "+p.StepsBackFor+")")
continue
}
parts = append(parts, p.Path)
}
return strings.Join(parts, ", ")
}
+19
View File
@@ -74,3 +74,22 @@ func TestAKindedBenchIsNotHandedOver(t *testing.T) {
}
}
}
// The listing shows each seat's paths, and the seat a path steps back for (novox/hq ADR 0287).
func TestTheSeatsListingShowsEachSeatsPaths(t *testing.T) {
rows, _ := seatsHeld(catalogue.Seats(), nil)
got := map[string]string{}
for _, r := range rows {
got[r.Seat] = seatPathsText(r.Paths)
}
for seat, want := range map[string]string{
"node-hostname": "/etc/hosts, /etc/hostname",
"node-uplink": "/etc/resolv.conf (steps back for node-resolver)",
"node-resolver": "/etc/resolv.conf",
"node-packet-filter": "—",
} {
if got[seat] != want {
t.Errorf("%s lists its paths as %q, not %q", seat, got[seat], want)
}
}
}
+32 -2
View File
@@ -336,6 +336,9 @@ func (a *verbArguments) commandLine() ([]string, error) {
}
return argv, nil
case "plans":
if err := plainPlanIDs(a.given); err != nil {
return nil, err
}
if r := str("repository"); r != "" {
argv := []string{"plans", "--what-if", r}
if p := str("paths"); p != "" {
@@ -365,6 +368,21 @@ func (a *verbArguments) commandLine() ([]string, error) {
return argv, nil
}
}
if id := str("wait"); id != "" {
// Following a plan (novox/hq issue 229): checked here too, so a call asking what no wait can do is
// refused before a command is started.
if _, _, err := planWaitAsked(a.given["until"], a.given["bound"]); err != nil {
return nil, err
}
argv := []string{"plans", "wait", id}
if u := str("until"); u != "" {
argv = append(argv, "--until", u)
}
if b := str("bound"); b != "" {
argv = append(argv, "--bound", b)
}
return argv, nil
}
if id := str("id"); id != "" {
return []string{"plans", id}, nil
}
@@ -985,12 +1003,14 @@ var jsonVerbs = map[string]bool{"status": true, "seats": true, "plan": true, "co
// The delivery's owner's verbs answer JSON where they read (plan, order, check, walks) — novox/hq ADR 0239.
"delivery": true,
// The build queue, each ask with its position among the waiting (novox/hq issue 455).
"queue": true}
"queue": true,
// A plan followed (novox/hq issue 229): `plans wait` answers JSON; its other forms say text, and so answer no data.
"plans": true}
// tableBesideVerbs are the JSON verbs whose document carries the table a person reads as its `text`: the
// verb answers that table as its output, as it did before it answered data, and the rest as `answer` —
// both from the one read, so they never disagree (novox/hq issue 455).
var tableBesideVerbs = map[string]bool{"queue": true}
var tableBesideVerbs = map[string]bool{"queue": true, "plans": true}
// overviewVerbs are the JSON verbs whose answer is said once, as data, and not again as the text the
// command printed: the overviews, which grow with the mesh (novox/hq issue 314). Every reader of theirs
@@ -1264,6 +1284,16 @@ func seatToolHandlers() (map[string]link.ToolHandler, []string, error) {
// Whatever it did, `status` is composed again once it has.
defer statusFrom.nudge()
}
if verb == "plans" && len(argv) > 1 && argv[1] == "wait" {
// A wait holds its answer up to its bound (novox/hq issue 229), and is answered in full inside it.
asked := ""
if i := slices.Index(argv, "--bound"); i > 0 && i+1 < len(argv) {
asked = argv[i+1]
}
if _, bound, err := planWaitAsked("", asked); err == nil {
link.AnswersWithin(ctx, bound+planWaitStart)
}
}
if answer, read := readHere(ctx, argv); read {
return answer, nil
}
+15
View File
@@ -18,6 +18,12 @@ import (
// make a machine look out of date for ever, or send something `plan` never showed.
type sendable struct {
Resources []map[string]any
// Groups, Identities and Libraries are what the declaration says of libraries and joining modules (novox/hq ADR
// 0328): composed only for an engine that keeps libraries, and not sent at all when empty — an older engine
// refuses a key it does not know, whole.
Groups []catalogue.GroupEntry
Identities []catalogue.IdentityEntry
Libraries []catalogue.LibraryEntry
// Sequence orders this send against every other to the same node: one higher each time, taken
// under the node's hold just before the body is made (novox/hq 04-ISSUES/107). Zero is not sent
// at all, which a host reads as "no order claimed" — the shape of every declaration before this.
@@ -110,6 +116,15 @@ func (s sendable) Body() ([]byte, error) {
if s.Generation > 0 {
envelope["generation"] = s.Generation
}
if len(s.Groups) > 0 {
envelope["groups"] = s.Groups
}
if len(s.Identities) > 0 {
envelope["identities"] = s.Identities
}
if len(s.Libraries) > 0 {
envelope["libraries"] = s.Libraries
}
// An empty declaration is deliberate here — the node owns nothing the mesh put there
// (novox/hq issue 127) — and the host refuses an empty body unless it is told the emptiness
// is meant, so a truncated or mis-composed body is never mistaken for "own nothing".
+2 -1
View File
@@ -8,6 +8,7 @@ require (
github.com/nats-io/nats-server/v2 v2.11.17
github.com/nats-io/nats.go v1.54.0
github.com/novox/mesh-host v0.0.0
github.com/therootcompany/xz v1.0.1
golang.org/x/crypto v0.57.0
golang.org/x/net v0.58.0
golang.org/x/sys v0.48.0
@@ -35,4 +36,4 @@ require (
// committed. Every build (the build agent's `go build`, the Dockerfile) compiles from vendor/ and
// fetches nothing; go refuses to build when vendor/ and this file disagree, so a pin moved without
// `go mod vendor` fails loudly, at once, everywhere.
replace github.com/novox/mesh-host => git.novox.be/novox/mesh-host v0.0.0-20261011092230-20d5af9b2d5f
replace github.com/novox/mesh-host => git.novox.be/novox/mesh-host v0.0.0-20261012222214-7a41e120635b
+4 -2
View File
@@ -1,5 +1,5 @@
git.novox.be/novox/mesh-host v0.0.0-20261011092230-20d5af9b2d5f h1:8N5OW2mdTNIck2pe4EciTYX5NsrPsHrTLENGNIWYNTU=
git.novox.be/novox/mesh-host v0.0.0-20261011092230-20d5af9b2d5f/go.mod h1:tcTK4LMs1d6JpZUwy3hSHMFG/MIuDr/XFs7/nbeaW/c=
git.novox.be/novox/mesh-host v0.0.0-20261012222214-7a41e120635b h1:PGdZJHUU9rCDucdrZIqSs4DGyxlo4lokvwmofpGk9RI=
git.novox.be/novox/mesh-host v0.0.0-20261012222214-7a41e120635b/go.mod h1:tcTK4LMs1d6JpZUwy3hSHMFG/MIuDr/XFs7/nbeaW/c=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261010191001-33917f91ac3c h1:l5onJwoIeH8yE/PjVlEeoPyXBsHp2NQiWLllz2fwX7s=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261010191001-33917f91ac3c/go.mod h1:GFuZUElBZ9A++mxgIKo97aXXo+kV0uJ/UkbhQPPIbrY=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
@@ -38,6 +38,8 @@ github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UV
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
github.com/therootcompany/xz v1.0.1 h1:CmOtsn1CbtmyYiusbfmhmkpAAETj0wBIH6kCYaX+xzw=
github.com/therootcompany/xz v1.0.1/go.mod h1:3K3UH1yCKgBneZYhuQUvJ9HPD19UEXEI0BWbMn8qNMY=
golang.org/x/crypto v0.57.0 h1:3ZVCjf8Ggz7zneR/EHRVx68Ctf+2pmIMP2UFhh9cC6M=
golang.org/x/crypto v0.57.0/go.mod h1:Fdz0i5U6CoizGwLda9DttjSk6qlZo25zYNtR+ycvuZA=
golang.org/x/net v0.58.0 h1:ynWG7rqYi4ccpTEuPZ2QGWHktVEM9DMCj9yzDE0Q7To=
+1 -1
View File
@@ -82,7 +82,7 @@ func theCarriedAccounts(t *testing.T) string {
// the commit the mesh runs, or as captured in testdata/beside (internal/beside, novox/hq issue 432).
func theTemplate(t *testing.T) []map[string]any {
t.Helper()
path := filepath.Join(beside.Dir(t, "mesh-host"), "examples", "foundation-first-node-nats.lock")
path := filepath.Join(beside.Dir(t, "mesh-host"), "examples", "foundation-first-node.lock")
raw, err := os.ReadFile(path)
if err != nil {
t.Fatal(err)
+8 -1
View File
@@ -343,6 +343,11 @@ func AskJudgeDirectoriesSubject(node string) string {
return "mesh.node." + node + ".ask.judge-directories"
}
// AskAccessActSubject is where the controller asks one machine's node-engine to take a shared folder on the
// operator's warrant (novox/hq ADR 0325 §8; mesh-host internal/link AskAccessActSubject): a request answered once,
// signed as a hand-over is (link.AccessActContext), for the same reason.
func AskAccessActSubject(node string) string { return "mesh.node." + node + ".ask.access-act" }
// inbox is a principal's own reply space. No user is ever granted a bare `_INBOX.>` (design 25
// §4): with one account, inbox privacy is the permission list or it is nothing, so each user's
// inbox is derived from its own identity and its permissions name that prefix and no other.
@@ -623,7 +628,9 @@ func PermissionsFor(p Principal) (Permissions, error) {
// And asking it for a fresh search for setuid programs (novox/hq issue 361), answered the same way.
AskSetuidSearchSubject(p.Node),
// And asking it to judge its declared directories, read-only (novox/hq issue 358), answered the same way.
AskJudgeDirectoriesSubject(p.Node)}
AskJudgeDirectoriesSubject(p.Node),
// And asking it to take a shared folder on the operator's warrant (novox/hq ADR 0325 §8), answered the same way.
AskAccessActSubject(p.Node)}
// The node-engine witnesses the core builds it places (novox/hq to-be 45 §8, ADR 0236; the
// contract is lease/witness.go): it asks its own machine's node tools PING, and, where the
// machine runs the controller, reads the lease's one key — read, never written.
+1 -1
View File
@@ -34,7 +34,7 @@ accounts {
} }
{ user: "node.one", password: "$2a$11$nnnnnnnnnnnnnnnnnnnnnn", permissions: {
publish: { allow: ["$JS.ACK.NODES.one.>", "$JS.API.CONSUMER.INFO.NODES.one", "$SRV.PING.node-tools.one", "mesh.control.one.>"] }
subscribe: { allow: ["_DELIVER.one", "_DELIVER.one.>", "_INBOX.node.one.>", "mesh.node.one.ask.hand-over", "mesh.node.one.ask.judge-directories", "mesh.node.one.ask.report", "mesh.node.one.ask.setuid-search", "mesh.node.one.declare"] }
subscribe: { allow: ["_DELIVER.one", "_DELIVER.one.>", "_INBOX.node.one.>", "mesh.node.one.ask.access-act", "mesh.node.one.ask.hand-over", "mesh.node.one.ask.judge-directories", "mesh.node.one.ask.report", "mesh.node.one.ask.setuid-search", "mesh.node.one.declare"] }
allow_responses: { max: 1, ttl: "1m" }
} }
{ user: "one.nats", password: "$2a$11$bbbbbbbbbbbbbbbbbbbbbb", permissions: {
+5
View File
@@ -101,6 +101,11 @@ var WritersTable = []WriterRow{
{State: "a judgement of a machine's directories asked of it", Writer: "controller",
KeptIn: "nowhere: the engine answers it and keeps nothing", Others: "the engine verifies the mesh's signature and answers, or refuses",
Subjects: []string{"mesh.node.*.ask.judge-directories"}, Writes: isController},
// The take of a shared folder asked of a machine's engine on the operator's warrant (novox/hq ADR 0325 §8):
// signed as a hand-over is, under a signing context of its own.
{State: "the take of a shared folder asked of a machine", Writer: "controller, on the operator's warrant",
KeptIn: "the machine, which records the act", Others: "the engine verifies the mesh's signature and takes it, or refuses",
Subjects: []string{"mesh.node.*.ask.access-act"}, Writes: isController},
{State: "a machine's applied state and its report", Writer: "the node-engine's apply queue",
KeptIn: "the machine; the report on the bus", Others: "the reconcile and a delivery enqueue, never apply",
// And its health statement between reports (novox/hq ADR 0240): the same writer stating the same
+1
View File
@@ -18,6 +18,7 @@ var designRows = []string{
"a hand-over asked of a machine",
"a fresh setuid search asked of a machine",
"a judgement of a machine's directories asked of it",
"the take of a shared folder asked of a machine",
"a machine's applied state and its report",
"the controller lease",
"plans and their tiers",
+4 -2
View File
@@ -939,14 +939,16 @@ func replaceGoFiles(from, into string) error {
}
// problemLines are the lines of a module check's output that name a problem: not a manifest found ok, not
// the count, not the closing note.
// the count, not the closing note, not a warning.
func problemLines(s string) []string {
var out []string
for _, line := range strings.Split(s, "\n") {
line = strings.TrimSpace(line)
switch {
case line == "", strings.Contains(line, ": ok"), strings.Contains(line, "problem(s) in"),
strings.Contains(line, "manifest(s) checked"):
strings.Contains(line, "manifest(s) checked"),
// A warning is said and fails nothing (novox/hq ADR 0325: what is refused once the catalogues moved).
strings.HasPrefix(line, "warning: "):
continue
}
out = append(out, line)
Binary file not shown.
+19 -40
View File
@@ -14,13 +14,13 @@ import (
"io"
"net/http"
"os"
"os/exec"
"path"
"path/filepath"
"strings"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/therootcompany/xz"
)
// An archive whose source is somebody else's published file, pinned by URL and sha256 (novox/hq ADR 0316
@@ -183,7 +183,6 @@ func unpackUpstream(ctx context.Context, file, into, within string) error {
buffered := bufio.NewReader(f)
magic, _ := buffered.Peek(6)
var stream io.Reader
var wait func() error
switch {
case bytes.HasPrefix(magic, []byte{0x1f, 0x8b}):
zipped, err := gzip.NewReader(buffered)
@@ -195,53 +194,33 @@ func unpackUpstream(ctx context.Context, file, into, within string) error {
case bytes.HasPrefix(magic, []byte("BZh")):
stream = bzip2.NewReader(buffered)
case bytes.HasPrefix(magic, []byte{0xfd, '7', 'z', 'X', 'Z', 0x00}):
// Go's standard library reads no xz; the build seat's image carries xz (cmd/mesh-builder/Dockerfile).
if _, err := exec.LookPath("xz"); err != nil {
return errors.New("it is compressed with xz, and this build seat has no xz to read it")
// Read in Go (github.com/therootcompany/xz, a port of xz-embedded), never by a tool the build seat's image
// may not carry: the seat's image is the catalogue's build-agent recipe, which installs git and docker-cli
// alone, and a release pinned as .tar.xz failed there on 2026-10-12 (hq issue 345). xzDictMax is a real
// maximum: a block whose LZMA2 dictionary is larger is refused before its dictionary is allocated
// (ErrMemlimit). The reader keeps a filter decoder for every block of a stream, so the file's framing is
// walked first and a file of more blocks than xzBlocksMax is refused before anything is decoded
// (xz_framing.go). What the stream unpacks to is bounded by untar (maxUpstream, maxEntries).
if err := xzWithinLimits(file); err != nil {
return err
}
cmd := exec.CommandContext(ctx, "xz", "-dc")
cmd.Stdin = buffered
var stderr bytes.Buffer
cmd.Stderr = &stderr
out, err := cmd.StdoutPipe()
zipped, err := xz.NewReader(buffered, xzDictMax)
if err != nil {
return err
return fmt.Errorf("it is not an xz stream this build seat reads (a dictionary of %d MiB at most): %v",
xzDictMax>>20, err)
}
if err := cmd.Start(); err != nil {
return err
}
stream = out
wait = func() error {
if err := cmd.Wait(); err != nil {
return fmt.Errorf("xz could not read it: %v %s", err, strings.TrimSpace(stderr.String()))
}
return nil
}
defer func() {
if wait != nil {
_ = cmd.Process.Kill()
_ = cmd.Wait()
}
}()
stream = zipped
default:
return errors.New("it is not a tar archive compressed with gzip, bzip2 or xz; a single file is kept " +
"whole by naming it in `file`")
}
if err := untar(stream, into, within); err != nil {
return err
}
if wait != nil {
// Whatever follows the tar's end is read before xz is waited for: xz blocks on a full pipe otherwise.
if _, err := io.Copy(io.Discard, stream); err != nil {
return err
}
w := wait
wait = nil
return w()
}
return nil
return untar(stream, into, within)
}
// xzDictMax is the largest xz dictionary read: xz's own presets use at most 64 MiB (-9); a stream asking for
// more is refused rather than given the memory.
const xzDictMax = 64 << 20
// untar writes a tar stream's files and directories within `within` into a directory.
func untar(stream io.Reader, into, within string) error {
reader := tar.NewReader(stream)
+84
View File
@@ -6,8 +6,10 @@ import (
"compress/gzip"
"context"
"crypto/sha256"
"encoding/binary"
"encoding/hex"
"fmt"
"hash/crc32"
"io"
"net"
"net/http"
@@ -15,6 +17,7 @@ import (
"os"
"os/exec"
"path/filepath"
"runtime"
"sort"
"strings"
"testing"
@@ -383,3 +386,84 @@ func TestARedirectOffHttpsIsNotFollowed(t *testing.T) {
t.Fatalf("%v", err)
}
}
func TestAnXzReleaseIsReadWithoutAnyXzToolOnTheBuildSeat(t *testing.T) {
// The build seat's image (the catalogue's build-agent recipe) carries git and docker-cli alone: Betterbird's
// .tar.xz failed there on 2026-10-12 with "this build seat has no xz to read it" (hq issue 345). It is
// read in Go, with nothing on PATH.
t.Setenv("PATH", "")
body, err := os.ReadFile(filepath.Join("testdata", "release.tar.xz"))
if err != nil {
t.Fatal(err)
}
v := &vendor{files: map[string][]byte{"/r.tar.xz": body}}
base := serveVendor(t, v)
packed, err := upstreamArchive(context.Background(), t.TempDir(), catalogue.Artifact{Name: "archive", Kind: "archive",
URL: base + "/r.tar.xz", SHA256: sumHex(body), Version: "1", Within: "betterbird"}, func(string, string, ...any) {})
if err != nil {
t.Fatal(err)
}
e := entriesOf(t, packed)
if len(e) != 2 || e["betterbird"] != "#!/bin/sh\n -rwxr-xr-x" || e["defaults/p.js"] != "pref(\"a\", 1);\n -rw-r--r--" {
t.Fatalf("%v", e)
}
}
// withDictionary is an xz stream whose first block's LZMA2 dictionary property is props, its header's CRC32
// recomputed: a stream asking for a dictionary of its own choosing.
func withDictionary(t *testing.T, stream []byte, props byte) []byte {
t.Helper()
out := append([]byte{}, stream...)
const at = 12 // the stream header
size := (int(out[at]) + 1) * 4
header := out[at : at+size]
flags := header[1]
i := 2
readVarint := func() {
for header[i]&0x80 != 0 {
i++
}
i++
}
if flags&0x40 != 0 {
readVarint()
}
if flags&0x80 != 0 {
readVarint()
}
if header[i] != 0x21 || header[i+1] != 0x01 {
t.Fatalf("the fixture's first filter is not LZMA2: % x", header)
}
header[i+2] = props
binary.LittleEndian.PutUint32(header[size-4:], crc32.ChecksumIEEE(header[:size-4]))
return out
}
func TestAnXzStreamAskingForTooLargeADictionaryIsRefused(t *testing.T) {
body, err := os.ReadFile(filepath.Join("testdata", "release.tar.xz"))
if err != nil {
t.Fatal(err)
}
// 1 GiB (props 36) and 4 GiB less one (props 40): over the 64 MiB maximum, refused before allocation.
for _, props := range []byte{36, 40} {
crafted := withDictionary(t, body, props)
v := &vendor{files: map[string][]byte{"/r.tar.xz": crafted}}
base := serveVendor(t, v)
var before, after runtime.MemStats
runtime.GC()
runtime.ReadMemStats(&before)
_, err := upstreamArchive(context.Background(), t.TempDir(), catalogue.Artifact{Name: "archive", Kind: "archive",
URL: base + "/r.tar.xz", SHA256: sumHex(crafted), Version: "1", Within: "betterbird"}, func(string, string, ...any) {})
runtime.ReadMemStats(&after)
if err == nil || !strings.Contains(err.Error(), "dictionary") {
t.Errorf("props %d: a dictionary over the maximum was not refused: %v", props, err)
}
if grew := after.TotalAlloc - before.TotalAlloc; grew > 256<<20 {
t.Errorf("props %d: %d bytes were allocated for a refused stream", props, grew)
}
}
// The fixture as it is (a small dictionary) still reads.
if crafted := withDictionary(t, body, 0); len(crafted) == 0 {
t.Fatal("no fixture")
}
}
+290
View File
@@ -0,0 +1,290 @@
package builder
import (
"bufio"
"bytes"
"encoding/binary"
"errors"
"fmt"
"hash/crc32"
"io"
"os"
)
// xzBlocksMax and xzStreamsMax are the most blocks and the most concatenated streams an xz file may hold for
// the build seat to read it.
//
// **Why a block count is limited at all.** The xz reader (github.com/therootcompany/xz v1.0.1) keeps one
// filter decoder for every block that names a delta or BCJ filter, and frees them only when a stream ends:
// its block counters are reset per stream, not per block. A stream of many tiny blocks therefore grows the
// heap with its length: an `xz --block-size=1 --delta` file of 6 MB kept about 67 MiB live, which comes to
// about 5.6 GiB at the 512 MiB download cap (review 68 on mesh-controller pull request 264, hq issue 345).
//
// **Why 1,024.** At about 340 bytes of decoder a block, 1,024 blocks keep at most about 0.35 MiB live. A
// real release is far below it: Betterbird's archive is 1 stream with 1 block, and multi-threaded xz cuts
// blocks of three times the dictionary (24 MiB at the default preset), so even 512 MiB unpacked in blocks
// of 512 KiB stays within it. Streams are limited to 16 because a release is one stream; the decoders are
// reused from one stream to the next, so the stream limit only bounds the work of this walk.
const (
xzBlocksMax = 1024
xzStreamsMax = 16
)
// xzWithinLimits walks an xz file's framing, without decompressing anything, and refuses one holding more
// blocks or streams than the build seat reads.
//
// **The framing, not the index.** Each stream's index at its end lists its blocks, but the reader compares
// the index with the blocks only after it has decoded them all, so a file whose index lies would still be
// decoded block by block. This walk follows the same framing the reader follows: stream header, each block
// header, the block's LZMA2 chunks (each chunk header carries its own compressed size, so a chunk is skipped
// without being decoded), its padding and check, then the index and the footer. It counts the blocks the
// reader would decode, and it also refuses an index whose count of blocks is not the count walked.
func xzWithinLimits(file string) error {
f, err := os.Open(file)
if err != nil {
return err
}
defer f.Close()
_, _, err = xzFraming(bufio.NewReader(f))
if errors.Is(err, errXzTooManyBlocks) || errors.Is(err, errXzTooManyStreams) {
return err
}
if err != nil {
return fmt.Errorf("it is not an xz file this build seat reads: %v", err)
}
return nil
}
var (
errXzTooManyBlocks = fmt.Errorf("it holds more than %d xz blocks, the most this build seat reads: "+
"its xz reader keeps a filter decoder for every block (hq issue 345)", xzBlocksMax)
errXzTooManyStreams = fmt.Errorf("it holds more than %d concatenated xz streams, the most this build "+
"seat reads", xzStreamsMax)
)
// xzFraming counts the streams and blocks of an xz file, checking its framing as it goes. It stops with
// errXzTooManyBlocks or errXzTooManyStreams at the first block or stream over the limit.
func xzFraming(r *bufio.Reader) (streams, blocks int, err error) {
full := func(n int) ([]byte, error) {
b := make([]byte, n)
if _, err := io.ReadFull(r, b); err != nil {
if err == io.EOF {
err = io.ErrUnexpectedEOF
}
return nil, err
}
return b, nil
}
skip := func(n int) error {
if got, err := r.Discard(n); err != nil || got != n {
return io.ErrUnexpectedEOF
}
return nil
}
for {
header, err := full(12)
if err != nil {
return streams, blocks, fmt.Errorf("stream %d: header: %v", streams+1, err)
}
if !bytes.Equal(header[:6], []byte{0xfd, '7', 'z', 'X', 'Z', 0x00}) {
return streams, blocks, fmt.Errorf("stream %d: no xz stream header", streams+1)
}
if header[6] != 0 || header[7]&0xf0 != 0 ||
crc32.ChecksumIEEE(header[6:8]) != binary.LittleEndian.Uint32(header[8:12]) {
return streams, blocks, fmt.Errorf("stream %d: a corrupt stream header", streams+1)
}
if streams++; streams > xzStreamsMax {
return streams, blocks, errXzTooManyStreams
}
checkSize := 0
if check := int(header[7]); check != 0 {
checkSize = 4 << ((check - 1) / 3)
}
inStream := 0
for {
first, err := r.ReadByte()
if err != nil {
return streams, blocks, fmt.Errorf("stream %d: no index: %v", streams, io.ErrUnexpectedEOF)
}
if first == 0 {
break // the index indicator
}
rest, err := full(int(first)*4 + 3)
if err != nil {
return streams, blocks, fmt.Errorf("stream %d, block %d: header: %v", streams, inStream+1, err)
}
if err := xzBlockHeader(append([]byte{first}, rest...)); err != nil {
return streams, blocks, fmt.Errorf("stream %d, block %d: %v", streams, inStream+1, err)
}
inStream++
if blocks++; blocks > xzBlocksMax {
return streams, blocks, errXzTooManyBlocks
}
// The block's LZMA2 chunks, up to the end marker.
size := 0
for {
control, err := r.ReadByte()
if err != nil {
return streams, blocks, fmt.Errorf("stream %d, block %d: %v", streams, inStream, io.ErrUnexpectedEOF)
}
size++
if control == 0 {
break
}
switch {
case control == 1 || control == 2: // stored: a 16-bit size less one, then the bytes
b, err := full(2)
if err != nil {
return streams, blocks, err
}
n := int(binary.BigEndian.Uint16(b)) + 1
if err := skip(n); err != nil {
return streams, blocks, err
}
size += 2 + n
case control >= 0x80: // LZMA: unpacked and packed sizes, the properties byte on a reset
b, err := full(4)
if err != nil {
return streams, blocks, err
}
n := int(binary.BigEndian.Uint16(b[2:])) + 1
size += 4
if control >= 0xc0 {
if err := skip(1); err != nil {
return streams, blocks, err
}
size++
}
if err := skip(n); err != nil {
return streams, blocks, err
}
size += n
default:
return streams, blocks, fmt.Errorf("stream %d, block %d: an LZMA2 chunk of kind %#x",
streams, inStream, control)
}
}
if pad := (4 - size%4) % 4; pad > 0 {
b, err := full(pad)
if err != nil || !bytes.Equal(b, make([]byte, pad)) {
return streams, blocks, fmt.Errorf("stream %d, block %d: corrupt block padding", streams, inStream)
}
}
if err := skip(checkSize); err != nil {
return streams, blocks, err
}
}
// The index: its count of records, two numbers each, padding, CRC32.
used := 1
count, n, err := xzVLI(r)
if err != nil {
return streams, blocks, fmt.Errorf("stream %d: index: %v", streams, err)
}
used += n
if count != uint64(inStream) {
return streams, blocks, fmt.Errorf("stream %d: its index lists %d blocks, but it holds %d",
streams, count, inStream)
}
for i := 0; i < 2*inStream; i++ {
_, n, err := xzVLI(r)
if err != nil {
return streams, blocks, fmt.Errorf("stream %d: index: %v", streams, err)
}
used += n
}
if err := skip((4-used%4)%4 + 4); err != nil {
return streams, blocks, fmt.Errorf("stream %d: index: %v", streams, err)
}
footer, err := full(12)
if err != nil || footer[10] != 'Y' || footer[11] != 'Z' {
return streams, blocks, fmt.Errorf("stream %d: no stream footer", streams)
}
// Stream padding: zero bytes, a multiple of four, then the end or another stream.
zeros := 0
for {
b, err := r.ReadByte()
if err == io.EOF {
if zeros%4 != 0 {
return streams, blocks, errors.New("stream padding not a multiple of four bytes")
}
return streams, blocks, nil
}
if err != nil {
return streams, blocks, err
}
if b != 0 {
_ = r.UnreadByte()
break
}
zeros++
}
if zeros%4 != 0 {
return streams, blocks, errors.New("stream padding not a multiple of four bytes")
}
}
}
// xzBlockHeader checks one block header: its CRC32, its flags, and that its last filter is LZMA2, the only
// one whose chunks this walk can skip (and the only last filter the reader decodes).
func xzBlockHeader(h []byte) error {
size := len(h)
if crc32.ChecksumIEEE(h[:size-4]) != binary.LittleEndian.Uint32(h[size-4:]) {
return errors.New("a corrupt block header")
}
flags := h[1]
if flags&0x3c != 0 {
return errors.New("a block header with reserved flags")
}
body := bytes.NewReader(h[2 : size-4])
if flags&0x40 != 0 {
if _, _, err := xzVLI(body); err != nil {
return err
}
}
if flags&0x80 != 0 {
if _, _, err := xzVLI(body); err != nil {
return err
}
}
var last uint64
for i := 0; i <= int(flags&3); i++ {
id, _, err := xzVLI(body)
if err != nil {
return err
}
props, _, err := xzVLI(body)
if err != nil {
return err
}
if props > uint64(body.Len()) {
return errors.New("a corrupt block header")
}
if _, err := body.Seek(int64(props), io.SeekCurrent); err != nil {
return err
}
last = id
}
if last != 0x21 {
return fmt.Errorf("a block whose last filter is %#x, not LZMA2", last)
}
return nil
}
// xzVLI reads one of xz's variable-length integers (at most 9 bytes, 7 bits each), and how many bytes it took.
func xzVLI(r io.ByteReader) (uint64, int, error) {
var v uint64
for i := 0; i < 9; i++ {
b, err := r.ReadByte()
if err != nil {
return 0, i, io.ErrUnexpectedEOF
}
v |= uint64(b&0x7f) << (7 * i)
if b&0x80 == 0 {
if i > 0 && b == 0 {
return 0, i + 1, errors.New("a number not in its shortest form")
}
return v, i + 1, nil
}
}
return 0, 9, errors.New("a number longer than nine bytes")
}
+207
View File
@@ -0,0 +1,207 @@
package builder
import (
"archive/tar"
"bufio"
"bytes"
"context"
"encoding/binary"
"errors"
"hash/crc32"
"os"
"os/exec"
"path/filepath"
"runtime"
"strings"
"testing"
)
// xzStream is one xz stream (no integrity check) of data cut into blocks of blockSize bytes, each block's
// LZMA2 data stored uncompressed, each block a delta filter before LZMA2 when delta is set: the shape of
// `xz --block-size=N --filters=delta,lzma2`, built without any xz tool. listed is how many blocks its index
// lists (-1: as many as it holds), so a stream whose index lies can be built too.
func xzStream(data []byte, blockSize int, delta bool, listed int) []byte {
var out bytes.Buffer
flags := []byte{0, 0}
out.Write([]byte{0xfd, '7', 'z', 'X', 'Z', 0x00})
out.Write(flags)
out.Write(binary.LittleEndian.AppendUint32(nil, crc32.ChecksumIEEE(flags)))
type record struct{ unpadded, uncompressed uint64 }
var records []record
for at := 0; at < len(data); at += blockSize {
block := data[at:min(at+blockSize, len(data))]
if delta {
// Delta-encoded with a distance of one; the decoder's history starts empty at each block.
encoded := make([]byte, len(block))
for i := range block {
encoded[i] = block[i]
if i > 0 {
encoded[i] -= block[i-1]
}
}
block = encoded
}
header := []byte{2, 0} // 12 bytes
if delta {
header[1] = 1 // two filters
header = append(header, 0x03, 0x01, 0x00)
}
header = append(header, 0x21, 0x01, 0x00) // LZMA2, a 4 KiB dictionary
for len(header) < 8 {
header = append(header, 0)
}
header = binary.LittleEndian.AppendUint32(header, crc32.ChecksumIEEE(header))
out.Write(header)
size := 0
for i, chunk := 0, block; len(chunk) > 0; i++ {
n := min(len(chunk), 1<<16)
control := byte(2)
if i == 0 {
control = 1 // the first chunk of a block resets the dictionary
}
out.WriteByte(control)
out.Write(binary.BigEndian.AppendUint16(nil, uint16(n-1)))
out.Write(chunk[:n])
size += 3 + n
chunk = chunk[n:]
}
out.WriteByte(0)
size++
out.Write(make([]byte, (4-size%4)%4))
records = append(records, record{uint64(len(header) + size), uint64(len(block))})
}
if listed >= 0 {
for len(records) < listed {
records = append(records, records[0])
}
records = records[:listed]
}
vli := func(b []byte, v uint64) []byte {
for v >= 0x80 {
b = append(b, byte(v)|0x80)
v >>= 7
}
return append(b, byte(v))
}
index := vli([]byte{0}, uint64(len(records)))
for _, r := range records {
index = vli(vli(index, r.unpadded), r.uncompressed)
}
for len(index)%4 != 0 {
index = append(index, 0)
}
index = binary.LittleEndian.AppendUint32(index, crc32.ChecksumIEEE(index))
out.Write(index)
footer := binary.LittleEndian.AppendUint32(nil, uint32(len(index)/4-1))
footer = append(footer, flags...)
out.Write(binary.LittleEndian.AppendUint32(nil, crc32.ChecksumIEEE(footer)))
out.Write(footer)
out.WriteString("YZ")
return out.Bytes()
}
// tarHolding is a tar holding betterbird/data with body.
func tarHolding(t *testing.T, body []byte) []byte {
t.Helper()
var out bytes.Buffer
w := tar.NewWriter(&out)
if err := w.WriteHeader(&tar.Header{Name: "betterbird/data", Mode: 0o644, Size: int64(len(body)),
Typeflag: tar.TypeReg}); err != nil {
t.Fatal(err)
}
w.Write(body)
if err := w.Close(); err != nil {
t.Fatal(err)
}
return out.Bytes()
}
func writeTemp(t *testing.T, body []byte) string {
t.Helper()
file := filepath.Join(t.TempDir(), "r.tar.xz")
if err := os.WriteFile(file, body, 0o644); err != nil {
t.Fatal(err)
}
return file
}
func TestAnXzFileOfAFewDeltaBlocksAndTwoStreamsIsRead(t *testing.T) {
// The streams this test builds are ones the reader decodes, so what the walk refuses below is the
// same framing the reader follows.
body := bytes.Repeat([]byte("betterbird "), 300)
whole := tarHolding(t, body)
half := len(whole) / 2
for name, file := range map[string][]byte{
"one stream of delta blocks": xzStream(whole, 512, true, -1),
"two streams with padding": append(append(xzStream(whole[:half], 700, true, -1), 0, 0, 0, 0),
xzStream(whole[half:], 100, false, -1)...),
} {
path := writeTemp(t, file)
if xzPath, err := exec.LookPath("xz"); err == nil {
if out, err := exec.Command(xzPath, "-t", path).CombinedOutput(); err != nil {
t.Fatalf("%s: xz itself refuses the built stream: %v %s", name, err, out)
}
}
into := t.TempDir()
if err := unpackUpstream(context.Background(), path, into, "betterbird"); err != nil {
t.Fatalf("%s: %v", name, err)
}
got, err := os.ReadFile(filepath.Join(into, "data"))
if err != nil || !bytes.Equal(got, body) {
t.Fatalf("%s: unpacked %d bytes, want %d: %v", name, len(got), len(body), err)
}
}
}
func TestAnXzFileOfManyDeltaBlocksIsRefusedBeforeItIsDecoded(t *testing.T) {
// Review 68 on pull request 264: `xz --block-size=1 --delta` of a 200 kB tar kept about 67 MiB live in
// the reader, one delta decoder per block. Its index here lists one block, as a file crafted to pass a
// check of the index alone would; the walk counts the blocks the reader would decode.
body := make([]byte, 200_000)
for i := range body {
body[i] = byte(i * 7)
}
crafted := xzStream(tarHolding(t, body), 1, true, 1)
path := writeTemp(t, crafted)
var before, after runtime.MemStats
runtime.GC()
runtime.ReadMemStats(&before)
err := unpackUpstream(context.Background(), path, t.TempDir(), "betterbird")
runtime.ReadMemStats(&after)
if err == nil || !strings.Contains(err.Error(), "more than 1024 xz blocks") {
t.Fatalf("a file of %d bytes in blocks of one byte was not refused for its blocks: %v", len(crafted), err)
}
if grew := after.TotalAlloc - before.TotalAlloc; grew > 4<<20 {
t.Fatalf("%d bytes were allocated for a refused file", grew)
}
}
func TestTheXzBlockLimitIsExact(t *testing.T) {
data := make([]byte, xzBlocksMax+1)
streams, blocks, err := xzFraming(bufio.NewReader(bytes.NewReader(xzStream(data[:xzBlocksMax], 1, true, -1))))
if err != nil || streams != 1 || blocks != xzBlocksMax {
t.Fatalf("%d blocks: %d streams, %d blocks, %v", xzBlocksMax, streams, blocks, err)
}
if _, _, err := xzFraming(bufio.NewReader(bytes.NewReader(xzStream(data, 1, true, -1)))); !errors.Is(err, errXzTooManyBlocks) {
t.Fatalf("%d blocks: %v", xzBlocksMax+1, err)
}
}
func TestAnXzFileOfTooManyStreamsIsRefused(t *testing.T) {
var file []byte
for i := 0; i <= xzStreamsMax; i++ {
file = append(file, xzStream([]byte("x"), 1, false, -1)...)
}
err := unpackUpstream(context.Background(), writeTemp(t, file), t.TempDir(), "")
if err == nil || !strings.Contains(err.Error(), "more than 16 concatenated xz streams") {
t.Fatalf("%v", err)
}
}
func TestAnXzIndexThatDoesNotListItsBlocksIsRefused(t *testing.T) {
file := xzStream(tarHolding(t, []byte("x")), 512, false, 1)
err := unpackUpstream(context.Background(), writeTemp(t, file), t.TempDir(), "betterbird")
if err == nil || !strings.Contains(err.Error(), "its index lists 1 blocks, but it holds 4") {
t.Fatalf("%v", err)
}
}
+68 -5
View File
@@ -246,6 +246,25 @@ type Rendering struct {
// statement's contract is link.RootContract or later). To an older, strict engine the field is not
// sent, and the account it names is not judged — which the self-check says, as not judged.
JudgesRoot bool
// KeepsLibraries says this machine's node-engine makes the mesh's users and library groups and keeps libraries
// (novox/hq ADR 0328: its statement's contract is link.LibraryContract or later). To an older, strict engine none
// of it is sent, and a module that joins is left out of its declaration, naming why.
KeepsLibraries bool
// Identities is each joining module's allocated number, by module (ADR 0328 §4): the same on every machine. A
// joining module with none here is left out, as at a question asked before its first send.
Identities map[string]ModuleIdentity
// Libraries is every library the operator defined, by name (§1); LibraryGroups each one's allocated group number
// (§3); TakenLibraries those whose data the operator let the mesh take on their home, on the filesystem it is on
// now (§7); LibraryPaths every sub-path any module in the mesh asks of each, kept on its home (§6).
Libraries map[string]Library
LibraryGroups map[string]int
TakenLibraries map[string]bool
LibraryPaths map[string][]string
// RunnerJoinsOwnGroups says this machine's engine puts the tool runner's account in a joining module's own group
// and takes a directory's setgid mode (novox/hq ADR 0334 §1, §2: its statement's contract is link.OwnGroupContract
// or later); to an older one neither the identity's runner nor the setgid bit is sent.
RunnerJoinsOwnGroups bool
}
// RootField is a user resource's field saying the account must never become root without a person
@@ -320,6 +339,11 @@ type Composed struct {
// naming its module and why (novox/hq ADR 0255): that piece is left out and the machine is told
// everything else. Push and plan say it; the merge gate refuses a change that adds one.
Unplaced []string
// Groups, Identities and Libraries are what the declaration says of libraries and joining modules (novox/hq ADR
// 0328): composed only for an engine that keeps libraries, and sent only when not empty.
Groups []GroupEntry
Identities []IdentityEntry
Libraries []LibraryEntry
}
// ForeseenSealed is what stands for an own secret a send will make, in a composition asked ahead of
@@ -358,7 +382,9 @@ func (r Resolution) LeftOutBecause(settings SettingsBy, adopted bool) map[string
out[m.Module] = &UnreadManifestError{Module: m.Module, said: why}
continue
}
if err := JudgeSettings(m, settings[m.Module], adopted); err != nil {
// A placement stored ahead of the build that declares its access is passed by, not a reason to leave the
// module out (novox/hq issue 575): it was judged against that build where it was stored.
if err := ComposesWith(m, settings[m.Module], adopted); err != nil {
out[m.Module] = err
}
}
@@ -380,7 +406,8 @@ func (r Resolution) Compose(with Rendering) (Composed, error) {
received := map[string]map[string][]Contribution{}
leftOut := map[string]string{}
var foreseen, unplaced []string
resources, err := r.compose(with, owner, received, leftOut, &foreseen, &unplaced)
var joined []Joined
resources, err := r.compose(with, owner, received, leftOut, &foreseen, &unplaced, &joined)
if err != nil {
return Composed{}, err
}
@@ -394,8 +421,10 @@ func (r Resolution) Compose(with Rendering) (Composed, error) {
})
}
sort.Strings(unplaced)
return Composed{Resources: resources, Owner: owner, Received: received, LeftOut: leftOut,
Foreseen: foreseen, Unplaced: unplaced}, nil
composed := Composed{Resources: resources, Owner: owner, Received: received, LeftOut: leftOut,
Foreseen: foreseen, Unplaced: unplaced}
composed.Groups, composed.Identities, composed.Libraries = r.libraryEnvelope(with, joined)
return composed, nil
}
// BusMembershipID names the resource carrying a machine's membership for the new bus, and
@@ -406,13 +435,26 @@ const BusMembershipPath = "/var/lib/mesh/membership-next.json"
func (r Resolution) compose(with Rendering, owner map[string]string,
received map[string]map[string][]Contribution, leftOut map[string]string,
foreseen, unplaced *[]string) ([]map[string]any, error) {
foreseen, unplaced *[]string, joinedOut *[]Joined) ([]map[string]any, error) {
// **A setting is judged where it is stored, and an impossible one costs a module, not a
// machine** (novox/hq ADR 0163, rule 6). A definition that moved under a stored setting makes
// this module uncomposable; it is left out of the declaration — its held things kept, its
// containers untouched, the machine told so by name — and the machine is told everything else.
// Before placing, because a placement is a setting too.
left := r.LeftOut(with.Settings, with.Adopted)
// And a module whose libraries cannot be composed here, or that waits for one to be taken (novox/hq ADR 0328 §2,
// §8): left out, naming why, its containers untouched — so a container keeps the user it ran as.
for _, m := range r.Modules {
if _, already := left[m.Module]; already {
continue
}
if why := libraryLeftOut(m, r.Node, with); why != "" {
if left == nil {
left = map[string]string{}
}
left[m.Module] = why
}
}
kept := make([]Manifest, 0, len(r.Modules))
var stillBackedUp []Manifest
for _, m := range r.Modules {
@@ -553,6 +595,20 @@ func (r Resolution) compose(with Rendering, owner map[string]string,
}
resources := m.Resources
// Its identity, where it joins (novox/hq ADR 0328 §4): its user and groups, and where its libraries are here.
var joined Joined
joins := Joins(m)
var libPaths map[string]string
if joins {
if joined, err = joinedOf(m, with); err != nil {
return nil, err
}
libPaths = libraryPaths(m, with)
if joinedOut != nil {
*joinedOut = append(*joinedOut, joined)
}
}
// What the mesh computes for this module goes FIRST, before the module's own resources.
//
// **Order is stated, not derived — the host does not sort** (novox/hq ADR 0005), so
@@ -1003,6 +1059,13 @@ func (r Resolution) compose(with Rendering, owner map[string]string,
return nil, err
}
ownerInto(copied, placed)
// And its identity and libraries, where it joins (novox/hq ADR 0328 §4, §5, §10): after the placement's
// owner, which a joining module may not set.
if joins {
if err := joinedInto(copied, m, joined, libPaths, with.RunnerJoinsOwnGroups); err != nil {
return nil, err
}
}
// **After settings, and that is the whole reason it is here.** A module's file
// content is where a setting lands, so a placeholder may only exist once the setting
// has been put in — filling secrets first would look at content that is not yet what
+907
View File
@@ -0,0 +1,907 @@
package catalogue
import (
"fmt"
"path/filepath"
"regexp"
"sort"
"strings"
)
// Libraries, and the modules that join them (novox/hq ADR 0328).
//
// **A library is a named tree of shared data with one home** (§1). The operator defines it — its name, the machine it
// lives on, its root there and the machines it reaches — as the mesh-wide `libraries` setting of the controller's own
// module, a trusted setting set at the controller's terminal or on the operator's Approve of a proposal (ADR 0277),
// never through a verb alone. Libraries never overlap on a machine (§13), and a library never shares a name with a
// module (§4), so `mesh-<library>` and `mesh-<module>` never collide.
//
// **A module asks for a library by name, sub-path and use** (§2): a manifest's `libraries` entries {id, library,
// path, use}. On each machine an entry resolves to the library's root joined with the path; a module assigned to a
// machine the library does not reach is left out there, naming the library. `${library:<id>}` is that host path, and
// is filled only where a host path belongs (§5): a container's volume source, and a word of the module's tools, which
// run on the machine. A `read` entry is mounted read-only (§10).
//
// **A module joins by using the model** (§14): it declares `libraries` or uses `${identity:…}`. It is given one number
// from the mesh's range, as its user's and its own group's (§4); its user's primary group is the group of the first
// library it uses, the others are added groups, and with no library its own group is primary. `${identity:user}`,
// `${identity:uid}`, `${identity:group}` and `${identity:gid}` are filled wherever its resources take values (§5), and
// its directories are owned by its user and primary group (§4). A module that does neither is composed exactly as
// before.
//
// **The range is 50000–59999** for every number the mesh allocates — a joining module's and a library's group — from
// one sequence, so a module's own group and a library's never share a number. It lies above the numbers distributions
// give people and below the 60000s Debian keeps for its daemons, and nowhere near root's 0 or a root group's (0, adm's 4,
// wheel's 10): no allocated number is root's (§15). The store refuses a row outside it.
//
// **Only to an engine that reads it** (Rendering.KeepsLibraries: its statement's contract is link.LibraryContract or
// later), the declaration carries `groups`, `identities` and `libraries` (libraryEnvelope). To an older engine a module
// that joins is left out of the declaration, naming why — its placeholders could not be filled — and everything else is
// composed as before.
// The range the mesh allocates user and group numbers from (novox/hq ADR 0328 §3, §4, §15). The store holds the same
// bounds.
const (
IdentityLow = 50000
IdentityHigh = 59999
)
// AllocatableID says whether n is a number the mesh may give: inside the range, so never 0 nor a root group's.
func AllocatableID(n int) bool { return n >= IdentityLow && n <= IdentityHigh }
// ADR0328 is how the checks name the decision their rules come from.
const ADR0328 = "novox/hq ADR 0328"
// MeshName is the user or group a module or a library is on every machine: mesh-<name>.
func MeshName(name string) string { return "mesh-" + name }
// IdentityName is the user a joining module runs as, and its own group.
func IdentityName(module string) string { return MeshName(module) }
// ---- libraries, as the operator defines them -----------------------------------------------------------------
// LibrariesSetting is the key of the controller's mesh-wide settings layer that defines the libraries (§1).
const LibrariesSetting = "libraries"
// LibrariesModule is the module whose mesh-wide layer holds LibrariesSetting: the controller's own.
const LibrariesModule = "mesh-controller"
// Library is one library as the operator defined it.
type Library struct {
Name string `json:"-"`
// Home is the machine the data lives on; Root its path there, the same on every machine it reaches.
Home string `json:"home"`
Root string `json:"root"`
// Reach is every machine the library reaches, its home among them.
Reach []string `json:"reach,omitempty"`
}
// Reaches says whether the library reaches a machine: its home, or one it names.
func (l Library) Reaches(node string) bool { return node == l.Home || hasName(l.Reach, node) }
// Machines is every machine the library reaches, its home first.
func (l Library) Machines() []string {
out := []string{l.Home}
for _, n := range l.Reach {
if n != l.Home && !hasName(out, n) {
out = append(out, n)
}
}
return out
}
// Group is the library's group: mesh-<library>.
func (l Library) Group() string { return MeshName(l.Name) }
var libraryName = regexp.MustCompile(`^[a-z][a-z0-9-]*$`)
// ParseLibraries reads the `libraries` setting: {<name>: {home, root, reach}}, or refuses it in words.
func ParseLibraries(v any) (map[string]Library, error) {
if v == nil {
return nil, nil
}
blocks, ok := v.(map[string]any)
if !ok {
return nil, fmt.Errorf("%s is a { <library>: {home, root, reach} } map (%s §1)", LibrariesSetting, ADR0328)
}
out := map[string]Library{}
for name, raw := range blocks {
if !libraryName.MatchString(name) {
return nil, fmt.Errorf("the library %q: a library's name is lowercase letters, digits and dashes, starting "+
"with a letter", name)
}
fields, ok := raw.(map[string]any)
if !ok {
return nil, fmt.Errorf("the library %s is {home, root, reach}, not %v", name, raw)
}
l := Library{Name: name}
for k, v := range fields {
switch k {
case "home":
l.Home, _ = v.(string)
case "root":
l.Root, _ = v.(string)
case "reach":
list, ok := v.([]any)
if !ok {
return nil, fmt.Errorf("the library %s's reach is a list of machines", name)
}
for _, n := range list {
s, _ := n.(string)
if s == "" {
return nil, fmt.Errorf("the library %s's reach names a machine that is not a name: %v", name, n)
}
l.Reach = append(l.Reach, s)
}
default:
return nil, fmt.Errorf("the library %s says %q; a library is its home, its root and its reach, and "+
"nothing about an owner, a group, a mode or a number (%s §1)", name, k, ADR0328)
}
}
if strings.TrimSpace(l.Home) == "" {
return nil, fmt.Errorf("the library %s names no home machine", name)
}
if !strings.HasPrefix(l.Root, "/") {
return nil, fmt.Errorf("the library %s's root %q is not an absolute path", name, l.Root)
}
l.Root = filepath.Clean(l.Root)
if why := systemPath(l.Root); why != "" {
return nil, fmt.Errorf("the library %s's root is %s: %s, and the node-engine keeps a library's root in its "+
"group — no library is ever there", name, l.Root, why)
}
sort.Strings(l.Reach)
out[name] = l
}
return out, nil
}
// LibrariesProblems refuses what no library may be together (§4, §13): a library named as a registered module, and two
// libraries on one machine whose roots lie inside each other — each named.
func LibrariesProblems(libs map[string]Library, modules []string) []string {
var problems []string
names := make([]string, 0, len(libs))
for n := range libs {
names = append(names, n)
}
sort.Strings(names)
for _, n := range names {
if hasName(modules, n) {
problems = append(problems, fmt.Sprintf("the library %s has the name of the module %s: %s would be both "+
"(%s §4)", n, n, MeshName(n), ADR0328))
}
}
for i, a := range names {
for _, b := range names[i+1:] {
la, lb := libs[a], libs[b]
shared := ""
for _, m := range la.Machines() {
if lb.Reaches(m) {
shared = m
break
}
}
if shared == "" {
continue
}
if within(la.Root, lb.Root) || within(lb.Root, la.Root) {
problems = append(problems, fmt.Sprintf("the libraries %s (%s) and %s (%s) overlap on %s: a library's "+
"root lies neither inside another's nor around it — divide one library by the paths its modules "+
"ask for (%s §13)", a, la.Root, b, lb.Root, shared, ADR0328))
}
}
}
return problems
}
// within says path is root or lies below it.
func within(path, root string) bool {
return path == root || strings.HasPrefix(path, strings.TrimSuffix(root, "/")+"/")
}
func hasName(list []string, s string) bool {
for _, x := range list {
if x == s {
return true
}
}
return false
}
// ---- a module's use of them ---------------------------------------------------------------------------------
// LibraryUse is one `libraries` entry of a manifest: the id the module's resources name it by, the library, the
// sub-path within it, and the use.
type LibraryUse struct {
ID string `json:"id"`
Library string `json:"library"`
Path string `json:"path"`
Use string `json:"use"`
}
// At is the entry's use, read by default.
func (u LibraryUse) At() string {
if u.Use == "" {
return AccessRead
}
return u.Use
}
// libraryRef is how a module names one of its library entries: ${library:<id>}.
var libraryRef = regexp.MustCompile(`\$\{library:([a-z0-9][a-z0-9-]*)\}`)
// identityRef is a module's identity by name: ${identity:user|uid|group|gid}.
var identityRef = regexp.MustCompile(`\$\{identity:(user|uid|group|gid)\}`)
// Joins says a module joins the model (§14): it declares libraries or uses its identity anywhere.
func Joins(m Manifest) bool {
if len(m.Libraries) > 0 {
return true
}
found := false
for _, r := range m.Resources {
walkStrings(r, func(s string) {
found = found || identityRef.MatchString(s) || strings.Contains(s, "${identity:")
})
}
for _, b := range m.Bundles {
for _, v := range b.Env {
found = found || strings.Contains(v, "${identity:")
}
}
return found
}
// walkStrings calls see with every string a value holds, however deep.
func walkStrings(v any, see func(string)) {
switch v := v.(type) {
case string:
see(v)
case []any:
for _, x := range v {
walkStrings(x, see)
}
case map[string]any:
for _, x := range v {
walkStrings(x, see)
}
}
}
// mapStrings is v with every string it holds replaced by fill's answer, however deep; the first error stops it.
func mapStrings(v any, fill func(string) (string, error)) (any, error) {
switch v := v.(type) {
case string:
return fill(v)
case []any:
out := make([]any, len(v))
for i, x := range v {
y, err := mapStrings(x, fill)
if err != nil {
return nil, err
}
out[i] = y
}
return out, nil
case map[string]any:
out := make(map[string]any, len(v))
for k, x := range v {
y, err := mapStrings(x, fill)
if err != nil {
return nil, err
}
out[k] = y
}
return out, nil
}
return v, nil
}
var libraryEntryID = regexp.MustCompile(`^[a-z0-9][a-z0-9-]*$`)
// numberWords are the setting names that carry a user or group number or an owner: none for a module that joins (§5).
// "user" and "group" are left out: an application's own login or group name is a setting like any other.
var numberWords = map[string]bool{"uid": true, "gid": true, "puid": true, "pgid": true, "user-id": true,
"group-id": true, "owner": true}
// libraryProblems is what the module check refuses of a manifest's libraries and identity (§2, §4, §5, §14):
//
// - an entry that is not {id, library, path, use} with a relative sub-path inside its library, or names a host
// path, an owner, a group, a mode or a number (the entry's shape refuses any other key);
// - two entries with one id, or a `${library:<id>}` naming none;
// - `${library:<id>}` anywhere but a container's volume source (and a word of its tools, a bundle's env);
// - `${identity:…}` other than user, uid, group or gid;
// - in a module that joins: a directory naming its owner. A setting of it naming a number or an owner is refused
// where it is set (JoiningSettingProblem).
func libraryProblems(m Manifest) []string {
var problems []string
ids := map[string]bool{}
for _, u := range m.Libraries {
where := fmt.Sprintf("%s's library entry %q", m.Module, u.ID)
switch {
case !libraryEntryID.MatchString(u.ID):
problems = append(problems, where+": an id is lowercase letters, digits and dashes")
case ids[u.ID]:
problems = append(problems, where+": two entries share the id")
}
ids[u.ID] = true
if !libraryName.MatchString(u.Library) {
problems = append(problems, fmt.Sprintf("%s names the library %q, which is not a library's name", where,
u.Library))
}
clean := filepath.Clean(u.Path)
if u.Path == "" || strings.HasPrefix(u.Path, "/") || clean == ".." || strings.HasPrefix(clean, "../") ||
strings.ContainsAny(u.Path, "${}") {
problems = append(problems, fmt.Sprintf("%s's path %q is a sub-path inside the library, relative and "+
"staying inside it — never a host path (%s §2)", where, u.Path, ADR0328))
}
if u.At() != AccessRead && u.At() != AccessReadWrite {
problems = append(problems, fmt.Sprintf("%s is used %q; a use is %q or %q", where, u.Use, AccessRead,
AccessReadWrite))
}
}
for _, r := range m.Resources {
kind := fmt.Sprint(r["type"])
for field, v := range r {
walkStrings(v, func(s string) {
for _, tok := range regexp.MustCompile(`\$\{identity:[^}]*\}`).FindAllString(s, -1) {
if !identityRef.MatchString(tok) {
problems = append(problems, fmt.Sprintf("%s's resource %v says %s: the identity is "+
"${identity:user}, ${identity:uid}, ${identity:group} or ${identity:gid} (%s §5)", m.Module,
r["id"], tok, ADR0328))
}
}
for _, match := range libraryRef.FindAllStringSubmatch(s, -1) {
if !ids[match[1]] {
problems = append(problems, fmt.Sprintf("%s's resource %v says ${library:%s}, and declares no "+
"library entry %q", m.Module, r["id"], match[1], match[1]))
}
source := ""
if kind == "container" && field == "volumes" {
source = volumeSource(s)
}
if !strings.Contains(source, match[0]) {
problems = append(problems, fmt.Sprintf("%s's resource %v says %s in its %s: a library's path "+
"is a host path, filled only in a container's volume source — the process reads the "+
"container side its volume gives (%s §5)", m.Module, r["id"], match[0], field, ADR0328))
}
}
})
}
}
if !Joins(m) {
return problems
}
problems = append(problems, toolsDirectoryProblems(m)...)
for _, r := range m.Resources {
if fmt.Sprint(r["type"]) == "directory" {
if owner, said := r["owner"]; said {
problems = append(problems, fmt.Sprintf("%s: the directory %v names its owner %v; a module that joins "+
"owns its directories by its user and primary group, and names no owner (%s §4)", m.Module,
r["id"], owner, ADR0328))
}
}
}
return problems
}
// toolsDirectoryProblems refuses a joining module whose tools name one of its directories whose mode closes it to its
// group (novox/hq ADR 0334 §3): the tool runner reads it through the module's own group, so it needs group read and
// enter.
func toolsDirectoryProblems(m Manifest) []string {
modes := map[string]string{}
for _, r := range m.Resources {
if fmt.Sprint(r["type"]) == "directory" {
mode, _ := r["mode"].(string)
modes[fmt.Sprint(r["id"])] = mode
}
}
var words []string
for _, b := range m.Bundles {
for _, v := range b.Env {
words = append(words, v)
}
}
if m.Build != nil {
for _, a := range m.Build.Artifacts {
for _, v := range a.Env {
words = append(words, v)
}
}
}
seen := map[string]bool{}
var problems []string
for _, w := range words {
for _, match := range dirRef.FindAllStringSubmatch(w, -1) {
id := match[1]
mode, declared := modes[id]
if !declared || seen[id] {
continue
}
seen[id] = true
var n uint64
if _, err := fmt.Sscanf(mode, "%o", &n); err != nil || mode == "" {
continue // its mode is judged where the directory is
}
if n&0o050 != 0o050 {
problems = append(problems, fmt.Sprintf("%s's tools name its directory %s, whose mode %s closes it to its "+
"own group: the tool runner reads it through that group, so it needs group read and enter (0750; "+
"the mesh adds setgid, 2750) (novox/hq ADR 0334 §3)", m.Module, id, mode))
}
}
}
sort.Strings(problems)
return problems
}
// LibraryWarnings is what the module check warns of a module's libraries (§4): an image that drops added groups
// reaches only the first library the module uses.
func LibraryWarnings(m Manifest) []string {
seen := map[string]bool{}
var libs []string
for _, u := range m.Libraries {
if !seen[u.Library] {
seen[u.Library] = true
libs = append(libs, u.Library)
}
}
if len(libs) < 2 {
return nil
}
return []string{fmt.Sprintf("%s uses the libraries %s: its user's primary group is %s, and %s are added groups — an "+
"image that drops added groups when it changes user reaches only %s (%s §4)", m.Module, strings.Join(libs, ", "),
MeshName(libs[0]), strings.Join(meshNames(libs[1:]), ", "), libs[0], ADR0328)}
}
func meshNames(names []string) []string {
out := make([]string, len(names))
for i, n := range names {
out[i] = MeshName(n)
}
return out
}
// JoiningSettingProblem refuses, where a setting is set, a layer of a module that joins that carries a user or group
// number or an owner (§5): a key whose name says so, or an owner a `places` setting names — its directories are owned
// by its user and primary group, and it reads its own as ${identity:…}.
func JoiningSettingProblem(m Manifest, layers []Layer) error {
if !Joins(m) {
return nil
}
for _, l := range layers {
for k := range l.Values {
if numberWords[k] {
return fmt.Errorf("%s joins, and the layer %q sets %q: no setting of a module that joins carries a user "+
"or group number or an owner — it reads its own as ${identity:…} (%s §5)", m.Module, l.From, k, ADR0328)
}
}
}
return PlacesOwnerProblem(m, layers)
}
// PlacesOwnerProblem refuses an owner a `places` setting names for a module that joins (§4, §5): its directories are
// owned by its user and primary group.
func PlacesOwnerProblem(m Manifest, layers []Layer) error {
if !Joins(m) {
return nil
}
placed, err := Places(m, layers)
if err != nil {
return nil // refused by name where it is judged
}
var owned []string
for id, p := range placed {
if p.Owner != "" {
owned = append(owned, id)
}
}
if len(owned) == 0 {
return nil
}
sort.Strings(owned)
return fmt.Errorf("%s places %s with an owner, and it joins: its directories are owned by its user and primary "+
"group, and no setting of it carries an owner (%s §4, §5)", m.Module, strings.Join(owned, ", "), ADR0328)
}
// ---- composition ------------------------------------------------------------------------------------------------
// ModuleIdentity is a joining module's allocated number: its user's and its own group's.
type ModuleIdentity struct {
Module string
UID int
}
// Joined is a joining module's identity on a machine, as its placeholders are filled and its directories owned.
type Joined struct {
Module string
UID int
// Primary is its user's primary group, by name and number; Added its other groups, by name, in the order used.
Primary string
PrimaryGID int
Added []string
AddedGIDs []int
}
// librariesOf is the libraries a module uses, by name, in the order its entries first name them.
func librariesOf(m Manifest) []string {
var out []string
for _, u := range m.Libraries {
if !hasName(out, u.Library) {
out = append(out, u.Library)
}
}
return out
}
// joinedOf is module m's identity here: its number, and its groups from the libraries it uses. Refused, naming
// what, where a number is missing or outside the range (§15).
func joinedOf(m Manifest, with Rendering) (Joined, error) {
id, has := with.Identities[m.Module]
if !has {
return Joined{}, fmt.Errorf("%s joins, and has no number yet: it is allocated at its first send", m.Module)
}
if !AllocatableID(id.UID) {
return Joined{}, fmt.Errorf("%s's number is %d, outside %d–%d: no number the mesh gives is root's or a root "+
"group's, and none is sent (%s §15)", m.Module, id.UID, IdentityLow, IdentityHigh, ADR0328)
}
j := Joined{Module: m.Module, UID: id.UID, Primary: IdentityName(m.Module), PrimaryGID: id.UID}
for i, lib := range librariesOf(m) {
gid, has := with.LibraryGroups[lib]
if !has || !AllocatableID(gid) {
return Joined{}, fmt.Errorf("%s uses the library %s, whose group has no number the mesh may give (%s §3, "+
"§15)", m.Module, lib, ADR0328)
}
if i == 0 {
j.Primary, j.PrimaryGID = MeshName(lib), gid
continue
}
j.Added = append(j.Added, MeshName(lib))
j.AddedGIDs = append(j.AddedGIDs, gid)
}
return j, nil
}
// libraryLeftOut is why module m is left out of this machine's declaration for its libraries, or "": a library it
// uses not defined or not reaching the machine (§2), or an engine that cannot be told it joins. A module held at the
// user it ran as until its libraries are taken (§8) is not left out: it is composed at the build it ran before it
// joined (cmd/mesh-controller holdJoining).
func libraryLeftOut(m Manifest, node string, with Rendering) string {
if !Joins(m) {
return ""
}
if !with.KeepsLibraries {
return fmt.Sprintf("%s joins the mesh's users and libraries (%s), and the node-engine on %s is older than "+
"them: it is left out until that engine is updated", m.Module, ADR0328, node)
}
for _, lib := range librariesOf(m) {
l, defined := with.Libraries[lib]
switch {
case !defined:
return fmt.Sprintf("%s uses the library %s, which is not defined: the operator defines it (%s §1)",
m.Module, lib, ADR0328)
case !l.Reaches(node):
return fmt.Sprintf("%s uses the library %s, which does not reach %s (%s §2)", m.Module, lib, node, ADR0328)
}
}
if _, has := with.Identities[m.Module]; !has {
return fmt.Sprintf("%s joins, and its number is allocated at its first send", m.Module)
}
return ""
}
// libraryPaths is where each of module m's library entries is on this machine: its library's root joined with the
// entry's path, by entry id.
func libraryPaths(m Manifest, with Rendering) map[string]string {
out := map[string]string{}
for _, u := range m.Libraries {
if l, ok := with.Libraries[u.Library]; ok {
out[u.ID] = filepath.Join(l.Root, filepath.Clean(u.Path))
}
}
return out
}
// identityFill fills ${identity:…} in one string.
func identityFill(s string, j Joined) string {
return identityRef.ReplaceAllStringFunc(s, func(tok string) string {
switch identityRef.FindStringSubmatch(tok)[1] {
case "user":
return IdentityName(j.Module)
case "uid":
return fmt.Sprint(j.UID)
case "group":
return j.Primary
default:
return fmt.Sprint(j.PrimaryGID)
}
})
}
// joinedInto composes one resource of a joining module (§4, §5, §10): ${identity:…} filled everywhere, a container's
// ${library:<id>} volume sources filled with the host path and a read entry mounted read-only, and a directory owned by
// the module's user and primary group.
func joinedInto(resource map[string]any, m Manifest, j Joined, paths map[string]string, setgid bool) error {
for k, v := range resource {
if k == "id" || k == "type" {
continue
}
filled, err := mapStrings(v, func(s string) (string, error) { return identityFill(s, j), nil })
if err != nil {
return err
}
resource[k] = filled
}
// Its own directories belong to its user and its own group, with setgid (ADR 0334 §1, v4): the library's group is
// its processes' primary, so what they make in a library keeps that group — and what they make in their own
// directories keeps mesh-<module> by the setgid bit. The bit only to an engine that takes a mode with it
// (Rendering.RunnerJoinsOwnGroups, link.OwnGroupContract): the one pinned refuses a mode not starting with 0.
if fmt.Sprint(resource["type"]) == "directory" {
resource["owner"] = fmt.Sprintf("%d:%d", j.UID, j.UID)
if mode, ok := resource["mode"].(string); ok && setgid {
var n uint64
if _, err := fmt.Sscanf(mode, "%o", &n); err == nil && mode != "" {
resource["mode"] = fmt.Sprintf("%04o", n|0o2000)
}
}
}
if fmt.Sprint(resource["type"]) != "container" {
return nil
}
volumes, _ := resource["volumes"].([]any)
out := make([]any, len(volumes))
for i, v := range volumes {
out[i] = v
s, ok := v.(string)
if !ok {
continue
}
source := volumeSource(s)
match := libraryRef.FindStringSubmatch(source)
if match == nil {
continue
}
path, has := paths[match[1]]
if !has {
return fmt.Errorf("%s says ${library:%s}, and has no library entry %q", m.Module, match[1], match[1])
}
filled := strings.Replace(source, match[0], path, 1) + strings.TrimPrefix(s, source)
for _, u := range m.Libraries {
if u.ID == match[1] && u.At() == AccessRead {
filled = readOnly(filled)
}
}
out[i] = filled
}
if len(volumes) > 0 {
resource["volumes"] = out
}
return nil
}
// readOnly is a volume mounted read-only: "src:dst" becomes "src:dst:ro", and options gain ro in place of rw.
func readOnly(v string) string {
parts := strings.Split(v, ":")
if len(parts) < 2 {
return v
}
if len(parts) == 2 {
return v + ":ro"
}
opts := strings.Split(parts[len(parts)-1], ",")
kept := []string{"ro"}
for _, o := range opts {
if o != "ro" && o != "rw" && o != "" {
kept = append(kept, o)
}
}
parts[len(parts)-1] = strings.Join(kept, ",")
return strings.Join(parts, ":")
}
// volumeSource is a volume's source as written: up to its first colon outside a ${…} placeholder, so
// "${dir:config}:/config" is "${dir:config}".
func volumeSource(v string) string {
depth := 0
for i, r := range v {
switch {
case r == '{' && i > 0 && v[i-1] == '$':
depth++
case r == '}' && depth > 0:
depth--
case r == ':' && depth == 0:
return v[:i]
}
}
return v
}
// bundleLibraryFill fills a joining module's tool words (a bundle's env): ${identity:…}, and ${library:<id>} with the
// host path, since its tools run on the machine (§5).
func bundleLibraryFill(value string, m Manifest, with Rendering) (string, error) {
if !strings.Contains(value, "${identity:") && !strings.Contains(value, "${library:") {
return value, nil
}
j, err := joinedOf(m, with)
if err != nil {
return "", err
}
paths := libraryPaths(m, with)
var missing error
value = libraryRef.ReplaceAllStringFunc(identityFill(value, j), func(tok string) string {
id := libraryRef.FindStringSubmatch(tok)[1]
p, has := paths[id]
if !has {
missing = fmt.Errorf("%s's tools say ${library:%s}, and it has no library entry %q", m.Module, id, id)
return tok
}
return p
})
return value, missing
}
// toolsAccount is the account module m's tools run as on this machine, or empty where they run nowhere as one: no
// runtime here, no bundle it loads, or a module left out of an adopted machine's declaration.
func (r Resolution) toolsAccount(m Manifest, with Rendering) string {
if !r.runtimeHere() || (with.Adopted && m.Filtering != nil) {
return ""
}
loads := false
for _, b := range m.Bundles {
if len(b.Loads) > 0 {
loads = true
}
}
if !loads {
return ""
}
if m.RunsAs != "" {
return m.RunsAs
}
return r.Account
}
// LibraryEntry is one library as a machine's declaration names it (§3, §6, §7, §11).
type LibraryEntry struct {
Library string `json:"library"`
Root string `json:"root"`
Group string `json:"group"`
GID int `json:"gid"`
// Home says this machine is the library's home, where the engine keeps it; Taken that the operator let the mesh
// take its existing data there (§7).
Home bool `json:"home"`
Taken bool `json:"taken"`
// Paths are the sub-paths its modules ask for, relative to its root.
Paths []string `json:"paths"`
// Runner is the tool runner's account where a module whose tools run there uses the library: a member of its
// group (§11). Empty otherwise.
Runner string `json:"runner,omitempty"`
}
// GroupEntry is one group a machine's declaration names: a library's, made on every machine the library reaches (§3).
type GroupEntry struct {
Name string `json:"name"`
GID int `json:"gid"`
}
// IdentityEntry is one joining module's user as a machine's declaration names it (§4; mesh-host internal/declaration
// Identity): the engine always makes its own group with its uid; GID is its primary group's number — its first
// library's, or its own — and Groups the numbers of its added groups.
type IdentityEntry struct {
Module string `json:"module"`
Name string `json:"name"`
UID int `json:"uid"`
GID int `json:"gid"`
Groups []int `json:"groups,omitempty"`
// Runner is the tool runner's account, put in this module's own group, where its tools run there (ADR 0334 §2).
// Only to an engine that reads it (Rendering.RunnerJoinsOwnGroups).
Runner string `json:"runner,omitempty"`
}
// libraryEnvelope is what a machine's declaration says of libraries and joining modules: the group of every library
// that reaches it, each joining module composed here, and each library a module here uses or that has its home here.
func (r Resolution) libraryEnvelope(with Rendering, joined []Joined) ([]GroupEntry, []IdentityEntry, []LibraryEntry) {
if !with.KeepsLibraries {
return nil, nil, nil
}
var groups []GroupEntry
for name, l := range with.Libraries {
if gid, has := with.LibraryGroups[name]; has && l.Reaches(r.Node) && AllocatableID(gid) {
groups = append(groups, GroupEntry{Name: l.Group(), GID: gid})
}
}
sort.Slice(groups, func(i, j int) bool { return groups[i].Name < groups[j].Name })
var ids []IdentityEntry
for _, j := range joined {
e := IdentityEntry{Module: j.Module, Name: IdentityName(j.Module), UID: j.UID, GID: j.PrimaryGID,
Groups: append([]int(nil), j.AddedGIDs...)}
if with.RunnerJoinsOwnGroups {
for _, m := range r.Modules {
if m.Module == j.Module {
e.Runner = r.toolsAccount(m, with)
}
}
}
ids = append(ids, e)
}
sort.Slice(ids, func(a, b int) bool { return ids[a].Module < ids[b].Module })
used := map[string]map[string]bool{}
runner := map[string]string{}
for _, m := range r.Modules {
if libraryLeftOut(m, r.Node, with) != "" {
continue
}
for _, u := range m.Libraries {
if used[u.Library] == nil {
used[u.Library] = map[string]bool{}
}
used[u.Library][filepath.Clean(u.Path)] = true
if a := r.toolsAccount(m, with); a != "" && runner[u.Library] == "" {
runner[u.Library] = a
}
}
}
var libs []LibraryEntry
for name, l := range with.Libraries {
home := l.Home == r.Node
if used[name] == nil && !home {
continue
}
gid, has := with.LibraryGroups[name]
if !has || !l.Reaches(r.Node) {
continue
}
paths := map[string]bool{}
for p := range used[name] {
paths[p] = true
}
if home {
// On its home the engine keeps every sub-path any module asks for, wherever it runs (§6).
for _, p := range with.LibraryPaths[name] {
paths[filepath.Clean(p)] = true
}
}
e := LibraryEntry{Library: name, Root: l.Root, Group: l.Group(), GID: gid, Home: home,
Taken: with.TakenLibraries[name], Paths: []string{}, Runner: runner[name]}
for p := range paths {
e.Paths = append(e.Paths, p)
}
sort.Strings(e.Paths)
libs = append(libs, e)
}
sort.Slice(libs, func(i, j int) bool { return libs[i].Library < libs[j].Library })
return groups, ids, libs
}
// LibraryUseShape is a module's libraries and whether it joins, in a form that compares: a move that changes either
// changes who uses its libraries (the gate's reading of an unknown or a held library).
func LibraryUseShape(m Manifest) string {
var out []string
for _, u := range m.Libraries {
out = append(out, u.ID+"\x00"+u.Library+"\x00"+filepath.Clean(u.Path)+"\x00"+u.At())
}
sort.Strings(out)
return fmt.Sprint(Joins(m), out)
}
// LibraryOfEntry is the library a module's entry id names, or "".
func LibraryOfEntry(m Manifest, id string) string {
for _, u := range m.Libraries {
if u.ID == id {
return u.Library
}
}
return ""
}
// AccessPathOn is where access id of module m is on a machine with these settings: its placement, else the
// definition's default; false where neither says, or the module declares no such access.
func AccessPathOn(m Manifest, layers []Layer, id string) (string, bool) {
_, byID, err := accessesFor(m, layers)
if err != nil {
return "", false
}
path, has := byID[id]
if !has || path == "" {
return "", false
}
return filepath.Clean(path), true
}
+429
View File
@@ -0,0 +1,429 @@
package catalogue
import (
"encoding/json"
"fmt"
"reflect"
"strings"
"testing"
"github.com/novox/mesh-host/validate"
)
// Defends novox/hq ADR 0328 on the controller's side, row by row of its "How it is checked" table where the row is the
// controller's: libraries defined (1, 13), entries resolved (2), groups and numbers (3, 4, 15), the identity and library
// placeholders (5), a read use mounted read-only (10), the tool runner in a library's group (11), a joining module held
// until its libraries are taken (8), and a module that does not join composed exactly as before (14).
const libraryImage = "x@sha256:1111111111111111111111111111111111111111111111111111111111111111"
var mediaLibrary = Library{Name: "media", Home: "ace", Root: "/storage/media", Reach: []string{"novox"}}
// aRadarr writes movies and reads books, tells its image its identity by PUID and PGID, and has tools on the runner.
func aRadarr(t *testing.T) Manifest {
t.Helper()
m := aToolsModule(t, "radarr", "tools/index.js")
m.Libraries = []LibraryUse{
{ID: "movies", Library: "media", Path: "movies", Use: AccessReadWrite},
{ID: "books", Library: "books", Path: "shelf", Use: AccessRead},
}
m.Resources = []map[string]any{
{"id": "config", "type": "directory", "mode": "0750"},
{"id": "env", "type": "file", "path": "${dir:config}/identity.env", "mode": "0644",
"content": "PUID=${identity:uid}\nPGID=${identity:gid}\nUSER=${identity:user}\nGROUP=${identity:group}\nUMASK=002\n"},
{"id": "app", "type": "container", "name": "radarr", "image": libraryImage,
"env": map[string]any{"PUID": "${identity:uid}", "PGID": "${identity:gid}"},
"volumes": []any{"${dir:config}:/config", "${library:movies}:/movies", "${library:books}:/books:z"}},
}
return m
}
// aPlain is a module that neither declares libraries nor uses its identity.
func aPlain() Manifest {
return Manifest{Module: "plain", Version: "1", Resources: []map[string]any{
{"id": "data", "type": "directory", "mode": "0700", "owner": "999:999"},
{"id": "app", "type": "container", "name": "plain", "image": libraryImage, "volumes": []any{"${dir:data}:/data"}},
}}
}
func librariesRendering() Rendering {
books := Library{Name: "books", Home: "ace", Root: "/storage/books"}
return Rendering{KeepsLibraries: true,
Libraries: map[string]Library{"media": mediaLibrary, "books": books},
LibraryGroups: map[string]int{"media": 50010, "books": 50011},
Identities: map[string]ModuleIdentity{"radarr": {Module: "radarr", UID: 50020}},
TakenLibraries: map[string]bool{"media": true, "books": true},
LibraryPaths: map[string][]string{"media": {"movies", "series"}},
}
}
func composeLibraries(t *testing.T, r Resolution, with Rendering) Composed {
t.Helper()
with.Needed = map[string]map[string]string{RuntimeModule: {"broker": "sealed-credential"}}
with.ArtifactStore = "anchor.internal:5101"
composed, err := r.Compose(with)
if err != nil {
t.Fatal(err)
}
return composed
}
// Rows 2, 4, 5, 10: a joining module's identity is filled everywhere, its library entries resolve to root + path in its
// volume sources, a read entry mounts read-only, its directories are owned by its user and primary group — the first
// library it uses — and its other libraries are added groups.
func TestAJoiningModuleIsComposedWithItsIdentityAndLibraries(t *testing.T) {
r := Resolution{Node: "ace", Account: "jochen", Modules: []Manifest{aRadarr(t), theRuntime(t)}}
composed := composeLibraries(t, r, librariesRendering())
got := byID(composed.Resources)
app := got["radarr.app"]
if fmt.Sprint(app["env"]) != fmt.Sprint(map[string]any{"PUID": "50020", "PGID": "50010"}) {
t.Errorf("the container's environment is %v", app["env"])
}
want := []any{"/var/lib/radarr/config:/config", "/storage/media/movies:/movies", "/storage/books/shelf:/books:ro,z"}
if !reflect.DeepEqual(app["volumes"], want) {
t.Errorf("the volumes are %v; want %v", app["volumes"], want)
}
if content := fmt.Sprint(got["radarr.env"]["content"]); content !=
"PUID=50020\nPGID=50010\nUSER=mesh-radarr\nGROUP=mesh-media\nUMASK=002\n" {
t.Errorf("the identity file reads %q", content)
}
if owner := got["radarr.config"]["owner"]; owner != "50020:50020" {
t.Errorf("its directory is owned by %v; want its user and own group (ADR 0334 §1)", owner)
}
if len(composed.Identities) != 1 || !reflect.DeepEqual(composed.Identities[0], IdentityEntry{Module: "radarr",
Name: "mesh-radarr", UID: 50020, GID: 50010, Groups: []int{50011}}) {
t.Errorf("the identities are %+v", composed.Identities)
}
if w := LibraryWarnings(aRadarr(t)); len(w) != 1 || !strings.Contains(w[0], "reaches only media") {
t.Errorf("a module of two libraries is warned %q", w)
}
}
// Rows 3, 7, 11: every library reaching the machine has its group, with its number; each library used here or homed
// here is named with its root, whether this is its home, whether it is taken, the sub-paths kept on its home, and the
// tool runner's account where a module whose tools run there uses it.
func TestTheDeclarationNamesTheLibrariesAndTheirGroups(t *testing.T) {
with := librariesRendering()
with.TakenLibraries = map[string]bool{"media": false, "books": true}
r := Resolution{Node: "ace", Account: "jochen", Modules: []Manifest{aRadarr(t), theRuntime(t)}}
composed := composeLibraries(t, r, with)
raw, _ := json.Marshal(map[string]any{"groups": composed.Groups, "libraries": composed.Libraries})
want := `{"groups":[{"name":"mesh-books","gid":50011},{"name":"mesh-media","gid":50010}],` +
`"libraries":[{"library":"books","root":"/storage/books","group":"mesh-books","gid":50011,"home":true,` +
`"taken":true,"paths":["shelf"],"runner":"jochen"},` +
`{"library":"media","root":"/storage/media","group":"mesh-media","gid":50010,"home":true,"taken":false,` +
`"paths":["movies","series"],"runner":"jochen"}]}`
if string(raw) != want {
t.Errorf("the declaration says\n %s\nwant\n %s", raw, want)
}
// On a machine it reaches but is not home: the group, the library as used there, and no sub-path from elsewhere.
elsewhere := composeLibraries(t, Resolution{Node: "novox", Account: "jochen",
Modules: []Manifest{aPlain(), theRuntime(t)}}, with)
raw, _ = json.Marshal(map[string]any{"groups": elsewhere.Groups, "libraries": elsewhere.Libraries})
if string(raw) != `{"groups":[{"name":"mesh-media","gid":50010}],"libraries":null}` {
t.Errorf("on a machine it only reaches: %s", raw)
}
}
// Row 2: a module assigned to a machine its library does not reach is left out there, naming the library; one whose
// library is not defined likewise.
func TestALibraryNotReachingTheMachineLeavesItsModuleOut(t *testing.T) {
with := librariesRendering()
with.Identities["radarr"] = ModuleIdentity{Module: "radarr", UID: 50020}
composed := composeLibraries(t, Resolution{Node: "far", Account: "jochen",
Modules: []Manifest{aRadarr(t), theRuntime(t)}}, with)
if why := composed.LeftOut["radarr"]; !strings.Contains(why, "does not reach far") {
t.Errorf("radarr on a machine media does not reach: %q", why)
}
delete(with.Libraries, "books")
composed = composeLibraries(t, Resolution{Node: "ace", Account: "jochen",
Modules: []Manifest{aRadarr(t), theRuntime(t)}}, with)
if why := composed.LeftOut["radarr"]; !strings.Contains(why, "books, which is not defined") {
t.Errorf("radarr with an undefined library: %q", why)
}
}
// Row 14: a module that neither declares libraries nor uses its identity is composed exactly as before — the whole
// declaration: its resources, the filter its port reaches, the contributions it makes — on an engine that keeps
// libraries or not; on a machine a library reaches, only that library's group is added. A joining module on an older
// engine is left out, naming why.
func TestAModuleThatDoesNotJoinIsComposedAsBefore(t *testing.T) {
plain := aPlain()
plain.Listens = []Listening{{Port: 8080, From: FromEverywhere, Name: "web", Why: "its page"}}
plain.Contributes = map[string]map[string]any{"reverse-proxy": {"host": "plain", "port": 8080}}
plain.Requires = []string{"reverse-proxy"}
modules := []Manifest{plain, proxy(), {Module: "nftables", Filtering: &Filtering{Into: "/etc/mesh/filter.nft"}},
theRuntime(t)}
base := Rendering{Mesh: []string{"198.51.100.2"}, OutwardLinks: []string{"eth0"}, TunnelInterface: "mesh0"}
keeping := func(node string) Rendering {
with := librariesRendering()
with.Mesh, with.OutwardLinks, with.TunnelInterface = base.Mesh, base.OutwardLinks, base.TunnelInterface
return with
}
for _, node := range []string{"far", "ace"} {
r := Resolution{Node: node, Account: "jochen", Modules: modules}
before := composeLibraries(t, r, base)
after := composeLibraries(t, r, keeping(node))
if node == "far" {
if !reflect.DeepEqual(before, after) {
t.Fatalf("on a machine no library reaches, a module that does not join composed otherwise:\n %+v\n %+v",
before, after)
}
continue
}
groups := after.Groups
after.Groups, after.Libraries = nil, nil
if !reflect.DeepEqual(before, after) {
t.Fatalf("on %s, a module that does not join composed otherwise:\n %+v\n %+v", node, before, after)
}
if len(groups) == 0 || len(after.Identities) > 0 {
t.Fatalf("on the library's home: groups %v, identities %v", groups, after.Identities)
}
filter := fmt.Sprint(byID(before.Resources)["nftables."+FilteringID()]["content"])
if !strings.Contains(filter, "8080") || received(t, before.Resources)[0].From != "plain" {
t.Fatalf("the comparison held no filter or contribution of the module: %s", filter)
}
}
if Joins(aPlain()) || !Joins(aRadarr(t)) {
t.Fatal("joining is not read from the libraries and the identity")
}
older := librariesRendering()
older.KeepsLibraries = false
composed := composeLibraries(t, Resolution{Node: "ace", Account: "jochen",
Modules: []Manifest{aRadarr(t), aPlain(), theRuntime(t)}}, older)
if why := composed.LeftOut["radarr"]; !strings.Contains(why, "older than them") {
t.Fatalf("a joining module on an older engine: %q", why)
}
if len(composed.Groups)+len(composed.Identities)+len(composed.Libraries) > 0 {
t.Fatal("an older engine was sent groups, identities or libraries")
}
if byID(composed.Resources)["plain.app"] == nil {
t.Fatal("a module that does not join was left out beside one that does")
}
}
// Row 15: no number outside the range is sent, root's above all; the composition refuses, naming why.
func TestNoNumberTheMeshGivesIsRoot(t *testing.T) {
for _, n := range []int{0, 4, 10, IdentityLow - 1, IdentityHigh + 1} {
if AllocatableID(n) {
t.Errorf("%d is allocatable", n)
}
with := librariesRendering()
with.Identities["radarr"] = ModuleIdentity{Module: "radarr", UID: n}
with.Needed = map[string]map[string]string{RuntimeModule: {"broker": "sealed-credential"}}
with.ArtifactStore = "anchor.internal:5101"
r := Resolution{Node: "ace", Account: "jochen", Modules: []Manifest{aRadarr(t), theRuntime(t)}}
if _, err := r.Compose(with); err == nil || !strings.Contains(err.Error(), "§15") {
t.Errorf("a module numbered %d composed: %v", n, err)
}
with = librariesRendering()
with.LibraryGroups["media"] = n
with.Needed = map[string]map[string]string{RuntimeModule: {"broker": "sealed-credential"}}
with.ArtifactStore = "anchor.internal:5101"
if _, err := r.Compose(with); err == nil || !strings.Contains(err.Error(), "§15") {
t.Errorf("a library group numbered %d composed: %v", n, err)
}
}
}
func libraryManifest(t *testing.T, extra map[string]any) []byte {
t.Helper()
m := map[string]any{"module": "app", "version": "1", "capabilities": []any{"container-runtime"},
"libraries": []any{map[string]any{"id": "movies", "library": "media", "path": "movies", "use": "read-write"}},
"resources": []any{map[string]any{"id": "app", "type": "container", "name": "app", "image": libraryImage,
"volumes": []any{"${library:movies}:/movies"}}}}
for k, v := range extra {
m[k] = v
}
raw, err := json.Marshal(m)
if err != nil {
t.Fatal(err)
}
return raw
}
// Rows 2 and 5 at the module check: library entries are accepted by name, sub-path and use, and refused with a host
// path, an owner, a group, a mode or a number; ${library:<id>} only in a volume source; ${identity:…} only by its four
// names; a joining module names no directory owner and declares no number setting.
func TestTheModuleCheckHoldsLibrariesAndIdentityToTheDecision(t *testing.T) {
if _, err := ParseManifest(libraryManifest(t, nil)); err != nil {
t.Fatalf("a library entry was refused: %v", err)
}
entry := func(extra map[string]any) map[string]any {
e := map[string]any{"id": "movies", "library": "media", "path": "movies", "use": "read-write"}
for k, v := range extra {
e[k] = v
}
return map[string]any{"libraries": []any{e}}
}
for name, c := range map[string]struct {
extra map[string]any
want string
}{
"a host path": {entry(map[string]any{"path": "/storage/media/movies"}), "never a host path"},
"out of it": {entry(map[string]any{"path": "../other"}), "never a host path"},
"an owner": {entry(map[string]any{"owner": "1000"}), "owner"},
"a group": {entry(map[string]any{"group": "media"}), "group"},
"a mode": {entry(map[string]any{"mode": "0775"}), "mode"},
"a number": {entry(map[string]any{"gid": 1000}), "gid"},
"a use": {entry(map[string]any{"use": "write"}), "is used \"write\""},
"in the env": {map[string]any{"resources": []any{map[string]any{"id": "app", "type": "container",
"name": "app", "image": libraryImage, "env": map[string]any{"MOVIES": "${library:movies}"}}}}, "volume source"},
"in a file": {map[string]any{"resources": []any{map[string]any{"id": "f", "type": "file", "path": "/etc/app.conf",
"mode": "0644", "content": "movies=${library:movies}"}}}, "volume source"},
"on the container side": {map[string]any{"resources": []any{map[string]any{"id": "app", "type": "container",
"name": "app", "image": libraryImage, "volumes": []any{"/srv:${library:movies}"}}}}, "volume source"},
"an unknown identity": {map[string]any{"resources": []any{map[string]any{"id": "app", "type": "container",
"name": "app", "image": libraryImage, "env": map[string]any{"HOME": "${identity:home}"}}}}, "${identity:home}"},
"a directory owner": {map[string]any{"resources": []any{map[string]any{"id": "d", "type": "directory",
"mode": "0750", "owner": "1000:1000"}}}, "names its owner"},
} {
_, err := ParseManifest(libraryManifest(t, c.extra))
if err == nil || !strings.Contains(err.Error(), c.want) {
t.Errorf("%s: %v; want refused saying %q", name, err, c.want)
}
}
// A module that does not join keeps its directory owner and its settings as they are (§14).
plain := map[string]any{"module": "plain", "version": "1",
"resources": []any{map[string]any{"id": "d", "type": "directory", "mode": "0750", "owner": "1000:1000"}}}
raw, _ := json.Marshal(plain)
if _, err := ParseManifest(raw); err != nil {
t.Errorf("a module that does not join was refused: %v", err)
}
// A setting carrying a number, or a placement's owner, is refused for a joining module, and kept for one that does
// not.
if err := JoiningSettingProblem(aRadarr(t), []Layer{{From: "ace", Values: map[string]any{"puid": "1000"}}}); err == nil ||
!strings.Contains(err.Error(), "carries a user or group number") {
t.Errorf("a joining module's puid setting: %v", err)
}
if err := JoiningSettingProblem(aPlain(), []Layer{{From: "ace", Values: map[string]any{"puid": "1000"}}}); err != nil {
t.Errorf("a plain module's puid setting: %v", err)
}
placed := []Layer{{From: "ace", Values: map[string]any{PlacesSetting: map[string]any{
"config": map[string]any{"path": "/srv/x", "owner": "1000:1000"}}}}}
if err := PlacesOwnerProblem(aRadarr(t), placed); err == nil || !strings.Contains(err.Error(), "ADR 0328") {
t.Errorf("a joining module's placed owner: %v", err)
}
if err := PlacesOwnerProblem(aPlain(), placed); err != nil {
t.Errorf("a plain module's placed owner: %v", err)
}
}
// Row 1: the libraries are the controller's own mesh-wide setting, the terminal's or the operator's Approve alone.
func TestTheLibrariesAreTheOperatorsSetting(t *testing.T) {
controller := Manifest{Module: LibrariesModule, Version: "1"}
keys := TerminalKeys(controller)
if !hasName(keys, LibrariesSetting) {
t.Fatalf("the libraries are not among the terminal's keys: %v", keys)
}
if hasName(TerminalKeys(Manifest{Module: "radarr"}), LibrariesSetting) {
t.Fatal("another module's libraries key is the terminal's")
}
if err := ComposesWith(controller, []Layer{{From: "ace", Values: map[string]any{LibrariesSetting: map[string]any{}}}},
false); err == nil {
t.Fatal("libraries set on one machine were accepted")
}
if _, err := ParseLibraries(map[string]any{"media": map[string]any{"home": "ace", "root": "/etc/media"}}); err == nil {
t.Fatal("a library in the machine's own tree was accepted")
}
}
// What the node-engine this controller pins (mesh-host #95, ADR 0328's half) makes of a composed declaration with
// libraries and a joining module: nothing refused.
func TestTheEngineTakesWhatIsComposedForLibraries(t *testing.T) {
for _, node := range []string{"ace", "novox"} {
r := Resolution{Node: node, Account: "jochen", Modules: []Manifest{aRadarr(t), theRuntime(t)}}
with := librariesRendering()
with.RunnerJoinsOwnGroups = true // contract 7 (mesh-host #97): the runner on identities, setgid on own directories
with.Libraries["books"] = Library{Name: "books", Home: "ace", Root: "/storage/books", Reach: []string{"novox"}}
composed := composeLibraries(t, r, with)
body, err := json.Marshal(map[string]any{"declaration": validate.Version, "resources": composed.Resources,
"groups": composed.Groups, "identities": composed.Identities, "libraries": composed.Libraries})
if err != nil {
t.Fatal(err)
}
if problems := validate.Declaration(body); len(problems) > 0 {
t.Fatalf("%s: the pinned node-engine refuses it: %v", node, problems)
}
}
}
// novox/hq ADR 0334 §1, §2 (the review of 7ea6d36e, item 2): a joining module's own directories are owned by its user
// and its own group, not the library's; and the tool runner's account joins the own group of every joining module whose
// tools run on the machine — sent only to an engine that reads it (link.OwnGroupContract).
func TestAJoiningModulesOwnDirectoriesAreItsOwnGroups(t *testing.T) {
r := Resolution{Node: "ace", Account: "jochen", Modules: []Manifest{aRadarr(t), theRuntime(t)}}
got := byID(composeLibraries(t, r, librariesRendering()).Resources)
if owner := got["radarr.config"]["owner"]; owner != "50020:50020" {
t.Errorf("its directory is owned by %v; want its user and its own group", owner)
}
// ADR 0334 v4: with setgid, so what is made in it later keeps mesh-radarr though the process's primary group is the
// library's.
if mode := got["radarr.config"]["mode"]; mode != "0750" {
t.Errorf("an engine that refuses setgid modes was sent %v", mode)
}
if env := got["radarr.app"]["env"].(map[string]any); env["PGID"] != "50010" {
t.Errorf("its process's primary group is %v; want the library's, so what it makes there keeps the group", env["PGID"])
}
older := composeLibraries(t, r, librariesRendering())
if older.Identities[0].Runner != "" {
t.Errorf("an engine that cannot put the runner in a module's own group was told %q", older.Identities[0].Runner)
}
with := librariesRendering()
with.RunnerJoinsOwnGroups = true
newer := composeLibraries(t, r, with)
if got := newer.Identities[0].Runner; got != "jochen" {
t.Errorf("the runner joins %q's own group; want jochen, whose tools it runs", got)
}
if mode := byID(newer.Resources)["radarr.config"]["mode"]; mode != "2750" {
t.Errorf("its directory's mode is %v; want 2750, setgid on its own 0750", mode)
}
}
// novox/hq ADR 0334 §3: a joining module whose tools name a directory whose mode closes it to its group is refused at
// the module check, naming the directory.
func TestAToolsDirectoryIsOpenToItsGroup(t *testing.T) {
closed := map[string]any{
"libraries": []any{map[string]any{"id": "movies", "library": "media", "path": "movies", "use": "read"}},
"resources": []any{
map[string]any{"id": "config", "type": "directory", "mode": "0700"},
map[string]any{"id": "app", "type": "container", "name": "app", "image": libraryImage,
"volumes": []any{"${dir:config}:/config", "${library:movies}:/movies"}}},
"build": map[string]any{"artifacts": []any{map[string]any{"name": "tools", "kind": "bundle",
"language": "go", "system": "arch", "from": "cmd/app", "binary": "app", "loads": []any{"app"},
"env": map[string]any{"APP_CONFIG": "${dir:config}/config.xml"}}}}}
_, err := ParseManifest(libraryManifest(t, closed))
if err == nil || !strings.Contains(err.Error(), "config") || !strings.Contains(err.Error(), "ADR 0334") {
t.Fatalf("a tools' directory closed to its group: %v", err)
}
open := closed
open["resources"] = []any{
map[string]any{"id": "config", "type": "directory", "mode": "0750"},
map[string]any{"id": "app", "type": "container", "name": "app", "image": libraryImage,
"volumes": []any{"${dir:config}:/config", "${library:movies}:/movies"}}}
if _, err := ParseManifest(libraryManifest(t, open)); err != nil {
t.Fatalf("a tools' directory open to its group was refused: %v", err)
}
}
// The review of 7ea6d36e, item 6: a service's unit takes the identity too — `User=${identity:user}` — at the module
// check and in what is composed.
func TestAServicesUnitTakesTheIdentity(t *testing.T) {
unit := "[Service]\nUser=${identity:user}\nGroup=${identity:group}\nUMask=0002\nExecStart=/usr/bin/importer\n"
raw := libraryManifest(t, map[string]any{"resources": []any{
map[string]any{"id": "app", "type": "container", "name": "app", "image": libraryImage,
"volumes": []any{"${library:movies}:/movies"}},
map[string]any{"id": "watch", "type": "service", "name": "app-watch", "unit": unit}}})
m, err := ParseManifest(raw)
if err != nil {
t.Fatalf("a unit naming the identity was refused: %v", err)
}
with := librariesRendering()
with.Identities["app"] = ModuleIdentity{Module: "app", UID: 50030}
composed := composeLibraries(t, Resolution{Node: "ace", Modules: []Manifest{m}}, with)
got := fmt.Sprint(byID(composed.Resources)["app.watch"]["unit"])
if !strings.Contains(got, "User=mesh-app\nGroup=mesh-media\n") {
t.Fatalf("the unit reads %q", got)
}
}
+5
View File
@@ -536,6 +536,9 @@ type Manifest struct {
// `directory` resource, which the mesh creates and owns: an access is mounted and nothing
// about it is reconciled, and several modules may name the same one without conflict.
Accesses []Access `json:"accesses,omitempty"`
// Libraries are the libraries this module uses, each by name, sub-path and use (novox/hq ADR 0328 §2): where a
// library is, and who may write it, is the operator's definition and the mesh's, never the module's.
Libraries []LibraryUse `json:"libraries,omitempty"`
// Computed names something in the control plane that works this module's resources out per
// node, instead of them being fixed here.
@@ -1860,6 +1863,8 @@ func ParseManifest(raw []byte) (Manifest, error) {
problems = append(problems, replacesProblems(m)...)
problems = append(problems, UnitSettingProblems(m)...)
problems = append(problems, OwnerSettingProblems(m)...)
// Its libraries and its identity, and what a module that joins is held to (novox/hq ADR 0328).
problems = append(problems, libraryProblems(m)...)
problems = append(problems, endpointNameProblems(m)...)
problems = append(problems, RouteProblems(m)...)
for _, port := range m.Guards {
+29 -34
View File
@@ -1,5 +1,7 @@
package catalogue
import "path"
// The machine's own resolver (novox/hq ADR 0247).
//
// **Most machines have none.** Every machine lists the mesh's resolvers in /etc/resolv.conf and nothing
@@ -66,43 +68,39 @@ func holdsSeat(m Manifest, seat string) bool {
return false
}
// rendersResolverFile says whether a module asks the mesh to render the machine's resolver file.
func rendersResolverFile(m Manifest) bool {
for _, f := range m.Facts {
if !f.Home && f.Path == ResolverFile {
return true
}
}
return false
}
// ResolverFileOwner is the module that writes the machine's resolver file among the modules on one
// machine (novox/hq ADR 0247): the holder of node-resolver when one renders it, else nobody is named here
// and the file is whoever's it always was — the uplink holder's (ADR 0223).
func ResolverFileOwner(modules []Manifest) string {
for _, m := range modules {
if holdsSeat(m, ResolverSeat) && rendersResolverFile(m) {
return m.Module
}
}
return ""
}
// stepsBack is the module as it composes on a machine whose resolver file is the node-resolver holder's:
// **the uplink holder's rendering of that file is left out**, and nothing else of it changes. Only the
// uplink's holder steps back — any other module rendering the file beside the resolver's is still two
// owners of one path, and is refused as before.
// stepsBack is the module as it composes among the modules of one machine (novox/hq ADR 0287,
// ADR 0247):
// **where it holds a seat that names a path stepping back for another seat, and that other seat's holder
// is on the machine and declares the path, its own rendering of the path is left out**, and nothing else
// of it changes. Only a file the mesh renders for it steps back, as before: a resource on the path is
// composed as declared (the declaration copies a module's resources whole), so an uplink holder declaring
// the resolver file as a resource beside the resolver's holder is two owners of one path, and refused. The stepping back is the seat's definition — node-uplink stepping back from the resolver
// file for node-resolver — so any other module declaring the path beside the owner's is still two owners
// of one path, and is refused as before.
func stepsBack(m Manifest, modules []Manifest) Manifest {
if !holdsSeat(m, "node-uplink") || !rendersResolverFile(m) {
return m
drop := map[string]bool{}
for _, c := range m.Claims {
seat, known := SeatNamed(c.Name)
if !known {
continue
}
for _, p := range seat.Paths {
if p.StepsBackFor == "" {
continue
}
for _, o := range modules {
if o.Module != m.Module && holdsSeat(o, p.StepsBackFor) && declaresPath(o, p.Path) {
drop[p.Path] = true
}
}
}
}
owner := ResolverFileOwner(modules)
if owner == "" || owner == m.Module {
if len(drop) == 0 {
return m
}
facts := make(map[string]RosterFile, len(m.Facts))
for name, f := range m.Facts {
if !f.Home && f.Path == ResolverFile {
if !f.Home && drop[path.Clean(f.Path)] {
continue
}
facts[name] = f
@@ -113,9 +111,6 @@ func stepsBack(m Manifest, modules []Manifest) Manifest {
// steppedBack is every module of one machine as it composes there (stepsBack, each).
func steppedBack(modules []Manifest) []Manifest {
if ResolverFileOwner(modules) == "" {
return modules
}
out := make([]Manifest, len(modules))
for i, m := range modules {
out[i] = stepsBack(m, modules)
+32 -3
View File
@@ -1,6 +1,7 @@
package catalogue
import (
"errors"
"fmt"
"path/filepath"
"regexp"
@@ -270,6 +271,31 @@ func Places(m Manifest, layers []Layer) (map[string]Placement, error) {
// It refuses an id the module declares no access under, and a path that is not absolute. An
// access declared by path alone cannot be placed — it has no name to place it by.
func AccessPlaces(m Manifest, layers []Layer) (map[string]string, error) {
return accessPlaces(m, layers, true)
}
// UndeclaredAccessError is a layer placing an access the definition does not declare (novox/hq issue 575): refused
// where a setting is stored, unless a newer build of the module declares it; passed by where a machine is composed.
type UndeclaredAccessError struct {
Module, ID string
said string
}
func (e *UndeclaredAccessError) Error() string { return e.said }
// UndeclaredAccessPlaces refuses a layer that places an access the definition does not declare.
func UndeclaredAccessPlaces(m Manifest, layers []Layer) error {
_, err := accessPlaces(m, layers, true)
var undeclared *UndeclaredAccessError
if errors.As(err, &undeclared) {
return err
}
return nil
}
// accessPlaces is AccessPlaces; where strict is false an access the definition does not declare is passed by
// (novox/hq issue 575), as a placement stored ahead of the build that declares it.
func accessPlaces(m Manifest, layers []Layer, strict bool) (map[string]string, error) {
declared := map[string]bool{}
for _, a := range m.Accesses {
if a.ID != "" {
@@ -289,9 +315,12 @@ func AccessPlaces(m Manifest, layers []Layer) (map[string]string, error) {
}
for id, body := range blocks {
if !declared[id] {
return nil, fmt.Errorf(
if !strict {
continue
}
return nil, &UndeclaredAccessError{Module: m.Module, ID: id, said: fmt.Sprintf(
"%s places the access %q, which it does not declare — the setting reaches "+
"nothing. It declares %s", m.Module, id, orNothing(namesOfAccesses(m)))
"nothing. It declares %s", m.Module, id, orNothing(namesOfAccesses(m)))}
}
path, _ := body.(string)
path = strings.TrimSpace(path)
@@ -324,7 +353,7 @@ type placedAccess struct {
// placed one, the definition's where it carries a default, and refused where neither says — an
// access that resolves to nowhere would reach the machine as a mount of nothing.
func accessesFor(m Manifest, layers []Layer) ([]placedAccess, map[string]string, error) {
placed, err := AccessPlaces(m, layers)
placed, err := accessPlaces(m, layers, false)
if err != nil {
return nil, nil, err
}
+3 -1
View File
@@ -957,7 +957,9 @@ func checkResources(modules []Manifest) []string {
// Judged as the machine composes them: where the node-resolver holder writes the resolver file, the
// uplink holder's rendering of it is not composed, so it is not a second owner (novox/hq ADR 0247).
modules = steppedBack(modules)
var problems []string
// And only a claimant of a seat naming a path declares it — the controller's generated resources
// included, which are judged here at composition (novox/hq ADR 0287).
problems := seatPathsOnMachine(modules)
owner := map[string]string{}
ownedPath := map[string]string{} // path → owning module, for the access check below
packagePresent, packageAbsent := map[string][]string{}, map[string][]string{}
+4
View File
@@ -410,6 +410,10 @@ func bundleWords(m Manifest, with Rendering) (map[string]string, error) {
if value, err = portsFilledInto(value, m.Module+"'s bundle "+b.Name+" ("+word+")", m.Module, m.Listens, with); err != nil {
return nil, err
}
// A joining module's identity and its libraries' host paths: its tools run on the machine (ADR 0328 §5).
if value, err = bundleLibraryFill(value, m, with); err != nil {
return nil, err
}
if out == nil {
out = map[string]string{}
}
+315
View File
@@ -0,0 +1,315 @@
package catalogue
import (
"fmt"
"path"
"strings"
)
// The paths a seat's holder owns (novox/hq ADR 0287).
//
// **Who owns a machine-wide file is the role's question, answered once.** Until this, which module owned
// /etc/hosts or /etc/resolv.conf was decided record by record (ADRs 0199, 0222, 0223, 0241) and enforced
// only after the fact, by checkResources finding two modules on one machine declaring the same path: a
// module that did not hold node-hostname could declare /etc/hosts on a machine where nobody held it, and
// nothing refused it. Now a seat's definition names the paths the role owns on any machine, and three
// refusals follow from it, each where the mesh can still say no:
//
// - a claim whose module does not declare every path its seat names, by CanHold, at registration and at
// handover;
// - a module declaring a path a seat names while claiming none of the seats that name it, at
// registration (CatalogueProblems), and what the controller generates for a machine, at composition;
// - a module's own seat naming a path another seat already names, at registration — the second loses.
//
// Reading a seat's path is access, never a resource, so it stays allowed.
// SeatPath is one path a seat's holder owns: absolute, literal, on the machine where the holder runs.
type SeatPath struct {
Path string `json:"path"`
// StepsBackFor is a node seat whose holder owns the path instead, on a machine where that seat is held
// (novox/hq ADR 0287, ADR 0247): this seat's holder still declares the path, and its declaration is not
// composed there. Only the mesh's own seats step back, each by a record.
StepsBackFor string `json:"steps-back-for,omitempty"`
}
// seatPathProblem is what is wrong with one path a seat names, judged alone: absolute, written literally
// with no placeholder, and clean, so it compares with what a module declares byte for byte.
func seatPathProblem(seat, scope, p string) string {
switch {
case scope != ScopeNode:
return fmt.Sprintf("the seat %s names the path %q and is a %s seat; only a node seat names paths, "+
"because a path is a machine's (novox/hq ADR 0287)", seat, p, scope)
case strings.Contains(p, "${"):
return fmt.Sprintf("the seat %s names the path %q, which has a placeholder; a seat's path is written "+
"literally (novox/hq ADR 0287)", seat, p)
case !strings.HasPrefix(p, "/") || path.Clean(p) != p:
return fmt.Sprintf("the seat %s names the path %q, which is not a clean absolute path (novox/hq ADR 0287)",
seat, p)
case path.Dir(p) == "/":
// The root or a top-level directory is every role's, never one role's own (ADR 0287).
return fmt.Sprintf("the seat %s names the path %q, the root or a top-level directory, which no one "+
"role owns (novox/hq ADR 0287)", seat, p)
}
return ""
}
// SeatPathProblems is what is wrong with the paths a set of the mesh's own seats names (novox/hq ADR
// 0287): only a node seat names one, each absolute and literal; every path has exactly one seat naming it
// without stepping back; and a seat steps back only for another node seat naming the same path without
// stepping back, so the stepping back cannot cycle.
func SeatPathProblems(set []Seat) []string {
var problems []string
owners := map[string][]string{} // path → the seats naming it without stepping back
stepping := map[string][]SeatPath{} // seat → the paths it names stepping back
byName := map[string]Seat{}
for _, s := range set {
byName[s.Name] = s
for _, p := range s.Paths {
if why := seatPathProblem(s.Name, s.Scope, p.Path); why != "" {
problems = append(problems, why)
}
if p.StepsBackFor == "" {
owners[p.Path] = append(owners[p.Path], s.Name)
} else {
stepping[s.Name] = append(stepping[s.Name], p)
}
}
}
for _, s := range set {
for _, p := range stepping[s.Name] {
other, known := byName[p.StepsBackFor]
switch {
case !known || other.Scope != ScopeNode:
problems = append(problems, fmt.Sprintf("the seat %s steps back from %q for %q, which is not a "+
"node seat of the mesh's (novox/hq ADR 0287)", s.Name, p.Path, p.StepsBackFor))
case !namesWithoutSteppingBack(other, p.Path):
problems = append(problems, fmt.Sprintf("the seat %s steps back from %q for %s, which does not "+
"own that path itself; a seat steps back only for the one that owns it (novox/hq ADR 0287)",
s.Name, p.Path, other.Name))
}
}
}
for _, p := range sortedKeys(owners) {
if len(owners[p]) > 1 {
problems = append(problems, fmt.Sprintf("the path %q is named by %s, each without stepping back; "+
"no two seats name one path (novox/hq ADR 0287)", p, strings.Join(owners[p], " and ")))
}
}
for _, s := range set {
for _, p := range stepping[s.Name] {
if len(owners[p.Path]) == 0 {
problems = append(problems, fmt.Sprintf("the path %q has no seat naming it without stepping back "+
"(novox/hq ADR 0287)", p.Path))
}
}
}
return problems
}
func namesWithoutSteppingBack(s Seat, p string) bool {
for _, sp := range s.Paths {
if sp.Path == p && sp.StepsBackFor == "" {
return true
}
}
return false
}
// declaredPaths is every machine path a module declares it owns: a resource's literal path, and a file
// the mesh renders for it outside the operator's home. An access is not here — reading is not owning.
//
// **Cleaned, because the machine writes the cleaned path**: /etc//hosts, /etc/./hosts, /etc/../etc/hosts and
// /etc/hosts/ are all /etc/hosts to it, and a seat's path is clean by its own rule, so a declaration compared
// as written would pass for a path nobody named.
func declaredPaths(m Manifest) map[string]bool {
out := map[string]bool{}
for _, r := range m.Resources {
if p, ok := r["path"].(string); ok && strings.HasPrefix(p, "/") {
out[path.Clean(p)] = true
}
}
for _, f := range m.Facts {
if !f.Home && strings.HasPrefix(f.Path, "/") {
out[path.Clean(f.Path)] = true
}
}
return out
}
// declaresPath says whether a module declares one path, as a resource or as a file the mesh renders.
func declaresPath(m Manifest, p string) bool { return declaredPaths(m)[p] }
// missingPaths is each path a seat names that a module does not declare, in the seat's order.
func missingPaths(m Manifest, paths []SeatPath) []string {
if len(paths) == 0 {
return nil
}
has := declaredPaths(m)
var missing []string
for _, p := range paths {
if !has[p.Path] {
missing = append(missing, p.Path)
}
}
return missing
}
// pathSeats is which seats name each path: the mesh's own, and then those modules declare, in the order
// given. A module's seat naming a path a seat already names loses it, and the refusal says so (novox/hq
// ADR 0287 — as for a seat's name, the second declaration loses).
func pathSeats(order []string, shelf Shelf, declaredBy map[string]string) (map[string][]string, []string) {
named := map[string][]string{}
for _, s := range Seats() {
for _, p := range s.Paths {
named[p.Path] = append(named[p.Path], s.Name)
}
}
var problems []string
for _, module := range order {
for _, s := range shelf[module].DefinesSeats {
if declaredBy != nil && declaredBy[s.Name] != module {
continue // a second declaration of the seat's name, refused for that already
}
for _, p := range s.Paths {
if others := named[p]; len(others) > 0 {
problems = append(problems, fmt.Sprintf(
"%s declares the seat %s naming the path %q, which %s already names; no two seats "+
"name one path (novox/hq ADR 0287)", module, s.Name, p, strings.Join(others, " and ")))
continue
}
named[p] = append(named[p], s.Name)
}
}
}
return named, problems
}
// unclaimedSeatPaths is each path a module declares that a seat names while it claims none of the seats
// naming it (novox/hq ADR 0287): only a claimant declares a seat's path, and a module claiming any one of
// them may.
func unclaimedSeatPaths(m Manifest, named map[string][]string) []string {
claims := map[string]bool{}
for _, c := range m.Claims {
claims[canonicalSeat(c.Name)] = true
}
var problems []string
declared := declaredPaths(m)
for _, p := range sortedKeys(declared) {
seats := named[p]
if len(seats) == 0 {
continue
}
claimsOne := false
for _, s := range seats {
claimsOne = claimsOne || claims[s]
}
if !claimsOne {
problems = append(problems, fmt.Sprintf(
"%s declares the path %q, which the seat %s names, and claims none of them; only a holder of "+
"a seat naming a path declares it (novox/hq ADR 0287)", m.Module, p, strings.Join(seats, " and ")))
}
}
return problems
}
// declaredSeatPathProblems is what one module's own seat declaration can be judged on alone: a path
// only on a node seat, absolute, literal and clean.
func declaredSeatPathProblems(m Manifest, s SeatDeclaration) []string {
var problems []string
for _, p := range s.Paths {
if why := seatPathProblem(s.Name, s.At(), p); why != "" {
problems = append(problems, m.Module+": "+why)
}
}
return problems
}
// asSeatPaths is a module's own seat's paths in the shape the mesh's seats name theirs. A module's seat
// never steps back: only the mesh's own seats do, each by a record.
func asSeatPaths(paths []string) []SeatPath {
out := make([]SeatPath, 0, len(paths))
for _, p := range paths {
out = append(out, SeatPath{Path: p})
}
return out
}
// PathOwner is the module that owns one seat's path on one machine (novox/hq ADR 0287): the seat naming
// the path, and that seat's holder there, with the stepping back applied.
type PathOwner struct {
Path string `json:"path"`
Node string `json:"node"`
Seat string `json:"seat"`
Module string `json:"module"`
}
// PathOwners is, for every path a seat in the set names and every machine where a seat naming it is held,
// who owns the path there. A seat that steps back for another held on the same machine is passed over.
func PathOwners(set []Seat, held []Held) []PathOwner {
// node → seat → holding module
holders := map[string]map[string]string{}
for _, h := range held {
s, ok := SeatNamed(h.Claim)
if !ok || h.Scope != ScopeNode || s.Scope != ScopeNode {
continue
}
if holders[h.Node] == nil {
holders[h.Node] = map[string]string{}
}
if _, taken := holders[h.Node][s.Name]; !taken {
holders[h.Node][s.Name] = h.Module
}
}
naming := map[string][]Seat{} // path → the seats naming it
steps := map[string]map[string]string{} // path → seat → the seat it steps back for
for _, s := range set {
for _, p := range s.Paths {
naming[p.Path] = append(naming[p.Path], s)
if p.StepsBackFor != "" {
if steps[p.Path] == nil {
steps[p.Path] = map[string]string{}
}
steps[p.Path][s.Name] = p.StepsBackFor
}
}
}
var out []PathOwner
for _, p := range sortedKeys(naming) {
for _, node := range sortedKeys(holders) {
for _, s := range naming[p] {
module, holds := holders[node][s.Name]
if !holds {
continue
}
if forSeat := steps[p][s.Name]; forSeat != "" {
if _, there := holders[node][forSeat]; there {
continue
}
}
out = append(out, PathOwner{Path: p, Node: node, Seat: s.Name, Module: module})
break
}
}
}
return out
}
// seatPathsOnMachine is each path the modules of one machine declare that a seat names, for a module
// claiming none of the seats naming it (novox/hq ADR 0287). Registration refuses the same of a manifest;
// this holds what the controller generates for a machine to it too, because a computed module's resources
// exist only once composed (checkResources is asked them there). The seats are the mesh's own and those the
// machine's modules declare.
func seatPathsOnMachine(modules []Manifest) []string {
shelf := Shelf{}
var order []string
for _, m := range modules {
shelf[m.Module] = m
order = append(order, m.Module)
}
named, _ := pathSeats(order, shelf, nil)
var problems []string
for _, m := range modules {
problems = append(problems, unclaimedSeatPaths(m, named)...)
}
return problems
}
+311
View File
@@ -0,0 +1,311 @@
package catalogue
import (
"strings"
"testing"
)
// Defends novox/hq ADR 0287 (issue 412): a seat names the paths its holder owns, a claim without them is
// refused, and only a claimant declares a seat's path.
// The starting set, exactly: node-hostname names /etc/hosts and /etc/hostname, node-uplink the resolver
// file stepping back for node-resolver, node-resolver the resolver file, and every other seat names none.
func TestTheSeatsThatNamePathsAreTheStartingSet(t *testing.T) {
want := map[string]string{
"node-hostname": "/etc/hosts /etc/hostname",
"node-uplink": "/etc/resolv.conf>node-resolver",
ResolverSeat: "/etc/resolv.conf",
}
for _, s := range DefaultSeats() {
var got []string
for _, p := range s.Paths {
if p.StepsBackFor != "" {
got = append(got, p.Path+">"+p.StepsBackFor)
} else {
got = append(got, p.Path)
}
}
if strings.Join(got, " ") != want[s.Name] {
t.Errorf("%s names %v; ADR 0287 gives it %q", s.Name, got, want[s.Name])
}
}
}
// The compiled set keeps the rules: node seats only, absolute and literal, one seat per path without
// stepping back, and a stepping back that cannot cycle.
func TestTheCompiledSeatPathsKeepTheRules(t *testing.T) {
if p := SeatPathProblems(DefaultSeats()); len(p) != 0 {
t.Fatalf("the compiled set breaks ADR 0287: %v", p)
}
for _, c := range []struct {
why string
set []Seat
want string
}{
{"a mesh seat naming a path", []Seat{{Name: "mesh-x", Scope: ScopeMesh, Paths: []SeatPath{{Path: "/etc/x"}}}},
"only a node seat names paths"},
{"a placeholder", []Seat{{Name: "node-x", Scope: ScopeNode, Paths: []SeatPath{{Path: "/etc/${setting:x}"}}}},
"placeholder"},
{"a relative path", []Seat{{Name: "node-x", Scope: ScopeNode, Paths: []SeatPath{{Path: "etc/x"}}}},
"not a clean absolute path"},
{"two owners", []Seat{
{Name: "node-a", Scope: ScopeNode, Paths: []SeatPath{{Path: "/etc/x"}}},
{Name: "node-b", Scope: ScopeNode, Paths: []SeatPath{{Path: "/etc/x"}}}}, "no two seats name one path"},
{"a cycle", []Seat{
{Name: "node-a", Scope: ScopeNode, Paths: []SeatPath{{Path: "/etc/x", StepsBackFor: "node-b"}}},
{Name: "node-b", Scope: ScopeNode, Paths: []SeatPath{{Path: "/etc/x", StepsBackFor: "node-a"}}}},
"does not own that path itself"},
{"stepping back for nobody", []Seat{
{Name: "node-a", Scope: ScopeNode, Paths: []SeatPath{{Path: "/etc/x", StepsBackFor: "node-z"}}}},
"not a node seat"},
} {
if got := strings.Join(SeatPathProblems(c.set), "; "); !strings.Contains(got, c.want) {
t.Errorf("%s: %q does not say %q", c.why, got, c.want)
}
}
}
// What the store's rows carry is never the paths: a row read back keeps the compiled ones.
func TestTheSeatPathsSurviveTheStoresRows(t *testing.T) {
before := seats
t.Cleanup(func() { seats = before })
UseSeats([]Seat{{Name: "node-hostname", Scope: ScopeNode}})
s, _ := SeatNamed("node-hostname")
if len(s.Paths) != 2 {
t.Fatalf("node-hostname read back from its row names %v", s.Paths)
}
}
func hostnameSeat(t *testing.T) Seat {
t.Helper()
s, ok := SeatNamed("node-hostname")
if !ok {
t.Fatal("no node-hostname seat")
}
return s
}
func hostnameHolder(resources ...map[string]any) Manifest {
return Manifest{Module: "names", Version: "1", Claims: []Claim{{Name: "node-hostname", Scope: ScopeNode,
Serves: []string{"entries", "add", "remove"}}}, Resources: resources}
}
// A holder declares every path its seat names, as a resource or as a file the mesh renders for it, or
// CanHold refuses it naming the seat and each missing path — the judgement registration and handover share.
func TestAClaimWithoutTheSeatsPathsIsRefused(t *testing.T) {
seat := hostnameSeat(t)
hosts := map[string]any{"id": "hosts", "type": "file", "path": "/etc/hosts", "content": "x"}
name := map[string]any{"id": "name", "type": "file", "path": "/etc/hostname", "content": "x"}
if err := CanHold(hostnameHolder(hosts, name), seat); err != nil {
t.Errorf("a holder declaring both paths as resources was refused: %v", err)
}
rendered := hostnameHolder(hosts)
rendered.Facts = map[string]RosterFile{"name": {Path: "/etc/hostname", Template: "{{.Node}}\n"}}
if err := CanHold(rendered, seat); err != nil {
t.Errorf("a holder declaring a path as a file the mesh renders was refused: %v", err)
}
err := CanHold(hostnameHolder(), seat)
if err == nil {
t.Fatal("a holder declaring neither path was accepted")
}
for _, want := range []string{"node-hostname", "/etc/hosts", "/etc/hostname", "ADR 0287"} {
if !strings.Contains(err.Error(), want) {
t.Errorf("the refusal does not name %s: %v", want, err)
}
}
// A file under the operator's home is placed per account: it is not the machine's path.
home := hostnameHolder(hosts)
home.Facts = map[string]RosterFile{"name": {Path: "/etc/hostname", Home: true, Template: "x"}}
if err := CanHold(home, seat); err == nil {
t.Error("a file in the operator's home counted as the machine's /etc/hostname")
}
// At registration, the same refusal.
if got := strings.Join(CatalogueProblems(Shelf{"names": hostnameHolder(hosts)}), "; "); !strings.Contains(got,
"does not declare /etc/hostname") {
t.Errorf("registration let a holder without /etc/hostname through: %q", got)
}
}
// Only a claimant declares a seat's path: a module claiming no seat that names /etc/hosts and declaring
// it is refused at registration naming the seat; the uplink's holder and the resolver's both declaring
// the resolver file are each a claimant of a seat naming it. Reading a seat's path is access, and allowed.
func TestOnlyAClaimantDeclaresASeatsPath(t *testing.T) {
stranger := Manifest{Module: "stranger", Version: "1", Resources: []map[string]any{
{"id": "hosts", "type": "file", "path": "/etc/hosts", "into": "block", "content": "10.0.0.1 x\n"}}}
got := strings.Join(CatalogueProblems(Shelf{"stranger": stranger}), "; ")
if !strings.Contains(got, `stranger declares the path "/etc/hosts", which the seat node-hostname names`) {
t.Errorf("a module declaring /etc/hosts without claiming node-hostname was not refused: %q", got)
}
// However the path is spelled (novox/hq issue 412 review).
unclean := Manifest{Module: "unclean", Version: "1", Resources: []map[string]any{
{"id": "hosts", "type": "file", "path": "/etc/../etc/hosts", "content": "10.0.0.1 x\n"}}}
if got := strings.Join(CatalogueProblems(Shelf{"unclean": unclean}), "; "); !strings.Contains(got, "node-hostname") {
t.Errorf("/etc/../etc/hosts from a module not claiming node-hostname was not refused: %q", got)
}
reader := Manifest{Module: "reader", Version: "1", Accesses: []Access{{ID: "hosts", Path: "/etc/hosts"}}}
if got := CatalogueProblems(Shelf{"reader": reader}); len(got) != 0 {
t.Errorf("reading /etc/hosts was refused: %v", got)
}
uplink := Manifest{Module: "uplink", Version: "1", Claims: []Claim{{Name: "node-uplink", Scope: ScopeNode}},
Facts: map[string]RosterFile{"resolvers": {Path: ResolverFile, Template: "nameserver 10.42.0.1\n"}}}
resolver := localResolver()
resolver.Claims[0].Serves = []string{"routes", "route", "unroute"}
if got := CatalogueProblems(Shelf{"uplink": uplink, resolver.Module: resolver}); len(got) != 0 {
t.Errorf("the uplink's holder and the resolver's declaring the resolver file were refused: %v", got)
}
}
// A module's own seat names paths under the same rules: a node seat only, literal, and none another seat
// names — the second declaration loses. Its holder declares them, as a mesh seat's does.
func TestAModulesOwnSeatNamesPathsUnderTheSameRules(t *testing.T) {
meshScoped := Manifest{Module: "a", Version: "1", DefinesSeats: []SeatDeclaration{
{Name: "a-seat", Paths: []string{"/etc/a"}}}}
if got := strings.Join(declaredSeatProblems(meshScoped), "; "); !strings.Contains(got, "only a node seat names paths") {
t.Errorf("a module's mesh seat naming a path was accepted: %q", got)
}
placeholder := Manifest{Module: "a", Version: "1", DefinesSeats: []SeatDeclaration{
{Name: "a-seat", Scope: ScopeNode, Paths: []string{"${dir:state}/a"}}}}
if got := strings.Join(declaredSeatProblems(placeholder), "; "); !strings.Contains(got, "placeholder") {
t.Errorf("a module's seat naming a path with a placeholder was accepted: %q", got)
}
taken := Manifest{Module: "b", Version: "1", DefinesSeats: []SeatDeclaration{
{Name: "b-seat", Scope: ScopeNode, Paths: []string{"/etc/hosts"}}}}
if got := strings.Join(CatalogueProblems(Shelf{"b": taken}), "; "); !strings.Contains(got,
`b declares the seat b-seat naming the path "/etc/hosts", which node-hostname already names`) {
t.Errorf("a module's seat naming /etc/hosts was accepted: %q", got)
}
own := Manifest{Module: "c", Version: "1",
DefinesSeats: []SeatDeclaration{{Name: "c-seat", Scope: ScopeNode, Paths: []string{"/etc/c.conf"}}},
Claims: []Claim{{Name: "c-seat", Scope: ScopeNode}}}
if got := strings.Join(CatalogueProblems(Shelf{"c": own}), "; "); !strings.Contains(got,
"c claims c-seat but does not declare /etc/c.conf") {
t.Errorf("a holder of a module's seat without its path was accepted: %q", got)
}
own.Resources = []map[string]any{{"id": "conf", "type": "file", "path": "/etc/c.conf", "content": "x"}}
if got := CatalogueProblems(Shelf{"c": own}); len(got) != 0 {
t.Errorf("a holder declaring its own seat's path was refused: %v", got)
}
other := Manifest{Module: "d", Version: "1", Resources: []map[string]any{
{"id": "conf", "type": "file", "path": "/etc/c.conf", "content": "x"}}}
if got := strings.Join(CatalogueProblems(Shelf{"c": own, "d": other}), "; "); !strings.Contains(got,
`d declares the path "/etc/c.conf", which the seat c-seat names`) {
t.Errorf("a module not claiming c-seat declared its path: %q", got)
}
}
type writesHosts struct{}
func (writesHosts) Resources(string) ([]map[string]any, bool, error) {
return []map[string]any{{"id": "hosts", "type": "file", "path": "/etc/hosts", "into": "block",
"content": "10.42.0.1 anchor\n"}}, true, nil
}
// The controller is bound too: what it generates for a machine on a seat's path is refused at composition.
func TestAGeneratedResourceOnASeatsPathIsRefused(t *testing.T) {
r := Resolution{Node: "workstation", Modules: []Manifest{
{Module: "mesh-names", Computed: "mesh-names"},
}}
_, err := r.Declaration(Rendering{Generators: map[string]Generator{"mesh-names": writesHosts{}}})
if err == nil {
t.Fatal("a generated resource on /etc/hosts was composed")
}
for _, want := range []string{"mesh-names", "/etc/hosts", "node-hostname"} {
if !strings.Contains(err.Error(), want) {
t.Errorf("the refusal does not name %s: %v", want, err)
}
}
}
// Who owns a seat's path on a machine: the seat naming it and its holder there, with the stepping back
// applied — the resolver's holder where node-resolver is held, else the uplink's.
func TestPathOwnersApplyTheSteppingBack(t *testing.T) {
held := []Held{
{Claim: "node-uplink", Scope: ScopeNode, Node: "laptop", Module: "networkmanager"},
{Claim: ResolverSeat, Scope: ScopeNode, Node: "laptop", Module: "systemd-resolved"},
{Claim: "node-uplink", Scope: ScopeNode, Node: "server", Module: "systemd-networkd"},
{Claim: "node-hostname", Scope: ScopeNode, Node: "server", Module: "hostname"},
}
var got []string
for _, o := range PathOwners(Seats(), held) {
got = append(got, o.Path+" "+o.Node+" "+o.Seat+" "+o.Module)
}
want := []string{
"/etc/hostname server node-hostname hostname",
"/etc/hosts server node-hostname hostname",
"/etc/resolv.conf laptop node-resolver systemd-resolved",
"/etc/resolv.conf server node-uplink systemd-networkd",
}
if strings.Join(got, "\n") != strings.Join(want, "\n") {
t.Errorf("the owners are\n%s\nnot\n%s", strings.Join(got, "\n"), strings.Join(want, "\n"))
}
}
// The stepping back comes from the seat, and is of a rendered file only: an uplink holder declaring the
// resolver file as a resource beside a node-resolver holder is two writers of one path, refused when the
// machine is resolved — never composed twice — as before the seat named its paths.
func TestAnUplinkDeclaringTheResolverFileAsAResourceIsRefusedBesideTheResolver(t *testing.T) {
shelf := resolverShelf(t)
uplink := shelf["systemd-networkd"]
uplink.Facts = nil
uplink.Resources = append(append([]map[string]any(nil), uplink.Resources...),
map[string]any{"id": "resolvers", "type": "file", "path": ResolverFile, "content": "nameserver 10.42.0.1\n"})
shelf["systemd-networkd"] = uplink
shelf["local-resolver"] = localResolver()
_, err := Resolve(shelf, []string{"dnsmasq", "systemd-networkd", "local-resolver"}, splitDNSLaptop(), World{})
if err == nil || !strings.Contains(err.Error(), `both declare the path "/etc/resolv.conf"`) {
t.Fatalf("an uplink declaring the resolver file as a resource beside the resolver was not refused: %v", err)
}
// Its rendered file steps back, by the seat's definition, and nothing else of it does.
rendered := Manifest{Module: "uplink", Version: "1", Claims: []Claim{{Name: "node-uplink"}},
Facts: map[string]RosterFile{"resolvers": {Path: "/etc//resolv.conf", Template: "x"},
"other": {Path: "/etc/uplink.conf", Template: "x"}}}
got := stepsBack(rendered, []Manifest{rendered, localResolver()})
if len(got.Facts) != 1 || got.Facts["other"].Path == "" {
t.Errorf("the uplink's holder renders %v beside the resolver", got.Facts)
}
}
// A path is compared as the machine writes it: cleaned. Each spelling of /etc/hosts is /etc/hosts.
func TestAnUncleanSpellingOfASeatsPathIsTheSeatsPath(t *testing.T) {
for _, spelled := range []string{"/etc//hosts", "/etc/./hosts", "/etc/../etc/hosts", "/etc/hosts/"} {
stranger := Manifest{Module: "stranger", Version: "1", Resources: []map[string]any{
{"id": "hosts", "type": "file", "path": spelled, "content": "10.0.0.1 x\n"}}}
if got := strings.Join(CatalogueProblems(Shelf{"stranger": stranger}), "; "); !strings.Contains(got,
`stranger declares the path "/etc/hosts", which the seat node-hostname names`) {
t.Errorf("%s from a module not claiming node-hostname was not refused: %q", spelled, got)
}
holder := hostnameHolder(map[string]any{"id": "hosts", "type": "file", "path": spelled, "content": "x"},
map[string]any{"id": "name", "type": "file", "path": "/etc/hostname", "content": "x"})
if err := CanHold(holder, hostnameSeat(t)); err != nil {
t.Errorf("a holder declaring %s was refused: %v", spelled, err)
}
}
}
// A seat names a role's own path, never the root or a top-level directory.
func TestASeatNamesNoRootOrTopLevelDirectory(t *testing.T) {
for _, p := range []string{"/", "/etc"} {
m := Manifest{Module: "a", Version: "1", DefinesSeats: []SeatDeclaration{
{Name: "a-seat", Scope: ScopeNode, Paths: []string{p}}}}
if got := strings.Join(declaredSeatProblems(m), "; "); !strings.Contains(got, "top-level directory") {
t.Errorf("a module's seat naming %s was accepted: %q", p, got)
}
}
}
// The catalogue as it is passes: no holder of a seat naming a path is missing one, and nobody else
// declares one.
func TestTheCataloguePassesTheSeatPathRules(t *testing.T) {
shelf := Shelf{}
for _, m := range theCatalogue(t) {
shelf[m.Module] = m
}
for _, p := range CatalogueProblems(shelf) {
if strings.Contains(p, "ADR 0287") {
t.Errorf("the catalogue breaks ADR 0287: %s", p)
}
}
}
+25 -4
View File
@@ -53,6 +53,11 @@ type Seat struct {
// ${contribution:<seat>:<kind>}. Compiled, never stored: like the protocol, it is the mesh's
// definition of the role, and the store's rows carry no column for it.
Receives []Receivable
// Paths are the paths on its machine the seat's holder owns (novox/hq ADR 0287): a holder declares
// every one, and a module claiming no seat that names a path may not declare it. Only a node seat
// names paths. Compiled, never stored, like Receives: adding or taking away a path is a decision,
// recorded, and the store's rows carry no column for it.
Paths []SeatPath
// Decision is the record that made it a seat.
Decision string
}
@@ -178,6 +183,8 @@ var defaultSeats = append([]Seat{
// holds none of it. Named node-hosts-file until ADR 0223; the former name resolves to it as an
// alias on a mesh that knew it.
{Name: "node-hostname", Scope: ScopeNode, Decision: "novox/hq ADR 0199, ADR 0223",
// The machine's names are the role's files, whatever program fills it (novox/hq ADR 0287).
Paths: []SeatPath{{Path: "/etc/hosts"}, {Path: "/etc/hostname"}},
Serves: []Verb{
{Name: "entries", Description: "Every line of this machine's /etc/hosts, each marked whose it is: " +
"the operator's, or the block of the module or tool that writes it.",
@@ -299,12 +306,17 @@ var defaultSeats = append([]Seat{
// resolver file as it is and who wrote it, and every link with its addresses, routes and the resolvers
// the manager knows for it — the same for every holder, so a question about a machine's names is asked
// the same way whatever manages its network. Optional until both holders serve them.
{Name: "node-uplink", Scope: ScopeNode, Decision: "novox/hq ADR 0117", Serves: uplinkVerbs()},
//
// It owns the resolver file, stepping back for node-resolver where that is held (novox/hq ADR 0287,
// ADR 0247).
{Name: "node-uplink", Scope: ScopeNode, Decision: "novox/hq ADR 0117", Serves: uplinkVerbs(),
Paths: []SeatPath{{Path: ResolverFile, StepsBackFor: ResolverSeat}}},
// The machine's own resolver (novox/hq ADR 0247): held only where something requires `split-dns` — a
// VPN client whose domains must go to its own servers while every other name goes to the mesh's.
// Where it is held, its holder writes the resolver file and the uplink's holder steps back from it
// (node_resolver.go). It knows nothing of any VPN: its verbs route domains to servers over a link.
{Name: ResolverSeat, Scope: ScopeNode, Decision: "novox/hq ADR 0247", Serves: resolverVerbs()},
{Name: ResolverSeat, Scope: ScopeNode, Decision: "novox/hq ADR 0247", Serves: resolverVerbs(),
Paths: []SeatPath{{Path: ResolverFile}}},
// A machine's shares and the shares it mounts (novox/hq ADR 0263): the holder of node-nfs-server
// exports a machine's folders to the private network and provides each as `nfs-share`; the holder of
// node-mounts writes a mount and an automount unit per share on a machine that asks (shares.go).
@@ -364,10 +376,11 @@ func UseSeats(s []Seat) {
row.Accepts, row.Emits, row.Serves = d.Accepts, d.Emits, d.Serves
}
}
// What a seat receives and whether it is replicated are never stored (novox/hq ADR 0212, ADR
// 0223), so they are always the compiled ones.
// What a seat receives, whether it is replicated and the paths it names are never stored
// (novox/hq ADR 0212, ADR 0223, ADR 0287), so they are always the compiled ones.
if d, known := byName[row.Name]; known {
row.Receives = d.Receives
row.Paths = d.Paths
row.Replicated = d.Replicated
row.Serves = optionalAsCompiled(row.Serves, d.Serves)
}
@@ -541,6 +554,14 @@ func CanHold(m Manifest, seat Seat) error {
return fmt.Errorf("%s claims %s, whose holder answers for %q, and %s does not provide %q at %s scope",
m.Module, seat.Name, seat.Delivers, m.Module, seat.Delivers, seat.Scope)
}
// **A holder owns what the role owns** (novox/hq ADR 0287): every path the seat names, declared as
// a resource or as a file the mesh renders for it, so which module owns /etc/hosts on a machine is
// answered by the seat's definition and not by reading every manifest.
if missing := missingPaths(m, seat.Paths); len(missing) > 0 {
return fmt.Errorf("%s claims %s but does not declare %s, which that seat's holder owns (novox/hq "+
"ADR 0287) — a holder declares every path its seat names, as a resource or as a file the mesh "+
"renders for it", m.Module, seat.Name, strings.Join(missing, ", "))
}
// **Serving the seat's tools is a condition of holding it** (novox/hq ADR 0132). A holder that
// does not answer what the role promises is every caller's timeout, found at registration and
// at handover instead, naming the verbs rather than the fact that something is missing.
+17
View File
@@ -67,6 +67,10 @@ type SeatDeclaration struct {
// Records are state buckets of the declaring module that each user reads under its own name — the
// keys `<user>.…` and no other (ADR 0259 §3).
Records []string `json:"records,omitempty"`
// Paths are the paths on its machine the seat's holder owns (novox/hq ADR 0287), under the rules the
// mesh's own seats name theirs by: a node seat only, each absolute and literal, and none another seat
// names. A module's seat never steps back.
Paths []string `json:"paths,omitempty"`
}
// KindedBenches are the seats that may be kinded (novox/hq ADR 0234 §2): `channel` sends to the operator,
@@ -163,6 +167,7 @@ func declaredSeatProblems(m Manifest) []string {
}
}
problems = append(problems, trafficProblems(m, s)...)
problems = append(problems, declaredSeatPathProblems(m, s)...)
}
for _, u := range m.Uses {
@@ -271,6 +276,10 @@ func CatalogueProblems(shelf Shelf) []string {
return ok
}
// Which seats name each path (novox/hq ADR 0287): a module's seat naming one another seat names loses.
named, pathProblems := pathSeats(shelfOrder(shelf), shelf, declaredBy)
problems = append(problems, pathProblems...)
// Who claims each kind of a kinded bench, so a second claim of one kind is refused (ADR 0234 §2).
kindsTaken := map[string]string{}
@@ -327,7 +336,15 @@ func CatalogueProblems(shelf Shelf) []string {
"%s claims %s but does not serve %s, which that seat's protocol promises",
module, c.Name, strings.Join(missing, ", ")))
}
// And owns what the seat's holder owns, as a holder of the mesh's own seats does (ADR 0287).
if missing := missingPaths(m, asSeatPaths(s.Paths)); len(missing) > 0 {
problems = append(problems, fmt.Sprintf(
"%s claims %s but does not declare %s, which that seat's holder owns (novox/hq ADR 0287)",
module, c.Name, strings.Join(missing, ", ")))
}
}
// Only a claimant declares a seat's path (novox/hq ADR 0287).
problems = append(problems, unclaimedSeatPaths(m, named)...)
}
// **A trusted holder runs as its own account** (novox/hq ADR 0259 §8): a module saying warrants, or
// speaking for a kind that proves its sender, is never carried by a machine's runtime.
+32
View File
@@ -337,6 +337,10 @@ func UnusedSettings(m Manifest, layers []Layer) []string {
}
lands := settingKeysUsedBy(m)
// The libraries are the controller's own mesh-wide setting (novox/hq ADR 0328 §1).
if m.Module == LibrariesModule {
lands[LibrariesSetting] = true
}
// A key the module gives a default reaches what asks for it (novox/hq ADR 0262): setting it
// overrides the default, which is how a running module gains a setting without a gap between.
for key := range m.Settings {
@@ -509,6 +513,34 @@ var networkName = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9_.-]*$`)
// refused where it is stored (SetSettings, with UnusedSettings) and said where a plan is read,
// and never costs a module its place.
func JudgeSettings(m Manifest, layers []Layer, adopted bool) error {
if err := UndeclaredAccessPlaces(m, layers); err != nil {
return err
}
return ComposesWith(m, layers, adopted)
}
// ComposesWith is JudgeSettings for a layer stored ahead of the definition that reads it (novox/hq issue 575): an
// access placed that this definition does not declare yet is not refused, because composition passes it by — it
// reaches nothing until the build that declares it registers, and then it places that access. Where a machine is
// declared (LeftOutBecause), and where a setting judged against a newer build of the module is held to still compose
// with the one registered.
func ComposesWith(m Manifest, layers []Layer, adopted bool) error {
// The libraries, on the controller's own layer (novox/hq ADR 0328 §1): each well formed, and only mesh-wide.
if m.Module == LibrariesModule {
for _, l := range layers {
v, has := l.Values[LibrariesSetting]
if !has {
continue
}
if l.From != MeshWideLayer && !l.Default {
return fmt.Errorf("%s is the mesh's, set on the whole mesh and never on one machine (%s §1)",
LibrariesSetting, ADR0328)
}
if _, err := ParseLibraries(v); err != nil {
return err
}
}
}
if err := settingsHoldOneLine(m.Module, layers); err != nil {
return err
}
+4
View File
@@ -79,6 +79,10 @@ func TerminalKeys(m Manifest) []string {
}
delete(keys, PlacesSetting)
delete(keys, AccessesSetting)
// The libraries say where shared data is and who may write it (novox/hq ADR 0328 §1): the operator's alone.
if m.Module == LibrariesModule {
keys[LibrariesSetting] = true
}
rest := make([]string, 0, len(keys))
for k := range keys {
rest = append(rest, k)
@@ -59,6 +59,12 @@ var fillers = []filler{
{namespace: "setting", patterns: []*regexp.Regexp{settingRef}, fields: map[string][]string{"file": {"content"}, "service": {"unit"}, "directory": {"owner"}}},
{namespace: "dir", patterns: []*regexp.Regexp{dirRef}, fields: map[string][]string{"*": {"path", "content", "volumes", "env", "env-file"}}},
{namespace: "access", patterns: []*regexp.Regexp{accessRef}, fields: map[string][]string{"*": {"path", "content", "volumes", "env", "env-file"}}},
// A joining module's identity, wherever its resources take values; a library's host path, only in a container's
// volumes, whose source libraryProblems holds it to (novox/hq ADR 0328 §5).
{namespace: "identity", patterns: []*regexp.Regexp{identityRef}, fields: map[string][]string{"*": {"path", "content", "volumes", "env", "env-file", "run", "command", "user", "name", "unit"}}},
{namespace: "library", patterns: []*regexp.Regexp{libraryRef}, fields: map[string][]string{"container": {"volumes"}},
why: "a library's path is a host path, filled only in a container's volume source; the process reads the " +
"container side its volume gives (novox/hq ADR 0328 §5)"},
{namespace: "secret", patterns: []*regexp.Regexp{placeholder}, fields: map[string][]string{"file": {"content"}},
why: "a secret is never filled into an environment variable or any other field: put it in a file the " +
"module declares and mount that (novox/hq ADR 0086)"},
+5 -2
View File
@@ -33,7 +33,9 @@ func TestTheUplinkSeatPromisesItsVerbsAndRequiresNoneYet(t *testing.T) {
if !slices.Equal(names, []string{"resolvers", "links"}) {
t.Fatalf("node-uplink promises %v", names)
}
today := Manifest{Module: "networkmanager", Version: "1", Claims: []Claim{{Name: "node-uplink", Scope: ScopeNode}}}
// The resolver file is the seat's path (novox/hq ADR 0287), which every holder of today renders.
today := Manifest{Module: "networkmanager", Version: "1", Claims: []Claim{{Name: "node-uplink", Scope: ScopeNode}},
Facts: map[string]RosterFile{"resolvers": {Path: ResolverFile, Template: "x"}}}
if err := CanHold(today, seat); err != nil {
t.Fatalf("a holder that serves no verb yet is refused: %v", err)
}
@@ -55,7 +57,8 @@ func TestTheUplinkVerbsReadBackFromTheRowStayOptional(t *testing.T) {
// The row as seeding stores it: both verbs, the mark not a column.
UseSeats([]Seat{{Name: "node-uplink", Scope: ScopeNode, Serves: []Verb{{Name: "resolvers"}, {Name: "links"}}}})
seat := uplinkSeat(t)
holder := Manifest{Module: "systemd-networkd", Version: "1", Claims: []Claim{{Name: "node-uplink", Scope: ScopeNode}}}
holder := Manifest{Module: "systemd-networkd", Version: "1", Claims: []Claim{{Name: "node-uplink", Scope: ScopeNode}},
Facts: map[string]RosterFile{"resolvers": {Path: ResolverFile, Template: "x"}}}
if err := CanHold(holder, seat); err != nil {
t.Fatalf("read back from the row, the verbs are required: %v", err)
}
+8 -2
View File
@@ -116,9 +116,15 @@ var ControllerVerbs = []Verb{
"limit": "how many builds to list (default 20); not with log",
}, nil)},
{Name: "plans", Description: "What the last merges produced and where each stands (novox/hq ADR 0162): " +
"the tiers, the tier a plan is at, what it waits for and since when; one plan whole, given its id.",
"the tiers, the tier a plan is at, what it waits for and since when; one plan whole, given its id; " +
"or, given wait, one plan followed until it moves.",
Input: schema(map[string]string{
"id": "a plan's id (as `plans` lists them): that plan, tier by tier",
"id": "a plan's id (as `plans` lists them): that plan, tier by tier",
"wait": "a plan's id: wait until it moves, then answer it as data — the plan, and what ended the wait " +
"(moved, tier, done, or bound); ask again to follow it further (novox/hq issue 229)",
"until": "with wait: moved (any change, the default), tier (it reaches another tier) or done (it ends)",
"bound": "with wait: how long to wait at most, as 15s or seconds — 20s, the default, is the longest, " +
"inside what a call is answered within",
"stop": "a plan's id: stop it — what was asked still builds, nothing further is asked",
"close": "a plan's id: close a plan that will not move again, as failed by hand (novox/hq issue 254)",
"retry": "a failed plan's id: ask its failed builds again under new ids, and carry the plan on from that tier (novox/hq ADR 0219)",
+74 -1
View File
@@ -117,6 +117,10 @@ func (i *Inventory) RegisterModule(ctx context.Context, m catalogue.Manifest, fr
}
}
}
// A module never takes a defined library's name: mesh-<module> and mesh-<library> never collide (ADR 0328 §4).
if err := i.LibraryNamed(ctx, m.Module); err != nil {
return fmt.Errorf("%s is not registered: %w", m.Module, err)
}
raw, err := json.Marshal(m)
if err != nil {
return err
@@ -929,6 +933,10 @@ func (i *Inventory) setSettings(ctx context.Context, nodeName, module string, va
// the mesh-wide layer under it when the layer is one node's, and refuses the first thing that
// cannot work (ADR 0163, rule 6). The same judgement composition makes; what passes here composes.
func (i *Inventory) judgeSettings(ctx context.Context, nodeName, module string, values map[string]any) error {
// The libraries the controller's own layer defines (novox/hq ADR 0328 §1, §4, §13).
if err := i.judgeLibraries(ctx, nodeName, module, values); err != nil {
return err
}
m, err := i.declared(ctx, module)
if err != nil {
return fmt.Errorf("%w: %s", ErrNoSuchModule, module)
@@ -958,7 +966,38 @@ func (i *Inventory) judgeSettings(ctx context.Context, nodeName, module string,
}
layers = append(layers, catalogue.Layer{From: from, Values: values})
if err := catalogue.JudgeSettings(m, layers, adopted); err != nil {
return fmt.Errorf("refused: %s on %s cannot compose with the layer %q — %w", module, where, from, err)
// **A placement for an access the module's newer build brings** (novox/hq issue 575): judged against the
// manifest of the module's newest build on the branch it follows, where the registered one does not declare
// the access yet — so an access can be placed before or with the delivery that adds it. It must still compose
// with the registered manifest, which passes the placement by until the build registers.
var undeclared *catalogue.UndeclaredAccessError
if !errors.As(err, &undeclared) {
return fmt.Errorf("refused: %s on %s cannot compose with the layer %q — %w", module, where, from, err)
}
ahead, has, aerr := i.NewestManifestOnBranch(ctx, module)
if aerr != nil {
return aerr
}
if !has {
follows, ferr := i.followsABranch(ctx, module)
if ferr != nil {
return ferr
}
if !follows {
return fmt.Errorf("refused: %s on %s cannot compose with the layer %q — %w; and %s follows no branch "+
"(it was registered by hand), so no build of it can be judged ahead of what is registered", module,
where, from, err, module)
}
}
if !has || catalogue.JudgeSettings(ahead, layers, adopted) != nil || catalogue.ComposesWith(m, layers, adopted) != nil {
return fmt.Errorf("refused: %s on %s cannot compose with the layer %q — %w; nor does the newest build of "+
"%s on the branch it follows declare it", module, where, from, err, module)
}
m = ahead
}
// A module that joins carries no user or group number or owner in a setting (novox/hq ADR 0328 §5).
if err := catalogue.JoiningSettingProblem(m, layers[len(layers)-1:]); err != nil {
return fmt.Errorf("refused: %w", err)
}
// And a key that reaches nothing, refused here where somebody can still fix the spelling:
// stored, it would be a setting somebody believes they made.
@@ -969,6 +1008,40 @@ func (i *Inventory) judgeSettings(ctx context.Context, nodeName, module string,
return nil
}
// followsABranch says a module's record names the branch it follows.
func (i *Inventory) followsABranch(ctx context.Context, module string) (bool, error) {
var ref string
err := i.store.Pool().QueryRow(ctx, `select coalesce(ref, '') from module where name = $1`, module).Scan(&ref)
if errors.Is(err, pgx.ErrNoRows) {
return false, nil
}
return ref != "", err
}
// NewestManifestOnBranch is the manifest of a module's newest successful build on the branch its record follows
// (novox/hq issue 575): what the delivery under way brings, before it registers — or the registered one's own, when
// nothing newer was built. Only a build of that very branch: never one of another branch or of none. False when no
// such build is kept, and for a module whose record follows no branch (registered by hand).
func (i *Inventory) NewestManifestOnBranch(ctx context.Context, module string) (catalogue.Manifest, bool, error) {
var raw []byte
err := i.store.Pool().QueryRow(ctx,
`select b.manifest from build b join module m on m.name = b.module
where b.module = $1 and b.failed = '' and b.manifest is not null and b.manifest::text <> 'null'
and coalesce(m.ref, '') <> '' and b.ref = m.ref
order by b.at desc limit 1`, module).Scan(&raw)
if errors.Is(err, pgx.ErrNoRows) {
return catalogue.Manifest{}, false, nil
}
if err != nil {
return catalogue.Manifest{}, false, err
}
m, err := catalogue.ParseManifest(raw)
if err != nil {
return catalogue.Manifest{}, false, nil // a build the mesh cannot read judges nothing
}
return m, true, nil
}
// givenIn is the machine ports a node-level settings layer gives a module, software port →
// machine port (novox/hq ADR 0100). Nothing when the layer gives none; what is not a port is left
// for composition to refuse in its own words.
+12 -3
View File
@@ -38,13 +38,22 @@ type ResourceHealth struct {
Waits []Wait `json:"waits,omitempty"`
}
// Wait is one thing a waiting resource waits for the operator to give: exactly one of Secret and Setting
// (novox/hq ADR 0283), as link.Wait carries it.
// Wait is one thing a waiting resource waits for the operator to give: exactly one of Secret, Setting and Access
// (novox/hq ADR 0283, ADR 0325 §7), as link.Wait carries it.
type Wait struct {
Part string `json:"part"`
Secret string `json:"secret,omitempty"`
Setting string `json:"setting,omitempty"`
What string `json:"what"`
// Access is an access of the module by its own id for it, from the engine's access-waits, and Disk that it
// waits for the disk that holds its folder rather than for the operator's word (novox/hq ADR 0325 §7, §9).
Access string `json:"access,omitempty"`
Disk bool `json:"disk,omitempty"`
// Placed is the filesystem the folder is on now, without its path (ADR 0325 §8): a warrant for another does not
// answer the wait.
Placed string `json:"placed,omitempty"`
// Reason is the engine's reason for an access wait: "keep" or "disk".
Reason string `json:"reason,omitempty"`
What string `json:"what"`
}
// NodeHealth is a machine's newest statement, as kept.
+331
View File
@@ -0,0 +1,331 @@
package inventory
import (
"context"
"encoding/json"
"errors"
"fmt"
"path/filepath"
"strings"
"time"
"github.com/jackc/pgx/v5"
"github.com/novox/mesh-controller/internal/catalogue"
)
// Joining modules' numbers, library groups, taken libraries and the libraries themselves (novox/hq ADR 0328 §1, §3,
// §4, §7, §15; migration 0096).
// AllocateIdentity is a joining module's number, allocated now when it has none: one number from the mesh's range,
// mesh-wide, kept for ever, its user's and its own group's. Two controllers allocating at once agree: the second's
// insert finds the first's row.
func (i *Inventory) AllocateIdentity(ctx context.Context, module string) (catalogue.ModuleIdentity, error) {
if strings.TrimSpace(module) == "" {
return catalogue.ModuleIdentity{}, errors.New("a number is allocated for a module, and none was named")
}
if id, had, err := i.IdentityOf(ctx, module); err != nil || had {
return id, err
}
// Drawn only when the row is new; a draw lost to a concurrent insert is a gap in the range, never a number given
// twice.
if _, err := i.store.Pool().Exec(ctx,
`insert into module_identity (module, uid) select $1, nextval('mesh_number')::integer
on conflict (module) do nothing`, module); err != nil {
return catalogue.ModuleIdentity{}, fmt.Errorf("%s could not be given its number: %w", module, err)
}
id, had, err := i.IdentityOf(ctx, module)
if err == nil && !had {
err = fmt.Errorf("%s was given its number and it cannot be read back", module)
}
return id, err
}
// IdentityOf is a module's number, and false when it was never allocated one.
func (i *Inventory) IdentityOf(ctx context.Context, module string) (catalogue.ModuleIdentity, bool, error) {
id := catalogue.ModuleIdentity{Module: module}
err := i.store.Pool().QueryRow(ctx, `select uid from module_identity where module = $1`, module).Scan(&id.UID)
if errors.Is(err, pgx.ErrNoRows) {
return catalogue.ModuleIdentity{}, false, nil
}
if err != nil {
return catalogue.ModuleIdentity{}, false, err
}
return id, true, nil
}
// Identities is every joining module's number, by module.
func (i *Inventory) Identities(ctx context.Context) (map[string]catalogue.ModuleIdentity, error) {
rows, err := i.store.Pool().Query(ctx, `select module, uid from module_identity`)
if err != nil {
return nil, err
}
defer rows.Close()
out := map[string]catalogue.ModuleIdentity{}
for rows.Next() {
var id catalogue.ModuleIdentity
if err := rows.Scan(&id.Module, &id.UID); err != nil {
return nil, err
}
out[id.Module] = id
}
return out, rows.Err()
}
// AllocateLibraryGroup is a library's group number, allocated now when it has none, from the same sequence as the
// modules' numbers (§3).
func (i *Inventory) AllocateLibraryGroup(ctx context.Context, library string) (int, error) {
if strings.TrimSpace(library) == "" {
return 0, errors.New("a group number is allocated for a library, and none was named")
}
groups, err := i.LibraryGroups(ctx)
if err != nil {
return 0, err
}
if gid, had := groups[library]; had {
return gid, nil
}
if _, err := i.store.Pool().Exec(ctx,
`insert into library_group (library, gid) select $1, nextval('mesh_number')::integer
on conflict (library) do nothing`, library); err != nil {
return 0, fmt.Errorf("the library %s could not be given its group's number: %w", library, err)
}
if groups, err = i.LibraryGroups(ctx); err != nil {
return 0, err
}
gid, has := groups[library]
if !has {
return 0, fmt.Errorf("the library %s was given its group's number and it cannot be read back", library)
}
return gid, nil
}
// LibraryGroups is every library's group number, by library.
func (i *Inventory) LibraryGroups(ctx context.Context) (map[string]int, error) {
rows, err := i.store.Pool().Query(ctx, `select library, gid from library_group`)
if err != nil {
return nil, err
}
defer rows.Close()
out := map[string]int{}
for rows.Next() {
var name string
var gid int
if err := rows.Scan(&name, &gid); err != nil {
return nil, err
}
out[name] = gid
}
return out, rows.Err()
}
// Libraries is every library the operator defined (§1): the mesh-wide `libraries` layer of the controller's module.
// None when it was never set.
func (i *Inventory) Libraries(ctx context.Context) (map[string]catalogue.Library, error) {
var raw []byte
err := i.store.Pool().QueryRow(ctx,
`select values from settings where module = $1 and node is null`, catalogue.LibrariesModule).Scan(&raw)
if errors.Is(err, pgx.ErrNoRows) {
return nil, nil
}
if err != nil {
return nil, err
}
var layer map[string]any
if err := json.Unmarshal(raw, &layer); err != nil {
return nil, err
}
return catalogue.ParseLibraries(layer[catalogue.LibrariesSetting])
}
// judgeLibraries is what a layer of the controller's module is held to when it defines libraries (§1, §4, §13): only
// mesh-wide, well formed, no library named as a registered module, none overlapping another on a machine.
func (i *Inventory) judgeLibraries(ctx context.Context, nodeName, module string, values map[string]any) error {
v, defines := values[catalogue.LibrariesSetting]
if module != catalogue.LibrariesModule || !defines {
return nil
}
if nodeName != "" {
return fmt.Errorf("refused: %s is the mesh's, set on the whole mesh and never on one machine (%s §1)",
catalogue.LibrariesSetting, catalogue.ADR0328)
}
libs, err := catalogue.ParseLibraries(v)
if err != nil {
return fmt.Errorf("refused: %w", err)
}
shelf, err := i.Catalogue(ctx)
if err != nil {
return err
}
modules := make([]string, 0, len(shelf))
for name := range shelf {
modules = append(modules, name)
}
if problems := catalogue.LibrariesProblems(libs, modules); len(problems) > 0 {
return fmt.Errorf("refused: %s", strings.Join(problems, "; "))
}
return nil
}
// LibraryNamed refuses a module whose name is a defined library's (§4): mesh-<module> and mesh-<library> never collide.
func (i *Inventory) LibraryNamed(ctx context.Context, module string) error {
libs, err := i.Libraries(ctx)
if err != nil {
return err
}
if _, clash := libs[module]; clash {
return fmt.Errorf("the module %s has the name of the library %s: %s would be both (%s §4)", module, module,
catalogue.MeshName(module), catalogue.ADR0328)
}
return nil
}
// LibraryPaths is every sub-path a module on any machine asks of each library (§6): what its home keeps. A module on a
// machine is one assigned there or one a requirement pulled in — what the machine was last sent.
func (i *Inventory) LibraryPaths(ctx context.Context) (map[string][]string, error) {
shelf, err := i.Catalogue(ctx)
if err != nil {
return nil, err
}
// Every module assigned anywhere, and every module a machine was last sent — those a requirement pulled in too.
rows, err := i.store.Pool().Query(ctx,
`select module from assignment
union select jsonb_object_keys(sent_builds) from node where sent_builds is not null and jsonb_typeof(sent_builds) = 'object'`)
if err != nil {
return nil, err
}
defer rows.Close()
seen := map[string]map[string]bool{}
for rows.Next() {
var module string
if err := rows.Scan(&module); err != nil {
return nil, err
}
for _, u := range shelf[module].Libraries {
if seen[u.Library] == nil {
seen[u.Library] = map[string]bool{}
}
seen[u.Library][filepath.Clean(u.Path)] = true
}
}
if err := rows.Err(); err != nil {
return nil, err
}
out := map[string][]string{}
for lib, paths := range seen {
for p := range paths {
out[lib] = append(out[lib], p)
}
}
return out, nil
}
// TakenLibrary is one library the operator let the mesh take, on its home (§7).
type TakenLibrary struct {
Library string
Home string
Root string
Placed string
By string
Ask string
Warrant string
Said string
// Done is whether the engine stated the take's act done: only then is the library taken.
Done bool
}
// TakenLibraries is every library whose take was started, by name; Done says it is taken.
func (i *Inventory) TakenLibraries(ctx context.Context) (map[string]TakenLibrary, error) {
rows, err := i.store.Pool().Query(ctx,
`select library, home, root, placed, taken_by, ask, warrant, said, done from kept_library`)
if err != nil {
return nil, err
}
defer rows.Close()
out := map[string]TakenLibrary{}
for rows.Next() {
var t TakenLibrary
if err := rows.Scan(&t.Library, &t.Home, &t.Root, &t.Placed, &t.By, &t.Ask, &t.Warrant, &t.Said, &t.Done); err != nil {
return nil, err
}
out[t.Library] = t
}
return out, rows.Err()
}
// RecordTakeStarted keeps that a take of a library was started on its home, under the operator's warrant: the
// engine said it began. The library is taken only once the engine states the act done (TakeDone). Once per home, root
// and filesystem: a second start of the same keeps the first, and one on another (it moved, and the operator was asked
// again) replaces it, not done.
func (i *Inventory) RecordTakeStarted(ctx context.Context, t TakenLibrary) error {
switch {
case t.Library == "" || t.Home == "" || !strings.HasPrefix(t.Root, "/"):
return errors.New("a taken library names itself, its home and its root")
case strings.TrimSpace(t.Warrant) == "" || strings.TrimSpace(t.Ask) == "":
return errors.New("a library is taken only on the operator's warrant, and this names none")
}
_, err := i.store.Pool().Exec(ctx,
`insert into kept_library (library, home, root, placed, taken_by, ask, warrant, said, done)
values ($1, $2, $3, $4, $5, $6, $7, $8, false)
on conflict (library) do update
set home = excluded.home, root = excluded.root, placed = excluded.placed, taken_at = now(),
taken_by = excluded.taken_by, ask = excluded.ask, warrant = excluded.warrant, said = excluded.said,
done = false
where kept_library.home <> excluded.home or kept_library.root <> excluded.root
or kept_library.placed <> excluded.placed`,
t.Library, t.Home, t.Root, t.Placed, t.By, t.Ask, t.Warrant, t.Said)
return err
}
// TakeDone keeps that the engine stated a library's take done under a warrant: the library is taken. A warrant the
// recorded take was not started under changes nothing.
func (i *Inventory) TakeDone(ctx context.Context, library, warrant string) error {
_, err := i.store.Pool().Exec(ctx,
`update kept_library set done = true where library = $1 and warrant = $2 and not done`, library, warrant)
return err
}
// TakeFailed forgets a take the engine stated failed under a warrant, never one done: the library is asked about again.
func (i *Inventory) TakeFailed(ctx context.Context, library, warrant string) error {
_, err := i.store.Pool().Exec(ctx,
`delete from kept_library where library = $1 and warrant = $2 and not done`, library, warrant)
return err
}
// SettingsAsOf is a module's settings layers on a machine as they stood at a moment: each layer — the whole mesh's,
// then the machine's — as it was set then, read from the layer standing now or from its history. What a module held at
// the build it ran is composed with: the settings it last ran with (novox/hq ADR 0328 §8).
func (i *Inventory) SettingsAsOf(ctx context.Context, nodeName, module string, at time.Time) ([]catalogue.Layer, error) {
node, err := i.NodeByName(ctx, nodeName)
if err != nil {
return nil, err
}
var layers []catalogue.Layer
for _, scope := range []struct {
id *string
from string
}{{nil, catalogue.MeshWideLayer}, {&node.ID, nodeName}} {
var raw []byte
err := i.store.Pool().QueryRow(ctx,
`select values from settings
where module = $1 and node is not distinct from $2::uuid and set_at <= $3
union all
select values from (select values from settings_history
where module = $1 and node is not distinct from $2::uuid
and (set_at is null or set_at <= $3) and replaced_at > $3
order by replaced_at asc limit 1) h
limit 1`, module, scope.id, at).Scan(&raw)
if errors.Is(err, pgx.ErrNoRows) {
continue
}
if err != nil {
return nil, err
}
values := map[string]any{}
if err := json.Unmarshal(raw, &values); err != nil {
return nil, err
}
layers = append(layers, catalogue.Layer{From: scope.from, Values: values})
}
return layers, nil
}
+393
View File
@@ -0,0 +1,393 @@
package inventory
import (
"encoding/json"
"fmt"
"sort"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
)
// novox/hq ADR 0328 §3, §4, §15: one number per joining module (its user's and its own group's) and one per library's
// group, from one sequence and one range — never 0 or a root group's, never shared between a module and a library,
// never given twice, the same however often asked.
func TestOneNumberPerModuleAndLibraryNeverRoot(t *testing.T) {
inv := fresh(t)
ctx := t.Context()
radarr, err := inv.AllocateIdentity(ctx, "radarr")
if err != nil {
t.Fatal(err)
}
again, err := inv.AllocateIdentity(ctx, "radarr")
if err != nil || again != radarr {
t.Fatalf("radarr was given %+v, then %+v (%v): one number per module", radarr, again, err)
}
media, err := inv.AllocateLibraryGroup(ctx, "media")
if err != nil {
t.Fatal(err)
}
if mediaAgain, err := inv.AllocateLibraryGroup(ctx, "media"); err != nil || mediaAgain != media {
t.Fatalf("the media group was given %d, then %d (%v)", media, mediaAgain, err)
}
plex, err := inv.AllocateIdentity(ctx, "plex")
if err != nil {
t.Fatal(err)
}
seen := map[int]string{}
for who, n := range map[string]int{"radarr": radarr.UID, "plex": plex.UID, "mesh-media": media} {
if n == 0 || !catalogue.AllocatableID(n) {
t.Errorf("%s was given %d: within %d–%d, never 0", who, n, catalogue.IdentityLow, catalogue.IdentityHigh)
}
if other, twice := seen[n]; twice {
t.Errorf("%s and %s share %d: a module's own group and a library's never do", who, other, n)
}
seen[n] = who
}
// Read back as every machine's composition reads them: the same numbers.
ids, err := inv.Identities(ctx)
if err != nil {
t.Fatal(err)
}
groups, err := inv.LibraryGroups(ctx)
if err != nil {
t.Fatal(err)
}
if ids["radarr"].UID != radarr.UID || groups["media"] != media {
t.Fatalf("read back %+v and %v", ids, groups)
}
// A forgotten row frees no number: the next one is new.
if _, err := inv.store.Pool().Exec(ctx, `delete from module_identity where module = 'plex'`); err != nil {
t.Fatal(err)
}
sonarr, err := inv.AllocateIdentity(ctx, "sonarr")
if err != nil {
t.Fatal(err)
}
if _, given := seen[sonarr.UID]; given {
t.Fatalf("sonarr was given %d, a number given before", sonarr.UID)
}
// The store refuses a number outside the range, root's above all.
for _, n := range []int{0, 4, 10, 1000, 49999, 60000} {
if _, err := inv.store.Pool().Exec(ctx, `insert into module_identity (module, uid) values ('byhand', $1)`, n); err == nil {
t.Errorf("the store took %d as a module's number", n)
}
if _, err := inv.store.Pool().Exec(ctx, `insert into library_group (library, gid) values ('byhand', $1)`, n); err == nil {
t.Errorf("the store took %d as a library group's number", n)
}
}
}
// libraryController registers the controller's own module, whose mesh-wide layer defines the libraries.
func libraryController(t *testing.T, inv *Inventory) {
t.Helper()
m := catalogue.Manifest{Module: catalogue.LibrariesModule, Version: "1",
Resources: []map[string]any{{"id": "server", "type": "container", "name": "controller", "image": "x"}}}
if err := inv.RegisterModule(t.Context(), m, Source{}); err != nil {
t.Fatal(err)
}
}
// novox/hq ADR 0328 §1, §4, §13: a library is defined only on the controller's mesh-wide layer, as a trusted setting;
// never on one machine; never overlapping another on a machine it reaches; never named as a registered module — and no
// module is registered under a defined library's name.
func TestALibraryIsDefinedOnceForTheMeshAndNeverOverlaps(t *testing.T) {
inv := fresh(t)
ctx := t.Context()
for _, n := range []string{"ace", "novox"} {
if _, err := inv.AddNode(ctx, n); err != nil {
t.Fatal(err)
}
}
libraryController(t, inv)
radarr := catalogue.Manifest{Module: "radarr", Version: "1",
Resources: []map[string]any{{"id": "server", "type": "container", "name": "radarr", "image": "x"}}}
if err := inv.RegisterModule(ctx, radarr, Source{}); err != nil {
t.Fatal(err)
}
media := map[string]any{"home": "ace", "root": "/storage/media", "reach": []any{"novox"}}
define := func(libs map[string]any) map[string]any {
return map[string]any{catalogue.LibrariesSetting: libs}
}
if err := inv.SetSettings(ctx, "", catalogue.LibrariesModule, define(map[string]any{"media": media})); err != nil {
t.Fatalf("a library was not defined: %v", err)
}
libs, err := inv.Libraries(ctx)
if err != nil {
t.Fatal(err)
}
if l := libs["media"]; l.Home != "ace" || l.Root != "/storage/media" || !l.Reaches("novox") || !l.Reaches("ace") {
t.Fatalf("the library reads %+v", libs["media"])
}
for name, c := range map[string]struct {
node string
libs map[string]any
want string
}{
"on one machine": {"ace", map[string]any{"media": media}, "never on one machine"},
"inside another": {"", map[string]any{"media": media,
"movies": map[string]any{"home": "novox", "root": "/storage/media/movies"}}, "overlap on"},
"around another": {"", map[string]any{"movies": map[string]any{"home": "ace", "root": "/storage/media/movies"},
"media": media}, "overlap on"},
"named as a module": {"", map[string]any{"radarr": map[string]any{"home": "ace", "root": "/srv/radarr-data"}},
"name of the module radarr"},
"with an owner": {"", map[string]any{"books": map[string]any{"home": "ace", "root": "/srv/books",
"owner": "1000"}}, "nothing about an owner"},
} {
err := inv.SetSettings(ctx, c.node, catalogue.LibrariesModule, define(c.libs))
if err == nil || !strings.Contains(err.Error(), c.want) {
t.Errorf("%s: %v; want refused saying %q", name, err, c.want)
}
}
// Two libraries apart on different machines may share a path's prefix only where they do not meet.
apart := map[string]any{"media": media, "photos": map[string]any{"home": "elsewhere", "root": "/storage/media/photos"}}
if err := inv.JudgeSettings(ctx, "", catalogue.LibrariesModule, define(apart)); err != nil {
t.Errorf("libraries that meet on no machine were refused: %v", err)
}
// A module under a defined library's name is refused.
named := catalogue.Manifest{Module: "media", Version: "1",
Resources: []map[string]any{{"id": "s", "type": "container", "name": "media", "image": "x"}}}
if err := inv.RegisterModule(ctx, named, Source{}); err == nil || !strings.Contains(err.Error(), "name of the library media") {
t.Fatalf("a module named as a library: %v", err)
}
}
// novox/hq ADR 0328 §7: a library taken is recorded once per home and filesystem, under a warrant; one taken again on
// another filesystem replaces it.
func TestALibraryIsTakenOnceUnderAWarrant(t *testing.T) {
inv := fresh(t)
ctx := t.Context()
if err := inv.RecordTakeStarted(ctx, TakenLibrary{Library: "media", Home: "ace", By: "the operator"}); err == nil {
t.Fatal("a library was recorded taken with no warrant")
}
first := TakenLibrary{Library: "media", Home: "ace", Root: "/storage/media", Placed: "fs-000000000001",
By: "the operator", Ask: "c01", Warrant: "c01"}
if err := inv.RecordTakeStarted(ctx, first); err != nil {
t.Fatal(err)
}
again := first
again.Ask, again.Warrant, again.By = "c02", "c02", "someone else"
if err := inv.RecordTakeStarted(ctx, again); err != nil {
t.Fatal(err)
}
taken, err := inv.TakenLibraries(ctx)
if err != nil {
t.Fatal(err)
}
if got := taken["media"]; got.Ask != "c01" {
t.Fatalf("a second take of the same overwrote the first: %+v", got)
}
moved := first
moved.Placed, moved.Ask, moved.Warrant = "fs-0000000000ff", "c03", "c03"
if err := inv.RecordTakeStarted(ctx, moved); err != nil {
t.Fatal(err)
}
if taken, err = inv.TakenLibraries(ctx); err != nil || taken["media"].Ask != "c03" {
t.Fatalf("a take on another filesystem: %+v (%v)", taken["media"], err)
}
// The review of 7ea6d36e, item 3: the root is part of what was taken — a library whose root changed is taken
// again, and the new take replaces the old.
if got := taken["media"].Root; got != "/storage/media" {
t.Fatalf("the take recorded the root %q", got)
}
rooted := moved
rooted.Root, rooted.Ask, rooted.Warrant = "/storage/media2", "c04", "c04"
if err := inv.RecordTakeStarted(ctx, rooted); err != nil {
t.Fatal(err)
}
if taken, err = inv.TakenLibraries(ctx); err != nil || taken["media"].Ask != "c04" || taken["media"].Root != "/storage/media2" {
t.Fatalf("a take at another root: %+v (%v)", taken["media"], err)
}
}
// novox/hq issue 575: a module registered without the access movies, whose newest build on its branch declares it —
// the build the delivery under way brings — takes a proposal placing movies: judged against that build, not refused.
// Without such a build it is refused, as before.
func TestAPlacementIsJudgedAgainstTheBuildThatBringsTheAccess(t *testing.T) {
inv := fresh(t)
ctx := t.Context()
if _, err := inv.AddNode(ctx, "ace"); err != nil {
t.Fatal(err)
}
registered := catalogue.Manifest{Module: "media-importer", Version: "1",
Accesses: []catalogue.Access{{ID: "downloads", Path: "/services/media/downloads", Mode: catalogue.AccessReadWrite}},
Resources: []map[string]any{{"id": "server", "type": "container", "name": "importer", "image": "x",
"volumes": []any{"${access:downloads}:/downloads"}}}}
if err := inv.RegisterModule(ctx, registered, Source{Repository: "https://forge.invalid/media.git",
Ref: "main", BuiltFrom: "c0ffee33"}); err != nil {
t.Fatal(err)
}
placing := map[string]any{catalogue.AccessesSetting: map[string]any{"movies": "/storage/media/movies"}}
err := inv.JudgeSettings(ctx, "ace", "media-importer", placing)
if err == nil || !strings.Contains(err.Error(), `places the access "movies", which it does not declare`) {
t.Fatalf("a placement no build declares: %v", err)
}
// main's manifest — the build of #34 — declares movies.
ahead := registered
ahead.Accesses = append([]catalogue.Access{}, registered.Accesses...)
ahead.Accesses = append(ahead.Accesses, catalogue.Access{ID: "movies", Path: "/services/media/movies",
Mode: catalogue.AccessReadWrite})
raw, err := json.Marshal(ahead)
if err != nil {
t.Fatal(err)
}
b := aBuild("b34", "media-importer", "")
b.Ref, b.Manifest = "main", raw
if err := inv.RecordBuild(ctx, b); err != nil {
t.Fatal(err)
}
if err := inv.JudgeSettings(ctx, "ace", "media-importer", placing); err != nil {
t.Fatalf("the proposal placing movies, which main's build declares, was refused: %v", err)
}
if err := inv.SetSettings(ctx, "ace", "media-importer", placing); err != nil {
t.Fatalf("the layer placing movies was not stored: %v", err)
}
// Stored ahead, it does not cost the registered module its place: composition passes it by.
layers, err := inv.SettingsFor(ctx, "ace", "media-importer")
if err != nil {
t.Fatal(err)
}
if err := catalogue.ComposesWith(registered, layers, false); err != nil {
t.Fatalf("the registered module no longer composes with the layer stored ahead: %v", err)
}
// A typo is still a typo: an access neither build declares is refused.
typo := map[string]any{catalogue.AccessesSetting: map[string]any{"moveis": "/storage/media/movies"}}
if err := inv.JudgeSettings(ctx, "ace", "media-importer", typo); err == nil {
t.Fatal("a placement no build declares was taken")
}
}
// Review of e1c2b744: only a build of the branch the module's record follows judges a placement ahead of it — never one
// from another branch, nor one with no branch, and a module registered with no branch is refused as before, saying so.
func TestOnlyABuildOfTheFollowedBranchJudgesAPlacementAhead(t *testing.T) {
inv := fresh(t)
ctx := t.Context()
if _, err := inv.AddNode(ctx, "ace"); err != nil {
t.Fatal(err)
}
registered := catalogue.Manifest{Module: "importer", Version: "1",
Resources: []map[string]any{{"id": "server", "type": "container", "name": "importer", "image": "x"}}}
ahead := registered
ahead.Accesses = []catalogue.Access{{ID: "movies", Path: "/services/media/movies", Mode: catalogue.AccessReadWrite}}
raw, err := json.Marshal(ahead)
if err != nil {
t.Fatal(err)
}
placing := map[string]any{catalogue.AccessesSetting: map[string]any{"movies": "/storage/media/movies"}}
if err := inv.RegisterModule(ctx, registered, Source{Repository: "https://forge.invalid/media.git", Ref: "main",
BuiltFrom: "c0ffee"}); err != nil {
t.Fatal(err)
}
for i, ref := range []string{"feature", ""} {
b := aBuild(fmt.Sprintf("elsewhere%d", i), "importer", "")
b.Ref, b.Manifest = ref, raw
if err := inv.RecordBuild(ctx, b); err != nil {
t.Fatal(err)
}
}
if err := inv.JudgeSettings(ctx, "ace", "importer", placing); err == nil {
t.Fatal("a build of another branch, or of none, judged the placement")
}
// Registered by hand, with no branch: refused, saying why.
byHand := registered
byHand.Module = "byhand"
if err := inv.RegisterModule(ctx, byHand, Source{}); err != nil {
t.Fatal(err)
}
aheadByHand := ahead
aheadByHand.Module = "byhand"
if raw, err = json.Marshal(aheadByHand); err != nil {
t.Fatal(err)
}
b := aBuild("onmain", "byhand", "")
b.Ref, b.Manifest = "main", raw
if err := inv.RecordBuild(ctx, b); err != nil {
t.Fatal(err)
}
if err := inv.JudgeSettings(ctx, "ace", "byhand", placing); err == nil ||
!strings.Contains(err.Error(), "follows no branch") {
t.Fatalf("a module with no branch recorded: %v", err)
}
}
// The review of 7ea6d36e, item 4: every sub-path a module asks of a library is kept on its home, whether the module
// was assigned or pulled in by a requirement (it is in what its machine was last sent).
func TestTheSubPathsOfModulesPulledInByARequirementAreKept(t *testing.T) {
inv := fresh(t)
ctx := t.Context()
record, err := inv.AddNode(ctx, "ace")
if err != nil {
t.Fatal(err)
}
for _, m := range []catalogue.Manifest{
{Module: "radarr", Version: "1", Libraries: []catalogue.LibraryUse{{ID: "m", Library: "media", Path: "movies", Use: "read-write"}}},
{Module: "importer", Version: "1", Libraries: []catalogue.LibraryUse{{ID: "i", Library: "media", Path: "incoming", Use: "read-write"}}},
} {
if err := inv.RegisterModule(ctx, m, Source{}); err != nil {
t.Fatal(err)
}
}
if _, err := inv.Assign(ctx, "ace", "radarr"); err != nil {
t.Fatal(err)
}
if err := inv.RecordSentWith(ctx, record.ID, "d1", map[string]string{"radarr": "c1", "importer": "c2"}, 1, 1,
Send{Sequence: 1}); err != nil {
t.Fatal(err)
}
paths, err := inv.LibraryPaths(ctx)
if err != nil {
t.Fatal(err)
}
got := append([]string{}, paths["media"]...)
sort.Strings(got)
if strings.Join(got, ",") != "incoming,movies" {
t.Fatalf("the paths kept of media are %v; want the importer's, pulled in by a requirement, too", got)
}
}
// The review of 7ea6d36e, item 5: a take the engine says it started is recorded as started, not taken; it is taken when
// the engine states the act done under that warrant, and forgotten when it states it failed, so the operator is asked
// again.
func TestALibraryIsTakenWhenTheActIsDone(t *testing.T) {
inv := fresh(t)
ctx := t.Context()
started := TakenLibrary{Library: "media", Home: "ace", Root: "/storage/media", Placed: "fs-000000000001",
By: "the operator", Ask: "c01", Warrant: "c01"}
if err := inv.RecordTakeStarted(ctx, started); err != nil {
t.Fatal(err)
}
taken, err := inv.TakenLibraries(ctx)
if err != nil {
t.Fatal(err)
}
if got := taken["media"]; got.Done || got.Ask != "c01" {
t.Fatalf("a take only started reads %+v; want started, not done", got)
}
if err := inv.TakeDone(ctx, "media", "c99"); err != nil {
t.Fatal(err)
}
if taken, _ = inv.TakenLibraries(ctx); taken["media"].Done {
t.Fatal("another warrant's act made the take done")
}
if err := inv.TakeDone(ctx, "media", "c01"); err != nil {
t.Fatal(err)
}
if taken, _ = inv.TakenLibraries(ctx); !taken["media"].Done {
t.Fatal("the act done under the take's warrant did not make it taken")
}
// A failed act forgets a take not done, so it is asked again; never one done.
again := started
again.Root, again.Ask, again.Warrant = "/storage/media2", "c02", "c02"
if err := inv.RecordTakeStarted(ctx, again); err != nil {
t.Fatal(err)
}
if err := inv.TakeFailed(ctx, "media", "c02"); err != nil {
t.Fatal(err)
}
if taken, _ = inv.TakenLibraries(ctx); taken["media"].Ask == "c02" {
t.Fatalf("a failed take is still recorded: %+v", taken["media"])
}
}
@@ -0,0 +1,50 @@
-- A module that joins has a user and a group of its own, each library has a group of its own, and a library's
-- existing data is taken once, on the operator's word (novox/hq ADR 0328 §3, §4, §7, §15).
--
-- **One sequence for every number the mesh gives.** A joining module is given one number, its user's and its own
-- group's (mesh-<module>); a library is given one, its group's (mesh-<library>). Both are drawn from this sequence, so
-- a module's own group and a library's group never share a number. The range is 50000–59999 (catalogue.IdentityLow and
-- IdentityHigh): above the numbers distributions give people, below the 60000s Debian keeps for its daemons, and
-- nowhere near root's 0 or a root group's (0, 4, 10). The checks refuse a row outside it, so no write, by hand or by a
-- later bug, can make one root's (§15). A sequence never hands a number back, and a row is kept after its module or
-- library is gone, so a number once given is never given to another: a file left owned by a removed module's user is
-- never inherited by a new one.
create sequence mesh_number as integer minvalue 50000 maxvalue 59999 start 50000 no cycle;
create table module_identity (
module text primary key,
uid integer not null unique check (uid between 50000 and 59999),
allocated timestamptz not null default now()
);
create table library_group (
library text primary key,
gid integer not null unique check (gid between 50000 and 59999),
allocated timestamptz not null default now()
);
-- **A library taken** (§7). The first time a library's root on its home holds data not yet in its group, the operator
-- is asked once for that library; on the warrant the node-engine sets the group, group write and setgid under the root,
-- and the library is taken: kept from then on without asking. The row is the record of that act — on which machine
-- root and filesystem, who approved it, under which warrant. A library whose home, root or filesystem changed since
-- (it moved to another machine, path or disk) is not taken by that warrant, and the operator is asked again; a new take
-- replaces the row.
create table kept_library (
library text primary key,
home text not null,
-- The root it was taken at: a library whose root changed since is taken again.
root text not null check (root like '/%'),
-- The filesystem the root was on when taken, as the engine names it without a path ("fs-<12 hex>"); empty when
-- the engine named none.
placed text not null default '',
taken_at timestamptz not null default now(),
taken_by text not null,
ask text not null,
warrant text not null,
-- What the node-engine answered when it started the take.
said text not null default '',
-- Whether the engine has stated the take's act done under this warrant: only then is the library taken. Until
-- then the take is under way — the library still waits, and the operator is not asked again — and an act stated
-- failed forgets the row, so the operator is asked again.
done boolean not null default false
);
+45
View File
@@ -0,0 +1,45 @@
package inventory
import (
"strings"
"testing"
"time"
)
// **A plan's id is a value, never SQL** (novox/hq issue 561): the id reaches PlanByID from a caller of the
// controller's plans verb, so an id holding a quote is read as a plain id — no error from the store, and no
// other plan answered for it.
func TestAPlanIdHoldingAQuoteIsReadAsAPlainId(t *testing.T) {
inv := ForTest(t)
ctx := t.Context()
p := Plan{ID: "plan-561", Repository: "novox/mesh-controller", Commit: "abc", Created: time.Now().UTC(),
State: PlanBuilding, Tiers: [][]string{{"a"}}, Modules: map[string]*PlanModule{"a": {}}}
if err := inv.SavePlan(ctx, &p); err != nil {
t.Fatal(err)
}
for _, id := range []string{
"x' or '1'='1", // every plan, when pasted into the query
"plan-561' or id like '%", // the plan above, by a pattern
"it's", // a quote alone: a syntax error, when pasted
} {
got, err := inv.PlanByID(ctx, id)
if err == nil {
t.Fatalf("PlanByID(%q) answered %s, a plan of another id", id, got.ID)
}
if want := "no plan " + id; err.Error() != want {
t.Fatalf("PlanByID(%q) failed with %q, not as a plain id no plan has (%q)", id, err, want)
}
}
// A plan whose id holds a quote is found by it, and by nothing else.
quoted := p
quoted.ID, quoted.Revision = "plan-'561'", 0
if err := inv.SavePlan(ctx, &quoted); err != nil {
t.Fatal(err)
}
if got, err := inv.PlanByID(ctx, quoted.ID); err != nil || got.ID != quoted.ID {
t.Fatalf("the plan with a quote in its id was not read back: %v %+v", err, got.ID)
}
if recent, err := inv.RecentPlans(ctx, 1); err != nil || len(recent) != 1 || !strings.HasPrefix(recent[0].ID, "plan-") {
t.Fatalf("RecentPlans with a parameter: %v %+v", err, recent)
}
}
+4 -3
View File
@@ -623,7 +623,7 @@ func (i *Inventory) PlansSince(ctx context.Context, since time.Time) ([]Plan, er
// RecentPlans is the last few plans, newest first, open or not — what the overview shows.
func (i *Inventory) RecentPlans(ctx context.Context, limit int) ([]Plan, error) {
return i.plans(ctx, fmt.Sprintf(`order by created desc limit %d`, limit))
return i.plans(ctx, `order by created desc limit $1`, limit)
}
// NewestMergeOf is the plan, in any state, of the newest merge into a repository's branch that the mesh
@@ -638,9 +638,10 @@ func (i *Inventory) NewestMergeOf(ctx context.Context, repository, branch string
return plans[0], true, nil
}
// PlanByID is one plan.
// PlanByID is one plan. The id is a parameter, never part of the query (novox/hq issue 561): it comes from a
// caller of the plans verb.
func (i *Inventory) PlanByID(ctx context.Context, id string) (Plan, error) {
plans, err := i.plans(ctx, `where id = '`+id+`'`)
plans, err := i.plans(ctx, `where id = $1`, id)
if err != nil {
return Plan{}, err
}
+47 -10
View File
@@ -601,6 +601,24 @@ func Acknowledge(ctx context.Context) {
}
}
// LongestAnswer is the longest a handler that waits on purpose may hold its caller's one answer
// (AnswersWithin, novox/hq issue 229): inside the console's thirty seconds with room for the trip back,
// and inside the bus's window for an answer (broker.ResponseTTL). A variable so a test need not wait.
var LongestAnswer = 25 * time.Second
type answerWithinKey struct{}
// AnswersWithin asks that the call's caller be answered in full if the handler ends within d of the
// call's start, rather than within AnswerWithin: for a handler that waits on purpose, with a bound —
// `plans` waiting for a plan to move (novox/hq issue 229). Past d, or past LongestAnswer whichever is
// shorter, its caller is told it is running, as for any call. Never shortens the window; without effect
// outside a served call, and once the caller was answered.
func AnswersWithin(ctx context.Context, d time.Duration) {
if f, ok := ctx.Value(answerWithinKey{}).(func(time.Duration)); ok {
f(d)
}
}
// running is the interim answer: what a caller reads when the call has not finished.
func running(c *Call, within time.Duration, acknowledged bool, follow string) []byte {
why := fmt.Sprintf("it has not finished in %s", within)
@@ -645,6 +663,13 @@ func (l *CallLog) serveCallWithin(seat, verb string, args json.RawMessage, reply
acknowledged := make(chan struct{})
var once sync.Once
ctx = context.WithValue(ctx, answerNowKey{}, func() { once.Do(func() { close(acknowledged) }) })
longer := make(chan time.Duration, 1)
ctx = context.WithValue(ctx, answerWithinKey{}, func(d time.Duration) {
select {
case longer <- d:
default:
}
})
type outcome struct {
body []byte
@@ -703,17 +728,29 @@ func (l *CallLog) serveCallWithin(seat, verb string, args json.RawMessage, reply
}
}
}
timer := time.NewTimer(AnswerWithin)
within, start := AnswerWithin, time.Now()
timer := time.NewTimer(within)
defer timer.Stop()
select {
case o := <-done:
l.finish(c, o.body, o.failed, false)
say(o.body)
return
case <-acknowledged:
say(running(c, AnswerWithin, true, l.Follow))
case <-timer.C:
say(running(c, AnswerWithin, false, l.Follow))
waiting:
for {
select {
case o := <-done:
l.finish(c, o.body, o.failed, false)
say(o.body)
return
case <-acknowledged:
say(running(c, within, true, l.Follow))
break waiting
case d := <-longer:
// A handler waiting on purpose (AnswersWithin): its window is stretched, never past LongestAnswer.
if d = min(d, LongestAnswer); d > within {
within = d
timer.Reset(time.Until(start.Add(d)))
}
case <-timer.C:
say(running(c, within, false, l.Follow))
break waiting
}
}
o := <-done
l.finish(c, o.body, o.failed, true)
+55
View File
@@ -0,0 +1,55 @@
package link
import (
"context"
"encoding/json"
"testing"
"time"
)
// **A handler that waits on purpose is answered in full when it ends inside the window it asked for**
// (novox/hq issue 229): `plans` asked to wait for a plan holds its caller's one answer until the plan
// moves or its bound passes, rather than being answered "still running" at AnswerWithin.
func TestAHandlerThatWaitsOnPurposeIsAnsweredInFullInsideItsWindow(t *testing.T) {
shortWindow(t, 20*time.Millisecond)
l, a := NewCallLog(), newAnswers(t)
l.serveCall("mesh-controller", "plans", json.RawMessage(`{"wait":"plan-1"}`), "_INBOX.x.3",
func(ctx context.Context, _ json.RawMessage) (any, error) {
AnswersWithin(ctx, 200*time.Millisecond)
time.Sleep(80 * time.Millisecond) // past AnswerWithin, inside what it asked for
return "plan-1 moved", nil
}, a.respond, nil)
if got := a.only()["result"]; got != "plan-1 moved" {
t.Fatalf("a waiting handler inside its window was answered %v, not in full", got)
}
}
// What a handler may ask for is bounded: past LongestAnswer its caller is told it is running, as any
// call is, so no answer is ever later than the console waits.
func TestAWaitIsNeverLongerThanTheLongestAnswer(t *testing.T) {
shortWindow(t, 10*time.Millisecond)
was := LongestAnswer
LongestAnswer = 30 * time.Millisecond
t.Cleanup(func() { LongestAnswer = was })
l, a := NewCallLog(), newAnswers(t)
release := make(chan struct{})
finished := make(chan struct{})
go func() {
l.serveCall("mesh-controller", "plans", nil, "_INBOX.x.4", func(ctx context.Context, _ json.RawMessage) (any, error) {
AnswersWithin(ctx, time.Hour)
<-release
return "late", nil
}, a.respond, nil)
close(finished)
}()
select {
case <-a.sent:
case <-time.After(2 * time.Second):
t.Fatal("a handler asking for an hour held its caller past LongestAnswer")
}
if got, _ := a.only()["result"].(map[string]any); got["running"] != true {
t.Fatalf("past LongestAnswer the caller was answered %v, not that it is running", a.only())
}
close(release)
<-finished
}
+3
View File
@@ -325,6 +325,9 @@ type ChangePlan struct {
Steps []string `json:"steps,omitempty"`
// Summary is the plan in one line, for a commit status.
Summary string `json:"summary"`
// Class is the class of the slowest-budgeted module the change moves (novox/hq ADR 0282 §1, ADR 0313 §2):
// core, leaf, or none when it moves no module at all — a fix that moves nothing has no fix budget.
Class string `json:"class,omitempty"`
// Trunk is the trunk commit the head was merged into when it was judged (novox/hq ADR 0323 §3), twelve
// characters: set on the plan a verdict carries, which mesh-delivery keeps with its other inputs.
Trunk string `json:"trunk,omitempty"`
+70
View File
@@ -8,6 +8,7 @@ package link
import (
"encoding/base64"
"encoding/json"
"github.com/novox/mesh-host/rootsearch"
"strconv"
"strings"
@@ -436,6 +437,34 @@ const ReadinessContract = 2
// parses strictly and would refuse the whole declaration for it.
const RootContract = 3
// IdentityContract is the statement of an engine that gives each module its identity and keeps shared folders
// (novox/hq ADR 0325; mesh-host internal/link IdentityContract): it reads a declaration's `identities`, a container's
// resolved `identity`, an access's `users` and `taken`, states an access it may not keep yet as waiting
// (ResourceHealth.AccessWaits), and hears the take act (AskAccessAct). Only to such an engine is any of it sent: an
// older one parses strictly and would refuse the whole declaration for it. Four, the grant's number (ADR 0312,
// superseded), is what engines stated before; the controller sends no grant to any engine (#270).
const IdentityContract = 5
// LibraryContract is the statement of an engine that makes the mesh's users and library groups and keeps libraries
// (novox/hq ADR 0328, which supersedes ADR 0325): it reads a declaration's `groups`, `identities` and `libraries`,
// states a library it may not keep yet as waiting, and hears the take of a library. Only to such an engine is any of
// it sent; five, ADR 0325's, is sent nothing new.
const LibraryContract = 6
// OwnGroupContract is the statement of an engine that also puts the tool runner's account in a joining module's own
// group, named by an identity's `runner`, and takes a directory's mode with the setgid bit, such as "2750" (novox/hq
// ADR 0334 §1, §2). Not stated by any engine yet: the engine must add `runner` (an account's name) to its declaration's
// Identity and put that account in the group mesh-<module>, and accept a directory mode whose first digit is 2.
const OwnGroupContract = 7
// KindRunner is the tool runner's membership as the engine states it (novox/hq ADR 0334 §2; mesh-host internal/link):
// `runner:<module>` of a joining module, in its own group; `runner:library:<name>` of no module, in a library's group.
const KindRunner = "runner"
// KindLibrary is a library's own verdict on its home, as the engine may state it (novox/hq ADR 0328): healthy once
// kept. Read beside an access verdict naming the library until the engine's shape is settled.
const KindLibrary = "library"
// ReasonRoot starts the reason of an account verdict that found a way to root (ADR 0266); the node-engine's
// own words (mesh-host internal/accounts ReasonRoot).
const ReasonRoot = "can become root without a person"
@@ -580,6 +609,43 @@ type Wait struct {
What string `json:"what"`
}
// WaitKindAccess is the kind of a wait on an access (novox/hq ADR 0325 §7; mesh-host internal/link).
const WaitKindAccess = "access"
// AccessWait is one access waiting for the operator (novox/hq ADR 0325 §7): the access by its module's own id for
// it, the part (its module), and what the operator gives, in words with no path — "let the mesh keep the shared
// folder movies", or "mount the disk that holds books". Beside Waits because mesh-sdk's health.Wait names a secret or
// a setting and nothing else. From an engine at IdentityContract or later.
type AccessWait struct {
Kind string `json:"kind"`
Access string `json:"access"`
Part string `json:"part"`
What string `json:"what"`
// Placed names, with no path, the filesystem the folder is on now ("fs-<12 hex>"), on a wait for the operator's
// word: a warrant given for another filesystem (the folder moved) does not answer it, and the controller asks
// again (mesh-host internal/link AccessWait.Placed).
Placed string `json:"placed,omitempty"`
// Reason is what the wait is for: WaitKeep (the operator's word to take the library) or WaitDisk (its disk
// mounted).
Reason string `json:"reason"`
}
// The reasons of an access wait (mesh-host internal/link).
const (
WaitKeep = "keep"
WaitDisk = "disk"
)
// KindAccess is an access's verdict (novox/hq ADR 0325; mesh-host internal/link KindAccess): healthy while each user
// has its right, waiting while the mesh may not keep its folder yet or its disk is not there, unhealthy otherwise —
// and unknown when its filesystem could not be named this look (a hung mount), carrying no wait.
const KindAccess = "access"
// KindKeptEntries is the mesh's entries on a shared folder that nobody wants and that could not be taken out yet
// because its disk was away (novox/hq ADR 0325 §8; mesh-host internal/link): unhealthy, said under the access that
// last named it (`<access id>~entries`), which may no longer be declared. The node-engine's own.
const KindKeptEntries = "kept-entries"
// ResourceHealth is one long-running resource's state.
type ResourceHealth struct {
Module string `json:"module"`
@@ -606,6 +672,10 @@ type ResourceHealth struct {
// Waits is what a resource in StateWaiting waits for (novox/hq ADR 0283). Empty otherwise, and from an
// engine older than that.
Waits []Wait `json:"waits,omitempty"`
// AccessWaits is what an access stated waiting waits for (novox/hq ADR 0325 §7); Act the last walk under its
// folder, kept as the engine said it. From an engine at IdentityContract or later.
AccessWaits []AccessWait `json:"access-waits,omitempty"`
Act json.RawMessage `json:"act,omitempty"`
}
// HealthSaid is the health event's body: the machine and its statement. The machine is read from the
+114
View File
@@ -0,0 +1,114 @@
package link
import (
"context"
"crypto/rand"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"strings"
"time"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-controller/internal/broker"
)
// The take act: a shared folder that already holds data is taken by the mesh once, on the operator's word
// (novox/hq ADR 0325 §8).
//
// The controller asks the operator once for that folder on that machine (ADR 0259, at the level approve). On the
// warrant it asks the machine's node-engine, on the engine's own subject (broker.AskAccessActSubject), signed with
// the mesh's key as a hand-over is (SignedHandOver) under a signing context of its own (AccessActContext), to take
// the folder: the engine adds the entries of the users it was declared for that folder to everything under it —
// following no link, entering no other filesystem — answers at once that it started or why it refused, and states
// the walk on the access's verdict. **Nothing is taken without a warrant**: the ask names the warrant the operator
// gave, and the controller sends it only from the warrant (cmd/mesh-controller decideTake).
//
// The engine's own shape (mesh-host internal/link AccessActAsk, AnswerAccessAct): a hand-over's fields and
// {act: "take", access: <the access's resource id in the declaration>, warrant}.
// AccessActContext is prefixed to an act ask's bytes before signing: never another ask's signature, nor a
// declaration's. The engine holds the same words.
const AccessActContext = "novox-mesh access-act v2\n"
// ActTake is the one act the controller asks.
const ActTake = "take"
// AccessActAsk is what the controller asks: a hand-over's fields — the node it is for, the folder's absolute path as
// the declaration places it, who asked, when the ask stops being good and a nonce the engine takes once — and the
// act's own: which act, the access by its resource id, and the warrant the operator gave.
type AccessActAsk struct {
Node string `json:"node"`
Path string `json:"path"`
By string `json:"by"`
Expires time.Time `json:"expires"`
Nonce string `json:"nonce"`
Act string `json:"act"`
Access string `json:"access"`
Warrant string `json:"warrant"`
}
// MaxActField bounds an access id and a warrant, as the engine does.
const MaxActField = 200
// TakeWithin is how long the controller waits for the engine's answer: it starts the walk and answers, and the
// walk's end is stated on the access's verdict.
const TakeWithin = 30 * time.Second
// SignTake fills a take ask's act, expiry and nonce and signs it with the mesh's key over AccessActContext. It refuses
// an ask with no warrant, no path or no access — and one the engine would refuse for its shape: nothing is ever asked
// of an engine that it would have to refuse.
func SignTake(ctx context.Context, signer Signer, ask AccessActAsk) ([]byte, error) {
ask.Access, ask.Warrant = strings.TrimSpace(ask.Access), strings.TrimSpace(ask.Warrant)
switch {
case signer == nil:
return nil, errors.New("no signing key, so nothing can be asked of an engine")
case ask.Warrant == "":
return nil, errors.New("a take is asked only on the operator's warrant, and this one names none")
case ask.Node == "" || !strings.HasPrefix(ask.Path, "/"):
return nil, errors.New("a take names its machine and the folder's absolute path")
case ask.Access == "":
return nil, errors.New("a take names the access whose folder it takes")
}
// The warrant is one short name; the access is the resource id the declaration gave it, which may hold a path's
// slashes (catalogue.AccessID) — the engine judges it against what it was declared.
if len(ask.Warrant) > MaxActField || strings.ContainsAny(ask.Warrant, "\r\n\x00/") {
return nil, fmt.Errorf("the take's warrant %q is not one short name", ask.Warrant)
}
if len(ask.Access) > MaxActField || strings.ContainsAny(ask.Access, "\r\n\x00") {
return nil, fmt.Errorf("the take's access %q is not one line", ask.Access)
}
nonce := make([]byte, 16)
if _, err := rand.Read(nonce); err != nil {
return nil, err
}
ask.Act, ask.Nonce, ask.Expires = ActTake, hex.EncodeToString(nonce), time.Now().UTC().Add(HandOverGood)
raw, err := json.Marshal(ask)
if err != nil {
return nil, err
}
signature, err := signer.Sign(ctx, append([]byte(AccessActContext), raw...))
if err != nil {
return nil, fmt.Errorf("cannot sign the ask: %w", err)
}
return json.Marshal(SignedHandOver{Ask: raw, Signature: signature})
}
// AskTake asks one machine's node-engine to take a shared folder on the operator's warrant, and reads its answer. An
// error is the ask not reaching an engine, or an answer that is not one; a refusal is the engine's and comes back in
// the answer.
func AskTake(ctx context.Context, conn *nats.Conn, signer Signer, ask AccessActAsk, timeout time.Duration) (HandOverAnswer, error) {
if conn == nil {
return HandOverAnswer{}, errors.New("this controller is not on the bus")
}
body, err := SignTake(ctx, signer, ask)
if err != nil {
return HandOverAnswer{}, err
}
return askSigned(ctx, conn, broker.AskAccessActSubject(ask.Node), body, ask.Node, timeout, askWords{
what: "the take of a shared folder", nothing: "nothing was taken", issue: "ADR 0325",
unknown: "whether it started is not known — the access's verdict on the machine says whether it is kept",
record: "a start"})
}
+2 -2
View File
@@ -8,7 +8,7 @@ private addresses. That is acceptable while mesh-controller is private; if it ev
a second place to clean besides the catalogue itself.
mesh-catalog b9de001833b1b61b297182b8e3bcdae1cbdeace9 modules/*/module.json, modules/nats/Dockerfile
mesh-host a76baa4b47f07a8b2bab065603414530bd02b53a examples/foundation-first-node-nats.lock
mesh-host 7a41e120635b3f7b6cb94cca8b3fc4093e4e6110 examples/foundation-first-node.lock
mesh-sdk 33917f91ac3cc213b879b32f3c70e2aa70395f98 conformance/events/module-event.json
To move them, from this repository's root, with the two repositories checked out beside it:
@@ -18,7 +18,7 @@ To move them, from this repository's root, with the two repositories checked out
git -C ../mesh-catalog archive <commit> modules | tar -x -C testdata/beside/mesh-catalog --wildcards \
'modules/*/module.json' 'modules/nats/Dockerfile'
find testdata/beside -name module.json -exec mv {} {}.captured \;
git -C ../mesh-host archive <commit> examples/foundation-first-node-nats.lock | tar -x -C testdata/beside/mesh-host
git -C ../mesh-host archive <commit> examples/foundation-first-node.lock | tar -x -C testdata/beside/mesh-host
and write the commits here. The SDK is captured at the commit go.mod pins for git.novox.be/novox/mesh-sdk/go
(the last part of its pseudo-version; novox/hq issue 449), which internal/link's conformance test holds it to:
@@ -1,11 +1,17 @@
// foundation-first-node-nats.lock — what a machine must be before a mesh exists, on the bus being
// built (novox/hq ADR 0106, design 25).
// foundation-first-node.lock — what a machine must be before a mesh exists: the one genesis bundle
// (novox/hq ADR 0331 §2), on the bus the mesh runs on (ADR 0106, ADR 0131, design 25).
//
// The same twelve steps as foundation-first-node.lock, with one difference that matters: **the mesh
// writes its own user list, and at genesis there is no mesh yet to write it.** So this carries the
// first one — the controller's own account, at a well-known bootstrap password, exactly as the store
// is reached at `postgres:bootstrap` and the old bus at `guest:guest`. It is rotated with those, and
// from the controller's first composition onward the file is the controller's to write.
// The whole bootstrap (novox/hq 03-DESIGN/01-to-be/07-the-foundation.md): a container runtime, a
// filter, a store, a database per context, those contexts' schemas, the bus with its certificate
// and its first user list, and the control plane running on top of them.
//
// **The mesh writes its own user list, and at genesis there is no mesh yet to write it.** So this
// carries the first one — the controller's own account — and the control plane's address names the
// same account. Both say `bootstrap`, and both are placeholders: the installer makes the account's
// password, puts it in the address and its hash in the list (mesh-host internal/bootstrap
// rootsecrets.go), and refuses a bundle whose bus connection it does not recognise. Until the first
// node has enrolled the installer places the list the controller composes; from the bus's adoption
// as the `nats` module on, the module receives it like any module (ADR 0331 §1, §5).
//
// The accounts file is its own file beside the server's configuration, because the server's own
// settings belong to whoever raises it and the users belong to the mesh (design 25 §4). Both live in
@@ -15,7 +21,18 @@
//
// No `verify` on the TLS block, deliberately — that setting makes the server demand a *client*
// certificate, and nothing in the mesh presents one: a host pins this server's exact certificate and
// authenticates with a password (ADR 0004, design 25 §4).
// authenticates with a password (ADR 0004, design 25 §4). The control plane pins it too, through
// MESH_BROKER_CERTIFICATE, and dials the bus on the machine's own loopback.
//
// PINNED BY DIGEST, and the digest is not decoration: a tag can be made to point at a different
// image, and this file is applied on a machine with no mesh to ask about anything (ADR 0006). The
// registry prefix is the lab's; the digests are upstream's.
//
// The store's data is a NAMED VOLUME, not a directory on the machine. A directory the host
// creates is owned by root, and the database runs as somebody else inside the container — so it
// could not write, and the container crash-looped. A named volume lets the image set up its own
// ownership, and outlives the container, which is what you want for the thing holding the mesh's
// state.
{
"declaration": 1,
@@ -523,6 +523,20 @@ func refusedPlaces(resources []declaration.Resource) map[string]error {
switch res := r.(type) {
case *declaration.Directory:
err = refusePath(res.Path, placingDirectory, moduleOfID(res.ID), res.Owner)
// And outside every home, the walk's judgement before anything is applied (novox/hq issue 358): a
// link another account could make in its path refuses it, and with it what is below it, whatever
// order they are declared in. The apply walks again at the moment of use. A directory handed over
// keeps its own, stricter walk (issue 356), which refuses any link at all.
if err == nil && meshDirsByDescriptor && homeAbove(res.Path) == "" &&
!HandedOver()[handOverKey(res.ID, res.Path)] {
if judged := judgeMeshDir(res.Path); judged != nil {
err = judged
var link *LinkInPathError
if errors.As(judged, &link) {
err = &PlacementRefusedError{Path: res.Path, Why: "it " + link.why()}
}
}
}
case *declaration.Access:
err = refusePath(res.Path, placingAccess, moduleOfID(res.ID), "")
default:
+22
View File
@@ -7,12 +7,14 @@
./cmd/mesh-controller TestAStopNamesWhatStaysOnTheMachinesAndWhatStillCarriesIt
./cmd/mesh-controller TestAStopWithoutAMergeLeavesItOutOfABatch
./cmd/mesh-controller TestSavingPlansAndMovingMergesIsOneAct
./cmd/mesh-controller TestAPlanFollowedFromTheStoreAnswersWhenItMoves
./internal/inventory TestTheLastSendSaysWhatTheMachineAnswered
./internal/inventory TestAReportIsTheLastSendsAnswerOnlyWhenItIsAboutThatSend
./cmd/mesh-controller TestNodeShowAndStatusReadTheLastSendFromTheStore
./internal/inventory TestAnIssuedValueIsSealedToEveryHolderOnceAndNotAgain
./internal/inventory TestAnIssuedValueIsSealedAgainAfterAKeyChangeOrAnAcceptance
./internal/inventory TestAConditionsLooksAreKeptBesideTheBucket
./internal/inventory TestAPlanIdHoldingAQuoteIsReadAsAPlainId
./cmd/mesh-controller TestTheConditionsVerbReadsTheLooksTheBucketNoLongerHolds
./internal/inventory TestAWalksStartIsKeptAndARefusedSaveLeavesItUnsaid
./cmd/mesh-controller TestAWalkLetGoAndStoppedByVerbsIsSaidStartedAndEnded
@@ -29,3 +31,23 @@
./cmd/mesh-controller TestADirectoryOwnersKeyIsRefusedThroughAVerb
./cmd/mesh-controller TestTheSendMakesNoValueForAnOutsideSecretNobodyGave
./cmd/mesh-controller TestAQuestionComposesWithoutAnOutsideSecretNobodyGave
./cmd/mesh-controller TestReplay201
./cmd/mesh-controller TestARollbackWhoseSendWasRefusedIsCompletedByTheNextSend
./cmd/mesh-controller TestAGateJudgesTheOtherMovesBesideAKeptCore
./cmd/mesh-controller TestReplay401
# novox/hq ADR 0328 §1, §3, §4, §7, §13, §15 and issue 575: numbers allocated once and never root, libraries defined
# once and never overlapping, a library taken under a warrant, a placement judged against the build that brings it.
./internal/inventory TestOneNumberPerModuleAndLibraryNeverRoot
./internal/inventory TestALibraryIsDefinedOnceForTheMeshAndNeverOverlaps
./internal/inventory TestALibraryIsTakenOnceUnderAWarrant
./internal/inventory TestAPlacementIsJudgedAgainstTheBuildThatBringsTheAccess
./internal/inventory TestOnlyABuildOfTheFollowedBranchJudgesAPlacementAhead
./cmd/mesh-controller TestALibrarysVerdictIsSaidForTheModulesThatUseIt
./cmd/mesh-controller TestAHeldModuleIsComposedAsBeforeItJoined
./cmd/mesh-controller TestALibraryIsTakenOnlyWhenTheEngineSaysTheActIsDone
./internal/inventory TestTheSubPathsOfModulesPulledInByARequirementAreKept
./internal/inventory TestALibraryIsTakenWhenTheActIsDone
./cmd/mesh-controller TestADeliveryWaitsForALibraryAndJudgesTheRest
./cmd/mesh-controller TestAJoiningModuleIsHeldOnAnEngineOlderThanOwnGroups
./cmd/mesh-controller TestAHeldModuleKeepsTheSettingsItRanWith
./cmd/mesh-controller TestARunnersVerdictIsSaidAndJudgedByWhetherTheMoveChangesIt
+348 -5
View File
@@ -12,6 +12,7 @@ import (
"fmt"
"io"
"net"
"path/filepath"
"reflect"
"regexp"
"slices"
@@ -132,7 +133,13 @@ func (d *Directory) validate(where string, _ bool) []string {
if d.Path == "" {
problems = append(problems, where+": a directory needs a path")
}
return append(problems, checkMode(where, d.Mode)...)
// A directory's mode may carry the setgid bit (2xxx), so what is made beneath it keeps its group (novox/hq ADR
// 0334 §1); never setuid or sticky, which mean nothing the mesh asks of a directory it owns.
mode := d.Mode
if len(mode) == 4 && mode[0] == '2' {
mode = "0" + mode[1:]
}
return append(problems, checkMode(where, mode)...)
}
// File is a file with literal content. The host renders nothing.
@@ -456,14 +463,51 @@ const (
// **The distinction 04-ISSUES/036 and 026 turn on.** A `directory` resource is the mesh's own —
// it creates it, sets its owner and mode, and removes it when empty ([ADR 0030](novox/hq)). A
// media library, a download spool is the operator's: it existed before the mesh, several modules
// read and write it at once, and the mesh must not create, chown, reconcile or remove it. The
// host confirms it is there and mounts it; nothing else.
// read and write it at once, and the mesh must not create, chown, chmod or remove it. The host
// confirms it is there and mounts it; nothing else. Library data is a library's (novox/hq ADR 0328),
// not an access.
type Access struct {
ID string `json:"id"`
Type Type `json:"type"`
Path string `json:"path"`
// Mode is how this module reaches the path: read or read-write. Absent narrows to read.
Mode string `json:"mode,omitempty"`
// Users and Taken are what a controller speaking ADR 0325 sends (the identities that use the folder, and
// that the operator's warrant was carried out): accepted, so its declaration is not refused whole, and not
// acted on since ADR 0328 superseded ADR 0325 — no ACL entry is kept.
Users []AccessUser `json:"users,omitempty"`
Taken bool `json:"taken,omitempty"`
// Grant is what a controller older than ADR 0325 still sends (ADR 0312's group grant, superseded): accepted
// so its declaration is not refused whole, and never acted on — no account is put in a group for data (§5).
Grant json.RawMessage `json:"grant,omitempty"`
}
// The uses an access's user has (ADR 0051 §3), and where it runs (ADR 0325 §4).
const (
FromContainer = "container"
FromToolRunner = "tool-runner"
)
// AccessUser is one identity that uses an accessed folder: its name and uid as the controller allocated them, its
// use, and where it runs. A tool runner's account may come with no uid (0 or absent): the engine reads it by name.
type AccessUser struct {
Identity string `json:"identity"`
UID int `json:"uid,omitempty"`
Use string `json:"use"`
From string `json:"from"`
}
// UnmarshalJSON reads a user as strictly as a resource is read.
func (u *AccessUser) UnmarshalJSON(raw []byte) error {
type plain AccessUser
var p plain
if err := strictDecode(raw, &p); err != nil {
return fmt.Errorf("an access's user: %w", err)
}
*u = AccessUser(p)
return nil
}
func (a *Access) Identity() string { return a.ID }
@@ -482,9 +526,43 @@ func (a *Access) validate(where string, _ bool) []string {
problems = append(problems, fmt.Sprintf(
"%s: an access is %q or %q, not %q", where, AccessRead, AccessReadWrite, a.Mode))
}
for _, u := range a.Users {
switch {
case u.Identity == "" || !accountName.MatchString(u.Identity):
problems = append(problems, fmt.Sprintf("%s: a user of an access is an account's name, and %q is not one",
where, u.Identity))
case u.UID < 0 || (u.UID == 0 && u.From != FromToolRunner):
// Never root (novox/hq ADR 0325 §11): an entry for uid 0 would be a right root has anyway, and a
// number below it is no account. A tool runner's account alone may come without its number: the
// controller cannot know the operator's account's, and the engine reads it by name
// (apply.ResolveRunners), refusing one that is root's.
problems = append(problems, fmt.Sprintf("%s: the user %s has uid %d, and a module's identity is never "+
"root or below it (novox/hq ADR 0325 §11)", where, u.Identity, u.UID))
}
if u.Use != AccessRead && u.Use != AccessReadWrite {
problems = append(problems, fmt.Sprintf("%s: the user %s uses it %q, and a use is %q or %q", where,
u.Identity, u.Use, AccessRead, AccessReadWrite))
}
if u.Use == AccessReadWrite && a.Mode != AccessReadWrite {
problems = append(problems, fmt.Sprintf("%s: the user %s writes an access declared %s", where,
u.Identity, orRead(a.Mode)))
}
if u.From != FromContainer && u.From != FromToolRunner {
problems = append(problems, fmt.Sprintf("%s: the user %s runs from %q, and that is %q or %q", where,
u.Identity, u.From, FromContainer, FromToolRunner))
}
}
return problems
}
// orRead is an access's mode as said: absent narrows to read.
func orRead(mode string) string {
if mode == "" {
return AccessRead
}
return mode
}
func (u *User) Identity() string { return u.ID }
func (u *User) Kind() Type { return TypeUser }
func (u *User) Target() string { return u.Name }
@@ -1160,6 +1238,78 @@ type Container struct {
// Health is how this resource is ready (novox/hq ADR 0240 rule 2, Phase B): one kind and its
// timing, judged by the node-engine beside liveness. Absent: judged alive or not, and nothing more.
Health *Health `json:"health,omitempty"`
// RunAs is who the container runs as, resolved by the controller from the module's identity and the
// image's declared interface (novox/hq ADR 0325 §2, §3): told by environment (the image reads PUID and
// PGID, say) or by the runtime's own user option. Part of the spec, so a change recreates it. Absent: the
// container runs as its image says. Its key is `identity`, the manifest's word; the field is not, because
// a resource's Identity is its id.
RunAs *ContainerIdentity `json:"identity,omitempty"`
}
// The ways an image is told who to run as (novox/hq ADR 0325 §2).
const (
IdentityByEnv = "env"
IdentityByRun = "run"
)
// ContainerIdentity is who a container runs as, and how its image is told.
type ContainerIdentity struct {
// Mode is IdentityByEnv or IdentityByRun.
Mode string `json:"mode"`
// UserEnv and GroupEnv are the variables the image reads the user's and the group's number from, for
// IdentityByEnv: "PUID", "PGID".
UserEnv string `json:"user_env,omitempty"`
GroupEnv string `json:"group_env,omitempty"`
UID int `json:"uid"`
GID int `json:"gid"`
}
// UnmarshalJSON reads an identity as strictly as a resource is read.
func (i *ContainerIdentity) UnmarshalJSON(raw []byte) error {
type plain ContainerIdentity
var p plain
if err := strictDecode(raw, &p); err != nil {
return fmt.Errorf("a container's identity: %w", err)
}
*i = ContainerIdentity(p)
return nil
}
// envName is a variable's name as a runtime takes it.
var envName = regexp.MustCompile(`^[A-Za-z_][A-Za-z0-9_]*$`)
func (i *ContainerIdentity) problems(where string, env map[string]string) []string {
if i == nil {
return nil
}
var problems []string
switch i.Mode {
case IdentityByEnv:
if !envName.MatchString(i.UserEnv) || (i.GroupEnv != "" && !envName.MatchString(i.GroupEnv)) {
problems = append(problems, fmt.Sprintf("%s: an identity told by environment names the variables "+
"it is told in, and %q and %q are not both names", where, i.UserEnv, i.GroupEnv))
}
for _, name := range []string{i.UserEnv, i.GroupEnv} {
if _, set := env[name]; set && name != "" {
problems = append(problems, fmt.Sprintf("%s: env sets %s, which the identity sets; one value "+
"for one variable", where, name))
}
}
case IdentityByRun:
if i.UserEnv != "" || i.GroupEnv != "" {
problems = append(problems, where+": an identity told by the runtime's user option names no variable")
}
default:
problems = append(problems, fmt.Sprintf("%s: an identity is told by %q or %q, not %q", where,
IdentityByEnv, IdentityByRun, i.Mode))
}
// Never root (novox/hq ADR 0325 §11): refused here, so nothing of the declaration is made or started.
if i.UID <= 0 || i.GID <= 0 {
problems = append(problems, fmt.Sprintf("%s: the identity is uid %d, gid %d, and a module's identity is "+
"never root, nor a number below it (novox/hq ADR 0325 §11)", where, i.UID, i.GID))
}
return problems
}
func (c *Container) Identity() string { return c.ID }
@@ -1242,6 +1392,7 @@ func (c *Container) validate(where string, _ bool) []string {
problems = append(problems, where+": networks is for a container that keeps running; a step "+
"runs and exits, and joins nothing afterwards")
}
problems = append(problems, c.RunAs.problems(where+": container "+c.Name, c.Env)...)
return append(problems, checkImage(where, c.Image)...)
}
@@ -1392,6 +1543,185 @@ type Declaration struct {
// the generation of what it puts back, which may be older than what the machine applied, and is
// not refused for it.
PutBack bool
// Identities is the identity of every module assigned to this machine (novox/hq ADR 0328 §4): an account
// and a group `mesh-<module>` with the numbers the controller allocated once for the module, the same on
// every machine. The engine makes each before any resource is applied, and removes what it made of one no
// longer declared. Absent is a controller older than ADR 0325.
Identities []Identity
// Groups is the group of every library this machine is reached by (novox/hq ADR 0328 §3): `mesh-<library>`
// with the number the controller allocated once, mesh-wide. The engine makes each, and removes what it made of
// one no longer declared. Absent is a controller older than ADR 0328.
Groups []Group
// Libraries is every library this machine is reached by (ADR 0328 §1, §6, §9): its root here, its group, whether
// this is its home, whether the operator's warrant to take it was carried out, the sub-paths its modules ask for,
// and the tool runner's account to put in its group (§11). Absent is a controller older than ADR 0328.
Libraries []Library
}
// Identity is one module's identity, as the controller allocated it (ADR 0328 §4): its user `mesh-<module>` with
// uid, its own group of the same name and number, its primary group gid (its own, or its first library's group),
// and the groups of its other libraries as added groups.
type Identity struct {
Module string `json:"module"`
Name string `json:"name"`
UID int `json:"uid"`
GID int `json:"gid"`
Groups []int `json:"groups,omitempty"`
// Runner is the tool runner's account, put in the module's own group (novox/hq ADR 0334 §2) when its tools run
// on this machine. Absent for a module that declares no tools here, and from a controller older than ADR 0334.
Runner string `json:"runner,omitempty"`
}
// Group is one library's group, as the controller allocated it (ADR 0328 §3).
type Group struct {
Name string `json:"name"`
GID int `json:"gid"`
}
// GroupPrefix starts every library group's name, as IdentityPrefix starts every module's.
const GroupPrefix = "mesh-"
// LibraryAccessPrefix starts the access a library's wait and its take act name: `library:<name>`.
const LibraryAccessPrefix = "library:"
// Library is one library as this machine sees it (ADR 0328).
type Library struct {
Library string `json:"library"`
// Root is the library's root on this machine: on its home the data itself, elsewhere the mount point the mounts
// module mounts it at.
Root string `json:"root"`
Group string `json:"group"`
GID int `json:"gid"`
// Home is that the library's data lives on this machine: only there does the engine keep it.
Home bool `json:"home"`
// Taken is the controller saying the operator's warrant to take it was carried out (§7).
Taken bool `json:"taken,omitempty"`
// Paths are the sub-paths its modules ask for, relative to the root.
Paths []string `json:"paths,omitempty"`
// Runner is the tool runner's account, put in the library's group (§11). Absent when no module's tools use it.
Runner string `json:"runner,omitempty"`
}
// checkLibraries judges the groups and libraries on arrival: every library's group declared, named for it, never
// root's number; paths that stay below the root; a runner that is an account's name; and no two roots that nest
// (§13).
func checkLibraries(groups []Group, libs []Library) []string {
var problems []string
byName := map[string]int{}
numbers := map[int]string{}
for _, g := range groups {
where := fmt.Sprintf("group %q", g.Name)
if !strings.HasPrefix(g.Name, GroupPrefix) || !accountName.MatchString(g.Name) {
problems = append(problems, fmt.Sprintf("%s: a library's group is named %s<library>", where, GroupPrefix))
}
if g.GID <= 0 {
problems = append(problems, fmt.Sprintf("%s: gid %d, and a library's group is never root, nor a number "+
"below it (novox/hq ADR 0328 §15); nothing of this declaration was made", where, g.GID))
}
if _, twice := byName[g.Name]; twice {
problems = append(problems, where+": declared twice")
}
if other, used := numbers[g.GID]; used {
problems = append(problems, fmt.Sprintf("%s: gid %d is %s's too", where, g.GID, other))
}
byName[g.Name], numbers[g.GID] = g.GID, g.Name
}
seen := map[string]bool{}
for _, l := range libs {
where := fmt.Sprintf("library %q", l.Library)
switch {
case l.Library == "" || strings.ContainsAny(l.Library, "/ \t\n"):
problems = append(problems, where+": a library is named by one word")
case seen[l.Library]:
problems = append(problems, where+": declared twice")
}
seen[l.Library] = true
if !filepath.IsAbs(l.Root) || filepath.Clean(l.Root) != l.Root || l.Root == "/" {
problems = append(problems, fmt.Sprintf("%s: its root %q is not an absolute, clean path below /", where,
l.Root))
}
if gid, ok := byName[l.Group]; !ok || gid != l.GID {
problems = append(problems, fmt.Sprintf("%s: its group %s (gid %d) is not one this declaration makes",
where, l.Group, l.GID))
}
if l.Group != GroupPrefix+l.Library {
problems = append(problems, fmt.Sprintf("%s: its group is %s%s, and %s is not", where, GroupPrefix,
l.Library, l.Group))
}
for _, p := range l.Paths {
if p == "" || filepath.IsAbs(p) || filepath.Clean(p) != p || p == ".." || strings.HasPrefix(p, "../") {
problems = append(problems, fmt.Sprintf("%s: the path %q is not a clean path below its root", where, p))
}
}
if l.Runner != "" && !accountName.MatchString(l.Runner) {
problems = append(problems, fmt.Sprintf("%s: its runner %q is not an account's name", where, l.Runner))
}
}
for i, a := range libs {
for _, b := range libs[i+1:] {
if a.Root != "" && b.Root != "" && (a.Root == b.Root || strings.HasPrefix(a.Root, b.Root+"/") ||
strings.HasPrefix(b.Root, a.Root+"/")) {
problems = append(problems, fmt.Sprintf("library %q: its root and library %q's nest, and libraries "+
"never overlap (novox/hq ADR 0328 §13)", a.Library, b.Library))
}
}
}
return problems
}
// IdentityPrefix starts every module identity's name.
const IdentityPrefix = "mesh-"
// checkIdentities judges the identities on arrival: each named for its module, never root, and no name or number
// twice — two modules with one account would write as each other.
func checkIdentities(ids []Identity, groups []Group) []string {
var problems []string
names, uids := map[string]bool{}, map[int]string{}
libraryGroups := map[int]bool{}
for _, g := range groups {
libraryGroups[g.GID] = true
}
for _, id := range ids {
where := fmt.Sprintf("identity %q", id.Name)
if id.Module == "" || id.Name != IdentityPrefix+id.Module || !accountName.MatchString(id.Name) {
problems = append(problems, fmt.Sprintf("%s: a module's identity is named %s<module>, and this one is "+
"for the module %q", where, IdentityPrefix, id.Module))
}
if id.UID <= 0 || id.GID <= 0 {
problems = append(problems, fmt.Sprintf("%s: uid %d, gid %d, and a module's identity is never root, "+
"nor a number below it (novox/hq ADR 0328 §15); nothing of this declaration was made", where,
id.UID, id.GID))
}
if names[id.Name] {
problems = append(problems, where+": declared twice")
}
if other, used := uids[id.UID]; used {
problems = append(problems, fmt.Sprintf("%s: uid %d is %s's too", where, id.UID, other))
}
// Its primary group is its own (gid == uid) or a library's this declaration makes (ADR 0328 §4); so is
// every added group. A module's own number is never a library group's.
if id.GID != id.UID && !libraryGroups[id.GID] {
problems = append(problems, fmt.Sprintf("%s: its primary group gid %d is neither its own (%d) nor a "+
"library's this declaration makes", where, id.GID, id.UID))
}
if libraryGroups[id.UID] {
problems = append(problems, fmt.Sprintf("%s: uid %d is a library group's number too", where, id.UID))
}
for _, g := range id.Groups {
if !libraryGroups[g] {
problems = append(problems, fmt.Sprintf("%s: its added group gid %d is not a library's this "+
"declaration makes", where, g))
}
}
if id.Runner != "" && !accountName.MatchString(id.Runner) {
problems = append(problems, fmt.Sprintf("%s: its runner %q is not an account's name", where, id.Runner))
}
names[id.Name], uids[id.UID] = true, id.Name
}
return problems
}
// LeftOutModuleOf says which left-out module a recorded resource belongs to, if any: its id is the
@@ -1561,6 +1891,13 @@ type envelope struct {
// sends them only to a host whose reports say `reads_generation`.
Generation int64 `json:"generation,omitempty"`
PutBack bool `json:"put_back,omitempty"`
// Identities is optional on the wire too (novox/hq ADR 0325 §3, ADR 0328 §4). **An older host refuses this key**; a
// controller sends it only to a host whose health says link.IdentityContract.
Identities []Identity `json:"identities,omitempty"`
// Groups and Libraries are optional too (novox/hq ADR 0328). **An older host refuses these keys**; a controller
// sends them only to a host whose health says link.LibraryContract.
Groups []Group `json:"groups,omitempty"`
Libraries []Library `json:"libraries,omitempty"`
}
func parse(raw []byte, allowActions bool) (*Declaration, error) {
@@ -1578,8 +1915,10 @@ func parse(raw []byte, allowActions bool) (*Declaration, error) {
}
d := &Declaration{Version: env.Version, For: env.For, Adoption: env.Adoption, Sequence: env.Sequence,
Epoch: env.Epoch, LeftOut: env.LeftOut, Generation: env.Generation, PutBack: env.PutBack}
var problems []string
Epoch: env.Epoch, LeftOut: env.LeftOut, Generation: env.Generation, PutBack: env.PutBack,
Identities: env.Identities, Groups: env.Groups, Libraries: env.Libraries}
problems := checkIdentities(env.Identities, env.Groups)
problems = append(problems, checkLibraries(env.Groups, env.Libraries)...)
if env.Generation < 0 {
// Below zero would read as "none claimed" and pass every refusal of an older generation.
problems = append(problems, fmt.Sprintf("an assignment generation below zero (%d) is not one the "+
@@ -1732,6 +2071,10 @@ func checkWhileStopped(resources []Resource) []string {
return problems
}
// accountName is what an identity's or an access user's name may be: the portable account name the user tools
// take, optionally ending in `$`, never starting with a dash — it is handed to them as an argument.
var accountName = regexp.MustCompile(`^[A-Za-z0-9_.][A-Za-z0-9_.-]*\$?$`)
func strictDecode(raw []byte, into any) error {
// DisallowUnknownFields is the whole point rather than strictness for its own sake: a
// field the host does not know is a thing the control plane believes it asked for.
+8
View File
@@ -0,0 +1,8 @@
# Package xz authors
Michael Cross <https://github.com/xi2>
# XZ Embedded authors
Lasse Collin <lasse.collin@tukaani.org>
Igor Pavlov <http://7-zip.org/>
+86
View File
@@ -0,0 +1,86 @@
Creative Commons CC0 1.0 Universal
CREATIVE COMMONS CORPORATION IS NOT A LAW FIRM AND DOES NOT PROVIDE LEGAL
SERVICES. DISTRIBUTION OF THIS DOCUMENT DOES NOT CREATE AN ATTORNEY-CLIENT
RELATIONSHIP. CREATIVE COMMONS PROVIDES THIS INFORMATION ON AN "AS-IS" BASIS.
CREATIVE COMMONS MAKES NO WARRANTIES REGARDING THE USE OF THIS DOCUMENT OR THE
INFORMATION OR WORKS PROVIDED HEREUNDER, AND DISCLAIMS LIABILITY FOR DAMAGES
RESULTING FROM THE USE OF THIS DOCUMENT OR THE INFORMATION OR WORKS PROVIDED
HEREUNDER.
Statement of Purpose
The laws of most jurisdictions throughout the world automatically confer
exclusive Copyright and Related Rights (defined below) upon the creator and
subsequent owner(s) (each and all, an "owner") of an original work of
authorship and/or a database (each, a "Work").
Certain owners wish to permanently relinquish those rights to a Work for the
purpose of contributing to a commons of creative, cultural and scientific
works ("Commons") that the public can reliably and without fear of later
claims of infringement build upon, modify, incorporate in other works, reuse
and redistribute as freely as possible in any form whatsoever and for any
purposes, including without limitation commercial purposes. These owners may
contribute to the Commons to promote the ideal of a free culture and the
further production of creative, cultural and scientific works, or to gain
reputation or greater distribution for their Work in part through the use and
efforts of others.
For these and/or other purposes and motivations, and without any expectation
of additional consideration or compensation, the person associating CC0 with a
Work (the "Affirmer"), to the extent that he or she is an owner of Copyright
and Related Rights in the Work, voluntarily elects to apply CC0 to the Work
and publicly distribute the Work under its terms, with knowledge of his or her
Copyright and Related Rights in the Work and the meaning and intended legal
effect of CC0 on those rights.
1. Copyright and Related Rights. A Work made available under CC0 may be protected by copyright and related or neighboring rights ("Copyright and Related Rights"). Copyright and Related Rights include, but are not limited to, the following:
i. the right to reproduce, adapt, distribute, perform, display, communicate,
and translate a Work;
ii. moral rights retained by the original author(s) and/or performer(s);
iii. publicity and privacy rights pertaining to a person&apos;s image or
likeness depicted in a Work;
iv. rights protecting against unfair competition in regards to a Work, subject
to the limitations in paragraph 4(a), below;
v. rights protecting the extraction, dissemination, use and reuse of data in a
Work;
vi. database rights (such as those arising under Directive 96/9/EC of the
European Parliament and of the Council of 11 March 1996 on the legal
protection of databases, and under any national implementation thereof,
including any amended or successor version of such directive); and
vii. other similar, equivalent or corresponding rights throughout the world
based on applicable law or treaty, and any national implementations thereof.
2. Waiver. To the greatest extent permitted by, but not in contravention of, applicable law, Affirmer hereby overtly, fully, permanently, irrevocably and unconditionally waives, abandons, and surrenders all of Affirmer&apos;s Copyright and Related Rights and associated claims and causes of action, whether now known or unknown (including existing as well as future claims and causes of action), in the Work (i) in all territories worldwide, (ii) for the maximum duration provided by applicable law or treaty (including future time extensions), (iii) in any current or future medium and for any number of copies, and (iv) for any purpose whatsoever, including without limitation commercial, advertising or promotional purposes (the "Waiver"). Affirmer makes the Waiver for the benefit of each member of the public at large and to the detriment of Affirmer&apos;s heirs and successors, fully intending that such Waiver shall not be subject to revocation, rescission, cancellation, termination, or any other legal or equitable action to disrupt the quiet enjoyment of the Work by the public as contemplated by Affirmer&apos;s express Statement of Purpose.
3. Public License Fallback. Should any part of the Waiver for any reason be judged legally invalid or ineffective under applicable law, then the Waiver shall be preserved to the maximum extent permitted taking into account Affirmer&apos;s express Statement of Purpose. In addition, to the extent the Waiver is so judged Affirmer hereby grants to each affected person a royalty-free, non transferable, non sublicensable, non exclusive, irrevocable and unconditional license to exercise Affirmer&apos;s Copyright and Related Rights in the Work (i) in all territories worldwide, (ii) for the maximum duration provided by applicable law or treaty (including future time extensions), (iii) in any current or future medium and for any number of copies, and (iv) for any purpose whatsoever, including without limitation commercial, advertising or promotional purposes (the "License"). The License shall be deemed effective as of the date CC0 was applied by Affirmer to the Work. Should any part of the License for any reason be judged legally invalid or ineffective under applicable law, such partial invalidity or ineffectiveness shall not invalidate the remainder of the License, and in such case Affirmer hereby affirms that he or she will not (i) exercise any of his or her remaining Copyright and Related Rights in the Work or (ii) assert any associated claims and causes of action with respect to the Work, in either case contrary to Affirmer&apos;s express Statement of Purpose.
4. Limitations and Disclaimers.
a. No trademark or patent rights held by Affirmer are waived, abandoned,
surrendered, licensed or otherwise affected by this document.
b. Affirmer offers the Work as-is and makes no representations or warranties
of any kind concerning the Work, express, implied, statutory or otherwise,
including without limitation warranties of title, merchantability, fitness for
a particular purpose, non infringement, or the absence of latent or other
defects, accuracy, or the present or absence of errors, whether or not
discoverable, all to the greatest extent permissible under applicable law.
c. Affirmer disclaims responsibility for clearing rights of other persons that
may apply to the Work or any use thereof, including without limitation any
person&apos;s Copyright and Related Rights in the Work. Further, Affirmer
disclaims responsibility for obtaining any necessary consents, permissions or
other rights required for any use of the Work.
d. Affirmer understands and acknowledges that Creative Commons is not a party
to this document and has no duty or obligation with respect to this CC0 or use
of the Work.
+19
View File
@@ -0,0 +1,19 @@
# Xz
Package xz implements XZ decompression natively in Go.
Documentation at <https://pkg.go.dev/github.com/therootcompany/xz>.
Download and install with `go get github.com/xi2/xz`.
If you need compression as well as decompression, you might want to
look at <https://github.com/ulikunitz/xz>.
# LICENSE
This was originally released into the public domain by the AUTHORS.
Here it is licensed more explicitly as Creative Commons CC0 1.0 Universal
(CC0-1.0) so that it can be detected by automated tooling, and satisfy the legal
requirements for vendor integration for cases in which a "public domain"
statement is not sufficient.
+461
View File
@@ -0,0 +1,461 @@
/*
* Branch/Call/Jump (BCJ) filter decoders
*
* Authors: Lasse Collin <lasse.collin@tukaani.org>
* Igor Pavlov <http://7-zip.org/>
*
* Translation to Go: Michael Cross <https://github.com/xi2>
*
* This file has been put into the public domain.
* You can do whatever you want with this file.
*/
package xz
/* from linux/lib/xz/xz_dec_bcj.c *************************************/
type xzDecBCJ struct {
/* Type of the BCJ filter being used */
typ xzFilterID
/*
* Return value of the next filter in the chain. We need to preserve
* this information across calls, because we must not call the next
* filter anymore once it has returned xzStreamEnd
*/
ret xzRet
/*
* Absolute position relative to the beginning of the uncompressed
* data (in a single .xz Block).
*/
pos int
/* x86 filter state */
x86PrevMask uint32
/* Temporary space to hold the variables from xzBuf */
out []byte
outPos int
temp struct {
/* Amount of already filtered data in the beginning of buf */
filtered int
/*
* Buffer to hold a mix of filtered and unfiltered data. This
* needs to be big enough to hold Alignment + 2 * Look-ahead:
*
* Type Alignment Look-ahead
* x86 1 4
* PowerPC 4 0
* IA-64 16 0
* ARM 4 0
* ARM-Thumb 2 2
* SPARC 4 0
*/
buf []byte // slice buf will be backed by bufArray
bufArray [16]byte
}
}
/*
* This is used to test the most significant byte of a memory address
* in an x86 instruction.
*/
func bcjX86TestMSByte(b byte) bool {
return b == 0x00 || b == 0xff
}
func bcjX86Filter(s *xzDecBCJ, buf []byte) int {
var maskToAllowedStatus = []bool{
true, true, true, false, true, false, false, false,
}
var maskToBitNum = []byte{0, 1, 2, 2, 3, 3, 3, 3}
var i int
var prevPos int = -1
var prevMask uint32 = s.x86PrevMask
var src uint32
var dest uint32
var j uint32
var b byte
if len(buf) <= 4 {
return 0
}
for i = 0; i < len(buf)-4; i++ {
if buf[i]&0xfe != 0xe8 {
continue
}
prevPos = i - prevPos
if prevPos > 3 {
prevMask = 0
} else {
prevMask = (prevMask << (uint(prevPos) - 1)) & 7
if prevMask != 0 {
b = buf[i+4-int(maskToBitNum[prevMask])]
if !maskToAllowedStatus[prevMask] || bcjX86TestMSByte(b) {
prevPos = i
prevMask = prevMask<<1 | 1
continue
}
}
}
prevPos = i
if bcjX86TestMSByte(buf[i+4]) {
src = getLE32(buf[i+1:])
for {
dest = src - uint32(s.pos+i+5)
if prevMask == 0 {
break
}
j = uint32(maskToBitNum[prevMask]) * 8
b = byte(dest >> (24 - j))
if !bcjX86TestMSByte(b) {
break
}
src = dest ^ (1<<(32-j) - 1)
}
dest &= 0x01FFFFFF
dest |= 0 - dest&0x01000000
putLE32(dest, buf[i+1:])
i += 4
} else {
prevMask = prevMask<<1 | 1
}
}
prevPos = i - prevPos
if prevPos > 3 {
s.x86PrevMask = 0
} else {
s.x86PrevMask = prevMask << (uint(prevPos) - 1)
}
return i
}
func bcjPowerPCFilter(s *xzDecBCJ, buf []byte) int {
var i int
var instr uint32
for i = 0; i+4 <= len(buf); i += 4 {
instr = getBE32(buf[i:])
if instr&0xFC000003 == 0x48000001 {
instr &= 0x03FFFFFC
instr -= uint32(s.pos + i)
instr &= 0x03FFFFFC
instr |= 0x48000001
putBE32(instr, buf[i:])
}
}
return i
}
var bcjIA64BranchTable = [...]byte{
0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0,
4, 4, 6, 6, 0, 0, 7, 7,
4, 4, 0, 0, 4, 4, 0, 0,
}
func bcjIA64Filter(s *xzDecBCJ, buf []byte) int {
var branchTable = bcjIA64BranchTable[:]
/*
* The local variables take a little bit stack space, but it's less
* than what LZMA2 decoder takes, so it doesn't make sense to reduce
* stack usage here without doing that for the LZMA2 decoder too.
*/
/* Loop counters */
var i int
var j int
/* Instruction slot (0, 1, or 2) in the 128-bit instruction word */
var slot uint32
/* Bitwise offset of the instruction indicated by slot */
var bitPos uint32
/* bit_pos split into byte and bit parts */
var bytePos uint32
var bitRes uint32
/* Address part of an instruction */
var addr uint32
/* Mask used to detect which instructions to convert */
var mask uint32
/* 41-bit instruction stored somewhere in the lowest 48 bits */
var instr uint64
/* Instruction normalized with bit_res for easier manipulation */
var norm uint64
for i = 0; i+16 <= len(buf); i += 16 {
mask = uint32(branchTable[buf[i]&0x1f])
for slot, bitPos = 0, 5; slot < 3; slot, bitPos = slot+1, bitPos+41 {
if (mask>>slot)&1 == 0 {
continue
}
bytePos = bitPos >> 3
bitRes = bitPos & 7
instr = 0
for j = 0; j < 6; j++ {
instr |= uint64(buf[i+j+int(bytePos)]) << (8 * uint(j))
}
norm = instr >> bitRes
if (norm>>37)&0x0f == 0x05 && (norm>>9)&0x07 == 0 {
addr = uint32((norm >> 13) & 0x0fffff)
addr |= (uint32(norm>>36) & 1) << 20
addr <<= 4
addr -= uint32(s.pos + i)
addr >>= 4
norm &= ^(uint64(0x8fffff) << 13)
norm |= uint64(addr&0x0fffff) << 13
norm |= uint64(addr&0x100000) << (36 - 20)
instr &= 1<<bitRes - 1
instr |= norm << bitRes
for j = 0; j < 6; j++ {
buf[i+j+int(bytePos)] = byte(instr >> (8 * uint(j)))
}
}
}
}
return i
}
func bcjARMFilter(s *xzDecBCJ, buf []byte) int {
var i int
var addr uint32
for i = 0; i+4 <= len(buf); i += 4 {
if buf[i+3] == 0xeb {
addr = uint32(buf[i]) | uint32(buf[i+1])<<8 |
uint32(buf[i+2])<<16
addr <<= 2
addr -= uint32(s.pos + i + 8)
addr >>= 2
buf[i] = byte(addr)
buf[i+1] = byte(addr >> 8)
buf[i+2] = byte(addr >> 16)
}
}
return i
}
func bcjARMThumbFilter(s *xzDecBCJ, buf []byte) int {
var i int
var addr uint32
for i = 0; i+4 <= len(buf); i += 2 {
if buf[i+1]&0xf8 == 0xf0 && buf[i+3]&0xf8 == 0xf8 {
addr = uint32(buf[i+1]&0x07)<<19 |
uint32(buf[i])<<11 |
uint32(buf[i+3]&0x07)<<8 |
uint32(buf[i+2])
addr <<= 1
addr -= uint32(s.pos + i + 4)
addr >>= 1
buf[i+1] = byte(0xf0 | (addr>>19)&0x07)
buf[i] = byte(addr >> 11)
buf[i+3] = byte(0xf8 | (addr>>8)&0x07)
buf[i+2] = byte(addr)
i += 2
}
}
return i
}
func bcjSPARCFilter(s *xzDecBCJ, buf []byte) int {
var i int
var instr uint32
for i = 0; i+4 <= len(buf); i += 4 {
instr = getBE32(buf[i:])
if instr>>22 == 0x100 || instr>>22 == 0x1ff {
instr <<= 2
instr -= uint32(s.pos + i)
instr >>= 2
instr = (0x40000000 - instr&0x400000) |
0x40000000 | (instr & 0x3FFFFF)
putBE32(instr, buf[i:])
}
}
return i
}
/*
* Apply the selected BCJ filter. Update *pos and s.pos to match the amount
* of data that got filtered.
*/
func bcjApply(s *xzDecBCJ, buf []byte, pos *int) {
var filtered int
buf = buf[*pos:]
switch s.typ {
case idBCJX86:
filtered = bcjX86Filter(s, buf)
case idBCJPowerPC:
filtered = bcjPowerPCFilter(s, buf)
case idBCJIA64:
filtered = bcjIA64Filter(s, buf)
case idBCJARM:
filtered = bcjARMFilter(s, buf)
case idBCJARMThumb:
filtered = bcjARMThumbFilter(s, buf)
case idBCJSPARC:
filtered = bcjSPARCFilter(s, buf)
default:
/* Never reached */
}
*pos += filtered
s.pos += filtered
}
/*
* Flush pending filtered data from temp to the output buffer.
* Move the remaining mixture of possibly filtered and unfiltered
* data to the beginning of temp.
*/
func bcjFlush(s *xzDecBCJ, b *xzBuf) {
var copySize int
copySize = len(b.out) - b.outPos
if copySize > s.temp.filtered {
copySize = s.temp.filtered
}
copy(b.out[b.outPos:], s.temp.buf[:copySize])
b.outPos += copySize
s.temp.filtered -= copySize
copy(s.temp.buf, s.temp.buf[copySize:])
s.temp.buf = s.temp.buf[:len(s.temp.buf)-copySize]
}
/*
* Decode raw stream which has a BCJ filter as the first filter.
*
* The BCJ filter functions are primitive in sense that they process the
* data in chunks of 1-16 bytes. To hide this issue, this function does
* some buffering.
*/
func xzDecBCJRun(s *xzDecBCJ, b *xzBuf, chain func(*xzBuf) xzRet) xzRet {
var outStart int
/*
* Flush pending already filtered data to the output buffer. Return
* immediately if we couldn't flush everything, or if the next
* filter in the chain had already returned xzStreamEnd.
*/
if s.temp.filtered > 0 {
bcjFlush(s, b)
if s.temp.filtered > 0 {
return xzOK
}
if s.ret == xzStreamEnd {
return xzStreamEnd
}
}
/*
* If we have more output space than what is currently pending in
* temp, copy the unfiltered data from temp to the output buffer
* and try to fill the output buffer by decoding more data from the
* next filter in the chain. Apply the BCJ filter on the new data
* in the output buffer. If everything cannot be filtered, copy it
* to temp and rewind the output buffer position accordingly.
*
* This needs to be always run when len(temp.buf) == 0 to handle a special
* case where the output buffer is full and the next filter has no
* more output coming but hasn't returned xzStreamEnd yet.
*/
if len(s.temp.buf) < len(b.out)-b.outPos || len(s.temp.buf) == 0 {
outStart = b.outPos
copy(b.out[b.outPos:], s.temp.buf)
b.outPos += len(s.temp.buf)
s.ret = chain(b)
if s.ret != xzStreamEnd && s.ret != xzOK {
return s.ret
}
bcjApply(s, b.out[:b.outPos], &outStart)
/*
* As an exception, if the next filter returned xzStreamEnd,
* we can do that too, since the last few bytes that remain
* unfiltered are meant to remain unfiltered.
*/
if s.ret == xzStreamEnd {
return xzStreamEnd
}
s.temp.buf = s.temp.bufArray[:b.outPos-outStart]
b.outPos -= len(s.temp.buf)
copy(s.temp.buf, b.out[b.outPos:])
/*
* If there wasn't enough input to the next filter to fill
* the output buffer with unfiltered data, there's no point
* to try decoding more data to temp.
*/
if b.outPos+len(s.temp.buf) < len(b.out) {
return xzOK
}
}
/*
* We have unfiltered data in temp. If the output buffer isn't full
* yet, try to fill the temp buffer by decoding more data from the
* next filter. Apply the BCJ filter on temp. Then we hopefully can
* fill the actual output buffer by copying filtered data from temp.
* A mix of filtered and unfiltered data may be left in temp; it will
* be taken care on the next call to this function.
*/
if b.outPos < len(b.out) {
/* Make b.out temporarily point to s.temp. */
s.out = b.out
s.outPos = b.outPos
b.out = s.temp.bufArray[:]
b.outPos = len(s.temp.buf)
s.ret = chain(b)
s.temp.buf = s.temp.bufArray[:b.outPos]
b.out = s.out
b.outPos = s.outPos
if s.ret != xzOK && s.ret != xzStreamEnd {
return s.ret
}
bcjApply(s, s.temp.buf, &s.temp.filtered)
/*
* If the next filter returned xzStreamEnd, we mark that
* everything is filtered, since the last unfiltered bytes
* of the stream are meant to be left as is.
*/
if s.ret == xzStreamEnd {
s.temp.filtered = len(s.temp.buf)
}
bcjFlush(s, b)
if s.temp.filtered > 0 {
return xzOK
}
}
return s.ret
}
/*
* Allocate memory for BCJ decoders. xzDecBCJReset must be used before
* calling xzDecBCJRun.
*/
func xzDecBCJCreate() *xzDecBCJ {
return new(xzDecBCJ)
}
/*
* Decode the Filter ID of a BCJ filter and check the start offset is
* valid. Returns xzOK if the given Filter ID and offset is
* supported. Otherwise xzOptionsError is returned.
*/
func xzDecBCJReset(s *xzDecBCJ, id xzFilterID, offset int) xzRet {
switch id {
case idBCJX86:
case idBCJPowerPC:
case idBCJIA64:
case idBCJARM:
case idBCJARMThumb:
case idBCJSPARC:
default:
/* Unsupported Filter ID */
return xzOptionsError
}
// check offset is a multiple of alignment
switch id {
case idBCJPowerPC, idBCJARM, idBCJSPARC:
if offset%4 != 0 {
return xzOptionsError
}
case idBCJIA64:
if offset%16 != 0 {
return xzOptionsError
}
case idBCJARMThumb:
if offset%2 != 0 {
return xzOptionsError
}
}
s.typ = id
s.ret = xzOK
s.pos = offset
s.x86PrevMask = 0
s.temp.filtered = 0
s.temp.buf = nil
return xzOK
}
+55
View File
@@ -0,0 +1,55 @@
/*
* Delta decoder
*
* Author: Lasse Collin <lasse.collin@tukaani.org>
*
* Translation to Go: Michael Cross <https://github.com/xi2>
*
* This file has been put into the public domain.
* You can do whatever you want with this file.
*/
package xz
type xzDecDelta struct {
delta [256]byte
pos byte
distance int // in range [1, 256]
}
/*
* Decode raw stream which has a delta filter as the first filter.
*/
func xzDecDeltaRun(s *xzDecDelta, b *xzBuf, chain func(*xzBuf) xzRet) xzRet {
outStart := b.outPos
ret := chain(b)
for i := outStart; i < b.outPos; i++ {
tmp := b.out[i] + s.delta[byte(s.distance+int(s.pos))]
s.delta[s.pos] = tmp
b.out[i] = tmp
s.pos--
}
return ret
}
/*
* Allocate memory for a delta decoder. xzDecDeltaReset must be used
* before calling xzDecDeltaRun.
*/
func xzDecDeltaCreate() *xzDecDelta {
return new(xzDecDelta)
}
/*
* Returns xzOK if the given distance is valid. Otherwise
* xzOptionsError is returned.
*/
func xzDecDeltaReset(s *xzDecDelta, distance int) xzRet {
if distance < 1 || distance > 256 {
return xzOptionsError
}
s.delta = [256]byte{}
s.pos = 0
s.distance = distance
return xzOK
}
+1235
View File
File diff suppressed because it is too large Load Diff
+932
View File
@@ -0,0 +1,932 @@
/*
* .xz Stream decoder
*
* Author: Lasse Collin <lasse.collin@tukaani.org>
*
* Translation to Go: Michael Cross <https://github.com/xi2>
*
* This file has been put into the public domain.
* You can do whatever you want with this file.
*/
package xz
import (
"bytes"
"crypto/sha256"
"hash"
"hash/crc32"
"hash/crc64"
)
/* from linux/lib/xz/xz_stream.h **************************************/
/*
* See the .xz file format specification at
* http://tukaani.org/xz/xz-file-format.txt
* to understand the container format.
*/
const (
streamHeaderSize = 12
headerMagic = "\xfd7zXZ\x00"
footerMagic = "YZ"
)
/*
* Variable-length integer can hold a 63-bit unsigned integer or a special
* value indicating that the value is unknown.
*/
type vliType uint64
const (
vliUnknown vliType = ^vliType(0)
/* Maximum encoded size of a VLI */
vliBytesMax = 8 * 8 / 7 // (Sizeof(vliType) * 8 / 7)
)
/* from linux/lib/xz/xz_dec_stream.c **********************************/
/* Hash used to validate the Index field */
type xzDecHash struct {
unpadded vliType
uncompressed vliType
sha256 hash.Hash
}
// type of xzDec.sequence
type xzDecSeq int
const (
seqStreamHeader xzDecSeq = iota
seqBlockStart
seqBlockHeader
seqBlockUncompress
seqBlockPadding
seqBlockCheck
seqIndex
seqIndexPadding
seqIndexCRC32
seqStreamFooter
)
// type of xzDec.index.sequence
type xzDecIndexSeq int
const (
seqIndexCount xzDecIndexSeq = iota
seqIndexUnpadded
seqIndexUncompressed
)
/**
* xzDec - Opaque type to hold the XZ decoder state
*/
type xzDec struct {
/* Position in decMain */
sequence xzDecSeq
/* Position in variable-length integers and Check fields */
pos int
/* Variable-length integer decoded by decVLI */
vli vliType
/* Saved inPos and outPos */
inStart int
outStart int
/* CRC32 checksum hash used in Index */
crc32 hash.Hash
/* Hashes used in Blocks */
checkCRC32 hash.Hash
checkCRC64 hash.Hash
checkSHA256 hash.Hash
/* for checkTypes CRC32/CRC64/SHA256, check is one of the above 3 hashes */
check hash.Hash
/* Embedded stream header struct containing CheckType */
*Header
/*
* True if the next call to xzDecRun is allowed to return
* xzBufError.
*/
allowBufError bool
/* Information stored in Block Header */
blockHeader struct {
/*
* Value stored in the Compressed Size field, or
* vliUnknown if Compressed Size is not present.
*/
compressed vliType
/*
* Value stored in the Uncompressed Size field, or
* vliUnknown if Uncompressed Size is not present.
*/
uncompressed vliType
/* Size of the Block Header field */
size int
}
/* Information collected when decoding Blocks */
block struct {
/* Observed compressed size of the current Block */
compressed vliType
/* Observed uncompressed size of the current Block */
uncompressed vliType
/* Number of Blocks decoded so far */
count vliType
/*
* Hash calculated from the Block sizes. This is used to
* validate the Index field.
*/
hash xzDecHash
}
/* Variables needed when verifying the Index field */
index struct {
/* Position in decIndex */
sequence xzDecIndexSeq
/* Size of the Index in bytes */
size vliType
/* Number of Records (matches block.count in valid files) */
count vliType
/*
* Hash calculated from the Records (matches block.hash in
* valid files).
*/
hash xzDecHash
}
/*
* Temporary buffer needed to hold Stream Header, Block Header,
* and Stream Footer. The Block Header is the biggest (1 KiB)
* so we reserve space according to that. bufArray has to be aligned
* to a multiple of four bytes; the variables before it
* should guarantee this.
*/
temp struct {
pos int
buf []byte // slice buf will be backed by bufArray
bufArray [1024]byte
}
// chain is the function (or to be more precise, closure) which
// does the decompression and will call into the lzma2 and other
// filter code as needed. It is constructed by decBlockHeader
chain func(b *xzBuf) xzRet
// lzma2 holds the state of the last filter (which must be LZMA2)
lzma2 *xzDecLZMA2
// pointers to allocated BCJ/Delta filters
bcjs []*xzDecBCJ
deltas []*xzDecDelta
// number of currently in use BCJ/Delta filters from the above
bcjsUsed int
deltasUsed int
}
/* Sizes of the Check field with different Check IDs */
var checkSizes = [...]byte{
0,
4, 4, 4,
8, 8, 8,
16, 16, 16,
32, 32, 32,
64, 64, 64,
}
/*
* Fill s.temp by copying data starting from b.in[b.inPos]. Caller
* must have set s.temp.pos to indicate how much data we are supposed
* to copy into s.temp.buf. Return true once s.temp.pos has reached
* len(s.temp.buf).
*/
func fillTemp(s *xzDec, b *xzBuf) bool {
copySize := len(b.in) - b.inPos
tempRemaining := len(s.temp.buf) - s.temp.pos
if copySize > tempRemaining {
copySize = tempRemaining
}
copy(s.temp.buf[s.temp.pos:], b.in[b.inPos:])
b.inPos += copySize
s.temp.pos += copySize
if s.temp.pos == len(s.temp.buf) {
s.temp.pos = 0
return true
}
return false
}
/* Decode a variable-length integer (little-endian base-128 encoding) */
func decVLI(s *xzDec, in []byte, inPos *int) xzRet {
var byte byte
if s.pos == 0 {
s.vli = 0
}
for *inPos < len(in) {
byte = in[*inPos]
*inPos++
s.vli |= vliType(byte&0x7f) << uint(s.pos)
if byte&0x80 == 0 {
/* Don't allow non-minimal encodings. */
if byte == 0 && s.pos != 0 {
return xzDataError
}
s.pos = 0
return xzStreamEnd
}
s.pos += 7
if s.pos == 7*vliBytesMax {
return xzDataError
}
}
return xzOK
}
/*
* Decode the Compressed Data field from a Block. Update and validate
* the observed compressed and uncompressed sizes of the Block so that
* they don't exceed the values possibly stored in the Block Header
* (validation assumes that no integer overflow occurs, since vliType
* is uint64). Update s.check if presence of the CRC32/CRC64/SHA256
* field was indicated in Stream Header.
*
* Once the decoding is finished, validate that the observed sizes match
* the sizes possibly stored in the Block Header. Update the hash and
* Block count, which are later used to validate the Index field.
*/
func decBlock(s *xzDec, b *xzBuf) xzRet {
var ret xzRet
s.inStart = b.inPos
s.outStart = b.outPos
ret = s.chain(b)
s.block.compressed += vliType(b.inPos - s.inStart)
s.block.uncompressed += vliType(b.outPos - s.outStart)
/*
* There is no need to separately check for vliUnknown since
* the observed sizes are always smaller than vliUnknown.
*/
if s.block.compressed > s.blockHeader.compressed ||
s.block.uncompressed > s.blockHeader.uncompressed {
return xzDataError
}
switch s.CheckType {
case CheckCRC32, CheckCRC64, CheckSHA256:
_, _ = s.check.Write(b.out[s.outStart:b.outPos])
}
if ret == xzStreamEnd {
if s.blockHeader.compressed != vliUnknown &&
s.blockHeader.compressed != s.block.compressed {
return xzDataError
}
if s.blockHeader.uncompressed != vliUnknown &&
s.blockHeader.uncompressed != s.block.uncompressed {
return xzDataError
}
s.block.hash.unpadded +=
vliType(s.blockHeader.size) + s.block.compressed
s.block.hash.unpadded += vliType(checkSizes[s.CheckType])
s.block.hash.uncompressed += s.block.uncompressed
var buf [2 * 8]byte // 2*Sizeof(vliType)
putLE64(uint64(s.block.hash.unpadded), buf[:])
putLE64(uint64(s.block.hash.uncompressed), buf[8:])
_, _ = s.block.hash.sha256.Write(buf[:])
s.block.count++
}
return ret
}
/* Update the Index size and the CRC32 hash. */
func indexUpdate(s *xzDec, b *xzBuf) {
inUsed := b.inPos - s.inStart
s.index.size += vliType(inUsed)
_, _ = s.crc32.Write(b.in[s.inStart : s.inStart+inUsed])
}
/*
* Decode the Number of Records, Unpadded Size, and Uncompressed Size
* fields from the Index field. That is, Index Padding and CRC32 are not
* decoded by this function.
*
* This can return xzOK (more input needed), xzStreamEnd (everything
* successfully decoded), or xzDataError (input is corrupt).
*/
func decIndex(s *xzDec, b *xzBuf) xzRet {
var ret xzRet
for {
ret = decVLI(s, b.in, &b.inPos)
if ret != xzStreamEnd {
indexUpdate(s, b)
return ret
}
switch s.index.sequence {
case seqIndexCount:
s.index.count = s.vli
/*
* Validate that the Number of Records field
* indicates the same number of Records as
* there were Blocks in the Stream.
*/
if s.index.count != s.block.count {
return xzDataError
}
s.index.sequence = seqIndexUnpadded
case seqIndexUnpadded:
s.index.hash.unpadded += s.vli
s.index.sequence = seqIndexUncompressed
case seqIndexUncompressed:
s.index.hash.uncompressed += s.vli
var buf [2 * 8]byte // 2*Sizeof(vliType)
putLE64(uint64(s.index.hash.unpadded), buf[:])
putLE64(uint64(s.index.hash.uncompressed), buf[8:])
_, _ = s.index.hash.sha256.Write(buf[:])
s.index.count--
s.index.sequence = seqIndexUnpadded
}
if !(s.index.count > 0) {
break
}
}
return xzStreamEnd
}
/*
* Validate that the next 4 bytes match s.crc32.Sum(nil). s.pos must
* be zero when starting to validate the first byte.
*/
func crcValidate(s *xzDec, b *xzBuf) xzRet {
sum := s.crc32.Sum(nil)
// CRC32 - reverse slice
sum[0], sum[1], sum[2], sum[3] = sum[3], sum[2], sum[1], sum[0]
for {
if b.inPos == len(b.in) {
return xzOK
}
if sum[s.pos] != b.in[b.inPos] {
return xzDataError
}
b.inPos++
s.pos++
if !(s.pos < 4) {
break
}
}
s.crc32.Reset()
s.pos = 0
return xzStreamEnd
}
/*
* Validate that the next 4/8/32 bytes match s.check.Sum(nil). s.pos
* must be zero when starting to validate the first byte.
*/
func checkValidate(s *xzDec, b *xzBuf) xzRet {
sum := s.check.Sum(nil)
if s.CheckType == CheckCRC32 || s.CheckType == CheckCRC64 {
// CRC32/64 - reverse slice
for i, j := 0, len(sum)-1; i < j; i, j = i+1, j-1 {
sum[i], sum[j] = sum[j], sum[i]
}
}
for {
if b.inPos == len(b.in) {
return xzOK
}
if sum[s.pos] != b.in[b.inPos] {
return xzDataError
}
b.inPos++
s.pos++
if !(s.pos < len(sum)) {
break
}
}
s.check.Reset()
s.pos = 0
return xzStreamEnd
}
/*
* Skip over the Check field when the Check ID is not supported.
* Returns true once the whole Check field has been skipped over.
*/
func checkSkip(s *xzDec, b *xzBuf) bool {
for s.pos < int(checkSizes[s.CheckType]) {
if b.inPos == len(b.in) {
return false
}
b.inPos++
s.pos++
}
s.pos = 0
return true
}
/* polynomial table used in decStreamHeader below */
var xzCRC64Table = crc64.MakeTable(crc64.ECMA)
/* Decode the Stream Header field (the first 12 bytes of the .xz Stream). */
func decStreamHeader(s *xzDec) xzRet {
if string(s.temp.buf[:len(headerMagic)]) != headerMagic {
return xzFormatError
}
if crc32.ChecksumIEEE(s.temp.buf[len(headerMagic):len(headerMagic)+2]) !=
getLE32(s.temp.buf[len(headerMagic)+2:]) {
return xzDataError
}
if s.temp.buf[len(headerMagic)] != 0 {
return xzOptionsError
}
/*
* Of integrity checks, we support none (Check ID = 0),
* CRC32 (Check ID = 1), CRC64 (Check ID = 4) and SHA256 (Check ID = 10)
* However, we will accept other check types too, but then the check
* won't be verified and a warning (xzUnsupportedCheck) will be given.
*/
s.CheckType = CheckID(s.temp.buf[len(headerMagic)+1])
if s.CheckType > checkMax {
return xzOptionsError
}
switch s.CheckType {
case CheckNone:
// CheckNone: no action needed
case CheckCRC32:
if s.checkCRC32 == nil {
s.checkCRC32 = crc32.NewIEEE()
} else {
s.checkCRC32.Reset()
}
s.check = s.checkCRC32
case CheckCRC64:
if s.checkCRC64 == nil {
s.checkCRC64 = crc64.New(xzCRC64Table)
} else {
s.checkCRC64.Reset()
}
s.check = s.checkCRC64
case CheckSHA256:
if s.checkSHA256 == nil {
s.checkSHA256 = sha256.New()
} else {
s.checkSHA256.Reset()
}
s.check = s.checkSHA256
default:
return xzUnsupportedCheck
}
return xzOK
}
/* Decode the Stream Footer field (the last 12 bytes of the .xz Stream) */
func decStreamFooter(s *xzDec) xzRet {
if string(s.temp.buf[10:10+len(footerMagic)]) != footerMagic {
return xzDataError
}
if crc32.ChecksumIEEE(s.temp.buf[4:10]) != getLE32(s.temp.buf) {
return xzDataError
}
/*
* Validate Backward Size. Note that we never added the size of the
* Index CRC32 field to s->index.size, thus we use s->index.size / 4
* instead of s->index.size / 4 - 1.
*/
if s.index.size>>2 != vliType(getLE32(s.temp.buf[4:])) {
return xzDataError
}
if s.temp.buf[8] != 0 || CheckID(s.temp.buf[9]) != s.CheckType {
return xzDataError
}
/*
* Use xzStreamEnd instead of xzOK to be more convenient
* for the caller.
*/
return xzStreamEnd
}
/* Decode the Block Header and initialize the filter chain. */
func decBlockHeader(s *xzDec) xzRet {
var ret xzRet
/*
* Validate the CRC32. We know that the temp buffer is at least
* eight bytes so this is safe.
*/
crc := getLE32(s.temp.buf[len(s.temp.buf)-4:])
s.temp.buf = s.temp.buf[:len(s.temp.buf)-4]
if crc32.ChecksumIEEE(s.temp.buf) != crc {
return xzDataError
}
s.temp.pos = 2
/*
* Catch unsupported Block Flags.
*/
if s.temp.buf[1]&0x3C != 0 {
return xzOptionsError
}
/* Compressed Size */
if s.temp.buf[1]&0x40 != 0 {
if decVLI(s, s.temp.buf, &s.temp.pos) != xzStreamEnd {
return xzDataError
}
if s.vli >= 1<<63-8 {
// the whole block must stay smaller than 2^63 bytes
// the block header cannot be smaller than 8 bytes
return xzDataError
}
if s.vli == 0 {
// compressed size must be non-zero
return xzDataError
}
s.blockHeader.compressed = s.vli
} else {
s.blockHeader.compressed = vliUnknown
}
/* Uncompressed Size */
if s.temp.buf[1]&0x80 != 0 {
if decVLI(s, s.temp.buf, &s.temp.pos) != xzStreamEnd {
return xzDataError
}
s.blockHeader.uncompressed = s.vli
} else {
s.blockHeader.uncompressed = vliUnknown
}
// get total number of filters (1-4)
filterTotal := int(s.temp.buf[1]&0x03) + 1
// slice to hold decoded filters
filterList := make([]struct {
id xzFilterID
props uint32
}, filterTotal)
// decode the non-last filters which cannot be LZMA2
for i := 0; i < filterTotal-1; i++ {
/* Valid Filter Flags always take at least two bytes. */
if len(s.temp.buf)-s.temp.pos < 2 {
return xzDataError
}
s.temp.pos += 2
switch id := xzFilterID(s.temp.buf[s.temp.pos-2]); id {
case idDelta:
// delta filter
if s.temp.buf[s.temp.pos-1] != 0x01 {
return xzOptionsError
}
/* Filter Properties contains distance - 1 */
if len(s.temp.buf)-s.temp.pos < 1 {
return xzDataError
}
props := uint32(s.temp.buf[s.temp.pos])
s.temp.pos++
filterList[i] = struct {
id xzFilterID
props uint32
}{id: id, props: props}
case idBCJX86, idBCJPowerPC, idBCJIA64,
idBCJARM, idBCJARMThumb, idBCJSPARC:
// bcj filter
var props uint32
switch s.temp.buf[s.temp.pos-1] {
case 0x00:
props = 0
case 0x04:
if len(s.temp.buf)-s.temp.pos < 4 {
return xzDataError
}
props = getLE32(s.temp.buf[s.temp.pos:])
s.temp.pos += 4
default:
return xzOptionsError
}
filterList[i] = struct {
id xzFilterID
props uint32
}{id: id, props: props}
default:
return xzOptionsError
}
}
/*
* decode the last filter which must be LZMA2
*/
if len(s.temp.buf)-s.temp.pos < 2 {
return xzDataError
}
/* Filter ID = LZMA2 */
if xzFilterID(s.temp.buf[s.temp.pos]) != idLZMA2 {
return xzOptionsError
}
s.temp.pos++
/* Size of Properties = 1-byte Filter Properties */
if s.temp.buf[s.temp.pos] != 0x01 {
return xzOptionsError
}
s.temp.pos++
/* Filter Properties contains LZMA2 dictionary size. */
if len(s.temp.buf)-s.temp.pos < 1 {
return xzDataError
}
props := uint32(s.temp.buf[s.temp.pos])
s.temp.pos++
filterList[filterTotal-1] = struct {
id xzFilterID
props uint32
}{id: idLZMA2, props: props}
/*
* Process the filter list and create s.chain, going from last
* filter (LZMA2) to first filter
*
* First, LZMA2.
*/
ret = xzDecLZMA2Reset(s.lzma2, byte(filterList[filterTotal-1].props))
if ret != xzOK {
return ret
}
s.chain = func(b *xzBuf) xzRet {
return xzDecLZMA2Run(s.lzma2, b)
}
/*
* Now the non-last filters
*/
for i := filterTotal - 2; i >= 0; i-- {
switch id := filterList[i].id; id {
case idDelta:
// delta filter
var delta *xzDecDelta
if s.deltasUsed < len(s.deltas) {
delta = s.deltas[s.deltasUsed]
} else {
delta = xzDecDeltaCreate()
s.deltas = append(s.deltas, delta)
}
s.deltasUsed++
ret = xzDecDeltaReset(delta, int(filterList[i].props)+1)
if ret != xzOK {
return ret
}
chain := s.chain
s.chain = func(b *xzBuf) xzRet {
return xzDecDeltaRun(delta, b, chain)
}
case idBCJX86, idBCJPowerPC, idBCJIA64,
idBCJARM, idBCJARMThumb, idBCJSPARC:
// bcj filter
var bcj *xzDecBCJ
if s.bcjsUsed < len(s.bcjs) {
bcj = s.bcjs[s.bcjsUsed]
} else {
bcj = xzDecBCJCreate()
s.bcjs = append(s.bcjs, bcj)
}
s.bcjsUsed++
ret = xzDecBCJReset(bcj, id, int(filterList[i].props))
if ret != xzOK {
return ret
}
chain := s.chain
s.chain = func(b *xzBuf) xzRet {
return xzDecBCJRun(bcj, b, chain)
}
}
}
/* The rest must be Header Padding. */
for s.temp.pos < len(s.temp.buf) {
if s.temp.buf[s.temp.pos] != 0x00 {
return xzOptionsError
}
s.temp.pos++
}
s.temp.pos = 0
s.block.compressed = 0
s.block.uncompressed = 0
return xzOK
}
func decMain(s *xzDec, b *xzBuf) xzRet {
var ret xzRet
/*
* Store the start position for the case when we are in the middle
* of the Index field.
*/
s.inStart = b.inPos
for {
switch s.sequence {
case seqStreamHeader:
/*
* Stream Header is copied to s.temp, and then
* decoded from there. This way if the caller
* gives us only little input at a time, we can
* still keep the Stream Header decoding code
* simple. Similar approach is used in many places
* in this file.
*/
if !fillTemp(s, b) {
return xzOK
}
/*
* If decStreamHeader returns
* xzUnsupportedCheck, it is still possible
* to continue decoding. Thus, update s.sequence
* before calling decStreamHeader.
*/
s.sequence = seqBlockStart
ret = decStreamHeader(s)
if ret != xzOK {
return ret
}
fallthrough
case seqBlockStart:
/* We need one byte of input to continue. */
if b.inPos == len(b.in) {
return xzOK
}
/* See if this is the beginning of the Index field. */
if b.in[b.inPos] == 0 {
s.inStart = b.inPos
b.inPos++
s.sequence = seqIndex
break
}
/*
* Calculate the size of the Block Header and
* prepare to decode it.
*/
s.blockHeader.size = (int(b.in[b.inPos]) + 1) * 4
s.temp.buf = s.temp.bufArray[:s.blockHeader.size]
s.temp.pos = 0
s.sequence = seqBlockHeader
fallthrough
case seqBlockHeader:
if !fillTemp(s, b) {
return xzOK
}
ret = decBlockHeader(s)
if ret != xzOK {
return ret
}
s.sequence = seqBlockUncompress
fallthrough
case seqBlockUncompress:
ret = decBlock(s, b)
if ret != xzStreamEnd {
return ret
}
s.sequence = seqBlockPadding
fallthrough
case seqBlockPadding:
/*
* Size of Compressed Data + Block Padding
* must be a multiple of four. We don't need
* s->block.compressed for anything else
* anymore, so we use it here to test the size
* of the Block Padding field.
*/
for s.block.compressed&3 != 0 {
if b.inPos == len(b.in) {
return xzOK
}
if b.in[b.inPos] != 0 {
return xzDataError
}
b.inPos++
s.block.compressed++
}
s.sequence = seqBlockCheck
fallthrough
case seqBlockCheck:
switch s.CheckType {
case CheckCRC32, CheckCRC64, CheckSHA256:
ret = checkValidate(s, b)
if ret != xzStreamEnd {
return ret
}
default:
if !checkSkip(s, b) {
return xzOK
}
}
s.sequence = seqBlockStart
case seqIndex:
ret = decIndex(s, b)
if ret != xzStreamEnd {
return ret
}
s.sequence = seqIndexPadding
fallthrough
case seqIndexPadding:
for (s.index.size+vliType(b.inPos-s.inStart))&3 != 0 {
if b.inPos == len(b.in) {
indexUpdate(s, b)
return xzOK
}
if b.in[b.inPos] != 0 {
return xzDataError
}
b.inPos++
}
/* Finish the CRC32 value and Index size. */
indexUpdate(s, b)
/* Compare the hashes to validate the Index field. */
if !bytes.Equal(
s.block.hash.sha256.Sum(nil), s.index.hash.sha256.Sum(nil)) {
return xzDataError
}
s.sequence = seqIndexCRC32
fallthrough
case seqIndexCRC32:
ret = crcValidate(s, b)
if ret != xzStreamEnd {
return ret
}
s.temp.buf = s.temp.bufArray[:streamHeaderSize]
s.sequence = seqStreamFooter
fallthrough
case seqStreamFooter:
if !fillTemp(s, b) {
return xzOK
}
return decStreamFooter(s)
}
}
/* Never reached */
}
/**
* xzDecRun - Run the XZ decoder
* @s: Decoder state allocated using xzDecInit
* @b: Input and output buffers
*
* See xzRet for details of return values.
*
* xzDecRun is a wrapper for decMain to handle some special cases.
*
* We must return xzBufError when it seems clear that we are not
* going to make any progress anymore. This is to prevent the caller
* from calling us infinitely when the input file is truncated or
* otherwise corrupt. Since zlib-style API allows that the caller
* fills the input buffer only when the decoder doesn't produce any
* new output, we have to be careful to avoid returning xzBufError
* too easily: xzBufError is returned only after the second
* consecutive call to xzDecRun that makes no progress.
*/
func xzDecRun(s *xzDec, b *xzBuf) xzRet {
inStart := b.inPos
outStart := b.outPos
ret := decMain(s, b)
if ret == xzOK && inStart == b.inPos && outStart == b.outPos {
if s.allowBufError {
ret = xzBufError
}
s.allowBufError = true
} else {
s.allowBufError = false
}
return ret
}
/**
* xzDecInit - Allocate and initialize a XZ decoder state
* @dictMax: Maximum size of the LZMA2 dictionary (history buffer) for
* decoding. LZMA2 dictionary is always 2^n bytes
* or 2^n + 2^(n-1) bytes (the latter sizes are less common
* in practice), so other values for dictMax don't make sense.
*
* dictMax specifies the maximum allowed dictionary size that xzDecRun
* may allocate once it has parsed the dictionary size from the stream
* headers. This way excessive allocations can be avoided while still
* limiting the maximum memory usage to a sane value to prevent running the
* system out of memory when decompressing streams from untrusted sources.
*
* xzDecInit returns a pointer to an xzDec, which is ready to be used with
* xzDecRun.
*/
func xzDecInit(dictMax uint32, header *Header) *xzDec {
s := new(xzDec)
s.crc32 = crc32.NewIEEE()
s.Header = header
s.block.hash.sha256 = sha256.New()
s.index.hash.sha256 = sha256.New()
s.lzma2 = xzDecLZMA2Create(dictMax)
xzDecReset(s)
return s
}
/**
* xzDecReset - Reset an already allocated decoder state
* @s: Decoder state allocated using xzDecInit
*
* This function can be used to reset the decoder state without
* reallocating memory with xzDecInit.
*/
func xzDecReset(s *xzDec) {
s.sequence = seqStreamHeader
s.allowBufError = false
s.pos = 0
s.crc32.Reset()
s.check = nil
s.CheckType = checkUnset
s.block.compressed = 0
s.block.uncompressed = 0
s.block.count = 0
s.block.hash.unpadded = 0
s.block.hash.uncompressed = 0
s.block.hash.sha256.Reset()
s.index.sequence = seqIndexCount
s.index.size = 0
s.index.count = 0
s.index.hash.unpadded = 0
s.index.hash.uncompressed = 0
s.index.hash.sha256.Reset()
s.temp.pos = 0
s.temp.buf = s.temp.bufArray[:streamHeaderSize]
s.chain = nil
s.bcjsUsed = 0
s.deltasUsed = 0
}
+52
View File
@@ -0,0 +1,52 @@
/*
* XZ decompressor utility functions
*
* Author: Michael Cross <https://github.com/xi2>
*
* This file has been put into the public domain.
* You can do whatever you want with this file.
*/
package xz
func getLE32(buf []byte) uint32 {
return uint32(buf[0]) |
uint32(buf[1])<<8 |
uint32(buf[2])<<16 |
uint32(buf[3])<<24
}
func getBE32(buf []byte) uint32 {
return uint32(buf[0])<<24 |
uint32(buf[1])<<16 |
uint32(buf[2])<<8 |
uint32(buf[3])
}
func putLE32(val uint32, buf []byte) {
buf[0] = byte(val)
buf[1] = byte(val >> 8)
buf[2] = byte(val >> 16)
buf[3] = byte(val >> 24)
return
}
func putBE32(val uint32, buf []byte) {
buf[0] = byte(val >> 24)
buf[1] = byte(val >> 16)
buf[2] = byte(val >> 8)
buf[3] = byte(val)
return
}
func putLE64(val uint64, buf []byte) {
buf[0] = byte(val)
buf[1] = byte(val >> 8)
buf[2] = byte(val >> 16)
buf[3] = byte(val >> 24)
buf[4] = byte(val >> 32)
buf[5] = byte(val >> 40)
buf[6] = byte(val >> 48)
buf[7] = byte(val >> 56)
return
}
+124
View File
@@ -0,0 +1,124 @@
/*
* XZ decompressor
*
* Authors: Lasse Collin <lasse.collin@tukaani.org>
* Igor Pavlov <http://7-zip.org/>
*
* Translation to Go: Michael Cross <https://github.com/xi2>
*
* This file has been put into the public domain.
* You can do whatever you want with this file.
*/
package xz
/* from linux/include/linux/xz.h **************************************/
/**
* xzRet - Return codes
* @xzOK: Everything is OK so far. More input or more
* output space is required to continue.
* @xzStreamEnd: Operation finished successfully.
* @xzUnSupportedCheck: Integrity check type is not supported. Decoding
* is still possible by simply calling xzDecRun
* again.
* @xzMemlimitError: A bigger LZMA2 dictionary would be needed than
* allowed by the dictMax argument given to
* xzDecInit.
* @xzFormatError: File format was not recognized (wrong magic
* bytes).
* @xzOptionsError: This implementation doesn't support the requested
* compression options. In the decoder this means
* that the header CRC32 matches, but the header
* itself specifies something that we don't support.
* @xzDataError: Compressed data is corrupt.
* @xzBufError: Cannot make any progress.
*
* xzBufError is returned when two consecutive calls to XZ code cannot
* consume any input and cannot produce any new output. This happens
* when there is no new input available, or the output buffer is full
* while at least one output byte is still pending. Assuming your code
* is not buggy, you can get this error only when decoding a
* compressed stream that is truncated or otherwise corrupt.
*/
type xzRet int
const (
xzOK xzRet = iota
xzStreamEnd
xzUnsupportedCheck
xzMemlimitError
xzFormatError
xzOptionsError
xzDataError
xzBufError
)
/**
* xzBuf - Passing input and output buffers to XZ code
* @in: Input buffer.
* @inPos: Current position in the input buffer. This must not exceed
* input buffer size.
* @out: Output buffer.
* @outPos: Current position in the output buffer. This must not exceed
* output buffer size.
*
* Only the contents of the output buffer from out[outPos] onward, and
* the variables inPos and outPos are modified by the XZ code.
*/
type xzBuf struct {
in []byte
inPos int
out []byte
outPos int
}
/* All XZ filter IDs */
type xzFilterID int64
const (
idDelta xzFilterID = 0x03
idBCJX86 xzFilterID = 0x04
idBCJPowerPC xzFilterID = 0x05
idBCJIA64 xzFilterID = 0x06
idBCJARM xzFilterID = 0x07
idBCJARMThumb xzFilterID = 0x08
idBCJSPARC xzFilterID = 0x09
idLZMA2 xzFilterID = 0x21
)
// CheckID is the type of the data integrity check in an XZ stream
// calculated from the uncompressed data.
type CheckID int
func (id CheckID) String() string {
switch id {
case CheckNone:
return "None"
case CheckCRC32:
return "CRC32"
case CheckCRC64:
return "CRC64"
case CheckSHA256:
return "SHA256"
default:
return "Unknown"
}
}
const (
CheckNone CheckID = 0x00
CheckCRC32 CheckID = 0x01
CheckCRC64 CheckID = 0x04
CheckSHA256 CheckID = 0x0A
checkMax CheckID = 0x0F
checkUnset CheckID = -1
)
// An XZ stream contains a stream header which holds information about
// the stream. That information is exposed as fields of the
// Reader. Currently it contains only the stream's data integrity
// check type.
type Header struct {
CheckType CheckID // type of the stream's data integrity check
}
+35
View File
@@ -0,0 +1,35 @@
// Package xz implements XZ decompression natively in Go.
//
// Usage
//
// For ease of use, this package is designed to have a similar API to
// compress/gzip. See the examples for further details.
//
// Implementation
//
// This package is a translation from C to Go of XZ Embedded
// (http://tukaani.org/xz/embedded.html) with enhancements made so as
// to implement all mandatory and optional parts of the XZ file format
// specification v1.0.4. It supports all filters and block check
// types, supports multiple streams, and performs index verification
// using SHA-256 as recommended by the specification.
//
// Speed
//
// On the author's Intel Ivybridge i5, decompression speed is about
// half that of the standard XZ Utils (tested with a recent linux
// kernel tarball).
//
// Thanks
//
// Thanks are due to Lasse Collin and Igor Pavlov, the authors of XZ
// Embedded, on whose code package xz is based. It would not exist
// without their decision to allow others to modify and reuse their
// code.
//
// Bug reports
//
// For bug reports relating to this package please contact the author
// through https://github.com/xi2/xz/issues, and not the authors of XZ
// Embedded.
package xz
+256
View File
@@ -0,0 +1,256 @@
/*
* Package xz Go Reader API
*
* Author: Michael Cross <https://github.com/xi2>
*
* This file has been put into the public domain.
* You can do whatever you want with this file.
*/
package xz
import (
"errors"
"io"
)
// Package specific errors.
var (
ErrUnsupportedCheck = errors.New("xz: integrity check type not supported")
ErrMemlimit = errors.New("xz: LZMA2 dictionary size exceeds max")
ErrFormat = errors.New("xz: file format not recognized")
ErrOptions = errors.New("xz: compression options not supported")
ErrData = errors.New("xz: data is corrupt")
ErrBuf = errors.New("xz: data is truncated or corrupt")
)
// DefaultDictMax is the default maximum dictionary size in bytes used
// by the decoder. This value is sufficient to decompress files
// created with XZ Utils "xz -9".
const DefaultDictMax = 1 << 26 // 64 MiB
// inBufSize is the input buffer size used by the decoder.
const inBufSize = 1 << 13 // 8 KiB
// A Reader is an io.Reader that can be used to retrieve uncompressed
// data from an XZ file.
//
// In general, an XZ file can be a concatenation of other XZ
// files. Reads from the Reader return the concatenation of the
// uncompressed data of each.
type Reader struct {
Header
r io.Reader // the wrapped io.Reader
multistream bool // true if reader is in multistream mode
rEOF bool // true after io.EOF received on r
dEOF bool // true after decoder has completed
padding int // bytes of stream padding read (or -1)
in [inBufSize]byte // backing array for buf.in
buf *xzBuf // decoder input/output buffers
dec *xzDec // decoder state
err error // the result of the last decoder call
}
// NewReader creates a new Reader reading from r. The decompressor
// will use an LZMA2 dictionary size up to dictMax bytes in
// size. Passing a value of zero sets dictMax to DefaultDictMax. If
// an individual XZ stream requires a dictionary size greater than
// dictMax in order to decompress, Read will return ErrMemlimit.
//
// If NewReader is passed a value of nil for r then a Reader is
// created such that all read attempts will return io.EOF. This is
// useful if you just want to allocate memory for a Reader which will
// later be initialized with Reset.
//
// Due to internal buffering, the Reader may read more data than
// necessary from r.
func NewReader(r io.Reader, dictMax uint32) (*Reader, error) {
if dictMax == 0 {
dictMax = DefaultDictMax
}
z := &Reader{
r: r,
multistream: true,
padding: -1,
buf: &xzBuf{},
}
if r == nil {
z.rEOF, z.dEOF = true, true
}
z.dec = xzDecInit(dictMax, &z.Header)
var err error
if r != nil {
_, err = z.Read(nil) // read stream header
}
return z, err
}
// decode is a wrapper around xzDecRun that additionally handles
// stream padding. It treats the padding as a kind of stream that
// decodes to nothing.
//
// When decoding padding, z.padding >= 0
// When decoding a real stream, z.padding == -1
func (z *Reader) decode() (ret xzRet) {
if z.padding >= 0 {
// read all padding in input buffer
for z.buf.inPos < len(z.buf.in) &&
z.buf.in[z.buf.inPos] == 0 {
z.buf.inPos++
z.padding++
}
switch {
case z.buf.inPos == len(z.buf.in) && z.rEOF:
// case: out of padding. no more input data available
if z.padding%4 != 0 {
ret = xzDataError
} else {
ret = xzStreamEnd
}
case z.buf.inPos == len(z.buf.in):
// case: read more padding next loop iteration
ret = xzOK
default:
// case: out of padding. more input data available
if z.padding%4 != 0 {
ret = xzDataError
} else {
xzDecReset(z.dec)
ret = xzStreamEnd
}
}
} else {
ret = xzDecRun(z.dec, z.buf)
}
return
}
func (z *Reader) Read(p []byte) (n int, err error) {
// restore err
err = z.err
// set decoder output buffer to p
z.buf.out = p
z.buf.outPos = 0
for {
// update n
n = z.buf.outPos
// if last call to decoder ended with an error, return that error
if err != nil {
break
}
// if decoder has finished, return with err == io.EOF
if z.dEOF {
err = io.EOF
break
}
// if p full, return with err == nil, unless we have not yet
// read the stream header with Read(nil)
if n == len(p) && z.CheckType != checkUnset {
break
}
// if needed, read more data from z.r
if z.buf.inPos == len(z.buf.in) && !z.rEOF {
rn, e := z.r.Read(z.in[:])
if e != nil && e != io.EOF {
// read error
err = e
break
}
if e == io.EOF {
z.rEOF = true
}
// set new input buffer in z.buf
z.buf.in = z.in[:rn]
z.buf.inPos = 0
}
// decode more data
ret := z.decode()
switch ret {
case xzOK:
// no action needed
case xzStreamEnd:
if z.padding >= 0 {
z.padding = -1
if !z.multistream || z.rEOF {
z.dEOF = true
}
} else {
z.padding = 0
}
case xzUnsupportedCheck:
err = ErrUnsupportedCheck
case xzMemlimitError:
err = ErrMemlimit
case xzFormatError:
err = ErrFormat
case xzOptionsError:
err = ErrOptions
case xzDataError:
err = ErrData
case xzBufError:
err = ErrBuf
}
// save err
z.err = err
}
return
}
// Multistream controls whether the reader is operating in multistream
// mode.
//
// If enabled (the default), the Reader expects the input to be a
// sequence of XZ streams, possibly interspersed with stream padding,
// which it reads one after another. The effect is that the
// concatenation of a sequence of XZ streams or XZ files is
// treated as equivalent to the compressed result of the concatenation
// of the sequence. This is standard behaviour for XZ readers.
//
// Calling Multistream(false) disables this behaviour; disabling the
// behaviour can be useful when reading file formats that distinguish
// individual XZ streams. In this mode, when the Reader reaches the
// end of the stream, Read returns io.EOF. To start the next stream,
// call z.Reset(nil) followed by z.Multistream(false). If there is no
// next stream, z.Reset(nil) will return io.EOF.
func (z *Reader) Multistream(ok bool) {
z.multistream = ok
}
// Reset, for non-nil values of io.Reader r, discards the Reader z's
// state and makes it equivalent to the result of its original state
// from NewReader, but reading from r instead. This permits reusing a
// Reader rather than allocating a new one.
//
// If you wish to leave r unchanged use z.Reset(nil). This keeps r
// unchanged and ensures internal buffering is preserved. If the
// Reader was at the end of a stream it is then ready to read any
// follow on streams. If there are no follow on streams z.Reset(nil)
// returns io.EOF. If the Reader was not at the end of a stream then
// z.Reset(nil) does nothing.
func (z *Reader) Reset(r io.Reader) error {
switch {
case r == nil:
z.multistream = true
if !z.dEOF {
return nil
}
if z.rEOF {
return io.EOF
}
z.dEOF = false
_, err := z.Read(nil) // read stream header
return err
default:
z.r = r
z.multistream = true
z.rEOF = false
z.dEOF = false
z.padding = -1
z.buf.in = nil
z.buf.inPos = 0
xzDecReset(z.dec)
z.err = nil
_, err := z.Read(nil) // read stream header
return err
}
}
+5 -2
View File
@@ -78,11 +78,14 @@ github.com/nats-io/nkeys
# github.com/nats-io/nuid v1.0.1
## explicit
github.com/nats-io/nuid
# github.com/novox/mesh-host v0.0.0 => git.novox.be/novox/mesh-host v0.0.0-20261011092230-20d5af9b2d5f
# github.com/novox/mesh-host v0.0.0 => git.novox.be/novox/mesh-host v0.0.0-20261012222214-7a41e120635b
## explicit; go 1.26.0
github.com/novox/mesh-host/internal/declaration
github.com/novox/mesh-host/rootsearch
github.com/novox/mesh-host/validate
# github.com/therootcompany/xz v1.0.1
## explicit; go 1.15
github.com/therootcompany/xz
# golang.org/x/crypto v0.57.0
## explicit; go 1.26.0
golang.org/x/crypto/acme
@@ -135,4 +138,4 @@ golang.org/x/text/width
# golang.org/x/time v0.15.0
## explicit; go 1.25.0
golang.org/x/time/rate
# github.com/novox/mesh-host => git.novox.be/novox/mesh-host v0.0.0-20261011092230-20d5af9b2d5f
# github.com/novox/mesh-host => git.novox.be/novox/mesh-host v0.0.0-20261012222214-7a41e120635b