Compare commits

...
18 Commits
Author SHA1 Message Date
jochen dd78357b9b Make pending assignments safe to race, settle them on a tick, and say only what was checked
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery-group group fix/assign-says-why-a-module-is-not-there ready: every member ready, and composed together they pass
mesh/delivery superseded: a newer head of the same pull request
Review of #150: a withdrawal could land between the look and the act, a
failed ask read as a build in flight, a request kept the wrong asker, a
build being registered read as not built, build "true" could ask a build
nothing waited on, and a status read changed state. Claim a row under the
machine's hold before making it, keep a request only once asked, settle on
the controller's own tick, raise an assignment not made as a condition
until it is answered, and tie each row to its machine.
2026-10-08 16:21:53 +02:00
jochen 596148d8e6 Say why assign finds no module, and keep an assignment pending on its build
A merge asks for a new module's build, and assign answered "no module of that
name" until the build registered it, which read as a module nobody registered
(hq issue 325). Keep every build request, tell a build in flight, a module
known and not built, and an unknown name apart, and make an assignment made
while the build runs when the build registers the module.
2026-10-08 16:03:08 +02:00
mesh-admin 056414bc06 Merge pull request 'Record the bases a build copies, keep them by the builds that stood on them, copy each image once (hq ADR 0257, issue 321)' (#144) from feat/a-mirror-is-recorded into main 2026-10-08 14:02:47 +00:00
mesh-admin 59620fdacb Merge pull request 'Excuse no wait for a move from a build not known, and test the tier path's at-once put-back (hq issue 318 review)' (#152) from fix/318-follow-up-2 into main 2026-10-08 14:02:22 +00:00
jochen a44dc01c65 Excuse no wait for a move from a build not known, and count a module put back only once there is one (hq issue 318 review)
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer delivery to the same trunk took over its walk
2026-10-08 15:49:07 +02:00
jochen c6e372896b Leave an eligible index for a confirmed collect instead of letting it go alone
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer delivery to the same trunk took over its walk
Let go of alone after a build, an index's platforms stayed for ever under a record that
said collected, so no later collect could reach them (re-review of #144).
2026-10-08 15:47:42 +02:00
jochen c5663aa18b Let platform manifests go only on a confirmed collect, and copy again what a sweep took
An unrecorded index or a copy in progress can name a platform the records do not see, so
only a person's collect, after its dry run, takes an index's platforms, and only once every
kept index of each repository it touches was read. A copy missing a platform is copied
again, and a copy a build holds again is no longer recorded as collected (review of #144).
2026-10-08 15:47:42 +02:00
jochen 9907df6530 Record the bases a build copies, keep them by the builds that stood on them, and copy each image once
A copied base was named only in what a build stood on, and nowhere when the build failed,
so the store's sweep could never let one go (hq issue 321). One repository per upstream
image stops each module asking the public registry for the same image again, and letting
an index go now takes its own platform manifests, which otherwise kept every byte. A
person can record the copies no record names through the new mirrors verb (hq ADR 0257).

The forge test fix is the same commit as on feat/plain-notifications: main fails without it.
2026-10-08 15:47:42 +02:00
mesh-admin 41d6019fa0 Merge pull request 'Let a block show a value its module keeps on the bus, and grant the bar the read (hq ADR 0260)' (#149) from feat/the-bar-takes-blocks into main 2026-10-08 13:46:14 +00:00
mesh-admin 58e143bbc0 Merge pull request 'Give every module of a failed send a verdict, and excuse only the wait a build's own send brought (hq issue 318 review)' (#146) from fix/318-follow-up into main 2026-10-08 13:45:01 +00:00
jochen d9588b4f13 Let a block show a value its module keeps on the bus, and grant the bar the read (hq ADR 0260)
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
Modules talk over the bus, and the power draw reached the bar through a file. A block
may now show state its contributor keeps, the contributor only its own; the holder on
the same machine is granted the read without naming the module, and the template sees
which machine it renders for.
2026-10-08 15:32:51 +02:00
mesh-admin 34611c8e38 Merge pull request 'Let a seat receive blocks as data its holder renders, placed where the machine has the hardware (hq ADR 0255)' (#143) from feat/the-bar-takes-blocks into main 2026-10-08 13:31:48 +00:00
jochen a63939160e Put a broken module back at once, and excuse a wait only for a move that added an account group (hq issue 318 review)
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer delivery to the same trunk took over its walk
2026-10-08 15:25:58 +02:00
jochen 7ad9dbcb5d Say a pending new login in the same words everywhere, without 'session' (issue 318 review) 2026-10-08 15:25:58 +02:00
jochen 363898ec8a Give every module of a failed send a verdict, and excuse only the wait a build's own send brought (hq issue 318 review) 2026-10-08 15:25:58 +02:00
jochen 8984c3437f Leave out a block its holder cannot render, and keep if-capability to known names on offered kinds (hq ADR 0255)
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer delivery to the same trunk took over its walk
A piece whose shows the holder's template does not know rendered as nothing and failed
the whole machine's declaration; it is now left out and named, for push, plan and the
merge gate. quote escapes DEL, which TOML refuses bare. An if-capability nothing
detects, or on a kind a holder depends on, would drop a piece silently, so both are
refused.
2026-10-08 15:20:59 +02:00
jochen 9766338368 Build every artifact the forge declares in the forge tests
The catalogue's forge gained an npm-registry bundle (hq ADR 0251 §4), so resolving it
against its code bundle alone failed three tests on main and on every pull request.
2026-10-08 15:20:59 +02:00
jochen b1acb3d9de Let a seat receive blocks as data its holder renders, placed where the machine has the hardware (hq ADR 0255)
A module adding a battery to the bar had to write i3status-rust's TOML, so a second
bar could not take its place. node-bar now receives a bar-neutral block: the
contributor says what it shows, the holder renders it with its own template, places
every bar and place once, and a contribution may name a capability the machine must
report. The block is offered: the power module runs on servers without a bar.
2026-10-08 15:20:59 +02:00
70 changed files with 6256 additions and 179 deletions
+2
View File
@@ -264,6 +264,8 @@ func answer(ctx context.Context, publisher builder.Publisher, on, workspace stri
request.Repository, request.Path, request.Ref, workspace, request.Held, npmrc,
forgeFrom(), say, request.Seats)
}
// What it copied into the store, whatever became of the build (novox/hq ADR 0257).
result.Mirrored = built.Mirrored
// Only a build the kill ended: the kill came before its work did. One that finished — built, or
// failed on its own — in the moment the kill arrived says what it did, and the kill is refused.
killed := false
+41
View File
@@ -7,8 +7,10 @@ import (
"slices"
"sort"
"strings"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
)
// The things the mesh can be asked to do, separated from how it was asked.
@@ -41,6 +43,11 @@ import (
// It costs a resolution per machine. Assignment is a person typing a command, and being told which
// machines this just blocked is worth more than the milliseconds.
func assign(ctx context.Context, open *stores, node string, modules ...string) (string, error) {
return assignWith(ctx, open, node, assignOptions{}, modules...)
}
// assignWith is assign asked with its options: build, for a module known and not built (novox/hq issue 325).
func assignWith(ctx context.Context, open *stores, node string, opts assignOptions, modules ...string) (string, error) {
if len(modules) == 0 {
return "", fmt.Errorf("assign %s names no module", node)
}
@@ -51,6 +58,29 @@ func assign(ctx context.Context, open *stores, node string, modules ...string) (
return "", err
}
defer release()
// **A module the catalogue does not hold is looked for further before it is refused** (novox/hq issue
// 325): a build in flight keeps the assignment pending, a module known and not built is said with how to
// build it, and only a name the mesh never heard of is refused as unknown. Several modules in one act
// are judged together (ADR 0207), so an act naming one not registered is not made in part.
if missing, err := notInCatalogue(ctx, open, modules); err != nil {
return "", err
} else if len(missing) > 0 {
if len(modules) == 1 {
return notRegistered(ctx, open, node, modules[0], opts)
}
var lines []string
for _, m := range missing {
where, err := whereIs(ctx, open.inventory, m, time.Now())
if err != nil {
return "", err
}
lines = append(lines, where.said)
}
return "", fmt.Errorf("%w: nothing was assigned, since modules assigned together are judged together "+
"(ADR 0207) and %s not registered:\n %s\n assign them together once each is registered, or each "+
"alone to keep it pending on its build", inventory.ErrNoSuchModule, strings.Join(missing, ", "),
strings.Join(lines, "\n "))
}
// **The one assignment refused for what the node lacks** (novox/hq ADR 0207). Everything else
// an assignment leaves unresolved is kept, because assignment is not an ordering; a module whose
// resources are applied through a seat nothing on the node holds is refused, because that order
@@ -200,6 +230,17 @@ func unassign(ctx context.Context, open *stores, node string, modules ...string)
for _, a := range assigned {
runs[a] = true
}
// A module only pending on the node (novox/hq issue 325) is withdrawn, when it is the act's one module:
// that is the undo of an assignment kept while its build runs.
if len(modules) == 1 && !runs[modules[0]] {
said, withdrawn, err := withdrawPending(ctx, open.inventory, node, modules[0])
if err != nil {
return "", err
}
if withdrawn {
return said, nil
}
}
for _, module := range modules {
if !runs[module] {
return "", fmt.Errorf("%s is not assigned to %s", module, node)
+59 -2
View File
@@ -160,6 +160,9 @@ func buildFrom(result link.BuildResult) inventory.Build {
for _, ref := range result.Against {
kept.Against = append(kept.Against, catalogue.Recorded(ref))
}
for _, ref := range result.Mirrored {
kept.Mirrored = append(kept.Mirrored, catalogue.Recorded(ref))
}
for _, r := range result.Read {
kept.Read = append(kept.Read, inventory.ReadRepository{Repository: r.Repository, Ref: r.Ref})
}
@@ -396,15 +399,43 @@ func buildBehind(ctx context.Context, wait time.Duration) error {
//
// Separated from the command so `--behind` can walk a list without a second path to the same act.
func buildOne(ctx context.Context, source buildSource, path, ref string, wait time.Duration) error {
_, err := buildOneAsked(ctx, source, path, ref, wait, false)
_, err := buildOneAsked(ctx, source, path, ref, wait, false, "build")
return err
}
// recordAsked keeps a build request once it is asked (novox/hq issue 325), with who asked it, in a store
// opened for the purpose: the asks reach here from processes that hold none.
func recordAsked(ctx context.Context, r inventory.BuildRequest) {
open, err := openStores(ctx)
if err != nil {
fmt.Fprintf(os.Stderr, "build %s is asked and not kept as a build request: %v\n", r.ID, err)
return
}
defer open.Close()
recordBuildRequest(ctx, open.inventory, r)
}
// markNotAsked says a kept build request was not handed over, or not waited for.
func markNotAsked(ctx context.Context, id string, why error) {
open, err := openStores(ctx)
if err != nil {
fmt.Fprintf(os.Stderr, "build %s could not be marked as not asked: %v\n", id, err)
return
}
defer open.Close()
if err := open.inventory.MarkNotAsked(ctx, id, why.Error()); err != nil {
fmt.Fprintf(os.Stderr, "build %s could not be marked as not asked: %v\n", id, err)
}
}
// buildOneAsked is buildOne answering the id it asked with — what a plan keeps to match the outcome
// by (novox/hq ADR 0219) — and, for an ask not waited for, optionally a dry run: built and looked
// at, never taken in (issue 240), which is what `replay` asks unless told to register.
//
// asker is who asked, for the build request kept once the ask is made (novox/hq issue 325); empty for an
// asker that keeps the request itself, with its own name, once it knows the ask was made.
func buildOneAsked(ctx context.Context, source buildSource, path, ref string, wait time.Duration,
dryRun bool) (string, error) {
dryRun bool, asker string) (string, error) {
if dryRun && wait != 0 {
return "", errors.New("a dry run waited for is `build --dry-run`")
}
@@ -459,6 +490,12 @@ func buildOneAsked(ctx context.Context, source buildSource, path, ref string, wa
}
defer ask.Close()
fmt.Printf(" of %s\n", seat)
// Kept once it is asked, so `assign` knows a module is coming while its build runs (novox/hq issue 325);
// never before, so an ask that failed never reads as a build in flight. A dry run registers nothing, and
// is not kept.
keep := !dryRun && asker != ""
asked := inventory.BuildRequest{ID: request.ID, Repository: source.Repository, Seat: source.Seat, Path: path,
Ref: ref, For: asker}
if wait == 0 {
// Asked and not waited for (novox/hq issue 176): the outcome is the role's event, and the
@@ -468,6 +505,9 @@ func buildOneAsked(ctx context.Context, source buildSource, path, ref string, wa
if err := ask.Ask(ctx, request); err != nil {
return "", err
}
if keep {
recordAsked(ctx, asked)
}
if dryRun {
fmt.Printf("asked as a dry run, not waited for: `builds --log %s` follows it as it runs; "+
"its outcome is not taken in\n", request.ID)
@@ -478,8 +518,16 @@ func buildOneAsked(ctx context.Context, source buildSource, path, ref string, wa
return request.ID, nil
}
// Waited for: kept while it is waited for, since that can take minutes, and marked when the hand-over or
// the wait failed — an outcome heard later is still the last word.
if keep {
recordAsked(ctx, asked)
}
result, err := ask.Submit(ctx, request, wait)
if err != nil {
if keep {
markNotAsked(ctx, request.ID, err)
}
return request.ID, err
}
@@ -489,6 +537,10 @@ func buildOneAsked(ctx context.Context, source buildSource, path, ref string, wa
}
defer open.Close()
manifest, kept, err := takeIn(ctx, open.inventory, result)
// The assignments pending on this build, made or ended (novox/hq issue 325).
for _, line := range settlePendingOnBuild(ctx, open, result, manifest.Module, err) {
fmt.Printf(" %s\n", line)
}
if err != nil {
return request.ID, err
}
@@ -768,6 +820,11 @@ type answers struct {
// handed to the operator; healsUnread why it could not be read.
heals *healsCount
healsUnread string
// pending is every assignment waiting for its module's build, and every one ended in the last day
// (novox/hq issue 325); pendingUnread why they could not be read. One waiting, or one ended other than
// applied, is not well: an assignment somebody made is not made yet, or will not be.
pending []inventory.PendingAssignment
pendingUnread string
}
// heldBy is every artifact this mesh has built, for a build that may need one as its base.
+50
View File
@@ -31,6 +31,11 @@ type sweepBounds struct {
most int
// budget is the longest it keeps its caller waiting.
budget time.Duration
// platforms is whether an index is let go of with its own platform manifests (novox/hq ADR 0257).
// Only a sweep a person confirmed, after its dry run listed what stays and names a platform: the
// records cannot see an unrecorded index, or a copy in progress, naming the same platform, and the
// store's tools can.
platforms bool
}
// afterBuild are the bounds of the sweep inside somebody's build.
@@ -54,6 +59,9 @@ type sweepResult struct {
LetGo []string
// Skipped is how many references the sweep will not address (novox/hq issue 226).
Skipped int
// Indexes is how many eligible indexes a sweep a person did not confirm left for one that is: an
// index goes only with its platform manifests (novox/hq ADR 0257 §4).
Indexes int
// Left is how many eligible references were not asked about this time.
Left int
// Bounded is whether it stopped at its bounds rather than at a refusal.
@@ -76,6 +84,29 @@ func sweep(ctx context.Context, inv *inventory.Inventory, store artifacts.Store,
// as builds come, and is two HEADs each once done. A kept archive that could not be held stops
// the sweep before it deletes anything: "everything kept is held" is the precondition the
// collector's safety rests on, and a store refusing a hold would refuse the deletes too.
// **An index goes with its platform manifests only when a person confirmed it** (novox/hq ADR
// 0257), and a kept index keeps its own: which platform manifests the kept indexes name is read
// from the store for every repository the sweep will touch, before the first delete. A single
// read that fails stops the sweep before it deletes anything, as a kept archive that cannot be
// held does. The sweep after a build leaves an eligible index for such a collect, below.
store.Spare = nil
if bounds.platforms {
if inv == nil {
r.Stopped = "no records to read which platform manifests a kept index names, so nothing was let go"
r.Left = len(references)
return r
}
images, err := inv.KeptImages(ctx)
if err == nil {
store.Spare, err = store.SpareKeptIndexes(within, images, references)
}
if err != nil {
r.Stopped = fmt.Sprintf("which platform manifests a kept index names could not be read, so nothing was let go: %v", err)
r.Left = len(references)
return r
}
}
wrote, missing, err := holdKept(within, store, kept)
r.HoldersWritten, r.Missing = wrote, missing
if err != nil {
@@ -96,6 +127,22 @@ func sweep(ctx context.Context, inv *inventory.Inventory, store artifacts.Store,
}
break
}
if !bounds.platforms {
// **An index waits for a confirmed collect** (novox/hq ADR 0257 §4). Let go of alone,
// its platform manifests would stay in the store and the record would say collected, so
// no later sweep would offer it again and they would stay for ever. Left eligible, the
// next collect a person confirms takes it with them.
index, err := store.IsIndex(within, reference)
if err != nil && !errors.Is(err, artifacts.Gone) && !errors.Is(err, artifacts.ErrNotOurs) {
r.Stopped = fmt.Sprintf("the artifact store could not say whether %s is an index, so nothing more was asked of it: %v", reference, err)
r.Left = len(references) - i
break
}
if index {
r.Indexes++
continue
}
}
err := store.LetGo(within, reference)
if err == nil || errors.Is(err, artifacts.Gone) {
// Gone is the outcome wanted, already true. Recorded so the next sweep does not ask
@@ -187,6 +234,9 @@ func collect(ctx context.Context, inv *inventory.Inventory) {
if r.Skipped > 0 {
fmt.Fprintf(os.Stderr, "%d artifact(s) the sweep will not address were skipped\n", r.Skipped)
}
if r.Indexes > 0 {
fmt.Fprintf(os.Stderr, "%d eligible index(es) wait for a collect a person confirms, which takes their platforms too\n", r.Indexes)
}
}
// holdKept holds every kept archive by its manifest, stopping at the first refusal by the store.
+116 -36
View File
@@ -119,6 +119,9 @@ type gateFacts struct {
healthErr error
// heldOn is, per "<module>@<machine>", the provider its findings are held under (ADR 0240 rule 5).
heldOn map[string]string
// groupsAdded is, per module, whether the move judged puts an account in a group its previous build did
// not (issue 318 review): the only move whose wait for a new login is excused.
groupsAdded map[string]bool
}
// gatherGateFacts reads what a judging needs, from the store, the bus and this controller's memory. A
@@ -430,6 +433,7 @@ func judgeMoves(ctx context.Context, open *stores, g *inventory.PlanGate, pairs
if err != nil {
return "", err
}
facts.groupsAdded = movesAddingGroups(ctx, open.inventory, g, pairs, shelf)
// **What is wrong with a machine itself is the machine's** (novox/hq issue 281): read once for each
// machine judged, apart from what is wrong with a module there, and never pinned on the module the
// gate happens to be kept on.
@@ -442,11 +446,12 @@ func judgeMoves(ctx context.Context, open *stores, g *inventory.PlanGate, pairs
words[node] = aboutTheMachine(node, moved, *g.Since, facts)
}
// **Each module is judged on its own** (novox/hq ADR 0254, issue 318): its reading is the worst of its
// machines', its passes are counted apart, and the send's verdict is still one — but a module that was
// healthy on its own for the passes the gate asks keeps a pass when another beside it fails.
// machines', its passes are counted apart, and the send's verdict is still one. **Every module of a send
// leaves it with a verdict** (issue 318 review): one healthy for the passes the gate asks, after the
// settle time, keeps a pass when another beside it fails; one found broken holds the send's verdict until
// the modules beside it have theirs, or the bound; and at the verdict, every module that did not pass is
// put back with what failed, so none is left on the machine unjudged for other walks to wait on.
worst, why := healthGood, ""
var failing []string
broken := map[string]bool{}
reading := map[string]health{}
waits := map[string]string{}
var modules []string
@@ -465,18 +470,21 @@ func judgeMoves(ctx context.Context, open *stores, g *inventory.PlanGate, pairs
if _, seen := reading[j.module]; !seen {
modules = append(modules, j.module)
}
if h == healthBroken && !slices.Contains(g.Broken, j.module) {
g.Broken = append(g.Broken, j.module)
if g.BrokenWhy == "" {
g.BrokenWhy = said
}
}
if slices.Contains(g.Broken, j.module) {
h = healthBroken // found broken once, broken for the rest of the judging
}
if h > reading[j.module] {
reading[j.module] = h
}
if h == healthPerson {
waits[j.module] = joinSaid(waits[j.module], said)
}
if h > healthPerson && !slices.Contains(failing, j.module) {
failing = append(failing, j.module)
}
if h == healthBroken {
broken[j.module] = true
}
if h > worst {
worst, why = h, said
} else if h == worst && h > healthPerson && why == "" {
@@ -486,54 +494,78 @@ func judgeMoves(ctx context.Context, open *stores, g *inventory.PlanGate, pairs
if g.Healthy == nil {
g.Healthy = map[string]int{}
}
if g.HealthyAt == nil {
g.HealthyAt = map[string]time.Time{}
}
g.Waits = nil
var failing []string
for _, m := range modules {
switch {
case reading[m] > healthPerson:
if reading[m] > healthPerson {
failing = append(failing, m)
g.Healthy[m] = 0
default:
delete(g.HealthyAt, m)
continue
}
// **A pass is counted only gateEvery after the one before** (issue 318 review): a send judged more
// often while another module beside it is not yet healthy counts no faster.
if at, counted := g.HealthyAt[m]; !counted || now.Sub(at) >= gateEvery {
g.Healthy[m]++
if w := waits[m]; w != "" {
if g.Waits == nil {
g.Waits = map[string]string{}
}
g.Waits[m] = w
g.HealthyAt[m] = now
}
if w := waits[m]; w != "" {
if g.Waits == nil {
g.Waits = map[string]string{}
}
g.Waits[m] = w
}
}
// passedAlone is every module that passed on its own while the send as a whole did not.
passedAlone := func() []string {
var out []string
if now.Sub(*g.Since) < gateSettle {
return nil
}
settled := now.Sub(*g.Since) >= gateSettle
passedAlone := func(m string) bool {
return settled && reading[m] <= healthPerson && g.Healthy[m] >= gatePasses
}
// fail decides the send failed: what passed on its own keeps its pass, everything else is put back.
fail := func(why string) {
g.Passing, g.Failing = nil, nil
for _, m := range modules {
if reading[m] <= healthPerson && g.Healthy[m] >= gatePasses {
out = append(out, m)
if passedAlone(m) {
g.Passing = append(g.Passing, m)
} else {
g.Failing = append(g.Failing, m)
}
}
return out
decide(g, inventory.GateFailed, why, now)
}
pastBound := now.Sub(*g.Since) > gateBound
switch {
case worst == healthBroken:
// What broke is put back; what was only not yet healthy beside it is too — they moved together.
g.Failing, g.Passing = failing, passedAlone()
decide(g, inventory.GateFailed, why, now)
var judging []string
for _, m := range modules {
if reading[m] != healthBroken && !passedAlone(m) {
judging = append(judging, m)
}
}
if len(judging) == 0 || pastBound {
fail(g.BrokenWhy)
break
}
// What broke fails the send, and is put back at once by the caller (putBackBroken); what is beside
// it is judged to its own verdict first, within the bound.
g.Passes, g.LastPass, g.Failing = 0, nil, failing
g.Last = fmt.Sprintf("%s; %s judged to its own verdict before the send's", g.BrokenWhy, strings.Join(judging, ", "))
case worst == healthWaiting:
// Waiting on a provider that is unhealthy: not a pass, and not a failure at the bound either —
// the provider's own condition says what is wrong (ADR 0240 rule 5).
g.Passes, g.LastPass, g.Last, g.Failing = 0, nil, why, failing
case worst == healthNotYet:
g.Passes, g.LastPass, g.Last, g.Failing = 0, nil, why, failing
if now.Sub(*g.Since) > gateBound {
g.Passing = passedAlone()
decide(g, inventory.GateFailed, fmt.Sprintf("not healthy within %s of its apply: %s", gateBound, why), now)
if pastBound {
fail(fmt.Sprintf("not healthy within %s of its apply: %s", gateBound, why))
}
default:
// Healthy, or waiting for a person (ADR 0254): a pass, the wait carried along in the verdict.
g.Passes++
g.LastPass, g.Last, g.Failing = &now, "", nil
if g.Passes >= gatePasses && now.Sub(*g.Since) >= gateSettle {
if g.Passes >= gatePasses && settled {
decide(g, inventory.GatePassed, fmt.Sprintf("healthy %d times over %s", g.Passes,
now.Sub(*g.Since).Round(time.Second))+waitsSaid(g.Waits), now)
}
@@ -541,6 +573,18 @@ func judgeMoves(ctx context.Context, open *stores, g *inventory.PlanGate, pairs
return g.Verdict, nil
}
// whyFor is a passing gate's why as one module's verdict says it: the send's, and that module's own wait
// for a person, never another's (issue 318 review).
func whyFor(g *inventory.PlanGate, module string) string {
why, _, _ := strings.Cut(g.Why, waitsPrefix)
if w := g.Waits[module]; w != "" {
why += waitsPrefix + w
}
return why
}
const waitsPrefix = "; and it waits for a person: "
// waitsSaid is the waits for a person a passing gate carries, as its verdict says them.
func waitsSaid(waits map[string]string) string {
if len(waits) == 0 {
@@ -555,7 +599,7 @@ func waitsSaid(waits map[string]string) string {
for _, m := range modules {
said = append(said, waits[m])
}
return "; and it waits for a person: " + strings.Join(said, "; ")
return waitsPrefix + strings.Join(said, "; ")
}
func joinSaid(a, b string) string {
@@ -568,6 +612,42 @@ func joinSaid(a, b string) string {
return a + "; " + b
}
// movesAddingGroups is, per module a gate judges, whether its move puts an account in a group the build it
// moved from did not: read from the builds' manifests. A module whose move is not known adds none.
func movesAddingGroups(ctx context.Context, inv *inventory.Inventory, g *inventory.PlanGate, pairs []judged,
shelf map[string]catalogue.Manifest) map[string]bool {
out := map[string]bool{}
for _, j := range pairs {
if _, done := out[j.module]; done {
continue
}
from, to := g.From, g.To
for _, c := range g.Carried {
if c.Module == j.module {
from, to = c.From, c.To
break
}
}
target, found, err := inv.ManifestAt(ctx, j.module, to)
if err != nil || !found {
target = shelf[j.module]
}
// What it moved from is not known — no build named (a plan's own module whose previous build was not
// recorded), or no manifest kept for it: no wait is excused, rather than one the move did not bring.
if from == "" {
out[j.module] = false
continue
}
before, had, err := inv.ManifestAt(ctx, j.module, from)
if err != nil || !had {
out[j.module] = false
continue
}
out[j.module] = addsAccountGroups(before, had, target)
}
return out
}
// decide sets a gate's verdict.
func decide(g *inventory.PlanGate, verdict, why string, now time.Time) {
g.Verdict, g.Why, g.JudgedAt = verdict, why, &now
@@ -579,7 +659,7 @@ func gatePassed(ctx context.Context, open *stores, p *inventory.Plan, module str
g := state.Gate
err := open.inventory.RecordGate(ctx, inventory.GateVerdict{Build: state.Build, Module: module,
Commit: state.Commit, Previous: state.Previous, Plan: p.ID, Machines: g.Machines,
Verdict: inventory.GatePassed, Why: g.Why, Component: g.Component, JudgingFrom: g.Since})
Verdict: inventory.GatePassed, Why: whyFor(g, module), Component: g.Component, JudgingFrom: g.Since})
if err != nil && state.Build != "" {
fmt.Printf("%s: %s passed its gate, and the verdict could not be kept: %v\n", p.ID, module, err)
}
+453
View File
@@ -0,0 +1,453 @@
package main
import (
"context"
"encoding/json"
"reflect"
"slices"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/lease"
"github.com/novox/mesh-controller/internal/link"
)
// What the review of issue 318's fix found (novox/hq ADR 0254): every module of a send leaves it with a
// verdict, a pass is counted no faster than the gate's spacing, a carried build's verdict carries only its own
// wait, and a provider waiting for a login says who waits on it.
// withGateBounds sets the gate's bounds for one test.
func withGateBounds(t *testing.T, settle, every, bound time.Duration) {
t.Helper()
wasSettle, wasEvery, wasBound := gateSettle, gateEvery, gateBound
gateSettle, gateEvery, gateBound = settle, every, bound
t.Cleanup(func() { gateSettle, gateEvery, gateBound = wasSettle, wasEvery, wasBound })
}
// factsOn is a judging's facts for one machine that applied its send: the node tools answer, and what it says
// of its modules' resources.
func factsOn(t *testing.T, machine string, since time.Time, rs ...inventory.ResourceHealth) func() gateFacts {
return func() gateFacts {
now := time.Now()
at := now
return gateFacts{now: now,
reports: map[string]inventory.Reported{machine: {Node: machine, Outcome: inventory.OutcomeApplied, At: &at, Current: true}},
engines: map[string]string{},
rolledBack: map[string][]lease.Rollback{},
served: map[string]served{machine: {runtime: true, tools: map[string]bool{}}},
health: map[string]inventory.NodeHealth{machine: {Node: machine, HeardAt: now, Resources: rs}},
}
}
}
func healthyContainer(module string) inventory.ResourceHealth {
return inventory.ResourceHealth{Module: module, Resource: module + ".web", Kind: "container", Target: module,
State: link.StateHealthy}
}
// **A fault decided before the settle time does not strand the module beside it** (review (a)): the node
// tools' witness put its build back at once, and the send waits for the healthy module's own verdict — a pass,
// counted over the gate's spacing — before it says its own. The broken one alone is put back.
func TestAFaultBeforeTheSettleTimeWaitsForTheModuleBesideIt(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
withGateBounds(t, 150*time.Millisecond, 20*time.Millisecond, 10*time.Second)
since := time.Now().Add(-time.Second)
base := factsOn(t, "laptop", since, healthyContainer("app"))
wasGather := gatherGateFacts
t.Cleanup(func() { gatherGateFacts = wasGather })
gatherGateFacts = func(context.Context, *stores, string) (gateFacts, error) {
f := base()
f.rolledBack["laptop"] = []lease.Rollback{{Component: lease.ComponentNodeTools, Outcome: lease.OutcomeRolledBack,
At: since.Add(100 * time.Millisecond), Why: "the node tools did not answer"}}
return f, nil
}
g := &inventory.PlanGate{Machines: []string{"laptop"}, Since: &since}
pairs := []judged{{module: broker_runtime, node: "laptop"}, {module: "app", node: "laptop"}}
judgings := 0
for deadline := time.Now().Add(5 * time.Second); g.Verdict == "" && time.Now().Before(deadline); {
if _, err := judgeMoves(ctx, open, g, pairs, time.Now()); err != nil {
t.Fatal(err)
}
judgings++
time.Sleep(30 * time.Millisecond)
}
if g.Verdict != inventory.GateFailed || judgings < gatePasses {
t.Fatalf("verdict %q after %d judging(s): %+v; want failed, after the module beside it was judged", g.Verdict,
judgings, g)
}
if !reflect.DeepEqual(g.Passing, []string{"app"}) || !reflect.DeepEqual(g.Failing, []string{broker_runtime}) {
t.Fatalf("passing %v, failing %v; want app kept and the node tools put back", g.Passing, g.Failing)
}
if !strings.Contains(g.Why, "witness") {
t.Errorf("the verdict does not say what broke: %q", g.Why)
}
}
// broker_runtime is the node tools' module name, a core component a witness judges.
const broker_runtime = "node-tools"
// **A pass is counted only the gate's spacing after the one before** (review (c)): a send judged every tick
// while a module beside it is not yet healthy counts the healthy one once.
func TestAPassIsCountedNoFasterThanTheGatesSpacing(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
withGateBounds(t, 0, time.Hour, time.Hour)
since := time.Now().Add(-time.Minute)
base := factsOn(t, "laptop", since, healthyContainer("app"), inventory.ResourceHealth{Module: "late",
Resource: "late.web", Kind: "container", Target: "late", State: link.StateUnhealthy, Reason: "down"})
wasGather := gatherGateFacts
t.Cleanup(func() { gatherGateFacts = wasGather })
gatherGateFacts = func(context.Context, *stores, string) (gateFacts, error) { return base(), nil }
g := &inventory.PlanGate{Machines: []string{"laptop"}, Since: &since}
pairs := []judged{{module: "app", node: "laptop"}, {module: "late", node: "laptop"}}
now := time.Now()
for i := 0; i < 3; i++ {
if _, err := judgeMoves(ctx, open, g, pairs, now.Add(time.Duration(i)*time.Second)); err != nil {
t.Fatal(err)
}
}
if g.Healthy["app"] != 1 || g.Healthy["late"] != 0 {
t.Fatalf("counted %v in three judgings within a second; want app once and late never", g.Healthy)
}
}
// **A carried build's pass carries its own wait, never another's** (review (e)).
func TestACarriedPassCarriesOnlyItsOwnWait(t *testing.T) {
b := aBacklog(t)
ctx := t.Context()
inv := b.open.inventory
since := time.Now().Add(-time.Minute)
g := &inventory.PlanGate{Machines: []string{"laptop"}, Since: &since, Verdict: inventory.GatePassed,
Why: "healthy 3 times over 2m0s" + waitsSaid(map[string]string{"late": "relogin needed on laptop: late waits"}),
Waits: map[string]string{"late": "relogin needed on laptop: late waits"},
Carried: []inventory.CarriedMove{{Module: "app", Node: "laptop", From: "c1", To: "c2", Build: "build-app-c2"},
{Module: "late", Node: "laptop", From: "c1", To: "c2", Build: "build-late-c2"}}}
passCarried(ctx, b.open, &inventory.Plan{ID: "release-test"}, g, "")
app, _, _ := inv.GateOf(ctx, "build-app-c2")
late, _, _ := inv.GateOf(ctx, "build-late-c2")
if strings.Contains(app.Why, "waits for a person") || app.Verdict != inventory.GatePassed {
t.Errorf("app's pass: %+v; want it without late's wait", app)
}
if !strings.Contains(late.Why, "relogin needed on laptop") {
t.Errorf("late's pass: %+v; want its own wait", late)
}
}
// **The tier path keeps the pass of the module the gate is kept on** (review (b)): a plan's first send
// carried its own module, healthy, and a build waiting on that machine that never became healthy. At the
// bound the waiting one is put back and marked; the plan's own module keeps its pass, so no walk waits on it.
func TestThePlansOwnModuleKeepsItsPassWhenItsSendFails(t *testing.T) {
g := aGateMesh(t)
ctx := t.Context()
inv := g.open.inventory
// `late` waits on anchor for a gate: c1 sent, c2 registered and built.
for _, b := range []inventory.Build{
{ID: "build-late-1", Module: "late", Commit: "l1", Repository: "novox/mesh-catalog", Path: "modules/late",
Asked: time.Now().Add(-2 * time.Hour), At: time.Now().Add(-2 * time.Hour)},
{ID: "build-late-2", Module: "late", Commit: "l2", Repository: "novox/mesh-catalog", Path: "modules/late",
Asked: time.Now().Add(-time.Minute), At: time.Now().Add(-time.Minute)},
} {
manifest, _ := json.Marshal(catalogue.Manifest{Module: "late", Version: b.Commit})
b.Manifest = manifest
b.Made = []inventory.Artifact{{Name: "x", Kind: "bundle", Reference: "sha256:" + b.Commit}}
if err := inv.RecordBuild(ctx, b); err != nil {
t.Fatal(err)
}
if err := inv.RegisterModule(ctx, catalogue.Manifest{Module: "late", Version: b.Commit}, inventory.Source{
Repository: "novox/mesh-catalog", Seat: "git", Path: "modules/late", BuiltFrom: b.Commit, Head: b.Commit,
Asked: b.Asked}); err != nil {
t.Fatal(err)
}
if b.Commit == "l1" {
if _, err := inv.Assign(ctx, "anchor", "late"); err != nil {
t.Fatal(err)
}
if err := inv.RecordSent(ctx, nodeID(t, g.open, "anchor"), "d-anchor-late", map[string]string{"app": "c1",
"late": "l1"}); err != nil {
t.Fatal(err)
}
}
}
wasMoves := machineMoves
t.Cleanup(func() { machineMoves = wasMoves })
machineMoves = func(ctx context.Context, open *stores, f moveFacts, node string, all bool) ([]inventory.CarriedMove, error) {
modules, err := open.inventory.Assigned(ctx, node)
if err != nil {
return nil, err
}
sent, known, err := open.inventory.SentBuilds(ctx, node)
if err != nil {
return nil, err
}
return f.moves(node, modules, sent, known, all), nil
}
gather := gatherGateFacts
gatherGateFacts = func(ctx context.Context, open *stores, component string) (gateFacts, error) {
f, err := gather(ctx, open, component)
f.health = map[string]inventory.NodeHealth{"anchor": {Node: "anchor", HeardAt: time.Now(), Resources: []inventory.ResourceHealth{
healthyContainer("app"), {Module: "late", Resource: "late.web", Kind: "container", Target: "late",
State: link.StateUnhealthy, Reason: "down"}}}}
return f, err
}
gateEvery, gateBound = 0, 400*time.Millisecond
for deadline := time.Now().Add(5 * time.Second); time.Now().Before(deadline); {
advancePlans(ctx, g.open)
if p := g.plan(t); p.State == inventory.PlanFailed || p.State == inventory.PlanDone {
break
}
time.Sleep(30 * time.Millisecond)
}
p := g.plan(t)
if p.State != inventory.PlanFailed {
t.Fatalf("the plan is %s: %s; want it failed on late", p.State, p.Note)
}
if v, found, err := inv.GateOf(ctx, "build-2"); err != nil || !found || v.Verdict != inventory.GatePassed ||
!strings.Contains(v.Why, "on its own") {
t.Fatalf("app's verdict: %+v (found %v, %v); want its own pass kept", v, found, err)
}
if failed, _ := inv.GateFailed(ctx, "build-late-2"); !failed {
t.Fatal("late's build is not marked failed at its gate")
}
if current, _ := inv.CurrentBuilds(ctx); current["app"].Commit != "c2" || current["late"].Commit != "l1" {
t.Fatalf("registered app %s, late %s; want app kept at c2 and late put back to l1", current["app"].Commit,
current["late"].Commit)
}
}
// **A provider waiting for a login says who waits on it** (review (g)): its consumer, failing a check that
// needs it, is held under it, and the provider's relogin-needed condition lists it and is urgent.
func TestAProviderWaitingForALoginSaysWhoWaitsOnIt(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
k := conditionsFrom
register(t, open, catalogue.Manifest{Module: "db", Version: "1",
Provides: []catalogue.Offer{{Name: "postgres-database", Scope: catalogue.ScopeMesh}}})
register(t, open, catalogue.Manifest{Module: "shop", Version: "1", Requires: []string{"postgres-database"}})
for _, a := range [][2]string{{"anchor", "db"}, {"laptop", "shop"}} {
if _, err := inv.Assign(ctx, a[0], a[1]); err != nil {
t.Fatal(err)
}
}
if err := inv.RecordBindings(ctx, "laptop", []inventory.Binding{{Machine: "laptop", Consumer: "shop",
Provision: "postgres-database", Provider: catalogue.Chosen{Node: "anchor", Module: "db"}}}); err != nil {
t.Fatal(err)
}
at := h0
say := func(machine string, rs ...link.ResourceHealth) {
t.Helper()
at = at.Add(time.Second)
for i := range rs {
rs[i].Since = at
}
if err := stateHealth(ctx, inv, k, machine, link.Health{Contract: link.ReadinessContract, At: at, Resources: rs}, at); err != nil {
t.Fatal(err)
}
}
account := link.ResourceHealth{Module: "db", Resource: "db.operator", Kind: link.KindAccount, Target: "operator",
Account: "operator", State: link.StateUnhealthy, Reason: link.ReasonRelogin + ": operator is in the group db"}
unit := link.ResourceHealth{Module: "db", Resource: "db.server", Kind: link.KindUnit, Target: "db.service",
Account: "operator", State: link.StateUnhealthy, Reason: "failed in the account's own service manager (exit-code)"}
shop := link.ResourceHealth{Module: "shop", Resource: "shop.web", Kind: "container", Target: "shop",
State: link.StateUnhealthy, Reason: "http /health on web: answered 500", Check: "http", Needs: "postgres-database"}
for look := 0; look < 2; look++ {
say("anchor", account, unit)
say("laptop", shop)
}
list, err := k.Open(ctx)
if err != nil {
t.Fatal(err)
}
var keys []string
var c conditions.Condition
for _, x := range list {
keys = append(keys, x.Key)
if x.Key == "module.db.anchor.relogin-needed" {
c = x
}
}
if !reflect.DeepEqual(keys, []string{"module.db.anchor.relogin-needed"}) {
t.Fatalf("open %v; want the provider's wait alone, its consumer held under it", keys)
}
if c.Severity != conditions.Urgent || !strings.Contains(c.Evidence[0].Said, "shop on laptop") {
t.Fatalf("the provider's wait: %s, %q; want it urgent and naming its consumer", c.Severity, c.Evidence[0].Said)
}
}
// **A broken module is put back at once** (issue 318 review): the laptop's witness put the node tools back
// within a minute of a send that also moved `app`, and `app` reads not yet healthy because of it. The node
// tools are marked and put back in the very judging that found them broken — their registered build is the
// one before, and the laptop is sent it — while `app` goes on being judged, recovers, and keeps its own pass.
func TestABrokenModuleIsPutBackAtOnceAndTheOneBesideItGetsItsOwnVerdict(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
withConditionsInMemory(t)
was := doctorFrom
doctorFrom = nil
t.Cleanup(func() { doctorFrom = was })
old, recent := time.Now().Add(-3*time.Hour), time.Now().Add(-time.Minute)
build := func(module, commit string, asked time.Time) {
manifest, _ := json.Marshal(catalogue.Manifest{Module: module, Version: "1"})
if err := inv.RecordBuild(ctx, inventory.Build{ID: "build-" + module + "-" + commit, Module: module, Commit: commit,
Repository: "novox/mesh-catalog", Path: "modules/" + module, Manifest: manifest, Asked: asked, At: asked,
Made: []inventory.Artifact{{Name: "x", Kind: "bundle", Reference: "sha256:" + module + commit}}}); err != nil {
t.Fatal(err)
}
if err := inv.RegisterModule(ctx, catalogue.Manifest{Module: module, Version: "1"}, inventory.Source{
Repository: "novox/mesh-catalog", Seat: "git", Path: "modules/" + module, BuiltFrom: commit, Head: commit,
Asked: asked}); err != nil {
t.Fatal(err)
}
}
for _, m := range []string{"app", broker_runtime} {
build(m, "c1", old)
if _, err := inv.Assign(ctx, "laptop", m); err != nil {
t.Fatal(err)
}
}
if err := inv.RecordSent(ctx, nodeID(t, open, "laptop"), "d-laptop", map[string]string{"app": "c1", broker_runtime: "c1"}); err != nil {
t.Fatal(err)
}
build("app", "c2", recent)
build(broker_runtime, "c2", recent)
wasMoves, wasHeard, wasSend, wasGather := machineMoves, releaseHeard, sendRollout, gatherGateFacts
t.Cleanup(func() {
machineMoves, releaseHeard, sendRollout, gatherGateFacts = wasMoves, wasHeard, wasSend, wasGather
})
machineMoves = func(ctx context.Context, open *stores, f moveFacts, node string, all bool) ([]inventory.CarriedMove, error) {
modules, _ := open.inventory.Assigned(ctx, node)
sent, known, err := open.inventory.SentBuilds(ctx, node)
if err != nil {
return nil, err
}
return f.moves(node, modules, sent, known, all), nil
}
releaseHeard = func(context.Context, *stores) (map[string]bool, error) { return map[string]bool{"laptop": true}, nil }
var sends []string
n := 0
sendRollout = func(ctx context.Context, open *stores, names []string) ([]string, error) {
current, err := open.inventory.CurrentBuilds(ctx)
if err != nil {
return nil, err
}
for _, node := range names {
n++
carried := map[string]string{"app": current["app"].Commit, broker_runtime: current[broker_runtime].Commit}
sends = append(sends, node+":"+carried[broker_runtime])
digest := "d-" + node + "-" + time.Now().Format("150405.000000") + string(rune('a'+n))
if err := open.inventory.RecordSent(ctx, nodeID(t, open, node), digest, carried); err != nil {
return nil, err
}
if _, err := open.inventory.RecordDoing(ctx, nodeID(t, open, node), inventory.Doing{Node: node,
Outcome: inventory.OutcomeApplied, Declared: digest, Applied: 1, At: time.Now()}); err != nil {
return nil, err
}
}
return names, nil
}
var seen []string // the node tools' registered build at each judging
gatherGateFacts = func(ctx context.Context, open *stores, component string) (gateFacts, error) {
now := time.Now()
f := gateFacts{now: now, reports: map[string]inventory.Reported{}, engines: map[string]string{},
rolledBack: map[string][]lease.Rollback{}, served: map[string]served{}}
reports, _ := open.inventory.LastReports(ctx)
for _, r := range reports {
f.reports[r.Node] = r
}
current, _ := open.inventory.CurrentBuilds(ctx)
seen = append(seen, current[broker_runtime].Commit)
app := healthyContainer("app")
if current[broker_runtime].Commit == "c2" {
// The witness reverted the node tools; app cannot reach them yet.
f.rolledBack["laptop"] = []lease.Rollback{{Component: lease.ComponentNodeTools, Outcome: lease.OutcomeRolledBack,
From: "c2", To: "c1", At: now, Why: "the node tools did not answer"}}
app.State, app.Reason = link.StateUnhealthy, "down"
}
f.served["laptop"] = served{runtime: current[broker_runtime].Commit == "c1", tools: map[string]bool{}}
f.health = map[string]inventory.NodeHealth{"laptop": {Node: "laptop", HeardAt: now, Resources: []inventory.ResourceHealth{app}}}
return f, nil
}
withGateBounds(t, 100*time.Millisecond, 0, 10*time.Second)
release := func() inventory.Plan {
t.Helper()
plans, _ := inv.RecentPlans(ctx, 10)
for _, p := range plans {
if p.Release != nil {
return p
}
}
t.Fatal("no walk")
return inventory.Plan{}
}
// Judged until the first judging has found the node tools broken, and one more.
for i := 0; i < 20 && len(seen) < 2; i++ {
advancePlans(ctx, open)
}
if len(seen) < 2 || seen[0] != "c2" || seen[1] != "c1" {
t.Fatalf("the node tools' registered build at each judging: %v; want c2, then c1 at the very next judging", seen)
}
if failed, _ := inv.GateFailed(ctx, "build-"+broker_runtime+"-c2"); !failed {
t.Fatal("the node tools' build is not marked failed at once")
}
if p := release(); p.State != inventory.PlanRolling || p.Release.Gate == nil || p.Release.Gate.Verdict != "" {
t.Fatalf("the walk is %s with gate %+v; want it still judging app", p.State, p.Release.Gate)
}
if !slices.Contains(sends, "laptop:c1") {
t.Fatalf("sends %v; want the laptop sent the node tools' earlier build at once", sends)
}
for deadline := time.Now().Add(5 * time.Second); time.Now().Before(deadline); {
advancePlans(ctx, open)
if release().State != inventory.PlanRolling {
break
}
time.Sleep(20 * time.Millisecond)
}
p := release()
if p.State != inventory.PlanFailed {
t.Fatalf("the walk is %s: %s; want failed, for the node tools", p.State, p.Note)
}
if v, found, _ := inv.GateOf(ctx, "build-app-c2"); !found || v.Verdict != inventory.GatePassed || !strings.Contains(v.Why, "on its own") {
t.Fatalf("app's verdict: %+v; want its own pass", v)
}
if current, _ := inv.CurrentBuilds(ctx); current["app"].Commit != "c2" {
t.Fatalf("app is registered at %s; want it kept at c2", current["app"].Commit)
}
}
// **A move from a build not known excuses no wait** (issue 318 review): a plan's own module whose previous
// build was not recorded, or whose previous manifest is not kept, cannot be shown to have added the group.
func TestAMoveFromAnUnknownBuildExcusesNoWait(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
record := func(commit string, m catalogue.Manifest) {
raw, _ := json.Marshal(m)
at := time.Now().Add(-time.Hour)
if err := inv.RecordBuild(ctx, inventory.Build{ID: "build-lights-" + commit, Module: "lights", Commit: commit,
Repository: "novox/mesh-catalog", Path: "modules/lights", Manifest: raw, Asked: at, At: at}); err != nil {
t.Fatal(err)
}
}
record("c1", catalogue.Manifest{Module: "lights"})
record("c2", catalogue.Manifest{Module: "lights", Resources: []map[string]any{{"id": "operator", "type": "user",
"name": "operator", "groups": []any{"lights"}}}})
pairs := []judged{{module: "lights", node: "laptop"}}
shelf := map[string]catalogue.Manifest{}
for _, c := range []struct {
from string
want bool
}{{"c1", true}, {"", false}, {"c0-never-built", false}} {
g := &inventory.PlanGate{From: c.from, To: "c2"}
if got := movesAddingGroups(ctx, inv, g, pairs, shelf)["lights"]; got != c.want {
t.Errorf("a move from %q adds a group: %v; want %v", c.from, got, c.want)
}
}
}
+4
View File
@@ -86,6 +86,10 @@ var handActVerbs = []handActVerb{
// a moment the person chose (ADR 0251) — never a repair.
{Verb: "collect", Decision: "letting the store go of what the records keep for no reason, now rather " +
"than at the next build, is a person's word (ADR 0251)"},
// Recording a copy no record names as the mesh's: a person's reading of the store, never a repair
// (ADR 0257).
{Verb: "mirrors", Decision: "which copies in the store no record names are the mesh's is a person's " +
"word (ADR 0257)"},
{Verb: "bus upgrade", Decision: "the bus is never rolled by the mesh: replacing it is a planned step a " +
"person starts (ADR 0236)"},
{Verb: "upgrade release-backlog", Decision: "after a release plan failed, the next opens only when a " +
+11
View File
@@ -100,6 +100,8 @@ func run() error {
return collectCommand(ctx, args[1:])
case "images":
return imagesCommand(ctx, args[1:])
case "mirrors":
return mirrorsCommand(ctx, args[1:])
case "plans":
return plansCommand(ctx, args[1:])
case "delivery":
@@ -369,6 +371,15 @@ func (b builds) Built(ctx context.Context, result link.BuildResult) error {
return nil
}
manifest, _, err := takeIn(ctx, b.inv, result)
// The assignments pending on this build, made or ended (novox/hq issue 325), said in the daemon's log
// after what became of the build.
if b.open != nil {
defer func() {
for _, line := range settlePendingOnBuild(ctx, b.open, result, manifest.Module, err) {
fmt.Printf("%s: %s\n", result.ID, line)
}
}()
}
// When it was asked, so a plan takes as its outcome only a build asked for it or after it
// (novox/hq 04-ISSUES/219). Zero when the id does not say.
asked, _ := link.BuildAskedAt(result.ID)
+8
View File
@@ -135,6 +135,7 @@ type mergeComposed struct {
Problems []string `json:"problems,omitempty"`
Withheld []string `json:"withheld,omitempty"`
Unbound []string `json:"unbound,omitempty"`
Unplaced []string `json:"unplaced,omitempty"`
LeftOut map[string]string `json:"left-out,omitempty"`
Resources []string `json:"resources,omitempty"`
}
@@ -372,6 +373,12 @@ func judgeChange(ctx context.Context, in mergeCheckInput) (mergeVerdict, error)
v.Failures = append(v.Failures, fmt.Sprintf("%s: the change leaves a credential bound elsewhere — %s",
gm.Described, u))
}
// A contribution its holder's template renders nothing for (novox/hq ADR 0255): the machine
// would be sent without it, so a change that adds one is refused, naming it.
for _, u := range newOnly(gm.Change.Unplaced, gm.Base.Unplaced) {
v.Failures = append(v.Failures, fmt.Sprintf("%s: the change leaves a contribution unplaced — %s",
gm.Described, u))
}
for module, why := range gm.Change.LeftOut {
if _, was := gm.Base.LeftOut[module]; !was {
v.Failures = append(v.Failures, fmt.Sprintf("%s: the change leaves %s out of its declaration — %s",
@@ -780,6 +787,7 @@ func composeEveryMachine(ctx context.Context, in mergeCheckInput, shelf map[stri
for _, u := range declared.unbound {
c.Unbound = append(c.Unbound, u.String())
}
c.Unplaced = declared.unplaced
sort.Strings(c.Withheld)
sort.Strings(c.Unbound)
out[m.Name] = c
+237
View File
@@ -0,0 +1,237 @@
package main
import (
"context"
"encoding/json"
"errors"
"flag"
"fmt"
"os"
"regexp"
"strings"
"github.com/novox/mesh-controller/internal/artifacts"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
)
// The bases the mesh copied into its artifact store, and the copies no record names (novox/hq ADR 0257).
//
// A build that stands on an image published elsewhere copies it in first (ADR 0096, 0097). Since ADR 0257
// each copy is recorded where it is made, and a copy is kept while a kept build stood on it. Copies made
// before then by a build that failed — or before the records kept what a build stood on — are in the store
// with no record naming them, and ADR 0189 §3 keeps the sweep away from anything no record names. This
// verb is how a person says such a copy is the mesh's: it records it, and the sweep then decides as it
// does for every other copy. It deletes nothing.
// mirrorRepository is a repository only the mirror writes: one image's repository, or a module's own
// repository of a base from before ADR 0257.
var mirrorRepository = regexp.MustCompile(`^(upstream/[a-z0-9][a-z0-9._/-]*|[a-z0-9][a-z0-9._-]*/on-[a-z0-9_]+)$`)
// mirrorReference reads a reference a person gave into its recorded form, refusing anything that is not
// a manifest in a repository the mirror writes.
func mirrorReference(given string) (string, error) {
reference := catalogue.Recorded(strings.TrimSpace(given))
path, ours := catalogue.InArtifactStore(reference)
if !ours {
return "", fmt.Errorf("%q is not a reference into the artifact store (artifact-store://<repository>@sha256:<hex>)", given)
}
repository, digest, ok := strings.Cut(path, "@sha256:")
if !ok || len(digest) != 64 || strings.Trim(digest, "0123456789abcdef") != "" {
return "", fmt.Errorf("%q names no manifest by digest", given)
}
if !mirrorRepository.MatchString(repository) {
return "", fmt.Errorf("%q is in %s, which is not a repository the mirror writes "+
"(upstream/<host>/<path>, or <module>/on-<argument>)", given, repository)
}
return reference, nil
}
type mirrorEntry struct {
Reference string `json:"reference"`
State string `json:"state"`
Why []string `json:"why,omitempty"`
}
type mirrorsAnswer struct {
DryRun bool `json:"dry_run,omitempty"`
// Mirrors is every copy the records hold that is not yet let go of.
Mirrors []mirrorEntry `json:"mirrors"`
Counts struct {
Kept int `json:"kept"`
Eligible int `json:"eligible"`
Collected int `json:"collected"`
} `json:"counts"`
// Recorded, AlreadyRecorded and Refused answer a record: what was (or, a dry run, would be)
// recorded, what the records already held, and what was refused, with why.
Recorded []string `json:"recorded,omitempty"`
// Then says what recording does: a recorded copy is eligible, and the next sweep lets it go.
Then string `json:"then,omitempty"`
AlreadyRecorded []string `json:"already_recorded,omitempty"`
Refused map[string]string `json:"refused,omitempty"`
}
// mirrorsOf is the copies among the recorded states.
func mirrorsOf(states []inventory.ArtifactState, mirrored map[string]bool) mirrorsAnswer {
a := mirrorsAnswer{Mirrors: []mirrorEntry{}}
for _, s := range states {
if !mirrored[s.Reference] {
continue
}
switch s.State {
case inventory.ArtifactKept:
a.Counts.Kept++
case inventory.ArtifactEligible:
a.Counts.Eligible++
case inventory.ArtifactCollected:
a.Counts.Collected++
continue
}
a.Mirrors = append(a.Mirrors, mirrorEntry{Reference: s.Reference, State: s.State, Why: s.Why})
}
return a
}
// splitReferences reads references separated by spaces or commas.
func splitReferences(given string) []string {
return strings.FieldsFunc(given, func(r rune) bool { return r == ',' || r == ' ' || r == '\n' || r == '\t' })
}
// recordMirrors decides, for each reference given, whether it may be recorded: in a repository the
// mirror writes, not already recorded, and held by the store. A real run records those.
func recordMirrors(ctx context.Context, inv *inventory.Inventory, store artifacts.Store, given []string,
known map[string]bool, why string, real bool) (mirrorsAnswer, error) {
a := mirrorsAnswer{DryRun: !real, Mirrors: []mirrorEntry{}, Refused: map[string]string{}}
var record []string
seen := map[string]bool{}
for _, g := range given {
reference, err := mirrorReference(g)
if err != nil {
a.Refused[g] = err.Error()
continue
}
if seen[reference] {
continue
}
seen[reference] = true
if known[reference] {
a.AlreadyRecorded = append(a.AlreadyRecorded, reference)
continue
}
if store.Address == "" {
a.Refused[g] = "this mesh has no artifact store on its network to ask whether it holds this"
continue
}
held, err := store.HoldsManifest(ctx, reference)
switch {
case err != nil:
a.Refused[g] = err.Error()
continue
case !held:
a.Refused[g] = "the artifact store does not hold it"
continue
}
record = append(record, reference)
}
a.Recorded = record
if len(record) > 0 {
verb := "would become"
if real {
verb = "became"
}
a.Then = fmt.Sprintf("%d cop%s %s eligible: the next sweep (after any build, or collect) lets each go "+
"unless one of the five kept builds of a module the mesh holds stood on it", len(record),
map[bool]string{true: "y", false: "ies"}[len(record) == 1], verb)
}
if real && len(record) > 0 {
if err := inv.RecordMirrored(ctx, "", why, record); err != nil {
return a, fmt.Errorf("recording %d copies: %w", len(record), err)
}
}
return a, nil
}
func mirrorsCommand(ctx context.Context, args []string) error {
set := flag.NewFlagSet("mirrors", flag.ContinueOnError)
asJSON := set.Bool("json", false, "answer as JSON")
record := set.String("record", "", "references of copies no record names, separated by spaces or commas, to record as the mesh's")
confirm := set.Bool("confirm", false, "record them, rather than only say what would be recorded")
f := addHandActFlags(set)
positionals, err := parseAround(set, args)
if err != nil {
return err
}
if len(positionals) != 0 {
return errors.New("mirrors [--json] [--record <references> [--confirm --why <text>]]")
}
if *confirm {
if strings.TrimSpace(*record) == "" {
return errors.New("mirrors: --confirm records what --record names, and it names nothing. Nothing was done")
}
if err := f.require("mirrors"); err != nil {
return err
}
}
open, err := openStores(ctx)
if err != nil {
return err
}
defer open.Close()
inv := open.inventory
states, err := inv.Artifacts(ctx)
if err != nil {
return err
}
known, err := inv.Mirrored(ctx)
if err != nil {
return err
}
var answer mirrorsAnswer
if strings.TrimSpace(*record) == "" {
answer = mirrorsOf(states, known)
} else {
shelf, err := inv.Catalogue(ctx)
if err != nil {
return err
}
address, err := artifactStoreAddress(ctx, inv, shelf, "")
if err != nil {
return err
}
answer, err = recordMirrors(ctx, inv, artifacts.Store{Address: address}, splitReferences(*record), known,
strings.TrimSpace(*f.why), *confirm)
if err != nil {
return err
}
// The hand act is logged once the records say what it did, never before: an act that
// failed half-way is not logged as done.
if *confirm && len(answer.Recorded) > 0 {
f.record(ctx, "mirrors", append([]string{"--record"}, answer.Recorded...))
}
}
if *asJSON {
encoder := json.NewEncoder(os.Stdout)
encoder.SetIndent("", " ")
return encoder.Encode(answer)
}
if answer.DryRun && len(answer.Recorded) > 0 {
fmt.Println("a dry run: nothing was recorded (--confirm --why <text> to record)")
}
if answer.Then != "" {
fmt.Println(answer.Then)
}
for _, r := range answer.Recorded {
fmt.Printf(" record %s\n", r)
}
for _, r := range answer.AlreadyRecorded {
fmt.Printf(" already %s\n", r)
}
for g, why := range answer.Refused {
fmt.Printf(" refused %s: %s\n", g, why)
}
for _, m := range answer.Mirrors {
fmt.Printf(" %-9s %s %s\n", m.State, m.Reference, strings.Join(m.Why, ", "))
}
return nil
}
+298
View File
@@ -0,0 +1,298 @@
package main
import (
"fmt"
"net/http"
"net/http/httptest"
"slices"
"strings"
"sync"
"testing"
"time"
"github.com/novox/mesh-controller/internal/artifacts"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// The bases the mesh copied in, and recording a copy no record names (novox/hq ADR 0257).
func TestTheMirrorsVerbComposesItsCommandAndRefusesWhatItDoesNotTake(t *testing.T) {
r := ref("route-proxy", "on-go_base", 1)
for _, c := range []struct {
args map[string]any
want string
}{
{map[string]any{}, "mirrors --json"},
{map[string]any{"record": r}, "mirrors --json --record " + r},
{map[string]any{"record": r, "confirm": "true", "why": "copied before copies were recorded"},
"mirrors --json --record " + r + " --confirm --why copied before copies were recorded"},
} {
argv, err := argvFor("mirrors", c.args)
if err != nil || strings.Join(argv, " ") != c.want {
t.Errorf("mirrors %v: %q %v, want %q", c.args, argv, err, c.want)
}
}
for _, args := range []map[string]any{
{"record": r, "confirm": "true"}, // recorded without a why
{"record": r, "why": "x"}, // a why for a dry run
{"confirm": "true", "why": "x"}, // nothing to record
{"record": r, "confirm": "yes", "why": "x"}, // a switch is true or false
{"delete": "true"},
} {
if argv, err := argvFor("mirrors", args); err == nil {
t.Errorf("mirrors %v was composed as %q", args, argv)
}
}
if repairingCommand([]string{"mirrors", "--json", "--record", r, "--confirm", "--why", "x"}) != "mirrors" {
t.Error("recording a copy through the generic verb would go unrecorded")
}
if repairingCommand([]string{"mirrors", "--json", "--record", r}) != "" {
t.Error("a dry run was taken for an act by hand")
}
if !personsDecision(link.HandAct{Verb: "mirrors"}) {
t.Error("recording a copy would count toward a healer the mesh lacks")
}
}
func TestOnlyACopyInARepositoryTheMirrorWritesIsTaken(t *testing.T) {
for given, ok := range map[string]bool{
ref("route-proxy", "on-go_base", 1): true,
catalogue.ArtifactStoreScheme + "upstream/docker.io/library/golang@sha256:" + strings.Repeat("a", 64): true,
ref("route-proxy", "server", 1): false, // a module's own image
catalogue.ArtifactStoreScheme + "novox/invoicing-api@sha256:" + strings.Repeat("a", 64): false,
archiveRef("web", "on-tools", 1): false, // a blob
catalogue.ArtifactStoreScheme + "web/on-x@sha256:abc": false, // not a whole digest
"docker.io/library/golang@sha256:" + strings.Repeat("a", 64): false,
} {
if _, err := mirrorReference(given); (err == nil) != ok {
t.Errorf("%s: taken %v, want %v (%v)", given, err == nil, ok, err)
}
}
}
// heldStore answers a manifest HEAD with 200 for the digests it holds, and records every request.
func heldStore(t *testing.T, holds ...string) (artifacts.Store, *[]string) {
t.Helper()
var asked []string
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
asked = append(asked, r.Method+" "+r.URL.Path)
if r.Method == http.MethodHead && slices.ContainsFunc(holds, func(d string) bool { return strings.HasSuffix(r.URL.Path, d) }) {
w.WriteHeader(http.StatusOK)
return
}
w.WriteHeader(http.StatusNotFound)
}))
t.Cleanup(server.Close)
return artifacts.Store{Address: strings.TrimPrefix(server.URL, "http://")}, &asked
}
func TestRecordingACopyTakesWhatTheStoreHoldsAndDeletesNothing(t *testing.T) {
inv := inventory.ForTest(t)
held := ref("route-proxy", "on-go_base", 1)
absent := ref("route-proxy", "on-go_base", 2)
known := ref("nats", "on-nats_base", 3)
notACopy := ref("route-proxy", "server", 4)
if err := inv.RecordMirrored(t.Context(), "b1", "", []string{known}); err != nil {
t.Fatal(err)
}
store, asked := heldStore(t, fmt.Sprintf("%064x", 1))
given := []string{held, absent, known, notACopy}
mirrored, err := inv.Mirrored(t.Context())
if err != nil {
t.Fatal(err)
}
// A dry run says, and records nothing.
dry, err := recordMirrors(t.Context(), inv, store, given, mirrored, "", false)
if err != nil {
t.Fatal(err)
}
if !dry.DryRun || !slices.Equal(dry.Recorded, []string{held}) || !slices.Equal(dry.AlreadyRecorded, []string{known}) ||
len(dry.Refused) != 2 || dry.Refused[absent] == "" || dry.Refused[notACopy] == "" {
t.Fatalf("a dry run answered %+v", dry)
}
if !strings.Contains(dry.Then, "would become eligible") || !strings.Contains(dry.Then, "next sweep") {
t.Fatalf("a dry run did not say what recording does: %q", dry.Then)
}
if again, _ := inv.Mirrored(t.Context()); again[held] {
t.Fatal("a dry run recorded a copy")
}
// A real one records what the store holds, with the person's why, and the sweep may now decide.
real, err := recordMirrors(t.Context(), inv, store, given, mirrored, "copied before copies were recorded", true)
if err != nil {
t.Fatal(err)
}
if !slices.Equal(real.Recorded, []string{held}) {
t.Fatalf("recorded %v", real.Recorded)
}
left, err := inv.ToCollect(t.Context())
if err != nil {
t.Fatal(err)
}
if !slices.Contains(left, held) {
t.Fatalf("a recorded copy no kept build stood on is not offered to collect: %v", left)
}
for _, r := range *asked {
if !strings.HasPrefix(r, "HEAD ") {
t.Fatalf("recording a copy asked the store %s", r)
}
}
}
// indexRegistry holds indexes and platforms of one image's repository, answers GETs with their
// documents, refuses GETs for the digests in fail, and records every delete.
type indexRegistry struct {
mu sync.Mutex
indexes map[string][]string
held map[string]bool
fail map[string]bool
deleted []string
}
func (f *indexRegistry) serve(t *testing.T) artifacts.Store {
t.Helper()
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
f.mu.Lock()
defer f.mu.Unlock()
digest := r.URL.Path[strings.LastIndex(r.URL.Path, "/")+1:]
switch r.Method {
case http.MethodGet:
if f.fail[digest] {
w.WriteHeader(http.StatusInternalServerError)
return
}
if children, ok := f.indexes[digest]; ok && f.held[digest] {
var named []string
for _, c := range children {
named = append(named, `{"mediaType":"application/vnd.oci.image.manifest.v1+json","digest":"`+c+`"}`)
}
_, _ = w.Write([]byte(`{"schemaVersion":2,"mediaType":"application/vnd.oci.image.index.v1+json","manifests":[` +
strings.Join(named, ",") + `]}`))
return
}
if f.held[digest] {
_, _ = w.Write([]byte(`{"schemaVersion":2,"layers":[]}`))
return
}
w.WriteHeader(http.StatusNotFound)
case http.MethodHead:
if f.held[digest] {
w.WriteHeader(http.StatusOK)
return
}
w.WriteHeader(http.StatusNotFound)
case http.MethodDelete:
if !f.held[digest] {
w.WriteHeader(http.StatusNotFound)
return
}
delete(f.held, digest)
f.deleted = append(f.deleted, digest)
w.WriteHeader(http.StatusAccepted)
default:
w.WriteHeader(http.StatusBadRequest)
}
}))
t.Cleanup(server.Close)
return artifacts.Store{Address: strings.TrimPrefix(server.URL, "http://")}
}
func copyDigest(n int) string { return fmt.Sprintf("sha256:%064x", n) }
// twoCopies records, in one image's repository, a kept copy (stood on by a held module's build) naming
// platforms 11 and 12, and an eligible one (a failed build's) naming 12 and 13.
func twoCopies(t *testing.T, inv *inventory.Inventory) (kept, eligible string, reg *indexRegistry) {
t.Helper()
const repository = "upstream/docker.io/library/golang"
kept = catalogue.ArtifactStoreScheme + repository + "@" + copyDigest(1)
eligible = catalogue.ArtifactStoreScheme + repository + "@" + copyDigest(2)
if err := inv.RegisterModule(t.Context(), catalogue.Manifest{Module: "proxy", Version: "1"},
inventory.Source{Repository: "https://forge.invalid/proxy.git"}); err != nil {
t.Fatal(err)
}
failed := inventory.Build{ID: "f1", Repository: "https://forge.invalid/proxy.git", On: "a-build-machine",
Failed: "a recipe refused", Mirrored: []string{eligible}}
if err := inv.RecordBuild(t.Context(), failed); err != nil {
t.Fatal(err)
}
ok := inventory.Build{ID: "b1", Repository: "https://forge.invalid/proxy.git", Module: "proxy", On: "a-build-machine",
Commit: "c0ffee", Made: []inventory.Artifact{{Name: "app", Kind: "image", Reference: ref("proxy", "app", 7)}},
Against: []string{kept}, Mirrored: []string{kept}}
if err := inv.RecordBuild(t.Context(), ok); err != nil {
t.Fatal(err)
}
reg = &indexRegistry{
indexes: map[string][]string{copyDigest(1): {copyDigest(11), copyDigest(12)}, copyDigest(2): {copyDigest(12), copyDigest(13)}},
held: map[string]bool{copyDigest(1): true, copyDigest(2): true, copyDigest(11): true, copyDigest(12): true, copyDigest(13): true},
fail: map[string]bool{},
}
return kept, eligible, reg
}
// Through the records: a confirmed collect lets the eligible index go with the platform only it names,
// and leaves the platform the kept index names.
func TestAConfirmedCollectLetsAnIndexGoWithItsOwnPlatformsOnly(t *testing.T) {
inv := inventory.ForTest(t)
_, eligible, reg := twoCopies(t, inv)
store := reg.serve(t)
references, err := inv.ToCollect(t.Context())
if err != nil {
t.Fatal(err)
}
if !slices.Equal(references, []string{eligible}) {
t.Fatalf("offered %v, want the failed build's copy", references)
}
a := runCollect(t.Context(), inv, store, references, nil, true,
sweepBounds{most: 10, budget: 5 * time.Second, platforms: true})
if !slices.Equal(a.LetGo, []string{eligible}) || a.Stopped != "" {
t.Fatalf("let go %v, stopped %q", a.LetGo, a.Stopped)
}
if !slices.Equal(reg.deleted, []string{copyDigest(13), copyDigest(2)}) {
t.Fatalf("deleted %v; want its own platform, then the index", reg.deleted)
}
}
// The sweep after a build leaves an eligible index in place and eligible: let go of alone, its
// platforms would stay for ever under a record that says collected. A later collect a person confirms
// takes it with the platform only it names. An image the same sweep reaches is let go of as before.
func TestTheSweepAfterABuildLeavesAnIndexForAConfirmedCollect(t *testing.T) {
inv := inventory.ForTest(t)
_, eligible, reg := twoCopies(t, inv)
reg.held[copyDigest(5)] = true
image := catalogue.ArtifactStoreScheme + "upstream/docker.io/library/golang@" + copyDigest(5)
store := reg.serve(t)
r := sweep(t.Context(), inv, store, []string{eligible, image}, nil, afterBuild)
if r.Indexes != 1 || !slices.Equal(r.LetGo, []string{image}) || !slices.Equal(reg.deleted, []string{copyDigest(5)}) {
t.Fatalf("after a build: %d indexes left, let go %v, deleted %v; want the index left and the image gone",
r.Indexes, r.LetGo, reg.deleted)
}
left, err := inv.ToCollect(t.Context())
if err != nil {
t.Fatal(err)
}
if !slices.Equal(left, []string{eligible}) {
t.Fatalf("after the sweep the records offer %v; want the index still eligible", left)
}
a := runCollect(t.Context(), inv, store, left, nil, true,
sweepBounds{most: 10, budget: 5 * time.Second, platforms: true})
if !slices.Equal(a.LetGo, []string{eligible}) ||
!slices.Equal(reg.deleted, []string{copyDigest(5), copyDigest(13), copyDigest(2)}) {
t.Fatalf("a confirmed collect let go %v, deleted %v; want the index with its own platform", a.LetGo, reg.deleted)
}
}
// A kept index the store will not answer for stops a confirmed collect before its first delete.
func TestASpareListThatCannotBeReadStopsTheSweepBeforeAnyDelete(t *testing.T) {
inv := inventory.ForTest(t)
_, eligible, reg := twoCopies(t, inv)
reg.fail[copyDigest(1)] = true
store := reg.serve(t)
a := runCollect(t.Context(), inv, store, []string{eligible}, nil, true,
sweepBounds{most: 10, budget: 5 * time.Second, platforms: true})
if len(a.LetGo) != 0 || len(reg.deleted) != 0 || !strings.Contains(a.Stopped, "nothing was let go") {
t.Fatalf("let go %v, deleted %v, stopped %q", a.LetGo, reg.deleted, a.Stopped)
}
}
+154 -35
View File
@@ -152,14 +152,23 @@ func judgeModuleHealth(ctx context.Context, inv *inventory.Inventory, k *conditi
}
sort.Strings(modules)
seen := map[string]bool{}
// became is, per module, the kind of condition this statement says of it: what a standing one of the
// other kind turned into.
became := map[string]string{}
heldOn := map[string]string{}
providers := map[catalogue.Chosen]bool{}
for _, m := range modules {
// **A wait for a person's new login is said as that** (novox/hq ADR 0254): one plain sentence to the
// operator, never urgent, cleared on the first statement that no longer says it.
if said, waits := personWait(m, node, unhealthy[m]); waits {
o := reloginObservation(m, node, said, unhealthy[m])
o := reloginObservation(m, node, said, operatorOn(ctx, inv, node), unhealthy[m])
// **A provider waiting for a login says who waits on it** (novox/hq issue 318 review): its
// consumers are held under it, and its own condition is where they are said.
if hold != nil {
sayWaitingOn(&o, hold.waitersOn(catalogue.Chosen{Node: node, Module: m}))
}
seen[o.Key()] = true
became[m] = kindReloginNeeded
if _, isOpen := standing[o.Key()]; streaks[m] < moduleUnhealthyAfter && !isOpen {
continue
}
@@ -176,14 +185,10 @@ func judgeModuleHealth(ctx context.Context, inv *inventory.Inventory, k *conditi
providers[p] = true
continue
}
if waiters := hold.waitersOn(catalogue.Chosen{Node: node, Module: m}); len(waiters) > 0 {
o.Severity = conditions.Urgent
o.Said += "; " + waitingWords(waiters)
o.Summary += fmt.Sprintf("; %d consumer(s) wait on it", len(waiters))
o.Explanation += fmt.Sprintf(" %d module(s) that depend on it wait for it.", len(waiters))
}
sayWaitingOn(&o, hold.waitersOn(catalogue.Chosen{Node: node, Module: m}))
}
seen[o.Key()] = true
became[m] = kindModuleUnhealthy
c, isOpen := standing[o.Key()]
if streaks[m] < moduleUnhealthyAfter && !isOpen {
continue // unconfirmed: one statement can be wrong; `node show` lists it
@@ -207,7 +212,18 @@ func judgeModuleHealth(ctx context.Context, inv *inventory.Inventory, k *conditi
if on, held := heldOn[key]; held {
why = fmt.Sprintf("what %s finds on %s waits on %s, which is unhealthy: held under its condition", module, node, on)
}
if _, err := k.Clear(ctx, key, why); err != nil {
// **A condition that became the other kind** is not "working again" (issue 318 review): its clearing
// line says what it became.
resolved := ""
switch {
case c.Kind == kindModuleUnhealthy && became[module] == kindReloginNeeded:
why = fmt.Sprintf("%s on %s now waits only for a new login", module, node)
resolved = fmt.Sprintf("%s on %s now waits only for a new login", module, node)
case c.Kind == kindReloginNeeded && became[module] == kindModuleUnhealthy:
why = fmt.Sprintf("%s on %s no longer waits for a new login, and is not healthy", module, node)
resolved = fmt.Sprintf("The new login on %s is done, and %s still does not work", node, module)
}
if _, err := k.ClearSaying(ctx, key, why, resolved); err != nil {
problems = append(problems, err.Error())
}
}
@@ -229,7 +245,7 @@ func judgeModuleHealth(ctx context.Context, inv *inventory.Inventory, k *conditi
func sayWaiters(ctx context.Context, k *conditions.Keeper, hold *holding, p catalogue.Chosen, now time.Time) error {
var raisedAt *conditions.Condition
for i, c := range hold.open {
if c.Key == moduleUnhealthyKey(p.Module, p.Node) {
if c.Key == moduleUnhealthyKey(p.Module, p.Node) || c.Key == reloginKey(p.Module, p.Node) {
raisedAt = &hold.open[i]
}
}
@@ -246,35 +262,97 @@ func sayWaiters(ctx context.Context, k *conditions.Keeper, hold *holding, p cata
return nil
}
o := moduleUnhealthyObservation(p.Module, p.Node, rs)
if waiters := hold.waitersOn(p); len(waiters) > 0 {
o.Severity = conditions.Urgent
o.Said += "; " + waitingWords(waiters)
o.Summary += fmt.Sprintf("; %d consumer(s) wait on it", len(waiters))
if said, waits := personWait(p.Module, p.Node, rs); waits {
o = reloginObservation(p.Module, p.Node, said, operatorOn(ctx, hold.inv, p.Node), rs)
}
if o.Key() != raisedAt.Key {
return nil // its own statement says it next
}
sayWaitingOn(&o, hold.waitersOn(p))
_, err := k.Observe(ctx, o)
return err
}
// reloginKey is a module's relogin-needed condition on a machine.
func reloginKey(module, node string) string {
return conditions.Key(conditions.ScopeModule, module+"."+node, kindReloginNeeded)
}
// operatorOn is the operator's account on a machine, or "" when it is not known (to-be 29).
func operatorOn(ctx context.Context, inv *inventory.Inventory, node string) string {
if inv == nil {
return ""
}
n, err := inv.NodeByName(ctx, node)
if err != nil {
return ""
}
return n.Account
}
// reloginObservation is a module waiting for a person's new login on a machine (novox/hq ADR 0254): the
// operator's, a warning, its summary the one sentence that says what to do; the resources are evidence. Its
// plain words (ADR 0253) are the kind's: it needs the operator, and offers no answer — no verb can log a
// person in again, and a restart of the module's service would start it in the same session.
func reloginObservation(module, node, said string, rs []inventory.ResourceHealth) conditions.Observation {
// person in again, and a restart of the module's service would start it in the same session. operator is
// the machine's operator account, when known: the words say "your account" only for it.
func reloginObservation(module, node, said, operator string, rs []inventory.ResourceHealth) conditions.Observation {
o := moduleUnhealthyObservation(module, node, rs)
o.Token, o.Kind, o.Resolver, o.Summary = kindReloginNeeded, kindReloginNeeded, conditions.ResolverOperator, said
w := reloginWords(module, node)
accounts := waitingAccounts(module, rs)
yours := operator != "" && len(accounts) > 0
for _, a := range accounts {
yours = yours && a == operator
}
w := reloginWords(module, node, yours)
o.Headline, o.Explanation, o.Resolved, o.Needs, o.Actions = w.Headline, w.Explanation, w.Resolved, w.Needs, nil
return o
}
// reloginWords is what the operator reads of a module waiting for their new login on a machine (ADR 0253,
// ADR 0254): never quiet, since only the operator can do it, and no button, since nothing else can.
func reloginWords(module, node string) words {
// reloginWords is what the operator reads of a module waiting for a new login on a machine (ADR 0253,
// ADR 0254): never quiet, since only a person can do it, and no button, since nothing else can. yours says
// the account waiting is the operator's own on that machine; otherwise the words do not claim it is.
func reloginWords(module, node string, yours bool) words {
if yours {
return words{Headline: fmt.Sprintf("%s waits for a new login on %s", module, node),
Needs: fmt.Sprintf("log out of %s completely and log in again, or restart it.", node),
Explanation: fmt.Sprintf("%s put your account in a group it needs. You logged in before that, so %s "+
"cannot run until you log in again. Its update is in place and nothing was undone.",
conditions.Capital(module), module),
Resolved: fmt.Sprintf("%s runs on %s after your new login", module, node)}
}
return words{Headline: fmt.Sprintf("%s waits for a new login on %s", module, node),
Needs: fmt.Sprintf("log out of %s completely and log in again, or restart it.", node),
Explanation: fmt.Sprintf("%s put your account in a group it needs. You logged in before that, so %s "+
"cannot run until you log in again. Its update is in place and nothing was undone.", conditions.Capital(module), module),
Resolved: fmt.Sprintf("%s runs on %s after your new login", module, node)}
Needs: fmt.Sprintf("have the account it names log out of %s completely and log in again, or restart %s.", node, node),
Explanation: fmt.Sprintf("%s put an account on %s in a group it needs. That account logged in before that, "+
"so %s cannot run until it logs in again. Its update is in place and nothing was undone.",
conditions.Capital(module), node, module),
Resolved: fmt.Sprintf("%s runs on %s after the new login", module, node)}
}
// waitingAccounts is every account of a module whose resource says it waits for a new login, sorted.
func waitingAccounts(module string, rs []inventory.ResourceHealth) []string {
seen := map[string]bool{}
var out []string
for _, r := range rs {
if r.Module == module && r.Kind == link.KindAccount && r.State == link.StateUnhealthy &&
strings.HasPrefix(r.Reason, link.ReasonRelogin) && accountOf(r) != "" && !seen[accountOf(r)] {
seen[accountOf(r)] = true
out = append(out, accountOf(r))
}
}
sort.Strings(out)
return out
}
// sayWaitingOn adds to a module's condition the consumers held under it (to-be 48 §6): urgent while anyone
// waits on it, whether it is not working or waits for a new login.
func sayWaitingOn(o *conditions.Observation, waiters []string) {
if len(waiters) == 0 {
return
}
o.Severity = conditions.Urgent
o.Said += "; " + waitingWords(waiters)
o.Summary += fmt.Sprintf("; %d consumer(s) wait on it", len(waiters))
o.Explanation += fmt.Sprintf(" %d module(s) that depend on it wait for it.", len(waiters))
}
// moduleUnhealthyObservation is a module unhealthy on a machine, in words: the summary names the module,
@@ -310,6 +388,10 @@ func moduleUnhealthyObservation(module, node string, rs []inventory.ResourceHeal
// reasonWords is why a resource is unhealthy, as a person reads it.
func reasonWords(r inventory.ResourceHealth) string {
// An account waiting for a new login (ADR 0252) is said in the mesh's words, not the engine's.
if r.Kind == link.KindAccount && strings.HasPrefix(r.Reason, link.ReasonRelogin) {
return fmt.Sprintf("waits for a new login of %s, which is in the group and logged in before it was", accountOf(r))
}
switch r.Reason {
case "restarting":
return fmt.Sprintf("keeps restarting (%d restart(s) counted)", r.Restarts)
@@ -352,18 +434,14 @@ const kindReloginNeeded = "relogin-needed"
// not in its group or that could not be read — is not a wait, and the module is judged as before. A
// resource still starting is not unhealthy and does not make a wait either. Pure.
func personWait(module, machine string, rs []inventory.ResourceHealth) (string, bool) {
waiting := map[string]bool{}
var accounts []string
for _, r := range rs {
if r.Module == module && r.Kind == link.KindAccount && r.State == link.StateUnhealthy &&
strings.HasPrefix(r.Reason, link.ReasonRelogin) && accountOf(r) != "" && !waiting[accountOf(r)] {
waiting[accountOf(r)] = true
accounts = append(accounts, accountOf(r))
}
}
accounts := waitingAccounts(module, rs)
if len(accounts) == 0 {
return "", false
}
waiting := map[string]bool{}
for _, a := range accounts {
waiting[a] = true
}
var units []string
for _, r := range rs {
if r.Module != module || r.State != link.StateUnhealthy {
@@ -377,13 +455,12 @@ func personWait(module, machine string, rs []inventory.ResourceHealth) (string,
return "", false
}
}
sort.Strings(accounts)
said := fmt.Sprintf("relogin needed on %s: %s waits for a new login of %s, which is in its group and whose "+
"running session began before it was; log out of every session and in again, or reboot", machine, module,
"login began before it was; log out of %[1]s completely and log in again, or restart it", machine, module,
strings.Join(accounts, ", "))
if len(units) > 0 {
sort.Strings(units)
said += fmt.Sprintf(" (until then %s cannot run in that session)", strings.Join(units, ", "))
said += fmt.Sprintf(" (until then %s cannot run)", strings.Join(units, ", "))
}
return said, true
}
@@ -415,6 +492,13 @@ func moduleHealthWord(module, machine string, since time.Time, f gateFacts) (hea
return healthNotYet, fmt.Sprintf("%s has not said how what %s runs is since it was sent", machine, module)
}
wait, waits := personWait(module, machine, h.Resources)
// **Only a build whose own send put the account in a new group is excused** (issue 318 review): read from
// what the controller sent, never from when the machine says the wait began — that time is the engine's
// memory, reset by its restart and moved by a change of words. A build that adds no account group cannot
// have brought a wait, so a later build sent while the login is still owed is judged as before.
if waits && !f.groupsAdded[module] {
waits = false
}
for _, r := range h.Resources {
if r.Module != module {
continue
@@ -476,3 +560,38 @@ func healthLines(h inventory.NodeHealth, had bool, now time.Time) []string {
}
return out
}
// accountGroups is every account group a manifest declares, as "<account>/<group>": a user resource's
// groups (ADR 0252).
func accountGroups(m catalogue.Manifest) map[string]bool {
out := map[string]bool{}
for _, r := range m.Resources {
if t, _ := r["type"].(string); t != "user" {
continue
}
name, _ := r["name"].(string)
groups, _ := r["groups"].([]any)
for _, g := range groups {
if group, ok := g.(string); ok && group != "" {
out[name+"/"+group] = true
}
}
}
return out
}
// addsAccountGroups is whether a move from one manifest of a module to another puts an account in a group the
// earlier one did not (issue 318 review): the only send that can bring a wait for a new login. A module new to
// the machine (no earlier manifest) adds every group it declares.
func addsAccountGroups(from catalogue.Manifest, hadFrom bool, to catalogue.Manifest) bool {
before := map[string]bool{}
if hadFrom {
before = accountGroups(from)
}
for g := range accountGroups(to) {
if !before[g] {
return true
}
}
return false
}
+14 -1
View File
@@ -354,9 +354,20 @@ func moduleCommand(ctx context.Context, args []string) error {
func assignCommand(ctx context.Context, verb string, args []string) error {
// Several modules in one act (novox/hq ADR 0207): holders that depend on each other — the
// service manager and the package manager — can only go on, or come off, together.
set := flag.NewFlagSet(verb, flag.ContinueOnError)
// A module known and not built: ask for its build, and keep the assignment pending on it (novox/hq
// issue 325). Only assign reads it.
build := set.Bool("build", false, "for a module known and not built: ask for its build, and assign it when it registers")
args, err := parseAround(set, args)
if err != nil {
return err
}
if len(args) < 2 {
return fmt.Errorf("%s <node> <module> [<module>…]", verb)
}
if *build && verb == "unassign" {
return errors.New("unassign takes no --build")
}
open, err := openStores(ctx)
if err != nil {
return err
@@ -365,7 +376,9 @@ func assignCommand(ctx context.Context, verb string, args []string) error {
// The act itself is in acts.go, so the command API refuses exactly what this refuses
// (novox/hq ADR 0035). What differs between the surfaces is how the answer is printed.
act := assign
act := func(ctx context.Context, open *stores, node string, modules ...string) (string, error) {
return assignWith(ctx, open, node, assignOptions{Build: *build}, modules...)
}
if verb == "unassign" {
act = unassign
}
+695
View File
@@ -0,0 +1,695 @@
package main
import (
"context"
"errors"
"fmt"
"os"
"sort"
"strings"
"time"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// A module not registered yet, and the assignment that waits for it (novox/hq issue 325, ADR 0261).
//
// On 2026-10-08 the catalogue's pull request adding `sensors` merged; a minute later `assign g14 sensors`
// answered "no module of that name: sensors", and a few minutes later the same call worked. The merge had
// asked for the build (issue 300) and the build had not finished. The answer read as "you forgot to register
// it", and nothing in it said a build was on its way.
//
// So an assignment of a module the catalogue does not hold looks further before it refuses, and says which
// of three cases it is in:
//
// - **a build is in flight**: a build request for the module's directory has no outcome yet, or its build
// succeeded a moment ago and is being registered. The assignment is kept as a **pending assignment**,
// which the controller makes when the build registers the module (ADR 0261).
// - **known, not built**: the controller asked for the directory before, and the build failed, was not
// registered, said nothing within its bound, or could not be asked. Said, with `build` and assign's
// own build argument, which asks for the build and keeps the assignment pending on it.
// - **unknown**: no module registered and no build request kept by that name. Refused as before, with the
// closest names.
//
// **Pending by default while a build is in flight** (ADR 0261), without an argument to ask for it: an
// assignment is what a person meant and is kept even when its machine does not resolve (acts.go), and
// `unassign` withdraws it. Asking for a second act once the build lands is the two acts by hand issue 300
// removed. A build is never asked for by default: it runs on another machine, and a directory whose last
// build failed is a fault to look at before it is a thing to retry.
//
// **Settling is the controller's tick, never a read** (settlingPending): `status`, the board and the summary
// only read the rows.
// buildRequestBound is how long a build request may go without an outcome before it is no longer read as in
// flight, and a pending assignment waiting for it expires. Generous: a build waits behind others on the
// build seat, and a pending assignment that waits a little longer costs nothing.
const buildRequestBound = 2 * time.Hour
// registerGrace is how long a build heard as built is read as still in flight while it is not registered:
// the outcome is recorded first and registered after it, in the same take-in. Past it, the build was
// recorded and not registered, and says so.
const registerGrace = 5 * time.Minute
// pendingShownFor is how long an ended pending assignment is listed in `status`.
const pendingShownFor = 24 * time.Hour
// settleEvery is how often the controller settles the pending assignments.
const settleEvery = time.Minute
// claimStaleAfter is how long a pending assignment may be held as applying before the tick reads the claim
// as left by a controller that stopped, and settles it from what the mesh holds.
const claimStaleAfter = 5 * time.Minute
// kindPendingEnded is a pending assignment that ended without being made: expired or refused.
const kindPendingEnded = "assignment-not-made"
// assignOptions are what an assignment was asked with beside its machine and modules.
type assignOptions struct {
// Build asks for the build of a module known and not built, and keeps the assignment pending on it.
Build bool
}
// recordBuildRequest keeps a build request so `assign` can find it, once it is asked. Said, never fatal: the
// build was asked.
func recordBuildRequest(ctx context.Context, inv *inventory.Inventory, r inventory.BuildRequest) {
if err := inv.RecordBuildRequest(ctx, r); err != nil {
fmt.Fprintf(os.Stderr, "build %s was asked, and could not be kept as a build request, so `assign` "+
"will not know it is coming: %v\n", r.ID, err)
}
}
// The cases of a module the catalogue does not hold (novox/hq issue 325).
const (
unknownModule = iota
buildInFlight
knownNotBuilt
)
// notHeld is where a module the catalogue does not hold stands: its case, the build request it was read
// from, and the sentence that says it.
type notHeld struct {
kind int
request inventory.RequestOutcome
said string
}
// whereIs looks for a module the catalogue does not hold among the build requests the controller keeps.
func whereIs(ctx context.Context, inv *inventory.Inventory, module string, now time.Time) (notHeld, error) {
requests, err := inv.RequestsNamed(ctx, module)
if err != nil {
return notHeld{}, err
}
if r, ok := inFlight(requests, now); ok {
being := "being built"
if r.Heard {
being = "built and being registered"
}
return notHeld{kind: buildInFlight, request: r, said: fmt.Sprintf("%s is not registered yet: it is %s "+
"from %s since %s, as build %s; it can be assigned once that build registers it", module, being,
requestSource(r), clock(r.At), r.ID)}, nil
}
if len(requests) > 0 {
return notHeld{kind: knownNotBuilt, request: requests[0], said: fmt.Sprintf("%s is known and not "+
"registered: %s", module, whyNotBuilt(requests[0], now))}, nil
}
// Only what was looked at is claimed: the catalogue, and the build requests the controller keeps.
said := fmt.Sprintf("%v: %s — the catalogue holds no module of that name, and no build request the "+
"controller kept (the last 30 days) is for a directory of that name", inventory.ErrNoSuchModule, module)
if near := closestNames(ctx, inv, module); len(near) > 0 {
said += "; the closest names it holds: " + strings.Join(near, ", ")
}
return notHeld{kind: unknownModule, said: said}, nil
}
// notInCatalogue is the modules named that the catalogue does not hold.
func notInCatalogue(ctx context.Context, open *stores, modules []string) ([]string, error) {
shelf, err := open.inventory.Catalogue(ctx)
if err != nil {
return nil, err
}
var missing []string
for _, m := range modules {
if _, ok := shelf[m]; !ok {
missing = append(missing, m)
}
}
return missing, nil
}
// openPending is the open pending assignment of a module to a machine, if there is one.
func openPending(ctx context.Context, inv *inventory.Inventory, node, module string) (*inventory.PendingAssignment, error) {
rows, err := inv.PendingFor(ctx, node, module)
if err != nil {
return nil, err
}
for i := range rows {
if rows[i].Open() {
return &rows[i], nil
}
}
return nil, nil
}
// notRegistered answers an assignment of a module the catalogue does not hold: pending on a build in flight,
// known and not built, or unknown (novox/hq issue 325). An answer with no error is a pending assignment kept.
func notRegistered(ctx context.Context, open *stores, node, module string, opts assignOptions) (string, error) {
inv := open.inventory
if _, err := inv.NodeByName(ctx, node); err != nil {
return "", err
}
now := time.Now()
where, err := whereIs(ctx, inv, module, now)
if err != nil {
return "", err
}
waiting, err := openPending(ctx, inv, node, module)
if err != nil {
return "", err
}
switch where.kind {
case buildInFlight:
lead := where.said
if opts.Build {
lead += "\n no second build was asked: this one is running"
}
return keepPending(ctx, open, node, module, where.request, waiting, lead)
case knownNotBuilt:
last := where.request
if !opts.Build {
also := ""
if waiting != nil {
also = fmt.Sprintf("\n %s already has a pending assignment of %s, waiting for build %s; build "+
"\"true\" asks for a new build and makes it wait for that one", node, module, waiting.Build)
}
return "", fmt.Errorf("%w: %s%s\n assign it again with build \"true\" to ask for its build and keep this "+
"assignment until the build registers it; or `build` it (repository %s, path %s) and assign it "+
"once it is registered", inventory.ErrNoSuchModule, where.said, also, last.Repository, orRoot(last.Path))
}
if waiting != nil && waiting.State == inventory.PendingApplying {
return "", fmt.Errorf("%s is being assigned %s now; nothing was asked", node, module)
}
source := buildSource{Repository: last.Repository, Seat: last.Seat}
id, err := askABuild(ctx, source, last.Path, last.Ref)
if err != nil {
return "", fmt.Errorf("%s\n its build could not be asked for: %w; nothing was assigned", where.said, err)
}
asked := inventory.RequestOutcome{BuildRequest: inventory.BuildRequest{ID: id, Repository: last.Repository,
Seat: last.Seat, Path: last.Path, Ref: last.Ref, For: "assign", At: now.UTC()}}
recordBuildRequest(ctx, inv, asked.BuildRequest)
lead := fmt.Sprintf("%s\n build %s of %s is asked for now; %s can be assigned once it registers it",
where.said, id, requestSource(asked), module)
return keepPending(ctx, open, node, module, asked, waiting, lead)
}
answer := strings.TrimPrefix(where.said, inventory.ErrNoSuchModule.Error()) +
"\n a module in a repository is built with `build` (its repository and path), then assigned"
if opts.Build {
answer += "\n build \"true\" asks for nothing here: the controller has no build request saying where a " +
"module of that name is"
}
return "", fmt.Errorf("%w%s", inventory.ErrNoSuchModule, answer)
}
// keepPending records the assignment as pending on a build request — or, where one is already open, makes it
// wait for that build — and says so.
func keepPending(ctx context.Context, open *stores, node, module string, r inventory.RequestOutcome,
waiting *inventory.PendingAssignment, lead string) (string, error) {
inv := open.inventory
if waiting != nil {
if waiting.Build == r.ID {
return lead + fmt.Sprintf("\n %s already waits for %s on this build: nothing changed", node, module), nil
}
moved, err := inv.RepointPending(ctx, waiting.ID, r.ID, r.Repository, r.Path)
if err != nil {
return "", err
}
if !moved {
return lead + fmt.Sprintf("\n the pending assignment of %s to %s ended while this was asked: `status` "+
"says how", module, node), nil
}
return lead + fmt.Sprintf("\n the pending assignment of %s to %s, which waited for build %s, now waits "+
"for build %s", module, node, waiting.Build, r.ID), nil
}
_, err := inv.RecordPending(ctx, inventory.PendingAssignment{Node: node, Module: module, Build: r.ID,
Repository: r.Repository, Path: r.Path})
if errors.Is(err, inventory.ErrAlreadyPending) {
return lead + fmt.Sprintf("\n %s already waits for %s: nothing changed", node, module), nil
}
if err != nil {
return "", err
}
// The controller makes it, not this act (ADR 0261): when the build registers the module, or at its next
// tick if the module was registered between the look and the record.
return lead + fmt.Sprintf("\n the assignment is kept as pending: the controller assigns %s to %s when the "+
"build registers it, and `status` lists it until then. If the build fails it expires, says why and "+
"raises a condition; `unassign %s %s` withdraws it", module, node, node, module), nil
}
// inFlight is the newest build request for the module still to register it: no outcome yet within the
// bound, or a successful outcome heard a moment ago and being registered.
func inFlight(requests []inventory.RequestOutcome, now time.Time) (inventory.RequestOutcome, bool) {
for _, r := range requests {
if stillComing(r, now) {
return r, true
}
}
return inventory.RequestOutcome{}, false
}
func stillComing(r inventory.RequestOutcome, now time.Time) bool {
if r.Heard {
return r.Failed == "" && now.Sub(r.HeardAt) < registerGrace
}
return r.NotAsked == "" && now.Sub(r.At) < buildRequestBound
}
// whyNotBuilt says what came of a module's last build request. What was heard comes first: an outcome is the
// last word whatever its asker said.
func whyNotBuilt(r inventory.RequestOutcome, now time.Time) string {
where := fmt.Sprintf("%s in %s", orRoot(r.Path), r.Repository)
switch {
case r.Heard && r.Failed != "":
return fmt.Sprintf("%s; its last build, %s at %s, failed: %s", where, r.ID, clock(r.HeardAt),
firstLine(r.Failed))
case r.Heard && r.Module != "" && r.Module != r.Name():
return fmt.Sprintf("%s; its last build, %s, built it as %s", where, r.ID, r.Module)
case r.Heard:
return fmt.Sprintf("%s; its last build, %s at %s, was recorded and not registered — `builds` says why",
where, r.ID, clock(r.HeardAt))
case r.NotAsked != "" && r.For == "merge":
return fmt.Sprintf("%s; the merge that added it (%s) could not ask for its build: %s", where,
short(r.Commit), firstLine(r.NotAsked))
case r.NotAsked != "":
return fmt.Sprintf("%s; build %s, asked by %s, was not handed over or not waited for: %s", where, r.ID,
askerOr(r.For), firstLine(r.NotAsked))
default:
return fmt.Sprintf("%s; build %s was asked at %s, and no outcome has been heard in %s", where, r.ID,
clock(r.At), now.Sub(r.At).Round(time.Minute))
}
}
func askerOr(who string) string {
if who == "" {
return "the controller"
}
return who
}
// requestSource is a build request's source as a person reads it: repository@commit (directory).
func requestSource(r inventory.RequestOutcome) string {
at := short(r.Commit)
if at == "" {
at = r.Ref
}
if at == "" {
at = "its default branch"
}
return fmt.Sprintf("%s@%s (%s)", r.Repository, at, orRoot(r.Path))
}
// clock is a moment as a person reads it, in the controller's local time.
func clock(t time.Time) string { return t.Local().Format("2006-01-02 15:04:05 MST") }
// closestNames is up to three names near the one asked: registered modules and directories the controller
// asked to build.
func closestNames(ctx context.Context, inv *inventory.Inventory, name string) []string {
var candidates []string
if shelf, err := inv.Catalogue(ctx); err == nil {
for m := range shelf {
candidates = append(candidates, m)
}
}
if asked, err := inv.RequestedNames(ctx); err == nil {
candidates = append(candidates, asked...)
}
type scored struct {
name string
distance int
}
seen := map[string]bool{}
var near []scored
limit := max(2, len(name)/3)
for _, c := range candidates {
if seen[c] || c == name {
continue
}
seen[c] = true
d := editDistance(name, c)
if d <= limit || strings.Contains(c, name) || strings.Contains(name, c) {
near = append(near, scored{c, d})
}
}
sort.Slice(near, func(i, j int) bool {
if near[i].distance != near[j].distance {
return near[i].distance < near[j].distance
}
return near[i].name < near[j].name
})
var out []string
for i := 0; i < len(near) && i < 3; i++ {
out = append(out, near[i].name)
}
return out
}
// editDistance is the Levenshtein distance between two names.
func editDistance(a, b string) int {
ra, rb := []rune(a), []rune(b)
prev := make([]int, len(rb)+1)
for j := range prev {
prev[j] = j
}
for i := 1; i <= len(ra); i++ {
cur := make([]int, len(rb)+1)
cur[0] = i
for j := 1; j <= len(rb); j++ {
cost := 1
if ra[i-1] == rb[j-1] {
cost = 0
}
cur[j] = min(prev[j]+1, cur[j-1]+1, prev[j-1]+cost)
}
prev = cur
}
return prev[len(rb)]
}
// applyPending makes a pending assignment now that its module is registered (ADR 0261). Under the machine's
// hold it claims the row (waiting → applying), so a withdrawal cannot land between the look and the act,
// then assigns by the same act a person's assign is, then says what came of it: applied, or refused with the
// refusal. Returns what it said, or nothing when the row no longer waited.
func applyPending(ctx context.Context, open *stores, p inventory.PendingAssignment, by string) string {
inv := open.inventory
ctx, release, err := holdNodes(ctx, open, []string{p.Node})
if err != nil {
// Left waiting: the next tick tries again.
return fmt.Sprintf("the pending assignment of %s to %s waits: %s could not be held: %v", p.Module, p.Node,
p.Node, err)
}
defer release()
claimed, err := inv.ClaimPending(ctx, p.ID)
if err != nil {
return fmt.Sprintf("the pending assignment of %s to %s could not be taken for making: %v", p.Module, p.Node, err)
}
if !claimed {
return ""
}
answer, err := assign(ctx, open, p.Node, p.Module)
state, note := inventory.PendingApplied, ""
switch {
case err != nil:
state = inventory.PendingRefused
note = fmt.Sprintf("the pending assignment of %s to %s was refused when build %s registered it: %s",
p.Module, p.Node, by, firstLine(err.Error()))
case strings.Contains(answer, "already runs"):
note = fmt.Sprintf("%s already runs %s: the pending assignment is met", p.Node, p.Module)
default:
// The same as a person's assign: nothing is sent by this act.
note = fmt.Sprintf("%s is assigned %s: build %s registered it (pending since %s); `push %s` sends it",
p.Node, p.Module, by, clock(p.Since), p.Node)
}
settled, serr := inv.SettlePending(ctx, p.ID, inventory.PendingApplying, state, note)
if serr != nil {
return fmt.Sprintf("%s — and it could not be recorded as settled: %v", note, serr)
}
if !settled {
return ""
}
return note
}
// expirePending ends a waiting pending assignment whose build will not register its module, with why.
func expirePending(ctx context.Context, inv *inventory.Inventory, p inventory.PendingAssignment, why string) string {
note := fmt.Sprintf("the pending assignment of %s to %s expired: %s; nothing was assigned. Assign it again with "+
"build \"true\" to ask for the build again", p.Module, p.Node, why)
settled, err := inv.SettlePending(ctx, p.ID, inventory.PendingWaiting, inventory.PendingExpired, note)
if err != nil {
return fmt.Sprintf("%s — and it could not be recorded: %v", note, err)
}
if !settled {
return ""
}
return note
}
// settlePendingOnBuild is a build's outcome reaching the assignments waiting for it, wherever the outcome is
// taken in (novox/hq issue 325): a module registered makes every assignment pending on it; a build of a
// pending assignment that failed, or was not registered, ends it with why. Returns what was said.
func settlePendingOnBuild(ctx context.Context, open *stores, result link.BuildResult, module string, takeErr error) []string {
inv := open.inventory
waiting, err := inv.Pending(ctx, time.Time{})
if err != nil {
return []string{fmt.Sprintf("the pending assignments could not be read after build %s: %v", result.ID, err)}
}
registered := module != "" && (takeErr == nil || errors.Is(takeErr, inventory.ErrSuperseded))
var said []string
for _, p := range waiting {
switch {
case registered && p.Module == module:
if line := applyPending(ctx, open, p, result.ID); line != "" {
said = append(said, line)
}
case !registered && p.Build == result.ID:
why := firstLine(result.Failed)
if why == "" && takeErr != nil {
why = firstLine(takeErr.Error())
}
what := fmt.Sprintf("build %s of %s failed: %s", result.ID, orRoot(p.Path), why)
if result.Failed == "" {
what = fmt.Sprintf("build %s of %s was recorded and not registered: %s", result.ID, orRoot(p.Path), why)
}
if line := expirePending(ctx, inv, p, what); line != "" {
said = append(said, line)
}
}
}
return said
}
// settlingPending is the controller's tick over the pending assignments (ADR 0261), every settleEvery until
// the context ends. What it does is printed to the controller's log, kept in each row's note (which `status`
// lists for a day), and raised as a condition for an assignment that was not made.
func settlingPending(ctx context.Context, open *stores) {
for {
for _, line := range settlePending(ctx, open, time.Now()) {
fmt.Println(line)
}
select {
case <-ctx.Done():
return
case <-time.After(settleEvery):
}
}
}
// settlePending is one tick: claims left by a controller that stopped are settled from what the mesh holds;
// a module registered meanwhile is assigned; a build heard and not registered, or silent past its bound,
// ends its assignment; ended ones are raised as conditions and cleared once answered; ended rows older than
// KeptFor are deleted. Returns what it did.
func settlePending(ctx context.Context, open *stores, now time.Time) []string {
inv := open.inventory
var said []string
say := func(format string, args ...any) { said = append(said, fmt.Sprintf(format, args...)) }
shelf, err := inv.Catalogue(ctx)
if err != nil {
say("the pending assignments are not settled this time: the catalogue could not be read: %v", err)
return said
}
stuck, err := inv.Stuck(ctx, now.Add(-claimStaleAfter))
if err != nil {
say("claims left by a stopped controller could not be read: %v", err)
}
for _, p := range stuck {
assigned, err := inv.Assigned(ctx, p.Node)
if err != nil {
say("the claim on %s for %s could not be settled: %v", p.Node, p.Module, err)
continue
}
if containsString(assigned, p.Module) {
if _, err := inv.SettlePending(ctx, p.ID, inventory.PendingApplying, inventory.PendingApplied,
fmt.Sprintf("%s is assigned %s (settled after the controller that made it stopped)", p.Node, p.Module)); err != nil {
say("the claim on %s for %s could not be settled: %v", p.Node, p.Module, err)
}
continue
}
if err := inv.ReleaseClaim(ctx, p.ID); err != nil {
say("the claim on %s for %s could not be released: %v", p.Node, p.Module, err)
}
}
waiting, err := inv.Pending(ctx, time.Time{})
if err != nil {
say("the pending assignments could not be read to settle them: %v", err)
}
for _, p := range waiting {
if _, ok := shelf[p.Module]; ok {
if line := applyPending(ctx, open, p, p.Build); line != "" {
said = append(said, line)
}
continue
}
requests, err := inv.RequestsNamed(ctx, p.Module)
if err != nil {
say("the build requests for %s could not be read: %v", p.Module, err)
continue
}
var r *inventory.RequestOutcome
for i := range requests {
if requests[i].ID == p.Build {
r = &requests[i]
}
}
why := ""
switch {
case r == nil && now.Sub(p.Since) > buildRequestBound:
why = fmt.Sprintf("build %s is no longer on record", p.Build)
case r != nil && !stillComing(*r, now) && (r.Heard || r.NotAsked != ""):
why = whyNotBuilt(*r, now)
case r != nil && !stillComing(*r, now):
why = fmt.Sprintf("no outcome of build %s was heard within %s of asking", p.Build, buildRequestBound)
}
if why == "" {
continue
}
if line := expirePending(ctx, inv, p, why); line != "" {
said = append(said, line)
}
}
said = append(said, raisePendingEnded(ctx, inv)...)
if n, err := inv.ForgetEndedPending(ctx, now); err != nil {
say("ended pending assignments could not be deleted: %v", err)
} else if n > 0 {
say("deleted %d pending assignment(s) ended more than %s ago", n, inventory.KeptFor)
}
return said
}
func containsString(xs []string, x string) bool {
for _, y := range xs {
if y == x {
return true
}
}
return false
}
// pendingObservation is the condition for a pending assignment that was not made.
func pendingObservation(p inventory.PendingAssignment) conditions.Observation {
return conditions.Observation{Scope: conditions.ScopeMachine, ID: p.Node + "." + p.Module,
Token: kindPendingEnded, Kind: kindPendingEnded, Machine: p.Node, Severity: conditions.Warning,
Resolver: conditions.ResolverOperator, Source: "pending assignments",
Summary: p.Note}
}
// raisePendingEnded raises a condition for each pending assignment that expired or was refused, once, and
// clears it when it is answered: the module was assigned to that machine since, a newer pending assignment of
// it is open, or a person took it back with `unassign`. Where this process keeps no conditions, nothing is
// stamped, and the serving controller's tick raises it.
func raisePendingEnded(ctx context.Context, inv *inventory.Inventory) []string {
keeper := conditionsFrom
if keeper == nil {
return nil
}
var said []string
toRaise, err := inv.ToRaise(ctx)
if err != nil {
return []string{fmt.Sprintf("pending assignments not made could not be read to raise them: %v", err)}
}
for _, p := range toRaise {
if _, err := keeper.Observe(ctx, pendingObservation(p)); err != nil {
said = append(said, fmt.Sprintf("the condition for %s on %s could not be raised: %v", p.Module, p.Node, err))
continue
}
if err := inv.MarkPending(ctx, p.ID, "raised"); err != nil {
said = append(said, fmt.Sprintf("the condition for %s on %s was raised and not recorded: %v", p.Module, p.Node, err))
}
}
toClear, err := inv.ToClear(ctx)
if err != nil {
return append(said, fmt.Sprintf("pending assignments raised could not be read to clear them: %v", err))
}
for _, p := range toClear {
why, answered, err := pendingAnswered(ctx, inv, p)
if err != nil {
said = append(said, fmt.Sprintf("whether %s on %s is answered could not be read: %v", p.Module, p.Node, err))
continue
}
if !answered {
continue
}
if _, err := keeper.Clear(ctx, pendingObservation(p).Key(), why); err != nil {
said = append(said, fmt.Sprintf("the condition for %s on %s could not be cleared: %v", p.Module, p.Node, err))
continue
}
if err := inv.MarkPending(ctx, p.ID, "cleared"); err != nil {
said = append(said, fmt.Sprintf("the condition for %s on %s was cleared and not recorded: %v", p.Module, p.Node, err))
}
}
return said
}
// pendingAnswered says whether a pending assignment that was not made has been answered since, and how.
func pendingAnswered(ctx context.Context, inv *inventory.Inventory, p inventory.PendingAssignment) (string, bool, error) {
if p.Acknowledged != nil {
return "taken back with unassign", true, nil
}
assigned, err := inv.Assigned(ctx, p.Node)
if err != nil {
return "", false, err
}
if containsString(assigned, p.Module) {
return p.Module + " is assigned to " + p.Node + " since", true, nil
}
rows, err := inv.PendingFor(ctx, p.Node, p.Module)
if err != nil {
return "", false, err
}
for _, r := range rows {
if r.ID != p.ID && r.Since.After(p.Since) {
return "assigned again, pending on another build", true, nil
}
}
return "", false, nil
}
// withdrawPending is `unassign` of a module only pending on a machine, under the machine's hold: a waiting
// pending assignment is withdrawn; one being made now is refused, never overridden; one that expired or was
// refused is taken back, which answers its condition. False when there is none.
func withdrawPending(ctx context.Context, inv *inventory.Inventory, node, module string) (string, bool, error) {
rows, err := inv.PendingFor(ctx, node, module)
if err != nil {
return "", false, err
}
for _, p := range rows {
switch p.State {
case inventory.PendingApplying:
return "", true, fmt.Errorf("%s is being assigned %s now, as build %s registered it: nothing was "+
"withdrawn; `unassign %s %s` again once it is assigned takes it off", node, module, p.Build, node, module)
case inventory.PendingWaiting:
note := fmt.Sprintf("the pending assignment of %s to %s is withdrawn; build %s goes on, and registers "+
"%s assigned nowhere", module, node, p.Build, module)
settled, err := inv.SettlePending(ctx, p.ID, inventory.PendingWaiting, inventory.PendingWithdrawn, note)
if err != nil {
return "", false, err
}
if !settled {
return "", true, fmt.Errorf("the pending assignment of %s to %s changed while it was withdrawn: "+
"`status` says how; nothing was withdrawn", module, node)
}
return note, true, nil
case inventory.PendingExpired, inventory.PendingRefused:
if p.Cleared != nil || p.Acknowledged != nil {
continue
}
if err := inv.MarkPending(ctx, p.ID, "acknowledged"); err != nil {
return "", false, err
}
return fmt.Sprintf("the pending assignment of %s to %s, which %s, is taken back; its condition clears",
module, node, p.State), true, nil
}
}
return "", false, nil
}
+643
View File
@@ -0,0 +1,643 @@
package main
import (
"context"
"encoding/json"
"errors"
"slices"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// novox/hq issue 325: an assignment of a module the catalogue does not hold says which case it is in — a
// build in flight (kept pending), known and not built (said, with build), or unknown (refused, with the
// closest names) — and a pending assignment is made when its build registers the module, or ends with why.
// aCatalogueMesh is aMesh with one module held from the catalogue, so a merge of it is acted on.
func aCatalogueMesh(t *testing.T) *stores {
t.Helper()
open := aMesh(t)
if err := open.inventory.RegisterModule(t.Context(), catalogue.Manifest{Module: "networkmanager", Version: "1"},
inventory.Source{Repository: "novox/mesh-catalog", Seat: "git", Path: "modules/networkmanager", Ref: "main",
BuiltFrom: "c0", Head: "c0"}); err != nil {
t.Fatal(err)
}
return open
}
// mergeAdding is the catalogue's merge adding one module's directory, acted on as the bus hands it over.
func mergeAdding(t *testing.T, open *stores, dir, commit string) {
t.Helper()
m := link.SourceMoved{Owner: "novox", Repo: "mesh-catalog", Base: "main", Commit: commit,
Paths: []string{dir + "/module.json"}, ModuleDirs: []string{dir}, ModuleDirsSaid: true}
if err := (following{open: open}).SourceMoved(t.Context(), m); err != nil {
t.Fatal(err)
}
}
// outcome is a build's outcome as the build seat announces it: registered as module, or failed.
func outcome(id, dir, commit, module, failed string) link.BuildResult {
r := link.BuildResult{ID: id, Repository: "novox/mesh-catalog", Path: dir, Ref: "main", Commit: commit,
On: "anchor", Failed: failed, Source: &link.SourceOnSeat{Repository: "novox/mesh-catalog", Seat: "git"}}
if failed == "" {
r.Module = module
r.Manifest, _ = json.Marshal(catalogue.Manifest{Module: module, Version: "1"})
}
return r
}
func assignedTo(t *testing.T, open *stores, node string) []string {
t.Helper()
got, err := open.inventory.Assigned(t.Context(), node)
if err != nil {
t.Fatal(err)
}
return got
}
func pendingOf(t *testing.T, open *stores) []inventory.PendingAssignment {
t.Helper()
got, err := open.inventory.Pending(t.Context(), time.Now().Add(-time.Hour))
if err != nil {
t.Fatal(err)
}
return got
}
// A build in flight: the merge asked for it, the request is kept as the merge's, the assignment is kept
// pending and said so with the build, status reads it without settling anything, and the build's outcome
// makes it — saying that a push sends it, and nothing more.
func TestAnAssignmentWaitsForTheBuildInFlight(t *testing.T) {
open := aCatalogueMesh(t)
ctx := t.Context()
asksWithPaths(t)
mergeAdding(t, open, "modules/sensors", "3da80a4b00aa")
requests, err := open.inventory.RequestsNamed(ctx, "sensors")
if err != nil || len(requests) != 1 || requests[0].For != "merge" || requests[0].Commit != "3da80a4b00aa" {
t.Fatalf("the merge's build request: %+v %v", requests, err)
}
said, err := assign(ctx, open, "laptop", "sensors")
if err != nil {
t.Fatalf("an assignment while its build runs was refused: %v", err)
}
t.Logf("assign laptop sensors, while its build runs:\n%s", said)
for _, want := range []string{"being built from novox/mesh-catalog@3da80a4b (modules/sensors)",
"build b-modules/sensors", "kept as pending", "the controller assigns sensors to laptop", "`unassign laptop sensors`"} {
if !strings.Contains(said, want) {
t.Fatalf("the answer does not say %q:\n%s", want, said)
}
}
if slices.Contains(assignedTo(t, open, "laptop"), "sensors") {
t.Fatal("a module not registered was assigned")
}
pending := pendingOf(t, open)
if len(pending) != 1 || pending[0].State != inventory.PendingWaiting || pending[0].Build != "b-modules/sensors" {
t.Fatalf("pending: %+v", pending)
}
if again, err := assign(ctx, open, "laptop", "sensors"); err != nil || !strings.Contains(again, "already waits") {
t.Fatalf("a second assignment: %q %v", again, err)
}
asked, err := theThreeQuestions(ctx, open)
if err != nil {
t.Fatal(err)
}
if asked.well() {
t.Fatal("status called the mesh well while an assignment waits")
}
body, err := statusAsJSON(asked)
if err != nil {
t.Fatal(err)
}
var doc meshStatus
if err := json.Unmarshal(body, &doc); err != nil {
t.Fatal(err)
}
if len(doc.Pending) != 1 || doc.Pending[0].State != "waiting" || doc.Pending[0].Module != "sensors" {
t.Fatalf("status says pending %+v", doc.Pending)
}
if err := (builds{open.inventory, open}).Built(ctx, outcome("b-modules/sensors", "modules/sensors",
"3da80a4b00aa", "sensors", "")); err != nil {
t.Fatal(err)
}
if !slices.Contains(assignedTo(t, open, "laptop"), "sensors") {
t.Fatal("the build registered sensors and the pending assignment was not made")
}
pending = pendingOf(t, open)
if len(pending) != 1 || pending[0].State != inventory.PendingApplied ||
!strings.HasSuffix(pending[0].Note, "`push laptop` sends it") {
t.Fatalf("pending after the build: %+v", pending)
}
}
// **A read settles nothing** (review of #150, point 6): status — and so the board, the summary and the
// probe, which compose from the same reading — leaves a pending assignment whose module is registered as it
// is; the controller's tick makes it.
func TestAStatusReadSettlesNothing(t *testing.T) {
open := aCatalogueMesh(t)
ctx := t.Context()
asksWithPaths(t)
mergeAdding(t, open, "modules/sensors", "3da80a4b00aa")
if _, err := assign(ctx, open, "laptop", "sensors"); err != nil {
t.Fatal(err)
}
register(t, open, catalogue.Manifest{Module: "sensors", Version: "1"})
for range 2 {
if _, err := theThreeQuestions(ctx, open); err != nil {
t.Fatal(err)
}
}
if slices.Contains(assignedTo(t, open, "laptop"), "sensors") || pendingOf(t, open)[0].State != inventory.PendingWaiting {
t.Fatalf("a status read settled a pending assignment: %+v", pendingOf(t, open))
}
said := settlePending(ctx, open, time.Now())
if !slices.Contains(assignedTo(t, open, "laptop"), "sensors") || pendingOf(t, open)[0].State != inventory.PendingApplied {
t.Fatalf("the tick did not make it: %v %+v", said, pendingOf(t, open))
}
}
// The build of a pending assignment fails: it expires with the build's words and nothing is assigned; the
// tick raises it as a condition once; it then reads as known and not built; and `unassign` takes it back,
// after which the tick clears the condition. Status is not called well while the condition is open, and is
// again once it clears — no fixed day of "not well" (review point 6).
func TestAPendingAssignmentExpiresWhenItsBuildFails(t *testing.T) {
open := aCatalogueMesh(t)
ctx := t.Context()
asksWithPaths(t)
mergeAdding(t, open, "modules/sensors", "3da80a4b00aa")
if _, err := assign(ctx, open, "laptop", "sensors"); err != nil {
t.Fatal(err)
}
if err := (builds{open.inventory, open}).Built(ctx, outcome("b-modules/sensors", "modules/sensors",
"3da80a4b00aa", "", "go build: undefined: sensorsRead")); err != nil {
t.Fatal(err)
}
pending := pendingOf(t, open)
if len(pending) != 1 || pending[0].State != inventory.PendingExpired {
t.Fatalf("pending after a failed build: %+v", pending)
}
for _, want := range []string{"expired", "build b-modules/sensors of modules/sensors failed",
"undefined: sensorsRead", "nothing was assigned", "build \"true\""} {
if !strings.Contains(pending[0].Note, want) {
t.Fatalf("the expiry does not say %q: %s", want, pending[0].Note)
}
}
if slices.Contains(assignedTo(t, open, "laptop"), "sensors") {
t.Fatal("a failed build's module was assigned")
}
settlePending(ctx, open, time.Now())
key := pendingObservation(pending[0]).Key()
c, found, err := conditionsFrom.Get(ctx, key)
if err != nil || !found || c.Kind != kindPendingEnded {
t.Fatalf("no condition %s for the assignment not made: %+v %v %v", key, c, found, err)
}
if asked, err := theThreeQuestions(ctx, open); err != nil || asked.well() || len(asked.conditions) == 0 {
t.Fatalf("status does not hold the open condition: %v", err)
}
_, err = assign(ctx, open, "laptop", "sensors")
if !errors.Is(err, inventory.ErrNoSuchModule) {
t.Fatalf("a module whose build failed: %v", err)
}
for _, want := range []string{"sensors is known and not registered", "its last build, b-modules/sensors",
"failed: go build: undefined: sensorsRead", "build \"true\"", "repository novox/mesh-catalog, path modules/sensors"} {
if !strings.Contains(err.Error(), want) {
t.Fatalf("the refusal does not say %q:\n%v", want, err)
}
}
said, err := unassign(ctx, open, "laptop", "sensors")
if err != nil || !strings.Contains(said, "taken back") {
t.Fatalf("unassign of an expired pending assignment: %q %v", said, err)
}
settlePending(ctx, open, time.Now())
if _, found, _ := conditionsFrom.Get(ctx, key); found {
t.Fatal("the condition stayed open after the assignment was taken back")
}
// Nothing of it keeps status from being well any more: no open pending assignment, no open condition of it.
asked, err := theThreeQuestions(ctx, open)
if err != nil || pendingOpen(asked.pending) {
t.Fatalf("status still counts the pending assignment: %+v %v", asked.pending, err)
}
for _, c := range asked.conditions {
if c.Kind == kindPendingEnded {
t.Fatalf("status still holds the condition: %+v", c)
}
}
}
// The condition also clears when the module is later assigned to the machine.
func TestTheConditionClearsWhenTheModuleIsAssignedLater(t *testing.T) {
open := aCatalogueMesh(t)
ctx := t.Context()
asksWithPaths(t)
mergeAdding(t, open, "modules/sensors", "3da80a4b00aa")
if _, err := assign(ctx, open, "laptop", "sensors"); err != nil {
t.Fatal(err)
}
if err := (builds{open.inventory, open}).Built(ctx, outcome("b-modules/sensors", "modules/sensors",
"3da80a4b00aa", "", "boom")); err != nil {
t.Fatal(err)
}
settlePending(ctx, open, time.Now())
key := pendingObservation(pendingOf(t, open)[0]).Key()
register(t, open, catalogue.Manifest{Module: "sensors", Version: "1"})
if _, err := assign(ctx, open, "laptop", "sensors"); err != nil {
t.Fatal(err)
}
settlePending(ctx, open, time.Now())
if _, found, _ := conditionsFrom.Get(ctx, key); found {
t.Fatal("the condition stayed open after the module was assigned")
}
}
// Known and not built, asked with build: the build is asked from where the last one was, the request kept as
// assign's, and the assignment kept pending on the new build.
func TestAssignWithBuildAsksForAKnownModule(t *testing.T) {
open := aCatalogueMesh(t)
ctx := t.Context()
inv := open.inventory
if err := inv.RecordBuildRequest(ctx, inventory.BuildRequest{ID: "build-old", Repository: "novox/mesh-catalog",
Seat: "git", Path: "modules/sensors", Ref: "main", For: "build", At: time.Now().Add(-time.Hour)}); err != nil {
t.Fatal(err)
}
if err := inv.RecordBuild(ctx, inventory.Build{ID: "build-old", Repository: "novox/mesh-catalog", Ref: "main",
Path: "modules/sensors", On: "anchor", Failed: "no space left on device"}); err != nil {
t.Fatal(err)
}
asked := asksWithPaths(t)
said, err := assignWith(ctx, open, "laptop", assignOptions{Build: true}, "sensors")
if err != nil {
t.Fatal(err)
}
if len(*asked) != 1 || (*asked)[0] != [3]string{"novox/mesh-catalog", "modules/sensors", "main"} {
t.Fatalf("asked %v", *asked)
}
for _, want := range []string{"known and not registered", "no space left on device",
"build b-modules/sensors of novox/mesh-catalog@main (modules/sensors) is asked for now", "kept as pending"} {
if !strings.Contains(said, want) {
t.Fatalf("the answer does not say %q:\n%s", want, said)
}
}
pending := pendingOf(t, open)
if len(pending) != 1 || pending[0].Build != "b-modules/sensors" {
t.Fatalf("pending: %+v", pending)
}
requests, _ := inv.RequestsNamed(ctx, "sensors")
if len(requests) != 2 || requests[0].ID != "b-modules/sensors" || requests[0].For != "assign" {
t.Fatalf("the request is not kept as assign's: %+v", requests)
}
}
// **build "true" with a pending assignment already waiting** (review point 5): the waiting one is made to
// wait for the new build, and no build is asked that nothing waits on.
func TestAssignWithBuildRepointsTheWaitingAssignment(t *testing.T) {
open := aCatalogueMesh(t)
ctx := t.Context()
inv := open.inventory
if err := inv.RecordBuildRequest(ctx, inventory.BuildRequest{ID: "build-old", Repository: "novox/mesh-catalog",
Seat: "git", Path: "modules/sensors", Ref: "main", For: "merge", At: time.Now().Add(-time.Hour)}); err != nil {
t.Fatal(err)
}
if err := inv.RecordBuild(ctx, inventory.Build{ID: "build-old", Repository: "novox/mesh-catalog", Ref: "main",
Path: "modules/sensors", On: "anchor", Failed: "boom"}); err != nil {
t.Fatal(err)
}
// Waiting on the failed build, its expiry not yet settled.
if _, err := inv.RecordPending(ctx, inventory.PendingAssignment{Node: "laptop", Module: "sensors",
Build: "build-old", Repository: "novox/mesh-catalog", Path: "modules/sensors"}); err != nil {
t.Fatal(err)
}
_, err := assign(ctx, open, "laptop", "sensors")
if err == nil || !strings.Contains(err.Error(), "already has a pending assignment of sensors, waiting for build build-old") {
t.Fatalf("known and not built with a pending assignment waiting: %v", err)
}
asked := asksWithPaths(t)
said, err := assignWith(ctx, open, "laptop", assignOptions{Build: true}, "sensors")
if err != nil || !strings.Contains(said, "which waited for build build-old, now waits for build b-modules/sensors") {
t.Fatalf("assign with build: %q %v", said, err)
}
if len(*asked) != 1 {
t.Fatalf("asked %v", *asked)
}
pending := pendingOf(t, open)
if len(pending) != 1 || pending[0].Build != "b-modules/sensors" || pending[0].State != inventory.PendingWaiting {
t.Fatalf("pending: %+v", pending)
}
// The new build in flight now: build "true" again asks nothing.
if said, err := assignWith(ctx, open, "laptop", assignOptions{Build: true}, "sensors"); err != nil ||
!strings.Contains(said, "no second build was asked") || len(*asked) != 1 {
t.Fatalf("a second build true while the build runs: %q %v, asked %v", said, err, *asked)
}
}
// **A failed ask never reads as in flight** (review point 2): a merge whose ask could not be made keeps the
// request as not asked, and assign says the merge could not ask; a request whose asker could not hand it over
// or stopped waiting reads the same, unless its outcome was heard.
func TestAFailedAskIsNotABuildInFlight(t *testing.T) {
open := aCatalogueMesh(t)
ctx := t.Context()
inv := open.inventory
was := askABuild
askABuild = func(context.Context, buildSource, string, string) (string, error) {
return "", errors.New("cannot submit a build: nats: timeout")
}
t.Cleanup(func() { askABuild = was })
mergeAdding(t, open, "modules/sensors", "3da80a4b00aa")
_, err := assign(ctx, open, "laptop", "sensors")
if err == nil || !strings.Contains(err.Error(), "the merge that added it (3da80a4b) could not ask for its build: cannot submit") {
t.Fatalf("a merge that could not ask: %v", err)
}
if len(pendingOf(t, open)) != 0 {
t.Fatal("a pending assignment waits on a build never asked")
}
if err := inv.RecordBuildRequest(ctx, inventory.BuildRequest{ID: "build-waited", Repository: "novox/mesh-catalog",
Seat: "git", Path: "modules/gauges", Ref: "main", For: "build"}); err != nil {
t.Fatal(err)
}
if err := inv.MarkNotAsked(ctx, "build-waited", "cannot submit a build: no responders"); err != nil {
t.Fatal(err)
}
_, err = assign(ctx, open, "laptop", "gauges")
if err == nil || !strings.Contains(err.Error(), "build build-waited, asked by build, was not handed over or not waited for") {
t.Fatalf("a build not handed over: %v", err)
}
// Heard first, the outcome stands: marking it afterwards changes nothing.
if err := inv.RecordBuild(ctx, inventory.Build{ID: "build-waited", Repository: "novox/mesh-catalog", Ref: "main",
Path: "modules/gauges", On: "anchor", Failed: "the real failure"}); err != nil {
t.Fatal(err)
}
_, err = assign(ctx, open, "laptop", "gauges")
if err == nil || !strings.Contains(err.Error(), "failed: the real failure") {
t.Fatalf("an outcome heard after a failed wait: %v", err)
}
}
// A plan's tier keeps its requests as the plan's (review point 3).
func TestAPlansBuildRequestIsThePlans(t *testing.T) {
open := aCatalogueMesh(t)
ctx := t.Context()
asksWithPaths(t)
if err := open.inventory.RegisterModule(ctx, catalogue.Manifest{Module: "app", Version: "1"},
inventory.Source{Repository: "novox/mesh-catalog", Seat: "git", Path: "modules/app", Ref: "main",
BuiltFrom: "c0", Head: "c0"}); err != nil {
t.Fatal(err)
}
m := link.SourceMoved{Owner: "novox", Repo: "mesh-catalog", Base: "main", Commit: "c1aaaaaaaa",
Paths: []string{"modules/app/index.ts"}, ModuleDirs: []string{"modules/app"}, ModuleDirsSaid: true}
if err := (following{open: open}).SourceMoved(ctx, m); err != nil {
t.Fatal(err)
}
requests, err := open.inventory.RequestsNamed(ctx, "app")
if err != nil || len(requests) != 1 || requests[0].For != "plan" {
t.Fatalf("the plan's request: %+v %v", requests, err)
}
}
// **A build heard as built and not registered yet is in flight** (review point 4): the outcome is recorded
// before the module is registered; for that moment assign keeps the assignment pending. Past the grace it is
// recorded and not registered, and the tick ends the pending assignment with that.
func TestABuildBeingRegisteredIsInFlight(t *testing.T) {
open := aCatalogueMesh(t)
ctx := t.Context()
inv := open.inventory
if err := inv.RecordBuildRequest(ctx, inventory.BuildRequest{ID: "build-done", Repository: "novox/mesh-catalog",
Seat: "git", Path: "modules/sensors", Ref: "main", For: "merge"}); err != nil {
t.Fatal(err)
}
if err := inv.RecordBuild(ctx, inventory.Build{ID: "build-done", Repository: "novox/mesh-catalog", Ref: "main",
Path: "modules/sensors", Module: "sensors", On: "anchor"}); err != nil {
t.Fatal(err)
}
said, err := assign(ctx, open, "laptop", "sensors")
if err != nil || !strings.Contains(said, "built and being registered") || !strings.Contains(said, "kept as pending") {
t.Fatalf("a build being registered: %q %v", said, err)
}
settlePending(ctx, open, time.Now().Add(registerGrace+time.Minute))
p := pendingOf(t, open)
if len(p) != 1 || p[0].State != inventory.PendingExpired || !strings.Contains(p[0].Note, "recorded and not registered") {
t.Fatalf("past the grace: %+v", p)
}
}
// Unknown: refused as no module of that name, claiming only what was looked at, with the closest names; build
// asks for nothing.
func TestAnUnknownModuleIsRefusedWithTheClosestNames(t *testing.T) {
open := aCatalogueMesh(t)
ctx := t.Context()
register(t, open, catalogue.Manifest{Module: "sensord", Version: "1"})
asked := asksWithPaths(t)
_, err := assignWith(ctx, open, "laptop", assignOptions{Build: true}, "sensors")
if !errors.Is(err, inventory.ErrNoSuchModule) {
t.Fatalf("an unknown module: %v", err)
}
for _, want := range []string{"no module of that name: sensors", "the catalogue holds no module of that name",
"no build request the controller kept (the last 30 days)", "the closest names it holds: sensord",
"asks for nothing here"} {
if !strings.Contains(err.Error(), want) {
t.Fatalf("the refusal does not say %q:\n%v", want, err)
}
}
if strings.Contains(err.Error(), "merge") {
t.Fatalf("the refusal claims a merge it never looked at: %v", err)
}
if len(*asked) != 0 || len(pendingOf(t, open)) != 0 {
t.Fatalf("an unknown module asked %v, pending %+v", *asked, pendingOf(t, open))
}
}
// Several modules in one act, one of them not registered: nothing is assigned and nothing kept pending, and
// each missing one is said.
func TestAnActWithAModuleNotRegisteredIsRefusedWhole(t *testing.T) {
open := aCatalogueMesh(t)
ctx := t.Context()
asksWithPaths(t)
register(t, open, catalogue.Manifest{Module: "known", Version: "1"})
mergeAdding(t, open, "modules/sensors", "3da80a4b00aa")
_, err := assign(ctx, open, "laptop", "known", "sensors")
if err == nil || !strings.Contains(err.Error(), "nothing was assigned") || !strings.Contains(err.Error(), "being built") {
t.Fatalf("an act naming a module in flight: %v", err)
}
if slices.Contains(assignedTo(t, open, "laptop"), "known") || len(pendingOf(t, open)) != 0 {
t.Fatal("an act refused whole was made in part")
}
}
// unassign withdraws a waiting pending assignment; the build goes on and registers the module assigned
// nowhere: a withdrawn row is never claimed.
func TestUnassignWithdrawsAPendingAssignment(t *testing.T) {
open := aCatalogueMesh(t)
ctx := t.Context()
asksWithPaths(t)
mergeAdding(t, open, "modules/sensors", "3da80a4b00aa")
if _, err := assign(ctx, open, "laptop", "sensors"); err != nil {
t.Fatal(err)
}
said, err := unassign(ctx, open, "laptop", "sensors")
if err != nil || !strings.Contains(said, "withdrawn") {
t.Fatalf("unassign of a pending assignment: %q %v", said, err)
}
if err := (builds{open.inventory, open}).Built(ctx, outcome("b-modules/sensors", "modules/sensors",
"3da80a4b00aa", "sensors", "")); err != nil {
t.Fatal(err)
}
settlePending(ctx, open, time.Now())
if slices.Contains(assignedTo(t, open, "laptop"), "sensors") {
t.Fatal("a withdrawn assignment was made")
}
}
// **A claim is never overridden** (review point 1): a pending assignment being made is claimed
// (waiting → applying); an unassign meanwhile is refused and says so, and leaves the claim; making a row
// already withdrawn makes nothing. A claim left by a controller that stopped is settled by the tick from what
// the mesh holds: back to waiting when the module is not assigned, applied when it is.
func TestAWithdrawalNeverOverridesAClaim(t *testing.T) {
open := aCatalogueMesh(t)
ctx := t.Context()
inv := open.inventory
asksWithPaths(t)
mergeAdding(t, open, "modules/sensors", "3da80a4b00aa")
if _, err := assign(ctx, open, "laptop", "sensors"); err != nil {
t.Fatal(err)
}
p := pendingOf(t, open)[0]
if ok, err := inv.ClaimPending(ctx, p.ID); err != nil || !ok {
t.Fatalf("claim: %v %v", ok, err)
}
_, err := unassign(ctx, open, "laptop", "sensors")
if err == nil || !strings.Contains(err.Error(), "is being assigned sensors now") {
t.Fatalf("unassign of a claimed pending assignment: %v", err)
}
if got := pendingOf(t, open)[0]; got.State != inventory.PendingApplying {
t.Fatalf("the claim was overridden: %+v", got)
}
// Pending(zero) is the waiting ones alone (review point 7).
if waiting, err := inv.Pending(ctx, time.Time{}); err != nil || len(waiting) != 0 {
t.Fatalf("Pending(zero) read a claimed row: %+v %v", waiting, err)
}
// Left by a controller that stopped: back to waiting, since sensors is not assigned.
settlePending(ctx, open, time.Now().Add(claimStaleAfter+time.Minute))
if got := pendingOf(t, open)[0]; got.State != inventory.PendingWaiting {
t.Fatalf("a stale claim was not released: %+v", got)
}
// Withdrawn, then the build registers it: applyPending finds nothing to claim.
if _, err := unassign(ctx, open, "laptop", "sensors"); err != nil {
t.Fatal(err)
}
register(t, open, catalogue.Manifest{Module: "sensors", Version: "1"})
if line := applyPending(ctx, open, p, "b-modules/sensors"); line != "" {
t.Fatalf("a withdrawn pending assignment was made: %s", line)
}
if slices.Contains(assignedTo(t, open, "laptop"), "sensors") {
t.Fatal("a withdrawn pending assignment was assigned")
}
// A stale claim whose module was assigned is applied.
q, err := inv.RecordPending(ctx, inventory.PendingAssignment{Node: "anchor", Module: "sensors", Build: "b-x"})
if err != nil {
t.Fatal(err)
}
if ok, _ := inv.ClaimPending(ctx, q.ID); !ok {
t.Fatal("claim")
}
if _, err := inv.Assign(ctx, "anchor", "sensors"); err != nil {
t.Fatal(err)
}
settlePending(ctx, open, time.Now().Add(claimStaleAfter+time.Minute))
rows, _ := inv.PendingFor(ctx, "anchor", "sensors")
if len(rows) != 1 || rows[0].State != inventory.PendingApplied {
t.Fatalf("a stale claim of an assigned module: %+v", rows)
}
}
// A build that says nothing within the bound: the pending assignment expires at the next tick, saying so,
// rather than waiting for ever; and a module registered by a process that kept no pending assignment is
// assigned at the next tick.
func TestAPendingAssignmentNeverWaitsSilently(t *testing.T) {
open := aCatalogueMesh(t)
ctx := t.Context()
inv := open.inventory
for _, r := range []inventory.BuildRequest{
{ID: "build-lost", Repository: "novox/mesh-catalog", Seat: "git", Path: "modules/lost", Ref: "main",
For: "merge", At: time.Now().Add(-buildRequestBound / 2)},
{ID: "build-heard", Repository: "novox/mesh-catalog", Seat: "git", Path: "modules/heard", Ref: "main",
For: "merge", At: time.Now()},
} {
if err := inv.RecordBuildRequest(ctx, r); err != nil {
t.Fatal(err)
}
}
for _, m := range []string{"lost", "heard"} {
if _, err := assign(ctx, open, "laptop", m); err != nil {
t.Fatal(err)
}
}
register(t, open, catalogue.Manifest{Module: "heard", Version: "1"})
settlePending(ctx, open, time.Now().Add(buildRequestBound))
if !slices.Contains(assignedTo(t, open, "laptop"), "heard") {
t.Fatal("a module registered meanwhile was not assigned at the next tick")
}
byModule := map[string]inventory.PendingAssignment{}
for _, p := range pendingOf(t, open) {
byModule[p.Module] = p
}
if p := byModule["lost"]; p.State != inventory.PendingExpired || !strings.Contains(p.Note, "no outcome of build build-lost") {
t.Fatalf("a build that said nothing: %+v", p)
}
if p := byModule["heard"]; p.State != inventory.PendingApplied {
t.Fatalf("a module registered meanwhile: %+v", p)
}
// Ended rows are deleted after KeptFor (review point 7); open ones never.
if _, err := inv.RecordPending(ctx, inventory.PendingAssignment{Node: "anchor", Module: "lost", Build: "build-lost"}); err != nil {
t.Fatal(err)
}
said := settlePending(ctx, open, time.Now().Add(inventory.KeptFor+time.Hour))
rows, err := inv.Pending(ctx, time.Unix(0, 0))
if err != nil {
t.Fatal(err)
}
for _, r := range rows {
if !r.Open() {
t.Fatalf("an ended pending assignment outlived %s: %+v (%v)", inventory.KeptFor, r, said)
}
}
}
// The seat's assign passes build on; unassign takes none.
func TestTheSeatsAssignPassesBuild(t *testing.T) {
argv, err := argvFor("assign", map[string]any{"node": "g14", "module": "sensors", "build": "true"})
if err != nil || !slices.Equal(argv, []string{"assign", "g14", "sensors", "--build"}) {
t.Fatalf("assign with build: %v %v", argv, err)
}
if _, err := argvFor("unassign", map[string]any{"node": "g14", "module": "sensors", "build": "true"}); err == nil {
t.Fatal("unassign took build")
}
}
// The condition's words are plain.
func TestAnAssignmentNotMadeIsSaidPlainly(t *testing.T) {
o := pendingObservation(inventory.PendingAssignment{Node: "g14", Module: "sensors",
Note: "the pending assignment of sensors to g14 expired: build b-1 of modules/sensors failed: boom"})
w := plainWordings[kindPendingEnded](o)
if why, ok := conditions.PlainWords(w, "g14"); !ok {
t.Fatalf("not plain: %s %+v", why, w)
}
if w.Headline != "sensors was not put on g14" {
t.Fatalf("headline %q", w.Headline)
}
}
+103 -5
View File
@@ -6,6 +6,7 @@ import (
"testing"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
@@ -75,11 +76,20 @@ func TestOnlyWhatDependsOnTheNewLoginIsAWait(t *testing.T) {
func TestAWaitForAPersonIsAPassCarriedAlong(t *testing.T) {
now := time.Now()
since := now.Add(-time.Minute)
account := relogin("lights", "operator")
f := gateFacts{now: now, health: map[string]inventory.NodeHealth{"laptop": {Node: "laptop", HeardAt: now,
Resources: []inventory.ResourceHealth{relogin("lights", "operator"), userUnit("lights", "operator")}}}}
Resources: []inventory.ResourceHealth{account, userUnit("lights", "operator")}}},
groupsAdded: map[string]bool{"lights": true}}
if h, why := moduleHealthWord("lights", "laptop", since, f); h != healthPerson || !strings.Contains(why, "relogin needed on laptop") {
t.Fatalf("a wait for a new login reads %v %q; want a wait for a person", h, why)
}
// **Only a move that put the account in a new group is excused** (issue 318 review): a later build that
// adds no group did not bring the wait, and is judged as before.
f.groupsAdded["lights"] = false
if h, why := moduleHealthWord("lights", "laptop", since, f); h != healthNotYet {
t.Fatalf("a wait the move did not bring reads %v %q; want not yet", h, why)
}
f.groupsAdded["lights"] = true
h := f.health["laptop"]
h.Resources = append(h.Resources, inventory.ResourceHealth{Module: "lights", Resource: "lights.web", Kind: "container",
Target: "lights", State: link.StateUnhealthy, Reason: "down"})
@@ -197,7 +207,7 @@ func TestAModuleHealthyOnItsOwnKeepsItsPassWhenItsSendFails(t *testing.T) {
// What the operator reads of the wait (ADR 0253, ADR 0254): it needs them, so it is never quiet, and it offers
// no button, since nothing but their own new login can do it.
func TestTheReloginConditionNeedsTheOperatorAndOffersNoButton(t *testing.T) {
o := reloginObservation("openrazer", "g14", "relogin needed on g14: …", []inventory.ResourceHealth{
o := reloginObservation("openrazer", "g14", "relogin needed on g14: …", "operator", []inventory.ResourceHealth{
relogin("openrazer", "operator"), userUnit("openrazer", "operator")})
plainExample(t, o, "openrazer waits for a new login on g14",
"Needs you: log out of g14 completely and log in again, or restart it. Openrazer put your account in a "+
@@ -206,9 +216,97 @@ func TestTheReloginConditionNeedsTheOperatorAndOffersNoButton(t *testing.T) {
if len(o.Actions) != 0 || o.Needs == "" {
t.Errorf("relogin: needs %q, actions %+v; want needs and no button", o.Needs, o.Actions)
}
// The kind's own wording, for a condition raised without words, says the same.
if w := plainWordings[kindReloginNeeded](conditions.Observation{Scope: conditions.ScopeModule, ID: "openrazer.g14", Machine: "g14"}); w.Needs != o.Needs ||
w.Headline != o.Headline || len(w.Actions) != 0 {
// **Not "your account" when it is not the operator's** (issue 318 review).
other := reloginObservation("openrazer", "g14", "relogin needed on g14: …", "operator", []inventory.ResourceHealth{
relogin("openrazer", "guest"), userUnit("openrazer", "guest")})
plainExample(t, other, "openrazer waits for a new login on g14",
"Needs you: have the account it names log out of g14 completely and log in again, or restart g14. Openrazer "+
"put an account on g14 in a group it needs. That account logged in before that, so openrazer cannot run "+
"until it logs in again. Its update is in place and nothing was undone.")
if unknown := reloginObservation("openrazer", "g14", "…", "", []inventory.ResourceHealth{relogin("openrazer", "operator")}); strings.Contains(unknown.Explanation, "your account") {
t.Errorf("an operator not known is still told it is their account: %q", unknown.Explanation)
}
// The kind's own wording, for a condition raised without words, names the module whole, dots and all,
// and never claims the account is the operator's.
w := plainWordings[kindReloginNeeded](conditions.Observation{Scope: conditions.ScopeModule, ID: "razer.lights.g14", Machine: "g14"})
if w.Headline != "razer.lights waits for a new login on g14" || w.Needs == "" || len(w.Actions) != 0 ||
strings.Contains(w.Explanation, "your account") {
t.Errorf("the kind's wording: %+v", w)
}
}
// When a module not working turns into one waiting for a new login, and back, the clearing line says what it
// became, never that it works again (issue 318 review).
func TestAConditionThatBecameTheOtherKindSaysSoWhenItClears(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
store := conditions.NewInMemory()
told := &conditions.Told{}
k := conditions.NewKeeper(ctx, conditions.Options{Store: store, History: store, Teller: told})
t.Cleanup(func() { k.Close(context.Background()) })
at := h0
say := func(rs ...link.ResourceHealth) {
t.Helper()
at = at.Add(time.Second)
for i := range rs {
rs[i].Since = at
}
if err := stateHealth(ctx, open.inventory, k, "laptop", link.Health{Contract: link.ReadinessContract, At: at,
Resources: rs}, at); err != nil {
t.Fatal(err)
}
}
unit := link.ResourceHealth{Module: "lights", Resource: "lights.daemon", Kind: link.KindUnit, Target: "lights.service",
Account: "operator", State: link.StateUnhealthy, Reason: "failed in the account's own service manager (exit-code)"}
account := link.ResourceHealth{Module: "lights", Resource: "lights.operator", Kind: link.KindAccount, Target: "operator",
Account: "operator", State: link.StateUnhealthy, Reason: link.ReasonRelogin + ": operator is in the group lights"}
say(unit)
say(unit) // lights not working
say(account, unit)
say(account, unit) // now it waits for a login
var resolved []string
for deadline := time.Now().Add(5 * time.Second); time.Now().Before(deadline) && len(resolved) == 0; {
time.Sleep(20 * time.Millisecond)
for _, e := range told.Said() {
if e.Change == conditions.ChangeCleared {
resolved = append(resolved, e.Condition.Key+": "+e.Condition.Resolved)
// The clearing line is held to the plain rule too (ADR 0253).
w := conditions.Words{Headline: e.Condition.Headline, Explanation: e.Condition.Explanation,
Resolved: e.Condition.Resolved, Needs: e.Condition.Needs}
if why, ok := conditions.PlainWords(w, "laptop"); !ok {
t.Errorf("%s: its clearing words are not plain: %s", e.Condition.Key, why)
}
}
}
}
if len(resolved) != 1 || !strings.Contains(resolved[0], "module.lights.laptop.unhealthy: lights on laptop now waits only for a new login") {
t.Fatalf("cleared %v; want the not-working condition cleared as now waiting for a login", resolved)
}
}
// Which moves add an account group, read from the builds' manifests (issue 318 review).
func TestOnlyAMoveThatAddsAnAccountGroupCanBringAWait(t *testing.T) {
user := func(groups ...any) catalogue.Manifest {
return catalogue.Manifest{Module: "lights", Resources: []map[string]any{{"id": "operator", "type": "user",
"name": "operator", "groups": groups}}}
}
none := catalogue.Manifest{Module: "lights"}
for _, c := range []struct {
name string
from catalogue.Manifest
had bool
to catalogue.Manifest
addsSome bool
}{
{"a group added", none, true, user("lights"), true},
{"the same group kept", user("lights"), true, user("lights"), false},
{"a second group added", user("lights"), true, user("lights", "video"), true},
{"a group taken away", user("lights", "video"), true, user("lights"), false},
{"new to the machine, with a group", none, false, user("lights"), true},
{"new to the machine, no group", none, false, none, false},
} {
if got := addsAccountGroups(c.from, c.had, c.to); got != c.addsSome {
t.Errorf("%s: adds %v; want %v", c.name, got, c.addsSome)
}
}
}
+18 -4
View File
@@ -153,6 +153,19 @@ var plainWordings = map[string]func(conditions.Observation) words{
"asleep is normal.", m),
Resolved: m + " can be reached again"}
}),
// A pending assignment that was not made (novox/hq issue 325, ADR 0261).
kindPendingEnded: worded(func(o conditions.Observation) words {
m := machineOr(o, "a machine")
module := idPart(o, 1)
if module == "" {
module = "a module"
}
return words{Headline: module + " was not put on " + m,
Needs: "decide whether to build it again or take the assignment back; the details say why.",
Explanation: fmt.Sprintf("An assignment of %s to %s waited for its build, and the build did not "+
"register it, so it was not made.", module, m),
Resolved: "Resolved: " + module + " on " + m + " is settled"}
}),
kindBindingKept: worded(func(o conditions.Observation) words {
m := machineOr(o, "a machine")
return words{Headline: "A module's data source is held on " + m,
@@ -184,11 +197,12 @@ var plainWordings = map[string]func(conditions.Observation) words{
Resolved: conditions.Capital(thing) + " works again"}
}),
kindReloginNeeded: worded(func(o conditions.Observation) words {
// A module's name may hold dots: it is the id without its machine.
module := ""
if o.Scope == conditions.ScopeModule {
module = idPart(o, 0)
if o.Scope == conditions.ScopeModule && o.Machine != "" {
module = strings.TrimSuffix(o.ID, "."+o.Machine)
}
return reloginWords(orModule(module), machineOr(o, "a machine"))
return reloginWords(orModule(module), machineOr(o, "a machine"), false)
}),
kindProviderFailing: worded(func(o conditions.Observation) words {
thing, consumer := conditions.ThingWords(o), idPart(o, 2)
@@ -671,7 +685,7 @@ const FromMeshMCPServer = "from the mesh MCP server; this notification cannot do
// reloginNeeds is what an account waiting for its groups needs (ADR 0252).
func reloginNeeds(node string) string {
return fmt.Sprintf("log out of every session on %s and log in again.", node)
return fmt.Sprintf("log out of %s completely and log in again, or restart it.", node)
}
// stalledWords are the plain words of a delivery held past its bound, as mesh-delivery says it.
+4 -1
View File
@@ -94,7 +94,7 @@ func TestAModuleUnhealthyAsksForARestartInWords(t *testing.T) {
// An account waiting for a new login (ADR 0252) asks for the login, held to the plain rule.
o = moduleUnhealthyObservation("openrazer", "g14", []inventory.ResourceHealth{{Kind: "account",
Resource: "operator-in-group", Target: "jochen", Reason: "relogin needed: the account is in the group"}})
if o.Needs != "log out of every session on g14 and log in again." || len(o.Actions) != 0 {
if o.Needs != "log out of g14 completely and log in again, or restart it." || len(o.Actions) != 0 {
t.Errorf("relogin: %q %+v", o.Needs, o.Actions)
}
w := conditions.Words{Headline: o.Headline, Explanation: o.Explanation, Resolved: o.Resolved, Needs: o.Needs}
@@ -107,6 +107,9 @@ func TestAModuleUnhealthyAsksForARestartInWords(t *testing.T) {
if why, ok := conditions.PlainWords(rw, "g14"); !ok || rw.Needs == "" || len(rw.Actions) != 0 {
t.Errorf("relogin-needed: %s %+v", why, rw)
}
if strings.Contains(o.Summary, "session") || !strings.Contains(o.Summary, "waits for a new login of jochen") {
t.Errorf("relogin summary: %q", o.Summary)
}
}
// **Failed units on a machine**: "shanks's service manager is degraded: 3 failed unit(s) no module places —
+6 -1
View File
@@ -419,7 +419,7 @@ func declarationWith(ctx context.Context, open *stores, node string,
out := sendable{Resources: composed.Resources, Adoption: adoption,
Received: composed.Received, Mesh: with.Mesh, BusUsers: with.BusUsers,
LeftOut: sortedKeysOf(composed.LeftOut), leftOutWhy: composed.LeftOut, withheld: with.Withheld,
unbound: with.Unbound, foreseen: composed.Foreseen}
unbound: with.Unbound, foreseen: composed.Foreseen, unplaced: composed.Unplaced}
// And which build of each module it carries, for the send to record (novox/hq issue 259, ADR
// 0221). Read only on the send path: a question about what would be sent records nothing.
if choosing == Allocating {
@@ -528,6 +528,11 @@ func reportLeftOut(node string, declared sendable) {
for _, u := range declared.unbound {
fmt.Printf("%s: %s\n", node, u)
}
// And every contribution its holder could not render, which the machine is sent without (novox/hq
// ADR 0255).
for _, u := range declared.unplaced {
fmt.Printf("%s: %s\n", node, u)
}
}
// busCredentialIssued refuses an own secret called `broker` whose bus account nobody issued.
+4
View File
@@ -165,6 +165,10 @@ func serve(ctx context.Context) (err error) {
// Open plans move on a timer as well as on outcomes (novox/hq ADR 0162): a tier waiting for
// machines to report moves when they have, and a plan left by a replaced controller resumes.
go planTicker(ctx, open)
// And the pending assignments settled on a tick of their own (novox/hq ADR 0261): made once their module
// is registered, ended with why when its build will not register it, raised and cleared as conditions.
// Never by a read.
go settlingPending(ctx, open)
// The durations the core's bounds are set from are kept a month (novox/hq to-be 45 Phase 0).
go forgettingOldDurations(ctx, inv)
// And what the catalogue decided a build meant. The builder's own result is already handled
+7 -1
View File
@@ -406,6 +406,8 @@ func rebuildCommand(ctx context.Context, args []string) error {
if err != nil {
return err
}
recordAsked(ctx, inventory.BuildRequest{ID: id, Repository: source.Repository, Seat: source.Seat, Path: path,
Ref: ref, For: "rebuild"})
fmt.Printf("rebuild asked as %s\n", id)
return nil
}
@@ -490,7 +492,11 @@ func replayCommand(ctx context.Context, args []string) error {
if err := replayRefusal(b, module, history, *register, *older, outstanding); err != nil {
return err
}
id, err := buildOneAsked(ctx, source, b.Path, b.Commit, 0, !*register)
asker := ""
if *register {
asker = "replay"
}
id, err := buildOneAsked(ctx, source, b.Path, b.Commit, 0, !*register, asker)
if err != nil {
return err
}
+23
View File
@@ -101,6 +101,24 @@ type meshStatus struct {
// Overflowing is every module whose identity overflows the bound of a provision it requires, and
// so is left out of its provider's grants (novox/hq ADR 0225). Absent when every identity fits.
Overflowing []catalogue.Overflow `json:"overflowing,omitempty"`
// Pending is every assignment waiting for its module's build to register it, and every one ended in
// the last day with what ended it (novox/hq issue 325). Absent when there is none; PendingUnread says
// why they could not be read.
Pending []pendingStatus `json:"pending,omitempty"`
PendingUnread string `json:"pendingUnread,omitempty"`
}
// pendingStatus is one pending assignment as status says it.
type pendingStatus struct {
Node string `json:"node"`
Module string `json:"module"`
State string `json:"state"`
Build string `json:"build"`
Repository string `json:"repository,omitempty"`
Path string `json:"path,omitempty"`
Since time.Time `json:"since"`
Settled *time.Time `json:"settled,omitempty"`
Note string `json:"note,omitempty"`
}
// machineFiltered is one rule set on a converged machine that the mesh did not write and that
@@ -240,6 +258,11 @@ func statusAsJSON(asked answers) ([]byte, error) {
out.Conditions, out.ConditionsUnread = inBrief(asked.conditions), asked.conditionsUnread
out.Failing = providerStandings(asked.conditions)
out.Overflowing = asked.overflowing
out.PendingUnread = asked.pendingUnread
for _, p := range asked.pending {
out.Pending = append(out.Pending, pendingStatus{Node: p.Node, Module: p.Module, State: p.State,
Build: p.Build, Repository: p.Repository, Path: p.Path, Since: p.Since, Settled: p.Settled, Note: p.Note})
}
for name := range asked.refused {
out.Unresolved = append(out.Unresolved, machineUnresolved{
Node: name, Problem: asked.refused[name]})
+1 -1
View File
@@ -209,7 +209,7 @@ func collectCommand(ctx context.Context, args []string) error {
if *most < 1 {
return fmt.Errorf("collect: --most is at least 1, not %d", *most)
}
bounds := sweepBounds{most: min(*most, collectMostAtMost), budget: collectBudget}
bounds := sweepBounds{most: min(*most, collectMostAtMost), budget: collectBudget, platforms: true}
open, err := openStores(ctx)
if err != nil {
@@ -133,6 +133,10 @@ func (f *fakeStore) serve(t *testing.T) artifacts.Store {
w.WriteHeader(http.StatusOK)
case r.Method == http.MethodHead:
w.WriteHeader(http.StatusNotFound)
case r.Method == http.MethodGet && strings.Contains(r.URL.Path, "/manifests/"):
// An image, not an index: it names no platform manifests.
w.Header().Set("Content-Type", "application/vnd.oci.image.manifest.v1+json")
_, _ = w.Write([]byte(`{"schemaVersion":2,"mediaType":"application/vnd.oci.image.manifest.v1+json","layers":[]}`))
case r.Method == http.MethodPost:
w.Header().Set("Location", "/upload/x?state=1")
w.WriteHeader(http.StatusAccepted)
+77 -5
View File
@@ -319,7 +319,7 @@ func passCarried(ctx context.Context, open *stores, p *inventory.Plan, g *invent
}
seen[c.Build] = true
if err := open.inventory.RecordGate(ctx, inventory.GateVerdict{Build: c.Build, Module: c.Module, Commit: c.To,
Previous: c.From, Plan: p.ID, Machines: []string{c.Node}, Verdict: inventory.GatePassed, Why: g.Why,
Previous: c.From, Plan: p.ID, Machines: []string{c.Node}, Verdict: inventory.GatePassed, Why: whyFor(g, c.Module),
Component: coreComponent(c.Module), JudgingFrom: g.Since}); err != nil {
fmt.Printf("%s: %s passed its gate on %s, and the verdict could not be kept: %v\n", p.ID, c.Module, c.Node, err)
}
@@ -334,6 +334,9 @@ func failCarried(ctx context.Context, open *stores, p *inventory.Plan, g *invent
notes = append(notes, p.Note)
}
done := map[string]bool{except: true}
for _, m := range g.Returned {
done[m] = true // put back at once when it broke
}
// **What passed on its own keeps its pass** (novox/hq ADR 0254, issue 318): healthy for the passes the
// gate asks while another module of the send failed, it is neither put back nor left unjudged — a build
// left without a verdict is one more move every other walk would wait for.
@@ -375,10 +378,7 @@ func keepPassing(ctx context.Context, open *stores, p *inventory.Plan, g *invent
continue
}
seen[c.Build] = true
why := fmt.Sprintf("healthy %d times on its own while the send failed: %s", g.Healthy[c.Module], g.Why)
if w := g.Waits[c.Module]; w != "" {
why += "; and it waits for a person: " + w
}
why := passedAloneWhy(g, c.Module)
if err := open.inventory.RecordGate(ctx, inventory.GateVerdict{Build: c.Build, Module: c.Module, Commit: c.To,
Previous: c.From, Plan: p.ID, Machines: []string{c.Node}, Verdict: inventory.GatePassed, Why: why,
Component: coreComponent(c.Module), JudgingFrom: g.Since}); err != nil {
@@ -393,6 +393,75 @@ func keepPassing(ctx context.Context, open *stores, p *inventory.Plan, g *invent
return kept
}
// putBackBroken puts back, at once, every module a gate found broken and has not put back yet (issue 318
// review): a witness that reverted a core component, or a machine that refused or failed its send, is not
// left registered at the build that broke for as long as the modules beside it take to be judged — a resend
// meanwhile would declare it again. The send goes on judging the others to their own verdicts; its own
// verdict is still failed. lead and leadState are the module a plan's gate is kept on, when it is a plan's.
// Answers whether anything was put back.
func putBackBroken(ctx context.Context, open *stores, p *inventory.Plan, g *inventory.PlanGate, lead string,
leadState *inventory.PlanModule) bool {
var todo []string
for _, m := range g.Broken {
if !slices.Contains(g.Returned, m) {
todo = append(todo, m)
}
}
if len(todo) == 0 || g.Verdict != "" {
return false
}
stateWas, noteWas := p.State, p.Note
batched, back := batchingRollbacks(ctx)
var said []string
for _, m := range todo {
machines := g.Machines
var state *inventory.PlanModule
if m == lead && leadState != nil {
judged := *leadState
judged.Gate = nil // its own record of the failure; the send's gate goes on judging
state = &judged
} else {
i := slices.IndexFunc(g.Carried, func(c inventory.CarriedMove) bool { return c.Module == m })
if i < 0 {
continue
}
c := g.Carried[i]
state = &inventory.PlanModule{Build: c.Build, Previous: c.From, Commit: c.To}
if sent, err := sentTheBuild(ctx, open, m, c.To); err == nil && len(sent) > 0 {
machines = sent
} else {
machines = []string{c.Node}
}
}
// Counted as put back only once there is something to put back (a carried move or the plan's own).
g.Returned = append(g.Returned, m)
p.Note = ""
gateFailed(batched, open, p, m, state, machines, g.BrokenWhy)
if m == lead && leadState != nil {
leadState.Why = "put back at once, its send still judged: " + g.BrokenWhy
}
said = append(said, m)
}
sendRollbacks(ctx, open, p, back)
p.State = stateWas
p.Note = noteWas
if len(said) > 0 {
p.Note = fmt.Sprintf("put back at once: %s (%s); the rest of the send judged to their own verdicts",
strings.Join(said, ", "), g.BrokenWhy)
fmt.Printf("%s: %s\n", p.ID, p.Note)
}
return true
}
// passedAloneWhy is the verdict of a module that passed on its own while its send failed, with its own wait.
func passedAloneWhy(g *inventory.PlanGate, module string) string {
why := fmt.Sprintf("healthy %d times on its own while the send failed: %s", g.Healthy[module], g.Why)
if w := g.Waits[module]; w != "" {
why += waitsPrefix + w
}
return why
}
// sentTheBuild is every machine running a module that was last sent this build of it.
func sentTheBuild(ctx context.Context, open *stores, module, commit string) ([]string, error) {
running, err := open.inventory.Running(ctx, module)
@@ -611,6 +680,9 @@ func advanceRelease(ctx context.Context, open *stores, p *inventory.Plan) (bool,
}
switch verdict {
case "":
if putBackBroken(ctx, open, p, g, "", nil) {
return true, nil
}
note := fmt.Sprintf("judging %s: %s", strings.Join(g.Machines, ", "), gateLine(g))
changed := p.Note != note
p.Note = note
+37 -6
View File
@@ -337,8 +337,16 @@ func askTier(ctx context.Context, inv *inventory.Inventory, p *inventory.Plan) e
// askABuild is how a plan asks for one build, not waited for, and learns the id it asked under. A
// variable so a test of what a plan does around an ask needs no build machine.
var askABuild = func(ctx context.Context, source buildSource, path, ref string) (string, error) {
return buildOneAsked(ctx, source, path, ref, 0, false)
//
// Set in init, not where it is declared: an assignment pending on a build asks for one too (novox/hq issue
// 325), and a build's outcome makes pending assignments, so the two refer to each other.
var askABuild func(ctx context.Context, source buildSource, path, ref string) (string, error)
func init() {
askABuild = func(ctx context.Context, source buildSource, path, ref string) (string, error) {
// Kept by each asker with its own name once the ask is made (novox/hq issue 325).
return buildOneAsked(ctx, source, path, ref, 0, false, "")
}
}
// askModule asks the build machine for one module of a plan and marks it asked, with the id it was
@@ -369,6 +377,8 @@ func askModule(ctx context.Context, p *inventory.Plan, name string, byName map[s
p.Note = fmt.Sprintf("%s could not be asked for: %v", name, err)
return
}
recordAsked(ctx, inventory.BuildRequest{ID: id, Repository: e.Source.Repository, Seat: e.Source.Seat,
Path: e.Source.Path, Ref: followedBranch(e.Source.Ref), Commit: p.Commit, For: "plan"})
state.State = "asked"
state.AskedAt = &now
state.Build = id
@@ -703,7 +713,7 @@ func advanceOnce(ctx context.Context, open *stores, p *inventory.Plan,
if lead.Gate.Verdict != inventory.GatePassed {
continue
}
passedWith(state, state.GatedBy, lead.Gate)
passedWith(m, state, state.GatedBy, lead.Gate)
}
switch {
case step.failed != "":
@@ -729,6 +739,9 @@ func advanceOnce(ctx context.Context, open *stores, p *inventory.Plan,
}
switch verdict {
case "":
if putBackBroken(ctx, open, p, state.Gate, m, state) {
return true, nil
}
pending = append(pending, fmt.Sprintf("%s judged on %s: %s", m,
strings.Join(state.Gate.Machines, ", "), gateLine(state.Gate)))
continue
@@ -937,14 +950,14 @@ func firstSend(ctx context.Context, open *stores, p *inventory.Plan, node string
// passedWith keeps, on a module sent with others, the verdict of the gate that judged the send: the gate
// on the first of them passed, and its pass was kept for every build it carried (passCarried).
func passedWith(s *inventory.PlanModule, lead string, g *inventory.PlanGate) {
func passedWith(module string, s *inventory.PlanModule, lead string, g *inventory.PlanGate) {
if s.Gate == nil {
s.Gate = &inventory.PlanGate{Machines: g.Machines, From: s.Previous, To: s.Commit, Since: g.Since}
}
if s.Gate.Verdict != "" {
return
}
s.Gate.Verdict, s.Gate.Why, s.Gate.JudgedAt, s.Gate.Took = g.Verdict, "judged with "+lead+": "+g.Why, g.JudgedAt, g.Took
s.Gate.Verdict, s.Gate.Why, s.Gate.JudgedAt, s.Gate.Took = g.Verdict, "judged with "+lead+": "+whyFor(g, module), g.JudgedAt, g.Took
s.Gate.Passes, s.Gate.Kept = g.Passes, true
}
@@ -961,8 +974,26 @@ func failFirstSend(ctx context.Context, open *stores, p *inventory.Plan, module
fmt.Printf("%s: %s\n", p.ID, p.Note)
}()
g := state.Gate
if g == nil || g.Verdict == "" || len(g.Failing) == 0 || slices.Contains(g.Failing, module) {
if g != nil && slices.Contains(g.Returned, module) {
// Put back at once when it broke: its rollback was made then, and is not made again.
state.Why = "put back when it broke; its send failed: " + g.Why
p.State = inventory.PlanFailed
p.Note = fmt.Sprintf("the send to %s in tier %d failed its gate: %s; %s was put back when it broke",
strings.Join(g.Machines, ", "), p.Tier, g.Why, module)
} else if g == nil || g.Verdict == "" || len(g.Failing) == 0 || slices.Contains(g.Failing, module) {
gateFailed(batched, open, p, module, state, machines, why)
} else if slices.Contains(g.Passing, module) && state.Build != "" {
// **The module the gate is kept on keeps its own pass too** (issue 318 review): healthy for the passes
// asked while another of its send failed, its build has a verdict, and other walks do not wait on it.
if err := open.inventory.RecordGate(ctx, inventory.GateVerdict{Build: state.Build, Module: module,
Commit: state.Commit, Previous: state.Previous, Plan: p.ID, Machines: g.Machines, Verdict: inventory.GatePassed,
Why: passedAloneWhy(g, module), Component: g.Component, JudgingFrom: g.Since}); err != nil {
fmt.Printf("%s: %s passed its gate on its own, and the verdict could not be kept: %v\n", p.ID, module, err)
}
state.Why = "passed on its own; stopped with the send that carried it: " + g.Why
p.State = inventory.PlanFailed
p.Note = fmt.Sprintf("the send to %s in tier %d failed its gate: %s; %s passed on its own and is kept",
strings.Join(g.Machines, ", "), p.Tier, g.Why, module)
} else {
state.Why = "not found wanting; stopped with the send that carried it: " + g.Why
p.State = inventory.PlanFailed
+8 -2
View File
@@ -40,13 +40,19 @@ func TestReplay318(t *testing.T) {
t.Cleanup(func() { doctorFrom = was })
build := func(module, repository, commit string, asked time.Time) {
manifest, _ := json.Marshal(catalogue.Manifest{Module: module, Version: "1"})
m := catalogue.Manifest{Module: module, Version: "1"}
if module == "openrazer" && commit == "c78b5fc9" {
// The build that put the operator's account in the group `openrazer` (ADR 0252).
m.Resources = []map[string]any{{"id": "operator", "type": "user", "name": "operator",
"groups": []any{"openrazer"}}}
}
manifest, _ := json.Marshal(m)
if err := inv.RecordBuild(ctx, inventory.Build{ID: "build-" + module + "-" + commit, Module: module, Commit: commit,
Repository: repository, Path: "modules/" + module, Manifest: manifest, Asked: asked, At: asked,
Made: []inventory.Artifact{{Name: "x", Kind: "bundle", Reference: "sha256:" + module + "-" + commit}}}); err != nil {
t.Fatal(err)
}
if err := inv.RegisterModule(ctx, catalogue.Manifest{Module: module, Version: "1"}, inventory.Source{
if err := inv.RegisterModule(ctx, m, inventory.Source{
Repository: repository, Seat: "git", Path: "modules/" + module, BuiltFrom: commit, Head: commit,
Asked: asked}); err != nil {
t.Fatal(err)
+60
View File
@@ -0,0 +1,60 @@
package main
import (
"encoding/json"
"slices"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// novox/hq issue 325, replayed with only what the controller had before its fix, so it can be laid over the
// older commit. On 2026-10-08 the catalogue's pull request adding `sensors` merged, and the merge asked for
// its build (issue 300). About a minute later `assign g14 sensors` answered "no module of that name:
// sensors"; a few minutes later the same call worked, because the build had registered the module. The
// answer read as "nobody registered it". An assignment made while the build runs says the build — where it
// is from, since when, its id — is kept, and is made when the build registers the module.
func TestReplay325(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
if err := open.inventory.RegisterModule(ctx, catalogue.Manifest{Module: "networkmanager", Version: "1"},
inventory.Source{Repository: "novox/mesh-catalog", Seat: "git", Path: "modules/networkmanager", Ref: "main",
BuiltFrom: "c0", Head: "c0"}); err != nil {
t.Fatal(err)
}
asksWithPaths(t)
m := link.SourceMoved{Owner: "novox", Repo: "mesh-catalog", Base: "main", Commit: "5e450125aa",
Paths: []string{"modules/sensors/module.json", "modules/sensors/cmd/sensors/main.go"},
ModuleDirs: []string{"modules/sensors"}, ModuleDirsSaid: true}
if err := (following{open: open}).SourceMoved(ctx, m); err != nil {
t.Fatal(err)
}
said, err := assign(ctx, open, "laptop", "sensors")
if err != nil {
t.Fatalf("assign while the merge's build runs was refused: %v", err)
}
for _, want := range []string{"being built", "novox/mesh-catalog", "modules/sensors", "b-modules/sensors"} {
if !strings.Contains(said, want) {
t.Fatalf("the answer does not say %q:\n%s", want, said)
}
}
manifest, _ := json.Marshal(catalogue.Manifest{Module: "sensors", Version: "1"})
if err := (builds{open.inventory, open}).Built(ctx, link.BuildResult{ID: "b-modules/sensors",
Repository: "novox/mesh-catalog", Path: "modules/sensors", Ref: "main", Commit: "5e450125aa", On: "anchor",
Module: "sensors", Manifest: manifest,
Source: &link.SourceOnSeat{Repository: "novox/mesh-catalog", Seat: "git"}}); err != nil {
t.Fatal(err)
}
assigned, err := open.inventory.Assigned(ctx, "laptop")
if err != nil {
t.Fatal(err)
}
if !slices.Contains(assigned, "sensors") {
t.Fatalf("the build registered sensors and laptop runs %v", assigned)
}
}
+34 -2
View File
@@ -418,7 +418,13 @@ func (a *verbArguments) commandLine() ([]string, error) {
}
// Several modules comma-separated, judged as one act (novox/hq ADR 0207): the holders of
// the seats that apply resources depend on each other and go on together.
return append([]string{verb, str("node")}, splitModules(str("module"))...), nil
argv := append([]string{verb, str("node")}, splitModules(str("module"))...)
// A module known and not built: its build asked for, the assignment pending on it (novox/hq
// issue 325). unassign declares no build, so a call giving it is refused before this.
if verb == "assign" && on("build") {
argv = append(argv, "--build")
}
return argv, nil
case "pin":
if err := need("node", "provision", "from", "module"); err != nil {
return nil, err
@@ -605,6 +611,30 @@ func (a *verbArguments) commandLine() ([]string, error) {
return nil, err
}
return []string{"images", str("node"), "--json"}, nil
case "mirrors":
// novox/hq ADR 0257.
argv := []string{"mirrors", "--json"}
record := str("record")
why := str("why")
if record == "" {
if on("confirm") || why != "" {
return nil, errors.New("mirrors takes confirm and why only with record: there is nothing " +
"else it records. Nothing was done")
}
return argv, nil
}
argv = append(argv, "--record", record)
if !on("confirm") {
if why != "" {
return nil, errors.New("mirrors takes why only with confirm: without confirm it is a dry run, " +
"and a reason for nothing would be recorded nowhere. Nothing was done")
}
return argv, nil
}
if err := need("why"); err != nil {
return nil, fmt.Errorf("%w: recording a copy as the mesh's is a hand act, which says why. Nothing was done", err)
}
return append(argv, "--confirm", "--why", why), nil
case "data":
argv := []string{"data", "--json"}
if m := str("machine"); m != "" {
@@ -804,7 +834,7 @@ func (a *verbArguments) commandLine() ([]string, error) {
var jsonVerbs = map[string]bool{"status": true, "seats": true, "plan": true, "collection": true,
"hand-acts": true, "durations": true, "conditions": true, "doctor": true, "retire": true, "cleanup": true, "data": true,
// What the records say the registries may keep (novox/hq ADR 0251).
"artifacts": true, "collect": true, "images": true,
"artifacts": true, "collect": true, "images": true, "mirrors": true,
// The delivery's owner's verbs answer JSON where they read (plan, order, check, walks) — novox/hq ADR 0239.
"delivery": true}
@@ -835,6 +865,8 @@ func repairingCommand(argv []string) string {
return "cleanup delete"
case argv[0] == "collect" && slices.Contains(argv, "--confirm"):
return "collect"
case argv[0] == "mirrors" && slices.Contains(argv, "--confirm"):
return "mirrors"
}
return ""
}
@@ -284,6 +284,7 @@ var accountedFlags = map[string]map[string]string{
"artifacts": {"json": "set by the verb: the answer is data"},
"collect": {"json": "set by the verb: the answer is data"},
"images": {"json": "set by the verb: the answer is data"},
"mirrors": {"json": "set by the verb: the answer is data"},
"conditions history": {"json": "set by the verb: the answer is data"},
"conditions show": {"json": "set by the verb: the answer is data"},
"healers": {"json": "set by the verb: the answer is data"},
+3
View File
@@ -58,6 +58,9 @@ type sendable struct {
// make (Foreseeing, novox/hq issue 275). Never on the wire, and a declaration that has any is never
// sent.
foreseen []string
// unplaced is every contribution its holder's template could not render, naming its module and
// why (novox/hq ADR 0255): left out of this declaration, for push and plan to say, never on the wire.
unplaced []string
// Builds is the build of each module this declaration carries — module to the commit its build
// was made from — recorded with the send and never on the wire (novox/hq issue 259, ADR 0221).
// Composed only on the send path; nil records that it is not known.
+56 -1
View File
@@ -143,6 +143,8 @@ func printStatus(asked answers) error {
len(quiet), strings.Join(said, "\n "))
}
printPending(asked.pending, asked.pendingUnread)
if open, late := openPlans(asked.plans, time.Now(), asked.paused, asked.tierBounds); len(open) > 0 {
fmt.Printf("%d plan(s) open", len(open))
if late > 0 {
@@ -466,6 +468,12 @@ func theThreeQuestions(ctx context.Context, open *stores) (answers, error) {
if out.conditions, err = openConditions(ctx); err != nil {
out.conditionsUnread = err.Error()
}
// And the assignments waiting for their module's build (novox/hq issue 325, ADR 0261), read only: the
// controller's tick settles them, never a read.
if out.pending, err = inv.Pending(ctx, time.Now().Add(-pendingShownFor)); err != nil {
out.pendingUnread = err.Error()
err = nil
}
out.plans, err = inv.RecentPlans(ctx, 5)
if err != nil {
return answers{}, err
@@ -598,7 +606,54 @@ func (a answers) well() bool {
return len(a.wrong) == 0 && len(a.quiet) == 0 && len(a.behind) == 0 &&
len(a.waiting) == 0 && len(a.refused) == 0 && a.network == "" && len(a.untaken) == 0 &&
len(a.filtered) == 0 && len(a.unheld) == 0 && len(a.overflowing) == 0 &&
len(a.conditions) == 0 && a.conditionsUnread == ""
len(a.conditions) == 0 && a.conditionsUnread == "" && a.pendingUnread == "" && !pendingOpen(a.pending)
}
// pendingOpen is whether any pending assignment is still to be made. One that ended without being made is
// not counted here: it is a condition, open until it is answered (ADR 0261).
func pendingOpen(pending []inventory.PendingAssignment) bool {
for _, p := range pending {
if p.Open() {
return true
}
}
return false
}
// printPending says the assignments waiting for their module's build, and those ended in the last day
// (novox/hq issue 325).
func printPending(pending []inventory.PendingAssignment, unread string) {
if unread != "" {
fmt.Printf("the assignments waiting for a build could not be read: %s\n\n", unread)
return
}
var waiting, ended []inventory.PendingAssignment
for _, p := range pending {
if p.Open() {
waiting = append(waiting, p)
} else {
ended = append(ended, p)
}
}
if len(waiting) > 0 {
fmt.Printf("%d assignment(s) wait for their module's build to register it:\n", len(waiting))
for _, p := range waiting {
fmt.Printf(" %-12s %-20s build %s of %s %s, since %s\n", p.Node, p.Module, p.Build, p.Repository,
orRoot(p.Path), clock(p.Since))
}
fmt.Printf("\n each is made when its build registers the module; `unassign <node> <module>` withdraws it\n\n")
}
if len(ended) > 0 {
fmt.Printf("%d pending assignment(s) ended in the last day:\n", len(ended))
for _, p := range ended {
at := ""
if p.Settled != nil {
at = clock(*p.Settled)
}
fmt.Printf(" %-12s %-20s %-9s %s\n %-12s %s\n", p.Node, p.Module, p.State, at, "", p.Note)
}
fmt.Println()
}
}
// hostSplit is which machines report which host version, for every version more than one machine
+64 -2
View File
@@ -255,8 +255,13 @@ func TestACoreBehindWhileTheNodeToolsSettleFailsNoOtherModule(t *testing.T) {
t.Fatalf("%s's build was marked failed for the node tools' condition", m)
}
}
if !strings.Contains(p.Note, "app1 was not found wanting") {
t.Fatalf("the plan blames the module its gate was kept on: %s", p.Note)
// Healthy for the passes asked, the module the gate is kept on keeps its own pass (novox/hq issue 318
// review): it is not blamed, and not left without a verdict.
if !strings.Contains(p.Note, "app1 passed on its own and is kept") {
t.Fatalf("the plan blames the module its gate was kept on, or leaves it unjudged: %s", p.Note)
}
if v, found, _ := inv.GateOf(ctx, "build-app1-2"); !found || v.Verdict != inventory.GatePassed {
t.Fatalf("app1's own pass was not kept: %+v", v)
}
}
@@ -373,3 +378,60 @@ func TestAModuleItsSendChangedNothingOfIsLeftAsItWasAndRetried(t *testing.T) {
t.Fatalf("retried as %q: the plan is %s, app %+v", said, p.State, s)
}
}
// **In a plan's tier path, a broken module is put back at once too** (novox/hq issue 318 review): the node
// tools' witness reverts them on the first machine. Whether the gate is kept on them or on the module beside
// them, they are registered back and marked in the judging that finds them broken, the plan goes on judging
// the other module to its own pass, and only then fails.
func TestATierPutsABrokenModuleBackAtOnce(t *testing.T) {
for _, order := range [][]string{{broker.RuntimeModule, "app1"}, {"app1", broker.RuntimeModule}} {
t.Run("gate kept on "+order[0], func(t *testing.T) {
tm := aTierMesh(t, order...)
ctx := t.Context()
inv := tm.open.inventory
tierFacts := gatherGateFacts
var seen []string
gatherGateFacts = func(ctx context.Context, open *stores, component string) (gateFacts, error) {
f, err := tierFacts(ctx, open, component)
current, _ := open.inventory.CurrentBuilds(ctx)
seen = append(seen, current[broker.RuntimeModule].Commit)
if current[broker.RuntimeModule].Commit == "c2" {
f.rolledBack["anchor"] = []lease.Rollback{{Component: lease.ComponentNodeTools,
Outcome: lease.OutcomeRolledBack, From: "c2", To: "c1", At: time.Now(), Why: "the node tools did not answer"}}
}
return f, err
}
gateEvery, gateSettle, gateBound = 0, 100*time.Millisecond, 10*time.Second
for i := 0; i < 20 && len(seen) < 2; i++ {
advancePlans(ctx, tm.open)
}
if len(seen) < 2 || seen[0] != "c2" || seen[1] != "c1" {
t.Fatalf("the node tools' registered build at each judging: %v; want c2, then c1 at once", seen)
}
if failed, _ := inv.GateFailed(ctx, "build-"+broker.RuntimeModule+"-2"); !failed {
t.Fatal("the node tools' build is not marked failed at once")
}
if p := tm.plan(t); p.State == inventory.PlanFailed {
t.Fatalf("the plan failed at once: %s; want it judging app1 first", p.Note)
}
for deadline := time.Now().Add(5 * time.Second); time.Now().Before(deadline); {
advancePlans(ctx, tm.open)
if tm.plan(t).State == inventory.PlanFailed {
break
}
time.Sleep(20 * time.Millisecond)
}
p := tm.plan(t)
if p.State != inventory.PlanFailed {
t.Fatalf("the plan is %s: %s; want failed, for the node tools", p.State, p.Note)
}
if v, found, _ := inv.GateOf(ctx, "build-app1-2"); !found || v.Verdict != inventory.GatePassed {
t.Fatalf("app1's verdict: %+v; want its own pass (plan: %s)", v, p.Note)
}
if current, _ := inv.CurrentBuilds(ctx); current["app1"].Commit != "c2" || current[broker.RuntimeModule].Commit != "c1" {
t.Fatalf("registered app1 %s, node tools %s; want c2 and c1", current["app1"].Commit,
current[broker.RuntimeModule].Commit)
}
})
}
}
+11 -2
View File
@@ -360,7 +360,7 @@ func (f following) SourceMoved(ctx context.Context, m link.SourceMoved) error {
}
// **A new module is built and registered, and sent nowhere** (novox/hq issue 300): the delivery plan
// says so, and assigning it is a person's act, which needs it registered first.
built := askNewModules(ctx, m, from, added)
built := askNewModules(ctx, inv, m, from, added)
moved := append(append([]inventory.Entry{}, touched...), packaging...)
if len(moved) == 0 {
if len(built) > 0 {
@@ -491,7 +491,8 @@ func (f following) SourceMoved(ctx context.Context, m link.SourceMoved) error {
// new module either. Outside the plan: the plan walks modules the catalogue holds, and a new one has no
// machine to send to and nothing standing on it. What could not be asked is said, and left to `build`.
// Returns the directories asked for.
func askNewModules(ctx context.Context, m link.SourceMoved, from []inventory.Entry, added []string) []string {
func askNewModules(ctx context.Context, inv *inventory.Inventory, m link.SourceMoved, from []inventory.Entry,
added []string) []string {
if len(added) == 0 || len(from) == 0 {
return nil
}
@@ -503,11 +504,19 @@ func askNewModules(ctx context.Context, m link.SourceMoved, from []inventory.Ent
path = ""
}
id, err := askABuild(ctx, source, path, m.Base)
// Kept, asked or not, so `assign` can tell a build in flight, or a merge that could not ask for one,
// from a module nobody ever heard of (novox/hq issue 325).
request := inventory.BuildRequest{ID: id, Repository: source.Repository, Seat: source.Seat, Path: path,
Ref: m.Base, Commit: m.Commit, For: "merge"}
if err != nil {
request.ID = fmt.Sprintf("not-asked-%s-%s", short(m.Commit), strings.ReplaceAll(dir, "/", "-"))
request.NotAsked = err.Error()
recordBuildRequest(ctx, inv, request)
fmt.Printf(" %s is a new module in %s/%s and could not be built: %v — a hand `build` of it "+
"asks again\n", dir, m.Owner, m.Repo, err)
continue
}
recordBuildRequest(ctx, inv, request)
fmt.Printf(" %s is a new module in %s/%s: build %s asked at %s; registered when it lands, assigned "+
"nowhere\n", dir, m.Owner, m.Repo, id, m.Base)
asked = append(asked, dir)
+176
View File
@@ -0,0 +1,176 @@
package artifacts
import (
"context"
"errors"
"fmt"
"net/http"
"net/http/httptest"
"slices"
"strings"
"sync"
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
)
// An index is let go of with the platform manifests it names, except those a kept index of the same
// repository names too (novox/hq ADR 0257). Each platform manifest is a manifest of the repository in
// its own right, and the store's collector keeps every manifest a repository holds — so an index let go
// of alone frees nothing of the images it names.
func digestN(n int) string { return fmt.Sprintf("sha256:%064x", n) }
// indexStore holds indexes and images in one repository, answers GETs with their documents, and records
// every delete.
type indexStore struct {
mu sync.Mutex
indexes map[string][]string // digest → the platform manifests it names
images map[string]bool
deleted []string
}
func (s *indexStore) serve(t *testing.T) Store {
t.Helper()
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
s.mu.Lock()
defer s.mu.Unlock()
digest := r.URL.Path[strings.LastIndex(r.URL.Path, "/")+1:]
switch r.Method {
case http.MethodGet:
if children, ok := s.indexes[digest]; ok {
var named []string
for _, c := range children {
named = append(named, `{"mediaType":"application/vnd.oci.image.manifest.v1+json","digest":"`+c+`","size":1}`)
}
w.Header().Set("Content-Type", "application/vnd.oci.image.index.v1+json")
_, _ = w.Write([]byte(`{"schemaVersion":2,"mediaType":"application/vnd.oci.image.index.v1+json","manifests":[` +
strings.Join(named, ",") + `]}`))
return
}
if s.images[digest] {
w.Header().Set("Content-Type", "application/vnd.oci.image.manifest.v1+json")
_, _ = w.Write([]byte(`{"schemaVersion":2,"mediaType":"application/vnd.oci.image.manifest.v1+json","layers":[]}`))
return
}
w.WriteHeader(http.StatusNotFound)
case http.MethodHead:
if _, ok := s.indexes[digest]; ok || s.images[digest] {
w.WriteHeader(http.StatusOK)
return
}
w.WriteHeader(http.StatusNotFound)
case http.MethodDelete:
if _, ok := s.indexes[digest]; !ok && !s.images[digest] {
w.WriteHeader(http.StatusNotFound)
return
}
s.deleted = append(s.deleted, digest)
delete(s.indexes, digest)
delete(s.images, digest)
w.WriteHeader(http.StatusAccepted)
default:
w.WriteHeader(http.StatusBadRequest)
}
}))
t.Cleanup(server.Close)
return Store{Address: strings.TrimPrefix(server.URL, "http://")}
}
func TestAnIndexGoesWithItsPlatformsAndAKeptIndexKeepsItsOwn(t *testing.T) {
// Two indexes of one image's repository: the old one names platforms 11 and 12, the kept one 12 and 13.
s := &indexStore{
indexes: map[string][]string{digestN(1): {digestN(11), digestN(12)}, digestN(2): {digestN(12), digestN(13)}},
images: map[string]bool{digestN(11): true, digestN(12): true, digestN(13): true},
}
store := s.serve(t)
repository := "upstream/docker.io/library/golang"
old := catalogue.ArtifactStoreScheme + repository + "@" + digestN(1)
kept := catalogue.ArtifactStoreScheme + repository + "@" + digestN(2)
spare, err := store.SpareKeptIndexes(context.Background(), []string{kept}, []string{old})
if err != nil {
t.Fatal(err)
}
store.Spare = spare
if err := store.LetGo(context.Background(), old); err != nil {
t.Fatal(err)
}
// Its own platform first, the index last; the platform the kept index names is left in place.
if !slices.Equal(s.deleted, []string{digestN(11), digestN(1)}) {
t.Fatalf("deleted %v; want the old index's own platform, then the index", s.deleted)
}
if !s.images[digestN(12)] || !s.images[digestN(13)] {
t.Fatal("a platform a kept index names was let go of")
}
}
func TestWithoutSparingAnIndexGoesAloneAndAnImageNamesNoPlatforms(t *testing.T) {
s := &indexStore{indexes: map[string][]string{digestN(1): {digestN(11)}}, images: map[string]bool{digestN(11): true, digestN(5): true}}
store := s.serve(t)
// No Spare: as before ADR 0257, the index alone.
if err := store.LetGo(context.Background(), catalogue.ArtifactStoreScheme+"web/base@"+digestN(1)); err != nil {
t.Fatal(err)
}
if !slices.Equal(s.deleted, []string{digestN(1)}) {
t.Fatalf("without a Spare, deleted %v", s.deleted)
}
// An image is one delete, with a Spare or without.
s.deleted = nil
spare, err := store.SpareKeptIndexes(context.Background(), nil, []string{catalogue.ArtifactStoreScheme + "web/app@" + digestN(5)})
if err != nil {
t.Fatal(err)
}
store.Spare = spare
if err := store.LetGo(context.Background(), catalogue.ArtifactStoreScheme+"web/app@"+digestN(5)); err != nil {
t.Fatal(err)
}
if !slices.Equal(s.deleted, []string{digestN(5)}) {
t.Fatalf("an image was let go of as %v", s.deleted)
}
// An index the store no longer holds is Gone, and nothing is deleted on its account.
s.deleted = nil
err = store.LetGo(context.Background(), catalogue.ArtifactStoreScheme+"web/base@"+digestN(9))
if !errors.Is(err, Gone) || len(s.deleted) != 0 {
t.Fatalf("an index the store does not hold: %v, deleted %v", err, s.deleted)
}
}
func TestASpareThatCannotBeReadKeepsTheIndex(t *testing.T) {
s := &indexStore{indexes: map[string][]string{digestN(1): {digestN(11)}}, images: map[string]bool{digestN(11): true}}
store := s.serve(t)
store.Spare = func(context.Context, string) (map[string]bool, error) { return nil, fmt.Errorf("the store went away") }
err := store.LetGo(context.Background(), catalogue.ArtifactStoreScheme+"web/base@"+digestN(1))
if err == nil || !strings.Contains(err.Error(), "was kept") {
t.Fatalf("a spare that could not be read: %v", err)
}
if len(s.deleted) != 0 {
t.Fatalf("deleted %v without knowing what a kept index names", s.deleted)
}
}
// The spare list is read whole before the sweep: a kept index that cannot be read is an error before
// anything is deleted, and a repository not read before keeps its index.
func TestASpareListIsReadWholeBeforeAnythingGoes(t *testing.T) {
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusInternalServerError)
}))
t.Cleanup(server.Close)
store := Store{Address: strings.TrimPrefix(server.URL, "http://")}
kept := catalogue.ArtifactStoreScheme + "web/base@" + digestN(2)
old := catalogue.ArtifactStoreScheme + "web/base@" + digestN(1)
if _, err := store.SpareKeptIndexes(context.Background(), []string{kept}, []string{old}); err == nil {
t.Fatal("a kept index the store would not answer for was taken as naming nothing")
}
s := &indexStore{indexes: map[string][]string{digestN(1): {digestN(11)}}, images: map[string]bool{digestN(11): true}}
good := s.serve(t)
spare, err := good.SpareKeptIndexes(context.Background(), nil, []string{catalogue.ArtifactStoreScheme + "other/repo@" + digestN(3)})
if err != nil {
t.Fatal(err)
}
good.Spare = spare
if err := good.LetGo(context.Background(), old); err == nil || len(s.deleted) != 0 {
t.Fatalf("an index in a repository not read before the sweep was let go of: %v, deleted %v", err, s.deleted)
}
}
+170
View File
@@ -9,8 +9,10 @@ package artifacts
import (
"context"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"strings"
@@ -24,6 +26,10 @@ type Store struct {
Address string
// HTTP is the client used; nil is a client with a modest timeout.
HTTP *http.Client
// Spare says, for one repository, which manifests a kept index there names: the platform
// manifests an index being let go of must leave in place (novox/hq ADR 0257). Nil spares every
// platform manifest — an index is then let go of alone, as it was before.
Spare func(ctx context.Context, repository string) (map[string]bool, error)
}
// Gone is the answer when the store does not hold it: the outcome wanted, already true.
@@ -77,9 +83,142 @@ func (s Store) LetGo(ctx context.Context, reference string) error {
return err
}
}
if kind == "manifests" && s.Spare != nil {
// **An index's platform manifests go before the index** (novox/hq ADR 0257). Each was put
// under its own digest, so each is a manifest of the repository in its own right, and the
// store's collector keeps every manifest a repository holds: an index let go of alone frees
// no byte of the images it names. Before, not after, so a sweep cut short leaves the index
// to be asked about again, and with it the list of what is still to go.
if err := s.letGoOfPlatforms(ctx, repository, digest, reference); err != nil {
return err
}
}
return s.remove(ctx, s.url(repository, kind, digest), reference)
}
// letGoOfPlatforms deletes the manifests an index names, except those a kept index of the same
// repository also names. A manifest that is not an index names none.
func (s Store) letGoOfPlatforms(ctx context.Context, repository, digest, reference string) error {
children, err := s.Platforms(ctx, repository, digest)
if err != nil || len(children) == 0 {
// Gone: there is no index to read, and the delete that follows says so.
if errors.Is(err, Gone) {
return nil
}
return err
}
spared, err := s.Spare(ctx, repository)
if err != nil {
return fmt.Errorf("cannot say which platform manifests of %s a kept index names, so %s was kept: %w",
repository, reference, err)
}
for _, child := range children {
if spared[child] {
continue
}
if err := s.remove(ctx, s.url(repository, "manifests", child), reference+" (its "+child+")"); err != nil && !errors.Is(err, Gone) {
return err
}
}
return nil
}
// anyManifest is every manifest the store may hold, indexes included, so it answers with the
// document as it is.
var anyManifest = []string{
"application/vnd.oci.image.index.v1+json",
"application/vnd.docker.distribution.manifest.list.v2+json",
"application/vnd.oci.image.manifest.v1+json",
"application/vnd.docker.distribution.manifest.v2+json",
}
// Platforms is the manifests an index names, by digest; none for a manifest that is not an index.
// Gone when the store does not hold it.
func (s Store) Platforms(ctx context.Context, repository, digest string) ([]string, error) {
request, err := http.NewRequestWithContext(ctx, http.MethodGet, s.url(repository, "manifests", digest), nil)
if err != nil {
return nil, err
}
request.Header.Set("Accept", strings.Join(anyManifest, ", "))
response, err := s.client().Do(request)
if err != nil {
return nil, err
}
defer response.Body.Close()
switch response.StatusCode {
case http.StatusOK:
case http.StatusNotFound:
return nil, Gone
default:
return nil, fmt.Errorf("the artifact store answered %s when asked for %s@%s", response.Status, repository, digest)
}
var document struct {
Manifests []struct {
Digest string `json:"digest"`
} `json:"manifests"`
}
if err := json.NewDecoder(io.LimitReader(response.Body, 4<<20)).Decode(&document); err != nil {
return nil, fmt.Errorf("%s@%s is not a manifest: %w", repository, digest, err)
}
var out []string
for _, m := range document.Manifests {
if strings.HasPrefix(m.Digest, "sha256:") {
out = append(out, m.Digest)
}
}
return out, nil
}
// SpareKeptIndexes is a Spare for a sweep, read whole before it starts: for every repository an
// image reference of `touching` is in, each kept manifest there and every platform manifest a kept
// index there names. Any read that fails is an error, and the sweep lets nothing go. A repository not
// read here is answered with an error, so an index in it is kept. A kept index the store no longer
// holds names nothing.
func (s Store) SpareKeptIndexes(ctx context.Context, kept, touching []string) (func(context.Context, string) (map[string]bool, error), error) {
imageIn := func(reference string) (repository, digest string, ok bool) {
path, ours := catalogue.InArtifactStore(reference)
if !ours {
return "", "", false
}
repository, kind, digest, err := split(path)
if err != nil || kind != "manifests" {
return "", "", false
}
return repository, digest, true
}
read := map[string]map[string]bool{}
for _, reference := range touching {
if repository, _, ok := imageIn(reference); ok {
read[repository] = map[string]bool{}
}
}
for _, reference := range kept {
repository, digest, ok := imageIn(reference)
spared, touched := read[repository]
if !ok || !touched {
continue
}
spared[digest] = true
children, err := s.Platforms(ctx, repository, digest)
if errors.Is(err, Gone) {
continue
}
if err != nil {
return nil, fmt.Errorf("reading the kept %s@%s: %w", repository, digest, err)
}
for _, c := range children {
spared[c] = true
}
}
return func(_ context.Context, repository string) (map[string]bool, error) {
spared, done := read[repository]
if !done {
return nil, fmt.Errorf("%s was not read before the sweep began", repository)
}
return spared, nil
}, nil
}
// remove asks the store to delete what is at url. Gone when it has no such thing.
func (s Store) remove(ctx context.Context, url, what string) error {
request, err := http.NewRequestWithContext(ctx, http.MethodDelete, url, nil)
@@ -121,3 +260,34 @@ func split(path string) (repository, kind, digest string, err error) {
}
return "", "", "", fmt.Errorf("%w: %q names nothing the store holds by digest", ErrNotOurs, path)
}
// IsIndex is whether a recorded reference is an index the store holds: one that names manifests. A
// blob, or a manifest that names none, is not. Gone when the store does not hold it.
func (s Store) IsIndex(ctx context.Context, reference string) (bool, error) {
path, ours := catalogue.InArtifactStore(reference)
if !ours {
return false, fmt.Errorf("%w: %s", ErrNotOurs, reference)
}
repository, kind, digest, err := split(path)
if err != nil || kind != "manifests" {
return false, err
}
children, err := s.Platforms(ctx, repository, digest)
return len(children) > 0, err
}
// HoldsManifest is whether the store holds the manifest a recorded image reference names.
func (s Store) HoldsManifest(ctx context.Context, reference string) (bool, error) {
path, ours := catalogue.InArtifactStore(reference)
if !ours {
return false, fmt.Errorf("%w: %s", ErrNotOurs, reference)
}
repository, kind, digest, err := split(path)
if err != nil {
return false, err
}
if kind != "manifests" {
return false, fmt.Errorf("%w: %s names a blob, not a manifest", ErrNotOurs, reference)
}
return s.has(ctx, s.url(repository, kind, digest), anyManifest...)
}
+42 -5
View File
@@ -77,6 +77,11 @@ type Result struct {
// the default, and that branch is its trunk.
Branches []string
// Mirrored is every base this build copied into the artifact store, as pinned (novox/hq ADR 0257):
// said whether the build worked or not, because the copy is in the store either way, and the
// records are what decide whether it stays.
Mirrored []string
// Source is the build's source fingerprint (source.go): what it was made from — the module's tree,
// the contexts' trees, the bases and toolchains by digest — hashed. Empty where the source does not
// pin the build. Two builds with one fingerprint are one build, whatever digests they made
@@ -106,6 +111,19 @@ type GitCredential struct {
func Build(ctx context.Context, run Runner, publish Publisher,
repository, path, ref, workspace string, held map[string]string, npmrc Npmrc,
forge GitCredential, log Log, seats ...map[string]string) (Result, error) {
// **What was copied is said whether the build worked or not** (novox/hq ADR 0257): a base is in
// the store from the moment it is copied, and a build that failed after copying it is the
// one record that it is there.
var mirrored []string
result, err := build(ctx, run, publish, repository, path, ref, workspace, held, npmrc, forge, log,
&mirrored, seats...)
result.Mirrored = mirrored
return result, err
}
func build(ctx context.Context, run Runner, publish Publisher,
repository, path, ref, workspace string, held map[string]string, npmrc Npmrc,
forge GitCredential, log Log, mirrored *[]string, seats ...map[string]string) (Result, error) {
// The clone base of each seat a context may name (novox/hq ADR 0155); variadic so the callers
// that hand none — tests of everything but contexts — read as they did.
var seatBases map[string]string
@@ -222,10 +240,22 @@ func Build(ctx context.Context, run Runner, publish Publisher,
// An image published elsewhere that the build stands on is copied into the mesh's own
// registry first, like an upstream artifact (ADR 0096), and the recipe is handed the copy.
// Genesis has nowhere to copy to and pulls it into this machine's store instead.
mirror := func(ctx context.Context, from, repository string) (string, error) {
mirror := func(ctx context.Context, from, repository, former string) (string, error) {
if m, can := publish.(BaseMirrorer); can {
say("bases", "copying %s into the mesh's registry as %s", from, repository)
reference, err := m.MirrorBase(ctx, from, repository, former)
if err == nil {
*mirrored = append(*mirrored, reference)
}
return reference, err
}
if m, can := publish.(Mirrorer); can {
say("bases", "copying %s into the mesh's registry", from)
return m.MirrorImage(ctx, from, repository)
say("bases", "copying %s into the mesh's registry as %s", from, repository)
reference, err := m.MirrorImage(ctx, from, repository)
if err == nil {
*mirrored = append(*mirrored, reference)
}
return reference, err
}
if _, err := run(ctx, tree, "docker", "pull", from); err != nil {
return "", fmt.Errorf("cannot fetch %s: %w", from, err)
@@ -901,7 +931,7 @@ var _ io.Writer = (*stringWriter)(nil)
// The order is fixed so two builds of one commit invoke the same command. Returned alongside the
// arguments is every reference they resolved to, which is what the build stood on.
func standingOn(ctx context.Context, manifest catalogue.Manifest, held map[string]string,
mirror func(ctx context.Context, from, repository string) (string, error)) ([]string, []string, error) {
mirror func(ctx context.Context, from, repository, former string) (string, error)) ([]string, []string, error) {
if manifest.Build == nil || len(manifest.Build.On) == 0 {
return nil, nil, nil
}
@@ -924,7 +954,14 @@ func standingOn(ctx context.Context, manifest catalogue.Manifest, held map[strin
"%s stands on the image %q, which is not pinned by digest. A tag is what "+
"somebody else can move; name it as <image>@sha256:…", manifest.Module, base.Image)
}
reference, err := mirror(ctx, base.Image, manifest.Module+"/on-"+strings.ToLower(base.Arg))
// **One copy per upstream image, whichever modules stand on it** (novox/hq ADR 0257). Its
// repository is named for the image, not the module; the module's own repository of
// before is offered as a source, so the move to one copy asks upstream for nothing.
repository, err := MirrorRepository(base.Image)
if err != nil {
return nil, nil, fmt.Errorf("%s stands on %s: %w", manifest.Module, base.Image, err)
}
reference, err := mirror(ctx, base.Image, repository, FormerMirrorRepository(manifest.Module, base.Arg))
if err != nil {
return nil, nil, fmt.Errorf("%s stands on %s: %w", manifest.Module, base.Image, err)
}
+117 -10
View File
@@ -27,6 +27,40 @@ type Mirrorer interface {
MirrorImage(ctx context.Context, from, repository string) (string, error)
}
// BaseMirrorer copies a base a build stands on into the one repository the mesh keeps for that
// upstream image, taking what an earlier copy under `former` already holds rather than asking
// upstream for it again (novox/hq ADR 0257).
type BaseMirrorer interface {
MirrorBase(ctx context.Context, from, repository, former string) (string, error)
}
// MirrorPrefix is the namespace of the repositories a base is mirrored into: `upstream/<host>/<path>`,
// one repository per upstream image, whichever modules stand on it (novox/hq ADR 0257).
const MirrorPrefix = "upstream/"
// MirrorRepository is the repository the mesh keeps an upstream image's copy in: the image's own
// host and path under MirrorPrefix, so two modules standing on one image stand on one copy, and two
// images that share a path on different hosts are never confused. Lower case and without a port's
// colon, because a repository name allows neither.
func MirrorRepository(from string) (string, error) {
where, err := parseReference(from)
if err != nil {
return "", err
}
host := strings.TrimPrefix(strings.TrimPrefix(where.base, "https://"), "http://")
if host == "registry-1.docker.io" {
host = "docker.io"
}
host = strings.ReplaceAll(host, ":", "-")
return strings.ToLower(MirrorPrefix + host + "/" + where.repository), nil
}
// FormerMirrorRepository is where a module's base was copied before ADR 0257: under the module's own
// repository, one copy per module (ADR 0097). Read only, as a source of what is already held.
func FormerMirrorRepository(module, arg string) string {
return module + "/on-" + strings.ToLower(arg)
}
const (
mediaIndexOCI = "application/vnd.oci.image.index.v1+json"
mediaIndexDocker = "application/vnd.docker.distribution.manifest.list.v2+json"
@@ -86,6 +120,9 @@ func parseReference(ref string) (upstream, error) {
type source struct {
client *http.Client
token string
// mountFrom is a repository of the mesh's own registry the source is, so a blob is mounted from
// it rather than read and written again.
mountFrom string
}
// get fetches a registry URL, answering a bearer challenge once with an anonymous token — which is
@@ -176,6 +213,15 @@ type descriptor struct {
// under `repository`, and returns the reference the mesh will pin: this registry, the repository,
// and the digest of the document that was put last, which is the index where there is one.
func (r Registry) MirrorImage(ctx context.Context, from, repository string) (string, error) {
return r.MirrorBase(ctx, from, repository, "")
}
// MirrorBase is MirrorImage, and where `former` — a repository of this registry — already holds the
// image by its digest, the copy is made from there: manifests read from this registry, blobs mounted
// across rather than moved (novox/hq ADR 0257). Upstream is asked only for what no repository here
// holds, which is what kept the public hub's anonymous pull limit out of reach when every module's
// base moved into one shared repository.
func (r Registry) MirrorBase(ctx context.Context, from, repository, former string) (string, error) {
where, err := parseReference(from)
if err != nil {
return "", err
@@ -186,7 +232,11 @@ func (r Registry) MirrorImage(ctx context.Context, from, repository string) (str
// on the first merge that rebuilt a whole catalogue (2026-09-28), and every module whose base
// lives there failed on a copy it did not need.
if strings.HasPrefix(where.reference, "sha256:") {
held, err := r.has(ctx, "http://"+r.Address+"/v2/"+repository+"/manifests/"+where.reference, manifestAccept)
// **Held whole, not only its index** (novox/hq ADR 0257): every platform manifest the index
// names is asked about too. A sweep that stopped half-way, or that ran while this build was
// copying, may have left an index naming a platform the store no longer holds; such a copy
// is made again, and the copy puts back what is missing.
held, err := r.holdsWhole(ctx, repository, where.reference)
if err != nil {
return "", fmt.Errorf("asking %s whether it holds %s: %w", r.Address, from, err)
}
@@ -195,6 +245,17 @@ func (r Registry) MirrorImage(ctx context.Context, from, repository string) (str
}
}
src := &source{client: r.client()}
if former != "" && former != repository && strings.HasPrefix(where.reference, "sha256:") {
held, err := r.holdsWhole(ctx, former, where.reference)
if err != nil {
return "", fmt.Errorf("asking %s whether %s holds %s: %w", r.Address, former, from, err)
}
if held {
// The same bytes, already here: copied from this registry, each blob mounted.
where = upstream{base: "http://" + r.Address, repository: former, reference: where.reference}
src.mountFrom = former
}
}
digest, err := r.copyManifest(ctx, src, where, where.reference, repository)
if err != nil {
return "", fmt.Errorf("copying %s into %s/%s: %w", from, r.Address, repository, err)
@@ -202,6 +263,42 @@ func (r Registry) MirrorImage(ctx context.Context, from, repository string) (str
return r.Address + "/" + repository + "@" + digest, nil
}
// holdsWhole is whether this registry holds the manifest under the repository and, for an index, every
// manifest it names.
func (r Registry) holdsWhole(ctx context.Context, repository, digest string) (bool, error) {
url := "http://" + r.Address + "/v2/" + repository + "/manifests/"
held, err := r.has(ctx, url+digest, manifestAccept)
if err != nil || !held {
return false, err
}
request, err := http.NewRequestWithContext(ctx, http.MethodGet, url+digest, nil)
if err != nil {
return false, err
}
request.Header.Set("Accept", manifestAccept)
response, err := r.client().Do(request)
if err != nil {
return false, fmt.Errorf("cannot reach the registry at %s: %w", r.Address, err)
}
defer response.Body.Close()
if response.StatusCode != http.StatusOK {
return false, nil
}
var document struct {
Manifests []descriptor `json:"manifests"`
}
if err := json.NewDecoder(io.LimitReader(response.Body, 4<<20)).Decode(&document); err != nil {
return false, fmt.Errorf("%s@%s is not a manifest: %w", repository, digest, err)
}
for _, m := range document.Manifests {
held, err := r.has(ctx, url+m.Digest, manifestAccept)
if err != nil || !held {
return false, err
}
}
return true, nil
}
// copyManifest copies one manifest document and everything it names, and returns its digest. An
// index is copied by copying each manifest it names first, so the index never points at something
// the registry does not hold yet.
@@ -286,16 +383,14 @@ func (r Registry) copyBlob(ctx context.Context, src *source, where upstream, dig
} else if there {
return nil
}
response, err := src.get(ctx, where.base+"/v2/"+where.repository+"/blobs/"+digest, "")
if err != nil {
return err
// **Mounted where this registry already holds it** (novox/hq ADR 0257): a blob is stored once
// whichever repositories link it, so a mount moves no bytes. A registry that cannot mount answers
// with an ordinary upload's location, and the blob is moved as before.
uploads := base + "/blobs/uploads/"
if src.mountFrom != "" {
uploads += "?mount=" + digest + "&from=" + src.mountFrom
}
defer response.Body.Close()
if response.StatusCode != http.StatusOK {
return fmt.Errorf("%s/%s: blob %s: %s", where.base, where.repository, digest, response.Status)
}
start, err := http.NewRequestWithContext(ctx, http.MethodPost, base+"/blobs/uploads/", nil)
start, err := http.NewRequestWithContext(ctx, http.MethodPost, uploads, nil)
if err != nil {
return err
}
@@ -304,9 +399,21 @@ func (r Registry) copyBlob(ctx context.Context, src *source, where upstream, dig
return fmt.Errorf("cannot start an upload to %s: %w", base, err)
}
begun.Body.Close()
if begun.StatusCode == http.StatusCreated && src.mountFrom != "" {
return nil
}
if begun.StatusCode != http.StatusAccepted {
return fmt.Errorf("%s answered %s when asked where to put a blob", base, begun.Status)
}
response, err := src.get(ctx, where.base+"/v2/"+where.repository+"/blobs/"+digest, "")
if err != nil {
return err
}
defer response.Body.Close()
if response.StatusCode != http.StatusOK {
return fmt.Errorf("%s/%s: blob %s: %s", where.base, where.repository, digest, response.Status)
}
location := begun.Header.Get("Location")
if location == "" {
return fmt.Errorf("%s accepted an upload and said nowhere to put it", base)
+375
View File
@@ -0,0 +1,375 @@
package builder
import (
"context"
"net/http"
"net/http/httptest"
"strings"
"sync"
"testing"
)
// One copy per upstream image, whichever modules stand on it (novox/hq ADR 0257).
func TestAnUpstreamImageHasOneRepositoryNamedForIt(t *testing.T) {
for from, want := range map[string]string{
"golang@sha256:abc": "upstream/docker.io/library/golang",
"n8nio/n8n@sha256:abc": "upstream/docker.io/n8nio/n8n",
"quay.io/minio/mc@sha256:abc": "upstream/quay.io/minio/mc",
"ghcr.io/Mailu/Admin@sha256:abc": "upstream/ghcr.io/mailu/admin",
"localhost:5000/x/y@sha256:abc": "upstream/localhost-5000/x/y",
"docker.io/library/alpine:3.20@sha256:ab": "upstream/docker.io/library/alpine",
} {
got, err := MirrorRepository(from)
if err != nil {
t.Fatalf("%s: %v", from, err)
}
if got != want {
t.Errorf("%s: got %q want %q", from, got, want)
}
}
if got := FormerMirrorRepository("route-proxy", "GO_BASE"); got != "route-proxy/on-go_base" {
t.Fatalf("the former repository is %q", got)
}
}
// aRegistryOfRepositories is a registry the way the real one is: blobs stored once, and each
// repository linking the blobs and manifests it holds. It mounts a blob across repositories.
type aRegistryOfRepositories struct {
mu sync.Mutex
blobs map[string][]byte // digest → bytes, stored once
links map[string]map[string]bool // repository → blob digests it links
manifests map[string][]byte // repository@digest → document
uploads int
mounts int
gets int
// noMount answers every mount with an ordinary upload's location, as a registry that cannot
// mount across repositories does.
noMount bool
}
func newRegistryOfRepositories() *aRegistryOfRepositories {
return &aRegistryOfRepositories{blobs: map[string][]byte{}, links: map[string]map[string]bool{},
manifests: map[string][]byte{}}
}
func (m *aRegistryOfRepositories) link(repository, digest string) {
if m.links[repository] == nil {
m.links[repository] = map[string]bool{}
}
m.links[repository][digest] = true
}
// split reads /v2/<repository>/<kind>/<rest> with a repository of any depth.
func splitPath(path string) (repository, kind, rest string) {
path = strings.TrimPrefix(path, "/v2/")
for _, k := range []string{"/manifests/", "/blobs/uploads/", "/blobs/"} {
if i := strings.Index(path, k); i >= 0 {
return path[:i], strings.Trim(k, "/"), path[i+len(k):]
}
}
return "", "", ""
}
func (m *aRegistryOfRepositories) handler() http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
m.mu.Lock()
defer m.mu.Unlock()
repository, kind, rest := splitPath(r.URL.Path)
switch {
case kind == "manifests" && (r.Method == http.MethodHead || r.Method == http.MethodGet):
body, ok := m.manifests[repository+"@"+rest]
if !ok {
w.WriteHeader(http.StatusNotFound)
return
}
if r.Method == http.MethodGet {
m.gets++
w.Header().Set("Content-Type", mediaTypeOf(body))
_, _ = w.Write(body)
return
}
w.WriteHeader(http.StatusOK)
case kind == "manifests" && r.Method == http.MethodPut:
body, _ := readAll(r)
m.manifests[repository+"@"+digestOf(body)] = body
w.WriteHeader(http.StatusCreated)
case kind == "blobs" && (r.Method == http.MethodHead || r.Method == http.MethodGet):
if !m.links[repository][rest] {
w.WriteHeader(http.StatusNotFound)
return
}
if r.Method == http.MethodGet {
m.gets++
_, _ = w.Write(m.blobs[rest])
return
}
w.WriteHeader(http.StatusOK)
case kind == "manifests" && r.Method == http.MethodDelete:
delete(m.manifests, repository+"@"+rest)
w.WriteHeader(http.StatusAccepted)
case kind == "blobs/uploads" && r.Method == http.MethodPost:
if digest, from := r.URL.Query().Get("mount"), r.URL.Query().Get("from"); digest != "" && m.links[from][digest] && !m.noMount {
m.link(repository, digest)
m.mounts++
w.WriteHeader(http.StatusCreated)
return
}
w.Header().Set("Location", "/v2/"+repository+"/blobs/uploads/one")
w.WriteHeader(http.StatusAccepted)
case kind == "blobs/uploads" && r.Method == http.MethodPut:
body, _ := readAll(r)
digest := r.URL.Query().Get("digest")
if digestOf(body) != digest {
w.WriteHeader(http.StatusBadRequest)
return
}
m.blobs[digest] = body
m.link(repository, digest)
m.uploads++
w.WriteHeader(http.StatusCreated)
default:
w.WriteHeader(http.StatusNotFound)
}
})
}
func mediaTypeOf(body []byte) string {
switch {
case strings.Contains(string(body), mediaIndexOCI):
return mediaIndexOCI
default:
return mediaManifestOCI
}
}
// A base a module's own repository already holds is copied into the image's repository from there:
// every blob mounted, nothing asked of upstream — which may be gone, or rationing anonymous pulls.
func TestABaseHeldUnderTheModulesFormerRepositoryIsMountedNotFetchedAgain(t *testing.T) {
src, indexDigest, srcBlobs := anUpstreamRegistry(t)
dst := newRegistryOfRepositories()
dstServer := httptest.NewServer(dst.handler())
defer dstServer.Close()
address := strings.TrimPrefix(dstServer.URL, "http://")
r := Registry{Address: address, HTTP: src.Client()}
host := strings.TrimPrefix(src.URL, "http://")
// Before: copied the old way, under the module's repository.
if _, err := r.MirrorImage(context.Background(), host+"/library/thing:latest", "hello-web/on-thing_base"); err != nil {
t.Fatal(err)
}
uploaded := dst.uploads
if uploaded != len(srcBlobs) {
t.Fatalf("the first copy uploaded %d of %d blobs", uploaded, len(srcBlobs))
}
src.Close()
from := host + "/library/thing@" + indexDigest
repository, err := MirrorRepository(from)
if err != nil {
t.Fatal(err)
}
reference, err := r.MirrorBase(context.Background(), from, repository, "hello-web/on-thing_base")
if err != nil {
t.Fatalf("a base the registry holds was asked of an upstream that is gone: %v", err)
}
if reference != address+"/"+repository+"@"+indexDigest {
t.Fatalf("pinned as %q", reference)
}
if dst.uploads != uploaded {
t.Fatalf("the move to one repository uploaded %d blobs again", dst.uploads-uploaded)
}
if dst.mounts != len(srcBlobs) {
t.Fatalf("%d of %d blobs mounted", dst.mounts, len(srcBlobs))
}
// The index and both platform manifests are in the image's repository, and every blob is linked.
for key := range dst.manifests {
if strings.HasPrefix(key, "hello-web/") {
continue
}
if !strings.HasPrefix(key, repository+"@") {
t.Fatalf("a manifest went to %s", key)
}
}
if n := countPrefix(dst.manifests, repository+"@"); n != 3 {
t.Fatalf("%d manifests in %s, want the index and two platforms", n, repository)
}
for digest := range srcBlobs {
if !dst.links[repository][digest] {
t.Fatalf("blob %s is not linked into %s", digest, repository)
}
}
// A second module standing on the same image: already held, nothing copied, the same reference.
again, err := r.MirrorBase(context.Background(), from, repository, "other-module/on-thing_base")
if err != nil || again != reference {
t.Fatalf("a second module's copy: %q %v", again, err)
}
}
func countPrefix(m map[string][]byte, prefix string) int {
n := 0
for k := range m {
if strings.HasPrefix(k, prefix) {
n++
}
}
return n
}
// What a build copied is said whether it worked or not: the copy is in the store either way, and a
// build that failed after copying is the one record that it is there.
func TestABuildSaysWhatItMirroredEvenWhenItFails(t *testing.T) {
// A recipe that reaches for an image nobody declared is refused — after the base was copied.
r, workspace := aRepository(t, anImage, map[string]string{
"modules/bus/Dockerfile": "ARG BASE\nFROM ${BASE}\nCOPY --from=vendor/other:1 /x /x"})
r.contents["modules/bus/module.json"] = anImage
r.tree = "aaaa"
m := &mirroring{recorded: r, at: "registry-a:5000"}
got, err := Build(context.Background(), r.run, m, "https://forge.invalid/catalogue.git", "modules/bus", "", workspace,
nil, Npmrc{}, GitCredential{}, nil)
if err == nil {
t.Fatal("a recipe fetching an undeclared image was built")
}
want := "registry-a:5000/upstream/docker.io/vendor/server@sha256:" + strings.Repeat("1", 64)
if len(got.Mirrored) != 1 || got.Mirrored[0] != want {
t.Fatalf("a failed build said it mirrored %v, want [%s]", got.Mirrored, want)
}
// And a build that works says it too.
r2, workspace2 := aRepository(t, anImage, map[string]string{"modules/bus/Dockerfile": "ARG BASE\nFROM ${BASE}"})
r2.contents["modules/bus/module.json"] = anImage
r2.tree = "aaaa"
ok, err := Build(context.Background(), r2.run, &mirroring{recorded: r2, at: "registry-a:5000"},
"https://forge.invalid/catalogue.git", "modules/bus", "", workspace2, nil, Npmrc{}, GitCredential{}, nil)
if err != nil {
t.Fatal(err)
}
if len(ok.Mirrored) != 1 || ok.Mirrored[0] != want {
t.Fatalf("a build said it mirrored %v, want [%s]", ok.Mirrored, want)
}
}
// held puts one image in a repository the old way and answers its index digest and the registry.
func heldUnderAModule(t *testing.T) (*aRegistryOfRepositories, Registry, string, string, map[string][]byte, func()) {
t.Helper()
src, indexDigest, srcBlobs := anUpstreamRegistry(t)
t.Cleanup(src.Close)
dst := newRegistryOfRepositories()
dstServer := httptest.NewServer(dst.handler())
t.Cleanup(dstServer.Close)
r := Registry{Address: strings.TrimPrefix(dstServer.URL, "http://"), HTTP: src.Client()}
host := strings.TrimPrefix(src.URL, "http://")
if _, err := r.MirrorImage(context.Background(), host+"/library/thing:latest", "hello-web/on-thing_base"); err != nil {
t.Fatal(err)
}
return dst, r, host + "/library/thing@" + indexDigest, indexDigest, srcBlobs, src.Close
}
// An index whose platform manifests are not all held is not "already held": it is copied again, and the
// copy puts back what is missing — what a sweep that stopped half-way, or ran while a build was copying,
// leaves behind.
func TestAnIndexMissingAPlatformIsCopiedAgain(t *testing.T) {
dst, r, from, indexDigest, _, _ := heldUnderAModule(t)
repository, _ := MirrorRepository(from)
if _, err := r.MirrorBase(context.Background(), from, repository, ""); err != nil {
t.Fatal(err)
}
// A sweep lets one platform go, between this build's copy and the next.
var platform string
for key := range dst.manifests {
if strings.HasPrefix(key, repository+"@") && !strings.HasSuffix(key, indexDigest) {
platform = key
break
}
}
delete(dst.manifests, platform)
if _, err := r.MirrorBase(context.Background(), from, repository, ""); err != nil {
t.Fatal(err)
}
if _, back := dst.manifests[platform]; !back {
t.Fatalf("an index missing %s was taken as held", platform)
}
}
// The same, for the module's former copy: a former copy missing a platform is not a source; upstream
// is asked instead.
func TestAFormerCopyMissingAPlatformIsNotTheSource(t *testing.T) {
dst, r, from, indexDigest, _, _ := heldUnderAModule(t)
for key := range dst.manifests {
if strings.HasPrefix(key, "hello-web/on-thing_base@") && !strings.HasSuffix(key, indexDigest) {
delete(dst.manifests, key)
break
}
}
repository, _ := MirrorRepository(from)
if _, err := r.MirrorBase(context.Background(), from, repository, "hello-web/on-thing_base"); err != nil {
t.Fatal(err)
}
if dst.mounts != 0 {
t.Fatalf("a former copy missing a platform was mounted from (%d mounts)", dst.mounts)
}
if n := countPrefix(dst.manifests, repository+"@"); n != 3 {
t.Fatalf("%d manifests copied from upstream, want 3", n)
}
}
// A registry that answers a mount with an upload's location (202) gets the blob moved as before.
func TestAMountRefusedFallsBackToAnUpload(t *testing.T) {
dst, r, from, _, srcBlobs, _ := heldUnderAModule(t)
dst.noMount = true
uploaded := dst.uploads
repository, _ := MirrorRepository(from)
reference, err := r.MirrorBase(context.Background(), from, repository, "hello-web/on-thing_base")
if err != nil {
t.Fatal(err)
}
if !strings.HasSuffix(reference, repository+"@"+strings.SplitN(from, "@", 2)[1]) {
t.Fatalf("pinned as %s", reference)
}
if dst.mounts != 0 || dst.uploads-uploaded != len(srcBlobs) {
t.Fatalf("%d mounts, %d uploads; want every blob uploaded from the former copy", dst.mounts, dst.uploads-uploaded)
}
for digest := range srcBlobs {
if !dst.links[repository][digest] {
t.Fatalf("blob %s is not linked into %s", digest, repository)
}
}
}
// A sweep and a build at once: while builds copy the base, platforms are let go of under them. Each
// build that returns has the copy whole.
func TestABuildCopyingWhileASweepLetsGoEndsWhole(t *testing.T) {
dst, r, from, indexDigest, _, _ := heldUnderAModule(t)
repository, _ := MirrorRepository(from)
if _, err := r.MirrorBase(context.Background(), from, repository, ""); err != nil {
t.Fatal(err)
}
done := make(chan struct{})
go func() {
defer close(done)
for i := 0; i < 50; i++ {
dst.mu.Lock()
for key := range dst.manifests {
if strings.HasPrefix(key, repository+"@") && !strings.HasSuffix(key, indexDigest) {
delete(dst.manifests, key)
break
}
}
dst.mu.Unlock()
}
}()
for i := 0; i < 20; i++ {
if _, err := r.MirrorBase(context.Background(), from, repository, ""); err != nil {
t.Fatal(err)
}
}
<-done
// The sweep is over; the next build finds what it left and makes the copy whole.
if _, err := r.MirrorBase(context.Background(), from, repository, ""); err != nil {
t.Fatal(err)
}
if n := countPrefix(dst.manifests, repository+"@"); n != 3 {
t.Fatalf("after the sweep and the builds, %d manifests in %s, want 3", n, repository)
}
}
+7
View File
@@ -117,6 +117,13 @@ func (m *theMeshsRegistry) handler() http.Handler {
} else {
w.WriteHeader(http.StatusNotFound)
}
case r.Method == http.MethodGet && strings.Contains(r.URL.Path, "/manifests/"):
body, ok := m.manifests[r.URL.Path[strings.LastIndex(r.URL.Path, "/")+1:]]
if !ok {
w.WriteHeader(http.StatusNotFound)
return
}
_, _ = w.Write(body)
case r.Method == http.MethodHead && strings.Contains(r.URL.Path, "/blobs/"):
if _, ok := m.blobs[r.URL.Path[strings.LastIndex(r.URL.Path, "/")+1:]]; ok {
w.WriteHeader(http.StatusOK)
+9 -6
View File
@@ -72,7 +72,7 @@ func TestAnIncompleteBaseIsRefused(t *testing.T) {
}
// noMirror is a mirror for tests whose bases are all the mesh's own.
func noMirror(context.Context, string, string) (string, error) {
func noMirror(context.Context, string, string, string) (string, error) {
return "", fmt.Errorf("nothing to copy in this test")
}
@@ -86,17 +86,20 @@ func TestADeclaredVendorImageIsCopiedInAndHandedToTheRecipe(t *testing.T) {
},
}
var asked []string
args, _, err := standingOn(context.Background(), manifest, nil, func(_ context.Context, from, repository string) (string, error) {
asked = append(asked, from+" -> "+repository)
args, _, err := standingOn(context.Background(), manifest, nil, func(_ context.Context, from, repository, former string) (string, error) {
asked = append(asked, from+" -> "+repository+" (from "+former+")")
return "127.0.0.1:5000/" + repository + "@sha256:" + strings.Repeat("d", 64), nil
})
if err != nil {
t.Fatal(err)
}
if len(asked) != 1 || asked[0] != "quay.io/minio/mc@sha256:"+strings.Repeat("c", 64)+" -> minio/on-mc_base" {
t.Fatalf("the image was not copied under the module's repository: %v", asked)
// One repository per upstream image, named for the image (novox/hq ADR 0257); the module's
// repository of before offered as where it may already be held.
if len(asked) != 1 || asked[0] != "quay.io/minio/mc@sha256:"+strings.Repeat("c", 64)+
" -> upstream/quay.io/minio/mc (from minio/on-mc_base)" {
t.Fatalf("the image was not copied into the image's own repository: %v", asked)
}
if strings.Join(args, " ") != "--build-arg MC_BASE=127.0.0.1:5000/minio/on-mc_base@sha256:"+strings.Repeat("d", 64) {
if strings.Join(args, " ") != "--build-arg MC_BASE=127.0.0.1:5000/upstream/quay.io/minio/mc@sha256:"+strings.Repeat("d", 64) {
t.Fatalf("the recipe was not handed the copy: %v", args)
}
// Unpinned, it is refused: a tag is what somebody else can move.
+1 -1
View File
@@ -32,7 +32,7 @@ func TestHandWrittenBackupLinesOfAnOlderModuleArePlaced(t *testing.T) {
if err != nil {
t.Fatal(err)
}
placed, err := seatContributions([]Manifest{pg, mail}, BackupSeat, "backup", Rendering{}, nil)
placed, _, err := seatContributions([]Manifest{pg, mail}, Manifest{}, BackupSeat+":"+"backup", Rendering{}, nil, nil)
if err != nil {
t.Fatal(err)
}
+3 -3
View File
@@ -174,7 +174,7 @@ func TestTheBackupHoldersLinesAreDerivedFromTheData(t *testing.T) {
"resources":[{"id":"meta","type":"directory","mode":"0700"}]}`)
facts := map[string]string{"account-home": "/home/op"}
with := Rendering{Settings: SettingsBy{"media": {{From: "node", Values: map[string]any{AccessesSetting: map[string]any{"films": "/tank/films"}}}}}}
backup, err := seatContributions([]Manifest{pg, agent, media}, BackupSeat, BackupKindBackup, with, facts)
backup, _, err := seatContributions([]Manifest{pg, agent, media}, Manifest{}, BackupSeat+":"+BackupKindBackup, with, facts, nil)
if err != nil {
t.Fatal(err)
}
@@ -182,7 +182,7 @@ func TestTheBackupHoldersLinesAreDerivedFromTheData(t *testing.T) {
if backup != want {
t.Fatalf("backup lines:\n%s\nwant:\n%s", backup, want)
}
data, err := seatContributions([]Manifest{pg, agent, media}, BackupSeat, BackupKindData, with, facts)
data, _, err := seatContributions([]Manifest{pg, agent, media}, Manifest{}, BackupSeat+":"+BackupKindData, with, facts, nil)
if err != nil {
t.Fatal(err)
}
@@ -192,7 +192,7 @@ func TestTheBackupHoldersLinesAreDerivedFromTheData(t *testing.T) {
if data != want {
t.Fatalf("data lines:\n%s\nwant:\n%s", data, want)
}
if _, err := seatContributions([]Manifest{agent}, BackupSeat, BackupKindData, Rendering{}, nil); err == nil {
if _, _, err := seatContributions([]Manifest{agent}, Manifest{}, BackupSeat+":"+BackupKindData, Rendering{}, nil, nil); err == nil {
t.Fatal("a home directory with no account on the machine reached the holder as a placeholder")
}
// What keeps something irreplaceable depends on the machine's backup holder — it backs it up or
+10 -5
View File
@@ -291,6 +291,10 @@ type Composed struct {
// as `module/name`, sorted: what the next send will make. Never set on a declaration that is
// sent — a placeholder in a sealed file is a credential the process cannot read.
Foreseen []string
// Unplaced is every contribution the holder's template could not render into something, each
// naming its module and why (novox/hq ADR 0255): that piece is left out and the machine is told
// everything else. Push and plan say it; the merge gate refuses a change that adds one.
Unplaced []string
}
// ForeseenSealed is what stands for an own secret a send will make, in a composition asked ahead of
@@ -326,8 +330,8 @@ func (r Resolution) Compose(with Rendering) (Composed, error) {
owner := map[string]string{}
received := map[string]map[string][]Contribution{}
leftOut := map[string]string{}
var foreseen []string
resources, err := r.compose(with, owner, received, leftOut, &foreseen)
var foreseen, unplaced []string
resources, err := r.compose(with, owner, received, leftOut, &foreseen, &unplaced)
if err != nil {
return Composed{}, err
}
@@ -340,8 +344,9 @@ func (r Resolution) Compose(with Rendering) (Composed, error) {
"sealed": with.BusMembership, "mode": "0600",
})
}
sort.Strings(unplaced)
return Composed{Resources: resources, Owner: owner, Received: received, LeftOut: leftOut,
Foreseen: foreseen}, nil
Foreseen: foreseen, Unplaced: unplaced}, nil
}
// BusMembershipID names the resource carrying a machine's membership for the new bus, and
@@ -352,7 +357,7 @@ const BusMembershipPath = "/var/lib/mesh/membership-next.json"
func (r Resolution) compose(with Rendering, owner map[string]string,
received map[string]map[string][]Contribution, leftOut map[string]string,
foreseen *[]string) ([]map[string]any, error) {
foreseen, unplaced *[]string) ([]map[string]any, error) {
// **A setting is judged where it is stored, and an impossible one costs a module, not a
// machine** (novox/hq ADR 0163, rule 6). A definition that moved under a stored setting makes
// this module uncomposable; it is left out of the declaration — its held things kept, its
@@ -970,7 +975,7 @@ func (r Resolution) compose(with Rendering, owner map[string]string,
// seat that places them (novox/hq ADR 0203, ADR 0204). Gathered from every module on
// the node, as the jails are, and **last of every placeholder pass**: shell code is a
// shell's own syntax, full of `${…}` no pass above should ever be shown.
if err := contributionsInto(copied, m, r.Modules, thisMachine, with); err != nil {
if err := contributionsInto(copied, m, r.Modules, thisMachine, with, r.Capabilities, unplaced); err != nil {
return nil, err
}
copied["id"] = m.Module + "." + fmt.Sprint(resource["id"])
+5 -2
View File
@@ -547,7 +547,7 @@ func shellCode(modules []Manifest, shell, slot string) string {
// `${machine:…}` some module wrote for its shell to see. So nothing runs after them, the environment
// is filled before the shell's code is, and each is replaced in a single pass over what the holder
// wrote, so a contributed piece is never scanned again.
func contributionsInto(resource map[string]any, m Manifest, modules []Manifest, facts map[string]string, with Rendering) error {
func contributionsInto(resource map[string]any, m Manifest, modules []Manifest, facts map[string]string, with Rendering, caps map[string]bool, unplaced *[]string) error {
if problems := append(placeholderProblems(m, resource), seatPlaceholderProblems(m, resource)...); len(problems) > 0 {
return fmt.Errorf("%s", problems[0])
}
@@ -576,10 +576,13 @@ func contributionsInto(resource map[string]any, m Manifest, modules []Manifest,
found := ofContributed.FindStringSubmatch(placeholder)
first, second, _ := strings.Cut(found[2], ":")
if found[1] == "contribution" {
placed, err := seatContributions(modules, first, second, with, facts)
placed, left, err := seatContributions(modules, m, found[2], with, facts, caps)
if err != nil && failed == nil {
failed = err
}
if unplaced != nil {
*unplaced = append(*unplaced, left...)
}
return placed
}
return shellCode(modules, first, second)
+14 -13
View File
@@ -177,19 +177,8 @@ func TestTheForgeHoldsTheNpmAndGitSeats(t *testing.T) {
// wrong on every node whose assignment differs, and wrong for a second reason on a node given the
// port (ADR 0100). Composed through the whole path, because what proves the placeholder resolves
// in an `env` at all is a declaration, not a manifest.
// forgeBuilt is every artifact the forge's manifest builds: its code, and since hq ADR 0251 the npm
// registry it serves beside it. A manifest resolves only against all of what it asked to be built.
func forgeBuilt() []Built {
var out []Built
for _, name := range []string{"code", "npm-registry"} {
out = append(out, Built{Name: name, Kind: ArtifactBundle,
Reference: ArtifactStoreScheme + "gitea/" + name + "/blobs/" + bundleDigest, Digest: bundleDigest})
}
return out
}
func TestTheForgesOwnAddressFollowsThePortTheNodeGaveIt(t *testing.T) {
forge, err := catalogueManifest(t, "gitea").Resolve(forgeBuilt())
forge, err := catalogueManifest(t, "gitea").Resolve(forgeBuilt(catalogueManifest(t, "gitea")))
if err != nil {
t.Fatalf("the forge's manifest does not resolve against its own build: %v", err)
}
@@ -242,7 +231,7 @@ func TestTheForgesOwnAddressFollowsThePortTheNodeGaveIt(t *testing.T) {
func declaredGiteaSsh(t *testing.T, given map[int]int) map[string]any {
t.Helper()
forge := catalogueManifest(t, "gitea")
resolved, err := forge.Resolve(forgeBuilt())
resolved, err := forge.Resolve(forgeBuilt(forge))
if err != nil {
t.Fatalf("the forge's manifest does not resolve against its own build: %v", err)
}
@@ -314,3 +303,15 @@ func TestANodeMayGiveTheForgesSshPortADifferentNumber(t *testing.T) {
t.Fatalf("the forge is published on %v, not the port this node gave it", server["ports"])
}
}
// forgeBuilt is a build of every artifact the forge's manifest declares: its code bundle and, since the
// catalogue added it (novox/hq ADR 0251 §4), its npm registry's. Named from the manifest, so a third
// artifact does not break every forge test again.
func forgeBuilt(forge Manifest) []Built {
var out []Built
for _, a := range forge.Build.Artifacts {
out = append(out, Built{Name: a.Name, Kind: ArtifactBundle,
Reference: ArtifactStoreScheme + "gitea/" + a.Name + "/blobs/" + bundleDigest, Digest: bundleDigest})
}
return out
}
+121 -2
View File
@@ -1,5 +1,10 @@
package catalogue
import (
"fmt"
"strings"
)
// The graphical session's seats (novox/hq ADR 0208): one module per piece of software, each piece's
// role a node seat in the mesh's own set, so i3 and sway, xterm and foot, rofi and dmenu compete for
// a role rather than each inventing one — and a machine running two of one role is refused at
@@ -19,7 +24,8 @@ const (
)
// graphicalSessionSeats are the eleven, in the order ADR 0208's table reads, each with the verbs
// research 026/05 starts it with. Three have none yet: the bar, the compositor and the secret
// research 026/05 starts it with. Three have none yet: the bar (which receives blocks instead,
// ADR 0255), the compositor and the secret
// service are roles a second holder competes for, and nothing has needed to ask them anything.
func graphicalSessionSeats() []Seat {
const decided = "novox/hq ADR 0208"
@@ -93,7 +99,11 @@ func graphicalSessionSeats() []Seat {
{Name: "copy", Description: "Put text on the operator's clipboard.",
Input: schema(map[string]string{"text": "the text"}, []string{"text"})},
}},
{Name: BarSeat, Scope: ScopeNode, Decision: decided},
// It receives blocks as data (novox/hq ADR 0255): what a block shows, said for any bar, which
// the holder renders in its own grammar. A module that knows something about the machine — the
// power module its battery — gives the block; the bar knows no machine's hardware.
{Name: BarSeat, Scope: ScopeNode, Decision: decided,
Receives: []Receivable{{Kind: BarKindBlock, Comment: "#", Offered: true, Shape: barBlockShape()}}},
{Name: CompositorSeat, Scope: ScopeNode, Decision: decided},
{Name: SecretServiceSeat, Scope: ScopeNode, Decision: decided},
}
@@ -107,3 +117,112 @@ func withEnum(s map[string]any, property string, values ...string) map[string]an
p["enum"] = values
return s
}
// BarKindBlock is the bar's one received kind: a block (novox/hq ADR 0255).
const BarKindBlock = "block"
// barShows is what a contributed block may show, and the options each takes — the bar-neutral
// vocabulary every holder of node-bar renders. A new entry is a decision, and every holder's template
// renders it (the shape's examples) before it is accepted.
var barShows = map[string]map[string]barOption{
// The machine's battery: its charge, and how long it lasts or takes to fill.
"battery": {"device": {kind: "text"}},
// One line a command prints, run every interval seconds: what the bar cannot know itself.
"command": {"command": {kind: "text", required: true}, "interval": {kind: "seconds", required: true}},
// A value the contributor keeps on the bus (novox/hq ADR 0255): its state `<module>.<name>` and the
// key, this machine's name when absent. The value is an object whose `text` is what is shown. The
// holder is granted to read it and shows it however its bar shows a changing value; no file and
// no command passes between the two modules.
"state": {"state": {kind: "text", required: true}, "key": {kind: "text"}},
}
// barBlockReads is the state a block shows, if any.
func barBlockReads(data map[string]any) []string {
if data["shows"] != "state" {
return nil
}
options, _ := data["options"].(map[string]any)
if s, ok := options["state"].(string); ok && s != "" {
return []string{s}
}
return nil
}
type barOption struct {
kind string // text, or seconds (a whole number from 1 to 86400)
required bool
}
// barBlockShape is the shape of a block (novox/hq ADR 0255): which bar, where on it, in what order,
// what it shows, and that kind's options.
//
// - bar: top or bottom;
// - place: resources (beside the processor, memory and disks) or status (beside the sound, before the
// clock) — the two places every holder keeps for contributed blocks;
// - order: 0–99 within the place, 50 when absent;
// - shows: battery or command; options: what that kind takes.
func barBlockShape() *Shape {
return &Shape{
Places: []PlaceField{{Field: "bar", Values: []string{"bottom", "top"}},
{Field: "place", Values: []string{"resources", "status"}}},
Order: "order",
Check: barBlockProblems,
Reads: barBlockReads,
Examples: []map[string]any{
{"bar": "bottom", "place": "status", "shows": "battery"},
{"bar": "bottom", "place": "status", "shows": "battery", "options": map[string]any{"device": "BAT0"}},
{"bar": "bottom", "place": "status", "shows": "command",
"options": map[string]any{"command": "echo \"a line\"", "interval": float64(5)}},
{"bar": "bottom", "place": "status", "shows": "state", "options": map[string]any{"state": "power.draw"}},
{"bar": "bottom", "place": "status", "shows": "state",
"options": map[string]any{"state": "power.draw", "key": "a-machine"}},
},
}
}
func barBlockProblems(data map[string]any) []string {
var problems []string
for field := range data {
if !oneOf([]string{"bar", "place", "order", "shows", "options"}, field) {
problems = append(problems, fmt.Sprintf("a block has no field %q; it has bar, place, order, shows and options", field))
}
}
shows, _ := data["shows"].(string)
takes, known := barShows[shows]
if !known {
return append(problems, fmt.Sprintf("a block shows %v; it shows one of %s", data["shows"],
strings.Join(sortedKeys(barShows), ", ")))
}
options := map[string]any{}
if raw, has := data["options"]; has {
o, ok := raw.(map[string]any)
if !ok {
return append(problems, "a block's options are an object")
}
options = o
}
for name, v := range options {
opt, ok := takes[name]
if !ok {
problems = append(problems, fmt.Sprintf("a %s block has no option %q; it takes %s", shows, name,
orNothing(sortedKeys(takes))))
continue
}
switch opt.kind {
case "text":
if t, isText := v.(string); !isText || strings.TrimSpace(t) == "" {
problems = append(problems, fmt.Sprintf("a %s block's %s is text", shows, name))
}
case "seconds":
if _, whole := wholeIn(v, 1, 86400); !whole {
problems = append(problems, fmt.Sprintf("a %s block's %s is a whole number of seconds from 1 to 86400", shows, name))
}
}
}
for _, name := range sortedKeys(takes) {
if _, has := options[name]; takes[name].required && !has {
problems = append(problems, fmt.Sprintf("a %s block needs the option %s", shows, name))
}
}
return problems
}
+5
View File
@@ -60,6 +60,10 @@ type Claim struct {
// module's own `tools` must list every verb the seat promises, which is how a module named
// like its seat — the catalogue, the records — says they are one and the same.
Serves []string `json:"serves,omitempty"`
// Renders is how this holder writes each kind its seat receives as data, by kind: a Go
// text/template over one piece — its fields and `module` — in the tool's own grammar (novox/hq ADR
// 0255). The data is the mesh's, the format the holder's, as a module's facts template is.
Renders map[string]string `json:"renders,omitempty"`
}
// ServesFor is what this claim offers a seat's protocol: the verbs it names, else the module's
@@ -2030,6 +2034,7 @@ func ParseManifest(raw []byte) (Manifest, error) {
problems = append(problems, m.shellProblems()...)
problems = append(problems, m.contributionPlaceholderProblems()...)
problems = append(problems, m.seatContributionProblems()...)
problems = append(problems, m.placementProblems()...)
problems = append(problems, m.dataProblems()...)
problems = append(problems, m.upgradeProblems()...)
+5 -1
View File
@@ -134,6 +134,10 @@ type Resolution struct {
// here without a store lookup.
Account string
AccountHome string
// Capabilities are the machine's, as its profile reported them, carried from the node so a
// contribution placed only where the machine has something (`if-capability`, novox/hq ADR 0255)
// is decided here without a store lookup.
Capabilities map[string]bool
// Modules in the order they were resolved: assigned first, then what they pulled in.
Modules []Manifest
@@ -699,7 +703,7 @@ func Resolve(catalogue map[string]Manifest, assigned []string, node Node, world
}
resolution := Resolution{Node: node.Name, At: node.At, PublicDomain: node.PublicDomain,
Account: node.Account, AccountHome: node.AccountHome,
Account: node.Account, AccountHome: node.AccountHome, Capabilities: node.Capabilities,
Because: because, Needs: needs, Unhostable: unhostable, Kept: kept}
for _, n := range providersFirst(order, catalogue) {
resolution.Modules = append(resolution.Modules, catalogue[n])
+62 -8
View File
@@ -30,7 +30,14 @@ type SeatContribution struct {
// Kind is which of the seat's receivable kinds it is.
Kind string `json:"kind"`
// Content is the text, in the tool's own grammar. Never interpreted.
Content string `json:"content"`
Content string `json:"content,omitempty"`
// Data is the piece in the seat's own shape, for a kind the seat receives as data (novox/hq ADR
// 0255): the holder renders it with its template, and Content is then empty.
Data map[string]any `json:"data,omitempty"`
// IfCapability names a capability the machine must report for this contribution to be placed
// there (novox/hq ADR 0255): a battery's block only where the node-engine found a battery. The
// composition decides, from the machine's facts, not the tool at run time.
IfCapability string `json:"if-capability,omitempty"`
}
// ofContribution is where a holder places a kind: ${contribution:<seat>:<kind>}. Loose inside the
@@ -92,7 +99,31 @@ func (m Manifest) seatContributionProblems() []string {
"%s's contribution %d to %s is of the kind %q; %s (novox/hq ADR 0212)",
m.Module, i+1, s.Name, c.Kind, kindsOf(s)))
}
if strings.TrimSpace(c.Content) == "" {
// Only on a kind the seat offers (novox/hq ADR 0255): a piece shown where the hardware is. On a
// kind a holder depends on — what a backup keeps, a key's trigger — a machine missing the
// capability would lose the piece without a word.
if c.IfCapability != "" {
switch {
case !capabilityName.MatchString(c.IfCapability):
problems = append(problems, fmt.Sprintf("%s's contribution %d is if-capability %q, which is not a "+
"capability's name", m.Module, i+1, c.IfCapability))
case !oneOf(KnownCapabilities, c.IfCapability):
problems = append(problems, fmt.Sprintf("%s's contribution %d is if-capability %q, which no machine "+
"reports; the node-engine detects %s", m.Module, i+1, c.IfCapability, strings.Join(KnownCapabilities, ", ")))
case ok && !r.Offered:
problems = append(problems, fmt.Sprintf("%s's contribution %d to %s (%s) is if-capability %s; only a "+
"kind the seat offers may be left out where a machine lacks something (novox/hq ADR 0255)",
m.Module, i+1, s.Name, c.Kind, c.IfCapability))
}
}
switch {
case ok && r.Shape != nil:
problems = append(problems, shapedProblems(m.Module, i, r, c)...)
problems = append(problems, ownStateProblems(m, i, r, c)...)
case len(c.Data) > 0:
problems = append(problems, fmt.Sprintf("%s's contribution %d has data; %s receives %s as text, "+
"given as `content`", m.Module, i+1, s.Name, c.Kind))
case strings.TrimSpace(c.Content) == "":
problems = append(problems, fmt.Sprintf("%s's contribution %d has no content", m.Module, i+1))
}
}
@@ -120,8 +151,15 @@ func seatPlaceholderProblems(m Manifest, r map[string]any) []string {
continue
}
for _, f := range found {
seat, kind, two := strings.Cut(f[1], ":")
s, _, ok := receivable(seat, kind)
seat, rest, two := strings.Cut(f[1], ":")
// A kind received as data may be narrowed by its shape's placing fields (novox/hq ADR 0255).
kind, where, _ := strings.Cut(rest, ":")
s, rcv, ok := receivable(seat, kind)
if two && ok {
if _, err := placeholderWhere(rcv, where); err != nil {
problems = append(problems, fmt.Sprintf("%s's resource %v names %s: %v", m.Module, r["id"], f[0], err))
}
}
if !two || !ok {
detail := "the mesh defines no seat " + fmt.Sprintf("%q", seat)
if s.Name != "" {
@@ -152,17 +190,33 @@ func seatPlaceholderProblems(m Manifest, r map[string]any) []string {
// A directory placed in a person's home is named `${machine:account-home}/…` (novox/hq to-be 29); in a
// kind that takes directories that is filled from the machine's facts as well, so the holder reads a
// path, never a placeholder (novox/hq ADR 0233: an item in the operator's home is backed up too).
func seatContributions(modules []Manifest, seat, kind string, with Rendering, facts map[string]string) (string, error) {
//
// A kind received as data is rendered through the holder's template instead, narrowed by where
// (novox/hq ADR 0255); and a contribution naming a capability this machine did not report is left
// out, of either form.
//
// What it could not render is answered as unplaced, each naming its module and why, and left out: one
// piece the holder's template does not know is that piece's fault, not the machine's.
func seatContributions(modules []Manifest, holder Manifest, placeholder string, with Rendering, facts map[string]string, caps map[string]bool) (string, []string, error) {
seat, rest, _ := strings.Cut(placeholder, ":")
kind, whereText, _ := strings.Cut(rest, ":")
s, r, ok := receivable(seat, kind)
if !ok {
return "", nil
return "", nil, nil
}
if r.Shape != nil {
where, err := placeholderWhere(r, whereText)
if err != nil {
return "", nil, err
}
return shapedContributions(modules, holder, facts["name"], s, r, where, caps)
}
var failed error
var b strings.Builder
for _, m := range inModuleOrder(modules) {
named := false
for _, c := range m.allContributions() {
if c.Kind != kind {
if c.Kind != kind || !capable(c, caps) {
continue
}
if cs, known := SeatNamed(c.Seat); !known || cs.Name != s.Name {
@@ -207,5 +261,5 @@ func seatContributions(modules []Manifest, seat, kind string, with Rendering, fa
}
}
}
return b.String(), failed
return b.String(), nil, failed
}
+462
View File
@@ -0,0 +1,462 @@
package catalogue
import (
"bytes"
"encoding/json"
"fmt"
"math"
"regexp"
"sort"
"strings"
"text/template"
)
// A kind a seat receives as data, rendered by its holder (novox/hq ADR 0255).
//
// ADR 0212 made a contribution text in the tool's own grammar, which the controller places and never
// reads. That is right where the tool is the seat's for good — a hotkey daemon's trigger lines — and
// wrong where the seat's whole point is that its tool can be replaced: a module adding a battery to the
// bar would write i3status-rust's TOML, and a second bar would have to read it. So a seat may define a
// kind as data instead. The seat says the data's shape; a contributor gives data in that shape; the
// holder gives a template that renders one piece of it into its tool's grammar, as a module's facts
// template renders the roster (`facts`): the data is the mesh's, the format is the holder's.
//
// What the holder places is narrowed by the shape's placing fields — the bar's `bar` and `place` — in
// the placeholder, `${contribution:<seat>:<kind>:<field>=<value>,…}`, and every combination of them is
// placed by the holder exactly once, so nothing a module contributes is dropped by a holder that forgot
// a corner of the shape.
// Shape is what a kind received as data looks like.
type Shape struct {
// Places are the fields a holder narrows its placeholders by, each with every value it takes.
Places []PlaceField
// Order is the whole-number field that orders the pieces within one placeholder (lowest first,
// then module order); absent in a piece means OrderDefault.
Order string
// Check is what else is wrong with one piece's data, beyond its placing fields and its order.
Check func(data map[string]any) []string `json:"-"`
// Examples are pieces every holder's template must render, one for each variant the shape has,
// so a holder that cannot render one is refused at registration and not found on a machine.
Examples []map[string]any
// Reads is the state on the bus a piece shows, as `<module>.<name>`, or nothing (novox/hq ADR
// 0255): a contributor offers only state it keeps itself, and the holder on the same machine is
// granted to read it, so the holder's manifest names no contributor.
Reads func(data map[string]any) []string `json:"-"`
}
// PlaceField is one field a holder places by, and the values it takes.
type PlaceField struct {
Field string
Values []string
}
// OrderDefault is a piece's order when it gives none: the middle of 0–99.
const OrderDefault = 50
// capabilityName is the shape of a capability's name, as a node's profile reports it.
var capabilityName = regexp.MustCompile(`^[a-z0-9][a-z0-9-]*$`)
// KnownCapabilities is every capability the node-engine detects (mesh-host internal/profile), the
// names an `if-capability` may give (novox/hq ADR 0255). A name nothing detects would leave its
// contribution out on every machine, silently, so it is refused. A detector added there is added here.
var KnownCapabilities = []string{
"battery", "container-runtime", "firewall", "graphical-session", "overlay", "package-manager",
"power-meter", "privileged", "seat", "service-manager", "uplink-dhcpcd", "uplink-networkmanager",
"uplink-systemd-networkd", "virtualisation",
}
// renderFuncs are the functions a holder's template may call. `quote` writes a value as a JSON
// string with DEL escaped as well — JSON leaves it bare, and TOML refuses it — which makes it a valid
// basic string in TOML and in most tools' grammars, so a contributed command with quotes or control
// characters in it cannot break out of the holder's line.
var renderFuncs = template.FuncMap{"quote": quote}
func quote(v any) (string, error) {
b, err := json.Marshal(fmt.Sprint(v))
return strings.ReplaceAll(string(b), "\x7f", `\u007f`), err
}
// shapedProblems is what is wrong with one contribution of a kind received as data.
func shapedProblems(module string, i int, r Receivable, c SeatContribution) []string {
var problems []string
at := fmt.Sprintf("%s's contribution %d to %s (%s)", module, i+1, c.Seat, c.Kind)
if strings.TrimSpace(c.Content) != "" {
problems = append(problems, at+" has content; this kind is data, given as `data` (novox/hq ADR 0255)")
}
if len(c.Data) == 0 {
return append(problems, at+" has no data")
}
for _, p := range r.Shape.Places {
v, _ := c.Data[p.Field].(string)
if !oneOf(p.Values, v) {
problems = append(problems, fmt.Sprintf("%s says %s %q; it is one of %s",
at, p.Field, fmt.Sprint(c.Data[p.Field]), strings.Join(p.Values, ", ")))
}
}
if r.Shape.Order != "" {
if v, has := c.Data[r.Shape.Order]; has {
if _, ok := wholeIn(v, 0, 99); !ok {
problems = append(problems, fmt.Sprintf("%s says %s %v; it is a whole number from 0 to 99",
at, r.Shape.Order, v))
}
}
}
if r.Shape.Check != nil {
for _, p := range r.Shape.Check(c.Data) {
problems = append(problems, at+": "+p)
}
}
return problems
}
// ownStateProblems is a data contribution showing state its module does not keep (novox/hq ADR 0255):
// a module offers the holder its own state and nobody else's, so what it publishes stays its own.
func ownStateProblems(m Manifest, i int, r Receivable, c SeatContribution) []string {
if r.Shape == nil || r.Shape.Reads == nil {
return nil
}
keeps := map[string]bool{}
for _, s := range m.State {
keeps[s.Name] = true
}
var problems []string
for _, read := range r.Shape.Reads(c.Data) {
module, local, err := ReadState(read)
switch {
case err != nil:
problems = append(problems, fmt.Sprintf("%s's contribution %d shows state %v", m.Module, i+1, err))
case module != m.Module || !keeps[local]:
problems = append(problems, fmt.Sprintf("%s's contribution %d shows the state %s, which %s does not keep; a "+
"module offers its own state only (novox/hq ADR 0255)", m.Module, i+1, read, m.Module))
}
}
return problems
}
// ReadsGranted is the state a holder reads because modules on its machine offer it in pieces of a
// kind its seat receives as data (novox/hq ADR 0255), sorted and once each. The bus grants the holder
// these beside its own `reads`: the holder names no contributor, and reads only what is offered to it
// on the machine it runs on.
func ReadsGranted(holder Manifest, onMachine []Manifest) []string {
seen := map[string]bool{}
for _, claim := range holder.Claims {
s, known := SeatNamed(claim.Name)
if !known {
continue
}
for _, r := range s.Receives {
if r.Shape == nil || r.Shape.Reads == nil {
continue
}
for _, m := range onMachine {
for _, c := range m.allContributions() {
if cs, ok := SeatNamed(c.Seat); !ok || cs.Name != s.Name || c.Kind != r.Kind {
continue
}
for _, read := range r.Shape.Reads(c.Data) {
if module, _, err := ReadState(read); err == nil && module == m.Module && module != holder.Module {
seen[read] = true
}
}
}
}
}
}
return sortedKeys(seen)
}
// wholeIn is v as a whole number within [lo, hi]; JSON gives every number as a float.
func wholeIn(v any, lo, hi int) (int, bool) {
f, ok := v.(float64)
if !ok {
if i, isInt := v.(int); isInt {
f, ok = float64(i), true
}
}
if !ok || f != math.Trunc(f) || f < float64(lo) || f > float64(hi) {
return 0, false
}
return int(f), true
}
// placeholderWhere parses the narrowing part of a placeholder, `bar=bottom,place=status`, against the
// kind's shape. Empty narrows nothing.
func placeholderWhere(r Receivable, where string) (map[string]string, error) {
out := map[string]string{}
if where == "" {
return out, nil
}
if r.Shape == nil {
return nil, fmt.Errorf("%s is text, which a placeholder does not narrow", r.Kind)
}
for _, pair := range strings.Split(where, ",") {
field, value, ok := strings.Cut(pair, "=")
var p *PlaceField
for i := range r.Shape.Places {
if r.Shape.Places[i].Field == field {
p = &r.Shape.Places[i]
}
}
switch {
case !ok || p == nil:
return nil, fmt.Errorf("%q narrows by %s, and %s is placed by %s", where, field, r.Kind, placeFields(r))
case !oneOf(p.Values, value):
return nil, fmt.Errorf("%q says %s=%s; %s is one of %s", where, field, value, field, strings.Join(p.Values, ", "))
case out[field] != "":
return nil, fmt.Errorf("%q names %s twice", where, field)
}
out[field] = value
}
return out, nil
}
func placeFields(r Receivable) string {
var names []string
for _, p := range r.Shape.Places {
names = append(names, p.Field)
}
return strings.Join(names, ", ")
}
// holderTemplate is the holder's template for a kind, parsed.
func holderTemplate(holder Manifest, seat Seat, kind string) (*template.Template, error) {
for _, c := range holder.Claims {
if s, known := SeatNamed(c.Name); !known || s.Name != seat.Name {
continue
}
text, has := c.Renders[kind]
if !has || strings.TrimSpace(text) == "" {
break
}
t, err := template.New(seat.Name + ":" + kind).Funcs(renderFuncs).Parse(text)
if err != nil {
return nil, fmt.Errorf("%s's template for %s:%s does not parse: %v", holder.Module, seat.Name, kind, err)
}
return t, nil
}
return nil, fmt.Errorf("%s places %s:%s, which is data, and its claim gives no template for it in `renders` "+
"(novox/hq ADR 0255)", holder.Module, seat.Name, kind)
}
// renderPiece is one piece of data in the holder's grammar. The template sees the piece's fields,
// `module`, the contributor, and `machine`, the node it is composed for. A value the piece does not have reads as nothing in a `with` or an `if`;
// printed bare it would write `<no value>` into the tool's file, which is refused here instead.
func renderPiece(t *template.Template, module, machine string, data map[string]any) (string, error) {
view := map[string]any{"module": module, "machine": machine}
for k, v := range data {
view[k] = v
}
var b bytes.Buffer
if err := t.Execute(&b, view); err != nil {
return "", fmt.Errorf("rendering %s's piece: %v", module, err)
}
out := b.String()
if strings.Contains(out, "<no value>") {
return "", fmt.Errorf("rendering %s's piece printed a value it does not have: %q", module, out)
}
return out, nil
}
// placedPiece is one contributed piece on its way into a holder's file.
type placedPiece struct {
module string
data map[string]any
order int
}
// shapedContributions is every module's data of one kind to one seat, narrowed by where and by the
// machine's capabilities, ordered, and rendered through the holder's template — each piece under a
// comment line naming its module.
//
// A piece the template renders to nothing — a `shows` the holder does not know yet — or fails on is
// left out and answered as unplaced, naming its module: the other pieces and the rest of the machine's
// declaration go on (novox/hq ADR 0255).
func shapedContributions(modules []Manifest, holder Manifest, machine string, s Seat, r Receivable, where map[string]string, caps map[string]bool) (string, []string, error) {
var pieces []placedPiece
for _, m := range inModuleOrder(modules) {
for _, c := range m.allContributions() {
if c.Kind != r.Kind || !capable(c, caps) {
continue
}
if cs, known := SeatNamed(c.Seat); !known || cs.Name != s.Name {
continue
}
match := true
for field, value := range where {
if v, _ := c.Data[field].(string); v != value {
match = false
}
}
if !match {
continue
}
order := OrderDefault
if v, ok := wholeIn(c.Data[r.Shape.Order], 0, 99); ok {
order = v
}
pieces = append(pieces, placedPiece{module: m.Module, data: c.Data, order: order})
}
}
if len(pieces) == 0 {
return "", nil, nil
}
t, err := holderTemplate(holder, s, r.Kind)
if err != nil {
return "", nil, err
}
sort.SliceStable(pieces, func(a, b int) bool { return pieces[a].order < pieces[b].order })
var b strings.Builder
var unplaced []string
for _, p := range pieces {
out, err := renderPiece(t, p.module, machine, p.data)
if err == nil && strings.TrimSpace(out) == "" {
err = fmt.Errorf("%s's template renders nothing for it", holder.Module)
}
if err != nil {
unplaced = append(unplaced, fmt.Sprintf("%s's %s:%s %s is not placed by %s: %v",
p.module, s.Name, r.Kind, describePiece(p.data), holder.Module, err))
continue
}
fmt.Fprintf(&b, "%s %s\n", r.Comment, p.module)
b.WriteString(out)
if !strings.HasSuffix(out, "\n") {
b.WriteString("\n")
}
}
return b.String(), unplaced, nil
}
// describePiece is a piece in one line, its fields in order, for a person reading why it was left out.
func describePiece(data map[string]any) string {
var parts []string
for _, k := range sortedKeys(data) {
if _, nested := data[k].(map[string]any); nested {
continue
}
parts = append(parts, fmt.Sprintf("%s=%v", k, data[k]))
}
return "(" + strings.Join(parts, " ") + ")"
}
// capable is whether a contribution applies on a machine with these capabilities: always, unless it
// names one with `if-capability` the machine did not report (novox/hq ADR 0255). A machine whose
// capabilities are not known has none, so a conditional piece is left out rather than guessed in.
func capable(c SeatContribution, caps map[string]bool) bool {
return c.IfCapability == "" || caps[c.IfCapability]
}
// placementProblems is what is wrong with how a holder places the kinds its seats receive as data:
// once it places one at all, it needs a template that renders every example of the shape, and every combination of the placing
// fields placed exactly once across its files — or something contributed would be dropped, or written
// twice (novox/hq ADR 0255).
func (m Manifest) placementProblems() []string {
var problems []string
for _, c := range m.Claims {
s, known := SeatNamed(c.Name)
if !known {
continue
}
for _, r := range s.Receives {
// A holder that places none of the kind is older than it (novox/hq ADR 0255): refusing it
// would refuse every bar registered before the kind existed, and the controller that
// defines the kind ships first. Its contributions wait, unplaced, until it places them.
if r.Shape == nil || !placesKind(m, s, r.Kind) {
continue
}
t, err := holderTemplate(m, s, r.Kind)
if err != nil {
problems = append(problems, err.Error())
continue
}
for _, ex := range r.Shape.Examples {
if out, err := renderPiece(t, "example", "machine", ex); err != nil || strings.TrimSpace(out) == "" {
problems = append(problems, fmt.Sprintf("%s's template for %s:%s renders nothing for %v: %v",
m.Module, s.Name, r.Kind, ex, err))
}
}
placed := map[string]int{}
for _, res := range m.Resources {
content, _ := res["content"].(string)
for _, f := range ofContribution.FindAllStringSubmatch(content, -1) {
seat, rest, _ := strings.Cut(f[1], ":")
kind, whereText, _ := strings.Cut(rest, ":")
if cs, ok := SeatNamed(seat); !ok || cs.Name != s.Name || kind != r.Kind {
continue
}
where, err := placeholderWhere(r, whereText)
if err != nil {
continue // said by seatPlaceholderProblems
}
for _, combo := range combinations(r.Shape.Places) {
if covers(where, combo) {
placed[comboKey(combo)]++
}
}
}
}
for _, combo := range combinations(r.Shape.Places) {
switch n := placed[comboKey(combo)]; {
case n == 0:
problems = append(problems, fmt.Sprintf("%s never places %s:%s for %s; every combination of %s is "+
"placed once, or a contribution to it is lost (novox/hq ADR 0255)",
m.Module, s.Name, r.Kind, comboKey(combo), placeFields(r)))
case n > 1:
problems = append(problems, fmt.Sprintf("%s places %s:%s for %s %d times; once, or a contribution "+
"is written twice (novox/hq ADR 0255)", m.Module, s.Name, r.Kind, comboKey(combo), n))
}
}
}
}
return problems
}
// placesKind is whether any of a holder's files names the kind's placeholder.
func placesKind(m Manifest, s Seat, kind string) bool {
for _, res := range m.Resources {
content, _ := res["content"].(string)
for _, f := range ofContribution.FindAllStringSubmatch(content, -1) {
seat, rest, _ := strings.Cut(f[1], ":")
k, _, _ := strings.Cut(rest, ":")
if cs, ok := SeatNamed(seat); ok && cs.Name == s.Name && k == kind {
return true
}
}
}
return false
}
// combinations is every assignment of a value to each placing field.
func combinations(places []PlaceField) []map[string]string {
out := []map[string]string{{}}
for _, p := range places {
var next []map[string]string
for _, partial := range out {
for _, v := range p.Values {
c := map[string]string{p.Field: v}
for k, w := range partial {
c[k] = w
}
next = append(next, c)
}
}
out = next
}
return out
}
func covers(where, combo map[string]string) bool {
for field, value := range where {
if combo[field] != value {
return false
}
}
return true
}
func comboKey(combo map[string]string) string {
var parts []string
for _, k := range sortedKeys(combo) {
parts = append(parts, k+"="+combo[k])
}
return strings.Join(parts, ",")
}
+395
View File
@@ -0,0 +1,395 @@
package catalogue
import (
"encoding/json"
"fmt"
"strconv"
"strings"
"testing"
)
// Defends novox/hq ADR 0255: a seat may receive a kind as data in its own shape, which the holder
// renders with its template; the holder places every combination of the shape's placing fields once;
// a contribution may be placed only where the machine reports a capability; and an offered kind makes
// no dependency on its seat.
const barTemplate = `{{if eq .shows "battery"}}[[block]]
block = "battery"
{{with .options}}{{with .device}}device = {{quote .}}
{{end}}{{end}}{{else if eq .shows "command"}}[[block]]
block = "custom"
command = {{quote .options.command}}
interval = {{.options.interval}}
{{else if eq .shows "state"}}[[block]]
block = "custom"
command = {{quote (printf "show %s %s" .options.state (or .options.key .machine))}}
{{end}}`
func barHolder() Manifest {
place := func(bar string) string {
return "own-first\n${contribution:node-bar:block:bar=" + bar + ",place=resources}" +
"own-middle\n${contribution:node-bar:block:bar=" + bar + ",place=status}own-last\n"
}
return Manifest{Module: "a-bar", Claims: []Claim{{Name: BarSeat, Renders: map[string]string{BarKindBlock: barTemplate}}},
Resources: []map[string]any{
{"id": "bottom", "type": "file", "path": "/b", "content": place("bottom")},
{"id": "top", "type": "file", "path": "/t", "content": place("top")},
}}
}
func block(data string) SeatContribution {
var d map[string]any
if err := json.Unmarshal([]byte(data), &d); err != nil {
panic(err)
}
return SeatContribution{Seat: BarSeat, Kind: BarKindBlock, Data: d}
}
func composedFile(t *testing.T, r Resolution, id string) string {
t.Helper()
out, err := r.Declaration(Rendering{})
if err != nil {
t.Fatal(err)
}
for _, res := range out {
if res["id"] == id {
return res["content"].(string)
}
}
t.Fatalf("%s was not composed", id)
return ""
}
func TestABlockIsRenderedByTheHolderInItsPlaceAndOrderNamedByModule(t *testing.T) {
power := Manifest{Module: "power", Contributions: []SeatContribution{
block(`{"bar":"bottom","place":"status","shows":"battery"}`),
block(`{"bar":"bottom","place":"status","order":20,"shows":"command","options":{"command":"draw \"now\"","interval":5}}`),
}}
other := Manifest{Module: "another", Contributions: []SeatContribution{
block(`{"bar":"bottom","place":"status","order":70,"shows":"battery","options":{"device":"BAT1"}}`),
block(`{"bar":"top","place":"resources","shows":"command","options":{"command":"up","interval":60}}`),
}}
r := Resolution{Node: "laptop", Account: "op", Modules: []Manifest{barHolder(), power, other}}
// Ordered by order (20, 50, 70), then module order; each piece under its module's name; the
// command quoted so its own quotes cannot end the line.
want := "own-first\nown-middle\n" +
"# power\n[[block]]\nblock = \"custom\"\ncommand = \"draw \\\"now\\\"\"\ninterval = 5\n" +
"# power\n[[block]]\nblock = \"battery\"\n" +
"# another\n[[block]]\nblock = \"battery\"\ndevice = \"BAT1\"\n" +
"own-last\n"
if got := composedFile(t, r, "a-bar.bottom"); got != want {
t.Fatalf("the bottom bar is\n%s\nnot\n%s", got, want)
}
want = "own-first\n# another\n[[block]]\nblock = \"custom\"\ncommand = \"up\"\ninterval = 60\nown-middle\nown-last\n"
if got := composedFile(t, r, "a-bar.top"); got != want {
t.Fatalf("the top bar is\n%s\nnot\n%s", got, want)
}
}
func TestAContributionIfACapabilityIsPlacedOnlyWhereTheMachineReportsIt(t *testing.T) {
c := block(`{"bar":"bottom","place":"status","shows":"battery"}`)
c.IfCapability = "battery"
power := Manifest{Module: "power", Contributions: []SeatContribution{c}}
for _, c := range []struct {
caps map[string]bool
has bool
}{{map[string]bool{"battery": true}, true}, {map[string]bool{"seat": true}, false}, {nil, false}} {
r := Resolution{Node: "n", Account: "op", Capabilities: c.caps, Modules: []Manifest{barHolder(), power}}
if bar := composedFile(t, r, "a-bar.bottom"); strings.Contains(bar, `block = "battery"`) != c.has {
t.Errorf("with %v: the battery placed is not %v:\n%s", c.caps, c.has, bar)
}
}
}
func TestIfACapabilityNamesOneTheNodeEngineDetectsOnAnOfferedKindOnly(t *testing.T) {
cases := map[string]string{
// A name nothing detects would leave the piece out on every machine.
`{"seat":"node-bar","kind":"block","if-capability":"batery","data":{"bar":"top","place":"status","shows":"battery"}}`: "which no machine reports",
// A kind the holder depends on is never left out for want of something.
`{"seat":"node-hotkeys","kind":"trigger","if-capability":"battery","content":"KEY_BATTERY 1 x"}`: "only a kind the seat offers",
`{"seat":"node-backup","kind":"backup","if-capability":"seat","content":"path /x"}`: "only a kind the seat offers",
}
for c, want := range cases {
raw := `{"module":"power","contributions":[` + c + `]}`
if _, err := ParseManifest([]byte(raw)); err == nil || !strings.Contains(err.Error(), want) {
t.Errorf("%s: accepted, or refused without %q: %v", c, want, err)
}
}
for _, name := range []string{"battery", "power-meter", "seat"} {
if !oneOf(KnownCapabilities, name) {
t.Errorf("%s is not a known capability", name)
}
}
}
func TestResolveCarriesTheMachinesCapabilities(t *testing.T) {
node := workstation()
node.Capabilities["battery"] = true
r, err := Resolve(shelf(Manifest{Module: "a"}), []string{"a"}, node, World{})
if err != nil {
t.Fatal(err)
}
if !r.Capabilities["battery"] || !r.Capabilities["seat"] {
t.Fatalf("the resolution carries %v, not the node's %v", r.Capabilities, node.Capabilities)
}
}
func TestAPieceTheHolderCannotRenderIsLeftOutAndNamedNotTheMachine(t *testing.T) {
// A shows the shape gained after the holder's template was written: built here, past the check.
power := Manifest{Module: "power", Contributions: []SeatContribution{
block(`{"bar":"bottom","place":"status","shows":"gpu"}`),
block(`{"bar":"bottom","place":"status","shows":"battery"}`),
}}
r := Resolution{Node: "n", Account: "op", Modules: []Manifest{barHolder(), power}}
c, err := r.Compose(Rendering{})
if err != nil {
t.Fatalf("one piece failed the whole machine: %v", err)
}
if len(c.Unplaced) != 1 || !strings.Contains(c.Unplaced[0], "power's node-bar:block") ||
!strings.Contains(c.Unplaced[0], "shows=gpu") || !strings.Contains(c.Unplaced[0], "renders nothing") {
t.Fatalf("unplaced: %v", c.Unplaced)
}
for _, res := range c.Resources {
if res["id"] == "a-bar.bottom" && !strings.Contains(res["content"].(string), `block = "battery"`) {
t.Fatalf("the piece beside it was lost too: %s", res["content"])
}
}
}
func TestTheCataloguesBarRendersEveryExampleOfTheShape(t *testing.T) {
holder := catalogueManifest(t, "i3status-rust")
s, _ := SeatNamed(BarSeat)
if !placesKind(holder, s, BarKindBlock) {
t.Skip("the catalogue beside this checkout has a bar that places no blocks yet")
}
tmpl, err := holderTemplate(holder, s, BarKindBlock)
if err != nil {
t.Fatal(err)
}
for _, ex := range s.Receives[0].Shape.Examples {
out, err := renderPiece(tmpl, "example", "a-machine", ex)
if err != nil || !strings.Contains(out, "[[block]]") {
t.Errorf("the catalogue's bar renders %v as %q: %v", ex, out, err)
}
}
for shows := range barShows {
covered := false
for _, ex := range s.Receives[0].Shape.Examples {
covered = covered || ex["shows"] == shows
}
if !covered {
t.Errorf("the shape has no example that shows %s, so no holder is made to render it", shows)
}
}
}
func TestQuoteIsATomlBasicStringOfTheSameText(t *testing.T) {
for _, in := range []string{`plain`, `say "hi" \ there`, "tab\tnew\nline", "del\x7fend", "nul\x00 esc\x1b", "<&> é ☃"} {
q, err := quote(in)
if err != nil {
t.Fatal(err)
}
got, err := tomlBasicString(q)
if err != nil || got != in {
t.Errorf("quote(%q) = %s, which TOML reads as %q: %v", in, q, got, err)
}
}
}
// tomlBasicString reads one TOML basic string, as the TOML specification (1.0, "String") defines it:
// any character but a quote, a backslash and the control characters U+0000–U+0008, U+000A–U+001F
// and U+007F, which are written as escapes.
func tomlBasicString(s string) (string, error) {
if len(s) < 2 || s[0] != '"' || s[len(s)-1] != '"' {
return "", fmt.Errorf("not quoted")
}
var b strings.Builder
rs := []rune(s[1 : len(s)-1])
for i := 0; i < len(rs); i++ {
r := rs[i]
switch {
case r == '"':
return "", fmt.Errorf("a bare quote at %d", i)
case r == 0x7f || (r < 0x20 && r != '\t'):
return "", fmt.Errorf("a bare control character %U at %d", r, i)
case r != '\\':
b.WriteRune(r)
continue
}
i++
if i >= len(rs) {
return "", fmt.Errorf("an escape at the end")
}
switch rs[i] {
case 'b':
b.WriteRune('\b')
case 't':
b.WriteRune('\t')
case 'n':
b.WriteRune('\n')
case 'f':
b.WriteRune('\f')
case 'r':
b.WriteRune('\r')
case '"':
b.WriteRune('"')
case '\\':
b.WriteRune('\\')
case 'u', 'U':
n := 4
if rs[i] == 'U' {
n = 8
}
if i+n > len(rs)-1 {
return "", fmt.Errorf("a short \\u escape")
}
v, err := strconv.ParseUint(string(rs[i+1:i+1+n]), 16, 32)
if err != nil {
return "", err
}
b.WriteRune(rune(v))
i += n
default:
return "", fmt.Errorf("the escape \\%c is not TOML's", rs[i])
}
}
return b.String(), nil
}
func TestABlockOutsideTheShapeIsRefused(t *testing.T) {
cases := map[string]string{
`{"seat":"node-bar","kind":"block","content":"[[block]]"}`: "has no data",
`{"seat":"node-bar","kind":"block","content":"x","data":{"bar":"top","place":"status","shows":"battery"}}`: "has content",
`{"seat":"node-hotkeys","kind":"trigger","data":{"a":"b"}}`: "receives trigger as text",
`{"seat":"node-bar","kind":"block","data":{"bar":"left","place":"status","shows":"battery"}}`: `bar "left"`,
`{"seat":"node-bar","kind":"block","data":{"bar":"top","place":"middle","shows":"battery"}}`: `place "middle"`,
`{"seat":"node-bar","kind":"block","data":{"bar":"top","place":"status","shows":"gpu"}}`: "shows one of battery, command",
`{"seat":"node-bar","kind":"block","data":{"bar":"top","place":"status","shows":"battery","colour":"red"}}`: `no field "colour"`,
`{"seat":"node-bar","kind":"block","data":{"bar":"top","place":"status","shows":"battery","order":100}}`: "whole number from 0 to 99",
`{"seat":"node-bar","kind":"block","data":{"bar":"top","place":"status","shows":"command"}}`: "needs the option command",
`{"seat":"node-bar","kind":"block","data":{"bar":"top","place":"status","shows":"battery","options":{"x":"y"}}}`: `no option "x"`,
`{"seat":"node-bar","kind":"block","data":{"bar":"top","place":"status","shows":"command","options":{"command":"c","interval":0}}}`: "seconds",
`{"seat":"node-bar","kind":"block","if-capability":"Has Battery","data":{"bar":"top","place":"status","shows":"battery"}}`: "not a capability",
}
for c, want := range cases {
raw := `{"module":"power","contributions":[` + c + `]}`
if _, err := ParseManifest([]byte(raw)); err == nil || !strings.Contains(err.Error(), want) {
t.Errorf("%s: accepted, or refused without %q: %v", c, want, err)
}
}
ok := `{"module":"power","contributions":[{"seat":"node-bar","kind":"block","if-capability":"battery",` +
`"data":{"bar":"bottom","place":"status","order":10,"shows":"command","options":{"command":"c","interval":5}}}]}`
if _, err := ParseManifest([]byte(ok)); err != nil {
t.Errorf("a block in the shape was refused: %v", err)
}
}
func TestAHolderPlacesEveryCombinationOnceWithATemplateThatRendersEveryExample(t *testing.T) {
holder := func(template string, contents ...string) string {
var res []string
for i, c := range contents {
b, _ := json.Marshal(c)
res = append(res, `{"id":"f`+string(rune('a'+i))+`","type":"file","path":"/f`+string(rune('a'+i))+`","content":`+string(b)+`}`)
}
tb, _ := json.Marshal(template)
renders := ""
if template != "" {
renders = `,"renders":{"block":` + string(tb) + `}`
}
return `{"module":"a-bar","claims":[{"name":"node-bar"` + renders + `}],"resources":[` + strings.Join(res, ",") + `]}`
}
all := "${contribution:node-bar:block:bar=bottom,place=resources}${contribution:node-bar:block:bar=bottom,place=status}" +
"${contribution:node-bar:block:bar=top,place=resources}${contribution:node-bar:block:bar=top,place=status}"
cases := map[string]string{
holder(barTemplate, "${contribution:node-bar:block:bar=bottom}"): "never places node-bar:block for bar=top,place=resources",
holder(barTemplate, all, "${contribution:node-bar:block:place=status}"): "for bar=bottom,place=status 2 times",
holder("", all): "gives no template",
holder(`{{if eq .shows "battery"}}b{{end}}`, all): "renders nothing",
holder(`{{.nothing}}`, all): "renders nothing",
holder(`{{if}`, all): "does not parse",
holder(barTemplate, "${contribution:node-bar:block:bar=left}"+all): "bar is one of bottom, top",
holder(barTemplate, "${contribution:node-bar:block:colour=red}"+all): "placed by bar, place",
holder(barTemplate, "${contribution:node-hotkeys:trigger:bar=top}"): "does not claim node-hotkeys",
}
for raw, want := range cases {
if _, err := ParseManifest([]byte(raw)); err == nil || !strings.Contains(err.Error(), want) {
t.Errorf("%s:\naccepted, or refused without %q: %v", raw, want, err)
}
}
for _, raw := range []string{holder(barTemplate, all), holder(barTemplate, "${contribution:node-bar:block}"),
// A holder that places no block at all is older than the kind, and is not refused for it.
holder("", "no blocks here")} {
if _, err := ParseManifest([]byte(raw)); err != nil {
t.Errorf("%s: refused: %v", raw, err)
}
}
}
func TestAnOfferedBlockMakesNoDependencyOnTheBar(t *testing.T) {
power := Manifest{Module: "power", Contributions: []SeatContribution{
block(`{"bar":"bottom","place":"status","shows":"battery"}`),
{Seat: HotkeysSeat, Kind: "trigger", Content: "x"},
}}
for _, seat := range DependsOn(power) {
if seat == BarSeat {
t.Fatalf("a block made the power module depend on the bar: %v", DependsOn(power))
}
}
// And on a machine with no bar it is placed nowhere, and nothing fails.
r := Resolution{Node: "server", Account: "op", Capabilities: map[string]bool{"battery": true},
Modules: []Manifest{hotkeysHolder(), power}}
if _, err := r.Declaration(Rendering{}); err != nil {
t.Fatal(err)
}
}
func TestTheBarReceivesBlocksAsData(t *testing.T) {
s, _ := SeatNamed(BarSeat)
if len(s.Receives) != 1 || s.Receives[0].Kind != BarKindBlock || s.Receives[0].Shape == nil || !s.Receives[0].Offered {
t.Fatalf("node-bar receives %+v", s.Receives)
}
for _, ex := range s.Receives[0].Shape.Examples {
if p := barBlockProblems(ex); len(p) > 0 {
t.Errorf("the shape's own example %v is refused: %v", ex, p)
}
}
}
func TestAStateBlockIsRenderedForThisMachineAndTheHolderIsGrantedToReadIt(t *testing.T) {
power := Manifest{Module: "power", State: []StateDeclaration{{Name: "draw"}}, Contributions: []SeatContribution{
block(`{"bar":"bottom","place":"status","shows":"state","options":{"state":"power.draw"}}`),
}}
r := Resolution{Node: "laptop", Account: "op", Modules: []Manifest{barHolder(), power}}
if bar := composedFile(t, r, "a-bar.bottom"); !strings.Contains(bar, `command = "show power.draw laptop"`) {
t.Fatalf("the block does not read this machine's key:\n%s", bar)
}
if got := ReadsGranted(barHolder(), []Manifest{barHolder(), power}); len(got) != 1 || got[0] != "power.draw" {
t.Fatalf("the holder is granted %v", got)
}
// Only the holder, and only for what the modules on its machine offer.
if got := ReadsGranted(power, []Manifest{barHolder(), power}); len(got) != 0 {
t.Fatalf("a contributor is granted %v", got)
}
if got := ReadsGranted(barHolder(), []Manifest{barHolder()}); len(got) != 0 {
t.Fatalf("a holder alone is granted %v", got)
}
}
func TestAModuleOffersItsOwnStateOnly(t *testing.T) {
block := `{"seat":"node-bar","kind":"block","data":{"bar":"bottom","place":"status","shows":"state","options":{"state":"%s"}}}`
cases := map[string]string{
`{"module":"power","state":["draw"],"contributions":[` + fmt.Sprintf(block, "other.draw") + `]}`: "which power does not keep",
`{"module":"power","contributions":[` + fmt.Sprintf(block, "power.draw") + `]}`: "which power does not keep",
`{"module":"power","state":["draw"],"contributions":[` + fmt.Sprintf(block, "draw") + `]}`: "does not name a module",
}
for raw, want := range cases {
if _, err := ParseManifest([]byte(raw)); err == nil || !strings.Contains(err.Error(), want) {
t.Errorf("%s: accepted, or refused without %q: %v", raw, want, err)
}
}
ok := `{"module":"power","state":["draw"],"contributions":[` + fmt.Sprintf(block, "power.draw") + `]}`
if _, err := ParseManifest([]byte(ok)); err != nil {
t.Errorf("its own state was refused: %v", err)
}
}
+4 -1
View File
@@ -118,8 +118,11 @@ func contributedTo(m Manifest) []string {
}
// Any other seat's contribution (novox/hq ADR 0212 §4), by the seat's canonical name; one the
// mesh does not define is refused at registration and depends on nothing here.
// A kind the seat only offers to place makes no dependency (novox/hq ADR 0255).
for _, c := range m.Contributions {
if s, known := SeatNamed(c.Seat); known {
if s, r, ok := receivable(c.Seat, c.Kind); ok && r.Offered {
continue
} else if s.Name != "" {
out = append(out, s.Name)
}
}
+7
View File
@@ -68,6 +68,13 @@ type Receivable struct {
// only the mesh knows where. Off for every kind written in a tool's grammar that has its own
// `${…}` — a shell's — where the mesh filling one would change what the tool reads.
Dirs bool
// Shape, when set, makes the kind data rather than text (novox/hq ADR 0255): a contributor gives
// `data` in this shape, and the holder renders each piece with the template its claim gives.
Shape *Shape
// Offered says a contribution of this kind does not depend on the seat (novox/hq ADR 0255): where
// nobody holds it nothing is placed, and the contributor runs there all the same. A battery's bar
// block is offered — the power module is on every machine, and a server has no bar.
Offered bool
}
// defaultSeats is the set the mesh ships with — the seed for the control plane's seat table and the
+28 -3
View File
@@ -178,9 +178,17 @@ var ControllerVerbs = []Verb{
"diff": "\"true\": what a push would change on this machine, against what it was last sent; not with files",
}, []string{"node"}, "files", "diff")},
{Name: "assign", Description: "Put a module on a machine. Refused with the mesh's own words when it cannot resolve there, " +
"or when a seat its resources are applied through is held by nothing on the machine (novox/hq ADR 0207).",
"or when a seat its resources are applied through is held by nothing on the machine (novox/hq ADR 0207). " +
"A module not registered yet is looked for (novox/hq issue 325, ADR 0261): while its build runs — a merge " +
"that adds a module asks for one — the assignment is kept as pending and the controller makes it when the " +
"build registers it; `status` lists it, and if the build fails it expires with why and raises a condition. " +
"A module known and not built is said, and build asks for its build; a name with no module and no build " +
"request kept is refused with the closest names.",
Input: schema(map[string]string{"node": "the machine's name",
"module": "the module's name; several comma-separated are judged together"}, []string{"node", "module"})},
"module": "the module's name; several comma-separated are judged together",
"build": "\"true\": for a module known and not built (its last build failed, or was never asked), ask " +
"for its build and keep this assignment pending until the build registers it"},
[]string{"node", "module"}, "build")},
{Name: "unassign", Description: "Take a module off a machine. Refused when it holds a seat a module left there depends on.",
Input: schema(map[string]string{"node": "the machine's name",
"module": "the module's name; several comma-separated are judged together"}, []string{"node", "module"})},
@@ -449,7 +457,10 @@ var ControllerVerbs = []Verb{
{Name: "collect", Description: "The artifact store's sweep, now: what the mesh made and keeps for no reason " +
"is let go of. Without confirm a dry run — every kept archive asked about with HEADs, what would be let go " +
"listed, nothing held or deleted. With confirm and why: every kept archive held first, then each eligible " +
"artifact let go of, oldest first, and recorded collected — a hand act, recorded with its why. Bounded by " +
"artifact let go of, oldest first, and recorded collected — a hand act, recorded with its why. A confirmed " +
"collect lets an index go with the platform manifests no kept index names (read first; any read that " +
"fails lets nothing go); the sweep after a build leaves an eligible index for such a collect (novox/hq ADR " +
"0257). Bounded by " +
"most (500 by default, at most 5000) and 45 seconds; what is left is said. Bytes are reclaimed by the " +
"store's nightly collector. Never touches a digest the mesh did not record (novox/hq ADR 0189, ADR 0251).",
Input: schema(map[string]string{
@@ -457,6 +468,20 @@ var ControllerVerbs = []Verb{
"why": "with confirm: why — required, and recorded in the hand-act log",
"most": "let go of at most this many (500 by default, at most 5000)",
}, nil, "confirm")},
{Name: "mirrors", Description: "The bases the mesh copied into its artifact store for builds to stand on, as " +
"the records hold them: each with its state — kept because a kept build stood on it, or eligible — and " +
"which build or person recorded it. With record: references of copies no record names (a copy made " +
"before copies were recorded, or by a build that failed before then) to record as the mesh's, so the " +
"sweep keeps or lets go of them like any other; only a repository the mirror writes is taken " +
"(upstream/<host>/<path>, or a module's <module>/on-<argument>), and each must be in the store. Without " +
"confirm a dry run; with confirm and why, recorded — a hand act. A recorded copy is eligible: the next " +
"sweep lets it go unless a kept build stood on it. Records only: this verb deletes nothing " +
"(novox/hq ADR 0257).",
Input: schema(map[string]string{
"record": "references to record, artifact-store://<repository>@sha256:<hex>, separated by spaces or commas",
"confirm": "\"true\": record them (needs why); without it nothing is recorded",
"why": "with confirm: why — required, and recorded with each copy and in the hand-act log",
}, nil, "confirm")},
{Name: "images", Description: "Every container image one machine's declaration names — the declaration the " +
"mesh would send it now and the one it was last sent — each with the resources naming it: what the " +
"machine keeps when it prunes its images. sent_known is false when the last one sent is not kept with " +
+10
View File
@@ -280,6 +280,13 @@ func (k *Keeper) Observe(ctx context.Context, o Observation) (Condition, error)
// Clear removes a condition an observation says is resolved, and says so. False when none was open.
func (k *Keeper) Clear(ctx context.Context, key, why string) (bool, error) {
return k.ClearSaying(ctx, key, why, "")
}
// ClearSaying is Clear with the line the operator reads when it clears, in place of the condition's own
// resolved line: for a condition that ends because it became another (novox/hq ADR 0254: a module not
// working that now waits for a new login is not "working again"). Empty keeps the condition's own.
func (k *Keeper) ClearSaying(ctx context.Context, key, why, resolved string) (bool, error) {
for i := 0; i < tries; i++ {
entry, found, err := k.store.Get(ctx, key)
if err != nil {
@@ -296,6 +303,9 @@ func (k *Keeper) Clear(ctx context.Context, key, why string) (bool, error) {
if err := k.stamp(&c); err != nil {
return false, err
}
if resolved != "" {
c.Resolved = resolved
}
if err := k.store.Delete(ctx, key, entry.Revision); errors.Is(err, ErrMoved) {
continue
} else if err != nil {
+38 -1
View File
@@ -39,6 +39,9 @@ type Build struct {
// Against is every artifact this build stood on, as references rather than module names —
// what makes a build edge derived rather than declared (ADR 0009).
Against []string
// Mirrored is every base this build copied into the artifact store (novox/hq ADR 0257), said
// whether the build worked or not: the copy is the mesh's, recorded in artifact_mirrored.
Mirrored []string
// Read is every repository this build read source from besides the module's own (novox/hq
// 04-ISSUES/131), at the ref it read.
Read []ReadRepository
@@ -120,7 +123,41 @@ func (i *Inventory) RecordBuild(ctx context.Context, b Build) error {
on conflict (id) do nothing`,
b.ID, b.Repository, b.Ref, module, b.Commit, b.On, b.Failed, made,
b.Path, manifestOrNil(b.Manifest), against, read, asked, b.SourceFingerprint)
return err
if err != nil {
return err
}
return i.RecordMirrored(ctx, b.ID, "", b.Mirrored)
}
// RecordMirrored keeps that the mesh copied these bases into the artifact store (novox/hq ADR 0257):
// by the build that said so, or by a person who says why. The first record of a copy stands.
//
// **A copy a build says it holds is held again.** The builder answers a copy only once the store holds
// it whole, copying again what was let go of; so a copy the records say was collected, and a build now
// says it copied, is no longer collected. Left collected, it would stay in the store and out of every
// sweep for ever.
func (i *Inventory) RecordMirrored(ctx context.Context, build, why string, references []string) error {
var by *string
if build != "" {
by = &build
}
for _, reference := range references {
if reference == "" {
continue
}
if _, err := i.store.Pool().Exec(ctx,
`insert into artifact_mirrored (reference, build, why) values ($1, $2, $3)
on conflict (reference) do nothing`, reference, by, why); err != nil {
return err
}
if build != "" {
if _, err := i.store.Pool().Exec(ctx,
`delete from artifact_collected where reference = $1`, reference); err != nil {
return err
}
}
}
return nil
}
// Builds is what has happened lately, newest first.
+15
View File
@@ -79,6 +79,10 @@ func (i *Inventory) BusRecords(ctx context.Context) (broker.Records, error) {
"be derived", module, n.Name)
}
d := declaredFor(m, seats)
// And the state offered to it as a seat's holder by the modules beside it (novox/hq ADR 0255).
if granted := catalogue.ReadsGranted(m, onMachine(declared, modules)); len(granted) > 0 {
d.Reads = append(append([]string(nil), d.Reads...), granted...)
}
d.Holds = heldHere(d.Holds, holdings, n.Name, module)
out.Assigned[n.Name] = append(out.Assigned[n.Name], d)
if m.Instances == catalogue.InstancesInterchangeable {
@@ -103,6 +107,17 @@ func (i *Inventory) BusRecords(ctx context.Context) (broker.Records, error) {
return out, nil
}
// onMachine is the manifests of the modules assigned to one machine that the catalogue knows.
func onMachine(declared map[string]catalogue.Manifest, modules []string) []catalogue.Manifest {
var out []catalogue.Manifest
for _, module := range modules {
if m, known := declared[module]; known {
out = append(out, m)
}
}
return out
}
// declaredFor is one module's manifest as the composer needs it: what it says about itself, and the
// protocol of every seat it holds or uses.
func declaredFor(m catalogue.Manifest, seats map[string]catalogue.SeatDeclaration) broker.Declared {
+45
View File
@@ -296,3 +296,48 @@ func TestTheBussOwnModuleBecomesTheSnapshotUser(t *testing.T) {
t.Fatalf("users derived: %v", seen)
}
}
// The bar's holder reads the state the modules beside it offer in its blocks (novox/hq ADR 0255),
// without naming them: granted on the machine where both are, and nowhere else.
func TestABarIsGrantedTheStateItsMachinesModulesOfferIt(t *testing.T) {
bar := catalogue.Manifest{Module: "a-bar", Version: "1",
Claims: []catalogue.Claim{{Name: catalogue.BarSeat, Scope: catalogue.ScopeNode}}}
power := catalogue.Manifest{Module: "power", Version: "1",
State: []catalogue.StateDeclaration{{Name: "draw"}},
Contributions: []catalogue.SeatContribution{{Seat: catalogue.BarSeat, Kind: catalogue.BarKindBlock,
Data: map[string]any{"bar": "bottom", "place": "status", "shows": "state",
"options": map[string]any{"state": "power.draw"}}}}}
inv, ctx := aMeshWith(t, bar, power)
for node, modules := range map[string][]string{"laptop": {"a-bar", "power"}, "desk": {"a-bar"}} {
if _, err := inv.AddNode(ctx, node); err != nil {
t.Fatal(err)
}
for _, module := range modules {
if _, err := inv.Assign(ctx, node, module); err != nil {
t.Fatal(err)
}
}
}
records, err := inv.BusRecords(ctx)
if err != nil {
t.Fatal(err)
}
reads := func(node, module string) []string {
for _, d := range records.Assigned[node] {
if d.Module == module {
return d.Reads
}
}
t.Fatalf("%s is not on %s", module, node)
return nil
}
if got := reads("laptop", "a-bar"); len(got) != 1 || got[0] != "power.draw" {
t.Fatalf("the laptop's bar reads %v", got)
}
if got := reads("desk", "a-bar"); len(got) != 0 {
t.Fatalf("a bar with no power beside it reads %v", got)
}
if got := reads("laptop", "power"); len(got) != 0 {
t.Fatalf("power reads %v", got)
}
}
+118 -10
View File
@@ -37,6 +37,10 @@ const (
// KeptUnaddressable: it names no digest, so nothing here can speak for it, and it is kept rather
// than guessed about.
KeptUnaddressable = "unaddressable"
// KeptStoodOn: it is a base the mesh copied in, and one of the KeptBuilds most recent successful
// builds of a module the mesh still holds stood on it (novox/hq ADR 0257). A copy has no builds of
// its own; it is kept by the builds that used it.
KeptStoodOn = "stood-on"
)
// The states an artifact the mesh recorded making is in (novox/hq ADR 0251).
@@ -77,7 +81,7 @@ func (i *Inventory) Artifacts(ctx context.Context) ([]ArtifactState, error) {
if err != nil {
return nil, err
}
all, err := i.everyReferenceMade(ctx)
all, err := i.everyReferenceRecorded(ctx)
if err != nil {
return nil, err
}
@@ -156,8 +160,9 @@ func (i *Inventory) KeepSet(ctx context.Context) (map[string][]string, error) {
// join every module ever built kept five builds' artifacts for ever — and once the store's
// collector runs for real, what the keep set says is what the disk holds.
recent, err := i.store.Pool().Query(ctx,
`select made from (
select b.made, row_number() over (partition by b.module order by b.at desc, b.id desc) as back
`select made, built_against from (
select b.made, b.built_against,
row_number() over (partition by b.module order by b.at desc, b.id desc) as back
from build b
join module m on m.name = b.module
where b.failed = '' and b.module is not null and b.module <> ''
@@ -166,18 +171,26 @@ func (i *Inventory) KeepSet(ctx context.Context) (map[string][]string, error) {
return nil, err
}
defer recent.Close()
var stoodOn []string
for recent.Next() {
var raw []byte
if err := recent.Scan(&raw); err != nil {
var raw, against []byte
if err := recent.Scan(&raw, &against); err != nil {
return nil, err
}
var made []Artifact
if err := json.Unmarshal(raw, &made); err != nil {
continue
if err := json.Unmarshal(raw, &made); err == nil {
for _, a := range made {
if reference := asRecorded(a.Reference); reference != "" {
because(reference, KeptByRecentBuild)
}
}
}
for _, a := range made {
if reference := asRecorded(a.Reference); reference != "" {
because(reference, KeptByRecentBuild)
var bases []string
if err := json.Unmarshal(against, &bases); err == nil {
for _, b := range bases {
if reference := asRecorded(b); reference != "" {
stoodOn = append(stoodOn, reference)
}
}
}
}
@@ -185,6 +198,27 @@ func (i *Inventory) KeepSet(ctx context.Context) (map[string][]string, error) {
return nil, err
}
// **A copied base is kept by the kept builds that stood on it, and by nothing else** (novox/hq
// ADR 0257). It has no builds of its own to count five of, and no machine is ever handed a base,
// so a definition is never its reason: a definition names the image upstream, by the digest every
// copy of it shares, and read as a reason it would keep every module's copy for as long as any
// module stood on that image. A copy already let go of stays let go of.
mirrored, err := i.mirroredSet(ctx)
if err != nil {
return nil, err
}
if len(mirrored) > 0 {
collected, err := i.alreadyCollected(ctx)
if err != nil {
return nil, err
}
for _, reference := range stoodOn {
if mirrored[reference] && !collected[reference] {
because(reference, KeptStoodOn)
}
}
}
// And anything a manifest mentions — every reference ever made, the recent ones included, so a
// kept artifact carries each of its reasons and the operator reads why, not only that (novox/hq
// ADR 0251). A manifest holds a reference composed with the store's address or kept bare, so the
@@ -195,6 +229,9 @@ func (i *Inventory) KeepSet(ctx context.Context) (map[string][]string, error) {
return nil, err
}
for _, reference := range all {
if mirrored[reference] {
continue
}
digest := digestIn(reference)
if digest == "" {
// Not something the store holds by digest; nothing here can speak for it, so it
@@ -237,6 +274,72 @@ func (i *Inventory) KeptArchives(ctx context.Context) ([]string, error) {
return out, nil
}
// KeptImages is every image the mesh keeps, in a stated order: the manifests whose platform manifests
// a sweep spares when it lets an index of the same repository go (novox/hq ADR 0257).
func (i *Inventory) KeptImages(ctx context.Context) ([]string, error) {
keep, err := i.KeepSet(ctx)
if err != nil {
return nil, err
}
var out []string
for reference := range keep {
if path, ours := catalogue.InArtifactStore(reference); ours && strings.Contains(path, "@sha256:") {
out = append(out, reference)
}
}
sort.Strings(out)
return out, nil
}
// everyReferenceRecorded is every artifact the mesh recorded putting in its store: what each
// successful build made, and every base a build — or a person — recorded copying in (novox/hq ADR
// 0257). Oldest first, a copy placed at the moment it was recorded.
func (i *Inventory) everyReferenceRecorded(ctx context.Context) ([]string, error) {
made, err := i.everyReferenceMade(ctx)
if err != nil {
return nil, err
}
rows, err := i.store.Pool().Query(ctx, `select reference from artifact_mirrored order by at asc, reference asc`)
if err != nil {
return nil, err
}
defer rows.Close()
seen := map[string]bool{}
for _, r := range made {
seen[r] = true
}
out := made
for rows.Next() {
var reference string
if err := rows.Scan(&reference); err != nil {
return nil, err
}
if reference = asRecorded(reference); reference != "" && !seen[reference] {
seen[reference] = true
out = append(out, reference)
}
}
return out, rows.Err()
}
// mirroredSet is every base the records say the mesh copied in, as recorded.
func (i *Inventory) mirroredSet(ctx context.Context) (map[string]bool, error) {
rows, err := i.store.Pool().Query(ctx, `select reference from artifact_mirrored`)
if err != nil {
return nil, err
}
defer rows.Close()
out := map[string]bool{}
for rows.Next() {
var reference string
if err := rows.Scan(&reference); err != nil {
return nil, err
}
out[asRecorded(reference)] = true
}
return out, rows.Err()
}
// everyReferenceMade is every artifact reference any successful build recorded, oldest build first.
func (i *Inventory) everyReferenceMade(ctx context.Context) ([]string, error) {
rows, err := i.store.Pool().Query(ctx,
@@ -331,3 +434,8 @@ func (i *Inventory) MarkCollected(ctx context.Context, references []string) erro
}
return nil
}
// Mirrored is every base the records say the mesh copied into its store (novox/hq ADR 0257).
func (i *Inventory) Mirrored(ctx context.Context) (map[string]bool, error) {
return i.mirroredSet(ctx)
}
+5
View File
@@ -115,6 +115,11 @@ func dependenciesOf(entries []Entry, against map[string][]string, read map[strin
}
for _, ref := range against[name] {
if rest, ok := strings.CutPrefix(ref, catalogue.ArtifactStoreScheme); ok {
// A copy of an upstream image is no module's artifact (novox/hq ADR 0257): its
// repository is named for the image, and its first segment names no module.
if strings.HasPrefix(rest, builder.MirrorPrefix) {
continue
}
if base, _, found := strings.Cut(rest, "/"); found {
add(name, base, EdgeStandsOn)
}
+17
View File
@@ -154,3 +154,20 @@ func TestNothingStandsOnTheBuildAgentOnlyIsBuiltByIt(t *testing.T) {
t.Errorf("everything source-built is built by the agent: %d edges", builtBy)
}
}
// A copy of an upstream image is no module's artifact (novox/hq ADR 0257), even when a module happens to
// carry the name its repository starts with.
func TestACopyOfAnUpstreamImageIsNoEdge(t *testing.T) {
entries := []Entry{
{Manifest: catalogue.Manifest{Module: "route-proxy"}},
{Manifest: catalogue.Manifest{Module: "upstream"}},
}
against := map[string][]string{
"route-proxy": {catalogue.ArtifactStoreScheme + "upstream/docker.io/library/golang@sha256:a"},
}
for _, e := range dependenciesOf(entries, against, nil) {
if e.From == "route-proxy" && e.To == "upstream" {
t.Fatalf("a copy of an upstream image was read as a module's artifact: %+v", e)
}
}
}
@@ -0,0 +1,30 @@
-- A base a build copies into the artifact store is recorded where it is copied (novox/hq ADR 0257).
--
-- A build that stands on an image published elsewhere copies it into the store first (ADR 0096, 0097),
-- and until now nothing recorded the copy as something the mesh made: the build named it only in what
-- it stood on, and a build that failed after copying named it nowhere. So on 2026-10-08 the store held
-- 374 manifests of copied bases, in 27 repositories, that the sweep could never let go of, because no
-- record said the mesh had put them there (ADR 0189 §3).
--
-- One row per copy, by the reference as the mesh records it. `build` is the build that first said it
-- copied it, whether that build worked or not; `why` is a person's words when the copy was recorded by
-- hand (the `mirrors` verb), and empty otherwise. Not a foreign key to build: a build's row may be
-- written after its copy is, and a copy recorded by hand has no build.
create table artifact_mirrored (
reference text primary key,
build text,
at timestamptz not null default now(),
why text not null default ''
);
-- The copies the records already name. A build that worked kept what it stood on, and every reference
-- into a module's own `on-<argument>` repository there is a copy that build's mirror made: only the
-- mirror ever wrote a repository so named (ADR 0097). The earliest build to name each is its maker.
insert into artifact_mirrored (reference, build, at)
select distinct on (stood_on) stood_on, b.id, b.at
from build b,
jsonb_array_elements_text(case when jsonb_typeof(b.built_against) = 'array'
then b.built_against else '[]'::jsonb end) as stood_on
where stood_on ~ '^artifact-store://[a-z0-9][a-z0-9._-]*/on-[a-z0-9_]+@sha256:[0-9a-f]{64}$'
order by stood_on, b.at asc, b.id asc
on conflict (reference) do nothing;
@@ -0,0 +1,49 @@
-- An assignment waits for its module's build (novox/hq issue 325, ADR 0261).
--
-- A merge that adds a module asks for its build (issue 300), and the module is registered when the build's
-- outcome is taken in, minutes later. An `assign` in between was answered "no module of that name", which
-- read as "nobody registered it". The controller now keeps every build it asks for, so `assign` can tell a
-- build in flight from a module it never heard of, and an assignment made while the build runs is kept as
-- pending and made by the controller when the build registers the module, or ended with the reason.
-- Every build the controller asked for: a merge's new module, a plan's tier, a person's `build`, an `assign`
-- with build. Kept once the ask is made, never before. Its outcome is the `build` row of the same id;
-- an ask without one is still running, or was lost. not_asked is why the ask could not be made or waited
-- for; for a merge that could not ask, the id is the controller's and no build's.
create table build_request (
id text primary key,
repository text not null,
seat text not null default '',
source_path text not null default '',
ref text not null default '',
commit_hash text not null default '',
asked_for text not null default '',
not_asked text,
asked_at timestamptz not null default now()
);
create index build_request_asked_at on build_request (asked_at);
-- An assignment kept until its module is registered (ADR 0261). waiting until the build registers the
-- module; applying while the controller makes it, under the machine's hold; then applied, refused (the
-- assignment was refused then), expired (the build failed, was not registered, or said nothing within its
-- bound) or withdrawn (`unassign`). raised_at is when an expiry or refusal was raised as a condition,
-- acknowledged_at when a person took it back with `unassign`, cleared_at when its condition was cleared.
-- Gone with its machine. Ended rows are deleted after 30 days.
create table pending_assignment (
id bigserial primary key,
node uuid not null references node (id) on delete cascade,
module text not null,
build_id text not null,
repository text not null default '',
source_path text not null default '',
since timestamptz not null default now(),
state text not null default 'waiting'
check (state in ('waiting', 'applying', 'applied', 'refused', 'expired', 'withdrawn')),
note text not null default '',
settled_at timestamptz,
raised_at timestamptz,
acknowledged_at timestamptz,
cleared_at timestamptz
);
create unique index pending_assignment_open on pending_assignment (node, module)
where state in ('waiting', 'applying');
+211
View File
@@ -0,0 +1,211 @@
package inventory
import (
"context"
"fmt"
"os"
"slices"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
)
// A base a build copies into the store is recorded where it is copied, and kept while a kept build stood
// on it (novox/hq ADR 0257).
func mirrorRef(repository string, n int) string {
return fmt.Sprintf("%s%s@sha256:%064x", catalogue.ArtifactStoreScheme, repository, n)
}
// builtOn records a successful build of a module that made one image and stood on these bases.
func builtOn(t *testing.T, inv *Inventory, id, module string, n int, mirrored []string, against ...string) {
t.Helper()
b := aBuild(id, module, "")
b.Made = []Artifact{{Name: "app", Kind: "image", Reference: ref(module, "app", n)}}
b.Against = against
b.Mirrored = mirrored
if err := inv.RecordBuild(context.Background(), b); err != nil {
t.Fatal(err)
}
}
func stateOf(t *testing.T, inv *Inventory, reference string) ArtifactState {
t.Helper()
states, err := inv.Artifacts(context.Background())
if err != nil {
t.Fatal(err)
}
for _, s := range states {
if s.Reference == reference {
return s
}
}
return ArtifactState{Reference: reference, State: "unrecorded"}
}
func TestACopyAFailedBuildMadeIsRecordedAndNothingKeepsIt(t *testing.T) {
inv := fresh(t)
golang := mirrorRef("upstream/docker.io/library/golang", 1)
failed := aBuild("f1", "", "the recipe fetched an undeclared image")
failed.Mirrored = []string{golang}
if err := inv.RecordBuild(context.Background(), failed); err != nil {
t.Fatal(err)
}
if s := stateOf(t, inv, golang); s.State != ArtifactEligible {
t.Fatalf("a copy only a failed build made reads %+v; want eligible", s)
}
}
func TestACopyIsKeptByTheKeptBuildsThatStoodOnItAndByNothingElse(t *testing.T) {
inv := fresh(t)
ctx := context.Background()
holding(t, inv, "proxy")
old := mirrorRef("upstream/docker.io/library/golang", 1)
current := mirrorRef("upstream/docker.io/library/golang", 2)
// The first build stood on the old copy; six more on the current one. The old copy falls out of
// the five kept builds; the current one is stood on by all five.
builtOn(t, inv, "b01", "proxy", 1, []string{old}, old)
for i := 2; i <= 7; i++ {
builtOn(t, inv, fmt.Sprintf("b%02d", i), "proxy", i, []string{current}, current)
}
if s := stateOf(t, inv, old); s.State != ArtifactEligible {
t.Fatalf("a copy no kept build stood on reads %+v", s)
}
if s := stateOf(t, inv, current); s.State != ArtifactKept || !slices.Equal(s.Why, []string{KeptStoodOn}) {
t.Fatalf("a copy the kept builds stood on reads %+v", s)
}
// A definition naming the image upstream, by the digest every copy shares, keeps no copy.
m := catalogue.Manifest{Module: "proxy", Version: "2",
Build: &catalogue.Build{On: []catalogue.BuildsOn{{Arg: "GO_BASE", Image: "golang@" + strings.SplitN(old, "@", 2)[1]}}}}
if err := inv.RegisterModule(ctx, m, Source{Repository: "https://forge.invalid/proxy.git"}); err != nil {
t.Fatal(err)
}
if s := stateOf(t, inv, old); s.State != ArtifactEligible {
t.Fatalf("a definition naming the upstream image kept the old copy: %+v", s)
}
go_, err := inv.ToCollect(ctx)
if err != nil {
t.Fatal(err)
}
if !slices.Contains(go_, old) || slices.Contains(go_, current) {
t.Fatalf("offered %v to collect", go_)
}
// Once let go of, a copy stays let go of, even should a kept build still name it.
if err := inv.MarkCollected(ctx, []string{current}); err != nil {
t.Fatal(err)
}
if s := stateOf(t, inv, current); s.State != ArtifactCollected {
t.Fatalf("a collected copy reads %+v", s)
}
}
func TestAModulesBaseIsNotKeptAsACopy(t *testing.T) {
// A base that is another module's artifact is kept by that module's own builds, as before: the
// stood-on reason is a copy's alone.
inv := fresh(t)
holding(t, inv, "app")
base := ref("go-base", "build", 1)
b := aBuild("g1", "go-base", "")
b.Made = []Artifact{{Name: "build", Kind: "image", Reference: base}}
if err := inv.RecordBuild(context.Background(), b); err != nil {
t.Fatal(err)
}
builtOn(t, inv, "a1", "app", 1, nil, base)
if s := stateOf(t, inv, base); s.State != ArtifactEligible {
t.Fatalf("a forgotten module's artifact read %+v because a build stood on it", s)
}
}
func TestAPersonRecordsACopyAndTheFirstRecordStands(t *testing.T) {
inv := fresh(t)
ctx := context.Background()
copyRef := mirrorRef("route-proxy/on-go_base", 3)
if err := inv.RecordMirrored(ctx, "", "copied before copies were recorded", []string{copyRef}); err != nil {
t.Fatal(err)
}
if err := inv.RecordMirrored(ctx, "b9", "", []string{copyRef}); err != nil {
t.Fatal(err)
}
known, err := inv.Mirrored(ctx)
if err != nil || !known[copyRef] {
t.Fatalf("recorded copies %v %v", known, err)
}
var why string
if err := inv.store.Pool().QueryRow(ctx, `select why from artifact_mirrored where reference = $1`, copyRef).Scan(&why); err != nil {
t.Fatal(err)
}
if why != "copied before copies were recorded" {
t.Fatalf("the first record did not stand: %q", why)
}
if s := stateOf(t, inv, copyRef); s.State != ArtifactEligible {
t.Fatalf("a copy a person recorded reads %+v", s)
}
}
// The migration records the copies the build records already name: only a module's own on-<argument>
// repository, and only once, by the earliest build naming it.
func TestTheCopiesTheRecordsAlreadyNameAreRecorded(t *testing.T) {
inv := fresh(t)
ctx := context.Background()
copyRef := mirrorRef("route-proxy/on-go_base", 1)
notACopy := ref("go-base", "build", 2)
builtOn(t, inv, "b1", "route-proxy", 1, nil, copyRef, notACopy)
builtOn(t, inv, "b2", "route-proxy", 2, nil, copyRef)
// The rows a mesh had before the table: written by builds that did not say what they copied.
if _, err := inv.store.Pool().Exec(ctx, `truncate artifact_mirrored`); err != nil {
t.Fatal(err)
}
sql, err := os.ReadFile("migrations/0081-a-mirrored-base-is-recorded.sql")
if err != nil {
t.Fatal(err)
}
_, backfill, _ := strings.Cut(string(sql), "insert into artifact_mirrored")
if _, err := inv.store.Pool().Exec(ctx, "insert into artifact_mirrored"+backfill); err != nil {
t.Fatal(err)
}
var reference, build string
var n int
if err := inv.store.Pool().QueryRow(ctx, `select count(*) from artifact_mirrored`).Scan(&n); err != nil || n != 1 {
t.Fatalf("%d copies recorded (%v); want the one", n, err)
}
if err := inv.store.Pool().QueryRow(ctx, `select reference, build from artifact_mirrored`).Scan(&reference, &build); err != nil {
t.Fatal(err)
}
if reference != copyRef || build != "b1" {
t.Fatalf("recorded %s by %s", reference, build)
}
}
// A copy let go of and copied again by a later build is held again: its record is back, and the sweep
// decides about it as before.
func TestACopyLetGoOfAndCopiedAgainIsHeldAgain(t *testing.T) {
inv := fresh(t)
ctx := context.Background()
holding(t, inv, "proxy")
golang := mirrorRef("upstream/docker.io/library/golang", 1)
builtOn(t, inv, "b01", "proxy", 1, []string{golang}, golang)
if err := inv.MarkCollected(ctx, []string{golang}); err != nil {
t.Fatal(err)
}
if s := stateOf(t, inv, golang); s.State != ArtifactCollected {
t.Fatalf("a let-go copy reads %+v", s)
}
builtOn(t, inv, "b02", "proxy", 2, []string{golang}, golang)
if s := stateOf(t, inv, golang); s.State != ArtifactKept || !slices.Equal(s.Why, []string{KeptStoodOn}) {
t.Fatalf("a copy copied again reads %+v; want kept again", s)
}
// A person's record never undoes a collection: only a build that copied it says it is there.
if err := inv.MarkCollected(ctx, []string{golang}); err != nil {
t.Fatal(err)
}
if err := inv.RecordMirrored(ctx, "", "again", []string{golang}); err != nil {
t.Fatal(err)
}
if s := stateOf(t, inv, golang); s.State != ArtifactCollected {
t.Fatalf("a person's record brought a collected copy back: %+v", s)
}
}
+354
View File
@@ -0,0 +1,354 @@
package inventory
import (
"context"
"errors"
"path"
"strings"
"time"
"github.com/jackc/pgx/v5"
)
// What the controller asked to be built, and the assignments waiting for it (novox/hq issue 325, ADR 0261).
//
// A build's outcome was kept, and its asking was not: between a merge asking for a new module and the
// outcome registering it, the mesh had no record that anything was coming, and `assign` answered "no module
// of that name" for a module minutes from existing. Kept here, a build request lets `assign` say which case
// it is in, and an assignment made while the build runs is kept until the build registers the module.
// KeptFor is how long a build request, and an ended pending assignment, is kept.
const KeptFor = 30 * 24 * time.Hour
// BuildRequest is one build the controller asked for.
type BuildRequest struct {
// ID is the build's id, as its outcome will carry it; for a merge whose ask could not be made, an id
// of the controller's making that no build carries.
ID string
// Repository and Seat are the source as the mesh spells it (novox/hq ADR 0111): a path on the seat's
// holder, or a URL with no seat. Path is the module's directory in it, Ref the branch or commit asked.
Repository, Seat, Path, Ref string
// Commit is the commit that made the ask, where one did: a merge's.
Commit string
// For says who asked: "merge", "plan", "build" (a person), "assign".
For string
// NotAsked is why the ask could not be made, or not waited for; empty when it was.
NotAsked string
At time.Time
}
// Name is the module this request is expected to register, read from its directory: the last element of
// its path, or of its repository for a module at the repository's root. A guess until the build says, and
// the one a person reads too: `modules/sensors` is sensors.
func (a BuildRequest) Name() string {
if p := strings.Trim(a.Path, "/"); p != "" && p != "." {
return path.Base(p)
}
return strings.TrimSuffix(path.Base(strings.TrimRight(a.Repository, "/")), ".git")
}
// RequestOutcome is a build request beside what came of it, where anything did.
type RequestOutcome struct {
BuildRequest
// Heard is whether an outcome of this id was taken in; Failed is its builder's words when it failed,
// Module what the build said it built, HeardAt when it was taken in.
Heard bool
Failed string
Module string
HeardAt time.Time
}
// RecordBuildRequest keeps one build request, once it is asked. Idempotent on the id. Requests older than
// KeptFor are deleted in the same act.
func (i *Inventory) RecordBuildRequest(ctx context.Context, a BuildRequest) error {
at := a.At
if at.IsZero() {
at = time.Now().UTC()
}
var notAsked *string
if a.NotAsked != "" {
notAsked = &a.NotAsked
}
if _, err := i.store.Pool().Exec(ctx,
`insert into build_request (id, repository, seat, source_path, ref, commit_hash, asked_for, not_asked, asked_at)
values ($1, $2, $3, $4, $5, $6, $7, $8, $9)
on conflict (id) do nothing`,
a.ID, a.Repository, a.Seat, strings.Trim(a.Path, "/"), a.Ref, a.Commit, a.For, notAsked, at); err != nil {
return err
}
_, err := i.store.Pool().Exec(ctx, `delete from build_request where asked_at < $1`, time.Now().Add(-KeptFor))
return err
}
// MarkNotAsked says a build request was asked and its asker could not hand it over, or stopped waiting for
// it: the words are kept, unless its outcome was heard first.
func (i *Inventory) MarkNotAsked(ctx context.Context, id, why string) error {
_, err := i.store.Pool().Exec(ctx,
`update build_request set not_asked = $2
where id = $1 and not exists (select 1 from build where build.id = $1)`, id, why)
return err
}
// RequestsNamed is every build request kept whose directory names the module, or whose build said it built
// it, newest first, each with its outcome.
func (i *Inventory) RequestsNamed(ctx context.Context, name string) ([]RequestOutcome, error) {
all, err := i.requests(ctx)
if err != nil {
return nil, err
}
var out []RequestOutcome
for _, a := range all {
if a.Name() == name || a.Module == name {
out = append(out, a)
}
}
return out, nil
}
// RequestedNames is the name of every module a kept build request is expected to register: what `assign`
// offers as a closest name beside the modules registered.
func (i *Inventory) RequestedNames(ctx context.Context) ([]string, error) {
all, err := i.requests(ctx)
if err != nil {
return nil, err
}
seen := map[string]bool{}
var out []string
for _, a := range all {
if n := a.Name(); n != "" && !seen[n] {
seen[n] = true
out = append(out, n)
}
}
return out, nil
}
func (i *Inventory) requests(ctx context.Context) ([]RequestOutcome, error) {
rows, err := i.store.Pool().Query(ctx,
`select a.id, a.repository, a.seat, a.source_path, a.ref, a.commit_hash, a.asked_for,
coalesce(a.not_asked, ''), a.asked_at,
b.id is not null, coalesce(b.failed, ''), coalesce(b.module, ''), coalesce(b.at, a.asked_at)
from build_request a left join build b on b.id = a.id
order by a.asked_at desc, a.id desc`)
if err != nil {
return nil, err
}
defer rows.Close()
var out []RequestOutcome
for rows.Next() {
var a RequestOutcome
if err := rows.Scan(&a.ID, &a.Repository, &a.Seat, &a.Path, &a.Ref, &a.Commit, &a.For, &a.NotAsked,
&a.At, &a.Heard, &a.Failed, &a.Module, &a.HeardAt); err != nil {
return nil, err
}
out = append(out, a)
}
return out, rows.Err()
}
// The states of a pending assignment.
const (
PendingWaiting = "waiting"
PendingApplying = "applying"
PendingApplied = "applied"
PendingRefused = "refused"
PendingExpired = "expired"
PendingWithdrawn = "withdrawn"
)
// PendingAssignment is an assignment kept until its module is registered (ADR 0261).
type PendingAssignment struct {
ID int64
Node string
Module string
// Build is the build it waits for; Repository and Path where that build is from.
Build, Repository, Path string
Since time.Time
State string
// Note is what ended it, in the words said when it ended.
Note string
Settled *time.Time
// Raised is when an expiry or refusal was raised as a condition; Acknowledged when a person took it
// back; Cleared when its condition was cleared.
Raised, Acknowledged, Cleared *time.Time
}
// Open is whether the pending assignment is still to be made.
func (p PendingAssignment) Open() bool {
return p.State == PendingWaiting || p.State == PendingApplying
}
// ErrAlreadyPending is a second pending assignment of one module to one machine.
var ErrAlreadyPending = errors.New("that assignment is already waiting for its build")
const pendingColumns = `p.id, n.name, p.module, p.build_id, p.repository, p.source_path, p.since, p.state, p.note,
p.settled_at, p.raised_at, p.acknowledged_at, p.cleared_at`
func scanPending(rows pgx.Rows) ([]PendingAssignment, error) {
defer rows.Close()
var out []PendingAssignment
for rows.Next() {
var p PendingAssignment
if err := rows.Scan(&p.ID, &p.Node, &p.Module, &p.Build, &p.Repository, &p.Path, &p.Since, &p.State,
&p.Note, &p.Settled, &p.Raised, &p.Acknowledged, &p.Cleared); err != nil {
return nil, err
}
out = append(out, p)
}
return out, rows.Err()
}
// RecordPending keeps an assignment until its module is registered. One is open per machine and module.
func (i *Inventory) RecordPending(ctx context.Context, p PendingAssignment) (PendingAssignment, error) {
node, err := i.NodeByName(ctx, p.Node)
if err != nil {
return PendingAssignment{}, err
}
err = i.store.Pool().QueryRow(ctx,
`insert into pending_assignment (node, module, build_id, repository, source_path)
values ($1, $2, $3, $4, $5)
on conflict (node, module) where state in ('waiting', 'applying') do nothing
returning id, since`,
node.ID, p.Module, p.Build, p.Repository, strings.Trim(p.Path, "/")).Scan(&p.ID, &p.Since)
if errors.Is(err, pgx.ErrNoRows) {
return PendingAssignment{}, ErrAlreadyPending
}
if err != nil {
return PendingAssignment{}, err
}
p.State = PendingWaiting
return p, nil
}
// RepointPending makes a waiting pending assignment wait for another build. False when it no longer waits.
func (i *Inventory) RepointPending(ctx context.Context, id int64, build, repository, sourcePath string) (bool, error) {
tag, err := i.store.Pool().Exec(ctx,
`update pending_assignment set build_id = $2, repository = $3, source_path = $4
where id = $1 and state = 'waiting'`, id, build, repository, strings.Trim(sourcePath, "/"))
if err != nil {
return false, err
}
return tag.RowsAffected() == 1, nil
}
// Pending is every pending assignment still waiting, oldest first, when endedSince is zero. Otherwise it is
// every open one and every one ended since then.
func (i *Inventory) Pending(ctx context.Context, endedSince time.Time) ([]PendingAssignment, error) {
if endedSince.IsZero() {
rows, err := i.store.Pool().Query(ctx, `select `+pendingColumns+`
from pending_assignment p join node n on n.id = p.node
where p.state = 'waiting' order by p.since, p.id`)
if err != nil {
return nil, err
}
return scanPending(rows)
}
rows, err := i.store.Pool().Query(ctx, `select `+pendingColumns+`
from pending_assignment p join node n on n.id = p.node
where p.state in ('waiting', 'applying') or p.settled_at >= $1
order by p.since, p.id`, endedSince)
if err != nil {
return nil, err
}
return scanPending(rows)
}
// PendingFor is every pending assignment of one module to one machine, newest first.
func (i *Inventory) PendingFor(ctx context.Context, node, module string) ([]PendingAssignment, error) {
rows, err := i.store.Pool().Query(ctx, `select `+pendingColumns+`
from pending_assignment p join node n on n.id = p.node
where n.name = $1 and p.module = $2 order by p.since desc, p.id desc`, node, module)
if err != nil {
return nil, err
}
return scanPending(rows)
}
// Stuck is every pending assignment left applying since before the time given: a controller that claimed
// it and stopped before it said what came of it.
func (i *Inventory) Stuck(ctx context.Context, before time.Time) ([]PendingAssignment, error) {
rows, err := i.store.Pool().Query(ctx, `select `+pendingColumns+`
from pending_assignment p join node n on n.id = p.node
where p.state = 'applying' and p.settled_at < $1 order by p.id`, before)
if err != nil {
return nil, err
}
return scanPending(rows)
}
// ToRaise is every pending assignment that expired or was refused and has not been raised as a condition;
// ToClear every one raised and not cleared.
func (i *Inventory) ToRaise(ctx context.Context) ([]PendingAssignment, error) {
rows, err := i.store.Pool().Query(ctx, `select `+pendingColumns+`
from pending_assignment p join node n on n.id = p.node
where p.state in ('expired', 'refused') and p.raised_at is null order by p.id`)
if err != nil {
return nil, err
}
return scanPending(rows)
}
func (i *Inventory) ToClear(ctx context.Context) ([]PendingAssignment, error) {
rows, err := i.store.Pool().Query(ctx, `select `+pendingColumns+`
from pending_assignment p join node n on n.id = p.node
where p.raised_at is not null and p.cleared_at is null order by p.id`)
if err != nil {
return nil, err
}
return scanPending(rows)
}
// MarkPending stamps a pending assignment raised, acknowledged or cleared, now.
func (i *Inventory) MarkPending(ctx context.Context, id int64, what string) error {
column := map[string]string{"raised": "raised_at", "acknowledged": "acknowledged_at", "cleared": "cleared_at"}[what]
if column == "" {
return errors.New("a pending assignment is marked raised, acknowledged or cleared")
}
_, err := i.store.Pool().Exec(ctx,
`update pending_assignment set `+column+` = coalesce(`+column+`, now()) where id = $1`, id)
return err
}
// ClaimPending takes a waiting pending assignment for making, so nothing else settles or withdraws it while
// it is made. False when it no longer waits: another process took it, or it was withdrawn.
func (i *Inventory) ClaimPending(ctx context.Context, id int64) (bool, error) {
tag, err := i.store.Pool().Exec(ctx,
`update pending_assignment set state = 'applying', settled_at = now() where id = $1 and state = 'waiting'`, id)
if err != nil {
return false, err
}
return tag.RowsAffected() == 1, nil
}
// ReleaseClaim puts a claimed pending assignment back to waiting.
func (i *Inventory) ReleaseClaim(ctx context.Context, id int64) error {
_, err := i.store.Pool().Exec(ctx,
`update pending_assignment set state = 'waiting', settled_at = null where id = $1 and state = 'applying'`, id)
return err
}
// SettlePending ends a pending assignment in the state it is read in (from): waiting for an expiry or a
// withdrawal, applying for the outcome of making it. False when it was no longer in that state.
func (i *Inventory) SettlePending(ctx context.Context, id int64, from, state, note string) (bool, error) {
if state == PendingWaiting || state == PendingApplying {
return false, errors.New("a pending assignment is settled to applied, refused, expired or withdrawn")
}
tag, err := i.store.Pool().Exec(ctx,
`update pending_assignment set state = $3, note = $4, settled_at = now()
where id = $1 and state = $2`, id, from, state, note)
if err != nil {
return false, err
}
return tag.RowsAffected() == 1, nil
}
// ForgetEndedPending deletes ended pending assignments settled more than KeptFor ago, and says how many.
func (i *Inventory) ForgetEndedPending(ctx context.Context, now time.Time) (int64, error) {
tag, err := i.store.Pool().Exec(ctx,
`delete from pending_assignment where state not in ('waiting', 'applying') and settled_at < $1`,
now.Add(-KeptFor))
if err != nil {
return 0, err
}
return tag.RowsAffected(), nil
}
+25
View File
@@ -0,0 +1,25 @@
package inventory
import (
"testing"
"time"
)
// A pending assignment goes with its machine (novox/hq issue 325, review of mesh-controller#150 point 7).
func TestAPendingAssignmentGoesWithItsMachine(t *testing.T) {
inv := ForTest(t)
ctx := t.Context()
if _, err := inv.AddNode(ctx, "leaving"); err != nil {
t.Fatal(err)
}
if _, err := inv.RecordPending(ctx, PendingAssignment{Node: "leaving", Module: "sensors", Build: "b-1"}); err != nil {
t.Fatal(err)
}
if _, err := inv.store.Pool().Exec(ctx, `delete from node where name = 'leaving'`); err != nil {
t.Fatalf("a machine with a pending assignment could not be removed: %v", err)
}
rows, err := inv.Pending(ctx, time.Unix(0, 0))
if err != nil || len(rows) != 0 {
t.Fatalf("a pending assignment outlived its machine: %+v %v", rows, err)
}
}
+9
View File
@@ -152,6 +152,15 @@ type PlanGate struct {
// Passing names the modules that passed on their own when the send failed (ADR 0254): each keeps its
// pass, and is not put back.
Passing []string `json:"passing,omitempty"`
// HealthyAt is, per module, when its newest healthy judging was counted: a pass is counted only gateEvery
// after the one before (issue 318 review).
HealthyAt map[string]time.Time `json:"healthy_at,omitempty"`
// Broken names the modules a judging found broken, kept for the rest of the judging, and BrokenWhy the
// first reason: the send fails, and the modules beside them are judged to their own verdict first.
Broken []string `json:"broken,omitempty"`
BrokenWhy string `json:"broken_why,omitempty"`
// Returned names the broken modules already put back, at once, while the rest of the send is judged.
Returned []string `json:"returned,omitempty"`
}
// CarriedMove is one module's build moving on a machine with a gated send.
+4
View File
@@ -215,6 +215,10 @@ type BuildResult struct {
// (novox/hq ADR 0009). The catalogue turns these into edges; nothing else need care.
Against []string `json:"against,omitempty"`
// Mirrored is every base this build copied into the artifact store, as pinned (novox/hq ADR
// 0257). Carried on a failed build too: the copy is in the store whether the build worked or not.
Mirrored []string `json:"mirrored,omitempty"`
// Read is every repository this build read source from besides the module's own. A module whose
// recipe packages source that lives elsewhere is affected when that repository moves, and the
// manifest the mesh keeps says nothing about it (novox/hq 04-ISSUES/131).
+2 -1
View File
@@ -75,7 +75,8 @@
"build",
"artifacts",
"collect",
"images"
"images",
"mirrors"
],
"resources": [
{