Compare commits

...
Author SHA1 Message Date
jochen f0a62c1b47 Store optional requirements beside a plain requires list (hq issue 166, review of #281)
mesh/merge-gate head f0a62c1b473f merged into trunk 454a3db69a1e — fail: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; head f…
mesh/repo-check head f0a62c1b473f merged into trunk 454a3db69a1e — fail: head f0a62c1b473f does not merge cleanly into trunk 454a3db69a1e: in conflict test…
mesh/delivery rejected: the gate failed or could not run, or the repository's own check failed
A controller older than optional requirements reads `requires` as names, and Inventory.Catalogue
stops at the first manifest it cannot read: the long form stored inside `requires` would have left
a rollback below this controller composing no machine. Stored, `requires` stays plain and the
optional names sit in `optional-requires`, a key an older controller sets aside, leaving out only
the module carrying it. A refusal of a hard requirement now names a module that wants it hard.
2026-10-13 01:16:32 +02:00
jochen bc81029a03 Let a requirement be optional, binding only a provider that already runs (hq issue 166, ADR 0330)
mesh/merge-gate out of date: judged against trunk 56d6313262e8, the trunk is now cbac703e817d; judged again on a recheck or when it is merged
mesh/repo-check out of date: judged against trunk 56d6313262e8, the trunk is now cbac703e817d; judged again on a recheck or when it is merged
mesh/delivery superseded: a newer head of the same pull request
Making every soft dependency a provision made it hard: a mesh without influxdb could not assign
grafana, and one with influxdb in its catalogue would install it for grafana. A requirement may now
be written {"name": ..., "optional": true}. It resolves as a hard one when a provider is assigned or
reachable, and is otherwise absent: never refused, nothing installed for it. Its binding, credential
and only-when-bound files are left out, its ${bound:...|default:...} placeholders take their
defaults, the module check refuses one that could render empty, and plan and node list it unbound.
2026-10-13 00:39:58 +02:00
mesh-admin 21ac8fd858 Merge pull request 'Let a mergeable file name the setting keys it takes (hq issue 168, ADR 0326)' (#272) from fix/168-a-file-names-the-settings-it-takes into main 2026-10-12 22:07:08 +00:00
mesh-admin e63af0df17 Merge pull request 'End a walk's apply phase at each machine's own report of its send (hq issue 582)' (#278) from fix/582-apply-ends-at-the-machines-own-report into main 2026-10-12 21:36:33 +00:00
mesh-admin 961f2e82ea Merge pull request 'Hand a delivery build out before pull-request checks (issue 558)' (#276) from fix/558-a-delivery-build-goes-first into main 2026-10-12 21:20:19 +00:00
jschoubben e727589e48 End a walk's apply phase at each machine's own report of its send (hq issue 582)
mesh/merge-gate head e727589e483b merged into trunk 961f2e82ea1c — pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus…
mesh/repo-check head e727589e483b merged into trunk 961f2e82ea1c — pass: its merge-check.sh passed
mesh/delivery delivered
The walk records its send to the rest only after every machine was sent,
which takes 20-28 s, so the machine sent first fell outside the 2 s slack,
its report was missed and the phase ended at the next walk's send to it.
Keep each machine's own send time with what it was sent, match its apply
duration by that send and digest, and show each machine's report in the
walk's phases (ADR 0282 decision 6).
2026-10-12 23:18:28 +02:00
mesh-admin 2f157148c6 Merge pull request 'node shows what a machine said it is made of, with its age (issue 160)' (#269) from fix/160-node-shows-what-a-machine-is into main 2026-10-12 20:37:06 +00:00
mesh-admin 13be75eb65 Merge pull request 'Replay issue 201 with only what the controller had before its fix (for mesh-lab R201)' (#275) from test/201-replay into main 2026-10-12 20:22:37 +00:00
mesh-admin 13711fb5b9 Merge pull request 'Mark that the build agent takes work, so its health check can see it ready (hq issue 544)' (#277) from fix/544-the-build-agent-says-it-is-ready into main 2026-10-12 19:58:09 +00:00
jschoubben fc573f951a Remove only a move's proven second copy, and never delete a handed-out original (hq issue 558)
mesh/merge-gate head fc573f951a47 merged into trunk 2f157148c6c8 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head fc573f951a47 merged into trunk 2f157148c6c8 — pass: its merge-check.sh passed
mesh/delivery delivered
Second review of #276: a dead letter delivered again beside its original
shares its request id and was removed as a copy, losing the ask; and a
publish that outlived the pause could delete an original a holder had
taken. A copy now names the message it copies, removals are said, the
publish waits inside the pause's margin, and an original handed out
meanwhile keeps its place while the copy goes.
2026-10-12 21:46:49 +02:00
jschoubben 1ded7dff34 Make a queue move lossless and run its passes on the acting controller alone (hq issue 558)
mesh/merge-gate out of date: judged against trunk 4dbd4066ca54, the trunk is now c0c379af2b3c; judged again on a recheck or when it is merged
mesh/repo-check out of date: judged against trunk 4dbd4066ca54, the trunk is now c0c379af2b3c; judged again on a recheck or when it is merged
mesh/delivery superseded: a newer head of the same pull request
Review of #276: a move deleted before it published, so a failed publish
lost an ask; a verb's own process ran a pass beside the serving one, so
one could resume the queue while the other moved asks and a build ran
twice. The copy is now published first and second copies removed by id,
asks only signal the acting controller's loop, moves stop with 5 s of
the pause left, a cancel finds a moved ask by id, and an unreadable ask
keeps its age.
2026-10-12 21:38:11 +02:00
jschoubben 50c55a773e Mark that the build agent takes work, so its health check can see it ready (novox/hq issue 544)
mesh/merge-gate head 50c55a773eb8 merged into trunk c0c379af2b3c — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head 50c55a773eb8 merged into trunk c0c379af2b3c — pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group fix/544-the-build-agent-says-it-is-ready delivered: every member is delivered
The agent listens on nothing, so the node-engine could judge it only alive and
held it starting for the whole grace on every machine a walk moved it to. It
now writes a mark once it is bound to its worker, after its seat verbs are
served, and 'mesh-builder ready' passes only while the mark is there.
2026-10-12 21:33:12 +02:00
jschoubben c6e234455f Hand a delivery build out before pull-request checks (hq issue 558)
mesh/delivery superseded: a newer head of the same pull request
mesh/merge-gate out of date: judged against trunk 4dbd4066ca54, the trunk is now c0c379af2b3c; judged again on a recheck or when it is merged
mesh/repo-check out of date: judged against trunk 4dbd4066ca54, the trunk is now c0c379af2b3c; judged again on a recheck or when it is merged
A merged change's walk waited minutes on its build behind checks in a
first-come queue. The queue now hands deliveries out first, a check that
has waited ten minutes before deliveries asked after it, and says each
ask's kind; the order is kept in the work queue itself.
2026-10-12 21:25:46 +02:00
jochen 45a1783ba9 Replay issue 201 with only what the controller had before its fix
mesh/merge-gate head 45a1783ba9a4 merged into trunk 13711fb5b91b — pass: the change touches no module of the mesh's graph
mesh/repo-check head 45a1783ba9a4 merged into trunk 13711fb5b91b — pass: its merge-check.sh passed
mesh/delivery delivered
The fix's own TestReplay201 reads what #273 added (sendScope.puttingBack,
recordedKept's core keep, the condition), so it cannot be laid over the
commit before. TestReplay201ThePlansSend composes a plan's send to the
control node through keepingRecorded, planFor and declarationFor and holds
that it carries the controller the machine was last sent, not the older
record; it fails on 84deb53^1 and passes on 84deb53. For mesh-lab's R201
(hq issue 201).
2026-10-12 21:17:31 +02:00
jochen b538397055 Name the issue that holds the refusal back (hq issue 574)
mesh/merge-gate head b5383970550b merged into trunk e63af0df17f5 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head b5383970550b merged into trunk e63af0df17f5 — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-12 18:00:49 +02:00
jochen dd10889209 Hold back refusing a declaring module's silent mergeable file (hq issue 168)
mesh/delivery superseded: a newer head of the same pull request
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check pass: its merge-check.sh passed
betterbird declares a preference and owns a mergeable file that names no
keys, and the catalogue can only name them once this controller runs: the
refusal made the catalogue check fail against catalogue main. It follows
once mesh-catalog names the keys.
2026-10-12 18:00:33 +02:00
jochen 286257d802 Let a mergeable file name the setting keys it takes (hq issue 168, ADR 0326)
mesh/merge-gate pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without …
mesh/repo-check fail: its merge-check.sh failed: --- FAIL: TestModuleCheckPassesTheCatalogue (0.89s); and TestEveryCatalogueManifestParses, TestNoCatalogue…
mesh/delivery superseded: a newer head of the same pull request
Every key of a module's layers merged into every mergeable file it owns:
the mesh's own words, a key meant for a ${setting:} in another file, a
key for a second configuration file. A module could not own two
configurable files. A file now names its keys in takes-settings and
takes only those; a key no file names is stray where it is set; a
module that declares a setting names the keys of every mergeable file,
and one that declares none keeps today's form (ADR 0164 §15).
2026-10-12 17:47:05 +02:00
jochen 67d54d9d09 Say an account without its time as age unknown, not as two thousand years (issue 160)
mesh/merge-gate head 67d54d9d09b2 merged into trunk 13be75eb65c0 — pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus…
mesh/repo-check head 67d54d9d09b2 merged into trunk 13be75eb65c0 — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-12 17:43:38 +02:00
jochen 7953d2889d Show what a machine said it is made of in node, with its age (issue 160)
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
The architecture and kernel a machine reported were kept and read by nothing, and the facts the
node-engine now says (memory, filesystems, processor, distribution, virtual or physical, uptime,
timezone) would have joined them. node shows every one in one block with the age of the account,
and a fact the machine could not determine as unknown with its why, never as a zero.
2026-10-12 17:23:07 +02:00
36 changed files with 3211 additions and 118 deletions
+13 -1
View File
@@ -66,6 +66,11 @@ broker to take work from or a registry to publish into:
Without --registry the artifacts stay in this machine's container runtime, named by the
digest of their own configuration. The result is printed as JSON.
mesh-builder ready
exits 0 once the agent running in this container is bound to its build work and pulls it, and
1 before: its container's health check.
`
func run() error {
@@ -76,12 +81,18 @@ func run() error {
return nil
case "build":
return buildOnce(context.Background(), os.Args[2:])
case "ready":
// The container's health check (novox/hq issue 544): exits non-zero until the agent takes work.
return ready()
default:
fmt.Print(usage)
return nil
}
}
if err := notReady(); err != nil {
return err
}
credential, err := brokerFrom()
if err != nil {
return err
@@ -134,7 +145,8 @@ func run() error {
// **What an earlier holder here left goes before anything is taken** (novox/hq issue 306): a holder
// recreated mid-check left the check's store and bus running, and the ask went to another machine.
h.removeLeftBehind()
machine := link.MachineOverNATSWith(js, on, seat, link.MachineOptions{Paused: h.Paused})
machine := link.MachineOverNATSWith(js, on, seat, link.MachineOptions{Paused: h.Paused,
Taking: func() { taking(seat) }})
defer machine.Close()
// The seat's verbs, on this machine's subjects. Only the seat that declares them: the retired
+44
View File
@@ -0,0 +1,44 @@
package main
import (
"errors"
"fmt"
"os"
"path/filepath"
)
// **The build agent says it is ready** (novox/hq issue 544). It listens on nothing, so no http or tcp
// check can see it work; it writes a mark once it is bound to its worker and pulls build work, and its
// container's health check runs `mesh-builder ready`, which passes only while the mark is there. Without
// it the node-engine could judge the agent only alive, and every walk that moved it waited out the whole
// grace on each machine.
//
// The mark lives in the container's own temporary directory, never in the workspace the machine keeps:
// a new container starts without it, and a restarted one removes it before it binds again.
func readyMark() string { return filepath.Join(os.TempDir(), "mesh-builder.ready") }
// notReady removes the mark: a holder starting is not taking work until it says so again.
func notReady() error {
if err := os.Remove(readyMark()); err != nil && !errors.Is(err, os.ErrNotExist) {
return fmt.Errorf("cannot remove the mark that this agent takes build work (%s): %w", readyMark(), err)
}
return nil
}
// taking writes the mark. A mark that cannot be written is said, and the agent goes on building: its
// check then says it is not ready, which is the truth the gate should hear.
func taking(seat string) {
if err := os.WriteFile(readyMark(), []byte(seat+"\n"), 0o644); err != nil {
fmt.Fprintf(os.Stderr, "cannot mark that this agent takes build work (%s): %v\n", readyMark(), err)
return
}
fmt.Fprintf(os.Stderr, "taking build work from %s\n", seat)
}
// ready is the health check: nil when this container's agent is bound and pulling build work.
func ready() error {
if _, err := os.Stat(readyMark()); err != nil {
return fmt.Errorf("not taking build work yet: %s is not there", readyMark())
}
return nil
}
+32
View File
@@ -0,0 +1,32 @@
package main
import (
"os"
"testing"
)
// The health check passes only between the agent saying it takes work and its next start (novox/hq
// issue 544).
func TestTheAgentIsReadyOnlyOnceItTakesWork(t *testing.T) {
t.Setenv("TMPDIR", t.TempDir())
if err := ready(); err == nil {
t.Fatal("an agent that never took work is ready")
}
taking("mesh-build-machine")
if err := ready(); err != nil {
t.Fatalf("an agent taking work is not ready: %v", err)
}
// Started again, it is not ready until it binds again.
if err := notReady(); err != nil {
t.Fatal(err)
}
if err := ready(); err == nil {
t.Fatal("an agent starting again is ready before it takes work")
}
if err := notReady(); err != nil {
t.Fatalf("removing a mark that is not there failed: %v", err)
}
if _, err := os.Stat(readyMark()); err == nil {
t.Fatal("the mark is still there")
}
}
+5 -5
View File
@@ -1042,13 +1042,13 @@ func deliveryCommand(ctx context.Context, args []string) error {
return fmt.Errorf("delivery %s: plan, order, check, go, stop or walks", sub)
}
// appliedFrom answers a machine's first report after a send from the controller's `apply` durations; a lookup
// that fails is a report not read, which leaves the walk's end unknown rather than wrong.
// appliedFrom answers a machine's own report of the send a walk made to it from the controller's `apply` durations
// (novox/hq issue 582); a lookup that fails is a report not read, which leaves the walk's end unknown rather than wrong.
func appliedFrom(ctx context.Context, inv *inventory.Inventory) inventory.AppliedLookup {
return func(node string, sent time.Time) (inventory.AppliedReport, bool) {
r, ok, err := inv.FirstAppliedAfter(ctx, node, sent)
return func(node string, send inventory.SentDeclaration, walkSent time.Time) (inventory.AppliedReport, bool) {
r, ok, err := inv.FirstAppliedAfter(ctx, node, send, walkSent)
if err != nil {
fmt.Fprintf(os.Stderr, "the report of %s after %s could not be read: %v\n", node, sent.Format(time.RFC3339), err)
fmt.Fprintf(os.Stderr, "the report of %s after %s could not be read: %v\n", node, walkSent.Format(time.RFC3339), err)
return inventory.AppliedReport{}, false
}
return r, ok
+135
View File
@@ -0,0 +1,135 @@
package main
import (
"fmt"
"sort"
"strings"
"time"
"github.com/novox/mesh-controller/internal/inventory"
)
// machineLines is what a machine said it is made of (novox/hq issue 160), as `node` shows it: every fact in
// one block with its age, the architecture and kernel the profile always carried among them, and each fact
// the machine could not determine said as unknown with its why — never as a zero, and never left out.
func machineLines(a inventory.MachineAccount, had bool, now time.Time) []string {
if !had {
return []string{" what it is made of: never said — this machine has not reported a profile"}
}
m := a.Machine
if m == nil {
return []string{
" what it is made of: not said — its node-engine is older than the one that says it",
" " + factLine("architecture", orUnsaid(a.Architecture)),
" " + factLine("kernel", orUnsaid(a.Kernel)),
}
}
age := roughly(now.Sub(m.ObservedAt)) + " ago"
if m.ObservedAt.IsZero() {
// Said with no time: its age is not known, and is never read as two thousand years.
age = "at a time it did not say (age unknown)"
}
out := []string{fmt.Sprintf(" what it is made of, as it said %s:", age)}
add := func(name, value string) { out = append(out, " "+factLine(name, value)) }
unknown := map[string]string{}
for k, v := range m.Unknown {
unknown[k] = v
}
// said is the value, or the fact's own reason it is unknown; a fact neither said nor unknown is a
// node-engine that broke its own rule, and is said so rather than shown blank.
said := func(fact, value string) string {
if value != "" {
delete(unknown, fact)
return value
}
if why, ok := unknown[fact]; ok {
delete(unknown, fact)
return "unknown: " + why
}
return "unknown: the machine said neither it nor why"
}
add("architecture", orUnsaid(a.Architecture))
release := said("kernel release", m.KernelRelease)
if m.KernelRelease == "" {
release = "(release " + release + ")"
}
add("kernel", orUnsaid(a.Kernel)+" "+release)
dist := said("distribution", m.Distribution)
if m.Distribution != "" {
dist += " " + said("version", m.Version)
}
add("distribution", dist)
add("processor", said("processor", m.Processor))
add("cores", said("cores", countWords(m.Cores)))
add("threads", said("threads", countWords(m.Threads)))
add("memory", said("memory", bytesWords(m.MemoryBytes)))
add("virtual", said("virtual or physical", m.Virtual))
up := ""
if m.BootedAt != nil {
up = fmt.Sprintf("%s, since %s", roughly(now.Sub(*m.BootedAt)), m.BootedAt.Local().Format("2006-01-02 15:04"))
}
add("up", said("uptime", up))
add("timezone", said("timezone", m.Timezone))
if len(m.Filesystems) == 0 {
add("filesystems", said("filesystems", ""))
} else {
for i, f := range m.Filesystems {
name := ""
if i == 0 {
name = "filesystems"
}
add(name, fmt.Sprintf("%s %s, %s free (%s %s)", f.Mount, bytesWords(f.SizeBytes),
bytesWords(f.FreeBytes), f.Type, f.Device))
}
if why, ok := unknown["filesystems"]; ok {
add("", "and some unknown: "+why)
delete(unknown, "filesystems")
}
}
add("public address", said("public address", ""))
// Any other fact a newer node-engine named unknown, so nothing it said is dropped.
rest := make([]string, 0, len(unknown))
for k := range unknown {
rest = append(rest, k)
}
sort.Strings(rest)
for _, k := range rest {
add(k, "unknown: "+unknown[k])
}
return out
}
func factLine(name, value string) string { return fmt.Sprintf("%-15s %s", name, value) }
func orUnsaid(s string) string {
if strings.TrimSpace(s) == "" {
return "not said"
}
return s
}
// countWords is a number a machine said, or nothing when it said none (its unknown says why).
func countWords(n int) string {
if n <= 0 {
return ""
}
return fmt.Sprint(n)
}
// bytesWords is an amount of bytes a person reads, in binary units; nothing when none was said.
func bytesWords(n int64) string {
if n <= 0 {
return ""
}
units := []string{"B", "KiB", "MiB", "GiB", "TiB", "PiB"}
v, u := float64(n), 0
for v >= 1024 && u < len(units)-1 {
v /= 1024
u++
}
if u == 0 {
return fmt.Sprintf("%d B", n)
}
return fmt.Sprintf("%.1f %s", v, units[u])
}
+88
View File
@@ -0,0 +1,88 @@
package main
import (
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/inventory"
)
// Defends novox/hq issue 160: `node` shows every fact a machine said it is made of, in one block with its
// age, the architecture and kernel among them; and a fact it could not determine as unknown with why.
func TestNodeShowsWhatAMachineIsMadeOfWithItsAge(t *testing.T) {
now := time.Date(2026, 10, 12, 15, 0, 0, 0, time.UTC)
booted := now.Add(-75 * time.Hour)
a := inventory.MachineAccount{Architecture: "amd64", Kernel: "linux", Machine: &inventory.MadeOf{
ObservedAt: now.Add(-3 * time.Hour), KernelRelease: "7.2.8-arch1-2", Distribution: "Arch Linux",
Version: "rolling", Processor: "AMD Ryzen 9 8945HS", Cores: 8, Threads: 16, MemoryBytes: 32891494400,
Virtual: "physical", BootedAt: &booted, Timezone: "Europe/Brussels",
Filesystems: []inventory.MountedFilesystem{{Mount: "/", Device: "/dev/nvme0n1p3", Type: "ext4",
SizeBytes: 987978522624, FreeBytes: 31830110208}},
Unknown: map[string]string{"public address": "how it learns it is not decided yet"},
}}
got := strings.Join(machineLines(a, true, now), "\n")
t.Log("\n" + got)
for _, want := range []string{
"what it is made of, as it said 3h ago:",
"architecture amd64",
"kernel linux 7.2.8-arch1-2",
"distribution Arch Linux rolling",
"processor AMD Ryzen 9 8945HS",
"cores 8",
"threads 16",
"memory 30.6 GiB",
"virtual physical",
"up 3d, since",
"timezone Europe/Brussels",
"filesystems / 920.1 GiB, 29.6 GiB free (ext4 /dev/nvme0n1p3)",
"public address unknown: how it learns it is not decided yet",
} {
if !strings.Contains(got, want) {
t.Errorf("missing %q", want)
}
}
}
func TestAFactAMachineCouldNotReadIsShownUnknownNeverZero(t *testing.T) {
now := time.Date(2026, 10, 12, 15, 0, 0, 0, time.UTC)
a := inventory.MachineAccount{Architecture: "arm64", Kernel: "linux", Machine: &inventory.MadeOf{
ObservedAt: now, Unknown: map[string]string{"memory": "MemTotal not found in /proc/meminfo",
"cores": "/proc/cpuinfo says no core id", "kernel release": "permission denied"}}}
got := strings.Join(machineLines(a, true, now), "\n")
t.Log("\n" + got)
for _, want := range []string{
"memory unknown: MemTotal not found in /proc/meminfo",
"cores unknown: /proc/cpuinfo says no core id",
"kernel linux (release unknown: permission denied)",
// Neither said nor named unknown: the node-engine broke its rule, and that is shown.
"timezone unknown: the machine said neither it nor why",
} {
if !strings.Contains(got, want) {
t.Errorf("missing %q", want)
}
}
if strings.Contains(got, " 0 ") || strings.Contains(got, "0 B") {
t.Error("a fact the machine could not read was shown as a zero")
}
}
func TestAnOlderNodeEngineStillShowsArchitectureAndKernel(t *testing.T) {
got := strings.Join(machineLines(inventory.MachineAccount{Architecture: "amd64", Kernel: "linux"}, true,
time.Now()), "\n")
if !strings.Contains(got, "older than the one that says it") || !strings.Contains(got, "architecture amd64") ||
!strings.Contains(got, "kernel linux") {
t.Errorf("%s", got)
}
if got := strings.Join(machineLines(inventory.MachineAccount{}, false, time.Now()), "\n"); !strings.Contains(got, "never said") {
t.Errorf("%s", got)
}
}
func TestAnAccountWithoutItsTimeSaysAgeUnknown(t *testing.T) {
got := machineLines(inventory.MachineAccount{Architecture: "amd64", Kernel: "linux",
Machine: &inventory.MadeOf{KernelRelease: "7.2"}}, true, time.Now())[0]
if !strings.Contains(got, "age unknown") || strings.Contains(got, "d ago") {
t.Errorf("%s", got)
}
}
+9 -1
View File
@@ -237,7 +237,15 @@ func moduleCommand(ctx context.Context, args []string) error {
says = append(says, "provides "+describeOffers(m.Provides))
}
if len(m.Requires) > 0 {
says = append(says, "requires "+strings.Join(m.Requires, ", "))
// An optional requirement says so (novox/hq ADR 0330): it is not refused when absent.
required := make([]string, 0, len(m.Requires))
for _, r := range m.Requires {
if m.IsOptional(r) {
r += " (optional)"
}
required = append(required, r)
}
says = append(says, "requires "+strings.Join(required, ", "))
}
for _, c := range m.Claims {
says = append(says, "claims "+c.At()+"/"+c.Name)
+18
View File
@@ -875,6 +875,17 @@ func showNode(ctx context.Context, stored *stores, name string) error {
}
}
// What it is made of, in one block with its age (novox/hq issue 160): collected and shown in the same
// breath, so no fact the machine says is kept and read by nothing.
if account, had, err := inv.MachineOf(ctx, name); err != nil {
fmt.Printf("\n what it is made of could NOT be read: %v\n", err)
} else {
fmt.Println()
for _, line := range machineLines(account, had, time.Now()) {
fmt.Println(line)
}
}
held, err := inv.Profile(ctx, name)
if err != nil {
return err
@@ -918,6 +929,13 @@ func showNode(ctx context.Context, stored *stores, name string) error {
for _, line := range notAppliedLines(name, leftOutOf(plan, settings, node.Adopted)) {
fmt.Println(line)
}
// And which optional requirements of its modules nothing assigned answers (novox/hq ADR 0330 §7).
if lines := unboundLines(plan); len(lines) > 0 {
fmt.Println()
for _, line := range lines {
fmt.Println(line)
}
}
}
return nil
}
+11
View File
@@ -1315,6 +1315,14 @@ func planCommand(ctx context.Context, args []string) error {
fmt.Printf("%s is assigned nothing\n", args[0])
return nil
}
// Which optional requirements are unbound (novox/hq ADR 0330 §7): beside the document on the
// error stream where the answer is the declaration itself, which carries no such field — the
// node-engine refuses an envelope key it does not know.
if *asDiff || *asJSON {
for _, line := range unboundLines(plan) {
fmt.Fprintf(os.Stderr, "%s:%s\n", args[0], line)
}
}
if *asDiff {
declared, err := declarationFor(ctx, open, args[0], plan, settings)
if err != nil {
@@ -1396,6 +1404,9 @@ func planCommand(ctx context.Context, args []string) error {
for _, n := range plan.Needs {
fmt.Printf(" needs %s from %s, for %s\n", n.Name, n.From, n.For)
}
for _, line := range unboundLines(plan) {
fmt.Println(line)
}
declared, err := declarationFor(ctx, open, args[0], plan, settings)
if err != nil {
return err
+56 -14
View File
@@ -14,7 +14,6 @@ import (
"time"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/inventory"
@@ -81,6 +80,30 @@ func listQueue(ctx context.Context, args []string, w io.Writer) error {
return nil
}
// keepingTheQueueInOrder runs the build queue's passes while this controller acts (novox/hq issue 558):
// the one process that puts the queue in order, so no two passes pause and resume it over each other.
// The build seat is read from the catalogue at most every half minute.
func keepingTheQueueInOrder(ctx context.Context, open *stores, js *broker.JetStream) {
if js == nil {
return
}
seat, readAt := link.TheBuildMachine, time.Time{}
seatNow := func() string {
if time.Since(readAt) < watchEvery {
return seat
}
if entries, err := open.inventory.Catalogued(ctx); err == nil {
seat, readAt = buildSeatAmong(entries), time.Now()
} else {
fmt.Printf("the build seat is read as %s: the catalogue cannot be read: %v\n", seat, err)
}
return seat
}
link.KeepingInOrder(ctx, js, seatNow, link.Holding, func(format string, args ...any) {
fmt.Printf(format+"\n", args...)
})
}
// queueAnswer is the queue as data (novox/hq issue 455): what a caller reads to say how far back an ask
// waits — "waiting in the build queue behind 2" — which the table alone could not tell it. Text is the
// table, written from the same read: the verb answers it as its output and the rest as its answer, so the
@@ -100,7 +123,8 @@ type queueAnswer struct {
type queuedAnswer struct {
ID string `json:"id"`
State string `json:"state"`
// Kind is "check" for a pull request's merge check, "build" for anything else.
// Kind is "delivery" for a build a walk waits for, "check" for a pull request's check and any ask
// whose kind cannot be told (link.KindOf, novox/hq issue 558).
Kind string `json:"kind"`
Repository string `json:"repository"`
Path string `json:"path,omitempty"`
@@ -129,10 +153,7 @@ func queueAnswerOf(q link.Queue, now time.Time) queueAnswer {
out := queueAnswer{Seat: q.Seat, Waiting: len(waiting), InFlight: len(running), Dead: len(dead),
MaxDeliver: q.MaxDeliver, Asks: []queuedAnswer{}, Text: queueText(q, now)}
one := func(a link.QueuedAsk, position int) queuedAnswer {
kind := "build"
if a.Request.Check != nil {
kind = "check"
}
kind := link.KindOf(a.Request)
since := a.AskedAt
if a.State == link.AskInFlight && a.On != "" && !a.Started.IsZero() {
since = a.Started
@@ -189,19 +210,20 @@ func queueText(q link.Queue, now time.Time) string {
case a.Was != "":
where = fmt.Sprintf("handed back by %s, to be handed out again", a.Was)
}
fmt.Fprintf(&b, " %-26s %s — %s, %s (seq %d)\n", a.ID, what(a), where, ago(a.AskedAt), a.Seq)
fmt.Fprintf(&b, " %-26s %-8s %s — %s, %s (seq %d)\n", a.ID, link.KindOf(a.Request), what(a), where,
ago(a.AskedAt), a.Seq)
}
}
if len(waiting) > 0 {
fmt.Fprintln(&b, "\nwaiting:")
fmt.Fprintln(&b, "\nwaiting, in the order they are handed out:")
for _, a := range waiting {
fmt.Fprintf(&b, " %-26s %s — %s (seq %d)\n", a.ID, what(a), ago(a.AskedAt), a.Seq)
fmt.Fprintf(&b, " %-26s %-8s %s — %s (seq %d)\n", a.ID, link.KindOf(a.Request), what(a), ago(a.AskedAt), a.Seq)
}
}
if len(dead) > 0 {
fmt.Fprintf(&b, "\ndead — handed out as often as the worker allows (%d) and never settled, still held:\n", q.MaxDeliver)
for _, a := range dead {
fmt.Fprintf(&b, " %-26s %s — %s (seq %d)\n", a.ID, what(a), ago(a.AskedAt), a.Seq)
fmt.Fprintf(&b, " %-26s %-8s %s — %s (seq %d)\n", a.ID, link.KindOf(a.Request), what(a), ago(a.AskedAt), a.Seq)
}
}
switch {
@@ -275,6 +297,15 @@ func cancelAsk(ctx context.Context, js *broker.JetStream, open *stores, seat str
}
}
worker, _ := broker.HolderConsumerFor("", "", broker.DeclaredSeat{Name: seat, Accepts: []string{"build"}})
// **Where it waits now** (novox/hq issue 558): moved into order since it was read, a waiting ask has a
// later sequence, and the worker handing out what was behind its old one is not it being taken.
if ask.State == link.AskWaiting {
if q, err := link.ReadQueue(ctx, js, seat); err == nil {
if now, ok := q.Find(ask.ID); ok && now.State == link.AskWaiting {
ask.Seq = now.Seq
}
}
}
switch ask.State {
case link.AskWaiting:
if taken, err := takenSince(js, worker, ask.Seq); err != nil {
@@ -296,14 +327,25 @@ func cancelAsk(ctx context.Context, js *broker.JetStream, open *stores, seat str
"ends it there", ask.ID, started, ask.ID)
}
}
if err := js.Context().DeleteMsg(worker.Stream, ask.Seq); err != nil && !errors.Is(err, nats.ErrMsgNotFound) &&
!errors.Is(err, jetstream.ErrMsgNotFound) && !strings.Contains(err.Error(), "no message found") {
// Found by its id when it was moved into order since it was read (novox/hq issue 558).
removed, err := link.DeleteAsk(ctx, js, seat, ask)
if err != nil {
return "", fmt.Errorf("%s is marked cancelled and could not be deleted from the queue (seq %d): %w — a "+
"holder taking it ends it as cancelled", ask.ID, ask.Seq, err)
}
recordCancelled(ctx, open, ask)
return fmt.Sprintf("cancelled %s (%s, %s): deleted from the %s queue and recorded failed, %s — a plan "+
"that asked for it fails with that", ask.ID, ask.Repository, ask.State, seat, link.CancelledByHand), nil
return cancelSaid(ask, seat, removed), nil
}
// cancelSaid is what a cancel says: deleted from the queue only when it was.
func cancelSaid(ask link.QueuedAsk, seat string, removed uint64) string {
where := fmt.Sprintf("deleted from the %s queue", seat)
if removed == 0 {
where = fmt.Sprintf("no longer in the %s queue to delete, and marked cancelled, so a holder taking it "+
"ends it", seat)
}
return fmt.Sprintf("cancelled %s (%s, %s): %s and recorded failed, %s — a plan "+
"that asked for it fails with that", ask.ID, ask.Repository, ask.State, where, link.CancelledByHand)
}
// holderLooks is how long a cancel waits for a holder that took the ask before the mark to say it
+22 -2
View File
@@ -24,7 +24,7 @@ func aQueueOfEachKind(now time.Time) link.Queue {
State: link.AskInFlight, On: "ace", Started: started,
Request: link.BuildRequest{ID: "build-running", Check: &link.CheckRequest{Owner: "novox", Repo: "mesh-host"}}},
{Seq: 3, ID: "build-first", Repository: "ssh://forge/novox/mesh-catalog", Ref: "main", AskedAt: asked,
State: link.AskWaiting},
State: link.AskWaiting, Request: link.BuildRequest{ID: "build-first", Repository: "ssh://forge/novox/mesh-catalog"}},
{Seq: 4, ID: "build-second", Repository: "ssh://forge/novox/mesh-catalog", Path: "modules/gitea", Ref: aCommit,
AskedAt: now.Add(-time.Minute), State: link.AskWaiting,
Request: link.BuildRequest{ID: "build-second", Check: &link.CheckRequest{Owner: "novox", Repo: "mesh-catalog"}}},
@@ -51,7 +51,7 @@ func TestTheQueueVerbAnswersEachAskAsData(t *testing.T) {
t.Errorf("in flight: %+v", running)
}
first, second := byID["build-first"], byID["build-second"]
if first.Position != 1 || first.Kind != "build" || first.Commit != "" || first.Ref != "main" || first.Machine != "" ||
if first.Position != 1 || first.Kind != "delivery" || first.Commit != "" || first.Ref != "main" || first.Machine != "" ||
!first.Since.Equal(now.Add(-5*time.Minute)) {
t.Errorf("first waiting: %+v", first)
}
@@ -62,6 +62,13 @@ func TestTheQueueVerbAnswersEachAskAsData(t *testing.T) {
if dead := byID["build-dead"]; dead.Position != 0 || dead.State != link.AskDead {
t.Errorf("dead: %+v", dead)
}
// And the table says each ask's kind too (novox/hq issue 558).
for _, line := range []string{"build-running check ", "build-first delivery ",
"build-second check ", "build-dead check "} {
if !strings.Contains(got.Text, line) {
t.Errorf("the table does not say %q:\n%s", line, got.Text)
}
}
// In the table's order: in flight, then the waiting by position, then the dead.
var order []string
for _, a := range got.Asks {
@@ -135,3 +142,16 @@ func TestTheQueueNeverShowsARepositorysPassword(t *testing.T) {
}
}
}
// A cancel says it deleted the ask only when it did: moved into order and gone since, it says so (novox/hq
// issue 558).
func TestACancelSaysDeletedOnlyWhenItDeleted(t *testing.T) {
ask := link.QueuedAsk{ID: "build-1", Repository: "novox/x", State: link.AskWaiting, Seq: 4}
if said := cancelSaid(ask, link.TheBuildMachine, 9); !strings.Contains(said, "deleted from the") {
t.Errorf("removed, it says: %s", said)
}
if said := cancelSaid(ask, link.TheBuildMachine, 0); strings.Contains(said, "deleted from") ||
!strings.Contains(said, "no longer in the") {
t.Errorf("gone already, it says: %s", said)
}
}
+90
View File
@@ -0,0 +1,90 @@
package main
import (
"encoding/json"
"testing"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
)
// TestReplay201ThePlansSend replays novox/hq issue 201 (2026-10-02): two merges a minute apart made two plans; the
// controller's plan rolled the newer controller onto the control node, and the node-engine's plan then sent that
// machine the controller build before it, whose binary refused the seat row the newer one had widened. Here the
// control node, anchor, was last sent the newer controller c2 while the mesh's record of the controller is the
// build before it, c1, which passed its gate. The outcome asserted: a send that is no person's push — composed as a
// plan's send composes it (keepingRecorded, what sendKeeps gives every send through sendToEach) — carries the
// controller at c2, the build the machine was last sent, and not the older record; a person's push still carries
// the record. Written with only what the controller had before its fix (mesh-controller #273) — the inventory's
// build, gate, assignment and send records, keepingRecorded, planFor and declarationFor — so it is laid over the
// commit before; the fix's own TestReplay201 reads what the fix added. Needs the store (MESH_TEST_POSTGRES), and
// is named in testdata/store-tests.
func TestReplay201ThePlansSend(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
const older, newer = "c1aaaaaaaaaa", "c2bbbbbbbbbb"
old, now := time.Now().Add(-2*time.Hour), time.Now().Add(-time.Minute)
record := func(commit string, asked time.Time) inventory.Build {
t.Helper()
manifest, _ := json.Marshal(catalogue.Manifest{Module: "mesh-controller", Version: "1"})
b := inventory.Build{ID: "build-mesh-controller-" + commit, Module: "mesh-controller", Commit: commit,
Repository: "novox/mesh-controller", Manifest: manifest, Asked: asked, At: asked,
Made: []inventory.Artifact{{Name: "x", Kind: "bundle", Reference: "sha256:" + commit + "0000000000000000"}}}
if err := inv.RecordBuild(ctx, b); err != nil {
t.Fatal(err)
}
return b
}
// The older controller rolled out and passed its gate; the newer one was asked after it and registered.
before := record(older, old)
if err := inv.RecordGate(ctx, inventory.GateVerdict{Build: before.ID, Module: "mesh-controller", Commit: older,
Machines: []string{"anchor"}, Verdict: inventory.GatePassed}); err != nil {
t.Fatal(err)
}
record(newer, now)
if err := inv.RegisterModule(ctx, catalogue.Manifest{Module: "mesh-controller", Version: "1"}, inventory.Source{
Repository: "novox/mesh-controller", BuiltFrom: newer, Head: newer, Asked: now}); err != nil {
t.Fatal(err)
}
if _, err := inv.Assign(ctx, "anchor", "mesh-controller"); err != nil {
t.Fatal(err)
}
// The controller's plan sent anchor the newer controller.
if err := inv.RecordSent(ctx, nodeID(t, open, "anchor"), "d-anchor", map[string]string{"mesh-controller": newer}); err != nil {
t.Fatal(err)
}
// And the mesh's record of the controller is the build before it — whatever made it so.
if err := inv.RestoreModule(ctx, before); err != nil {
t.Fatal(err)
}
// The controller each send to anchor carries: a plan's send keeps recorded builds (keepingRecorded, what
// sendKeeps gives every send but a person's push); a person's push composes the mesh's record.
controllerSent := func(sends string, keeping bool) string {
t.Helper()
c := ctx
if keeping {
c = keepingRecorded(ctx)
}
plan, settings, err := planFor(c, open, "anchor")
if err != nil {
t.Fatalf("%s to anchor did not compose: %v", sends, err)
}
declared, err := declarationFor(c, open, "anchor", plan, settings)
if err != nil {
t.Fatalf("%s to anchor did not compose: %v", sends, err)
}
return declared.Builds["mesh-controller"]
}
if got := controllerSent("a plan's send", true); got != newer {
t.Fatalf("a plan's send carried anchor the controller %s, older than the %s anchor was last sent: the "+
"older binary would read the row its successor widened", short(got), short(newer))
}
if got := controllerSent("a person's push", false); got != older {
t.Fatalf("a person's push carried the controller %s, not the mesh's record %s", short(got), short(older))
}
}
+21
View File
@@ -0,0 +1,21 @@
package main
import (
"fmt"
"github.com/novox/mesh-controller/internal/catalogue"
)
// unboundLines say which optional requirements of a machine's modules nothing assigned answers
// (novox/hq ADR 0330 §7): left unbound rather than refused, with nothing installed for them, so the
// module runs without — and said, because a data source that never appears is otherwise a silence.
// Each binds without anybody's act on the first composition after a provider is assigned.
func unboundLines(plan catalogue.Resolution) []string {
lines := make([]string, 0, len(plan.Absent))
for _, a := range plan.Absent {
lines = append(lines, fmt.Sprintf(
" unbound %s, optional for %s — nothing assigned provides it; it binds once a provider is "+
"assigned (novox/hq ADR 0330)", a.Requirement, a.Consumer))
}
return lines
}
+59
View File
@@ -0,0 +1,59 @@
package main
import (
"strings"
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
)
// A machine running grafana without influxdb says so (novox/hq ADR 0330 §7, issue 166): `plan` and
// `node` list the optional requirement nothing assigned answers, and the requirement is left
// absent rather than refused — influxdb is in the catalogue and is not installed for it.
func TestPlanAndNodeListAnOptionalRequirementLeftUnbound(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
register(t, open, catalogue.Manifest{Module: "influxdb", Version: "1",
Provides: catalogue.FromAnywhere("influxdb-api")})
register(t, open, catalogue.Manifest{Module: "grafana", Version: "1",
Requires: []string{"influxdb-api"}, OptionalRequires: map[string]bool{"influxdb-api": true},
Resources: []map[string]any{{"id": "state", "type": "directory", "path": "/var/lib/grafana", "mode": "0700"}}})
if _, err := assign(ctx, open, "laptop", "grafana"); err != nil {
t.Fatalf("grafana could not be assigned on a mesh without influxdb: %v", err)
}
plan, _, err := planFor(ctx, open, "laptop")
if err != nil {
t.Fatal(err)
}
for _, m := range plan.Modules {
if m.Module == "influxdb" {
t.Fatal("influxdb was installed for grafana's optional requirement (ADR 0330 §3)")
}
}
const said = "unbound influxdb-api, optional for grafana"
if shown := printed(t, func() error { return planCommand(ctx, []string{"laptop"}) }); !strings.Contains(shown, said) {
t.Errorf("plan laptop does not say influxdb-api is unbound:\n%s", shown)
}
if shown := printed(t, func() error { return showNode(ctx, open, "laptop") }); !strings.Contains(shown, said) {
t.Errorf("node show laptop does not say influxdb-api is unbound:\n%s", shown)
}
// And once influxdb is assigned somewhere, grafana binds to it with no act on grafana.
if _, err := assign(ctx, open, "anchor", "influxdb"); err != nil {
t.Fatal(err)
}
plan, _, err = planFor(ctx, open, "laptop")
if err != nil {
t.Fatal(err)
}
if len(plan.Absent) != 0 {
t.Fatalf("still absent after influxdb was assigned: %+v", plan.Absent)
}
var bound bool
for _, n := range plan.Needs {
bound = bound || n.For == "grafana" && n.Name == "influxdb-api" && n.From == "anchor"
}
if !bound {
t.Fatalf("grafana was not bound to influxdb on anchor: %+v", plan.Needs)
}
}
+3
View File
@@ -739,6 +739,9 @@ func watchTheMesh(ctx context.Context, open *stores, server *link.Server, bus li
// and the answer chosen is performed on its warrant.
startAsking(watching, open, server, bus.Conn, keeper)
go forgettingOldHeals(watching, open.inventory)
// And the build queue's order (novox/hq issue 558): a check that has waited its while goes ahead of the
// deliveries asked after it even when nothing new is asked.
go keepingTheQueueInOrder(watching, open, server.JetStream())
fmt.Printf("watching the mesh: %d signal(s) every %s, %d probe(s) every %s; what is wrong is kept in %s "+
"and said as %s events\n", len(watchedRows()), watchEvery, len(runnableProbes()), doctorEvery,
broker.ConditionsBucket, conditions.Seat)
+35 -23
View File
@@ -26,21 +26,10 @@ import (
// serves — whose keys are agreed by the requirement's name, not by this file.
// bound is where a module says a value from one of its bindings belongs:
// ${bound:<provision>.<key>}.
var bound = regexp.MustCompile(`\$\{bound:([a-z0-9][a-z0-9.-]*[a-z0-9]):([a-z0-9][a-z0-9_-]*)\}`)
// boundUsed are the (provision, key) pairs a file's content asks for, first appearance first.
func boundUsed(content string) [][2]string {
var used [][2]string
seen := map[string]bool{}
for _, m := range bound.FindAllStringSubmatch(content, -1) {
if key := m[1] + ":" + m[2]; !seen[key] {
seen[key] = true
used = append(used, [2]string{m[1], m[2]})
}
}
return used
}
// ${bound:<provision>:<key>}, and for an optional requirement the value it takes while absent,
// ${bound:<provision>:<key>|default:<value>} (novox/hq ADR 0330 §4). A default is never empty: an
// absent requirement is never rendered as an empty string.
var bound = regexp.MustCompile(`\$\{bound:([a-z0-9][a-z0-9.-]*[a-z0-9]):([a-z0-9][a-z0-9_-]*)(?:\|default:([^}\n]+))?\}`)
// knownFor is everything a module may name from one of its bindings.
//
@@ -125,7 +114,11 @@ func plainly(value any) string {
// serve, is refused. Left as it was, the literal `${bound:x:y}` would be written into a
// configuration file and read as a value — a connection to a host called `${bound:x:y}`, failing
// somewhere that names neither the module nor the mesh.
func boundInto(resource map[string]any, known map[string]map[string]string, module string) error {
//
// **An optional requirement left absent takes the default its placeholder names** (novox/hq ADR
// 0330 §4), and one that names none is refused rather than rendered empty. The module check refuses
// that already; this is the same rule where the file is made, for a manifest registered before it.
func boundInto(resource map[string]any, known map[string]map[string]string, module string, absent map[string]bool) error {
if fmt.Sprint(resource["type"]) != "file" {
return nil
}
@@ -133,24 +126,43 @@ func boundInto(resource map[string]any, known map[string]map[string]string, modu
if !ok {
return nil
}
for _, pair := range boundUsed(content) {
provision, key := pair[0], pair[1]
var refused error
filled := bound.ReplaceAllStringFunc(content, func(token string) string {
if refused != nil {
return token
}
m := bound.FindStringSubmatch(token)
provision, key, fallback := m[1], m[2], m[3]
values, has := known[provision]
if !has {
return fmt.Errorf(
if absent[provision] {
if fallback != "" {
return fallback
}
refused = fmt.Errorf(
"%s has a file that says ${bound:%s:%s}, and %q is optional and absent here: it names no "+
"value for then (${bound:%s:%s|default:<value>}), so it would be written empty "+
"(novox/hq ADR 0330 §4)", module, provision, key, provision, provision, key)
return token
}
refused = fmt.Errorf(
"%s has a file that says ${bound:%s:%s}, and %s does not require %q. It may name %s",
module, provision, key, module, provision, orNothing(namesOfBindings(known)))
return token
}
value, said := values[key]
if !said {
return fmt.Errorf(
refused = fmt.Errorf(
"%s asks its %s binding for %q, and what answers it says %s",
module, provision, key, orNothing(namesOfKeys(values)))
return token
}
resource["content"] = strings.ReplaceAll(
content, fmt.Sprintf("${bound:%s:%s}", provision, key), value)
content = resource["content"].(string)
return value
})
if refused != nil {
return refused
}
resource["content"] = filled
return nil
}
+6 -1
View File
@@ -850,6 +850,11 @@ func (r Resolution) compose(with Rendering, owner map[string]string,
// An account depends on nothing the mesh computes.
accounts, rest := accountsFirst(resources)
resources = append(append(accounts, first...), rest...)
// **A file delivered only while its optional requirement is bound is left out while it is
// absent** (novox/hq ADR 0330 §5), and the field is dropped from the rest: the node-engine
// parses strictly and knows no such field.
absent := r.AbsentFor(m.Module)
resources = withoutAbsent(resources, absent)
// No container is given the mesh's names (novox/hq ADR 0148). It used to be: every
// container got the whole roster as `--add-host` entries at creation, and a name that
@@ -1013,7 +1018,7 @@ func (r Resolution) compose(with Rendering, owner map[string]string,
// **Substituted here, not on the machine.** A bound value is not secret — the mesh
// holds it in the clear — so there is nothing for the host to be the only witness of,
// and sending it already filled in means the host learns no new field.
if err := boundInto(copied, known, m.Module); err != nil {
if err := boundInto(copied, known, m.Module, absent); err != nil {
return nil, err
}
// And what the module could not have written: the machine it turned out to be
+48 -1
View File
@@ -385,6 +385,10 @@ type Manifest struct {
// Requires are names that must be provided by something assigned to the same node.
Requires []string `json:"requires,omitempty"`
// OptionalRequires is which of Requires the module can run without (novox/hq ADR 0330): filled
// by UnmarshalJSON from the long form of a `requires` entry, {name, optional}, or from the stored
// form's `optional-requires` list; never written by hand.
OptionalRequires map[string]bool `json:"-"`
// Emits are the events this module publishes, named **locally**: `order.placed`, not a subject
// and not a routing key. The mesh derives where it lands (design 29 §1), so reorganising the
@@ -1354,6 +1358,35 @@ func (m *Manifest) UnmarshalJSON(raw []byte) error {
}
delete(keys, "contributes")
}
var optionalRequires map[string]bool
if requires, ok := keys["requires"]; ok && string(requires) != "null" {
names, optional, err := readRequires(requires)
if err != nil {
return err
}
plainNames, err := json.Marshal(names)
if err != nil {
return err
}
keys["requires"] = plainNames
optionalRequires = optional
}
// The stored form: `requires` plain, and which are optional beside it (OptionalRequiresKey).
if stored, ok := keys[OptionalRequiresKey]; ok {
if string(stored) != "null" {
optional, err := readOptionalRequires(stored)
if err != nil {
return err
}
for name := range optional {
if optionalRequires == nil {
optionalRequires = map[string]bool{}
}
optionalRequires[name] = true
}
}
delete(keys, OptionalRequiresKey)
}
rest, err := json.Marshal(keys)
if err != nil {
return err
@@ -1385,6 +1418,7 @@ func (m *Manifest) UnmarshalJSON(raw []byte) error {
}
*m = Manifest(fields)
m.unknown = unknown
m.OptionalRequires = optionalRequires
if len(plain) > 0 {
m.Secrets = plain
}
@@ -1407,13 +1441,22 @@ func (m Manifest) MarshalJSON() ([]byte, error) {
if err != nil {
return nil, err
}
if len(m.SecretsMany) == 0 && len(m.ContributesMany) == 0 {
if len(m.SecretsMany) == 0 && len(m.ContributesMany) == 0 && len(m.OptionalRequires) == 0 {
return raw, nil
}
var keys map[string]json.RawMessage
if err := json.Unmarshal(raw, &keys); err != nil {
return nil, err
}
if optional := optionalRequiresList(m); len(optional) > 0 {
// Beside `requires`, never inside it: an older controller reads `requires` as plain names, and one
// manifest it cannot read stops the whole catalogue (OptionalRequiresKey).
listed, err := json.Marshal(optional)
if err != nil {
return nil, err
}
keys[OptionalRequiresKey] = listed
}
if len(m.ContributesMany) > 0 {
mergedContributes := map[string]any{}
for to, values := range m.Contributes {
@@ -1693,6 +1736,10 @@ func ParseManifest(raw []byte) (Manifest, error) {
problems = append(problems, LogProblems(m)...)
// And the defaults it gives its settings (setting_defaults.go, novox/hq ADR 0262).
problems = append(problems, SettingProblems(m)...)
// And what it requires optionally, and the files it delivers only while bound (novox/hq ADR 0330).
problems = append(problems, OptionalProblems(m)...)
// And which setting keys each mergeable file takes (setting_takes.go, novox/hq ADR 0326).
problems = append(problems, TakenSettingsProblems(m)...)
wellFormed := true
for _, c := range m.Claims {
if !name.MatchString(c.Name) {
+282
View File
@@ -0,0 +1,282 @@
package catalogue
import (
"bytes"
"encoding/json"
"fmt"
"sort"
"strings"
)
// A requirement a module can run without (novox/hq ADR 0330, issue 166).
//
// **Hard stays the default.** A requirement written as a bare name is hard, exactly as before: it
// resolves, or the assignment is refused, and a single catalogue candidate is installed for it. One
// written `{"name": "influxdb-api", "optional": true}` resolves the same way when a provider is
// already assigned in the mesh or reachable from the consumer (§2), and is otherwise **absent**: not
// refused, and nothing is installed for it, on this machine or elsewhere (§3).
//
// **An absent requirement is never rendered as an empty value** (§4–§6). Its binding, its pair
// credential and its grant are not composed, so a file that names one of them is delivered only while
// the requirement is bound (`only-when-bound`), and a `${bound:…}` placeholder of it outside such a
// file names the value it takes while absent: `${bound:<provision>:<key>|default:<value>}`. The
// module check refuses anything else, so the gap is found when the manifest is written, not on a
// machine where it reads as a blank.
// OnlyWhenBound is a resource's field naming the optional requirement it is delivered only while
// bound (ADR 0330 §5). Said in the catalogue and dropped before the declaration is sent: the
// node-engine parses strictly and knows no such field.
const OnlyWhenBound = "only-when-bound"
// requirementForm is one entry of `requires` in its long form.
type requirementForm struct {
Name string `json:"name"`
Optional bool `json:"optional,omitempty"`
}
// readRequires reads `requires` in both of its forms — a bare name, or {name, optional} — into the
// names and which of them are optional.
func readRequires(raw json.RawMessage) ([]string, map[string]bool, error) {
var entries []json.RawMessage
if err := json.Unmarshal(raw, &entries); err != nil {
return nil, nil, fmt.Errorf("requires: a list of names, or of {name, optional}: %w", err)
}
var names []string
optional := map[string]bool{}
for i, e := range entries {
var plain string
if err := json.Unmarshal(e, &plain); err == nil {
names = append(names, plain)
continue
}
var full requirementForm
dec := json.NewDecoder(bytes.NewReader(e))
dec.DisallowUnknownFields()
if err := dec.Decode(&full); err != nil {
return nil, nil, fmt.Errorf("requires[%d]: a name, or {name, optional} (novox/hq ADR 0330): %w", i, err)
}
if strings.TrimSpace(full.Name) == "" {
return nil, nil, fmt.Errorf("requires[%d] names no requirement: {name, optional}", i)
}
names = append(names, full.Name)
if full.Optional {
optional[full.Name] = true
}
}
if len(optional) == 0 {
optional = nil
}
return names, optional, nil
}
// OptionalRequiresKey is where a stored manifest keeps which of its requirements are optional: beside
// `requires`, which stays a plain list of names (mesh-controller #281's review). A controller older than
// optional requirements reads `requires` as names; the long form inside it failed that read, and with it
// the whole catalogue, so a rollback below this controller would have composed no machine. An older
// controller sets this unknown key aside instead and leaves only the module that carries it out. A
// manifest may be written either way; it is stored this way.
const OptionalRequiresKey = "optional-requires"
// readOptionalRequires reads the stored form's list of optional requirements.
func readOptionalRequires(raw json.RawMessage) (map[string]bool, error) {
var names []string
if err := json.Unmarshal(raw, &names); err != nil {
return nil, fmt.Errorf("%s: a list of the names in `requires` that are optional: %w", OptionalRequiresKey, err)
}
out := map[string]bool{}
for _, n := range names {
out[n] = true
}
return out, nil
}
// optionalRequiresList is the stored form's list: the optional requirements, sorted.
func optionalRequiresList(m Manifest) []string {
var out []string
for name, optional := range m.OptionalRequires {
if optional {
out = append(out, name)
}
}
sort.Strings(out)
return out
}
// IsOptional is whether this module requires the name optionally (ADR 0330 §1). A name it contributes
// to without requiring, or requires as a bare name, is hard.
func (m Manifest) IsOptional(requirement string) bool { return m.OptionalRequires[requirement] }
// wantsHard is whether this module wants the name as a hard requirement: required without
// `optional`, or contributed to.
func (m Manifest) wantsHard(name string) bool {
for _, w := range m.Wants() {
if w == name {
return !m.IsOptional(name)
}
}
return false
}
// Absent is one optional requirement of one consumer that nothing assigned answers (ADR 0330 §3):
// left unbound rather than refused, and listed by `plan` and `node` (§7).
type Absent struct {
// Consumer is the module that requires it optionally.
Consumer string `json:"consumer"`
// Requirement is what it requires.
Requirement string `json:"requirement"`
}
// AbsentFor is the optional requirements of one module this resolution leaves unbound.
func (r Resolution) AbsentFor(module string) map[string]bool {
out := map[string]bool{}
for _, a := range r.Absent {
if a.Consumer == module {
out[a.Requirement] = true
}
}
return out
}
// absentOf is, for each requirement left absent, every module of the set that requires it
// optionally, sorted by consumer then requirement.
func absentOf(names map[string]bool, order []string, catalogue map[string]Manifest) []Absent {
var out []Absent
for _, module := range order {
m := catalogue[module]
for _, r := range m.Requires {
if names[r] && m.IsOptional(r) {
out = append(out, Absent{Consumer: module, Requirement: r})
}
}
}
sort.Slice(out, func(i, j int) bool {
if out[i].Consumer != out[j].Consumer {
return out[i].Consumer < out[j].Consumer
}
return out[i].Requirement < out[j].Requirement
})
return out
}
// withoutAbsent drops the resources delivered only while a requirement is bound whose requirement is
// absent here (ADR 0330 §5), and the field itself from every resource that keeps it.
func withoutAbsent(resources []map[string]any, absent map[string]bool) []map[string]any {
out := make([]map[string]any, 0, len(resources))
for _, r := range resources {
when, has := r[OnlyWhenBound]
if !has {
out = append(out, r)
continue
}
if absent[fmt.Sprint(when)] {
continue
}
kept := make(map[string]any, len(r))
for k, v := range r {
if k != OnlyWhenBound {
kept[k] = v
}
}
out = append(out, kept)
}
return out
}
// secretNamesOf is every `${secret:…}` name that carries a credential of this requirement: its own
// name, and each local name it keeps several under (ADR 0094).
func secretNamesOf(m Manifest, requirement string) []string {
names := []string{requirement}
for local := range m.SecretsMany[requirement] {
names = append(names, local)
}
sort.Strings(names)
return names
}
// OptionalProblems is what the module check refuses of a manifest's optional requirements and its
// only-when-bound resources (ADR 0330 §4–§6).
func OptionalProblems(m Manifest) []string {
var problems []string
required := map[string]bool{}
for _, r := range m.Requires {
if required[r] && m.IsOptional(r) {
problems = append(problems, fmt.Sprintf("%s requires %q twice; say it once, optional or not", m.Module, r))
}
required[r] = true
}
for _, r := range optionalRequiresList(m) {
if !required[r] {
problems = append(problems, fmt.Sprintf(
"%s says %q is optional in %s and does not require it", m.Module, r, OptionalRequiresKey))
}
}
// A contribution is to whoever answers the requirement, so it goes with it: contributing to one
// that is optional is allowed, and composed only while it is bound.
for _, r := range m.Resources {
id := fmt.Sprint(r["id"])
when, marked := r[OnlyWhenBound]
on := ""
if marked {
name, ok := when.(string)
switch {
case !ok || name == "":
problems = append(problems, fmt.Sprintf(
"%s's resource %s says %s %v; it names one requirement the module requires optionally",
m.Module, id, OnlyWhenBound, when))
continue
case fmt.Sprint(r["type"]) != "file":
problems = append(problems, fmt.Sprintf(
"%s's resource %s is a %v and says %s: only a file is left out while a requirement is "+
"absent (novox/hq ADR 0330 §5)", m.Module, id, r["type"], OnlyWhenBound))
continue
case !required[name]:
problems = append(problems, fmt.Sprintf(
"%s's resource %s is delivered only when %q is bound, and %s does not require it",
m.Module, id, name, m.Module))
continue
case !m.IsOptional(name):
problems = append(problems, fmt.Sprintf(
"%s's resource %s is delivered only when %q is bound, which is a hard requirement and "+
"always bound: mark the requirement optional, or drop %s", m.Module, id, name, OnlyWhenBound))
continue
}
on = name
}
if fmt.Sprint(r["type"]) != "file" {
continue
}
content, _ := r["content"].(string)
for _, use := range bound.FindAllStringSubmatch(content, -1) {
provision, key, hasDefault := use[1], use[2], use[3] != ""
if !m.IsOptional(provision) {
if hasDefault {
problems = append(problems, fmt.Sprintf(
"%s's resource %s gives ${bound:%s:%s} a default, and %q is a hard requirement, always "+
"bound: a default it never renders only misleads a reader", m.Module, id, provision, key, provision))
}
continue
}
if !hasDefault && on != provision {
problems = append(problems, fmt.Sprintf(
"%s's resource %s says ${bound:%s:%s}, and %q is optional: while it is absent this would "+
"be empty. Name its value then — ${bound:%s:%s|default:<value>} — or deliver the file "+
"only while it is bound, with \"%s\": %q (novox/hq ADR 0330 §4)",
m.Module, id, provision, key, provision, provision, key, OnlyWhenBound, provision))
}
}
for _, provision := range m.Requires {
if !m.IsOptional(provision) || on == provision {
continue
}
for _, name := range secretNamesOf(m, provision) {
if strings.Contains(content, "${secret:"+name+"}") {
problems = append(problems, fmt.Sprintf(
"%s's resource %s says ${secret:%s}, the credential of %q, which is optional and not "+
"delivered while absent: deliver the file only while it is bound, with \"%s\": %q "+
"(novox/hq ADR 0330 §6)", m.Module, id, name, provision, OnlyWhenBound, provision))
}
}
}
}
return problems
}
@@ -0,0 +1,301 @@
package catalogue
import (
"encoding/json"
"strings"
"testing"
)
// A requirement a module can run without (novox/hq ADR 0330, issue 166). The shape of the issue:
// grafana runs without a data source and requires `influxdb-api` softly; a mesh without influxdb
// could not assign grafana at all, and one with influxdb in its catalogue would have had it installed
// for grafana.
func optionalShelf() map[string]Manifest {
grafana := Manifest{Module: "grafana", Version: "1", Requires: []string{"influxdb-api", "shell"},
OptionalRequires: map[string]bool{"influxdb-api": true, "shell": true}}
return shelf(
Manifest{Module: "influxdb", Version: "1", Provides: FromAnywhere("influxdb-api")},
Manifest{Module: "zsh", Version: "1", Provides: Offers("shell")},
grafana,
// What wants the same names hard, for the cases where one consumer's optional is another's hard.
Manifest{Module: "telegraf", Version: "1", Requires: []string{"influxdb-api"}},
Manifest{Module: "editor", Version: "1", Requires: []string{"shell"}},
Manifest{Module: "suite", Version: "1", Requires: []string{"editor"}},
)
}
func TestARequirementIsHardUnlessMarkedOptional(t *testing.T) {
m := mustParse(t, `{"module":"grafana","version":"1",
"requires":["route",{"name":"influxdb-api","optional":true},{"name":"oidc-client"}]}`)
if strings.Join(m.Requires, " ") != "route influxdb-api oidc-client" {
t.Fatalf("the requirements read as %v", m.Requires)
}
if m.IsOptional("route") || m.IsOptional("oidc-client") {
t.Error("a requirement without `optional` read as optional; hard is the default (ADR 0330 §1)")
}
if !m.IsOptional("influxdb-api") {
t.Error("a requirement marked optional read as hard")
}
// And written back as it was read, so a manifest through the store keeps its form.
raw, err := json.Marshal(m)
if err != nil {
t.Fatal(err)
}
var again Manifest
if err := json.Unmarshal(raw, &again); err != nil {
t.Fatal(err)
}
if !again.IsOptional("influxdb-api") || again.IsOptional("route") || len(again.Requires) != 3 {
t.Fatalf("written and read back, it says %v optional %v (%s)", again.Requires, again.OptionalRequires, raw)
}
if strings.Contains(string(raw), `"name":`) {
t.Errorf("a requirement was stored in the long form: %s", raw)
}
}
// The stored form reads on a controller older than optional requirements (review of mesh-controller
// #281). Inventory.Catalogue stops at the first manifest it cannot read, so a long-form entry inside
// `requires` would have stopped every plan after a rollback. Stored, `requires` is plain names, as main's
// manifest type reads it, and which are optional sits beside it in a key such a controller sets aside.
func TestTheStoredFormReadsWhereRequiresIsAListOfNames(t *testing.T) {
m := mustParse(t, `{"module":"grafana","version":"1",
"requires":["route",{"name":"influxdb-api","optional":true}]}`)
raw, err := json.Marshal(m)
if err != nil {
t.Fatal(err)
}
// main's field, exactly: `Requires []string`.
var older struct {
Requires []string `json:"requires"`
}
if err := json.Unmarshal(raw, &older); err != nil {
t.Fatalf("a controller reading requires as names cannot read the stored form: %v\n%s", err, raw)
}
if strings.Join(older.Requires, " ") != "route influxdb-api" {
t.Fatalf("it reads the requirements as %v", older.Requires)
}
var keys map[string]json.RawMessage
if err := json.Unmarshal(raw, &keys); err != nil {
t.Fatal(err)
}
if string(keys[OptionalRequiresKey]) != `["influxdb-api"]` {
t.Fatalf("which are optional is stored as %s", keys[OptionalRequiresKey])
}
// And this controller reads the stored form back the same, leniently and strictly.
var back Manifest
if err := json.Unmarshal(raw, &back); err != nil {
t.Fatal(err)
}
strict, err := ParseManifest(raw)
if err != nil {
t.Fatal(err)
}
for _, got := range []Manifest{back, strict} {
if !got.IsOptional("influxdb-api") || got.IsOptional("route") || len(got.Requires) != 2 {
t.Fatalf("read back as %v optional %v", got.Requires, got.OptionalRequires)
}
}
// A stored list naming something the module does not require is refused.
if _, err := ParseManifest([]byte(`{"module":"x","version":"1","requires":["route"],"optional-requires":["mqtt-topic"]}`)); err == nil {
t.Error("an optional requirement the module does not require passed the check")
}
}
func TestARequirementsLongFormRefusesWhatItDoesNotKnow(t *testing.T) {
for _, raw := range []string{
`{"module":"x","version":"1","requires":[{"name":"influxdb-api","optinal":true}]}`,
`{"module":"x","version":"1","requires":[{"optional":true}]}`,
} {
if _, err := ParseManifest([]byte(raw)); err == nil {
t.Errorf("accepted %s", raw)
}
}
}
func TestAnOptionalRequirementWithNoProviderIsAbsentAndNothingIsInstalled(t *testing.T) {
// Exactly one catalogue module could answer each — the case where a hard requirement installs it.
// Brokered (influxdb-api, from anywhere) and on this machine (shell); both passes.
for _, world := range []World{{}, {Unchecked: true}} {
got, err := Resolve(optionalShelf(), []string{"grafana"}, reachable(), world)
if err != nil {
t.Fatalf("an optional requirement with no provider refused the assignment: %v", err)
}
if have := strings.Join(names(got), " "); have != "grafana" {
t.Fatalf("assigning grafana put %s on the machine; nothing is installed for an optional requirement (ADR 0330 §3)", have)
}
if world.Unchecked {
continue
}
want := []Absent{{Consumer: "grafana", Requirement: "influxdb-api"}, {Consumer: "grafana", Requirement: "shell"}}
if len(got.Absent) != 2 || got.Absent[0] != want[0] || got.Absent[1] != want[1] {
t.Fatalf("absent: %+v, want %+v", got.Absent, want)
}
if len(got.Needs) != 0 {
t.Fatalf("an absent requirement was given a binding: %+v", got.Needs)
}
}
}
func TestAnOptionalRequirementWithAProviderBindsAsAHardOne(t *testing.T) {
world := World{Offered: map[string][]Provider{"influxdb-api": {
{Node: "anchor", At: "anchor.internal", Module: "influxdb", Serves: map[string]any{"port": 8086}}}}}
got, err := Resolve(optionalShelf(), []string{"grafana", "zsh"}, reachable(), world)
if err != nil {
t.Fatal(err)
}
if n := boundFrom(t, got, "grafana", "influxdb-api"); n.From != "anchor" || n.Module != "influxdb" {
t.Fatalf("bound to %s/%s", n.From, n.Module)
}
if len(got.Absent) != 0 {
t.Fatalf("a requirement with a provider was left absent: %+v", got.Absent)
}
// And the same assignment without the provider is absent: what binds it later is only the
// provider being assigned, with no act on grafana (ADR 0330 §7).
before, err := Resolve(optionalShelf(), []string{"grafana", "zsh"}, reachable(), World{})
if err != nil {
t.Fatal(err)
}
if len(before.Absent) != 1 || before.Absent[0].Requirement != "influxdb-api" {
t.Fatalf("absent before the provider was assigned: %+v", before.Absent)
}
}
func TestOneConsumersOptionalRequirementIsAnothersHardOne(t *testing.T) {
// telegraf wants influxdb-api hard: with nothing offering it, the assignment is refused as
// before, naming what to assign — grafana's optional does not soften telegraf's.
_, err := Resolve(optionalShelf(), []string{"grafana", "telegraf"}, reachable(), World{})
if err == nil || !strings.Contains(err.Error(), "assign influxdb") {
t.Fatalf("a hard requirement with no provider was not refused naming the candidate: %v", err)
}
// And the refusal names telegraf, whose remedy it is, not grafana, which met the name first.
if !strings.Contains(err.Error(), "wanted by telegraf") {
t.Errorf("the refusal does not name the module that wants it hard: %v", err)
}
// And a module pulled in later (suite → editor) that wants the shell hard installs it, even
// though grafana, met first, wanted it only optionally — and grafana is then bound to it.
got, err := Resolve(optionalShelf(), []string{"grafana", "suite"}, workstation(), World{})
if err != nil {
t.Fatal(err)
}
if have := strings.Join(names(got), " "); !strings.Contains(have, "zsh") {
t.Fatalf("a hard requirement met after an optional one was not installed: %s", have)
}
for _, a := range got.Absent {
if a.Requirement == "shell" {
t.Fatalf("the shell is installed and still listed absent: %+v", got.Absent)
}
}
}
func TestAHardRequirementWithNoProviderIsStillRefused(t *testing.T) {
_, err := Resolve(optionalShelf(), []string{"telegraf"}, reachable(), World{})
if err == nil || !strings.Contains(err.Error(), "influxdb") {
t.Fatalf("got %v", err)
}
}
func optionalModule(resources string) string {
return `{"module":"grafana","version":"1",
"requires":["route",{"name":"influxdb-api","optional":true}],
"secrets":{"influxdb-api":"/var/lib/grafana/influxdb-api"},
"resources":[` + resources + `]}`
}
func TestTheModuleCheckRefusesAnOptionalPlaceholderThatCouldRenderEmpty(t *testing.T) {
refused := map[string]string{
"a placeholder without a default": `{"id":"ds","type":"file","path":"/a","mode":"0644",
"content":"url: ${bound:influxdb-api:at}"}`,
"a secret outside an only-when-bound file": `{"id":"s","type":"file","path":"/a","mode":"0400",
"content":"${secret:influxdb-api}"}`,
"only-when-bound on a hard requirement": `{"id":"r","type":"file","path":"/a","mode":"0644",
"content":"x","only-when-bound":"route"}`,
"only-when-bound on something not required": `{"id":"r","type":"file","path":"/a","mode":"0644",
"content":"x","only-when-bound":"mqtt-topic"}`,
"only-when-bound on a directory": `{"id":"d","type":"directory","mode":"0700",
"only-when-bound":"influxdb-api"}`,
"a default on a hard requirement": `{"id":"r","type":"file","path":"/a","mode":"0644",
"content":"${bound:route:name|default:x}"}`,
}
for what, resource := range refused {
if _, err := ParseManifest([]byte(optionalModule(resource))); err == nil {
t.Errorf("%s passed the module check", what)
} else if what == "a placeholder without a default" && !strings.Contains(err.Error(), "|default:") {
t.Errorf("the refusal does not say how to write it: %v", err)
}
}
accepted := map[string]string{
"a placeholder with a default": `{"id":"ds","type":"file","path":"/a","mode":"0644",
"content":"url: ${bound:influxdb-api:at|default:none}"}`,
"a placeholder and a secret inside an only-when-bound file": `{"id":"ds","type":"file","path":"/a",
"mode":"0400","content":"url: ${bound:influxdb-api:at}\npassword: ${secret:influxdb-api}",
"only-when-bound":"influxdb-api"}`,
}
for what, resource := range accepted {
if _, err := ParseManifest([]byte(optionalModule(resource))); err != nil {
t.Errorf("%s was refused: %v", what, err)
}
}
}
func absentGrafana(t *testing.T, absent bool) Resolution {
t.Helper()
m := mustParse(t, optionalModule(`
{"id":"env","type":"file","path":"/var/lib/grafana/env","mode":"0644",
"content":"INFLUX=${bound:influxdb-api:at|default:none}:${bound:influxdb-api:port|default:0}\n"},
{"id":"ds","type":"file","path":"/var/lib/grafana/ds.yaml","mode":"0400",
"content":"url: ${bound:influxdb-api:at}\npassword: ${secret:influxdb-api}\n","only-when-bound":"influxdb-api"}`))
m.Requires = []string{"influxdb-api"}
m.Binds = map[string]string{"influxdb-api": "/var/lib/grafana/influxdb.json"}
r := Resolution{Node: "workstation", Modules: []Manifest{m}}
if absent {
r.Absent = []Absent{{Consumer: "grafana", Requirement: "influxdb-api"}}
} else {
r.Needs = []Needed{{Name: "influxdb-api", From: "anchor", At: "anchor.internal", For: "grafana",
Module: "influxdb", Sealed: "sealed-influx", Serves: map[string]any{"port": float64(8086)}}}
}
return r
}
func TestAnAbsentRequirementIsComposedWithItsDefaultsAndWithoutItsFiles(t *testing.T) {
out, err := absentGrafana(t, true).Declaration(Rendering{})
if err != nil {
t.Fatal(err)
}
env := fileNamed(out, "grafana.env")
if env == nil || env["content"] != "INFLUX=none:0\n" {
t.Fatalf("the defaults were not rendered: %v", env)
}
for _, id := range []string{"grafana.ds", "grafana." + SecretID("influxdb-api"), "grafana." + BoundID("influxdb-api")} {
if fileNamed(out, id) != nil {
t.Errorf("%s was delivered while influxdb-api is absent (ADR 0330 §5, §6)", id)
}
}
for _, r := range out {
if _, has := r[OnlyWhenBound]; has {
t.Errorf("%v carries %s to the node-engine, which knows no such field", r["id"], OnlyWhenBound)
}
}
}
func TestABoundOptionalRequirementIsComposedAsAHardOne(t *testing.T) {
out, err := absentGrafana(t, false).Declaration(Rendering{})
if err != nil {
t.Fatal(err)
}
if env := fileNamed(out, "grafana.env"); env == nil || env["content"] != "INFLUX=anchor.internal:8086\n" {
t.Fatalf("the bound values were not rendered: %v", env)
}
ds := fileNamed(out, "grafana.ds")
if ds == nil {
t.Fatalf("the only-when-bound file was left out while bound: %v", out)
}
if _, has := ds[OnlyWhenBound]; has {
t.Errorf("the file carries %s to the node-engine", OnlyWhenBound)
}
for _, id := range []string{"grafana." + SecretID("influxdb-api"), "grafana." + BoundID("influxdb-api")} {
if fileNamed(out, id) == nil {
t.Errorf("%s was not delivered while bound", id)
}
}
}
+92 -2
View File
@@ -183,6 +183,10 @@ type Resolution struct {
// that keeps its data, and did not (novox/hq ADR 0232): it stays bound where its data is, and the
// controller raises an urgent condition naming the move until a person pins one or the other.
Kept []KeptBinding
// Absent is every optional requirement of this node's modules that nothing assigned answers
// (novox/hq ADR 0330 §3): left unbound rather than refused, with nothing installed for it, and
// listed by `plan` and `node` (§7). It binds on the first composition after a provider is assigned.
Absent []Absent
}
// Unhostable is one directly-assigned module the machine cannot run.
@@ -374,12 +378,80 @@ func Resolve(catalogue map[string]Manifest, assigned []string, node Node, world
// identical lines before realising there is none.
reported := map[string]bool{}
for len(queue) > 0 {
// **An optional requirement is settled last** (novox/hq ADR 0330). Whether a name is optional is
// a question about every module that wants it, and a module pulled in later may want it hard; so
// a name only optional modules want so far is put aside until the walk has nothing else, and only
// then, if still nobody wants it hard, settled by the optional rule: bound to a provider that is
// already assigned or reachable, exactly as a hard one, and otherwise absent — never refused,
// and never answered by installing anything.
var deferred []string
settling := map[string]bool{}
absent := map[string]bool{}
wantedHard := func(want string) bool {
for _, name := range order {
if catalogue[name].wantsHard(want) {
return true
}
}
return false
}
wantedOptionally := func(want string) bool {
for _, name := range order {
if catalogue[name].IsOptional(want) {
return true
}
}
return false
}
leaveAbsent := func(want string) {
absent[want] = true
reported[want] = true
}
for {
if len(queue) == 0 {
if len(deferred) == 0 {
break
}
// What a module chosen since wants hard goes back to the walk, which may install for
// it; only when none does is the rest settled, since settling installs nothing and so
// can change no other name's answer.
var hard, optional []string
for _, d := range deferred {
if wantedHard(d) {
hard = append(hard, d)
} else {
optional = append(optional, d)
}
}
if len(hard) > 0 {
queue, deferred = hard, optional
continue
}
for _, d := range optional {
settling[d] = true
}
queue, deferred = optional, nil
}
want := queue[0]
queue = queue[1:]
if chosen[want] || reported[want] {
continue
}
if !settling[want] && because[want] != "assigned" && !wantedHard(want) && wantedOptionally(want) {
deferred = append(deferred, want)
continue
}
// Walked as a hard requirement: a refusal names a module that wants it hard, never one that
// wanted it only optionally and met it first — whose remedy it is not.
if first, known := catalogue[because[want]]; !settling[want] && known && !first.wantsHard(want) {
for _, name := range order {
if catalogue[name].wantsHard(want) {
because[want] = name
break
}
}
}
// Already answered by something in the set. This is the case that makes assigning zsh do
// what a person meant by it.
//
@@ -483,6 +555,12 @@ func Resolve(catalogue map[string]Manifest, assigned []string, node Node, world
if brokered[want] {
reported[want] = true
where := world.Offered[want]
if settling[want] && len(where) == 0 {
// Nothing assigned anywhere answers it: absent, and nothing is put on a machine
// for it (ADR 0330 §3), however few modules in the catalogue could.
leaveAbsent(want)
continue
}
names := providerNames(where)
take := func(p Provider) {
if node.At != "" && p.At == "" || node.At == "" && p.At != "" || node.At == "" && p.At == "" {
@@ -579,6 +657,10 @@ func Resolve(catalogue map[string]Manifest, assigned []string, node Node, world
// research 026 found. Another node's provider never counts — node scope is never brokered.
if machineReach[want] && !isModule(catalogue, want) {
reported[want] = true
if settling[want] {
leaveAbsent(want)
continue
}
if world.Unchecked {
// The first pass's refusals take a machine off the network; the second says it.
continue
@@ -591,6 +673,13 @@ func Resolve(catalogue map[string]Manifest, assigned []string, node Node, world
}
candidates := offers[want]
if settling[want] && !(len(candidates) == 0 && len(world.Licences[want]) > 0) {
// Not answered by anything in this node's set, and only installing could answer it
// here — which is never done for an optional requirement (ADR 0330 §3), not even
// for the only candidate. A licence record still answers it, in the post-pass.
leaveAbsent(want)
continue
}
switch len(candidates) {
case 0:
if len(world.Licences[want]) > 0 {
@@ -724,7 +813,8 @@ func Resolve(catalogue map[string]Manifest, assigned []string, node Node, world
resolution := Resolution{Node: node.Name, At: node.At, PublicDomain: node.PublicDomain,
Account: node.Account, AccountHome: node.AccountHome, AgentAccount: node.AgentAccount,
AgentAccountHome: node.AgentAccountHome, Capabilities: node.Capabilities,
Because: because, Needs: needs, Unhostable: unhostable, Kept: kept}
Because: because, Needs: needs, Unhostable: unhostable, Kept: kept,
Absent: absentOf(absent, order, catalogue)}
for _, n := range providersFirst(order, catalogue) {
resolution.Modules = append(resolution.Modules, catalogue[n])
}
+176
View File
@@ -0,0 +1,176 @@
package catalogue
import (
"fmt"
"strings"
"unicode"
)
// A mergeable file names the setting keys it takes (novox/hq ADR 0326, issue 168).
//
// Before this, every key of a module's layers merged into every one of its mergeable files: the
// mesh's own words, a key meant for a `${setting:<key>}` in another file, a key for a second
// configuration file. So a module could not own two configurable files, and every receiver had to
// ignore what it did not know. ADR 0164 §11 decided that a setting names the files that read it; this
// is that relation, stated from the file's side, so the operator's own values stay declared nowhere
// (ADR 0262 §2).
// TakesSettingsField is the key a mergeable file carries to name the top-level setting keys it takes.
const TakesSettingsField = "takes-settings"
// takenKeys is the keys a resource names in takes-settings, and whether it names any list at all. An
// empty list is a list: that file takes no setting.
func takenKeys(resource map[string]any) (map[string]bool, bool) {
raw, said := resource[TakesSettingsField]
if !said {
return nil, false
}
keys := map[string]bool{}
for _, k := range stringsOf(raw) {
keys[k] = true
}
return keys, true
}
// onlyTaken is the layers with every key the file does not name left out. The file's own defaults and
// what the mesh reads itself are not touched here: settle already keeps both out of a file.
func onlyTaken(layers []Layer, taken map[string]bool) []Layer {
kept := make([]Layer, 0, len(layers))
for _, layer := range layers {
values := map[string]any{}
for key, value := range layer.Values {
if taken[key] {
values[key] = value
}
}
kept = append(kept, Layer{From: layer.From, Values: values, Default: layer.Default})
}
return kept
}
// namesItsKeys says whether a module's mergeable files name the keys they take (ADR 0326 §2): true
// only when it has one and every one carries takes-settings. A module with a silent mergeable file
// keeps today's form (§3), where every key reaches every mergeable file.
func namesItsKeys(m Manifest) bool {
merging := 0
for _, r := range m.Resources {
if how, _ := r["merge"].(string); how == "" {
continue
}
merging++
if _, said := r[TakesSettingsField]; !said {
return false
}
}
return merging > 0
}
// TakenSettingsProblems is every way a definition's takes-settings is wrong (ADR 0326 §2 and §4), in
// its own words: on a file that does not merge, not a list of keys, a key named twice, one the mesh
// reads itself, one the file keeps for itself, or one mergeable file naming its keys beside another
// that names none.
func TakenSettingsProblems(m Manifest) []string {
// Not yet here: refusing a module that declares a setting and owns a mergeable file naming no keys
// (ADR 0326 §3). betterbird is one, and the catalogue cannot name its keys before this controller
// runs, so that refusal follows once the catalogue does (novox/hq issue 574).
return takenSettingsProblems(m)
}
// takenSettingsProblems is every rule of ADR 0326 §2 and §4, judged where a definition is checked and
// where a stored one is composed.
func takenSettingsProblems(m Manifest) []string {
var problems []string
var naming, silent []string
for _, r := range m.Resources {
id := fmt.Sprint(r["id"])
how, _ := r["merge"].(string)
raw, said := r[TakesSettingsField]
if how == "" {
if said {
problems = append(problems, fmt.Sprintf("%s's %s says %s and does not merge: only a mergeable "+
"file takes settings by key (novox/hq ADR 0326)", m.Module, id, TakesSettingsField))
}
continue
}
if !said {
silent = append(silent, id)
continue
}
naming = append(naming, id)
list, isList := raw.([]any)
if !isList {
problems = append(problems, fmt.Sprintf("%s's %s: %s is a list of setting keys, and this is %T",
m.Module, id, TakesSettingsField, raw))
continue
}
protected := map[string]bool{}
for _, k := range stringsOf(r["protected"]) {
protected[k] = true
}
seen := map[string]bool{}
for _, item := range list {
key, isString := item.(string)
switch {
case !isString || !usableKey(key):
problems = append(problems, fmt.Sprintf("%s's %s: %v in %s is not a usable setting key",
m.Module, id, item, TakesSettingsField))
case seen[key]:
problems = append(problems, fmt.Sprintf("%s's %s names %q twice in %s",
m.Module, id, key, TakesSettingsField))
case meshWords[key]:
problems = append(problems, fmt.Sprintf("%s's %s names %q in %s, and the mesh reads %q itself "+
"(novox/hq ADR 0164 §14)", m.Module, id, key, TakesSettingsField, key))
case protected[key]:
problems = append(problems, fmt.Sprintf("%s's %s names %q in %s and keeps it as protected: a "+
"key is the file's or the setting's, never both", m.Module, id, key, TakesSettingsField))
}
if isString {
seen[key] = true
}
}
}
if len(naming) > 0 {
for _, id := range silent {
problems = append(problems, fmt.Sprintf("%s's %s merges and names no %s while %s does: once one "+
"mergeable file of a module names the keys it takes, every one does, an empty list for none "+
"(novox/hq ADR 0326)", m.Module, id, TakesSettingsField, strings.Join(naming, ", ")))
}
}
return problems
}
// usableKey is a key a file can be said to take: not empty, and no space, line end or control
// character. Not the lower-case rule of a `${setting:<key>}`: a file's own keys are its software's
// spelling, such as `timeZone`.
func usableKey(key string) bool {
if key == "" {
return false
}
for _, r := range key {
if unicode.IsSpace(r) || unicode.IsControl(r) || !unicode.IsPrint(r) {
return false
}
}
return true
}
// takenSaid is what a stray key's message adds for a module whose files name their keys: which keys
// each file takes, so the person sees where the key would have had to be named.
func takenSaid(m Manifest) string {
if !namesItsKeys(m) {
return ""
}
var files []string
for _, r := range m.Resources {
if how, _ := r["merge"].(string); how == "" {
continue
}
keys := stringsOf(r[TakesSettingsField])
if len(keys) == 0 {
files = append(files, fmt.Sprintf("%v takes none", r["id"]))
continue
}
files = append(files, fmt.Sprintf("%v takes %s", r["id"], strings.Join(keys, ", ")))
}
return " (" + strings.Join(files, "; ") + ")"
}
+196
View File
@@ -0,0 +1,196 @@
package catalogue
import (
"encoding/json"
"strings"
"testing"
)
// twoFiles is a module that owns two configurable files, which issue 168 said no module could:
// every key of every layer reached both.
func twoFiles(app, sidecar []any) Manifest {
appFile := map[string]any{"id": "app", "type": "file", "path": "/etc/app.json", "merge": MergeJSON,
"content": `{"server":{"port":8080},"search":{"safe":1}}`}
sideFile := map[string]any{"id": "sidecar", "type": "file", "path": "/etc/sidecar.json", "merge": MergeJSON,
"content": `{"url":"http://127.0.0.1:8080"}`}
if app != nil {
appFile[TakesSettingsField] = app
}
if sidecar != nil {
sideFile[TakesSettingsField] = sidecar
}
return Manifest{Module: "search", Resources: []map[string]any{
appFile, sideFile,
{"id": "env", "type": "file", "path": "/etc/search.env", "content": "SITE=${setting:site-name}\n"},
}}
}
func settledContent(t *testing.T, resource map[string]any, layers []Layer) map[string]any {
t.Helper()
out, err := ApplySettings(resource, layers)
if err != nil {
t.Fatal(err)
}
if _, kept := out[TakesSettingsField]; kept {
t.Errorf("%v still carries %s; the host knows no such field", out["id"], TakesSettingsField)
}
var parsed map[string]any
if err := json.Unmarshal([]byte(out["content"].(string)), &parsed); err != nil {
t.Fatalf("%v is not JSON after settling: %v", out["id"], err)
}
return parsed
}
func TestEachMergeableFileTakesOnlyTheKeysItNames(t *testing.T) {
// novox/hq ADR 0326 §1, issue 168: two configurable files, each taking its own keys. A key for
// the other file, for a ${setting:…} elsewhere or for the mesh itself lands in neither.
m := twoFiles([]any{"server", "search"}, []any{"url"})
layers := []Layer{
{From: "the mesh", Values: map[string]any{
"server": map[string]any{"base_url": "https://search.example.tld/"},
"url": "http://127.0.0.1:9090", "site-name": "Search",
"endpoints": map[string]any{"web": map[string]any{"reach": "public"}}}},
{From: "node-a", Values: map[string]any{"search": map[string]any{"safe": 0.0}}},
}
app := settledContent(t, m.Resources[0], layers)
for _, stray := range []string{"url", "site-name", "endpoints"} {
if _, in := app[stray]; in {
t.Errorf("app took %q, which it does not name: %v", stray, app)
}
}
server, _ := app["server"].(map[string]any)
if server["base_url"] != "https://search.example.tld/" || server["port"] != 8080.0 {
t.Errorf("app's server block is %v; the setting should merge into the file's own", server)
}
if search, _ := app["search"].(map[string]any); search["safe"] != 0.0 {
t.Errorf("app's search block is %v; the node's layer should win", search)
}
side := settledContent(t, m.Resources[1], layers)
if side["url"] != "http://127.0.0.1:9090" {
t.Errorf("sidecar's url is %v; it names url", side["url"])
}
if len(side) != 1 {
t.Errorf("sidecar took keys it does not name: %v", side)
}
}
func TestAnEmptyListTakesNoSetting(t *testing.T) {
// ADR 0164 §12: a file that names no setting takes none.
m := twoFiles([]any{"server"}, []any{})
side := settledContent(t, m.Resources[1], []Layer{{From: "the mesh", Values: map[string]any{
"server": map[string]any{"port": 1.0}, "url": "x"}}})
if len(side) != 1 || side["url"] != "http://127.0.0.1:8080" {
t.Errorf("a file taking none was changed: %v", side)
}
}
func TestAFileThatNamesNothingKeepsTodaysForm(t *testing.T) {
// ADR 0326 §3, ADR 0164 §15: a module none of whose files names keys is left as it was.
m := twoFiles(nil, nil)
app := settledContent(t, m.Resources[0], []Layer{{From: "the mesh", Values: map[string]any{"url": "x"}}})
if app["url"] != "x" {
t.Errorf("a silent file should still take every key until its module names them: %v", app)
}
if unused := UnusedSettings(m, []Layer{{From: "the mesh", Values: map[string]any{"anything": 1}}}); len(unused) != 0 {
t.Errorf("today's form has nothing stray, and said: %v", unused)
}
}
func TestAKeyNoFileNamesIsStray(t *testing.T) {
// ADR 0326 §5: once the files name their keys, a key none of them names — and nothing else
// takes — is stray, and is said with what each file takes.
m := twoFiles([]any{"server", "search"}, []any{"url"})
unused := UnusedSettings(m, []Layer{{From: "the mesh", Values: map[string]any{
"server": 1, "url": "x", "site-name": "Search", "ui": map[string]any{"theme": "dark"}}}})
if len(unused) != 1 || !strings.Contains(unused[0], `"ui"`) {
t.Fatalf("only ui reaches nothing; named: %v", unused)
}
if !strings.Contains(unused[0], "app takes server, search") || !strings.Contains(unused[0], "sidecar takes url") {
t.Errorf("the stray key's message does not say what each file takes: %s", unused[0])
}
}
func TestOneNamingFileMakesEveryMergeableFileName(t *testing.T) {
// ADR 0326 §2: a module with one naming and one silent mergeable file is refused, naming the
// silent one — the silent one would otherwise still take every key.
m := twoFiles([]any{"server"}, nil)
problems := TakenSettingsProblems(m)
if len(problems) != 1 || !strings.Contains(problems[0], "sidecar merges and names no takes-settings") {
t.Fatalf("problems: %v", problems)
}
if err := JudgeSettings(m, nil, false); err == nil || !strings.Contains(err.Error(), "sidecar") {
t.Errorf("composing it should refuse it too, and said: %v", err)
}
raw, err := json.Marshal(map[string]any{"module": m.Module, "version": "1", "resources": m.Resources})
if err != nil {
t.Fatal(err)
}
if _, err := ParseManifest(raw); err == nil || !strings.Contains(err.Error(), "sidecar merges and names no") {
t.Errorf("the module check does not say it: %v", err)
}
}
func TestTakesSettingsIsRefusedWhereItCannotMeanAnything(t *testing.T) {
// ADR 0326 §4.
cases := map[string]struct {
resource map[string]any
says string
}{
"on a file that does not merge": {
map[string]any{"id": "plain", "type": "file", "path": "/x", "content": "x",
TakesSettingsField: []any{"a"}},
"does not merge"},
"not a list": {
map[string]any{"id": "conf", "type": "file", "path": "/x", "merge": MergeJSON, "content": "{}",
TakesSettingsField: "server"},
"is a list of setting keys"},
"a mesh word": {
map[string]any{"id": "conf", "type": "file", "path": "/x", "merge": MergeJSON, "content": "{}",
TakesSettingsField: []any{"endpoints"}},
"the mesh reads"},
"a protected key": {
map[string]any{"id": "conf", "type": "file", "path": "/x", "merge": MergeJSON, "content": "{}",
"protected": []any{"secret"}, TakesSettingsField: []any{"secret"}},
"keeps it as protected"},
"named twice": {
map[string]any{"id": "conf", "type": "file", "path": "/x", "merge": MergeJSON, "content": "{}",
TakesSettingsField: []any{"a", "a"}},
"twice"},
"not a key": {
map[string]any{"id": "conf", "type": "file", "path": "/x", "merge": MergeJSON, "content": "{}",
TakesSettingsField: []any{"two words"}},
"not a usable setting key"},
}
for name, c := range cases {
problems := TakenSettingsProblems(Manifest{Module: "m", Resources: []map[string]any{c.resource}})
if !containsText(problems, c.says) {
t.Errorf("%s: want a problem saying %q, got %v", name, c.says, problems)
}
}
// And a file that does not merge is refused where it is settled, not handed to the host.
plain := map[string]any{"id": "plain", "type": "file", "path": "/x", "content": "x",
TakesSettingsField: []any{"a"}}
if _, err := ApplySettings(plain, nil); err == nil {
t.Error("a file that does not merge and names keys was settled")
}
}
func TestASoftwareSpelledKeyIsUsable(t *testing.T) {
// A file's keys are its software's: nodered's timeZone is not lower case and is still a key.
m := Manifest{Module: "flows", Resources: []map[string]any{{"id": "settings", "type": "file",
"path": "/x", "merge": MergeJSON, "content": "{}", TakesSettingsField: []any{"timeZone"}}}}
if problems := TakenSettingsProblems(m); len(problems) != 0 {
t.Errorf("timeZone was refused: %v", problems)
}
}
func containsText(list []string, text string) bool {
for _, s := range list {
if strings.Contains(s, text) {
return true
}
}
return false
}
+32 -3
View File
@@ -46,11 +46,24 @@ type Layer struct {
// rather than by repeating everything the rest already say.
func ApplySettings(resource map[string]any, layers []Layer) (map[string]any, error) {
how, _ := resource["merge"].(string)
taken, names := takenKeys(resource)
if how == "" {
if names {
// Refused by the module check too (TakenSettingsProblems); said here as well, for a
// definition stored by an older binary, rather than handed to the host, which knows no
// such field.
return nil, fmt.Errorf("%v says %s and does not merge: only a mergeable file takes "+
"settings by key (novox/hq ADR 0326)", resource["id"], TakesSettingsField)
}
// Not a mergeable file. Settings for it are ignored rather than silently doing nothing
// somewhere else — see UnusedSettings, which is what says so.
return resource, nil
}
if names {
// **The file takes the keys it names, and no other** (novox/hq ADR 0326, issue 168): a key
// for another file, for a `${setting:<key>}` or for the mesh itself does not land here.
layers = onlyTaken(layers, taken)
}
if how != MergeJSON {
return nil, fmt.Errorf(
"%v says it merges as %q, and this control plane knows how to merge %q",
@@ -91,6 +104,7 @@ func ApplySettings(resource map[string]any, layers []Layer) (map[string]any, err
delete(out, "merge")
delete(out, "protected")
delete(out, TrustedField)
delete(out, TakesSettingsField)
return out, nil
}
@@ -324,8 +338,11 @@ func sortedKeysAny(m map[string]any) []string {
// (novox/hq 04-ISSUES/173); and the mesh's own words — `expose`, `ports`, `reach`, `endpoints` —
// are read by the mesh. A key none of those takes is stray, and is said so rather than dropped.
func UnusedSettings(m Manifest, layers []Layer) []string {
named := namesItsKeys(m)
for _, r := range m.Resources {
if how, _ := r["merge"].(string); how != "" {
if how, _ := r["merge"].(string); how != "" && !named {
// Today's form (novox/hq ADR 0326 §3, ADR 0164 §15): a mergeable file that names no
// keys takes every key, so nothing is stray.
return nil
}
}
@@ -337,6 +354,13 @@ func UnusedSettings(m Manifest, layers []Layer) []string {
}
lands := settingKeysUsedBy(m)
// A mergeable file takes the keys it names (novox/hq ADR 0326 §5).
for _, r := range m.Resources {
taken, _ := takenKeys(r)
for key := range taken {
lands[key] = true
}
}
// A key the module gives a default reaches what asks for it (novox/hq ADR 0262): setting it
// overrides the default, which is how a running module gains a setting without a gap between.
for key := range m.Settings {
@@ -401,8 +425,8 @@ func UnusedSettings(m Manifest, layers []Layer) []string {
}
unused = append(unused, fmt.Sprintf(
"%s sets %q, and %s has no file that merges it, asks for no ${setting:%s}, and "+
"declares no %q in what it contributes or serves",
layer.From, key, m.Module, key, key))
"declares no %q in what it contributes or serves%s",
layer.From, key, m.Module, key, key, takenSaid(m)))
}
}
sort.Strings(unused)
@@ -512,6 +536,11 @@ func JudgeSettings(m Manifest, layers []Layer, adopted bool) error {
if err := settingsHoldOneLine(m.Module, layers); err != nil {
return err
}
// Which keys each mergeable file takes is the definition's to say right (novox/hq ADR 0326);
// judged here too for a definition stored by an older binary.
if problems := takenSettingsProblems(m); len(problems) > 0 {
return fmt.Errorf("%s", problems[0])
}
// With no layers too: a definition may ask for a setting nobody made — an access placed by
// nobody, a file's ${setting:…} nothing sets — and that is the same statement, missing.
if _, err := GivenPorts(m, layers); err != nil {
+74
View File
@@ -0,0 +1,74 @@
package inventory
import (
"context"
"encoding/json"
"errors"
"fmt"
"time"
"github.com/jackc/pgx/v5"
)
// MachineAccount is what a machine said it is made of (novox/hq issue 160), from the profile it last
// reported: the architecture and kernel family the profile always carried, and the facts a node-engine that
// knows how says beside them. Read by `node`, so what a machine says of itself is visible in one place —
// collected and shown in the same breath, not collected and read by nothing (issue 160's pattern).
type MachineAccount struct {
Architecture string `json:"architecture"`
Kernel string `json:"kernel"`
// Machine is nil when the node-engine that reported is older than the one that says it.
Machine *MadeOf `json:"machine"`
}
// MadeOf is the node-engine's account of what it is made of, as it put it on the wire. A fact it could
// not determine is absent here and named in Unknown with why — never a zero.
type MadeOf struct {
ObservedAt time.Time `json:"observed_at"`
KernelRelease string `json:"kernel_release"`
Distribution string `json:"distribution"`
Version string `json:"version"`
Processor string `json:"processor"`
Cores int `json:"cores"`
Threads int `json:"threads"`
MemoryBytes int64 `json:"memory_bytes"`
Virtual string `json:"virtual"`
BootedAt *time.Time `json:"booted_at"`
Timezone string `json:"timezone"`
Filesystems []MountedFilesystem `json:"filesystems"`
Unknown map[string]string `json:"unknown"`
}
// MountedFilesystem is one local filesystem a machine has mounted.
type MountedFilesystem struct {
Mount string `json:"mount"`
Device string `json:"device"`
Type string `json:"type"`
SizeBytes int64 `json:"size_bytes"`
FreeBytes int64 `json:"free_bytes"`
}
// MachineOf is what a node last said it is made of. had is false for a node that has never reported a
// profile at all, which is a different answer from one whose node-engine does not say the facts yet.
func (i *Inventory) MachineOf(ctx context.Context, nodeName string) (account MachineAccount, had bool, err error) {
var raw []byte
err = i.store.Pool().QueryRow(ctx, `select profile from node where name = $1`, nodeName).Scan(&raw)
if errors.Is(err, pgx.ErrNoRows) {
return MachineAccount{}, false, fmt.Errorf("%w: %s", ErrNoSuchNode, nodeName)
}
if err != nil {
return MachineAccount{}, false, err
}
return machineFrom(raw)
}
func machineFrom(raw []byte) (MachineAccount, bool, error) {
if len(raw) == 0 {
return MachineAccount{}, false, nil
}
var account MachineAccount
if err := json.Unmarshal(raw, &account); err != nil {
return MachineAccount{}, false, err
}
return account, true, nil
}
+42
View File
@@ -0,0 +1,42 @@
package inventory
import (
"testing"
)
// The node-engine's words for a machine's facts (novox/hq issue 160), as mesh-host's profile.Machine puts
// them on the wire. A rename on either side fails here rather than as a blank on a real machine.
func TestAMachinesAccountIsReadInTheNodeEnginesWords(t *testing.T) {
raw := []byte(`{"architecture":"amd64","kernel":"linux","capabilities":[],
"machine":{"observed_at":"2026-10-12T15:00:00Z","kernel_release":"7.2.8-arch1-2",
"distribution":"Arch Linux","version":"rolling","processor":"AMD Ryzen 9","cores":8,"threads":16,
"memory_bytes":32891494400,"virtual":"physical","booted_at":"2026-10-09T08:50:34Z",
"timezone":"Europe/Brussels",
"filesystems":[{"mount":"/","device":"/dev/nvme0n1p3","type":"ext4","size_bytes":100,"free_bytes":40}],
"unknown":{"public address":"not decided"}}}`)
a, had, err := machineFrom(raw)
if err != nil || !had {
t.Fatalf("had %v, err %v", had, err)
}
m := a.Machine
if a.Architecture != "amd64" || a.Kernel != "linux" || m == nil {
t.Fatalf("%+v", a)
}
if m.KernelRelease != "7.2.8-arch1-2" || m.Distribution != "Arch Linux" || m.Version != "rolling" ||
m.Processor != "AMD Ryzen 9" || m.Cores != 8 || m.Threads != 16 || m.MemoryBytes != 32891494400 ||
m.Virtual != "physical" || m.BootedAt == nil || m.Timezone != "Europe/Brussels" ||
len(m.Filesystems) != 1 || m.Filesystems[0].FreeBytes != 40 || m.Unknown["public address"] == "" ||
m.ObservedAt.IsZero() {
t.Errorf("%+v", m)
}
}
func TestAProfileWithoutTheFactsIsAnOlderNodeEngineNotNothing(t *testing.T) {
a, had, err := machineFrom([]byte(`{"architecture":"amd64","kernel":"linux","capabilities":[]}`))
if err != nil || !had || a.Machine != nil || a.Architecture != "amd64" {
t.Errorf("%+v had %v err %v", a, had, err)
}
if _, had, _ := machineFrom(nil); had {
t.Error("a node that never reported read as having reported")
}
}
+12 -3
View File
@@ -1064,9 +1064,14 @@ type Reported struct {
// SentDeclaration is what one send carried to a machine, as a gate keeps it: the declaration's digest and
// its sequence (novox/hq issue 352). A report about this declaration, or about one sequenced after it, is a
// report on what the gate sent — whatever the machine was sent since.
//
// At is when the controller sent it to that machine: the node's own send time, which is where that machine's
// `apply` duration starts. A walk measures its rest from it, not from its own record of the send, made only after
// every machine of the rest was sent (novox/hq issue 582). Zero for a send kept before it was recorded.
type SentDeclaration struct {
Digest string `json:"digest"`
Sequence int64 `json:"sequence,omitempty"`
Digest string `json:"digest"`
Sequence int64 `json:"sequence,omitempty"`
At time.Time `json:"at,omitzero"`
}
// ReportsOn says a report is about this send: the declaration itself; one the same machine was sequenced
@@ -1085,7 +1090,8 @@ func (s SentDeclaration) ReportsOn(r Reported) bool {
func (i *Inventory) SentTo(ctx context.Context, name string) (SentDeclaration, bool, error) {
var digest *string
var seq *int64
err := i.store.Pool().QueryRow(ctx, `select sent, sequence from node where name = $1`, name).Scan(&digest, &seq)
var at *time.Time
err := i.store.Pool().QueryRow(ctx, `select sent, sequence, sent_at from node where name = $1`, name).Scan(&digest, &seq, &at)
if errors.Is(err, pgx.ErrNoRows) {
return SentDeclaration{}, false, fmt.Errorf("%w: %s", ErrNoSuchNode, name)
}
@@ -1096,6 +1102,9 @@ func (i *Inventory) SentTo(ctx context.Context, name string) (SentDeclaration, b
if seq != nil {
s.Sequence = *seq
}
if at != nil {
s.At = at.UTC()
}
return s, true, nil
}
+173 -44
View File
@@ -2,13 +2,10 @@ package inventory
import (
"context"
"errors"
"fmt"
"sort"
"strings"
"time"
"github.com/jackc/pgx/v5"
)
// Where a walk's time went (novox/hq ADR 0282 decision 6, issue 382): from its merge to every machine of its
@@ -59,11 +56,14 @@ type WalkPhases struct {
TotalMS int64 `json:"total_ms,omitempty"`
Total string `json:"total,omitempty"`
// UnknownMS is the time within the walk no phase could be measured over.
UnknownMS int64 `json:"unknown_ms,omitempty"`
Phases []WalkPhase `json:"phases"`
Silent []string `json:"silent,omitempty"`
Said string `json:"said"`
Merges []MergeStart `json:"merges,omitempty"`
UnknownMS int64 `json:"unknown_ms,omitempty"`
Phases []WalkPhase `json:"phases"`
Silent []string `json:"silent,omitempty"`
// Rest is each machine of the rest's own report of the send that carried the walk's build there (novox/hq ADR
// 0282 decision 6, issue 582): when it was sent, when it said, and what.
Rest []RestReport `json:"rest,omitempty"`
Said string `json:"said"`
Merges []MergeStart `json:"merges,omitempty"`
}
// MergeStart is one merge the walk answers and when it was made: where that merge's delivery time starts.
@@ -92,8 +92,34 @@ type AppliedReport struct {
Outcome string
}
// AppliedLookup answers a machine's first report after a send to it; false when it has not reported.
type AppliedLookup func(node string, sent time.Time) (AppliedReport, bool)
// AppliedLookup answers a machine's report of the send the walk made to it: send is what the walk kept of it, and
// walkSent the walk's own record of its send to the rest; false when it has not reported.
type AppliedLookup func(node string, send SentDeclaration, walkSent time.Time) (AppliedReport, bool)
// RestReport is one machine of the rest and its own report of a send of the walk (novox/hq issue 582).
type RestReport struct {
Node string `json:"node"`
Modules []string `json:"modules"`
// Sent is when that machine was sent, by its own send time where the walk kept it; else the walk's record of
// its send to the rest, and SentSaid says so.
Sent time.Time `json:"sent"`
SentSaid string `json:"sent_said,omitempty"`
Sequence int64 `json:"sequence,omitempty"`
Reported *time.Time `json:"reported,omitempty"`
Took string `json:"took,omitempty"`
Outcome string `json:"outcome,omitempty"`
// State is applied, failed, waiting (not yet heard, within ApplySilentAfter) or silent (not heard within it:
// listed, never moving the walk's end).
State string `json:"state"`
}
// The states of a machine of the rest.
const (
RestApplied = "applied"
RestFailed = "failed"
RestWaiting = "waiting"
RestSilent = "silent"
)
// point is one moment of the walk, ending the phase named.
type point struct {
@@ -307,8 +333,8 @@ func (p Plan) points(now time.Time, applied AppliedLookup, w *WalkPhases) []poin
}
} else {
rest = latest(rest, s.SentAt)
for node := range s.Rest {
lastSends = append(lastSends, restSend{module: m, node: node, at: *s.SentAt})
for node, what := range s.Rest {
lastSends = append(lastSends, restSend{module: m, node: node, walkSent: *s.SentAt, what: what})
}
}
}
@@ -347,22 +373,73 @@ func (p Plan) points(now time.Time, applied AppliedLookup, w *WalkPhases) []poin
if applied == nil {
return append(out, point{phase: PhaseApply, tier: -1, said: "the rest's reports were not read"})
}
var end *time.Time
var waiting, failed []string
// Each machine of the rest, by its own report of the send that carried the build there (novox/hq issue 582):
// never a report of a send made after it, which another walk's send may be.
reports := map[string]*RestReport{}
var order []string
for _, s := range lastSends {
r, ok := applied(s.node, s.at)
switch {
case !ok && now.Sub(s.at) > ApplySilentAfter, ok && r.At.Sub(s.at) > ApplySilentAfter:
w.Silent = appendOnce(w.Silent, s.node)
case !ok:
waiting = appendOnce(waiting, s.node)
case r.Outcome != OutcomeApplied:
failed = appendOnce(failed, s.node+" ("+r.Outcome+")")
default:
at := r.At
end = latest(end, &at)
at, said := s.walkSent, ""
if !s.what.At.IsZero() {
at = s.what.At
} else {
said = "the walk's record of its send to the rest: this machine's own send time was not kept"
}
key := s.node + "\x00" + s.what.Digest + "\x00" + at.Format(time.RFC3339Nano)
if r := reports[key]; r != nil {
r.Modules = appendOnce(r.Modules, s.module)
continue
}
rr := &RestReport{Node: s.node, Modules: []string{s.module}, Sent: at, SentSaid: said, Sequence: s.what.Sequence}
if r, ok := applied(s.node, s.what, s.walkSent); ok {
got := r.At
rr.Reported, rr.Outcome, rr.Took = &got, r.Outcome, words(r.At.Sub(at))
}
reports[key] = rr
order = append(order, key)
}
// A machine sent twice by the walk (one tier's rest, then the next's) that said nothing of the first send has
// said it of the second, which carries the first's builds too: both are this walk's.
for _, k := range order {
rr := reports[k]
if rr.Reported != nil {
continue
}
for _, l := range order {
later := reports[l]
if later.Node == rr.Node && later.Sent.After(rr.Sent) && later.Reported != nil &&
(rr.Reported == nil || later.Reported.Before(*rr.Reported)) {
got := *later.Reported
rr.Reported, rr.Outcome, rr.Took = &got, later.Outcome, words(got.Sub(rr.Sent))
}
}
}
var end *time.Time
var waiting, failed []string
for _, k := range order {
rr := reports[k]
switch {
case rr.Reported == nil && now.Sub(rr.Sent) > ApplySilentAfter, rr.Reported != nil && rr.Reported.Sub(rr.Sent) > ApplySilentAfter:
rr.State = RestSilent
w.Silent = appendOnce(w.Silent, rr.Node)
case rr.Reported == nil:
rr.State = RestWaiting
waiting = appendOnce(waiting, rr.Node)
case rr.Outcome != OutcomeApplied:
rr.State = RestFailed
failed = appendOnce(failed, rr.Node+" ("+rr.Outcome+")")
default:
rr.State = RestApplied
end = latest(end, rr.Reported)
}
sort.Strings(rr.Modules)
w.Rest = append(w.Rest, *rr)
}
sort.SliceStable(w.Rest, func(a, b int) bool {
if w.Rest[a].Node != w.Rest[b].Node {
return w.Rest[a].Node < w.Rest[b].Node
}
return w.Rest[a].Sent.Before(w.Rest[b].Sent)
})
switch {
case len(failed) > 0:
return append(out, point{phase: PhaseApply, tier: -1, said: "not applied on " + strings.Join(failed, ", ")})
@@ -393,7 +470,8 @@ func (p Plan) points(now time.Time, applied AppliedLookup, w *WalkPhases) []poin
type restSend struct {
module, node string
at time.Time
walkSent time.Time
what SentDeclaration
}
func askedAnyOf(p Plan, tier []string) bool {
@@ -451,28 +529,79 @@ func words(d time.Duration) string {
return d.Round(100 * time.Millisecond).String()
}
// FirstAppliedAfter is a machine's first report of a send made at or after a walk's send to it, within
// ApplySilentAfter of it — the controller's `apply` durations, which measure every send to its first report (to-be
// 45 Phase 0). A declaration sent later carries the build too, so its report counts; one sent past the bound is a
// machine that slept, listed as silent and never moving the walk's end (ADR 0282 decision 1).
func (i *Inventory) FirstAppliedAfter(ctx context.Context, node string, sent time.Time) (AppliedReport, bool, error) {
var started time.Time
var ms int64
var outcome string
err := i.store.Pool().QueryRow(ctx,
`select started, took_ms, detail from duration
// FirstAppliedAfter is a machine's report of the send a walk made to it: the controller's `apply` durations, which
// measure every send to its first report (to-be 45 Phase 0), each row starting at the machine's own send time and
// naming the declaration it was about. send is what the walk kept of that send; walkSent its own record of the
// send to the rest, made after every machine of the rest was sent (novox/hq issue 582). FirstReportOf chooses.
func (i *Inventory) FirstAppliedAfter(ctx context.Context, node string, send SentDeclaration, walkSent time.Time) (AppliedReport, bool, error) {
from, to := applyRowsBetween(send, walkSent)
rows, err := i.store.Pool().Query(ctx,
`select ref, started, took_ms, detail from duration
where kind = $1 and subject = $2 and started >= $3 and started < $4
order by started limit 1`,
DurationApply, node, sent.Add(-appliedSlack), sent.Add(ApplySilentAfter)).Scan(&started, &ms, &outcome)
if errors.Is(err, pgx.ErrNoRows) {
return AppliedReport{}, false, nil
}
order by started`,
DurationApply, node, from, to)
if err != nil {
return AppliedReport{}, false, err
}
return AppliedReport{At: started.Add(time.Duration(ms) * time.Millisecond).UTC(), Outcome: outcome}, true, nil
defer rows.Close()
var ds []Duration
for rows.Next() {
d := Duration{Kind: DurationApply, Subject: node, Node: node}
var ms int64
if err := rows.Scan(&d.Ref, &d.Started, &ms, &d.Detail); err != nil {
return AppliedReport{}, false, err
}
d.Took = time.Duration(ms) * time.Millisecond
ds = append(ds, d)
}
if err := rows.Err(); err != nil {
return AppliedReport{}, false, err
}
r, ok := FirstReportOf(ds, send, walkSent)
return r, ok, nil
}
// appliedSlack is how much earlier than a walk's record of its send the machine's own record of it may be:
// the send is recorded on the machine first, then on the walk.
// applyRowsBetween is the span of a machine's `apply` durations that may hold its report of a walk's send.
func applyRowsBetween(send SentDeclaration, walkSent time.Time) (time.Time, time.Time) {
switch {
case !send.At.IsZero():
return send.At.Add(-time.Millisecond), send.At.Add(time.Millisecond)
case send.Digest != "":
return walkSent.Add(-restSendBound), walkSent.Add(ApplySilentAfter)
}
return walkSent.Add(-appliedSlack), walkSent.Add(ApplySilentAfter)
}
// FirstReportOf picks, from a machine's `apply` durations ordered by their start, its report of the send a walk made
// to it (novox/hq issue 582):
// - with the machine's own send time kept (every walk since issue 582): the duration of that very send, and of the
// declaration sent where the walk kept its digest. A report of a later send is never this walk's: a send made
// later, by another walk, would end this walk at that walk's send.
// - with only the declaration's digest kept (a walk kept before): the first duration about that declaration
// within restSendBound before the walk's record of its send, the machine sent first in the rest included.
// - with neither (a walk kept before ADR 0282's rest): the first duration started no earlier than appliedSlack
// before the walk's record of its send, as it was read before.
//
// A send past ApplySilentAfter is never read: its machine slept, and is listed silent (ADR 0282 decision 1).
func FirstReportOf(ds []Duration, send SentDeclaration, walkSent time.Time) (AppliedReport, bool) {
from, to := applyRowsBetween(send, walkSent)
for _, d := range ds {
if d.Started.Before(from) || !d.Started.Before(to) {
continue
}
if send.Digest != "" && !strings.HasPrefix(d.Ref, send.Digest+"@") {
continue
}
return AppliedReport{At: d.Started.Add(d.Took).UTC(), Outcome: d.Detail}, true
}
return AppliedReport{}, false
}
// appliedSlack is how much earlier than a walk's record of its send the machine's own record of it may be, for a
// walk that kept neither the machine's send time nor what it sent.
const appliedSlack = 2 * time.Second
// restSendBound is how long before a walk's record of its send to the rest a machine of the rest may have been
// sent, for a walk that kept what it sent and not when: the send to the rest goes machine by machine, and took
// 19 to 28 seconds on 2026-10-12 (novox/hq issue 582).
const restSendBound = 5 * time.Minute
+188
View File
@@ -0,0 +1,188 @@
package inventory
import (
"strings"
"testing"
"time"
)
// novox/hq issue 582: the walk's own record of its send to the rest is made after every machine of the rest was
// sent, and that send took 19 to 28 s on 2026-10-12. The machine sent first reported 5 s after its own send, long
// before the walk's record; read from the walk's record, its report was missed and the walk ended at another walk's
// send to it, minutes later. A walk of one tier and three machines of the rest, the send to the rest taking 20 s.
type sendRest582 struct {
judged, walkSent time.Time
walk Plan
rows map[string][]Duration
}
func walk582(t *testing.T) sendRest582 {
t.Helper()
judged := time.Date(2026, 10, 12, 17, 45, 35, 137000000, time.UTC)
ownSend := map[string]time.Duration{"g14": 500 * time.Millisecond, "novox": 10 * time.Second, "shanks": 20 * time.Second}
walkSent := judged.Add(20200 * time.Millisecond)
merged := judged.Add(-10 * time.Minute)
built, first := judged.Add(-3*time.Minute), judged.Add(-2*time.Minute)
rest := map[string]SentDeclaration{}
rows := map[string][]Duration{}
took := map[string]time.Duration{"g14": 5 * time.Second, "novox": 13 * time.Second, "shanks": 15 * time.Second}
for i, n := range []string{"g14", "novox", "shanks"} {
at := judged.Add(ownSend[n])
digest := "walk-" + n
rest[n] = SentDeclaration{Digest: digest, Sequence: int64(647 + i), At: at}
rows[n] = append(rows[n],
// This walk's send, and its report.
Duration{Kind: DurationApply, Subject: n, Node: n, Ref: digest + "@" + at.Format("2006-01-02T15:04:05.000000"),
Started: at, Took: took[n], Detail: OutcomeApplied},
// The next walk's send to the same machine, nine minutes later, and its report.
Duration{Kind: DurationApply, Subject: n, Node: n, Ref: "next-" + n + "@x",
Started: judged.Add(9*time.Minute + ownSend[n]), Took: 4 * time.Second, Detail: OutcomeApplied})
}
p := Plan{ID: "plan-582", State: PlanDone, Tier: 1, Tiers: [][]string{{"build-agent"}}, Created: merged,
Delivery: &PlanDelivery{Merges: []PlanMerge{{Repository: "novox/mesh-controller", Commit: "c", Merged: merged}}},
Times: &PlanTimes{Cut: &merged, Class: ClassCore},
Modules: map[string]*PlanModule{"build-agent": {State: "built", AskedAt: &merged, BuiltAt: &built,
FirstAt: &first, SentAt: &walkSent, Gate: &PlanGate{JudgedAt: &judged}, Rest: rest}}}
return sendRest582{judged: judged, walkSent: walkSent, walk: p, rows: rows}
}
// lookup answers from the rows as the controller's durations would.
func (w sendRest582) lookup(node string, send SentDeclaration, walkSent time.Time) (AppliedReport, bool) {
return FirstReportOf(w.rows[node], send, walkSent)
}
// The apply phase ends at the last machine's own report of this walk's send — shanks, sent at the end of the 20 s
// and reporting 15 s later — and never at the next walk's send; every machine's report is shown.
func TestTheApplyPhaseEndsAtEachMachinesOwnReport(t *testing.T) {
c := walk582(t)
w := c.walk.WalkPhases(c.judged.Add(time.Hour), c.lookup)
sumsUp(t, w)
want := c.judged.Add(35 * time.Second) // shanks: sent 20 s after the judgement, reported 15 s later
if !w.End.Equal(want) {
t.Fatalf("the walk ends at %s, want %s: its rest %+v", w.End, want, w.Rest)
}
if got := phase(w, PhaseApply, -1); got.State != PhaseMeasured || got.TookMS != want.Sub(c.walkSent).Milliseconds() {
t.Fatalf("apply: %+v", got)
}
if len(w.Rest) != 3 || len(w.Silent) != 0 {
t.Fatalf("every machine of the rest is shown with its report: %+v, silent %v", w.Rest, w.Silent)
}
g14 := w.Rest[0]
if g14.Node != "g14" || g14.State != RestApplied || g14.Reported == nil ||
!g14.Reported.Equal(c.judged.Add(5500*time.Millisecond)) || g14.Took != "5s" || g14.Sequence != 647 ||
!g14.Sent.Equal(c.judged.Add(500*time.Millisecond)) || g14.SentSaid != "" {
t.Fatalf("g14, sent first and reporting 5 s after its own send: %+v", g14)
}
}
// A walk kept before its machines' own send times were: matched by the declaration it sent, within restSendBound
// of its record of the send, so the machine sent first is still found; a walk that kept neither is read as before.
func TestAnOlderWalkIsReadByWhatItSent(t *testing.T) {
c := walk582(t)
for n, s := range c.walk.Modules["build-agent"].Rest {
s.At = time.Time{}
c.walk.Modules["build-agent"].Rest[n] = s
}
w := c.walk.WalkPhases(c.judged.Add(time.Hour), c.lookup)
sumsUp(t, w)
if !w.End.Equal(c.judged.Add(35 * time.Second)) {
t.Fatalf("by the declaration sent: end %s, rest %+v", w.End, w.Rest)
}
if !strings.Contains(w.Rest[0].SentSaid, "walk's record") || !w.Rest[0].Sent.Equal(c.walkSent) {
t.Fatalf("a send time not kept is said: %+v", w.Rest[0])
}
// Nothing of the send kept: the first report no earlier than appliedSlack before the walk's record, as before.
if r, ok := FirstReportOf(c.rows["shanks"], SentDeclaration{}, c.walkSent); !ok || !r.At.Equal(c.judged.Add(35*time.Second)) {
t.Fatalf("a walk that kept nothing of its send: %+v %v", r, ok)
}
}
// A machine that said nothing of this walk's send is waiting, then silent past ApplySilentAfter: the report of a
// later send to it is never taken as this walk's, and never moves the walk's end.
func TestALaterSendsReportIsNeverThisWalks(t *testing.T) {
c := walk582(t)
c.rows["g14"] = c.rows["g14"][1:] // g14 slept through this walk's send; the next walk's send it reported
w := c.walk.WalkPhases(c.judged.Add(5*time.Minute), c.lookup)
if w.End != nil || !strings.Contains(phase(w, PhaseApply, -1).Said, "waiting for g14") {
t.Fatalf("g14 not heard from yet: end %v, apply %+v", w.End, phase(w, PhaseApply, -1))
}
w = c.walk.WalkPhases(c.judged.Add(time.Hour), c.lookup)
sumsUp(t, w)
if len(w.Silent) != 1 || w.Silent[0] != "g14" || !w.End.Equal(c.judged.Add(35*time.Second)) {
t.Fatalf("g14 silent, the end at shanks' report: silent %v, end %s", w.Silent, w.End)
}
if w.Rest[0].State != RestSilent || w.Rest[0].Reported != nil {
t.Fatalf("g14's later report taken as this walk's: %+v", w.Rest[0])
}
}
// A machine sent by two tiers of one walk that said nothing of the first send has said it of the second, which
// carries the first tier's build too.
func TestAMachineSentTwiceByOneWalkIsHeardByItsLaterSend(t *testing.T) {
c := walk582(t)
second := c.judged.Add(4 * time.Minute)
secondSent := second.Add(time.Second)
c.walk.Tiers = append(c.walk.Tiers, []string{"route-proxy"})
c.walk.Tier = 2
asked, built := c.judged.Add(time.Minute), c.judged.Add(2*time.Minute)
first := built.Add(time.Second)
c.walk.Modules["route-proxy"] = &PlanModule{State: "built", AskedAt: &asked, BuiltAt: &built, FirstAt: &first,
SentAt: &secondSent, Gate: &PlanGate{JudgedAt: &second},
Rest: map[string]SentDeclaration{"novox": {Digest: "walk-novox-2", Sequence: 1554, At: second.Add(500 * time.Millisecond)}}}
c.rows["novox"] = []Duration{{Kind: DurationApply, Subject: "novox", Ref: "walk-novox-2@x",
Started: second.Add(500 * time.Millisecond), Took: 14 * time.Second, Detail: OutcomeApplied}}
w := c.walk.WalkPhases(c.judged.Add(time.Hour), c.lookup)
sumsUp(t, w)
want := second.Add(14500 * time.Millisecond)
if !w.End.Equal(want) || len(w.Silent) != 0 {
t.Fatalf("end %s (want %s), silent %v, rest %+v", w.End, want, w.Silent, w.Rest)
}
for _, r := range w.Rest {
if r.Node == "novox" && (r.State != RestApplied || !r.Reported.Equal(want)) {
t.Fatalf("novox, heard by its later send of this walk: %+v", r)
}
}
}
// The store: a machine's own send time is read with what it was sent, and its `apply` duration is found by it —
// the send and the duration start at one moment, the node's sent_at.
func TestAWalksRestIsReadFromEachMachinesOwnSend(t *testing.T) {
inv := ForTest(t)
ctx := t.Context()
record, err := inv.AddNode(ctx, "g14")
if err != nil {
t.Fatal(err)
}
if _, err := inv.NextSequence(ctx, record.ID); err != nil {
t.Fatal(err)
}
if err := inv.RecordSent(ctx, record.ID, "d-582", map[string]string{"build-agent": "c1"}); err != nil {
t.Fatal(err)
}
sent, found, err := inv.SentTo(ctx, "g14")
if err != nil || !found || sent.At.IsZero() || sent.Digest != "d-582" {
t.Fatalf("the machine's own send time: %+v %v %v", sent, found, err)
}
if _, ok, err := inv.FirstAppliedAfter(ctx, "g14", sent, sent.At.Add(20*time.Second)); err != nil || ok {
t.Fatalf("no report yet: %v %v", ok, err)
}
if err := inv.RecordApplyDuration(ctx, "g14", "d-582", OutcomeApplied); err != nil {
t.Fatal(err)
}
// The walk's record of its send, 20 s after the machine's own: still found, by the machine's own send.
r, ok, err := inv.FirstAppliedAfter(ctx, "g14", sent, sent.At.Add(20*time.Second))
if err != nil || !ok || r.Outcome != OutcomeApplied || r.At.Before(sent.At) {
t.Fatalf("the machine's own report of its send: %+v %v %v", r, ok, err)
}
// A later send, reported: never this walk's.
if err := inv.RecordSent(ctx, record.ID, "d-next", map[string]string{"build-agent": "c2"}); err != nil {
t.Fatal(err)
}
if err := inv.RecordApplyDuration(ctx, "g14", "d-next", OutcomeApplied); err != nil {
t.Fatal(err)
}
again, _, _ := inv.FirstAppliedAfter(ctx, "g14", sent, sent.At.Add(20*time.Second))
if !again.At.Equal(r.At) {
t.Fatalf("a later send's report read as this walk's: %+v, was %+v", again, r)
}
}
+8 -8
View File
@@ -116,7 +116,7 @@ func TestAWalkOfTwoTiersEndsAtItsRestsLastReport(t *testing.T) {
}}
reports := map[string]AppliedReport{"ace": {At: *at("05:12"), Outcome: OutcomeApplied},
"shanks": {At: *at("08:15.25"), Outcome: OutcomeApplied}}
w := p.WalkPhases(*at("30:00"), func(node string, _ time.Time) (AppliedReport, bool) {
w := p.WalkPhases(*at("30:00"), func(node string, _ SentDeclaration, _ time.Time) (AppliedReport, bool) {
r, ok := reports[node]
return r, ok
})
@@ -135,7 +135,7 @@ func TestAWalkOfTwoTiersEndsAtItsRestsLastReport(t *testing.T) {
}
// A report not yet in: the walk has no end, and its apply is unknown — never zero.
delete(reports, "shanks")
w = p.WalkPhases(*at("10:00"), func(node string, _ time.Time) (AppliedReport, bool) {
w = p.WalkPhases(*at("10:00"), func(node string, _ SentDeclaration, _ time.Time) (AppliedReport, bool) {
r, ok := reports[node]
return r, ok
})
@@ -146,7 +146,7 @@ func TestAWalkOfTwoTiersEndsAtItsRestsLastReport(t *testing.T) {
t.Errorf("apply while shanks has not reported: %+v", got)
}
// Silent past the bound: listed, not waited for.
w = p.WalkPhases(at("08:00.5").Add(ApplySilentAfter+time.Second), func(node string, _ time.Time) (AppliedReport, bool) {
w = p.WalkPhases(at("08:00.5").Add(ApplySilentAfter+time.Second), func(node string, _ SentDeclaration, _ time.Time) (AppliedReport, bool) {
r, ok := reports[node]
return r, ok
})
@@ -156,7 +156,7 @@ func TestAWalkOfTwoTiersEndsAtItsRestsLastReport(t *testing.T) {
}
// A failed report: the walk has no end, said.
reports["shanks"] = AppliedReport{At: *at("08:10"), Outcome: OutcomeFailed}
w = p.WalkPhases(*at("30:00"), func(node string, _ time.Time) (AppliedReport, bool) {
w = p.WalkPhases(*at("30:00"), func(node string, _ SentDeclaration, _ time.Time) (AppliedReport, bool) {
r, ok := reports[node]
return r, ok
})
@@ -247,7 +247,7 @@ func TestAWalksMomentsAreKeptAndItsRestsReportRead(t *testing.T) {
t.Fatal("a merge's time was not kept")
}
sent := time.Now().UTC().Add(-time.Minute).Truncate(time.Millisecond)
if _, ok, err := inv.FirstAppliedAfter(ctx, "ace", sent); err != nil || ok {
if _, ok, err := inv.FirstAppliedAfter(ctx, "ace", SentDeclaration{}, sent); err != nil || ok {
t.Fatalf("no report yet: %v %v", ok, err)
}
for _, d := range []Duration{
@@ -258,7 +258,7 @@ func TestAWalksMomentsAreKeptAndItsRestsReportRead(t *testing.T) {
t.Fatal(err)
}
}
r, ok, err := inv.FirstAppliedAfter(ctx, "ace", sent)
r, ok, err := inv.FirstAppliedAfter(ctx, "ace", SentDeclaration{}, sent)
if err != nil || !ok || r.Outcome != OutcomeApplied || !r.At.Equal(sent.Add(11*time.Second)) {
t.Fatalf("the report after the send: %+v %v %v", r, ok, err)
}
@@ -281,7 +281,7 @@ func TestALateReportIsSilentNotTheEnd(t *testing.T) {
p := walkOf(t, recordedWalk)
p.Modules["mesh-delivery"].SentAt = &sent
p.Modules["mesh-delivery"].Rest = map[string]SentDeclaration{"laptop": {Digest: "d"}, "server": {Digest: "e"}}
w := p.WalkPhases(sent.Add(3*time.Hour), func(node string, _ time.Time) (AppliedReport, bool) {
w := p.WalkPhases(sent.Add(3*time.Hour), func(node string, _ SentDeclaration, _ time.Time) (AppliedReport, bool) {
if node == "server" {
return AppliedReport{At: sent.Add(20 * time.Second), Outcome: OutcomeApplied}, true
}
@@ -296,7 +296,7 @@ func TestALateReportIsSilentNotTheEnd(t *testing.T) {
Started: sent.Add(time.Hour), Took: time.Second, Detail: OutcomeApplied}); err != nil {
t.Fatal(err)
}
if _, ok, err := inv.FirstAppliedAfter(ctx, "laptop", sent); err != nil || ok {
if _, ok, err := inv.FirstAppliedAfter(ctx, "laptop", SentDeclaration{}, sent); err != nil || ok {
t.Fatalf("a send past the bound read as the walk's: %v %v", ok, err)
}
}
+11
View File
@@ -79,6 +79,8 @@ func (b *natsBuilds) Ask(ctx context.Context, request BuildRequest) error {
if _, err := b.js.Context().Publish(BuildWorkOf(b.role()), body, nats.Context(publish)); err != nil {
return fmt.Errorf("cannot submit a build: %w", err)
}
// The acting controller puts it in order (novox/hq issue 558); this only says an ask joined.
signalAsked()
return nil
}
@@ -113,6 +115,8 @@ func (b *natsBuilds) Submit(ctx context.Context, request BuildRequest,
if _, err := b.js.Context().Publish(BuildWorkOf(b.role()), body, nats.Context(publish)); err != nil {
return BuildResult{}, fmt.Errorf("%w: cannot submit a build: %w", ErrNotHandedOver, err)
}
// The acting controller puts it in order (novox/hq issue 558); this only says an ask joined.
signalAsked()
waiting, cancelWait := context.WithTimeout(ctx, wait)
defer cancelWait()
@@ -167,6 +171,10 @@ type MachineOptions struct {
// Paused is asked before every fetch: while it says so, nothing new is taken, and a build
// already running finishes. Nil is never paused.
Paused func() bool
// Taking is called once, when this machine is bound to its worker and pulls work: from then on it
// can take a build, which is what its container's health check reads (novox/hq issue 544). Nil is
// told nothing.
Taking func()
}
// MachineOverNATS takes build work from the current build role.
@@ -232,6 +240,9 @@ func (m *natsMachine) Take(ctx context.Context, do func(context.Context, Build))
"the mesh's to answer, not this machine's", m.seat, err)
}
m.sub = sub
if m.opts.Taking != nil {
m.opts.Taking()
}
for {
if ctx.Err() != nil {
+33
View File
@@ -368,3 +368,36 @@ func TestNatsAMachineOnTheRetiredBuildRoleTakesThatRolesAsks(t *testing.T) {
t.Errorf("the retired role's holder did not answer: %+v", result)
}
}
// A holder says once that it takes work, and only once it is bound to its worker (novox/hq issue 544):
// its container's health check reads that, so a holder that could not bind is never said ready.
func TestNatsAHolderSaysItTakesWorkOnceItIsBound(t *testing.T) {
js := aBusWithTheBuildRole(t)
ctx, stop := context.WithCancel(context.Background())
defer stop()
said := make(chan struct{}, 4)
machine := MachineOverNATSWith(js, "anchor", TheBuildMachine, MachineOptions{Taking: func() { said <- struct{}{} }})
defer machine.Close()
go func() { _ = machine.Take(ctx, func(context.Context, Build) {}) }()
select {
case <-said:
case <-time.After(5 * time.Second):
t.Fatal("a holder bound to its worker never said it takes work")
}
time.Sleep(300 * time.Millisecond)
if len(said) != 0 {
t.Fatalf("a holder said it takes work %d more time(s)", len(said))
}
// A role the mesh raised no worker for cannot be bound, and its holder says nothing.
bounded, end := context.WithTimeout(ctx, 10*time.Second)
defer end()
other := MachineOverNATSWith(js, "anchor", "mesh-no-such-seat", MachineOptions{Taking: func() { said <- struct{}{} }})
defer other.Close()
if err := other.Take(bounded, func(context.Context, Build) {}); err == nil {
t.Fatal("a machine with no worker took work")
}
if len(said) != 0 {
t.Fatal("a machine that could not bind said it takes work")
}
}
+26 -10
View File
@@ -62,7 +62,11 @@ type QueuedAsk struct {
Path string `json:"path,omitempty"`
Ref string `json:"ref,omitempty"`
AskedAt time.Time `json:"asked-at,omitempty"`
State string `json:"state"`
// Stored is when the stream first took it, a move into order kept (novox/hq issue 558).
Stored time.Time `json:"stored,omitempty"`
State string `json:"state"`
// movedFrom is the MovedFromHeader it carries: the sequence of the message a move copied it from.
movedFrom string
// On and Started are the holder building an in-flight ask and when it started, from its started
// event. Was is the holder that started an in-flight ask and is no longer building it — handed
// back, to be handed out again — with Started its start there.
@@ -236,15 +240,7 @@ func ReadQueue(ctx context.Context, js *broker.JetStream, seat string) (Queue, e
if err != nil {
return q, fmt.Errorf("reading the %s work queue: %w", seat, err)
}
ask := QueuedAsk{Seq: msg.Sequence}
if json.Unmarshal(msg.Data, &ask.Request) == nil {
r := ask.Request
ask.ID, ask.Repository, ask.Path, ask.Ref = r.ID, r.Repository, r.Path, r.Ref
if at, ok := BuildAskedAt(r.ID); ok {
ask.AskedAt = at
}
}
asks = append(asks, ask)
asks = append(asks, askOf(msg))
seq = msg.Sequence + 1
}
@@ -269,6 +265,26 @@ func ReadQueue(ctx context.Context, js *broker.JetStream, seat string) (Queue, e
return q, nil
}
// askOf is one ask as the stream holds it: what its request says, and when the stream took it.
func askOf(msg *jetstream.RawStreamMsg) QueuedAsk {
ask := QueuedAsk{Seq: msg.Sequence, Stored: msg.Time}
// Moved into order, it keeps when the stream first took it (novox/hq issue 558).
if at, err := time.Parse(time.RFC3339Nano, msg.Header.Get(QueuedAtHeader)); err == nil {
ask.Stored = at
}
if json.Unmarshal(msg.Data, &ask.Request) == nil {
r := ask.Request
ask.ID, ask.Repository, ask.Path, ask.Ref = r.ID, r.Repository, r.Path, r.Ref
if at, ok := BuildAskedAt(r.ID); ok {
ask.AskedAt = at
}
} else {
// Nothing read is nothing kept: a half-read request would pass for a delivery (novox/hq issue 558).
ask.Request = BuildRequest{}
}
return ask
}
// WithoutPassword is a repository as a listing may say it: a URL's password, should one ever be written
// into it, left out (novox/hq issue 455). Only for what is shown: the ask itself keeps what it was given,
// which a cancel's outcome and a rebuild's match are read from.
+426
View File
@@ -0,0 +1,426 @@
package link
import (
"context"
"errors"
"fmt"
"sort"
"strconv"
"strings"
"sync"
"time"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"github.com/novox/mesh-controller/internal/broker"
)
// The order the build queue hands asks out in (novox/hq issue 558).
//
// **A build a walk waits for goes before a pull request's check.** The queue was first come, first served,
// and a merged change's delivery build waited minutes behind checks while its walk, and every walk after
// it, waited on the build. So:
//
// - a delivery is handed out before any check;
// - a check that has waited CheckWaitsAtMost goes before every delivery asked after it, so no check
// waits for ever behind a stream of deliveries;
// - within a kind, first come, first served, by when it was asked;
// - an ask whose kind cannot be told counts as a check: a delivery taken late breaks a budget, a check
// taken late breaks nothing.
//
// **Kept in the stream, never beside it.** The queue is the seat's work queue, which the server hands out
// in stream order to whichever holder pulls; nothing else holds the order, so a restarted controller
// finds it as it was left. An ask that must go behind another is moved to the tail — deleted from where
// it was and published again, its body unchanged, under the same id — while the worker is paused, so no
// holder takes it in the moment it moves. Only waiting asks move: what a holder took is never touched.
// The kinds of ask, as the queue says them.
const (
KindDelivery = "delivery"
KindCheck = "check"
)
// CheckWaitsAtMost is how long a check waits before it goes ahead of the deliveries asked after it.
var CheckWaitsAtMost = 10 * time.Minute
// KindOf is an ask's kind: a check for a pull request's check — a recheck the merge tool asks included —
// a dry run, or an ask nothing could be read from; a delivery for any other build.
func KindOf(r BuildRequest) string {
if r.ID == "" || r.Check != nil || r.DryRun {
return KindCheck
}
return KindDelivery
}
// askedOrStored is when an ask was asked, as its id says, or else when the queue took it.
func (a QueuedAsk) askedOrStored() time.Time {
if !a.AskedAt.IsZero() {
return a.AskedAt
}
return a.Stored
}
// QueueOrder is the order the waiting asks are to be handed out in, now: the deliveries and every check
// that has waited CheckWaitsAtMost, by when each was asked, then the other checks, by when each was asked.
func QueueOrder(waiting []QueuedAsk, now time.Time) []QueuedAsk {
var first, then []QueuedAsk
for _, a := range waiting {
if KindOf(a.Request) == KindCheck && now.Sub(a.askedOrStored()) < CheckWaitsAtMost {
then = append(then, a)
continue
}
first = append(first, a)
}
byAsking := func(s []QueuedAsk) {
sort.SliceStable(s, func(i, j int) bool {
if ti, tj := s[i].askedOrStored(), s[j].askedOrStored(); !ti.Equal(tj) {
return ti.Before(tj)
}
return s[i].Seq < s[j].Seq
})
}
byAsking(first)
byAsking(then)
return append(first, then...)
}
// toMove is the asks to move to the tail, in this order, for the queue to stand as order says: everything
// after the longest start of order that the queue already holds in that order.
func toMove(queue, order []QueuedAsk) []QueuedAsk {
at := make(map[uint64]int, len(queue))
for i, a := range queue {
at[a.Seq] = i
}
kept, last := 0, -1
for kept < len(order) {
i, ok := at[order[kept].Seq]
if !ok || i < last {
break
}
last = i
kept++
}
return order[kept:]
}
// orderPasses keeps one pass at a time. Only the acting controller's loop runs one (KeepingInOrder), so
// there is one; the lock holds a test's two at once to that too.
var orderPasses sync.Mutex
// pausedFor bounds a pause the pass could not end itself: a controller that stops mid-pass leaves the
// worker paused this long at most. moveMargin is the part of it no move starts in: a pass stops moving
// with at least this left, and the next pass moves the rest, so no move runs past the pause.
const (
pausedFor = 20 * time.Second
moveMargin = 5 * time.Second
publishWaits = 3 * time.Second
)
// QueuedAtHeader carries, on a moved ask, when the stream first took it: a move keeps an ask's place in
// time, its id unread or not.
const QueuedAtHeader = "Mesh-Queued-At"
// MovedFromHeader names, on a move's copy, the sequence of the message it is a copy of: with QueuedAtHeader,
// what proves a second message of one id is a move's copy and not an ask delivered again (ADR 0293), whose
// headers are its original's — and so never name that original's own sequence.
const MovedFromHeader = "Mesh-Moved-From"
// Pass is what one pass did: the asks it moved, and each second copy it removed, in words.
type Pass struct {
Moved []string
Removed []string
}
// asked is the signal an ask sends the acting controller's pass loop, in this process. An ask asked in
// another process — a verb run as its own command — is seen by the loop as the queue's last sequence moving.
var asked = make(chan struct{}, 1)
// signalAsked tells the pass loop an ask joined the queue. Never blocks, never runs a pass.
func signalAsked() {
select {
case asked <- struct{}{}:
default:
}
}
// PutInOrder moves the seat's waiting asks into the order QueueOrder says, and says each it moved. A queue
// already in order is read and left alone; the worker is paused only while asks move. Run by the acting
// controller alone (KeepingInOrder): two passes in two processes would each resume what the other paused.
func PutInOrder(ctx context.Context, js *broker.JetStream, seat string) (Pass, error) {
var pass Pass
orderPasses.Lock()
defer orderPasses.Unlock()
api, err := jetstream.New(js.Conn())
if err != nil {
return pass, err
}
worker, _ := broker.HolderConsumerFor("", "", broker.DeclaredSeat{Name: seat, Accepts: []string{"build"}})
waiting, extra, delivered, err := readWaiting(ctx, api, worker, seat)
if err != nil || (len(extra) == 0 && len(toMove(waiting, QueueOrder(waiting, time.Now()))) == 0) {
return pass, err
}
// Paused, then read again: what the worker handed out up to the pause is not waiting any more.
pauseEnds := time.Now().Add(pausedFor)
if delivered >= 0 {
if _, err := api.PauseConsumer(ctx, worker.Stream, worker.Name, pauseEnds); err != nil {
return pass, fmt.Errorf("the %s worker could not be paused to put its queue in order, so it is left "+
"first come, first served: %w", seat, err)
}
defer func() {
resuming, cancel := context.WithTimeout(context.WithoutCancel(ctx), 5*time.Second)
defer cancel()
if _, err := api.ResumeConsumer(resuming, worker.Stream, worker.Name); err != nil {
fmt.Printf("the %s worker could not be resumed after its queue was put in order; it resumes by "+
"itself by %s: %v\n", seat, pauseEnds.Format(time.RFC3339), err)
}
}()
if waiting, extra, _, err = readWaiting(ctx, api, worker, seat); err != nil {
return pass, err
}
}
stream, err := api.Stream(ctx, worker.Stream)
if err != nil {
return pass, err
}
// A second copy of one ask — a move that published and stopped before it deleted — goes first.
for _, a := range extra {
if err := stream.DeleteMsg(ctx, a.Seq); err != nil && !errors.Is(err, jetstream.ErrMsgNotFound) {
return pass, fmt.Errorf("a second copy of %s (seq %d) could not be removed: %w", a.ID, a.Seq, err)
}
pass.Removed = append(pass.Removed, fmt.Sprintf("%s at seq %d, a move's second copy of the message at seq %s",
a.ID, a.Seq, a.movedFrom))
}
for _, a := range toMove(waiting, QueueOrder(waiting, time.Now())) {
if delivered >= 0 && time.Until(pauseEnds) < moveMargin {
// The rest waits for the next pass, under a pause of its own.
break
}
ok, err := moveToTheTail(ctx, js, stream, worker.Name, seat, a)
if err != nil {
return pass, err
}
if ok {
pass.Moved = append(pass.Moved, a.ID)
}
}
return pass, nil
}
// readWaiting is every ask the worker has not yet handed out, in queue order, but each second copy of one
// ask; the second copies, to remove; and the last sequence the worker handed out, -1 when there is no
// worker, so nothing is handed out to pause.
//
// **A second copy is proven, never guessed from an id**: a waiting message whose MovedFromHeader names an
// earlier message of the same request id still in the queue, and whose QueuedAtHeader is that message's
// first-queued time — a move that published its copy and stopped before it deleted the original. The
// original stays, waiting or handed out (a holder's then, and the copy would build it twice). Any other
// second message of one id — an ask a dead letter delivered again beside its original (ADR 0293) — is a
// real ask and stays.
func readWaiting(ctx context.Context, api jetstream.JetStream, worker broker.Consumer, seat string) (
waiting, extra []QueuedAsk, delivered int64, err error) {
stream, err := api.Stream(ctx, worker.Stream)
if err != nil {
return nil, nil, 0, fmt.Errorf("the %s work queue (%s) cannot be read: %w", seat, worker.Stream, err)
}
delivered = -1
if consumer, err := stream.Consumer(ctx, worker.Name); err == nil {
info, err := consumer.Info(ctx)
if err != nil {
return nil, nil, 0, fmt.Errorf("the worker of %s cannot be read: %w", seat, err)
}
delivered = int64(info.Delivered.Stream)
} else if !errors.Is(err, jetstream.ErrConsumerNotFound) {
return nil, nil, 0, fmt.Errorf("the worker of %s cannot be read: %w", seat, err)
}
// From the first: the stream holds only what is unsettled, so this is a handful of reads.
// Every message by sequence, with its id and first-queued time, to prove a copy against.
type held struct {
id string
stored time.Time
}
bySeq := map[string]held{}
var seq uint64 = 1
for n := 0; n < 10000; n++ {
msg, err := stream.GetMsg(ctx, seq, jetstream.WithGetMsgSubject(BuildWorkOf(seat)))
if errors.Is(err, jetstream.ErrMsgNotFound) {
break
}
if err != nil {
return nil, nil, 0, fmt.Errorf("reading the %s work queue: %w", seat, err)
}
seq = msg.Sequence + 1
a := askOf(msg)
a.movedFrom = msg.Header.Get(MovedFromHeader)
from, known := bySeq[a.movedFrom]
again := a.ID != "" && a.movedFrom != "" && known && from.id == a.ID && from.stored.Equal(a.Stored)
bySeq[strconv.FormatUint(a.Seq, 10)] = held{id: a.ID, stored: a.Stored}
switch {
case int64(a.Seq) <= delivered:
case again:
extra = append(extra, a)
default:
waiting = append(waiting, a)
}
}
return waiting, extra, delivered, nil
}
// moveToTheTail publishes the ask's body again at the queue's tail and then deletes it where it was. **The
// copy first**: a publish that fails deletes nothing, and a pass that stops between the two leaves two
// copies, which the next pass sees by their one id and makes one (readWaiting). A move never loses an ask.
// Looked at once more just before: one the worker has handed out since is a holder's now and stays. False
// when it was not moved.
func moveToTheTail(ctx context.Context, js *broker.JetStream, stream jetstream.Stream, workerName, seat string,
a QueuedAsk) (bool, error) {
if consumer, err := stream.Consumer(ctx, workerName); err == nil {
info, err := consumer.Info(ctx)
if err != nil {
return false, err
}
if info.Delivered.Stream >= a.Seq {
return false, nil
}
}
msg, err := stream.GetMsg(ctx, a.Seq)
if errors.Is(err, jetstream.ErrMsgNotFound) {
return false, nil
}
if err != nil {
return false, err
}
again := &nats.Msg{Subject: msg.Subject, Data: msg.Data, Header: nats.Header{}}
for k, v := range msg.Header {
// The server's own headers stay behind: a message id would be taken for a duplicate and dropped.
if !strings.HasPrefix(k, "Nats-") {
again.Header[k] = v
}
}
// Its place in time goes with it: when the stream first took it, as askOf read it; and the message it
// is a copy of, which proves it one.
again.Header.Set(QueuedAtHeader, askOf(msg).Stored.UTC().Format(time.RFC3339Nano))
again.Header.Set(MovedFromHeader, strconv.FormatUint(a.Seq, 10))
// Inside the margin, so the publish ends while the worker is still paused.
publishing, cancel := context.WithTimeout(ctx, publishWaits)
ack, err := js.Context().PublishMsg(again, nats.Context(publishing))
cancel()
if err != nil {
return false, fmt.Errorf("%s (seq %d) could not be put behind, and stays where it is: %w", a.ID, a.Seq, err)
}
// **Handed out meanwhile** — a bus slow enough that the pause ended: the original is a holder's, and the
// copy goes instead, so it is built once.
if consumer, err := stream.Consumer(ctx, workerName); err == nil {
info, err := consumer.Info(ctx)
if err != nil || info.Delivered.Stream >= a.Seq {
if derr := stream.DeleteMsg(ctx, ack.Sequence); derr != nil && !errors.Is(derr, jetstream.ErrMsgNotFound) {
return false, fmt.Errorf("%s was handed out while it moved and its copy (seq %d) could not be "+
"removed; the next pass removes it: %w", a.ID, ack.Sequence, derr)
}
if err != nil {
return false, fmt.Errorf("whether %s was handed out while it moved cannot be read, so its copy was "+
"removed and it stays where it was: %w", a.ID, err)
}
if info.Delivered.Stream >= ack.Sequence {
return false, fmt.Errorf("%s was handed out while it moved, and its copy too: it may be built twice", a.ID)
}
return false, nil
}
}
if err := stream.DeleteMsg(ctx, a.Seq); err != nil && !errors.Is(err, jetstream.ErrMsgNotFound) {
return false, fmt.Errorf("%s was copied behind and its original (seq %d) not removed; the next pass "+
"removes the second copy: %w", a.ID, a.Seq, err)
}
return true, nil
}
// KeepingInOrder runs the passes, on the acting controller alone, until ctx ends: when an ask in this
// process says so, when the queue's last sequence moved — an ask from another process — and every
// passEvery, for a check that comes of age with nothing asked. seat says the build seat to order; acting
// whether this controller acts (Holding); said what a pass did.
func KeepingInOrder(ctx context.Context, js *broker.JetStream, seat func() string, acting func() bool,
said func(format string, args ...any)) {
tick := time.NewTicker(orderLooks)
defer tick.Stop()
var lastSeq uint64
var lastPass time.Time
for {
signalled := false
select {
case <-ctx.Done():
return
case <-asked:
signalled = true
case <-tick.C:
}
if !acting() {
continue
}
name := seat()
worker, _ := broker.HolderConsumerFor("", "", broker.DeclaredSeat{Name: name, Accepts: []string{"build"}})
info, err := js.Context().StreamInfo(worker.Stream)
if err != nil {
continue
}
if !signalled && info.State.LastSeq == lastSeq && time.Since(lastPass) < passEvery {
continue
}
lastSeq, lastPass = info.State.LastSeq, time.Now()
passing, cancel := context.WithTimeout(ctx, pausedFor+10*time.Second)
pass, err := PutInOrder(passing, js, name)
cancel()
for _, r := range pass.Removed {
said("the %s queue: removed %s", name, r)
}
if len(pass.Moved) > 0 {
said("the %s queue put in order: %s moved behind what goes first", name, strings.Join(pass.Moved, ", "))
}
if err != nil {
said("the %s queue could not be put in order: %v", name, err)
}
if info, err := js.Context().StreamInfo(worker.Stream); err == nil {
// Its own moves are not an ask.
lastSeq = info.State.LastSeq
}
}
}
// orderLooks is how often the loop looks whether an ask joined from another process; passEvery how often
// it passes with nothing asked.
var (
orderLooks = 2 * time.Second
passEvery = 30 * time.Second
)
// DeleteAsk removes a waiting or dead ask from the queue: at its sequence, or — moved into order since it
// was read — the copy carrying its id. Says the sequence it removed, 0 when neither was there any more.
func DeleteAsk(ctx context.Context, js *broker.JetStream, seat string, ask QueuedAsk) (uint64, error) {
worker, _ := broker.HolderConsumerFor("", "", broker.DeclaredSeat{Name: seat, Accepts: []string{"build"}})
gone := func(err error) bool {
return errors.Is(err, nats.ErrMsgNotFound) || errors.Is(err, jetstream.ErrMsgNotFound) ||
(err != nil && strings.Contains(err.Error(), "no message found"))
}
err := js.Context().DeleteMsg(worker.Stream, ask.Seq)
if err == nil {
return ask.Seq, nil
}
if !gone(err) {
return 0, err
}
if ask.ID == "" {
return 0, nil
}
q, err := ReadQueue(ctx, js, seat)
if err != nil {
return 0, err
}
for _, a := range q.Asks {
if a.ID == ask.ID && a.Seq != ask.Seq && a.State != AskInFlight {
if err := js.Context().DeleteMsg(worker.Stream, a.Seq); err != nil && !gone(err) {
return 0, err
} else if err == nil {
return a.Seq, nil
}
}
}
return 0, nil
}
+441
View File
@@ -0,0 +1,441 @@
package link
import (
"context"
"encoding/json"
"fmt"
"reflect"
"sync"
"testing"
"time"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-controller/internal/broker"
)
// The order the build queue hands asks out in (novox/hq issue 558): a build a walk waits for before any
// pull request's check, a check that has waited ten minutes before every delivery asked after it, and
// first come, first served within a kind.
func anAsk(seq uint64, id string, asked time.Time, check bool) QueuedAsk {
r := BuildRequest{ID: id, Repository: "/r"}
if check {
r.Check = &CheckRequest{Owner: "novox", Repo: "r"}
}
return QueuedAsk{Seq: seq, ID: id, AskedAt: asked, Request: r, State: AskWaiting}
}
func idsOf(asks []QueuedAsk) []string {
out := []string{}
for _, a := range asks {
out = append(out, a.ID)
}
return out
}
func TestADeliveryIsHandedOutBeforeTheChecksAskedBeforeIt(t *testing.T) {
now := time.Date(2026, 10, 12, 12, 0, 0, 0, time.UTC)
waiting := []QueuedAsk{
anAsk(1, "check-a", now.Add(-3*time.Minute), true),
anAsk(2, "check-b", now.Add(-2*time.Minute), true),
anAsk(3, "delivery-d", now.Add(-time.Minute), false),
}
want := []string{"delivery-d", "check-a", "check-b"}
if got := idsOf(QueueOrder(waiting, now)); !reflect.DeepEqual(got, want) {
t.Fatalf("handed out as %v, want %v", got, want)
}
// Only the checks ahead of it move behind it; the delivery stays where it is.
if got := idsOf(toMove(waiting, QueueOrder(waiting, now))); !reflect.DeepEqual(got, []string{"check-a", "check-b"}) {
t.Fatalf("moved %v", got)
}
}
func TestACheckThatWaitedTenMinutesGoesBeforeALaterDelivery(t *testing.T) {
now := time.Date(2026, 10, 12, 12, 0, 0, 0, time.UTC)
// As the queue stands after a delivery put the young checks behind it, ten minutes ago and since.
waiting := []QueuedAsk{
anAsk(5, "delivery-early", now.Add(-12*time.Minute), false),
anAsk(6, "delivery-later", now.Add(-time.Minute), false),
anAsk(7, "check-old", now.Add(-11*time.Minute), true),
anAsk(8, "check-young", now.Add(-9*time.Minute), true),
}
want := []string{"delivery-early", "check-old", "delivery-later", "check-young"}
if got := idsOf(QueueOrder(waiting, now)); !reflect.DeepEqual(got, want) {
t.Fatalf("handed out as %v, want %v", got, want)
}
if got := idsOf(toMove(waiting, QueueOrder(waiting, now))); !reflect.DeepEqual(got, []string{"delivery-later", "check-young"}) {
t.Fatalf("moved %v", got)
}
}
func TestAnAskWhoseKindCannotBeToldIsACheck(t *testing.T) {
for _, r := range []BuildRequest{
{}, // unreadable: nothing was read from it
{ID: "build-1", Check: &CheckRequest{}}, // a pull request's check, or a recheck the merge tool asked
{ID: "build-2", DryRun: true}, // looked at, never delivered
} {
if got := KindOf(r); got != KindCheck {
t.Errorf("%+v is a %s", r, got)
}
}
if got := KindOf(BuildRequest{ID: "build-3", Repository: "/r"}); got != KindDelivery {
t.Errorf("a build is a %s", got)
}
}
// Against a real server: two checks wait, a delivery is asked, and the next free holder takes the delivery.
func TestNatsADeliveryIsTakenBeforeTheChecksQueuedAheadOfIt(t *testing.T) {
js := aBusWithTheBuildRole(t)
ctx, stop := context.WithCancel(context.Background())
defer stop()
asks := BuildsOn(js, TheBuildMachine)
now := time.Now()
for i, r := range []BuildRequest{
{ID: NewBuildID(now.Add(-3 * time.Minute)), Repository: "/a", Check: &CheckRequest{Owner: "novox", Repo: "a"}},
{ID: NewBuildID(now.Add(-2 * time.Minute)), Repository: "/b", Check: &CheckRequest{Owner: "novox", Repo: "b"}},
{ID: NewBuildID(now), Repository: "/d"},
} {
if i == 2 {
// And one nobody can read, whose kind cannot be told: it waits with the checks.
if _, err := js.Context().Publish(BuildWork(), []byte("not an ask")); err != nil {
t.Fatal(err)
}
}
if err := asks.Ask(ctx, r); err != nil {
t.Fatal(err)
}
}
// The acting controller's loop puts it in order, told by the asks.
go KeepingInOrder(ctx, js, func() string { return TheBuildMachine }, func() bool { return true },
func(string, ...any) {})
time.Sleep(500 * time.Millisecond)
took := make(chan string, 4)
machine := MachineOverNATS(js, "anchor")
defer machine.Close()
go func() {
_ = machine.Take(ctx, func(ctx context.Context, work Build) {
took <- work.Request().Repository
_ = work.Announce(ctx, BuildResult{ID: work.Request().ID, On: "anchor", Failed: "no"})
_ = work.Done()
})
}()
var order []string
for len(order) < 3 {
select {
case r := <-took:
order = append(order, r)
case <-time.After(10 * time.Second):
t.Fatalf("taken so far: %v", order)
}
}
if want := []string{"/d", "/a", "/b"}; !reflect.DeepEqual(order, want) {
t.Fatalf("taken as %v, want %v", order, want)
}
}
// --- the moves, against a real server ------------------------------------------------------------
func publishAsk(t *testing.T, js *broker.JetStream, r BuildRequest) {
t.Helper()
body, _ := json.Marshal(r)
if _, err := js.Context().Publish(BuildWork(), body); err != nil {
t.Fatal(err)
}
}
func aCheck(asked time.Time, repository string) BuildRequest {
return BuildRequest{ID: NewBuildID(asked), Repository: repository, Check: &CheckRequest{Owner: "novox", Repo: "r"}}
}
func aDelivery(asked time.Time, repository string) BuildRequest {
return BuildRequest{ID: NewBuildID(asked), Repository: repository}
}
// queued is the stream's asks, in queue order.
func queued(t *testing.T, js *broker.JetStream) []QueuedAsk {
t.Helper()
q, err := ReadQueue(context.Background(), js, TheBuildMachine)
if err != nil {
t.Fatal(err)
}
return q.Asks
}
// takeAll runs a holder that records how often it was handed each repository.
func takeAll(t *testing.T, ctx context.Context, js *broker.JetStream) (chan string, func() map[string]int) {
t.Helper()
took := make(chan string, 100)
var mu sync.Mutex
counts := map[string]int{}
machine := MachineOverNATS(js, "anchor")
t.Cleanup(machine.Close)
go func() {
_ = machine.Take(ctx, func(ctx context.Context, work Build) {
mu.Lock()
counts[work.Request().Repository]++
mu.Unlock()
took <- work.Request().Repository
_ = work.Announce(ctx, BuildResult{ID: work.Request().ID, On: "anchor", Failed: "no"})
_ = work.Done()
})
}()
return took, func() map[string]int {
mu.Lock()
defer mu.Unlock()
out := map[string]int{}
for k, v := range counts {
out[k] = v
}
return out
}
}
// A move that published its copy and stopped before deleting the original leaves two copies of one ask;
// the next pass makes them one, and a copy of one a holder already took is removed, never built twice.
func TestNatsAMoveCutShortLeavesOneCopyAfterTheNextPass(t *testing.T) {
js := aBusWithTheBuildRole(t)
ctx := t.Context()
now := time.Now()
check := aCheck(now.Add(-2*time.Minute), "/c")
publishAsk(t, js, check)
publishAsk(t, js, aDelivery(now.Add(-time.Minute), "/d"))
moveCopyOf(t, js, check.ID) // the copy, published; the original never deleted
if pass, err := PutInOrder(ctx, js, TheBuildMachine); err != nil || len(pass.Removed) != 1 {
t.Fatalf("removed %v: %v", pass.Removed, err)
}
var got []string
for _, a := range queued(t, js) {
got = append(got, a.Repository)
}
if want := []string{"/d", "/c"}; !reflect.DeepEqual(got, want) {
t.Fatalf("after the pass the queue holds %v, want %v", got, want)
}
// And one a holder took before the copy was published: the copy goes, the holder's stays.
taken := aCheck(now, "/taken")
publishAsk(t, js, taken)
worker, _ := broker.HolderConsumerFor("", "", broker.DeclaredSeat{Name: TheBuildMachine, Accepts: []string{"build"}})
sub, err := js.Context().PullSubscribe(worker.Filters[0], worker.Name, nats.Bind(worker.Stream, worker.Name), nats.ManualAck())
if err != nil {
t.Fatal(err)
}
defer func() { _ = sub.Unsubscribe() }()
// Take everything waiting up to and including /taken, and keep /taken unsettled.
for {
msgs, err := sub.Fetch(1, nats.MaxWait(2*time.Second))
if err != nil || len(msgs) == 0 {
t.Fatalf("nothing to take: %v", err)
}
var r BuildRequest
_ = json.Unmarshal(msgs[0].Data, &r)
if r.Repository == "/taken" {
break
}
_ = msgs[0].Ack()
}
moveCopyOf(t, js, taken.ID)
if _, err := PutInOrder(ctx, js, TheBuildMachine); err != nil {
t.Fatal(err)
}
n := 0
for _, a := range queued(t, js) {
if a.ID == taken.ID {
n++
}
}
if n != 1 {
t.Fatalf("%d copies of the ask a holder took are in the queue, want 1", n)
}
}
// moveCopyOf publishes what a move publishes for the ask with this id, as a pass stopped before its delete
// would leave it.
func moveCopyOf(t *testing.T, js *broker.JetStream, id string) {
t.Helper()
for _, a := range queued(t, js) {
if a.ID != id {
continue
}
body, _ := json.Marshal(a.Request)
msg := &nats.Msg{Subject: BuildWork(), Data: body, Header: nats.Header{}}
msg.Header.Set(QueuedAtHeader, a.Stored.UTC().Format(time.RFC3339Nano))
msg.Header.Set(MovedFromHeader, fmt.Sprint(a.Seq))
if _, err := js.Context().PublishMsg(msg); err != nil {
t.Fatal(err)
}
return
}
t.Fatalf("%s is not queued", id)
}
// A dead letter delivered again whose original stayed beside it (ADR 0293: its removal failed) is a real
// ask with the same id, and is never removed as a second copy — even when the original was itself a move's
// copy, whose headers the delivery carries.
func TestNatsAnAskDeliveredAgainBesideItsOriginalStays(t *testing.T) {
js := aBusWithTheBuildRole(t)
ctx := t.Context()
ask := aCheck(time.Now(), "/again")
publishAsk(t, js, aCheck(time.Now().Add(-time.Minute), "/before"))
publishAsk(t, js, ask)
publishAsk(t, js, aDelivery(time.Now(), "/d"))
if _, err := PutInOrder(ctx, js, TheBuildMachine); err != nil { // /again becomes a move's copy
t.Fatal(err)
}
worker, _ := broker.HolderConsumerFor("", "", broker.DeclaredSeat{Name: TheBuildMachine, Accepts: []string{"build"}})
sub, err := js.Context().PullSubscribe(worker.Filters[0], worker.Name, nats.Bind(worker.Stream, worker.Name), nats.ManualAck())
if err != nil {
t.Fatal(err)
}
defer func() { _ = sub.Unsubscribe() }()
var original *nats.Msg
for original == nil {
msgs, err := sub.Fetch(1, nats.MaxWait(2*time.Second))
if err != nil || len(msgs) == 0 {
t.Fatalf("nothing to take: %v", err)
}
var r BuildRequest
_ = json.Unmarshal(msgs[0].Data, &r)
if r.ID == ask.ID {
original = msgs[0] // given up on, never settled: dead, and kept
continue
}
_ = msgs[0].Ack()
}
// Delivered again as DeliverAgain does: its headers, and its own.
again := &nats.Msg{Subject: BuildWork(), Data: original.Data, Header: nats.Header{}}
for k, v := range original.Header {
again.Header[k] = v
}
again.Header.Set(AgainHeader, "7")
if _, err := js.Context().PublishMsg(again); err != nil {
t.Fatal(err)
}
pass, err := PutInOrder(ctx, js, TheBuildMachine)
if err != nil || len(pass.Removed) != 0 {
t.Fatalf("removed %v: %v", pass.Removed, err)
}
n := 0
for _, a := range queued(t, js) {
if a.ID == ask.ID {
n++
}
}
if n != 2 {
t.Fatalf("%d messages of the ask delivered again are queued, want its original and the delivery", n)
}
}
// Two passes at once: nothing is built twice, and nothing is lost.
func TestNatsTwoPassesAtOnceBuildNothingTwice(t *testing.T) {
js := aBusWithTheBuildRole(t)
ctx, stop := context.WithCancel(t.Context())
defer stop()
took, counts := takeAll(t, ctx, js)
now := time.Now()
const asks = 24
var passes sync.WaitGroup
for p := 0; p < 2; p++ {
passes.Add(1)
go func() {
defer passes.Done()
for i := 0; i < 15; i++ {
_, _ = PutInOrder(ctx, js, TheBuildMachine)
}
}()
}
for i := 0; i < asks; i++ {
repository := fmt.Sprintf("/%02d", i)
if i%3 == 2 {
publishAsk(t, js, aDelivery(now.Add(time.Duration(i)*time.Millisecond), repository))
} else {
publishAsk(t, js, aCheck(now.Add(time.Duration(i)*time.Millisecond), repository))
}
}
passes.Wait()
for got := 0; got < asks; got++ {
select {
case <-took:
case <-time.After(15 * time.Second):
t.Fatalf("taken %d of %d: %v", got, asks, counts())
}
}
time.Sleep(time.Second)
c := counts()
for i := 0; i < asks; i++ {
if n := c[fmt.Sprintf("/%02d", i)]; n != 1 {
t.Errorf("/%02d was built %d times", i, n)
}
}
}
// A check that has waited ten minutes goes before a delivery asked after it, even one queued ahead of it.
func TestNatsACheckThatWaitedTenMinutesIsTakenBeforeALaterDelivery(t *testing.T) {
js := aBusWithTheBuildRole(t)
ctx, stop := context.WithCancel(t.Context())
defer stop()
now := time.Now()
publishAsk(t, js, aDelivery(now.Add(-time.Minute), "/delivery"))
publishAsk(t, js, aCheck(now.Add(-11*time.Minute), "/old-check"))
if _, err := PutInOrder(ctx, js, TheBuildMachine); err != nil {
t.Fatal(err)
}
took, _ := takeAll(t, ctx, js)
var order []string
for len(order) < 2 {
select {
case r := <-took:
order = append(order, r)
case <-time.After(10 * time.Second):
t.Fatalf("taken so far: %v", order)
}
}
if want := []string{"/old-check", "/delivery"}; !reflect.DeepEqual(order, want) {
t.Fatalf("taken as %v, want %v", order, want)
}
}
// A cancel that read an ask before it moved deletes the moved copy, found by its id, and says when
// there was nothing left to delete.
func TestNatsACancelRacingAMoveDeletesTheMovedCopy(t *testing.T) {
js := aBusWithTheBuildRole(t)
ctx := t.Context()
now := time.Now()
publishAsk(t, js, aCheck(now.Add(-time.Minute), "/c"))
read := queued(t, js)[0]
publishAsk(t, js, aDelivery(now, "/d"))
if pass, err := PutInOrder(ctx, js, TheBuildMachine); err != nil || len(pass.Moved) != 1 {
t.Fatalf("moved %v: %v", pass.Moved, err)
}
removed, err := DeleteAsk(ctx, js, TheBuildMachine, read)
if err != nil || removed == 0 || removed == read.Seq {
t.Fatalf("removed seq %d (read at %d): %v", removed, read.Seq, err)
}
for _, a := range queued(t, js) {
if a.ID == read.ID {
t.Fatalf("the cancelled ask is still queued at seq %d", a.Seq)
}
}
if removed, err := DeleteAsk(ctx, js, TheBuildMachine, read); err != nil || removed != 0 {
t.Fatalf("an ask gone already is said removed at %d: %v", removed, err)
}
}
// An ask nothing can be read from keeps its place in time when it moves.
func TestNatsAnUnreadableAskKeepsItsAgeWhenItMoves(t *testing.T) {
js := aBusWithTheBuildRole(t)
ctx := t.Context()
if _, err := js.Context().Publish(BuildWork(), []byte("not an ask")); err != nil {
t.Fatal(err)
}
first := queued(t, js)[0].Stored
time.Sleep(50 * time.Millisecond)
publishAsk(t, js, aDelivery(time.Now(), "/d"))
if pass, err := PutInOrder(ctx, js, TheBuildMachine); err != nil || len(pass.Moved) != 1 {
t.Fatalf("moved %v: %v", pass.Moved, err)
}
asks := queued(t, js)
if len(asks) != 2 || asks[1].ID != "" || !asks[1].Stored.Equal(first) {
t.Fatalf("after the move the unreadable ask was taken at %v, first at %v: %+v", asks[1].Stored, first, asks)
}
}
+3
View File
@@ -35,3 +35,6 @@
./cmd/mesh-controller TestARollbackWhoseSendWasRefusedIsCompletedByTheNextSend
./cmd/mesh-controller TestAGateJudgesTheOtherMovesBesideAKeptCore
./cmd/mesh-controller TestReplay401
./cmd/mesh-controller TestReplay201ThePlansSend
./internal/inventory TestAWalksRestIsReadFromEachMachinesOwnSend
./cmd/mesh-controller TestPlanAndNodeListAnOptionalRequirementLeftUnbound