Author SHA1 Message Date
mesh-admin 1a26778823 Merge pull request 'A busy container is left running until it says idle, within 15 minutes (issue 594, hq ADR 0335)' (#98) from feat/335-a-busy-container-is-replaced-when-idle into main 2026-10-13 22:20:54 +00:00
jschoubben b9a293f8c8 Merge remote-tracking branch 'origin/main' into feat/335-a-busy-container-is-replaced-when-idle
mesh/merge-gate head b9a293f8c8bc merged into trunk 0e92ef3d1b21 — pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes wi…
mesh/repo-check head b9a293f8c8bc merged into trunk 0e92ef3d1b21 — pass: its merge-check.sh passed
mesh/delivery-group group feat/335-a-busy-container-is-replaced-when-idle delivered: every member is delivered
mesh/delivery delivered
# Conflicts:
#	cmd/mesh-host/main.go
2026-10-14 00:00:09 +02:00
jschoubben 695e022991 Keep a replacement owed after a busy wait, and clamp a wait from a later clock
A restart-on trigger that is no file counts as changed only in its own pass,
so after a busy wait the container read unchanged and was never replaced,
while the report stopped listing it (review of #98).
2026-10-13 23:59:58 +02:00
mesh-admin 0e92ef3d1b Merge pull request 'Report the networks the outward links are on, for the local network (issue 154, ADR 0340)' (#107) from fix/154-local-network into main 2026-10-13 13:07:08 +00:00
mesh-admin 12bc6a56a3 Merge pull request 'Adopt genesis's bus through the bus's planned step, its streams snapshotted first (issue 590, hq ADR 0333 §3)' (#108) from fix/590-adopt-the-bus-as-its-planned-step into main 2026-10-13 11:11:41 +00:00
jochen cdcfcd9380 Adopt genesis's bus through the bus's planned step, its streams snapshotted first (hq issue 590, ADR 0333 §3)
mesh/merge-gate head cdcfcd938016 merged into trunk 130c5d98a601 — pass: the change touches no module of the mesh's graph
mesh/repo-check head cdcfcd938016 merged into trunk 130c5d98a601 — pass: its merge-check.sh passed
mesh/delivery delivered
The broker step sent the nats module with a plain push, and the controller refuses a push that would
replace the bus: that is a planned step, bus upgrade, which snapshots the streams first and checks
them after (ADR 0236). Genesis stopped at step 18.

The step now snapshots every stream with the module's own snapshot program (no backup holder exists
yet to be asked), asks bus upgrade with that snapshot and why, says the one restart, and waits until
the bus runs the module's image, answers, and the controller ends the step done. The bundle's bus runs
the release the module's image is built on (2.11.17, by the same digest, where it ran 2.10.29), so the
step changes no server release and is said reversible.
2026-10-13 11:47:10 +02:00
mesh-admin 130c5d98a6 Merge pull request 'Build the Go toolchain at genesis's base step, beside the base (issue 616)' (#106) from fix/616-the-base-builds-every-genesis-toolchain into main 2026-10-13 09:23:04 +00:00
jochen cb7b71dcc5 Report the networks the outward links are on, for the local network (issue 154, ADR 0340)
mesh/merge-gate head cb7b71dcc568 merged into trunk 12bc6a56a38a — pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes wi…
mesh/repo-check head cb7b71dcc568 merged into trunk 12bc6a56a38a — pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group fix/154-local-network delivered: every member is delivered
A port reached from local-network is admitted from the private networks the machine is attached to, and only the machine can say which those are. Every network on a link facing outside is said among its facts, and a change is said at the next reconcile.
2026-10-13 11:20:21 +02:00
jochen 8108dd5c23 Build the Go toolchain at genesis's base step, beside the base (hq issue 616)
mesh/merge-gate head 8108dd5c23c6 merged into trunk f0d39b4a4ab0 — pass: the change touches no module of the mesh's graph
mesh/repo-check head 8108dd5c23c6 merged into trunk f0d39b4a4ab0 — pass: its merge-check.sh passed
mesh/delivery delivered
Genesis's base step built only mesh-tools, the TypeScript toolchain. The store's and the bus's modules
carry Go bundles, compiled in mesh-tools-go, so genesis stopped at the store step: "this mesh holds no
copy of it. Build mesh-tools-go first". The base step now builds mesh-tools-go too, from
--tools-go-source or the repository beside --tools-source, and a test holds every toolchain a genesis
module's bundles compile in, read from the catalogue, to the ones the step builds.
2026-10-13 11:13:44 +02:00
mesh-admin f0d39b4a4a Merge pull request 'Wait until the node applied the genesis document before the sdk step asks for a build (issue 615)' (#105) from fix/615-wait-for-the-genesis-document into main 2026-10-13 09:08:59 +00:00
jochen 89fd0c06e1 Wait until the node applied the genesis document before the sdk step asks for a build (hq issue 615)
mesh/merge-gate head 89fd0c06e102 merged into trunk 17038b2a074b — pass: the change touches no module of the mesh's graph
mesh/repo-check head 89fd0c06e102 merged into trunk 17038b2a074b — pass: its merge-check.sh passed
mesh/delivery delivered
The packages step sealed the build agent its genesis document and pushed, and the push returned as soon
as it was sent. The sdk step then asked for the SDK's build at once, and the agent reads the document
from the file the node writes when it applies that push: one genesis found it in time, the next was
refused for having neither a binding nor the document. That one push now waits until the node reports
it applied it (push --wait); every other push still returns on sending.
2026-10-13 10:59:45 +02:00
mesh-admin 17038b2a07 Merge pull request 'Raise genesis's store as the postgres module's server, so the store step adopts it as it runs (issue 609, hq ADR 0338)' (#104) from fix/609-genesis-raises-the-store-as-postgres into main 2026-10-13 08:55:41 +00:00
jochen 4dde21f0ab Raise genesis's store as the postgres module's server, so the store step adopts it as it runs (hq issue 609, ADR 0338)
mesh/merge-gate head 4dde21f0ab20 merged into trunk 7d1d6bffe30c — pass: the change touches no module of the mesh's graph
mesh/repo-check head 4dde21f0ab20 merged into trunk 7d1d6bffe30c — pass: its merge-check.sh passed
mesh/delivery delivered
The bundle raised the store as mesh-store, on upstream postgres, in a docker volume; the postgres
module keeps it as postgres, on its pgvector image, in /var/lib/mesh-store owned 999:70, reading its
superuser password from a file. The store step refused, and genesis stopped at step 17 of 24.

The bundle now raises the store with the module's name, image, data directory and password file, and
every step of the bundle and the installer reaches it by that name. The refusal to raise a store over
data it cannot open looks for a cluster in the store's directory, no longer for the old volume.
2026-10-13 10:47:53 +02:00
mesh-admin 7d1d6bffe3 Merge pull request 'Place the bus's users after every push until the bus is adopted (issue 608)' (#103) from fix/608-place-the-bus-users-after-every-push into main 2026-10-13 05:00:42 +00:00
jochen 7283209d1a Place the bus's users after every push until the bus is adopted (hq issue 608)
mesh/merge-gate head 7283209d1abc merged into trunk 64f0ab405101 — pass: the change touches no module of the mesh's graph
mesh/repo-check head 7283209d1abc merged into trunk 64f0ab405101 — pass: its merge-check.sh passed
mesh/delivery delivered
The bus reads its user list from a file that, until the broker step adopts it as the nats module,
only the installer writes, and the installer wrote it only at enrolment. The build agent's account,
minted with step 13's push, never reached the bus; the agent was refused (Authorization Violation)
and the sdk step's build waited until it was killed. Every push genesis makes now places the composed
list and has the bus re-read it, until the nats module is registered (ADR 0331 §5, ADR 0333 §4).
2026-10-13 06:58:39 +02:00
mesh-admin 64f0ab4051 Merge pull request 'genesis: seal the build agent the forge it raised, in place of the builder's npm-password (hq issue 121, ADR 0336)' (#102) from fix/121-genesis-package-registry into main 2026-10-13 03:37:15 +00:00
jochen 5f147fe6b9 Seal the build agent the forge genesis raised, in place of the builder's npm-password (hq issue 121, ADR 0336)
mesh/merge-gate head 5f147fe6b9c7 merged into trunk 0d11b4fc930d — pass: the change touches no module of the mesh's graph
mesh/repo-check head 5f147fe6b9c7 merged into trunk 0d11b4fc930d — pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group fix/121-genesis-package-registry delivered: every member is delivered
The build agent declares neither the retired builder's npm-password own-secret
nor a serves setting, so the packages step handed its credential to nothing
and genesis's sdk step had no way to publish. The packages step now seals the
genesis-package-registry own-secret to build-agent: the forge on this node's
loopback, the port given for the package registry, its npm path, and the
mesh-builder account with its password. setFoundationSettings no longer gives
the builder a serves.port.
2026-10-13 03:01:35 +02:00
jschoubben bb60e7b584 Leave a busy container running until it says idle, within 15 minutes
mesh/delivery superseded: a newer head of the same pull request
mesh/delivery-group group feat/335-a-busy-container-is-replaced-when-idle rejected: its order contradicts itself: novox/mesh-catalog@190608193764, novox/mesh-c…
mesh/merge-gate out of date: judged against trunk 7a41e120635b, the trunk is now 8431ac96562e; judged again on a recheck or when it is merged
mesh/repo-check out of date: judged against trunk 7a41e120635b, the trunk is now 8431ac96562e; judged again on a recheck or when it is merged
A send that replaced the build agent killed the check it ran, and the queue
started it again elsewhere (issue 594). A container may now declare busy; the
node-engine asks it before replacing it, applies everything else, says the
wait, and replaces it on the first idle pass or at the bound, kept across
restarts (novox/hq ADR 0335).
2026-10-13 00:41:32 +02:00
40 changed files with 2908 additions and 166 deletions
+6
View File
@@ -116,6 +116,8 @@ const usage = `mesh-bootstrap — make a bare machine into a mesh
--tools-source the repository the shared base is built from
--tools-ref what of it to build (default main)
--tools-go-source the repository the Go toolchain is built from (default: mesh-tools-go beside --tools-source)
--tools-go-ref what of it to build (default main)
--catalog-source the catalogue REPOSITORY, for building its modules;
--catalog is the checkout that says what they are
--catalog-ref what of it to build (default main)
@@ -308,6 +310,10 @@ func newFlagSet(opts *bootstrap.Options, jsonOut *bool) *flag.FlagSet {
"the repository the shared base is built from")
set.StringVar(&opts.ToolsSource.Ref, "tools-ref", opts.ToolsSource.Ref,
"what of it to build (default main)")
set.StringVar(&opts.ToolsGoSource.Repository, "tools-go-source", opts.ToolsGoSource.Repository,
"the repository the Go toolchain is built from (default: mesh-tools-go beside --tools-source)")
set.StringVar(&opts.ToolsGoSource.Ref, "tools-go-ref", opts.ToolsGoSource.Ref,
"what of it to build (default main)")
set.StringVar(&opts.CatalogSource.Repository, "catalog-source", opts.CatalogSource.Repository,
"the catalogue REPOSITORY, for building its modules — --catalog is the checkout that says what they are")
set.StringVar(&opts.CatalogSource.Ref, "catalog-ref", opts.CatalogSource.Ref,
+66
View File
@@ -0,0 +1,66 @@
package main
import (
"context"
"sync/atomic"
"testing"
"time"
"github.com/novox/mesh-host/internal/link"
)
// A container waiting because it said busy is news when it starts waiting and when it is replaced, never on
// its output alone (novox/hq ADR 0335): the walk waiting on this machine starts its gate at the report that
// says it replaced, and a reconcile is otherwise silent.
func TestABusyWaitIsSaidWhenItStartsAndWhenItEnds(t *testing.T) {
quiet := link.Report{Outward: []string{"eth0"}}
waiting := link.Report{Outward: []string{"eth0"}, Busy: []link.Busy{{ID: "build-agent.server",
Container: "mesh-build-agent", State: "waiting", Said: "building a"}}}
still := link.Report{Outward: []string{"eth0"}, Busy: []link.Busy{{ID: "build-agent.server",
Container: "mesh-build-agent", State: "waiting", Said: "building b, a minute on"}}}
replaced := link.Report{Outward: []string{"eth0"}, Busy: []link.Busy{{ID: "build-agent.server",
Container: "mesh-build-agent", State: "replaced-when-idle"}}}
if adoptionFingerprint(waiting) == adoptionFingerprint(quiet) {
t.Error("a container starting to wait is not news")
}
if adoptionFingerprint(still) != adoptionFingerprint(waiting) {
t.Error("a wait whose output changed reads as news on every ask")
}
if adoptionFingerprint(replaced) == adoptionFingerprint(waiting) {
t.Error("a container replaced after it waited is not news")
}
if !worthSaying(link.Report{Busy: replaced.Busy}) {
t.Error("a replacement after a wait is not worth saying")
}
}
// While a container waits, a pass is asked for every so often, not at the reconcile's five minutes; with
// nothing waiting, none (novox/hq ADR 0335 §2).
func TestAWaitIsAskedAgainSoonerThanTheReconcile(t *testing.T) {
if BusyAskEvery >= ReconcileEvery || BusyAskEvery > time.Minute {
t.Fatalf("a waiting container is asked every %s", BusyAskEvery)
}
var reconciles atomic.Int32
queue := &link.Queue{Reconcile: func(context.Context) (link.Report, bool) {
reconciles.Add(1)
return link.Report{}, false
}}
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
go queue.Run(ctx)
busyWaiting.Store(false)
t.Cleanup(func() { busyWaiting.Store(false) })
go busyPasses(ctx, queue, 10*time.Millisecond)
time.Sleep(100 * time.Millisecond)
if n := reconciles.Load(); n != 0 {
t.Fatalf("%d passes were asked with nothing waiting", n)
}
busyWaiting.Store(true)
deadline := time.Now().Add(2 * time.Second)
for reconciles.Load() == 0 && time.Now().Before(deadline) {
time.Sleep(10 * time.Millisecond)
}
if reconciles.Load() == 0 {
t.Fatal("no pass was asked while a container waits")
}
}
+29
View File
@@ -0,0 +1,29 @@
package main
import (
"testing"
"github.com/novox/mesh-host/internal/link"
"github.com/novox/mesh-host/internal/outward"
"github.com/novox/mesh-host/internal/profile"
)
func reportOn(networks ...outward.Network) link.Report {
return link.Report{Outward: []string{"eth0"},
Profile: profileAsReported(profile.Profile{Machine: &profile.Machine{Networks: networks}})}
}
// **A change of the networks a machine is attached to is said unasked** (novox/hq ADR 0340 §7): a port
// reached from the local network is admitted from them, so a laptop that joins another network is said at
// the next reconcile, not at the next push — and the same networks again are not news.
func TestAChangedNetworkIsSaidUnasked(t *testing.T) {
w := &adoptionWatch{}
home := reportOn(outward.Network{Link: "eth0", Network: "192.168.1.0/24"})
w.said(home)
if w.differs(reportOn(outward.Network{Link: "eth0", Network: "192.168.1.0/24"})) {
t.Fatal("the same networks read as a change, so the machine would speak on every reconcile")
}
if !w.differs(reportOn(outward.Network{Link: "eth0", Network: "10.20.0.0/16"})) {
t.Fatal("a machine that moved to another network does not say so until it is pushed to")
}
}
+70 -2
View File
@@ -24,6 +24,7 @@ import (
"sort"
"strings"
"sync"
"sync/atomic"
"syscall"
"text/tabwriter"
"time"
@@ -1259,6 +1260,7 @@ func runLink(ctx context.Context, opts options) error {
queue.ReconcileDue()
}
go holdTheMachine(ctx, queue)
go busyPasses(ctx, queue, BusyAskEvery)
// And whether what it runs stays up, looked at on its own clock and said when it changes (novox/hq
// ADR 0240) — between applies, which is when a container crash-loops.
if judging != nil {
@@ -1397,6 +1399,18 @@ func adoptionFingerprint(r link.Report) string {
// it could only speak when a declaration arrived, and a declaration cannot be composed until it
// has spoken: a machine waiting for a push that is waiting for the machine.
parts := []string{"firewall=" + r.Firewall, "outward=" + strings.Join(r.Outward, ",")}
// **And the networks those links are on** (novox/hq ADR 0340 §7): a port reached from the local
// network is admitted from them, so a laptop that joins another network, or a lease that moves, is
// said at the next reconcile rather than at the next push or the next day.
if networks := attachedNetworksOf(r); networks != "" {
parts = append(parts, "networks="+networks)
}
// And a container waiting because it said busy, or replaced after it waited (novox/hq ADR 0335): the walk
// waiting on this machine starts its gate at the report that says it replaced. Not its output, which
// may change on every ask.
for _, b := range r.Busy {
parts = append(parts, "busy "+b.ID+"="+b.State)
}
for _, h := range r.Held {
parts = append(parts, "held "+h.ID+"="+h.Changed)
}
@@ -1428,6 +1442,26 @@ func adoptionFingerprint(r link.Report) string {
return strings.Join(parts, "\n")
}
// attachedNetworksOf is the networks a report's profile says the machine is attached to, as one line: the
// part of what the machine is made of that the filter is composed from (novox/hq ADR 0340).
func attachedNetworksOf(r link.Report) string {
machine, ok := r.Profile["machine"].(map[string]any)
if !ok {
return ""
}
networks, ok := machine["networks"].([]any)
if !ok {
return ""
}
said := make([]string, 0, len(networks))
for _, n := range networks {
if entry, ok := n.(map[string]any); ok {
said = append(said, fmt.Sprintf("%v %v", entry["link"], entry["network"]))
}
}
return strings.Join(said, ",")
}
// applyState is how an apply went, as the fingerprint keeps it: refused, clean, or failed with what it did not
// apply and which keys its failure is said under.
func applyState(r link.Report) string {
@@ -1736,7 +1770,7 @@ func healthAsReported(st liveness.Statement, ns *network.Statement) *link.Health
// LibraryContract: this engine reads a resource's declared `health` and judges it (ADR 0240 Phase B), judges a
// user's declared `root` (novox/hq ADR 0266), and reads a declaration's identities, groups and libraries
// (novox/hq ADR 0328), which is what tells the controller it may be sent any of those fields.
h := &link.Health{Contract: link.OwnGroupContract, At: st.At.UTC(), Resources: []link.ResourceHealth{}}
h := &link.Health{Contract: link.BusyContract, At: st.At.UTC(), Resources: []link.ResourceHealth{}}
for _, r := range st.Resources {
h.Resources = append(h.Resources, link.ResourceHealth{Module: r.Module, Resource: r.ID, Kind: r.Kind,
Target: r.Target, State: r.State, Reason: r.Reason, Since: r.Since.UTC(), Streak: r.Streak,
@@ -1868,6 +1902,32 @@ func holdTheMachine(ctx context.Context, queue *link.Queue) {
}
}
// busyWaiting is whether the last apply left a container running because it said busy (novox/hq ADR 0335).
var busyWaiting atomic.Bool
// BusyAskEvery is how often a container waiting because it said busy is asked again (novox/hq ADR 0335 §2):
// each ask is a pass of its own, so a build that ends is followed by its replacement within this, not
// within the reconcile's five minutes — and the bound of 15 minutes is met within this of its end.
const BusyAskEvery = 20 * time.Second
// busyPasses asks for a pass every so often while a container waits because it said busy. **It asks; it
// does not apply**, as holdTheMachine: the queue's worker does, when its turn comes, so a pass never holds
// the machine for the wait, and a delivery waiting is that pass's apply.
func busyPasses(ctx context.Context, queue *link.Queue, every time.Duration) {
ticker := time.NewTicker(every)
defer ticker.Stop()
for {
select {
case <-ctx.Done():
return
case <-ticker.C:
if busyWaiting.Load() {
queue.ReconcileDue()
}
}
}
}
// reconcileKept is the reconcile's act, as the queue's worker runs it: hold the machine to what it
// was last told, read once it is this act's turn, and say what did not go.
//
@@ -1919,7 +1979,7 @@ func worthSaying(report link.Report) bool {
}
return len(report.Held) > 0 || report.Firewall != "" || len(report.Outward) > 0 ||
len(report.Filters) > 0 || report.FoundFirewall != nil || len(report.Windows) > 0 ||
len(report.Rollbacks) > 0
len(report.Rollbacks) > 0 || len(report.Busy) > 0
}
// announceOr is what the apply writes its detail with, given what the caller has to say things with.
@@ -2168,6 +2228,14 @@ func applyAndKeepHeld(ctx context.Context, opts options, raw []byte, signed *sto
for _, w := range outcome.Windows {
report.Windows = append(report.Windows, link.Window{Step: w.Step, Holds: w.Holds, Since: w.Opened, Until: w.Until})
}
// And a container left running because it said busy, or replaced after it waited (novox/hq ADR 0335):
// the walk waiting on this machine reads it. While one waits, this machine asks it again sooner than its
// reconcile (busyPasses).
for _, b := range outcome.Busy {
report.Busy = append(report.Busy, link.Busy{ID: b.ID, Container: b.Container, State: b.State,
Since: b.Since, Until: b.Until, Said: b.Said})
}
busyWaiting.Store(outcome.Waiting())
// And what runs here that nobody asked for (novox/hq ADR 0163).
if strays, err := apply.Strays(ctx, apply.ExecRunner, updated); err != nil {
fmt.Fprintf(os.Stderr, "mesh-host: applied, and could not list what else runs here: %v\n", err)
+4 -4
View File
@@ -87,10 +87,10 @@ func TestTheStatementSaysAnAccountJudgedForRoot(t *testing.T) {
Groups: []string{"openrazer"}}, State: accounts.Healthy, Since: at},
}}
h := withAccounts(healthAsReported(liveness.Statement{At: at}, nil), as)
// At RootContract or later: an engine that also reads an identity's runner says OwnGroupContract (novox/hq ADR
// 0334), one above ADR 0328's LibraryContract.
if h.Contract < link.RootContract || link.RootContract != 3 || h.Contract != link.OwnGroupContract ||
link.OwnGroupContract != link.LibraryContract+1 {
// At RootContract or later: an engine that also reads a container's busy says BusyContract (novox/hq ADR
// 0335), one above ADR 0334's OwnGroupContract.
if h.Contract < link.RootContract || link.RootContract != 3 || h.Contract != link.BusyContract ||
link.BusyContract != link.OwnGroupContract+1 {
t.Fatalf("contract %d", h.Contract)
}
got := map[string]string{}
+1 -1
View File
@@ -49,7 +49,7 @@ func TestTheBundleCarriesTheFoundationAndTheControlPlaneAndNothingElse(t *testin
}
sort.Strings(images)
want := []string{"mesh-controller", "nats", "postgres"}
want := []string{"mesh-controller", "nats", "pgvector"}
if strings.Join(images, ",") != strings.Join(want, ",") {
t.Fatalf("the bundle carries %v; expected exactly %v.\n\n"+
"Adding one is a change to what every first node becomes, and to ADR 0006's "+
+33 -18
View File
@@ -28,11 +28,12 @@
// image, and this file is applied on a machine with no mesh to ask about anything (ADR 0006). The
// registry prefix is the lab's; the digests are upstream's.
//
// The store's data is a NAMED VOLUME, not a directory on the machine. A directory the host
// creates is owned by root, and the database runs as somebody else inside the container — so it
// could not write, and the container crash-looped. A named volume lets the image set up its own
// ownership, and outlives the container, which is what you want for the thing holding the mesh's
// state.
// **The store is raised where the `postgres` module keeps it** (novox/hq ADR 0338): the container
// `postgres`, on the module's pgvector image, its data in the directory /var/lib/mesh-store owned by
// the image's postgres user, its superuser password read from the module's own-secret file — so the
// store step adopts it as it runs. Pinned upstream by the module's own digest, not through the lab's
// registry prefix, because the two must be the same reference (mesh-host internal/bootstrap holds
// them to each other against the module's manifest).
{
"declaration": 1,
@@ -83,17 +84,28 @@
"boot": "enabled",
"restart-on": ["base-filter"]
},
// The store's data is a directory owned by the image's postgres user (999), made before the
// container starts: a directory the runtime made would be root's, and the database could not
// write it. Measured on the module's image 2026-10-13: it initialises a new store on a directory
// made so (novox/hq ADR 0338 §5).
{
"id": "store-data",
"type": "directory",
"path": "/var/lib/mesh-store",
"mode": "0700",
"owner": "999:70"
},
{
"id": "store",
"type": "container",
"name": "mesh-store",
"image": "192.0.2.250:5000/postgres@sha256:7abf537131b66ed5af448d90653abf1679b0c7e9a1f07efdd4c3108a401b259a",
"name": "postgres",
"image": "pgvector/pgvector@sha256:cf134a767f474095eeba57e0117be8e568e011a63f33fbf252f14c9b760f8e6f",
"env": {
"POSTGRES_PASSWORD": "bootstrap",
"PGDATA": "/var/lib/postgresql/data/pgdata"
},
"ports": ["5432:5432"],
"volumes": ["mesh-store-data:/var/lib/postgresql/data"]
"volumes": ["/var/lib/mesh-store:/var/lib/postgresql/data"]
},
// Over TCP, not the socket. While the store initialises it runs a temporary server on the
// socket ONLY, then stops it and starts the real one — so a socket check passes, the action
@@ -102,21 +114,21 @@
{
"id": "store-ready",
"type": "action",
"in": "mesh-store",
"in": "postgres",
"command": ["sh", "-c", "for i in $(seq 1 180); do pg_isready -h 127.0.0.1 -U postgres >/dev/null 2>&1 && exit 0; sleep 1; done; echo 'the store did not answer within 180s; its own last words follow'; pg_isready -h 127.0.0.1 -U postgres; tail -n 20 /var/lib/postgresql/data/log/*.log 2>/dev/null; exit 1"],
"verify": ["pg_isready", "-h", "127.0.0.1", "-U", "postgres"]
},
{
"id": "inventory-database",
"type": "action",
"in": "mesh-store",
"in": "postgres",
"command": ["sh", "-c", "psql -U postgres -c 'CREATE DATABASE inventory'"],
"verify": ["sh", "-c", "psql -U postgres -lqt | cut -d'|' -f1 | grep -qw inventory"]
},
{
"id": "identity-database",
"type": "action",
"in": "mesh-store",
"in": "postgres",
"command": ["sh", "-c", "psql -U postgres -c 'CREATE DATABASE identity'"],
"verify": ["sh", "-c", "psql -U postgres -lqt | cut -d'|' -f1 | grep -qw identity"]
},
@@ -126,20 +138,20 @@
{
"id": "licences-database",
"type": "action",
"in": "mesh-store",
"in": "postgres",
"command": ["sh", "-c", "psql -U postgres -c 'CREATE DATABASE licences'"],
"verify": ["sh", "-c", "psql -U postgres -lqt | cut -d'|' -f1 | grep -qw licences"]
},
{
"id": "context-schemas",
"type": "action",
"command": ["docker", "run", "--rm", "--network", "container:mesh-store",
"command": ["docker", "run", "--rm", "--network", "container:postgres",
"-e", "MESH_STORE_INVENTORY=postgres://postgres:bootstrap@127.0.0.1:5432/inventory?sslmode=disable",
"-e", "MESH_STORE_IDENTITY=postgres://postgres:bootstrap@127.0.0.1:5432/identity?sslmode=disable",
"-e", "MESH_STORE_LICENCES=postgres://postgres:bootstrap@127.0.0.1:5432/licences?sslmode=disable",
"192.0.2.250:5000/mesh-controller@sha256:c67db38439ff0aee242b467486765467bb95801f52175fc5727cc4e437338ace",
"migrate"],
"verify": ["sh", "-c", "docker exec mesh-store psql -U postgres -d inventory -tAc \"select to_regclass('public.node')\" | grep -qx node && docker exec mesh-store psql -U postgres -d identity -tAc \"select to_regclass('public.signing_key')\" | grep -qx signing_key && docker exec mesh-store psql -U postgres -d licences -tAc \"select to_regclass('public.licence')\" | grep -qx licence"]
"verify": ["sh", "-c", "docker exec postgres psql -U postgres -d inventory -tAc \"select to_regclass('public.node')\" | grep -qx node && docker exec postgres psql -U postgres -d identity -tAc \"select to_regclass('public.signing_key')\" | grep -qx signing_key && docker exec postgres psql -U postgres -d licences -tAc \"select to_regclass('public.licence')\" | grep -qx licence"]
},
// **The bus is raised where the `nats` module keeps it** (novox/hq ADR 0333 §1): the container
// `mesh-broker-nats`, its configuration and user list in /var/lib/nats-module/conf, its streams in
@@ -212,14 +224,17 @@
"content": "// The first user list, carried by the installer because at genesis there is no mesh to\n// compose one. A bootstrap credential, rotated with the store's and replaced by the\n// controller's own composition from its first start onward.\naccounts {\n MESH {\n jetstream: enabled\n users = [\n { user: \"controller\", password: \"$2a$10$AHqJgOifIVbU41KmATiMhuXFs8xa7Wl2HuN4UVBCXdN2jIQzjqApy\", permissions: {\n publish: { allow: [\"$JS.ACK.CONTROL.controller.>\", \"$JS.ACK.EVENTS.controller.>\", \"$JS.ACK.DEAD_LETTER_NOTICES.controller.>\", \"$JS.API.>\", \"$KV.mesh-controller_asked.>\", \"$KV.mesh-controller_calls.>\", \"$KV.mesh-controller_hand-acts.>\", \"$KV.mesh-controller_conditions.>\", \"$KV.mesh-controller_condition-history.>\", \"$KV.mesh-controller_lease.>\", \"$KV.SEAT_MESH_BUILD_MACHINE_cancelled.>\", \"$KV.SEAT_NODE_BUILD_AGENT_cancelled.>\", \"_INBOX.enrol.>\", \"mesh.assignment.>\", \"mesh.mod.*.tool.>\", \"mesh.node.>\", \"mesh.seat.mesh-build-machine.accept.>\", \"mesh.seat.node-build-agent.accept.>\", \"mesh.seat.mesh-build-machine.tool.>\", \"mesh.seat.node-build-agent.tool.>\", \"mesh.seat.mesh-controller.event.applied\", \"mesh.seat.mesh-controller.event.built-before\", \"mesh.seat.mesh-controller.event.refused\", \"mesh.seat.mesh-controller.event.rolled-back\", \"mesh.seat.mesh-controller.event.checked\", \"mesh.seat.mesh-controller.event.plan-moved\", \"mesh.seat.mesh-controller.event.condition-raised\", \"mesh.seat.mesh-controller.event.condition-changed\", \"mesh.seat.mesh-controller.event.condition-cleared\", \"mesh.seat.mesh-controller.event.doctor-heartbeat\", \"mesh.seat.mesh-controller.event.secret-replaced\", \"mesh.seat.mesh-controller.event.healer-acted\", \"$SRV.INFO\", \"mesh.seat.node-intrusion-prevention.tool.banned.*\", \"mesh.seat.node-backup.tool.backed-up.*\", \"mesh.seat.node-backup.tool.now.*\", \"mesh.seat.mesh-delivery.tool.stalled\", \"mesh.seat.mesh-delivery.tool.close\", \"mesh.seat.node-launcher.tool.secret.*\", \"mesh.seat.mesh-delivery.tool.release\", \"mesh.seat.mesh-delivery.tool.stop\", \"mesh.seat.mesh-delivery.tool.times\", \"mesh.seat.mesh-delivery.tool.groups\", \"mesh.seat.node-service-manager.tool.restart.*\", \"mesh.seat.mesh-controller.tool.plans\", \"mesh.events.dead.>\", \"mesh.again.>\"] }\n subscribe: { allow: [\"$JS.API.>\", \"$JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES.>\", \"$JS.EVENT.ADVISORY.CONSUMER.DELETED.>\", \"_DELIVER.controller\", \"_DELIVER.controller.>\", \"_INBOX.controller.>\", \"mesh.control.>\", \"mesh.mod.*.event.provisioner.failing\", \"mesh.mod.*.event.provisioner.recovered\", \"mesh.mod.*.event.provisioner.retirement\", \"mesh.mod.gitea.event.pull.merged\", \"mesh.mod.gitea.event.pull.updated\", \"mesh.mod.mesh-catalog.event.catching-up\", \"mesh.mod.mesh-catalog.event.upgraded\", \"mesh.seat.mesh-build-machine.event.built\", \"mesh.seat.node-build-agent.event.built\", \"mesh.seat.operator-channel.event.decided.mesh-controller\", \"mesh.seat.mesh-controller.tool.>\", \"$SRV.PING\", \"$SRV.INFO\", \"$SRV.PING.mesh-controller\", \"$SRV.PING.mesh-controller.>\", \"$SRV.INFO.mesh-controller\", \"$SRV.INFO.mesh-controller.>\", \"$SRV.STATS\", \"$SRV.STATS.mesh-controller\", \"$SRV.STATS.mesh-controller.>\"] }\n allow_responses: { max: 1, ttl: \"1m\" }\n } }\n ]\n }\n}\n"
},
// The upstream server, until the broker step adopts it: the `nats` module's own image, built by the
// mesh, takes its place there once, on these same directories (ADR 0333 §3). The published port is the
// mesh, takes its place there once, on these same directories (ADR 0333 §3), as the bus's planned
// step (ADR 0236). The release is the one the module's image is built on (its NATS_BASE, 2.11.17),
// pinned upstream by that same digest: the adoption changes no server release, so it can be undone
// by putting the old image back and is said reversible. The published port is the
// node's bus port, 5671 unless given, which every enrolment token names; the installer gives the
// module the setting that maps its declared 4222 to it (ADR 0333 §2).
{
"id": "broker",
"type": "container",
"name": "mesh-broker-nats",
"image": "192.0.2.250:5000/nats@sha256:b83efabe3e7def1e0a4a31ec6e078999bb17c80363f881df35edc70fcb6bb927",
"image": "nats@sha256:e4bf19f15fd3218814a4e3c9e0064e1334bd8aa20d5984b9f1a0afd084f8cc00",
"ports": ["5671:4222", "127.0.0.1:8222:8222"],
"volumes": ["/var/lib/mesh-broker-nats:/data", "/var/lib/nats-module/conf:/etc/nats:ro", "/var/lib/mesh-broker-tls:/tls:ro"],
"args": ["-c", "/etc/nats/nats.conf", "-js"]
@@ -227,8 +242,8 @@
{
"id": "broker-ready",
"type": "action",
"command": ["sh", "-c", "for i in $(seq 1 60); do docker run --rm --network host --entrypoint sh 192.0.2.250:5000/nats@sha256:b83efabe3e7def1e0a4a31ec6e078999bb17c80363f881df35edc70fcb6bb927 -c 'nc -z 127.0.0.1 5671' >/dev/null 2>&1 && exit 0; sleep 1; done; exit 1"],
"verify": ["sh", "-c", "docker run --rm --network host --entrypoint sh 192.0.2.250:5000/nats@sha256:b83efabe3e7def1e0a4a31ec6e078999bb17c80363f881df35edc70fcb6bb927 -c 'nc -z 127.0.0.1 5671'"]
"command": ["sh", "-c", "for i in $(seq 1 60); do docker run --rm --network host --entrypoint sh nats@sha256:e4bf19f15fd3218814a4e3c9e0064e1334bd8aa20d5984b9f1a0afd084f8cc00 -c 'nc -z 127.0.0.1 5671' >/dev/null 2>&1 && exit 0; sleep 1; done; exit 1"],
"verify": ["sh", "-c", "docker run --rm --network host --entrypoint sh nats@sha256:e4bf19f15fd3218814a4e3c9e0064e1334bd8aa20d5984b9f1a0afd084f8cc00 -c 'nc -z 127.0.0.1 5671'"]
},
{
"id": "control-plane",
+48 -2
View File
@@ -92,6 +92,11 @@ type Outcome struct {
// reads is, for a container, the digest of each file it was created reading, by path — so
// the next apply can say which one changed (novox/hq 04-ISSUES/103).
reads map[string]string
// busySince is, for a container left running because it said busy, when the first apply that would
// have replaced it ran (novox/hq ADR 0335): kept in its record, so the bound holds across passes and
// across a restart of the node-engine. busy is what the report says of it.
busySince *time.Time
busy *BusyWait
}
// Report is what an apply did, in the order it did it.
@@ -108,6 +113,20 @@ type Report struct {
// 224): a machine whose server is stopped at 03:31 because its collector is running is
// working, not broken, and the report is where that difference is said.
Windows []Window `json:"windows,omitempty"`
// Busy is every container this apply left running because it said busy, or replaced after it had
// waited (novox/hq ADR 0335): a machine whose walk waits for it says why, and how it ended.
Busy []BusyWait `json:"busy,omitempty"`
}
// Waiting says whether a container of this apply waits to be replaced because it is busy: what makes the
// node-engine ask again sooner than its reconcile (novox/hq ADR 0335 §2).
func (r Report) Waiting() bool {
for _, b := range r.Busy {
if b.State == BusyWaiting {
return true
}
}
return false
}
// Changed reports whether anything about the machine actually moved. An apply that changed
@@ -118,7 +137,10 @@ func (r Report) Changed() bool {
// whose account's manager is not running, which nothing moved either (novox/hq ADR 0177);
// held still is a container a maintenance window stopped, which the apply did not touch
// (novox/hq issue 224).
if o.Action != "unchanged" && o.Action != "held" && o.Action != "waiting" && o.Action != heldStill {
// waits busy is a container left running because it said busy (novox/hq ADR 0335), which nothing
// moved either.
if o.Action != "unchanged" && o.Action != "held" && o.Action != "waiting" && o.Action != heldStill &&
o.Action != waitsBusy {
return true
}
}
@@ -987,6 +1009,7 @@ func ApplyMindingWindows(
KeptMode: keptMode,
KeptOwner: keptOwner,
Reads: outcome.reads,
BusySince: outcome.busySince,
Stateless: outcome.stateless,
Scope: outcome.scope,
User: outcome.user,
@@ -1046,7 +1069,10 @@ func ApplyMindingWindows(
Action: "forgotten", Detail: detail})
log(fmt.Sprintf(" forgotten %s (%s): %s", from.ID, from.Target, detail))
}
if outcome.Action == heldStill {
if outcome.busy != nil {
report.Busy = append(report.Busy, *outcome.busy)
}
if outcome.Action == heldStill || outcome.Action == waitsBusy {
// Not changed: nothing about it moved, so nothing that restarts on it restarts.
log(fmt.Sprintf(" %s %s (%s): %s", outcome.Action, outcome.ID, outcome.Target, outcome.Detail))
} else if outcome.Action != "unchanged" {
@@ -2642,6 +2668,12 @@ func applyContainer(ctx context.Context, r *declaration.Container, run Runner,
// (novox/hq 04-ISSUES/009). Recreating is how a container gets restart-on, which a service
// already has.
reasons := restartedBy(r.RestartOn, changed)
// **A replacement owed since a busy wait began is still owed** (novox/hq ADR 0335): a restart-on trigger
// counts as changed only in the pass it changed, so a container left running then would otherwise read
// unchanged at the next pass, never be replaced, and drop from the report as if it had been.
if owed := owedSinceBusy(r, previous); owed != "" {
reasons = append(reasons, owed)
}
// What this container was created reading, as last recorded. By its declared id first, and by
// its NAME when that id has no record: the bundle's `store` becomes the postgres module's
@@ -2692,6 +2724,8 @@ func applyContainer(ctx context.Context, r *declaration.Container, run Runner,
// stop, so a container this apply found stopped by a window is one whose window it can see.
if win, held := in.windows.holding(r.Name, time.Now()); held {
out.reads = wasReading
// A replacement owed since a busy wait stays owed through the window (novox/hq ADR 0335).
out.busySince = previous.BusySince
out.Action = heldStill
out.Detail = fmt.Sprintf("held still by the maintenance window of %s since %s; left as it is, "+
"and converged by the first apply after the window closes (novox/hq issue 224)",
@@ -2707,6 +2741,15 @@ func applyContainer(ctx context.Context, r *declaration.Container, run Runner,
out.Action = "unchanged"
return out, nil
case existed:
// **A container that says it is busy is asked before it is replaced** (novox/hq ADR 0335 §2): left
// running while it says busy, within the bound, and replaced on the first pass it says idle. Only a
// running one: a stopped container holds no work in hand.
if r.Busy != nil && before.Running {
if waits := askBeforeReplacing(ctx, r, run, cri, previous, &out); waits {
out.reads = wasReading
return out, nil
}
}
if _, err := run(ctx, cri, "rm", "-f", r.Name); err != nil {
return out, fmt.Errorf("replacing container %s: %w", r.Name, err)
}
@@ -2807,6 +2850,9 @@ func applyContainer(ctx context.Context, r *declaration.Container, run Runner,
default:
out.Detail = "replaced; a container's configuration is fixed when it is created"
}
if out.busy != nil {
out.Detail += "; " + out.busy.Words()
}
}
return out, nil
}
+153
View File
@@ -0,0 +1,153 @@
package apply
import (
"context"
"fmt"
"strings"
"time"
"unicode/utf8"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/store"
)
// A container that says it is busy is replaced once it is idle, within a bound (novox/hq ADR 0335).
//
// **Asked, never waited on.** The apply asks the container once, and while it says busy leaves it running
// as it is and goes on with everything else of the declaration: the machine's lock is held for one ask,
// never for the wait. The wait is carried between passes in the container's record (store.Applied's
// BusySince), so the next pass — the node-engine asks again sooner than its reconcile while anything waits
// — asks again, and so does a node-engine started again in between: the bound runs from the first apply
// that would have replaced the container, whoever applied it.
// waitsBusy is the action of a container left running because it said busy (novox/hq ADR 0335 §2). Its own
// word rather than "unchanged": its declaration moved, and the machine is not yet what was declared.
const waitsBusy = "waits-busy"
// What a report says of a busy container (novox/hq ADR 0335 §2, §3).
const (
// BusyWaiting is a container left running because it said busy, waiting to be replaced.
BusyWaiting = "waiting"
// BusyReplacedIdle is a container replaced on the first pass it said idle, after it had waited.
BusyReplacedIdle = "replaced-when-idle"
// BusyReplacedAtBound is a container still busy at the bound, replaced anyway.
BusyReplacedAtBound = "replaced-at-bound"
)
// BusyWait is one container a busy command held, as a report says it.
type BusyWait struct {
// ID is the resource's id (`<module>.<resource>`), Container its runtime name.
ID string `json:"id"`
Container string `json:"container"`
// State is BusyWaiting, BusyReplacedIdle or BusyReplacedAtBound.
State string `json:"state"`
// Since is when the first apply that would have replaced it ran; Until when the bound replaces it.
Since time.Time `json:"since"`
Until time.Time `json:"until"`
// Said is the last output its busy command gave.
Said string `json:"said,omitempty"`
}
// Words is the wait in a sentence, for an outcome's detail and the console.
func (b BusyWait) Words() string {
since := b.Since.UTC().Format(time.RFC3339)
said := ""
if b.Said != "" {
said = "; its busy command last said: " + b.Said
}
switch b.State {
case BusyReplacedAtBound:
return fmt.Sprintf("replaced while still busy: it said busy since %s and the bound of %s ran out "+
"(novox/hq ADR 0335 §3)%s", since, declaration.BusyBound, said)
case BusyReplacedIdle:
return fmt.Sprintf("replaced once it said idle, after waiting since %s (novox/hq ADR 0335 §2)%s", since, said)
}
return fmt.Sprintf("busy: left running and replaced on the first pass it says idle, or at %s at the latest; "+
"waiting since %s (novox/hq ADR 0335 §2)%s", b.Until.UTC().Format(time.RFC3339), since, said)
}
// owedSinceBusy is the reason a container still owes the replacement it was left running for, or empty: its
// record keeps when its busy wait began, under the same name.
func owedSinceBusy(r *declaration.Container, previous store.Applied) string {
if previous.BusySince == nil || previous.Target != r.Name {
return ""
}
return "the replacement owed since its busy wait began at " + previous.BusySince.UTC().Format(time.RFC3339)
}
// busyClock is the clock the bound is read on. A test moves it.
var busyClock = time.Now
// busySaidMost is how much of a busy command's output a report carries: its last lines, enough to say what
// it held.
const busySaidMost = 400
// askBeforeReplacing asks a running container that declares busy whether it may be replaced now, and says
// in out what came of it. True when it waits: out is then the whole outcome of this pass. False when the
// caller replaces it — idle, or past the bound — with out carrying the wait it ended, if there was one.
func askBeforeReplacing(ctx context.Context, r *declaration.Container, run Runner, cri string,
previous store.Applied, out *Outcome) bool {
now := busyClock().UTC()
busy, said := askBusy(ctx, r, run, cri)
var since time.Time
waited := previous.BusySince != nil && previous.Target == r.Name
if waited {
since = previous.BusySince.UTC()
// A wait kept from a clock that has since gone back reads as starting now, never as later: the bound
// holds from no later than this pass.
if since.After(now) {
since = now
}
}
wait := BusyWait{ID: r.ID, Container: r.Name, Since: since, Said: said}
switch {
case !busy && !waited:
// Idle at the first ask: replaced at once, as a container that declares nothing is.
return false
case !busy:
wait.State, wait.Until = BusyReplacedIdle, since.Add(declaration.BusyBound)
out.busy = &wait
return false
}
if !waited {
since = now
wait.Since = now
}
wait.Until = since.Add(declaration.BusyBound)
if !now.Before(wait.Until) {
wait.State = BusyReplacedAtBound
out.busy = &wait
return false
}
wait.State = BusyWaiting
out.Action, out.Detail = waitsBusy, wait.Words()
out.busySince, out.busy = &since, &wait
return true
}
// askBusy runs a container's busy command inside it, by its shell, as an exec health check is run: busy is
// an exit of 0 and nothing else. An ask that could not be made, or did not answer within its timeout, is
// not busy: the bound is what keeps a container whose command never answers from being held.
func askBusy(ctx context.Context, r *declaration.Container, run Runner, cri string) (bool, string) {
ask, cancel := context.WithTimeout(ctx, r.Busy.Within())
defer cancel()
output, err := run(ask, cri, "exec", r.Name, "/bin/sh", "-c", r.Busy.Command)
said := strings.TrimSpace(output)
if err != nil && said == "" {
said = err.Error()
}
return err == nil, lastOf(said, busySaidMost)
}
// lastOf is the end of a text, at most n bytes, on one line.
func lastOf(s string, n int) string {
s = strings.Join(strings.Fields(s), " ")
if len(s) <= n {
return s
}
from := len(s) - n
for from < len(s) && !utf8.RuneStart(s[from]) {
from++
}
return "…" + s[from:]
}
+433
View File
@@ -0,0 +1,433 @@
package apply
import (
"context"
"errors"
"path/filepath"
"slices"
"strings"
"sync"
"testing"
"time"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/store"
)
// A container that says it is busy is replaced once it is idle, within a bound (novox/hq ADR 0335). The
// machine of these tests runs two containers, a build agent that declares `busy` and a web server that
// does not; every call that replaces a container is counted.
type aBusyMachine struct {
mu sync.Mutex
// containers is each container's name and the spec it was created with; running whether it runs.
containers map[string]string
busy bool // what the agent's busy command answers
said string // what it prints
asked int
calls []string
}
func newBusyMachine() *aBusyMachine {
return &aBusyMachine{containers: map[string]string{}}
}
func (m *aBusyMachine) run(_ context.Context, _ string, args ...string) (string, error) {
m.mu.Lock()
defer m.mu.Unlock()
switch args[0] {
case "info", "image":
return "27.0\n", nil
case "container": // inspect
name := args[len(args)-1]
spec, there := m.containers[name]
if !there {
return "", errors.New("no such container")
}
return "true\t" + spec, nil
case "exec":
m.asked++
m.calls = append(m.calls, "ask "+args[1])
if strings.Join(args[2:], " ") != "/bin/sh -c mesh-builder busy" {
return "", errors.New("asked something else: " + strings.Join(args[2:], " "))
}
if m.busy {
return m.said + "\n", nil
}
return m.said + "\n", errors.New("exited 1")
case "rm":
name := args[len(args)-1]
m.calls = append(m.calls, "rm "+name)
delete(m.containers, name)
case "run":
name := args[slices.Index(args, "--name")+1]
m.calls = append(m.calls, "create "+name)
for i, a := range args {
if a == "--label" && strings.HasPrefix(args[i+1], specLabel+"=") {
m.containers[name] = strings.TrimPrefix(args[i+1], specLabel+"=")
}
}
}
return "", nil
}
func (m *aBusyMachine) mark() int {
m.mu.Lock()
defer m.mu.Unlock()
return len(m.calls)
}
// since is every call since mark that replaced a container, or asked one.
func (m *aBusyMachine) since(mark int) []string {
m.mu.Lock()
defer m.mu.Unlock()
return append([]string(nil), m.calls[mark:]...)
}
func (m *aBusyMachine) answer(busy bool, said string) {
m.mu.Lock()
defer m.mu.Unlock()
m.busy, m.said = busy, said
}
// twoContainers is the declaration at a version: v moves the agent's image tag through its env, w the web's.
func twoContainers(t *testing.T, agent, web string) *declaration.Declaration {
t.Helper()
return parse(t, `{"declaration":1,"resources":[
{"id":"build-agent.server","type":"container","name":"mesh-build-agent","image":"`+pinned+`",
"env":{"V":"`+agent+`"},"busy":{"command":"mesh-builder busy","timeout":"5s"}},
{"id":"web.server","type":"container","name":"web","image":"`+pinned+`","env":{"V":"`+web+`"}}
]}`)
}
// atClock moves the bound's clock for one test.
func atClock(t *testing.T, at *time.Time) {
t.Helper()
was := busyClock
busyClock = func() time.Time { return *at }
t.Cleanup(func() { busyClock = was })
}
func busyOf(r Report, id string) (BusyWait, bool) {
for _, b := range r.Busy {
if b.ID == id {
return b, true
}
}
return BusyWait{}, false
}
func applyBusy(t *testing.T, m *aBusyMachine, d *declaration.Declaration, known store.State) (Report, store.State) {
t.Helper()
report, state, err := Apply(context.Background(), archHost(t), d, known, store.OriginDeclared, m.run, nil, nil)
if err != nil {
t.Fatalf("the apply failed: %v", err)
}
return report, state
}
// The decision's first check: an apply that would replace a container whose busy command exits 0 leaves it
// running, applies the rest, and replaces it on the first pass the command exits non-zero.
func TestABusyContainerIsLeftRunningAndReplacedOnTheFirstPassItSaysIdle(t *testing.T) {
now := time.Date(2026, 10, 13, 1, 0, 0, 0, time.UTC)
atClock(t, &now)
m := newBusyMachine()
_, state := applyBusy(t, m, twoContainers(t, "1", "1"), store.State{})
// A merge moves both: the agent holds a build, the web server holds nothing.
m.answer(true, "building novox/mesh-controller (check of #280) since 00:58")
mark := m.mark()
report, state := applyBusy(t, m, twoContainers(t, "2", "2"), state)
if got := m.since(mark); !slices.Equal(got, []string{"ask mesh-build-agent", "rm web", "create web"}) {
t.Fatalf("a busy agent was not left running while the rest applied: %v", got)
}
o := outcomeOf(report, "build-agent.server")
if o.Action != waitsBusy {
t.Fatalf("a busy container was reported %q, want %q: %+v", o.Action, waitsBusy, o)
}
for _, says := range []string{"busy", "first pass it says idle", "check of #280", "ADR 0335"} {
if !strings.Contains(o.Detail, says) {
t.Errorf("the outcome does not say %q: %s", says, o.Detail)
}
}
if (Report{Outcomes: []Outcome{o}}).Changed() {
t.Errorf("a container left running reads as a change: %+v", o)
}
b, said := busyOf(report, "build-agent.server")
if !said || b.State != BusyWaiting || !b.Since.Equal(now) || !b.Until.Equal(now.Add(15*time.Minute)) ||
b.Container != "mesh-build-agent" || !strings.Contains(b.Said, "check of #280") {
t.Fatalf("the report does not say the agent waits, since when and until when: %+v", report.Busy)
}
if !report.Waiting() {
t.Error("a report with a container waiting does not say so")
}
if rec, _ := state.Find("build-agent.server"); rec.BusySince == nil || !rec.BusySince.Equal(now) {
t.Fatalf("the wait is not kept in the container's record: %+v", rec)
}
// Asked again a minute later, still busy: still left running, the wait counted from the first apply.
now = now.Add(time.Minute)
mark = m.mark()
report, state = applyBusy(t, m, twoContainers(t, "2", "2"), state)
if got := m.since(mark); !slices.Equal(got, []string{"ask mesh-build-agent"}) {
t.Fatalf("a second pass while busy did more than ask: %v", got)
}
if b, _ := busyOf(report, "build-agent.server"); b.State != BusyWaiting || !b.Since.Equal(now.Add(-time.Minute)) {
t.Fatalf("the second pass does not keep the wait from the first apply: %+v", report.Busy)
}
// The build ends: the next pass replaces it, and says it waited.
now = now.Add(time.Minute)
m.answer(false, "idle")
mark = m.mark()
report, state = applyBusy(t, m, twoContainers(t, "2", "2"), state)
if got := m.since(mark); !slices.Equal(got, []string{"ask mesh-build-agent", "rm mesh-build-agent", "create mesh-build-agent"}) {
t.Fatalf("the agent was not replaced on the first pass it said idle: %v", got)
}
if o := outcomeOf(report, "build-agent.server"); o.Action != "updated" || !strings.Contains(o.Detail, "once it said idle") {
t.Errorf("the replacement does not say it waited: %+v", o)
}
if b, _ := busyOf(report, "build-agent.server"); b.State != BusyReplacedIdle {
t.Errorf("the report does not say the agent was replaced when idle: %+v", report.Busy)
}
if report.Waiting() {
t.Error("a report with nothing waiting says something waits")
}
if rec, _ := state.Find("build-agent.server"); rec.BusySince != nil {
t.Errorf("the wait is still kept after the replacement: %+v", rec)
}
// And the pass after that has nothing to say about it.
report, _ = applyBusy(t, m, twoContainers(t, "2", "2"), state)
if len(report.Busy) != 0 {
t.Errorf("a container replaced is still said: %+v", report.Busy)
}
}
// An idle container is replaced at once, as one that declares nothing is: asked once, nothing waits.
func TestAnIdleContainerIsReplacedAtOnce(t *testing.T) {
m := newBusyMachine()
_, state := applyBusy(t, m, twoContainers(t, "1", "1"), store.State{})
m.answer(false, "")
mark := m.mark()
report, _ := applyBusy(t, m, twoContainers(t, "2", "1"), state)
if got := m.since(mark); !slices.Equal(got, []string{"ask mesh-build-agent", "rm mesh-build-agent", "create mesh-build-agent"}) {
t.Fatalf("an idle agent was not replaced at once: %v", got)
}
if len(report.Busy) != 0 {
t.Errorf("an idle container replaced at once is said as a wait: %+v", report.Busy)
}
// And a container whose declaration did not move is never asked.
mark = m.mark()
applyBusy(t, m, twoContainers(t, "2", "1"), state)
if got := m.since(mark); slices.Contains(got, "ask mesh-build-agent") {
t.Errorf("a container nothing replaces was asked: %v", got)
}
}
// Past the bound it is replaced anyway, the report carrying the command's last output — and the bound holds
// across a restart of the node-engine: the wait is read back from the state on disk.
func TestABusyContainerIsReplacedAtTheBoundEvenAfterARestart(t *testing.T) {
now := time.Date(2026, 10, 13, 1, 0, 0, 0, time.UTC)
atClock(t, &now)
m := newBusyMachine()
_, state := applyBusy(t, m, twoContainers(t, "1", "1"), store.State{})
m.answer(true, "building novox/mesh-host (a check that never ends)")
_, state = applyBusy(t, m, twoContainers(t, "2", "1"), state)
// The node-engine stops and starts again: what it knows is what it saved.
path := filepath.Join(t.TempDir(), "state.json")
if err := store.Save(path, state); err != nil {
t.Fatal(err)
}
reloaded, err := store.Load(path)
if err != nil {
t.Fatal(err)
}
now = now.Add(14 * time.Minute)
mark := m.mark()
report, reloaded := applyBusy(t, m, twoContainers(t, "2", "1"), reloaded)
if got := m.since(mark); slices.Contains(got, "rm mesh-build-agent") {
t.Fatalf("a container busy within the bound was replaced after a restart: %v", got)
}
if b, _ := busyOf(report, "build-agent.server"); b.State != BusyWaiting || !b.Since.Equal(now.Add(-14*time.Minute)) {
t.Fatalf("a restart started the wait again: %+v", report.Busy)
}
now = now.Add(time.Minute)
mark = m.mark()
report, _ = applyBusy(t, m, twoContainers(t, "2", "1"), reloaded)
if got := m.since(mark); !slices.Equal(got, []string{"ask mesh-build-agent", "rm mesh-build-agent", "create mesh-build-agent"}) {
t.Fatalf("a container still busy at the bound was not replaced: %v", got)
}
b, _ := busyOf(report, "build-agent.server")
if b.State != BusyReplacedAtBound || !strings.Contains(b.Said, "never ends") {
t.Fatalf("the report does not say it replaced a busy container, with its last output: %+v", report.Busy)
}
if o := outcomeOf(report, "build-agent.server"); !strings.Contains(o.Detail, "still busy") ||
!strings.Contains(o.Detail, "never ends") {
t.Errorf("the outcome does not say a busy container was replaced, with what it said: %s", o.Detail)
}
}
// While one container waits, a send that changes another module on the same machine is applied at once: the
// apply returns after one ask, and holds nothing for the wait.
func TestASendToAnotherModuleIsAppliedWhileOneWaits(t *testing.T) {
now := time.Date(2026, 10, 13, 1, 0, 0, 0, time.UTC)
atClock(t, &now)
m := newBusyMachine()
_, state := applyBusy(t, m, twoContainers(t, "1", "1"), store.State{})
m.answer(true, "building")
_, state = applyBusy(t, m, twoContainers(t, "2", "1"), state)
// The next send moves only the web server.
mark := m.mark()
done := make(chan struct{})
var report Report
go func() {
defer close(done)
report, _ = applyBusy(t, m, twoContainers(t, "2", "2"), state)
}()
select {
case <-done:
case <-time.After(5 * time.Second):
t.Fatal("an apply while a container waits did not return")
}
got := m.since(mark)
if !slices.Contains(got, "create web") || slices.Contains(got, "rm mesh-build-agent") {
t.Fatalf("the other module's send was not applied at once beside the wait: %v", got)
}
if o := outcomeOf(report, "web.server"); o.Action != "updated" {
t.Errorf("the web server was reported %q: %+v", o.Action, o)
}
}
// A container's busy is held to its shape where it arrives.
func TestBusyIsReadAndHeldToItsShape(t *testing.T) {
d := twoContainers(t, "1", "1")
c := d.Resources[0].(*declaration.Container)
if c.Busy == nil || c.Busy.Command != "mesh-builder busy" || c.Busy.Within() != 5*time.Second {
t.Fatalf("busy was not read: %+v", c.Busy)
}
for name, raw := range map[string]string{
"no command": `"busy":{"command":" "}`,
"a long timeout": `"busy":{"command":"x","timeout":"5m"}`,
"not a duration": `"busy":{"command":"x","timeout":"soon"}`,
"an unknown key": `"busy":{"command":"x","every":"1m"}`,
"on a step": `"run-once":true,"busy":{"command":"x"}`,
} {
_, err := declaration.Parse([]byte(`{"declaration":1,"resources":[{"id":"a.b","type":"container",` +
`"name":"b","image":"` + pinned + `",` + raw + `}]}`))
if err == nil {
t.Errorf("%s: a busy of that shape was accepted", name)
}
}
}
// A replacement owed to a restart-on trigger is still owed after the wait: a trigger that is no file or
// directory (a service, an identity's or a library's change) is in no container's spec and counts as changed only
// in the pass it changed, so the container must not read unchanged at the next pass and drop from the report as if
// replaced (novox/hq ADR 0335).
func TestAReplacementOwedToARestartTriggerIsMadeOnceIdle(t *testing.T) {
now := time.Date(2026, 10, 13, 1, 0, 0, 0, time.UTC)
atClock(t, &now)
d := parse(t, `{"declaration":1,"resources":[
{"id":"build-agent.server","type":"container","name":"mesh-build-agent","image":"`+pinned+`",
"restart-on":["identity:build-agent"],"busy":{"command":"mesh-builder busy"}}
]}`)
r := d.Resources[0].(*declaration.Container)
in := inputs{declares: map[string]string{}}
m := newBusyMachine()
m.containers["mesh-build-agent"] = containerSpec(r, in)
ctx := context.Background()
// The trigger moved this pass, and the agent holds a build: left running.
m.answer(true, "building")
out, err := applyContainer(ctx, r, m.run, map[string]bool{"identity:build-agent": true}, in, store.Applied{})
if err != nil || out.Action != waitsBusy || out.busySince == nil {
t.Fatalf("a busy container was not left running for its trigger: %+v %v", out, err)
}
record := store.Applied{ID: r.ID, Target: r.Name, BusySince: out.busySince}
// The next pass: nothing triggers now, and the replacement is still owed.
now = now.Add(time.Minute)
m.answer(false, "idle")
mark := m.mark()
out, err = applyContainer(ctx, r, m.run, map[string]bool{}, in, record)
if err != nil {
t.Fatal(err)
}
if got := m.since(mark); !slices.Equal(got, []string{"ask mesh-build-agent", "rm mesh-build-agent", "create mesh-build-agent"}) {
t.Fatalf("the replacement owed to the trigger was dropped: %v (%+v)", got, out)
}
if out.busy == nil || out.busy.State != BusyReplacedIdle || out.busySince != nil {
t.Fatalf("the replacement does not say it was made once idle: %+v", out)
}
}
// A wait kept from a clock that later went back is never longer than the bound from now.
func TestAWaitFromAClockThatWentBackIsClampedToNow(t *testing.T) {
now := time.Date(2026, 10, 13, 1, 0, 0, 0, time.UTC)
atClock(t, &now)
m := newBusyMachine()
_, state := applyBusy(t, m, twoContainers(t, "1", "1"), store.State{})
m.answer(true, "building")
_, state = applyBusy(t, m, twoContainers(t, "2", "1"), state)
now = now.Add(-time.Hour) // the clock went back an hour
report, _ := applyBusy(t, m, twoContainers(t, "2", "1"), state)
b, _ := busyOf(report, "build-agent.server")
if b.State != BusyWaiting || b.Since.After(now) || !b.Until.Equal(now.Add(15*time.Minute)) {
t.Fatalf("a wait from a later clock is not clamped to now: %+v (now %s)", b, now)
}
}
// An ask that fails — no shell in the image, a timeout, a paused container — is not busy: replaced at once.
func TestAnAskThatFailsIsIdle(t *testing.T) {
m := newBusyMachine()
_, state := applyBusy(t, m, twoContainers(t, "1", "1"), store.State{})
failing := func(ctx context.Context, name string, args ...string) (string, error) {
if args[0] == "exec" {
m.mu.Lock()
m.calls = append(m.calls, "ask "+args[1])
m.mu.Unlock()
return "", errors.New("OCI runtime exec failed: exec: \"/bin/sh\": no such file or directory")
}
return m.run(ctx, name, args...)
}
mark := m.mark()
report, _, err := Apply(context.Background(), archHost(t), twoContainers(t, "2", "1"), state,
store.OriginDeclared, failing, nil, nil)
if err != nil {
t.Fatal(err)
}
if got := m.since(mark); !slices.Equal(got, []string{"ask mesh-build-agent", "rm mesh-build-agent", "create mesh-build-agent"}) {
t.Fatalf("an ask that failed held the container: %v", got)
}
if len(report.Busy) != 0 {
t.Errorf("an ask that failed is said as a wait: %+v", report.Busy)
}
}
// A stopped container holds no work: it is replaced without being asked.
func TestAStoppedContainerIsNotAsked(t *testing.T) {
m := newBusyMachine()
_, state := applyBusy(t, m, twoContainers(t, "1", "1"), store.State{})
m.answer(true, "would say busy")
stopped := func(ctx context.Context, name string, args ...string) (string, error) {
if args[0] == "container" && args[len(args)-1] == "mesh-build-agent" {
out, err := m.run(ctx, name, args...)
return strings.Replace(out, "true\t", "false\t", 1), err
}
return m.run(ctx, name, args...)
}
mark := m.mark()
if _, _, err := Apply(context.Background(), archHost(t), twoContainers(t, "2", "1"), state,
store.OriginDeclared, stopped, nil, nil); err != nil && !strings.Contains(err.Error(), "not running") {
t.Fatal(err)
}
got := m.since(mark)
if slices.Contains(got, "ask mesh-build-agent") || !slices.Contains(got, "rm mesh-build-agent") {
t.Fatalf("a stopped container was asked, or not replaced: %v", got)
}
}
+60 -26
View File
@@ -36,6 +36,7 @@ import (
"time"
"github.com/novox/mesh-host/internal/firewall"
"github.com/novox/mesh-host/internal/system"
"github.com/novox/mesh-host/internal/tunnel"
)
@@ -173,6 +174,10 @@ type Options struct {
// ToolsSource is where the shared base is built from. Everything with code of its own
// compiles against it, so it is the first thing the mesh builds for itself.
ToolsSource Source
// ToolsGoSource is where the Go toolchain is built from (`mesh-tools-go`, ADR 0142): every Go
// bundle compiles in it, and genesis's own modules have Go code (issue 616). Empty means the
// repository beside ToolsSource named mesh-tools-go, as the mesh keeps it.
ToolsGoSource Source
// CatalogSource is where the catalogue REPOSITORY is, for building its modules. The catalogue
// CHECKOUT (Catalogue, above) says what a module is; this is where a builder clones it.
CatalogSource Source
@@ -292,12 +297,14 @@ type Result struct {
Agent string `json:"host-agent,omitempty"`
// Registry is the mesh's own artifact store, once it answers.
Registry string `json:"registry,omitempty"`
RegistryReplied int `json:"registry-replied,omitempty"`
RegistryKnown bool `json:"registry-already-registered,omitempty"`
RegistryRunning string `json:"registry-container,omitempty"`
PublishedAs string `json:"control-plane-image,omitempty"`
PublishedAlready bool `json:"control-plane-image-already-published,omitempty"`
Registry string `json:"registry,omitempty"`
RegistryReplied int `json:"registry-replied,omitempty"`
RegistryKnown bool `json:"registry-already-registered,omitempty"`
RegistryRunning string `json:"registry-container,omitempty"`
// RegistryTrusted are the names the runtime was told to trust the registry by, before the push.
RegistryTrusted []string `json:"registry-trusted,omitempty"`
PublishedAs string `json:"control-plane-image,omitempty"`
PublishedAlready bool `json:"control-plane-image-already-published,omitempty"`
// Permanent is the control plane as an ordinary module.
Permanent string `json:"permanent-container,omitempty"`
@@ -555,7 +562,9 @@ func Run(ctx context.Context, o Options, d Deps, say func(string)) (Result, erro
// Named rather than attempted. Everything from step 6 on is a conversation with a
// control plane that a dry run has not raised, so there is nothing to ask and nothing
// honest to report about the answers.
say(" would then enrol " + o.Node + ", install the registry from " +
say(" would then enrol " + o.Node + ", have the runtime trust the registry as " +
strings.Join(RegistryNames(o), ", ") + ",")
say(" install the registry from " +
o.Catalogue + ", push the control plane's image into it,")
say(" reinstall the control plane as a module, and drop " +
rewritten.TempName)
@@ -654,26 +663,10 @@ func Run(ctx context.Context, o Options, d Deps, say func(string)) (Result, erro
return result, failed(StepOperator, err)
}
// ---- 7. registry ----------------------------------------------------------------------
say("registry — somewhere for this mesh to keep its own images")
registry, err := InstallRegistry(ctx, o, d, temporary, say)
result.Registry, result.RegistryRunning = registry.Address, registry.Container
result.RegistryKnown, result.RegistryReplied = registry.Known, registry.Answered
// ---- 7. registry, 8. publish -----------------------------------------------------------
published, err := RegistryAndPublish(ctx, o, d, sys, temporary, controlPlaneImage, &result, say)
if err != nil {
return result, failed(StepRegistry, err)
}
// ---- 8. publish -----------------------------------------------------------------------
say("publish — the control plane's image gets its first manifest digest")
// **The image this mesh built, not the one the installer carried.** The carried one is the
// builder; publishing it here would put a builder in the registry under the control plane's
// name and install it as the control plane — which is exactly what happened the first time
// this ran, and presented as a control plane that started, printed a builder's usage, and
// exited cleanly over and over.
published, err := PublishControlPlane(ctx, o, d, controlPlaneImage, say)
result.PublishedAs, result.PublishedAlready = published.Reference, published.Already
if err != nil {
return result, failed(StepPublish, err)
return result, err
}
// ---- 9. control plane -----------------------------------------------------------------
@@ -803,3 +796,44 @@ func Run(ctx context.Context, o Options, d Deps, say func(string)) (Result, erro
return result, nil
}
// RegistryAndPublish is steps 7 and 8: the runtime trusts the mesh's registry, the registry is
// installed and answers, and the control plane's image is pushed into it.
//
// **One function so the order is held by a test** (novox/hq issue 92). The runtime refuses plain
// HTTP to a registry it has not been told to trust, and the module that tells it is applied long
// after the push — so the trust is written and confirmed here, before anything is tagged or pushed.
// A refusal names the step it happened in.
func RegistryAndPublish(ctx context.Context, o Options, d Deps, sys system.System, temporary controlPlane,
controlPlaneImage string, result *Result, say func(string)) (Published, error) {
// ---- 7. registry ----------------------------------------------------------------------
say("registry — somewhere for this mesh to keep its own images")
// Trusted by the runtime before anything is pushed to it, by every name the mesh will use for
// it here: the module that writes this trust is applied long after the push (novox/hq issue 92).
trusted, err := TrustTheRegistry(ctx, o, sys, d.Run, say)
result.RegistryTrusted = trusted
if err != nil {
return Published{}, failed(StepRegistry, err)
}
registry, err := InstallRegistry(ctx, o, d, temporary, say)
result.Registry, result.RegistryRunning = registry.Address, registry.Container
result.RegistryKnown, result.RegistryReplied = registry.Known, registry.Answered
if err != nil {
return Published{}, failed(StepRegistry, err)
}
// ---- 8. publish -----------------------------------------------------------------------
say("publish — the control plane's image gets its first manifest digest")
// **The image this mesh built, not the one the installer carried.** The carried one is the
// builder; publishing it here would put a builder in the registry under the control plane's
// name and install it as the control plane — which is exactly what happened the first time
// this ran, and presented as a control plane that started, printed a builder's usage, and
// exited cleanly over and over.
published, err := PublishControlPlane(ctx, o, d, controlPlaneImage, say)
result.PublishedAs, result.PublishedAlready = published.Reference, published.Already
if err != nil {
return published, failed(StepPublish, err)
}
return published, nil
}
-21
View File
@@ -38,27 +38,6 @@ func recording(t *testing.T) (*controlRecorder, controlPlane) {
return c, controlPlane{container: "temp-mesh-controller", run: c.run, timeout: time.Second}
}
func TestTheBuildersPackageRegistryPortIsASettingAndNotTheManifest(t *testing.T) {
c, control := recording(t)
o := Options{Node: "anchor", Ports: FoundationPorts{Packages: 3100}, Wait: time.Second}
if _, err := installModule(context.Background(), o, control, BuilderModule,
[]byte(theBuildersBinding), quietly); err != nil {
t.Fatal(err)
}
if got := c.settings[BuilderModule+"-settings.json"]; got != `{"serves":{"port":3100}}` {
t.Errorf("the builder was told %q about the package registry's port", got)
}
// The manifest reaches the mesh as the catalogue wrote it. A port rewritten into it here is a
// port the next registration from the catalogue silently takes back.
if got := c.settings[BuilderModule+"-module.json"]; got != theBuildersBinding {
t.Errorf("the installer changed the builder's manifest:\n%s", got)
}
set, push := c.index("settings set "+BuilderModule), c.index("push anchor")
if set < 0 || push < 0 || set > push {
t.Fatalf("the port was not set before the push that raises the builder: %v", c.told)
}
}
func TestTheBuilderOnTheDefaultPackagesPortIsToldNothing(t *testing.T) {
c, control := recording(t)
o := Options{Node: "anchor", Wait: time.Second}
+179
View File
@@ -0,0 +1,179 @@
package bootstrap
import (
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"encoding/pem"
"errors"
"fmt"
"os"
"regexp"
"strconv"
"strings"
"time"
"github.com/novox/mesh-host/internal/declaration"
)
// Adopting the bus is the mesh's planned step for replacing it (novox/hq ADR 0333 §3, ADR 0236).
//
// The adoption replaces the bus's image once: the module's built image takes the place of the upstream
// one the bundle raised, on the same streams, users and certificate. The controller refuses a push that
// would replace the bus — "a planned step: `bus upgrade --why …` snapshots its streams first and checks
// them after" — so genesis stopped at step 18 (issue 590). The installer takes that step: it snapshots the
// streams itself, because at genesis no backup holder exists to be asked, and hands the snapshot to
// `bus upgrade`, which sends the machine, opens `bus-maintenance`, and ends the step done only once the
// bus answers healthy again — every stream, every durable consumer, a round trip (H-bus). The installer
// waits for that end.
// adoptionWhy is what the step is recorded as, for the record of hand acts and the bus's step.
const adoptionWhy = "genesis adopts its bus as the nats module (ADR 0333)"
// Where the installer keeps the streams' snapshot before the adoption, and the credential the snapshot
// program reads while it runs. Beside genesis's own credentials; variables so a test writes elsewhere.
var (
busSnapshotFile = "/var/lib/mesh-genesis/bus-before-adoption.tar"
busSnapshotCredential = "/var/lib/mesh-genesis/bus-snapshot.credential"
busCertificateFile = "/var/lib/mesh-broker-tls/tls.crt"
busControllerSecret = BusControllerFile
// busStepWithin bounds the wait for the controller to say the step ended: its own bound (15 min,
// mesh-controller busStepBound), and its doctor's next look after that.
busStepWithin = 17 * time.Minute
)
// adoptTheBus sends the nats module to this machine through the bus's planned step. A bus already on the
// module's image is the module's: it is sent like any module, and nothing is replaced.
func adoptTheBus(ctx context.Context, o Options, control controlPlane, bus *declaration.Container,
say func(string)) error {
if running := runningImage(ctx, control, bus.Name); running != "" && running != bus.Image {
say(" already adopted " + bus.Name + " — it runs the module's image")
_, err := pushNode(ctx, o, control, say)
return err
}
image, err := theModulesBusImage(ctx, o, control)
if err != nil {
return err
}
if err := snapshotTheBus(ctx, o, control, image, say); err != nil {
return err
}
say(" restarting " + bus.Name + " — once, onto the " + BusModule + " module's image, as the bus's " +
"planned step; its streams, users and certificate stay where they are (novox/hq ADR 0333 §3, ADR 0236)")
if _, err := control.within(2*o.Wait).tell(ctx, "bus", "upgrade", "--why", adoptionWhy,
"--reversible", "--snapshot-taken", busSnapshotFile); err != nil {
return fmt.Errorf("the bus's planned step was refused, so the bus is not replaced: %w", err)
}
return nil
}
// meshBuiltImage is an image the mesh built, as it records it: `artifact-store://<module>/<artifact>@sha256:…`.
var meshBuiltImage = regexp.MustCompile(`artifact-store://([a-z0-9][a-z0-9-]*/[a-z0-9][a-z0-9-]*@sha256:[0-9a-f]{64})`)
// theModulesBusImage is the nats module's server image, as the build just made it, pulled from this
// mesh's registry: the image that carries the snapshot program.
func theModulesBusImage(ctx context.Context, o Options, control controlPlane) (string, error) {
said, err := control.tell(ctx, "builds", BusModule, "-n", "1")
if err != nil {
return "", err
}
for _, m := range meshBuiltImage.FindAllStringSubmatch(said, -1) {
if strings.HasPrefix(m[1], BusModule+"/") {
return strings.TrimSuffix(o.Registry, "/") + "/" + m[1], nil
}
}
return "", fmt.Errorf("the mesh says no image it built for the %s module, so its snapshot program cannot "+
"be run before the bus is replaced:\n%s", BusModule, indent(strings.TrimSpace(said)))
}
// snapshotTheBus writes every stream of the running bus to busSnapshotFile with the module's own snapshot
// program (mesh-catalog modules/nats, ADR 0235), as the controller's account, pinned to the bus's
// certificate. The credential is written for the program's one run and removed after it.
func snapshotTheBus(ctx context.Context, o Options, control controlPlane, image string, say func(string)) error {
password, err := readCredentialFile(busControllerSecret)
if err != nil {
return fmt.Errorf("the controller's bus credential is not at %s, so the streams cannot be snapshotted "+
"before the bus is replaced: %w", busControllerSecret, err)
}
fingerprint, err := certificateFingerprint(busCertificateFile)
if err != nil {
return err
}
port := o.Ports.orDefaults().Bus
server := "tls://127.0.0.1:" + strconv.Itoa(port)
credential, err := json.Marshal(map[string]string{"url": server, "fingerprint": fingerprint,
"user": busControllerUser, "password": password})
if err != nil {
return err
}
if err := os.WriteFile(busSnapshotCredential, credential, 0o600); err != nil {
return fmt.Errorf("the snapshot's credential could not be written: %w", err)
}
defer os.Remove(busSnapshotCredential)
script := "docker run --rm -i --network host --entrypoint /usr/local/bin/mesh-nats-snapshot " + image +
" snapshot --server " + server + " --credential - < " + busSnapshotCredential + " > " + busSnapshotFile +
".partial && mv " + busSnapshotFile + ".partial " + busSnapshotFile
running, cancel := context.WithTimeout(ctx, busStepWithin)
defer cancel()
if out, err := control.run(running, "sh", "-c", script); err != nil {
_, _ = control.run(ctx, "rm", "-f", busSnapshotFile+".partial")
return fmt.Errorf("the bus's streams could not be snapshotted, so the bus is not replaced: %w\n%s",
err, indent(strings.TrimSpace(out)))
}
say(" snapshotted every stream of the bus, at " + busSnapshotFile + " (its module's snapshot program)")
return nil
}
// certificateFingerprint is the pin of the bus's certificate: sha256 of its DER, as every machine pins it.
func certificateFingerprint(path string) (string, error) {
raw, err := os.ReadFile(path)
if err != nil {
return "", fmt.Errorf("the bus's certificate is not at %s, so it cannot be pinned for the snapshot: %w", path, err)
}
block, _ := pem.Decode(raw)
if block == nil {
return "", fmt.Errorf("%s holds no PEM certificate", path)
}
sum := sha256.Sum256(block.Bytes)
return "sha256:" + hex.EncodeToString(sum[:]), nil
}
// waitForTheBusStep waits until the controller says the bus's planned step ended: done once the bus
// answered healthy — every stream, every durable consumer, a round trip — or failed, with its snapshot as
// the way back. The step's own bound is the controller's; this waits a little past it.
func waitForTheBusStep(ctx context.Context, o Options, control controlPlane, say func(string)) error {
deadline := time.Now().Add(busStepWithin)
var last string
for {
said, err := control.tell(ctx, "bus")
if err == nil {
for _, line := range strings.Split(said, "\n") {
if strings.HasPrefix(strings.TrimSpace(line), "last step ") {
last = strings.TrimSpace(line)
}
}
switch {
case strings.Contains(last, ": done at "):
say(" bus step done " + last)
return nil
case strings.Contains(last, ": failed at "):
return fmt.Errorf("the bus's planned step ended failed: %s\n%s", last, indent(strings.TrimSpace(said)))
}
}
if !time.Now().Before(deadline) {
break
}
select {
case <-ctx.Done():
return ctx.Err()
case <-time.After(answerEvery):
}
}
if last == "" {
return errors.New("the controller says no bus step after the adoption was sent; `mesh-controller bus` says how it stands")
}
return fmt.Errorf("the bus's planned step did not end within %s: %s", busStepWithin, last)
}
+135
View File
@@ -0,0 +1,135 @@
package bootstrap
import (
"context"
"encoding/json"
"os"
"strings"
"testing"
"time"
)
// Defends novox/hq issue 590 at its broker step (ADR 0333 §3, ADR 0236): the adoption replaces the bus's
// image, and the mesh replaces its bus only as a planned step. A plain push was refused — "sending anchor
// would replace the bus … a planned step: `bus upgrade --why …`" — and genesis stopped at step 18.
func TestTheBrokerStepAdoptsTheBusThroughItsPlannedStep(t *testing.T) {
catalogue := aCatalogue(t, map[string]string{BusModule: aBusModule(theModulesLayout)})
rewritten, err := Rewrite(theRealBundle(t), held)
if err != nil {
t.Fatal(err)
}
bundleImage := containerNamed(rewritten.Declaration, "mesh-broker-nats").Image
upgraded := false
var credential []byte
runtime := &asked{}
agrees := aMeshThatAgrees(nil)
runtime.answer = func(name string, args []string) (string, error) {
joined := strings.Join(args, " ")
switch {
case name == "sh" && strings.Contains(joined, "mesh-nats-snapshot"):
credential, _ = os.ReadFile(busSnapshotCredential) // as the program reads it
case strings.Contains(joined, "bus upgrade --why"):
upgraded = true
case strings.Contains(joined, "inspect --format {{.Config.Image}} mesh-broker-nats"):
if upgraded {
return theModulesImage + "\n", nil
}
return bundleImage + "\n", nil
}
return agrees(name, args)
}
_, fetch := aBusThatAdopts()
o := installing(t, catalogue)
o.CatalogSource = Source{Repository: "git://forge/mesh-catalog.git"}
o.Wait = 5 * time.Second
said, err := busStepWith(t, o, runtime, fetch)
if err != nil {
t.Fatal(err)
}
snapshot, upgrade := -1, -1
for i, c := range runtime.commands {
switch {
case strings.HasPrefix(c, "sh -c") && strings.Contains(c, "mesh-nats-snapshot"):
snapshot = i
for _, want := range []string{"127.0.0.1:5000/nats/server@sha256:" + strings.Repeat("4", 64),
"snapshot --server tls://127.0.0.1:5671 --credential - < " + busSnapshotCredential,
"> " + busSnapshotFile + ".partial && mv " + busSnapshotFile + ".partial " + busSnapshotFile} {
if !strings.Contains(c, want) {
t.Errorf("the snapshot is not taken with %q: %s", want, c)
}
}
case strings.Contains(c, "/mesh-controller bus upgrade"):
upgrade = i
if want := "bus upgrade --why " + adoptionWhy + " --reversible --snapshot-taken " + busSnapshotFile; !strings.HasSuffix(c, want) {
t.Errorf("the planned step is asked as %q; want it to end %q", c, want)
}
case strings.HasSuffix(c, "/mesh-controller push anchor"):
t.Errorf("the bus's module was sent by a plain push, which the mesh refuses for the bus: %v", runtime.commands)
}
}
if snapshot < 0 || upgrade < 0 || snapshot > upgrade {
t.Fatalf("the streams were not snapshotted before the bus's planned step (snapshot %d, upgrade %d): %v",
snapshot, upgrade, runtime.commands)
}
var cred map[string]string
if err := json.Unmarshal(credential, &cred); err != nil {
t.Fatalf("the snapshot program was given no credential: %v", err)
}
fingerprint, _ := certificateFingerprint(busCertificateFile)
if cred["user"] != "controller" || cred["password"] != "BUS-PW" || cred["fingerprint"] != fingerprint ||
cred["url"] != "tls://127.0.0.1:5671" {
t.Errorf("the snapshot's credential is %v", cred)
}
if _, err := os.Stat(busSnapshotCredential); !os.IsNotExist(err) {
t.Error("the snapshot's credential was left on the machine")
}
text := strings.Join(said, "\n")
for _, want := range []string{"snapshotted every stream", "restarting mesh-broker-nats — once",
"bus answers mesh-broker-nats", "bus step done last step 1", "adopted mesh-broker-nats"} {
if !strings.Contains(text, want) {
t.Errorf("the step did not say %q:\n%s", want, text)
}
}
}
// A planned step the controller ends failed is the broker step failing, with the controller's words.
func TestABusStepThatEndsFailedIsRefused(t *testing.T) {
aGenesisBusOnDisk(t)
runtime := &asked{answer: func(name string, args []string) (string, error) {
return "last step 1: not known → 53e51864 on anchor by installer (x): failed at 2026-10-13 11:59; snapshot s\n" +
" stream KV_x has no messages\n", nil
}}
control := controlPlane{container: "mesh-controller", run: runtime.run, timeout: time.Second}
err := waitForTheBusStep(context.Background(), Options{}, control, quietly)
if err == nil || !strings.Contains(err.Error(), "failed at") || !strings.Contains(err.Error(), "KV_x") {
t.Fatalf("a failed bus step was not refused with its words: %v", err)
}
}
// The bundle's bus runs the release the module's image is built on, pinned by the same digest, so the
// adoption changes no server release and is reversible as the step says (ADR 0333 §3).
func TestTheBundlesBusRunsTheReleaseTheModulesImageIsBuiltOn(t *testing.T) {
var m struct {
Build struct {
On []struct{ Arg, Image string } `json:"on"`
} `json:"build"`
}
if err := json.Unmarshal(theCataloguesModule(t, BusModule), &m); err != nil {
t.Fatal(err)
}
base := ""
for _, on := range m.Build.On {
if on.Arg == "NATS_BASE" {
base = on.Image
}
}
rewritten, err := Rewrite(theRealBundle(t), held)
if err != nil {
t.Fatal(err)
}
if got := containerNamed(rewritten.Declaration, "mesh-broker-nats").Image; got != base {
t.Errorf("the bundle raises the bus on %s; the nats module's image is built on %s", got, base)
}
}
@@ -0,0 +1,65 @@
package bootstrap
import (
"context"
"encoding/json"
"strings"
"testing"
"time"
)
// Genesis seals the build agent the forge it raised (novox/hq ADR 0336 §3, issue 121): the agent's
// npm-package-registry requirement is optional and binds only once the gitea module is assigned, after
// the base, so until then the packages step's genesis document is how the sdk and base steps reach the
// forge.
func TestThePackagesStepSealsTheGenesisDocumentToTheBuildAgent(t *testing.T) {
if BuilderModule != "build-agent" {
t.Fatalf("genesis installs %q; the document is the build agent's", BuilderModule)
}
c, control := recording(t)
o := Options{Node: "anchor", Ports: FoundationPorts{Packages: 3100}, Wait: time.Second}
doc := genesisRegistry(o.Ports.orDefaults(), "builder-password")
if err := deliverGenesisRegistry(context.Background(), o, control, doc, quietly); err != nil {
t.Fatal(err)
}
carried := c.settings["accepting-"+GenesisRegistrySecret]
var got GenesisRegistry
if err := json.Unmarshal([]byte(carried), &got); err != nil {
t.Fatalf("the genesis document carried is not JSON: %q", carried)
}
want := GenesisRegistry{Scheme: "http", At: "127.0.0.1", Port: 3100, NpmPath: "/api/packages/novox/npm/",
As: "mesh-builder", Password: "builder-password"}
if got != want {
t.Fatalf("the genesis document says %+v, want %+v", got, want)
}
accept := c.index("secret accept anchor build-agent genesis-package-registry --from /accepting-genesis-package-registry")
push := c.index("push anchor")
if accept < 0 || push < 0 || accept > push {
t.Fatalf("the genesis document was not sealed to the build agent before the push: %v", c.told)
}
}
// What the retired builder was given and the build agent declares neither of: an `npm-password`
// own-secret and a `serves` setting for its carried binding (ADR 0336 §3).
func TestNoStepGivesTheBuildAgentAServesSettingOrAnNpmPassword(t *testing.T) {
for _, module := range []string{BuilderModule} {
c, control := recording(t)
o := Options{Node: "anchor", Ports: FoundationPorts{Packages: 3100}, Wait: time.Second}
if _, err := installModule(context.Background(), o, control, module, []byte(`{"module":"`+module+`"}`), quietly); err != nil {
t.Fatal(err)
}
if got, set := c.settings[module+"-settings.json"]; set {
t.Errorf("%s was given a setting on a node with a non-default packages port: %s", module, got)
}
if err := deliverGenesisRegistry(context.Background(), o, control,
genesisRegistry(o.Ports.orDefaults(), "p"), quietly); err != nil {
t.Fatal(err)
}
for _, told := range c.told {
if strings.Contains(told, "npm-password") || strings.Contains(told, "serves") {
t.Errorf("genesis told the controller %q", told)
}
}
}
}
@@ -0,0 +1,89 @@
package bootstrap
import (
"context"
"encoding/json"
"strings"
"testing"
"time"
)
// Defends novox/hq issue 616: genesis's base step builds the toolchain of every language a module
// genesis installs is compiled in. It built only mesh-tools, and the store's module, with Go code,
// stopped genesis at step 17: "this mesh holds no copy of it. Build mesh-tools-go first".
// genesisModules is every catalogue module genesis installs on its own, read from the steps that
// install them: the registry, the build agent, the store, the bus, the vault, the catalogue and the
// filter (extras are the operator's choice and come after the base too, but are not genesis's own).
var genesisModules = []string{RegistryModule, BuilderModule, "postgres", BusModule, "mesh-vault", "mesh-catalog", "nftables"}
// Every genesis module's bundles are compiled in a toolchain the base step builds, read from the real
// catalogue's manifests: a module that gains a language genesis does not build fails here, not at a
// genesis.
func TestTheBaseStepBuildsEveryToolchainAGenesisModuleIsCompiledIn(t *testing.T) {
built := map[string]bool{}
for _, tc := range genesisToolchains {
built[tc.Language] = true
}
for _, module := range genesisModules {
var m struct {
Build struct {
Artifacts []struct {
Name, Kind, Language string
} `json:"artifacts"`
} `json:"build"`
}
if err := json.Unmarshal(theCataloguesModule(t, module), &m); err != nil {
t.Fatalf("%s: %v", module, err)
}
for _, a := range m.Build.Artifacts {
if a.Kind != "bundle" {
continue // an image builds on the bases its Dockerfile names, pinned upstream
}
if !built[a.Language] {
t.Errorf("%s's %q is a %s bundle, and genesis's base step builds no %s toolchain",
module, a.Name, a.Language, a.Language)
}
}
}
}
// And the base step asks for each toolchain's build: the base, then the Go toolchain beside it.
func TestTheBaseStepBuildsTheGoToolchainBesideTheBase(t *testing.T) {
runtime := &asked{answer: aMeshThatAgrees(nil)}
control := controlPlane{container: "mesh-controller", run: runtime.run, timeout: time.Second}
o := Options{ToolsSource: Source{Repository: "git://192.0.2.30/mesh-tools.git"}}
if err := BuildBase(context.Background(), o, control, quietly); err != nil {
t.Fatal(err)
}
var builds []string
for _, c := range runtime.commands {
if strings.Contains(c, "/mesh-controller build ") {
builds = append(builds, c[strings.Index(c, "build "):])
}
}
want := []string{
"build git://192.0.2.30/mesh-tools.git --ref main --wait 1200s",
"build git://192.0.2.30/mesh-tools-go.git --ref main --wait 1200s",
}
if strings.Join(builds, "\n") != strings.Join(want, "\n") {
t.Fatalf("the base step built\n%s\nand not\n%s", strings.Join(builds, "\n"), strings.Join(want, "\n"))
}
}
func TestTheGoToolchainIsAsGivenOrBesideTheBase(t *testing.T) {
for _, c := range []struct{ tools, given, want string }{
{"http://forge/novox/mesh-tools.git", "", "http://forge/novox/mesh-tools-go.git"},
{"http://forge/novox/mesh-tools", "", "http://forge/novox/mesh-tools-go"},
{"http://forge/other/base.git", "http://forge/other/go.git", "http://forge/other/go.git"},
} {
got, err := toolsGoSource(Options{ToolsSource: Source{Repository: c.tools}, ToolsGoSource: Source{Repository: c.given}})
if err != nil || got.Repository != c.want {
t.Errorf("tools %s, given %q: %q %v; want %s", c.tools, c.given, got.Repository, err, c.want)
}
}
if _, err := toolsGoSource(Options{ToolsSource: Source{Repository: "http://forge/other/base.git"}}); err == nil ||
!strings.Contains(err.Error(), "--tools-go-source") {
t.Errorf("a base not named mesh-tools, and no Go toolchain given, was not refused naming the flag: %v", err)
}
}
+34 -2
View File
@@ -137,7 +137,26 @@ func registerAndAssign(ctx context.Context, o Options, control controlPlane, mod
// should hold, seals every secret in them and publishes them, and on a first node that is the
// slowest thing the mesh does.
func pushNode(ctx context.Context, o Options, control controlPlane, say func(string)) (string, error) {
said, err := control.within(o.Wait).tell(ctx, "push", o.Node)
return pushing(ctx, o, control, false, say)
}
// pushNodeApplied is pushNode that returns only once the node reports it applied exactly what it was
// sent (`push <node> --wait`, novox/hq ADR 0010): for a push whose content the very next step reads
// on the machine. A plain push returns as soon as the declaration is sent, and the node applies it a
// moment later (issue 615).
func pushNodeApplied(ctx context.Context, o Options, control controlPlane, say func(string)) (string, error) {
return pushing(ctx, o, control, true, say)
}
func pushing(ctx context.Context, o Options, control controlPlane, applied bool, say func(string)) (string, error) {
args := []string{"push", o.Node}
patience := o.Wait
if applied {
args = append(args, "--wait", o.Wait.String())
// The controller's own wait, and time to compose and send before it starts.
patience = 2 * o.Wait
}
said, err := control.within(patience).tell(ctx, args...)
if err != nil {
return "", fmt.Errorf(
"%w\n\nWhat was registered and assigned is registered and assigned, and this node has "+
@@ -145,7 +164,20 @@ func pushNode(ctx context.Context, o Options, control controlPlane, say func(str
"at all. Fix what it named and run this installer again: it will find the module "+
"already registered and try the push again", err)
}
say(" pushed " + o.Node)
if applied {
say(" pushed " + o.Node + ", and it applied it")
} else {
say(" pushed " + o.Node)
}
// **Every push may carry a new bus user, and until the adoption the bus hears of it only from here**
// (novox/hq issue 608, ADR 0331 §5, ADR 0333 §4). A module's account is minted into the mesh's
// records and sealed to this node with the push; the bus reads its user list from a file, and until
// the broker step adopts the bus as the nats module nothing but this installer writes that file. The
// list was placed only when the node enrolled, so the build agent's account (step 13) never reached
// the bus and the agent was refused ("Authorization Violation") until the sdk step gave up.
if err := placeUntilAdopted(ctx, control, say); err != nil {
return "", err
}
return strings.TrimSpace(said), nil
}
+47 -5
View File
@@ -25,20 +25,62 @@ import (
// buildWait bounds one module build. Generous, because the first build compiles a toolchain.
const buildWait = 20 * time.Minute
// BuildBase asks the mesh to build the shared base every module with code of its own stands on.
// BuildBase asks the mesh to build the shared base every module with code of its own stands on, and
// the toolchain of every language genesis's own modules are written in.
//
// **First, because until it exists nothing else with code can be built.** Not registered as a
// module here: it is never assigned — it runs nowhere — and the build itself records what was
// made, which is all anything downstream reads.
//
// **Every toolchain a genesis module compiles in** (novox/hq issue 616). A bundle is compiled in its
// language's base (mesh-controller internal/builder toolchain.go): TypeScript in mesh-tools, Go in
// mesh-tools-go. The store's and the bus's modules carry Go code, so a genesis that built only
// mesh-tools stopped at the store step: "this mesh holds no copy of it. Build mesh-tools-go first".
// The bases are the ones genesisToolchains names, and a test holds them to the languages of every
// module genesis installs, read from the catalogue.
func BuildBase(ctx context.Context, o Options, control controlPlane, say func(string)) error {
if o.ToolsSource.Repository == "" {
return fmt.Errorf("phase two needs --tools-source: the shared base is built from its " +
"own repository, and an installer told nothing cannot know where that is")
}
say(" building " + o.ToolsSource.Repository + " at " + refOr(o.ToolsSource.Ref))
_, err := control.within(buildWait).tell(ctx,
"build", o.ToolsSource.Repository, "--ref", refOr(o.ToolsSource.Ref), "--wait", "1200s")
return err
goSource, err := toolsGoSource(o)
if err != nil {
return err
}
for _, src := range []Source{o.ToolsSource, goSource} {
say(" building " + src.Repository + " at " + refOr(src.Ref))
if _, err := control.within(buildWait).tell(ctx,
"build", src.Repository, "--ref", refOr(src.Ref), "--wait", "1200s"); err != nil {
return err
}
}
return nil
}
// genesisToolchains is the base each language's bundles compile in, for the languages genesis's own
// modules are written in — the same pairs as mesh-controller internal/builder toolchain.go — in the
// order BuildBase builds them.
var genesisToolchains = []struct{ Language, Base string }{
{"typescript", "mesh-tools"},
{"go", "mesh-tools-go"},
}
// toolsGoSource is where the Go toolchain is built from: as given, or the repository beside the base's
// named mesh-tools-go, which is how the mesh keeps its repositories side by side.
func toolsGoSource(o Options) (Source, error) {
if o.ToolsGoSource.Repository != "" {
return o.ToolsGoSource, nil
}
repo := strings.TrimSuffix(o.ToolsSource.Repository, "/")
suffix := ""
if strings.HasSuffix(repo, ".git") {
repo, suffix = strings.TrimSuffix(repo, ".git"), ".git"
}
if !strings.HasSuffix(repo, "mesh-tools") {
return Source{}, fmt.Errorf("the Go toolchain is built from mesh-tools-go, and --tools-source (%s) is not a "+
"repository named mesh-tools to find it beside; give --tools-go-source", o.ToolsSource.Repository)
}
return Source{Repository: repo + "-go" + suffix, Ref: o.ToolsGoSource.Ref}, nil
}
// InstallFromCatalogue builds a catalogue module and installs it on this machine.
+16 -22
View File
@@ -84,7 +84,7 @@ func InstallStore(ctx context.Context, o Options, control controlPlane,
say); err != nil {
return err
}
say(" adopted mesh-store — the foundation's store is now the " + module + " module")
say(" adopted " + store.Name + " — the foundation's store is now the " + module + " module")
return nil
}
@@ -93,7 +93,8 @@ func InstallStore(ctx context.Context, o Options, control controlPlane,
// be in it. InstallFromCatalogue is the same sequence without that moment; the store, the broker
// and the vault each need it or need the shape, and three copies of it drifted.
func installProvider(ctx context.Context, o Options, control controlPlane, module string,
manifest []byte, buildNote string, beforePush func() error, say func(string)) error {
manifest []byte, buildNote string, beforePush func() error, say func(string),
send ...func() error) error {
remote := "/" + module + "-module.json"
if err := control.carrying(ctx, module+"-module.json", manifest, remote); err != nil {
@@ -132,6 +133,11 @@ func installProvider(ctx context.Context, o Options, control controlPlane, modul
return err
}
}
// A module whose first send is not a plain push says how it is sent: the bus, whose replacement is a
// planned step (ADR 0236), is sent by that step (adoptTheBus).
if len(send) > 0 && send[0] != nil {
return send[0]()
}
_, err := pushNode(ctx, o, control, say)
return err
}
@@ -176,9 +182,10 @@ const BusModule = "nats"
// **The image is the one thing that changes, once** (ADR 0333 §3). The store's module pins the upstream
// postgres the foundation runs; the bus's module builds its server image (it carries the mesh's snapshot
// tool), so no pinned reference can be equal to the foundation's. The applier recreates the container on
// the module's image, on the same streams, users and certificate. This says so before the push and waits
// after it until the container runs the module's image and the bus answers, so nothing after this step
// dials a bus that is restarting.
// the module's image, on the same streams, users and certificate. The mesh replaces its bus only as a
// planned step (ADR 0236), so the module is sent by that step: its streams snapshotted first, the restart
// said, and the installer waits until the container runs the module's image, the bus answers, and the
// controller ends the step done (bus_adoption.go), so nothing after this step dials a bus that is restarting.
func InstallBroker(ctx context.Context, o Options, control controlPlane,
foundation *declaration.Declaration, fetch func(context.Context, string) (int, string, error),
say func(string)) error {
@@ -197,20 +204,15 @@ func InstallBroker(ctx context.Context, o Options, control controlPlane,
}
say(" adopting " + bus.Name + " — the bus the foundation raised, its streams kept")
if err := installProvider(ctx, o, control, module, manifest, "(the server, adopting the foundation's bus)",
func() error {
if running := runningImage(ctx, control, bus.Name); running != "" && running != bus.Image {
say(" already restarted " + bus.Name + " — it runs the module's image")
return nil
}
say(" restarting " + bus.Name + " — once, onto the " + module + " module's image; its " +
"streams, users and certificate stay where they are (novox/hq ADR 0333 §3)")
return nil
}, say); err != nil {
nil, say, func() error { return adoptTheBus(ctx, o, control, bus, say) }); err != nil {
return err
}
if err := waitForTheAdoptedBus(ctx, o, control, bus, fetch, say); err != nil {
return err
}
if err := waitForTheBusStep(ctx, o, control, say); err != nil {
return err
}
say(" adopted " + bus.Name + " — the foundation's bus is now the " + module +
" module, and its user list is the module's from here on")
return nil
@@ -478,11 +480,3 @@ func serverMatchesFoundation(manifest []byte, store *declaration.Container, modu
"store with. Its server container has to carry the name the foundation raised",
module, store.Name)
}
// orUnknown is s, or a word for nothing: trust.go held it, and this branch takes trust.go out.
func orUnknown(s string) string {
if strings.TrimSpace(s) == "" {
return "(nothing)"
}
return s
}
+30 -2
View File
@@ -69,6 +69,7 @@ func aBusThatAdopts() (*asked, func(context.Context, string) (int, string, error
func busStepWith(t *testing.T, o Options, runtime *asked,
fetch func(context.Context, string) (int, string, error)) ([]string, error) {
t.Helper()
aGenesisBusOnDisk(t)
rewritten, err := Rewrite(theRealBundle(t), held)
if err != nil {
t.Fatal(err)
@@ -222,7 +223,7 @@ func TestTheBrokerStepSaysTheRestartAndWaitsForTheBusToAnswer(t *testing.T) {
runtime.answer = func(name string, args []string) (string, error) {
joined := strings.Join(args, " ")
switch {
case strings.Contains(joined, "push anchor"):
case strings.Contains(joined, "bus upgrade --why"):
pushed = true
case strings.Contains(joined, "inspect --format {{.Config.Image}} mesh-broker-nats"):
if !pushed {
@@ -304,7 +305,7 @@ func TestARerunAfterTheAdoptionSaysTheBusIsTheModulesAlready(t *testing.T) {
t.Fatal(err)
}
text := strings.Join(said, "\n")
if strings.Contains(text, "restarting") || !strings.Contains(text, "already restarted mesh-broker-nats") {
if strings.Contains(text, "restarting") || !strings.Contains(text, "already adopted mesh-broker-nats") {
t.Errorf("a re-run announced a restart, or did not say the bus is the module's already:\n%s", text)
}
}
@@ -417,3 +418,30 @@ func TestTheBundlesBusFilesAreSeeds(t *testing.T) {
}
}
}
// aGenesisBusOnDisk puts what the adoption reads on the machine in a test's own directory: the bus's
// certificate, the controller's bus credential, and where the snapshot goes. The step's wait is short.
func aGenesisBusOnDisk(t *testing.T) {
t.Helper()
dir := t.TempDir()
cert := filepath.Join(dir, "tls.crt")
if err := os.WriteFile(cert, []byte(aCertificatePEM), 0o644); err != nil {
t.Fatal(err)
}
secret := filepath.Join(dir, "bus-controller.secret")
if err := os.WriteFile(secret, []byte("BUS-PW"), 0o600); err != nil {
t.Fatal(err)
}
was := []string{busSnapshotFile, busSnapshotCredential, busCertificateFile, busControllerSecret}
wasWithin := busStepWithin
busSnapshotFile, busSnapshotCredential = filepath.Join(dir, "bus.tar"), filepath.Join(dir, "bus.credential")
busCertificateFile, busControllerSecret = cert, secret
busStepWithin = 50 * time.Millisecond
t.Cleanup(func() {
busSnapshotFile, busSnapshotCredential, busCertificateFile, busControllerSecret = was[0], was[1], was[2], was[3]
busStepWithin = wasWithin
})
}
// aCertificatePEM is a PEM block standing for the bus's certificate; its pin is the sha256 of these bytes.
const aCertificatePEM = "-----BEGIN CERTIFICATE-----\nAAECAwQFBgcICQ==\n-----END CERTIFICATE-----\n"
+67 -24
View File
@@ -2,6 +2,7 @@ package bootstrap
import (
"context"
"encoding/json"
"fmt"
"net/http"
"os"
@@ -24,7 +25,7 @@ import (
const (
// foundationStore is the foundation's postgres container — the mesh's own memory, raised from the
// bundle. gitea's bootstrap database lives here too, so a mesh runs one postgres (issue 051).
foundationStore = "mesh-store"
foundationStore = "postgres"
// giteaBootstrap is the gitea server raised directly at genesis, before gitea is a module —
// under the name the gitea MODULE declares for its container, so the module finds it and holds
// it rather than raising a second forge beside it (novox/hq ADR 0163, rule 7; issue 090).
@@ -49,8 +50,8 @@ const (
packagesTeam = "packages"
// giteaAdminUser is the admin the bootstrap creates and the provisioner later authenticates as.
giteaAdminUser = "mesh-admin"
// builderGiteaUser is the gitea account the builder publishes and pulls with at genesis. It is
// the `as` the builder's static package binding names.
// builderGiteaUser is the gitea account the build agent publishes and pulls with at genesis: the
// `as` of the genesis document (novox/hq ADR 0336 §3). Retiring it is issue 602.
builderGiteaUser = "mesh-builder"
// giteaDBRole/giteaDBName is gitea's own database in the foundation store.
giteaDBRole = "mesh_gitea"
@@ -133,8 +134,8 @@ func RaisePackageRegistry(ctx context.Context, o Options, d Deps, control contro
return err
}
say(" delivering the builder its registry credential")
if err := deliverBuilderNpm(ctx, o, control, builderPassword, say); err != nil {
say(" sealing the build agent the genesis document for the forge it raised")
if err := deliverGenesisRegistry(ctx, o, control, genesisRegistry(ports, builderPassword), say); err != nil {
return err
}
@@ -362,27 +363,69 @@ func packagePasswords() (db, admin, builder string, err error) {
return db, admin, builder, nil
}
// deliverBuilderNpm seals the builder's registry password to this node as its `npm-password`
// own-secret, the same way the control plane's store connections are delivered — carry the value in,
// `secret accept`, and the next push writes it sealed where the builder reads it.
func deliverBuilderNpm(ctx context.Context, o Options, control controlPlane, password string,
say func(string)) error {
at := "/accepting-npm-password"
if err := control.carryingSecret(ctx, "mesh-accepting-npm-password", []byte(password), at); err != nil {
return err
}
if _, err := control.tell(ctx, "secret", "accept", o.Node, BuilderModule, "npm-password", "--from", at); err != nil {
return err
}
// Push so the sealed secret reaches the builder, which restarts on it and comes back credentialed.
if _, err := control.tell(ctx, "push", o.Node); err != nil {
return err
}
return nil
// GenesisRegistrySecret is the build agent's own-secret the genesis document is sealed as (novox/hq
// ADR 0336 §3). The build-agent manifest declares it, issued outside the mesh: only this installer
// gives one, on genesis's node.
const GenesisRegistrySecret = "genesis-package-registry"
// GenesisRegistry is the genesis document (novox/hq ADR 0336 §3): the forge this step raised, the port
// this node gave the package registry, its npm path, and the account this step made with its password.
// mesh-controller's cmd/mesh-builder reads it under the same field names while the build agent's
// npm-package-registry requirement is unbound, and prefers the binding once the gitea module is
// assigned and it binds.
type GenesisRegistry struct {
Scheme string `json:"scheme"`
At string `json:"at"`
Port int `json:"port"`
NpmPath string `json:"npm-path"`
As string `json:"as"`
Password string `json:"password"`
}
// PublishTheSDK dispatches a build of the SDK to the builder. The builder has its registry
// credential by now, so the build's `npm publish` authenticates; the artifact is a `package`, built
// genesisRegistry is the document for the forge raised here: on the machine's own loopback, where it
// answers on the host network and where its ROOT_URL says it is (raiseGiteaServer), on the port this
// node gave the package registry.
func genesisRegistry(ports FoundationPorts, password string) GenesisRegistry {
return GenesisRegistry{Scheme: "http", At: "127.0.0.1", Port: ports.Packages,
NpmPath: "/api/packages/" + packagesOrg + "/npm/", As: builderGiteaUser, Password: password}
}
// deliverGenesisRegistry seals the genesis document to the build agent on this node as its
// `genesis-package-registry` own-secret (novox/hq ADR 0336 §3), the same way the control plane's store
// connections are delivered — carry the value in, `secret accept`, and the next push writes it sealed
// where the build agent reads it.
//
// **The build agent's one way to the forge until the registry binds.** Its npm-package-registry
// requirement is optional (ADR 0336 §1), and nothing provides it until the gitea module is assigned
// after the base, so the sdk step (15) publishes and the base step (16) resolves with the account in
// this document. It replaces the retired builder's `npm-password` own-secret and the `serves.port`
// setting its carried binding took, neither of which the build agent declares.
func deliverGenesisRegistry(ctx context.Context, o Options, control controlPlane, doc GenesisRegistry,
say func(string)) error {
raw, err := json.Marshal(doc)
if err != nil {
return err
}
at := "/accepting-" + GenesisRegistrySecret
if err := control.carryingSecret(ctx, "mesh-accepting-"+GenesisRegistrySecret, raw, at); err != nil {
return err
}
if _, err := control.tell(ctx, "secret", "accept", o.Node, BuilderModule, GenesisRegistrySecret, "--from", at); err != nil {
return err
}
// Push so the sealed document reaches the build agent, which reads it at its next build — and,
// like every push before the adoption, the bus hears the users it carries (issue 608).
//
// **Waited for until the node applied it** (issue 615). The next step asks the build agent to
// publish the SDK, and the agent reads the document from the file the node writes when it applies
// this push. A push that returned on sending let the ask race the apply: one genesis found the
// document in time, the next was refused, "neither its binding nor the genesis document".
_, err = pushNodeApplied(ctx, o, control, say)
return err
}
// PublishTheSDK dispatches a build of the SDK to the build agent. The agent has the genesis document by
// now (novox/hq ADR 0336 §7), so the build's `npm publish` authenticates; the artifact is a `package`, built
// on a public base, so this needs no toolchain — which is the whole point of doing it before the base.
func PublishTheSDK(ctx context.Context, o Options, control controlPlane, say func(string)) error {
if o.SDKSource.Repository == "" {
+2 -25
View File
@@ -117,24 +117,6 @@ const busModulePort = 4222
// PortsSetting is the controller's settings key for a module's given ports.
const PortsSetting = "ports"
// ServesSetting is the settings key that carries what a provider serves, inside the one binding a
// module carries rather than resolves — the builder's package binding (novox/hq 04-ISSUES/085).
//
// **A port given at genesis has to be a setting wherever it is read, or something puts it back.**
// The builder reaches the package registry through a binding written in its own manifest, because
// at genesis no module yet provides `package-registry` and so there is nothing for the controller
// to resolve it from. That binding used to be rewritten, as text, when the installer registered the
// builder — which a later registration from the catalogue undid silently. Set as a node's setting
// instead, it is held by the controller rather than by the manifest, so re-registering the builder
// leaves it where it was.
//
// **A settings layer is the module's, not one resource's.** The controller lays it over every file
// of that module which merges as JSON, so `serves` lands in the package binding only because that
// binding is the builder's one mergeable file. A second mergeable file added to the builder would
// be given a `serves` key too, meaning nothing to whatever reads it. Worth knowing before adding
// one.
const ServesSetting = "serves"
// setFoundationSettings tells the controller the ports this node gave a foundation module — and,
// on an adopted node, that the registry is reached from anywhere, as a node pulls from it before it
// has a private-network address (novox/hq ADR 0100). Done after the module is registered and before
@@ -148,13 +130,8 @@ func setFoundationSettings(ctx context.Context, o Options, control controlPlane,
if o.Adopted && module == RegistryModule {
values["expose"] = map[string]string{strconv.Itoa(DefaultPorts().Registry): "anywhere"}
}
if packages := o.Ports.orDefaults().Packages; module == BuilderModule &&
packages != DefaultPorts().Packages {
// The one binding nothing resolves: the builder dials the forge by a number it carries.
// The rest of the binding — the provision, the forge it is from, the account it presents —
// is the manifest's and stays there.
values[ServesSetting] = map[string]int{"port": packages}
}
// No `serves` setting for the build agent: the port given for the package registry reaches it in
// the genesis document (novox/hq ADR 0336 §3), and from the binding once the registry binds.
if len(values) == 0 {
return nil
}
+7 -1
View File
@@ -68,7 +68,7 @@ func TestGivenPortsMoveOnlyTheMachinesSide(t *testing.T) {
if got.Places == 0 {
t.Fatal("nothing was rewritten")
}
storeC := containerNamed(r.Declaration, "mesh-store")
storeC := containerNamed(r.Declaration, "postgres")
if len(storeC.Ports) != 1 || storeC.Ports[0] != "5433:5432" {
t.Errorf("the store publishes %v", storeC.Ports)
}
@@ -239,6 +239,12 @@ func (c *controlRecorder) run(_ context.Context, name string, args ...string) (s
}
if name == "docker" && args[0] == "exec" {
c.told = append(c.told, strings.Join(args[3:], " "))
if strings.Join(args[3:], " ") == "broker accounts" {
return "accounts {}\n", nil // the composed user list, placed after every push (issue 608)
}
}
if name == "sh" {
c.told = append(c.told, "placed the bus's users")
}
return "", nil
}
+7
View File
@@ -127,6 +127,13 @@ func Preflight(ctx context.Context, o Options, d Deps, say func(string)) ([]byte
if err := RefuseAMachineInUse(ctx, o, d.Run, say); err != nil {
return nil, err
}
// The registry's trust is written at step 7, before the push (novox/hq issue 92). What can be
// known about whether it will take, without writing anything, is asked now.
if o.pivots() {
if err := RuntimeWillTakeTheTrust(ctx, o, d.Run, say); err != nil {
return nil, err
}
}
// 4. Can this machine reach what the bundle's images come from?
//
@@ -0,0 +1,134 @@
package bootstrap
import (
"context"
"strings"
"testing"
"time"
)
// Defends novox/hq issue 608 (ADR 0331 §5, ADR 0333 §4): until the bus is adopted as the nats module,
// every push genesis makes is followed by the installer placing the composed user list where the bus
// reads it, and making the bus re-read it. The build agent's account was minted at step 13 and sealed
// with that push, and the list was placed only at enrolment, so the bus refused the agent
// ("Authorization Violation") and genesis stopped at step 15.
// placing answers the controller as a mesh whose bus is or is not the nats module yet, and records
// what was run, in order.
func placing(adopted bool) (*asked, controlPlane) {
listed := "build-agent 1\n"
if adopted {
listed += "nats 1\n"
}
runtime := &asked{}
runtime.answer = func(name string, args []string) (string, error) {
joined := strings.Join(args, " ")
switch {
case name == "sh":
return "", nil
case strings.HasSuffix(joined, "module list"):
return listed, nil
case strings.HasSuffix(joined, "broker accounts"):
return "accounts {\n MESH { users = [ { user: \"anchor.build-agent\" } ] }\n}\n", nil
}
return "", nil
}
return runtime, controlPlane{container: "mesh-controller", run: runtime.run, timeout: time.Second}
}
// placedAfterThePush is whether the bus's users were placed — written at the module's bus-users path,
// the bus told to re-read them — after the push, and how often.
func placedAfterThePush(t *testing.T, commands []string) int {
t.Helper()
pushed, placed := -1, 0
for i, c := range commands {
if strings.Contains(c, "/mesh-controller push anchor") && pushed < 0 {
pushed = i
}
if strings.HasPrefix(c, "sh -c") && strings.Contains(c, busAccounts) &&
strings.Contains(c, "kill -s HUP "+busContainer) {
if pushed < 0 || i < pushed {
t.Fatalf("the bus's users were placed before the push that carries the account: %v", commands)
}
placed++
}
}
if pushed < 0 {
t.Fatalf("nothing was pushed: %v", commands)
}
return placed
}
func TestTheBusHearsTheBuildAgentsAccountAfterItsPush(t *testing.T) {
t.Setenv("TMPDIR", t.TempDir())
runtime, control := placing(false)
o := Options{Node: "anchor", Wait: time.Second}
if _, err := installModule(context.Background(), o, control, BuilderModule, []byte(`{}`), quietly); err != nil {
t.Fatal(err)
}
if n := placedAfterThePush(t, runtime.commands); n != 1 {
t.Fatalf("the bus's users were placed %d time(s) after the build agent's push; once: %v", n, runtime.commands)
}
}
func TestTheGenesisDocumentsPushIsHeardByTheBusToo(t *testing.T) {
t.Setenv("TMPDIR", t.TempDir())
runtime, control := placing(false)
o := Options{Node: "anchor", Wait: time.Second}
if err := deliverGenesisRegistry(context.Background(), o, control, genesisRegistry(DefaultPorts(), "pw"), quietly); err != nil {
t.Fatal(err)
}
if n := placedAfterThePush(t, runtime.commands); n != 1 {
t.Fatalf("the bus's users were placed %d time(s) after the genesis document's push; once: %v", n, runtime.commands)
}
}
// From the adoption on the nats module writes the list, and the installer does not (ADR 0333 §4).
func TestOnceTheBusIsTheNatsModulesThePushPlacesNothing(t *testing.T) {
t.Setenv("TMPDIR", t.TempDir())
runtime, control := placing(true)
o := Options{Node: "anchor", Wait: time.Second}
if _, err := installModule(context.Background(), o, control, BuilderModule, []byte(`{}`), quietly); err != nil {
t.Fatal(err)
}
if n := placedAfterThePush(t, runtime.commands); n != 0 {
t.Fatalf("the installer placed the bus's users over the nats module's: %v", runtime.commands)
}
}
// Defends novox/hq issue 615: the packages step's push is waited for until the node applied it, because
// the sdk step right after asks the build agent for a build that reads the genesis document from the
// file that apply writes. A push that returned on sending let the ask race the apply.
func TestThePackagesStepWaitsUntilTheNodeAppliedTheGenesisDocument(t *testing.T) {
t.Setenv("TMPDIR", t.TempDir())
runtime, control := placing(false)
o := Options{Node: "anchor", Wait: 3 * time.Minute}
if err := deliverGenesisRegistry(context.Background(), o, control, genesisRegistry(DefaultPorts(), "pw"), quietly); err != nil {
t.Fatal(err)
}
for _, c := range runtime.commands {
if strings.Contains(c, "/mesh-controller push anchor") {
if !strings.HasSuffix(c, "push anchor --wait 3m0s") {
t.Fatalf("the genesis document's push does not wait for the node to apply it: %s", c)
}
return
}
}
t.Fatalf("the genesis document was never pushed: %v", runtime.commands)
}
// Every other push of genesis still returns on sending: only a push whose content the next step reads on
// the machine waits.
func TestAnOrdinaryPushDoesNotWait(t *testing.T) {
t.Setenv("TMPDIR", t.TempDir())
runtime, control := placing(false)
o := Options{Node: "anchor", Wait: time.Second}
if _, err := installModule(context.Background(), o, control, BuilderModule, []byte(`{}`), quietly); err != nil {
t.Fatal(err)
}
for _, c := range runtime.commands {
if strings.Contains(c, "push anchor --wait") {
t.Fatalf("an ordinary push waits: %s", c)
}
}
}
+12
View File
@@ -58,8 +58,20 @@ func aMeshThatAgrees(answers map[string]string) func(string, []string) (string,
}
}
switch {
case name == "sh" && len(args) == 2 && strings.Contains(args[1], "accounts.conf"):
return "", nil // the bus's users placed after a push (issue 608)
case name == "sh" && len(args) == 2 && strings.Contains(args[1], "mesh-nats-snapshot"):
return "", nil // the streams snapshotted before the bus's planned step (issue 590)
case name == "docker" && strings.HasSuffix(joined, "/mesh-controller bus"):
return "the bus is nats, built 53e51864, on anchor\nlast step 1: not known → 53e51864 on anchor by installer (" +
adoptionWhy + "): done at 2026-10-13 11:40; snapshot " + busSnapshotFile + "\n", nil
case name == "docker" && strings.Contains(joined, "/mesh-controller builds nats -n 1"):
return "nats built 53e51864 anchor\n image artifact-store://nats/server@sha256:" +
strings.Repeat("4", 64) + "\n bundle artifact-store://nats/tools/blobs/sha256:" + strings.Repeat("5", 64) + "\n", nil
case name != "docker":
return "", fmt.Errorf("unexpected program %q", name)
case strings.HasSuffix(joined, "broker accounts"):
return "accounts {}\n", nil
case args[0] == "cp":
return "", nil
case args[0] == "container":
+13 -7
View File
@@ -47,6 +47,9 @@ const (
// StoreSuperuserFile is where the store's superuser password lives on the machine — the
// postgres module's own-secret path, so genesis and adoption write the same file.
StoreSuperuserFile = "/var/lib/postgres/superuser.secret"
// StoreDataDir is where the store keeps its data — the postgres module's data directory, so
// genesis raises the store where the module adopts it (novox/hq ADR 0338 §1).
StoreDataDir = "/var/lib/mesh-store"
// BusControllerFile is where the controller's password on the bus lives on the machine. Not a
// module's path: no module reads it — the permanent controller is handed the address the
// temporary one was given — and genesis keeps it only so a second run puts the same account
@@ -62,7 +65,7 @@ const (
// What the template says, matched exactly. A template that says something else is a template
// this installer does not know how to make safe, and it says so rather than guessing.
templateStorePassword = `"POSTGRES_PASSWORD": "bootstrap"`
templateStoreVolumes = `"volumes": ["mesh-store-data:/var/lib/postgresql/data"]`
templateStoreVolumes = `"volumes": ["` + StoreDataDir + `:/var/lib/postgresql/data"]`
templateStoreURL = "postgres:bootstrap@"
// BusURLPlaceholder is the controller's account in its bus address, as the template names it.
@@ -265,8 +268,8 @@ func Masking(say func(string), c RootCredentials) func(string) {
// RefuseExistingServers stops a run that would put a made credential in front of a store raised
// by an earlier installer with the template's.
//
// The store's password is set by initdb, once, on an empty volume. A machine that already holds
// `mesh-store-data` and has no credential file was raised with `bootstrap`, and minting a new
// The store's password is set by initdb, once, on an empty data directory. A machine that already holds
// a cluster in the store's data directory and has no credential file was raised with `bootstrap`, and minting a new
// value here would make a bundle that dials with a password the server does not have — failing
// three steps later, in the schemas, with nothing pointing back here. Refused by name instead,
// with the way forward.
@@ -278,11 +281,14 @@ func RefuseExistingServers(ctx context.Context, run Runner, c RootCredentials) e
if !c.StoreMade {
return nil
}
if _, err := run(ctx, "docker", "volume", "inspect", "mesh-store-data"); err != nil {
return nil // no such volume: a fresh machine, which is the case this installer makes
// Where the store keeps its data, as the postgres module does (novox/hq ADR 0338 §2): a cluster
// there is one initdb already set a password for. The docker volume genesis used before is not
// the store's data any more, and a machine holding only that is not refused.
if _, err := run(ctx, "test", "-e", StoreDataDir+"/pgdata/PG_VERSION"); err != nil {
return nil // no cluster there: a fresh machine, which is the case this installer makes
}
return fmt.Errorf(
"this machine already holds the store's data (volume mesh-store-data) and no credential at %s, so it was raised "+
"this machine already holds the store's data (in "+StoreDataDir+") and no credential at %s, so it was raised "+
"by an earlier installer with the template's password. A new one made here would not open it. "+
"Put the password the store has into %s (0600, the value alone) and run again; then change it "+
"on the server and accept the new value — this installer does not rotate a running store",
@@ -311,7 +317,7 @@ func RewriteRoot(r *Rewritten, c RootCredentials) (RootRewrite, error) {
return out, err
}
if bundle, err = replaceOnce(bundle, templateStoreVolumes,
`"volumes": ["mesh-store-data:/var/lib/postgresql/data", "`+StoreSuperuserFile+":"+storeSuperuserMount+`:ro"]`,
`"volumes": ["`+StoreDataDir+`:/var/lib/postgresql/data", "`+StoreSuperuserFile+":"+storeSuperuserMount+`:ro"]`,
"the store's volumes"); err != nil {
return out, err
}
+1 -1
View File
@@ -59,7 +59,7 @@ func TestTheProducedBundleCarriesNoWellKnownCredential(t *testing.T) {
switch x := res.(type) {
case *declaration.Container:
switch x.Name {
case "mesh-store":
case "postgres":
store = true
if _, has := x.Env["POSTGRES_PASSWORD"]; has {
t.Error("the store still takes its password from its environment")
+163
View File
@@ -0,0 +1,163 @@
package bootstrap
import (
"context"
"encoding/json"
"errors"
"strings"
"testing"
"github.com/novox/mesh-host/internal/declaration"
)
// Defends novox/hq ADR 0338 (issue 609): genesis raises its store as the postgres module's server, so
// the store step adopts it as it runs. Until then the bundle raised `mesh-store`, on another image, in a
// docker volume, and genesis stopped at step 17 of 24.
// theStoreGenesisRaises is the store container of the bundle as the installer applies it: rewritten for
// this machine's credentials, so its password is the file the module reads too.
func theStoreGenesisRaises(t *testing.T) *declaration.Container {
t.Helper()
r := producedBundle(t)
store, err := storeIn(r.Declaration)
if err != nil {
t.Fatal(err)
}
return store
}
// The bundle's store is exactly the catalogue's postgres module's server, read from the module's own
// manifest (ADR 0338 §1): its name, its image (so a drift of either pin fails here), where its data is,
// and the file its superuser password is read from, with the environment that says so.
func TestTheBundleRaisesTheStoreAsTheCataloguesPostgresModuleDeclaresIt(t *testing.T) {
manifest := theCataloguesModule(t, "postgres")
var m struct {
Resources []struct {
ID string `json:"id"`
Type string `json:"type"`
Name string `json:"name"`
Image string `json:"image"`
Env map[string]string `json:"env"`
Ports []string `json:"ports"`
Volumes []string `json:"volumes"`
} `json:"resources"`
}
if err := json.Unmarshal(manifest, &m); err != nil {
t.Fatal(err)
}
ours := theStoreGenesisRaises(t)
found := false
for _, r := range m.Resources {
if r.Type != "container" || r.ID != "server" {
continue
}
found = true
if ours.Name != r.Name {
t.Errorf("the bundle raises the store as %q; the module's server is %q", ours.Name, r.Name)
}
if ours.Image != r.Image {
t.Errorf("the bundle raises the store on %s; the module pins %s — the two pins move together", ours.Image, r.Image)
}
for _, key := range []string{"POSTGRES_PASSWORD_FILE", "PGDATA"} {
if ours.Env[key] != r.Env[key] {
t.Errorf("the bundle's store has %s=%q; the module's %q", key, ours.Env[key], r.Env[key])
}
}
if _, has := ours.Env["POSTGRES_PASSWORD"]; has {
t.Error("the bundle's store takes its password from its environment; the module reads a file")
}
// The module's own directory for its state is <data root>/<module>, where genesis keeps the
// superuser's file (StoreSuperuserFile).
var theirs []string
for _, v := range r.Volumes {
theirs = append(theirs, strings.ReplaceAll(v, "${dir:state}", "/var/lib/postgres"))
}
if strings.Join(ours.Volumes, " ") != strings.Join(theirs, " ") {
t.Errorf("the bundle's store mounts %v; the module's %v", ours.Volumes, theirs)
}
if strings.Join(ours.Ports, " ") != strings.Join(r.Ports, " ") {
t.Errorf("the bundle publishes the store on %v; the module on %v", ours.Ports, r.Ports)
}
// And the store step's own check agrees: by name and image, before anything is registered.
if err := serverMatchesFoundation(manifest, ours, "postgres"); err != nil {
t.Errorf("the store step would refuse the store the bundle raises: %v", err)
}
}
if !found {
t.Fatal("the postgres module declares no server container")
}
}
// The store's data directory is made by the bundle before the store starts, owned by the image's
// postgres user, as the module declares it (ADR 0338 §1, and the measurement under its §5).
func TestTheBundleMakesTheStoresDataDirectoryAsTheModuleDoes(t *testing.T) {
r := producedBundle(t)
at, store := -1, -1
for i, res := range r.Declaration.Resources {
if d, ok := res.(*declaration.Directory); ok && d.Path == StoreDataDir {
at = i
if d.Mode != "0700" || d.Owner != "999:70" {
t.Errorf("the store's data directory is made %s %s; the module makes it 0700 999:70", d.Mode, d.Owner)
}
}
if res.Identity() == StoreID {
store = i
}
}
if at < 0 || store < 0 || at > store {
t.Fatalf("the store's data directory %s is not made before the store (directory at %d, store at %d)", StoreDataDir, at, store)
}
}
// No step of the bundle and no step of the installer names `mesh-store`, which is the seat's name only
// (ADR 0338 §2): every step that reaches the store reaches the container the bundle raised.
func TestNoStepReachesTheStoreByTheSeatsName(t *testing.T) {
text := string(theRealBundle(t))
for _, by := range []string{`"in": "mesh-store"`, `container:mesh-store`, `exec mesh-store`, `"name": "mesh-store"`, `mesh-store-data`} {
if strings.Contains(text, by) {
t.Errorf("the bundle still reaches the store as %s", by)
}
}
store := theStoreGenesisRaises(t)
if foundationStore != store.Name {
t.Errorf("the packages step reaches the store as %q; the bundle raises it as %q", foundationStore, store.Name)
}
r := producedBundle(t)
for _, res := range r.Declaration.Resources {
if a, ok := res.(*declaration.Action); ok && a.In != "" && a.In != store.Name && strings.Contains(a.ID, "database") {
t.Errorf("the step %s reaches the store in %q", a.ID, a.In)
}
}
}
// The installer's refusal to raise a store over data it cannot open looks where the store keeps its data
// (ADR 0338 §2): a cluster there and no credential is refused; the old docker volume alone is not.
func TestTheRefusalOverExistingDataLooksInTheStoresDirectory(t *testing.T) {
made := RootCredentials{Store: "s", Bus: "b", StoreMade: true}
machine := func(cluster, volume bool) Runner {
return func(_ context.Context, name string, args ...string) (string, error) {
joined := name + " " + strings.Join(args, " ")
switch {
case joined == "test -e "+StoreDataDir+"/pgdata/PG_VERSION":
if cluster {
return "", nil
}
case strings.HasPrefix(joined, "docker volume inspect"):
if volume {
return "[]", nil
}
}
return "", errors.New("exit status 1")
}
}
err := RefuseExistingServers(context.Background(), machine(true, false), made)
if err == nil || !strings.Contains(err.Error(), StoreDataDir) {
t.Fatalf("a cluster in %s and no credential was not refused naming it: %v", StoreDataDir, err)
}
if err := RefuseExistingServers(context.Background(), machine(false, true), made); err != nil {
t.Fatalf("a machine holding only the old volume was refused: %v", err)
}
if err := RefuseExistingServers(context.Background(), machine(true, false), RootCredentials{Store: "s"}); err != nil {
t.Fatalf("a kept credential was refused: %v", err)
}
}
+317
View File
@@ -0,0 +1,317 @@
package bootstrap
import (
"context"
"encoding/json"
"fmt"
"net"
"os"
"strconv"
"strings"
"time"
"github.com/novox/mesh-host/internal/apply"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/store"
"github.com/novox/mesh-host/internal/system"
)
// The registry's trust, written by genesis before anything is pushed to it (novox/hq issue 92).
//
// The mesh's registry speaks plain HTTP on purpose: every path to it is inside the private
// network's encryption (novox/hq ADR 0082). The container runtime refuses plain HTTP to a registry
// it has not been told to trust, and the module that tells it — `docker`, which writes the
// artifact store's reach into the runtime's configuration — is applied long after genesis pushes
// the control plane's image. So genesis writes that trust itself, first, with the same writer the
// module uses: into the runtime's file beside what is there (ADR 0102), the runtime reloaded and
// never restarted. It then asks the runtime whether it will speak to every name, and refuses with
// the runtime's answer before anything is pushed.
//
// **The handover.** What genesis writes is kept in no record of its own: no record of the foundation
// could hold it, because the retire step re-applies the bundle and takes away a carried record the
// bundle does not name. The `docker` module, applied later, declares the artifact store's reach —
// the mesh name on the node's port — finds it already in the file it has no record of, and takes it
// as its own (ADR 0102: a declared member found in an unrecorded file is the mesh's). That member
// does not change at the handover. The module's first apply usually still changes the file and
// reloads the runtime once, for the keys it writes beside it (`live-restore`, `log-driver`,
// `log-opts`); a reload stops nothing. The address the operator gave, when it is not the mesh name,
// stays as part of the machine's own configuration, as the hand-written trust it replaces would have.
//
// **This rests on an assumption about order.** The module takes the mesh name's member over only
// because its FIRST apply on this node already declares it: genesis assigns no `docker` before the
// network step (16), and once the node is on the network the controller composes the reach as
// `<node>.<suffix>:<port>`.
// Were `docker` applied earlier — while the reach is empty or `127.0.0.1:<port>`, as it is before the
// network — its record would exist without the mesh name, and the member would stay the machine's
// for good: undeclaring the module would never take it out. Installing the node's seat holders
// earlier in genesis must revisit this.
//
// **The suffix** is read from this process's environment, and the controller reads its own: a mesh
// with a custom `MESH_INTERNAL_SUFFIX` sets it for both.
// runtimeConfig is the container runtime's configuration, which the trust is written into. A
// variable so a test writes into a file of its own.
var runtimeConfig = "/etc/docker/daemon.json"
// runtimeUnit is the runtime's service, reloaded when the file changed.
const runtimeUnit = "docker.service"
// internalSuffixVar and defaultInternalSuffix are what a node's mesh name ends in: the same variable
// and default the controller composes `<node>.<suffix>` from (mesh-controller internal/overlay),
// which is the name the artifact store's reach — and so every push by the mesh's builder — uses.
const (
internalSuffixVar = "MESH_INTERNAL_SUFFIX"
defaultInternalSuffix = "internal"
)
// RegistryNames are the names the mesh will use for its registry on this node, each as host:port.
//
// The address the operator gave, and the node's mesh name: the controller's reach for the artifact
// store, which the builder pushes to without being told. Both on the port the registry module
// serves on this node — the node's setting, never the module's default (novox/hq issue 088). A
// loopback name is left out: the runtime trusts loopback without being told, and the mesh never
// names it to another node.
func RegistryNames(o Options) []string {
port := strconv.Itoa(o.Ports.orDefaults().Registry)
var names []string
add := func(hostport string) {
host, _, err := net.SplitHostPort(hostport)
if err != nil || host == "" || isLoopback(host) {
return
}
for _, n := range names {
if n == hostport {
return
}
}
names = append(names, hostport)
}
if host, _, err := net.SplitHostPort(o.Registry); err == nil {
add(net.JoinHostPort(strings.Trim(host, "[]"), port))
}
if node := strings.TrimSpace(o.Node); node != "" {
add(net.JoinHostPort(node+"."+internalSuffix(), port))
}
return names
}
func internalSuffix() string {
if v := strings.TrimSpace(os.Getenv(internalSuffixVar)); v != "" {
return strings.TrimPrefix(v, ".")
}
return defaultInternalSuffix
}
func isLoopback(host string) bool {
if strings.EqualFold(host, "localhost") {
return true
}
ip := net.ParseIP(strings.Trim(host, "[]"))
return ip != nil && ip.IsLoopback()
}
// trustDeclaration is the declaration genesis applies to write the trust: the same two resources
// the `docker` module declares for it — the runtime's file written into, the runtime reloaded on it.
func trustDeclaration(names []string) (*declaration.Declaration, error) {
keys, err := json.Marshal(map[string][]string{"insecure-registries": names})
if err != nil {
return nil, err
}
raw, err := json.Marshal(map[string]any{
"declaration": 1,
"resources": []map[string]any{
{"id": "genesis.registry-trust", "type": "file", "path": runtimeConfig, "mode": "0644",
"into": "json", "content": string(keys) + "\n"},
{"id": "genesis.registry-trust-reload", "type": "service", "unit": runtimeUnit,
"state": "running", "reload-on": []string{"genesis.registry-trust"}},
},
})
if err != nil {
return nil, err
}
return declaration.ParseTrusted(raw)
}
// TrustTheRegistry writes the runtime's trust of every name the mesh serves its registry on here,
// then refuses unless the runtime says it will speak plain HTTP to each.
func TrustTheRegistry(ctx context.Context, o Options, sys system.System, run Runner,
say func(string)) ([]string, error) {
names := RegistryNames(o)
if len(names) == 0 {
say(" trusted only loopback names this registry, which the runtime trusts already")
return nil, nil
}
d, err := trustDeclaration(names)
if err != nil {
return names, err
}
// One apply at a time on this machine, as ApplyBundle does: a host already running here may be
// writing the same file.
unlock, err := store.Lock(o.State, func() { say(" waiting another apply holds this node's state") })
if err != nil {
return names, err
}
report, _, err := apply.Apply(ctx, sys, d, store.State{}, store.OriginCarried, run,
func(line string) { say(" " + strings.TrimPrefix(line, " ")) }, refuseSealed)
unlock()
if err != nil {
return names, fmt.Errorf("the registry's trust could not be written into %s: %w\n"+
"The registry is plain HTTP over the private network (novox/hq ADR 0082), and the container "+
"runtime refuses to push to it until it trusts it. Nothing has been pushed", runtimeConfig, err)
}
if report.Changed() {
say(" trusted " + strings.Join(names, ", ") + " — written into " + runtimeConfig +
", the runtime reloaded")
} else {
say(" trusted " + strings.Join(names, ", ") + " — " + runtimeConfig + " already says so")
}
return names, runtimeWillSpeakTo(ctx, o, run, names)
}
// runtimeWillSpeakTo asks the runtime, until it says it trusts every name or the wait runs out.
//
// The file being right is not the runtime trusting it: a reload the runtime refused, or a flag on
// its command line that contradicts the file, leaves the push to fail at step 8 with an HTTPS error.
func runtimeWillSpeakTo(ctx context.Context, o Options, run Runner, names []string) error {
deadline := time.Now().Add(o.Wait)
var missing []string
var last error
for {
asking, cancel := context.WithTimeout(ctx, o.Timeout)
out, err := run(asking, "docker", "info", "--format", "{{json .RegistryConfig}}")
cancel()
last = err
if err == nil {
missing, last = untrusted(out, names)
if last == nil && len(missing) == 0 {
return nil
}
}
if time.Now().After(deadline) {
break
}
select {
case <-ctx.Done():
return ctx.Err()
case <-time.After(answerEvery):
}
}
if last != nil {
return fmt.Errorf("the container runtime could not be asked whether it trusts the registry: %w", last)
}
return fmt.Errorf(
"the container runtime does not trust %s after %s says it should, so it would refuse to push to "+
"it.\nThe file was written and the runtime reloaded; a runtime that still refuses was started "+
"with its own registry options, or did not take the reload — `journalctl -u %s` says which. "+
"Nothing has been pushed", strings.Join(missing, ", "), runtimeConfig, runtimeUnit)
}
// untrusted reads the runtime's registry configuration and gives back the names it would speak
// HTTPS to: a name is trusted when the runtime lists it as not secure, or when its address falls in
// a range the runtime trusts.
func untrusted(registryConfig string, names []string) ([]string, error) {
var c struct {
InsecureRegistryCIDRs []string `json:"InsecureRegistryCIDRs"`
IndexConfigs map[string]struct {
Secure bool `json:"Secure"`
} `json:"IndexConfigs"`
}
if err := json.Unmarshal([]byte(strings.TrimSpace(registryConfig)), &c); err != nil {
return nil, fmt.Errorf("the runtime's registry configuration is not readable: %w", err)
}
var missing []string
for _, name := range names {
if index, ok := c.IndexConfigs[name]; ok && !index.Secure {
continue
}
host, _, _ := net.SplitHostPort(name)
if ip := net.ParseIP(strings.Trim(host, "[]")); ip != nil && inAny(ip, c.InsecureRegistryCIDRs) {
continue
}
missing = append(missing, name)
}
return missing, nil
}
func inAny(ip net.IP, cidrs []string) bool {
for _, cidr := range cidrs {
if _, n, err := net.ParseCIDR(cidr); err == nil && n.Contains(ip) {
return true
}
}
return false
}
// procRoot is where a process's command line is read from. A variable so a test gives its own.
var procRoot = "/proc"
// RuntimeWillTakeTheTrust is preflight's half of the registry's trust: everything that can be known
// about it without writing anything, asked at step 1 so a machine that cannot take the trust is
// refused before the foundation is raised, rather than at step 7 after it (novox/hq issue 92).
//
// - The runtime's file is absent or a JSON object: the mesh writes into an object, and refuses to
// write over anything else (ADR 0102).
// - The runtime's service is known to systemd and active: the trust is taken by reloading it.
// - The runtime was not started with `--insecure-registry`: dockerd refuses a directive given both
// as a flag and in its file, so it would refuse the reload and keep distrusting the registry.
func RuntimeWillTakeTheTrust(ctx context.Context, o Options, run Runner, say func(string)) error {
names := RegistryNames(o)
if len(names) == 0 {
return nil
}
if raw, err := os.ReadFile(runtimeConfig); err == nil {
var object map[string]json.RawMessage
if len(strings.TrimSpace(string(raw))) > 0 {
if err := json.Unmarshal(raw, &object); err != nil || object == nil {
return fmt.Errorf("%s is not a JSON object, and the mesh writes the trust of its registry "+
"(%s) into it beside what is there, never over it (novox/hq ADR 0102). Make it an object "+
"or remove it; nothing has been changed", runtimeConfig, strings.Join(names, ", "))
}
}
} else if !os.IsNotExist(err) {
return fmt.Errorf("the runtime's configuration %s cannot be read: %w", runtimeConfig, err)
}
asking, cancel := context.WithTimeout(ctx, o.Timeout)
out, err := run(asking, "systemctl", "show", runtimeUnit, "--property=LoadState,ActiveState,MainPID")
cancel()
if err != nil {
return fmt.Errorf("systemd could not be asked about %s: %w\nThe registry's trust is taken by "+
"reloading it; nothing has been changed", runtimeUnit, err)
}
props := map[string]string{}
for _, line := range strings.Split(out, "\n") {
if k, v, ok := strings.Cut(strings.TrimSpace(line), "="); ok {
props[k] = v
}
}
if props["LoadState"] != "loaded" || props["ActiveState"] != "active" {
return fmt.Errorf("%s is %s and %s, and the registry's trust (%s) is taken by reloading it. "+
"Genesis needs the runtime run by systemd as %s; nothing has been changed",
runtimeUnit, orUnknown(props["LoadState"]), orUnknown(props["ActiveState"]),
strings.Join(names, ", "), runtimeUnit)
}
if pid := props["MainPID"]; pid != "" && pid != "0" {
if cmdline, err := os.ReadFile(procRoot + "/" + pid + "/cmdline"); err == nil {
for _, arg := range strings.Split(string(cmdline), "\x00") {
if strings.HasPrefix(arg, "--insecure-registry") {
return fmt.Errorf("the runtime was started with %s, and dockerd refuses insecure registries "+
"given both as a flag and in %s — so it would refuse the reload that makes it trust "+
"%s. Move the flag's registries into the file and restart the runtime once; nothing "+
"has been changed", arg, runtimeConfig, strings.Join(names, ", "))
}
}
}
}
say(" registry trust " + runtimeConfig + " can be written into and " + runtimeUnit + " reloaded")
return nil
}
func orUnknown(s string) string {
if s == "" {
return "unknown"
}
return s
}
+347
View File
@@ -0,0 +1,347 @@
package bootstrap
import (
"context"
"encoding/json"
"fmt"
"net/http"
"os"
"path/filepath"
"strings"
"testing"
"time"
"github.com/novox/mesh-host/internal/apply"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/store"
"github.com/novox/mesh-host/internal/system"
)
// novox/hq issue 92: genesis pushed the control plane's image to its registry before anything had
// told the container runtime to trust that registry, so the push failed with "server gave HTTP
// response to HTTPS client" — first by the address given, then by the mesh name the builder used,
// and the trust the mesh wrote later named the registry's default port rather than the node's.
// aRuntime is a container runtime on systemd whose registry configuration is what its file said at
// its last reload: what a real dockerd does, and why a written file is not yet a trusted registry.
type aRuntime struct {
path string
trusted []string
commands []string
// ignoresReload is a runtime started with options of its own, which a reload does not change.
ignoresReload bool
}
func (r *aRuntime) run(_ context.Context, name string, args ...string) (string, error) {
line := name + " " + strings.Join(args, " ")
r.commands = append(r.commands, line)
switch {
case name == "docker" && len(args) > 0 && args[0] == "info":
index := map[string]map[string]any{"docker.io": {"Secure": true}}
for _, n := range r.trusted {
index[n] = map[string]any{"Name": n, "Secure": false}
}
out, _ := json.Marshal(map[string]any{"InsecureRegistryCIDRs": []string{"127.0.0.0/8"}, "IndexConfigs": index})
return string(out), nil
case name == "systemctl" && strings.Contains(line, "reload"):
if !r.ignoresReload {
r.trusted = r.trusted[:0]
raw, _ := os.ReadFile(r.path)
var c struct {
Insecure []string `json:"insecure-registries"`
}
_ = json.Unmarshal(raw, &c)
r.trusted = append(r.trusted, c.Insecure...)
}
case strings.Contains(line, "is-enabled"):
return "enabled", nil
case strings.Contains(line, "show") && strings.Contains(line, "ActiveState"):
return "LoadState=loaded\nActiveState=active\nSubState=running", nil
}
return "", nil
}
func (r *aRuntime) did(word string) int {
n := 0
for _, c := range r.commands {
if strings.Contains(c, "systemctl") && strings.Contains(c, " "+word+" ") || strings.HasSuffix(c, " "+word) {
n++
}
}
return n
}
func trustSetup(t *testing.T, existing string) (Options, *aRuntime, system.System) {
t.Helper()
restore := apply.ForTests(t.TempDir(), t.TempDir())
t.Cleanup(restore)
dir := t.TempDir()
was := runtimeConfig
runtimeConfig = filepath.Join(dir, "daemon.json")
t.Cleanup(func() { runtimeConfig = was })
if existing != "" {
if err := os.WriteFile(runtimeConfig, []byte(existing), 0o644); err != nil {
t.Fatal(err)
}
}
t.Setenv(internalSuffixVar, "")
sys, err := system.For("arch")
if err != nil {
t.Fatal(err)
}
o := Options{Node: "anchor", Registry: "192.0.2.10:5100", Ports: FoundationPorts{Registry: 5100},
State: filepath.Join(dir, "state", "state.json"), Timeout: time.Second, Wait: 0}
return o, &aRuntime{path: runtimeConfig}, sys
}
func insecureIn(t *testing.T) (string, map[string]json.RawMessage) {
t.Helper()
raw, err := os.ReadFile(runtimeConfig)
if err != nil {
t.Fatal(err)
}
var o map[string]json.RawMessage
if err := json.Unmarshal(raw, &o); err != nil {
t.Fatal(err)
}
var list []string
_ = json.Unmarshal(o["insecure-registries"], &list)
return fmt.Sprint(list), o
}
func TestGenesisTrustsTheRegistryByEveryNameBeforeItPushes(t *testing.T) {
o, runtime, sys := trustSetup(t, `{"data-root":"/srv/docker","insecure-registries":["198.51.100.7:5000"]}`)
names, err := TrustTheRegistry(context.Background(), o, sys, runtime.run, func(string) {})
if err != nil {
t.Fatal(err)
}
// The address given and the mesh name the builder pushes by, both on the node's port.
if fmt.Sprint(names) != "[192.0.2.10:5100 anchor.internal:5100]" {
t.Errorf("trusted by %v", names)
}
list, object := insecureIn(t)
if list != "[198.51.100.7:5000 192.0.2.10:5100 anchor.internal:5100]" {
t.Errorf("the runtime's list is %s", list)
}
if string(object["data-root"]) != `"/srv/docker"` {
t.Errorf("the machine's own key was not kept: %s", object["data-root"])
}
if runtime.did("reload") != 1 || runtime.did("restart") != 0 || runtime.did("stop") != 0 {
t.Errorf("the runtime was not reloaded exactly once and never restarted: %v", runtime.commands)
}
// Re-run: nothing changes, nothing is reloaded.
runtime.commands = nil
if _, err := TrustTheRegistry(context.Background(), o, sys, runtime.run, func(string) {}); err != nil {
t.Fatal(err)
}
if runtime.did("reload") != 0 {
t.Errorf("a re-run reloaded the runtime: %v", runtime.commands)
}
}
// The port is the node's setting for the registry, not the module's default, whatever the operator
// typed the address with — and loopback, which the runtime trusts already, is not written.
func TestTheRegistrysTrustNamesThePortTheNodeServesItOn(t *testing.T) {
o := Options{Node: "anchor", Registry: "127.0.0.1:5100", Ports: FoundationPorts{Registry: 5100}}
t.Setenv(internalSuffixVar, "")
if got := fmt.Sprint(RegistryNames(o)); got != "[anchor.internal:5100]" {
t.Errorf("names %s", got)
}
o.Ports = FoundationPorts{}
o.Registry = "192.0.2.10:5000"
t.Setenv(internalSuffixVar, ".mesh.example")
if got := fmt.Sprint(RegistryNames(o)); got != "[192.0.2.10:5000 anchor.mesh.example:5000]" {
t.Errorf("names %s", got)
}
}
// A runtime that still refuses after the write is refused here, with the names, before any push.
func TestGenesisRefusesARuntimeThatStillDoesNotTrustTheRegistry(t *testing.T) {
o, runtime, sys := trustSetup(t, "")
runtime.ignoresReload = true
_, err := TrustTheRegistry(context.Background(), o, sys, runtime.run, func(string) {})
if err == nil {
t.Fatal("a runtime that does not trust the registry was let through to the push")
}
if !strings.Contains(err.Error(), "192.0.2.10:5100, anchor.internal:5100") ||
!strings.Contains(err.Error(), "Nothing has been pushed") {
t.Errorf("the refusal does not say what: %v", err)
}
}
// The handover: the `docker` module, applied later, writes the reach the controller composes for
// the artifact store — the mesh name and the node's port. It finds it there: the file does not
// change, the runtime is not reloaded, and the member it declares is recorded as the module's from
// then on (a member found in a file with no record of it is the mesh's, ADR 0102), so the module
// owns the trust it writes and taking the module away takes that member out.
func TestTheDockerModuleTakesTheTrustOverWithoutAChange(t *testing.T) {
o, runtime, sys := trustSetup(t, `{"live-restore":true}`)
if _, err := TrustTheRegistry(context.Background(), o, sys, runtime.run, func(string) {}); err != nil {
t.Fatal(err)
}
before, _ := os.ReadFile(runtimeConfig)
module, err := declaration.ParseTrusted([]byte(fmt.Sprintf(`{"declaration":1,"resources":[
{"id":"docker.daemon","type":"file","path":%q,"mode":"0644","into":"json",
"content":"{\"live-restore\": true, \"insecure-registries\": [\"anchor.internal:5100\"]}\n"},
{"id":"docker.runtime","type":"service","unit":"docker.service","state":"running","boot":"enabled",
"reload-on":["docker.daemon"]}]}`, runtimeConfig)))
if err != nil {
t.Fatal(err)
}
runtime.commands = nil
_, state, err := apply.Apply(context.Background(), sys, module, store.State{}, store.OriginDeclared,
runtime.run, nil, nil)
if err != nil {
t.Fatal(err)
}
after, _ := os.ReadFile(runtimeConfig)
if string(before) != string(after) {
t.Errorf("the handover changed the file:\n%s\n%s", before, after)
}
if runtime.did("reload") != 0 || runtime.did("restart") != 0 {
t.Errorf("the handover touched the runtime: %v", runtime.commands)
}
if rec, _ := state.Find("docker.daemon"); rec.Into == nil || len(rec.Into.Added["insecure-registries"]) != 1 {
t.Errorf("the module did not take the mesh name's member over: %+v", rec.Into)
}
}
func TestARangeTheRuntimeTrustsCoversAnAddressInIt(t *testing.T) {
missing, err := untrusted(`{"InsecureRegistryCIDRs":["10.42.0.0/16"],"IndexConfigs":{}}`,
[]string{"10.42.0.1:5000", "anchor.internal:5000"})
if err != nil {
t.Fatal(err)
}
if fmt.Sprint(missing) != "[anchor.internal:5000]" {
t.Errorf("missing %v", missing)
}
}
// **The order, which is the fault issue 92 is about.** Steps 7 and 8 against a runtime that behaves
// as dockerd does: a push to a registry it has not been told to trust is refused with the error the
// first genesis met. The trust must be written and confirmed before anything is tagged or pushed;
// moving the trust after the push, or dropping it, fails this test.
func TestNothingIsPushedToTheRegistryBeforeTheRuntimeTrustsIt(t *testing.T) {
o, runtime, sys := trustSetup(t, `{"data-root":"/srv/docker"}`)
o.Catalogue = catalogueWith(t, RegistryModule, upstreamRegistryManifest)
o.Registry = "192.0.2.10:5100"
pushed := false
mesh := aMeshThatAgrees(nil)
run := func(ctx context.Context, name string, args ...string) (string, error) {
if name == "systemctl" || (name == "docker" && args[0] == "info") {
return runtime.run(ctx, name, args...)
}
runtime.commands = append(runtime.commands, name+" "+strings.Join(args, " "))
if name == "docker" {
switch args[0] {
case "tag":
return "", nil
case "push":
trusted := false
for _, n := range runtime.trusted {
trusted = trusted || strings.HasPrefix(args[1], n+"/")
}
if !trusted {
return "", fmt.Errorf(`Get "https://%s/v2/": http: server gave HTTP response to HTTPS client`, o.Registry)
}
pushed = true
return "", nil
case "image":
return `["` + o.Registry + `/mesh-controller@sha256:` + strings.Repeat("a", 64) + `"]`, nil
}
}
return mesh(name, args)
}
d := Deps{Run: run, Fetch: func(_ context.Context, url string) (int, string, error) {
if strings.HasSuffix(url, "/v2/") || pushed {
return http.StatusOK, `{"name":"mesh-controller","tags":["genesis"]}`, nil
}
return http.StatusNotFound, "", nil
}}
var result Result
published, err := RegistryAndPublish(context.Background(), o, d, sys,
controlPlane{container: "temp-mesh-controller", run: run, timeout: time.Second}, held, &result, func(string) {})
if err != nil {
t.Fatalf("genesis did not get past the push: %v\ncommands: %v", err, runtime.commands)
}
if !strings.HasPrefix(published.Reference, o.Registry+"/mesh-controller@sha256:") {
t.Errorf("published as %q", published.Reference)
}
firstPush, reloaded, confirmed := -1, -1, -1
for i, c := range runtime.commands {
switch {
case firstPush < 0 && (strings.HasPrefix(c, "docker tag") || strings.HasPrefix(c, "docker push")):
firstPush = i
case strings.HasPrefix(c, "systemctl") && strings.Contains(c, "reload"):
reloaded = i
case strings.HasPrefix(c, "docker info"):
confirmed = i
}
}
if firstPush < 0 || reloaded < 0 || confirmed < 0 || !(reloaded < confirmed && confirmed < firstPush) {
t.Errorf("the trust was not written (%d) and confirmed (%d) before the first tag or push (%d): %v",
reloaded, confirmed, firstPush, runtime.commands)
}
if fmt.Sprint(result.RegistryTrusted) != "[192.0.2.10:5100 anchor.internal:5100]" {
t.Errorf("the result says it trusted %v", result.RegistryTrusted)
}
}
// Preflight asks, without writing, what would stop the trust from taking at step 7.
func TestPreflightRefusesARuntimeThatCouldNotTakeTheTrust(t *testing.T) {
proc := t.TempDir()
was := procRoot
procRoot = proc
t.Cleanup(func() { procRoot = was })
if err := os.MkdirAll(filepath.Join(proc, "42"), 0o755); err != nil {
t.Fatal(err)
}
unit := func(load, active string) Runner {
return func(_ context.Context, name string, args ...string) (string, error) {
return "LoadState=" + load + "\nActiveState=" + active + "\nMainPID=42\n", nil
}
}
cmdline := func(args ...string) {
if err := os.WriteFile(filepath.Join(proc, "42", "cmdline"), []byte(strings.Join(args, "\x00")+"\x00"), 0o644); err != nil {
t.Fatal(err)
}
}
cmdline("/usr/bin/dockerd", "-H", "fd://")
for _, c := range []struct {
name, file string
run Runner
flag bool
refused string
}{
{"a runtime that can take it", `{"data-root":"/srv/docker"}`, unit("loaded", "active"), false, ""},
{"no file yet", "", unit("loaded", "active"), false, ""},
{"a file that is not an object", `["not", "an", "object"]`, unit("loaded", "active"), false, "not a JSON object"},
{"a runtime systemd does not run", `{}`, unit("not-found", "inactive"), false, "not-found and inactive"},
{"a runtime with the flag", `{}`, unit("loaded", "active"), true, "--insecure-registry=10.0.0.1:5000"},
} {
t.Run(c.name, func(t *testing.T) {
o, _, _ := trustSetup(t, c.file)
if c.flag {
cmdline("/usr/bin/dockerd", "--insecure-registry=10.0.0.1:5000")
defer cmdline("/usr/bin/dockerd", "-H", "fd://")
}
before, _ := os.ReadFile(runtimeConfig)
err := RuntimeWillTakeTheTrust(context.Background(), o, c.run, func(string) {})
switch {
case c.refused == "" && err != nil:
t.Errorf("refused: %v", err)
case c.refused != "" && (err == nil || !strings.Contains(err.Error(), c.refused)):
t.Errorf("wanted a refusal naming %q, got %v", c.refused, err)
}
if after, _ := os.ReadFile(runtimeConfig); string(after) != string(before) {
t.Error("preflight wrote the runtime's file")
}
})
}
}
+1 -1
View File
@@ -89,7 +89,7 @@ func TestAFoundationThatIsUpAndAnsweringIsAccepted(t *testing.T) {
// run-once and scheduled shapes are excluded on purpose — a step that has exited is not a
// fault. The name is `temp-mesh-controller` because the permanent one is a module and takes the
// plain name (novox/hq ADR 0067), which is what makes the two of them coexist at all.
want := []string{"mesh-store", "mesh-broker-nats", "temp-mesh-controller"}
want := []string{"postgres", "mesh-broker-nats", "temp-mesh-controller"}
if len(verified.Running) != len(want) {
t.Fatalf("confirmed %v running, want %v", verified.Running, want)
}
+87
View File
@@ -0,0 +1,87 @@
package declaration
import (
"bytes"
"encoding/json"
"fmt"
"strings"
"time"
)
// Busy is how a long-running container says it holds work that would be lost if it were stopped now
// (novox/hq ADR 0335 §1): a command run inside it, by its shell, as an exec health check is, that exits 0
// while it is busy and non-zero when it is idle. Being asked is also the container's notice that it is
// about to be replaced: from the first ask it takes no new work, so its answer ends once the work in hand
// ends.
//
// **Not part of what the container is.** Busy is not in the container's spec: a changed command changes
// when the container is replaced, never whether — a container is not recreated for its busy command alone.
type Busy struct {
Command string `json:"command"`
// Timeout is how long one ask may take; empty is BusyTimeoutDefault.
Timeout string `json:"timeout,omitempty"`
}
// The bounds of an ask (novox/hq ADR 0335).
const (
BusyTimeoutDefault = 10 * time.Second
// BusyTimeoutCeiling is the longest one ask may take: an ask runs inside an apply, and an apply waits
// for nothing longer than a look.
BusyTimeoutCeiling = time.Minute
// BusyBound is how long a container that says busy is left running after the first apply that would
// have replaced it (ADR 0335 §3). The node-engine's, not the module's: no module holds its own
// replacement for ever.
BusyBound = 15 * time.Minute
)
// UnmarshalJSON reads a busy strictly, as everything a declaration carries is read.
func (b *Busy) UnmarshalJSON(raw []byte) error {
type plain Busy
var p plain
dec := json.NewDecoder(bytes.NewReader(raw))
dec.DisallowUnknownFields()
if err := dec.Decode(&p); err != nil {
return fmt.Errorf("busy: %w", err)
}
*b = Busy(p)
return nil
}
// Within is how long one ask may take. Validated on arrival, so an unreadable timeout is the default.
func (b *Busy) Within() time.Duration {
if d, err := time.ParseDuration(b.Timeout); err == nil && d > 0 {
return d
}
return BusyTimeoutDefault
}
// problems holds a container's busy to its shape: a one-line command, a timeout within its bounds, and only
// on a container that stays up.
func (b *Busy) problems(where string, longRunning bool) []string {
if b == nil {
return nil
}
var problems []string
say := func(format string, args ...any) {
problems = append(problems, where+": "+fmt.Sprintf(format, args...))
}
if !longRunning {
say("busy is asked of a container that stays up; a step or a scheduled run is never replaced mid-run")
}
if strings.TrimSpace(b.Command) == "" {
say("busy needs a command")
}
if strings.ContainsAny(b.Command, "\n\r") {
say("busy's command is one line")
}
if b.Timeout != "" {
d, err := time.ParseDuration(b.Timeout)
switch {
case err != nil:
say("busy's timeout is a duration, not %q", b.Timeout)
case d <= 0 || d > BusyTimeoutCeiling:
say("an ask of busy takes more than nothing and at most %s, not %s", BusyTimeoutCeiling, d)
}
}
return problems
}
+6
View File
@@ -1239,6 +1239,11 @@ type Container struct {
// timing, judged by the node-engine beside liveness. Absent: judged alive or not, and nothing more.
Health *Health `json:"health,omitempty"`
// Busy is how this container says it holds work a replacement would lose (novox/hq ADR 0335): asked
// before it is replaced, and while it says busy it is left running, within the bound. Absent: never
// busy, and replaced at once, as before.
Busy *Busy `json:"busy,omitempty"`
// RunAs is who the container runs as, resolved by the controller from the module's identity and the
// image's declared interface (novox/hq ADR 0325 §2, §3): told by environment (the image reads PUID and
// PGID, say) or by the runtime's own user option. Part of the spec, so a change recreates it. Absent: the
@@ -1318,6 +1323,7 @@ func (c *Container) Target() string { return c.Name }
func (c *Container) validate(where string, _ bool) []string {
problems := c.Health.problems(where, true, !c.RunOnce && c.Schedule == "")
problems = append(problems, c.Busy.problems(where, !c.RunOnce && c.Schedule == "")...)
if c.Name == "" {
problems = append(problems, where+": a container needs a name")
}
+26
View File
@@ -174,6 +174,14 @@ type Report struct {
// broken.
Windows []Window `json:"windows,omitempty"`
// Busy is every container the apply that reports left running because it said busy, and every one it
// replaced after it had waited, with how (novox/hq ADR 0335): `waiting`, `replaced-when-idle` or
// `replaced-at-bound`, since when, until when at the latest, and the last output of its busy command.
// A walk waiting on this machine for a module whose container is listed `waiting` waits for the report
// that no longer lists it so: its first-machine gate starts then (ADR 0335 §4). Absent from an engine
// older than BusyContract, which never waits.
Busy []Busy `json:"busy,omitempty"`
// Strays is what runs on the machine that the mesh neither wrote nor holds (novox/hq ADR
// 0163): containers nobody declared and nobody holds, the ones a cutover leaves behind.
Strays []Stray `json:"strays,omitempty"`
@@ -280,6 +288,13 @@ const LibraryContract = 6
// before it, its presence is what tells the controller this host may be sent `runner` on an identity.
const OwnGroupContract = 7
// BusyContract is the statement of a host that also reads a container's `busy` (novox/hq ADR 0335): it asks a
// container that declares it before replacing it, leaves it running while it says busy, within a bound of
// 15 minutes, and says the wait in its report's `busy`. Like the fields before it, its presence is what tells
// the controller this host may be sent `busy` on a container; an older host refuses the whole declaration
// for it.
const BusyContract = 8
// Health is one statement of every long-running resource's state on this machine (to-be 48 §4). Said in
// every report, as an event on each change, and again every minute while anything is not healthy — so a
// lost statement is not a lost fault.
@@ -731,6 +746,17 @@ type Window struct {
Until time.Time `json:"until"`
}
// Busy is one container a busy command held (novox/hq ADR 0335), as a report says it. ID is the resource's
// id, `<module>.<resource>`.
type Busy struct {
ID string `json:"id"`
Container string `json:"container"`
State string `json:"state"`
Since time.Time `json:"since"`
Until time.Time `json:"until"`
Said string `json:"said,omitempty"`
}
// A Stray is a container the mesh neither wrote nor holds (ADR 0163).
type Stray struct {
Kind string `json:"kind"`
+92
View File
@@ -0,0 +1,92 @@
package outward
import (
"errors"
"fmt"
"net"
"net/netip"
"sort"
)
// Network is one network this machine is directly attached to, and the link it is on (novox/hq ADR
// 0340 §7): the address and prefix of the network, not of the machine's own address on it.
//
// **Every one is said, and the machine judges none of them.** Which count as the machine's local
// network — private ranges only, never a public subnet, never a link-local one — is the controller's
// judgement (ADR 0340 §2), made where the filter is composed and shown with why; a machine that left a
// network out would make that judgement silently.
type Network struct {
Link string `json:"link"`
Network string `json:"network"`
}
// AddressesOf is the addresses on one link, as address and prefix. A parameter so a test can hold a
// machine's links without one.
type AddressesOf func(link string) ([]netip.Prefix, error)
// RealAddresses reads one link's addresses from the kernel.
func RealAddresses(link string) ([]netip.Prefix, error) {
iface, err := net.InterfaceByName(link)
if err != nil {
return nil, err
}
addrs, err := iface.Addrs()
if err != nil {
return nil, err
}
var out []netip.Prefix
for _, a := range addrs {
ipnet, ok := a.(*net.IPNet)
if !ok {
continue
}
addr, ok := netip.AddrFromSlice(ipnet.IP)
if !ok {
continue
}
bits, _ := ipnet.Mask.Size()
out = append(out, netip.PrefixFrom(addr.Unmap(), bits))
}
return out, nil
}
// Networks is the networks on each of these links — the links that face outside (Links) — sorted by link
// and network, without repeats. A link that has gone since Links read it has no networks rather than
// failing the rest; a link whose addresses cannot be read fails, so the machine says the fact is unknown
// rather than reporting a partial account as the whole.
func Networks(links []string, addressesOf AddressesOf) ([]Network, error) {
if addressesOf == nil {
addressesOf = RealAddresses
}
seen := map[Network]bool{}
var out []Network
for _, link := range links {
prefixes, err := addressesOf(link)
if err != nil {
if isNoSuchInterface(err) {
continue
}
return nil, fmt.Errorf("cannot read the addresses on %s: %w", link, err)
}
for _, p := range prefixes {
n := Network{Link: link, Network: p.Masked().String()}
if !seen[n] {
seen[n] = true
out = append(out, n)
}
}
}
sort.Slice(out, func(i, j int) bool {
if out[i].Link != out[j].Link {
return out[i].Link < out[j].Link
}
return out[i].Network < out[j].Network
})
return out, nil
}
// isNoSuchInterface is the error net.InterfaceByName gives for a link that is not there.
func isNoSuchInterface(err error) bool {
var op *net.OpError
return errors.As(err, &op) && op.Err != nil && op.Err.Error() == "no such network interface"
}
+59
View File
@@ -0,0 +1,59 @@
package outward
import (
"errors"
"net/netip"
"reflect"
"testing"
)
func prefixes(s ...string) []netip.Prefix {
var out []netip.Prefix
for _, p := range s {
out = append(out, netip.MustParsePrefix(p))
}
return out
}
// **Each link facing outside says the networks it is attached to** (novox/hq ADR 0340 §7), as the
// network's address and prefix, never the machine's own address on it — and every one, public and
// link-local included, because which count is the controller's judgement and said with why.
func TestEachOutwardLinkSaysTheNetworksItIsOn(t *testing.T) {
addrs := map[string][]netip.Prefix{
"eth0": prefixes("192.168.1.10/24", "fe80::1/64", "fd12:3456:789a::5/64"),
"eth1": prefixes("203.0.113.7/24"),
// Not asked: the bridge is not a link facing outside.
"docker0": prefixes("172.17.0.1/16"),
}
got, err := Networks([]string{"eth1", "eth0"}, func(link string) ([]netip.Prefix, error) { return addrs[link], nil })
if err != nil {
t.Fatal(err)
}
want := []Network{
{Link: "eth0", Network: "192.168.1.0/24"},
{Link: "eth0", Network: "fd12:3456:789a::/64"},
{Link: "eth0", Network: "fe80::/64"},
{Link: "eth1", Network: "203.0.113.0/24"},
}
if !reflect.DeepEqual(got, want) {
t.Fatalf("got %+v\nwant %+v", got, want)
}
}
// A link whose addresses cannot be read fails the whole, so the machine says the fact is unknown rather
// than reporting part of its networks as all of them.
func TestAnUnreadableLinkIsNotAPartialAccount(t *testing.T) {
_, err := Networks([]string{"eth0"}, func(string) ([]netip.Prefix, error) { return nil, errors.New("denied") })
if err == nil {
t.Fatal("an unreadable link was reported as having no networks")
}
}
// A link gone between reading the routing table and reading its addresses has no networks, and the
// rest are still said.
func TestALinkGoneSinceItWasListedHasNoNetworks(t *testing.T) {
got, err := Networks([]string{"gone0"}, RealAddresses)
if err != nil || len(got) != 0 {
t.Fatalf("got %v, %v", got, err)
}
}
+35 -2
View File
@@ -10,6 +10,8 @@ import (
"sync"
"syscall"
"time"
"github.com/novox/mesh-host/internal/outward"
)
// Machine is what this machine is made of, as it says itself (novox/hq issue 160): what somebody
@@ -50,6 +52,11 @@ type Machine struct {
// Filesystems is each local filesystem mounted, by where, with its size and what is free.
Filesystems []Filesystem `json:"filesystems,omitempty"`
// Networks is every network the machine is directly attached to on a link facing outside it, as the
// link and the network's address and prefix (novox/hq ADR 0340 §7): what a port reached from
// `local-network` is resolved from. Every one is said; which count is the controller's judgement.
Networks []outward.Network `json:"networks,omitempty"`
// Unknown names each fact that could not be determined, and why.
Unknown map[string]string `json:"unknown,omitempty"`
}
@@ -79,11 +86,12 @@ const (
FactTimezone = "timezone"
FactFilesystems = "filesystems"
FactPublicAddress = "public address"
FactNetworks = "networks"
)
// MachineFacts is every fact Unknown may name, in the order a reader shows them.
var MachineFacts = []string{FactKernelRelease, FactDistribution, FactVersion, FactProcessor, FactCores,
FactThreads, FactMemory, FactVirtual, FactUptime, FactTimezone, FactFilesystems, FactPublicAddress}
FactThreads, FactMemory, FactVirtual, FactUptime, FactTimezone, FactFilesystems, FactNetworks, FactPublicAddress}
// publicAddressUnknown is why the public uplink address is not said. A machine behind a NAT cannot see
// its public address from itself, and asking a service outside the mesh is a dependency the mesh does not
@@ -101,11 +109,24 @@ type Sources struct {
// Statfs is (size, free) in bytes of the filesystem mounted at a path.
Statfs func(path string) (size, free int64, err error)
Now func() time.Time
// Networks is the networks on the links facing outside this machine (novox/hq ADR 0340).
Networks func() ([]outward.Network, error)
}
// RealSources reads this machine.
func RealSources() Sources {
return Sources{Read: os.ReadFile, Readlink: os.Readlink, Run: ExecRunner, Statfs: statfs, Now: time.Now}
return Sources{Read: os.ReadFile, Readlink: os.Readlink, Run: ExecRunner, Statfs: statfs, Now: time.Now,
Networks: attachedNetworks}
}
// attachedNetworks is this machine's networks on the links that face outside it, read the way the report
// reads those links (novox/hq ADR 0140), so the two never disagree about which links those are.
func attachedNetworks() ([]outward.Network, error) {
links, err := outward.Links("", "")
if err != nil {
return nil, err
}
return outward.Networks(links, nil)
}
func statfs(path string) (int64, int64, error) {
@@ -193,6 +214,18 @@ func DescribeMachine(ctx context.Context, src Sources) Machine {
}
}
// The networks it is attached to, for the local network (novox/hq ADR 0340): every one on a link facing
// outside, or why none is said — never an empty list that reads as a machine on no network.
if src.Networks == nil {
unknown(FactNetworks, "this node-engine was given no way to read its links")
} else if networks, err := src.Networks(); err != nil {
unknown(FactNetworks, err.Error())
} else if len(networks) == 0 {
unknown(FactNetworks, "no link facing outside this machine carries an address")
} else {
m.Networks = networks
}
unknown(FactPublicAddress, publicAddressUnknown)
return m
}
+29
View File
@@ -10,6 +10,8 @@ import (
"sync/atomic"
"testing"
"time"
"github.com/novox/mesh-host/internal/outward"
)
// The parsing layer, with the machine's files given (novox/hq ADR 0017 allows a fake here and only here);
@@ -40,6 +42,9 @@ func given(read map[string]string) Sources {
Run: func(context.Context, string, ...string) (string, error) { return "none\n", errors.New("exited 1") },
Statfs: func(p string) (int64, int64, error) { return 1000, 400, nil },
Now: func() time.Time { return time.Date(2026, 10, 12, 15, 0, 0, 0, time.UTC) },
Networks: func() ([]outward.Network, error) {
return []outward.Network{{Link: "eth0", Network: "192.168.1.0/24"}, {Link: "eth1", Network: "203.0.113.0/24"}}, nil
},
}
}
@@ -92,6 +97,7 @@ func TestMachine_aFactItCannotReadIsSaidUnknownNeverZero(t *testing.T) {
src := given(map[string]string{})
src.Readlink = func(string) (string, error) { return "", os.ErrNotExist }
src.Run = func(context.Context, string, ...string) (string, error) { return "", errors.New("not found") }
src.Networks = func() ([]outward.Network, error) { return nil, errors.New("cannot read the routing table") }
m := DescribeMachine(context.Background(), src)
raw, _ := json.Marshal(m)
@@ -183,3 +189,26 @@ func TestAgainstThisMachine_machineSaysWhatItIs(t *testing.T) {
raw, _ := json.MarshalIndent(m, "", " ")
t.Logf("%s", raw)
}
// **The networks it is attached to are among its facts** (novox/hq ADR 0340 §7), every one — the public
// one too, which the controller judges and never counts — and a machine with none on a link facing
// outside says why rather than reporting an empty list.
func TestMachine_saysTheNetworksItIsAttachedTo(t *testing.T) {
m := DescribeMachine(context.Background(), given(files))
raw, _ := json.Marshal(m)
var said struct {
Networks []map[string]string `json:"networks"`
}
_ = json.Unmarshal(raw, &said)
if len(said.Networks) != 2 || said.Networks[0]["link"] != "eth0" || said.Networks[0]["network"] != "192.168.1.0/24" ||
said.Networks[1]["network"] != "203.0.113.0/24" {
t.Fatalf("networks said as %s", raw)
}
src := given(files)
src.Networks = func() ([]outward.Network, error) { return nil, nil }
m = DescribeMachine(context.Background(), src)
if len(m.Networks) != 0 || !strings.Contains(m.Unknown[FactNetworks], "no link facing outside") {
t.Fatalf("a machine with no network said %v, unknown %q", m.Networks, m.Unknown[FactNetworks])
}
}
+5
View File
@@ -155,6 +155,11 @@ type Applied struct {
// container's spec label; this is the same information kept per file, so that when the spec
// no longer matches the host can say WHICH file changed rather than only that something did.
Reads map[string]string `json:"reads,omitempty"`
// BusySince is, for a container left running because it said busy, when the first apply that would
// have replaced it ran (novox/hq ADR 0335 §3): kept here, beside the machine's state, so the bound runs
// from then across every pass and across a restart of the node-engine. Absent while nothing waits.
BusySince *time.Time `json:"busy_since,omitempty"`
}
// Into is what a file written into held before the mesh's keys, or before the mesh's block.