Compare commits

..
Author SHA1 Message Date
jschoubben fb9c9c3ee8 A taken container keeps a found network, a left-out module is kept, and genesis raises the forge as its module declares (hq ADR 0163)
A container may name networks it also joins once created, for the per-machine
setting that keeps a found network while a neighbour still resolves it there:
joined after the run, part of the spec, refused when it cannot be joined.

A declaration may say which modules the mesh left out because a stored setting
cannot compose with its definition. Absence used to read as removal; a left-out
module's records are kept and said, and its holds are not released.

Genesis raises the bootstrap forge under the gitea module's container name, with
its image digest and its data directory mounted at /data, so the module holds it
by the found rule instead of raising a second forge beside it (issue 090). The
network is the one difference left for a take to say. Before this the forge had
no volume: its repositories were the container's, lost with it.
2026-10-01 23:45:05 +02:00
mesh-admin d53e626366 Merge pull request 'A take is a comparison: the host's facts, former targets, and strays (hq ADR 0163)' (#63) from feat/a-take-is-a-comparison-the-hosts-facts into main 2026-10-01 19:25:56 +00:00
jschoubben 83b3d20e68 A take is a comparison: the host's facts, former targets, and strays (hq ADR 0163)
Every held thing carries what a take compares: for a found container its image and the image's date,
the networks it is on and the other containers on each, its mounts and published ports, beside the
declared image (and its date once pulled), ports and volumes, with the downgrade decided when both
dates are known; for a found file whether the declared content differs and how, as lines lost and
lines new. A resource whose target moved keeps the former target on record as an orphan, so the next
apply removes the container or file the host wrote under the old name (issue 097). Every apply reports
the strays: containers the mesh neither wrote nor holds.
2026-10-01 21:24:37 +02:00
mesh-admin e030aa2387 Merge pull request 'The first user list lets the controller publish assignments and hear its seat's tools (hq #251)' (#62) from fix/first-user-list-matches-the-controller into main 2026-10-01 15:29:55 +00:00
jschoubben c670bef4e1 The first user list lets the controller publish assignments and hear its seat's tools
The controller's own composition (mesh-controller internal/broker, 2026-10-01)
publishes memberships into the assignments stream after each push and
subscribes to its seat's tool subjects; the list the installer carries did
not say so, and a controller on it is refused on the first thing it tries:
"Permissions Violation for Publish to mesh.assignment.novox.builder" (hq #251),
which is why every build asked through the console was lost. The controller's
test that compares the two (TestTheInstallersFirstUserListIsWhatTheControllerWouldCompose,
run with this checkout beside it) named exactly these two subjects, and passes.
2026-10-01 17:29:48 +02:00
mesh-admin 45529bfec2 Merge pull request 'The profile names the network manager that is running, and travels in every report (hq ADR 0161)' (#61) from feat/the-profile-names-the-uplink-and-travels-in-the-report into main 2026-10-01 14:02:16 +00:00
jschoubben b1e9ccff6d The profile names the network manager that is running, and travels in every report (hq ADR 0161)
One capability per manager — uplink-networkmanager, uplink-systemd-networkd, uplink-dhcpcd — from
systemctl is-active, so the uplink seat's holder for a manager this machine does not run is refused
the way any missing capability is, naming it (issue 138). The apply that reports detects the profile
again and sends it, the same shape enrolment sends, so a machine that switched managers reaches the
mesh at its next push.
2026-10-01 15:58:31 +02:00
mesh-admin cbcf0bcc93 Merge pull request 'A node can join the bus the mesh runs on' (#51) from fix/a-node-can-join-the-bus-the-mesh-runs-on into main 2026-10-01 11:18:02 +00:00
jschoubben 6910f07d75 Merge pull request 'Say what a container's host entries now are' (#60) from feat/148-names-are-resolved-not-copied into main 2026-09-30 12:38:15 +00:00
jschoubben 88037b33b7 Say what a container's host entries now are
novox/hq ADR 0148: the mesh's names are no longer among them, only what
the module declared. Comment only; the digest is unchanged.
2026-09-30 14:38:11 +02:00
jschoubben 422ad516d5 Merge pull request 'A declaration carries its order, and a host refuses an older one' (#59) from feat/107-a-declaration-carries-its-order into main 2026-09-30 12:03:10 +00:00
jschoubben 8431ecfb48 A declaration carries its order, and a host refuses an older one
novox/hq 04-ISSUES/107. A declaration's only identity was the digest of
its bytes: a host could say "not the last" and could not say "older". On
the link, nothing refused an older one at all, and the drain picked the
last to arrive — wrong exactly when it mattered, a backlog drained out of
order or a broker that split a burst.

A declaration may now carry a sequence, one higher per send. A host
refuses one lower than what it kept, whole, and says why. The drain keeps
the highest sequence in a batch rather than the last to arrive.

Only when both sides claim an order. Absent reads as zero — "no ordering
claimed", not "first" — so a controller that sends none is still
understood and a host that kept one before it understood them compares
nothing. That is what lets hosts go first and the controller follow, which
is the order 087 says a new field needs.
2026-09-30 14:03:03 +02:00
jschoubben f107d68b2d Merge pull request 'A delivered host knows it is the delivered one' (#58) from fix/163-a-delivered-host-knows-it-is-the-delivered-one into main 2026-09-30 11:46:58 +00:00
jschoubben 1028193c8a A delivered host knows it is the delivered one
novox/hq 04-ISSUES/163. The host asks after every apply whether a newer
host is delivered than the one running, and asked with the link-time
version stamp — which every delivered host carries as "development
build", because the version comes from where the binary sits now (0142).
So a delivered host never matched the newest delivered version, stood
aside on every push for ever, and because standing aside cancels the
report, the mesh never heard from it again.

Measured on two machines: each push produced "host <v> is delivered;
standing aside" for the version already running, then "applied, and could
not tell the mesh: reporting: context canceled". A machine restarting its
host on every push and reporting nothing, reading as healthy.

Asked with the running version now. Half of 0142 was applied to the
report and the known-good record and not here; this is the other half.
2026-09-30 13:46:51 +02:00
jschoubben f576287b51 Merge pull request 'A delivered host reads its version from where it sits' (#57) from fix/161-a-delivered-host-reads-its-version-from-its-path into main 2026-09-30 10:40:29 +00:00
jschoubben 6c6495f6d9 A delivered host reads its version from where it sits
novox/hq ADR 0142, which decided this and was not implemented: "It is
unpacked into a directory named for its version, so it can read its own
version from its path. The stamp goes, and with it the need for a build to
know what it will be called."

The mesh's toolchain stamps no version, on purpose, so a delivered host
called itself "development build" and the mesh could not tell which host
any machine ran — which is the whole of what 087 added. A delivered host
lives at <libexec>/versions/<version>/<binary>, and that directory is the
answer.

A host placed by hand keeps its stamp, which is the honest answer for one
the mesh did not deliver, and is every machine until a delivery reaches
it. A binary sitting anywhere else is not read as a version at all.

The decision is split from the reading so a test can ask about a path
without being that binary.
2026-09-30 12:40:00 +02:00
jschoubben e6d48cf537 Merge pull request 'The mesh delivers the launcher, which is the last link in self-update' (#56) from feat/142-the-mesh-delivers-the-launcher into main 2026-09-30 10:19:07 +00:00
jschoubben b8a766f234 The mesh delivers the launcher, which is the last link in self-update
novox/hq ADR 0141 and 04-ISSUES/142. A version was being delivered to a
machine and nothing started it: the launcher on these machines predates
the versions mechanism and runs the fixed binary path, so the delivery was
correct and inert.

Delivered as a FILE resource, not as part of an archive, and the
difference is the whole reason this is safe. A file is written atomically —
temp file in the same directory, then rename — so the running launcher
keeps the inode it was started from and the next start picks up the new
one. An archive writes in place with truncate, which would cut the file a
running shell is reading halfway through.

The manifest therefore carries a second copy of the script, and a test
refuses any difference between it and packaging/nox-mesh-host-launch.
Proven by drifting one and watching it fail. Two copies of a script is a
bad thing to accept, and the alternative was writing over a running
supervisor.

Together the two resources complete the loop: the version lands, the
running host stands aside because it sees one delivered, and the launcher
that starts next is the one that looks in versions/ and picks the newest
by arrival.
2026-09-30 12:19:00 +02:00
jschoubben 9caea5bc32 Merge pull request 'The delivered binary is named as every machine runs it' (#55) from fix/142-the-delivered-binary-is-named-as-machines-run-it into main 2026-09-30 09:41:52 +00:00
jschoubben df27cee7b7 The delivered binary is named as every machine runs it
nox-mesh-host, not mesh-host. The command directory is cmd/mesh-host and
the launcher looks inside a delivered version for nox-mesh-host — the name
this is installed at and the name in its unit. The first delivery landed
the package's name, reported success, and would have been invisible.
2026-09-30 11:41:45 +02:00
jschoubben d275e64ed3 Merge pull request 'The host declares its own successor, as an archive at a versioned path' (#54) from feat/142-the-host-delivers-its-successor into main 2026-09-30 09:33:19 +00:00
jschoubben 1a628a4d22 The host declares its own successor, as an archive at a versioned path
novox/hq ADR 0141 and 04-ISSUES/142. The host half of the delivery has
been built and tested since 0141 and has never had a version to work on:
versions side by side, the newest runs, the running one stands aside
between reconciles, rollback picks a directory. This is the declaration
that gives it one.

One archive, unpacked to /usr/lib/nox-mesh-host/versions/${version}. The
version resolves to the artifact's digest, so an unchanged build lands at
the path it already had and re-composing a declaration moves nothing.

Not circular: the host applying this is a different version from the one
being written, and neither writes over the other — the kernel refuses to
truncate a running executable, which is the reason the path carries the
version rather than a link pointing at "current".

**The launcher is deliberately not delivered here.** The one on these
machines predates the versions mechanism and runs the fixed binary path,
so a delivered version is inert until it is replaced — and replacing it
from the mesh means writing over a running shell script, which sh reads
incrementally. That wants a designed swap rather than a file resource, and
it is the last piece rather than this one.
2026-09-30 11:33:12 +02:00
jschoubben b5196e974c Merge pull request 'The host is a module, so the mesh can build it' (#53) from feat/142-the-host-is-a-module into main 2026-09-30 07:56:35 +00:00
jschoubben 5162c3b05f The host is a module, so the mesh can build it
novox/hq 04-ISSUES/142 and ADR 0142. Nothing delivered the host because
nothing could compile it, and nothing could compile it partly because the
host was not a thing the mesh builds at all — it had no manifest.

One bundle in Go, for arch, built from cmd/mesh-host. A system is
required for a compiled artifact because a binary is pinned at link time
so a host refuses to touch a machine it was not built for (ADR 0005), and
`arch` is what all four of this mesh's machines report themselves to be.
Another system is another artifact and another build, which is what ADR
0142 means by one per target.

No resources yet. What places a version into a directory named for it
needs an archive resource whose path carries the version, and nothing
interpolates one — the second half of 0141's insight, and the next piece.
2026-09-30 09:56:18 +02:00
jschoubben 98fe8edf35 Merge pull request 'An apply says what it held, not only what it applied' (#52) from fix/125-a-hold-is-a-line-in-the-report into main 2026-09-30 06:47:45 +00:00
jschoubben cbf50185d0 An apply says what it held, not only what it applied
novox/hq 04-ISSUES/125. An adopted node keeps what it found until its
module is taken, which is correct and was recorded only in the node's own
state file. On the edge cut-over the mesh sent 346 resources, the journal
said it applied 330, and nothing anywhere said which sixteen or why —
reading it meant opening state.json by hand, and not reading it took every
public name on the machine down.

The line that reports the apply now carries it, grouped by module and
ordered by name, because the sentence an operator needs is "route-proxy is
assigned and not taken" and the module is the thing `take` acts on. An
apply that held nothing says nothing extra: a line that reports "0 held"
on every converged apply is a line that stops being read.
2026-09-30 08:46:19 +02:00
jschoubben 197258c88c A node can join the bus the mesh runs on
novox/hq 04-ISSUES/146, the layers behind the three already fixed.

A new membership says which bus it is for. Empty meant 'whatever the mesh runs
today' while two buses existed, and became a refusal the moment one did: an
enrolled node came up and reconnected for ever against its own record.

The enrolling client takes its inboxes in the space its user may listen in. A
JetStream publish waits for the stream's acknowledgement on an inbox the client
picks, and its default is one this user may not subscribe to — so the enrolment
failed with a permissions violation on a subject nobody had chosen.

And the enrolment publish carries a message id, so the client's own retry is
discarded by the stream rather than enrolling the machine twice. That one is
not finished: the duplicate survives it, and the issue says where the trail
stops.
2026-09-29 17:36:59 +02:00
jschoubben a3b810f1f0 Merge pull request 'A first node gets as far as its own bus: three faults on the way' (#50) from fix/one-foundation-on-the-bus-the-mesh-runs-on into main 2026-09-29 14:06:33 +00:00
jschoubben 971a6d6d03 A first node gets as far as its own bus: three faults on the way
novox/hq 04-ISSUES/146. Each was right while the mesh ran on the previous
broker, and nothing has raised a foundation since it changed.

The bus's certificate is made by the program that needs it rather than by
openssl inside the broker's image — the bus's image is Alpine with a shell and
no openssl, so the step exited 127 and no mesh could be raised. Self-signed as
before and on purpose; --user 0:0 because the volume is root's and the control
plane's image runs as nobody.

Enrolment no longer opens a raw TLS connection to check the pin: NATS speaks
its own protocol and upgrades afterwards, so the handshake met a plaintext
greeting. The client that presents the token carries the same pinned config
and verifies inside its own handshake, so the secret still leaves only after
the certificate is checked. The raw dial stays as what its tests prove, and is
no longer a path anything takes.

And the token says which bus it is for. Empty meant 'whatever the mesh runs
today' while two buses existed and became a refusal the moment one did.

It now stops at the bus's user list, which is the genesis half of 146.
2026-09-29 15:42:43 +02:00
mesh-admin 04a27caa43 Merge pull request 'A host running as a service says what its apply did' (#49) from fix/a-host-running-as-a-service-says-what-it-did into main 2026-09-29 07:16:01 +00:00
jschoubben 6e90c2692d A host running as a service says what its apply did
The serving path passed nil where the apply writes its detail. Nil is silence, so
everything the apply says — a file held, a container replaced, the found firewall
retired — was visible when a person ran the one-shot command and discarded in the
way the host actually runs, which is always.

Measured: after a machine was converged and its found firewall was not retired,
what the host decided was unrecoverable, because it had said it to nobody. That is
why issue 143 has candidates instead of a cause.

say already reaches stdout and the unit sends that to the journal, so this needed
no new mechanism — only for the argument to be passed. Both paths now reach the
apply through one named helper, so a reader asking where the apply's output goes
finds one answer.

The test asserts the log is never nil and cannot catch the fault it was written
for, which is wiring; that is proved by a deployed host whose journal carries the
detail.
2026-09-29 09:15:53 +02:00
mesh-admin ced54d489f Merge pull request 'A converged machine can speak unasked' (#48) from fix/a-converged-machine-can-speak-unasked into main 2026-09-28 23:13:13 +00:00
29 changed files with 1467 additions and 64 deletions
+166 -26
View File
@@ -51,6 +51,43 @@ var builtFor = ""
var version = "development build" var version = "development build"
// runningVersion is this host's version: the directory it was delivered into, or the link-time stamp
// for one placed by hand.
//
// **From where it sits, not from its linker** (novox/hq ADR 0142): "It is unpacked into a directory
// named for its version, so it can read its own version from its path. The stamp goes, and with it the
// need for a build to know what it will be called."
//
// The mesh's toolchain does not stamp a version, on purpose — a build does not know what it will be
// called — so a delivered host read as "development build" and the mesh could not tell which host any
// machine ran (novox/hq 04-ISSUES/161, and 087 for why that matters). The path knows: a delivered host
// lives at `<libexec>/versions/<version>/<binary>`.
//
// A host placed by hand keeps its stamp, which is the honest answer for one the mesh did not deliver.
func runningVersion() string {
self, err := os.Executable()
if err != nil {
return version
}
return versionAt(self, version)
}
// versionAt is runningVersion's decision, with the executable's path and the link-time stamp given —
// so a test can ask it about a path without being that binary.
func versionAt(self, stamped string) string {
// .../versions/<version>/<binary> — the parent is the version, and its parent is the versions
// directory. Checked rather than assumed, so a binary somewhere else does not read a directory
// name as a version.
dir := filepath.Dir(self)
if filepath.Base(filepath.Dir(dir)) != upgrade.VersionsDirName {
return stamped
}
if name := filepath.Base(dir); name != "" && name != "." && name != string(filepath.Separator) {
return name
}
return stamped
}
const usage = `mesh-host — the node host const usage = `mesh-host — the node host
profile what this machine can be asked to do profile what this machine can be asked to do
@@ -254,7 +291,7 @@ func run(ctx context.Context, command string, opts options) error {
return runLink(ctx, opts) return runLink(ctx, opts)
case "version": case "version":
fmt.Println(version) fmt.Println(runningVersion())
return nil return nil
case "", "help", "-h", "--help": case "", "help", "-h", "--help":
@@ -420,10 +457,10 @@ func short(digest string) string {
// them again — and everything the mesh declared read as no longer declared and removed. Even the // them again — and everything the mesh declared read as no longer declared and removed. Even the
// very declaration the mesh last sent, applied from a file, would plan to remove the foundation. // very declaration the mesh last sent, applied from a file, would plan to remove the foundation.
// `apply FILE` is for a machine the mesh has not spoken to, and is refused saying so. // `apply FILE` is for a machine the mesh has not spoken to, and is refused saying so.
func refuseStale(known store.State, kept store.Declared, keptErr error, digest string, from provenance) error { func refuseStale(known store.State, kept store.Declared, keptErr error, digest string, from provenance, sequence int64) error {
switch from { switch from {
case fromDeclared: case fromDeclared:
return nil return refuseOlder(kept, keptErr, sequence)
case fromBundle: case fromBundle:
if known.Genesis == nil || known.Genesis.Digest == digest { if known.Genesis == nil || known.Genesis.Digest == digest {
return nil return nil
@@ -520,7 +557,7 @@ func runApply(ctx context.Context, opts options, d *declaration.Declaration, raw
if err := apply.CheckMode(known, d); err != nil { if err := apply.CheckMode(known, d); err != nil {
return err return err
} }
if err := refuseStale(known, kept, keptErr, digest, from); err != nil { if err := refuseStale(known, kept, keptErr, digest, from, d.Sequence); err != nil {
return err return err
} }
@@ -594,8 +631,8 @@ func runApply(ctx context.Context, opts options, d *declaration.Declaration, raw
// //
// A failure to record is reported and does not fail the apply. The apply worked; what is // A failure to record is reported and does not fail the apply. The apply worked; what is
// lost is a rollback's ability to come back here, which is worse to hide than to say. // lost is a rollback's ability to come back here, which is worse to hide than to say.
if version != "" { if v := runningVersion(); v != "" {
if err := upgrade.RecordKnownGood(upgrade.KnownGoodPath(opts.state), version); err != nil { if err := upgrade.RecordKnownGood(upgrade.KnownGoodPath(opts.state), v); err != nil {
fmt.Fprintf(os.Stderr, fmt.Fprintf(os.Stderr,
"mesh-host: applied, but could not record %s as known-good: %v\n"+ "mesh-host: applied, but could not record %s as known-good: %v\n"+
" a rollback would have nothing to return to.\n", version, err) " a rollback would have nothing to return to.\n", version, err)
@@ -706,15 +743,18 @@ func enrol(ctx context.Context, opts options) error {
fmt.Printf(" signing key %s\n", fmt.Printf(" signing key %s\n",
base64.StdEncoding.EncodeToString(token.Signer)[:16]+"...") base64.StdEncoding.EncodeToString(token.Signer)[:16]+"...")
// The check that has to happen before this machine says anything. // **The pin is checked by the connection that presents this token, not by a dial of our own**
conn, err := link.Dial(token.Broker, token.Fingerprint, opts.timeout) // (novox/hq 04-ISSUES/146). This opened a raw TLS connection to the bus first, which worked
if err != nil { // against the broker the mesh used to run and cannot work against the one it runs now: NATS
return err // speaks its own protocol before it upgrades to TLS, so an immediate handshake is answered
} // with a plaintext line and the enrolment failed with "first record does not look like a TLS
defer conn.Close() // handshake" — on every node that has tried to join since the bus changed, which is why this
fmt.Println("\nthe broker presented the certificate this token pins") // went unnoticed: none had.
//
conn.Close() // What ADR 0004 requires still holds, and holds better: the client that presents the token
// carries the same pinned configuration, reads the server's greeting, upgrades, and the
// verification runs inside that handshake — so the one-time secret is sent only after the
// certificate has been checked, and nothing of this node's reaches an impostor.
mine, err := identity.Generate(*name) mine, err := identity.Generate(*name)
if err != nil { if err != nil {
@@ -778,20 +818,23 @@ func enrol(ctx context.Context, opts options) error {
// control plane cannot decide what a node should run without it, so it travels with the // control plane cannot decide what a node should run without it, so it travels with the
// request instead of being asked for in a second round trip. // request instead of being asked for in a second round trip.
detected := profile.Detect(ctx, profile.Default(nil), opts.timeout) detected := profile.Detect(ctx, profile.Default(nil), opts.timeout)
reported := map[string]any{} reported := profileAsReported(detected)
if raw, err := json.Marshal(detected); err == nil {
_ = json.Unmarshal(raw, &reported)
}
// Signed with the identity just generated, so the mesh can tell this machine from anyone else // Signed with the identity just generated, so the mesh can tell this machine from anyone else
// who knows its public key (novox/hq issue 083). // who knows its public key (novox/hq issue 083).
proof := mine.Sign(link.EnrolProof(token.Secret, mine.Public, mine.Overlay.Public, proof := mine.Sign(link.EnrolProof(token.Secret, mine.Public, mine.Overlay.Public,
sealing.Public, serving.Public)) sealing.Public, serving.Public))
// The token says where to go and which certificate that address must present. It says nothing // The token says where to go and which certificate that address must present. It says nothing
// about which bus is there, and does not need to: every token names the one the mesh runs on // about which bus is there, and does not need to: there is one, and this host knows which
// today until the rollout (novox/hq ADR 0116 step 5), and that is what an empty Transport is. // (novox/hq ADR 0131 — the mesh speaks to one seat and the old transport is gone).
//
// **It used to leave this empty** and mean "whatever the mesh runs today", which was true
// while two buses existed and became a refusal the moment one did: an empty transport is not
// the bus's name, so every enrolment ended at "this token is for the \"\" bus"
// (novox/hq 04-ISSUES/146). Nothing caught it because nothing had enrolled since the bus
// changed.
reply, err := link.Enrol(ctx, reply, err := link.Enrol(ctx,
link.Approach{Address: token.Broker, Fingerprint: token.Fingerprint}, link.Approach{Address: token.Broker, Fingerprint: token.Fingerprint, Transport: link.OnNATS},
*name, token.Secret, *name, token.Secret,
mine.Public, mine.Overlay.Public, sealing.Public, serving.Public, reported, proof, found, mine.Public, mine.Overlay.Public, sealing.Public, serving.Public, reported, proof, found,
opts.timeout) opts.timeout)
@@ -807,6 +850,13 @@ func enrol(ctx context.Context, opts options) error {
Fingerprint: firstNonEmpty(reply.Fingerprint, token.Fingerprint), Fingerprint: firstNonEmpty(reply.Fingerprint, token.Fingerprint),
Signer: firstNonEmpty2(reply.Signer, token.Signer), Signer: firstNonEmpty2(reply.Signer, token.Signer),
Password: reply.Password, Password: reply.Password,
// **Which bus this membership is for, said rather than left empty** (novox/hq
// 04-ISSUES/146). The link refuses a membership that names another bus, and an empty name
// is not this one's — so a node enrolled without it came up and reconnected for ever
// against its own record: "this membership is for \"\", and the mesh's bus is nats". The
// reply does not carry it because there is one bus and the host knows which (ADR 0131);
// what was missing was writing that down where the link reads it.
Transport: link.OnNATS,
} }
if mine.Membership.Password == "" { if mine.Membership.Password == "" {
// The mesh did not replace the token's secret, so it is still this node's broker // The mesh did not replace the token's secret, so it is still this node's broker
@@ -1008,7 +1058,12 @@ func runLink(ctx context.Context, opts options) error {
// standing before this machine leaves the one it is on (novox/hq design 28, task 5.2). // standing before this machine leaves the one it is on (novox/hq design 28, task 5.2).
adoptDeliveredMembership(identity.Path(opts.state), &mine, say) adoptDeliveredMembership(identity.Path(opts.state), &mine, say)
switch next, waiting, err := upgrade.Successor(upgrade.VersionsDir(""), version); { // Asked with the version this host is RUNNING, read from where it sits — not the link-time
// stamp, which every delivered host carries as "development build". Asked with the stamp,
// a delivered host never matched the newest delivered version, so it stood aside on every
// push for ever, and standing aside cancels the report, so the mesh never heard from it
// again (novox/hq 04-ISSUES/163).
switch next, waiting, err := upgrade.Successor(upgrade.VersionsDir(""), runningVersion()); {
case err != nil: case err != nil:
// Said, not fatal. A host that cannot read the delivered versions is still running this // Said, not fatal. A host that cannot read the delivered versions is still running this
// machine correctly; what it has lost is the ability to be replaced. // machine correctly; what it has lost is the ability to be replaced.
@@ -1246,6 +1301,21 @@ func applyDeclared(ctx context.Context, opts options, raw []byte, sched *apply.S
return applyAndKeep(ctx, opts, raw, nil, sched, say) return applyAndKeep(ctx, opts, raw, nil, sched, say)
} }
// announceOr is what the apply writes its detail with, given what the caller has to say things with.
//
// **Never nil.** This argument was nil on the serving path, and nil is silence: everything the apply
// says — a file held, a container replaced, the found firewall retired — was visible when a person ran
// the one-shot command and discarded in the way the host actually runs (novox/hq 04-ISSUES/143).
//
// Named rather than written inline at the call site so both paths reach the apply the same way, and so
// a reader asking "where does the apply's output go" finds one answer.
func announceOr(say link.Announce) func(string) {
if say == nil {
return func(string) {}
}
return say
}
// applying serialises applies within this process. // applying serialises applies within this process.
// //
// **Two things apply here: the link and the reconcile loop**, and each reads the node's state, // **Two things apply here: the link and the reconcile loop**, and each reads the node's state,
@@ -1308,8 +1378,18 @@ func applyAndKeep(ctx context.Context, opts options, raw []byte, signed *store.D
// Declared, not carried. A declaration from the mesh removes only what the mesh previously // Declared, not carried. A declaration from the mesh removes only what the mesh previously
// declared — never what this machine raised for itself from its bundle (04-ISSUES/010). // declared — never what this machine raised for itself from its bundle (04-ISSUES/010).
// **What the apply says goes to the console, which is the journal when this runs as a service.**
//
// It was nil, and nil is silence. The one-shot path has always passed a real one, so every detail
// the apply produces — a file held, a container replaced, the found firewall retired — was visible
// when a person ran it by hand and discarded in the way the host actually runs. Measured: after a
// machine was converged and its found firewall was not retired, what the host decided was
// unrecoverable, because it had said it to nobody (novox/hq 04-ISSUES/143).
//
// `say` already reaches stdout, and the launcher's unit sends that to the journal, so this needs
// no new mechanism — only for the argument to be passed.
outcome, updated, applyErr := apply.ApplyKeeping(ctx, built, declared, known, store.OriginDeclared, outcome, updated, applyErr := apply.ApplyKeeping(ctx, built, declared, known, store.OriginDeclared,
apply.ExecRunner, nil, sealOpener(opts.state), apply.KeepIn(filepath.Dir(opts.state))) apply.ExecRunner, announceOr(say), sealOpener(opts.state), apply.KeepIn(filepath.Dir(opts.state)))
// The mode the mesh said, recorded whichever way the apply went: the declaration is kept // The mode the mesh said, recorded whichever way the apply went: the declaration is kept
// either way, and the node is held to it from the next reconcile (novox/hq ADR 0100). // either way, and the node is held to it from the next reconcile (novox/hq ADR 0100).
@@ -1335,7 +1415,8 @@ func applyAndKeep(ctx context.Context, opts options, raw []byte, signed *store.D
sched.Sync(declared, held) sched.Sync(declared, held)
} }
report := link.Report{Carried: carriedPorts(updated), Declared: digestOf(raw), Host: version} report := link.Report{Carried: carriedPorts(updated), Declared: digestOf(raw), Host: runningVersion(),
Profile: profileAsReported(profile.Detect(ctx, profile.Default(nil), opts.timeout))}
// Which of this machine's links face outside, for the filter the mesh writes around them // Which of this machine's links face outside, for the filter the mesh writes around them
// (novox/hq ADR 0140). Reported whatever the node's mode: a converged node's filter needs it, // (novox/hq ADR 0140). Reported whatever the node's mode: a converged node's filter needs it,
// and an adopted one becomes converged without a further round trip. A machine that cannot read // and an adopted one becomes converged without a further round trip. A machine that cannot read
@@ -1349,7 +1430,15 @@ func applyAndKeep(ctx context.Context, opts options, raw []byte, signed *store.D
// node never reads as converged (novox/hq ADR 0100). // node never reads as converged (novox/hq ADR 0100).
for _, h := range updated.Held { for _, h := range updated.Held {
report.Held = append(report.Held, link.Held{ID: h.ID, Module: h.Module, Kind: h.Kind, report.Held = append(report.Held, link.Held{ID: h.ID, Module: h.Module, Kind: h.Kind,
Target: h.Target, Since: h.Since, Changed: h.Changed, Kept: h.Kept}) Target: h.Target, Since: h.Since, Changed: h.Changed, Kept: h.Kept, Facts: factsAsReported(h.Facts)})
}
// And what runs here that nobody asked for (novox/hq ADR 0163).
if strays, err := apply.Strays(ctx, apply.ExecRunner, updated); err != nil {
fmt.Fprintf(os.Stderr, "mesh-host: applied, and could not list what else runs here: %v\n", err)
} else {
for _, s := range strays {
report.Strays = append(report.Strays, link.Stray{Kind: s.Kind, Name: s.Name, Detail: s.Detail})
}
} }
if declared.Adoption != nil { if declared.Adoption != nil {
if updated.Firewall != nil { if updated.Firewall != nil {
@@ -1518,3 +1607,54 @@ func adoptDeliveredMembership(identityPath string, mine *identity.Identity, say
say(fmt.Sprintf("moving to the %s bus at %s — restarting to dial it", next.Transport, next.Broker)) say(fmt.Sprintf("moving to the %s bus at %s — restarting to dial it", next.Transport, next.Broker))
os.Exit(0) os.Exit(0)
} }
// refuseOlder refuses a declaration from the mesh that is older than the one this node holds.
//
// **By sequence, not by arrival** (novox/hq 04-ISSUES/107). What the host kept is the last thing
// the mesh said, signed; a declaration whose sequence is lower was composed before it, whatever
// order they arrived in — a backlog drained after the node was away, or a broker that split a burst.
// Applying it would make the machine into something the mesh had already moved past, which is the
// incident of issue 104 by another door.
//
// Only when both sides claim an order. A declaration with no sequence is one an older controller
// sent, and one kept with no sequence is one this host received before it understood them; in either
// case there is no order to compare, and refusing on a guess would strand the node the moment the
// controller is older than the host. Equal is the same declaration again, which reconciling is for.
func refuseOlder(kept store.Declared, keptErr error, sequence int64) error {
if sequence == 0 || keptErr != nil {
return nil
}
last, err := declaration.ParseTrusted(kept.Declaration)
if err != nil || last.Sequence == 0 {
return nil
}
if sequence < last.Sequence {
return fmt.Errorf("this declaration is older than what the mesh last said to this node: it "+
"is sequence %d, and the one kept here is %d. It arrived late — a backlog, or a broker "+
"that split a burst — and applying it would make this machine into something the mesh has "+
"already moved past. Refused whole; nothing was applied", sequence, last.Sequence)
}
return nil
}
// profileAsReported is the profile as the mesh reads it — the same bytes enrolment sends, so a
// report's profile and an enrolment's are one shape on the controller's side (novox/hq ADR 0161).
func profileAsReported(detected profile.Profile) map[string]any {
reported := map[string]any{}
if raw, err := json.Marshal(detected); err == nil {
_ = json.Unmarshal(raw, &reported)
}
return reported
}
// factsAsReported is a held thing's facts as the mesh reads them: the same bytes the host keeps.
func factsAsReported(f *store.Facts) map[string]any {
if f == nil {
return nil
}
out := map[string]any{}
if raw, err := json.Marshal(f); err == nil {
_ = json.Unmarshal(raw, &out)
}
return out
}
+25
View File
@@ -568,3 +568,28 @@ func TestAConvergedMachineSaysItsOutwardLinksUnasked(t *testing.T) {
t.Fatal("a refused report is offered as news about the machine") t.Fatal("a refused report is offered as news about the machine")
} }
} }
// **A host running as a service says what its apply did.**
//
// The serving path passed nil where the apply writes its detail, and nil is silence. The one-shot path
// has always passed a real function, so everything the apply says was visible when a person ran it by
// hand and discarded in the way the host actually runs. Measured before this was written: a machine was
// converged, its found firewall was not retired, and what the host decided was unrecoverable because it
// had been said to nobody (novox/hq 04-ISSUES/143).
//
// This asserts only that the apply's log is never nil and that a line reaches what the caller gave.
// **It cannot catch the fault it was written for** — a call site passing nil directly — because that is
// wiring, and wiring is only proved by running the thing. That proof is a deployed host whose journal
// carries the apply's detail, which is how this fix was verified.
func TestTheApplysLogIsNeverNil(t *testing.T) {
if announceOr(nil) == nil {
t.Fatal("a host with nowhere to say things got a nil log, which the apply will call")
}
announceOr(nil)("this goes nowhere and must not panic")
var said []string
announceOr(func(line string) { said = append(said, line) })(" disabled ufw")
if len(said) != 1 || !strings.Contains(said[0], "disabled ufw") {
t.Fatalf("the apply's detail did not reach the caller's announce: %v", said)
}
}
+58
View File
@@ -0,0 +1,58 @@
package main
import (
"encoding/json"
"strings"
"testing"
"github.com/novox/mesh-host/internal/store"
)
// A declaration carries no order, so a host cannot tell an older one from a newer (novox/hq
// 04-ISSUES/107). Its only identity was the digest of its bytes: "not the last" could be said,
// "older" could not.
func keptWith(t *testing.T, sequence int64) store.Declared {
t.Helper()
body, err := json.Marshal(map[string]any{
"declaration": 1, "resources": []any{}, "owns_nothing": true, "sequence": sequence,
})
if err != nil {
t.Fatal(err)
}
return store.Declared{Declaration: body, Signature: []byte("x")}
}
func TestAnOlderDeclarationFromTheMeshIsRefused(t *testing.T) {
err := refuseOlder(keptWith(t, 7), nil, 5)
if err == nil {
t.Fatal("sequence 5 was accepted over a kept 7")
}
if !strings.Contains(err.Error(), "older") || !strings.Contains(err.Error(), "nothing was applied") {
t.Fatalf("the refusal does not say what it is: %v", err)
}
}
func TestANewerOrEqualDeclarationIsNot(t *testing.T) {
if err := refuseOlder(keptWith(t, 7), nil, 8); err != nil {
t.Fatalf("sequence 8 was refused over a kept 7: %v", err)
}
// Equal is the same declaration again, which reconciling is for.
if err := refuseOlder(keptWith(t, 7), nil, 7); err != nil {
t.Fatalf("the same sequence was refused: %v", err)
}
}
func TestNoOrderClaimedMeansNoOrderCompared(t *testing.T) {
// An older controller sends none; a host that received before it understood them kept none.
// Refusing on a guess would strand a node the moment the controller is older than the host.
if err := refuseOlder(keptWith(t, 7), nil, 0); err != nil {
t.Fatalf("a declaration claiming no order was refused: %v", err)
}
if err := refuseOlder(keptWith(t, 0), nil, 3); err != nil {
t.Fatalf("a declaration was refused against a kept one that claimed no order: %v", err)
}
if err := refuseOlder(store.Declared{}, store.ErrNothingDeclared, 3); err != nil {
t.Fatalf("a first declaration was refused: %v", err)
}
}
+48
View File
@@ -0,0 +1,48 @@
package main
import (
"os"
"path/filepath"
"testing"
)
// A component's version comes from where it sits, not from its linker (novox/hq ADR 0142). The mesh's
// toolchain stamps no version — a build does not know what it will be called — so a delivered host
// read as "development build" and the mesh could not tell which host a machine ran (04-ISSUES/161).
func TestADeliveredHostReadsItsVersionFromItsPath(t *testing.T) {
// A delivered host lives at <libexec>/versions/<version>/<binary>.
dir := t.TempDir()
versioned := filepath.Join(dir, "versions", "637f65559d16")
if err := os.MkdirAll(versioned, 0o755); err != nil {
t.Fatal(err)
}
self := filepath.Join(versioned, "nox-mesh-host")
if err := os.WriteFile(self, []byte("#!/bin/sh\n"), 0o755); err != nil {
t.Fatal(err)
}
if got := versionAt(self, "development build"); got != "637f65559d16" {
t.Fatalf("a delivered host read its version as %q", got)
}
}
func TestAHostPlacedByHandKeepsItsStamp(t *testing.T) {
// The honest answer for one the mesh did not deliver — and every machine is in that state until
// a delivery reaches it.
if got := versionAt("/usr/bin/nox-mesh-host", "04a27ca"); got != "04a27ca" {
t.Fatalf("a hand-placed host read its version as %q", got)
}
}
func TestADirectoryThatIsNotAVersionIsNotReadAsOne(t *testing.T) {
// A binary sitting anywhere else must not have its parent directory's name read as a version.
for _, path := range []string{
"/opt/somewhere/nox-mesh-host",
"/usr/lib/nox-mesh-host/launch",
"/home/someone/build/nox-mesh-host",
} {
if got := versionAt(path, "the stamp"); got != "the stamp" {
t.Fatalf("%s read its version as %q", path, got)
}
}
}
+16 -7
View File
@@ -127,12 +127,21 @@
{ {
"id": "bus-certificate", "id": "bus-certificate",
"type": "action", "type": "action",
"command": ["docker", "run", "--rm", "--entrypoint", "sh", "-v", "mesh-broker-tls:/tls", // **The mesh makes its own** (novox/hq 04-ISSUES/146). This ran `openssl` inside the
"192.0.2.250:5000/nats@sha256:b83efabe3e7def1e0a4a31ec6e078999bb17c80363f881df35edc70fcb6bb927", // broker's image while the broker was one that carried it; the bus that replaced it has a
"-c", "test -f /tls/tls.crt || (openssl req -x509 -newkey rsa:2048 -nodes -keyout /tls/tls.key -out /tls/tls.crt -days 3650 -subj '/CN=mesh-broker' -addext 'subjectAltName=DNS:mesh-broker,IP:127.0.0.1' >/dev/null 2>&1 && chmod 644 /tls/tls.crt && chmod 600 /tls/tls.key)"], // shell and no openssl, and no other image the bundle names has one either. So the program
"verify": ["docker", "run", "--rm", "--entrypoint", "sh", "-v", "mesh-broker-tls:/tls", // that needs the certificate writes it — already on this machine, since the schema step ran
"192.0.2.250:5000/nats@sha256:b83efabe3e7def1e0a4a31ec6e078999bb17c80363f881df35edc70fcb6bb927", // it, and asking nothing of the image it writes into. Self-signed on purpose: a host pins
"-c", "test -s /tls/tls.crt && openssl x509 -in /tls/tls.crt -noout"] // this server's exact certificate (novox/hq ADR 0004), and at this moment there is no mesh
// to ask an authority of.
// `--user 0:0` because the volume is root's and this image runs as nobody, which is right
// for the long-running control plane and wrong for a one-shot writing into a fresh volume.
"command": ["docker", "run", "--rm", "--user", "0:0", "-v", "mesh-broker-tls:/tls",
"192.0.2.250:5000/mesh-controller@sha256:c67db38439ff0aee242b467486765467bb95801f52175fc5727cc4e437338ace",
"broker", "certificate", "--into", "/tls"],
"verify": ["docker", "run", "--rm", "--user", "0:0", "-v", "mesh-broker-tls:/tls",
"192.0.2.250:5000/mesh-controller@sha256:c67db38439ff0aee242b467486765467bb95801f52175fc5727cc4e437338ace",
"broker", "certificate", "--check", "--into", "/tls"]
}, },
{ {
"id": "bus-conf-dir", "id": "bus-conf-dir",
@@ -152,7 +161,7 @@
"type": "file", "type": "file",
"path": "/var/lib/mesh-bus-conf/accounts.conf", "path": "/var/lib/mesh-bus-conf/accounts.conf",
"mode": "0600", "mode": "0600",
"content": "// The first user list, carried by the installer because at genesis there is no mesh to\n// compose one. A bootstrap credential, rotated with the store's and replaced by the\n// controller's own composition from its first start onward.\naccounts {\n MESH {\n jetstream: enabled\n users = [\n { user: \"controller\", password: \"$2a$10$AHqJgOifIVbU41KmATiMhuXFs8xa7Wl2HuN4UVBCXdN2jIQzjqApy\", permissions: {\n publish: { allow: [\"$JS.API.>\", \"$JS.ACK.CONTROL.controller.>\", \"$JS.ACK.EVENTS.controller.>\", \"_INBOX.enrol.>\", \"mesh.control.>\", \"mesh.mod.*.tool.>\", \"mesh.node.>\", \"mesh.seat.mesh-build-machine.accept.>\", \"mesh.seat.mesh-controller.event.applied\", \"mesh.seat.mesh-controller.event.built-before\", \"mesh.seat.mesh-controller.event.refused\"] }\n subscribe: { allow: [\"$JS.API.>\", \"_DELIVER.controller\", \"_DELIVER.controller.>\", \"_INBOX.controller.>\", \"mesh.control.>\", \"mesh.mod.mesh-catalog.event.catching-up\", \"mesh.mod.mesh-catalog.event.upgraded\", \"mesh.mod.gitea.event.pull.merged\", \"mesh.seat.mesh-build-machine.event.built\"] }\n allow_responses: { max: 1, ttl: \"1m\" }\n } }\n ]\n }\n}\n" "content": "// The first user list, carried by the installer because at genesis there is no mesh to\n// compose one. A bootstrap credential, rotated with the store's and replaced by the\n// controller's own composition from its first start onward.\naccounts {\n MESH {\n jetstream: enabled\n users = [\n { user: \"controller\", password: \"$2a$10$AHqJgOifIVbU41KmATiMhuXFs8xa7Wl2HuN4UVBCXdN2jIQzjqApy\", permissions: {\n publish: { allow: [\"$JS.ACK.CONTROL.controller.>\", \"$JS.ACK.EVENTS.controller.>\", \"$JS.API.>\", \"_INBOX.enrol.>\", \"mesh.assignment.>\", \"mesh.control.>\", \"mesh.mod.*.tool.>\", \"mesh.node.>\", \"mesh.seat.mesh-build-machine.accept.>\", \"mesh.seat.mesh-controller.event.applied\", \"mesh.seat.mesh-controller.event.built-before\", \"mesh.seat.mesh-controller.event.refused\"] }\n subscribe: { allow: [\"$JS.API.>\", \"_DELIVER.controller\", \"_DELIVER.controller.>\", \"_INBOX.controller.>\", \"mesh.control.>\", \"mesh.mod.gitea.event.pull.merged\", \"mesh.mod.mesh-catalog.event.catching-up\", \"mesh.mod.mesh-catalog.event.upgraded\", \"mesh.seat.mesh-build-machine.event.built\", \"mesh.seat.mesh-controller.tool.>\"] }\n allow_responses: { max: 1, ttl: \"1m\" }\n } }\n ]\n }\n}\n"
}, },
{ {
"id": "broker", "id": "broker",
+41 -7
View File
@@ -20,6 +20,7 @@ import (
"os" "os"
"os/exec" "os/exec"
"path/filepath" "path/filepath"
"slices"
"sort" "sort"
"strconv" "strconv"
"strings" "strings"
@@ -230,6 +231,18 @@ func ApplyKeeping(
protecting = append(protecting, orphan) protecting = append(protecting, orphan)
continue continue
} }
// **A module the mesh left out is not a module the mesh removed** (novox/hq ADR 0163, rule
// 6): its resources are absent because a setting stored for it cannot compose, and the
// mesh said so by name. What the host wrote for it stays as it is, recorded, until the
// module is declared again or unassigned.
if module, left := d.LeftOutModuleOf(orphan.ID); left {
report.Outcomes = append(report.Outcomes, Outcome{
ID: orphan.ID, Type: orphan.Type, Target: orphan.Target,
Action: "unchanged", Detail: "kept: " + module + " was left out of this declaration by the mesh, not removed",
})
log(fmt.Sprintf(" kept %s (%s): %s was left out of this declaration by the mesh, not removed", orphan.ID, orphan.Target, module))
continue
}
orphans = append(orphans, orphan) orphans = append(orphans, orphan)
} }
ordered := d.Resources ordered := d.Resources
@@ -270,6 +283,11 @@ func ApplyKeeping(
if declared[h.ID] { if declared[h.ID] {
continue continue
} }
if slices.Contains(d.LeftOut, h.Module) {
// Left out, not unassigned (ADR 0163, rule 6): still held for the module, as the
// mesh asked.
continue
}
known.Release(h.ID) known.Release(h.ID)
report.Outcomes = append(report.Outcomes, Outcome{ID: h.ID, Type: h.Kind, Target: h.Target, report.Outcomes = append(report.Outcomes, Outcome{ID: h.ID, Type: h.Kind, Target: h.Target,
Action: "forgotten", Detail: "no longer declared; left as found"}) Action: "forgotten", Detail: "no longer declared; left as found"})
@@ -1501,13 +1519,15 @@ func containerSpecReading(r *declaration.Container, declares, reads map[string]s
for _, a := range r.Args { for _, a := range r.Args {
b.WriteString("arg " + a + "\n") b.WriteString("arg " + a + "\n")
} }
// **The mesh's names are part of what a container is** (novox/hq 04-ISSUES/135). A container // **A container's declared names are part of what it is** (novox/hq 04-ISSUES/135). What is
// resolves every other machine and every public name through the entries the mesh gives it at // here is what the module declared for itself and nothing else: the mesh's own names are no
// creation, and nothing re-reads them afterwards — so a container left alone when the roster // longer written into a container (ADR 0148) — they were once, every container got the whole
// moved is one that cannot reach anything by name, for ever, while every check reports it // roster at creation and nothing re-read it, so one left alone when the roster moved could not
// running. That is exactly what happened when this mesh's overlay range changed: one container // reach anything by name for as long as it ran while every check reported it running; and once
// whose image and files never changed kept an address five days out of date and restarted // the roster was in this digest so that could be caught, one name moving anywhere replaced
// 2286 times against a database it could no longer find. // every container in the mesh (04-ISSUES/151). A container resolves a mesh name through the
// machine's resolver at the moment it asks. What a module declares does not move when the
// roster does, so hashing it costs nothing and catches a manifest that changed.
// //
// Sorted, so the digest does not move for a reordering nobody made. // Sorted, so the digest does not move for a reordering nobody made.
hosts := append([]string(nil), r.Hosts...) hosts := append([]string(nil), r.Hosts...)
@@ -1524,6 +1544,11 @@ func containerSpecReading(r *declaration.Container, declares, reads map[string]s
if r.IP != "" { if r.IP != "" {
b.WriteString("ip " + r.IP + "\n") b.WriteString("ip " + r.IP + "\n")
} }
// The networks it also joins are part of what it is (ADR 0163, rule 4): kept or let go, the
// container is recreated, and a neighbour's reach changes with it.
for _, n := range r.Networks {
b.WriteString("also-on " + n + "\n")
}
// The cadence is part of what was declared, so a changed schedule is a changed spec — the marker // The cadence is part of what was declared, so a changed schedule is a changed spec — the marker
// moves and the install is reported "updated" and re-established. Added only when present, so no // moves and the install is reported "updated" and re-established. Added only when present, so no
// ordinary container's or run-once step's digest moves for a field it does not set. // ordinary container's or run-once step's digest moves for a field it does not set.
@@ -1764,6 +1789,15 @@ func applyContainer(ctx context.Context, r *declaration.Container, run Runner,
if after.Spec != want { if after.Spec != want {
return out, fmt.Errorf("container %s is not the one that was declared after creating it", r.Name) return out, fmt.Errorf("container %s is not the one that was declared after creating it", r.Name)
} }
// The found networks a per-machine setting keeps for it (novox/hq ADR 0163, rule 4), joined
// once it runs: a runtime starts a container on one network, and the others are connected.
// Refused, not skipped, when one cannot be joined — a neighbour that was promised to keep
// reaching this container by name would silently not.
for _, n := range r.Networks {
if _, err := run(ctx, cri, "network", "connect", n, r.Name); err != nil {
return out, fmt.Errorf("container %s could not join the kept network %s: %w", r.Name, n, err)
}
}
out.Action = "created" out.Action = "created"
if existed { if existed {
+97
View File
@@ -0,0 +1,97 @@
package apply
import (
"context"
"os"
"path/filepath"
"strings"
"testing"
"github.com/novox/mesh-host/internal/store"
)
// A take is a comparison (novox/hq ADR 0163): while a module's container is held, the host reports
// the found image and its age beside the declared one, the networks and who else is on them, the
// mounts and the ports — and says when the declared image is the older.
func TestAHeldContainerCarriesTheFactsATakeCompares(t *testing.T) {
dir, page, m := predecessor(t)
m.containers["hello-web"].image = "web:1.27"
m.containers["hello-web"].imageID = "sha256:found"
m.containers["hello-web"].networks = []string{"predecessor_default"}
m.containers["hello-web"].mounts = []string{"/srv/web:/data"}
m.containers["hello-web"].ports = []string{"80/tcp>0.0.0.0:8080"}
m.images = map[string]string{"sha256:found": "2026-09-17T10:00:00Z", pinned: "2026-08-20T10:00:00Z"}
m.members = map[string][]string{"predecessor_default": {"hello-web", "office", "db"}}
_, state := applyAdopted(t, adopted(t, untakenWeb, webResources(page)), store.State{}, m, dir)
h, ok := state.HeldAt("hello-web.server")
if !ok || h.Facts == nil {
t.Fatalf("a held container carries no facts: %+v", h)
}
f := h.Facts
if f.Image != "web:1.27" || f.ImageCreated != "2026-09-17T10:00:00Z" {
t.Errorf("the found image and its age: %+v", f)
}
if f.DeclaredImage != pinned || f.DeclaredImageCreated != "2026-08-20T10:00:00Z" || !f.Downgrade {
t.Errorf("the declared image, its age, and that it is a downgrade: %+v", f)
}
if got := f.Networks["predecessor_default"]; len(got) != 2 || got[0] != "db" || got[1] != "office" {
t.Errorf("the neighbours on the found network, without the container itself: %v", f.Networks)
}
if len(f.Mounts) != 1 || f.Mounts[0] != "/srv/web:/data" || len(f.Ports) != 1 || f.Ports[0] != "80/tcp>0.0.0.0:8080" {
t.Errorf("mounts and ports as found: %+v", f)
}
// And the held file carries how the declared content differs from what was found.
p, ok := state.HeldAt("hello-web.page")
if !ok || p.Facts == nil || !p.Facts.Differs {
t.Fatalf("a held file that differs from the declared content does not say so: %+v", p)
}
joined := strings.Join(p.Facts.Difference, "\n")
if !strings.Contains(joined, "- the predecessor's page") || !strings.Contains(joined, "+ the mesh's page") {
t.Errorf("the difference does not show what is lost and what is new: %q", joined)
}
_ = os.Remove(filepath.Join(dir, "unused"))
}
// A declared image not yet on the machine leaves its age unknown and the comparison undecided.
func TestAnImageNotYetPulledLeavesTheDowngradeUndecided(t *testing.T) {
dir, page, m := predecessor(t)
m.containers["hello-web"].image = "web:1.27"
m.containers["hello-web"].imageID = "sha256:found"
m.images = map[string]string{"sha256:found": "2026-09-17T10:00:00Z"}
_, state := applyAdopted(t, adopted(t, untakenWeb, webResources(page)), store.State{}, m, dir)
h, _ := state.HeldAt("hello-web.server")
if h.Facts == nil || h.Facts.DeclaredImageCreated != "" || h.Facts.Downgrade {
t.Fatalf("an unknown declared age decided a downgrade: %+v", h.Facts)
}
}
func TestTheDifferenceIsWhatIsLostAndWhatIsNew(t *testing.T) {
differs, lines := differenceOf("a\nprivate scope: local\nb\n", "a\nb\nupstream: public\n")
if !differs || len(lines) != 2 || lines[0] != "- private scope: local" || lines[1] != "+ upstream: public" {
t.Fatalf("got %v %v", differs, lines)
}
if differs, lines := differenceOf("same\n", "same\n"); differs || lines != nil {
t.Fatalf("identical content differs: %v %v", differs, lines)
}
}
// What runs on the machine that the mesh neither wrote nor holds is reported (ADR 0163).
func TestStraysAreWhatRunsHereThatNobodyAsked(t *testing.T) {
m := &machine{containers: map[string]*fakeContainer{
"hello-web": {id: "ours", running: true, image: "web:1"},
"gitea-old": {id: "left-behind", running: true, image: "gitea:1.22"},
"held-thing": {id: "found", running: true, image: "x:1"},
}}
known := store.State{
Resources: []store.Applied{{ID: "hello-web.server", Type: "container", Target: "hello-web"}},
Held: []store.Held{{ID: "other.server", Kind: "container", Target: "held-thing"}},
}
strays, err := Strays(context.Background(), m.run, known)
if err != nil {
t.Fatal(err)
}
if len(strays) != 1 || strays[0].Name != "gitea-old" || !strings.Contains(strays[0].Detail, "gitea:1.22") {
t.Fatalf("strays: %+v", strays)
}
}
+125 -4
View File
@@ -8,6 +8,7 @@ import (
"fmt" "fmt"
"os" "os"
"path/filepath" "path/filepath"
"sort"
"strings" "strings"
"syscall" "syscall"
"time" "time"
@@ -433,6 +434,32 @@ type foundContainer struct {
id string id string
running bool running bool
spec string spec string
// What a take compares (novox/hq ADR 0163): the image and its id, the networks the container
// is on, its mounts and its published ports — empty from a runtime (or a test's fake) that
// answers the short form.
image string
imageID string
networks []string
mounts []string
ports []string
}
// foundFormat is what inspectFound asks the runtime for, tab-separated: the three a hold has
// always needed, then the facts a take compares.
const foundFormat = "{{.Id}}\t{{.State.Running}}\t{{index .Config.Labels \"" + specLabel + "\"}}" +
"\t{{.Config.Image}}\t{{.Image}}" +
"\t{{range $k, $v := .NetworkSettings.Networks}}{{$k}},{{end}}" +
"\t{{range .Mounts}}{{.Source}}:{{.Destination}},{{end}}" +
"\t{{range $p, $b := .NetworkSettings.Ports}}{{$p}}{{range $b}}>{{.HostIp}}:{{.HostPort}}{{end}},{{end}}"
func splitList(s string) []string {
var out []string
for _, part := range strings.Split(s, ",") {
if part = strings.TrimSpace(part); part != "" {
out = append(out, part)
}
}
return out
} }
// inspectFound reads a container by name the way a hold needs it: its id, whether it runs, and // inspectFound reads a container by name the way a hold needs it: its id, whether it runs, and
@@ -451,8 +478,7 @@ func inspectFound(ctx context.Context, name string, run Runner) (foundContainer,
if err != nil { if err != nil {
return foundContainer{}, false, fmt.Errorf("%w, so nothing can be said about %q", err, name) return foundContainer{}, false, fmt.Errorf("%w, so nothing can be said about %q", err, name)
} }
out, err := run(ctx, cri, "container", "inspect", "--format", out, err := run(ctx, cri, "container", "inspect", "--format", foundFormat, name)
"{{.Id}}\t{{.State.Running}}\t{{index .Config.Labels \""+specLabel+"\"}}", name)
if err != nil { if err != nil {
if absent(err) { if absent(err) {
return foundContainer{}, false, nil return foundContainer{}, false, nil
@@ -466,14 +492,100 @@ func inspectFound(ctx context.Context, name string, run Runner) (foundContainer,
name, err) name, err)
} }
parts := strings.Split(strings.TrimSpace(out), "\t") parts := strings.Split(strings.TrimSpace(out), "\t")
for len(parts) < 3 { for len(parts) < 8 {
parts = append(parts, "") parts = append(parts, "")
} }
spec := strings.TrimSpace(parts[2]) spec := strings.TrimSpace(parts[2])
if spec == "<no value>" { if spec == "<no value>" {
spec = "" spec = ""
} }
return foundContainer{id: strings.TrimSpace(parts[0]), running: parts[1] == "true", spec: spec}, true, nil return foundContainer{id: strings.TrimSpace(parts[0]), running: parts[1] == "true", spec: spec,
image: strings.TrimSpace(parts[3]), imageID: strings.TrimSpace(parts[4]),
networks: splitList(parts[5]), mounts: splitList(parts[6]), ports: splitList(parts[7])}, true, nil
}
// factsOf is what a take would compare for a found container (novox/hq ADR 0163): the found
// image and when it was made, the networks and who else is on them, mounts and ports — beside
// what the module declares, and the declared image's date when that image is on the machine.
// Every question the runtime cannot answer leaves its fact empty; a preview says so rather than
// guesses.
func factsOf(ctx context.Context, seen foundContainer, res *declaration.Container, run Runner) *Facts {
cri, err := containerRuntime(ctx, run)
if err != nil {
return nil
}
f := &store.Facts{Image: seen.image, Mounts: seen.mounts, Ports: seen.ports,
DeclaredImage: res.Image, DeclaredPorts: res.Ports, DeclaredVolumes: res.Volumes}
if seen.imageID != "" {
if out, err := run(ctx, cri, "image", "inspect", "--format", "{{.Created}}", seen.imageID); err == nil {
f.ImageCreated = strings.TrimSpace(out)
}
}
if res.Image != "" {
if out, err := run(ctx, cri, "image", "inspect", "--format", "{{.Created}}", res.Image); err == nil {
f.DeclaredImageCreated = strings.TrimSpace(out)
}
}
if found, err := time.Parse(time.RFC3339Nano, f.ImageCreated); err == nil {
if declared, err := time.Parse(time.RFC3339Nano, f.DeclaredImageCreated); err == nil {
f.Downgrade = declared.Before(found)
}
}
for _, network := range seen.networks {
if f.Networks == nil {
f.Networks = map[string][]string{}
}
var members []string
if out, err := run(ctx, cri, "network", "inspect", "--format",
"{{range .Containers}}{{.Name}},{{end}}", network); err == nil {
for _, m := range splitList(out) {
if m != res.Name {
members = append(members, m)
}
}
}
sort.Strings(members)
f.Networks[network] = members
}
return (*Facts)(f)
}
// Facts is store.Facts, named here so hold's callers read as one vocabulary.
type Facts = store.Facts
// differenceOf is how a found file differs from the declared content: the lines only the found
// file has, marked -, then the lines only the declared content has, marked +, in their own order,
// bounded so a report stays a report. Not a diff tool's output: the question a take answers is
// "what would be lost and what would be new", and that is these two lists.
func differenceOf(found, declared string) (bool, []string) {
if found == declared {
return false, nil
}
const bound = 40
count := func(s string) map[string]int {
out := map[string]int{}
for _, line := range strings.Split(s, "\n") {
out[line]++
}
return out
}
inFound, inDeclared := count(found), count(declared)
var out []string
add := func(mark, s string, other map[string]int) {
seen := map[string]int{}
for _, line := range strings.Split(s, "\n") {
seen[line]++
if seen[line] > other[line] && len(out) < bound {
out = append(out, mark+" "+line)
}
}
}
add("-", found, inDeclared)
add("+", declared, inFound)
if len(out) >= bound {
out = append(out, "… and more")
}
return true, out
} }
// absent is whether a runtime said the thing is not there, rather than failing to answer. Its own // absent is whether a runtime said the thing is not there, rather than failing to answer. Its own
@@ -540,6 +652,12 @@ func hold(ctx context.Context, sys system.System, r declaration.Resource, module
} else if digestOf(string(content)) != h.Digest { } else if digestOf(string(content)) != h.Digest {
changed = "rewritten" changed = "rewritten"
} }
// What a take would replace it with, and how that differs (novox/hq ADR 0163): a
// file declared whole is compared whole; one written into is not replaced at all.
if res.Into == "" {
differs, lines := differenceOf(string(content), res.Content)
h.Facts = &Facts{Differs: differs, Difference: lines}
}
} }
case *declaration.Directory: case *declaration.Directory:
info, err := os.Lstat(res.Path) info, err := os.Lstat(res.Path)
@@ -628,6 +746,9 @@ func hold(ctx context.Context, sys system.System, r declaration.Resource, module
case h.Running && !seen.running: case h.Running && !seen.running:
changed = "stopped" changed = "stopped"
} }
if exists {
h.Facts = factsOf(ctx, seen, res, run)
}
default: default:
return out, h, fmt.Errorf("a %s cannot be held", r.Kind()) return out, h, fmt.Errorf("a %s cannot be held", r.Kind())
} }
+39 -2
View File
@@ -5,6 +5,7 @@ import (
"errors" "errors"
"os" "os"
"path/filepath" "path/filepath"
"sort"
"strings" "strings"
"testing" "testing"
@@ -18,7 +19,11 @@ import (
// label when a host made it. Every command it is asked is written down. // label when a host made it. Every command it is asked is written down.
type machine struct { type machine struct {
containers map[string]*fakeContainer containers map[string]*fakeContainer
asked []string // images is what `image inspect --format {{.Created}}` answers per image or id; members is
// what `network inspect` lists per network (ADR 0163).
images map[string]string
members map[string][]string
asked []string
// wgUp is what `wg show interfaces` answers: the tunnels up on the machine. // wgUp is what `wg show interfaces` answers: the tunnels up on the machine.
wgUp string wgUp string
// handshakes is what `wg show <interface> latest-handshakes` answers, and handshakesFail the // handshakes is what `wg show <interface> latest-handshakes` answers, and handshakesFail the
@@ -97,6 +102,9 @@ type fakeContainer struct {
id string id string
running bool running bool
spec string spec string
// What a take compares (ADR 0163), answered in the long inspect form when set.
image, imageID string
networks, mounts, ports []string
} }
func (m *machine) run(_ context.Context, name string, args ...string) (string, error) { func (m *machine) run(_ context.Context, name string, args ...string) (string, error) {
@@ -140,9 +148,38 @@ func (m *machine) run(_ context.Context, name string, args ...string) (string, e
running = "true" running = "true"
} }
if strings.HasPrefix(args[3], "{{.Id}}") { if strings.HasPrefix(args[3], "{{.Id}}") {
return c.id + "\t" + running + "\t" + c.spec + "\n", nil line := c.id + "\t" + running + "\t" + c.spec
if c.image != "" {
line += "\t" + c.image + "\t" + c.imageID + "\t" + strings.Join(c.networks, ",") + "," +
"\t" + strings.Join(c.mounts, ",") + "," + "\t" + strings.Join(c.ports, ",") + ","
}
return line + "\n", nil
} }
return running + "\t" + c.spec + "\n", nil return running + "\t" + c.spec + "\n", nil
case "image":
if len(args) > 1 && args[1] == "inspect" {
if created, ok := m.images[args[len(args)-1]]; ok {
return created + "\n", nil
}
return "", errors.New("no such image")
}
return "", nil
case "network":
if len(args) > 1 && args[1] == "inspect" {
return strings.Join(m.members[args[len(args)-1]], ",") + ",\n", nil
}
return "", nil
case "ps":
var lines []string
for name, c := range m.containers {
state := "exited"
if c.running {
state = "running"
}
lines = append(lines, name+"\t"+c.image+"\t"+state)
}
sort.Strings(lines)
return strings.Join(lines, "\n") + "\n", nil
case "rm": case "rm":
delete(m.containers, args[len(args)-1]) delete(m.containers, args[len(args)-1])
return "", nil return "", nil
+136
View File
@@ -0,0 +1,136 @@
package apply
import (
"context"
"errors"
"strings"
"testing"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/store"
)
// A taken container keeps a found network by a per-machine setting (novox/hq ADR 0163, rule 4):
// joined once it runs, part of its spec, and refused when it cannot be joined.
func TestAContainerJoinsTheNetworksItKeeps(t *testing.T) {
var ran []string
connectFails := false
run := func(_ context.Context, name string, args ...string) (string, error) {
if name != "docker" {
return "", errors.New("not installed")
}
ran = append(ran, strings.Join(args, " "))
switch args[0] {
case "info":
return "29.0.0\n", nil
case "container":
if len(ran) > 2 {
return "true\t" + specOfLast, nil
}
return "false\t\n", errors.New("no such container")
case "run":
return "deadbeef\n", nil
case "network":
if connectFails {
return "", errors.New("network predecessor_default not found")
}
}
return "", nil
}
d := parseTrusted(t, `{"declaration":1,"resources":[
{"id":"app","type":"container","name":"app","image":"`+pinned+`",
"networks":["predecessor_default"]}
]}`)
specOfLast = containerSpec(d.Resources[0].(*declaration.Container), inputs{})
alone := *d.Resources[0].(*declaration.Container)
alone.Networks = nil
if specOfLast == containerSpec(&alone, inputs{}) {
t.Fatal("the kept network is not part of the container's spec: kept or let go, the container would be left alone")
}
report, _, err := Apply(context.Background(), archHost(t), d, store.State{}, store.OriginCarried, run, nil, nil)
if err != nil {
t.Fatal(err)
}
joined := false
for i, line := range ran {
if line == "network connect predecessor_default app" {
joined = true
if ran[i-1] != "container inspect --format {{.State.Running}}\t{{index .Config.Labels \""+specLabel+"\"}} app" &&
!strings.HasPrefix(ran[i-1], "container inspect") {
t.Errorf("joined before the container was read back as running: %v", ran)
}
}
}
if !joined || report.Outcomes[0].Action != "created" {
t.Fatalf("the container did not join the kept network: %v\n%+v", ran, report.Outcomes)
}
connectFails, ran = true, nil
if _, _, err := Apply(context.Background(), archHost(t), d, store.State{}, store.OriginCarried, run, nil, nil); err == nil ||
!strings.Contains(err.Error(), "could not join the kept network predecessor_default") {
t.Fatalf("a network that cannot be joined was passed over: %v", err)
}
}
var specOfLast string
// A module the mesh left out of a declaration is not a module the mesh removed (novox/hq ADR 0163,
// rule 6): what the host wrote for it stays, recorded and said; what it holds for it stays held.
// A module simply absent is removed as it always was.
func TestALeftOutModuleIsNeitherRemovedNorForgotten(t *testing.T) {
var removed []string
gone := map[string]bool{}
run := func(_ context.Context, name string, args ...string) (string, error) {
if name != "docker" {
return "", nil
}
switch args[0] {
case "info":
return "29.0.0\n", nil
case "rm":
removed = append(removed, args[len(args)-1])
gone[args[len(args)-1]] = true
case "container":
if gone[args[len(args)-1]] {
return "", errors.New("no such container")
}
return "true\tspec", nil
}
return "", nil
}
known := store.State{
Resources: []store.Applied{
{ID: "web.server", Type: "container", Target: "web", Origin: store.OriginDeclared},
{ID: "old.server", Type: "container", Target: "old", Origin: store.OriginDeclared},
},
Held: []store.Held{{ID: "web.page", Module: "web", Kind: "file", Target: "/srv/web/index.html"}},
}
d := parse(t, `{"declaration":1,"left_out":["web"],"resources":[
{"id":"notes.conf","type":"file","path":"`+t.TempDir()+`/notes.conf","content":"x"}
]}`)
report, state, err := Apply(context.Background(), archHost(t), d, known, store.OriginDeclared, run, nil, nil)
if err != nil {
t.Fatal(err)
}
if len(removed) != 1 || removed[0] != "old" {
t.Fatalf("removed %v; only the module that is absent goes", removed)
}
if _, kept := state.At("container", "web"); !kept {
t.Fatal("the left-out module's record was forgotten")
}
if _, held := state.HeldAt("web.page"); !held {
t.Fatal("the left-out module's hold was released")
}
said := false
for _, o := range report.Outcomes {
if o.ID == "web.server" && o.Action == "unchanged" && strings.Contains(o.Detail, "web was left out of this declaration by the mesh") {
said = true
}
if o.ID == "web.server" && o.Action != "unchanged" {
t.Errorf("the left-out module's container was %s", o.Action)
}
}
if !said {
t.Fatalf("keeping the left-out module's container was not said: %+v", report.Outcomes)
}
}
+54
View File
@@ -0,0 +1,54 @@
package apply
import (
"context"
"sort"
"strings"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/store"
)
// Strays is what runs on the machine that the mesh neither wrote nor holds (novox/hq ADR 0163):
// every container the runtime has that no record names and no hold names. The question nothing
// answered on 2026-09-23, when a renamed resource left its old container running for a day; asked
// on every apply now, and reported, so a thing left behind is seen the day it is left.
//
// Containers only, today. A listener nobody declared is harder to attribute to a thing, and the
// machine's own services are not strays; that account is issue 160's.
func Strays(ctx context.Context, run Runner, known store.State) ([]store.Stray, error) {
cri, err := containerRuntime(ctx, run)
if err != nil {
return nil, nil // a machine with no runtime has no containers to stray
}
out, err := run(ctx, cri, "ps", "-a", "--format", "{{.Names}}\t{{.Image}}\t{{.State}}")
if err != nil {
return nil, err
}
ours := map[string]bool{}
for _, r := range known.Resources {
if declaration.Type(r.Type) == declaration.TypeContainer {
ours[r.Target] = true
}
}
for _, h := range known.Held {
if h.Kind == string(declaration.TypeContainer) {
ours[h.Target] = true
}
}
var strays []store.Stray
for _, line := range strings.Split(strings.TrimSpace(out), "\n") {
parts := strings.Split(line, "\t")
name := strings.TrimSpace(parts[0])
if name == "" || ours[name] {
continue
}
detail := ""
if len(parts) > 2 {
detail = strings.TrimSpace(parts[1]) + ", " + strings.TrimSpace(parts[2])
}
strays = append(strays, store.Stray{Kind: string(declaration.TypeContainer), Name: name, Detail: detail})
}
sort.Slice(strays, func(i, j int) bool { return strays[i].Name < strays[j].Name })
return strays, nil
}
+79
View File
@@ -0,0 +1,79 @@
package bootstrap
import (
"context"
"encoding/json"
"os"
"path/filepath"
"strings"
"testing"
"time"
)
// What genesis raises, it raises as the module that succeeds it declares — name, data directory
// and image — so the module adopts it by the found rule that already exists (novox/hq ADR 0163,
// rule 7; issue 090). The network is the one difference left: the bootstrap forge runs on the
// machine's network to reach the store on its loopback, and a take says so.
func TestGenesisRaisesTheForgeAsTheModuleDeclaresIt(t *testing.T) {
var ran [][]string
run := func(_ context.Context, name string, args ...string) (string, error) {
if name == "docker" && args[0] == "container" {
return "", nil // not raised yet
}
ran = append(ran, append([]string{name}, args...))
return "", nil
}
if err := raiseGiteaServer(context.Background(), run, time.Second, "pw", DefaultPorts(), quietly); err != nil {
t.Fatal(err)
}
var raised []string
for _, r := range ran {
if r[0] == "docker" && r[1] == "run" {
raised = r
}
}
line := strings.Join(raised, " ")
for _, want := range []string{"--name gitea ", "--volume " + giteaDataDir + ":/data", " " + giteaImage} {
if !strings.Contains(line+" ", want) {
t.Errorf("the forge is not raised with %q: %s", want, line)
}
}
if giteaBootstrap != ForgeModule {
t.Errorf("the bootstrap forge is %q and the module names its container %q", giteaBootstrap, ForgeModule)
}
// Against the module's own manifest, where the catalogue is checked out beside this repository.
var manifest []byte
for _, candidate := range []string{"../../../mesh-catalog/modules/gitea/module.json", "../../../../../mesh-catalog/modules/gitea/module.json"} {
if raw, err := os.ReadFile(filepath.Clean(candidate)); err == nil {
manifest = raw
break
}
}
if manifest == nil {
t.Skip("the catalogue is not beside this checkout; the module's pin is not compared")
}
var m struct {
Resources []struct {
ID, Type, Name, Image string
Volumes []string
} `json:"resources"`
}
if err := json.Unmarshal(manifest, &m); err != nil {
t.Fatal(err)
}
for _, r := range m.Resources {
if r.Type != "container" || r.ID != "server" {
continue
}
if r.Name != giteaBootstrap {
t.Errorf("the module names its container %q; genesis raises %q", r.Name, giteaBootstrap)
}
if r.Image != giteaImage {
t.Errorf("the module pins %s; genesis raises %s — the two must move together", r.Image, giteaImage)
}
if len(r.Volumes) != 1 || !strings.HasSuffix(r.Volumes[0], ":/data") {
t.Errorf("the module mounts %v; genesis mounts %s:/data", r.Volumes, giteaDataDir)
}
}
}
+1 -1
View File
@@ -91,7 +91,7 @@ func TestARerunOfGenesisIsNotAMachineInUse(t *testing.T) {
if err := store.Save(o.State, store.State{Resources: []store.Applied{{ID: "store", Type: "container", Target: "mesh-store"}}}); err != nil { if err := store.Save(o.State, store.State{Resources: []store.Applied{{ID: "store", Type: "container", Target: "mesh-store"}}}); err != nil {
t.Fatal(err) t.Fatal(err)
} }
m := inUseRunner{ps: "mesh-gitea-server\t\n", ss: servingSockets} m := inUseRunner{ps: giteaBootstrap + "\t\n", ss: servingSockets}
if err := RefuseAMachineInUse(context.Background(), o, m.run, quietly); err != nil { if err := RefuseAMachineInUse(context.Background(), o, m.run, quietly); err != nil {
t.Errorf("what an earlier genesis raised was counted as a machine in use: %v", err) t.Errorf("what an earlier genesis raised was counted as a machine in use: %v", err)
} }
+23 -6
View File
@@ -25,11 +25,24 @@ const (
// foundationStore is the foundation's postgres container — the mesh's own memory, raised from the // foundationStore is the foundation's postgres container — the mesh's own memory, raised from the
// bundle. gitea's bootstrap database lives here too, so a mesh runs one postgres (issue 051). // bundle. gitea's bootstrap database lives here too, so a mesh runs one postgres (issue 051).
foundationStore = "mesh-store" foundationStore = "mesh-store"
// giteaBootstrap is the gitea server raised directly at genesis, before gitea is a module. // giteaBootstrap is the gitea server raised directly at genesis, before gitea is a module —
giteaBootstrap = "mesh-gitea-server" // under the name the gitea MODULE declares for its container, so the module finds it and holds
// giteaImage is the same upstream image the gitea module runs, pinned identically so the module // it rather than raising a second forge beside it (novox/hq ADR 0163, rule 7; issue 090).
// adopts the running server rather than replacing it. giteaBootstrap = "gitea"
giteaImage = "gitea/gitea@sha256:dfc61e347c8b582df918f4556401bf2cecdfbdb56c5282ae9488dd76fca3e41c" // giteaImage is the image the gitea module declares for that container, pinned to the same
// digest, so taking the module over is not a downgrade and not an upgrade. **Moves with the
// module's pin**: the two are compared by a take, and a difference is said there — but a
// genesis that raised an older image than the module declares would be taken over as an
// upgrade on first push, which a forge holding the mesh's packages must not have done to it
// unannounced. Checked in TestGenesisRaisesTheForgeAsTheModuleDeclaresIt against the module's
// manifest where the catalogue is beside this checkout.
giteaImage = "gitea/gitea@sha256:87a67ee09d3ae0d1df5fda5dcda3e2a1f9236a45b0a59025d6e00e46adc43bef"
// giteaDataDir is where the module's `data` directory resolves on a machine with the default
// layout (<data root>/<module>/<id>, novox/hq ADR 0112): mounted at /data as the module mounts
// it, so the repositories, attachments and indexes the bootstrap forge accumulates are the
// module's the day it is taken — before this, the forge had no volume and its data was the
// container's, lost with it.
giteaDataDir = "/var/lib/gitea/data"
// packagesOrg is the npm owner: every module consumes `@novox/*` from this gitea org. // packagesOrg is the npm owner: every module consumes `@novox/*` from this gitea org.
packagesOrg = "novox" packagesOrg = "novox"
// packagesTeam is the org team whose members may read and write the org's packages. // packagesTeam is the org team whose members may read and write the org's packages.
@@ -262,9 +275,13 @@ func raiseGiteaServer(ctx context.Context, run Runner, timeout time.Duration, db
"run", "-d", "--name", giteaBootstrap, "run", "-d", "--name", giteaBootstrap,
// Host network, like the control plane: it reaches the foundation store on the machine's // Host network, like the control plane: it reaches the foundation store on the machine's
// loopback (where the store publishes 5432) and answers on the machine's own 3000, which is // loopback (where the store publishes 5432) and answers on the machine's own 3000, which is
// where mesh-bootstrap and the builder's build containers look for it. // where mesh-bootstrap and the builder's build containers look for it. The module runs
// bridged and publishes its ports; that is the one difference a take still has to say
// (ADR 0163, rule 7) — the data, the name and the image are the module's already.
"--network", "host", "--network", "host",
"--restart", "unless-stopped", "--restart", "unless-stopped",
// The module's data directory, so what the forge accumulates is the module's when taken.
"--volume", giteaDataDir + ":/data",
}, env...) }, env...)
args = append(args, giteaImage) args = append(args, giteaImage)
+70 -1
View File
@@ -940,6 +940,13 @@ type Container struct {
// its siblings can name before any of them can resolve anything. // its siblings can name before any of them can resolve anything.
Dns []string `json:"dns,omitempty"` Dns []string `json:"dns,omitempty"`
// Networks are networks this container also joins once created, by name — a found network a
// per-machine setting keeps for a taken container (novox/hq ADR 0163, rule 4), so a
// neighbour that resolves it there keeps resolving it until the neighbour is taken too.
// Joined after creation, because a runtime starts a container on one network; part of the
// container's spec, so a network kept or let go recreates it.
Networks []string `json:"networks,omitempty"`
// IP is this container's address on its network, passed to the runtime unchanged. // IP is this container's address on its network, passed to the runtime unchanged.
// //
// Only meaningful on a user-defined network, and refused by the runtime elsewhere. Exists for // Only meaningful on a user-defined network, and refused by the runtime elsewhere. Exists for
@@ -1032,6 +1039,16 @@ func (c *Container) validate(where string, _ bool) []string {
"static address anywhere but a user-defined one") "static address anywhere but a user-defined one")
} }
} }
for _, n := range c.Networks {
problems = append(problems, (&Network{Name: n}).validate(where+": networks", false)...)
if n == c.Network {
problems = append(problems, where+": networks names "+n+", which is already the container's network")
}
}
if len(c.Networks) > 0 && (c.RunOnce || c.Schedule != "") {
problems = append(problems, where+": networks is for a container that keeps running; a step "+
"runs and exits, and joins nothing afterwards")
}
return append(problems, checkImage(where, c.Image)...) return append(problems, checkImage(where, c.Image)...)
} }
@@ -1137,6 +1154,41 @@ type Declaration struct {
// converged node — which is every node the mesh raised before adoption existed, and so the // converged node — which is every node the mesh raised before adoption existed, and so the
// only form an older controller ever sends (novox/hq ADR 0100). // only form an older controller ever sends (novox/hq ADR 0100).
Adoption *Adoption Adoption *Adoption
// Sequence orders this declaration against every other the mesh has sent this node: each
// send is one higher than the last, assigned under the control plane's hold on the node
// (novox/hq 04-ISSUES/107). Zero is a declaration that carries no order — every one an older
// controller sent, and the bundle genesis applies — and a host makes no ordering claim about
// one of those.
//
// **The one property a declaration needs that its signature does not give it.** A signature
// says the mesh sent this; it cannot say the mesh sent it AFTER the one the host is holding.
// Before this, "older" was inferred from arrival within a batch and a 750ms window, and a
// backlog longer than the batch, or a slow broker, applied a declaration the mesh had already
// superseded.
Sequence int64
// LeftOut names the modules of this machine's set the mesh left out of this declaration,
// because a setting stored for one cannot compose with its definition (novox/hq ADR 0163,
// rule 6). A machine is told everything or nothing about what it IS told; this is what it is
// not told, said. The host keeps what it holds for a left-out module and touches none of
// what it wrote for it — its resources are absent from the declaration, and absence would
// otherwise read as removal.
LeftOut []string
}
// LeftOutModuleOf says which left-out module a recorded resource belongs to, if any: its id is the
// module's name, a dot, and the module's own id for it. A module's name may contain a dot, so the
// longest left-out name that prefixes the id wins; a false match keeps a thing an apply would
// otherwise remove, which is the conservative mistake.
func (d *Declaration) LeftOutModuleOf(id string) (string, bool) {
best := ""
for _, m := range d.LeftOut {
if strings.HasPrefix(id, m+".") && len(m) > len(best) {
best = m
}
}
return best, best != ""
} }
// Adoption is a node's mode, as the controller records it: the node is adopted, and these are // Adoption is a node's mode, as the controller records it: the node is adopted, and these are
@@ -1274,6 +1326,11 @@ type envelope struct {
// a bug quietly strip a machine. // a bug quietly strip a machine.
OwnsNothing bool `json:"owns_nothing,omitempty"` OwnsNothing bool `json:"owns_nothing,omitempty"`
Resources []json.RawMessage `json:"resources"` Resources []json.RawMessage `json:"resources"`
// Sequence is optional on the wire, so a controller that does not send one is still
// understood: absent reads as zero, which is "no ordering claimed" rather than "first".
Sequence int64 `json:"sequence,omitempty"`
// LeftOut is optional on the wire too, and absent when nothing was left out (ADR 0163).
LeftOut []string `json:"left_out,omitempty"`
} }
func parse(raw []byte, allowActions bool) (*Declaration, error) { func parse(raw []byte, allowActions bool) (*Declaration, error) {
@@ -1290,8 +1347,20 @@ func parse(raw []byte, allowActions bool) (*Declaration, error) {
env.Version, Version)}} env.Version, Version)}}
} }
d := &Declaration{Version: env.Version, For: env.For, Adoption: env.Adoption} d := &Declaration{Version: env.Version, For: env.For, Adoption: env.Adoption, Sequence: env.Sequence,
LeftOut: env.LeftOut}
var problems []string var problems []string
if len(env.LeftOut) > 0 && allowActions {
// The bundle is carried with the binary and leaves nothing out: which module a setting
// stopped composing for is the mesh's record (ADR 0163).
problems = append(problems, "a carried bundle says modules were left out, and only the "+
"mesh can say that")
}
for _, m := range env.LeftOut {
if strings.TrimSpace(m) == "" {
problems = append(problems, "left_out names a module with no name")
}
}
if len(env.Resources) == 0 && !env.OwnsNothing { if len(env.Resources) == 0 && !env.OwnsNothing {
problems = append(problems, "no resources. An empty declaration is a mistake, not a "+ problems = append(problems, "no resources. An empty declaration is a mistake, not a "+
+40
View File
@@ -464,3 +464,43 @@ func TestAnExplicitlyEmptyDeclarationIsAccepted(t *testing.T) {
t.Fatalf("an unmarked empty declaration must still be refused; got %v", err) t.Fatalf("an unmarked empty declaration must still be refused; got %v", err)
} }
} }
// A container's kept networks are names, not its own network, and not for a step (novox/hq ADR
// 0163, rule 4); and the mesh may say which modules it left out, which a carried bundle may not.
func TestKeptNetworksAndLeftOutModulesAreReadStrictly(t *testing.T) {
pinnedImage := "postgres@sha256:" + strings.Repeat("a", 64)
d, err := Parse([]byte(`{"declaration":1,"left_out":["web"],"resources":[
{"id":"app","type":"container","name":"app","image":"` + pinnedImage + `","networks":["predecessor_default"]}
]}`))
if err != nil {
t.Fatal(err)
}
if got := d.Resources[0].(*Container).Networks; len(got) != 1 || got[0] != "predecessor_default" {
t.Fatalf("the kept network was not read: %v", got)
}
if m, left := d.LeftOutModuleOf("web.server"); !left || m != "web" {
t.Fatalf("web.server is not web's: %q %v", m, left)
}
if _, left := d.LeftOutModuleOf("webapp.server"); left {
t.Fatal("webapp.server was taken for web's")
}
for name, raw := range map[string]string{
"a bad network name": `{"declaration":1,"resources":[
{"id":"app","type":"container","name":"app","image":"` + pinnedImage + `","networks":["a/b"]}]}`,
"its own network": `{"declaration":1,"resources":[
{"id":"app","type":"container","name":"app","image":"` + pinnedImage + `","network":"own","networks":["own"]}]}`,
"a step": `{"declaration":1,"resources":[
{"id":"app","type":"container","name":"app","image":"` + pinnedImage + `","run-once":true,"networks":["x"]}]}`,
"a nameless module": `{"declaration":1,"left_out":[""],"resources":[
{"id":"app","type":"container","name":"app","image":"` + pinnedImage + `"}]}`,
} {
if _, err := Parse([]byte(raw)); err == nil {
t.Errorf("%s was accepted", name)
}
}
if _, err := ParseTrusted([]byte(`{"declaration":1,"left_out":["web"],"resources":[
{"id":"app","type":"container","name":"app","image":"` + pinnedImage + `"}]}`)); err == nil ||
!strings.Contains(err.Error(), "only the mesh can say that") {
t.Fatalf("a carried bundle leaving modules out was accepted: %v", err)
}
}
+21 -1
View File
@@ -3,6 +3,7 @@ package link
import ( import (
"context" "context"
"crypto/rand" "crypto/rand"
"crypto/sha256"
"encoding/hex" "encoding/hex"
"errors" "errors"
"fmt" "fmt"
@@ -63,8 +64,15 @@ func presentNats(_ context.Context, to Approach, node, secret string,
// subscribe its own inbox and nothing else (design 25 §6). The secret is its password, the same // subscribe its own inbox and nothing else (design 25 §6). The secret is its password, the same
// string the request claims, so the server proves somebody holds the token and the request // string the request claims, so the server proves somebody holds the token and the request
// proves the same thing to the controller without it having to ask the server who connected. // proves the same thing to the controller without it having to ask the server who connected.
// **Its own inbox space, because that is the only one it may listen in** (novox/hq
// 04-ISSUES/146). A JetStream publish waits for the stream's acknowledgement on an inbox the
// client picks, and the client's default is `_INBOX.<random>` — which this user may not
// subscribe to, so the enrolment failed with a permissions violation on a subject nobody had
// chosen. The permission is `_INBOX.enrol.<node>.>` (design 25 §6), so the client is told to
// pick its inboxes there; the reply address below is in the same space for the same reason.
conn, err := nats.Connect(natsURL(to.Address), conn, err := nats.Connect(natsURL(to.Address),
nats.Secure(config), nats.Secure(config),
nats.CustomInboxPrefix("_INBOX.enrol."+node),
nats.UserInfo("enrol."+node, secret), nats.UserInfo("enrol."+node, secret),
nats.Name("mesh-host/enrol/"+node), nats.Name("mesh-host/enrol/"+node),
nats.Timeout(timeout), nats.Timeout(timeout),
@@ -124,7 +132,19 @@ func (a *natsAsking) Ask(ctx context.Context, request []byte, wait time.Duration
defer cancel() defer cancel()
// Into the stream and awaited: an enrolment the bus never accepted must fail here rather than be // Into the stream and awaited: an enrolment the bus never accepted must fail here rather than be
// assumed, because the node has nothing else to go on. // assumed, because the node has nothing else to go on.
if _, err := a.js.Publish(EnrolSubject, addressed, nats.Context(publish)); err != nil { //
// **Once, however many times it is sent** (novox/hq 04-ISSUES/146). The client re-publishes when
// an acknowledgement is slow, and the mesh enrolled the machine on each copy — minting a second
// credential, which replaced the first, which is the one the node had already been given. The
// machine then reconnected for ever as a user whose password the mesh had rotated out from under
// it, and the controller's log said "enrolled anchor" twice in the same second.
//
// The id is the message: the same bytes carry the same id, so the stream discards the client's
// own retry, and a genuine second attempt — which carries a new reply address — is a different
// message and is let through.
sum := sha256.Sum256(addressed)
if _, err := a.js.Publish(EnrolSubject, addressed,
nats.MsgId(hex.EncodeToString(sum[:])), nats.Context(publish)); err != nil {
return nil, fmt.Errorf("cannot ask the mesh to enrol this node: %w", err) return nil, fmt.Errorf("cannot ask the mesh to enrol this node: %w", err)
} }
+46
View File
@@ -0,0 +1,46 @@
package link
import (
"strings"
"testing"
)
// The count that did not add up was the only symptom sixteen held resources had, and reading it meant
// opening the node's state file by hand (novox/hq 04-ISSUES/125). The line that says what an apply did
// says what it did not, too.
func TestTheApplyLineSaysWhatItHeldAndForWhichModule(t *testing.T) {
got := heldNote([]Held{
{ID: "ca", Module: "route-proxy", Kind: "directory"},
{ID: "certs", Module: "route-proxy", Kind: "directory"},
{ID: "server", Module: "route-proxy", Kind: "container"},
{ID: "mail", Module: "mailu", Kind: "container"},
})
if !strings.Contains(got, "4 held") {
t.Fatalf("the count of what was held is not in the line: %q", got)
}
// The module is the thing an operator can act on: `take` takes a module.
if !strings.Contains(got, "route-proxy: 3") || !strings.Contains(got, "mailu: 1") {
t.Fatalf("the line does not break the holds down by module: %q", got)
}
// Ordered, so two machines holding the same things read the same and a diff of two reports is
// about what changed.
if strings.Index(got, "mailu") > strings.Index(got, "route-proxy") {
t.Fatalf("modules are not in a stated order: %q", got)
}
// It says why, because "held" alone reads as a failure and this is correct behaviour.
if !strings.Contains(got, "taken") {
t.Fatalf("the line does not say a hold ends when the module is taken: %q", got)
}
}
func TestAnApplyThatHeldNothingSaysNothingExtra(t *testing.T) {
// A converged machine holds nothing, which is most applies. Reporting "0 held" on every one of
// them is how a line stops being read.
if got := heldNote(nil); got != "" {
t.Fatalf("an apply with no holds added %q to its line", got)
}
if got := heldNote([]Held{}); got != "" {
t.Fatalf("an apply with no holds added %q to its line", got)
}
}
+19
View File
@@ -110,6 +110,15 @@ type Report struct {
// and the mesh's up in its place, and where the found configuration's original was kept. // and the mesh's up in its place, and where the found configuration's original was kept.
Tunnel *CarriedTunnel `json:"tunnel,omitempty"` Tunnel *CarriedTunnel `json:"tunnel,omitempty"`
// Strays is what runs on the machine that the mesh neither wrote nor holds (novox/hq ADR
// 0163): containers nobody declared and nobody holds, the ones a cutover leaves behind.
Strays []Stray `json:"strays,omitempty"`
// Profile is what this machine can do, detected again by the apply that reports (novox/hq
// ADR 0161) — the same shape enrolment sends — so a capability gained or lost since enrolment,
// a network manager switched, reaches the mesh at the next push rather than never.
Profile map[string]any `json:"profile,omitempty"`
// Host is the version of the host that produced this report (novox/hq ADR 0141). // Host is the version of the host that produced this report (novox/hq ADR 0141).
// //
// Without it nothing can say a machine is behind, so "every machine current with its source" // Without it nothing can say a machine is behind, so "every machine current with its source"
@@ -200,6 +209,16 @@ type Held struct {
Changed string `json:"changed,omitempty"` Changed string `json:"changed,omitempty"`
// Kept is where a file's original was kept. // Kept is where a file's original was kept.
Kept string `json:"kept,omitempty"` Kept string `json:"kept,omitempty"`
// Facts is the found thing beside what the module declares — what a take compares (novox/hq
// ADR 0163). The same shape the host keeps; the controller reads it as data.
Facts map[string]any `json:"facts,omitempty"`
}
// A Stray is a container the mesh neither wrote nor holds (ADR 0163).
type Stray struct {
Kind string `json:"kind"`
Name string `json:"name"`
Detail string `json:"detail,omitempty"`
} }
// Reach is one thing reachable on the machine: a listening socket, or a published container port. // Reach is one thing reachable on the machine: a listening socket, or a published container port.
+50
View File
@@ -0,0 +1,50 @@
package link
import (
"encoding/json"
"testing"
"time"
)
// The drain picked the last to arrive. A backlog longer than the batch, or a broker that split a
// burst, delivered a superseded declaration last (novox/hq 04-ISSUES/107).
func sequenced(t *testing.T, n int64) *said {
t.Helper()
inner, err := json.Marshal(map[string]any{"declaration": 1, "resources": []any{}, "sequence": n})
if err != nil {
t.Fatal(err)
}
body, err := json.Marshal(Signed{Declaration: inner, Signature: []byte("s")})
if err != nil {
t.Fatal(err)
}
return &said{body: body}
}
func TestTheDrainKeepsTheHighestSequenceNotTheLastToArrive(t *testing.T) {
waiting := make(chan Declaration, 8)
waiting <- sequenced(t, 9)
waiting <- sequenced(t, 4) // arrived last, composed earlier
latest, aside := newest(waiting, sequenced(t, 8), 30*time.Millisecond)
if got := sequenceOf(latest.Body()); got != 9 {
t.Fatalf("the drain kept sequence %d, and 9 was waiting", got)
}
if len(aside) != 2 {
t.Fatalf("%d set aside, wanted 2 (the 8 and the late 4)", len(aside))
}
}
func TestWithoutSequencesTheLastToArriveStillWins(t *testing.T) {
// The behaviour this had before, kept for a controller that sends no order.
apply, aside := newest(arriving("two", "three"), &said{body: []byte("one")}, 30*time.Millisecond)
if string(apply.Body()) != "three" || len(aside) != 2 {
t.Fatalf("applied %q with %d set aside", apply.Body(), len(aside))
}
}
func TestAnUnreadableBodyClaimsNoOrder(t *testing.T) {
if got := sequenceOf([]byte("not json")); got != 0 {
t.Fatalf("garbage claimed sequence %d", got)
}
}
+14 -3
View File
@@ -73,8 +73,19 @@ func PinnedConfig(pin string) (*tls.Config, error) {
}, nil }, nil
} }
// Dial opens a TLS connection to the broker, refusing anything but the pinned certificate. // dialPinned completes a TLS handshake against an address, refusing anything but the pinned
func Dial(address, pin string, timeout time.Duration) (*tls.Conn, error) { // certificate.
//
// **Not how the bus is reached, and it used to be** (novox/hq 04-ISSUES/146). Enrolment opened one
// of these before it said anything, which was right while the broker answered TLS immediately and
// wrong the moment the mesh moved to a bus that speaks its own protocol first. The pin itself was
// never the problem — PinnedConfig is what the NATS client is given, and the verification runs
// inside the handshake that client performs.
//
// It stays here because this is where the pin is proven: the tests beside it run a real TLS server
// and assert that a wrong certificate is refused before a byte of application data is sent. What it
// must not become again is something a caller uses to reach the bus.
func dialPinned(address, pin string, timeout time.Duration) (*tls.Conn, error) {
config, err := PinnedConfig(pin) config, err := PinnedConfig(pin)
if err != nil { if err != nil {
return nil, err return nil, err
@@ -86,7 +97,7 @@ func Dial(address, pin string, timeout time.Duration) (*tls.Conn, error) {
if errors.Is(err, ErrWrongCertificate) { if errors.Is(err, ErrWrongCertificate) {
return nil, err return nil, err
} }
return nil, fmt.Errorf("cannot reach the broker at %s: %w", address, err) return nil, fmt.Errorf("cannot reach %s: %w", address, err)
} }
return conn, nil return conn, nil
} }
+4 -4
View File
@@ -63,7 +63,7 @@ func server(t *testing.T) (address string, fingerprint string) {
func TestTheRightBrokerIsAccepted(t *testing.T) { func TestTheRightBrokerIsAccepted(t *testing.T) {
address, pin := server(t) address, pin := server(t)
conn, err := Dial(address, pin, 5*time.Second) conn, err := dialPinned(address, pin, 5*time.Second)
if err != nil { if err != nil {
t.Fatalf("the broker its token describes was refused: %v", err) t.Fatalf("the broker its token describes was refused: %v", err)
} }
@@ -76,7 +76,7 @@ func TestADifferentBrokerIsRefused(t *testing.T) {
address, _ := server(t) address, _ := server(t)
_, other := server(t) _, other := server(t)
_, err := Dial(address, other, 5*time.Second) _, err := dialPinned(address, other, 5*time.Second)
if err == nil { if err == nil {
t.Fatal("a broker presenting a different certificate was accepted") t.Fatal("a broker presenting a different certificate was accepted")
} }
@@ -130,7 +130,7 @@ func TestNothingIsSentToTheWrongBroker(t *testing.T) {
// A pin for a certificate this server does not have. // A pin for a certificate this server does not have.
_, elsewhere := server(t) _, elsewhere := server(t)
if _, err := Dial(listener.Addr().String(), elsewhere, 5*time.Second); err == nil { if _, err := dialPinned(listener.Addr().String(), elsewhere, 5*time.Second); err == nil {
t.Fatal("the impostor was accepted") t.Fatal("the impostor was accepted")
} }
if n := <-received; n > 0 { if n := <-received; n > 0 {
@@ -160,7 +160,7 @@ func TestAnUnreachableBrokerIsAnOrdinaryFailure(t *testing.T) {
address := listener.Addr().String() address := listener.Addr().String()
listener.Close() listener.Close()
_, err = Dial(address, pin, 2*time.Second) _, err = dialPinned(address, pin, 2*time.Second)
if err == nil { if err == nil {
t.Fatal("dialling a closed port succeeded") t.Fatal("dialling a closed port succeeded")
} }
+67 -2
View File
@@ -6,6 +6,8 @@ import (
"encoding/json" "encoding/json"
"errors" "errors"
"fmt" "fmt"
"sort"
"strings"
"time" "time"
) )
@@ -250,9 +252,11 @@ func Run(ctx context.Context, m Membership, apply Applier, say Announce, timeout
case report.Refused != "": case report.Refused != "":
say("refused a declaration: " + report.Refused) say("refused a declaration: " + report.Refused)
case len(report.Failed) > 0: case len(report.Failed) > 0:
say(fmt.Sprintf("applied %d and failed: %v", len(report.Applied), report.Failed)) say(fmt.Sprintf("applied %d and failed: %v%s",
len(report.Applied), report.Failed, heldNote(report.Held)))
default: default:
say(fmt.Sprintf("applied %d resource(s)", len(report.Applied))) say(fmt.Sprintf("applied %d resource(s)%s",
len(report.Applied), heldNote(report.Held)))
} }
publishReport(ctx, link, m, report, say, timeout) publishReport(ctx, link, m, report, say, timeout)
// Settled after the report is published. A node that dies between applying and // Settled after the report is published. A node that dies between applying and
@@ -296,6 +300,16 @@ func newest(arriving <-chan Declaration, first Declaration, window time.Duration
if !ok { if !ok {
return latest, superseded return latest, superseded
} }
// **By sequence when both carry one, by arrival when either does not** (novox/hq
// 04-ISSUES/107). Arrival is what this window had to go on, and it is wrong exactly
// when it matters — a backlog drained out of order. A declaration that says where it
// stands is believed over when it turned up; one that does not is the older
// controller's, and arrival is all there is.
if sequenceOf(next.Body()) < sequenceOf(latest.Body()) &&
sequenceOf(next.Body()) > 0 && sequenceOf(latest.Body()) > 0 {
superseded = append(superseded, next)
continue
}
superseded = append(superseded, latest) superseded = append(superseded, latest)
latest = next latest = next
case <-time.After(window): case <-time.After(window):
@@ -304,6 +318,24 @@ func newest(arriving <-chan Declaration, first Declaration, window time.Duration
} }
} }
// sequenceOf is the order a signed declaration claims, or zero when it claims none or cannot be
// read. Read from the envelope alone; the signature is verified later, when the winner is applied,
// and a forged message that lied about its sequence would only set aside real ones — which are
// reported as set aside, and the next push sends the current one again.
func sequenceOf(body []byte) int64 {
var signed Signed
if err := json.Unmarshal(body, &signed); err != nil {
return 0
}
var d struct {
Sequence int64 `json:"sequence"`
}
if err := json.Unmarshal(signed.Declaration, &d); err != nil {
return 0
}
return d.Sequence
}
// declaredIn is the id a signed declaration carries, for a report about one that was not applied. // declaredIn is the id a signed declaration carries, for a report about one that was not applied.
// Empty if the message is not one — a forged or garbled message is refused by handleBody when its // Empty if the message is not one — a forged or garbled message is refused by handleBody when its
// turn comes; here it is only named. // turn comes; here it is only named.
@@ -392,3 +424,36 @@ func publishAlive(ctx context.Context, bus Bus, m Membership, say Announce,
say("could not tell the mesh this node is here: " + err.Error()) say("could not tell the mesh this node is here: " + err.Error())
} }
} }
// heldNote is what this apply did NOT do, for the line that says what it did.
//
// **A count that does not add up is the only symptom a held resource had** (novox/hq 04-ISSUES/125).
// An adopted node keeps what it found until its module is taken (ADR 0100), and that is correct — but
// it was recorded only in the node's own state file. On the edge cut-over the mesh sent 346 resources,
// the journal said it applied 330, and nothing anywhere said which sixteen or why. Reading it took
// opening state.json by hand; not reading it took every public name on the machine down, because the
// operator had four green surfaces and a discrepancy nobody could interpret.
//
// So the line that reports the apply carries it. Grouped by module and ordered by name, because the
// sentence an operator needs is "route-proxy is assigned and not taken", and the module is the thing
// they can act on — `take` is the verb, and it takes a module.
func heldNote(held []Held) string {
if len(held) == 0 {
return ""
}
byModule := map[string]int{}
for _, h := range held {
byModule[h.Module]++
}
names := make([]string, 0, len(byModule))
for name := range byModule {
names = append(names, name)
}
sort.Strings(names)
parts := make([]string, 0, len(names))
for _, name := range names {
parts = append(parts, fmt.Sprintf("%s: %d", name, byModule[name]))
}
return fmt.Sprintf(", %d held until their module is taken (%s)",
len(held), strings.Join(parts, ", "))
}
+23
View File
@@ -20,6 +20,14 @@ const (
// state: whether one IS running. Assignment needs the first. // state: whether one IS running. Assignment needs the first.
CapSeat = "seat" CapSeat = "seat"
CapPrivileged = "privileged" CapPrivileged = "privileged"
// The network manager this machine runs, one capability per dialect (novox/hq ADR 0161): the
// uplink seat's holder declares its own, so the holder for a manager the machine does not run
// is refused the way any missing capability is, naming it. Active, not installed — a machine
// may have two of these on disk and runs one.
CapUplinkNetworkManager = "uplink-networkmanager"
CapUplinkSystemdNetworkd = "uplink-systemd-networkd"
CapUplinkDhcpcd = "uplink-dhcpcd"
) )
// commandCapability is the shape most detectors take: run something, and treat a working // commandCapability is the shape most detectors take: run something, and treat a working
@@ -202,6 +210,21 @@ func Default(runner Runner) []Detector {
why: "asks the kernel for interfaces — needs the module, not just the tool", why: "asks the kernel for interfaces — needs the module, not just the tool",
runner: runner, runner: runner,
}, },
commandCapability{
name: CapUplinkNetworkManager, command: "systemctl", args: []string{"is-active", "NetworkManager.service"},
why: "asks the init whether NetworkManager is running — the dialect the uplink seat's holder must speak",
runner: runner,
},
commandCapability{
name: CapUplinkSystemdNetworkd, command: "systemctl", args: []string{"is-active", "systemd-networkd.service"},
why: "asks the init whether systemd-networkd is running — the dialect the uplink seat's holder must speak",
runner: runner,
},
commandCapability{
name: CapUplinkDhcpcd, command: "systemctl", args: []string{"is-active", "dhcpcd.service"},
why: "asks the init whether dhcpcd is running — the dialect the uplink seat's holder must speak",
runner: runner,
},
} }
} }
+39
View File
@@ -0,0 +1,39 @@
package profile
import (
"context"
"errors"
"testing"
)
// The uplink seat's holder must be the dialect the machine runs (novox/hq ADR 0161): the profile
// names the network manager found active, one capability per manager, and nothing for one that is
// merely installed.
func TestTheProfileNamesTheNetworkManagerThatIsRunning(t *testing.T) {
runner := func(_ context.Context, name string, args ...string) (string, error) {
if name == "systemctl" && len(args) == 2 && args[0] == "is-active" {
if args[1] == "NetworkManager.service" {
return "active\n", nil
}
return "inactive\n", errors.New("exit status 3")
}
return "", errors.New("not here")
}
var have []Detector
for _, d := range Default(Runner(runner)) {
switch d.Name() {
case CapUplinkNetworkManager, CapUplinkSystemdNetworkd, CapUplinkDhcpcd:
have = append(have, d)
}
}
if len(have) != 3 {
t.Fatalf("expected a detector per manager, found %d", len(have))
}
for _, d := range have {
v := d.Detect(context.Background())
want := d.Name() == CapUplinkNetworkManager
if v.Present != want {
t.Errorf("%s: present=%v, want %v (%s)", d.Name(), v.Present, want, v.Detail)
}
}
}
+29
View File
@@ -0,0 +1,29 @@
package store
import "testing"
// A resource whose target moves leaves what the host wrote under the old target on record as a
// former one, undeclared by construction, so the next apply removes it (novox/hq issue 097, ADR 0163).
func TestARecordWhoseTargetMovedKeepsTheFormerTargetToRemove(t *testing.T) {
s := State{}
s.Record(Applied{ID: "gitea.server", Type: "container", Target: "mesh-gitea", Origin: OriginDeclared})
s.Record(Applied{ID: "gitea.server", Type: "container", Target: "gitea", Origin: OriginDeclared})
if len(s.Resources) != 2 {
t.Fatalf("a moved target produced %d record(s): %+v", len(s.Resources), s.Resources)
}
orphans := s.Orphans(map[string]bool{"gitea.server": true}, OriginDeclared)
if len(orphans) != 1 || orphans[0].Target != "mesh-gitea" || !IsFormer(orphans[0].ID) {
t.Fatalf("the former target is not an orphan to remove: %+v", orphans)
}
s.Forget(orphans[0].ID)
if len(s.Resources) != 1 || s.Resources[0].Target != "gitea" {
t.Fatalf("forgetting the former target touched the current one: %+v", s.Resources)
}
// The same target again is not a move; a carried record is not the host's to remove.
s.Record(Applied{ID: "gitea.server", Type: "container", Target: "gitea", Origin: OriginDeclared})
s.Record(Applied{ID: "bundle", Type: "file", Target: "/a"})
s.Record(Applied{ID: "bundle", Type: "file", Target: "/b"})
if len(s.Resources) != 2 {
t.Fatalf("an unmoved or carried record grew the list: %+v", s.Resources)
}
}
+57
View File
@@ -18,6 +18,7 @@ import (
"os" "os"
"path/filepath" "path/filepath"
"sort" "sort"
"strings"
"time" "time"
) )
@@ -274,6 +275,41 @@ type Held struct {
// reverted: that is how a predecessor still writing is caught. // reverted: that is how a predecessor still writing is caught.
Changed string `json:"changed,omitempty"` Changed string `json:"changed,omitempty"`
ChangedAt time.Time `json:"changed_at,omitempty"` ChangedAt time.Time `json:"changed_at,omitempty"`
// Facts is what a take would compare: the found thing beside what the module declares
// (novox/hq ADR 0163). Read fresh on every apply while held, so the controller's preview
// speaks of the machine as it is.
Facts *Facts `json:"facts,omitempty"`
}
// Facts is a held thing beside what its module declares — what a take compares (ADR 0163).
type Facts struct {
// A found container: the image it runs and when that image was made; the networks it is on
// and the other containers on each; what it mounts; what it publishes.
Image string `json:"image,omitempty"`
ImageCreated string `json:"image_created,omitempty"`
Networks map[string][]string `json:"networks,omitempty"`
Mounts []string `json:"mounts,omitempty"`
Ports []string `json:"ports,omitempty"`
// What the module declares for it, and the declared image's creation date when the image
// is on the machine already.
DeclaredImage string `json:"declared_image,omitempty"`
DeclaredImageCreated string `json:"declared_image_created,omitempty"`
DeclaredPorts []string `json:"declared_ports,omitempty"`
DeclaredVolumes []string `json:"declared_volumes,omitempty"`
// Downgrade is true when both creation dates are known and the declared image is the older.
Downgrade bool `json:"downgrade,omitempty"`
// A found file: whether the declared content differs from what was found, and how, as lines
// only in the found file (-) and lines only in the declared one (+), bounded.
Differs bool `json:"differs,omitempty"`
Difference []string `json:"difference,omitempty"`
}
// A Stray is something running on the machine that the mesh neither wrote nor holds
// (novox/hq ADR 0163): the answer to "what is here that nobody asked for".
type Stray struct {
Kind string `json:"kind"`
Name string `json:"name"`
Detail string `json:"detail,omitempty"`
} }
// Recorded reports whether this host has a record, of any origin, of putting something of this // Recorded reports whether this host has a record, of any origin, of putting something of this
@@ -462,6 +498,20 @@ func Save(path string, s State) error {
func (s *State) Record(a Applied) { func (s *State) Record(a Applied) {
for i, existing := range s.Resources { for i, existing := range s.Resources {
if existing.ID == a.ID { if existing.ID == a.ID {
// A resource whose target moved leaves what the host wrote under the old target
// behind — a container under the old name, a file at the old path. Rewriting the
// record would erase the only trace of it (novox/hq issue 097, ADR 0163), so the old
// target stays on record as a former one, undeclared by construction, until the next
// apply removes it the way it removes anything the host wrote and no longer declares.
// What was found is held, never recorded here, and so never removed by this.
if originOf(existing) == OriginDeclared && existing.Target != "" && a.Target != "" &&
existing.Target != a.Target && existing.Type == a.Type {
former := existing
former.ID = FormerID(existing.ID, existing.Target)
s.Resources[i] = a
s.Resources = append(s.Resources, former)
return
}
s.Resources[i] = a s.Resources[i] = a
return return
} }
@@ -469,6 +519,13 @@ func (s *State) Record(a Applied) {
s.Resources = append(s.Resources, a) s.Resources = append(s.Resources, a)
} }
// FormerID names the record of a resource's former target: the resource's id and the target it
// had, so the record is distinct from the current one and is never what a declaration names.
func FormerID(id, target string) string { return id + "@former:" + target }
// IsFormer says whether a record names a former target.
func IsFormer(id string) bool { return strings.Contains(id, "@former:") }
// Forget drops a resource from what the node owns. // Forget drops a resource from what the node owns.
func (s *State) Forget(id string) { func (s *State) Forget(id string) {
kept := s.Resources[:0] kept := s.Resources[:0]
+32
View File
File diff suppressed because one or more lines are too long
+48
View File
@@ -0,0 +1,48 @@
package packaging_test
import (
"encoding/json"
"os"
"testing"
)
// The mesh delivers the launcher, so the manifest carries a copy of it (novox/hq 04-ISSUES/142).
//
// **Two copies of one script is a drift waiting to happen**, and the only reason to accept it is that
// a file resource is written atomically — temp file, then rename — while an archive writes in place
// with truncate. The running launcher keeps the inode it was started from and the next start picks up
// the new one; unpacking an archive over it would truncate the file a running shell is reading.
//
// So: two copies, and this is the check that they are the same one.
func TestTheManifestCarriesTheLauncherExactly(t *testing.T) {
onDisk, err := os.ReadFile("nox-mesh-host-launch")
if err != nil {
t.Fatal(err)
}
raw, err := os.ReadFile("../module.json")
if err != nil {
t.Fatal(err)
}
var manifest struct {
Resources []struct {
ID string `json:"id"`
Content string `json:"content"`
} `json:"resources"`
}
if err := json.Unmarshal(raw, &manifest); err != nil {
t.Fatal(err)
}
for _, r := range manifest.Resources {
if r.ID != "launcher" {
continue
}
if r.Content != string(onDisk) {
t.Fatal("the launcher the mesh would deliver is not the launcher in this repository. " +
"Copy packaging/nox-mesh-host-launch into module.json's `launcher` resource — the " +
"machines run what the manifest says, and this file is what gets reviewed")
}
return
}
t.Fatal("module.json declares no `launcher` resource, so nothing delivers the launcher and a " +
"delivered host version is never started")
}