Compare commits

..
14 Commits
Author SHA1 Message Date
mesh-admin e030aa2387 Merge pull request 'The first user list lets the controller publish assignments and hear its seat's tools (hq #251)' (#62) from fix/first-user-list-matches-the-controller into main 2026-10-01 15:29:55 +00:00
jschoubben c670bef4e1 The first user list lets the controller publish assignments and hear its seat's tools
The controller's own composition (mesh-controller internal/broker, 2026-10-01)
publishes memberships into the assignments stream after each push and
subscribes to its seat's tool subjects; the list the installer carries did
not say so, and a controller on it is refused on the first thing it tries:
"Permissions Violation for Publish to mesh.assignment.novox.builder" (hq #251),
which is why every build asked through the console was lost. The controller's
test that compares the two (TestTheInstallersFirstUserListIsWhatTheControllerWouldCompose,
run with this checkout beside it) named exactly these two subjects, and passes.
2026-10-01 17:29:48 +02:00
mesh-admin 45529bfec2 Merge pull request 'The profile names the network manager that is running, and travels in every report (hq ADR 0161)' (#61) from feat/the-profile-names-the-uplink-and-travels-in-the-report into main 2026-10-01 14:02:16 +00:00
jschoubben b1e9ccff6d The profile names the network manager that is running, and travels in every report (hq ADR 0161)
One capability per manager — uplink-networkmanager, uplink-systemd-networkd, uplink-dhcpcd — from
systemctl is-active, so the uplink seat's holder for a manager this machine does not run is refused
the way any missing capability is, naming it (issue 138). The apply that reports detects the profile
again and sends it, the same shape enrolment sends, so a machine that switched managers reaches the
mesh at its next push.
2026-10-01 15:58:31 +02:00
mesh-admin cbcf0bcc93 Merge pull request 'A node can join the bus the mesh runs on' (#51) from fix/a-node-can-join-the-bus-the-mesh-runs-on into main 2026-10-01 11:18:02 +00:00
jschoubben 6910f07d75 Merge pull request 'Say what a container's host entries now are' (#60) from feat/148-names-are-resolved-not-copied into main 2026-09-30 12:38:15 +00:00
jschoubben 88037b33b7 Say what a container's host entries now are
novox/hq ADR 0148: the mesh's names are no longer among them, only what
the module declared. Comment only; the digest is unchanged.
2026-09-30 14:38:11 +02:00
jschoubben 422ad516d5 Merge pull request 'A declaration carries its order, and a host refuses an older one' (#59) from feat/107-a-declaration-carries-its-order into main 2026-09-30 12:03:10 +00:00
jschoubben 8431ecfb48 A declaration carries its order, and a host refuses an older one
novox/hq 04-ISSUES/107. A declaration's only identity was the digest of
its bytes: a host could say "not the last" and could not say "older". On
the link, nothing refused an older one at all, and the drain picked the
last to arrive — wrong exactly when it mattered, a backlog drained out of
order or a broker that split a burst.

A declaration may now carry a sequence, one higher per send. A host
refuses one lower than what it kept, whole, and says why. The drain keeps
the highest sequence in a batch rather than the last to arrive.

Only when both sides claim an order. Absent reads as zero — "no ordering
claimed", not "first" — so a controller that sends none is still
understood and a host that kept one before it understood them compares
nothing. That is what lets hosts go first and the controller follow, which
is the order 087 says a new field needs.
2026-09-30 14:03:03 +02:00
jschoubben f107d68b2d Merge pull request 'A delivered host knows it is the delivered one' (#58) from fix/163-a-delivered-host-knows-it-is-the-delivered-one into main 2026-09-30 11:46:58 +00:00
jschoubben 1028193c8a A delivered host knows it is the delivered one
novox/hq 04-ISSUES/163. The host asks after every apply whether a newer
host is delivered than the one running, and asked with the link-time
version stamp — which every delivered host carries as "development
build", because the version comes from where the binary sits now (0142).
So a delivered host never matched the newest delivered version, stood
aside on every push for ever, and because standing aside cancels the
report, the mesh never heard from it again.

Measured on two machines: each push produced "host <v> is delivered;
standing aside" for the version already running, then "applied, and could
not tell the mesh: reporting: context canceled". A machine restarting its
host on every push and reporting nothing, reading as healthy.

Asked with the running version now. Half of 0142 was applied to the
report and the known-good record and not here; this is the other half.
2026-09-30 13:46:51 +02:00
jschoubben f576287b51 Merge pull request 'A delivered host reads its version from where it sits' (#57) from fix/161-a-delivered-host-reads-its-version-from-its-path into main 2026-09-30 10:40:29 +00:00
jschoubben 6c6495f6d9 A delivered host reads its version from where it sits
novox/hq ADR 0142, which decided this and was not implemented: "It is
unpacked into a directory named for its version, so it can read its own
version from its path. The stamp goes, and with it the need for a build to
know what it will be called."

The mesh's toolchain stamps no version, on purpose, so a delivered host
called itself "development build" and the mesh could not tell which host
any machine ran — which is the whole of what 087 added. A delivered host
lives at <libexec>/versions/<version>/<binary>, and that directory is the
answer.

A host placed by hand keeps its stamp, which is the honest answer for one
the mesh did not deliver, and is every machine until a delivery reaches
it. A binary sitting anywhere else is not read as a version at all.

The decision is split from the reading so a test can ask about a path
without being that binary.
2026-09-30 12:40:00 +02:00
jschoubben 197258c88c A node can join the bus the mesh runs on
novox/hq 04-ISSUES/146, the layers behind the three already fixed.

A new membership says which bus it is for. Empty meant 'whatever the mesh runs
today' while two buses existed, and became a refusal the moment one did: an
enrolled node came up and reconnected for ever against its own record.

The enrolling client takes its inboxes in the space its user may listen in. A
JetStream publish waits for the stream's acknowledgement on an inbox the client
picks, and its default is one this user may not subscribe to — so the enrolment
failed with a permissions violation on a subject nobody had chosen.

And the enrolment publish carries a message id, so the client's own retry is
discarded by the stream rather than enrolling the machine twice. That one is
not finished: the duplicate survives it, and the issue says where the trail
stops.
2026-09-29 17:36:59 +02:00
12 changed files with 397 additions and 22 deletions
+98 -12
View File
@@ -51,6 +51,43 @@ var builtFor = ""
var version = "development build"
// runningVersion is this host's version: the directory it was delivered into, or the link-time stamp
// for one placed by hand.
//
// **From where it sits, not from its linker** (novox/hq ADR 0142): "It is unpacked into a directory
// named for its version, so it can read its own version from its path. The stamp goes, and with it the
// need for a build to know what it will be called."
//
// The mesh's toolchain does not stamp a version, on purpose — a build does not know what it will be
// called — so a delivered host read as "development build" and the mesh could not tell which host any
// machine ran (novox/hq 04-ISSUES/161, and 087 for why that matters). The path knows: a delivered host
// lives at `<libexec>/versions/<version>/<binary>`.
//
// A host placed by hand keeps its stamp, which is the honest answer for one the mesh did not deliver.
func runningVersion() string {
self, err := os.Executable()
if err != nil {
return version
}
return versionAt(self, version)
}
// versionAt is runningVersion's decision, with the executable's path and the link-time stamp given —
// so a test can ask it about a path without being that binary.
func versionAt(self, stamped string) string {
// .../versions/<version>/<binary> — the parent is the version, and its parent is the versions
// directory. Checked rather than assumed, so a binary somewhere else does not read a directory
// name as a version.
dir := filepath.Dir(self)
if filepath.Base(filepath.Dir(dir)) != upgrade.VersionsDirName {
return stamped
}
if name := filepath.Base(dir); name != "" && name != "." && name != string(filepath.Separator) {
return name
}
return stamped
}
const usage = `mesh-host — the node host
profile what this machine can be asked to do
@@ -254,7 +291,7 @@ func run(ctx context.Context, command string, opts options) error {
return runLink(ctx, opts)
case "version":
fmt.Println(version)
fmt.Println(runningVersion())
return nil
case "", "help", "-h", "--help":
@@ -420,10 +457,10 @@ func short(digest string) string {
// them again — and everything the mesh declared read as no longer declared and removed. Even the
// very declaration the mesh last sent, applied from a file, would plan to remove the foundation.
// `apply FILE` is for a machine the mesh has not spoken to, and is refused saying so.
func refuseStale(known store.State, kept store.Declared, keptErr error, digest string, from provenance) error {
func refuseStale(known store.State, kept store.Declared, keptErr error, digest string, from provenance, sequence int64) error {
switch from {
case fromDeclared:
return nil
return refuseOlder(kept, keptErr, sequence)
case fromBundle:
if known.Genesis == nil || known.Genesis.Digest == digest {
return nil
@@ -520,7 +557,7 @@ func runApply(ctx context.Context, opts options, d *declaration.Declaration, raw
if err := apply.CheckMode(known, d); err != nil {
return err
}
if err := refuseStale(known, kept, keptErr, digest, from); err != nil {
if err := refuseStale(known, kept, keptErr, digest, from, d.Sequence); err != nil {
return err
}
@@ -594,8 +631,8 @@ func runApply(ctx context.Context, opts options, d *declaration.Declaration, raw
//
// A failure to record is reported and does not fail the apply. The apply worked; what is
// lost is a rollback's ability to come back here, which is worse to hide than to say.
if version != "" {
if err := upgrade.RecordKnownGood(upgrade.KnownGoodPath(opts.state), version); err != nil {
if v := runningVersion(); v != "" {
if err := upgrade.RecordKnownGood(upgrade.KnownGoodPath(opts.state), v); err != nil {
fmt.Fprintf(os.Stderr,
"mesh-host: applied, but could not record %s as known-good: %v\n"+
" a rollback would have nothing to return to.\n", version, err)
@@ -781,10 +818,7 @@ func enrol(ctx context.Context, opts options) error {
// control plane cannot decide what a node should run without it, so it travels with the
// request instead of being asked for in a second round trip.
detected := profile.Detect(ctx, profile.Default(nil), opts.timeout)
reported := map[string]any{}
if raw, err := json.Marshal(detected); err == nil {
_ = json.Unmarshal(raw, &reported)
}
reported := profileAsReported(detected)
// Signed with the identity just generated, so the mesh can tell this machine from anyone else
// who knows its public key (novox/hq issue 083).
@@ -816,6 +850,13 @@ func enrol(ctx context.Context, opts options) error {
Fingerprint: firstNonEmpty(reply.Fingerprint, token.Fingerprint),
Signer: firstNonEmpty2(reply.Signer, token.Signer),
Password: reply.Password,
// **Which bus this membership is for, said rather than left empty** (novox/hq
// 04-ISSUES/146). The link refuses a membership that names another bus, and an empty name
// is not this one's — so a node enrolled without it came up and reconnected for ever
// against its own record: "this membership is for \"\", and the mesh's bus is nats". The
// reply does not carry it because there is one bus and the host knows which (ADR 0131);
// what was missing was writing that down where the link reads it.
Transport: link.OnNATS,
}
if mine.Membership.Password == "" {
// The mesh did not replace the token's secret, so it is still this node's broker
@@ -1017,7 +1058,12 @@ func runLink(ctx context.Context, opts options) error {
// standing before this machine leaves the one it is on (novox/hq design 28, task 5.2).
adoptDeliveredMembership(identity.Path(opts.state), &mine, say)
switch next, waiting, err := upgrade.Successor(upgrade.VersionsDir(""), version); {
// Asked with the version this host is RUNNING, read from where it sits — not the link-time
// stamp, which every delivered host carries as "development build". Asked with the stamp,
// a delivered host never matched the newest delivered version, so it stood aside on every
// push for ever, and standing aside cancels the report, so the mesh never heard from it
// again (novox/hq 04-ISSUES/163).
switch next, waiting, err := upgrade.Successor(upgrade.VersionsDir(""), runningVersion()); {
case err != nil:
// Said, not fatal. A host that cannot read the delivered versions is still running this
// machine correctly; what it has lost is the ability to be replaced.
@@ -1369,7 +1415,8 @@ func applyAndKeep(ctx context.Context, opts options, raw []byte, signed *store.D
sched.Sync(declared, held)
}
report := link.Report{Carried: carriedPorts(updated), Declared: digestOf(raw), Host: version}
report := link.Report{Carried: carriedPorts(updated), Declared: digestOf(raw), Host: runningVersion(),
Profile: profileAsReported(profile.Detect(ctx, profile.Default(nil), opts.timeout))}
// Which of this machine's links face outside, for the filter the mesh writes around them
// (novox/hq ADR 0140). Reported whatever the node's mode: a converged node's filter needs it,
// and an adopted one becomes converged without a further round trip. A machine that cannot read
@@ -1552,3 +1599,42 @@ func adoptDeliveredMembership(identityPath string, mine *identity.Identity, say
say(fmt.Sprintf("moving to the %s bus at %s — restarting to dial it", next.Transport, next.Broker))
os.Exit(0)
}
// refuseOlder refuses a declaration from the mesh that is older than the one this node holds.
//
// **By sequence, not by arrival** (novox/hq 04-ISSUES/107). What the host kept is the last thing
// the mesh said, signed; a declaration whose sequence is lower was composed before it, whatever
// order they arrived in — a backlog drained after the node was away, or a broker that split a burst.
// Applying it would make the machine into something the mesh had already moved past, which is the
// incident of issue 104 by another door.
//
// Only when both sides claim an order. A declaration with no sequence is one an older controller
// sent, and one kept with no sequence is one this host received before it understood them; in either
// case there is no order to compare, and refusing on a guess would strand the node the moment the
// controller is older than the host. Equal is the same declaration again, which reconciling is for.
func refuseOlder(kept store.Declared, keptErr error, sequence int64) error {
if sequence == 0 || keptErr != nil {
return nil
}
last, err := declaration.ParseTrusted(kept.Declaration)
if err != nil || last.Sequence == 0 {
return nil
}
if sequence < last.Sequence {
return fmt.Errorf("this declaration is older than what the mesh last said to this node: it "+
"is sequence %d, and the one kept here is %d. It arrived late — a backlog, or a broker "+
"that split a burst — and applying it would make this machine into something the mesh has "+
"already moved past. Refused whole; nothing was applied", sequence, last.Sequence)
}
return nil
}
// profileAsReported is the profile as the mesh reads it — the same bytes enrolment sends, so a
// report's profile and an enrolment's are one shape on the controller's side (novox/hq ADR 0161).
func profileAsReported(detected profile.Profile) map[string]any {
reported := map[string]any{}
if raw, err := json.Marshal(detected); err == nil {
_ = json.Unmarshal(raw, &reported)
}
return reported
}
+58
View File
@@ -0,0 +1,58 @@
package main
import (
"encoding/json"
"strings"
"testing"
"github.com/novox/mesh-host/internal/store"
)
// A declaration carries no order, so a host cannot tell an older one from a newer (novox/hq
// 04-ISSUES/107). Its only identity was the digest of its bytes: "not the last" could be said,
// "older" could not.
func keptWith(t *testing.T, sequence int64) store.Declared {
t.Helper()
body, err := json.Marshal(map[string]any{
"declaration": 1, "resources": []any{}, "owns_nothing": true, "sequence": sequence,
})
if err != nil {
t.Fatal(err)
}
return store.Declared{Declaration: body, Signature: []byte("x")}
}
func TestAnOlderDeclarationFromTheMeshIsRefused(t *testing.T) {
err := refuseOlder(keptWith(t, 7), nil, 5)
if err == nil {
t.Fatal("sequence 5 was accepted over a kept 7")
}
if !strings.Contains(err.Error(), "older") || !strings.Contains(err.Error(), "nothing was applied") {
t.Fatalf("the refusal does not say what it is: %v", err)
}
}
func TestANewerOrEqualDeclarationIsNot(t *testing.T) {
if err := refuseOlder(keptWith(t, 7), nil, 8); err != nil {
t.Fatalf("sequence 8 was refused over a kept 7: %v", err)
}
// Equal is the same declaration again, which reconciling is for.
if err := refuseOlder(keptWith(t, 7), nil, 7); err != nil {
t.Fatalf("the same sequence was refused: %v", err)
}
}
func TestNoOrderClaimedMeansNoOrderCompared(t *testing.T) {
// An older controller sends none; a host that received before it understood them kept none.
// Refusing on a guess would strand a node the moment the controller is older than the host.
if err := refuseOlder(keptWith(t, 7), nil, 0); err != nil {
t.Fatalf("a declaration claiming no order was refused: %v", err)
}
if err := refuseOlder(keptWith(t, 0), nil, 3); err != nil {
t.Fatalf("a declaration was refused against a kept one that claimed no order: %v", err)
}
if err := refuseOlder(store.Declared{}, store.ErrNothingDeclared, 3); err != nil {
t.Fatalf("a first declaration was refused: %v", err)
}
}
+48
View File
@@ -0,0 +1,48 @@
package main
import (
"os"
"path/filepath"
"testing"
)
// A component's version comes from where it sits, not from its linker (novox/hq ADR 0142). The mesh's
// toolchain stamps no version — a build does not know what it will be called — so a delivered host
// read as "development build" and the mesh could not tell which host a machine ran (04-ISSUES/161).
func TestADeliveredHostReadsItsVersionFromItsPath(t *testing.T) {
// A delivered host lives at <libexec>/versions/<version>/<binary>.
dir := t.TempDir()
versioned := filepath.Join(dir, "versions", "637f65559d16")
if err := os.MkdirAll(versioned, 0o755); err != nil {
t.Fatal(err)
}
self := filepath.Join(versioned, "nox-mesh-host")
if err := os.WriteFile(self, []byte("#!/bin/sh\n"), 0o755); err != nil {
t.Fatal(err)
}
if got := versionAt(self, "development build"); got != "637f65559d16" {
t.Fatalf("a delivered host read its version as %q", got)
}
}
func TestAHostPlacedByHandKeepsItsStamp(t *testing.T) {
// The honest answer for one the mesh did not deliver — and every machine is in that state until
// a delivery reaches it.
if got := versionAt("/usr/bin/nox-mesh-host", "04a27ca"); got != "04a27ca" {
t.Fatalf("a hand-placed host read its version as %q", got)
}
}
func TestADirectoryThatIsNotAVersionIsNotReadAsOne(t *testing.T) {
// A binary sitting anywhere else must not have its parent directory's name read as a version.
for _, path := range []string{
"/opt/somewhere/nox-mesh-host",
"/usr/lib/nox-mesh-host/launch",
"/home/someone/build/nox-mesh-host",
} {
if got := versionAt(path, "the stamp"); got != "the stamp" {
t.Fatalf("%s read its version as %q", path, got)
}
}
}
+1 -1
View File
@@ -161,7 +161,7 @@
"type": "file",
"path": "/var/lib/mesh-bus-conf/accounts.conf",
"mode": "0600",
"content": "// The first user list, carried by the installer because at genesis there is no mesh to\n// compose one. A bootstrap credential, rotated with the store's and replaced by the\n// controller's own composition from its first start onward.\naccounts {\n MESH {\n jetstream: enabled\n users = [\n { user: \"controller\", password: \"$2a$10$AHqJgOifIVbU41KmATiMhuXFs8xa7Wl2HuN4UVBCXdN2jIQzjqApy\", permissions: {\n publish: { allow: [\"$JS.API.>\", \"$JS.ACK.CONTROL.controller.>\", \"$JS.ACK.EVENTS.controller.>\", \"_INBOX.enrol.>\", \"mesh.control.>\", \"mesh.mod.*.tool.>\", \"mesh.node.>\", \"mesh.seat.mesh-build-machine.accept.>\", \"mesh.seat.mesh-controller.event.applied\", \"mesh.seat.mesh-controller.event.built-before\", \"mesh.seat.mesh-controller.event.refused\"] }\n subscribe: { allow: [\"$JS.API.>\", \"_DELIVER.controller\", \"_DELIVER.controller.>\", \"_INBOX.controller.>\", \"mesh.control.>\", \"mesh.mod.mesh-catalog.event.catching-up\", \"mesh.mod.mesh-catalog.event.upgraded\", \"mesh.mod.gitea.event.pull.merged\", \"mesh.seat.mesh-build-machine.event.built\"] }\n allow_responses: { max: 1, ttl: \"1m\" }\n } }\n ]\n }\n}\n"
"content": "// The first user list, carried by the installer because at genesis there is no mesh to\n// compose one. A bootstrap credential, rotated with the store's and replaced by the\n// controller's own composition from its first start onward.\naccounts {\n MESH {\n jetstream: enabled\n users = [\n { user: \"controller\", password: \"$2a$10$AHqJgOifIVbU41KmATiMhuXFs8xa7Wl2HuN4UVBCXdN2jIQzjqApy\", permissions: {\n publish: { allow: [\"$JS.ACK.CONTROL.controller.>\", \"$JS.ACK.EVENTS.controller.>\", \"$JS.API.>\", \"_INBOX.enrol.>\", \"mesh.assignment.>\", \"mesh.control.>\", \"mesh.mod.*.tool.>\", \"mesh.node.>\", \"mesh.seat.mesh-build-machine.accept.>\", \"mesh.seat.mesh-controller.event.applied\", \"mesh.seat.mesh-controller.event.built-before\", \"mesh.seat.mesh-controller.event.refused\"] }\n subscribe: { allow: [\"$JS.API.>\", \"_DELIVER.controller\", \"_DELIVER.controller.>\", \"_INBOX.controller.>\", \"mesh.control.>\", \"mesh.mod.gitea.event.pull.merged\", \"mesh.mod.mesh-catalog.event.catching-up\", \"mesh.mod.mesh-catalog.event.upgraded\", \"mesh.seat.mesh-build-machine.event.built\", \"mesh.seat.mesh-controller.tool.>\"] }\n allow_responses: { max: 1, ttl: \"1m\" }\n } }\n ]\n }\n}\n"
},
{
"id": "broker",
+9 -7
View File
@@ -1501,13 +1501,15 @@ func containerSpecReading(r *declaration.Container, declares, reads map[string]s
for _, a := range r.Args {
b.WriteString("arg " + a + "\n")
}
// **The mesh's names are part of what a container is** (novox/hq 04-ISSUES/135). A container
// resolves every other machine and every public name through the entries the mesh gives it at
// creation, and nothing re-reads them afterwards — so a container left alone when the roster
// moved is one that cannot reach anything by name, for ever, while every check reports it
// running. That is exactly what happened when this mesh's overlay range changed: one container
// whose image and files never changed kept an address five days out of date and restarted
// 2286 times against a database it could no longer find.
// **A container's declared names are part of what it is** (novox/hq 04-ISSUES/135). What is
// here is what the module declared for itself and nothing else: the mesh's own names are no
// longer written into a container (ADR 0148) — they were once, every container got the whole
// roster at creation and nothing re-read it, so one left alone when the roster moved could not
// reach anything by name for as long as it ran while every check reported it running; and once
// the roster was in this digest so that could be caught, one name moving anywhere replaced
// every container in the mesh (04-ISSUES/151). A container resolves a mesh name through the
// machine's resolver at the moment it asks. What a module declares does not move when the
// roster does, so hashing it costs nothing and catches a manifest that changed.
//
// Sorted, so the digest does not move for a reordering nobody made.
hosts := append([]string(nil), r.Hosts...)
+17 -1
View File
@@ -1137,6 +1137,19 @@ type Declaration struct {
// converged node — which is every node the mesh raised before adoption existed, and so the
// only form an older controller ever sends (novox/hq ADR 0100).
Adoption *Adoption
// Sequence orders this declaration against every other the mesh has sent this node: each
// send is one higher than the last, assigned under the control plane's hold on the node
// (novox/hq 04-ISSUES/107). Zero is a declaration that carries no order — every one an older
// controller sent, and the bundle genesis applies — and a host makes no ordering claim about
// one of those.
//
// **The one property a declaration needs that its signature does not give it.** A signature
// says the mesh sent this; it cannot say the mesh sent it AFTER the one the host is holding.
// Before this, "older" was inferred from arrival within a batch and a 750ms window, and a
// backlog longer than the batch, or a slow broker, applied a declaration the mesh had already
// superseded.
Sequence int64
}
// Adoption is a node's mode, as the controller records it: the node is adopted, and these are
@@ -1274,6 +1287,9 @@ type envelope struct {
// a bug quietly strip a machine.
OwnsNothing bool `json:"owns_nothing,omitempty"`
Resources []json.RawMessage `json:"resources"`
// Sequence is optional on the wire, so a controller that does not send one is still
// understood: absent reads as zero, which is "no ordering claimed" rather than "first".
Sequence int64 `json:"sequence,omitempty"`
}
func parse(raw []byte, allowActions bool) (*Declaration, error) {
@@ -1290,7 +1306,7 @@ func parse(raw []byte, allowActions bool) (*Declaration, error) {
env.Version, Version)}}
}
d := &Declaration{Version: env.Version, For: env.For, Adoption: env.Adoption}
d := &Declaration{Version: env.Version, For: env.For, Adoption: env.Adoption, Sequence: env.Sequence}
var problems []string
if len(env.Resources) == 0 && !env.OwnsNothing {
+21 -1
View File
@@ -3,6 +3,7 @@ package link
import (
"context"
"crypto/rand"
"crypto/sha256"
"encoding/hex"
"errors"
"fmt"
@@ -63,8 +64,15 @@ func presentNats(_ context.Context, to Approach, node, secret string,
// subscribe its own inbox and nothing else (design 25 §6). The secret is its password, the same
// string the request claims, so the server proves somebody holds the token and the request
// proves the same thing to the controller without it having to ask the server who connected.
// **Its own inbox space, because that is the only one it may listen in** (novox/hq
// 04-ISSUES/146). A JetStream publish waits for the stream's acknowledgement on an inbox the
// client picks, and the client's default is `_INBOX.<random>` — which this user may not
// subscribe to, so the enrolment failed with a permissions violation on a subject nobody had
// chosen. The permission is `_INBOX.enrol.<node>.>` (design 25 §6), so the client is told to
// pick its inboxes there; the reply address below is in the same space for the same reason.
conn, err := nats.Connect(natsURL(to.Address),
nats.Secure(config),
nats.CustomInboxPrefix("_INBOX.enrol."+node),
nats.UserInfo("enrol."+node, secret),
nats.Name("mesh-host/enrol/"+node),
nats.Timeout(timeout),
@@ -124,7 +132,19 @@ func (a *natsAsking) Ask(ctx context.Context, request []byte, wait time.Duration
defer cancel()
// Into the stream and awaited: an enrolment the bus never accepted must fail here rather than be
// assumed, because the node has nothing else to go on.
if _, err := a.js.Publish(EnrolSubject, addressed, nats.Context(publish)); err != nil {
//
// **Once, however many times it is sent** (novox/hq 04-ISSUES/146). The client re-publishes when
// an acknowledgement is slow, and the mesh enrolled the machine on each copy — minting a second
// credential, which replaced the first, which is the one the node had already been given. The
// machine then reconnected for ever as a user whose password the mesh had rotated out from under
// it, and the controller's log said "enrolled anchor" twice in the same second.
//
// The id is the message: the same bytes carry the same id, so the stream discards the client's
// own retry, and a genuine second attempt — which carries a new reply address — is a different
// message and is let through.
sum := sha256.Sum256(addressed)
if _, err := a.js.Publish(EnrolSubject, addressed,
nats.MsgId(hex.EncodeToString(sum[:])), nats.Context(publish)); err != nil {
return nil, fmt.Errorf("cannot ask the mesh to enrol this node: %w", err)
}
+5
View File
@@ -110,6 +110,11 @@ type Report struct {
// and the mesh's up in its place, and where the found configuration's original was kept.
Tunnel *CarriedTunnel `json:"tunnel,omitempty"`
// Profile is what this machine can do, detected again by the apply that reports (novox/hq
// ADR 0161) — the same shape enrolment sends — so a capability gained or lost since enrolment,
// a network manager switched, reaches the mesh at the next push rather than never.
Profile map[string]any `json:"profile,omitempty"`
// Host is the version of the host that produced this report (novox/hq ADR 0141).
//
// Without it nothing can say a machine is behind, so "every machine current with its source"
+50
View File
@@ -0,0 +1,50 @@
package link
import (
"encoding/json"
"testing"
"time"
)
// The drain picked the last to arrive. A backlog longer than the batch, or a broker that split a
// burst, delivered a superseded declaration last (novox/hq 04-ISSUES/107).
func sequenced(t *testing.T, n int64) *said {
t.Helper()
inner, err := json.Marshal(map[string]any{"declaration": 1, "resources": []any{}, "sequence": n})
if err != nil {
t.Fatal(err)
}
body, err := json.Marshal(Signed{Declaration: inner, Signature: []byte("s")})
if err != nil {
t.Fatal(err)
}
return &said{body: body}
}
func TestTheDrainKeepsTheHighestSequenceNotTheLastToArrive(t *testing.T) {
waiting := make(chan Declaration, 8)
waiting <- sequenced(t, 9)
waiting <- sequenced(t, 4) // arrived last, composed earlier
latest, aside := newest(waiting, sequenced(t, 8), 30*time.Millisecond)
if got := sequenceOf(latest.Body()); got != 9 {
t.Fatalf("the drain kept sequence %d, and 9 was waiting", got)
}
if len(aside) != 2 {
t.Fatalf("%d set aside, wanted 2 (the 8 and the late 4)", len(aside))
}
}
func TestWithoutSequencesTheLastToArriveStillWins(t *testing.T) {
// The behaviour this had before, kept for a controller that sends no order.
apply, aside := newest(arriving("two", "three"), &said{body: []byte("one")}, 30*time.Millisecond)
if string(apply.Body()) != "three" || len(aside) != 2 {
t.Fatalf("applied %q with %d set aside", apply.Body(), len(aside))
}
}
func TestAnUnreadableBodyClaimsNoOrder(t *testing.T) {
if got := sequenceOf([]byte("not json")); got != 0 {
t.Fatalf("garbage claimed sequence %d", got)
}
}
+28
View File
@@ -300,6 +300,16 @@ func newest(arriving <-chan Declaration, first Declaration, window time.Duration
if !ok {
return latest, superseded
}
// **By sequence when both carry one, by arrival when either does not** (novox/hq
// 04-ISSUES/107). Arrival is what this window had to go on, and it is wrong exactly
// when it matters — a backlog drained out of order. A declaration that says where it
// stands is believed over when it turned up; one that does not is the older
// controller's, and arrival is all there is.
if sequenceOf(next.Body()) < sequenceOf(latest.Body()) &&
sequenceOf(next.Body()) > 0 && sequenceOf(latest.Body()) > 0 {
superseded = append(superseded, next)
continue
}
superseded = append(superseded, latest)
latest = next
case <-time.After(window):
@@ -308,6 +318,24 @@ func newest(arriving <-chan Declaration, first Declaration, window time.Duration
}
}
// sequenceOf is the order a signed declaration claims, or zero when it claims none or cannot be
// read. Read from the envelope alone; the signature is verified later, when the winner is applied,
// and a forged message that lied about its sequence would only set aside real ones — which are
// reported as set aside, and the next push sends the current one again.
func sequenceOf(body []byte) int64 {
var signed Signed
if err := json.Unmarshal(body, &signed); err != nil {
return 0
}
var d struct {
Sequence int64 `json:"sequence"`
}
if err := json.Unmarshal(signed.Declaration, &d); err != nil {
return 0
}
return d.Sequence
}
// declaredIn is the id a signed declaration carries, for a report about one that was not applied.
// Empty if the message is not one — a forged or garbled message is refused by handleBody when its
// turn comes; here it is only named.
+23
View File
@@ -20,6 +20,14 @@ const (
// state: whether one IS running. Assignment needs the first.
CapSeat = "seat"
CapPrivileged = "privileged"
// The network manager this machine runs, one capability per dialect (novox/hq ADR 0161): the
// uplink seat's holder declares its own, so the holder for a manager the machine does not run
// is refused the way any missing capability is, naming it. Active, not installed — a machine
// may have two of these on disk and runs one.
CapUplinkNetworkManager = "uplink-networkmanager"
CapUplinkSystemdNetworkd = "uplink-systemd-networkd"
CapUplinkDhcpcd = "uplink-dhcpcd"
)
// commandCapability is the shape most detectors take: run something, and treat a working
@@ -202,6 +210,21 @@ func Default(runner Runner) []Detector {
why: "asks the kernel for interfaces — needs the module, not just the tool",
runner: runner,
},
commandCapability{
name: CapUplinkNetworkManager, command: "systemctl", args: []string{"is-active", "NetworkManager.service"},
why: "asks the init whether NetworkManager is running — the dialect the uplink seat's holder must speak",
runner: runner,
},
commandCapability{
name: CapUplinkSystemdNetworkd, command: "systemctl", args: []string{"is-active", "systemd-networkd.service"},
why: "asks the init whether systemd-networkd is running — the dialect the uplink seat's holder must speak",
runner: runner,
},
commandCapability{
name: CapUplinkDhcpcd, command: "systemctl", args: []string{"is-active", "dhcpcd.service"},
why: "asks the init whether dhcpcd is running — the dialect the uplink seat's holder must speak",
runner: runner,
},
}
}
+39
View File
@@ -0,0 +1,39 @@
package profile
import (
"context"
"errors"
"testing"
)
// The uplink seat's holder must be the dialect the machine runs (novox/hq ADR 0161): the profile
// names the network manager found active, one capability per manager, and nothing for one that is
// merely installed.
func TestTheProfileNamesTheNetworkManagerThatIsRunning(t *testing.T) {
runner := func(_ context.Context, name string, args ...string) (string, error) {
if name == "systemctl" && len(args) == 2 && args[0] == "is-active" {
if args[1] == "NetworkManager.service" {
return "active\n", nil
}
return "inactive\n", errors.New("exit status 3")
}
return "", errors.New("not here")
}
var have []Detector
for _, d := range Default(Runner(runner)) {
switch d.Name() {
case CapUplinkNetworkManager, CapUplinkSystemdNetworkd, CapUplinkDhcpcd:
have = append(have, d)
}
}
if len(have) != 3 {
t.Fatalf("expected a detector per manager, found %d", len(have))
}
for _, d := range have {
v := d.Detect(context.Background())
want := d.Name() == CapUplinkNetworkManager
if v.Present != want {
t.Errorf("%s: present=%v, want %v (%s)", d.Name(), v.Present, want, v.Detail)
}
}
}