Compare commits

..
Author SHA1 Message Date
mesh-admin 08bb56f1d3 Merge pull request 'The controller composes one tool runtime per node: its principal, the bundles, its process, and the gate (hq ADR 0175, to-be 38 WP2)' (#223) from feat/the-operators-machine into main 2026-10-02 19:27:52 +00:00
mesh-admin 1a44d281c2 Merge pull request 'node show cites ADR 0180 for a removed front end (hq ADR 0186)' (#224) from fix/a-ban-list-never-holds-a-neighbour into main 2026-10-02 16:47:25 +00:00
jschoubben 1c8fe65601 node show cites ADR 0180 for a removed front end (hq ADR 0186)
Another session took 0175 while that record was in review; the line printed on every converged
machine was pointing at an unrelated decision.
2026-10-02 18:42:16 +02:00
jochen 8eb6c3e94a A tool container on the runtime's image is refused once the runtime is registered (hq ADR 0175, to-be 38 WP2.4)
A module declaring tools, a container, and a build on mesh-tools' runtime image is a container whose
purpose is serving tools — the pattern the node's tool runtime retires. Once node-tools is in the
catalogue, registering one is refused by name, with the record that says why; before, it is accepted
as it always was, so a mesh converts in the design's order and nothing is refused before there is
anything to move to. This is the mechanism that keeps the old pattern from returning by habit.

Judged from a repository manifest's own build.on, and for a built manifest — which carries no build
— from what its build stood on, now recorded beside the commit as part of a module's provenance.
2026-10-02 18:30:14 +02:00
jochen 9d4d847dc4 The machine runs one tool runtime, loading every delivered bundle (hq ADR 0175, to-be 38 WP2.3)
Where node-tools is in a node's set, the declaration ends with one process: the runtime module's own
bundle, run from its one entrypoint by its language's interpreter, told in MESH_TOOL_MODULES every
<module>=<file> the machine's bundles load, where its credential is (the module's own broker secret
as this node places it), and — on a machine with an operator account — who the operator is, running
as that account so a tool that needs root can escalate as the operator would. Restarted when any
bundle it loads or the credential changes. A machine with no account runs it as root without the two
operator words; a machine without the runtime is sent nothing new.

A bundle says which of its entrypoints the runtime LOADS (`loads`), because one bundle may carry a
daemon beside its tools and importing the daemon into the runtime would start it there; absent, a
module declaring tools has every entrypoint loaded. And the TypeScript toolchain is rooted at the
module, so an entrypoint lands at the path it is named by — the runtime loading bundles by their
declared paths is what made the compiler's common-directory default visible.
2026-10-02 18:26:54 +02:00
mesh-admin bea1a1c513 Merge pull request 'A control plane behind its seat's row serves what it can (hq ADR 0185)' (#222) from fix/a-service-asked-to-run-is-still-running into main 2026-10-02 16:21:55 +00:00
jochen b1df688c62 A module's tools bundle reaches the machine as an archive where the runtime runs (hq ADR 0175, to-be 38 WP2.2)
The resolved manifest now carries what the build compiled — each bundle's source, digest, language
and entrypoints — because a tools bundle is named by no resource of the module's own: the node's
runtime loads it, and until this the mesh held no trace of the one artifact that runtime needs. A
repository manifest that writes `bundles` beside its build is refused: the mesh derives it.

Where the runtime module is in a node's set, every assigned module's bundle with entrypoints is
composed as an archive under the mesh's own directory, routed through the artifact store like any
image or archive the mesh built. A bundle without entrypoints is run rather than loaded and is
delivered by the process that runs it. A node without the runtime is sent exactly what it was.
2026-10-02 18:19:10 +02:00
jschoubben 21d38c9b0e A control plane behind its seat's row serves what it can (hq ADR 0185)
One verb in the row that this binary cannot run aborted the start, and a stale push that put an
older control plane back took the whole mesh off the bus for ten minutes — recoverable only by a
person running the binary outside its service, because the push that repairs it is one of the verbs
that had stopped being served. Now the verbs it knows are served, the ones it does not answer the
reason, and the start names them once.
2026-10-02 18:16:24 +02:00
jochen 1f86ed4135 One runtime principal per node carries every assigned module's tools (hq ADR 0175, to-be 38 WP2.1)
Where the node-tools module is assigned, the machine's bus user list gains one principal of kind
node-tools in place of that module's own: it may subscribe every carried module's tool namespace
and every held seat's verbs on its node, read and follow every membership on its node, call any
tool anywhere, answer what it is asked — and consume nothing, because tools are what it runs.
Every other module keeps its own principal, so a module still serving tools from its container
holds its own credential until it moves.

Named exactly as the module it stands for, so `module issue` and `rollout mint` deliver its
credential through the path a module's already takes, into node-tools' own `broker` secret. The
runtime module's name is one constant in each of the broker and catalogue packages, held to one
string by the agreement test, because a rule turns on it.
2026-10-02 18:16:14 +02:00
36 changed files with 1107 additions and 371 deletions
+1 -1
View File
@@ -95,7 +95,7 @@ func showFiltering(f inventory.Filtering, adopted bool) {
case fw.Active:
fmt.Printf(" found firewall %s is ACTIVE on this converged machine; the next apply retires it again\n", fw.Kind)
case fw.RetiredBy == "removed":
fmt.Printf(" found firewall %s, removed: the mesh's filter is what filters this machine (novox/hq ADR 0175)\n", fw.Kind)
fmt.Printf(" found firewall %s, removed: the mesh's filter is what filters this machine (novox/hq ADR 0180)\n", fw.Kind)
case fw.RetiredBy == inventory.FilterMesh || fw.RetiredBy == "mesh":
fmt.Printf(" found firewall %s, retired by the mesh; its configuration stays on disk\n", fw.Kind)
case fw.RetiredBy != "":
+2
View File
@@ -522,6 +522,8 @@ func takeIn(ctx context.Context, inv *inventory.Inventory, result link.BuildResu
recorded := inventory.Source{
Repository: result.Repository, Path: result.Path, Ref: result.Ref,
BuiltFrom: result.Commit, Head: result.Commit,
// What it stood on, so registration can judge a built manifest's base (to-be 38 WP2.4).
Against: kept.Against,
}
if result.Source != nil && result.Source.Seat != "" {
recorded.Repository, recorded.Seat = result.Source.Repository, result.Source.Seat
+11 -1
View File
@@ -557,7 +557,7 @@ func issueOnTheNewBus(ctx context.Context, inv *inventory.Inventory, m catalogue
user := broker.Principal{Kind: broker.KindModule, Node: node, Module: m.Module}.Username()
password, err := inv.MintBusPassword(ctx, inventory.BusUser{
Username: user, Kind: inventory.BusModule, Node: node, Module: m.Module,
Username: user, Kind: busKindOf(m.Module), Node: node, Module: m.Module,
})
if err != nil {
return err
@@ -577,6 +577,16 @@ func issueOnTheNewBus(ctx context.Context, inv *inventory.Inventory, m catalogue
return issueWith(ctx, inv, m, node, busAddress, known, reachable, user, password)
}
// busKindOf is what a module's bus user is recorded as: the node's tool runtime where the module is
// the runtime (novox/hq ADR 0175), a module otherwise. The username is the same either way — the
// runtime is issued through this same path — and the kind is what a reader of the records sees.
func busKindOf(module string) string {
if module == catalogue.RuntimeModule {
return inventory.BusNodeTools
}
return inventory.BusModule
}
// issueWith is the delivery half: the minted password sealed to the machine as the module's broker
// secret, and the module's consumer created where the bus can be reached. Split from the minting
// so the move can issue every module against a bus whose address it worked out itself
+1 -14
View File
@@ -232,22 +232,9 @@ func network(ctx context.Context, inv *inventory.Inventory, on map[string]bool,
if err != nil {
return nil, err
}
// **A machine joining is on the network before it is anything else** (novox/hq ADR 0169). Its
// token was issued for its tunnel key and gave it an address, so while that token can still be
// used the hub carries it as a peer: it brings its tunnel up from the token and enrols over it.
// When the token is spent the machine is on the network by what it runs, as every other is; when
// it expires unused, the peer goes with it at the hub's next composition.
joining, err := inv.NodesWithALiveToken(ctx)
if err != nil {
return nil, err
}
isJoining := map[string]bool{}
for _, name := range joining {
isJoining[name] = true
}
nodes := make([]overlay.Node, 0, len(places))
for _, p := range places {
if !on[p.Name] && !(isJoining[p.Name] && p.Key != "" && p.Address != "") {
if !on[p.Name] {
continue
}
n := overlay.Node{
-72
View File
@@ -2,11 +2,9 @@ package main
import (
"context"
"encoding/base64"
"errors"
"flag"
"fmt"
"net"
"strings"
"time"
@@ -232,8 +230,6 @@ func tokenCommand(ctx context.Context, args []string) error {
validFor := set.Duration("for", time.Hour, "how long the token may be used")
adopted := set.Bool("adopted", false,
"the machine joining is in use: it is adopted, and keeps what is found on it")
tunnelKey := set.String("overlay-key", "",
"the public half of the tunnel key the machine made (`nox-mesh-host key`): it joins through the tunnel")
if err := set.Parse(args[1:]); err != nil {
return err
}
@@ -287,14 +283,6 @@ func tokenCommand(ctx context.Context, args []string) error {
default:
return err
}
// **Through the tunnel** (novox/hq ADR 0169): the machine's key recorded, its address given, the
// hub sent it as a peer — all before the token is shown, so the tunnel answers the first time the
// machine knocks. The bus is then reached at its address on the private network.
if *tunnelKey != "" {
if made.Tunnel, made.Broker, err = throughTheTunnel(ctx, open, issued.Node, *tunnelKey, made.Broker); err != nil {
return err
}
}
encoded, err := made.Encode()
if err != nil {
return err
@@ -319,66 +307,6 @@ func tokenCommand(ctx context.Context, args []string) error {
return nil
}
// throughTheTunnel makes a machine a peer of the hub for its token, and says what the token carries
// for it: its first tunnel, and the bus at its address on the private network (novox/hq ADR 0169).
//
// The hub is pushed here, before the token is shown. A token shown before the hub knew the key is a
// tunnel that does not answer, and a machine that cannot tell that from a bus that is down.
func throughTheTunnel(ctx context.Context, open *stores, node inventory.Node, key, busAt string) (
*token.Tunnel, string, error) {
inv := open.inventory
key = strings.TrimSpace(key)
if raw, err := base64.StdEncoding.DecodeString(key); err != nil || len(raw) != 32 {
return nil, "", fmt.Errorf("%q is not a tunnel public key: it is 32 bytes in base64, as "+
"`nox-mesh-host key` prints it", key)
}
// The bus on the private network is the hub's address at the bus's own port, so the port must be
// known before anything is recorded.
_, port, err := net.SplitHostPort(busAt)
if err != nil || port == "" {
return nil, "", fmt.Errorf("the bus's address %q has no port to reach it on", busAt)
}
places, err := inv.Overlays(ctx)
if err != nil {
return nil, "", err
}
var hub *inventory.Overlay
for i := range places {
if places[i].Hub {
hub = &places[i]
}
}
if hub == nil || hub.Key == "" || hub.Endpoint == "" || hub.Address == "" {
return nil, "", errors.New("this mesh has no hub with a key, an address and an endpoint to " +
"dial, so there is no tunnel to join through: place one (`overlay place <node> --hub " +
"--endpoint <host>:<port>`), or issue the token without --overlay-key")
}
if err := inv.RecordOverlayKey(ctx, node.ID, key); err != nil {
return nil, "", err
}
if err := inv.BindTokenToKey(ctx, node.ID, key); err != nil {
return nil, "", err
}
cidr, err := overlayRange(ctx, inv)
if err != nil {
return nil, "", err
}
address, err := inv.AssignAddress(ctx, node.ID, cidr)
if err != nil {
return nil, "", err
}
if err := sendTo(ctx, open, []string{hub.Name}); err != nil {
return nil, "", fmt.Errorf("%s was made a peer of the hub, and the hub could not be sent "+
"it, so the tunnel would not answer — the token is not shown; issue it again once %s "+
"can be pushed: %w", node.Name, hub.Name, err)
}
// An address, not a name — nothing resolves before the machine has joined (novox/hq ADR 0004).
return &token.Tunnel{
Key: key, Address: address + "/32", Range: cidr,
HubKey: hub.Key, HubEndpoint: hub.Endpoint,
}, net.JoinHostPort(hub.Address, port), nil
}
// issueFor is the inventory's half of issuing a token: the record, made when it is new, adopted
// when the operator says so, and the one-time secret for it. The node in what it returns carries
// its mode, which is what the token says.
+2 -1
View File
@@ -1,6 +1,7 @@
package main
import (
"reflect"
"strings"
"testing"
"time"
@@ -191,7 +192,7 @@ func TestWhatAHandedOverModuleRecordsAboutItsSource(t *testing.T) {
t.Fatalf("the source records as %+v", from)
}
// A manifest with no provenance at all is legitimate: fixing something in a hurry.
if from, err := whereItComesFrom("", "", "", "", false); err != nil || from != (inventory.Source{}) {
if from, err := whereItComesFrom("", "", "", "", false); err != nil || !reflect.DeepEqual(from, inventory.Source{}) {
t.Fatalf("a manifest handed over with no provenance was refused: %+v, %v", from, err)
}
for _, c := range []struct {
+8 -1
View File
@@ -131,10 +131,17 @@ func serve(ctx context.Context) error {
// And the mesh's own verbs, as the seat this control plane holds (novox/hq ADR 0154). Served
// from the store's row, so what the seat declares is what is answered.
handlers, err := seatToolHandlers()
handlers, behind, err := seatToolHandlers()
if err != nil {
return err
}
if len(behind) > 0 {
// Said once, loudly, and then served anyway (novox/hq ADR 0185): the mesh keeps answering
// while whatever put an older control plane here is undone.
fmt.Printf("this control plane is behind the %s row: it cannot run %s. "+
"Those answer the reason when called; everything else is served as usual\n",
catalogue.ControllerSeatName, strings.Join(behind, ", "))
}
bus, isNATS := server.Bus().(link.OverNATS)
if !isNATS {
return errors.New("the mesh's verbs are served over the bus, and this control plane is not on it")
+4 -2
View File
@@ -346,7 +346,9 @@ func rolloutMint(ctx context.Context, again bool) error {
}
machines++
case broker.KindModule:
case broker.KindModule, broker.KindNodeTools:
// The runtime is minted and delivered exactly as a module is (novox/hq ADR 0175): it is
// issued as the module it stands for, to that module's `broker` secret.
if p.Module == "mesh-controller" {
// The control plane is a module too, and its `broker` secret is the old bus's
// credential it is still using while this runs. Writing the new bus's blob there
@@ -365,7 +367,7 @@ func rolloutMint(ctx context.Context, again bool) error {
skipped++
continue
}
password, err := inv.MintBusPassword(ctx, inventory.BusUser{Username: p.Username(), Kind: inventory.BusModule, Node: p.Node, Module: p.Module})
password, err := inv.MintBusPassword(ctx, inventory.BusUser{Username: p.Username(), Kind: busKindOf(p.Module), Node: p.Node, Module: p.Module})
if err != nil {
return err
}
+28 -27
View File
@@ -144,26 +144,6 @@ func argvFor(verb string, args map[string]any) ([]string, error) {
// Half of either shape: the command says its usage, which names both shapes, and that is
// the answer the caller needs.
return []string{"rotate"}, nil
case "token":
// `token issue` at a shell (novox/hq ADR 0169). Exactly one of node or new; the command
// refuses both or neither in its own words.
argv := []string{"token", "issue"}
if n := str("node"); n != "" {
argv = append(argv, "--node", n)
}
if n := str("new"); n != "" {
argv = append(argv, "--new", n)
}
if k := str("overlay_key"); k != "" {
argv = append(argv, "--overlay-key", k)
}
if d := str("for"); d != "" {
argv = append(argv, "--for", d)
}
if str("adopted") == "true" {
argv = append(argv, "--adopted")
}
return argv, nil
case "settings":
// `settings set|clear` at a shell (novox/hq issue 198). The values travel as an argument
// because a tool has no file to hand the command; the command reads either.
@@ -250,13 +230,15 @@ func runVerb(ctx context.Context, argv []string) (verbAnswer, error) {
}
// seatToolHandlers are the handlers for every verb the mesh-controller seat declares, from the
// store's row, so a verb the row does not carry is not served and a verb it carries that this binary
// cannot run is said at start rather than at the first call.
func seatToolHandlers() (map[string]link.ToolHandler, error) {
// store's row, so a verb the row does not carry is not served. A verb it carries that this binary
// cannot run is named at start and answers the reason when called — never a refusal to serve, which
// would take the whole control plane down for one word (novox/hq ADR 0185).
func seatToolHandlers() (map[string]link.ToolHandler, []string, error) {
seat, known := catalogue.SeatNamed(catalogue.ControllerSeatName)
if !known {
return nil, fmt.Errorf("this mesh defines no %s seat", catalogue.ControllerSeatName)
return nil, nil, fmt.Errorf("this mesh defines no %s seat", catalogue.ControllerSeatName)
}
var behind []string
handlers := map[string]link.ToolHandler{}
for _, v := range seat.Serves {
verb := v.Name
@@ -267,8 +249,27 @@ func seatToolHandlers() (map[string]link.ToolHandler, error) {
continue
}
if _, err := argvFor(verb, sampleArguments(v)); err != nil {
return nil, fmt.Errorf("the %s seat's row declares %q, which this control plane cannot run: %w",
catalogue.ControllerSeatName, verb, err)
// **A row ahead of this binary is not a reason to go silent.**
//
// The row is the store's and a control plane follows it (novox/hq ADR 0154), so a verb
// this build does not know means the row was widened by a newer one — the ordinary
// state of a roll-out, and of a push that put an older control plane back. Refusing to
// serve at all made that transient fatal: on 2026-10-02 one unknown verb took the whole
// mesh off the bus for ten minutes, and the way back was a human running the binary by
// hand, because the thing that would have repaired it is the thing that was down
// (novox/hq 04-ISSUES/201, ADR 0185).
//
// So the verbs this binary knows are served, and this one answers the reason instead of
// nothing: a caller gets a sentence naming the fault, and everything else keeps working
// — including the push that replaces this binary with the one whose verb it is.
behind = append(behind, verb)
reason := err
handlers[verb] = func(context.Context, json.RawMessage) (any, error) {
return nil, fmt.Errorf("%s is in this mesh's %s row and the control plane running "+
"here cannot run it: %w. It is a verb of a newer build; this one is behind",
verb, catalogue.ControllerSeatName, reason)
}
continue
}
handlers[verb] = func(ctx context.Context, raw json.RawMessage) (any, error) {
args := map[string]any{}
@@ -284,7 +285,7 @@ func seatToolHandlers() (map[string]link.ToolHandler, error) {
return runVerb(ctx, argv)
}
}
return handlers, nil
return handlers, behind, nil
}
// seatTools is what `tools` answers: every seat with a protocol, and the tools each serves, from the
+55 -9
View File
@@ -1,6 +1,7 @@
package main
import (
"context"
"strings"
"testing"
@@ -73,14 +74,6 @@ func TestRotateTakesAProvisionOrAnOwnSecret(t *testing.T) {
}
}
// `token` is `token issue` at a shell, with the machine's tunnel key (novox/hq ADR 0169).
func TestTokenIssuesForAMachineAndItsTunnelKey(t *testing.T) {
argv, err := argvFor("token", map[string]any{"new": "laptop", "overlay_key": "k", "for": "2h"})
if err != nil || strings.Join(argv, " ") != "token issue --new laptop --overlay-key k --for 2h" {
t.Fatalf("token: %v %v", argv, err)
}
}
// `settings` is `settings set|clear` at a shell, with the values passed inline (novox/hq issue 198).
func TestSettingsSetsOrClearsALayer(t *testing.T) {
argv, err := argvFor("settings", map[string]any{"module": "dnsmasq", "values": `{"a":1}`, "node": "ace"})
@@ -138,10 +131,13 @@ func TestActsDoNotBlockTheCall(t *testing.T) {
// What `tools` answers is the seats' records, with each verb's schema.
func TestToolsAnswersTheSeatsRecords(t *testing.T) {
handlers, err := seatToolHandlers()
handlers, behind, err := seatToolHandlers()
if err != nil {
t.Fatal(err)
}
if len(behind) != 0 {
t.Fatalf("this build cannot run %v of its own seat's verbs", behind)
}
if len(handlers) != len(catalogue.ControllerVerbs) {
t.Fatalf("%d handlers for %d verbs", len(handlers), len(catalogue.ControllerVerbs))
}
@@ -203,3 +199,53 @@ func TestCommandRunsTheLineAsGiven(t *testing.T) {
t.Fatal("an unclosed quote was accepted")
}
}
// A verb in the row that this binary cannot run does not take the control plane off the bus: the
// rest are served, the unknown one answers the reason, and the start-up names it (novox/hq ADR
// 0185). One unknown word cost the mesh ten minutes of silence on 2026-10-02, recoverable only by
// a person running the binary by hand — the push that would have repaired it needs the control
// plane that was down.
func TestARowAheadOfThisBuildIsServedAnyway(t *testing.T) {
seat, known := catalogue.SeatNamed(catalogue.ControllerSeatName)
if !known {
t.Fatal("no controller seat")
}
// The row as a newer control plane would have written it: every verb this build knows, and one
// it does not.
widened := seat
widened.Serves = append(append([]catalogue.Verb{}, seat.Serves...),
catalogue.Verb{Name: "teleport", Description: "a verb from a build that does not exist yet"})
rows := catalogue.DefaultSeats()
for i := range rows {
if rows[i].Name == catalogue.ControllerSeatName {
rows[i] = widened
}
}
catalogue.UseSeats(rows)
t.Cleanup(func() { catalogue.UseSeats(catalogue.DefaultSeats()) })
handlers, behind, err := seatToolHandlers()
if err != nil {
t.Fatalf("a row with one unknown verb refused to serve at all: %v", err)
}
if len(behind) != 1 || behind[0] != "teleport" {
t.Fatalf("the verbs this build cannot run were reported as %v", behind)
}
if len(handlers) != len(widened.Serves) {
t.Fatalf("%d handlers for %d verbs in the row", len(handlers), len(widened.Serves))
}
for _, known := range []string{"status", "nodes", "push"} {
if handlers[known] == nil {
t.Errorf("%s is not served although this build knows it", known)
}
}
_, err = handlers["teleport"](context.Background(), nil)
if err == nil {
t.Fatal("the unknown verb answered as though it had run")
}
for _, want := range []string{"teleport", "cannot run it", "behind"} {
if !strings.Contains(err.Error(), want) {
t.Errorf("the answer does not say %q: %v", want, err)
}
}
}
@@ -234,3 +234,13 @@ func admitsSubject(pattern, subject []string) bool {
}
return len(pattern) == len(subject)
}
// The two packages name the runtime module separately — the broker's types stay free of the
// catalogue's on purpose — so this is what holds them to one string. A rename that reached only one
// side would compose a runtime principal for a module nobody assigns, silently, and leave the one
// that is assigned with a module's own grants.
func TestTheBrokerAndTheCatalogueAgreeOnTheRuntimeModule(t *testing.T) {
if RuntimeModule != catalogue.RuntimeModule {
t.Fatalf("the broker calls the runtime %q and the catalogue %q", RuntimeModule, catalogue.RuntimeModule)
}
}
+34
View File
@@ -73,3 +73,37 @@ func TestAnAccountMayReadItsOwnMembershipAndNoOthers(t *testing.T) {
has(t, perms.Publish, "$JS.API.DIRECT.GET.ASSIGNMENTS.mesh.assignment.anchor.postgres")
hasNot(t, perms.Subscribe, "mesh.assignment.>")
}
// The runtime arriving on a machine changes nothing about what each module is issued (to-be 38 WP2):
// the memberships are composed as before and the runtime reads several of them. What the machine's
// user list gains is one runtime principal, and loses nothing but the runtime module's own.
func TestTheRuntimeArrivingLeavesEveryMembershipAsItWas(t *testing.T) {
filter := Seat{Name: "node-packet-filter", Scope: "node", Serves: []string{"rules", "reload"}}
three := []Declared{
{Module: "nftables", Holds: []Seat{filter}, Serves: []string{"firewall_rules"}},
{Module: "zsh", Serves: []string{"execute"}},
{Module: "systemd", Serves: []string{"units"}},
}
before := Records{Nodes: []string{"anchor"}, Assigned: map[string][]Declared{"anchor": three}}
after := Records{Nodes: []string{"anchor"}, Assigned: map[string][]Declared{
"anchor": append(append([]Declared{}, three...), Declared{Module: RuntimeModule}),
}}
for _, d := range three {
was := MembershipFor("anchor", d, PlacementsOf(before, nil))
is := MembershipFor("anchor", d, PlacementsOf(after, nil))
if !reflect.DeepEqual(was, is) {
t.Errorf("%s's membership changed when the runtime arrived:\n%+v\n%+v", d.Module, was, is)
}
}
users, err := Users(after)
if err != nil {
t.Fatal(err)
}
kinds := map[Kind]int{}
for _, p := range users {
kinds[p.Kind]++
}
if kinds[KindNodeTools] != 1 || kinds[KindModule] != 3 || kinds[KindNode] != 1 || kinds[KindController] != 1 {
t.Errorf("the machine's users are %v; one runtime, the three modules, the host and the controller", kinds)
}
}
+85 -2
View File
@@ -34,8 +34,20 @@ const (
// authority is a list of tools and nothing else — not control, not declarations, not builds,
// and no ability to answer anything, because a person asks.
KindPerson Kind = "person"
// KindNodeTools is a machine's tool runtime (novox/hq ADR 0175, to-be 38): one process per
// node, on the host side, serving every assigned module's tools and every held seat's verbs.
// Its authority is the union of what the modules it carries would each have had for their
// tools — and nothing of what they consume, because tools are what it runs, not reactions.
KindNodeTools Kind = "node-tools"
)
// RuntimeModule is the module that IS the node's tool runtime (novox/hq ADR 0175). Where it is
// assigned, the mesh composes one runtime principal for the machine in place of that module's own,
// and the per-module containers that served tools until then stop being the way tools reach a node.
// Mirrored in the catalogue package, which the agreement test holds to the same string; one
// constant, so a rename is one edit and the two packages cannot drift.
const RuntimeModule = "node-tools"
// Seat is a role on the bus as a principal relates to it: the subjects it accepts, and those it
// emits (novox/hq ADR 0118, design 29 §5).
type Seat struct {
@@ -74,6 +86,13 @@ type Principal struct {
// a namespace no such module owns. Every service started and the graph stayed empty.
Watches []Seat
// Carries are the modules whose tools this principal serves, for a KindNodeTools principal
// (novox/hq ADR 0175): every module assigned to its node, as each declares itself. Its
// serving authority is the union of theirs — each module's own tool namespace and each held
// seat's verbs on this node — derived from the same declarations the modules' own principals
// are, so the runtime can serve nothing a module could not have served for itself.
Carries []Declared
// Invokes are the tools this principal may call, as `<module>.<tool>`; a single `*` is every
// tool. A person's whole authority (design 25 §7), and a module's only if its manifest says so
// (novox/hq ADR 0152) — the console's does, and nothing else's.
@@ -112,7 +131,10 @@ func (p Principal) Username() string {
switch p.Kind {
case KindPerson:
return "person." + p.Module
case KindModule:
case KindModule, KindNodeTools:
// The runtime is named exactly as the module it stands for would have been: the mesh
// issues its credential through the same path a module's takes (`module issue`), and
// that path knows the node and the module, not the kind.
return p.Node + "." + p.Module
case KindNode:
return "node." + p.Node
@@ -375,6 +397,48 @@ func PermissionsFor(p Principal) (Permissions, error) {
pub = append(pub, seatToolSubject(s, t, "*"))
}
}
case KindNodeTools:
// **One process serves what every module on the machine would have served for itself**
// (novox/hq ADR 0175). Each carried module's whole tool namespace — the same grant that
// module's own principal has, for the same reason: the tools a module serves are what its
// code answers, and a list here would be a second copy of it. Each held seat's verbs on
// this node, as the holder's own principal would be granted them.
for _, d := range p.Carries {
if !safeSubject.MatchString(d.Module) {
return Permissions{}, fmt.Errorf(
"%q cannot be part of a subject: a permission is a subject pattern, and this would widen it", d.Module)
}
own := "mesh.mod." + d.Module
sub = append(sub, own+".tool.>")
// A tool that emits an event is the module's code and emits under the module's name
// (ADR 0042); the runtime carrying that code may publish what the module declared it
// emits, and nothing it did not.
for _, e := range d.Emits {
pub = append(pub, own+".event."+e)
}
for _, s := range d.Holds {
for _, t := range s.Serves {
sub = append(sub, seatToolSubject(s, t, p.Node))
}
}
}
// Every assigned module's membership on this node (ADR 0160): one per module, read
// directly from the stream and followed live. This node's and no other's — the one token
// that varies is the module, so the pattern is the machine's own assignments.
sub = append(sub, "mesh.assignment."+p.Node+".*")
pub = append(pub, "$JS.API.DIRECT.GET."+AssignmentsStream+".mesh.assignment."+p.Node+".*")
// And every tool on the mesh (ADR 0175, decision 5): any node may call any tool on any
// node, as the console already could — the runtime is the console's serving mode.
invoked, err := invokedSubjects([]string{"*"})
if err != nil {
return Permissions{}, err
}
pub = append(pub, invoked...)
// Nothing about consumers: it consumes nothing. A module's reactions to events are its
// own long-lived process, which ADR 0175 leaves where it is; what moves here is tools.
sub = unique(sub)
pub = unique(pub)
}
if p.Kind == KindPerson {
@@ -382,6 +446,11 @@ func PermissionsFor(p Principal) (Permissions, error) {
// consumer, because nothing is delivered to a person — they ask and are answered.
sub = append(sub, p.inbox())
}
if p.Kind == KindNodeTools {
// Its reply space, so the answers to what its tools call come back to it. No ack subject
// for the same reason a person has none: nothing is delivered to it.
sub = append(sub, p.inbox())
}
if p.Kind == KindModule || p.Kind == KindNode || p.Kind == KindController {
// Its own reply space, and nothing wider.
@@ -403,7 +472,7 @@ func PermissionsFor(p Principal) (Permissions, error) {
// A module answers what it was asked — a tool call reaches it on its own namespace, so the
// authority is bounded by having been asked — and so does the controller. A node and a
// person are never asked anything, and are granted nothing here.
AllowResponses: p.Kind == KindModule || p.Kind == KindController,
AllowResponses: p.Kind == KindModule || p.Kind == KindController || p.Kind == KindNodeTools,
}, nil
}
@@ -625,6 +694,20 @@ func ComposeAccounts(principals []Principal) (string, error) {
return b.String(), nil
}
// unique is a sorted list with each subject once. Two carried modules holding seats with the same
// verb, or the runtime module itself carried beside the others, would otherwise write a grant twice
// — harmless to the server, and noise in a file that is read as the mesh's authority model.
func unique(values []string) []string {
sort.Strings(values)
out := values[:0]
for i, v := range values {
if i == 0 || v != values[i-1] {
out = append(out, v)
}
}
return out
}
func quoted(values []string) string {
if len(values) == 0 {
return ""
+70
View File
@@ -371,3 +371,73 @@ func TestAModulePullsItsOwnConsumerAndNoOthers(t *testing.T) {
}
}
}
// The runtime's authority is the union of what the modules it carries would have been granted for
// their tools (novox/hq ADR 0175): every carried module's tool namespace, every held seat's verbs
// on this node, every module's membership on this node, and a call to anything. Nothing it
// consumes, because it reacts to nothing.
func TestTheRuntimeServesTheUnionAndConsumesNothing(t *testing.T) {
filter := Seat{Name: "node-packet-filter", Scope: "node", Serves: []string{"rules", "reload"}}
p := Principal{Kind: KindNodeTools, Node: "anchor", Module: RuntimeModule, Carries: []Declared{
{Module: "nftables", Holds: []Seat{filter}, Serves: []string{"firewall_rules"}},
{Module: "zsh", Emits: []string{"shell.opened"}, Consumes: []string{"shop.order.placed"}},
{Module: RuntimeModule},
}}
perms, err := PermissionsFor(p)
if err != nil {
t.Fatal(err)
}
for _, want := range []string{
"mesh.mod.nftables.tool.>", "mesh.mod.zsh.tool.>", "mesh.mod." + RuntimeModule + ".tool.>",
"mesh.seat.node-packet-filter.tool.rules.anchor", "mesh.seat.node-packet-filter.tool.reload.anchor",
"mesh.assignment.anchor.*",
"_INBOX.anchor." + RuntimeModule + ".>",
} {
if !contains(perms.Subscribe, want) {
t.Errorf("the runtime may not subscribe %s: %v", want, perms.Subscribe)
}
}
for _, want := range []string{
"mesh.mod.*.tool.>", "mesh.seat.*.tool.>",
"$JS.API.DIRECT.GET.ASSIGNMENTS.mesh.assignment.anchor.*",
"mesh.mod.zsh.event.shell.opened",
} {
if !contains(perms.Publish, want) {
t.Errorf("the runtime may not publish %s: %v", want, perms.Publish)
}
}
// Nothing of what a carried module consumes, and no consumer of its own to ack.
for _, s := range perms.Subscribe {
if strings.Contains(s, ".event.") || strings.HasPrefix(s, "_DELIVER.") {
t.Errorf("the runtime was granted a delivery it has no consumer for: %s", s)
}
}
for _, s := range perms.Publish {
if strings.HasPrefix(s, "$JS.ACK.") || strings.Contains(s, "CONSUMER") {
t.Errorf("the runtime was granted a consumer's subject and has no consumer: %s", s)
}
}
if !perms.AllowResponses {
t.Error("the runtime answers what it is asked, and may not reply")
}
if _, needed := ConsumerFor(p); needed {
t.Error("a consumer would be made for the runtime, which consumes nothing")
}
// Each subject once: the file is read as the mesh's authority model.
seen := map[string]bool{}
for _, s := range append(append([]string{}, perms.Subscribe...), perms.Publish...) {
if seen[s] {
t.Errorf("%s is granted twice", s)
}
seen[s] = true
}
}
func contains(list []string, want string) bool {
for _, s := range list {
if s == want {
return true
}
}
return false
}
+20
View File
@@ -62,13 +62,33 @@ func Users(r Records) ([]Principal, error) {
for _, node := range sortedCopy(r.Nodes) {
out = append(out, Principal{Kind: KindNode, Node: node})
// **Where the runtime is assigned, the machine gets one runtime principal in place of the
// runtime module's own** (novox/hq ADR 0175, to-be 38). It carries every module on the
// node: its serving grants are the union of theirs. Every other module keeps its own
// principal — a module still serving tools from its own container holds its own
// credential until it moves, and the two serve side by side in the meantime.
runtimeHere := false
for _, d := range r.Assigned[node] {
if d.Module == RuntimeModule {
runtimeHere = true
}
}
for _, d := range r.Assigned[node] {
if runtimeHere && d.Module == RuntimeModule {
continue
}
out = append(out, Principal{
Kind: KindModule, Node: node, Module: d.Module,
Emits: d.Emits, Consumes: d.Consumes, Serves: d.Serves,
Holds: d.Holds, Uses: d.Uses, Watches: d.Watches, Invokes: d.Invokes,
})
}
if runtimeHere {
out = append(out, Principal{
Kind: KindNodeTools, Node: node, Module: RuntimeModule,
Carries: append([]Declared(nil), r.Assigned[node]...),
})
}
}
for _, node := range sortedCopy(r.Enrolling) {
out = append(out, Principal{Kind: KindEnrolment, Node: node})
+51
View File
@@ -245,3 +245,54 @@ func TestAUserListIsComposedBeforeAnythingMovesOntoTheBus(t *testing.T) {
t.Errorf("the composed list does not contain the machine running the bus")
}
}
// Where the runtime module is assigned, the machine gets one runtime principal in place of the
// runtime module's own (novox/hq ADR 0175, to-be 38). Every other module keeps its own: a module
// still serving tools from its own container holds its own credential until it moves.
func TestTheRuntimeModuleBecomesTheMachinesRuntimePrincipal(t *testing.T) {
r := someRecords()
r.Assigned["one"] = append(r.Assigned["one"], Declared{Module: RuntimeModule})
users, err := Users(r)
if err != nil {
t.Fatal(err)
}
var runtime *Principal
for i := range users {
p := &users[i]
if p.Node == "one" && p.Module == RuntimeModule {
if p.Kind == KindModule {
t.Fatalf("%s on one was composed as an ordinary module beside the runtime", RuntimeModule)
}
runtime = p
}
}
if runtime == nil || runtime.Kind != KindNodeTools {
t.Fatalf("one runs %s and got no runtime principal: %v", RuntimeModule, namesOf(t, r))
}
if runtime.Username() != "one."+RuntimeModule {
t.Errorf("the runtime is named %q; `module issue` names it as the module it stands for", runtime.Username())
}
carried := map[string]bool{}
for _, d := range runtime.Carries {
carried[d.Module] = true
}
if !carried["telegram"] || !carried[RuntimeModule] {
t.Errorf("the runtime carries %v; it carries every module on its node", carried)
}
// And the other node, where the runtime is not assigned, is exactly as before.
for _, p := range users {
if p.Node == "two" && p.Kind == KindNodeTools {
t.Fatal("two runs no runtime and was given a runtime principal")
}
}
// A module serving its own tools beside the runtime keeps its own principal.
found := false
for _, p := range users {
if p.Kind == KindModule && p.Node == "one" && p.Module == "telegram" {
found = true
}
}
if !found {
t.Error("telegram lost its own principal when the runtime arrived on its node")
}
}
+7 -1
View File
@@ -107,10 +107,16 @@ var toolchains = []Toolchain{
// symlinks to a launcher that requires its library relatively — and the base image's own
// assembly resolves them away, leaving a launcher whose relative require points nowhere.
// Every module's hand-written Dockerfile had to know this. Now none of them does.
// **Rooted at the module, so an entrypoint lands where it is named.** Without a root the
// compiler takes the common directory of the files it is given: a module compiling only
// `tools/index.ts` had its output at `index.js`, and the entrypoint it declared —
// `tools/index.js`, "named as it will be found" — named a file the bundle did not
// contain. The runtime that loads bundles by their declared entrypoints (novox/hq ADR
// 0175) is what made this visible.
Compile: []string{
"node", "/app/node_modules/typescript/bin/tsc",
"--module", "NodeNext", "--moduleResolution", "NodeNext",
"--target", "ES2022",
"--target", "ES2022", "--rootDir", ".",
},
OutputFlag: "--outDir",
Unit: UnitSources,
+39
View File
@@ -67,6 +67,31 @@ func (m Manifest) Resolve(built []Built) (Manifest, error) {
out := m
out.Build = nil
out.Resources = nil
// What the build compiled, kept on the resolved manifest (novox/hq ADR 0175): a tools bundle is
// named by no resource of the module's own — the node's runtime loads it — so this is the only
// place the mesh would otherwise not have it. In artifact order, so two resolutions of one
// build compare equal.
out.Bundles = nil
if m.Build != nil {
for _, a := range m.Build.Artifacts {
if a.Kind != ArtifactBundle {
continue
}
made := by[a.Name]
// What the runtime loads: what the artifact said, else every entrypoint of a module
// that declares tools, else nothing (the field's own rule; see Artifact.Loads).
loads := append([]string(nil), a.Loads...)
if a.Loads == nil && len(m.Tools) > 0 {
loads = append([]string(nil), a.Entrypoints...)
}
out.Bundles = append(out.Bundles, Bundle{
Name: a.Name, Source: made.Reference, Digest: made.Digest,
Language: a.Language, Entrypoints: append([]string(nil), a.Entrypoints...),
Loads: loads,
})
}
sort.Slice(out.Bundles, func(i, j int) bool { return out.Bundles[i].Name < out.Bundles[j].Name })
}
for _, r := range m.Resources {
named, _ := r["artifact"].(string)
if named == "" {
@@ -191,6 +216,20 @@ func (b *Build) problems(module string) []string {
"%s: %q is a bundle and says no language, so nothing can choose a compiler "+
"for it", module, a.Name))
}
// What the runtime loads is among what was compiled (ADR 0175): a name here that is
// not an entrypoint is a file the bundle does not contain, and the runtime would
// fail to import it on every machine rather than here.
for _, load := range a.Loads {
found := false
for _, e := range a.Entrypoints {
found = found || e == load
}
if !found {
problems = append(problems, fmt.Sprintf(
"%s: %q says the runtime loads %q, which is not among its entrypoints — "+
"what is loaded is compiled, so it is named there too", module, a.Name, load))
}
}
// **A system, for a language that compiles to a binary** (novox/hq ADR 0142). A binary
// is pinned to one operating system at link time so a host refuses to touch a machine
// it was not built for (novox/hq ADR 0005); an artifact that says nothing would be
+18
View File
@@ -512,6 +512,13 @@ func (r Resolution) compose(with Rendering, owner map[string]string,
"id": NeedID(name), "type": "file", "path": m.OwnSecrets[name].Path, "sealed": sealed,
}))
}
// This module's tools bundles, where the machine runs the node's tool runtime (novox/hq
// ADR 0175, to-be 38 WP2). Mesh-computed like everything above it, and before the module's
// own resources for the same reason: the runtime's process names the files inside these
// and is restarted when one changes, so they are on the machine before it is.
if r.runtimeHere() {
first = append(first, bundleArchives(m)...)
}
// Operator-owned paths this module is granted use of (novox/hq ADR 0051). Written before
// the module's own resources, and so before the container that mounts them: the host must
// find each present — refusing clearly if the operator has not provided it — before it
@@ -885,6 +892,17 @@ func (r Resolution) compose(with Rendering, owner map[string]string,
out = append(out, fact)
}
}
// The node's tool runtime, last (novox/hq ADR 0175, to-be 38 WP2.3): one process loading every
// bundle delivered above and holding the credential sealed above, so both exist before it starts
// — the order written here is the order the machine applies.
if r.runtimeHere() {
process, err := r.runtimeProcess(with)
if err != nil {
return nil, err
}
owner[fmt.Sprint(process["id"])] = RuntimeModule
out = append(out, process)
}
if with.Adopted {
// First, before anything a module declares: what the mesh needs reachable, then its guard.
// The order a machine applies is the order written here.
+49
View File
@@ -572,6 +572,36 @@ type Manifest struct {
// a module that could ask for it could read every credential on the bus — and the claim on
// `mesh-broker` is what authorises it, checked from this manifest alone.
BusUsers string `json:"bus-users,omitempty"`
// Bundles are this module's compiled bundles as the build produced them: what each is called,
// where it is, what it hashes to, what language it is in and which files a tool runtime loads
// from it (novox/hq ADR 0175, to-be 38).
//
// **Derived, never written.** The manifest in a repository says `build.artifacts`; the manifest
// the mesh holds says what came out, the way a resource naming an artifact comes to name a
// digest. Kept here because a tools bundle is referenced by no resource of the module's own —
// the node's runtime loads it, and the runtime is composed by the mesh — so without this the
// resolved manifest would carry no trace of the one artifact the runtime needs. A repository
// manifest that writes this beside a build is refused: it would be stating the build's output
// by hand.
Bundles []Bundle `json:"bundles,omitempty"`
}
// Bundle is one compiled bundle after it exists, as the resolved manifest carries it.
type Bundle struct {
Name string `json:"name"`
// Source is where a machine fetches it, kept without the store's address like every reference
// the mesh records (artifacts.go); Digest is what it must hash to.
Source string `json:"source"`
Digest string `json:"digest"`
// Language is what it was compiled from, which is what says how it is run.
Language string `json:"language,omitempty"`
// Entrypoints are the compiled files it was built around, relative to its root.
Entrypoints []string `json:"entrypoints,omitempty"`
// Loads are the entrypoints a node's tool runtime imports from it: what the artifact said, or
// every entrypoint for a module declaring tools that said nothing. Empty for a bundle that is
// run rather than loaded.
Loads []string `json:"loads,omitempty"`
}
// Build says how to produce this module's artifacts from its source.
@@ -708,6 +738,16 @@ type Artifact struct {
// somebody adds a helper. An empty list is a bundle that is run rather than loaded — a
// provisioner or a step, named by whatever runs it.
Entrypoints []string `json:"entrypoints,omitempty"`
// Loads are the entrypoints of this bundle the node's tool runtime loads (novox/hq ADR 0175,
// to-be 38): the module's tool code, each file registering its tools as it is imported. A
// subset of Entrypoints, for a bundle that also carries things that are RUN — a daemon, a
// step, a report — and must not have them imported into the runtime.
//
// Absent means every entrypoint, for a module that declares `tools`: a bundle holding the
// module's tools and nothing else is the ordinary case and should not have to say the same
// list twice. A module declaring no tools has nothing the runtime loads, whatever it compiles.
Loads []string `json:"loads,omitempty"`
}
// Kinds an artifact may be.
@@ -1333,6 +1373,15 @@ func ParseManifest(raw []byte) (Manifest, error) {
//
// Refused here because the alternative is a build that never returns, on a mesh new enough
// that nobody is watching it yet.
if m.Build != nil && len(m.Bundles) > 0 {
// The output of a build, written beside the build that produces it (ADR 0175). A resource
// naming a digest beside an `artifact` would be the same mistake, and is caught the same way:
// what the mesh derives, a repository does not state.
problems = append(problems, fmt.Sprintf(
"%s writes `bundles` beside its build. The mesh derives that from what the build "+
"produced; a manifest states `build.artifacts` and nothing about what came out",
m.Module))
}
if m.Build != nil && len(m.Build.Artifacts) > 0 {
for _, o := range m.Offers() {
if o != ArtifactStoreProvision {
+251
View File
@@ -0,0 +1,251 @@
package catalogue
import (
"fmt"
"sort"
"strings"
)
// The node's tool runtime, as the catalogue knows it (novox/hq ADR 0175, to-be 38).
//
// **One module is the runtime.** Where it is assigned, one process per machine serves every assigned
// module's tools and every held seat's verbs, on the host side, from the bundles each module's build
// produced — and no module needs a container to reach the bus with its tools. The name is a constant
// rather than a manifest field because a rule turns on it: the composer places the runtime's process
// where this module is, and registration refuses the old pattern once this module exists.
// RuntimeModule is the module that is the node's tool runtime. Mirrored in the broker package,
// which composes a principal of its own for it; the agreement test there holds the two to one string.
const RuntimeModule = "node-tools"
// BundleRoot is where a machine keeps the tools bundles the mesh delivers to it: under the mesh's
// own directory, beside the daemons the host unpacks there, and never where a package manager also
// writes. One directory per module, one per bundle beneath it, at a path that does not move with
// the version — so the runtime's process names each entrypoint once and is restarted, not
// recomposed, when a bundle changes.
const BundleRoot = "/var/lib/mesh/bundles"
// BundleID names the archive resource that delivers one of a module's bundles; prefixed with the
// module like every resource of its own.
func BundleID(bundle string) string { return "bundle-" + bundle }
// BundlePath is where one module's bundle is unpacked on a machine.
func BundlePath(module, bundle string) string { return BundleRoot + "/" + module + "/" + bundle }
// runtimeHere says whether this node's set includes the runtime module, which is what decides
// whether anything about tools changes on the machine (to-be 38 WP2): until the runtime is assigned,
// a node is sent exactly what it was sent before, bundles included, because a bundle nothing loads
// is bytes nobody reads.
func (r Resolution) runtimeHere() bool {
for _, m := range r.Modules {
if m.Module == RuntimeModule {
return true
}
}
return false
}
// bundleArchives is one archive per tools bundle of a module — a bundle the runtime LOADS something
// from — as the host fetches and unpacks any artifact (novox/hq ADR 0175 §3: a module brings its
// tools as a bundle, delivered by the host like any artifact, never an image). A bundle it loads
// nothing from is run rather than loaded: a daemon, a step, the runtime itself — delivered by the
// process that runs it, and not again here.
//
// The source is the kept reference; the per-resource pass that follows routes it through the
// artifact store as this network reaches it now, as it does every image and archive the mesh built.
func bundleArchives(m Manifest) []map[string]any {
var out []map[string]any
for _, b := range m.Bundles {
if len(b.Loads) == 0 {
continue
}
out = append(out, map[string]any{
"id": BundleID(b.Name), "type": "archive",
"source": b.Source, "digest": b.Digest,
"path": BundlePath(m.Module, b.Name),
})
}
return out
}
// RuntimeProcessID names the one process the mesh composes for a machine's runtime; prefixed with
// the runtime module like a resource of its own, because that module is what the host sees it as.
func RuntimeProcessID() string { return "runtime" }
// RuntimeToolModules is the variable the runtime reads the modules it serves from: one
// `<module>=<entrypoint>` per file it loads, comma-separated — several entries may name one module.
// RuntimeBrokerFile is where it reads the node's credential; RuntimeOperatorAccount and
// RuntimeOperatorHome are the machine's operator account and home, handed to every tool's
// environment (to-be 38 WP1), and absent on a machine with no account.
const (
RuntimeToolModules = "MESH_TOOL_MODULES"
RuntimeBrokerFile = "MESH_BROKER_FILE"
RuntimeOperatorAccount = "MESH_OPERATOR_ACCOUNT"
RuntimeOperatorHome = "MESH_OPERATOR_HOME"
)
// interpreterFor is how a bundle in a language is run: the program the host's unit starts, with the
// bundle's entrypoint after it. The one thing the composer takes from a language, and said here
// rather than in a manifest because the runtime's process is the mesh's to compose (to-be 38 WP3).
func interpreterFor(language string) (string, error) {
switch language {
case "typescript":
return "node", nil
}
return "", fmt.Errorf(
"%s is written in %q, and the mesh knows no interpreter to run a %q bundle with",
RuntimeModule, language, language)
}
// runtimeProcess is the one process a machine runs the node's tool runtime as (novox/hq ADR 0175,
// to-be 38 WP2.3): the runtime module's own bundle, run by its language's interpreter, told which
// modules it serves and from which files, where its credential is, and who the machine's operator
// is — and restarted when any bundle it loads or the credential it holds changes.
//
// Composed from the placed manifests, so the credential's path is where this node puts it. The
// runtime runs as the operator's account when the machine has one, which is what lets a tool that
// needs root escalate as the operator would (ADR 0175 §4); on a machine with no account it runs as
// root, and the two operator words are not set.
func (r Resolution) runtimeProcess(with Rendering) (map[string]any, error) {
var runtime *Manifest
for i := range r.Modules {
if r.Modules[i].Module == RuntimeModule {
runtime = &r.Modules[i]
}
}
if runtime == nil {
return nil, nil
}
if len(runtime.Bundles) != 1 {
return nil, fmt.Errorf(
"%s is assigned to %s and its build produced %d bundle(s); the runtime is one bundle "+
"the mesh runs, so the module declares exactly one (novox/hq to-be 38)",
RuntimeModule, r.Node, len(runtime.Bundles))
}
bundle := runtime.Bundles[0]
if len(bundle.Entrypoints) != 1 {
return nil, fmt.Errorf(
"%s's bundle %q names %d entrypoint(s); the runtime is run from one, so the module "+
"declares exactly one (novox/hq to-be 38)", RuntimeModule, bundle.Name, len(bundle.Entrypoints))
}
interpreter, err := interpreterFor(bundle.Language)
if err != nil {
return nil, err
}
credential, declared := runtime.OwnSecrets["broker"]
if !declared {
return nil, fmt.Errorf(
"%s declares no own secret named broker, and the node's credential is delivered there: "+
"a module that speaks on the bus declares \"own-secrets\": {\"broker\": <path>}",
RuntimeModule)
}
// What it serves, and from which files: every module on this machine that composes here, in
// name order, each bundle it loads from in the order the manifest gave. A module left out of
// the declaration — a filter on an adopted machine — is left out of this too, or the runtime
// would be told to load files that were never delivered.
var served []string
var restartOn []string
for _, m := range r.Modules {
if with.Adopted && m.Filtering != nil {
continue
}
for _, b := range m.Bundles {
if len(b.Loads) == 0 {
continue
}
for _, load := range b.Loads {
served = append(served, m.Module+"="+BundlePath(m.Module, b.Name)+"/"+load)
}
restartOn = append(restartOn, m.Module+"."+BundleID(b.Name))
}
}
sort.Strings(served)
restartOn = append(restartOn, RuntimeModule+"."+NeedID("broker"))
sort.Strings(restartOn)
env := map[string]string{
RuntimeToolModules: strings.Join(served, ","),
RuntimeBrokerFile: credential.Path,
}
process := map[string]any{
"id": RuntimeModule + "." + RuntimeProcessID(), "type": "process", "name": RuntimeModule,
"source": bundle.Source, "digest": bundle.Digest,
"run": []any{interpreter, bundle.Entrypoints[0]},
"env": env,
"restart-on": toAny(restartOn),
}
if r.Account != "" {
env[RuntimeOperatorAccount] = r.Account
env[RuntimeOperatorHome] = accountHomeOf(r.Account, r.AccountHome)
process["user"] = r.Account
}
// Routed through the artifact store as this network reaches it now, like everything the mesh
// built; refused with the same words when there is no store to route through.
if err := artifactsInto(process, RuntimeModule, with); err != nil {
return nil, err
}
return process, nil
}
func toAny(in []string) []any {
out := make([]any, 0, len(in))
for _, s := range in {
out = append(out, s)
}
return out
}
// RuntimeImageModule and RuntimeImageArtifact name the image every per-module tool container was
// built on: the tool runtime's own runtime image. With the runtime a module of its own, that image
// stays the way a module's SERVICE may be built and stops being the way tools reach a node (ADR 0175).
const (
RuntimeImageModule = "mesh-tools"
RuntimeImageArtifact = "runtime"
)
// ToolContainerOnTheRuntime says why a manifest is the pattern ADR 0175 retires — a module whose tools
// are served from a container built on the tool runtime's image — or nothing when it is not. Judged
// from the manifest's own `build.on` when it is a repository manifest, and from what its build stood
// on when it is a built one, because a resolved manifest carries no build. The gate itself is
// registration's (to-be 38 WP2.4): once the runtime module is in the catalogue, this is refused.
//
// Three things must hold, and each alone is fine: declaring tools (a bundle does that); a container
// (a module's service may well be one); building on the runtime's image (a service written against
// the SDK may). All three is a container whose purpose is tools, which the runtime now serves.
func ToolContainerOnTheRuntime(m Manifest, against []string) string {
if len(m.Tools) == 0 {
return ""
}
container := false
for _, r := range m.Resources {
if fmt.Sprint(r["type"]) == "container" {
container = true
}
}
if !container {
return ""
}
onTheRuntime := false
if m.Build != nil {
for _, on := range m.Build.On {
if on.Module == RuntimeImageModule && on.Artifact == RuntimeImageArtifact {
onTheRuntime = true
}
}
}
for _, ref := range against {
path, kept := InArtifactStore(Recorded(ref))
if kept && strings.HasPrefix(path, RuntimeImageModule+"/"+RuntimeImageArtifact+"@") {
onTheRuntime = true
}
}
if !onTheRuntime {
return ""
}
return fmt.Sprintf(
"%s declares tools and a container built on %s's %s image — a container whose purpose is "+
"serving tools. The node's tool runtime (%s) serves every module's tools from its bundle "+
"now (novox/hq ADR 0175, to-be 38); declare the tools as a bundle and drop the container",
m.Module, RuntimeImageModule, RuntimeImageArtifact, RuntimeModule)
}
+88
View File
@@ -0,0 +1,88 @@
package catalogue
import (
"strings"
"testing"
)
// The packet-filter manifest as it was the day the runtime was decided (novox/hq ADR 0175): tools,
// served from a container built on the tool runtime's image, with NET_ADMIN so the container could
// reach the filter. The exact pattern to-be 38 WP4 moves it off, and the one the gate refuses.
const thePacketFilterAsItWas = `{
"module": "nftables",
"version": "1",
"capabilities": ["firewall", "container-runtime"],
"claims": [{"name": "node-packet-filter", "scope": "node", "serves": ["rules", "reload", "remove"]}],
"filtering": {"into": "/etc/nftables.conf"},
"resources": [
{"id": "mesh-state", "type": "directory", "mode": "0700", "place": "mesh"},
{"id": "package", "type": "package", "package": "nftables"},
{"id": "unit", "type": "file", "path": "/etc/systemd/system/mesh-filter.service",
"content": "[Unit]\nDescription=The mesh's packet filter\n[Service]\nType=oneshot\nExecStart=nft -f /etc/nftables.conf\n", "mode": "0644"},
{"id": "load", "type": "service", "unit": "mesh-filter.service", "state": "running", "boot": "enabled",
"restart-on": ["unit"], "reload-on": ["filtering"]},
{"id": "runtime", "type": "container", "name": "mesh-nftables", "network": "host",
"capabilities": ["NET_ADMIN"],
"volumes": ["${dir:mesh-state}/broker:/run/secrets/broker:ro", "/etc/nftables.conf:/etc/nftables.conf:ro"],
"env": {"MESH_BROKER_FILE": "/run/secrets/broker", "MESH_FILTER_FILE": "/etc/nftables.conf"},
"artifact": "runtime"}
],
"tools": ["firewall_rules"],
"own-secrets": {"broker": "${dir:mesh-state}/broker"},
"build": {
"on": [
{"arg": "BUILD_BASE", "module": "mesh-tools", "artifact": "build"},
{"arg": "RUNTIME_BASE", "module": "mesh-tools", "artifact": "runtime"}
],
"artifacts": [{"name": "runtime", "kind": "image", "from": "Dockerfile"}]
}
}`
func TestAToolContainerOnTheRuntimeImageIsNamedForWhatItIs(t *testing.T) {
m, err := ParseManifest([]byte(thePacketFilterAsItWas))
if err != nil {
t.Fatal(err)
}
// From the repository: the manifest says what it builds on.
why := ToolContainerOnTheRuntime(m, nil)
if why == "" {
t.Fatal("the packet filter's tool container was not recognised from its build")
}
for _, word := range []string{"nftables", "mesh-tools", "runtime", "ADR 0175", "bundle"} {
if !strings.Contains(why, word) {
t.Errorf("the refusal does not say %q: %s", word, why)
}
}
// Built: the manifest carries no build, and what it stood on says the same.
built, err := m.Resolve([]Built{{Name: "runtime", Kind: ArtifactImage,
Reference: ArtifactStoreScheme + "nftables/runtime@" + digest}})
if err != nil {
t.Fatal(err)
}
stoodOn := []string{"anchor.internal:5100/mesh-tools/build@" + digest, "anchor.internal:5100/mesh-tools/runtime@" + digest}
if ToolContainerOnTheRuntime(built, stoodOn) == "" {
t.Error("the packet filter's tool container was not recognised from what its build stood on")
}
if ToolContainerOnTheRuntime(built, nil) != "" {
t.Error("a built manifest with no record of its base was judged to be on the runtime")
}
// Each of the three alone is an ordinary module.
bundle := m
bundle.Resources = m.Resources[:len(m.Resources)-1]
if ToolContainerOnTheRuntime(bundle, nil) != "" {
t.Error("a module with tools and no container is the pattern the runtime serves, and was refused")
}
service := m
service.Tools = nil
if ToolContainerOnTheRuntime(service, nil) != "" {
t.Error("a service built against the SDK, declaring no tools, was refused")
}
elsewhere := m
elsewhere.Build = &Build{On: []BuildsOn{{Arg: "NODE_BASE", Image: "node@" + digest}},
Artifacts: m.Build.Artifacts}
if ToolContainerOnTheRuntime(elsewhere, nil) != "" {
t.Error("a tool container on a public base was refused as though it were on the runtime's")
}
}
+204
View File
@@ -0,0 +1,204 @@
package catalogue
import (
"fmt"
"strings"
"testing"
)
// The node's tool runtime (novox/hq ADR 0175, to-be 38): where the runtime module is assigned, a
// machine is sent every assigned module's tools bundle as an archive, and the runtime's own process
// loading them. Where it is not, the machine is sent exactly what it was sent before.
var bundleDigest = "sha256:" + strings.Repeat("b", 64)
// aToolsModule is a module whose tools come as a compiled bundle and nothing else — the shape every
// module takes once its tool container goes (to-be 38 WP4).
func aToolsModule(t *testing.T, name string, entrypoints ...string) Manifest {
t.Helper()
m := Manifest{Module: name, Version: "1", Tools: []string{"status"},
Build: &Build{Artifacts: []Artifact{
{Name: "tools", Kind: ArtifactBundle, Language: "typescript", Entrypoints: entrypoints},
}}}
resolved, err := m.Resolve([]Built{{Name: "tools", Kind: ArtifactBundle,
Reference: ArtifactStoreScheme + name + "/tools/blobs/" + bundleDigest, Digest: bundleDigest}})
if err != nil {
t.Fatal(err)
}
return resolved
}
// theRuntime is the runtime module as the catalogue holds it: its own bundle, run rather than
// loaded, and its broker secret to receive the node's credential in.
func theRuntime(t *testing.T) Manifest {
t.Helper()
m := Manifest{Module: RuntimeModule, Version: "1",
OwnSecrets: OwnSecrets{"broker": {Path: "/var/lib/mesh/" + RuntimeModule + "/broker"}},
Build: &Build{Artifacts: []Artifact{{Name: "runtime", Kind: ArtifactBundle, Language: "typescript",
Entrypoints: []string{"src/main.js"}}}}}
resolved, err := m.Resolve([]Built{{Name: "runtime", Kind: ArtifactBundle,
Reference: ArtifactStoreScheme + RuntimeModule + "/runtime/blobs/" + bundleDigest, Digest: bundleDigest}})
if err != nil {
t.Fatal(err)
}
return resolved
}
func TestABuildsBundlesAreCarriedOnTheResolvedManifest(t *testing.T) {
m := aToolsModule(t, "nftables", "tools/index.js")
if len(m.Bundles) != 1 {
t.Fatalf("the resolved manifest carries %d bundle(s), not the one the build made", len(m.Bundles))
}
b := m.Bundles[0]
if b.Name != "tools" || b.Digest != bundleDigest || b.Language != "typescript" ||
b.Source != ArtifactStoreScheme+"nftables/tools/blobs/"+bundleDigest ||
len(b.Entrypoints) != 1 || b.Entrypoints[0] != "tools/index.js" {
t.Errorf("the bundle is carried as %+v", b)
}
// A repository manifest may not write what the build derives.
raw := `{"module":"x","version":"1","build":{"artifacts":[{"name":"t","kind":"bundle","language":"typescript"}]},` +
`"bundles":[{"name":"t","source":"s","digest":"` + bundleDigest + `"}]}`
if _, err := ParseManifest([]byte(raw)); err == nil || !strings.Contains(err.Error(), "bundles") {
t.Errorf("a manifest stating its build's output by hand was accepted: %v", err)
}
}
func TestEveryToolsBundleIsDeliveredWhereTheRuntimeRuns(t *testing.T) {
store := Rendering{ArtifactStore: "anchor.internal:5101",
Needed: map[string]map[string]string{RuntimeModule: {"broker": "sealed-credential"}}}
nftables := aToolsModule(t, "nftables", "tools/index.js")
zsh := aToolsModule(t, "zsh", "tools/index.js", "tools/more.js")
t.Run("with the runtime, one archive per tools bundle", func(t *testing.T) {
r := Resolution{Node: "anchor", Modules: []Manifest{nftables, zsh, theRuntime(t)}}
out, err := r.Declaration(store)
if err != nil {
t.Fatal(err)
}
archive := fileNamed(out, "nftables."+BundleID("tools"))
if archive == nil {
t.Fatalf("nftables' tools bundle was not delivered: %v", ids(out))
}
if archive["type"] != "archive" || archive["digest"] != bundleDigest ||
archive["path"] != BundleRoot+"/nftables/tools" {
t.Errorf("delivered as %v", archive)
}
if archive["source"] != "http://anchor.internal:5101/v2/nftables/tools/blobs/"+bundleDigest {
t.Errorf("fetched from %v, not through the store as this network reaches it", archive["source"])
}
if fileNamed(out, "zsh."+BundleID("tools")) == nil {
t.Errorf("zsh's tools bundle was not delivered: %v", ids(out))
}
// The runtime's own bundle is run, not loaded: its process delivers it, not an archive.
if fileNamed(out, RuntimeModule+"."+BundleID("runtime")) != nil {
t.Error("the runtime's own bundle was delivered as an archive beside its process")
}
})
t.Run("without the runtime, nothing changes", func(t *testing.T) {
r := Resolution{Node: "anchor", Modules: []Manifest{nftables, zsh}}
out, err := r.Declaration(store)
if err != nil {
t.Fatal(err)
}
for _, id := range ids(out) {
if strings.Contains(id, BundleID("")) {
t.Errorf("%s was delivered to a machine running no runtime to load it", id)
}
}
})
}
func ids(out []map[string]any) []string {
var names []string
for _, r := range out {
names = append(names, r["id"].(string))
}
return names
}
// One process per machine runs the runtime from its own bundle, told what it serves and from where,
// where its credential is, and who the operator is — restarted when any of that changes.
func TestTheMachineRunsOneRuntimeLoadingEveryDeliveredBundle(t *testing.T) {
with := Rendering{ArtifactStore: "anchor.internal:5101",
Needed: map[string]map[string]string{RuntimeModule: {"broker": "sealed-credential"}}}
nftables := aToolsModule(t, "nftables", "tools/index.js")
// A bundle carrying a daemon beside its tools says which files the runtime loads.
showcase := Manifest{Module: "showcase", Version: "1", Tools: []string{"greet"},
Build: &Build{Artifacts: []Artifact{{Name: "code", Kind: ArtifactBundle, Language: "typescript",
Entrypoints: []string{"daemon/index.js", "tools/index.js"}, Loads: []string{"tools/index.js"}}}}}
showcase, err := showcase.Resolve([]Built{{Name: "code", Kind: ArtifactBundle,
Reference: ArtifactStoreScheme + "showcase/code/blobs/" + bundleDigest, Digest: bundleDigest}})
if err != nil {
t.Fatal(err)
}
r := Resolution{Node: "anchor", Account: "ops", Modules: []Manifest{nftables, showcase, theRuntime(t)}}
out, err := r.Declaration(with)
if err != nil {
t.Fatal(err)
}
process := fileNamed(out, RuntimeModule+"."+RuntimeProcessID())
if process == nil {
t.Fatalf("no runtime process was composed: %v", ids(out))
}
if process["type"] != "process" || process["name"] != RuntimeModule || process["digest"] != bundleDigest ||
process["source"] != "http://anchor.internal:5101/v2/"+RuntimeModule+"/runtime/blobs/"+bundleDigest {
t.Errorf("the runtime's process is %v", process)
}
if fmt.Sprint(process["run"]) != "[node src/main.js]" {
t.Errorf("the runtime is run as %v; its bundle's one entrypoint, by its language's interpreter", process["run"])
}
env := process["env"].(map[string]string)
if env[RuntimeToolModules] != "nftables="+BundleRoot+"/nftables/tools/tools/index.js,"+
"showcase="+BundleRoot+"/showcase/code/tools/index.js" {
t.Errorf("the runtime is told to serve %q: every loaded file, by module, and nothing a bundle runs", env[RuntimeToolModules])
}
if env[RuntimeBrokerFile] != "/var/lib/mesh/"+RuntimeModule+"/broker" {
t.Errorf("the runtime reads its credential at %q, not where the module's own secret is placed", env[RuntimeBrokerFile])
}
if env[RuntimeOperatorAccount] != "ops" || env[RuntimeOperatorHome] != "/home/ops" || process["user"] != "ops" {
t.Errorf("the operator is not handed to the runtime: %v as %v", env, process["user"])
}
restarts := fmt.Sprint(process["restart-on"])
for _, want := range []string{"nftables." + BundleID("tools"), "showcase." + BundleID("code"), RuntimeModule + "." + NeedID("broker")} {
if !strings.Contains(restarts, want) {
t.Errorf("the runtime is not restarted when %s changes: %s", want, restarts)
}
}
// After every bundle and the credential, so both exist before it starts.
names := ids(out)
if names[len(names)-1] != RuntimeModule+"."+RuntimeProcessID() {
t.Errorf("the runtime's process is not last: %v", names)
}
t.Run("a machine with no account runs it as root without the operator words", func(t *testing.T) {
out, err := Resolution{Node: "anchor", Modules: []Manifest{nftables, theRuntime(t)}}.Declaration(with)
if err != nil {
t.Fatal(err)
}
process := fileNamed(out, RuntimeModule+"."+RuntimeProcessID())
env := process["env"].(map[string]string)
if _, set := env[RuntimeOperatorAccount]; set {
t.Error("an operator account was named on a machine that has none")
}
if _, set := process["user"]; set {
t.Error("a user was set on a machine with no account")
}
})
t.Run("a runtime module built wrong is refused by name", func(t *testing.T) {
two := Manifest{Module: RuntimeModule, Version: "1", OwnSecrets: OwnSecrets{"broker": {Path: "/b"}},
Build: &Build{Artifacts: []Artifact{{Name: "runtime", Kind: ArtifactBundle, Language: "typescript",
Entrypoints: []string{"a.js", "b.js"}}}}}
resolved, err := two.Resolve([]Built{{Name: "runtime", Kind: ArtifactBundle,
Reference: ArtifactStoreScheme + "x/runtime/blobs/" + bundleDigest, Digest: bundleDigest}})
if err != nil {
t.Fatal(err)
}
_, err = Resolution{Node: "anchor", Modules: []Manifest{resolved}}.Declaration(with)
if err == nil || !strings.Contains(err.Error(), "entrypoint") {
t.Errorf("a runtime bundle with two entrypoints was composed: %v", err)
}
})
}
-10
View File
@@ -136,16 +136,6 @@ var ControllerVerbs = []Verb{
"node": "the machine that runs the module",
"module": "the module's name",
}, []string{"node", "module"})},
{Name: "token", Description: "Issue a one-time token for a machine to join with. Give the public half of the " +
"tunnel key the machine made (`nox-mesh-host key`): the machine is given its address and made a peer of " +
"the hub, and joins through the tunnel. The token is shown once, in the answer.",
Input: schema(map[string]string{
"node": "a machine the mesh already has a record for",
"new": "or the name of a machine to create the record for",
"overlay_key": "the public half of the machine's tunnel key",
"for": "how long it may be used, as a duration (default 1h)",
"adopted": "\"true\" when the machine is in use and joins adopted",
}, nil)},
{Name: "settings", Description: "Set what an assignment is configured with: a module's settings for the whole mesh, " +
"or for one machine. Replaces that layer whole — what it does not name, it no longer sets — and takes effect " +
"at the next push. With clear, removes the layer and the module is back to what its definition says.",
+3
View File
@@ -42,6 +42,9 @@ const (
BusModule = "module"
BusEnrolment = "enrolment"
BusPerson = "person"
// BusNodeTools is a machine's tool runtime (novox/hq ADR 0175): named like the module it
// stands for, recorded as what it is.
BusNodeTools = "node-tools"
)
// MintBusPassword makes a bus password and records its hash under a username, replacing whatever was
+31
View File
@@ -42,6 +42,11 @@ type Source struct {
// Seen is when the source was last looked at — by a build, by hand, or by the forge saying it
// moved. What a late report of an older move is judged against.
Seen time.Time
// Against is every artifact the build this manifest came from stood on, as recorded. Part of a
// module's provenance like the commit is, and what tells a built manifest's base when the manifest
// itself no longer carries its build (novox/hq to-be 38 WP2.4). Empty for a manifest handed over
// by hand, which carries its `build.on` itself.
Against []string
}
// Current reports whether what the mesh holds is what the source last had.
@@ -61,6 +66,22 @@ func (s Source) Current() bool {
// gains a requirement, a claim, a resource. What matters is that the change is visible the next
// time a node is resolved, which it is.
func (i *Inventory) RegisterModule(ctx context.Context, m catalogue.Manifest, from Source) error {
// **Once the node's tool runtime is in the catalogue, the pattern it retires is refused**
// (novox/hq ADR 0175, to-be 38 WP2.4): a module serving its tools from a container built on the
// runtime's image. Refused at registration, by name, because this is the mechanism that keeps
// the old pattern from returning by habit — a rebuild of an unmoved module stops here with the
// record that says why. Before the runtime exists the pattern is accepted as it always was.
if m.Module != catalogue.RuntimeModule {
if why := catalogue.ToolContainerOnTheRuntime(m, from.Against); why != "" {
runtime, err := i.hasModule(ctx, catalogue.RuntimeModule)
if err != nil {
return err
}
if runtime {
return fmt.Errorf("%s is not registered: %s", m.Module, why)
}
}
}
raw, err := json.Marshal(m)
if err != nil {
return err
@@ -89,6 +110,16 @@ func (i *Inventory) RegisterModule(ctx context.Context, m catalogue.Manifest, fr
return err
}
// hasModule is whether the catalogue holds a module of that name.
func (i *Inventory) hasModule(ctx context.Context, name string) (bool, error) {
var one int
err := i.store.Pool().QueryRow(ctx, `select 1 from module where name = $1`, name).Scan(&one)
if errors.Is(err, pgx.ErrNoRows) {
return false, nil
}
return err == nil, err
}
// SourceMoved records that a module's source has a newer commit than the mesh has built.
//
// This is the whole of noticing. Nothing here builds anything — it writes down that the two
+29
View File
@@ -685,3 +685,32 @@ func TestRegisteringWithoutProvenanceKeepsTheSeat(t *testing.T) {
t.Fatalf("a hand-registered manifest erased where the module comes from: %+v", got)
}
}
// Once the node's tool runtime is in the catalogue, a module serving its tools from a container
// built on the runtime's image is refused at registration, naming the record (novox/hq ADR 0175,
// to-be 38 WP2.4). Before, it is accepted as it always was — so a mesh converts in the order the
// design says and nothing is refused before there is anything to move to.
func TestAToolContainerIsRefusedOnceTheRuntimeIsRegistered(t *testing.T) {
inv := fresh(t)
filter := catalogue.Manifest{Module: "nftables", Version: "1", Tools: []string{"firewall_rules"},
Resources: []map[string]any{{"id": "runtime", "type": "container", "name": "mesh-nftables"}}}
stoodOn := []string{catalogue.ArtifactStoreScheme + "mesh-tools/runtime@sha256:" + strings.Repeat("d", 64)}
if err := inv.RegisterModule(t.Context(), filter, Source{Repository: "/r", Against: stoodOn}); err != nil {
t.Fatalf("before the runtime exists the old pattern is accepted: %v", err)
}
runtime := catalogue.Manifest{Module: catalogue.RuntimeModule, Version: "1"}
if err := inv.RegisterModule(t.Context(), runtime, Source{Repository: "/r"}); err != nil {
t.Fatal(err)
}
err := inv.RegisterModule(t.Context(), filter, Source{Repository: "/r", Against: stoodOn})
if err == nil || !strings.Contains(err.Error(), "ADR 0175") {
t.Fatalf("the old pattern was registered beside the runtime: %v", err)
}
// A module that moved its tools to a bundle registers.
moved := filter
moved.Resources = nil
if err := inv.RegisterModule(t.Context(), moved, Source{Repository: "/r", Against: stoodOn}); err != nil {
t.Fatalf("a module whose tools are a bundle was refused: %v", err)
}
}
@@ -1,7 +0,0 @@
-- A token issued for a tunnel key (novox/hq ADR 0169).
--
-- A machine that joins through the tunnel makes its key first, and the token is issued for it: the
-- hub is told the key before the token is shown. So enrolment must take that key and no other — a
-- different one is a machine the hub does not know, offering a tunnel that would never answer. Null
-- for a token issued without one, which enrols as before.
alter table enrolment_token add column overlay_key text;
-27
View File
@@ -356,33 +356,6 @@ func (i *Inventory) Claim(ctx context.Context, secret, by string, again bool) (N
return scanNode(i.store.Pool().QueryRow(ctx, `select `+nodeColumns+` from node where id = $1`, id))
}
// BindTokenToKey records the tunnel key a node's live token was issued for (novox/hq ADR 0169), so
// enrolment takes that key and no other. Refused when the node has no live token to bind: a key
// recorded against nothing would be a promise nothing keeps.
func (i *Inventory) BindTokenToKey(ctx context.Context, node, key string) error {
tag, err := i.store.Pool().Exec(ctx,
`update enrolment_token set overlay_key = $2
where node = $1 and redeemed is null and expires > now()`, node, key)
if err != nil {
return err
}
if tag.RowsAffected() == 0 {
return fmt.Errorf("no live token to issue for the tunnel key")
}
return nil
}
// TokenKey is the tunnel key a token was issued for, or empty when it was issued without one.
func (i *Inventory) TokenKey(ctx context.Context, secret string) (string, error) {
var key *string
err := i.store.Pool().QueryRow(ctx,
`select overlay_key from enrolment_token where secret = $1`, hashSecret(secret)).Scan(&key)
if err != nil || key == nil {
return "", err
}
return *key, nil
}
// Spend makes a claimed token used, only for the presenter holding the claim. The last write to the
// store in an enrolment, so a token is spent exactly when the node it enrolled is complete. Spent
// again by the same presenter is not an error: an answer lost after the first spend.
-1
View File
@@ -12,7 +12,6 @@ func ConnectNats(js *broker.JetStream, enroller Enroller, listener Listener) *Se
inbound: Nats(js),
bus: OverNATS{JS: js.Context(), Conn: js.Conn()},
js: js,
consumers: js,
enroller: enroller,
listener: listener,
log: newLog(),
-51
View File
@@ -1,51 +0,0 @@
package link
import (
"context"
"encoding/json"
"io"
"log"
"testing"
"time"
"github.com/novox/mesh-controller/internal/broker"
)
type ensured struct{ consumers []broker.Consumer }
func (e *ensured) EnsureConsumer(c broker.Consumer) error {
e.consumers = append(e.consumers, c)
return nil
}
type acceptsAs string
func (n acceptsAs) Enrol(context.Context, EnrolRequest) (EnrolReply, error) {
return EnrolReply{Accepted: true, Node: string(n)}, nil
}
type anEnrolment struct{ body []byte }
func (m anEnrolment) Kind() string { return "enrol" }
func (m anEnrolment) Body() []byte { return m.body }
func (m anEnrolment) Redelivered() bool { return false }
func (m anEnrolment) HeldFor() time.Duration { return 0 }
func (m anEnrolment) Answer(context.Context, []byte) error { return nil }
func (m anEnrolment) Took() error { return nil }
func (m anEnrolment) Hold(time.Duration) error { return nil }
func (m anEnrolment) Drop() error { return nil }
// **A node that enrols can hear its declarations at once** (novox/hq 04-ISSUES/146): its consumer is
// made as it enrols, not only when the control plane next starts — the first machine of a mesh
// enrols after the control plane is up, and heard nothing.
func TestAnEnrolledNodeIsGivenHowItHearsItsDeclarations(t *testing.T) {
made := &ensured{}
s := &Server{enroller: acceptsAs("anchor"), consumers: made, log: log.New(io.Discard, "", 0)}
body, _ := json.Marshal(EnrolRequest{Node: "anchor"})
s.enrolling(context.Background(), anEnrolment{body: body})
want := broker.NodeConsumer("anchor")
if len(made.consumers) != 1 || made.consumers[0].Name != want.Name || made.consumers[0].Stream != want.Stream {
t.Fatalf("the enrolled node was given %v, want its own declaration consumer %v", made.consumers, want)
}
}
-37
View File
@@ -1,37 +0,0 @@
package link_test
import (
"context"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/link"
)
// **A token issued for a tunnel key takes that key and no other** (novox/hq ADR 0169). The hub was
// sent the key before the token was shown, so another key is a machine it does not know.
func TestATokenIssuedForATunnelKeyTakesThatKeyAndNoOther(t *testing.T) {
inv, ident := aMeshReadyToEnrol(t)
ctx := context.Background()
secret, public := aTokenFor(t, inv, "joiner")
node, err := inv.NodeByName(ctx, "joiner")
if err != nil {
t.Fatal(err)
}
const issuedFor = "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA="
if err := inv.BindTokenToKey(ctx, node.ID, issuedFor); err != nil {
t.Fatal(err)
}
e := link.Enrolment{Inventory: inv, Identity: ident}
_, err = e.Enrol(ctx, link.EnrolRequest{Node: "joiner", Secret: secret, PublicKey: public,
OverlayKey: "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB="})
if err == nil || !strings.Contains(err.Error(), "issued for the tunnel key") {
t.Fatalf("a token issued for one key took another: %v", err)
}
if _, err := e.Enrol(ctx, link.EnrolRequest{Node: "joiner", Secret: secret, PublicKey: public,
OverlayKey: issuedFor}); err != nil {
t.Fatalf("the key the token was issued for was refused: %v", err)
}
}
-12
View File
@@ -86,18 +86,6 @@ func (e Enrolment) Enrol(ctx context.Context, request EnrolRequest) (reply Enrol
if err != nil {
return EnrolReply{}, err
}
// **A token issued for a tunnel key takes that key and no other** (novox/hq ADR 0169). The hub
// was told it before the token was shown; another key is a machine the hub does not know.
// Checked before anything is recorded, so a refusal changes nothing.
bound, err := e.Inventory.TokenKey(ctx, secret)
if err != nil {
return EnrolReply{}, err
}
if bound != "" && request.OverlayKey != bound {
return EnrolReply{}, fmt.Errorf("%s's token was issued for the tunnel key %s and the machine "+
"offered %q — the key it made with `nox-mesh-host key` is the one to issue for",
node.Name, bound, request.OverlayKey)
}
if _, err := e.Identity.RecordNodeKey(ctx, node.ID, public); err != nil {
return EnrolReply{}, fmt.Errorf("%s's key could not be recorded: %w", node.Name, err)
-21
View File
@@ -73,8 +73,6 @@ type Server struct {
inbound Inbound
bus Bus
js *broker.JetStream
// consumers makes a node's declaration consumer as it enrols; the bus connection, or a stand-in.
consumers interface{ EnsureConsumer(broker.Consumer) error }
enroller Enroller
listener Listener
@@ -385,7 +383,6 @@ func (s *Server) enrolling(ctx context.Context, m Control) {
default:
reply = accepted
s.log.Printf("enrolled %s", accepted.Node)
s.hearsItsDeclarations(accepted.Node)
}
}
@@ -405,24 +402,6 @@ func (s *Server) enrolling(ctx context.Context, m Control) {
_ = m.Took()
}
// hearsItsDeclarations makes the consumer a node reads its declarations through, as it enrols and
// before it is answered.
//
// **Created at enrolment, as the consumer's own doc has always said** (novox/hq 04-ISSUES/146). It
// was asserted only when the control plane started, so the first machine of a mesh — which enrols
// after the control plane is already up — joined and then heard nothing, its host retrying "consumer
// not found" for ever. Failing here is said and does not unspend the token: the next start of the
// control plane asserts it again.
func (s *Server) hearsItsDeclarations(node string) {
if s.consumers == nil {
return
}
if err := s.consumers.EnsureConsumer(broker.NodeConsumer(node)); err != nil {
s.log.Printf("%s enrolled, and how it hears its declarations could not be made — it will hear "+
"nothing until the control plane next starts: %v", node, err)
}
}
// wasBuilt keeps what a builder said, whichever way it went.
//
// This is for results nobody was waiting for. A build asked for with `build` is answered directly
-33
View File
@@ -55,26 +55,6 @@ type Token struct {
// that it speaks the firewall found on the machine, because an adopted node keeps that firewall
// in force. Absent for a converged node, so a converged token is byte for byte what it was.
Adopted bool `json:"adopted,omitempty"`
// Tunnel is the one peer a joining machine needs, when the token was issued for its tunnel key
// (novox/hq ADR 0169). The machine brings its tunnel up from this alone and reaches the bus over
// it, at an address on the private network — so the bus is never open to the internet. Absent
// on a token issued without a key, which then reads byte for byte as before.
Tunnel *Tunnel `json:"tunnel,omitempty"`
}
// Tunnel is the joining machine's side of its first tunnel: its own address and the hub to reach.
type Tunnel struct {
// Key is the public half of the key the machine made itself, which this token was issued for.
// The private half never left the machine (novox/hq ADR 0004).
Key string `json:"key"`
// Address is the machine's own address on the private network, with its prefix.
Address string `json:"address"`
// Range is the private network, routed through the hub until the machine is told more.
Range string `json:"range"`
// HubKey and HubEndpoint are the hub's tunnel key and where it is dialled.
HubKey string `json:"hub_key"`
HubEndpoint string `json:"hub_endpoint"`
}
// Missing names the parts that are not filled in.
@@ -103,19 +83,6 @@ func (t Token) Missing() []string {
if strings.TrimSpace(t.Secret) == "" {
missing = append(missing, "the one-time secret — nothing to present")
}
if t.Tunnel != nil {
for _, part := range []struct{ value, says string }{
{t.Tunnel.Key, "the machine's own tunnel key — the hub would not know it"},
{t.Tunnel.Address, "the machine's address on the private network"},
{t.Tunnel.Range, "the private network's range — nothing to route through the hub"},
{t.Tunnel.HubKey, "the hub's tunnel key — nothing to dial"},
{t.Tunnel.HubEndpoint, "where the hub's tunnel is dialled"},
} {
if strings.TrimSpace(part.value) == "" {
missing = append(missing, part.says)
}
}
}
return missing
}
-35
View File
@@ -172,38 +172,3 @@ func TestATokenWithNoNameIsRefused(t *testing.T) {
t.Fatalf("the refusal does not say what is missing: %v", without.Missing())
}
}
// A token issued for a tunnel key carries the one peer a joining machine needs (novox/hq ADR 0169),
// and says which part is missing rather than producing a tunnel that never answers.
func TestATokenThroughTheTunnelCarriesThePeerOrSaysWhatIsMissing(t *testing.T) {
whole := Token{Node: "n", Broker: "10.42.0.1:4222", Fingerprint: "sha256:x", Signer: make([]byte, 32), Secret: "s",
Tunnel: &Tunnel{Key: "k", Address: "10.42.0.9/32", Range: "10.42.0.0/16", HubKey: "h", HubEndpoint: "198.51.100.1:51820"}}
if !whole.Complete() {
t.Fatalf("a whole token through the tunnel reads as missing %v", whole.Missing())
}
encoded, err := whole.Encode()
if err != nil {
t.Fatal(err)
}
back, err := Decode(encoded)
if err != nil {
t.Fatal(err)
}
if back.Tunnel == nil || *back.Tunnel != *whole.Tunnel {
t.Fatalf("the tunnel did not survive the round trip: %+v", back.Tunnel)
}
part := whole
part.Tunnel = &Tunnel{Key: "k", Address: "10.42.0.9/32"}
if len(part.Missing()) != 3 {
t.Errorf("a tunnel without the hub and the range should name three missing parts: %v", part.Missing())
}
// And a token issued without a key carries no tunnel at all, byte for byte as before.
plain := whole
plain.Tunnel = nil
raw, _ := plain.Encode()
if strings.Contains(raw, "tunnel") {
t.Error("a token without a key mentions a tunnel")
}
}