Compare commits

...
Author SHA1 Message Date
jschoubben cfb6554e69 A node is given how it hears its declarations as it enrols
The node's declaration consumer was asserted only when the control plane
started, so the first machine of a mesh, enrolling after the control
plane was up, joined and then heard nothing: its host retried 'consumer
not found' for ever (novox/hq issue 146).
2026-10-02 16:55:50 +02:00
jschoubben 73790bd967 A token can be issued for a machine's tunnel key, and it joins through the tunnel
token issue --overlay-key records the key the machine made, binds the
token to it, gives the machine its address and makes it a peer of the
hub, pushing the hub before the token is shown. The token carries the
hub's tunnel and the bus at its address on the private network, and
enrolment refuses any other key (novox/hq ADR 0169). Tokens without a
key enrol as before until the bus is closed. Also a token verb.
2026-10-02 16:55:27 +02:00
mesh-admin 37b8edeec6 Merge pull request 'node show says a found firewall that was uninstalled is removed (hq ADR 0175)' (#217) from feat/the-found-front-end-is-uninstalled into main 2026-10-02 14:29:02 +00:00
jschoubben 424406b2d6 node show says a found firewall that was uninstalled is removed (hq ADR 0175) 2026-10-02 16:27:39 +02:00
mesh-admin 90455c61f6 Merge pull request 'The example images build from the Go the manifest pins' (#216) from jschoubben/example-images-go-base into main 2026-10-02 13:56:22 +00:00
jschoubben 1f6793cf0c The example images build from the Go the manifest pins
Four example Dockerfiles named golang:1.25 while go.mod requires 1.26;
the control plane's image takes GO_BASE from the manifest and these did
not, so a build that could not fetch a newer toolchain failed at go mod
download (found running the lab through the mesh).
2026-10-02 15:52:25 +02:00
mesh-admin 6ef522fbda Merge pull request 'Cite hq ADR 0170, not 0169: the firewall seat's record was renumbered' (#215) from fix/adr-0170-cited into main 2026-10-02 12:53:13 +00:00
jschoubben 46f65c12a0 Cite hq ADR 0170, not 0169: the firewall seat's record was renumbered after a collision on hq main 2026-10-02 14:52:24 +02:00
mesh-admin 8f7c02d77a Merge pull request 'The virtualisation capability grants the lab its daemon's socket (hq ADR 0172)' (#214) from jschoubben/the-lab-is-a-module into main 2026-10-02 12:47:45 +00:00
jschoubben a637df01ea The virtualisation capability grants the lab its daemon's socket
The lab raises machines on the virtualisation daemon, and a module may
mount a machine's socket only through the capability that grants it
(novox/hq ADR 0172). Also brings the resolver's tests to the setting
dnsmasq's listen addresses now come from, and to a module left out
rather than refused.
2026-10-02 14:46:17 +02:00
mesh-admin 252eb786a7 Merge pull request 'A filter module's own filter file counts as declared for a mount (hq ADR 0169)' (#213) from feat/the-firewall-seat-serves-its-verbs into main 2026-10-02 12:05:09 +00:00
jschoubben 9d13b0593b A filter module's own filter file counts as declared for a mount (hq ADR 0169)
The nftables module's runtime mounts the file filtering.into names, to reload
the mesh's table; the mount check knew every other declaration of a path and
not this one, and the module's first build was refused for it.
2026-10-02 14:04:35 +02:00
mesh-admin 30d548a762 Merge pull request 'The node-packet-filter seat serves rules, reload and remove (hq ADR 0169)' (#212) from feat/the-firewall-seat-serves-its-verbs into main 2026-10-02 11:28:57 +00:00
jschoubben 550e4c6acb The node-packet-filter seat serves rules, reload and remove (hq ADR 0169)
What a person asks a machine's packet filter whatever filter answers: the
rules as enforced, reload the mesh's own, remove one rule set the mesh did
not write — named as the host reports it under ADR 0168. Every holder serves
all three; a running mesh widens its seat row at the next controller start.
2026-10-02 13:27:04 +02:00
mesh-admin 1587fd97f9 Merge pull request 'The mesh says what filters a converged machine: filters kept per node, shown, named by status, previewed with fates (hq ADR 0168)' (#211) from feat/one-thing-filters-a-converged-machine into main 2026-10-02 10:03:01 +00:00
24 changed files with 429 additions and 22 deletions
+4 -4
View File
@@ -59,7 +59,7 @@ PROVISIONER_IMAGE ?= mesh-provision-postgres:$(VERSION)
PROVISIONER_DEV_TAG ?= mesh-provision-postgres:development
provisioner-image:
docker build -f examples/postgres-provisioner/Dockerfile \
docker build --build-arg GO_BASE=$(GO_BASE) -f examples/postgres-provisioner/Dockerfile \
-t $(PROVISIONER_IMAGE) -t $(PROVISIONER_DEV_TAG) .
@echo
@docker image inspect $(PROVISIONER_IMAGE) --format 'built {{.RepoTags}} {{.Size}} bytes'
@@ -70,7 +70,7 @@ OBJECTSTORE_IMAGE ?= mesh-provision-objectstore:$(VERSION)
OBJECTSTORE_DEV_TAG ?= mesh-provision-objectstore:development
objectstore-image:
docker build -f examples/objectstore-provisioner/Dockerfile \
docker build --build-arg GO_BASE=$(GO_BASE) -f examples/objectstore-provisioner/Dockerfile \
-t $(OBJECTSTORE_IMAGE) -t $(OBJECTSTORE_DEV_TAG) .
@echo
@docker image inspect $(OBJECTSTORE_IMAGE) --format 'built {{.RepoTags}} {{.Size}} bytes'
@@ -81,7 +81,7 @@ REDIS_PROVISIONER_IMAGE ?= mesh-provision-redis:$(VERSION)
REDIS_PROVISIONER_DEV_TAG ?= mesh-provision-redis:development
redis-provisioner-image:
docker build -f examples/redis-provisioner/Dockerfile \
docker build --build-arg GO_BASE=$(GO_BASE) -f examples/redis-provisioner/Dockerfile \
-t $(REDIS_PROVISIONER_IMAGE) -t $(REDIS_PROVISIONER_DEV_TAG) .
@echo
@docker image inspect $(REDIS_PROVISIONER_IMAGE) --format 'built {{.RepoTags}} {{.Size}} bytes'
@@ -91,7 +91,7 @@ PROXY_IMAGE ?= mesh-route-proxy:$(VERSION)
PROXY_DEV_TAG ?= mesh-route-proxy:development
proxy-image:
docker build -f examples/route-proxy/Dockerfile -t $(PROXY_IMAGE) -t $(PROXY_DEV_TAG) .
docker build --build-arg GO_BASE=$(GO_BASE) -f examples/route-proxy/Dockerfile -t $(PROXY_IMAGE) -t $(PROXY_DEV_TAG) .
@echo
@docker image inspect $(PROXY_IMAGE) --format 'built {{.RepoTags}} {{.Size}} bytes'
+2
View File
@@ -94,6 +94,8 @@ func showFiltering(f inventory.Filtering, adopted bool) {
switch {
case fw.Active:
fmt.Printf(" found firewall %s is ACTIVE on this converged machine; the next apply retires it again\n", fw.Kind)
case fw.RetiredBy == "removed":
fmt.Printf(" found firewall %s, removed: the mesh's filter is what filters this machine (novox/hq ADR 0175)\n", fw.Kind)
case fw.RetiredBy == inventory.FilterMesh || fw.RetiredBy == "mesh":
fmt.Printf(" found firewall %s, retired by the mesh; its configuration stays on disk\n", fw.Kind)
case fw.RetiredBy != "":
+14 -1
View File
@@ -232,9 +232,22 @@ func network(ctx context.Context, inv *inventory.Inventory, on map[string]bool,
if err != nil {
return nil, err
}
// **A machine joining is on the network before it is anything else** (novox/hq ADR 0169). Its
// token was issued for its tunnel key and gave it an address, so while that token can still be
// used the hub carries it as a peer: it brings its tunnel up from the token and enrols over it.
// When the token is spent the machine is on the network by what it runs, as every other is; when
// it expires unused, the peer goes with it at the hub's next composition.
joining, err := inv.NodesWithALiveToken(ctx)
if err != nil {
return nil, err
}
isJoining := map[string]bool{}
for _, name := range joining {
isJoining[name] = true
}
nodes := make([]overlay.Node, 0, len(places))
for _, p := range places {
if !on[p.Name] {
if !on[p.Name] && !(isJoining[p.Name] && p.Key != "" && p.Address != "") {
continue
}
n := overlay.Node{
+72
View File
@@ -2,9 +2,11 @@ package main
import (
"context"
"encoding/base64"
"errors"
"flag"
"fmt"
"net"
"strings"
"time"
@@ -230,6 +232,8 @@ func tokenCommand(ctx context.Context, args []string) error {
validFor := set.Duration("for", time.Hour, "how long the token may be used")
adopted := set.Bool("adopted", false,
"the machine joining is in use: it is adopted, and keeps what is found on it")
tunnelKey := set.String("overlay-key", "",
"the public half of the tunnel key the machine made (`nox-mesh-host key`): it joins through the tunnel")
if err := set.Parse(args[1:]); err != nil {
return err
}
@@ -283,6 +287,14 @@ func tokenCommand(ctx context.Context, args []string) error {
default:
return err
}
// **Through the tunnel** (novox/hq ADR 0169): the machine's key recorded, its address given, the
// hub sent it as a peer — all before the token is shown, so the tunnel answers the first time the
// machine knocks. The bus is then reached at its address on the private network.
if *tunnelKey != "" {
if made.Tunnel, made.Broker, err = throughTheTunnel(ctx, open, issued.Node, *tunnelKey, made.Broker); err != nil {
return err
}
}
encoded, err := made.Encode()
if err != nil {
return err
@@ -307,6 +319,66 @@ func tokenCommand(ctx context.Context, args []string) error {
return nil
}
// throughTheTunnel makes a machine a peer of the hub for its token, and says what the token carries
// for it: its first tunnel, and the bus at its address on the private network (novox/hq ADR 0169).
//
// The hub is pushed here, before the token is shown. A token shown before the hub knew the key is a
// tunnel that does not answer, and a machine that cannot tell that from a bus that is down.
func throughTheTunnel(ctx context.Context, open *stores, node inventory.Node, key, busAt string) (
*token.Tunnel, string, error) {
inv := open.inventory
key = strings.TrimSpace(key)
if raw, err := base64.StdEncoding.DecodeString(key); err != nil || len(raw) != 32 {
return nil, "", fmt.Errorf("%q is not a tunnel public key: it is 32 bytes in base64, as "+
"`nox-mesh-host key` prints it", key)
}
// The bus on the private network is the hub's address at the bus's own port, so the port must be
// known before anything is recorded.
_, port, err := net.SplitHostPort(busAt)
if err != nil || port == "" {
return nil, "", fmt.Errorf("the bus's address %q has no port to reach it on", busAt)
}
places, err := inv.Overlays(ctx)
if err != nil {
return nil, "", err
}
var hub *inventory.Overlay
for i := range places {
if places[i].Hub {
hub = &places[i]
}
}
if hub == nil || hub.Key == "" || hub.Endpoint == "" || hub.Address == "" {
return nil, "", errors.New("this mesh has no hub with a key, an address and an endpoint to " +
"dial, so there is no tunnel to join through: place one (`overlay place <node> --hub " +
"--endpoint <host>:<port>`), or issue the token without --overlay-key")
}
if err := inv.RecordOverlayKey(ctx, node.ID, key); err != nil {
return nil, "", err
}
if err := inv.BindTokenToKey(ctx, node.ID, key); err != nil {
return nil, "", err
}
cidr, err := overlayRange(ctx, inv)
if err != nil {
return nil, "", err
}
address, err := inv.AssignAddress(ctx, node.ID, cidr)
if err != nil {
return nil, "", err
}
if err := sendTo(ctx, open, []string{hub.Name}); err != nil {
return nil, "", fmt.Errorf("%s was made a peer of the hub, and the hub could not be sent "+
"it, so the tunnel would not answer — the token is not shown; issue it again once %s "+
"can be pushed: %w", node.Name, hub.Name, err)
}
// An address, not a name — nothing resolves before the machine has joined (novox/hq ADR 0004).
return &token.Tunnel{
Key: key, Address: address + "/32", Range: cidr,
HubKey: hub.Key, HubEndpoint: hub.Endpoint,
}, net.JoinHostPort(hub.Address, port), nil
}
// issueFor is the inventory's half of issuing a token: the record, made when it is new, adopted
// when the operator says so, and the one-time secret for it. The node in what it returns carries
// its mode, which is what the token says.
+20
View File
@@ -129,6 +129,26 @@ func argvFor(verb string, args map[string]any) ([]string, error) {
// Half of either shape: the command says its usage, which names both shapes, and that is
// the answer the caller needs.
return []string{"rotate"}, nil
case "token":
// `token issue` at a shell (novox/hq ADR 0169). Exactly one of node or new; the command
// refuses both or neither in its own words.
argv := []string{"token", "issue"}
if n := str("node"); n != "" {
argv = append(argv, "--node", n)
}
if n := str("new"); n != "" {
argv = append(argv, "--new", n)
}
if k := str("overlay_key"); k != "" {
argv = append(argv, "--overlay-key", k)
}
if d := str("for"); d != "" {
argv = append(argv, "--for", d)
}
if str("adopted") == "true" {
argv = append(argv, "--adopted")
}
return argv, nil
case "settings":
// `settings set|clear` at a shell (novox/hq issue 198). The values travel as an argument
// because a tool has no file to hand the command; the command reads either.
+8
View File
@@ -73,6 +73,14 @@ func TestRotateTakesAProvisionOrAnOwnSecret(t *testing.T) {
}
}
// `token` is `token issue` at a shell, with the machine's tunnel key (novox/hq ADR 0169).
func TestTokenIssuesForAMachineAndItsTunnelKey(t *testing.T) {
argv, err := argvFor("token", map[string]any{"new": "laptop", "overlay_key": "k", "for": "2h"})
if err != nil || strings.Join(argv, " ") != "token issue --new laptop --overlay-key k --for 2h" {
t.Fatalf("token: %v %v", argv, err)
}
}
// `settings` is `settings set|clear` at a shell, with the values passed inline (novox/hq issue 198).
func TestSettingsSetsOrClearsALayer(t *testing.T) {
argv, err := argvFor("settings", map[string]any{"module": "dnsmasq", "values": `{"a":1}`, "node": "ace"})
+4 -1
View File
@@ -10,7 +10,10 @@
# The client is copied from the vendor's own image rather than installed from a distribution:
# `apk add mc` on Alpine installs Midnight Commander, which is a different program with the same
# name, and the failure would be a provisioner that starts cleanly and cannot do anything.
FROM golang:1.25-alpine AS build
# The Go it builds with is the one the manifest pins (build.on GO_BASE), passed by the Makefile and the
# build machine alike; the default only serves a hand build, and matches go.mod.
ARG GO_BASE=golang:1.26-alpine
FROM ${GO_BASE} AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
+4 -1
View File
@@ -3,7 +3,10 @@
# Built here so a machine can be given it by the mesh rather than by somebody putting a binary on
# it. Static and FROM scratch for the same reason the control plane's image is: it is fetched by
# digest and run on a machine, and everything in it is something a person would have to audit.
FROM golang:1.25-alpine AS build
# The Go it builds with is the one the manifest pins (build.on GO_BASE), passed by the Makefile and the
# build machine alike; the default only serves a hand build, and matches go.mod.
ARG GO_BASE=golang:1.26-alpine
FROM ${GO_BASE} AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
+4 -1
View File
@@ -2,7 +2,10 @@
#
# FROM scratch, like the postgres one and unlike the bucket one: it speaks the store's own wire
# protocol directly and needs no client in the image.
FROM golang:1.25-alpine AS build
# The Go it builds with is the one the manifest pins (build.on GO_BASE), passed by the Makefile and the
# build machine alike; the default only serves a hand build, and matches go.mod.
ARG GO_BASE=golang:1.26-alpine
FROM ${GO_BASE} AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
+4 -1
View File
@@ -2,7 +2,10 @@
#
# Static and FROM scratch like the control plane's image, and for the same reason: it is fetched
# by digest and run on a machine, so everything in it is something a person would have to audit.
FROM golang:1.25-alpine AS build
# The Go it builds with is the one the manifest pins (build.on GO_BASE), passed by the Makefile and the
# build machine alike; the default only serves a hand build, and matches go.mod.
ARG GO_BASE=golang:1.26-alpine
FROM ${GO_BASE} AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
+8
View File
@@ -1769,6 +1769,8 @@ func (m Manifest) MachineSide(port int) (at int, mayAssign bool) {
var facilitiesOf = map[string][]string{
// Both spellings: /var/run is a link to /run on every machine the mesh runs on.
"container-runtime": {"/var/run/docker.sock", "/run/docker.sock"},
// The virtualisation daemon's socket, for the lab (novox/hq ADR 0172): it raises machines there.
"virtualisation": {"/var/lib/incus/unix.socket"},
}
// undeclaredMounts is every bind-mount source no declaration covers — see the check above.
@@ -1810,6 +1812,12 @@ func (m Manifest) undeclaredMounts() []string {
claim(p)
}
}
// The file a filter module's rule set is written to is declared by `filtering.into`: the mesh
// writes it, the module loads it, and the module's runtime may read it back to reload the
// mesh's own table (novox/hq ADR 0170).
if m.Filtering != nil {
claim(m.Filtering.Into)
}
// Under a declared directory is declared: a module that says where its data lives has said so
// for what it puts inside.
covers := func(path string) bool {
+10
View File
@@ -98,3 +98,13 @@ func TestAMountOfABoundFactIsAccepted(t *testing.T) {
t.Fatalf("a mount of the file the mesh writes a binding to was refused: %v", err)
}
}
// The file a filter module's rule set is written to is declared by `filtering.into` (novox/hq ADR
// 0169): the module's runtime mounts it to reload the mesh's own table, and nothing else declares it.
func TestAMountOfTheFilterFileIsDeclaredByFilteringInto(t *testing.T) {
_, err := ParseManifest([]byte(`{"module":"nftables","filtering":{"into":"/etc/nftables.conf"},` +
`"resources":[` + strings.Replace(aContainerMounting, "%s", "/etc/nftables.conf", 1) + `]}`))
if err != nil {
t.Fatalf("a filter module mounting its own filter file was refused: %v", err)
}
}
+16 -6
View File
@@ -48,7 +48,9 @@ func TestTheResolverForwardsToFixedUpstreamsAndNeverReadsResolvConf(t *testing.T
}
for _, want := range []string{
"\nno-resolv\n", "\nserver=1.1.1.1\n", "\nserver=8.8.8.8\n",
"\nlisten-address=127.0.0.1\n", "\nlisten-address=${machine:address}\n", "\nbind-dynamic\n",
// Loopback is the mesh-wide setting's default; a machine answering its own LAN adds its
// address there (novox/hq issue 198).
"\nlisten-address=${setting:listen-addresses}\n", "\nlisten-address=${machine:address}\n", "\nbind-dynamic\n",
"\ndomain-needed\n", "\nbogus-priv\n",
"\nconf-file=" + m.Facts["node-zones"].Path + "\n",
} {
@@ -113,7 +115,8 @@ func TestTheResolverAndWhatAsksItComposeOnOneMachine(t *testing.T) {
// issue 111) — the resolver's zones read only the second, and in this scenario the two
// happen to be the same map, since nothing routed is part of it.
Names: twoMachines, Machines: twoMachines, Suffix: "internal",
Needed: map[string]map[string]string{"dnsmasq": {"broker": "sealed"}},
Needed: map[string]map[string]string{"dnsmasq": {"broker": "sealed"}},
Settings: SettingsBy{"dnsmasq": {{From: "the mesh", Values: map[string]any{"listen-addresses": "127.0.0.1"}}}},
})
if err != nil {
t.Fatal(err)
@@ -207,9 +210,16 @@ func TestTheResolverOnAMachineOffTheNetworkIsRefused(t *testing.T) {
if err != nil {
t.Fatal(err)
}
_, err = got.Declaration(Rendering{Names: twoMachines, Suffix: "internal",
Needed: map[string]map[string]string{"dnsmasq": {"broker": "sealed"}}})
if err == nil || !strings.Contains(err.Error(), "${machine:address}") {
t.Fatalf("a machine off the network was composed a resolver, or refused for another reason: %v", err)
// Left out of the declaration and said, rather than composed listening nowhere: a module that
// cannot compose on a machine is kept as it is there, with the reason (hq ADR 0163).
composed, err := got.Compose(Rendering{Names: twoMachines, Suffix: "internal",
Needed: map[string]map[string]string{"dnsmasq": {"broker": "sealed"}},
Settings: SettingsBy{"dnsmasq": {{From: "the mesh", Values: map[string]any{"listen-addresses": "127.0.0.1"}}}}})
if err == nil && !strings.Contains(composed.LeftOut["dnsmasq"], "${machine:address}") {
t.Fatalf("a machine off the network was composed a resolver, or left out for another reason: %v",
composed.LeftOut)
}
if err != nil && !strings.Contains(err.Error(), "${machine:address}") {
t.Fatalf("a machine off the network was refused for another reason: %v", err)
}
}
+19 -1
View File
@@ -100,7 +100,25 @@ var defaultSeats = []Seat{
Accepts: []string{"build"}, Emits: []string{"started", "built", "log.*"}, Decision: "novox/hq ADR 0121"},
{Name: "node-dns-resolver", Scope: ScopeNode, Decision: "novox/hq ADR 0121"},
{Name: "node-intrusion-prevention", Scope: ScopeNode, Decision: "novox/hq ADR 0121"},
{Name: "node-packet-filter", Scope: ScopeNode, Decision: "novox/hq ADR 0121"},
// The packet filter's verbs (novox/hq ADR 0170): what a person asks a machine's filter whatever
// filter answers — the rules as enforced, reload the mesh's own, remove one thing the mesh did
// not write. Every holder serves all three; what differs by filter is the holder's own tools.
{Name: "node-packet-filter", Scope: ScopeNode, Decision: "novox/hq ADR 0121",
Serves: []Verb{
{Name: "rules", Description: "The packet filter as this machine enforces it now: the nftables " +
"ruleset and, where the tool exists, the legacy filter's listings. Narrowed to one table or " +
"chain when asked.",
Input: schema(map[string]string{"table": "one nftables table, as `family name` (optional)",
"chain": "one chain of that table (optional)"}, nil)},
{Name: "reload", Description: "Load the mesh's own filter again from the file the mesh writes, " +
"and answer with the mesh's table as loaded.",
Input: schema(map[string]string{}, nil)},
{Name: "remove", Description: "Remove one rule set the mesh did not write, named exactly as the " +
"host reports it (novox/hq ADR 0168) — `chain X (iptables-legacy)` or `table ip6 filter, chain " +
"DOCKER-USER`. Refuses the mesh's tables, the runtime's own chains, a built-in chain and an " +
"active found firewall's chains. An operator's act, by name, never a flush.",
Input: schema(map[string]string{"where": "the rule set, as `node show` lists it"}, []string{"where"})},
}},
// Deferred (novox/hq ADR 0121): renaming to mesh-private-network is a scope + server/client
// model change, not a rename, so it stays until that is built.
{Name: "the-private-network", Scope: ScopeNode, Decision: "novox/hq ADR 0110"},
+10
View File
@@ -136,6 +136,16 @@ var ControllerVerbs = []Verb{
"node": "the machine that runs the module",
"module": "the module's name",
}, []string{"node", "module"})},
{Name: "token", Description: "Issue a one-time token for a machine to join with. Give the public half of the " +
"tunnel key the machine made (`nox-mesh-host key`): the machine is given its address and made a peer of " +
"the hub, and joins through the tunnel. The token is shown once, in the answer.",
Input: schema(map[string]string{
"node": "a machine the mesh already has a record for",
"new": "or the name of a machine to create the record for",
"overlay_key": "the public half of the machine's tunnel key",
"for": "how long it may be used, as a duration (default 1h)",
"adopted": "\"true\" when the machine is in use and joins adopted",
}, nil)},
{Name: "settings", Description: "Set what an assignment is configured with: a module's settings for the whole mesh, " +
"or for one machine. Replaces that layer whole — what it does not name, it no longer sets — and takes effect " +
"at the next push. With clear, removes the layer and the module is back to what its definition says.",
@@ -0,0 +1,7 @@
-- A token issued for a tunnel key (novox/hq ADR 0169).
--
-- A machine that joins through the tunnel makes its key first, and the token is issued for it: the
-- hub is told the key before the token is shown. So enrolment must take that key and no other — a
-- different one is a machine the hub does not know, offering a tunnel that would never answer. Null
-- for a token issued without one, which enrols as before.
alter table enrolment_token add column overlay_key text;
+27
View File
@@ -356,6 +356,33 @@ func (i *Inventory) Claim(ctx context.Context, secret, by string, again bool) (N
return scanNode(i.store.Pool().QueryRow(ctx, `select `+nodeColumns+` from node where id = $1`, id))
}
// BindTokenToKey records the tunnel key a node's live token was issued for (novox/hq ADR 0169), so
// enrolment takes that key and no other. Refused when the node has no live token to bind: a key
// recorded against nothing would be a promise nothing keeps.
func (i *Inventory) BindTokenToKey(ctx context.Context, node, key string) error {
tag, err := i.store.Pool().Exec(ctx,
`update enrolment_token set overlay_key = $2
where node = $1 and redeemed is null and expires > now()`, node, key)
if err != nil {
return err
}
if tag.RowsAffected() == 0 {
return fmt.Errorf("no live token to issue for the tunnel key")
}
return nil
}
// TokenKey is the tunnel key a token was issued for, or empty when it was issued without one.
func (i *Inventory) TokenKey(ctx context.Context, secret string) (string, error) {
var key *string
err := i.store.Pool().QueryRow(ctx,
`select overlay_key from enrolment_token where secret = $1`, hashSecret(secret)).Scan(&key)
if err != nil || key == nil {
return "", err
}
return *key, nil
}
// Spend makes a claimed token used, only for the presenter holding the claim. The last write to the
// store in an enrolment, so a token is spent exactly when the node it enrolled is complete. Spent
// again by the same presenter is not an error: an answer lost after the first spend.
+7 -6
View File
@@ -9,12 +9,13 @@ import (
// the streams and the controller's consumers are asserted by Raise, before anything is served.
func ConnectNats(js *broker.JetStream, enroller Enroller, listener Listener) *Server {
return &Server{
inbound: Nats(js),
bus: OverNATS{JS: js.Context(), Conn: js.Conn()},
js: js,
enroller: enroller,
listener: listener,
log: newLog(),
inbound: Nats(js),
bus: OverNATS{JS: js.Context(), Conn: js.Conn()},
js: js,
consumers: js,
enroller: enroller,
listener: listener,
log: newLog(),
}
}
+51
View File
@@ -0,0 +1,51 @@
package link
import (
"context"
"encoding/json"
"io"
"log"
"testing"
"time"
"github.com/novox/mesh-controller/internal/broker"
)
type ensured struct{ consumers []broker.Consumer }
func (e *ensured) EnsureConsumer(c broker.Consumer) error {
e.consumers = append(e.consumers, c)
return nil
}
type acceptsAs string
func (n acceptsAs) Enrol(context.Context, EnrolRequest) (EnrolReply, error) {
return EnrolReply{Accepted: true, Node: string(n)}, nil
}
type anEnrolment struct{ body []byte }
func (m anEnrolment) Kind() string { return "enrol" }
func (m anEnrolment) Body() []byte { return m.body }
func (m anEnrolment) Redelivered() bool { return false }
func (m anEnrolment) HeldFor() time.Duration { return 0 }
func (m anEnrolment) Answer(context.Context, []byte) error { return nil }
func (m anEnrolment) Took() error { return nil }
func (m anEnrolment) Hold(time.Duration) error { return nil }
func (m anEnrolment) Drop() error { return nil }
// **A node that enrols can hear its declarations at once** (novox/hq 04-ISSUES/146): its consumer is
// made as it enrols, not only when the control plane next starts — the first machine of a mesh
// enrols after the control plane is up, and heard nothing.
func TestAnEnrolledNodeIsGivenHowItHearsItsDeclarations(t *testing.T) {
made := &ensured{}
s := &Server{enroller: acceptsAs("anchor"), consumers: made, log: log.New(io.Discard, "", 0)}
body, _ := json.Marshal(EnrolRequest{Node: "anchor"})
s.enrolling(context.Background(), anEnrolment{body: body})
want := broker.NodeConsumer("anchor")
if len(made.consumers) != 1 || made.consumers[0].Name != want.Name || made.consumers[0].Stream != want.Stream {
t.Fatalf("the enrolled node was given %v, want its own declaration consumer %v", made.consumers, want)
}
}
+37
View File
@@ -0,0 +1,37 @@
package link_test
import (
"context"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/link"
)
// **A token issued for a tunnel key takes that key and no other** (novox/hq ADR 0169). The hub was
// sent the key before the token was shown, so another key is a machine it does not know.
func TestATokenIssuedForATunnelKeyTakesThatKeyAndNoOther(t *testing.T) {
inv, ident := aMeshReadyToEnrol(t)
ctx := context.Background()
secret, public := aTokenFor(t, inv, "joiner")
node, err := inv.NodeByName(ctx, "joiner")
if err != nil {
t.Fatal(err)
}
const issuedFor = "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA="
if err := inv.BindTokenToKey(ctx, node.ID, issuedFor); err != nil {
t.Fatal(err)
}
e := link.Enrolment{Inventory: inv, Identity: ident}
_, err = e.Enrol(ctx, link.EnrolRequest{Node: "joiner", Secret: secret, PublicKey: public,
OverlayKey: "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB="})
if err == nil || !strings.Contains(err.Error(), "issued for the tunnel key") {
t.Fatalf("a token issued for one key took another: %v", err)
}
if _, err := e.Enrol(ctx, link.EnrolRequest{Node: "joiner", Secret: secret, PublicKey: public,
OverlayKey: issuedFor}); err != nil {
t.Fatalf("the key the token was issued for was refused: %v", err)
}
}
+12
View File
@@ -86,6 +86,18 @@ func (e Enrolment) Enrol(ctx context.Context, request EnrolRequest) (reply Enrol
if err != nil {
return EnrolReply{}, err
}
// **A token issued for a tunnel key takes that key and no other** (novox/hq ADR 0169). The hub
// was told it before the token was shown; another key is a machine the hub does not know.
// Checked before anything is recorded, so a refusal changes nothing.
bound, err := e.Inventory.TokenKey(ctx, secret)
if err != nil {
return EnrolReply{}, err
}
if bound != "" && request.OverlayKey != bound {
return EnrolReply{}, fmt.Errorf("%s's token was issued for the tunnel key %s and the machine "+
"offered %q — the key it made with `nox-mesh-host key` is the one to issue for",
node.Name, bound, request.OverlayKey)
}
if _, err := e.Identity.RecordNodeKey(ctx, node.ID, public); err != nil {
return EnrolReply{}, fmt.Errorf("%s's key could not be recorded: %w", node.Name, err)
+21
View File
@@ -73,6 +73,8 @@ type Server struct {
inbound Inbound
bus Bus
js *broker.JetStream
// consumers makes a node's declaration consumer as it enrols; the bus connection, or a stand-in.
consumers interface{ EnsureConsumer(broker.Consumer) error }
enroller Enroller
listener Listener
@@ -383,6 +385,7 @@ func (s *Server) enrolling(ctx context.Context, m Control) {
default:
reply = accepted
s.log.Printf("enrolled %s", accepted.Node)
s.hearsItsDeclarations(accepted.Node)
}
}
@@ -402,6 +405,24 @@ func (s *Server) enrolling(ctx context.Context, m Control) {
_ = m.Took()
}
// hearsItsDeclarations makes the consumer a node reads its declarations through, as it enrols and
// before it is answered.
//
// **Created at enrolment, as the consumer's own doc has always said** (novox/hq 04-ISSUES/146). It
// was asserted only when the control plane started, so the first machine of a mesh — which enrols
// after the control plane is already up — joined and then heard nothing, its host retrying "consumer
// not found" for ever. Failing here is said and does not unspend the token: the next start of the
// control plane asserts it again.
func (s *Server) hearsItsDeclarations(node string) {
if s.consumers == nil {
return
}
if err := s.consumers.EnsureConsumer(broker.NodeConsumer(node)); err != nil {
s.log.Printf("%s enrolled, and how it hears its declarations could not be made — it will hear "+
"nothing until the control plane next starts: %v", node, err)
}
}
// wasBuilt keeps what a builder said, whichever way it went.
//
// This is for results nobody was waiting for. A build asked for with `build` is answered directly
+33
View File
@@ -55,6 +55,26 @@ type Token struct {
// that it speaks the firewall found on the machine, because an adopted node keeps that firewall
// in force. Absent for a converged node, so a converged token is byte for byte what it was.
Adopted bool `json:"adopted,omitempty"`
// Tunnel is the one peer a joining machine needs, when the token was issued for its tunnel key
// (novox/hq ADR 0169). The machine brings its tunnel up from this alone and reaches the bus over
// it, at an address on the private network — so the bus is never open to the internet. Absent
// on a token issued without a key, which then reads byte for byte as before.
Tunnel *Tunnel `json:"tunnel,omitempty"`
}
// Tunnel is the joining machine's side of its first tunnel: its own address and the hub to reach.
type Tunnel struct {
// Key is the public half of the key the machine made itself, which this token was issued for.
// The private half never left the machine (novox/hq ADR 0004).
Key string `json:"key"`
// Address is the machine's own address on the private network, with its prefix.
Address string `json:"address"`
// Range is the private network, routed through the hub until the machine is told more.
Range string `json:"range"`
// HubKey and HubEndpoint are the hub's tunnel key and where it is dialled.
HubKey string `json:"hub_key"`
HubEndpoint string `json:"hub_endpoint"`
}
// Missing names the parts that are not filled in.
@@ -83,6 +103,19 @@ func (t Token) Missing() []string {
if strings.TrimSpace(t.Secret) == "" {
missing = append(missing, "the one-time secret — nothing to present")
}
if t.Tunnel != nil {
for _, part := range []struct{ value, says string }{
{t.Tunnel.Key, "the machine's own tunnel key — the hub would not know it"},
{t.Tunnel.Address, "the machine's address on the private network"},
{t.Tunnel.Range, "the private network's range — nothing to route through the hub"},
{t.Tunnel.HubKey, "the hub's tunnel key — nothing to dial"},
{t.Tunnel.HubEndpoint, "where the hub's tunnel is dialled"},
} {
if strings.TrimSpace(part.value) == "" {
missing = append(missing, part.says)
}
}
}
return missing
}
+35
View File
@@ -172,3 +172,38 @@ func TestATokenWithNoNameIsRefused(t *testing.T) {
t.Fatalf("the refusal does not say what is missing: %v", without.Missing())
}
}
// A token issued for a tunnel key carries the one peer a joining machine needs (novox/hq ADR 0169),
// and says which part is missing rather than producing a tunnel that never answers.
func TestATokenThroughTheTunnelCarriesThePeerOrSaysWhatIsMissing(t *testing.T) {
whole := Token{Node: "n", Broker: "10.42.0.1:4222", Fingerprint: "sha256:x", Signer: make([]byte, 32), Secret: "s",
Tunnel: &Tunnel{Key: "k", Address: "10.42.0.9/32", Range: "10.42.0.0/16", HubKey: "h", HubEndpoint: "198.51.100.1:51820"}}
if !whole.Complete() {
t.Fatalf("a whole token through the tunnel reads as missing %v", whole.Missing())
}
encoded, err := whole.Encode()
if err != nil {
t.Fatal(err)
}
back, err := Decode(encoded)
if err != nil {
t.Fatal(err)
}
if back.Tunnel == nil || *back.Tunnel != *whole.Tunnel {
t.Fatalf("the tunnel did not survive the round trip: %+v", back.Tunnel)
}
part := whole
part.Tunnel = &Tunnel{Key: "k", Address: "10.42.0.9/32"}
if len(part.Missing()) != 3 {
t.Errorf("a tunnel without the hub and the range should name three missing parts: %v", part.Missing())
}
// And a token issued without a key carries no tunnel at all, byte for byte as before.
plain := whole
plain.Tunnel = nil
raw, _ := plain.Encode()
if strings.Contains(raw, "tunnel") {
t.Error("a token without a key mentions a tunnel")
}
}