Compare commits

...
Author SHA1 Message Date
jschoubben 0412653920 The store keeps what the records name (hq ADR 0189)
The mesh names what may go from its own build records — a digest it did not
record making is never named, which is what keeps the sweep away from the
images genesis pushed. An artifact stays because a definition the mesh holds
names it, or because it belongs to one of the five most recent successful
builds of its module.

internal/artifacts asks the store to let go of one; internal/inventory
decides and remembers (migration 0055); the sweep runs after a build the mesh
recorded, which is when both the bytes and the keep set moved. Never fatal to
a build.

And the manifest side of while-stopped, refused from the definition alone:
no schedule, run-once, a container the module does not declare, itself.
2026-10-02 21:49:03 +02:00
jschoubben e09a14ba4c A served value may name the consumer it is served to (hq ADR 0188)
${consumer:as} and ${consumer:as:dns} in a serves block are filled per
consumer at resolution, and the one filled value reaches both ends: the
consumer's binding and its ${bound:...} substitutions, and the provider's
contributions entry as `derived`. A fact or alphabet the mesh does not have
is refused at parse; a consumer whose own file already holds the derived
value is refused at resolution, naming the placeholder to write instead.
2026-10-02 21:24:43 +02:00
mesh-admin 1a44d281c2 Merge pull request 'node show cites ADR 0180 for a removed front end (hq ADR 0186)' (#224) from fix/a-ban-list-never-holds-a-neighbour into main 2026-10-02 16:47:25 +00:00
jschoubben 1c8fe65601 node show cites ADR 0180 for a removed front end (hq ADR 0186)
Another session took 0175 while that record was in review; the line printed on every converged
machine was pointing at an unrelated decision.
2026-10-02 18:42:16 +02:00
mesh-admin bea1a1c513 Merge pull request 'A control plane behind its seat's row serves what it can (hq ADR 0185)' (#222) from fix/a-service-asked-to-run-is-still-running into main 2026-10-02 16:21:55 +00:00
jschoubben 21d38c9b0e A control plane behind its seat's row serves what it can (hq ADR 0185)
One verb in the row that this binary cannot run aborted the start, and a stale push that put an
older control plane back took the whole mesh off the bus for ten minutes — recoverable only by a
person running the binary outside its service, because the push that repairs it is one of the verbs
that had stopped being served. Now the verbs it knows are served, the ones it does not answer the
reason, and the start names them once.
2026-10-02 18:16:24 +02:00
mesh-admin 689dd060b0 Merge pull request 'A jail's pattern names <HOST> once, and is refused by name when it does not (hq ADR 0179)' (#221) from fix/a-jails-pattern-names-the-host-once into main 2026-10-02 15:32:14 +00:00
jschoubben 99c4c4ef04 A jail's pattern names <HOST> once, and is refused by name when it does not (hq ADR 0179)
fail2ban expands <HOST> into a named capture group, so a pattern naming it twice is a duplicate
group name: the daemon refuses its whole configuration and exits, and the machine keeps no bans at
all — for every jail, not the one at fault. Hit live on the control node the day the jails shipped.
A jail with no name, no pattern, or a name another of the module's jails took is refused too.
2026-10-02 17:25:34 +02:00
mesh-admin e5e1666f91 Merge pull request 'The intrusion-prevention seat serves status, banned, ban and unban; the proxy logs a refused name (hq ADR 0179)' (#219) from feat/the-intrusion-seat-serves-its-verbs into main 2026-10-02 15:15:41 +00:00
jschoubben bd58d1c3ef The intrusion-prevention seat serves status, banned, ban and unban; the proxy logs a refused name (hq ADR 0179)
Every holder of node-intrusion-prevention owes the four verbs, as the packet filter's holder owes
its three (ADR 0170). The proxy says in its log when a name this mesh does not serve is asked for,
with the asking address last, so its own jail can read it.
2026-10-02 17:02:49 +02:00
mesh-admin d134c89a7c Merge pull request 'The service manager is a node seat with the unit verbs, and a user shape names the account (hq ADR 0176, 0177)' (#218) from feat/the-operators-machine into main 2026-10-02 14:58:50 +00:00
jochen a9307d9f33 The controller seat gains a generic verb: command runs one line of the binary and answers what it printed
Beside the named verbs, `command` takes a command line as the controller's
own shell would — `node account g14 jochen`, `node show ace`, `module list` —
splits it as a shell does (quotes group, backslash escapes, nothing expanded)
and runs it in this binary like every other verb. The named verbs keep their
schemas; this is the whole binary, added because the operator decided any
node may call any tool (hq ADR 0175) and a verb per command was the only
thing keeping the rest behind a shell on the control node. ADR 0154 carries
the dated note. Tests: a plain line, quoted words, an empty line and an
unclosed quote refused.
2026-10-02 16:53:27 +02:00
jochen 5eb1c9c2b7 The service manager is a node seat with the unit verbs, and a user shape names the account (hq ADR 0176, 0177)
node-service-manager joins the mesh's own seats, node-scoped, serving eight
verbs — units, status, start, stop, restart, enable, disable, journal — each
with a schema that takes an optional scope, "system" or the operator
account's "user" manager. Sixteen seats now; the count test says so and names
the record.

${machine:account} resolves in a resource's `name` and `user` as it already
did in path, owner and content: the shell module makes the operator's account
its holder's login shell through the `user` shape, and the desktop's watchers
run as that account through a user-scoped unit (host change alongside).
Neither can name the person. A machine with no account refuses by name.
2026-10-02 16:48:16 +02:00
mesh-admin 37b8edeec6 Merge pull request 'node show says a found firewall that was uninstalled is removed (hq ADR 0175)' (#217) from feat/the-found-front-end-is-uninstalled into main 2026-10-02 14:29:02 +00:00
jschoubben 424406b2d6 node show says a found firewall that was uninstalled is removed (hq ADR 0175) 2026-10-02 16:27:39 +02:00
mesh-admin 90455c61f6 Merge pull request 'The example images build from the Go the manifest pins' (#216) from jschoubben/example-images-go-base into main 2026-10-02 13:56:22 +00:00
jschoubben 1f6793cf0c The example images build from the Go the manifest pins
Four example Dockerfiles named golang:1.25 while go.mod requires 1.26;
the control plane's image takes GO_BASE from the manifest and these did
not, so a build that could not fetch a newer toolchain failed at go mod
download (found running the lab through the mesh).
2026-10-02 15:52:25 +02:00
mesh-admin 6ef522fbda Merge pull request 'Cite hq ADR 0170, not 0169: the firewall seat's record was renumbered' (#215) from fix/adr-0170-cited into main 2026-10-02 12:53:13 +00:00
jschoubben 46f65c12a0 Cite hq ADR 0170, not 0169: the firewall seat's record was renumbered after a collision on hq main 2026-10-02 14:52:24 +02:00
mesh-admin 8f7c02d77a Merge pull request 'The virtualisation capability grants the lab its daemon's socket (hq ADR 0172)' (#214) from jschoubben/the-lab-is-a-module into main 2026-10-02 12:47:45 +00:00
jschoubben a637df01ea The virtualisation capability grants the lab its daemon's socket
The lab raises machines on the virtualisation daemon, and a module may
mount a machine's socket only through the capability that grants it
(novox/hq ADR 0172). Also brings the resolver's tests to the setting
dnsmasq's listen addresses now come from, and to a module left out
rather than refused.
2026-10-02 14:46:17 +02:00
mesh-admin 252eb786a7 Merge pull request 'A filter module's own filter file counts as declared for a mount (hq ADR 0169)' (#213) from feat/the-firewall-seat-serves-its-verbs into main 2026-10-02 12:05:09 +00:00
29 changed files with 1911 additions and 39 deletions
+4 -4
View File
@@ -59,7 +59,7 @@ PROVISIONER_IMAGE ?= mesh-provision-postgres:$(VERSION)
PROVISIONER_DEV_TAG ?= mesh-provision-postgres:development
provisioner-image:
docker build -f examples/postgres-provisioner/Dockerfile \
docker build --build-arg GO_BASE=$(GO_BASE) -f examples/postgres-provisioner/Dockerfile \
-t $(PROVISIONER_IMAGE) -t $(PROVISIONER_DEV_TAG) .
@echo
@docker image inspect $(PROVISIONER_IMAGE) --format 'built {{.RepoTags}} {{.Size}} bytes'
@@ -70,7 +70,7 @@ OBJECTSTORE_IMAGE ?= mesh-provision-objectstore:$(VERSION)
OBJECTSTORE_DEV_TAG ?= mesh-provision-objectstore:development
objectstore-image:
docker build -f examples/objectstore-provisioner/Dockerfile \
docker build --build-arg GO_BASE=$(GO_BASE) -f examples/objectstore-provisioner/Dockerfile \
-t $(OBJECTSTORE_IMAGE) -t $(OBJECTSTORE_DEV_TAG) .
@echo
@docker image inspect $(OBJECTSTORE_IMAGE) --format 'built {{.RepoTags}} {{.Size}} bytes'
@@ -81,7 +81,7 @@ REDIS_PROVISIONER_IMAGE ?= mesh-provision-redis:$(VERSION)
REDIS_PROVISIONER_DEV_TAG ?= mesh-provision-redis:development
redis-provisioner-image:
docker build -f examples/redis-provisioner/Dockerfile \
docker build --build-arg GO_BASE=$(GO_BASE) -f examples/redis-provisioner/Dockerfile \
-t $(REDIS_PROVISIONER_IMAGE) -t $(REDIS_PROVISIONER_DEV_TAG) .
@echo
@docker image inspect $(REDIS_PROVISIONER_IMAGE) --format 'built {{.RepoTags}} {{.Size}} bytes'
@@ -91,7 +91,7 @@ PROXY_IMAGE ?= mesh-route-proxy:$(VERSION)
PROXY_DEV_TAG ?= mesh-route-proxy:development
proxy-image:
docker build -f examples/route-proxy/Dockerfile -t $(PROXY_IMAGE) -t $(PROXY_DEV_TAG) .
docker build --build-arg GO_BASE=$(GO_BASE) -f examples/route-proxy/Dockerfile -t $(PROXY_IMAGE) -t $(PROXY_DEV_TAG) .
@echo
@docker image inspect $(PROXY_IMAGE) --format 'built {{.RepoTags}} {{.Size}} bytes'
+2
View File
@@ -94,6 +94,8 @@ func showFiltering(f inventory.Filtering, adopted bool) {
switch {
case fw.Active:
fmt.Printf(" found firewall %s is ACTIVE on this converged machine; the next apply retires it again\n", fw.Kind)
case fw.RetiredBy == "removed":
fmt.Printf(" found firewall %s, removed: the mesh's filter is what filters this machine (novox/hq ADR 0180)\n", fw.Kind)
case fw.RetiredBy == inventory.FilterMesh || fw.RetiredBy == "mesh":
fmt.Printf(" found firewall %s, retired by the mesh; its configuration stays on disk\n", fw.Kind)
case fw.RetiredBy != "":
+4
View File
@@ -533,6 +533,10 @@ func takeIn(ctx context.Context, inv *inventory.Inventory, result link.BuildResu
if err := inv.RegisterModule(ctx, manifest, recorded); err != nil {
return manifest, kept, err
}
// The keep set just moved, and new bytes just landed (novox/hq ADR 0189). Asked here rather
// than on a timer of its own: this is the only moment either is true. Never fatal — the build
// worked and the module is registered.
collect(ctx, inv)
return manifest, kept, nil
}
+84
View File
@@ -0,0 +1,84 @@
package main
import (
"context"
"errors"
"fmt"
"os"
"github.com/novox/mesh-controller/internal/artifacts"
"github.com/novox/mesh-controller/internal/inventory"
)
// Letting the artifact store go of what the mesh no longer keeps (novox/hq ADR 0189, issue 108).
//
// **Run where the records change.** A build is the moment new bytes landed in the store and the
// moment the keep set moved, so it is the moment to say what may go — and it needs no timer of
// its own. Reclaiming the bytes is the store's own nightly step; this only decides.
//
// Never fatal to a build. The build succeeded, the module is registered, and a store that could
// not be reached is a thing to say rather than a reason to undo any of that. The next build asks
// again, and the references it could not collect are still uncollected, so nothing is lost by
// having failed.
// collect asks the store to let go of everything the mesh made and no longer keeps, and records
// what it let go of. Says what it did and what it could not; returns nothing, because nothing
// upstream should branch on it.
func collect(ctx context.Context, inv *inventory.Inventory) {
references, err := inv.ToCollect(ctx)
if err != nil {
fmt.Fprintf(os.Stderr, "could not work out what the artifact store may let go of: %v\n", err)
return
}
if len(references) == 0 {
return
}
shelf, err := inv.Catalogue(ctx)
if err != nil {
fmt.Fprintf(os.Stderr, "could not read the catalogue to find the artifact store: %v\n", err)
return
}
// As the mesh reaches it from the network. Empty means the store is not on the network — on a
// mesh being raised it is not yet, and there the store holds one build of anything and has
// nothing to collect.
address, err := artifactStoreAddress(ctx, inv, shelf, "")
if err != nil || address == "" {
if err != nil {
fmt.Fprintf(os.Stderr, "could not find the artifact store to collect from: %v\n", err)
}
return
}
store := artifacts.Store{Address: address}
var done []string
var refused int
for _, reference := range references {
switch err := store.LetGo(ctx, reference); {
case err == nil, errors.Is(err, artifacts.Gone):
// Gone is the outcome wanted, already true. Recorded so the next sweep does not ask
// again for ever.
done = append(done, reference)
default:
refused++
if refused == 1 {
// Once per sweep. A store that refuses one refuses all of them, and a hundred
// identical lines would bury the reason.
fmt.Fprintf(os.Stderr, "the artifact store kept %s: %v\n", reference, err)
}
}
}
if len(done) > 0 {
if err := inv.MarkCollected(ctx, done); err != nil {
// Said, and that is all: the artifacts are gone either way, and the only cost of an
// unrecorded collection is that the next sweep asks about them again.
fmt.Fprintf(os.Stderr, "the store let go of %d artifact(s) and the record of it did not keep: %v\n",
len(done), err)
return
}
fmt.Fprintf(os.Stderr, "the artifact store let go of %d artifact(s) the mesh no longer keeps\n",
len(done))
}
if refused > 0 {
fmt.Fprintf(os.Stderr, "%d artifact(s) were not collected; the next build asks again\n", refused)
}
}
+8 -1
View File
@@ -131,10 +131,17 @@ func serve(ctx context.Context) error {
// And the mesh's own verbs, as the seat this control plane holds (novox/hq ADR 0154). Served
// from the store's row, so what the seat declares is what is answered.
handlers, err := seatToolHandlers()
handlers, behind, err := seatToolHandlers()
if err != nil {
return err
}
if len(behind) > 0 {
// Said once, loudly, and then served anyway (novox/hq ADR 0185): the mesh keeps answering
// while whatever put an older control plane here is undone.
fmt.Printf("this control plane is behind the %s row: it cannot run %s. "+
"Those answer the reason when called; everything else is served as usual\n",
catalogue.ControllerSeatName, strings.Join(behind, ", "))
}
bus, isNATS := server.Bus().(link.OverNATS)
if !isNATS {
return errors.New("the mesh's verbs are served over the bus, and this control plane is not on it")
+97 -7
View File
@@ -48,6 +48,21 @@ func argvFor(verb string, args map[string]any) ([]string, error) {
return nil
}
switch verb {
case "command":
// The generic verb: the command line as given, split as a shell would split it, with
// nothing added — the named verbs add flags a caller cannot reach; this one is the whole
// binary and says so in its description (novox/hq ADR 0154, 0175).
if err := need("command"); err != nil {
return nil, err
}
argv, err := splitCommandLine(str("command"))
if err != nil {
return nil, err
}
if len(argv) == 0 {
return nil, errors.New("command names no command")
}
return argv, nil
case "status":
return []string{"status", "--json"}, nil
case "nodes":
@@ -215,13 +230,15 @@ func runVerb(ctx context.Context, argv []string) (verbAnswer, error) {
}
// seatToolHandlers are the handlers for every verb the mesh-controller seat declares, from the
// store's row, so a verb the row does not carry is not served and a verb it carries that this binary
// cannot run is said at start rather than at the first call.
func seatToolHandlers() (map[string]link.ToolHandler, error) {
// store's row, so a verb the row does not carry is not served. A verb it carries that this binary
// cannot run is named at start and answers the reason when called — never a refusal to serve, which
// would take the whole control plane down for one word (novox/hq ADR 0185).
func seatToolHandlers() (map[string]link.ToolHandler, []string, error) {
seat, known := catalogue.SeatNamed(catalogue.ControllerSeatName)
if !known {
return nil, fmt.Errorf("this mesh defines no %s seat", catalogue.ControllerSeatName)
return nil, nil, fmt.Errorf("this mesh defines no %s seat", catalogue.ControllerSeatName)
}
var behind []string
handlers := map[string]link.ToolHandler{}
for _, v := range seat.Serves {
verb := v.Name
@@ -232,8 +249,27 @@ func seatToolHandlers() (map[string]link.ToolHandler, error) {
continue
}
if _, err := argvFor(verb, sampleArguments(v)); err != nil {
return nil, fmt.Errorf("the %s seat's row declares %q, which this control plane cannot run: %w",
catalogue.ControllerSeatName, verb, err)
// **A row ahead of this binary is not a reason to go silent.**
//
// The row is the store's and a control plane follows it (novox/hq ADR 0154), so a verb
// this build does not know means the row was widened by a newer one — the ordinary
// state of a roll-out, and of a push that put an older control plane back. Refusing to
// serve at all made that transient fatal: on 2026-10-02 one unknown verb took the whole
// mesh off the bus for ten minutes, and the way back was a human running the binary by
// hand, because the thing that would have repaired it is the thing that was down
// (novox/hq 04-ISSUES/201, ADR 0185).
//
// So the verbs this binary knows are served, and this one answers the reason instead of
// nothing: a caller gets a sentence naming the fault, and everything else keeps working
// — including the push that replaces this binary with the one whose verb it is.
behind = append(behind, verb)
reason := err
handlers[verb] = func(context.Context, json.RawMessage) (any, error) {
return nil, fmt.Errorf("%s is in this mesh's %s row and the control plane running "+
"here cannot run it: %w. It is a verb of a newer build; this one is behind",
verb, catalogue.ControllerSeatName, reason)
}
continue
}
handlers[verb] = func(ctx context.Context, raw json.RawMessage) (any, error) {
args := map[string]any{}
@@ -249,7 +285,7 @@ func seatToolHandlers() (map[string]link.ToolHandler, error) {
return runVerb(ctx, argv)
}
}
return handlers, nil
return handlers, behind, nil
}
// seatTools is what `tools` answers: every seat with a protocol, and the tools each serves, from the
@@ -289,3 +325,57 @@ func sampleArguments(v catalogue.Verb) map[string]any {
}
return sample
}
// splitCommandLine splits a command line into words the way a POSIX shell does for the simple
// cases a controller command needs: spaces separate, single or double quotes group, a backslash
// escapes the next character inside double quotes or outside any. No expansion of anything.
func splitCommandLine(line string) ([]string, error) {
var words []string
var cur strings.Builder
inWord := false
quote := rune(0)
runes := []rune(line)
for i := 0; i < len(runes); i++ {
r := runes[i]
switch {
case quote == '\'':
if r == '\'' {
quote = 0
} else {
cur.WriteRune(r)
}
case quote == '"':
if r == '"' {
quote = 0
} else if r == '\\' && i+1 < len(runes) {
i++
cur.WriteRune(runes[i])
} else {
cur.WriteRune(r)
}
case r == '\'' || r == '"':
quote = r
inWord = true
case r == '\\' && i+1 < len(runes):
i++
cur.WriteRune(runes[i])
inWord = true
case r == ' ' || r == '\t' || r == '\n':
if inWord {
words = append(words, cur.String())
cur.Reset()
inWord = false
}
default:
cur.WriteRune(r)
inWord = true
}
}
if quote != 0 {
return nil, fmt.Errorf("command has an unclosed %c quote", quote)
}
if inWord {
words = append(words, cur.String())
}
return words, nil
}
+79 -1
View File
@@ -1,6 +1,7 @@
package main
import (
"context"
"strings"
"testing"
@@ -130,10 +131,13 @@ func TestActsDoNotBlockTheCall(t *testing.T) {
// What `tools` answers is the seats' records, with each verb's schema.
func TestToolsAnswersTheSeatsRecords(t *testing.T) {
handlers, err := seatToolHandlers()
handlers, behind, err := seatToolHandlers()
if err != nil {
t.Fatal(err)
}
if len(behind) != 0 {
t.Fatalf("this build cannot run %v of its own seat's verbs", behind)
}
if len(handlers) != len(catalogue.ControllerVerbs) {
t.Fatalf("%d handlers for %d verbs", len(handlers), len(catalogue.ControllerVerbs))
}
@@ -171,3 +175,77 @@ func TestAJSONVerbsAnswerIsItsStandardOutput(t *testing.T) {
t.Fatalf("stderr and stdout are both what the command said: %s", answer.Output)
}
}
// `command` is the generic verb: the command line as given, split as a shell would, nothing added —
// so an operator's `node account g14 jochen` is one call through the console rather than a shell on
// the control node (novox/hq ADR 0154, ADR 0175).
func TestCommandRunsTheLineAsGiven(t *testing.T) {
argv, err := argvFor("command", map[string]any{"command": "node account g14 jochen"})
if err != nil || strings.Join(argv, " ") != "node account g14 jochen" {
t.Fatalf("a plain line: %v %v", argv, err)
}
argv, err = argvFor("command", map[string]any{"command": `settings set dnsmasq '{"a": "b c"}' --node ace`})
if err != nil || len(argv) != 6 || argv[3] != `{"a": "b c"}` {
t.Fatalf("a quoted word stays one word: %q %v", argv, err)
}
argv, err = argvFor("command", map[string]any{"command": `node add "the box" --adopted`})
if err != nil || len(argv) != 4 || argv[2] != "the box" {
t.Fatalf("double quotes group: %q %v", argv, err)
}
if _, err := argvFor("command", map[string]any{"command": " "}); err == nil {
t.Fatal("an empty line was accepted")
}
if _, err := argvFor("command", map[string]any{"command": `node "unclosed`}); err == nil {
t.Fatal("an unclosed quote was accepted")
}
}
// A verb in the row that this binary cannot run does not take the control plane off the bus: the
// rest are served, the unknown one answers the reason, and the start-up names it (novox/hq ADR
// 0185). One unknown word cost the mesh ten minutes of silence on 2026-10-02, recoverable only by
// a person running the binary by hand — the push that would have repaired it needs the control
// plane that was down.
func TestARowAheadOfThisBuildIsServedAnyway(t *testing.T) {
seat, known := catalogue.SeatNamed(catalogue.ControllerSeatName)
if !known {
t.Fatal("no controller seat")
}
// The row as a newer control plane would have written it: every verb this build knows, and one
// it does not.
widened := seat
widened.Serves = append(append([]catalogue.Verb{}, seat.Serves...),
catalogue.Verb{Name: "teleport", Description: "a verb from a build that does not exist yet"})
rows := catalogue.DefaultSeats()
for i := range rows {
if rows[i].Name == catalogue.ControllerSeatName {
rows[i] = widened
}
}
catalogue.UseSeats(rows)
t.Cleanup(func() { catalogue.UseSeats(catalogue.DefaultSeats()) })
handlers, behind, err := seatToolHandlers()
if err != nil {
t.Fatalf("a row with one unknown verb refused to serve at all: %v", err)
}
if len(behind) != 1 || behind[0] != "teleport" {
t.Fatalf("the verbs this build cannot run were reported as %v", behind)
}
if len(handlers) != len(widened.Serves) {
t.Fatalf("%d handlers for %d verbs in the row", len(handlers), len(widened.Serves))
}
for _, known := range []string{"status", "nodes", "push"} {
if handlers[known] == nil {
t.Errorf("%s is not served although this build knows it", known)
}
}
_, err = handlers["teleport"](context.Background(), nil)
if err == nil {
t.Fatal("the unknown verb answered as though it had run")
}
for _, want := range []string{"teleport", "cannot run it", "behind"} {
if !strings.Contains(err.Error(), want) {
t.Errorf("the answer does not say %q: %v", want, err)
}
}
}
+4 -1
View File
@@ -10,7 +10,10 @@
# The client is copied from the vendor's own image rather than installed from a distribution:
# `apk add mc` on Alpine installs Midnight Commander, which is a different program with the same
# name, and the failure would be a provisioner that starts cleanly and cannot do anything.
FROM golang:1.25-alpine AS build
# The Go it builds with is the one the manifest pins (build.on GO_BASE), passed by the Makefile and the
# build machine alike; the default only serves a hand build, and matches go.mod.
ARG GO_BASE=golang:1.26-alpine
FROM ${GO_BASE} AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
+4 -1
View File
@@ -3,7 +3,10 @@
# Built here so a machine can be given it by the mesh rather than by somebody putting a binary on
# it. Static and FROM scratch for the same reason the control plane's image is: it is fetched by
# digest and run on a machine, and everything in it is something a person would have to audit.
FROM golang:1.25-alpine AS build
# The Go it builds with is the one the manifest pins (build.on GO_BASE), passed by the Makefile and the
# build machine alike; the default only serves a hand build, and matches go.mod.
ARG GO_BASE=golang:1.26-alpine
FROM ${GO_BASE} AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
+4 -1
View File
@@ -2,7 +2,10 @@
#
# FROM scratch, like the postgres one and unlike the bucket one: it speaks the store's own wire
# protocol directly and needs no client in the image.
FROM golang:1.25-alpine AS build
# The Go it builds with is the one the manifest pins (build.on GO_BASE), passed by the Makefile and the
# build machine alike; the default only serves a hand build, and matches go.mod.
ARG GO_BASE=golang:1.26-alpine
FROM ${GO_BASE} AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
+4 -1
View File
@@ -2,7 +2,10 @@
#
# Static and FROM scratch like the control plane's image, and for the same reason: it is fetched
# by digest and run on a machine, so everything in it is something a person would have to audit.
FROM golang:1.25-alpine AS build
# The Go it builds with is the one the manifest pins (build.on GO_BASE), passed by the Makefile and the
# build machine alike; the default only serves a hand build, and matches go.mod.
ARG GO_BASE=golang:1.26-alpine
FROM ${GO_BASE} AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
+5
View File
@@ -732,6 +732,11 @@ func handler(held *table) http.Handler {
// And since a host may now be routed only on some paths, those are a third thing:
// saying "no route for this name" while listing that very name as served is a
// contradiction an operator would have to disbelieve the proxy to get past.
// **Said in the log as well as to the client.** A name this mesh does not serve, asked
// for from outside, is what a scanner does, and the machine's intrusion prevention reads
// this proxy's log for exactly that line (novox/hq ADR 0179): the address last, as the
// jail's filter expects it.
log.Printf("refused: no route for %q, asked from %s", r.Host, r.RemoteAddr)
w.Header().Set("Content-Type", "text/plain; charset=utf-8")
w.WriteHeader(http.StatusNotFound)
if !hidden && held.routed(r.Host) {
+99
View File
@@ -0,0 +1,99 @@
// Package artifacts speaks to the mesh's artifact store over its own door.
//
// Only what the mesh needs that nothing else does: letting go of something it put there
// (novox/hq ADR 0189, issue 108). Pushing is the builder's, through the container runtime; reading
// is every machine's, through its runtime. This is the one operation that belongs to the thing
// holding the records, because it is the only one that is a decision rather than a transfer.
package artifacts
import (
"context"
"fmt"
"net/http"
"strings"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
)
// Store is the artifact store at an address, as this machine reaches it.
type Store struct {
// Address is `host:port` — the store as the caller reaches it now, composed and never
// recorded (novox/hq 04-ISSUES/102).
Address string
// HTTP is the client used; nil is a client with a modest timeout.
HTTP *http.Client
}
// Gone is the answer when the store does not hold it: the outcome wanted, already true.
var Gone = fmt.Errorf("the store does not hold it")
// LetGo asks the store to drop one artifact the mesh recorded making.
//
// Takes a reference as the mesh records it — `artifact-store://<module>/<artifact>@sha256:…` for
// an image, `…/blobs/sha256:…` for an archive — because that is the identity every record uses,
// and composes the address here at the moment of use.
//
// Returns Gone when the store answers that it does not have it. That is not a failure: the sweep
// wants the artifact absent, and it is. It is distinguished from success only so a caller can say
// which of the two happened.
func (s Store) LetGo(ctx context.Context, reference string) error {
path, kept := catalogue.InArtifactStore(reference)
if !kept {
// Nothing the mesh put in its own store. Refused rather than attempted: composing a
// delete for a reference of unknown shape is how a sweep reaches something that is not
// the mesh's.
return fmt.Errorf("%s is not a reference into the mesh's artifact store", reference)
}
if s.Address == "" {
return fmt.Errorf("this mesh has no artifact store on its network to ask about %s", reference)
}
repository, kind, digest, err := split(path)
if err != nil {
return err
}
url := "http://" + s.Address + "/v2/" + repository + "/" + kind + "/" + digest
request, err := http.NewRequestWithContext(ctx, http.MethodDelete, url, nil)
if err != nil {
return err
}
client := s.HTTP
if client == nil {
client = &http.Client{Timeout: 30 * time.Second}
}
response, err := client.Do(request)
if err != nil {
return err
}
defer response.Body.Close()
switch response.StatusCode {
case http.StatusAccepted, http.StatusOK, http.StatusNoContent:
return nil
case http.StatusNotFound:
return Gone
case http.StatusMethodNotAllowed:
// The registry was started without deletion enabled. Said plainly, because the remedy is
// a setting on the store's module and not anything about this artifact.
return fmt.Errorf(
"the artifact store refuses deletion: its server was started without it enabled "+
"(REGISTRY_STORAGE_DELETE_ENABLED), so nothing can be collected until the store "+
"module is applied again (novox/hq ADR 0189). Asking about %s", reference)
default:
return fmt.Errorf("the artifact store answered %s for %s", response.Status, reference)
}
}
// split reads a recorded path into the repository, which endpoint names the thing, and the digest.
//
// Two shapes, which are the two the mesh records: `<repository>@sha256:<hex>` is a manifest, and
// `<repository>/blobs/sha256:<hex>` is a blob.
func split(path string) (repository, kind, digest string, err error) {
if before, after, ok := strings.Cut(path, "@sha256:"); ok {
return before, "manifests", "sha256:" + after, nil
}
if before, after, ok := strings.Cut(path, "/blobs/sha256:"); ok {
return before, "blobs", "sha256:" + after, nil
}
return "", "", "", fmt.Errorf("%q names nothing the store holds by digest", path)
}
+94
View File
@@ -0,0 +1,94 @@
package artifacts
import (
"context"
"errors"
"net/http"
"net/http/httptest"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
)
// Asking the store to let go of what the mesh no longer keeps (novox/hq ADR 0189, issue 108).
//
// A fake store records what it was asked to delete, so what is asserted is the mesh's decision
// and the shape of the request — not the registry's behaviour, which is the registry's to test.
func fakeStore(t *testing.T, answer int) (Store, *[]string) {
t.Helper()
var asked []string
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodDelete {
t.Errorf("the store was asked %s %s; collecting is a delete", r.Method, r.URL.Path)
}
asked = append(asked, r.URL.Path)
w.WriteHeader(answer)
}))
t.Cleanup(server.Close)
return Store{Address: strings.TrimPrefix(server.URL, "http://")}, &asked
}
func TestAnImageAndAnArchiveAreAskedForAtTheirOwnEndpoints(t *testing.T) {
// The two shapes the mesh records: a manifest by digest, and a blob by digest. They are
// different endpoints, and asking at the wrong one answers 404 — which this would then
// record as collected, leaving the bytes on disk for ever while the record says otherwise.
store, asked := fakeStore(t, http.StatusAccepted)
ctx := context.Background()
image := catalogue.ArtifactStoreScheme + "web/app@sha256:abc123"
archive := catalogue.ArtifactStoreScheme + "web/config/blobs/sha256:def456"
if err := store.LetGo(ctx, image); err != nil {
t.Fatal(err)
}
if err := store.LetGo(ctx, archive); err != nil {
t.Fatal(err)
}
want := []string{"/v2/web/app/manifests/sha256:abc123", "/v2/web/config/blobs/sha256:def456"}
if len(*asked) != 2 || (*asked)[0] != want[0] || (*asked)[1] != want[1] {
t.Fatalf("the store was asked %v; want %v", *asked, want)
}
}
func TestAStoreThatDoesNotHaveItAnswersGone(t *testing.T) {
// The outcome wanted, already true. Told apart from success only so the sweep can say which
// happened; both are recorded, because retrying for ever is the thing to avoid.
store, _ := fakeStore(t, http.StatusNotFound)
err := store.LetGo(context.Background(), catalogue.ArtifactStoreScheme+"web/app@sha256:abc123")
if !errors.Is(err, Gone) {
t.Fatalf("a store that does not hold it answered %v, want Gone", err)
}
}
func TestAStoreWithDeletionOffSaysSoAndNamesTheRemedy(t *testing.T) {
// The registry answers 405 when it was started without deletion enabled. The remedy is a
// setting on the store's module, and saying "405" would send somebody to the wrong place.
store, _ := fakeStore(t, http.StatusMethodNotAllowed)
err := store.LetGo(context.Background(), catalogue.ArtifactStoreScheme+"web/app@sha256:abc123")
if err == nil {
t.Fatal("a store that refuses deletion was read as success")
}
if !strings.Contains(err.Error(), "REGISTRY_STORAGE_DELETE_ENABLED") {
t.Fatalf("the refusal does not name the remedy: %v", err)
}
}
func TestAReferenceThatIsNotTheMeshsOwnIsNeverAsked(t *testing.T) {
// The whole safety of the sweep is that it names only what the mesh recorded putting there.
// A reference of another shape — a vendor's image, a package version — is refused rather
// than composed into a delete somewhere that is not the mesh's store.
store, asked := fakeStore(t, http.StatusAccepted)
for _, reference := range []string{
"docker.io/library/registry@sha256:abc123",
"registry@sha256:abc123",
"1.4.2",
} {
if err := store.LetGo(context.Background(), reference); err == nil {
t.Errorf("%s was asked about; it is not a reference into the mesh's store", reference)
}
}
if len(*asked) != 0 {
t.Fatalf("the store was asked about %v", *asked)
}
}
+12 -4
View File
@@ -46,7 +46,7 @@ func boundUsed(content string) [][2]string {
// Three facts the mesh states about any provision, plus whatever the provider said it serves. A
// module may not reach a binding it does not have — the same boundary as a secret, for the same
// reason.
func knownFor(m Manifest, needs []Needed, node string) map[string]map[string]string {
func knownFor(m Manifest, needs []Needed, node string) (map[string]map[string]string, error) {
out := map[string]map[string]string{}
for _, want := range m.Wants() {
for i := range needs {
@@ -54,12 +54,20 @@ func knownFor(m Manifest, needs []Needed, node string) map[string]map[string]str
if n.Name != want || n.For != m.Module {
continue
}
as := ConsumerIdentity(node, IdentitySource(m.Slug, m.Module))
values := map[string]string{
"at": n.At,
"from": n.From,
"as": ConsumerIdentity(node, IdentitySource(m.Slug, m.Module)),
"as": as,
}
for key, value := range n.Serves {
// What the provider derives for this consumer rather than for all of them
// (novox/hq ADR 0188). Filled here, the one place a provision and the module
// requiring it are both in hand.
served, err := ServedTo(n.Serves, as)
if err != nil {
return nil, fmt.Errorf("%s requires %s: %w", m.Module, want, err)
}
for key, value := range served {
// The provider's own vocabulary. Rendered plainly: a port is 5432, not 5432.000000,
// which is what a float would write and what a connection string would refuse.
values[key] = plainly(value)
@@ -67,7 +75,7 @@ func knownFor(m Manifest, needs []Needed, node string) map[string]map[string]str
out[want] = values
}
}
return out
return out, nil
}
// withOwnNames adds a module's own composed names to what it may name from one binding:
+290
View File
@@ -0,0 +1,290 @@
package catalogue
import (
"fmt"
"regexp"
"sort"
"strings"
)
// What a provider derives for one consumer, said once in the provider's definition and delivered
// to both ends (novox/hq ADR 0188, issue 124).
//
// A `serves` block is otherwise literal: the same values for every consumer. Where the provider
// *names the resource* — a bucket, a database, a vhost — the name is derived from who is asking,
// and before this the mesh had no channel for it. The provider recomputed it in its own code and
// every consumer transcribed it into its own definition by hand, which is a copy of somebody
// else's rule kept in agreement by nobody. One of three transcriptions was wrong for months.
//
// **The mesh learns no protocol here; it spells its own name in an alphabet it already knows.**
// The only fact a served value may name is the identity the mesh itself minted for the consumer,
// in one of two alphabets: as it was minted, and as a DNS label. Everything a provider wants
// around it — a prefix, a suffix, a separator — it writes around the placeholder, because a
// served value is a string.
// consumerFact is `${consumer:<fact>}` or `${consumer:<fact>:<alphabet>}`.
var consumerFact = regexp.MustCompile(`\$\{consumer:([a-z][a-z0-9-]*)(?::([a-z][a-z0-9-]*))?\}`)
// consumerFacts are what a served value may name about the consumer it is being derived for.
// One entry, deliberately: the identity is the one thing about a consumer the mesh itself chose,
// so it is the one thing the mesh can hand to a provider without either end guessing.
var consumerFacts = []string{"as"}
// consumerAlphabets are the ways the mesh will write that identity. `dns` is the mesh's own
// identifier with its separator written `-` instead of `_` — the whole of the difference between
// the alphabet the mesh mints in and the one buckets, vhosts and hostnames accept.
var consumerAlphabets = []string{"dns"}
// ServedTo fills a provider's served values for one consumer.
//
// `as` is the identity the mesh minted for that consumer — the same string it is told to present
// as a login. Values with no placeholder are returned exactly as they were, and a block with no
// placeholder at all is returned unchanged, so this costs nothing for the providers that derive
// nothing.
//
// Only strings carry placeholders. A number, a boolean or a nested object is a value the provider
// stated outright, and is left alone.
func ServedTo(serves map[string]any, as string) (map[string]any, error) {
if len(serves) == 0 {
return serves, nil
}
var out map[string]any
for _, key := range sortedAnyKeys(serves) {
text, ok := serves[key].(string)
if !ok || !strings.Contains(text, "${consumer:") {
continue
}
filled, err := consumerInto(text, as)
if err != nil {
return nil, fmt.Errorf("the value served as %q: %w", key, err)
}
if out == nil {
// Copied only once something actually changes: the caller's map is the manifest's,
// and a provider that derives nothing must not have it rewritten underneath it.
out = make(map[string]any, len(serves))
for k, v := range serves {
out[k] = v
}
}
out[key] = filled
}
if out == nil {
return serves, nil
}
return out, nil
}
// consumerInto replaces every `${consumer:…}` in one value.
//
// **A fact or an alphabet the mesh does not have is refused, not left standing.** Written through,
// the literal `${consumer:as}` would reach a configuration file and be read as a bucket name,
// failing somewhere that names neither the module nor the mesh — the same reasoning `${bound:…}`
// is refused by (boundInto).
func consumerInto(value, as string) (string, error) {
var failed error
out := consumerFact.ReplaceAllStringFunc(value, func(match string) string {
parts := consumerFact.FindStringSubmatch(match)
fact, alphabet := parts[1], parts[2]
if fact != "as" {
if failed == nil {
failed = fmt.Errorf(
"says %s, and the mesh states %s about a consumer", match, orNothing(consumerFacts))
}
return match
}
switch alphabet {
case "":
return as
case "dns":
return asDNSLabel(as)
default:
if failed == nil {
failed = fmt.Errorf(
"says %s, and the mesh writes an identity as %s", match, orNothing(consumerAlphabets))
}
return match
}
})
if failed != nil {
return "", failed
}
return out, nil
}
// asDNSLabel writes a minted identity as a DNS label.
//
// The mesh's identities are already lower-case letters, digits and `_` (ConsumerIdentity), and
// already short enough for the tightest backend they reach (CheckIdentity, twenty characters). So
// this is the separator and nothing else — no lower-casing of what is already lower case, no
// truncation to a limit the identity is already inside, no padding of a name that is already long
// enough. Each of those would be the mesh guessing at a rule it has not been given.
func asDNSLabel(as string) string {
return strings.ReplaceAll(as, "_", "-")
}
// CheckServes refuses a `serves` block that names a consumer fact or an alphabet the mesh does not
// have, when the definition is parsed rather than when a consumer is resolved.
//
// A provision nobody consumes yet still has its rule read: a definition that would be refused the
// first time somebody required it is a definition that is wrong now.
func CheckServes(m Manifest) []string {
var problems []string
for _, provision := range sortedServes(m.Serves) {
for _, key := range sortedAnyKeys(m.Serves[provision]) {
text, ok := m.Serves[provision][key].(string)
if !ok {
continue
}
// A probe identity, because what is checked is the shape of the statement and not
// what any consumer is called.
if _, err := consumerInto(text, "mesh_node_module"); err != nil {
problems = append(problems, fmt.Sprintf(
"%s serves %s, and the value it serves as %q %s", m.Module, provision, key, err))
}
}
}
return problems
}
func sortedServes(serves map[string]map[string]any) []string {
out := make([]string, 0, len(serves))
for k := range serves {
out = append(out, k)
}
sort.Strings(out)
return out
}
func sortedAnyKeys(values map[string]any) []string {
out := make([]string, 0, len(values))
for k := range values {
out = append(out, k)
}
sort.Strings(out)
return out
}
// derivedFor is what the provider on this machine derives for one consumer of one provision
// (novox/hq ADR 0188).
//
// Settled first, then derived: an operator may set a prefix on what the provider serves and the
// mesh still fills the consumer's half of it ([ADR 0174]). Only the keys that actually name the
// consumer are returned — the rest of a `serves` block is the same for every consumer and is
// already in the provider's own definition, so repeating it here would be a second copy to go
// stale.
//
// The first module in the resolved order that says it serves the provision answers, which is the
// choice servedOnThisMachine makes for the consumer's half. Nothing serving it on this machine is
// not an error: a contribution can reach a machine whose provider is a record or an adapter, and
// then there is nothing derived to tell.
func (r Resolution) derivedFor(provision, as string, settings SettingsBy) (map[string]any, error) {
for _, m := range r.Modules {
serves, said := m.Serves[provision]
if !said {
continue
}
var names map[string]any
for key, value := range serves {
if text, ok := value.(string); ok && strings.Contains(text, "${consumer:") {
if names == nil {
names = map[string]any{}
}
names[key] = value
}
}
if names == nil {
return nil, nil
}
settled, err := Settle(names, settings[m.Module])
if err != nil {
return nil, fmt.Errorf("%s serving %s: %w", m.Module, provision, err)
}
derived, err := ServedTo(settled, as)
if err != nil {
return nil, fmt.Errorf("%s serving %s to %s: %w", m.Module, provision, as, err)
}
return derived, nil
}
return nil, nil
}
// notTranscribed refuses a consumer's file that writes out the value its provider derives for it,
// instead of asking for it (novox/hq ADR 0188, issue 124).
//
// **What would have caught the one wrong instance.** The object store's three consumers each wrote
// their bucket into their own configuration by hand. One of them named a predecessor's bucket, and
// nothing compared it to what the provider would actually create: the module would have
// authenticated successfully and been refused on every object, which reads like a credential fault
// and is not one. It looked authoritative for months.
//
// The test is exact and costs one string search: a definition whose file already contains the
// value the mesh is about to derive for it has written down somebody else's rule. It cannot be a
// coincidence — a derived value carries the identity the mesh minted for this very consumer on
// this very machine, which nothing else would spell out — and it cannot be checked afterwards,
// because after substitution every consumer's file contains it legitimately.
//
// Only values that actually name the consumer are judged. A provider that serves a constant under
// the same key serves the same constant to everyone, and a consumer repeating it is redundant
// rather than wrong.
func notTranscribed(resource map[string]any, known map[string]map[string]string, module string) error {
if fmt.Sprint(resource["type"]) != "file" {
return nil
}
content, ok := resource["content"].(string)
if !ok || content == "" {
return nil
}
for _, provision := range sortedKnown(known) {
values := known[provision]
identity := values["as"]
if identity == "" {
continue
}
for _, key := range sortedStringKeys(values) {
if key == "as" {
// The login is not derived from itself, and a consumer that must present it in a
// connection string legitimately has it from `${bound:…}` — which is what it will
// be after substitution, so this would judge the substitution, not the module.
continue
}
value := values[key]
if value == "" || !namesTheConsumer(value, identity) {
continue
}
if !strings.Contains(content, value) {
continue
}
return fmt.Errorf(
"%s writes %q into %v, and that is exactly what %s derives for it — a definition "+
"keeping its own copy of somebody else's naming rule is one that can disagree "+
"with it, silently. Say ${bound:%s:%s} and be told",
module, value, resource["id"], provision, provision, key)
}
}
return nil
}
// namesTheConsumer is whether a derived value was built from this consumer's identity — in the
// alphabet it was minted in, or as a DNS label. A value that does not contain it was not derived
// from it, whatever else it may be.
func namesTheConsumer(value, identity string) bool {
return strings.Contains(value, identity) || strings.Contains(value, asDNSLabel(identity))
}
func sortedKnown(known map[string]map[string]string) []string {
out := make([]string, 0, len(known))
for k := range known {
out = append(out, k)
}
sort.Strings(out)
return out
}
func sortedStringKeys(values map[string]string) []string {
out := make([]string, 0, len(values))
for k := range values {
out = append(out, k)
}
sort.Strings(out)
return out
}
+50 -5
View File
@@ -628,7 +628,16 @@ func (r Resolution) compose(with Rendering, owner map[string]string,
if err != nil {
return nil, err
}
file, err := boundFile(*found, m.Binds[to], ConsumerIdentity(r.Node, IdentitySource(m.Slug, m.Module)), own)
as := ConsumerIdentity(r.Node, IdentitySource(m.Slug, m.Module))
// What the provider derives for THIS consumer, filled here where the consumer is
// known (novox/hq ADR 0188). The same fill knownFor does below, so the binding file
// and the module's `${bound:…}` substitutions cannot say different things.
told := *found
told.Serves, err = ServedTo(told.Serves, as)
if err != nil {
return nil, fmt.Errorf("%s is told about %s: %w", m.Module, to, err)
}
file, err := boundFile(told, m.Binds[to], as, own)
if err != nil {
return nil, err
}
@@ -694,7 +703,10 @@ func (r Resolution) compose(with Rendering, owner map[string]string,
return nil, err
}
// And what its bindings say, for the half of a connection that is not secret.
known := knownFor(m, r.Needs, r.Node)
known, err := knownFor(m, r.Needs, r.Node)
if err != nil {
return nil, err
}
// A requirement answered on this same machine is not in r.Needs — its binding file is
// written from `here` (above) — and so `${bound:…}` could not name it, though the file
// beside it said the same facts. Filled from the same answer, so the two cannot disagree.
@@ -711,7 +723,11 @@ func (r Resolution) compose(with Rendering, owner map[string]string,
}
local := *answered
local.For = m.Module
for provision, values := range knownFor(m, []Needed{local}, r.Node) {
here, err := knownFor(m, []Needed{local}, r.Node)
if err != nil {
return nil, err
}
for provision, values := range here {
known[provision] = values
}
}
@@ -736,6 +752,17 @@ func (r Resolution) compose(with Rendering, owner map[string]string,
// And the machine underneath, which no binding of its own can tell it.
thisMachine := machineFacts(r, with.Names, with.MeshRange)
// **A definition that already holds the answer transcribed it** (novox/hq ADR 0188).
// Judged over what the module itself declares, and before anything is substituted: the
// mesh's own generated files — the binding, the contributions — legitimately carry the
// derived value, and after substitution so does every consumer's file, so this is the one
// moment the two can be told apart.
for _, own := range m.Resources {
if err := notTranscribed(own, known, m.Module); err != nil {
return nil, err
}
}
// Which of this module's files carry a secret, for the rule that a container may not read
// one of them as its environment without saying so (ADR 0086, issue 041).
secretFiles := secretFilesOf(resources)
@@ -1100,6 +1127,19 @@ type Contribution struct {
// requirement's name — everything providing `reverse-proxy` understands the same shape, which
// is what makes swapping one for another cost nothing.
Values map[string]any `json:"values"`
// Derived is what this provider's own definition said it derives for this consumer, already
// derived (novox/hq ADR 0188).
//
// **The provider is told, rather than recomputing it.** A served value may name the consumer's
// identity — a bucket named for who is asking, a database prefixed with it — and before this
// the rule lived twice: once in the provisioner's code, once transcribed into every consumer's
// definition. The mesh fills the provider's own statement here and delivers the same filled
// value to the consumer, so the two cannot disagree: there is no second computation to
// disagree with.
//
// Only the keys that are per-consumer. The rest of what the provider serves is the same for
// everyone and is in its own definition, where it already is.
Derived map[string]any `json:"derived,omitempty"`
}
// grantPath is where one consumer's sealed credential lands on the providing machine.
@@ -1191,12 +1231,17 @@ func (r Resolution) contributions(settings SettingsBy, grants []Grant,
// told about it and withdraws the login on its next pass.
continue
}
as := holderAs(ConsumerIdentity(g.Consumer, IdentitySource(g.Slug, g.From)), g.Local)
derived, err := r.derivedFor(g.Provision, as, settings)
if err != nil {
return nil, err
}
out[g.Provision] = append(out[g.Provision], Contribution{
From: g.From, Node: g.Consumer, At: g.At, Values: g.Values,
From: g.From, Node: g.Consumer, At: g.At, Values: g.Values, Derived: derived,
// One holder per local name: the identity the consumer is known by, and the local name
// after it where the module keeps several (ADR 0094). Not a login any backend checks —
// a secret is not a login — so the identity limit does not apply to the suffix.
As: holderAs(ConsumerIdentity(g.Consumer, IdentitySource(g.Slug, g.From)), g.Local),
As: as,
Secret: grantPath(directories[g.Provision], g.Consumer, holderAs(g.From, g.Local)),
})
if granted[g.Provision] == nil {
@@ -0,0 +1,297 @@
package catalogue
import (
"encoding/json"
"strings"
"testing"
)
// What a provider derives for each consumer, said once and delivered to both ends
// (novox/hq ADR 0188, issue 124).
//
// The failure these are written against: the object store's provisioner derived each consumer's
// bucket from the login the mesh minted, in its own code, and the mesh had no channel to tell the
// consumer which bucket that was — so all three consumers wrote the answer into their own
// definitions by hand. Two were right. One named a predecessor's bucket and would have
// authenticated successfully and been refused on every object. Each of them also named the
// machine the module happens to run on, which a definition may not do.
// store is an object store in the shape minio has: it serves a region and a port to everyone, and
// a bucket named for whoever is asking.
func store() Manifest {
return Manifest{
Module: "store", Version: "1",
Provides: FromAnywhere("s3-bucket"),
Listens: []Listening{{Port: 9000, Protocol: "tcp", From: FromMesh}},
Serves: map[string]map[string]any{"s3-bucket": {
"region": "eu-west",
"bucket": "${consumer:as:dns}",
}},
Receives: map[string]string{"s3-bucket": "/var/lib/store/grants/mesh.json"},
Grants: map[string]string{"s3-bucket": "/var/lib/store/grants"},
Resources: []map[string]any{{
"id": "server", "type": "container", "name": "store", "ports": []any{"9000"},
}},
}
}
// files is a consumer that writes the bucket into its own configuration — which is the thing it
// could not do before, and had to transcribe.
func files() Manifest {
return Manifest{
Module: "files", Version: "1", Slug: "files",
Requires: []string{"s3-bucket"},
Binds: map[string]string{"s3-bucket": "/var/lib/files/store.json"},
Secrets: map[string]string{"s3-bucket": "/var/lib/files/store.secret"},
Resources: []map[string]any{{
"id": "env", "type": "file", "path": "/var/lib/files/env", "mode": "0600",
"content": "BUCKET=${bound:s3-bucket:bucket}\nREGION=${bound:s3-bucket:region}\n",
}},
}
}
// pics is a second consumer of the same provider on the same machine: two derivations, neither
// the other's.
func pics() Manifest {
return Manifest{
Module: "pics", Version: "1", Slug: "pics",
Requires: []string{"s3-bucket"},
Binds: map[string]string{"s3-bucket": "/var/lib/pics/store.json"},
Secrets: map[string]string{"s3-bucket": "/var/lib/pics/store.secret"},
Resources: []map[string]any{{
"id": "env", "type": "file", "path": "/var/lib/pics/env", "mode": "0600",
"content": "BUCKET=${bound:s3-bucket:bucket}\n",
}},
}
}
// The three places the derived value lands must agree, because agreeing is the whole point: the
// consumer's own file, the binding it reads as JSON, and the provider's contributions entry.
func TestADerivedValueReachesBothEndsAndAgrees(t *testing.T) {
r, err := Resolve(shelf(store(), files()), []string{"store", "files"}, reachable(), World{})
if err != nil {
t.Fatal(err)
}
out, err := r.Declaration(Rendering{Grants: []Grant{{
Provision: "s3-bucket", Consumer: "workstation", From: "files", Slug: "files",
Values: map[string]any{}, Sealed: "c2VhbGVk",
}}})
if err != nil {
t.Fatal(err)
}
// The mesh minted this identity for the consumer; the bucket is that identity as a DNS label.
// Derived here with the mesh's own function, so the test cannot agree with a wrong rule.
as := ConsumerIdentity("workstation", IdentitySource("files", "files"))
want := strings.ReplaceAll(as, "_", "-")
if want == as || !strings.Contains(as, "_") {
t.Fatalf("the mesh's identity %q has no separator to rewrite; this test proves nothing", as)
}
env := fileNamed(out, "files.env")
if env == nil {
t.Fatalf("the consumer was given no file: %v", out)
}
if got := env["content"].(string); !strings.Contains(got, "BUCKET="+want+"\n") {
t.Errorf("the consumer's own file was not told the bucket:\n%s\nwant BUCKET=%s", got, want)
}
binding := fileNamed(out, "files.bound-s3-bucket")
if binding == nil {
t.Fatalf("the consumer was given no binding: %v", out)
}
var said struct {
Serves map[string]any `json:"serves"`
}
if err := json.Unmarshal([]byte(binding["content"].(string)), &said); err != nil {
t.Fatal(err)
}
if said.Serves["bucket"] != want {
t.Errorf("the binding says the bucket is %q, want %q", said.Serves["bucket"], want)
}
// And what is the same for everybody is still the same for everybody.
if said.Serves["region"] != "eu-west" {
t.Errorf("the binding lost what the provider serves to all: %v", said.Serves)
}
given := storeGrants(t, out)
if len(given) != 1 {
t.Fatalf("the provider was told about %d consumer(s): %v", len(given), given)
}
if given[0].Derived["bucket"] != want {
t.Errorf("the provider was told the bucket is %v, and the consumer was told %q — "+
"the two ends disagree, which is the whole failure", given[0].Derived["bucket"], want)
}
// Only the per-consumer half. The region is the same for everyone and is already in the
// provider's own definition; repeating it here would be a copy to go stale.
if _, carried := given[0].Derived["region"]; carried {
t.Errorf("the provider was handed back what it already says for everyone: %v", given[0].Derived)
}
}
// Two consumers of one provider on one machine get two buckets, and neither gets the other's.
func TestTwoConsumersOfOneProviderGetTheirOwnDerivation(t *testing.T) {
r, err := Resolve(shelf(store(), files(), pics()),
[]string{"store", "files", "pics"}, reachable(), World{})
if err != nil {
t.Fatal(err)
}
out, err := r.Declaration(Rendering{Grants: []Grant{
{Provision: "s3-bucket", Consumer: "workstation", From: "files", Slug: "files",
Values: map[string]any{}, Sealed: "c2VhbGVk"},
{Provision: "s3-bucket", Consumer: "workstation", From: "pics", Slug: "pics",
Values: map[string]any{}, Sealed: "c2VhbGVk"},
}})
if err != nil {
t.Fatal(err)
}
forFiles := strings.ReplaceAll(ConsumerIdentity("workstation", IdentitySource("files", "files")), "_", "-")
forPics := strings.ReplaceAll(ConsumerIdentity("workstation", IdentitySource("pics", "pics")), "_", "-")
if forFiles == forPics {
t.Fatal("the two consumers were given the same identity; this test proves nothing")
}
if got := fileNamed(out, "files.env")["content"].(string); !strings.Contains(got, "BUCKET="+forFiles+"\n") {
t.Errorf("files was not given its own bucket:\n%s", got)
}
if got := fileNamed(out, "pics.env")["content"].(string); !strings.Contains(got, "BUCKET="+forPics+"\n") {
t.Errorf("pics was not given its own bucket:\n%s", got)
}
var buckets []any
for _, g := range storeGrants(t, out) {
buckets = append(buckets, g.Derived["bucket"])
}
if len(buckets) != 2 || buckets[0] == buckets[1] {
t.Errorf("the provider was told %v; it must be told one bucket per consumer", buckets)
}
}
// An operator may still set what the provider serves, and the mesh still derives the rest: the
// setting is laid on first, then the consumer's half is filled.
func TestASettingComposesWithADerivedValue(t *testing.T) {
r, err := Resolve(shelf(store(), files()), []string{"store", "files"}, reachable(), World{})
if err != nil {
t.Fatal(err)
}
out, err := r.Declaration(Rendering{
Settings: SettingsBy{"store": {{From: "the operator",
Values: map[string]any{"bucket": "team-${consumer:as:dns}"}}}},
Grants: []Grant{{Provision: "s3-bucket", Consumer: "workstation", From: "files", Slug: "files",
Values: map[string]any{}, Sealed: "c2VhbGVk"}},
})
if err != nil {
t.Fatal(err)
}
want := "team-" + strings.ReplaceAll(ConsumerIdentity("workstation", IdentitySource("files", "files")), "_", "-")
if got := fileNamed(out, "files.env")["content"].(string); !strings.Contains(got, "BUCKET="+want+"\n") {
t.Errorf("the operator's prefix did not survive the derivation:\n%s\nwant BUCKET=%s", got, want)
}
if given := storeGrants(t, out); given[0].Derived["bucket"] != want {
t.Errorf("the provider was told %v, the consumer %q", given[0].Derived["bucket"], want)
}
}
// A fact or an alphabet the mesh does not have is refused where the definition is, not where a
// consumer happens to be resolved — and the refusal says what may be said instead.
func TestAServedValueNamingSomethingTheMeshDoesNotHaveIsRefused(t *testing.T) {
for _, c := range []struct{ value, says string }{
{"${consumer:node}", "as"},
{"${consumer:as:punycode}", "dns"},
} {
m := store()
m.Serves["s3-bucket"]["bucket"] = c.value
raw, err := json.Marshal(m)
if err != nil {
t.Fatal(err)
}
_, err = ParseManifest(raw)
if err == nil {
t.Fatalf("%s was accepted", c.value)
}
if !strings.Contains(err.Error(), c.value) {
t.Errorf("the refusal of %s does not quote it: %v", c.value, err)
}
if !strings.Contains(err.Error(), c.says) {
t.Errorf("the refusal of %s does not say what may be said (%q): %v", c.value, c.says, err)
}
}
}
// `dns` is checked against an identity the mesh actually mints, not an invented string.
func TestTheDNSAlphabetIsTheMintedIdentityWithItsSeparatorRewritten(t *testing.T) {
as := ConsumerIdentity("anchor", IdentitySource("ncloud", "nextcloud"))
if err := CheckIdentity("anchor", IdentitySource("ncloud", "nextcloud")); err != nil {
t.Fatalf("the mesh would not mint this identity at all: %v", err)
}
label := asDNSLabel(as)
if strings.Contains(label, "_") {
t.Errorf("%q is not a DNS label", label)
}
if strings.ReplaceAll(label, "-", "_") != as {
t.Errorf("%q is not %q with its separator rewritten", label, as)
}
}
// The check that would have caught the one wrong instance: a consumer that writes the derived
// value into its own definition instead of asking for it is refused, whether it transcribed the
// right answer or a predecessor's.
func TestAConsumerThatTranscribesWhatItsProviderDerivesIsRefused(t *testing.T) {
as := ConsumerIdentity("workstation", IdentitySource("files", "files"))
transcribed := strings.ReplaceAll(as, "_", "-")
m := files()
m.Resources = []map[string]any{{
"id": "env", "type": "file", "path": "/var/lib/files/env", "mode": "0600",
// Exactly what the provider will create — correct today, and a copy of a rule that is
// not this module's.
"content": "BUCKET=" + transcribed + "\n",
}}
r, err := Resolve(shelf(store(), m), []string{"store", "files"}, reachable(), World{})
if err != nil {
t.Fatal(err)
}
_, err = r.Declaration(Rendering{Grants: []Grant{{
Provision: "s3-bucket", Consumer: "workstation", From: "files", Slug: "files",
Values: map[string]any{}, Sealed: "c2VhbGVk",
}}})
if err == nil {
t.Fatal("a definition holding its own copy of the provider's naming rule was accepted")
}
if !strings.Contains(err.Error(), "${bound:s3-bucket:bucket}") {
t.Errorf("the refusal does not say what to write instead: %v", err)
}
// And a constant the provider serves to everyone is not a transcription: repeating it is
// redundant, not wrong, and refusing it would be the mesh policing style.
m.Resources = []map[string]any{{
"id": "env", "type": "file", "path": "/var/lib/files/env", "mode": "0600",
"content": "REGION=eu-west\n",
}}
r, err = Resolve(shelf(store(), m), []string{"store", "files"}, reachable(), World{})
if err != nil {
t.Fatal(err)
}
if _, err := r.Declaration(Rendering{Grants: []Grant{{
Provision: "s3-bucket", Consumer: "workstation", From: "files", Slug: "files",
Values: map[string]any{}, Sealed: "c2VhbGVk",
}}}); err != nil {
t.Errorf("a value the provider serves to everyone was judged a transcription: %v", err)
}
}
func storeGrants(t *testing.T, out []map[string]any) []Contribution {
t.Helper()
for _, r := range out {
if r["path"] != "/var/lib/store/grants/mesh.json" {
continue
}
var parsed struct {
Given []Contribution `json:"given"`
}
if err := json.Unmarshal([]byte(r["content"].(string)), &parsed); err != nil {
t.Fatal(err)
}
return parsed.Given
}
t.Fatalf("the provider was given no contributions file: %v", out)
return nil
}
+5 -2
View File
@@ -102,8 +102,11 @@ func accountHomeOf(account, home string) string {
func machineInto(resource map[string]any, facts map[string]string, module string) error {
// Content, and now the path and owner too: a module that writes into a person's home names it
// with ${machine:account-home} and ${machine:account}, which it cannot know until assigned
// (novox/hq to-be 29), the same reason its content names ${machine:address}.
for _, field := range []string{"path", "owner", "content"} {
// (novox/hq to-be 29), the same reason its content names ${machine:address}. And the name a
// `user` shape sets the login shell of, and the user a user-scoped unit or a process runs as:
// the shell module makes the operator's account its holder's login shell, and the desktop's
// watchers run as that account (novox/hq ADR 0176, ADR 0177) — neither can name the person.
for _, field := range []string{"path", "owner", "content", "name", "user"} {
s, ok := resource[field].(string)
if !ok {
continue
+121 -1
View File
@@ -1360,6 +1360,10 @@ func ParseManifest(raw []byte) (Manifest, error) {
"%s serves %q to whoever requires it, and does not provide it", m.Module, to))
}
}
// A served value may be derived for the consumer it is served to (novox/hq ADR 0188). Read
// here, where the definition is, rather than when somebody first requires it: a rule that
// would be refused at the first consumer is wrong from the moment it is written.
problems = append(problems, CheckServes(m)...)
for to, where := range m.Binds {
if !placedOrAbsolute(where) {
problems = append(problems, fmt.Sprintf(
@@ -1507,6 +1511,13 @@ func ParseManifest(raw []byte) (Manifest, error) {
}
}
}
// **A scheduled step may hold this module's own containers still while it runs**
// (novox/hq ADR 0189). What the host judges is the declaration it receives — whether each
// id is a container placed on that machine; what belongs here is what only the definition
// shows: that the ids are this module's, that they are containers, and that the step is
// scheduled. A module naming a neighbour's container would be a module that can stop the
// mesh, and the manifest is where that is visible.
problems = append(problems, whileStoppedProblems(m, r, hasSchedule(r))...)
}
for name, own := range m.OwnSecrets {
if !placedOrAbsolute(own.Path) {
@@ -1667,6 +1678,7 @@ func ParseManifest(raw []byte) (Manifest, error) {
problems = append(problems, m.undeclaredMounts()...)
problems = append(problems, m.unknownDirRefs()...)
problems = append(problems, m.unknownAccessRefs()...)
problems = append(problems, m.jailProblems()...)
for i, r := range m.Resources {
id, _ := r["id"].(string)
@@ -1769,6 +1781,46 @@ func (m Manifest) MachineSide(port int) (at int, mayAssign bool) {
var facilitiesOf = map[string][]string{
// Both spellings: /var/run is a link to /run on every machine the mesh runs on.
"container-runtime": {"/var/run/docker.sock", "/run/docker.sock"},
// The virtualisation daemon's socket, for the lab (novox/hq ADR 0172): it raises machines there.
"virtualisation": {"/var/lib/incus/unix.socket"},
}
// jailProblems is every jail this module declares that the machine's intrusion prevention would
// refuse (novox/hq ADR 0179).
//
// **Because one bad pattern stops every jail, not its own.** fail2ban expands `<HOST>` into a named
// capture group, so a pattern naming it twice is a duplicate group name, and the daemon refuses the
// whole configuration and exits — the machine keeps no bans at all, for any jail, including the one
// watching its ssh. Caught live on the control node the day this was built, where a proxy's pattern
// matched two shapes of refusal in one line. A pattern matches one shape; several shapes are several
// patterns, one per line, as fail2ban's own filters are written.
func (m Manifest) jailProblems() []string {
var problems []string
seen := map[string]bool{}
for _, j := range m.Jails {
switch {
case strings.TrimSpace(j.Name) == "":
problems = append(problems, m.Module+" declares a jail with no name")
case seen[j.Name]:
problems = append(problems, m.Module+" declares two jails called "+strconv.Quote(j.Name))
}
seen[j.Name] = true
if strings.TrimSpace(j.Failregex) == "" {
problems = append(problems, m.Module+"'s jail "+strconv.Quote(j.Name)+" says nothing a failed attempt looks like")
}
for _, line := range strings.Split(j.Failregex, "\n") {
if strings.TrimSpace(line) == "" {
continue
}
if n := strings.Count(line, "<HOST>"); n > 1 {
problems = append(problems, fmt.Sprintf("%s's jail %s names <HOST> %d times in one pattern; "+
"fail2ban reads it as one capture group and refuses the whole configuration, so the machine "+
"keeps no bans at all — write one pattern per shape, each naming <HOST> once",
m.Module, strconv.Quote(j.Name), n))
}
}
}
return problems
}
// undeclaredMounts is every bind-mount source no declaration covers — see the check above.
@@ -1812,7 +1864,7 @@ func (m Manifest) undeclaredMounts() []string {
}
// The file a filter module's rule set is written to is declared by `filtering.into`: the mesh
// writes it, the module loads it, and the module's runtime may read it back to reload the
// mesh's own table (novox/hq ADR 0169).
// mesh's own table (novox/hq ADR 0170).
if m.Filtering != nil {
claim(m.Filtering.Into)
}
@@ -1997,3 +2049,71 @@ func (o OwnSecrets) Paths() map[string]string {
// InstancesInterchangeable is the one value of a definition's `instances`: the module is the same
// on every machine, so any instance may answer for the module.
const InstancesInterchangeable = "interchangeable"
// WhileStopped is the resource key naming the containers a scheduled step holds still while it
// runs (novox/hq ADR 0189). Carried to the host unchanged, like `schedule`.
const WhileStopped = "while-stopped"
// hasSchedule is whether a resource declares a cadence, as a string.
func hasSchedule(r map[string]any) bool {
s, _ := r["schedule"].(string)
return s != ""
}
// whileStoppedProblems judges one container's maintenance window against its own definition
// (novox/hq ADR 0189).
//
// Three things the manifest is the only place to see: that the step is scheduled (a one-time
// offline job says *before* rather than *instead of* — at apply the host already has a window,
// because the declaration is applied in order and a run-once step gates what follows); that every
// id it names is **this module's own** container; and that it does not name itself.
//
// The host checks the fourth — that the container is actually placed on that machine — because
// that is a fact about the declaration and not about the definition.
func whileStoppedProblems(m Manifest, r map[string]any, scheduled bool) []string {
raw, present := r[WhileStopped]
if !present {
return nil
}
ids, ok := raw.([]any)
if !ok {
return []string{fmt.Sprintf(
"%s declares %s on %v as a %T; it is a list of this module's container ids",
m.Module, WhileStopped, r["id"], raw)}
}
var problems []string
if len(ids) > 0 && !scheduled {
problems = append(problems, fmt.Sprintf(
"%s declares %s on %v, which has no schedule. A maintenance window is for a recurring "+
"step: at apply the mesh already has one, because a run-once step gates what is "+
"declared after it (novox/hq ADR 0189)", m.Module, WhileStopped, r["id"]))
}
containers := map[string]bool{}
for _, own := range m.Resources {
if fmt.Sprint(own["type"]) == "container" {
containers[fmt.Sprint(own["id"])] = true
}
}
for _, each := range ids {
id, ok := each.(string)
if !ok {
problems = append(problems, fmt.Sprintf(
"%s declares %s on %v naming a %T; each entry is a container's id",
m.Module, WhileStopped, r["id"], each))
continue
}
if id == fmt.Sprint(r["id"]) {
problems = append(problems, fmt.Sprintf(
"%s declares %s on %v naming itself", m.Module, WhileStopped, r["id"]))
continue
}
if !containers[id] {
problems = append(problems, fmt.Sprintf(
"%s declares %s on %v naming %q, which is not a container this module declares. "+
"A step may hold still its own module's containers and nobody else's — one "+
"that could quiesce a neighbour could stop the mesh",
m.Module, WhileStopped, r["id"], id))
}
}
return problems
}
@@ -0,0 +1,60 @@
package catalogue
import (
"strings"
"testing"
)
// A `user` shape and a user-scoped unit name the operator account the way a home file does
// (novox/hq ADR 0176, ADR 0177): with ${machine:account}, resolved when the module is assigned.
func TestAUserShapeAndAUserScopedUnitNameTheAccount(t *testing.T) {
facts := map[string]string{"account": "ops", "account-home": "/home/ops"}
login := map[string]any{"type": "user", "id": "login", "name": "${machine:account}", "shell": "/usr/bin/zsh"}
if err := machineInto(login, facts, "zsh"); err != nil {
t.Fatal(err)
}
if login["name"] != "ops" {
t.Fatalf("the user shape did not learn the account: %v", login["name"])
}
watcher := map[string]any{"type": "service", "id": "watcher", "unit": "i3-reload-watcher.service",
"scope": "user", "user": "${machine:account}"}
if err := machineInto(watcher, facts, "i3"); err != nil {
t.Fatal(err)
}
if watcher["user"] != "ops" {
t.Fatalf("the user-scoped unit did not learn the account: %v", watcher["user"])
}
// A machine with no operator account refuses rather than writing the literal.
err := machineInto(map[string]any{"type": "user", "id": "login", "name": "${machine:account}"},
map[string]string{"address": "10.0.0.1"}, "zsh")
if err == nil || !strings.Contains(err.Error(), "${machine:account}") {
t.Fatalf("a user shape on a machine with no account was not refused by name: %v", err)
}
}
// The service manager is a seat of the mesh's own with the unit verbs as its contract (novox/hq
// ADR 0177): every verb described, with a schema, taking a scope.
func TestTheServiceManagerSeatServesTheUnitVerbs(t *testing.T) {
seat, ok := SeatNamed("node-service-manager")
if !ok {
t.Fatal("node-service-manager is not a seat the mesh defines")
}
if seat.Scope != ScopeNode {
t.Fatalf("the service manager is a role each machine has once, and the seat is %s-scoped", seat.Scope)
}
want := []string{"units", "status", "start", "stop", "restart", "enable", "disable", "journal"}
var got []string
for _, v := range seat.Serves {
got = append(got, v.Name)
if v.Description == "" || v.Input == nil {
t.Fatalf("%s is promised without a description or a schema", v.Name)
}
props, _ := v.Input["properties"].(map[string]any)
if _, has := props["scope"]; !has {
t.Fatalf("%s takes no scope, and a user unit could not be asked for", v.Name)
}
}
if strings.Join(got, ",") != strings.Join(want, ",") {
t.Fatalf("the seat serves %v, not %v", got, want)
}
}
+16 -6
View File
@@ -48,7 +48,9 @@ func TestTheResolverForwardsToFixedUpstreamsAndNeverReadsResolvConf(t *testing.T
}
for _, want := range []string{
"\nno-resolv\n", "\nserver=1.1.1.1\n", "\nserver=8.8.8.8\n",
"\nlisten-address=127.0.0.1\n", "\nlisten-address=${machine:address}\n", "\nbind-dynamic\n",
// Loopback is the mesh-wide setting's default; a machine answering its own LAN adds its
// address there (novox/hq issue 198).
"\nlisten-address=${setting:listen-addresses}\n", "\nlisten-address=${machine:address}\n", "\nbind-dynamic\n",
"\ndomain-needed\n", "\nbogus-priv\n",
"\nconf-file=" + m.Facts["node-zones"].Path + "\n",
} {
@@ -113,7 +115,8 @@ func TestTheResolverAndWhatAsksItComposeOnOneMachine(t *testing.T) {
// issue 111) — the resolver's zones read only the second, and in this scenario the two
// happen to be the same map, since nothing routed is part of it.
Names: twoMachines, Machines: twoMachines, Suffix: "internal",
Needed: map[string]map[string]string{"dnsmasq": {"broker": "sealed"}},
Needed: map[string]map[string]string{"dnsmasq": {"broker": "sealed"}},
Settings: SettingsBy{"dnsmasq": {{From: "the mesh", Values: map[string]any{"listen-addresses": "127.0.0.1"}}}},
})
if err != nil {
t.Fatal(err)
@@ -207,9 +210,16 @@ func TestTheResolverOnAMachineOffTheNetworkIsRefused(t *testing.T) {
if err != nil {
t.Fatal(err)
}
_, err = got.Declaration(Rendering{Names: twoMachines, Suffix: "internal",
Needed: map[string]map[string]string{"dnsmasq": {"broker": "sealed"}}})
if err == nil || !strings.Contains(err.Error(), "${machine:address}") {
t.Fatalf("a machine off the network was composed a resolver, or refused for another reason: %v", err)
// Left out of the declaration and said, rather than composed listening nowhere: a module that
// cannot compose on a machine is kept as it is there, with the reason (hq ADR 0163).
composed, err := got.Compose(Rendering{Names: twoMachines, Suffix: "internal",
Needed: map[string]map[string]string{"dnsmasq": {"broker": "sealed"}},
Settings: SettingsBy{"dnsmasq": {{From: "the mesh", Values: map[string]any{"listen-addresses": "127.0.0.1"}}}}})
if err == nil && !strings.Contains(composed.LeftOut["dnsmasq"], "${machine:address}") {
t.Fatalf("a machine off the network was composed a resolver, or left out for another reason: %v",
composed.LeftOut)
}
if err != nil && !strings.Contains(err.Error(), "${machine:address}") {
t.Fatalf("a machine off the network was refused for another reason: %v", err)
}
}
+57 -2
View File
@@ -99,8 +99,24 @@ var defaultSeats = []Seat{
{Name: "mesh-build-machine", Scope: ScopeMesh,
Accepts: []string{"build"}, Emits: []string{"started", "built", "log.*"}, Decision: "novox/hq ADR 0121"},
{Name: "node-dns-resolver", Scope: ScopeNode, Decision: "novox/hq ADR 0121"},
{Name: "node-intrusion-prevention", Scope: ScopeNode, Decision: "novox/hq ADR 0121"},
// The packet filter's verbs (novox/hq ADR 0169): what a person asks a machine's filter whatever
// The intrusion prevention's verbs (novox/hq ADR 0179): what a person asks a machine's ban list
// whatever keeps it — who is banned and why, ban one address, let one go. Every holder serves all
// four; the jails themselves are composed from the modules the machine runs (to-be 31).
{Name: "node-intrusion-prevention", Scope: ScopeNode, Decision: "novox/hq ADR 0121",
Serves: []Verb{
{Name: "status", Description: "Every jail on this machine with how many it is watching and " +
"holding now, and the totals since the jail started; one jail's detail when named.",
Input: schema(map[string]string{"jail": "one jail (optional)"}, nil)},
{Name: "banned", Description: "Every address banned on this machine right now, with the jail " +
"that holds it and when the ban ends.",
Input: schema(map[string]string{"jail": "one jail (optional)"}, nil)},
{Name: "ban", Description: "Ban one address in one jail now, for the jail's ban time — an " +
"operator's act on the live ban list, which the mesh never writes itself.",
Input: schema(map[string]string{"ip": "the address", "jail": "the jail to hold it"}, []string{"ip", "jail"})},
{Name: "unban", Description: "Let one address go, from one jail or from every jail when none is named.",
Input: schema(map[string]string{"ip": "the address", "jail": "one jail (optional)"}, []string{"ip"})},
}},
// The packet filter's verbs (novox/hq ADR 0170): what a person asks a machine's filter whatever
// filter answers — the rules as enforced, reload the mesh's own, remove one thing the mesh did
// not write. Every holder serves all three; what differs by filter is the holder's own tools.
{Name: "node-packet-filter", Scope: ScopeNode, Decision: "novox/hq ADR 0121",
@@ -119,6 +135,12 @@ var defaultSeats = []Seat{
"active found firewall's chains. An operator's act, by name, never a flush.",
Input: schema(map[string]string{"where": "the rule set, as `node show` lists it"}, []string{"where"})},
}},
// The machine's service manager (novox/hq ADR 0177). The host applies every declared unit,
// system or user scope; the holder answers questions and operator acts about them, each verb
// taking the unit and an optional scope. The holder runs nothing of its own: its verbs are
// served by the node tools runtime (ADR 0175).
{Name: "node-service-manager", Scope: ScopeNode, Decision: "novox/hq ADR 0177",
Serves: serviceManagerVerbs()},
// Deferred (novox/hq ADR 0121): renaming to mesh-private-network is a scope + server/client
// model change, not a rename, so it stays until that is built.
{Name: "the-private-network", Scope: ScopeNode, Decision: "novox/hq ADR 0110"},
@@ -392,3 +414,36 @@ func SeatsWithAProtocol() []Seat {
}
return out
}
// serviceManagerVerbs is the contract every holder of node-service-manager serves (novox/hq ADR
// 0177): the units on the machine in both scopes, read and acted on by name. Every verb takes an
// optional scope — "system" when absent, "user" for the operator account's own manager — so a
// caller asks for a user unit the way it asks for a system one.
func serviceManagerVerbs() []Verb {
scoped := func(more map[string]string, required []string) map[string]any {
props := map[string]string{"scope": "\"system\" (the default) or \"user\": the operator account's own manager"}
for k, v := range more {
props[k] = v
}
return schema(props, required)
}
unit := map[string]string{"unit": "the unit's name, as the service manager knows it"}
return []Verb{
{Name: "units", Description: "The units the service manager knows in a scope, each with its load, active and sub state; narrowed to a pattern when asked.",
Input: scoped(map[string]string{"pattern": "a glob the unit's name must match (optional)"}, nil)},
{Name: "status", Description: "One unit as the service manager sees it now: its states, whether it starts at boot, its main process, and whether the mesh declares it.",
Input: scoped(unit, []string{"unit"})},
{Name: "start", Description: "Start one unit. For a unit the mesh declares, the answer says the host will restore what its declaration says at the next apply.",
Input: scoped(unit, []string{"unit"})},
{Name: "stop", Description: "Stop one unit; for a mesh-declared unit the answer says the host will restore its declared state.",
Input: scoped(unit, []string{"unit"})},
{Name: "restart", Description: "Restart one unit.",
Input: scoped(unit, []string{"unit"})},
{Name: "enable", Description: "Make one unit start at boot (or at the account's login, in user scope).",
Input: scoped(unit, []string{"unit"})},
{Name: "disable", Description: "Stop one unit starting at boot (or at login, in user scope).",
Input: scoped(unit, []string{"unit"})},
{Name: "journal", Description: "The last lines of one unit's journal.",
Input: scoped(map[string]string{"unit": unit["unit"], "lines": "how many lines from the end (default 100)"}, []string{"unit"})},
}
}
+3 -2
View File
@@ -44,8 +44,9 @@ func TestTheSeatsAreAClosedSetAndEachNamesItsDecision(t *testing.T) {
delivered[s.Delivers] = s.Name
}
}
if len(Seats()) != 15 {
t.Errorf("the mesh defines %d seats rather than 14; the set is closed, so a change here is "+
// Sixteen since node-service-manager (novox/hq ADR 0177).
if len(Seats()) != 16 {
t.Errorf("the mesh defines %d seats rather than 16; the set is closed, so a change here is "+
"a decision (novox/hq ADR 0110): %s", len(Seats()), seatNames())
}
}
+7
View File
@@ -145,6 +145,13 @@ var ControllerVerbs = []Verb{
"node": "one machine; the whole mesh when absent",
"clear": "\"true\" to remove the layer instead of setting it",
}, []string{"module"})},
{Name: "command", Description: "Run one command line of the controller's own, as you would type it at its " +
"shell — `node account g14 jochen`, `node show ace`, `module list` — and answer what it printed. The " +
"generic verb beside the named ones (novox/hq ADR 0154): everything the binary can do, without a verb " +
"per command. Any node may call any tool (ADR 0175), so nothing is held back here.",
Input: schema(map[string]string{
"command": "the command line, as the controller's binary takes it; quotes group a word with spaces",
}, []string{"command"})},
{Name: "build", Description: "Have the build machine build a repository. Answers at once with the build's id: " +
"`builds` with that id follows it line by line, and the module is registered when the outcome comes.",
Input: schema(map[string]string{
+89
View File
@@ -0,0 +1,89 @@
package catalogue
import (
"encoding/json"
"strings"
"testing"
)
// A scheduled step may hold its module's own containers still while it runs (novox/hq ADR 0189).
//
// The host judges what it receives — whether each id is a container on that machine. What the
// definition is the only place to see is judged here, near whoever wrote it.
func aStoreManifest(step map[string]any) []byte {
m := map[string]any{
"module": "distribution", "version": "1",
"resources": []any{
map[string]any{"id": "store", "type": "container", "name": "mesh-registry",
"image": "registry@sha256:" + strings.Repeat("a", 64)},
step,
},
}
raw, _ := json.Marshal(m)
return raw
}
func TestAMaintenanceWindowOnItsOwnModulesContainerIsAccepted(t *testing.T) {
raw := aStoreManifest(map[string]any{
"id": "collect", "type": "container", "name": "mesh-registry-collect",
"image": "registry@sha256:" + strings.Repeat("a", 64),
"schedule": "30 3 * * *", "while-stopped": []any{"store"},
})
if _, err := ParseManifest(raw); err != nil {
t.Fatalf("a step holding its own module's container still was refused: %v", err)
}
}
func TestAMaintenanceWindowIsRefusedWhereTheDefinitionShowsItCannotMean(t *testing.T) {
for _, c := range []struct {
name string
step map[string]any
says string
}{
{
"on a step with no schedule",
map[string]any{"id": "collect", "type": "container", "name": "c",
"image": "registry@sha256:" + strings.Repeat("a", 64),
"while-stopped": []any{"store"}},
"gates what is declared after it",
},
{
"on a run-once step, which already has order",
map[string]any{"id": "collect", "type": "container", "name": "c",
"image": "registry@sha256:" + strings.Repeat("a", 64),
"run-once": true, "while-stopped": []any{"store"}},
"A maintenance window is for a recurring step",
},
{
"naming a container this module does not declare",
map[string]any{"id": "collect", "type": "container", "name": "c",
"image": "registry@sha256:" + strings.Repeat("a", 64),
"schedule": "30 3 * * *", "while-stopped": []any{"the-broker"}},
"could quiesce a neighbour could stop the mesh",
},
{
"naming itself",
map[string]any{"id": "collect", "type": "container", "name": "c",
"image": "registry@sha256:" + strings.Repeat("a", 64),
"schedule": "30 3 * * *", "while-stopped": []any{"collect"}},
"naming itself",
},
{
"written as something that is not a list",
map[string]any{"id": "collect", "type": "container", "name": "c",
"image": "registry@sha256:" + strings.Repeat("a", 64),
"schedule": "30 3 * * *", "while-stopped": "store"},
"a list of this module's container ids",
},
} {
_, err := ParseManifest(aStoreManifest(c.step))
if err == nil {
t.Errorf("%s was accepted", c.name)
continue
}
if !strings.Contains(err.Error(), c.says) {
t.Errorf("%s: the refusal does not say %q:\n%v", c.name, c.says, err)
}
}
}
+242
View File
@@ -0,0 +1,242 @@
package inventory
import (
"context"
"encoding/json"
"strings"
)
// What the artifact store keeps, and what it may let go (novox/hq ADR 0189, issue 108).
//
// The store has never collected anything: every build pushes another layer set and nothing has
// ever removed one. The registry's own answer — collect what no tag names — is wrong here, because
// the mesh pushes each artifact under one moving tag and pins machines by digest, so every build
// but the newest is untagged and some machine may still be running it.
//
// **So the mesh decides, from its own records, and it never has to look in the store to do it.**
// It has never put anything there it did not record, which means every digest it could remove is
// already in a build row. A digest the mesh did not record making is therefore never named here —
// not as a safety margin but as the rule restated, and it is what keeps the sweep away from the
// images genesis pushed before any record existed (04-ISSUES/102, F4).
// KeptBuilds is how many successful builds of each module keep their artifacts, counting the
// newest. The newest is what the mesh hands a machine now; the four behind it are how far back a
// release that turns out wrong can be taken.
const KeptBuilds = 5
// ToCollect is every artifact the mesh made, no longer keeps, and has not already collected.
//
// Three reasons an artifact stays, and nothing else is a reason:
//
// - **a definition names it** — the reference appears in a module's recorded manifest, which is
// what the mesh would hand a machine now. No age limit: this is the floor;
// - **the mesh can still go back to it** — it is an artifact of one of the KeptBuilds most
// recent successful builds of its module;
// - it was already collected, in which case there is nothing left to do.
//
// Returned in a stated order so two runs over the same records ask for the same things in the
// same sequence, which is what makes a failed sweep safe to simply run again.
func (i *Inventory) ToCollect(ctx context.Context) ([]string, error) {
keep, err := i.keptReferences(ctx)
if err != nil {
return nil, err
}
rows, err := i.store.Pool().Query(ctx,
// Every artifact of every successful build, oldest first, minus what has already been
// collected. A failed build published nothing, so it names nothing to remove.
`select b.made
from build b
where b.failed = '' and b.module is not null and b.module <> ''
order by b.at asc, b.id asc`)
if err != nil {
return nil, err
}
defer rows.Close()
collected, err := i.alreadyCollected(ctx)
if err != nil {
return nil, err
}
seen := map[string]bool{}
var out []string
for rows.Next() {
var raw []byte
if err := rows.Scan(&raw); err != nil {
return nil, err
}
var made []Artifact
if err := json.Unmarshal(raw, &made); err != nil {
// One unreadable record must not stop the rest being collected — and an artifact this
// row named is simply not offered, which errs toward keeping.
continue
}
for _, a := range made {
if a.Reference == "" || keep[a.Reference] || collected[a.Reference] || seen[a.Reference] {
continue
}
seen[a.Reference] = true
out = append(out, a.Reference)
}
}
return out, rows.Err()
}
// keptReferences is every artifact reference the mesh still keeps, for either of the two reasons.
func (i *Inventory) keptReferences(ctx context.Context) (map[string]bool, error) {
keep := map[string]bool{}
// **Whatever a definition the mesh holds names.** Read as text rather than by walking the
// resource shapes: a reference may be a container's image, a bundle's source, or a field some
// later kind of resource grows, and what matters is only whether the mesh could hand this
// string to a machine. A manifest that mentions it is a manifest that might.
manifests, err := i.store.Pool().Query(ctx, `select manifest::text from module where manifest is not null`)
if err != nil {
return nil, err
}
defer manifests.Close()
var named []string
for manifests.Next() {
var text string
if err := manifests.Scan(&text); err != nil {
return nil, err
}
named = append(named, text)
}
if err := manifests.Err(); err != nil {
return nil, err
}
// The KeptBuilds most recent successful builds of each module, whole.
recent, err := i.store.Pool().Query(ctx,
`select made from (
select made, row_number() over (partition by module order by at desc, id desc) as back
from build
where failed = '' and module is not null and module <> ''
) ranked where back <= $1`, KeptBuilds)
if err != nil {
return nil, err
}
defer recent.Close()
for recent.Next() {
var raw []byte
if err := recent.Scan(&raw); err != nil {
return nil, err
}
var made []Artifact
if err := json.Unmarshal(raw, &made); err != nil {
continue
}
for _, a := range made {
if a.Reference != "" {
keep[a.Reference] = true
}
}
}
if err := recent.Err(); err != nil {
return nil, err
}
// And anything a manifest mentions. Done after the recent set so the scan runs over the
// candidates rather than over every reference ever recorded: a manifest holds a reference
// composed with the store's address or kept bare, so the search is for the digest within it.
if len(named) > 0 {
all, err := i.everyReferenceMade(ctx)
if err != nil {
return nil, err
}
for _, reference := range all {
if keep[reference] {
continue
}
digest := digestIn(reference)
if digest == "" {
// Not something the store holds by digest; nothing here can speak for it, so it
// is kept rather than guessed about.
keep[reference] = true
continue
}
for _, text := range named {
if strings.Contains(text, digest) {
keep[reference] = true
break
}
}
}
}
return keep, nil
}
// everyReferenceMade is every artifact reference any successful build recorded.
func (i *Inventory) everyReferenceMade(ctx context.Context) ([]string, error) {
rows, err := i.store.Pool().Query(ctx,
`select made from build where failed = '' and module is not null and module <> ''`)
if err != nil {
return nil, err
}
defer rows.Close()
seen := map[string]bool{}
var out []string
for rows.Next() {
var raw []byte
if err := rows.Scan(&raw); err != nil {
return nil, err
}
var made []Artifact
if err := json.Unmarshal(raw, &made); err != nil {
continue
}
for _, a := range made {
if a.Reference == "" || seen[a.Reference] {
continue
}
seen[a.Reference] = true
out = append(out, a.Reference)
}
}
return out, rows.Err()
}
// digestIn is the `sha256:<hex>` a reference names, empty when it names none.
func digestIn(reference string) string {
for _, marker := range []string{"@sha256:", "/sha256:"} {
if _, after, ok := strings.Cut(reference, marker); ok {
return "sha256:" + after
}
}
return ""
}
// alreadyCollected is what the store has already been asked to let go.
func (i *Inventory) alreadyCollected(ctx context.Context) (map[string]bool, error) {
rows, err := i.store.Pool().Query(ctx, `select reference from artifact_collected`)
if err != nil {
return nil, err
}
defer rows.Close()
out := map[string]bool{}
for rows.Next() {
var reference string
if err := rows.Scan(&reference); err != nil {
return nil, err
}
out[reference] = true
}
return out, rows.Err()
}
// MarkCollected records that the store no longer holds these.
//
// **A store that answered "not found" is recorded too.** The outcome wanted is that the artifact
// is gone, and it is; retrying it every sweep for ever is the failure this table exists to
// prevent. Only a store that could not be reached, or refused, leaves a reference unmarked — and
// then the next sweep asks again, which is what should happen.
func (i *Inventory) MarkCollected(ctx context.Context, references []string) error {
for _, reference := range references {
if _, err := i.store.Pool().Exec(ctx,
`insert into artifact_collected (reference) values ($1) on conflict (reference) do nothing`,
reference); err != nil {
return err
}
}
return nil
}
+147
View File
@@ -0,0 +1,147 @@
package inventory
import (
"context"
"fmt"
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
)
// What the store keeps, and what it may let go (novox/hq ADR 0189, issue 108).
//
// The store has collected nothing since it was raised, and the registry's own answer — collect
// what no tag names — would delete images machines are running, because the mesh pushes under one
// moving tag and pins by digest. So the rule is the mesh's, read from its own records, and these
// are the three reasons an artifact stays and the one reason it goes.
// ref is an artifact reference as the mesh records one.
func ref(module, artifact string, n int) string {
return fmt.Sprintf("%s%s/%s@sha256:%064x", catalogue.ArtifactStoreScheme, module, artifact, n)
}
// built records one successful build of a module publishing one image.
func built(t *testing.T, inv *Inventory, id, module string, n int) string {
t.Helper()
reference := ref(module, "app", n)
b := aBuild(id, module, "")
b.Made = []Artifact{{Name: "app", Kind: "image", Reference: reference}}
if err := inv.RecordBuild(context.Background(), b); err != nil {
t.Fatal(err)
}
return reference
}
func TestTheStoreKeepsTheRecentBuildsAndLetsGoOfTheRest(t *testing.T) {
inv := fresh(t)
ctx := context.Background()
// Eight builds of one module, oldest first. Five are kept — the newest, and the four a
// release that turns out wrong can be taken back to.
var made []string
for i := 1; i <= 8; i++ {
made = append(made, built(t, inv, fmt.Sprintf("b%02d", i), "web", i))
}
go_, err := inv.ToCollect(ctx)
if err != nil {
t.Fatal(err)
}
want := made[:3] // the three oldest
if len(go_) != len(want) {
t.Fatalf("offered %v to collect; want the %d oldest of %d", go_, len(want), len(made))
}
for i := range want {
if go_[i] != want[i] {
t.Fatalf("offered %v; want %v — and in that order, so a failed sweep is safe to run again",
go_, want)
}
}
}
func TestADefinitionNamingAnArtifactKeepsItHoweverOldItIs(t *testing.T) {
// The floor: no age limit. A module recorded at an older commit still names what the mesh
// would hand a machine now, and that is what must not be collected out from under it.
inv := fresh(t)
ctx := context.Background()
var made []string
for i := 1; i <= 8; i++ {
made = append(made, built(t, inv, fmt.Sprintf("b%02d", i), "web", i))
}
oldest := made[0]
// A definition the mesh holds, whose container runs that oldest image.
m := catalogue.Manifest{Module: "web", Version: "1", Resources: []map[string]any{{
"id": "app", "type": "container", "name": "web", "image": oldest,
}}}
if err := inv.RegisterModule(ctx, m, Source{Repository: "https://forge.invalid/web.git"}); err != nil {
t.Fatal(err)
}
go_, err := inv.ToCollect(ctx)
if err != nil {
t.Fatal(err)
}
for _, reference := range go_ {
if reference == oldest {
t.Fatalf("the mesh offered to collect %s, which a definition it holds names", oldest)
}
}
if len(go_) != 2 {
t.Fatalf("offered %v; want the two oldest that nothing names", go_)
}
}
func TestWhatHasBeenCollectedIsNotOfferedAgain(t *testing.T) {
// Without this the sweep reissues a delete for every artifact it has ever collected, every
// time it runs, for ever — a number of requests that grows with the mesh's whole history.
inv := fresh(t)
ctx := context.Background()
for i := 1; i <= 7; i++ {
built(t, inv, fmt.Sprintf("b%02d", i), "web", i)
}
first, err := inv.ToCollect(ctx)
if err != nil {
t.Fatal(err)
}
if len(first) != 2 {
t.Fatalf("offered %v, want two", first)
}
if err := inv.MarkCollected(ctx, first); err != nil {
t.Fatal(err)
}
again, err := inv.ToCollect(ctx)
if err != nil {
t.Fatal(err)
}
if len(again) != 0 {
t.Fatalf("offered %v again after collecting it", again)
}
}
func TestAFailedBuildNamesNothingToCollectAndEachModuleIsCountedOnItsOwn(t *testing.T) {
inv := fresh(t)
ctx := context.Background()
// A failed build published nothing, so it is neither kept nor collected — and it must not
// count against the module's five.
for i := 1; i <= 6; i++ {
built(t, inv, fmt.Sprintf("w%02d", i), "web", i)
}
if err := inv.RecordBuild(ctx, aBuild("w99", "web", "the recipe would not build")); err != nil {
t.Fatal(err)
}
// And a second module with three builds keeps all three: five each, not five between them.
for i := 1; i <= 3; i++ {
built(t, inv, fmt.Sprintf("d%02d", i), "db", 100+i)
}
go_, err := inv.ToCollect(ctx)
if err != nil {
t.Fatal(err)
}
if len(go_) != 1 || go_[0] != ref("web", "app", 1) {
t.Fatalf("offered %v; want only web's oldest — db's three are all within its five", go_)
}
}
@@ -0,0 +1,23 @@
-- What the artifact store no longer keeps (novox/hq ADR 0189, issue 108).
--
-- The mesh removes from its store only what it put there and can account for: every digest it
-- could remove is already in a build record, so the sweep reads its own records rather than
-- enumerating the store. What it does not get from those records is whether it has already
-- removed something -- `build.made` says what that build published, for ever, which is history
-- and not an index of what is on disk.
--
-- Without this the sweep would reissue a delete for every artifact it has ever collected, every
-- time it runs, and each one would answer 404 -- a number of requests that grows with the mesh's
-- whole history and never shrinks.
--
-- Keyed by the reference as the mesh records it (`artifact-store://<module>/<artifact>@sha256:…`),
-- because that is the identity the record uses everywhere else. Not a foreign key to build: two
-- builds can publish the same digest (the same source built twice produces the same bytes), and
-- what is collected is the artifact, not the attempt that made it.
create table artifact_collected (
reference text primary key,
-- When the store answered. Kept so a reader of an old build record can tell "this artifact is
-- gone" from "this artifact was never there", which are different kinds of surprise.
at timestamptz not null default now()
);