Compare commits

..
Author SHA1 Message Date
mesh-admin f6685ed22d Merge pull request 'An assignment places a module's directories and its accesses (hq 153)' (#176) from feat/153-an-assignment-places-directories-and-accesses into main 2026-09-30 22:03:33 +00:00
jschoubben 52c18f7a45 The path-preservation proof resolves access ids to their default paths too
An access named by id (issue 153) resolves to the path the definition still carries when the
assignment says nothing, and the proof compares that — the same rule as a placed directory.
2026-10-01 00:01:41 +02:00
jschoubben fa7415fcd0 Merge main into the branch: the assignment's placement sits beside the mesh's own place (issue 174) 2026-09-30 23:47:09 +02:00
mesh-admin f8947a806d Merge pull request 'The controller's own manifest names its paths for one more release' (#177) from fix/the-controllers-own-manifest-waits-a-release into main 2026-09-30 21:15:10 +00:00
jschoubben b98e503a61 The controller's own manifest names its paths for one more release
A manifest word ships one release after the code that reads it. The merged manifest already said
`place: "mesh"`, and the running controller, which does not know the word, refused its own build
result — so the controller that knows it could never be built. The literal paths come back here;
the conversion follows once this release runs.
2026-09-30 23:11:36 +02:00
jschoubben 5b832918df Merge pull request 'A setting reaches only what declares it, the mesh places its own files, registration refuses a name (issues 173, 174, ADR 0155)' (#175) from feat/the-mesh-places-its-own-files into main 2026-09-30 20:50:59 +00:00
jschoubben 17bbcc1596 An assignment places a module's directories and its accesses (hq 153)
A definition names no host path (ADR 0112); an adopted machine keeps its
data where the predecessor put it. Two settings, validated like endpoints:

  places:   {<directory id>: <path> | {path, owner}}
  accesses: {<access id>: <path>}

An access may now be declared by id (`{"id": "series", "mode": "read-write"}`)
and named in mounts, env and content as ${access:<id>}; the assignment
says where it is on this node, and an access nobody placed is refused by
name. A definition still carrying a path keeps it as the default the
assignment replaces. A placed directory takes the assignment's owner
where it says one. Resolved in composition, so the host receives paths
and owners exactly as before.
2026-09-30 22:42:50 +02:00
jschoubben 29be985c23 A served value or a contribution's may be the operator's: ${setting:…} fills there too, refused by name when unset
Issue 173's rule needs it: a mail provider serves its domain and an identity provider its issuer,
and neither is the definition's to state. Declared as ${setting:<key>}, filled from the layers after
the overrides; a key so asked for is not stray.
2026-09-30 22:34:43 +02:00
jschoubben 05d977666a A setting reaches only what declares it, the mesh places its own files, registration refuses a name
Three of novox/hq's group-4 leftovers, one branch.

Issue 173: a module's settings reached every route it contributed, every database it asked for and
every served fact its consumers read — a mail server's site name arrived at the proxy as a route
fact. A setting now overrides a key a contribution or served fact declares and adds none; a file
still merges any key, and a key nothing takes is named as stray instead of dropped silently.

Issue 174: the mesh's own files for a module — its bus credential, its merged config, its bindings —
were placed by the definition under /var/lib/mesh/<module>, 232 host paths in 50 definitions. A
directory may now say `place: "mesh"` and resolves to <root>/mesh/<module>; a directory beneath a
placed one may state its path as `${dir:<id>}/<rest>` and moves with it. The proof test resolves
both catalogues and compares: 48 definitions, no path moved. The controller's own manifest is
converted here; the catalogue in mesh-catalog.

ADR 0155: the installation check moves to registration. `module add` and a build's result both
refuse a definition that names an installation, in the check's words, with the way out; the build
stays recorded.
2026-09-30 22:29:28 +02:00
jschoubben e871991495 Merge pull request 'The catalogue-wide checks run against the sibling checkout by default' (#174) from fix/the-catalogue-checks-run-by-default into main 2026-09-30 20:10:44 +00:00
jschoubben f9e19814eb The catalogue-wide checks run against the checkout beside this one by default
A check that only ran when somebody remembered a variable was a check nobody ran (novox/hq issue
134). MESH_CATALOGUE still overrides; the checks skip only when no catalogue can be found.
2026-09-30 22:10:42 +02:00
jschoubben af31315a5f Merge pull request 'The controller takes one announcement at a time' (#173) from fix/one-announcement-at-a-time into main 2026-09-30 19:37:56 +00:00
jschoubben 474f68b34c The controller takes one announcement at a time
A merge's handler builds for minutes and keeps its own delivery alive; the announcements handed over
behind it timed out on the client and came back, and a merge that came back rebuilt what it had just
built, five times over (novox/hq issue 175). MaxAckPending 1 on the events consumer: the server holds
the rest.
2026-09-30 21:37:53 +02:00
jschoubben 1a724f20fe Merge pull request 'The seat rename survives a row seeded under the new name' (#172) from fix/the-seat-rename-survives-a-seeded-row into main 2026-09-30 19:34:37 +00:00
jschoubben 0da0bb2157 The seat rename survives a row seeded under the new name
A controller whose defaults carry the new name seeds it before the migration runs, and the first
form renamed into a duplicate key; the control node's prepare failed on every attempt (2026-09-30).
If the new row exists, the old row's holding moves to it and the old row goes; otherwise it is
renamed. The old name becomes an alias either way.
2026-09-30 21:34:34 +02:00
jschoubben 118e333ff8 Merge pull request 'The artifact store's seat is named for its scope: mesh-artifact-store' (#171) from feat/the-artifact-store-seat-is-named-for-its-scope into main
Reviewed-on: #171
2026-09-30 19:16:47 +00:00
22 changed files with 1091 additions and 56 deletions
+6
View File
@@ -479,6 +479,12 @@ func buildOne(ctx context.Context, source buildSource, path, ref string, wait ti
if source.Seat != "" { if source.Seat != "" {
recorded.Repository, recorded.Seat = source.Repository, source.Seat recorded.Repository, recorded.Seat = source.Repository, source.Seat
} }
// The build is kept; the module is not. A definition naming an installation is refused where
// it would enter the catalogue, and the build log says which build it was.
if err := namesNoInstallation(manifest); err != nil {
return fmt.Errorf("%s built %s (%s), and the mesh does not register it: %w",
result.On, result.Repository, short(result.Commit), err)
}
if err := inv.RegisterModule(ctx, manifest, recorded); err != nil { if err := inv.RegisterModule(ctx, manifest, recorded); err != nil {
return err return err
} }
+2 -2
View File
@@ -51,8 +51,8 @@ func moduleCheck(paths []string, out io.Writer) error {
failed++ failed++
continue continue
} }
// A definition names no installation (novox/hq ADR 0112, ADR 0155): judged here and in the // A definition names no installation (novox/hq ADR 0112, ADR 0155): judged here, in the
// catalogue-wide test, not yet at registration, while the declared exceptions shrink. // catalogue-wide test, and at registration, which refuses in the same words.
if named := catalogue.InstallationProblems(m); len(named) > 0 { if named := catalogue.InstallationProblems(m); len(named) > 0 {
for _, p := range named { for _, p := range named {
fmt.Fprintf(out, "%s: %s\n", path, p) fmt.Fprintf(out, "%s: %s\n", path, p)
+25
View File
@@ -5,6 +5,8 @@ import (
"os" "os"
"path/filepath" "path/filepath"
"strings" "strings"
"github.com/novox/mesh-controller/internal/catalogue"
"testing" "testing"
) )
@@ -67,3 +69,26 @@ func TestModuleCheckPassesTheCatalogue(t *testing.T) {
t.Fatalf("the catalogue does not pass its own check: %v\n%s", err, out.String()) t.Fatalf("the catalogue does not pass its own check: %v\n%s", err, out.String())
} }
} }
func TestRegistrationRefusesADefinitionNamingAnInstallation(t *testing.T) {
// novox/hq ADR 0155: the check moves to registration once the catalogue passes it. Both
// ways in — `module add` and a build's result — go through this, and a name declared on
// purpose passes with its reason.
named := catalogue.Manifest{Module: "idp", Resources: []map[string]any{
{"id": "server", "type": "container", "image": "x@sha256:aa",
"env": map[string]any{"KC_HOSTNAME": "https://login.mesh-one.be"}},
}}
err := namesNoInstallation(named)
if err == nil || !strings.Contains(err.Error(), "login.mesh-one.be") ||
!strings.Contains(err.Error(), catalogue.NamesOnPurpose) {
t.Fatalf("a definition naming an installation is refused with the name and the way out; got %v", err)
}
meant := catalogue.Manifest{Module: "site", Resources: []map[string]any{
{"id": "server", "type": "container", "image": "registry.mesh-one.be/org/site@sha256:cc",
catalogue.NamesOnPurpose: map[string]any{
"registry.mesh-one.be": "built outside the mesh until its repository is a build source here"}},
}}
if err := namesNoInstallation(meant); err != nil {
t.Fatalf("a name declared on purpose passes; got %v", err)
}
}
+18
View File
@@ -113,6 +113,9 @@ func moduleCommand(ctx context.Context, args []string) error {
if err != nil { if err != nil {
return err return err
} }
if err := namesNoInstallation(m); err != nil {
return err
}
if err := inv.RegisterModule(ctx, m, from); err != nil { if err := inv.RegisterModule(ctx, m, from); err != nil {
return err return err
} }
@@ -662,3 +665,18 @@ func whereItComesFrom(repository, ref, commit, path string, self bool) (inventor
} }
return from, nil return from, nil
} }
// namesNoInstallation is the mesh refusing a definition that names an installation, at the moment
// it would enter the catalogue (novox/hq ADR 0112, ADR 0155). `module check` says the same thing
// earlier, where the author is; this is the last moment the mesh can still say no, and a
// definition that got past the check — written elsewhere, or checked by nobody — is refused here
// in the same words. A name meant on purpose is declared with its reason and passes.
func namesNoInstallation(m catalogue.Manifest) error {
named := catalogue.InstallationProblems(m)
if len(named) == 0 {
return nil
}
return fmt.Errorf("%s names an installation, and a definition names none — declare a name meant "+
"on purpose under %s with its reason, or take it out:\n - %s",
m.Module, catalogue.NamesOnPurpose, strings.Join(named, "\n - "))
}
+8 -1
View File
@@ -40,7 +40,14 @@ type Consumer struct {
AckWaitSeconds int AckWaitSeconds int
// MaxDeliver before the message is dead-lettered; zero for the mesh's default. // MaxDeliver before the message is dead-lettered; zero for the mesh's default.
MaxDeliver int MaxDeliver int
Why string // MaxAckPending is how many deliveries the server lets stand unacknowledged at once; zero for
// the server's default, which is many. **One, for a consumer handled one at a time**
// (novox/hq issue 175): a handler that builds for minutes keeps its own message alive with a
// heartbeat, but everything handed over behind it times out unacknowledged and comes back —
// and a merge that came back rebuilt what it had just built, five times over on 2026-09-30.
// With one outstanding, the server holds the rest, and the heartbeat is keeping the message.
MaxAckPending int
Why string
} }
// seatStreamName is the stream holding a seat's inbound work. Named after the seat rather than // seatStreamName is the stream holding a seat's inbound work. Named after the seat rather than
+1
View File
@@ -179,6 +179,7 @@ func (j *JetStream) EnsureConsumer(c Consumer) error {
AckPolicy: nats.AckExplicitPolicy, AckPolicy: nats.AckExplicitPolicy,
AckWait: time.Duration(c.AckWaitSeconds) * time.Second, AckWait: time.Duration(c.AckWaitSeconds) * time.Second,
MaxDeliver: c.MaxDeliver, MaxDeliver: c.MaxDeliver,
MaxAckPending: c.MaxAckPending,
DeliverGroup: c.Queue, DeliverGroup: c.Queue,
DeliverSubject: "", DeliverSubject: "",
Description: c.Why, Description: c.Why,
+7 -2
View File
@@ -244,8 +244,13 @@ func MeshConsumers() []Consumer {
Push: true, Push: true,
AckWaitSeconds: 30, AckWaitSeconds: 30,
MaxDeliver: 5, MaxDeliver: 5,
Why: "the two events the mesh's own controller reacts to; after max-deliver it " + // One at a time (novox/hq issue 175): acting on a merge builds for minutes, and an
"dead-letters, because an announcement it cannot act on will not become actionable", // announcement handed over behind it must wait on the server, not time out on the
// client and come back to be acted on again.
MaxAckPending: 1,
Why: "the two events the mesh's own controller reacts to, one at a time; after " +
"max-deliver it dead-letters, because an announcement it cannot act on will not " +
"become actionable",
}, },
} }
} }
+11
View File
@@ -260,3 +260,14 @@ func TestNoTwoConsumersDeliverOntoTheSameSubject(t *testing.T) {
seen[subject] = c.Name + " on " + c.Stream seen[subject] = c.Name + " on " + c.Stream
} }
} }
// The controller's events consumer is handed one announcement at a time (novox/hq issue 175): a
// merge's handler builds for minutes, and what is queued behind it must wait on the server rather
// than time out on the client and be acted on twice.
func TestTheControllerTakesOneAnnouncementAtATime(t *testing.T) {
for _, c := range MeshConsumers() {
if c.Stream == "EVENTS" && c.Name == ControllerName && c.MaxAckPending != 1 {
t.Fatalf("the events consumer may have %d outstanding; one announcement at a time", c.MaxAckPending)
}
}
}
+18 -8
View File
@@ -11,11 +11,24 @@ import (
// //
// Not a fixture: the point is whether the manifests as written are accepted by the control plane that // Not a fixture: the point is whether the manifests as written are accepted by the control plane that
// will read them, and a copy of one manifest proves nothing about the other seventy-one. // will read them, and a copy of one manifest proves nothing about the other seventy-one.
func TestEveryCatalogueManifestParses(t *testing.T) { // catalogueRoot is the catalogue these checks run over: MESH_CATALOGUE when set, else the checkout
root := os.Getenv("MESH_CATALOGUE") // beside this one, the way the main layout has it. A check that only ran when somebody remembered a
if root == "" { // variable was a check nobody ran (novox/hq issue 134, 2026-09-30); it skips only when there is no
t.Skip("set MESH_CATALOGUE to a catalogue checkout to run this") // catalogue to be found at all.
func catalogueRoot(t *testing.T) string {
t.Helper()
if root := os.Getenv("MESH_CATALOGUE"); root != "" {
return root
} }
sibling := filepath.Join("..", "..", "..", "mesh-catalog")
if _, err := os.Stat(filepath.Join(sibling, "modules")); err != nil {
t.Skip("no catalogue beside this checkout and MESH_CATALOGUE unset")
}
return sibling
}
func TestEveryCatalogueManifestParses(t *testing.T) {
root := catalogueRoot(t)
found, err := filepath.Glob(filepath.Join(root, "modules", "*", "module.json")) found, err := filepath.Glob(filepath.Join(root, "modules", "*", "module.json"))
if err != nil || len(found) == 0 { if err != nil || len(found) == 0 {
t.Fatalf("no manifests under %s: %v", root, err) t.Fatalf("no manifests under %s: %v", root, err)
@@ -51,10 +64,7 @@ func TestEveryCatalogueManifestParses(t *testing.T) {
// definition names a domain or a public address the mesh acts on, and every value that must for now // definition names a domain or a public address the mesh acts on, and every value that must for now
// carries its reason (novox/hq ADR 0155, issue 134). The list it prints is the one that shrinks. // carries its reason (novox/hq ADR 0155, issue 134). The list it prints is the one that shrinks.
func TestNoCatalogueManifestNamesAnInstallation(t *testing.T) { func TestNoCatalogueManifestNamesAnInstallation(t *testing.T) {
root := os.Getenv("MESH_CATALOGUE") root := catalogueRoot(t)
if root == "" {
t.Skip("set MESH_CATALOGUE to a catalogue checkout to run this")
}
found, err := filepath.Glob(filepath.Join(root, "modules", "*", "module.json")) found, err := filepath.Glob(filepath.Join(root, "modules", "*", "module.json"))
if err != nil || len(found) == 0 { if err != nil || len(found) == 0 {
t.Fatalf("no manifests under %s: %v", root, err) t.Fatalf("no manifests under %s: %v", root, err)
+21
View File
@@ -197,6 +197,27 @@ func TestARouteCanBeSetPerMesh(t *testing.T) {
} }
} }
func TestASettingReachesAContributionOnlyWhereItDeclaresTheKey(t *testing.T) {
// novox/hq 04-ISSUES/173: the mail module's site name, set so its environment file could read
// it, arrived in every route it contributed. A setting overrides a key the contribution
// declares and adds none — the provider reads the contribution as a contract.
got, _ := Resolve(shelf(proxy(), published("board", "board", 8080)), []string{"board"}, workstation(), World{})
out, err := got.Declaration(Rendering{Settings: SettingsBy{
"board": {{From: "the mesh", Values: map[string]any{"host": "dashboard", "sitename": "Board"}}},
}})
if err != nil {
t.Fatal(err)
}
given := received(t, out)
if given[0].Values["host"] != "dashboard" {
t.Fatalf("the setting did not override the route's host: %v", given[0].Values)
}
if _, leaked := given[0].Values["sitename"]; leaked {
t.Fatalf("a setting the route never declared reached the proxy: %v", given[0].Values)
}
}
func TestReceivingWhatYouDoNotProvideIsRefused(t *testing.T) { func TestReceivingWhatYouDoNotProvideIsRefused(t *testing.T) {
// It would create a file nobody ever writes to, on a machine where nothing asked for it. // It would create a file nobody ever writes to, on a machine where nothing asked for it.
_, err := ParseManifest([]byte(`{"module":"traefik","version":"1", _, err := ParseManifest([]byte(`{"module":"traefik","version":"1",
+24 -3
View File
@@ -473,9 +473,15 @@ func (r Resolution) compose(with Rendering, owner map[string]string) ([]map[stri
// find each present — refusing clearly if the operator has not provided it — before it // find each present — refusing clearly if the operator has not provided it — before it
// starts anything that depends on it. The mesh creates, chowns and reconciles none of it; // starts anything that depends on it. The mesh creates, chowns and reconciles none of it;
// an `access` resource says only *this path must exist, and this module reaches it*. // an `access` resource says only *this path must exist, and this module reaches it*.
for _, a := range m.Accesses { // Where each is on THIS machine is the assignment's (novox/hq issue 153): placed by id
// where the operator said, the definition's default otherwise, refused where neither.
accesses, accessPaths, err := accessesFor(m, with.Settings[m.Module])
if err != nil {
return nil, err
}
for _, a := range accesses {
first = append(first, map[string]any{ first = append(first, map[string]any{
"id": AccessID(a.Path), "type": "access", "path": a.Path, "mode": a.At(), "id": AccessID(a.Path), "type": "access", "path": a.Path, "mode": a.Mode,
}) })
} }
for _, to := range m.SecretRequirements() { for _, to := range m.SecretRequirements() {
@@ -670,6 +676,12 @@ func (r Resolution) compose(with Rendering, owner map[string]string) ([]map[stri
} }
// And where this node places the directories the module declared without a path // And where this node places the directories the module declared without a path
// (novox/hq ADR 0112) — resolved once per module, named by ${dir:…} from any resource. // (novox/hq ADR 0112) — resolved once per module, named by ${dir:…} from any resource.
// — and, on an adopted machine, where the assignment says they already are, with the
// owner the data already has (novox/hq issue 153). Malformed placements are refused here.
placed, err := Places(m, with.Settings[m.Module])
if err != nil {
return nil, err
}
dirs := dirsFor(m, with) dirs := dirsFor(m, with)
// And the machine underneath, which no binding of its own can tell it. // And the machine underneath, which no binding of its own can tell it.
thisMachine := machineFacts(r, with.Names, with.MeshRange) thisMachine := machineFacts(r, with.Names, with.MeshRange)
@@ -710,6 +722,12 @@ func (r Resolution) compose(with Rendering, owner map[string]string) ([]map[stri
if err := dirInto(copied, dirs, m.Module); err != nil { if err := dirInto(copied, dirs, m.Module); err != nil {
return nil, err return nil, err
} }
// The operator's data the same way: ${access:…} becomes where this node keeps it,
// and a placed directory takes the owner the assignment said (issue 153).
if err := accessInto(copied, accessPaths, m.Module); err != nil {
return nil, err
}
ownerInto(copied, placed)
// **After settings, and that is the whole reason it is here.** A module's file // **After settings, and that is the whole reason it is here.** A module's file
// content is where a setting lands, so a placeholder may only exist once the setting // content is where a setting lands, so a placeholder may only exist once the setting
// has been put in — filling secrets first would look at content that is not yet what // has been put in — filling secrets first would look at content that is not yet what
@@ -1161,7 +1179,10 @@ func (r Resolution) contributions(settings SettingsBy, grants []Grant,
// module wrote another into its configuration. // module wrote another into its configuration.
func (r Resolution) composed(m Manifest, to string, raw map[string]any, layers []Layer, what string) ( func (r Resolution) composed(m Manifest, to string, raw map[string]any, layers []Layer, what string) (
map[string]any, error) { map[string]any, error) {
values, err := settle(raw, layers, nil, what) // Overridden, not merged: a setting changes a key the contribution declares and adds none.
// The provider reads the contribution as a contract, and a setting made for one of this
// module's files is no part of it (novox/hq 04-ISSUES/173).
values, err := overridden(raw, layers, what)
if err != nil { if err != nil {
return nil, fmt.Errorf("%s: %w", what, err) return nil, fmt.Errorf("%s: %w", what, err)
} }
+53 -10
View File
@@ -20,6 +20,12 @@ import (
// declared with the path as the exception it is, and everything else in the module names it by // declared with the path as the exception it is, and everything else in the module names it by
// id — so moving it later is one line, not a search. // id — so moving it later is one line, not a search.
// //
// **The mesh's own files for a module are placed too** (novox/hq issue 174). What the mesh writes
// *for* a module — its sealed bus credential, its merged configuration, its bindings — is the
// mesh's plumbing, not the module's data, and sits under `<root>/mesh/<module>`. A directory
// saying `"place": "mesh"` is that place; the definition names the files beneath it by
// `${dir:<id>}` and states no path.
//
// **Resolved here, not on the machine.** The host receives concrete paths exactly as it always // **Resolved here, not on the machine.** The host receives concrete paths exactly as it always
// has; nothing new reaches it and it learns no field. Which also means a resolved path changing // has; nothing new reaches it and it learns no field. Which also means a resolved path changing
// is a spec change like any other — and the spec comparison must see it (novox/hq issue 126). // is a spec change like any other — and the spec comparison must see it (novox/hq issue 126).
@@ -27,6 +33,15 @@ import (
// defaultDataRoot is where module data lands when a node states no root of its own. // defaultDataRoot is where module data lands when a node states no root of its own.
const defaultDataRoot = "/var/lib" const defaultDataRoot = "/var/lib"
// The two places a pathless directory may name, beside its own id.
const (
// placeOwn is the assignment's own root, <root>/<module> — to-be 27's one directory per
// assignment, which every other placed thing of the module sits beneath.
placeOwn = "."
// placeMesh is where the mesh keeps what it writes for the module, <root>/mesh/<module>.
placeMesh = "mesh"
)
// dirRef is how a module names one of its placed directories: ${dir:<id>}. // dirRef is how a module names one of its placed directories: ${dir:<id>}.
var dirRef = regexp.MustCompile(`\$\{dir:([a-z0-9][a-z0-9-]*)\}`) var dirRef = regexp.MustCompile(`\$\{dir:([a-z0-9][a-z0-9-]*)\}`)
@@ -40,26 +55,53 @@ func dataRoot(with Rendering) string {
// dirsFor is every placed directory of a module, id → the path it resolves to on this node. // dirsFor is every placed directory of a module, id → the path it resolves to on this node.
// //
// A pathless directory saying `"place": "."` is the assignment's own root, <root>/<module> — // A pathless directory saying `"place": "."` is the assignment's own root, <root>/<module>; one
// to-be 27's one directory per assignment, which every other placed thing sits beneath. At most // saying `"place": "mesh"` is the mesh's directory for the module, <root>/mesh/<module>; one
// one makes sense; nothing enforces one, because two ids resolving to one path is a mistake the // saying neither is <root>/<module>/<id>. At most one of each place makes sense; nothing enforces
// module's own files make visible immediately. // one, because two ids resolving to one path is a mistake the module's own files make visible
// immediately.
//
// A stated path may itself begin with a placed reference — `${dir:mesh-state}/state` — and is
// filled after the directories it can name are resolved; one level, because a directory beneath
// a placed one is the whole of what an adopted layout needs (issue 174's `state` and `out`).
func dirsFor(m Manifest, with Rendering) map[string]string { func dirsFor(m Manifest, with Rendering) map[string]string {
dirs := map[string]string{} dirs := map[string]string{}
var beneath []map[string]any
// The assignment's placement wins over both (novox/hq issue 153). Refused elsewhere when
// malformed; here an invalid setting simply places nothing.
placed, _ := Places(m, with.Settings[m.Module])
for _, r := range m.Resources { for _, r := range m.Resources {
if fmt.Sprint(r["type"]) != "directory" { if fmt.Sprint(r["type"]) != "directory" {
continue continue
} }
id := fmt.Sprint(r["id"]) id := fmt.Sprint(r["id"])
if p, said := placed[id]; said {
dirs[id] = p.Path
continue
}
if path, stated := r["path"].(string); stated && path != "" { if path, stated := r["path"].(string); stated && path != "" {
if strings.HasPrefix(path, "${dir:") {
beneath = append(beneath, r)
continue
}
dirs[id] = strings.TrimRight(path, "/") dirs[id] = strings.TrimRight(path, "/")
continue continue
} }
if place, said := r["place"].(string); said && place == "." { switch place, _ := r["place"].(string); place {
case placeOwn:
dirs[id] = dataRoot(with) + "/" + m.Module dirs[id] = dataRoot(with) + "/" + m.Module
continue case placeMesh:
dirs[id] = dataRoot(with) + "/mesh/" + m.Module
default:
dirs[id] = dataRoot(with) + "/" + m.Module + "/" + id
} }
dirs[id] = dataRoot(with) + "/" + m.Module + "/" + id }
for _, r := range beneath {
path := strings.TrimRight(r["path"].(string), "/")
// A reference to no directory is left as written and refused where the resource is
// placed (dirInto), with the message that names what exists.
filled, _ := dirFill(path, dirs, m.Module)
dirs[fmt.Sprint(r["id"])] = filled
} }
return dirs return dirs
} }
@@ -244,10 +286,11 @@ func (m Manifest) unknownDirRefs() []string {
"%s states both path and place on %v — a stated path IS the placement", "%s states both path and place on %v — a stated path IS the placement",
m.Module, r["id"])) m.Module, r["id"]))
} }
if place != "." { if place != placeOwn && place != placeMesh {
problems = append(problems, fmt.Sprintf( problems = append(problems, fmt.Sprintf(
"%s says place %q on %v, and the only place is %q — the assignment's own root", "%s says place %q on %v, and the places are %q — the assignment's own root — and "+
m.Module, place, r["id"], ".")) "%q — where the mesh keeps what it writes for the module",
m.Module, place, r["id"], placeOwn, placeMesh))
} }
} }
seen := map[string]bool{} seen := map[string]bool{}
+75 -2
View File
@@ -216,8 +216,48 @@ func TestPlaceIsValidatedAtTheManifest(t *testing.T) {
wrong := Manifest{Module: "x", Resources: []map[string]any{ wrong := Manifest{Module: "x", Resources: []map[string]any{
{"id": "d", "type": "directory", "place": "sub/dir"}, {"id": "d", "type": "directory", "place": "sub/dir"},
}} }}
if got := wrong.unknownDirRefs(); len(got) != 1 || !strings.Contains(got[0], `the only place is "."`) { if got := wrong.unknownDirRefs(); len(got) != 1 || !strings.Contains(got[0], `the places are "."`) {
t.Fatalf("a place that is not the root refuses; got %v", got) t.Fatalf("a place that is neither root refuses; got %v", got)
}
}
func TestTheMeshsDirectoryForAModuleIsAPlace(t *testing.T) {
// novox/hq issue 174. What the mesh writes for a module — its bus credential, its bindings —
// is the mesh's plumbing under <root>/mesh/<module>, and the definition names it by id.
m := Manifest{Module: "umami",
Resources: []map[string]any{
{"id": "mesh-state", "type": "directory", "place": "mesh"},
{"id": "state", "type": "directory", "place": "."},
{"id": "server", "type": "container", "image": "x@sha256:aa",
"volumes": []any{"${dir:mesh-state}/broker:/run/secrets/broker:ro"}},
},
OwnSecrets: map[string]string{"broker": "${dir:mesh-state}/broker"},
Binds: map[string]string{"route": "${dir:state}/route.json"},
}
if got := m.unknownDirRefs(); len(got) != 0 {
t.Fatalf("place %q is a place; got %v", "mesh", got)
}
dirs := dirsFor(m, Rendering{})
if dirs["mesh-state"] != "/var/lib/mesh/umami" || dirs["state"] != "/var/lib/umami" {
t.Fatalf("the mesh's directory sits beside the module's, not in it; got %v", dirs)
}
dirs = dirsFor(m, Rendering{DataRoot: "/srv"})
if dirs["mesh-state"] != "/srv/mesh/umami" {
t.Fatalf("a node's root moves the mesh's files with the module's; got %v", dirs)
}
placed, err := placedManifest(m, Rendering{})
if err != nil {
t.Fatal(err)
}
if placed.OwnSecrets["broker"] != "/var/lib/mesh/umami/broker" {
t.Fatalf("own-secrets are placed under the mesh's directory; got %v", placed.OwnSecrets)
}
container := shallowCopy(m.Resources[2])
if err := dirInto(container, dirsFor(m, Rendering{}), m.Module); err != nil {
t.Fatal(err)
}
if container["volumes"].([]any)[0] != "/var/lib/mesh/umami/broker:/run/secrets/broker:ro" {
t.Fatalf("the mount's host side is placed; got %v", container["volumes"])
} }
} }
@@ -250,3 +290,36 @@ func shallowCopy(resource map[string]any) map[string]any {
} }
return copied return copied
} }
func TestADirectoryBeneathAPlacedOneIsPlacedWithIt(t *testing.T) {
// An adopted layout keeps a subdirectory the predecessor made under the mesh's directory
// (issue 174: a forge's runtime state, a manager's output). Stated as beneath the placed one,
// it moves with it — a node's root moves both, and the definition names no host path.
m := Manifest{Module: "gitea", Resources: []map[string]any{
{"id": "mesh-state", "type": "directory", "place": "mesh"},
{"id": "runtime-state", "type": "directory", "path": "${dir:mesh-state}/state"},
{"id": "server", "type": "container", "image": "x@sha256:aa",
"volumes": []any{"${dir:runtime-state}:/data"}},
}}
if got := m.unknownDirRefs(); len(got) != 0 {
t.Fatalf("a path beneath a placed directory is well formed; got %v", got)
}
dirs := dirsFor(m, Rendering{DataRoot: "/srv"})
if dirs["runtime-state"] != "/srv/mesh/gitea/state" {
t.Fatalf("the subdirectory follows the placed one; got %v", dirs)
}
sub := shallowCopy(m.Resources[1])
if err := dirInto(sub, dirs, m.Module); err != nil {
t.Fatal(err)
}
if sub["path"] != "/srv/mesh/gitea/state" {
t.Fatalf("the directory resource itself is resolved; got %v", sub["path"])
}
container := shallowCopy(m.Resources[2])
if err := dirInto(container, dirs, m.Module); err != nil {
t.Fatal(err)
}
if container["volumes"].([]any)[0] != "/srv/mesh/gitea/state:/data" {
t.Fatalf("a reference to the subdirectory resolves whole; got %v", container["volumes"])
}
}
+18 -3
View File
@@ -91,8 +91,13 @@ const (
// If the path is absent when a machine applies, the host refuses clearly rather than creating it: // If the path is absent when a machine applies, the host refuses clearly rather than creating it:
// the mesh does not own it, so conjuring it would be a lie the host then acts on. // the mesh does not own it, so conjuring it would be a lie the host then acts on.
type Access struct { type Access struct {
// Path is the absolute path on the machine, as the operator provides it. // ID is the name the module gives this access, which the assignment places
Path string `json:"path"` // (`accesses: {<id>: <path>}`, novox/hq ADR 0112, issue 153) and the module's mounts name as
// ${access:<id>}. The shape a definition should use: it names no path of any machine.
ID string `json:"id,omitempty"`
// Path is the absolute path on the machine. A definition carrying one names an installation;
// tolerated as the default the assignment may replace, for accesses declared before ids.
Path string `json:"path,omitempty"`
// Mode is "read" or "read-write". Absent narrows to read. // Mode is "read" or "read-write". Absent narrows to read.
Mode string `json:"mode,omitempty"` Mode string `json:"mode,omitempty"`
} }
@@ -1525,7 +1530,16 @@ func ParseManifest(raw []byte) (Manifest, error) {
} }
} }
for _, a := range m.Accesses { for _, a := range m.Accesses {
if !strings.HasPrefix(a.Path, "/") { if a.ID == "" && a.Path == "" {
problems = append(problems, fmt.Sprintf(
"%s declares an access with neither an id nor a path — an id, which the assignment places",
m.Module))
}
if a.ID != "" && !accessRef.MatchString("${access:"+a.ID+"}") {
problems = append(problems, fmt.Sprintf(
"%s accesses %q; an access id is lowercase letters, digits and dashes", m.Module, a.ID))
}
if a.Path != "" && !strings.HasPrefix(a.Path, "/") {
problems = append(problems, fmt.Sprintf( problems = append(problems, fmt.Sprintf(
"%s accesses %q, which is not an absolute path", m.Module, a.Path)) "%s accesses %q, which is not an absolute path", m.Module, a.Path))
} }
@@ -1557,6 +1571,7 @@ func ParseManifest(raw []byte) (Manifest, error) {
// the time it sees the mount it is being asked to create the directory, which it can do. // the time it sees the mount it is being asked to create the directory, which it can do.
problems = append(problems, m.undeclaredMounts()...) problems = append(problems, m.undeclaredMounts()...)
problems = append(problems, m.unknownDirRefs()...) problems = append(problems, m.unknownDirRefs()...)
problems = append(problems, m.unknownAccessRefs()...)
for i, r := range m.Resources { for i, r := range m.Resources {
id, _ := r["id"].(string) id, _ := r["id"].(string)
+62 -3
View File
@@ -16,7 +16,9 @@ import (
// //
// Two checkouts: MESH_CATALOGUE_BEFORE, the catalogue as it was, and MESH_CATALOGUE, as it is now. // Two checkouts: MESH_CATALOGUE_BEFORE, the catalogue as it was, and MESH_CATALOGUE, as it is now.
// Every module in both is resolved with the controller's own rule (dirsFor, dirFill) and compared // Every module in both is resolved with the controller's own rule (dirsFor, dirFill) and compared
// whole — not only the directories, but every string a directory's id was written into. // whole — not only the directories, but every string a directory's id was written into. Both
// sides are resolved, so a manifest converted in two steps (issue 119, then issue 174) is judged
// against the paths it named, not the text it used to name them with.
func TestPlacedDirectoriesKeepTheirPaths(t *testing.T) { func TestPlacedDirectoriesKeepTheirPaths(t *testing.T) {
before, after := os.Getenv("MESH_CATALOGUE_BEFORE"), os.Getenv("MESH_CATALOGUE") before, after := os.Getenv("MESH_CATALOGUE_BEFORE"), os.Getenv("MESH_CATALOGUE")
if before == "" || after == "" { if before == "" || after == "" {
@@ -42,7 +44,11 @@ func TestPlacedDirectoriesKeepTheirPaths(t *testing.T) {
t.Errorf("%s: %v", module, err) t.Errorf("%s: %v", module, err)
continue continue
} }
dirs := dirsFor(m, Rendering{}) earlier, err := ParseManifest(old)
if err != nil {
t.Errorf("%s before: %v", module, err)
continue
}
var was, is any var was, is any
if err := json.Unmarshal(old, &was); err != nil { if err := json.Unmarshal(old, &was); err != nil {
t.Fatal(err) t.Fatal(err)
@@ -50,7 +56,15 @@ func TestPlacedDirectoriesKeepTheirPaths(t *testing.T) {
if err := json.Unmarshal(now, &is); err != nil { if err := json.Unmarshal(now, &is); err != nil {
t.Fatal(err) t.Fatal(err)
} }
resolved := resolvedTree(is, dirs, module, t) // Both sides resolved: the manifest before may itself already place some directories
// (issue 119's conversion), and what must not move is the path a machine sees.
was = resolvedTree(was, dirsFor(earlier, Rendering{}), module, t)
resolved := resolvedTree(is, dirsFor(m, Rendering{}), module, t)
// An access named by id resolves to the path the definition still carries as its default
// (issue 153) — the same rule as a placed directory: an assignment that says nothing moves
// nothing.
was = accessesResolved(was, earlier)
resolved = accessesResolved(resolved, m)
if !reflect.DeepEqual(was, resolved) { if !reflect.DeepEqual(was, resolved) {
wasJSON, _ := json.MarshalIndent(was, "", " ") wasJSON, _ := json.MarshalIndent(was, "", " ")
isJSON, _ := json.MarshalIndent(resolved, "", " ") isJSON, _ := json.MarshalIndent(resolved, "", " ")
@@ -119,3 +133,48 @@ func firstDifference(a, b string) string {
} }
return "(the difference is beyond the shorter document)" return "(the difference is beyond the shorter document)"
} }
// accessesResolved fills every ${access:<id>} with the default path the definition carries for that
// access, and drops the id, so a manifest that names its accesses is compared by the paths a machine
// with no placement receives.
func accessesResolved(node any, m Manifest) any {
defaults := map[string]string{}
for _, a := range m.Accesses {
if a.ID != "" && a.Path != "" {
defaults[a.ID] = a.Path
}
}
var walk func(any) any
walk = func(n any) any {
switch v := n.(type) {
case map[string]any:
out := map[string]any{}
for k, child := range v {
if k == "id" {
if _, isAccess := v["mode"]; isAccess && v["type"] == nil {
if _, hasPath := v["path"]; hasPath {
continue
}
}
}
out[k] = walk(child)
}
return out
case []any:
out := make([]any, len(v))
for i, child := range v {
out[i] = walk(child)
}
return out
case string:
return accessRef.ReplaceAllStringFunc(v, func(ref string) string {
if p, ok := defaults[accessRef.FindStringSubmatch(ref)[1]]; ok {
return p
}
return ref
})
}
return n
}
return walk(node)
}
+382
View File
@@ -0,0 +1,382 @@
package catalogue
import (
"fmt"
"regexp"
"sort"
"strings"
)
// Where a module's data is on THIS machine is the assignment's (novox/hq ADR 0112, issue 153).
//
// A definition names no host path. It declares the directories it owns by id, and the operator's
// shared data it needs by id too (an `access`, ADR 0051). A node has a default layout for the
// former — <root>/<module>/<id> — and nothing at all for the latter, because shared data is
// wherever the operator keeps it. An adopted machine keeps its data where the predecessor put it:
// a 40 TB library on its own pool, a configuration on a second disk. Both halves are said on the
// assignment, validated the way `endpoints` is — an id the module does not declare is refused,
// because a setting that reaches nothing is a mistake — and resolved here, so the host receives
// concrete paths exactly as it always has and learns no field.
//
// {"places": {"config": "/services/sonarr/config",
// "data": {"path": "/mnt/plex/data", "owner": "1000:1000"}},
// "accesses": {"series": "/storage/media/series",
// "downloads": "/storage/downloads"}}
//
// A placed directory is still the mesh's: created, chowned to the owner the assignment says (or
// the manifest's), removed when empty and no longer declared. A placed access is still the
// operator's: mounted, never created, chowned or removed.
// PlacesSetting is the settings key that places a module's declared directories, by id.
const PlacesSetting = "places"
// AccessesSetting is the settings key that says where a module's accesses are on this node, by id.
const AccessesSetting = "accesses"
// Placement is what an assignment says about one of a module's directories.
type Placement struct {
// Path is where the directory is on this machine. Absolute.
Path string
// Owner is "uid:gid" when the assignment overrides the manifest's — the predecessor's data is
// owned by whoever it ran as, and that is one machine's fact.
Owner string
}
var ownerShape = regexp.MustCompile(`^[0-9]+:[0-9]+$`)
// accessRef is how a module names one of its accesses: ${access:<id>}.
var accessRef = regexp.MustCompile(`\$\{access:([a-z0-9][a-z0-9-]*)\}`)
// Places reads where this node places the module's directories, by directory id.
//
// It refuses an id the module declares no directory for, a path that is not absolute, and an
// owner that is not uid:gid. A directory the assignment does not mention keeps the manifest's
// stated path or the node's default layout.
func Places(m Manifest, layers []Layer) (map[string]Placement, error) {
declared := map[string]bool{}
for _, r := range m.Resources {
if fmt.Sprint(r["type"]) == "directory" {
declared[fmt.Sprint(r["id"])] = true
}
}
out := map[string]Placement{}
for _, layer := range layers {
raw, ok := layer.Values[PlacesSetting]
if !ok {
continue
}
blocks, ok := raw.(map[string]any)
if !ok {
return nil, fmt.Errorf("%s: %s is a { directory: path | { path, owner } } map, and %q set it to something else",
m.Module, PlacesSetting, layer.From)
}
for id, body := range blocks {
if !declared[id] {
return nil, fmt.Errorf(
"%s places the directory %q, which it does not declare — the setting reaches "+
"nothing. It declares %s", m.Module, id, orNothing(namesOfDirs(directoriesOf(m))))
}
p := out[id]
switch v := body.(type) {
case string:
p.Path = strings.TrimSpace(v)
case map[string]any:
if path, said := v["path"]; said {
text, _ := path.(string)
p.Path = strings.TrimSpace(text)
}
if owner, said := v["owner"]; said {
text, _ := owner.(string)
if !ownerShape.MatchString(strings.TrimSpace(text)) {
return nil, fmt.Errorf("%s places %q with owner %v; an owner is uid:gid, numeric",
m.Module, id, owner)
}
p.Owner = strings.TrimSpace(text)
}
default:
return nil, fmt.Errorf("%s places %q with %v; a placement is a path, or { path, owner }",
m.Module, id, body)
}
if p.Path == "" {
return nil, fmt.Errorf("%s places %q without a path", m.Module, id)
}
if !strings.HasPrefix(p.Path, "/") {
return nil, fmt.Errorf("%s places %q at %q, which is not an absolute path", m.Module, id, p.Path)
}
p.Path = strings.TrimRight(p.Path, "/")
out[id] = p
}
}
if len(out) == 0 {
return nil, nil
}
return out, nil
}
// AccessPlaces reads where this node keeps the operator's data the module accesses, by access id.
//
// It refuses an id the module declares no access under, and a path that is not absolute. An
// access declared by path alone cannot be placed — it has no name to place it by.
func AccessPlaces(m Manifest, layers []Layer) (map[string]string, error) {
declared := map[string]bool{}
for _, a := range m.Accesses {
if a.ID != "" {
declared[a.ID] = true
}
}
out := map[string]string{}
for _, layer := range layers {
raw, ok := layer.Values[AccessesSetting]
if !ok {
continue
}
blocks, ok := raw.(map[string]any)
if !ok {
return nil, fmt.Errorf("%s: %s is a { access: path } map, and %q set it to something else",
m.Module, AccessesSetting, layer.From)
}
for id, body := range blocks {
if !declared[id] {
return nil, fmt.Errorf(
"%s places the access %q, which it does not declare — the setting reaches "+
"nothing. It declares %s", m.Module, id, orNothing(namesOfAccesses(m)))
}
path, _ := body.(string)
path = strings.TrimSpace(path)
if !strings.HasPrefix(path, "/") {
return nil, fmt.Errorf("%s places the access %q at %v, which is not an absolute path",
m.Module, id, body)
}
out[id] = strings.TrimRight(path, "/")
}
}
if len(out) == 0 {
return nil, nil
}
return out, nil
}
// placedAccess is one access with the path it resolves to on this node.
type placedAccess struct {
ID string
Path string
Mode string
}
// accessesFor is every access of a module with its path on this node: the assignment's where it
// placed one, the definition's where it carries a default, and refused where neither says — an
// access that resolves to nowhere would reach the machine as a mount of nothing.
func accessesFor(m Manifest, layers []Layer) ([]placedAccess, map[string]string, error) {
placed, err := AccessPlaces(m, layers)
if err != nil {
return nil, nil, err
}
var out []placedAccess
byID := map[string]string{}
for _, a := range m.Accesses {
path := a.Path
if a.ID != "" {
if at, said := placed[a.ID]; said {
path = at
}
}
if path == "" {
return nil, nil, fmt.Errorf(
"%s accesses %q, and nothing says where that is on this node — the definition "+
"carries no path (it must not, novox/hq ADR 0112) and the assignment places "+
"none. Set %s: {%q: \"/where/it/is\"}",
m.Module, a.ID, AccessesSetting, a.ID)
}
out = append(out, placedAccess{ID: a.ID, Path: path, Mode: a.At()})
if a.ID != "" {
byID[a.ID] = path
}
}
return out, byID, nil
}
// accessFill resolves every ${access:…} in one string, or refuses a reference naming no access.
func accessFill(s string, accesses map[string]string, module string) (string, error) {
var missing error
out := accessRef.ReplaceAllStringFunc(s, func(ref string) string {
id := accessRef.FindStringSubmatch(ref)[1]
path, has := accesses[id]
if !has {
missing = fmt.Errorf(
"%s says ${access:%s}, and %s declares no access %q. It declares %s",
module, id, module, id, orNothing(namesOfAccessIDs(accesses)))
return ref
}
return path
})
return out, missing
}
// accessInto fills every ${access:…} a resource carries — in its path, its content, its mounts,
// its environment and its env-files — with the path this node resolved for it. The same walk as
// dirInto, for the same reason: a literal `${access:x}` reaching the machine would be mounted as
// a directory called that.
func accessInto(resource map[string]any, accesses map[string]string, module string) error {
if !mentionsAccess(resource) {
return nil
}
fill := func(s string) (string, error) { return accessFill(s, accesses, module) }
var err error
if path, ok := resource["path"].(string); ok {
if resource["path"], err = fill(path); err != nil {
return err
}
}
if content, ok := resource["content"].(string); ok {
if resource["content"], err = fill(content); err != nil {
return err
}
}
for _, field := range []string{"volumes", "env-file"} {
list, ok := resource[field].([]any)
if !ok {
continue
}
filled := make([]any, len(list))
for i, v := range list {
filled[i] = v
if s, ok := v.(string); ok {
if filled[i], err = fill(s); err != nil {
return err
}
}
}
resource[field] = filled
}
if env, ok := resource["env"].(map[string]any); ok {
filled := make(map[string]any, len(env))
for key, v := range env {
filled[key] = v
if s, ok := v.(string); ok {
if filled[key], err = fill(s); err != nil {
return err
}
}
}
resource["env"] = filled
}
return nil
}
func mentionsAccess(resource map[string]any) bool {
for _, field := range []string{"path", "content"} {
if s, ok := resource[field].(string); ok && accessRef.MatchString(s) {
return true
}
}
for _, field := range []string{"volumes", "env-file"} {
if list, ok := resource[field].([]any); ok {
for _, v := range list {
if s, ok := v.(string); ok && accessRef.MatchString(s) {
return true
}
}
}
}
if env, ok := resource["env"].(map[string]any); ok {
for _, v := range env {
if s, ok := v.(string); ok && accessRef.MatchString(s) {
return true
}
}
}
return false
}
// ownerInto gives a placed directory the owner the assignment said, where it said one. The
// manifest's owner is what the image expects on any machine; the assignment's is what this
// machine's data already is.
func ownerInto(resource map[string]any, placed map[string]Placement) {
if fmt.Sprint(resource["type"]) != "directory" {
return
}
if p, ok := placed[fmt.Sprint(resource["id"])]; ok && p.Owner != "" {
resource["owner"] = p.Owner
}
}
// unknownAccessRefs is every ${access:…} in the definition that names no access the definition
// declares by id — refused where the author is, as unknownDirRefs does for directories.
func (m Manifest) unknownAccessRefs() []string {
declared := map[string]bool{}
for _, a := range m.Accesses {
if a.ID != "" {
declared[a.ID] = true
}
}
seen := map[string]bool{}
var problems []string
refuse := func(s string, where any) {
for _, match := range accessRef.FindAllStringSubmatch(s, -1) {
id := match[1]
if declared[id] || seen[id] {
continue
}
seen[id] = true
problems = append(problems, fmt.Sprintf(
"%s says ${access:%s} in %v, and declares no access %q — a reference the mesh "+
"cannot place would reach the machine as a literal path",
m.Module, id, where, id))
}
}
for _, r := range m.Resources {
for _, field := range []string{"path", "content"} {
if s, ok := r[field].(string); ok {
refuse(s, r["id"])
}
}
for _, field := range []string{"volumes", "env-file"} {
if list, ok := r[field].([]any); ok {
for _, v := range list {
if s, ok := v.(string); ok {
refuse(s, r["id"])
}
}
}
}
if env, ok := r["env"].(map[string]any); ok {
for _, v := range env {
if s, ok := v.(string); ok {
refuse(s, r["id"])
}
}
}
}
sort.Strings(problems)
return problems
}
func directoriesOf(m Manifest) map[string]string {
dirs := map[string]string{}
for _, r := range m.Resources {
if fmt.Sprint(r["type"]) == "directory" {
dirs[fmt.Sprint(r["id"])] = ""
}
}
return dirs
}
func namesOfAccesses(m Manifest) []string {
var names []string
for _, a := range m.Accesses {
if a.ID != "" {
names = append(names, fmt.Sprintf("%q", a.ID))
}
}
sort.Strings(names)
return names
}
func namesOfAccessIDs(accesses map[string]string) []string {
var names []string
for id := range accesses {
names = append(names, fmt.Sprintf("%q", id))
}
sort.Strings(names)
return names
}
+172
View File
@@ -0,0 +1,172 @@
package catalogue
import (
"strings"
"testing"
)
// The case novox/hq issue 153 records: an adopted machine keeps its data where the predecessor put
// it — a library on its own pool that must never move, a configuration on a second disk owned by
// whoever the predecessor ran as. A definition may name none of that (ADR 0112); the assignment
// says it, by the ids the definition declared, and the machine receives concrete paths as always.
func placeable() Manifest {
m := mod("arr", nil, nil, nil)
m.Resources = []map[string]any{
{"id": "state", "type": "directory", "place": ".", "mode": "0700"},
{"id": "config", "type": "directory", "mode": "0755", "owner": "1000:1000"},
{"id": "server", "type": "container", "name": "arr", "image": "arr:1",
"volumes": []any{"${dir:config}:/config", "${access:series}:/series", "${access:spool}:/downloads:ro"},
"env": map[string]any{"SPOOL": "${access:spool}"}},
}
m.Accesses = []Access{{ID: "series", Mode: AccessReadWrite}, {ID: "spool"}}
return m
}
func placedBy(values map[string]any) Rendering {
return Rendering{Settings: SettingsBy{"arr": {{From: "node anchor", Values: values}}}}
}
func TestAnAssignmentPlacesDirectoriesAndAccesses(t *testing.T) {
got, err := Resolve(shelf(placeable()), []string{"arr"}, workstation(), World{})
if err != nil {
t.Fatal(err)
}
out, err := got.Declaration(placedBy(map[string]any{
PlacesSetting: map[string]any{
"config": map[string]any{"path": "/services/arr/config/", "owner": "1001:2000"},
},
AccessesSetting: map[string]any{
"series": "/storage/media/series",
"spool": "/storage/downloads",
},
}))
if err != nil {
t.Fatal(err)
}
seen := map[string]map[string]any{}
for _, r := range out {
seen[r["id"].(string)] = r
}
config := seen["arr.config"]
if config["path"] != "/services/arr/config" || config["owner"] != "1001:2000" {
t.Fatalf("the placed directory is %v %v; want the assignment's path and owner", config["path"], config["owner"])
}
if seen["arr.state"]["path"] != "/var/lib/arr" {
t.Fatalf("an unplaced directory left the default layout: %v", seen["arr.state"]["path"])
}
var accesses []string
for _, r := range out {
if r["type"] == "access" {
accesses = append(accesses, r["path"].(string)+" "+r["mode"].(string))
}
}
if strings.Join(accesses, ",") != "/storage/media/series read-write,/storage/downloads read" {
t.Fatalf("the accesses reached the machine as %v", accesses)
}
server := seen["arr.server"]
mounts := server["volumes"].([]any)
if mounts[0] != "/services/arr/config:/config" || mounts[1] != "/storage/media/series:/series" ||
mounts[2] != "/storage/downloads:/downloads:ro" {
t.Fatalf("the mounts were not filled with the placed paths: %v", mounts)
}
if server["env"].(map[string]any)["SPOOL"] != "/storage/downloads" {
t.Fatalf("the environment was not filled: %v", server["env"])
}
}
// An access declared by id and placed by nobody resolves to nowhere, and that is refused with the
// setting to write — not mounted as the literal, not skipped.
func TestAnUnplacedAccessIsRefusedByName(t *testing.T) {
got, err := Resolve(shelf(placeable()), []string{"arr"}, workstation(), World{})
if err != nil {
t.Fatal(err)
}
_, err = got.Declaration(placedBy(map[string]any{
AccessesSetting: map[string]any{"series": "/storage/media/series"},
}))
if err == nil || !strings.Contains(err.Error(), `"spool"`) || !strings.Contains(err.Error(), AccessesSetting) {
t.Fatalf("an access nobody placed was not refused by name: %v", err)
}
}
// Validated like endpoints: an id the module does not declare reaches nothing, and the refusal
// says what it does declare; a relative path and a non-numeric owner are refused too.
func TestPlacementsAreValidated(t *testing.T) {
m := placeable()
layers := func(values map[string]any) []Layer { return placedBy(values).Settings["arr"] }
_, err := Places(m, layers(map[string]any{PlacesSetting: map[string]any{"data": "/mnt/data"}}))
if err == nil || !strings.Contains(err.Error(), "does not declare") || !strings.Contains(err.Error(), `"config"`) {
t.Fatalf("placing an undeclared directory was accepted: %v", err)
}
_, err = Places(m, layers(map[string]any{PlacesSetting: map[string]any{"config": "services/arr"}}))
if err == nil || !strings.Contains(err.Error(), "absolute") {
t.Fatalf("a relative placement was accepted: %v", err)
}
_, err = Places(m, layers(map[string]any{PlacesSetting: map[string]any{
"config": map[string]any{"path": "/services/arr", "owner": "media"}}}))
if err == nil || !strings.Contains(err.Error(), "uid:gid") {
t.Fatalf("a non-numeric owner was accepted: %v", err)
}
_, err = AccessPlaces(m, layers(map[string]any{AccessesSetting: map[string]any{"movies": "/storage/media/movies"}}))
if err == nil || !strings.Contains(err.Error(), "does not declare") || !strings.Contains(err.Error(), `"series"`) {
t.Fatalf("placing an undeclared access was accepted: %v", err)
}
_, err = AccessPlaces(m, layers(map[string]any{AccessesSetting: map[string]any{"series": "media/series"}}))
if err == nil || !strings.Contains(err.Error(), "absolute") {
t.Fatalf("a relative access was accepted: %v", err)
}
// And the two keys are never stray: they are validated here, not merged into a file.
if stray := UnusedSettings(m, layers(map[string]any{
PlacesSetting: map[string]any{"config": "/services/arr/config"},
AccessesSetting: map[string]any{"series": "/storage/media/series"},
})); len(stray) != 0 {
t.Fatalf("the placement keys were reported as unused: %v", stray)
}
}
// A definition that carries a path still works, as the default the assignment may replace — and
// the assignment's placement wins where both say.
func TestADefinitionsPathIsTheDefaultTheAssignmentReplaces(t *testing.T) {
m := placeable()
m.Accesses = []Access{{ID: "series", Path: "/services/media/series", Mode: AccessReadWrite}, {ID: "spool", Path: "/services/media/downloads"}}
got, err := Resolve(shelf(m), []string{"arr"}, workstation(), World{})
if err != nil {
t.Fatal(err)
}
out, err := got.Declaration(placedBy(map[string]any{
PlacesSetting: map[string]any{"config": "/services/arr/config"},
AccessesSetting: map[string]any{"series": "/storage/media/series"},
}))
if err != nil {
t.Fatal(err)
}
var paths []string
for _, r := range out {
if r["type"] == "access" {
paths = append(paths, r["path"].(string))
}
}
if strings.Join(paths, ",") != "/storage/media/series,/services/media/downloads" {
t.Fatalf("placed one, defaulted the other: got %v", paths)
}
}
// A reference to an access the definition does not declare is refused where the author is.
func TestAnUnknownAccessReferenceIsRefusedAtParse(t *testing.T) {
_, err := ParseManifest([]byte(`{
"module": "arr", "version": "1",
"accesses": [{"id": "series", "mode": "read-write"}],
"resources": [
{"id": "state", "type": "directory", "place": ".", "mode": "0700"},
{"id": "server", "type": "container", "name": "arr", "image": "arr:1",
"volumes": ["${access:movies}:/movies"]}
]}`))
if err == nil || !strings.Contains(err.Error(), "${access:movies}") {
t.Fatalf("a reference to an undeclared access was accepted: %v", err)
}
_, err = ParseManifest([]byte(`{"module": "arr", "version": "1", "accesses": [{"mode": "read"}]}`))
if err == nil || !strings.Contains(err.Error(), "neither an id nor a path") {
t.Fatalf("an access with no id and no path was accepted: %v", err)
}
}
+3
View File
@@ -791,6 +791,9 @@ func checkResources(modules []Manifest) []string {
// which is what lets the stack in 04-ISSUES/036 co-resolve. // which is what lets the stack in 04-ISSUES/036 co-resolve.
for _, m := range modules { for _, m := range modules {
for _, a := range m.Accesses { for _, a := range m.Accesses {
if a.Path == "" {
continue // placed by the assignment; nothing to compare at registration
}
switch other := ownedPath[a.Path]; other { switch other := ownedPath[a.Path]; other {
case "": case "":
// Nobody owns it — the ordinary, correct case for shared data. // Nobody owns it — the ordinary, correct case for shared data.
+25 -4
View File
@@ -91,19 +91,40 @@ func orNoSettings(layers []Layer) string {
return "; set today: " + strings.Join(keys, ", ") return "; set today: " + strings.Join(keys, ", ")
} }
// settingKeysUsedBy is every key a module's files ask for, so a setting that lands in one is not // settingKeysUsedBy is every key a module's files, contributions and served facts ask for, so a
// called stray. // setting that lands in one is not called stray.
func settingKeysUsedBy(m Manifest) map[string]bool { func settingKeysUsedBy(m Manifest) map[string]bool {
used := map[string]bool{} used := map[string]bool{}
note := func(s string) {
for _, k := range settingsUsed(s) {
used[k] = true
}
}
for _, r := range m.Resources { for _, r := range m.Resources {
if fmt.Sprint(r["type"]) != "file" { if fmt.Sprint(r["type"]) != "file" {
continue continue
} }
if content, ok := r["content"].(string); ok { if content, ok := r["content"].(string); ok {
for _, k := range settingsUsed(content) { note(content)
used[k] = true }
}
inValues := func(values map[string]any) {
for _, v := range values {
if s, ok := v.(string); ok {
note(s)
} }
} }
} }
for _, values := range m.Contributes {
inValues(values)
}
for _, locals := range m.ContributesMany {
for _, values := range locals {
inValues(values)
}
}
for _, values := range m.Serves {
inValues(values)
}
return used return used
} }
+90 -16
View File
@@ -85,17 +85,67 @@ func ApplySettings(resource map[string]any, layers []Layer) (map[string]any, err
return out, nil return out, nil
} }
// Settle lays settings over a module's own values. Exported for what a provider serves, which is // Settle lays settings over what a provider serves. Exported because a served fact is settled where
// settled where the mesh is walked rather than where a node is declared. // the mesh is walked rather than where a node is declared.
//
// **A setting overrides a served key; it never adds one** (novox/hq 04-ISSUES/173). What a consumer
// is told is the provider's contract, and a setting made for one of the provider's files — a site
// name, a public address — is not part of it. Before this, every setting of a module reached every
// consumer of every provision it served.
func Settle(base map[string]any, layers []Layer) (map[string]any, error) { func Settle(base map[string]any, layers []Layer) (map[string]any, error) {
return settle(base, layers, nil, "what is served") return overridden(base, layers, "what is served")
}
// overridden lays settings over a map whose keys are its contract: a contribution, a served fact.
// Only the keys the map already declares are touched; the rest of a layer is somebody else's
// business (a file's, another destination's) and is left to reach it there.
//
// A declared value may itself be the operator's, `${setting:<key>}` (ADR 0155): a mail provider
// serves its domain, an identity provider its issuer, and neither is the definition's to state.
// Filled from the layers after the overrides, and refused by name when nothing sets it — a literal
// placeholder handed to a consumer is a service configured against a string nobody meant.
func overridden(base map[string]any, layers []Layer, what string) (map[string]any, error) {
kept := make([]Layer, 0, len(layers))
for _, layer := range layers {
values := map[string]any{}
for key, value := range layer.Values {
if _, declared := base[key]; declared {
values[key] = value
}
}
kept = append(kept, Layer{From: layer.From, Values: values})
}
merged, err := settle(base, kept, nil, what)
if err != nil {
return nil, err
}
for key, value := range merged {
s, ok := value.(string)
if !ok {
continue
}
for _, asked := range settingsUsed(s) {
v, set := settingValue(layers, asked)
if !set {
return nil, fmt.Errorf(
"%s says ${setting:%s} for %q, and nothing sets %q — an operator's value is the "+
"assignment's, never the definition's (novox/hq ADR 0112)%s",
what, asked, key, asked, orNoSettings(layers))
}
s = strings.ReplaceAll(s, "${setting:"+asked+"}", plainly(v))
}
merged[key] = s
}
return merged, nil
} }
// settle lays the layers over a module's own values, in order. // settle lays the layers over a module's own values, in order.
// //
// Shared by a file's content and a module's contributions, because they are the same act: the // Shared by a file's content and a module's contributions, because they are the same act: the
// module says what it means by default, and somebody says what it means here. A contribution that // module says what it means by default, and somebody says what it means here. A contribution that
// could not be settled would have to be edited to be reused anywhere else. // could not be settled would have to be edited to be reused anywhere else. The two differ in one
// respect, and the caller decides it: a file takes keys it did not declare (a setting may add to a
// configuration), a contribution or served fact does not (overridden).
func settle(base map[string]any, layers []Layer, protected map[string]bool, what string) ( func settle(base map[string]any, layers []Layer, protected map[string]bool, what string) (
map[string]any, error) { map[string]any, error) {
merged := deepCopy(base) merged := deepCopy(base)
@@ -149,23 +199,22 @@ func deepCopy(in map[string]any) map[string]any {
return out return out
} }
// UnusedSettings names settings that reached no file. // UnusedSettings names settings that reach nothing.
// //
// Somebody who sets a key on a module with nothing mergeable, or misspells one, has changed // Somebody who sets a key on a module with nothing mergeable, or misspells one, has changed
// nothing — and would find out by the machine not behaving differently, which is the slowest // nothing — and would find out by the machine not behaving differently, which is the slowest
// way there is. This is what makes that visible at the moment they set it. // way there is. This is what makes that visible at the moment they set it.
//
// Where a key can land: any mergeable file takes any key; a file asking for `${setting:<key>}`
// takes that key (ADR 0155); a contribution or a served fact takes a key it declares, and no other
// (novox/hq 04-ISSUES/173); and the mesh's own words — `expose`, `ports`, `reach`, `endpoints` —
// are read by the mesh. A key none of those takes is stray, and is said so rather than dropped.
func UnusedSettings(m Manifest, layers []Layer) []string { func UnusedSettings(m Manifest, layers []Layer) []string {
for _, r := range m.Resources { for _, r := range m.Resources {
if how, _ := r["merge"].(string); how != "" { if how, _ := r["merge"].(string); how != "" {
return nil return nil
} }
} }
// A contribution is a destination too. A route's hostname is exactly the kind of thing that
// differs between one mesh and the next, and calling it stray would refuse the one setting
// most modules that publish anything will have.
if len(m.Contributes) > 0 {
return nil
}
// A computed module has no resources here to look at — they are worked out per node, and // A computed module has no resources here to look at — they are worked out per node, and
// whether a setting lands is not knowable until then. Silence rather than a wrong answer: // whether a setting lands is not knowable until then. Silence rather than a wrong answer:
// claiming every setting on the private network is stray would be worse than saying nothing. // claiming every setting on the private network is stray would be worse than saying nothing.
@@ -173,12 +222,28 @@ func UnusedSettings(m Manifest, layers []Layer) []string {
return nil return nil
} }
// A key a file's content asks for with ${setting:<key>} is a destination too (ADR 0155). lands := settingKeysUsedBy(m)
asked := settingKeysUsedBy(m) for _, values := range m.Contributes {
for key := range values {
lands[key] = true
}
}
for _, locals := range m.ContributesMany {
for _, values := range locals {
for key := range values {
lands[key] = true
}
}
}
for _, served := range m.Serves {
for key := range served {
lands[key] = true
}
}
var unused []string var unused []string
for _, layer := range layers { for _, layer := range layers {
for key := range layer.Values { for key := range layer.Values {
if asked[key] { if lands[key] {
continue continue
} }
// `expose` is a real destination for a module that listens: it overrides a port's // `expose` is a real destination for a module that listens: it overrides a port's
@@ -202,9 +267,18 @@ func UnusedSettings(m Manifest, layers []Layer) []string {
if key == EndpointsSetting && len(m.Listens) > 0 { if key == EndpointsSetting && len(m.Listens) > 0 {
continue continue
} }
// `places` puts a declared directory where this machine keeps it, `accesses` says where
// the operator's data is (novox/hq issue 153). Validated in Places and AccessPlaces.
if key == PlacesSetting && len(directoriesOf(m)) > 0 {
continue
}
if key == AccessesSetting && len(m.Accesses) > 0 {
continue
}
unused = append(unused, fmt.Sprintf( unused = append(unused, fmt.Sprintf(
"%s sets %q, and %s has no file or contribution to merge it into", "%s sets %q, and %s has no file that merges it, asks for no ${setting:%s}, and "+
layer.From, key, m.Module)) "declares no %q in what it contributes or serves",
layer.From, key, m.Module, key, key))
} }
} }
sort.Strings(unused) sort.Strings(unused)
+56 -1
View File
@@ -167,11 +167,45 @@ func TestSettingsThatReachNothingAreNamed(t *testing.T) {
{"id": "conf", "type": "file", "path": "/etc/thing", "content": "plain"}, {"id": "conf", "type": "file", "path": "/etc/thing", "content": "plain"},
}} }}
unused := UnusedSettings(m, []Layer{{From: "node", Values: map[string]any{"port": 1}}}) unused := UnusedSettings(m, []Layer{{From: "node", Values: map[string]any{"port": 1}}})
if len(unused) != 1 || !strings.Contains(unused[0], "no file or contribution to merge it into") { if len(unused) != 1 || !strings.Contains(unused[0], "no file that merges it") {
t.Errorf("settings that reached nothing were not named: %v", unused) t.Errorf("settings that reached nothing were not named: %v", unused)
} }
} }
func TestASettingLandsOnlyWhereSomethingDeclaresIt(t *testing.T) {
// novox/hq 04-ISSUES/173. A module that contributes a route and serves a provision takes a
// setting for a key either declares, and a setting for a key neither declares is stray — it
// would not reach the route or the served fact, so it must be said rather than dropped.
m := Manifest{Module: "mail",
Contributes: map[string]map[string]any{"reverse-proxy": {"host": "mail", "port": 8080}},
Serves: map[string]map[string]any{"smtp": {"host": "mail", "port": 25}},
Resources: []map[string]any{
{"id": "env", "type": "file", "path": "/etc/mail.env", "content": "SITE=${setting:sitename}\n"},
}}
layers := []Layer{{From: "the mesh", Values: map[string]any{
"host": "post", "sitename": "Mail", "website": "https://www.example.tld"}}}
unused := UnusedSettings(m, layers)
if len(unused) != 1 || !strings.Contains(unused[0], `"website"`) {
t.Errorf("only website reaches nothing; named: %v", unused)
}
}
func TestASettingOverridesAServedKeyAndAddsNone(t *testing.T) {
// What a consumer is told is the provider's contract. A setting made for one of the
// provider's files — its site name, its public address — is not part of it.
served, err := Settle(map[string]any{"host": "mail", "port": 25},
[]Layer{{From: "the mesh", Values: map[string]any{"host": "post", "sitename": "Mail"}}})
if err != nil {
t.Fatal(err)
}
if served["host"] != "post" {
t.Errorf("the setting did not override the served host: %v", served)
}
if _, leaked := served["sitename"]; leaked {
t.Errorf("a setting for a file reached the consumers: %v", served)
}
}
func TestContentThatIsNotJSONIsRefusedWhereSomebodyIsLooking(t *testing.T) { func TestContentThatIsNotJSONIsRefusedWhereSomebodyIsLooking(t *testing.T) {
// Rather than on the machine, at apply time, as a file the program cannot read. // Rather than on the machine, at apply time, as a file the program cannot read.
_, err := ApplySettings(file(`this is not json`), nil) _, err := ApplySettings(file(`this is not json`), nil)
@@ -179,3 +213,24 @@ func TestContentThatIsNotJSONIsRefusedWhereSomebodyIsLooking(t *testing.T) {
t.Fatal("a module claiming to merge as JSON shipped something else and was accepted") t.Fatal("a module claiming to merge as JSON shipped something else and was accepted")
} }
} }
func TestAServedValueMayBeTheOperators(t *testing.T) {
// A mail provider serves its domain and an identity provider its issuer; neither is the
// definition's to state (ADR 0155). Filled from the layers, refused by name when unset.
served, err := Settle(map[string]any{"port": 587, "domain": "${setting:domain}"},
[]Layer{{From: "the mesh", Values: map[string]any{"domain": "example.tld"}}})
if err != nil {
t.Fatal(err)
}
if served["domain"] != "example.tld" {
t.Errorf("the operator's value did not fill the served key: %v", served)
}
_, err = Settle(map[string]any{"domain": "${setting:domain}"}, nil)
if err == nil || !strings.Contains(err.Error(), `"domain"`) {
t.Errorf("a served value nothing sets must be refused by name; got %v", err)
}
m := Manifest{Module: "mail", Serves: map[string]map[string]any{"smtp": {"domain": "${setting:domain}"}}}
if unused := UnusedSettings(m, []Layer{{From: "the mesh", Values: map[string]any{"domain": "x"}}}); len(unused) != 0 {
t.Errorf("a setting a served fact asks for is not stray: %v", unused)
}
}
@@ -5,7 +5,20 @@
-- than for the mesh; ADR 0121 decided the rename and deferred it because a delivering seat that stops -- than for the mesh; ADR 0121 decided the rename and deferred it because a delivering seat that stops
-- resolving mid-flight takes a provision away from every consumer. ADR 0122 removed that risk: a seat's -- resolving mid-flight takes a provision away from every consumer. ADR 0122 removed that risk: a seat's
-- former name is an alias that resolves to it forever, a held record follows the rename by cascade, and -- former name is an alias that resolves to it forever, a held record follows the rename by cascade, and
-- a claim written with the old name still holds. So the rename is one update and one alias. -- a claim written with the old name still holds.
--
-- **Both rows may exist when this runs.** A controller whose compiled defaults already carry the new
-- name seeds it as a new seat the moment it can, and on the mesh this was written for that happened
-- before the rename: the first form of this migration renamed into a duplicate key and the control
-- node's prepare failed on every attempt (2026-09-30). So: if the new row is already there, the old
-- row's holding moves to it and the old row goes; otherwise the old row is renamed. Either way the old
-- name becomes an alias.
update seat_holding set seat = 'mesh-artifact-store'
where seat = 'the-artifact-store'
and exists (select 1 from seat where name = 'mesh-artifact-store');
delete from seat
where name = 'the-artifact-store'
and exists (select 1 from seat where name = 'mesh-artifact-store');
update seat set name = 'mesh-artifact-store' where name = 'the-artifact-store'; update seat set name = 'mesh-artifact-store' where name = 'the-artifact-store';
insert into seat_alias (alias, seat) values ('the-artifact-store', 'mesh-artifact-store') insert into seat_alias (alias, seat) values ('the-artifact-store', 'mesh-artifact-store')
on conflict (alias) do update set seat = excluded.seat; on conflict (alias) do update set seat = excluded.seat;