novox/mesh-controller
3
0
Fork 0
Code Issues Pull Requests 2 Releases Wiki Activity

A consumer is a module on a machine, not a machine

Browse Source 
novox/hq 04-ISSUES/022. A credential was keyed by provision, consumer
node and provider node, so "who is asking" was answered by naming a
host. The node this mesh exists to take over runs eight modules against
one database server.

The symptom had two halves and only one was loud. The provider refused,
naming the modules and explaining they would share one credential, which
reads as a decision rather than a limit. The consumer did not refuse: it
resolved cleanly, wrote one module's credential file and left the others
absent — a service that starts and cannot authenticate, with nothing
saying why. That is 021 again on a different axis.

Three modules wanting one database produced one need, carrying whichever
module mentioned it first, because the resolution walk is a work-list
over names. The fan-out now happens in one place, after the walk. The
record path already did this correctly and said why: a consumer here is
a module on a machine. It is the same rule.

Downstream: the secret's key gains the consuming module, the grant file
is named after both halves, needs are matched by provision and module
rather than provision alone, and the provisioners name the role and the
access key after the module. The refusal in ContributionsTo is gone
because there is nothing left to refuse.

Worth stating plainly: without that refusal, gitea's login would have
opened keycloak's database. From the provisioner's side it created
exactly what it was asked to create.

Existing secrets are discarded rather than backfilled. They cannot say
which module they were for, and a secret is remade and delivered to both
ends on the next push — so this costs one rotation and invents nothing.

Also guards the role name against PostgreSQL's 63-byte truncation, which
is a notice rather than an error and would reintroduce exactly this
collision at a length nobody tests.

Three faults injected — the fan-out removed, needs matched by name
alone, the grant file named after the machine — each caught.
This commit is contained in:
jschoubben
2026-09-01 02:40:09 +02:00
parent 1314be5282
commit 0af3ea1acf
14 changed files with 440 additions and 98 deletions
Download Patch File Download Diff File Expand all files Collapse all files
internal/catalogue/brokered_test.go
+7 -3
View File
@@ -242,15 +242,19 @@ func TestAnAppIsToldWhereItsDatabaseIs(t *testing.T) {
}
}
func TestItSaysItCarriesNoCredential(t *testing.T) {
func TestItSaysWhereTheCredentialIsInstead(t *testing.T) {
// A missing field looks like a bug; a stated absence looks like a boundary. Somebody wiring
// this up must not spend an afternoon looking for the password field.
//
// It used to say only that the mesh had no way to issue one, which stopped being true when
// 021 was fixed. A stated absence is only useful while it is accurate — once it is not, it
// sends the reader somewhere there is nothing to find.
got, _ := Resolve(boundShelf(), []string{"meshboard"}, reachable(),
World{Offered: map[string][]Provider{"postgres-database": {{Node: "anchor", At: "anchor.internal"}}}})
told := binding(t, mustDeclare(t, got))
note, _ := told["generated"].(string)
if !strings.Contains(note, "no credential") {
t.Fatalf("the file does not say what it does not carry: %v", note)
if !strings.Contains(note, "not here") || !strings.Contains(note, "secrets") {
t.Fatalf("the file does not say where the credential is instead: %v", note)
}
for key := range told {
if strings.Contains(key, "password") || strings.Contains(key, "secret") {
internal/catalogue/contributes_test.go
+4 -2
View File
@@ -287,7 +287,9 @@ func TestTheManifestNamesTheFileRatherThanCarryingTheCredential(t *testing.T) {
// where to find it — and the readable half therefore stays readable.
out := oneGrant(t)
given := grantedTo(t, out)
if given[0].Secret != "/var/lib/postgres/grants/workstation.secret" {
// Named after the machine and the module, because a consumer is both (novox/hq
// 04-ISSUES/022). The machine alone, two modules on one node wrote to one path.
if given[0].Secret != "/var/lib/postgres/grants/workstation.meshboard.secret" {
t.Fatalf("the manifest does not name the credential's file: %q", given[0].Secret)
}
for _, r := range out {
@@ -302,7 +304,7 @@ func TestTheManifestNamesTheFileRatherThanCarryingTheCredential(t *testing.T) {
func TestTheCredentialItselfLandsSealedBesideIt(t *testing.T) {
for _, r := range oneGrant(t) {
if r["path"] != "/var/lib/postgres/grants/workstation.secret" {
if r["path"] != "/var/lib/postgres/grants/workstation.meshboard.secret" {
continue
}
if r["sealed"] != "c2VhbGVk" {
internal/catalogue/declaration.go
+50 -32
View File
@@ -46,10 +46,14 @@ type OpensPorts interface {
type Grant struct {
// Provision is what was required.
Provision string
// Consumer is the node that will use it, which is also what names the file.
// Consumer is the node that will use it.
Consumer string
// From is the module on that machine which asked, so the provider can name what it creates
// after the thing using it rather than after the machine.
// From is the module on that machine which asked.
//
// **Part of who this credential is for, not a label** (novox/hq 04-ISSUES/022). Together with
// Consumer it names one consumer; the node alone does not, because a machine routinely runs
// several modules wanting the same thing. It is also what the provider names the role or the
// bucket or the client after, so withdrawing one consumer does not take another's away.
From string
// Values are what that module contributed — the name it wants, and anything else the
// provision's own vocabulary defines.
@@ -180,7 +184,12 @@ func (r Resolution) Declaration(with Rendering) ([]map[string]any, error) {
for _, to := range sortedKeys(m.Secrets) {
var found *Needed
for i, n := range r.Needs {
if n.Name == to {
// **This module's need, not the provision's** (novox/hq 04-ISSUES/022). Matching
// on the name alone, every consumer of a provision took whichever credential
// happened to be last in the list — so on a node with two of them, one module
// would be given the other's password and fail to authenticate with a valid
// credential belonging to somebody else.
if n.Name == to && n.For == m.Module {
found = &r.Needs[i]
}
}
@@ -222,9 +231,9 @@ func (r Resolution) Declaration(with Rendering) ([]map[string]any, error) {
continue
}
first = append(first, map[string]any{
"id": GrantID(to, g.Consumer),
"id": GrantID(to, g.Consumer+"."+g.From),
"type": "file",
"path": grantPath(m.Grants[to], g.Consumer),
"path": grantPath(m.Grants[to], g.Consumer, g.From),
"sealed": g.Sealed,
})
}
@@ -232,7 +241,7 @@ func (r Resolution) Declaration(with Rendering) ([]map[string]any, error) {
for _, to := range sortedKeys(m.Binds) {
var found *Needed
for i, n := range r.Needs {
if n.Name == to {
if n.Name == to && n.For == m.Module {
found = &r.Needs[i]
}
}
@@ -389,8 +398,13 @@ type Contribution struct {
//
// Suffixed, so the directory can also hold whatever the module writing it keeps there and so a
// node named like something else in that directory cannot collide with it.
func grantPath(directory, consumer string) string {
return strings.TrimRight(directory, "/") + "/" + consumer + ".secret"
// One file per consumer, and a consumer is a module on a machine (novox/hq 04-ISSUES/022).
//
// Named after both. Named after the machine alone, two modules on one node wrote to one path: the
// second overwrote the first, and the provisioner — reading a directory — saw one consumer where
// there were two.
func grantPath(directory, consumer, module string) string {
return strings.TrimRight(directory, "/") + "/" + consumer + "." + module + ".secret"
}
// contributions collects what every module in this set contributes, by requirement.
@@ -411,7 +425,10 @@ func (r Resolution) contributions(settings SettingsBy, grants []Grant,
if sorted[i].Provision != sorted[j].Provision {
return sorted[i].Provision < sorted[j].Provision
}
return sorted[i].Consumer < sorted[j].Consumer
if sorted[i].Consumer != sorted[j].Consumer {
return sorted[i].Consumer < sorted[j].Consumer
}
return sorted[i].From < sorted[j].From
})
for _, g := range sorted {
if g.From == "" {
@@ -421,7 +438,7 @@ func (r Resolution) contributions(settings SettingsBy, grants []Grant,
}
out[g.Provision] = append(out[g.Provision], Contribution{
From: g.From, Node: g.Consumer, At: g.At, Values: g.Values,
Secret: grantPath(directories[g.Provision], g.Consumer),
Secret: grantPath(directories[g.Provision], g.Consumer, g.From),
})
}
for _, m := range modules {
@@ -495,8 +512,12 @@ func boundFile(n Needed, path string) (map[string]any, error) {
"from": n.From,
"at": where,
"serves": n.Serves,
// **Where the credential is, not what it is.** It stopped being true that the mesh
// cannot issue one when 021 was fixed, and a comment asserting a fact about the mesh that
// has become false is worse than none — somebody reads it and stops looking.
"generated": "by the mesh — do not edit; replaced whenever this changes. " +
"It carries no credential: the mesh has no way to issue one yet",
"The credential is not here: it is sealed, in the file this module's manifest " +
"names under `secrets`",
}, "", " ")
if err != nil {
return nil, err
@@ -507,34 +528,31 @@ func boundFile(n Needed, path string) (map[string]any, error) {
}, nil
}
// ContributionsTo is what this node's set asked of one requirement, settled.
// ContributionsFrom is what one module on this node asked of one requirement, settled.
//
// Exported because a provider's grants are assembled from its consumers' resolutions, one machine
// at a time, and the alternative was for the control plane to reimplement settling.
func (r Resolution) ContributionsTo(requirement string, settings SettingsBy) (
string, map[string]any, error) {
//
// **One module, not one machine** (novox/hq 04-ISSUES/022). This used to take a requirement alone
// and refuse whenever two modules wanted it — correctly, given what it had: the credential was
// keyed by node, so the two would have shared one, and sharing is worse than refusing. But the
// arrangement refused is the ordinary one. A node running eight services against one database is
// not an edge case; it is what a machine looks like. Now each consumer has its own credential and
// there is nothing left to refuse.
func (r Resolution) ContributionsFrom(requirement, module string, settings SettingsBy) (
map[string]any, bool, error) {
all, err := r.contributions(settings, nil, nil)
if err != nil {
return "", nil, err
return nil, false, err
}
given := all[requirement]
if len(given) == 0 {
return "", nil, nil
}
if len(given) > 1 {
// Two modules on one machine wanting the same provision would share one credential, and
// the provider would be told to create one thing under two names. Refused rather than
// resolved by picking, which is the rule everywhere else here.
var who []string
for _, g := range given {
who = append(who, g.From)
for _, g := range all[requirement] {
if g.From == module {
return g.Values, true, nil
}
sort.Strings(who)
return "", nil, fmt.Errorf(
"%s has %d modules asking for %q and they would share one credential: %s",
r.Node, len(given), requirement, strings.Join(who, ", "))
}
return given[0].From, given[0].Values, nil
// Nothing on that machine asks for this any more. Said as "not found" rather than as an
// error: it is how a credential is withdrawn, and the provider removes what nobody asks for.
return nil, false, nil
}
// here is the module on this same machine that answers a requirement, as a binding.
internal/catalogue/resolve.go
+47
View File
@@ -388,6 +388,18 @@ func Resolve(catalogue map[string]Manifest, assigned []string, node Node, world
}
}
// One need per module that wants it, rather than one per name (novox/hq 04-ISSUES/022).
//
// **A consumer is a module on a machine, not a machine.** The walk above is a work-list over
// names, so a requirement three modules share is visited once and produced one need, carrying
// whichever module happened to mention it first. Everything downstream inherited that: one
// credential, named after a node, and the other two consumers given nothing at all — a
// service that resolves cleanly and then cannot authenticate, which is the exact shape of
// 021.
//
// The record pass below already gets this right and says so. It is the same rule.
needs = perConsumer(needs, order, catalogue)
// What is answered by a record rather than by a machine.
//
// A post-pass, deliberately: nothing about it depends on the order requirements were walked
@@ -664,3 +676,38 @@ func providersFirst(order []string, shelf map[string]Manifest) []string {
}
return out
}
// perConsumer turns one need per provision into one need per module that wants it.
//
// Order follows the resolved modules rather than a map, so the same set always produces the same
// needs — a declaration whose contents move for no reason makes every push look like a change.
//
// A need nothing in the set wants is kept as it is rather than dropped. That should not happen;
// if it does, the honest outcome is an extra credential nobody reads, not a consumer silently
// losing the one it depends on.
func perConsumer(needs []Needed, order []string, catalogue map[string]Manifest) []Needed {
out := make([]Needed, 0, len(needs))
for _, n := range needs {
var wanted bool
for _, name := range order {
m, known := catalogue[name]
if !known {
continue
}
for _, want := range m.Wants() {
if want != n.Name {
continue
}
copied := n
copied.For = m.Module
out = append(out, copied)
wanted = true
break
}
}
if !wanted {
out = append(out, n)
}
}
return out
}
internal/catalogue/two_consumers_test.go
+143
View File
@@ -0,0 +1,143 @@
package catalogue
import (
"strings"
"testing"
)
// A node runs several modules that all want one database (novox/hq 04-ISSUES/022).
//
// **The ordinary arrangement, and it could not be planned at all.** The walk that resolves a node
// is a work-list over names, so a requirement three modules shared was visited once and produced
// one need — carrying whichever module mentioned it first. Everything downstream inherited that:
// one credential, keyed by machine, named after a machine by the provisioner.
//
// The symptom had two halves and only one was loud. The provider refused, naming the modules and
// saying they would share one credential, which reads as a decision. The consumer did not: it
// resolved cleanly, wrote one module's credential file, and left the other two absent — a service
// that starts and cannot authenticate, with nothing anywhere saying why. That is the shape of 021
// again, on a different axis.
func threeConsumers() map[string]Manifest {
return shelf(
Manifest{Module: "postgres", Version: "1", Provides: FromAnywhere("postgres-database")},
Manifest{Module: "gitea", Version: "1", Requires: []string{"postgres-database"},
Contributes: map[string]map[string]any{"postgres-database": {"name": "gitea"}},
Secrets: map[string]string{"postgres-database": "/var/lib/gitea/db.secret"}},
Manifest{Module: "keycloak", Version: "1", Requires: []string{"postgres-database"},
Contributes: map[string]map[string]any{"postgres-database": {"name": "keycloak"}},
Secrets: map[string]string{"postgres-database": "/var/lib/keycloak/db.secret"}},
Manifest{Module: "umami", Version: "1", Requires: []string{"postgres-database"},
Contributes: map[string]map[string]any{"postgres-database": {"name": "umami"}},
Secrets: map[string]string{"postgres-database": "/var/lib/umami/db.secret"}},
)
}
func TestEveryConsumerOnANodeGetsItsOwnCredential(t *testing.T) {
got, err := Resolve(threeConsumers(), []string{"gitea", "keycloak", "umami"},
reachable(), World{Offered: onNetwork("anchor")})
if err != nil {
t.Fatal(err)
}
if len(got.Needs) != 3 {
t.Fatalf("three modules want a database and the node has %d need(s): %v",
len(got.Needs), got.Needs)
}
for _, want := range []string{"gitea", "keycloak", "umami"} {
var found bool
for _, n := range got.Needs {
if n.For == want {
found = true
}
}
if !found {
t.Errorf("%s wants a database and no credential is made for it", want)
}
}
}
// Each consumer's own credential reaches its own file. Matching on the provision alone, every
// consumer took whichever need was last — a module handed somebody else's password, which is a
// valid credential and therefore fails in a way that looks like a configuration error.
func TestEachConsumerGetsItsOwnCredentialAndNotAnothersFile(t *testing.T) {
got, err := Resolve(threeConsumers(), []string{"gitea", "keycloak", "umami"},
reachable(), World{Offered: onNetwork("anchor")})
if err != nil {
t.Fatal(err)
}
for i := range got.Needs {
got.Needs[i].Sealed = "sealed-for-" + got.Needs[i].For
}
out, err := got.Declaration(Rendering{})
if err != nil {
t.Fatal(err)
}
for _, want := range []string{"gitea", "keycloak", "umami"} {
var seen bool
for _, r := range out {
if r["path"] != "/var/lib/"+want+"/db.secret" {
continue
}
seen = true
if r["sealed"] != "sealed-for-"+want {
t.Errorf("%s was given %v, which belongs to something else", want, r["sealed"])
}
}
if !seen {
t.Errorf("%s resolved and its credential file was never written", want)
}
}
}
// The provider is told about all three, separately, and names each grant after the module.
func TestAProviderIsToldAboutEveryConsumerOnOneMachine(t *testing.T) {
provider, err := Resolve(shelf(Manifest{
Module: "postgres", Version: "1", Provides: FromAnywhere("postgres-database"),
Grants: map[string]string{"postgres-database": "/var/lib/postgres/grants"},
Receives: map[string]string{"postgres-database": "/var/lib/postgres/grants/mesh.json"},
}), []string{"postgres"}, reachable(), World{})
if err != nil {
t.Fatal(err)
}
var grants []Grant
for _, who := range []string{"gitea", "keycloak", "umami"} {
grants = append(grants, Grant{
Provision: "postgres-database", Consumer: "anchor", From: who,
Values: map[string]any{"name": who}, Sealed: "sealed-for-" + who})
}
out, err := provider.Declaration(Rendering{Grants: grants})
if err != nil {
t.Fatal(err)
}
for _, who := range []string{"gitea", "keycloak", "umami"} {
path := "/var/lib/postgres/grants/anchor." + who + ".secret"
var found bool
for _, r := range out {
if r["path"] == path {
found = true
if r["sealed"] != "sealed-for-"+who {
t.Errorf("%s's grant holds %v", who, r["sealed"])
}
}
}
if !found {
t.Errorf("the provider was never told to create a login for %s", who)
}
}
// And all three appear in the readable manifest, so the provisioner sees three consumers
// where there are three. Named after one machine, they were one path and the last won.
for _, r := range out {
if r["path"] != "/var/lib/postgres/grants/mesh.json" {
continue
}
body := r["content"].(string)
for _, who := range []string{"gitea", "keycloak", "umami"} {
if !strings.Contains(body, `"`+who+`"`) {
t.Errorf("the provider's manifest never mentions %s: %s", who, body)
}
}
}
}