Compare commits

..
Author SHA1 Message Date
jschoubben 1c3f44a526 Two faults found on review
A second Accept value would have overwritten the first, because the header was Set per value rather
than Added. One value is all any caller passes today, so nothing was wrong — but a helper that
quietly keeps only the last of what it was given is a trap for whoever passes two.

And a replayed announcement that could not be written was published as an empty body: a fact on the
mesh that says nothing, which the reader can only log and drop. It is now said and skipped, because a
body that cannot be marshalled is this program's fault rather than the bus's.
2026-09-28 16:32:41 +02:00
mesh-admin 89e152dfe2 Merge pull request 'The mesh says what it applied, and the replay has an address it may use' (#129) from feat/the-mesh-says-what-it-applied into main 2026-09-28 14:07:20 +00:00
jschoubben 1ebad3786c The mesh says what it applied, and the replay has an address it may use
The pipeline was observable from a merge to an artifact and went dark where it touched a machine: a
node's report is control traffic only the control plane reads, so nothing said which version a
machine runs, or that it refused to (novox/hq ADR 0134). The control plane now states both under the
seat it holds — a role's events belong to the role and keep their address when the holder is
replaced — and only when the report is news, because a machine reconciles every minute and a fact per
report would be a fact per minute per machine.

Whether a report is news is the store's answer: it holds the previous one, so the listener returns it
and the server states the fact. That also gives the catch-up replay a subject the controller may
publish: it was published as a module's event from a module called "control-plane", which does not
exist, so the controller's own account refused it and every catalogue that asked what it missed was
answered with nothing.
2026-09-28 16:07:18 +02:00
mesh-admin f2f526a60a Merge pull request 'A module's name may contain a dot, so the derived step adds none' (#128) from fix/a-modules-name-may-contain-a-dot into main 2026-09-28 13:44:21 +00:00
jschoubben 4b4c7e0e0d A module's name may contain a dot, so the derived step adds none
A resource's id is `<module>.<its own id>` and a module's name may itself contain a dot — novox.be is
one — so the owner of a resource is everything before the *last* dot. The preparation step's id used a
dot, which made its owner unreadable by that rule; it uses a hyphen, and the id says what it belongs
to whichever way a reader splits it.
2026-09-28 15:44:18 +02:00
mesh-admin cec792ce9d Merge pull request 'A manifest HEAD says what it accepts, or the registry answers 404' (#127) from fix/a-manifest-head-says-what-it-accepts into main 2026-09-28 11:02:10 +00:00
jschoubben 338d033632 A manifest HEAD says what it accepts, or the registry answers 404
The check that skips copying a base the mesh already holds asked with no Accept header, and a
registry answers a manifest only in a media type the caller named: the same digest answered 200 with
the manifest types and 404 without them. So the builder concluded it held nothing, copied every
vendor base again, and exhausted the public hub's pull limit a second time today.

The test could not have caught it, because the fake registry answered a manifest HEAD regardless of
Accept — more permissive than the thing it stands in for. It is now as strict as a real registry, and
fails without the fix.
2026-09-28 13:02:08 +02:00
mesh-admin 1be926cec4 Merge pull request 'The control plane prepares its own state, like any module' (#126) from feat/the-control-plane-prepares-its-own-state into main 2026-09-28 10:51:30 +00:00
jschoubben 2134768dfe The control plane prepares its own state, like any module
Now that every parser on the mesh knows the word, the control plane's manifest says it. Its schema
stops being a special case: the mesh derives the step from its own resource and gates its server on
it, which is the failure of novox/hq 04-ISSUES/133 closed by the mechanism rather than by a
hand-written step in one manifest.
2026-09-28 12:51:27 +02:00
mesh-admin ef825688ee Merge pull request 'The control plane learns 'prepares' one release before its manifest uses it' (#125) from fix/the-word-ships-before-the-manifest-uses-it into main 2026-09-28 10:49:36 +00:00
jschoubben 77a14360df The control plane learns 'prepares' one release before its manifest uses it
A manifest word has to reach every parser before a manifest carries it. The builder refused
mesh-controller's manifest with `unknown field "prepares"` until it was rebuilt; then the running
control plane could not read the manifest in the build result either, so the build was recorded with
no module and the version never moved. Strict parsing is deliberate (novox/hq 04-ISSUES/003), so the
word ships first and a manifest uses it next: this takes `prepares` back out of the control plane's
own manifest, leaving the code that understands it, and the manifest says it again once this is
running everywhere.
2026-09-28 12:49:33 +02:00
mesh-admin 9be2fb4750 Merge pull request 'A version prepares its state before it runs' (#124) from feat/a-version-prepares-its-state into main 2026-09-28 10:43:17 +00:00
jschoubben 5a963aec10 A version prepares its state before it runs
The mesh derives the preparation from the module's own resource instead of each module hand-writing
a step beside it (novox/hq ADR 0135). A manifest says one word — `prepares` — and the mesh runs that
module's own program in its preparation mode, in the module's own context: the same image, the same
environment, the same mounts, because it is the same code. A published port and a fixed address are
taken away rather than copied, since the version being replaced still holds them.

One word for every kind of module: a Go binary receives `prepare` as its argument, a bundle receives
it through the runtime whose entry takes the same word. The control plane answers it like anything
else — its own schema stops being a special case, and its hand-written step is gone.
2026-09-28 12:43:15 +02:00
mesh-admin 45d1c28a28 Merge pull request 'The control plane migrates before it serves' (#123) from fix/the-control-plane-migrates-before-it-serves into main 2026-09-28 08:27:44 +00:00
jschoubben a3e7683c63 The control plane migrates before it serves
The mesh replaced its own control plane with a build carrying a migration, applied none of it, and
then refused every build it recorded for three quarters of an hour while reporting itself healthy
(novox/hq 04-ISSUES/133). The module now declares the step ADR 0052 prescribes: a run-once
`migrate` before the server, re-run whenever the image moves because the image is part of a step's
digest, and gating — a migration that fails stops the new server from starting rather than letting
it serve against a schema it does not have.
2026-09-28 10:27:42 +02:00
mesh-admin da394b45e6 Merge pull request 'Work slower than the window says so, and one address is the bus's' (#122) from fix/work-longer-than-the-window-says-so into main 2026-09-28 07:51:52 +00:00
jschoubben 2f3bfda8c0 Work slower than the window says so, and one address is the bus's
Three faults the mesh's own logs showed this morning. A handler that outlives the acknowledgement
window was handed its message again while it was still working: acting on a merge builds modules,
minutes against a thirty-second window, so one merge ran the whole catalogue five times over. The
transport now says the work is in progress while it runs, which is where the window belongs.

Everything the mesh hands out — a token, a membership, a person's credential — took its address
from the enrolment setting, which on a mesh that has moved still names the broker it moved from:
the first person issued after the move was handed the retired broker's port. There is one bus, and
its address is the one the control plane is connected to.

And `operator issue` documented an argument order its parser refused.
2026-09-28 09:51:50 +02:00
mesh-admin 208388978a Merge pull request 'A merge rebuilds what it changed, and what packages it' (#121) from feat/a-merge-rebuilds-what-it-changed into main 2026-09-28 07:20:03 +00:00
jschoubben aa771616bb A merge rebuilds what it changed, and what packages it
Three faults in one path. A merge rebuilt every module built from the repository, so one change in
a repository holding twenty-six of them meant twenty-six builds. A merge into a repository a module
only *packages* source from rebuilt nothing — two modules are built from the control plane's own
repository and neither had ever been rebuilt when it moved — because the manifest the mesh keeps
carries no build section, so a build now says which repositories it read and the mesh keeps that
beside what it stood on. And a module handed over by hand could record a repository with no
directory inside it, which is a module nothing can ever rebuild (novox/hq 04-ISSUES/131, /132).

A change inside no module's own directory is a change to what they share, and everything built from
that repository is rebuilt: rebuilding too much is the safe direction, because the fault this whole
path exists for is a mesh that believes it is current and is not.
2026-09-28 09:20:01 +02:00
mesh-admin 1513bbaac9 Merge pull request 'An older merge does not move a source' (#120) from fix/an-older-merge-does-not-move-a-source into main 2026-09-28 03:12:40 +00:00
mesh-admin d6e49dbd68 Merge pull request 'A base the registry already holds is not pulled from upstream again' (#119) from fix/a-mirrored-base-is-not-pulled-twice into main 2026-09-28 02:48:35 +00:00
jschoubben 3756bb3460 A base the registry already holds is not pulled from upstream again
A base is named by digest, and a digest the mesh's registry holds under the module's repository
is the same bytes whatever upstream would say. Asked on every build, the public hub's anonymous
pull limit was reached on the first merge that rebuilt a whole catalogue, and every module whose
base lives there failed on a copy it did not need.
2026-09-28 04:48:32 +02:00
39 changed files with 1320 additions and 117 deletions
+3
View File
@@ -194,6 +194,9 @@ func answer(ctx context.Context, publisher builder.Publisher, on, workspace stri
})
}
result.Against = built.Against
for _, r := range built.Read {
result.Read = append(result.Read, link.ReadRepository{Repository: r.Repository, Ref: r.Ref})
}
fmt.Fprintf(os.Stderr, " built %s from %s\n", built.Manifest.Module, short(built.Commit))
}
}
+18 -8
View File
@@ -106,6 +106,9 @@ func buildOnce(ctx context.Context, args []string) error {
Manifest: built.Manifest,
Against: built.Against,
}
for _, r := range built.Read {
out.Read = append(out.Read, readRepository{Repository: r.Repository, Ref: r.Ref})
}
for _, made := range built.Built {
out.Made = append(out.Made, madeArtifact{Name: made.Name, Kind: made.Kind, Reference: made.Reference})
}
@@ -123,14 +126,21 @@ func buildOnce(ctx context.Context, args []string) error {
// The same fields the mesh records for a build, so a reader comparing a genesis build against an
// ordinary one is comparing the same thing said the same way.
type onceResult struct {
Module string `json:"module"`
Commit string `json:"commit"`
Repository string `json:"repository"`
Path string `json:"path,omitempty"`
Ref string `json:"ref,omitempty"`
Manifest any `json:"manifest"`
Made []madeArtifact `json:"made"`
Against []string `json:"against,omitempty"`
Module string `json:"module"`
Commit string `json:"commit"`
Repository string `json:"repository"`
Path string `json:"path,omitempty"`
Ref string `json:"ref,omitempty"`
Manifest any `json:"manifest"`
Made []madeArtifact `json:"made"`
Against []string `json:"against,omitempty"`
Read []readRepository `json:"read,omitempty"`
}
// readRepository is a repository this build read source from besides the module's own.
type readRepository struct {
Repository string `json:"repository"`
Ref string `json:"ref,omitempty"`
}
type madeArtifact struct {
+1 -1
View File
@@ -88,7 +88,7 @@ func reportsReaching(t *testing.T, open *stores, reachable []link.Reach, held ..
if err := open.inventory.RecordSent(ctx, record.ID, digestOf(body)); err != nil {
t.Fatal(err)
}
if err := (link.Enrolment{Inventory: open.inventory}).Heard(ctx, link.Report{
if _, err := (link.Enrolment{Inventory: open.inventory}).Heard(ctx, link.Report{
Node: "anchor", Applied: []string{"hello-web.x"}, Declared: digestOf(body),
Firewall: "ufw", Held: held, Reachable: reachable,
}); err != nil {
+3
View File
@@ -149,6 +149,9 @@ func buildFrom(result link.BuildResult) inventory.Build {
for _, ref := range result.Against {
kept.Against = append(kept.Against, catalogue.Recorded(ref))
}
for _, r := range result.Read {
kept.Read = append(kept.Read, inventory.ReadRepository{Repository: r.Repository, Ref: r.Ref})
}
var announced []inventory.Artifact
for _, made := range result.Made {
announced = append(announced, inventory.Artifact{
+5 -1
View File
@@ -76,7 +76,10 @@ func run() error {
return pinCommand(ctx, args[1:], true)
case "unpin":
return pinCommand(ctx, args[1:], false)
case "migrate":
// `prepare` is how the mesh asks any module to bring its state to the shape this version needs
// (novox/hq ADR 0135), and the control plane answers it the same way as everything else — its
// own schema is not a special case. `migrate` remains the word a person types.
case "prepare", "migrate":
return migrate(ctx)
case "node":
return nodeCommand(ctx, args[1:])
@@ -138,6 +141,7 @@ func usage() {
fmt.Fprint(os.Stderr, `mesh-controller — the control plane
migrate bring each context's schema up to date
prepare the same, asked the way the mesh asks any module (ADR 0135)
node add <name> [--adopted] create a node record; --adopted: the machine is in use
node list the nodes this mesh knows about
node show <name> what one machine reported it can do, and why
+54 -11
View File
@@ -69,12 +69,20 @@ func moduleCommand(ctx context.Context, args []string) error {
repo := set.String("source", "", "where this module comes from")
ref := set.String("ref", "", "the branch followed there")
commit := set.String("commit", "", "the commit this manifest was read at")
// **Where inside the repository the module is** (novox/hq ADR 0069). A module is a
// repository *and* a directory, and a record that carries only the repository names a
// module.json at its root — so every later build of it looks in the wrong place and fails
// with "no module.json at its root". Nine modules on this mesh were registered that way
// and none of them could be rebuilt (2026-09-28).
path := set.String("path", "", "the module's directory inside that repository")
self := set.Bool("self", false, "the source is a path on the forge holding the git seat")
positionals, err := parseAround(set, args[1:])
if err != nil {
return err
}
if len(positionals) != 1 {
return errors.New("module add <manifest.json> [--source <repo> --ref <branch> --commit <sha>]")
return errors.New("module add <manifest.json> [--source <repo> [--self] [--path P] " +
"--ref <branch> --commit <sha>]")
}
raw, err := os.ReadFile(positionals[0])
if err != nil {
@@ -84,23 +92,24 @@ func moduleCommand(ctx context.Context, args []string) error {
if err != nil {
return err
}
// Provenance together or not at all. A source with no commit cannot be compared against
// anything, so it would record where the module came from and still never be able to say
// the mesh is behind it — which is the one thing recording it is for.
if (*repo == "") != (*commit == "") {
return errors.New("--source and --commit go together: a source with no commit " +
"cannot be compared against anything, and a commit with no source has nothing " +
"to be compared with")
from, err := whereItComesFrom(*repo, *ref, *commit, *path, *self)
if err != nil {
return err
}
if err := inv.RegisterModule(ctx, m, inventory.Source{
Repository: *repo, Ref: *ref, BuiltFrom: *commit,
}); err != nil {
if err := inv.RegisterModule(ctx, m, from); err != nil {
return err
}
fmt.Printf("%s registered", m.Module)
if *commit != "" {
fmt.Printf(" from %s", short(*commit))
}
if *repo != "" && *path == "" {
// Said, not refused: a module really at the root is the ordinary case for a repository
// of its own. But a repository holding many modules and a record naming none of them is
// a module nothing can rebuild, and the person adding it is the one who knows which.
fmt.Printf("\n no directory inside %s, so it is built from that repository's root — "+
"`--path` if the module lives in a directory there", *repo)
}
if len(m.Provides) > 0 {
fmt.Printf(", providing %s", describeOffers(m.Provides))
}
@@ -602,3 +611,37 @@ func issueWith(ctx context.Context, inv *inventory.Inventory, m catalogue.Manife
"machine holding mesh-broker\n")
return nil
}
// whereItComesFrom is the provenance a module handed over by hand records, and what a record must
// say to be worth anything later.
//
// **A module is a repository and a directory inside it** (novox/hq ADR 0069). A record carrying only
// the repository names a module.json at its root, so every later build of it looks in the wrong
// place — nine modules on this mesh were registered that way and none of them could be rebuilt
// (2026-09-28). The directory cannot be checked from here, because the control plane does not clone;
// what can be checked is that the record is whole.
func whereItComesFrom(repository, ref, commit, path string, self bool) (inventory.Source, error) {
// Provenance together or not at all. A source with no commit cannot be compared against
// anything, so it would record where the module came from and still never be able to say the
// mesh is behind it — which is the one thing recording it is for.
if (repository == "") != (commit == "") {
return inventory.Source{}, errors.New("--source and --commit go together: a source with " +
"no commit cannot be compared against anything, and a commit with no source has " +
"nothing to be compared with")
}
// A directory or a forge with no repository is half a location, and the half it keeps is the
// half nothing can be found with.
if repository == "" && (path != "" || self) {
return inventory.Source{}, errors.New("--path and --self say where inside a source and " +
"which forge holds it, so they need --source: without one there is nothing for them " +
"to be part of")
}
from := inventory.Source{Repository: repository, Ref: ref, BuiltFrom: commit, Path: path}
if self {
if err := onASeat(repository); err != nil {
return inventory.Source{}, err
}
from.Seat = gitSeat
}
return from, nil
}
+7 -3
View File
@@ -189,13 +189,17 @@ func readPrivateKey(path string) (string, error) {
func personIssue(ctx context.Context, args []string) error {
set := flag.NewFlagSet("operator issue", flag.ContinueOnError)
invokes := set.String("invokes", "", "the tools this person may call, comma-separated, or * for every one")
if err := set.Parse(args); err != nil {
// Flags on either side of the name, because the usage this command prints puts them after it —
// and the standard parser stops at the first thing that is not a flag, so the order the command
// documents was the one order it refused (2026-09-28).
positionals, err := parseAround(set, args)
if err != nil {
return err
}
if set.NArg() != 1 {
if len(positionals) != 1 {
return errors.New("operator issue <name> --invokes <tool,tool|*>")
}
name := set.Arg(0)
name := positionals[0]
if *invokes == "" {
return errors.New(
"say what this person may call: --invokes mesh-catalog.catalog_tools,gitea.repo_create, " +
+104
View File
@@ -106,3 +106,107 @@ func TestAMergeOlderThanTheLastLookIsHistory(t *testing.T) {
t.Fatal("a merge or a source with no time on it was refused")
}
}
// A module as the catalogue holds it: built from a repository, at a directory inside it.
func fromRepo(module, repository, path string) inventory.Entry {
return inventory.Entry{
Manifest: catalogue.Manifest{Module: module},
Source: inventory.Source{Repository: repository, Path: path, Ref: "main"},
}
}
// A merge rebuilds the modules whose own directories it changed, and everything when what it changed
// is shared. One repository holding many modules is the ordinary case here, and rebuilding all of
// them for a change to one is what exhausted a registry's pull limit the first night this ran.
func TestAMergeRebuildsTheModulesItChanged(t *testing.T) {
const repo = "http://forge.internal:20000/novox/mesh-catalog.git"
gitea := fromRepo("gitea", repo, "modules/gitea")
keycloak := fromRepo("keycloak", repo, "modules/keycloak")
known := []inventory.Entry{gitea, keycloak, fromRepo("plex", repo, "modules/plex")}
candidates := []inventory.Entry{gitea, keycloak}
merge := func(paths []string, truncated bool) link.SourceMoved {
return link.SourceMoved{Owner: "novox", Repo: "mesh-catalog", Base: "main",
Paths: paths, PathsTruncated: truncated}
}
named := func(entries []inventory.Entry) string {
var names []string
for _, e := range entries {
names = append(names, e.Manifest.Module)
}
return strings.Join(names, ",")
}
for _, c := range []struct {
what string
m link.SourceMoved
want string
}{
{"one module's own files", merge([]string{"modules/gitea/index.ts", "modules/gitea/client.ts"}, false), "gitea"},
{"two modules' files", merge([]string{"modules/gitea/index.ts", "modules/keycloak/module.json"}, false), "gitea,keycloak"},
{"a file they share", merge([]string{"tsconfig.json"}, false), "gitea,keycloak"},
{"a module the mesh does not hold", merge([]string{"modules/plex/index.ts"}, false), ""},
{"nothing said about the files", merge(nil, false), "gitea,keycloak"},
{"more files than were listed", merge([]string{"modules/gitea/index.ts"}, true), "gitea,keycloak"},
} {
if got := named(whatTheMergeTouched(candidates, known, c.m)); got != c.want {
t.Errorf("%s: rebuilt %q, wanted %q", c.what, got, c.want)
}
}
}
// A module whose recipe packages source from another repository is affected when that repository
// moves — the manifest the mesh keeps says nothing about it, so the record of what the build read is
// the only thing that can say so.
func TestAModuleIsAffectedByTheRepositoryItPackages(t *testing.T) {
m := link.SourceMoved{Owner: "novox", Repo: "mesh-controller", Base: "main",
CloneURL: "http://forge.internal:20000/novox/mesh-controller.git"}
for _, read := range [][]inventory.ReadRepository{
{{Repository: "http://forge.internal:20000/novox/mesh-controller.git", Ref: "main"}},
{{Repository: "novox/mesh-controller"}},
{{Repository: "https://elsewhere.example/novox/other"}, {Repository: "novox/mesh-controller.git", Ref: "main"}},
} {
if !readsFrom(read, m) {
t.Errorf("%+v was not matched by the merge", read)
}
}
for _, read := range [][]inventory.ReadRepository{
nil,
{{Repository: "novox/mesh-host", Ref: "main"}},
{{Repository: "novox/mesh-controller", Ref: "release"}},
} {
if readsFrom(read, m) {
t.Errorf("%+v was matched by a merge that is not its", read)
}
}
}
// What a module handed over by hand records about where it came from, and what is refused.
func TestWhatAHandedOverModuleRecordsAboutItsSource(t *testing.T) {
// The whole location: a repository on the mesh's own forge, the directory inside it, the branch
// and the commit the manifest was read at.
from, err := whereItComesFrom("novox/mesh-catalog", "main", "c0ffee", "modules/gitea", true)
if err != nil {
t.Fatal(err)
}
if from.Path != "modules/gitea" || from.Seat != "git" || from.Repository != "novox/mesh-catalog" {
t.Fatalf("the source records as %+v", from)
}
// A manifest with no provenance at all is legitimate: fixing something in a hurry.
if from, err := whereItComesFrom("", "", "", "", false); err != nil || from != (inventory.Source{}) {
t.Fatalf("a manifest handed over with no provenance was refused: %+v, %v", from, err)
}
for _, c := range []struct {
what string
repository, ref, commit, path string
self bool
}{
{what: "a source with no commit", repository: "novox/mesh-catalog", commit: ""},
{what: "a commit with no source", commit: "c0ffee"},
{what: "a directory inside nothing", path: "modules/gitea"},
{what: "a forge holding nothing", self: true},
{what: "an address given as a path on the forge", repository: "http://forge.internal:20000/novox/x.git", commit: "c0ffee", self: true},
} {
if _, err := whereItComesFrom(c.repository, c.ref, c.commit, c.path, c.self); err == nil {
t.Errorf("%s was recorded as a source", c.what)
}
}
}
+160 -20
View File
@@ -232,29 +232,67 @@ func (f following) SourceMoved(ctx context.Context, m link.SourceMoved) error {
if err != nil {
return notNow(err)
}
var moved []inventory.Entry
read, err := inv.ReadRepositories(ctx)
if err != nil {
return notNow(err)
}
// Two kinds of module are affected by one merge, and they are affected differently.
//
// A module **built from** this repository and branch has moved: the mesh records the new commit
// as what its source now has, and only what the merge actually changed is rebuilt. A module that
// only **packages source from** it has not moved — its own source is somewhere else, at the
// commit it already records — so it is rebuilt and its record left alone. Writing this commit as
// its source would make it permanently behind a repository its manifest does not come from.
var from, packaging []inventory.Entry
already := 0
for _, e := range entries {
if !sourceIs(e.Source, m) {
continue
switch {
case sourceIs(e.Source, m):
if e.Source.BuiltFrom == m.Commit {
already++
continue
}
// **A merge older than the last look at the source is history, not a move.** The forge
// announces what it finds merged, and an old merge surfacing late would otherwise move
// the recorded head backwards and rebuild everything built from that repository, once
// per old merge (2026-09-28).
if isHistory(m.MergedAt, e.Source.Seen) {
continue
}
from = append(from, e)
case readsFrom(read[e.Manifest.Module], m):
packaging = append(packaging, e)
}
if e.Source.BuiltFrom == m.Commit {
continue
}
// **A merge older than the last look at the source is history, not a move.** The forge
// announces what it finds merged, and an old merge surfacing late would otherwise move the
// recorded head backwards and rebuild everything built from that repository, once per old
// merge (2026-09-28).
if isHistory(m.MergedAt, e.Source.Seen) {
continue
}
if len(from) == 0 && len(packaging) == 0 {
// "Already built from it" and "nothing reads it" are different facts, and reading the first
// as the second sends somebody looking for a broken trigger when the mesh is up to date.
if already > 0 {
fmt.Printf("%s/%s merged into %s (%.8s); %d module(s) the mesh holds are already built "+
"from it\n", m.Owner, m.Repo, m.Base, m.Commit, already)
return nil
}
fmt.Printf("%s/%s merged into %s (%.8s); nothing the mesh holds reads it\n",
m.Owner, m.Repo, m.Base, m.Commit)
return nil
}
// The same judgement for the packaging kind, against the newest look at that repository by
// anything built from it: they keep no record of it themselves, and a replayed old merge should
// not rebuild them either.
if isHistory(m.MergedAt, lastLookAt(entries, m)) {
packaging = nil
}
touched := whatTheMergeTouched(from, entries, m)
for _, e := range touched {
if err := inv.SourceMoved(ctx, e.Manifest.Module, m.Commit); err != nil {
return notNow(err)
}
moved = append(moved, e)
}
moved := append(append([]inventory.Entry{}, touched...), packaging...)
if len(moved) == 0 {
fmt.Printf("%s/%s merged into %s (%.8s); nothing the mesh holds is built from it\n",
m.Owner, m.Repo, m.Base, m.Commit)
fmt.Printf("%s/%s merged into %s (%.8s); it changed nothing any module the mesh holds is "+
"built from\n", m.Owner, m.Repo, m.Base, m.Commit)
return nil
}
against, err := inv.BuiltAgainst(ctx)
@@ -268,6 +306,14 @@ func (f following) SourceMoved(ctx context.Context, m link.SourceMoved) error {
}
fmt.Printf("%s/%s merged into %s (%.8s); building %s\n",
m.Owner, m.Repo, m.Base, m.Commit, strings.Join(names, ", "))
if len(packaging) > 0 {
var also []string
for _, e := range packaging {
also = append(also, e.Manifest.Module)
}
fmt.Printf(" %s package source from it, so they are rebuilt and their own source record "+
"is left where it is\n", strings.Join(also, ", "))
}
var failed []string
for _, e := range ordered {
source := buildSource{Repository: e.Source.Repository, Seat: e.Source.Seat}
@@ -293,16 +339,110 @@ func (f following) SourceMoved(ctx context.Context, m link.SourceMoved) error {
// An empty recorded ref is the repository's default branch, which is what a merge into the base
// branch of the forge's default means.
func sourceIs(s inventory.Source, m link.SourceMoved) bool {
want := strings.ToLower(m.Owner + "/" + m.Repo)
repo := strings.ToLower(strings.TrimSuffix(s.Repository, ".git"))
matches := repo == want || strings.HasSuffix(repo, "/"+want) ||
(m.CloneURL != "" && strings.EqualFold(strings.TrimSuffix(s.Repository, ".git"), strings.TrimSuffix(m.CloneURL, ".git")))
if !matches {
if !sameRepository(s.Repository, m) {
return false
}
return s.Ref == "" || s.Ref == m.Base
}
// sameRepository is whether a recorded repository is the one a merge names, in either spelling it
// may have been recorded in: a path on the git seat, or the URL it was cloned from.
func sameRepository(repository string, m link.SourceMoved) bool {
want := strings.ToLower(m.Owner + "/" + m.Repo)
repo := strings.ToLower(strings.TrimSuffix(repository, ".git"))
return repo == want || strings.HasSuffix(repo, "/"+want) ||
(m.CloneURL != "" && repo == strings.ToLower(strings.TrimSuffix(m.CloneURL, ".git")))
}
// readsFrom is whether a module's build read the repository a merge names: the second repository its
// recipe packages source from. Its ref must be the branch that moved, or unset — the same rule a
// module's own source follows.
func readsFrom(read []inventory.ReadRepository, m link.SourceMoved) bool {
for _, r := range read {
if sameRepository(r.Repository, m) && (r.Ref == "" || r.Ref == m.Base) {
return true
}
}
return false
}
// lastLookAt is the most recent look at this repository by anything built from it.
func lastLookAt(entries []inventory.Entry, m link.SourceMoved) time.Time {
var newest time.Time
for _, e := range entries {
if sameRepository(e.Source.Repository, m) && e.Source.Seen.After(newest) {
newest = e.Source.Seen
}
}
return newest
}
// whatTheMergeTouched narrows the modules built from a repository to the ones the merge changed.
//
// **A change inside no module's own directory is a change to what they share.** The forge lists the
// files a merge changed; a module is affected when one of them is inside its own directory, when it
// is built from the repository's root — everything there is its source — or when some changed file
// belongs to no module's directory at all, which is how a shared file, a build recipe or a
// dependency at the root rebuilds everything built from that repository.
//
// A change inside *another* module's directory is that module's business and not this one's, even
// when the mesh does not hold that module: `known` is every module this repository is known to hold,
// whatever branch it was registered from. That is also the limit of this — a repository whose shared
// code sits inside a directory the mesh has never seen a module in reads as shared, and everything
// is rebuilt. Rebuilding too much is the safe direction: the fault this whole path exists for is a
// mesh that believes it is current and is not (novox/hq 04-ISSUES/131).
func whatTheMergeTouched(candidates, known []inventory.Entry, m link.SourceMoved) []inventory.Entry {
// Nothing said about the files, or not all of them said: everything built from it is affected.
if len(m.Paths) == 0 || m.PathsTruncated {
return candidates
}
var dirs []string
for _, e := range known {
if e.Source.Path != "" && sameRepository(e.Source.Repository, m) {
dirs = append(dirs, e.Source.Path)
}
}
for _, p := range m.Paths {
if !insideAny(p, dirs) {
return candidates
}
}
var out []inventory.Entry
for _, e := range candidates {
if e.Source.Path == "" || anyInside(m.Paths, e.Source.Path) {
out = append(out, e)
}
}
return out
}
// inside is whether a changed file is in a directory: that directory itself, or under it.
func inside(path, dir string) bool {
dir = strings.Trim(dir, "/")
path = strings.TrimPrefix(path, "/")
return path == dir || strings.HasPrefix(path, dir+"/")
}
// insideAny is whether a changed file is in any of these directories.
func insideAny(path string, dirs []string) bool {
for _, dir := range dirs {
if inside(path, dir) {
return true
}
}
return false
}
// anyInside is whether any of these changed files is in a directory.
func anyInside(paths []string, dir string) bool {
for _, p := range paths {
if inside(p, dir) {
return true
}
}
return false
}
// orderByBases is the entries with every base before what stands on it: a module whose build stood
// on another's artifact comes after that module. Entries outside the set are not waited for — they
// are not being rebuilt. Stable for what has no order between it.
+13
View File
@@ -66,6 +66,19 @@ func FromEnvironment() (Broker, error) {
if err != nil {
return Broker{}, err
}
// **One bus, one address** (novox/hq ADR 0131). Everything the mesh hands out — a token, a
// machine's membership, a person's credential — must name the bus the control plane itself is
// connected to; the setting above predates the move and, on a mesh that has moved, still names
// the broker it moved from. The first person issued after the move was handed the retired
// broker's port and could not connect to anything (2026-09-28).
//
// Read from the credential rather than from a second setting somebody keeps in step: the
// control plane cannot be wrong about where it is connected.
if bus, on, err := OnNATS(); err == nil && on {
if where := strings.TrimPrefix(BareAddress(bus), "nats://"); where != "" {
address = where
}
}
return Broker{Address: address, Fingerprint: fingerprint}, nil
}
+8
View File
@@ -181,6 +181,14 @@ func PermissionsFor(p Principal) (Permissions, error) {
for _, seat := range meshSeatsTheControllerUses {
pub = append(pub, "mesh.seat."+seat+".accept.>")
}
// **And what the mesh says it did** (novox/hq ADR 0134). The control plane states its own
// facts under the seat it holds, because a role's events belong to the role and keep their
// address while the holder is replaced. Named one by one rather than as a whole namespace:
// least authority, and a fact nothing states is authority nobody uses.
for _, event := range ControllerStates {
pub = append(pub, seatEventSubject(ControllerSeat, event))
}
// Every module's tools: **the control plane is the way in** (novox/hq ADR 0095). A person
// or an agent asks through it and every question passes one process where an audit
// belongs — so it, alone among principals, may call any tool by name. The first `ask` on
+30
View File
@@ -0,0 +1,30 @@
package broker_test
import (
"slices"
"testing"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/link"
)
// The facts the control plane states are named twice — in the grant that permits them and in the code
// that states them — because `link` imports `broker` and the dependency cannot go the other way. So a
// test keeps them agreeing: a subject the grant omits is refused at the moment the mesh has something
// to say, and one the grant adds that nothing states is authority nobody uses.
//
// An external test package, because it may import both while neither imports the other.
func TestTheFactsTheGrantPermitsAreTheFactsTheMeshStates(t *testing.T) {
if broker.ControllerSeat != link.MeshControllerSeat {
t.Fatalf("the grant is written for the %q seat and the mesh states its facts under %q",
broker.ControllerSeat, link.MeshControllerSeat)
}
for _, event := range []string{link.KeyApplied, link.KeyRefused, link.KeyBuiltBefore} {
if !slices.Contains(broker.ControllerStates, event) {
t.Errorf("the mesh states %q and its account may not publish it", event)
}
}
if len(broker.ControllerStates) != 3 {
t.Errorf("the grant permits %v, which is more than the mesh states", broker.ControllerStates)
}
}
+11
View File
@@ -160,6 +160,17 @@ func Overlaps() []string {
// ack subject is derived from (nats.go: `$JS.ACK.<stream>.controller.>`).
const ControllerName = "controller"
// ControllerSeat is the role the control plane holds, and ControllerStates are the facts it states
// under it (novox/hq ADR 0134).
//
// **Written here as well as in `link`, and a test keeps them agreeing.** `link` imports `broker`, so
// `broker` cannot import `link`; a grant naming a subject the controller never publishes is authority
// nobody uses, and a controller publishing one the grant omits is refused at the moment it has
// something to say.
const ControllerSeat = "mesh-controller"
var ControllerStates = []string{"applied", "refused", "built-before"}
// ControllerFollows are the events the controller reacts to: the catalogue saying a module's
// current version moved, and a catalogue that has just started saying it may have missed builds.
//
+1 -1
View File
@@ -24,7 +24,7 @@ accounts {
jetstream: enabled
users = [
{ user: "controller", password: "$2a$11$cccccccccccccccccccccc", permissions: {
publish: { allow: ["$JS.ACK.CONTROL.controller.>", "$JS.ACK.EVENTS.controller.>", "$JS.API.>", "_INBOX.enrol.>", "mesh.control.>", "mesh.mod.*.tool.>", "mesh.node.>", "mesh.seat.mesh-build-machine.accept.>"] }
publish: { allow: ["$JS.ACK.CONTROL.controller.>", "$JS.ACK.EVENTS.controller.>", "$JS.API.>", "_INBOX.enrol.>", "mesh.control.>", "mesh.mod.*.tool.>", "mesh.node.>", "mesh.seat.mesh-build-machine.accept.>", "mesh.seat.mesh-controller.event.applied", "mesh.seat.mesh-controller.event.built-before", "mesh.seat.mesh-controller.event.refused"] }
subscribe: { allow: ["$JS.API.>", "_DELIVER.controller", "_DELIVER.controller.>", "_INBOX.controller.>", "mesh.control.>", "mesh.mod.gitea.event.pull.merged", "mesh.mod.mesh-catalog.event.catching-up", "mesh.mod.mesh-catalog.event.upgraded", "mesh.seat.mesh-build-machine.event.built"] }
allow_responses: { max: 1, ttl: "1m" }
} }
+35 -1
View File
@@ -61,6 +61,12 @@ type Result struct {
Commit string
// Built is each artifact, for reporting.
Built []catalogue.Built
// Read is every repository this build read source from besides the module's own — the second
// repository an artifact's recipe names (ArtifactContext). Reported because the manifest the
// mesh keeps carries no build section, so nothing else could say that a merge there is a
// change to this module (novox/hq 04-ISSUES/131).
Read []catalogue.ArtifactContext
}
// GitCredential is the forge credential a clone may present when the server asks for one.
@@ -220,7 +226,7 @@ func Build(ctx context.Context, run Runner, publish Publisher,
}
say("done", "%s at %s — %d artifact(s) pinned", manifest.Module, short(commit), len(built))
return Result{Manifest: resolved, Commit: commit, Built: built,
Against: against(within, manifest, stoodOn)}, nil
Against: against(within, manifest, stoodOn), Read: readBy(manifest)}, nil
}
// Log is where a build says what it is doing, step by step. Nil is silent — the tests pass none,
@@ -1016,3 +1022,31 @@ func instructions(recipe string) []string {
flush()
return out
}
// readBy is every repository other than the module's own that this build's recipes read source from,
// each once and in a fixed order, so two builds of one commit report the same thing the same way.
func readBy(manifest catalogue.Manifest) []catalogue.ArtifactContext {
if manifest.Build == nil {
return nil
}
seen := map[string]bool{}
var out []catalogue.ArtifactContext
for _, a := range manifest.Build.Artifacts {
if a.Context == nil || a.Context.Repository == "" {
continue
}
key := a.Context.Repository + "#" + a.Context.Ref
if seen[key] {
continue
}
seen[key] = true
out = append(out, *a.Context)
}
sort.Slice(out, func(i, j int) bool {
if out[i].Repository != out[j].Repository {
return out[i].Repository < out[j].Repository
}
return out[i].Ref < out[j].Ref
})
return out
}
+14
View File
@@ -180,6 +180,20 @@ func (r Registry) MirrorImage(ctx context.Context, from, repository string) (str
if err != nil {
return "", err
}
// **Already held is already mirrored.** A base is named by digest, and a digest this registry
// holds under the module's repository is the same bytes whatever upstream would say — so
// upstream is not asked. Asked every build, the public hub's anonymous pull limit was reached
// on the first merge that rebuilt a whole catalogue (2026-09-28), and every module whose base
// lives there failed on a copy it did not need.
if strings.HasPrefix(where.reference, "sha256:") {
held, err := r.has(ctx, "http://"+r.Address+"/v2/"+repository+"/manifests/"+where.reference, manifestAccept)
if err != nil {
return "", fmt.Errorf("asking %s whether it holds %s: %w", r.Address, from, err)
}
if held {
return r.Address + "/" + repository + "@" + where.reference, nil
}
}
src := &source{client: r.client()}
digest, err := r.copyManifest(ctx, src, where, where.reference, repository)
if err != nil {
+38
View File
@@ -103,6 +103,20 @@ func (m *theMeshsRegistry) handler() http.Handler {
m.mu.Lock()
defer m.mu.Unlock()
switch {
case r.Method == http.MethodHead && strings.Contains(r.URL.Path, "/manifests/"):
// **As strictly as a real registry.** A manifest is answered only in a media type the
// caller named; a request with no Accept is answered as if nothing were there. The fake
// used to answer regardless, which is why it could not catch a check that asked without
// one — and the mesh copied every base again (2026-09-28).
if !strings.Contains(r.Header.Get("Accept"), "manifest") && !strings.Contains(r.Header.Get("Accept"), "index") {
w.WriteHeader(http.StatusNotFound)
return
}
if _, ok := m.manifests[r.URL.Path[strings.LastIndex(r.URL.Path, "/")+1:]]; ok {
w.WriteHeader(http.StatusOK)
} else {
w.WriteHeader(http.StatusNotFound)
}
case r.Method == http.MethodHead && strings.Contains(r.URL.Path, "/blobs/"):
if _, ok := m.blobs[r.URL.Path[strings.LastIndex(r.URL.Path, "/")+1:]]; ok {
w.WriteHeader(http.StatusOK)
@@ -219,3 +233,27 @@ func TestATagBeforeTheDigestIsNotPartOfTheRepository(t *testing.T) {
t.Fatalf("got %+v", got)
}
}
// A base this registry already holds by digest is not asked of upstream at all: the public hub
// limits anonymous pulls, and a catalogue rebuilt on one merge asked it once per module.
func TestABaseAlreadyHeldIsNotAskedOfUpstream(t *testing.T) {
src, indexDigest, _ := anUpstreamRegistry(t)
dst := &theMeshsRegistry{blobs: map[string][]byte{}, manifests: map[string][]byte{}}
dstServer := httptest.NewServer(dst.handler())
defer dstServer.Close()
address := strings.TrimPrefix(dstServer.URL, "http://")
r := Registry{Address: address, HTTP: src.Client()}
host := strings.TrimPrefix(src.URL, "http://")
if _, err := r.MirrorImage(context.Background(), host+"/library/thing:latest", "hello-web/server"); err != nil {
t.Fatal(err)
}
// Upstream gone: the pinned base is answered from what the mesh holds.
src.Close()
reference, err := r.MirrorImage(context.Background(), host+"/library/thing@"+indexDigest, "hello-web/server")
if err != nil {
t.Fatalf("a base the registry holds was asked of an upstream that is gone: %v", err)
}
if reference != address+"/hello-web/server@"+indexDigest {
t.Fatalf("pinned as %q", reference)
}
}
+13 -1
View File
@@ -122,11 +122,23 @@ func (r Registry) PublishArchive(ctx context.Context, repository string, body []
return final, nil
}
func (r Registry) has(ctx context.Context, url string) (bool, error) {
// has is whether this registry already holds what is at that URL.
//
// **A manifest HEAD must say what it accepts.** A registry answers a manifest request only in a media
// type the caller named, and a bare HEAD — no Accept at all — is answered 404 for a manifest it holds
// perfectly well. Measured against the mesh's own registry (2026-09-28): the same digest answered 200
// with the manifest media types and 404 without them, so a check written without them concluded the
// registry held nothing, copied every base again, and exhausted the public hub's pull limit. A blob
// needs no Accept, which is why this went unnoticed: the same helper was right for blobs and wrong
// for manifests.
func (r Registry) has(ctx context.Context, url string, accept ...string) (bool, error) {
request, err := http.NewRequestWithContext(ctx, http.MethodHead, url, nil)
if err != nil {
return false, err
}
for _, media := range accept {
request.Header.Add("Accept", media)
}
response, err := r.client().Do(request)
if err != nil {
return false, fmt.Errorf("cannot reach the registry at %s: %w", r.Address, err)
+21
View File
@@ -179,3 +179,24 @@ func TestTheBasesABuildWasHandedAreWhatItStoodOn(t *testing.T) {
t.Fatalf("the bases the build was handed were not what it stood on: %v", got)
}
}
// What a build read besides its module's own repository is the second repository its recipes name,
// each once: a module that packages source living elsewhere is affected when that source moves.
func TestWhatABuildReadIsTheRepositoriesItsRecipesName(t *testing.T) {
elsewhere := catalogue.ArtifactContext{Repository: "http://forge.internal:20000/novox/mesh-controller.git", Ref: "main"}
manifest := catalogue.Manifest{
Module: "builder",
Build: &catalogue.Build{Artifacts: []catalogue.Artifact{
{Name: "server", Kind: catalogue.ArtifactImage, From: "Dockerfile", Context: &elsewhere},
{Name: "tools", Kind: catalogue.ArtifactImage, From: "Dockerfile", Context: &elsewhere},
{Name: "config", Kind: catalogue.ArtifactArchive, From: "etc"},
}},
}
read := readBy(manifest)
if len(read) != 1 || read[0] != elsewhere {
t.Fatalf("the repositories this build read are %+v", read)
}
if readBy(catalogue.Manifest{Module: "gitea", Build: &catalogue.Build{}}) != nil {
t.Fatal("a module whose recipes name no other repository read one")
}
}
+88
View File
@@ -623,6 +623,9 @@ func (r Resolution) compose(with Rendering, owner map[string]string) ([]map[stri
// one of them as its environment without saying so (ADR 0086, issue 041).
secretFiles := secretFilesOf(resources)
// Which of this module's resources its preparation runs before, if it prepares anything.
prepareBefore := preparationTarget(m)
for _, unsettled := range resources {
resource, err := ApplySettings(unsettled, with.Settings[m.Module])
if err != nil {
@@ -708,6 +711,18 @@ func (r Resolution) compose(with Rendering, owner map[string]string) ([]map[stri
if renamed := reflectsRenamed(m.Module, resource["reload-on"]); renamed != nil {
copied["reload-on"] = renamed
}
// **A version prepares its state before it runs** (novox/hq ADR 0135). Derived from the
// module's own resource rather than declared beside it: what prepares the state is the
// module's own code, so what it is given has to be what that code is given — and a
// second resource written by hand is a second copy to drift from the first. Placed
// immediately before it, because a run-once step stops everything the declaration
// places after it (ADR 0052), which is how a version whose preparation failed does not
// serve.
if prepareBefore != "" && fmt.Sprint(resource["id"]) == prepareBefore {
step := prepared(copied)
owner[fmt.Sprint(step["id"])] = m.Module
out = append(out, step)
}
owner[fmt.Sprint(copied["id"])] = m.Module
out = append(out, copied)
}
@@ -1610,3 +1625,76 @@ func atMachinePort(serves map[string]any, module string, ports map[string]map[in
func AtPublishedPort(values map[string]any, module string, published map[int]int) map[string]any {
return atMachinePort(values, module, map[string]map[int]int{module: published})
}
// PreparationArgument is how the mesh asks a module to prepare its state: one word, to the module's
// own program, whatever that program is (novox/hq ADR 0135).
//
// **One word for every kind of module.** A module built as a Go binary receives it as its argument;
// one built as a bundle receives it through the runtime, whose entry takes the same word. So the
// mesh has one way of asking and a module has one way of answering, and neither learns the other's
// shape.
const PreparationArgument = "prepare"
// preparationTarget is the resource a module's preparation runs before: its own workload.
//
// The first container carrying an artifact this module built, and not itself a step — that is the
// thing that runs the module's code, and therefore the thing whose state must be ready. Empty when
// the module prepares nothing, or when nothing it declares could run its code.
//
// **A module with two own workloads gates the first of them.** Five modules in the catalogue declare
// more than one container of their own, none of them preparing anything today. If one ever does and
// its second workload shares the state, the gate is in front of the first — stated here because the
// alternative is a field asking an author to restate what the mesh can see.
func preparationTarget(m Manifest) string {
if !m.Prepares {
return ""
}
for _, r := range m.Resources {
if fmt.Sprint(r["type"]) != "container" || !ownArtifact(r, m.Module) {
continue
}
if once, _ := r["run-once"].(bool); once {
continue
}
return fmt.Sprint(r["id"])
}
return ""
}
// ownArtifact is whether a resource runs something this module built, in either spelling a manifest
// may be in: naming the artifact, before a build resolved it, or carrying the reference a build
// recorded — this mesh's own store, under this module's name.
func ownArtifact(resource map[string]any, module string) bool {
if named, _ := resource["artifact"].(string); named != "" {
return true
}
image, _ := resource["image"].(string)
return strings.HasPrefix(image, ArtifactStoreScheme+module+"/")
}
// prepared is the module's own resource as the step that prepares its state: the same image, the same
// context, run to completion with the mesh's preparation argument.
//
// Three things are taken away rather than copied, each because the step runs while the version it
// prepares for is still running. A published port cannot be bound twice, and a step that tried would
// fail for a reason that has nothing to do with the state. A fixed address cannot be held twice, for
// the same reason. And a cadence is what a step is the opposite of: a container runs once and gates,
// or on a schedule, or stays up, never two (ADR 0053).
func prepared(from map[string]any) map[string]any {
step := map[string]any{}
for k, v := range from {
step[k] = v
}
// **A hyphen, not a dot.** A resource's id is `<module>.<its own id>`, and a module's name may
// itself contain a dot (`novox.be`), so the module is everything before the *last* dot — which
// only works if what the mesh derives adds no dot of its own.
step["id"] = fmt.Sprint(from["id"]) + "-prepare"
step["name"] = fmt.Sprint(from["name"]) + "-prepare"
step["run-once"] = true
step["args"] = []any{PreparationArgument}
delete(step, "ports")
delete(step, "ip")
delete(step, "schedule")
delete(step, "reload-on")
return step
}
+24
View File
@@ -225,6 +225,19 @@ type Manifest struct {
// subscription to the queue it writes to (design 29 §2).
Uses []string `json:"uses,omitempty"`
// Prepares says this module has state that must be brought to the shape this version needs
// before this version runs, and that the module's own code does it (novox/hq ADR 0135).
//
// **A word, not an arrangement.** The mesh runs the module's own program in its preparation
// mode, in the module's own context — every binding, credential and setting its code receives,
// because it *is* its code. Nothing here names a container, a command, a mount or a variable:
// the module already said all of that once, and a second copy is a second thing to drift.
//
// **Declared, never inferred.** The control plane cannot read what is inside an artifact, so a
// module that ships a migration and does not say this breaks on its first upgrade. That is
// stated in the record rather than guarded here, because nothing mechanical can guard it.
Prepares bool `json:"prepares,omitempty"`
// Tools are the tools this module answers — request and reply, awaited.
//
// **New, and not `serves`**, which this manifest already uses for the facts a consumer needs
@@ -1276,6 +1289,17 @@ func ParseManifest(raw []byte) (Manifest, error) {
"program that reads what the mesh delivered and reconciles",
m.Module, r["id"]))
}
// **A module that prepares its state must have code the mesh can run** (novox/hq ADR 0135). The
// preparation is the module's own program in its preparation mode, so it is derived from the
// resource that runs that program — and a module declaring none has asked for something the mesh
// cannot compose. Said here, where the manifest is read, rather than by a declaration that
// quietly prepares nothing.
if m.Prepares && preparationTarget(m) == "" {
problems = append(problems, fmt.Sprintf(
"%s says it prepares its state, and declares no container running an artifact it built — "+
"the preparation is this module's own program, so there has to be one for the mesh to "+
"run it in", m.Module))
}
// **A run-once container is a step the host runs to completion** (novox/hq ADR 0052). It is a
// boolean modifier on the container shape — the host runs the container, requires it to exit 0,
// and starts whatever the declaration places after it only once it has. A value that is not a
+141
View File
@@ -0,0 +1,141 @@
package catalogue
import (
"encoding/json"
"fmt"
"strings"
"testing"
)
// A module version prepares its state before it runs (novox/hq ADR 0135).
//
// What the mesh derives is the module's own resource, run once with one word, placed immediately in
// front of the thing it prepares for. What matters in these tests is that the derivation is a copy
// rather than a second description: the failure it replaces was a hand-written step repeating six
// fields of the resource it preceded, each free to drift from it.
func aPreparingModule() Manifest {
return Manifest{
Module: "gitea",
Prepares: true,
Resources: []map[string]any{
{"id": "state", "type": "directory", "path": "/var/lib/gitea", "mode": "0700"},
{"id": "server", "type": "container", "name": "mesh-gitea-server",
"image": "gitea/gitea@" + digest, "ports": []any{"3000:3000"}},
{"id": "runtime", "type": "container", "name": "mesh-gitea",
"image": ArtifactStoreScheme + "gitea/runtime@" + digest, "network": "host",
"env": map[string]any{"MESH_GITEA_STATE_DIR": "/run/state"},
"volumes": []any{"/var/lib/gitea:/run/state:ro"},
"ports": []any{"9000:9000"}},
},
}
}
func declaredFor(t *testing.T, m Manifest) []map[string]any {
t.Helper()
// The manifest as the mesh holds it: a build resolved the module's own artifact into the
// reference it recorded, which is also how the composition knows whose code a resource runs.
out, err := Resolution{Node: "anchor", Modules: []Manifest{m}}.Declaration(
Rendering{ArtifactStore: "anchor.internal:5100"})
if err != nil {
t.Fatal(err)
}
return out
}
func idsOf(resources []map[string]any) []string {
var ids []string
for _, r := range resources {
ids = append(ids, fmt.Sprint(r["id"]))
}
return ids
}
// The step runs the module's own code, and comes immediately before it — not before the upstream
// server the module packages, which may be the very thing the state lives in.
func TestThePreparationRunsTheModulesOwnCodeAndComesRightBeforeIt(t *testing.T) {
out := declaredFor(t, aPreparingModule())
ids := idsOf(out)
at := -1
for i, id := range ids {
if id == "gitea.runtime-prepare" {
at = i
}
}
if at < 0 {
t.Fatalf("nothing prepares this module's state: %v", ids)
}
// A module's name may contain a dot, so a resource's module is everything before the last one —
// which the derived id must not add to, or a machine reads the wrong owner from it.
if strings.Count("gitea.runtime-prepare", ".") != 1 {
t.Fatal("the derived id adds a dot, so what owns it cannot be read from it")
}
if ids[at+1] != "gitea.runtime" {
t.Fatalf("the preparation is not immediately before the module's own code: %v", ids)
}
for _, id := range ids[:at] {
if id == "gitea.runtime" {
t.Fatalf("the module's own code runs before its state is prepared: %v", ids)
}
}
}
// It is given exactly what the module's own code is given. Asserted field by field against the
// resource it was derived from, because writing it twice is the fault this replaces.
func TestThePreparationIsGivenWhatTheModuleIsGiven(t *testing.T) {
out := declaredFor(t, aPreparingModule())
declared := byID(out)
step, workload := declared["gitea.runtime-prepare"], declared["gitea.runtime"]
if step == nil || workload == nil {
t.Fatalf("expected both, got %v", idsOf(out))
}
for _, field := range []string{"image", "network", "env", "volumes", "type"} {
if fmt.Sprint(step[field]) != fmt.Sprint(workload[field]) {
t.Errorf("the preparation's %s is %v and the module's is %v", field, step[field], workload[field])
}
}
if once, _ := step["run-once"].(bool); !once {
t.Error("the preparation is not a step, so nothing waits for it and nothing is gated by it")
}
if fmt.Sprint(step["args"]) != fmt.Sprint([]any{PreparationArgument}) {
t.Errorf("the preparation is asked for as %v", step["args"])
}
if fmt.Sprint(step["name"]) == fmt.Sprint(workload["name"]) {
t.Error("the preparation and the workload have one name, so one removes the other")
}
// A published port cannot be bound twice, and the version being replaced is still running.
if _, published := step["ports"]; published {
t.Errorf("the preparation publishes a port the running version holds: %v", step["ports"])
}
}
// A module that says nothing about preparing gets nothing, which is most modules.
func TestAModuleThatPreparesNothingGetsNoStep(t *testing.T) {
m := aPreparingModule()
m.Prepares = false
for _, id := range idsOf(declaredFor(t, m)) {
if id == "gitea.runtime-prepare" {
t.Fatal("a module that prepares nothing was given a preparation")
}
}
}
// A module whose own code the mesh cannot find has nothing to ask, and saying so where the manifest
// is read beats a declaration that quietly prepares nothing.
func TestAModuleThatPreparesAndRunsNoneOfItsOwnCodeIsRefused(t *testing.T) {
m := Manifest{
Module: "gitea",
Prepares: true,
Resources: []map[string]any{
// Only the upstream server it packages: nothing here runs gitea's own code.
{"id": "server", "type": "container", "name": "mesh-gitea-server", "image": "gitea/gitea@" + digest},
},
}
raw, err := json.Marshal(m)
if err != nil {
t.Fatal(err)
}
if _, err := ParseManifest(raw); err == nil {
t.Fatal("a module that prepares its state with nothing of its own to run was accepted")
}
}
+54 -3
View File
@@ -36,12 +36,21 @@ type Build struct {
// Against is every artifact this build stood on, as references rather than module names —
// what makes a build edge derived rather than declared (ADR 0009).
Against []string
// Read is every repository this build read source from besides the module's own (novox/hq
// 04-ISSUES/131), at the ref it read.
Read []ReadRepository
// Failed is the builder's own words, empty when it worked.
Failed string
Made []Artifact
At time.Time
}
// ReadRepository is a repository a build read source from besides the module's own.
type ReadRepository struct {
Repository string `json:"repository"`
Ref string `json:"ref,omitempty"`
}
// Artifact is one thing a build published.
type Artifact struct {
Name string `json:"name"`
@@ -66,17 +75,21 @@ func (i *Inventory) RecordBuild(ctx context.Context, b Build) error {
if err != nil {
return err
}
read, err := json.Marshal(b.Read)
if err != nil {
return err
}
var module *string
if b.Module != "" {
module = &b.Module
}
_, err = i.store.Pool().Exec(ctx,
`insert into build (id, repository, ref, module, commit_hash, built_on, failed, made,
source_path, manifest, built_against)
values ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11)
source_path, manifest, built_against, built_contexts)
values ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12)
on conflict (id) do nothing`,
b.ID, b.Repository, b.Ref, module, b.Commit, b.On, b.Failed, made,
b.Path, manifestOrNil(b.Manifest), against)
b.Path, manifestOrNil(b.Manifest), against, read)
return err
}
@@ -199,6 +212,44 @@ func (i *Inventory) BuiltAgainst(ctx context.Context) (map[string][]string, erro
return against, rows.Err()
}
// ReadRepositories is what each module's newest successful build read source from besides its own
// repository, by module name.
//
// The mirror of BuiltAgainst, and derived the same way and for the same reason: a merge into a
// repository a module only packages is a change to that module, and the manifest the mesh keeps
// carries nothing that would say so (novox/hq 04-ISSUES/131).
func (i *Inventory) ReadRepositories(ctx context.Context) (map[string][]ReadRepository, error) {
rows, err := i.store.Pool().Query(ctx,
`select distinct on (module) module, built_contexts
from build
where module is not null and module <> '' and failed = ''
order by module, at desc`)
if err != nil {
return nil, err
}
defer rows.Close()
read := map[string][]ReadRepository{}
for rows.Next() {
var module string
var raw []byte
if err := rows.Scan(&module, &raw); err != nil {
return nil, err
}
if len(raw) == 0 {
continue
}
var of []ReadRepository
if err := json.Unmarshal(raw, &of); err != nil {
continue
}
if len(of) > 0 {
read[module] = of
}
}
return read, rows.Err()
}
// manifestOrNil keeps the difference between "declared nothing" and "predates this being kept".
//
// A build recorded before the mesh kept manifests has no manifest, and that is not the same as one
+33
View File
@@ -136,3 +136,36 @@ func ids(builds []Build) []string {
}
return out
}
// What a build read besides its module's own repository comes back for the newest build of each
// module, and only for builds that worked. Nothing recorded is absent rather than empty, which is how
// a build made before the mesh kept this is told from one that read nothing (novox/hq 04-ISSUES/131).
func TestWhatABuildReadComesBackForTheNewestBuildOfEachModule(t *testing.T) {
inv := fresh(t)
ctx := context.Background()
older := aBuild("older", "builder", "")
older.Read = []ReadRepository{{Repository: "novox/mesh-controller", Ref: "release"}}
newer := aBuild("newer", "builder", "")
newer.Read = []ReadRepository{{Repository: "novox/mesh-controller", Ref: "main"}}
plain := aBuild("plain", "gitea", "")
failed := aBuild("failed", "route-proxy", "cannot clone")
failed.Read = []ReadRepository{{Repository: "novox/mesh-controller", Ref: "main"}}
for _, b := range []Build{older, newer, plain, failed} {
if err := inv.RecordBuild(ctx, b); err != nil {
t.Fatal(err)
}
}
read, err := inv.ReadRepositories(ctx)
if err != nil {
t.Fatal(err)
}
if len(read["builder"]) != 1 || read["builder"][0].Ref != "main" {
t.Fatalf("the newest build's reading is %+v", read["builder"])
}
if _, has := read["gitea"]; has {
t.Fatalf("a build that read nothing but its own repository reads as %+v", read["gitea"])
}
if _, has := read["route-proxy"]; has {
t.Fatal("a failed build's reading was kept as what that module reads")
}
}
+12 -12
View File
@@ -30,12 +30,12 @@ func TestRefusedAndFailedAreDifferentSituations(t *testing.T) {
refuser := nodeNamed(t, inv, "refuser")
failer := nodeNamed(t, inv, "failer")
if err := inv.RecordDoing(ctx, refuser, Doing{
if _, err := inv.RecordDoing(ctx, refuser, Doing{
Outcome: OutcomeRefused, Refused: "resource \"x\": a file needs a path",
}); err != nil {
t.Fatal(err)
}
if err := inv.RecordDoing(ctx, failer, Doing{
if _, err := inv.RecordDoing(ctx, failer, Doing{
Outcome: OutcomeFailed,
Failed: []FailedResource{{ID: "svc", Error: "unit not found"}},
Applied: 4,
@@ -70,7 +70,7 @@ func TestAMachineDoingWhatItWasToldIsNotOnTheList(t *testing.T) {
inv := fresh(t)
ctx := context.Background()
id := nodeNamed(t, inv, "fine")
if err := inv.RecordDoing(ctx, id, Doing{Outcome: OutcomeApplied, Applied: 6}); err != nil {
if _, err := inv.RecordDoing(ctx, id, Doing{Outcome: OutcomeApplied, Applied: 6}); err != nil {
t.Fatal(err)
}
wrong, err := inv.NotDoingWhatTheyWereTold(ctx)
@@ -97,12 +97,12 @@ func TestTheLastReportReplacesTheOneBefore(t *testing.T) {
inv := fresh(t)
ctx := context.Background()
id := nodeNamed(t, inv, "recovered")
if err := inv.RecordDoing(ctx, id, Doing{
if _, err := inv.RecordDoing(ctx, id, Doing{
Outcome: OutcomeFailed, Failed: []FailedResource{{ID: "a", Error: "no"}},
}); err != nil {
t.Fatal(err)
}
if err := inv.RecordDoing(ctx, id, Doing{Outcome: OutcomeApplied, Applied: 3}); err != nil {
if _, err := inv.RecordDoing(ctx, id, Doing{Outcome: OutcomeApplied, Applied: 3}); err != nil {
t.Fatal(err)
}
wrong, err := inv.NotDoingWhatTheyWereTold(ctx)
@@ -141,7 +141,7 @@ func TestWhatANodeSaidGoesWhenTheNodeDoes(t *testing.T) {
inv := fresh(t)
ctx := context.Background()
id := nodeNamed(t, inv, "leaving")
if err := inv.RecordDoing(ctx, id, Doing{Outcome: OutcomeFailed}); err != nil {
if _, err := inv.RecordDoing(ctx, id, Doing{Outcome: OutcomeFailed}); err != nil {
t.Fatal(err)
}
if _, err := inv.store.Pool().Exec(ctx, `delete from node where name = 'leaving'`); err != nil {
@@ -257,7 +257,7 @@ func TestTheSameFailureReportedAgainIsCountedNotRestarted(t *testing.T) {
id := nodeNamed(t, inv, "looping")
same := Doing{Outcome: OutcomeFailed, Failed: []FailedResource{{ID: "img", Error: "no such image"}}}
if err := inv.RecordDoing(ctx, id, same); err != nil {
if _, err := inv.RecordDoing(ctx, id, same); err != nil {
t.Fatal(err)
}
first, _, err := inv.DoingOf(ctx, "looping")
@@ -269,7 +269,7 @@ func TestTheSameFailureReportedAgainIsCountedNotRestarted(t *testing.T) {
}
for range StuckAfter - 1 {
if err := inv.RecordDoing(ctx, id, same); err != nil {
if _, err := inv.RecordDoing(ctx, id, same); err != nil {
t.Fatal(err)
}
}
@@ -287,7 +287,7 @@ func TestTheSameFailureReportedAgainIsCountedNotRestarted(t *testing.T) {
// The same resource failing with different words — a duration, a counter — is still the same
// failure: it is the resource that loops, not the sentence.
reworded := Doing{Outcome: OutcomeFailed, Failed: []FailedResource{{ID: "img", Error: "no such image (after 31s)"}}}
if err := inv.RecordDoing(ctx, id, reworded); err != nil {
if _, err := inv.RecordDoing(ctx, id, reworded); err != nil {
t.Fatal(err)
}
still, _, err := inv.DoingOf(ctx, "looping")
@@ -300,7 +300,7 @@ func TestTheSameFailureReportedAgainIsCountedNotRestarted(t *testing.T) {
// A different failure is a new situation, not a longer one.
other := Doing{Outcome: OutcomeFailed, Failed: []FailedResource{{ID: "svc", Error: "unit not found"}}}
if err := inv.RecordDoing(ctx, id, other); err != nil {
if _, err := inv.RecordDoing(ctx, id, other); err != nil {
t.Fatal(err)
}
changed, _, err := inv.DoingOf(ctx, "looping")
@@ -312,7 +312,7 @@ func TestTheSameFailureReportedAgainIsCountedNotRestarted(t *testing.T) {
}
// And a clean apply clears it: the machine is doing what it was told, since nothing.
if err := inv.RecordDoing(ctx, id, Doing{Outcome: OutcomeApplied, Applied: 2}); err != nil {
if _, err := inv.RecordDoing(ctx, id, Doing{Outcome: OutcomeApplied, Applied: 2}); err != nil {
t.Fatal(err)
}
fine, _, err := inv.DoingOf(ctx, "looping")
@@ -324,7 +324,7 @@ func TestTheSameFailureReportedAgainIsCountedNotRestarted(t *testing.T) {
}
// The list of what is wrong carries the count, so `status` can say it.
if err := inv.RecordDoing(ctx, id, same); err != nil {
if _, err := inv.RecordDoing(ctx, id, same); err != nil {
t.Fatal(err)
}
wrong, err := inv.NotDoingWhatTheyWereTold(ctx)
@@ -0,0 +1,13 @@
-- A build says which repositories it read, so a merge can find everything it affects.
--
-- A module is built from the one repository the mesh records — where its module.json lives — and some
-- modules' recipes reach into a second for the source they package: the packaging and the source are
-- allowed to live apart (catalogue's ArtifactContext). That second repository is named in the manifest
-- the build read, and the manifest the mesh *keeps* carries no build section, so nothing on the mesh
-- could say that a merge into the other repository is a change to this module at all. Two modules are
-- built from the control plane's own repository, and neither had ever been rebuilt when it moved
-- (novox/hq 04-ISSUES/131).
--
-- Nullable, like the two derived columns beside it: null is a build recorded before the mesh kept
-- this, which is not the same as a build that read nothing but its module's own repository.
alter table build add column built_contexts jsonb;
+29 -18
View File
@@ -670,31 +670,39 @@ func sameFailure(a, b Doing) bool {
// a clean apply clears both (novox/hq 04-ISSUES/065). The previous row is read first and the
// comparison made here, so "the same" is a rule this package states rather than a jsonb equality
// that would restart the count on a changed word in an error.
func (i *Inventory) RecordDoing(ctx context.Context, node string, d Doing) error {
// **And whether this report was news**, which is what makes a fact about it worth stating (novox/hq
// ADR 0134). A machine reconciles continuously and reports each time; the same outcome about the same
// declaration is the same state said again, and a fact per report would be a fact per minute per
// machine that tells nobody anything. Read here because the previous row is read here anyway.
func (i *Inventory) RecordDoing(ctx context.Context, node string, d Doing) (news bool, err error) {
failed, err := json.Marshal(d.Failed)
if err != nil {
return err
return false, err
}
var before Doing
var beforeFailed []byte
found := i.store.Pool().QueryRow(ctx,
`select outcome, refused, failed, failing_since, failures, coalesce(declared,'')
from node_report where node = $1`,
node).Scan(&before.Outcome, &before.Refused, &beforeFailed, &before.Since, &before.Times,
&before.Declared)
switch {
case errors.Is(found, pgx.ErrNoRows):
news = true
case found != nil:
return false, found
default:
if err := json.Unmarshal(beforeFailed, &before.Failed); err != nil {
return false, err
}
news = before.Outcome != d.Outcome || before.Declared != d.Declared || !sameFailure(before, d)
}
var since *time.Time
times := 0
if d.Outcome != OutcomeApplied {
var before Doing
var beforeFailed []byte
err := i.store.Pool().QueryRow(ctx,
`select outcome, refused, failed, failing_since, failures from node_report where node = $1`,
node).Scan(&before.Outcome, &before.Refused, &beforeFailed, &before.Since, &before.Times)
switch {
case errors.Is(err, pgx.ErrNoRows):
case err != nil:
return err
default:
if err := json.Unmarshal(beforeFailed, &before.Failed); err != nil {
return err
}
}
now := time.Now()
since, times = &now, 1
if err == nil && sameFailure(before, d) && before.Since != nil {
if found == nil && sameFailure(before, d) && before.Since != nil {
since, times = before.Since, before.Times+1
}
}
@@ -707,7 +715,10 @@ func (i *Inventory) RecordDoing(ctx context.Context, node string, d Doing) error
declared = excluded.declared,
failing_since = excluded.failing_since, failures = excluded.failures`,
node, d.Outcome, d.Refused, failed, d.Applied, d.Declared, since, times)
return err
if err != nil {
return false, err
}
return news, nil
}
// NotDoingWhatTheyWereTold is every machine whose last report was not a clean apply.
+13
View File
@@ -88,10 +88,23 @@ type BuildResult struct {
// (novox/hq ADR 0009). The catalogue turns these into edges; nothing else need care.
Against []string `json:"against,omitempty"`
// Read is every repository this build read source from besides the module's own. A module whose
// recipe packages source that lives elsewhere is affected when that repository moves, and the
// manifest the mesh keeps says nothing about it (novox/hq 04-ISSUES/131).
Read []ReadRepository `json:"read,omitempty"`
// Failed is why, when it did.
Failed string `json:"failed,omitempty"`
}
// ReadRepository is a repository a build read source from besides the module's own, at the branch,
// tag or commit it read. Spelled here as well as in the catalogue and the inventory, for the reason
// MadeArtifact is: one direction of dependency.
type ReadRepository struct {
Repository string `json:"repository"`
Ref string `json:"ref,omitempty"`
}
// MadeArtifact is one thing a build produced, as a person would want it reported.
type MadeArtifact struct {
Name string `json:"name"`
+33
View File
@@ -30,6 +30,11 @@ type Bus interface {
// (design 29 §4, the *state* shape).
PublishDeclaration(ctx context.Context, node string, body []byte) error
// PublishSeatEvent states a fact under a role's own name, for the holder of that role. A
// module's event is addressed to the module; a role's is addressed to the role, so it keeps
// meaning when the holder changes (novox/hq ADR 0121, ADR 0129).
PublishSeatEvent(ctx context.Context, seat, event string, body []byte) error
// AskTool sends one question to a module's tool and awaits one answer. A tool nobody serves
// must say so **at once** rather than after the whole wait: the difference between "that
// module is down" and "that tool is slow" is the first thing a person asking wants.
@@ -81,6 +86,13 @@ func EventSubject(source, key string) string {
return "mesh.mod." + source + ".event." + key
}
// SeatEventSubject is where a role's own event lands. Derived from the role, never from its holder:
// a fact about the build machine or about the control plane keeps its address when the module holding
// that role is replaced (novox/hq ADR 0121, ADR 0129).
func SeatEventSubject(seat, event string) string {
return "mesh.seat." + seat + ".event." + event
}
// DeclareSubject is where one node's declaration lands. Last-per-subject on the NODES stream, so
// a node that was away gets exactly the current one and a replayed older one is refused by
// sequence — the wire-level answer to novox/hq issue 107.
@@ -112,6 +124,27 @@ func (b OverNATS) PublishEvent(ctx context.Context, key, source, node string, bo
return nil
}
// PublishSeatEvent states a role's own fact. Same envelope as a module's event and a different
// address: the source header is the role, because that is what the fact is about.
func (b OverNATS) PublishSeatEvent(ctx context.Context, seat, event string, body []byte) error {
id, err := eventID()
if err != nil {
return err
}
h := nats.Header{}
h.Set("x-event-id", id)
h.Set("x-source", seat)
_, err = b.JS.PublishMsg(&nats.Msg{
Subject: SeatEventSubject(seat, event),
Header: h,
Data: body,
}, nats.MsgId(id), nats.Context(ctx))
if err != nil {
return fmt.Errorf("stating %s of the %s seat: %w", event, seat, err)
}
return nil
}
func (b OverNATS) PublishDeclaration(ctx context.Context, node string, body []byte) error {
_, err := b.JS.Publish(DeclareSubject(node), body, nats.Context(ctx))
if err != nil {
+16 -13
View File
@@ -265,7 +265,7 @@ func (e Enrolment) Outstanding(ctx context.Context, node string) (string, error)
return e.Inventory.Outstanding(ctx, node)
}
func (e Enrolment) Heard(ctx context.Context, report Report) (err error) {
func (e Enrolment) Heard(ctx context.Context, report Report) (news bool, err error) {
// A store that could not be asked right now is said as such, so the report is kept for
// another attempt rather than acknowledged and lost (novox/hq issue 082).
defer func() {
@@ -274,11 +274,11 @@ func (e Enrolment) Heard(ctx context.Context, report Report) (err error) {
}
}()
if report.Node == "" {
return errors.New("a report named no node")
return false, errors.New("a report named no node")
}
node, err := e.Inventory.NodeByName(ctx, report.Node)
if err != nil {
return err
return false, err
}
// What an adopted node holds, which firewall it found, and what is reachable on it (novox/hq
@@ -298,7 +298,7 @@ func (e Enrolment) Heard(ctx context.Context, report Report) (err error) {
Port: r.Port, By: r.By, Published: r.Published, ContainerPort: r.ContainerPort})
}
if err := e.Inventory.RecordAdoption(ctx, node.ID, held, report.Firewall, reachable); err != nil {
return err
return false, err
}
}
// What it says about the tunnel it carried (novox/hq ADR 0105), whenever it says it.
@@ -308,7 +308,7 @@ func (e Enrolment) Heard(ctx context.Context, report Report) (err error) {
Peers: report.Tunnel.Peers, State: report.Tunnel.State, Note: report.Tunnel.Note,
Kept: report.Tunnel.Kept,
}); err != nil {
return err
return false, err
}
}
// A node taking a found tunnel's key after enrolment (novox/hq ADR 0105). Verified against the
@@ -317,9 +317,9 @@ func (e Enrolment) Heard(ctx context.Context, report Report) (err error) {
// does not verify or is stale — a refusal, not "not now", so the node hears why.
if report.Rekey != nil {
if err := e.rekey(ctx, node, *report.Rekey); err != nil {
return err
return false, err
}
return e.Inventory.Seen(ctx, node.ID)
return false, e.Inventory.Seen(ctx, node.ID)
}
// A bare word that a node is there is not an account of what the machine did or holds: it
@@ -334,7 +334,7 @@ func (e Enrolment) Heard(ctx context.Context, report Report) (err error) {
if report.Superseded != "" {
log.Printf("%s set aside declaration %s for the newer %s", report.Node, report.Declared, report.Superseded)
}
return e.Inventory.Seen(ctx, node.ID)
return false, e.Inventory.Seen(ctx, node.ID)
}
// What it did is kept whichever way it went. Until this, a refusal or a failure moved
// last_seen and the reason went to a log line, so "which machine is not doing what it was
@@ -361,17 +361,20 @@ func (e Enrolment) Heard(ctx context.Context, report Report) (err error) {
// on top of it (novox/hq ADR 0038). Kept even when the declaration was refused: what the
// machine carries is true regardless of what it thought of the last thing it was sent.
if err := e.Inventory.RecordCarried(ctx, report.Node, report.Carried); err != nil {
return err
return false, err
}
if err := e.Inventory.RecordDoing(ctx, node.ID, doing); err != nil {
return err
// **Whether this is news** is the store's answer: it holds the previous report, and a machine
// that reconciles every minute says the same thing until something changes (novox/hq ADR 0134).
news, err = e.Inventory.RecordDoing(ctx, node.ID, doing)
if err != nil {
return false, err
}
// A refusal, a failure, or a bare word that the node is there — none of them is an account of
// what the machine holds, so each moves last_seen and nothing else. Recording a partial list
// as though it were the whole would tell a rebuilding node to remove what it still has.
if report.Refused != "" || len(report.Failed) > 0 || report.Applied == nil {
return e.Inventory.Seen(ctx, node.ID)
return news, e.Inventory.Seen(ctx, node.ID)
}
return e.Inventory.RecordOwned(ctx, node.ID, report.Applied)
return news, e.Inventory.RecordOwned(ctx, node.ID, report.Applied)
}
+43
View File
@@ -49,6 +49,39 @@ func eventID() (string, error) {
return hex.EncodeToString(raw), nil
}
// MeshControllerSeat is the role the control plane holds, and therefore where its own facts live: a
// role's events belong to the role, not to whichever container is holding it today (novox/hq ADR 0121,
// ADR 0129). It is what makes them addressable while the control plane itself is being replaced.
const MeshControllerSeat = "mesh-controller"
// The facts the mesh states about its own work (novox/hq ADR 0134).
const (
// KeyApplied: a machine now runs what it was sent.
KeyApplied = "applied"
// KeyRefused: a machine did not take what it was sent, and why.
KeyRefused = "refused"
// KeyBuiltBefore: a build the mesh already held, for a catalogue that asked what it missed. Not
// `built` — that is the build machine's, said as it happens, and a replay is neither.
KeyBuiltBefore = "built-before"
)
// Applied is what a machine now runs, as the mesh states it.
type Applied struct {
Node string `json:"node"`
Declared string `json:"declared,omitempty"`
// Resources is how many the machine applied, not which: the list is the machine's own account
// of itself and belongs in the records, not in a fact every listener has to read past.
Resources int `json:"resources"`
}
// Refused is a machine that would not take what it was sent.
type Refused struct {
Node string `json:"node"`
Declared string `json:"declared,omitempty"`
Refused string `json:"refused,omitempty"`
Failed map[string]string `json:"failed,omitempty"`
}
// KeyModuleBuilt is what the builder announces when it has built something. The catalogue places
// it in the module graph; nothing else need care.
const KeyModuleBuilt = "module.builder.built"
@@ -130,6 +163,16 @@ type SourceMoved struct {
HTMLURL string `json:"html_url"`
// MergedAt is when the forge merged it, RFC 3339. What decides whether this is news.
MergedAt string `json:"merged_at"`
// Paths are the files the merge changed, from the repository's root. Empty means the forge said
// nothing about them, and every module built from the repository is treated as affected.
Paths []string `json:"paths,omitempty"`
// PathsTruncated says the merge changed more files than the forge was asked to list, so Paths is
// a beginning rather than the whole change — and again, everything is treated as affected. Said
// rather than inferred from a round number, because "this is all of it" and "this is as much as
// I asked for" are the difference between rebuilding a module and leaving it stale.
PathsTruncated bool `json:"paths_truncated,omitempty"`
}
type Upgraded struct {
+6 -6
View File
@@ -22,7 +22,7 @@ func heardFrom(t *testing.T, report link.Report) (*inventory.Inventory, inventor
if _, err := inv.AddNode(ctx, report.Node); err != nil {
t.Fatal(err)
}
if err := (link.Enrolment{Inventory: inv}).Heard(ctx, report); err != nil {
if _, err := (link.Enrolment{Inventory: inv}).Heard(ctx, report); err != nil {
t.Fatal(err)
}
doing, said, err := inv.DoingOf(ctx, report.Node)
@@ -103,7 +103,7 @@ func TestABareAliveDoesNotWipeTheDeclarationThatSaysANodeIsCurrent(t *testing.T)
if err := inv.RecordSent(ctx, node.ID, digest); err != nil {
t.Fatal(err)
}
if err := (link.Enrolment{Inventory: inv}).Heard(ctx, link.Report{
if _, err := (link.Enrolment{Inventory: inv}).Heard(ctx, link.Report{
Node: "anchor", Applied: []string{"a", "b"}, Declared: digest, Carried: []int{5432},
}); err != nil {
t.Fatal(err)
@@ -126,7 +126,7 @@ func TestABareAliveDoesNotWipeTheDeclarationThatSaysANodeIsCurrent(t *testing.T)
}
// Now the node says only that it is there, as it does every minute.
if err := (link.Enrolment{Inventory: inv}).Heard(ctx, link.Report{Node: "anchor"}); err != nil {
if _, err := (link.Enrolment{Inventory: inv}).Heard(ctx, link.Report{Node: "anchor"}); err != nil {
t.Fatal(err)
}
if !currentOf("anchor") {
@@ -162,7 +162,7 @@ func TestAFailureDoesNotBecomeTheAccountOfWhatTheMachineHolds(t *testing.T) {
if err := inv.RecordOwned(ctx, node.ID, []string{"one", "two", "three"}); err != nil {
t.Fatal(err)
}
if err := (link.Enrolment{Inventory: inv}).Heard(ctx, link.Report{
if _, err := (link.Enrolment{Inventory: inv}).Heard(ctx, link.Report{
Node: "workstation", Applied: []string{"one"}, Failed: map[string]string{"two": "no"},
}); err != nil {
t.Fatal(err)
@@ -200,13 +200,13 @@ func TestWhatAnAdoptedNodeHoldsIsKeptAndAnAliveWordDoesNotWipeIt(t *testing.T) {
}
check("after the report")
if err := (link.Enrolment{Inventory: inv}).Heard(ctx, link.Report{Node: "anchor"}); err != nil {
if _, err := (link.Enrolment{Inventory: inv}).Heard(ctx, link.Report{Node: "anchor"}); err != nil {
t.Fatal(err)
}
check("after an alive word")
// A reconcile report carrying only adoption is recorded, though it applied nothing.
if err := (link.Enrolment{Inventory: inv}).Heard(ctx, link.Report{Node: "anchor",
if _, err := (link.Enrolment{Inventory: inv}).Heard(ctx, link.Report{Node: "anchor",
Firewall: "ufw"}); err != nil {
t.Fatal(err)
}
+37
View File
@@ -154,10 +154,47 @@ func (n *natsInbound) deliver(ctx context.Context, act func(context.Context, Con
m.seq = meta.Sequence.Stream
m.delivered = meta.NumDelivered
}
// **Work that outlives the acknowledgement window says so while it runs.**
//
// The bus waits a fixed time to be told a message was taken, and then hands it to whoever
// consumes next — which is right for a consumer that died and wrong for one that is busy.
// Acting on a merge builds every module the merge changed: minutes of work against a
// thirty-second window. So the same merge was handed over again while the first build was
// still running, and again after that — on 2026-09-28 one merge ran the mesh's whole
// catalogue five times over and exhausted a public registry's pull limit.
//
// Here rather than in each handler, because the window belongs to the transport and every
// handler would otherwise have to remember it. It changes nothing about a handler that
// dies: a message is kept alive only while this goroutine is, so a controller that stops
// stops saying so, and the bus redelivers exactly as it should.
working := make(chan struct{})
defer close(working)
go stillWorking(msg, working)
}
act(ctx, m)
}
// heartbeatWhileWorking is how often a handler still running tells the bus so — comfortably inside
// the shortest acknowledgement window the mesh gives any of its consumers.
const heartbeatWhileWorking = 10 * time.Second
// stillWorking keeps one message alive until the work on it returns.
//
// An error is not worth reporting: what the bus does when it is not told is redeliver, which is
// exactly what happens if this fails, and the handler's own outcome is the thing worth logging.
func stillWorking(msg *nats.Msg, done <-chan struct{}) {
tick := time.NewTicker(heartbeatWhileWorking)
defer tick.Stop()
for {
select {
case <-done:
return
case <-tick.C:
_ = msg.InProgress()
}
}
}
// kindOfSubject is how this transport's addressing becomes what the mesh calls a message.
//
// By subject, which is the only thing the server enforces: a body claiming to be a report does not
+167 -5
View File
@@ -86,14 +86,15 @@ type counted struct {
heard []Report
}
func (c *counted) Heard(_ context.Context, r Report) error {
func (c *counted) Heard(_ context.Context, r Report) (bool, error) {
c.mu.Lock()
defer c.mu.Unlock()
if c.err != nil {
return c.err
return false, c.err
}
c.heard = append(c.heard, r)
return nil
// News, so what the mesh states about a report is exercised wherever a report is.
return true, nil
}
func (c *counted) refusing(err error) {
@@ -207,11 +208,11 @@ type sentAndHeardSafely struct {
heard []Report
}
func (s *sentAndHeardSafely) Heard(_ context.Context, r Report) error {
func (s *sentAndHeardSafely) Heard(_ context.Context, r Report) (bool, error) {
s.mu.Lock()
defer s.mu.Unlock()
s.heard = append(s.heard, r)
return nil
return true, nil
}
func (s *sentAndHeardSafely) Outstanding(context.Context, string) (string, error) {
@@ -395,3 +396,164 @@ func TestEverySubjectTheControllerFollowsDecodesToAKind(t *testing.T) {
t.Error("a subject nobody follows decoded to a kind")
}
}
// **A handler slower than the acknowledgement window is not handed its message again.**
//
// The bus waits a fixed time to be told a message was taken and then redelivers, which is right for
// a consumer that died and wrong for one that is busy. Acting on a merge builds modules — minutes
// against a thirty-second window — and the same merge was handed over five times while the first
// build was still running (2026-09-28). Here the window is two seconds and the work takes six.
func TestNatsWorkSlowerThanTheWindowIsNotHandedOverAgain(t *testing.T) {
js := aBus(t)
// The controller's own consumer, with a window short enough to outlive in a test.
if err := js.EnsureConsumer(broker.Consumer{
Name: broker.ControllerName, Stream: "CONTROL", Push: true, AckWaitSeconds: 2,
Why: "a window short enough to outlive in a test",
}); err != nil {
t.Fatal(err)
}
var mu sync.Mutex
handled := 0
slow := make(chan struct{})
s, stop := servingOn(t, js, nil)
defer stop()
s.listener = slowly{func() {
mu.Lock()
handled++
first := handled == 1
mu.Unlock()
if first {
time.Sleep(6 * time.Second)
close(slow)
}
}}
body, err := json.Marshal(Report{Node: "anchor", Declared: "d1", Applied: []string{"store"}})
if err != nil {
t.Fatal(err)
}
if _, err := js.Context().Publish(ReportSubject("anchor"), body); err != nil {
t.Fatal(err)
}
select {
case <-slow:
case <-time.After(30 * time.Second):
t.Fatal("the slow work never finished")
}
// A moment for a redelivery to arrive, if the bus were going to send one.
time.Sleep(3 * time.Second)
mu.Lock()
defer mu.Unlock()
if handled != 1 {
t.Fatalf("one report was handled %d times, so slow work is run again while it is running", handled)
}
}
// slowly is a listener that runs whatever it was given.
type slowly struct{ work func() }
func (s slowly) Heard(context.Context, Report) (bool, error) { s.work(); return true, nil }
// **The mesh says what it applied** (novox/hq ADR 0134), under the seat the control plane holds — and
// says nothing when a report is the same state said again, which is what a machine reconciling every
// minute sends.
func TestNatsTheMeshSaysWhatAMachineApplied(t *testing.T) {
js := aBus(t)
heard := make(chan *nats.Msg, 4)
sub, err := js.Conn().Subscribe(SeatEventSubject(MeshControllerSeat, ">"), func(m *nats.Msg) {
heard <- m
})
if err != nil {
t.Fatal(err)
}
defer sub.Unsubscribe() //nolint:errcheck // the subscription dies with the connection
_, stop := servingOn(t, js, &counted{})
defer stop()
// A report that changed something: the store says it was news.
body, err := json.Marshal(Report{Node: "anchor", Declared: "d1", Applied: []string{"store", "broker"}})
if err != nil {
t.Fatal(err)
}
if _, err := js.Context().Publish(ReportSubject("anchor"), body); err != nil {
t.Fatal(err)
}
select {
case m := <-heard:
if m.Subject != SeatEventSubject(MeshControllerSeat, KeyApplied) {
t.Fatalf("the mesh stated %q", m.Subject)
}
var said Applied
if err := json.Unmarshal(m.Data, &said); err != nil {
t.Fatal(err)
}
if said.Node != "anchor" || said.Declared != "d1" || said.Resources != 2 {
t.Fatalf("it said %+v", said)
}
case <-time.After(10 * time.Second):
t.Fatal("the mesh said nothing about a machine that now runs something else")
}
// A refusal is its own fact, with the reason in it rather than only in a log.
refusal, err := json.Marshal(Report{Node: "anchor", Declared: "d2",
Failed: map[string]string{"gitea.server": "no such image"}})
if err != nil {
t.Fatal(err)
}
if _, err := js.Context().Publish(ReportSubject("anchor"), refusal); err != nil {
t.Fatal(err)
}
select {
case m := <-heard:
if m.Subject != SeatEventSubject(MeshControllerSeat, KeyRefused) {
t.Fatalf("a refusal was stated as %q", m.Subject)
}
var said Refused
if err := json.Unmarshal(m.Data, &said); err != nil {
t.Fatal(err)
}
if said.Failed["gitea.server"] == "" {
t.Fatalf("the refusal does not say which resource or why: %+v", said)
}
case <-time.After(10 * time.Second):
t.Fatal("the mesh said nothing about a machine that refused what it was sent")
}
}
// And a report that is not news is not a fact. A machine reconciles every minute; a fact per report
// would be a fact per minute per machine, which is a stream nobody reads.
func TestNatsAReportThatIsNotNewsIsNotStated(t *testing.T) {
js := aBus(t)
heard := make(chan *nats.Msg, 4)
sub, err := js.Conn().Subscribe(SeatEventSubject(MeshControllerSeat, ">"), func(m *nats.Msg) {
heard <- m
})
if err != nil {
t.Fatal(err)
}
defer sub.Unsubscribe() //nolint:errcheck // the subscription dies with the connection
// A store that records the report and says it was nothing new — which is what the mesh's own
// store says about a machine repeating itself.
_, stop := servingOn(t, js, sameAgain{})
defer stop()
body, err := json.Marshal(Report{Node: "anchor", Declared: "d1", Applied: []string{"store"}})
if err != nil {
t.Fatal(err)
}
if _, err := js.Context().Publish(ReportSubject("anchor"), body); err != nil {
t.Fatal(err)
}
select {
case m := <-heard:
t.Fatalf("the mesh stated %q about a machine that changed nothing", m.Subject)
case <-time.After(3 * time.Second):
}
}
// sameAgain records a report and says it was the same state said again.
type sameAgain struct{}
func (sameAgain) Heard(context.Context, Report) (bool, error) { return false, nil }
+4 -4
View File
@@ -60,7 +60,7 @@ func TestASignedRekeyMovesTheHubOntoItsTunnel(t *testing.T) {
rekey := &link.Rekey{Previous: ownKey, OverlayKey: tunnelKey, Tunnel: theTunnel()}
rekey.Proof = ed25519.Sign(private, link.RekeyProof("anchor", ownKey, tunnelKey, theTunnel()))
if err := e.Heard(ctx, link.Report{Node: "anchor", Rekey: rekey}); err != nil {
if _, err := e.Heard(ctx, link.Report{Node: "anchor", Rekey: rekey}); err != nil {
t.Fatal(err)
}
placed, err := e.Inventory.Overlays(ctx)
@@ -77,7 +77,7 @@ func TestASignedRekeyMovesTheHubOntoItsTunnel(t *testing.T) {
_ = hub
// Replayed, it is stale: the previous key it names is no longer the node's.
err = e.Heard(ctx, link.Report{Node: "anchor", Rekey: rekey})
_, err = e.Heard(ctx, link.Report{Node: "anchor", Rekey: rekey})
if err == nil || !strings.Contains(err.Error(), "previous overlay key") {
t.Fatalf("a replayed rekey was accepted: %v", err)
}
@@ -93,7 +93,7 @@ func TestARekeySignedByAnotherKeyIsRefusedAndChangesNothing(t *testing.T) {
rekey := &link.Rekey{Previous: ownKey, OverlayKey: tunnelKey, Tunnel: theTunnel()}
rekey.Proof = ed25519.Sign(stranger, link.RekeyProof("anchor", ownKey, tunnelKey, theTunnel()))
err = e.Heard(ctx, link.Report{Node: "anchor", Rekey: rekey})
_, err = e.Heard(ctx, link.Report{Node: "anchor", Rekey: rekey})
if err == nil || !strings.Contains(err.Error(), "not signed by anchor's identity key") {
t.Fatalf("a rekey signed by a stranger was accepted: %v", err)
}
@@ -111,7 +111,7 @@ func TestARekeySignedByAnotherKeyIsRefusedAndChangesNothing(t *testing.T) {
other := theTunnel()
other.Port = 51820
moved.Proof = ed25519.Sign(mustPrivate(t, e, "anchor"), link.RekeyProof("anchor", ownKey, tunnelKey, other))
if err := e.Heard(ctx, link.Report{Node: "anchor", Rekey: moved}); err == nil {
if _, err := e.Heard(ctx, link.Report{Node: "anchor", Rekey: moved}); err == nil {
t.Fatal("a proof over another tunnel was accepted")
}
}
+2 -2
View File
@@ -9,12 +9,12 @@ import (
type heardWith struct{ err error }
func (h heardWith) Heard(context.Context, Report) error { return h.err }
func (h heardWith) Heard(context.Context, Report) (bool, error) { return h.err == nil, h.err }
// switchable answers with whatever it is set to — the store away, then back.
type switchable struct{ err error }
func (h *switchable) Heard(context.Context, Report) error { return h.err }
func (h *switchable) Heard(context.Context, Report) (bool, error) { return h.err == nil, h.err }
func aReport(node, declared string) Report {
return Report{Node: node, Declared: declared, Applied: []string{"store"}}
+62 -4
View File
@@ -29,7 +29,12 @@ type Enroller interface {
// Listener is what the controller does with a report. Separate from Enroller so the two can be
// given independently, and so a server that only sends declarations needs neither.
type Listener interface {
Heard(ctx context.Context, report Report) error
// Heard records what a node said, and says whether it was **news** — a machine that now runs
// something else, or refuses something it did not refuse before. A machine reconciles
// continuously and reports each time, so what is news is the store's answer rather than the
// bus's: only this side has the previous report to compare with. What the mesh states about it
// is the server's (novox/hq ADR 0134).
Heard(ctx context.Context, report Report) (news bool, err error)
}
// Recorder keeps what builders say.
@@ -257,7 +262,7 @@ func (s *Server) heartbeat(m Control) {
return
}
if s.listener != nil {
if err := s.listener.Heard(context.Background(), Report{Node: alive.Node}); err != nil {
if _, err := s.listener.Heard(context.Background(), Report{Node: alive.Node}); err != nil {
s.log.Printf("could not record that %s is here: %v", alive.Node, err)
}
}
@@ -291,7 +296,7 @@ func (s *Server) reported(ctx context.Context, m Control) {
return
}
err := s.listener.Heard(context.Background(), report)
news, err := s.listener.Heard(context.Background(), report)
switch s.decide(ctx, m, what, declaredIn, outstanding, err) {
case Hold:
// Held, not settled, while the store cannot take it: the node reports an apply once,
@@ -307,6 +312,13 @@ func (s *Server) reported(ctx context.Context, m Control) {
// node whose recovery copy is silently older than it looks.
s.log.Printf("could not record %s's report: %v", report.Node, err)
}
// **And the mesh says what it did** (novox/hq ADR 0134). Only when the report was news: a
// machine reports every convergence, and a fact per report would be a fact per minute per
// machine saying nothing. Stated after it is recorded, so nothing is announced that the
// mesh does not hold.
if err == nil && news {
s.saysWhatItDid(ctx, report)
}
}
switch {
@@ -460,7 +472,19 @@ func (s *Server) catchingUp(ctx context.Context, m Control) {
sent := 0
for _, a := range announcements {
a.Replay = true
if err := EmitEvent(ctx, s.bus, KeyModuleBuilt, "control-plane", "", a); err != nil {
// Under the control plane's own seat (novox/hq ADR 0134). It used to be published as a
// module's event from a module called "control-plane", which does not exist — so the
// controller's own account refused it, every catalogue that asked what it missed was
// answered with nothing, and its graph kept the gap (found 2026-09-28).
body, err := json.Marshal(a)
if err != nil {
// A body that cannot be written is this program's fault, not the bus's, and publishing
// an empty one would put a fact on the mesh that says nothing.
s.log.Printf("cannot re-announce %s at %s: %v", a.Module, short(a.Commit), err)
_ = m.Took()
return
}
if err := s.bus.PublishSeatEvent(ctx, MeshControllerSeat, KeyBuiltBefore, body); err != nil {
// Said and abandoned rather than retried: the catalogue asks again every time it
// starts, and half a graph delivered twice is no better than half delivered once.
s.log.Printf("replaying %s at %s failed, and the rest is abandoned: %v",
@@ -551,3 +575,37 @@ func (s *Server) sourceMoved(ctx context.Context, m Control) {
}
_ = m.Took()
}
// saysWhatItDid states what a machine now runs, or what it would not take, as a fact on the bus
// (novox/hq ADR 0134).
//
// **The control plane speaks, as the holder of its seat.** A node's report is control traffic only
// this process may read, so the chain from a merge to a machine went dark exactly where it touched
// one: nothing said which version a machine runs, or that it refused to. The facts are second-hand
// on purpose — one emitter, one ordering — and a machine that cannot reach the bus produces none, so
// absence is not health.
//
// A failure to state a fact is logged and nothing else: the report is recorded, which is the part
// that must not be lost, and the next change says the same thing again.
func (s *Server) saysWhatItDid(ctx context.Context, report Report) {
if s.bus == nil {
return
}
event, body := KeyApplied, any(Applied{
Node: report.Node, Declared: report.Declared, Resources: len(report.Applied),
})
if report.Refused != "" || len(report.Failed) > 0 {
event, body = KeyRefused, Refused{
Node: report.Node, Declared: report.Declared,
Refused: report.Refused, Failed: report.Failed,
}
}
raw, err := json.Marshal(body)
if err != nil {
s.log.Printf("could not say what %s did: %v", report.Node, err)
return
}
if err := s.bus.PublishSeatEvent(ctx, MeshControllerSeat, event, raw); err != nil {
s.log.Printf("could not say that %s %s: %v", report.Node, event, err)
}
}
+3 -3
View File
@@ -23,12 +23,12 @@ type sentAndHeard struct {
err error
}
func (s *sentAndHeard) Heard(_ context.Context, r Report) error {
func (s *sentAndHeard) Heard(_ context.Context, r Report) (bool, error) {
if s.err != nil {
return s.err
return false, s.err
}
s.heard = append(s.heard, r)
return nil
return true, nil
}
func (s *sentAndHeard) Outstanding(context.Context, string) (string, error) { return s.sent, nil }
+1
View File
@@ -27,6 +27,7 @@
"bus": "/var/lib/mesh/mesh-controller/bus"
},
"secrets-owner": "65534:65534",
"prepares": true,
"resources": [
{
"id": "mesh-state",