Compare commits

..
Author SHA1 Message Date
mesh-admin 5c832f2d19 Merge pull request 'Compose a process's environment as a container's' (#250) from feat/a-process-env-is-composed-like-a-containers into main 2026-10-03 22:29:11 +00:00
jochen cdebb7d1a5 Compose a process's environment as a container's
A module's own code moving out of its container (novox/hq to-be 38 WP4c)
becomes a process on the machine, and still has to be told what its
container was: the port this machine gave the module and where the
foundation's seats are. ${port:…} and ${seat:…} were filled only in a
file's content and a container's env, so in a process's env they reached
the machine as literals, and the modules that moved first (mesh-catalog
#245) wrote their run-once steps a 0600 env file instead. A process's env
now takes the same resolution and the same refusals; ${dir:…} and
${access:…} already did, and a bundle's env (ADR 0192) already resolves
${dir:…} and ${port:…}.
2026-10-04 00:27:42 +02:00
mesh-admin 6a803ea5b3 Merge pull request 'Issue 219: an older build request never replaces a newer one's artifact' (#249) from fix/issue-219-an-older-build-never-replaces-a-newer into main 2026-10-03 22:23:31 +00:00
jochen 9745c1ab31 An older build request never replaces a newer one's artifact
Builds of one module in flight together finish in any order, and the mesh
took whatever it heard last as what the module is: RegisterModule overwrote
the module's manifest unconditionally, and Held/BuiltAgainst/ReadRepositories
ordered builds by when they were recorded. A postgres build asked before the
mesh-tools runtime fix finished after the one asked after it, and the next
push deployed the stale image (novox/hq issue 219).

A build is now ordered by when it was asked, read from the build-<nanos> id
the controller writes: build.asked and module.built_asked (migration 0055).
A registration from an earlier request than the module's current one is
recorded and refused as superseded. A plan takes as its outcome only a build
asked at or after its own ask, so an earlier plan's leftover build cannot
settle a later plan. Ids of any other shape keep the old order.
2026-10-04 00:22:11 +02:00
mesh-admin c0c3c3fed4 Merge pull request 'A TypeScript bundle is one file per entrypoint, bundled in the toolchain (hq ADR 0193); a toolchain follows the SDK it stands on (hq issue 212)' (#247) from feat/a-typescript-bundle-is-one-file into main 2026-10-03 21:50:44 +00:00
mesh-admin c1449fffe9 Merge pull request 'Issue only the recorded holder a seat held once for the mesh (hq issue 218)' (#248) from fix/issue-218-only-the-holder-serves-a-mesh-seat into main 2026-10-03 21:30:50 +00:00
jochen f873c97db5 Issue only the recorded holder a seat held once for the mesh (novox/hq issue 218)
A module claiming a mesh-scoped seat was granted and issued the seat's subjects on every machine
it runs on, so the store's verbs answered from whichever postgres replied first. Where the mesh
records the seat's holder, only that (node, module) is now issued it; the module's own tools are
untouched everywhere.
2026-10-03 23:30:27 +02:00
jochen b0b3d87fe2 Run the toolchain's esbuild as itself: npm installs its native binary in place of the script 2026-10-03 23:26:08 +02:00
jochen ba189e6943 A TypeScript bundle is one file per entrypoint, bundled in the toolchain (hq ADR 0193); a toolchain follows the SDK it stands on (hq issue 212)
Every served bundle is its own process now, so each carries its own copy of what it imports: after
the compile and the launchers, the toolchain image's esbuild bundles every entrypoint in place and
every launcher under its own name into one ES module file, the SDK inlined, require provided to
inlined CommonJS, the launcher's shebang kept and its mode 0755. The toolchain's node_modules is
copied only for packages an artifact names external. An image without the bundler is refused by
name. Issue 212: build.on already passes a published package by its exact version and plans the
toolchain after it; tests say so.
2026-10-03 23:24:08 +02:00
mesh-admin cadf74a176 Merge pull request 'Tools pipeline: issues 214, 215, 216, Go tools bundles served, and the runtime consumes for its modules (ADR 0193, 0198)' (#246) from fix/tools-pipeline-issues-214-216-and-0198 into main 2026-10-03 21:16:13 +00:00
jochen 74efe8e2e7 Tests follow the grants and issue 203: a person may ask what answers; the resolver test mints its credential 2026-10-03 23:15:57 +02:00
jochen 68af9eff44 Merge remote-tracking branch 'origin/feat/0198-the-runtime-consumes-for-its-modules' into integrate 2026-10-03 23:12:21 +02:00
jochen 518eeb7941 integrate: go served 2026-10-03 23:12:21 +02:00
jochen bf2da878a0 Merge remote-tracking branch 'origin/fix/issue-216-a-bundle-nothing-delivers-is-refused' into integrate 2026-10-03 23:12:03 +02:00
jochen 50cf253a43 Merge remote-tracking branch 'origin/fix/issue-215-a-commit-is-never-a-branch-to-follow' into integrate 2026-10-03 23:12:03 +02:00
jochen 980a0dee93 integrate: 214 2026-10-03 23:12:03 +02:00
jochen 8b016cc62b A Go tools bundle is served by its binary (hq ADR 0193)
A bundle compiled to a binary has no entrypoints, and loads had to name one, so a Go bundle could
not be served. Its binary is what the runtime starts: loads names the binary, derived when the
module lists tools, and the runtime is told the binary's path, delivered like any tools bundle.
2026-10-03 22:27:01 +02:00
jochen cf2bb3b87d A bundle nothing would deliver is refused at registration (hq issue 216)
The composer delivers a bundle when the runtime loads from it, a resource names it, or it is the
runtime; one reached by none of them was built, recorded and pushed as success and was simply
absent. Seven modules' tools went missing that way. Refused at registration, naming the field that
would deliver it.
2026-10-03 22:25:34 +02:00
jochen 6784efae75 A commit is never a branch to follow (hq issue 215)
A build asked at a commit recorded that commit as the module's ref. Every merge after it failed to
match the module and its plan left it out without a word, and every plan that rebuilt it asked for
the same old commit again. Registration now keeps the branch the module followed (the default
branch for a new one); matching and re-asking read a recorded commit as the default branch, which
heals records already pinned this way; and a merge says which modules of its repository it leaves
out because they follow another branch.
2026-10-03 22:22:08 +02:00
jochen d86baebe9a A plan settles an asked build from the build records (hq issue 214)
A merge to the controller's own repository replaces the controller in its first tier; the build
that produced the new one was recorded, the plan never heard it, and it waited for ever with every
later plan behind it. The record is the fact: a build recorded after the ask is the tier's outcome,
whoever was listening when it came.
2026-10-03 22:20:33 +02:00
27 changed files with 1055 additions and 57 deletions
+23 -2
View File
@@ -148,6 +148,11 @@ func buildFrom(result link.BuildResult) inventory.Build {
// rebuild the graph rather than a list of names. // rebuild the graph rather than a list of names.
Path: result.Path, Path: result.Path,
} }
// When it was asked, which is what orders it against another build of the same module
// (novox/hq 04-ISSUES/219) — not when it was heard.
if asked, ok := link.BuildAskedAt(result.ID); ok {
kept.Asked = asked
}
for _, ref := range result.Against { for _, ref := range result.Against {
kept.Against = append(kept.Against, catalogue.Recorded(ref)) kept.Against = append(kept.Against, catalogue.Recorded(ref))
} }
@@ -409,7 +414,7 @@ func buildOne(ctx context.Context, source buildSource, path, ref string, wait ti
// Correlated by something the control plane makes, not by the module's name: two builds of one // Correlated by something the control plane makes, not by the module's name: two builds of one
// module can be in flight, and the second answer is not the first one's. // module can be in flight, and the second answer is not the first one's.
request := link.BuildRequest{ request := link.BuildRequest{
ID: fmt.Sprintf("%s-%d", "build", time.Now().UnixNano()), ID: link.NewBuildID(time.Now()),
Repository: repository, Repository: repository,
Path: path, Path: path,
Ref: ref, Ref: ref,
@@ -526,15 +531,31 @@ func takeIn(ctx context.Context, inv *inventory.Inventory, result link.BuildResu
BuiltFrom: result.Commit, Head: result.Commit, BuiltFrom: result.Commit, Head: result.Commit,
// What it stood on, so registration can judge a built manifest's base (to-be 38 WP2.4). // What it stood on, so registration can judge a built manifest's base (to-be 38 WP2.4).
Against: kept.Against, Against: kept.Against,
// When it was asked, so an older request heard later does not replace a newer one
// (novox/hq 04-ISSUES/219).
Asked: kept.Asked,
} }
if result.Source != nil && result.Source.Seat != "" { if result.Source != nil && result.Source.Seat != "" {
recorded.Repository, recorded.Seat = result.Source.Repository, result.Source.Seat recorded.Repository, recorded.Seat = result.Source.Repository, result.Source.Seat
} }
// **A build at a commit does not change the branch a module follows** (novox/hq 04-ISSUES/215):
// the commit is built and recorded as what it was built from, and the module keeps following
// what it followed before — the repository's default branch for one new to the catalogue.
if followedBranch(result.Ref) == "" && result.Ref != "" {
recorded.Ref = ""
if was, err := inv.SourceOf(ctx, manifest.Module); err == nil {
recorded.Ref = followedBranch(was.Ref)
}
}
if err := namesNoInstallation(manifest); err != nil { if err := namesNoInstallation(manifest); err != nil {
return manifest, kept, fmt.Errorf("%s built %s (%s), and the mesh does not register it: %w", return manifest, kept, fmt.Errorf("%s built %s (%s), and the mesh does not register it: %w",
result.On, result.Repository, short(result.Commit), err) result.On, result.Repository, short(result.Commit), err)
} }
if err := inv.RegisterModule(ctx, manifest, recorded); err != nil { if err := inv.RegisterModule(ctx, manifest, recorded); err != nil {
if errors.Is(err, inventory.ErrSuperseded) {
return manifest, kept, fmt.Errorf("%s built %s (%s), recorded and not registered: %w",
result.On, manifest.Module, short(result.Commit), err)
}
return manifest, kept, err return manifest, kept, err
} }
return manifest, kept, nil return manifest, kept, nil
@@ -564,7 +585,7 @@ func buildAndShow(ctx context.Context, source buildSource, path, ref string, wai
defer ask.Close() defer ask.Close()
result, err := ask.Submit(ctx, link.BuildRequest{ result, err := ask.Submit(ctx, link.BuildRequest{
ID: fmt.Sprintf("%s-%d", "build", time.Now().UnixNano()), ID: link.NewBuildID(time.Now()),
Repository: repository, Path: path, Ref: ref, Repository: repository, Path: path, Ref: ref,
Held: heldBy(ctx), Seats: seatBases(ctx), Held: heldBy(ctx), Seats: seatBases(ctx),
}, wait) }, wait)
+87
View File
@@ -2,9 +2,12 @@ package main
import ( import (
"encoding/json" "encoding/json"
"errors"
"strings" "strings"
"testing" "testing"
"time"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link" "github.com/novox/mesh-controller/internal/link"
) )
@@ -63,3 +66,87 @@ func TestABuildHeardIsRecordedAndRegistered(t *testing.T) {
t.Fatalf("a failure is said in the builder's words: %v", err) t.Fatalf("a failure is said in the builder's words: %v", err)
} }
} }
// novox/hq 04-ISSUES/215: a build asked at a commit is recorded as built from that commit, and the
// module keeps following the branch it followed — a new one, the default branch.
func TestABuildAtACommitKeepsTheBranchTheModuleFollows(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
manifest, _ := json.Marshal(map[string]any{"module": "unifi", "version": "1"})
result := func(id, ref, commit string) link.BuildResult {
return link.BuildResult{ID: id, Repository: "http://forge.internal:20000/novox/mesh-catalog.git",
Path: "modules/unifi", Ref: ref, On: "anchor", Commit: commit, Manifest: manifest,
Source: &link.SourceOnSeat{Seat: "git", Repository: "novox/mesh-catalog"}}
}
if _, _, err := takeIn(ctx, open.inventory, result("b-1", "main", "1111111aaaa")); err != nil {
t.Fatal(err)
}
if _, _, err := takeIn(ctx, open.inventory, result("b-2", "9c97a8a", "9c97a8a1d2c3")); err != nil {
t.Fatal(err)
}
src, err := open.inventory.SourceOf(ctx, "unifi")
if err != nil {
t.Fatal(err)
}
if src.Ref != "main" || src.BuiltFrom != "9c97a8a1d2c3" {
t.Errorf("after a build at a commit the module follows %q, built from %q; want main, 9c97a8a1d2c3", src.Ref, src.BuiltFrom)
}
// One new to the catalogue, first built at a commit, follows the default branch.
other, _ := json.Marshal(map[string]any{"module": "letta", "version": "1"})
r := result("b-3", "deadbeef", "deadbeefcafe")
r.Manifest, r.Path = other, "modules/letta"
if _, _, err := takeIn(ctx, open.inventory, r); err != nil {
t.Fatal(err)
}
if src, _ := open.inventory.SourceOf(ctx, "letta"); src.Ref != "" {
t.Errorf("a module first built at a commit follows %q, want the default branch", src.Ref)
}
}
// novox/hq 04-ISSUES/219: an older request heard after a newer one is recorded and not registered,
// so a push sends what the newer request built.
func TestAnOlderBuildHeardLaterDoesNotReplaceTheNewer(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
older := time.Date(2026, 10, 3, 21, 33, 45, 0, time.UTC)
newer := time.Date(2026, 10, 3, 21, 51, 57, 0, time.UTC)
result := func(asked time.Time, image string) link.BuildResult {
manifest, _ := json.Marshal(map[string]any{"module": "postgres", "version": image})
return link.BuildResult{ID: link.NewBuildID(asked), Repository: "http://forge.internal:20000/novox/mesh-catalog.git",
Path: "modules/postgres", Ref: "main", On: "anchor", Commit: "efff5415", Manifest: manifest,
Source: &link.SourceOnSeat{Seat: "git", Repository: "novox/mesh-catalog"}}
}
if _, _, err := takeIn(ctx, open.inventory, result(newer, "4bcd5f73")); err != nil {
t.Fatal(err)
}
_, _, err := takeIn(ctx, open.inventory, result(older, "0ab07fa9"))
if !errors.Is(err, inventory.ErrSuperseded) {
t.Fatalf("the older request's outcome was taken in as current: %v", err)
}
shelf, err := open.inventory.Catalogue(ctx)
if err != nil {
t.Fatal(err)
}
if got := shelf["postgres"].Version; got != "4bcd5f73" {
t.Errorf("postgres is %q; want the newer request's 4bcd5f73", got)
}
if builds, _ := open.inventory.Builds(ctx, "postgres", 5); len(builds) != 2 {
t.Errorf("the late build was not recorded: %v", builds)
}
}
func TestABuildIDSaysWhenItWasAsked(t *testing.T) {
at := time.Date(2026, 10, 3, 21, 51, 57, 392539762, time.UTC)
if got, ok := link.BuildAskedAt(link.NewBuildID(at)); !ok || !got.Equal(at) {
t.Errorf("read back %v %v; want %v", got, ok, at)
}
if got, ok := link.BuildAskedAt("build-1791064317392539762"); !ok || got.Format(time.TimeOnly) != "21:51:57" {
t.Errorf("the incident's id reads as %v %v", got, ok)
}
for _, id := range []string{"b-1", "build-2", "build-", "build-x", ""} {
if _, ok := link.BuildAskedAt(id); ok {
t.Errorf("%q read as a request time", id)
}
}
}
+13 -3
View File
@@ -8,6 +8,7 @@ package main
import ( import (
"context" "context"
"errors"
"flag" "flag"
"fmt" "fmt"
"os" "os"
@@ -253,25 +254,34 @@ func parseAround(set *flag.FlagSet, args []string) ([]string, error) {
// became of a build nobody was watching. // became of a build nobody was watching.
func (b builds) Built(ctx context.Context, result link.BuildResult) error { func (b builds) Built(ctx context.Context, result link.BuildResult) error {
manifest, _, err := takeIn(ctx, b.inv, result) manifest, _, err := takeIn(ctx, b.inv, result)
// When it was asked, so a plan takes as its outcome only a build asked for it or after it
// (novox/hq 04-ISSUES/219). Zero when the id does not say.
asked, _ := link.BuildAskedAt(result.ID)
switch { switch {
case err != nil && result.Failed != "": case err != nil && result.Failed != "":
fmt.Printf("%s: %v\n", result.ID, err) fmt.Printf("%s: %v\n", result.ID, err)
if result.Module != "" { if result.Module != "" {
planBuilt(ctx, b.open, result.Module, result.Commit, result.Failed) planBuilt(ctx, b.open, result.Module, result.Commit, result.Failed, asked)
} else { } else {
planFailedBuild(ctx, b.open, result) planFailedBuild(ctx, b.open, result)
} }
return nil return nil
case errors.Is(err, inventory.ErrSuperseded):
// Not a failure: the module is already at what a later request built. A plan that asked
// before that later request is answered by it; one that asked after it ignores this.
fmt.Printf("%s: %v\n", result.ID, err)
planBuilt(ctx, b.open, manifest.Module, result.Commit, "", asked)
return nil
case err != nil: case err != nil:
fmt.Printf("%s: heard and recorded, and not registered: %v\n", result.ID, err) fmt.Printf("%s: heard and recorded, and not registered: %v\n", result.ID, err)
if manifest.Module != "" { if manifest.Module != "" {
planBuilt(ctx, b.open, manifest.Module, result.Commit, err.Error()) planBuilt(ctx, b.open, manifest.Module, result.Commit, err.Error(), asked)
} }
return nil return nil
} }
fmt.Printf("%s: %s %s registered, built on %s from %s\n", fmt.Printf("%s: %s %s registered, built on %s from %s\n",
result.ID, manifest.Module, manifest.Version, result.On, short(result.Commit)) result.ID, manifest.Module, manifest.Version, result.On, short(result.Commit))
saysWhenThePolicyActs(ctx, b.inv, manifest.Module) saysWhenThePolicyActs(ctx, b.inv, manifest.Module)
planBuilt(ctx, b.open, manifest.Module, result.Commit, "") planBuilt(ctx, b.open, manifest.Module, result.Commit, "", asked)
return nil return nil
} }
+6
View File
@@ -266,6 +266,12 @@ func TestTheResolverIsToldEveryMachineOnTheNetworkAndToldAgainWhenOneLeaves(t *t
if _, err := assign(ctx, open, "anchor", "dnsmasq"); err != nil { if _, err := assign(ctx, open, "anchor", "dnsmasq"); err != nil {
t.Fatal(err) t.Fatal(err)
} }
// Its bus credential, as assigning issues it where the bus is reachable (novox/hq issue 203):
// no bus is known to this test, so it is minted here, or composing refuses the placeholder.
if _, err := open.inventory.MintBusPassword(ctx, inventory.BusUser{
Username: "anchor.dnsmasq", Kind: inventory.BusModule, Node: "anchor", Module: "dnsmasq"}); err != nil {
t.Fatal(err)
}
zones := func() string { zones := func() string {
t.Helper() t.Helper()
for _, r := range composed(t, open, "anchor").Resources { for _, r := range composed(t, open, "anchor").Resources {
+24
View File
@@ -211,3 +211,27 @@ func TestWhatAHandedOverModuleRecordsAboutItsSource(t *testing.T) {
} }
} }
} }
// novox/hq 04-ISSUES/215: a module once built at a commit still follows its branch — a merge into it
// matches the module, and a plan re-asks the branch, not the old commit.
func TestAModuleBuiltAtACommitStillFollowsItsBranch(t *testing.T) {
m := link.SourceMoved{Owner: "novox", Repo: "mesh-catalog", Base: "main"}
pinned := inventory.Source{Repository: "novox/mesh-catalog", Seat: "git", Ref: "9c97a8a"}
if !sourceIs(pinned, m) {
t.Error("a module whose record names a commit is left out of a merge into its branch")
}
full := inventory.Source{Repository: "novox/mesh-catalog", Seat: "git", Ref: "9c97a8a1d2c3b4a5f60718293a4b5c6d7e8f9012"}
if !sourceIs(full, m) {
t.Error("a full commit hash is read as a branch")
}
if got := followedBranch("9c97a8a"); got != "" {
t.Errorf("a plan would re-ask the old commit %q", got)
}
if got := followedBranch("release"); got != "release" {
t.Errorf("a branch is not followed as named: %q", got)
}
// A module that follows another branch is still not this merge's.
if sourceIs(inventory.Source{Repository: "novox/mesh-catalog", Seat: "git", Ref: "release"}, m) {
t.Error("a module following another branch was matched")
}
}
+81 -3
View File
@@ -272,7 +272,8 @@ func askTier(ctx context.Context, inv *inventory.Inventory, p *inventory.Plan) e
} }
source := buildSource{Repository: e.Source.Repository, Seat: e.Source.Seat} source := buildSource{Repository: e.Source.Repository, Seat: e.Source.Seat}
fmt.Printf(" tier %d: ", p.Tier) fmt.Printf(" tier %d: ", p.Tier)
if err := buildOne(ctx, source, e.Source.Path, e.Source.Ref, 0); err != nil { // The branch it follows, never a commit a build once named (novox/hq 04-ISSUES/215).
if err := buildOne(ctx, source, e.Source.Path, followedBranch(e.Source.Ref), 0); err != nil {
state.State = "failed" state.State = "failed"
state.Why = err.Error() state.Why = err.Error()
p.State = inventory.PlanFailed p.State = inventory.PlanFailed
@@ -287,7 +288,13 @@ func askTier(ctx context.Context, inv *inventory.Inventory, p *inventory.Plan) e
// planBuilt marks a module built (or failed) in every open plan whose current tier holds it, and // planBuilt marks a module built (or failed) in every open plan whose current tier holds it, and
// advances what that completes. Called from the daemon's take-in of every outcome. // advances what that completes. Called from the daemon's take-in of every outcome.
func planBuilt(ctx context.Context, open *stores, module, commit, failed string) { //
// **Only a build asked at or after the plan's ask is its outcome** (novox/hq 04-ISSUES/219). Two
// plans a few minutes apart both ask for a module; the earlier plan's build, finishing late, is not
// the later plan's answer — it stood on the bases from before the later plan's merge, and taking it
// would send machines, and the next tier, what the later merge replaced. asked is zero when the
// build's request time is not known, and such an outcome is taken as before.
func planBuilt(ctx context.Context, open *stores, module, commit, failed string, asked time.Time) {
inv := open.inventory inv := open.inventory
plans, err := inv.OpenPlans(ctx) plans, err := inv.OpenPlans(ctx)
if err != nil { if err != nil {
@@ -314,6 +321,9 @@ func planBuilt(ctx context.Context, open *stores, module, commit, failed string)
state = &inventory.PlanModule{} state = &inventory.PlanModule{}
p.Modules[module] = state p.Modules[module] = state
} }
if askedBefore(asked, state.AskedAt) {
continue
}
if failed != "" { if failed != "" {
state.State = "failed" state.State = "failed"
state.Why = failed state.Why = failed
@@ -399,6 +409,24 @@ func advanceOnce(ctx context.Context, open *stores, p *inventory.Plan,
} }
return true, nil return true, nil
} }
// **Asked: settle from the build records first** (novox/hq 04-ISSUES/214). An outcome is taken
// in by whichever controller hears it, and a merge to the controller's own repository replaces
// the controller in its first tier: the build that produced the new one is recorded, and the
// plan never hears it. The record is the fact; a build recorded after the ask is that tier's
// outcome, whoever was listening.
recorded := map[string][]inventory.Build{}
for _, m := range tier {
if s := p.Modules[m]; s != nil && s.State == "asked" {
builds, err := inv.Builds(ctx, m, 5)
if err != nil {
return false, err
}
recorded[m] = builds
}
}
if settleFromRecords(p, tier, recorded) {
return true, nil
}
// Asked: wait for every build. // Asked: wait for every build.
var latest time.Time var latest time.Time
for _, m := range tier { for _, m := range tier {
@@ -530,7 +558,8 @@ func planFailedBuild(ctx context.Context, open *stores, result link.BuildResult)
} }
for _, e := range entries { for _, e := range entries {
if repositoryMatches(e.Source.Repository, result.Repository) && e.Source.Path == result.Path { if repositoryMatches(e.Source.Repository, result.Repository) && e.Source.Path == result.Path {
planBuilt(ctx, open, e.Manifest.Module, result.Commit, result.Failed) asked, _ := link.BuildAskedAt(result.ID)
planBuilt(ctx, open, e.Manifest.Module, result.Commit, result.Failed, asked)
return return
} }
} }
@@ -755,3 +784,52 @@ func splitList(s string) []string {
} }
return out return out
} }
// settleFromRecords marks every module of the tier still `asked` built — or failed — from a build
// recorded after it was asked, and says whether it changed anything (novox/hq 04-ISSUES/214).
// Newest first, as Builds answers: the first record after the ask is the outcome of that ask.
func settleFromRecords(p *inventory.Plan, tier []string, recorded map[string][]inventory.Build) bool {
changed := false
for _, m := range tier {
s := p.Modules[m]
if s == nil || s.State != "asked" || s.AskedAt == nil {
continue
}
var outcome *inventory.Build
for i := range recorded[m] {
b := recorded[m][i]
if b.At.Before(*s.AskedAt) {
break
}
// Recorded after the ask and asked before it: an earlier ask's late outcome, not this
// one's (novox/hq 04-ISSUES/219).
if askedBefore(b.Asked, s.AskedAt) {
continue
}
outcome = &b
}
if outcome == nil {
continue
}
at := outcome.At
if outcome.Worked() {
s.State = "built"
s.BuiltAt = &at
s.Commit = outcome.Commit
} else {
s.State = "failed"
s.Why = outcome.Failed
p.State = inventory.PlanFailed
p.Note = fmt.Sprintf("%s failed to build in tier %d", m, p.Tier)
}
fmt.Printf("%s: %s settled from the build records as %s (%s)\n", p.ID, m, s.State, outcome.ID)
changed = true
}
return changed
}
// askedBefore is whether a build asked at asked was asked before a plan asked for its module — and
// so is not that plan's outcome (novox/hq 04-ISSUES/219). False when either time is not known.
func askedBefore(asked time.Time, planAsked *time.Time) bool {
return !asked.IsZero() && planAsked != nil && asked.Before(*planAsked)
}
+55
View File
@@ -126,3 +126,58 @@ func TestABundleIsPlannedAfterTheToolchainItIsCompiledIn(t *testing.T) {
t.Fatalf("the toolchain, then the bundle: %v", p.Tiers) t.Fatalf("the toolchain, then the bundle: %v", p.Tiers)
} }
} }
// novox/hq 04-ISSUES/214: a plan whose build outcome was recorded while no controller followed it —
// the controller rebuilding itself — settles from the build records instead of waiting for ever.
func TestAPlanSettlesAnAskedBuildFromTheRecords(t *testing.T) {
asked := time.Date(2026, 10, 3, 19, 20, 0, 0, time.UTC)
p := inventory.Plan{ID: "plan-1", Tiers: [][]string{{"mesh-controller", "builder"}, {"route-proxy"}},
Modules: map[string]*inventory.PlanModule{
"mesh-controller": {State: "asked", AskedAt: &asked},
"builder": {State: "asked", AskedAt: &asked},
}}
records := map[string][]inventory.Build{
// Newest first, as Builds answers: the build after the ask is the outcome.
"mesh-controller": {
{ID: "build-2", Commit: "2ebbb799", At: asked.Add(4 * time.Minute)},
{ID: "build-1", Commit: "06ea2168", At: asked.Add(-10 * time.Minute)},
},
// Only a build from before the ask: not this ask's outcome.
"builder": {{ID: "build-0", Commit: "06ea2168", At: asked.Add(-time.Hour)}},
}
if !settleFromRecords(&p, p.Tiers[0], records) {
t.Fatal("nothing settled, though the controller's build is recorded after the ask")
}
if s := p.Modules["mesh-controller"]; s.State != "built" || s.Commit != "2ebbb799" || s.BuiltAt == nil {
t.Errorf("the controller's ask is %+v, want built from 2ebbb799", s)
}
if s := p.Modules["builder"]; s.State != "asked" {
t.Errorf("an ask with no record after it was settled: %+v", s)
}
// novox/hq 04-ISSUES/219: a build recorded after the ask but asked before it — an earlier
// plan's late outcome — is not this ask's, built or failed.
r := inventory.Plan{ID: "plan-3", Tiers: [][]string{{"postgres"}},
Modules: map[string]*inventory.PlanModule{"postgres": {State: "asked", AskedAt: &asked}}}
late := map[string][]inventory.Build{"postgres": {
{ID: "build-old", Commit: "efff5415", Asked: asked.Add(-18 * time.Minute), At: asked.Add(12 * time.Minute)},
}}
if settleFromRecords(&r, r.Tiers[0], late) || r.Modules["postgres"].State != "asked" {
t.Errorf("an earlier ask's late outcome settled this ask: %+v", r.Modules["postgres"])
}
// Newest heard first: the earlier ask's late outcome, then this ask's own, heard before it.
late["postgres"] = append(late["postgres"], inventory.Build{ID: "build-mine", Commit: "4bcd5f73",
Asked: asked.Add(time.Second), At: asked.Add(5 * time.Minute)})
if !settleFromRecords(&r, r.Tiers[0], late) || r.Modules["postgres"].State != "built" ||
r.Modules["postgres"].Commit != "4bcd5f73" {
t.Errorf("this ask's own outcome, heard before the earlier ask's, did not settle it: %+v", r.Modules["postgres"])
}
// A failure recorded after the ask fails the plan, as hearing it would have.
q := inventory.Plan{ID: "plan-2", Tiers: [][]string{{"x"}},
Modules: map[string]*inventory.PlanModule{"x": {State: "asked", AskedAt: &asked}}}
settleFromRecords(&q, q.Tiers[0], map[string][]inventory.Build{"x": {{ID: "b", Failed: "no", At: asked.Add(time.Minute)}}})
if q.State != inventory.PlanFailed || q.Modules["x"].State != "failed" {
t.Errorf("a recorded failure did not fail the plan: %+v %+v", q, q.Modules["x"])
}
}
+27 -1
View File
@@ -5,6 +5,7 @@ import (
"errors" "errors"
"flag" "flag"
"fmt" "fmt"
"regexp"
"strings" "strings"
"time" "time"
@@ -283,6 +284,14 @@ func (f following) SourceMoved(ctx context.Context, m link.SourceMoved) error {
if isHistory(m.MergedAt, lastLookAt(entries, m)) { if isHistory(m.MergedAt, lastLookAt(entries, m)) {
packaging = nil packaging = nil
} }
// Said, never silent (novox/hq 04-ISSUES/215): a module built from this repository that follows
// another branch is not part of this merge, and whoever is waiting for its change should read why.
for _, e := range entries {
if sameRepository(e.Source.Repository, m) && !sourceIs(e.Source, m) {
fmt.Printf(" %s is built from %s/%s and follows %s, not %s; this merge leaves it out\n",
e.Manifest.Module, m.Owner, m.Repo, e.Source.Ref, m.Base)
}
}
touched := whatTheMergeTouched(from, entries, m) touched := whatTheMergeTouched(from, entries, m)
for _, e := range touched { for _, e := range touched {
if err := inv.SourceMoved(ctx, e.Manifest.Module, m.Commit); err != nil { if err := inv.SourceMoved(ctx, e.Manifest.Module, m.Commit); err != nil {
@@ -345,7 +354,24 @@ func sourceIs(s inventory.Source, m link.SourceMoved) bool {
if !sameRepository(s.Repository, m) { if !sameRepository(s.Repository, m) {
return false return false
} }
return s.Ref == "" || s.Ref == m.Base ref := followedBranch(s.Ref)
return ref == "" || ref == m.Base
}
// commitRef is a ref that names a commit rather than a branch: what `build --ref <commit>` asks for.
var commitRef = regexp.MustCompile(`^[0-9a-f]{7,40}$`)
// followedBranch is the branch a recorded ref means a module follows (novox/hq 04-ISSUES/215). **A
// commit is never a branch to follow.** A build asked at a commit — to try one, or to pin it during a
// fix — recorded that commit as the module's ref; every merge after it then failed to match the
// module, its plan left it out without saying so, and every plan that rebuilt it asked for that same
// old commit again. A commit recorded so is read as the repository's default branch, which is what
// the module followed before it; a branch is followed as named.
func followedBranch(ref string) string {
if commitRef.MatchString(strings.TrimSpace(ref)) {
return ""
}
return ref
} }
// sameRepository is whether a recorded repository is the one a merge names, in either spelling it // sameRepository is whether a recorded repository is the one a merge names, in either spelling it
+99 -1
View File
@@ -593,6 +593,12 @@ func one(ctx context.Context, run Runner, publish Publisher,
if err != nil { if err != nil {
return catalogue.Built{}, fmt.Errorf("%s: writing %s's launchers failed: %w", module, a.Name, err) return catalogue.Built{}, fmt.Errorf("%s: writing %s's launchers failed: %w", module, a.Name, err)
} }
if chain.Bundler != "" {
say("bundle", "bundling each entrypoint into one file")
if compiled, err = bundled(ctx, run, tree, chain, base, a, launchers); err != nil {
return catalogue.Built{}, fmt.Errorf("%s: bundling %s failed: %w", module, a.Name, err)
}
}
say("bundle", "compiled, packing") say("bundle", "compiled, packing")
body, err := pack(compiled) body, err := pack(compiled)
if err != nil { if err != nil {
@@ -974,7 +980,7 @@ func compile(ctx context.Context, run Runner, tree string, chain Toolchain,
if _, err := run(ctx, tree, "docker", invocation...); err != nil { if _, err := run(ctx, tree, "docker", invocation...); err != nil {
return "", err return "", err
} }
if chain.Dependencies != "" { if chain.Dependencies != "" && chain.Bundler == "" {
// **What the bundle runs with, from the image it was compiled in** (Toolchain.Dependencies). // **What the bundle runs with, from the image it was compiled in** (Toolchain.Dependencies).
// A second run in the same image rather than a shell wrapped around the compiler: the // A second run in the same image rather than a shell wrapped around the compiler: the
// compile line stays a plain command a reader can run by hand, and the copy is one more // compile line stays a plain command a reader can run by hand, and the copy is one more
@@ -1224,3 +1230,95 @@ func writeLaunchers(root string, chain Toolchain, a catalogue.Artifact) (map[str
} }
return out, nil return out, nil
} }
// bundledSuffix is where a bundle's one-file output is written, beside what the compiler wrote.
const bundledSuffix = ".bundled"
// bundled makes every entrypoint and every launcher of a compiled bundle ONE file, in the toolchain
// image's bundler, and answers the directory to pack (novox/hq ADR 0193).
//
// **What a launched bundle runs is what it imports, and nothing else.** Every served bundle is its
// own process, so it carries its own copy of the SDK and its own dependencies inlined — the
// toolchain's whole node_modules no longer travels in every bundle. An entrypoint a process runs by
// name (`node daemon/index.js`) is bundled in place under its own name; a launcher keeps its name
// and its first line, and stays executable. A package the bundler cannot inline is named by the
// artifact (`external`), kept as an import, and only then is the toolchain's runtime directory
// copied beside the files. CommonJS inlined into an ES module still finds `require`.
func bundled(ctx context.Context, run Runner, tree string, chain Toolchain, base string,
a catalogue.Artifact, launchers map[string]string) (string, error) {
const within = "/app/modules/module"
out, final := Out(a.Name), Out(a.Name)+bundledSuffix
if err := os.RemoveAll(filepath.Join(tree, final)); err != nil {
return "", err
}
if err := os.MkdirAll(filepath.Join(tree, final), 0o755); err != nil {
return "", err
}
common := []string{"--bundle", "--platform=node", "--format=esm", "--target=node22",
"--outbase=" + out, "--outdir=" + final, "--log-level=warning",
"--banner:js=import { createRequire as __meshRequire } from 'node:module'; const require = __meshRequire(import.meta.url);"}
for _, x := range a.External {
common = append(common, "--external:"+x)
}
var plain []string
for _, e := range a.Entrypoints {
if strings.HasSuffix(e, ".js") {
plain = append(plain, out+"/"+e)
}
}
var launch []string
for _, l := range sortedValues(launchers) {
launch = append(launch, out+"/"+l)
}
// Refused by name in an image that predates the bundler, as the dependencies copy is: a bundle
// packed without it would carry nothing it imports. Run as itself: npm installs esbuild's native
// binary in place of its script, which `node` cannot run.
guard := `test -x "$0" || { echo "the toolchain image carries no bundler at $0: it predates one-file bundles, rebuild mesh-tools first" >&2; exit 1; }; exec "$0" "$@"`
step := func(entries []string, extra ...string) error {
if len(entries) == 0 {
return nil
}
invocation := []string{"run", "--rm", "--volume", tree + ":" + within, "--workdir", within, base,
"sh", "-c", guard, chain.Bundler}
invocation = append(invocation, entries...)
invocation = append(invocation, common...)
invocation = append(invocation, extra...)
_, err := run(ctx, tree, "docker", invocation...)
return err
}
if err := step(plain); err != nil {
return "", err
}
if err := step(launch, "--out-extension:.js=.mjs"); err != nil {
return "", err
}
// Plain `.js` output is an ES module; said once, as the runtime directory used to say it.
if err := os.WriteFile(filepath.Join(tree, final, "package.json"), []byte(`{"type":"module","private":true}`+"\n"), 0o644); err != nil {
return "", err
}
for _, l := range launchers {
path := filepath.Join(tree, final, filepath.FromSlash(l))
if _, err := os.Stat(path); err == nil {
if err := os.Chmod(path, 0o755); err != nil {
return "", err
}
}
}
if len(a.External) > 0 && chain.Dependencies != "" {
copying := []string{"run", "--rm", "--volume", tree + ":" + within, "--workdir", within, base,
"sh", "-c", `cp -a "$0/node_modules" "$1/"`, chain.Dependencies, final}
if _, err := run(ctx, tree, "docker", copying...); err != nil {
return "", fmt.Errorf("copying the packages %s keeps external: %w", a.Name, err)
}
}
return filepath.Join(tree, final), nil
}
func sortedValues(m map[string]string) []string {
out := make([]string, 0, len(m))
for _, v := range m {
out = append(out, v)
}
sort.Strings(out)
return out
}
+38 -14
View File
@@ -83,25 +83,49 @@ func TestABundleIsCompiledAndPackedWithNoDockerfile(t *testing.T) {
t.Fatalf("the bundle was not pinned: %v", got.Manifest.Resources[0]) t.Fatalf("the bundle was not pinned: %v", got.Manifest.Resources[0])
} }
// **And what it runs with, from the image it was compiled in** (novox/hq to-be 38 WP3). A // **One file per entrypoint and launcher, in the toolchain's bundler** (novox/hq ADR 0193). A
// second run in the same toolchain image copies the toolchain's runtime directory — the // second run in the same toolchain image bundles each into the artifact's bundled output, the SDK
// `"type": "module"` package.json and the pruned node_modules — into the output's root, and // inlined, refusing by name in an image that predates the bundler; and the toolchain's
// refuses by name when the image carries none rather than packing a bundle that starts nowhere. // node_modules is no longer copied into a bundle that keeps nothing external.
var copied string var bundling []string
for _, line := range r.ran { for _, line := range r.ran {
if strings.HasPrefix(line, "docker run") && strings.Contains(line, "/app/runtime") { if strings.HasPrefix(line, "docker run") && strings.Contains(line, "esbuild") {
copied = line bundling = append(bundling, line)
} }
} }
if copied == "" { if len(bundling) != 2 {
t.Fatalf("the bundle's dependencies were not copied in after the compile:\n%s", strings.Join(r.ran, "\n")) t.Fatalf("want one bundling run for the entrypoints and one for the launchers:\n%s", strings.Join(r.ran, "\n"))
} }
if !strings.Contains(copied, "mesh-tools/build@sha256:") || !strings.Contains(copied, "predates") || for _, want := range []string{"mesh-tools/build@sha256:", "predates one-file bundles", "--bundle", "--format=esm",
!strings.Contains(copied, Out("code")) { "--platform=node", "--outdir=" + Out("code") + ".bundled", Out("code") + "/index.js"} {
t.Fatalf("the copy does not run in the same toolchain, refuse an older image by name, or land in the artifact's output: %s", copied) if !strings.Contains(bundling[0], want) {
t.Errorf("the entrypoints' bundling lacks %q: %s", want, bundling[0])
}
} }
if strings.Index(strings.Join(r.ran, "\n"), "--outDir") > strings.Index(strings.Join(r.ran, "\n"), "/app/runtime") { if !strings.Contains(bundling[1], Out("code")+"/index.serve.mjs") || !strings.Contains(bundling[1], "--out-extension:.js=.mjs") {
t.Fatal("the dependencies were copied before the compile wrote its output") t.Errorf("the launcher is not bundled under its own name: %s", bundling[1])
}
if strings.Contains(strings.Join(r.ran, "\n"), "/app/runtime") {
t.Errorf("the toolchain's node_modules was copied into a bundle that keeps nothing external:\n%s", strings.Join(r.ran, "\n"))
}
if strings.Index(strings.Join(r.ran, "\n"), "--outDir") > strings.Index(strings.Join(r.ran, "\n"), "esbuild") {
t.Fatal("the bundler ran before the compile wrote its output")
}
}
// A bundle naming packages it keeps external is bundled with them as imports, and carries the
// toolchain's node_modules for them — the one case it still does.
func TestABundleKeepingAPackageExternalCarriesTheToolchainsModules(t *testing.T) {
manifest := strings.Replace(aBundle, `"entrypoints":["index.js"]`, `"entrypoints":["index.js"],"external":["sharp"]`, 1)
r, workspace := aRepository(t, manifest, map[string]string{"index.ts": "console.log(1)"})
held := map[string]string{"mesh-tools/build": "registry.invalid/mesh-tools/build@sha256:" + strings.Repeat("b", 64)}
if _, err := Build(context.Background(), compiling{r}.run, r,
"https://forge.invalid/greeter.git", "", "", workspace, held, Npmrc{}, GitCredential{}, nil); err != nil {
t.Fatal(err)
}
all := strings.Join(r.ran, "\n")
if !strings.Contains(all, "--external:sharp") || !strings.Contains(all, "/app/runtime") {
t.Errorf("an external package was not kept as an import with the toolchain's modules beside it:\n%s", all)
} }
} }
+20
View File
@@ -200,3 +200,23 @@ func TestWhatABuildReadIsTheRepositoriesItsRecipesName(t *testing.T) {
t.Fatal("a module whose recipes name no other repository read one") t.Fatal("a module whose recipes name no other repository read one")
} }
} }
// novox/hq 04-ISSUES/212: a toolchain stands on the SDK's published package, and is built with the
// exact version the mesh published — an argument that changes when the SDK does, so a rebuild after
// a release never reuses an install of the version before it.
func TestAPackageTheMeshPublishedIsPassedByItsExactVersion(t *testing.T) {
manifest := catalogue.Manifest{
Module: "mesh-tools",
Build: &catalogue.Build{
On: []catalogue.BuildsOn{{Arg: "MESH_SDK", Module: "mesh-sdk", Artifact: "lib"}},
},
}
held := map[string]string{"mesh-sdk/lib": "@novox/mesh-sdk@0.1.6"}
args, resolved, err := standingOn(context.Background(), manifest, held, noMirror)
if err != nil {
t.Fatal(err)
}
if fmt.Sprint(args) != "[--build-arg MESH_SDK=@novox/mesh-sdk@0.1.6]" || fmt.Sprint(resolved) != "[@novox/mesh-sdk@0.1.6]" {
t.Errorf("the package was passed as %v, recorded as %v", args, resolved)
}
}
+10
View File
@@ -73,7 +73,16 @@ type Toolchain struct {
// //
// A toolchain image without the directory fails the build by name rather than packing a bundle // A toolchain image without the directory fails the build by name rather than packing a bundle
// that starts nowhere: the image predates this and must be rebuilt first. // that starts nowhere: the image predates this and must be rebuilt first.
//
// *Since the bundler (below):* copied only for a bundle that names packages it keeps external,
// which cannot be inlined; a bundle with none carries no node_modules at all.
Dependencies string Dependencies string
// Bundler is the bundler inside the toolchain image that makes each compiled entrypoint and each
// launcher ONE self-contained file (novox/hq ADR 0193): every served bundle is its own process
// now, so each carries its own copy of what it imports — the SDK included — and nothing else.
// A bundle shrinks from the toolchain's whole node_modules to the code it runs. Empty for a
// language whose build is already one file.
Bundler string
// SystemStamp is the variable this language's linker fills with the artifact's declared system, // SystemStamp is the variable this language's linker fills with the artifact's declared system,
// for a language whose binaries are pinned to one at link time (novox/hq ADR 0005). // for a language whose binaries are pinned to one at link time (novox/hq ADR 0005).
// //
@@ -139,6 +148,7 @@ var toolchains = []Toolchain{
Unit: UnitSources, Unit: UnitSources,
SourceExt: ".ts", SourceExt: ".ts",
Dependencies: "/app/runtime", Dependencies: "/app/runtime",
Bundler: "/app/node_modules/esbuild/bin/esbuild",
}, },
{ {
Language: "go", Language: "go",
+46
View File
@@ -87,6 +87,12 @@ func (m Manifest) Resolve(built []Built) (Manifest, error) {
loads := append([]string(nil), a.Loads...) loads := append([]string(nil), a.Loads...)
if a.Loads == nil && len(m.Tools) > 0 { if a.Loads == nil && len(m.Tools) > 0 {
loads = append([]string(nil), a.Entrypoints...) loads = append([]string(nil), a.Entrypoints...)
// A bundle compiled to a binary has no entrypoints: the binary is what it is, and what
// the runtime starts to serve it (novox/hq ADR 0193). So a Go tools bundle is served
// as Go — the runtime execs it — exactly as a TypeScript one is through its launcher.
if bin := BinaryOf(a); bin != "" {
loads = []string{bin}
}
} }
// **Kept, never routed** (ADR 0155): the builder publishes to the store at the address // **Kept, never routed** (ADR 0155): the builder publishes to the store at the address
// it reached it by, and a manifest carrying that address names an installation — // it reached it by, and a manifest carrying that address names an installation —
@@ -208,6 +214,11 @@ func (b *Build) problems(module string) []string {
// A bundle's source is the module's own directory by definition, and what it needs to say // A bundle's source is the module's own directory by definition, and what it needs to say
// is which compiler — because the mesh chooses that, and cannot choose for a module that // is which compiler — because the mesh chooses that, and cannot choose for a module that
// has not said. // has not said.
if len(a.External) > 0 && (a.Kind != ArtifactBundle || a.Language != "typescript") {
problems = append(problems, fmt.Sprintf(
"%s: %q names packages it keeps external, and only a TypeScript bundle is bundled into "+
"one file with some kept out (novox/hq ADR 0193)", module, a.Name))
}
if len(a.Env) > 0 && a.Kind != ArtifactBundle { if len(a.Env) > 0 && a.Kind != ArtifactBundle {
problems = append(problems, fmt.Sprintf( problems = append(problems, fmt.Sprintf(
"%s: %q is a %q and says what it is given (env). Only a bundle the node's runtime "+ "%s: %q is a %q and says what it is given (env). Only a bundle the node's runtime "+
@@ -242,6 +253,11 @@ func (b *Build) problems(module string) []string {
for _, e := range a.Entrypoints { for _, e := range a.Entrypoints {
found = found || e == load found = found || e == load
} }
// A bundle compiled to a binary is one executable: the runtime loads that or nothing
// (novox/hq ADR 0193).
if bin := BinaryOf(a); bin != "" {
found = load == bin
}
if !found { if !found {
problems = append(problems, fmt.Sprintf( problems = append(problems, fmt.Sprintf(
"%s: %q says the runtime loads %q, which is not among its entrypoints — "+ "%s: %q says the runtime loads %q, which is not among its entrypoints — "+
@@ -436,3 +452,33 @@ func BinaryOf(a Artifact) string {
} }
return a.Name return a.Name
} }
// undeliveredBundles says which of a module's bundles nothing would ever put on a machine (novox/hq
// 04-ISSUES/216). A bundle reaches a machine three ways: the node's runtime serves it (it says
// `loads`, or its module declares `tools`), a resource names it (a process, a step, an archive), or
// it is the runtime itself. One reached by none of them was built, recorded and pushed as success,
// and was simply absent — seven modules' tools went missing that way on 2026-10-03. Refused here,
// naming the field that would deliver it.
func undeliveredBundles(m Manifest) []string {
if m.Build == nil || m.Module == RuntimeModule {
return nil
}
named := map[string]bool{}
for _, r := range m.Resources {
if a, ok := r["artifact"].(string); ok && a != "" {
named[a] = true
}
}
var problems []string
for _, a := range m.Build.Artifacts {
if a.Kind != ArtifactBundle || named[a.Name] || len(a.Loads) > 0 || len(m.Tools) > 0 {
continue
}
problems = append(problems, fmt.Sprintf(
"%s: the bundle %q would be built and never reach a machine: nothing loads it, runs it or "+
"unpacks it. A tools bundle says `loads` (the entrypoints the node's runtime serves) or its "+
"module lists its `tools`; a daemon or a step is a resource naming it (novox/hq 04-ISSUES/216)",
m.Module, a.Name))
}
return problems
}
+6
View File
@@ -757,6 +757,11 @@ type Artifact struct {
// list twice. A module declaring no tools has nothing the runtime loads, whatever it compiles. // list twice. A module declaring no tools has nothing the runtime loads, whatever it compiles.
Loads []string `json:"loads,omitempty"` Loads []string `json:"loads,omitempty"`
// External are packages a TypeScript bundle keeps as imports rather than inlining — a native
// addon, a package that reads its own files — and so carries the toolchain's node_modules for
// (novox/hq ADR 0193). Absent for nearly every bundle, which is then one file per entrypoint.
External []string `json:"external,omitempty"`
// Env is what a tools bundle is given on a machine (novox/hq ADR 0192): words and their values, // Env is what a tools bundle is given on a machine (novox/hq ADR 0192): words and their values,
// paths and constants composed with ${dir:…} and ${port:…} exactly as a container's environment // paths and constants composed with ${dir:…} and ${port:…} exactly as a container's environment
// is, never a secret's content. The node's runtime hands it to this bundle and to no other. // is, never a secret's content. The node's runtime hands it to this bundle and to no other.
@@ -1373,6 +1378,7 @@ func ParseManifest(raw []byte) (Manifest, error) {
} }
} }
problems = append(problems, m.Build.problems(m.Module)...) problems = append(problems, m.Build.problems(m.Module)...)
problems = append(problems, undeliveredBundles(m)...)
// **What provides the artifact store cannot be delivered through it** (novox/hq 04-ISSUES/029). // **What provides the artifact store cannot be delivered through it** (novox/hq 04-ISSUES/029).
// //
// Building publishes to the store, and the builder will not start without one. So a module // Building publishes to the store, and the builder will not start without one. So a module
+10 -5
View File
@@ -31,7 +31,7 @@ import (
// //
// So a module asks. `${port:8080}` is "the machine-side port you gave me for the 8080 I said I // So a module asks. `${port:8080}` is "the machine-side port you gave me for the 8080 I said I
// listen on", and the module writes that where it would otherwise have written a literal — in a // listen on", and the module writes that where it would otherwise have written a literal — in a
// file's content, or in a value of a container's `env`. // file's content, or in a value of a container's or a process's `env`.
// //
// **The environment is filled by the control plane, exactly as a bound value is.** A port is not // **The environment is filled by the control plane, exactly as a bound value is.** A port is not
// secret — the mesh holds it in the clear — so there is nothing for the host to be the only // secret — the mesh holds it in the clear — so there is nothing for the host to be the only
@@ -64,7 +64,12 @@ func portsUsed(content string) []int {
} }
// portInto replaces a resource's ${port:…} placeholders with what this machine assigned — in a // portInto replaces a resource's ${port:…} placeholders with what this machine assigned — in a
// file's content, and in a value of a container's environment. // file's content, and in a value of a container's or a process's environment.
//
// **A process's environment is a container's** (novox/hq to-be 38 WP4c). A module's code moving out
// of its container becomes a process on the machine and still has to be told what the container
// was told; filled for one kind and not the other, the literal reached the process and was read as
// a port, and the modules that moved first wrote their run-once steps a 0600 env file instead.
// //
// A port the module did not say it listens on is refused, for the same reason a binding's unknown // A port the module did not say it listens on is refused, for the same reason a binding's unknown
// key is: the module is asking about something it never declared, and the answer would be a guess. // key is: the module is asking about something it never declared, and the answer would be a guess.
@@ -84,7 +89,7 @@ func portInto(resource map[string]any, module string, listens []Listening, with
} }
resource["content"] = filled resource["content"] = filled
case "container": case "container", "process":
env, ok := resource["env"].(map[string]any) env, ok := resource["env"].(map[string]any)
if !ok { if !ok {
return nil return nil
@@ -106,8 +111,8 @@ func portInto(resource map[string]any, module string, listens []Listening, with
continue continue
} }
value, err := portsFilledInto(written, value, err := portsFilledInto(written,
fmt.Sprintf("%s's container %s sets %s to something that", fmt.Sprintf("%s's %s %s sets %s to something that",
module, resource["name"], key), module, listens, with) module, resource["type"], resource["name"], key), module, listens, with)
if err != nil { if err != nil {
return err return err
} }
+80
View File
@@ -0,0 +1,80 @@
package catalogue
import (
"strings"
"testing"
)
// **A process's environment is composed as a container's is** (novox/hq to-be 38 WP4c).
//
// A module's code moving out of its container becomes a process on the machine, and what its
// container's environment asked for — the port this machine gave the module, the place it put the
// module's directory — it still has to be told. Filled for a container and not for a process, the
// literal `${port:8080}` reached the process as its environment and was read as a port; the modules
// that moved first wrote their run-once steps an env file instead.
func processModule(env map[string]any) Manifest {
return Manifest{
Module: "showcase",
Listens: []Listening{{Port: 8080, From: FromMesh}},
Resources: []map[string]any{
{"id": "data", "type": "directory", "mode": "0700"},
{"id": "setup", "type": "process", "name": "showcase-setup", "run-once": true,
"run": []any{"/usr/bin/showcase", "setup"}, "env": env},
},
}
}
func TestAProcessIsToldItsPortAndItsPlaceInItsEnvironment(t *testing.T) {
env := map[string]any{
"SHOWCASE_URL": "http://127.0.0.1:${port:8080}",
"SHOWCASE_DATA": "${dir:data}/objects",
"SHOWCASE_DB": "127.0.0.1:${seat:mesh-store:5432}",
"GREETING": "hello",
}
out, err := Resolution{Node: "anchor", Modules: []Manifest{processModule(env)}}.Declaration(Rendering{
Ports: map[string]map[int]int{"showcase": {8080: 21000}},
Seats: map[string]map[int]int{"mesh-store": {5432: 6852}},
})
if err != nil {
t.Fatalf("a process asking for its port and its place does not compose: %v", err)
}
setup := fileNamed(out, "showcase.setup")
if setup == nil {
t.Fatalf("the process is not in the declaration: %v", out)
}
got, _ := setup["env"].(map[string]any)
for key, want := range map[string]string{
"SHOWCASE_URL": "http://127.0.0.1:21000",
"SHOWCASE_DATA": "/var/lib/showcase/data/objects",
"SHOWCASE_DB": "127.0.0.1:6852",
"GREETING": "hello",
} {
if got[key] != want {
t.Errorf("the process is told %s=%v, want %q", key, got[key], want)
}
}
if env["SHOWCASE_URL"] != "http://127.0.0.1:${port:8080}" {
t.Fatalf("composing for one machine edited the module's own manifest: %v", env)
}
}
// An unknown reference in a process's environment is refused as a container's is, naming the
// process and the variable — left alone, it would reach the machine as a literal.
func TestAProcessAskingAboutAnUndeclaredPortIsRefused(t *testing.T) {
env := map[string]any{"SHOWCASE_URL": "http://127.0.0.1:${port:9999}"}
_, err := Resolution{Node: "anchor", Modules: []Manifest{processModule(env)}}.Declaration(Rendering{})
if err == nil {
t.Fatal("a process was told a port its module never said it listens on")
}
for _, said := range []string{"showcase-setup", "SHOWCASE_URL", "${port:9999}", "8080"} {
if !strings.Contains(err.Error(), said) {
t.Errorf("the refusal does not say %q: %v", said, err)
}
}
env = map[string]any{"SHOWCASE_DATA": "${dir:date}/objects"}
if _, err := (Resolution{Node: "anchor", Modules: []Manifest{processModule(env)}}).Declaration(Rendering{}); err == nil ||
!strings.Contains(err.Error(), "${dir:date}") {
t.Fatalf("a process naming no directory of its module was not refused: %v", err)
}
}
+68
View File
@@ -389,3 +389,71 @@ func TestARuntimeCompiledToABinaryRunsItself(t *testing.T) {
t.Errorf("the Go runtime is not told what to serve or whose it is: %v %v", env, process["user"]) t.Errorf("the Go runtime is not told what to serve or whose it is: %v %v", env, process["user"])
} }
} }
// novox/hq 04-ISSUES/216: a bundle nothing loads, runs or unpacks is refused at registration; saying
// `loads`, listing `tools`, or a resource naming it admits it.
func TestABundleNothingDeliversIsRefused(t *testing.T) {
base := func() Manifest {
return Manifest{Module: "baserow", Version: "1", Build: &Build{Artifacts: []Artifact{
{Name: "tools", Kind: ArtifactBundle, Language: "typescript", Entrypoints: []string{"tools/index.js"}}}}}
}
if p := undeliveredBundles(base()); len(p) != 1 || !strings.Contains(p[0], "never reach a machine") {
t.Fatalf("a bundle nothing delivers was admitted: %v", p)
}
loads := base()
loads.Build.Artifacts[0].Loads = []string{"tools/index.js"}
tools := base()
tools.Tools = []string{"baserow_list_rows"}
run := base()
run.Resources = []map[string]any{{"id": "daemon", "type": "process", "artifact": "tools", "run": []any{"node", "tools/index.js"}}}
runtime := base()
runtime.Module = RuntimeModule
for name, m := range map[string]Manifest{"loads": loads, "tools": tools, "a process": run, "the runtime": runtime} {
if p := undeliveredBundles(m); len(p) != 0 {
t.Errorf("a bundle delivered by %s was refused: %v", name, p)
}
}
}
// novox/hq ADR 0193: a Go tools bundle is served — its binary is what the runtime starts, delivered
// like any tools bundle, named to the runtime where a TypeScript bundle names its launcher.
func TestAGoToolsBundleIsServedByItsBinary(t *testing.T) {
with := Rendering{ArtifactStore: "anchor.internal:5101",
Needed: map[string]map[string]string{RuntimeModule: {"broker": "sealed-credential"}}}
lamp := Manifest{Module: "lamp", Version: "1", Tools: []string{"on"},
Build: &Build{Artifacts: []Artifact{{Name: "tools", Kind: ArtifactBundle, Language: "go",
System: "arch", From: "cmd/lamp-tools"}}}}
if p := lamp.Build.problems("lamp"); len(p) != 0 {
t.Fatalf("a Go tools bundle was refused: %v", p)
}
lamp, err := lamp.Resolve([]Built{{Name: "tools", Kind: ArtifactBundle,
Reference: ArtifactStoreScheme + "lamp/tools/blobs/" + bundleDigest, Digest: bundleDigest}})
if err != nil {
t.Fatal(err)
}
if fmt.Sprint(lamp.Bundles[0].Loads) != "[lamp-tools]" {
t.Fatalf("the runtime loads %v from a Go bundle, want its binary", lamp.Bundles[0].Loads)
}
out, err := Resolution{Node: "anchor", Account: "ops", Modules: []Manifest{lamp, theRuntime(t)}}.Declaration(with)
if err != nil {
t.Fatal(err)
}
if fileNamed(out, "lamp."+BundleID("tools")) == nil {
t.Errorf("the Go bundle is not delivered: %v", ids(out))
}
env := fileNamed(out, RuntimeModule+"."+RuntimeProcessID())["env"].(map[string]string)
if env[RuntimeToolModules] != "lamp="+BundlePath("lamp", "tools")+"/lamp-tools" {
t.Errorf("the runtime is told %q, want the binary", env[RuntimeToolModules])
}
// An artifact may say it explicitly; naming anything but the binary is refused.
said := Manifest{Module: "lamp", Version: "1", Build: &Build{Artifacts: []Artifact{{Name: "tools",
Kind: ArtifactBundle, Language: "go", System: "arch", Binary: "lamp", Loads: []string{"lamp"}}}}}
if p := said.Build.problems("lamp"); len(p) != 0 {
t.Errorf("loads naming the binary was refused: %v", p)
}
said.Build.Artifacts[0].Loads = []string{"tools/index.js"}
if p := said.Build.problems("lamp"); len(p) == 0 {
t.Error("a Go bundle loading a file it does not contain was admitted")
}
}
+5 -5
View File
@@ -43,8 +43,8 @@ import (
var ofSeat = regexp.MustCompile(`\$\{seat:([a-z0-9][a-z0-9-]*):([0-9]+)\}`) var ofSeat = regexp.MustCompile(`\$\{seat:([a-z0-9][a-z0-9-]*):([0-9]+)\}`)
// seatInto replaces a resource's ${seat:…} placeholders with where this machine put each seat's // seatInto replaces a resource's ${seat:…} placeholders with where this machine put each seat's
// holder — in a file's content, and in a value of a container's environment. The same two places // holder — in a file's content, and in a value of a container's or a process's environment. The
// portInto fills, for the same reason: they are where a process reads a number from. // same places portInto fills, for the same reason: they are where a program reads a number from.
func seatInto(resource map[string]any, module string, with Rendering) error { func seatInto(resource map[string]any, module string, with Rendering) error {
switch fmt.Sprint(resource["type"]) { switch fmt.Sprint(resource["type"]) {
case "file": case "file":
@@ -58,7 +58,7 @@ func seatInto(resource map[string]any, module string, with Rendering) error {
} }
resource["content"] = filled resource["content"] = filled
case "container": case "container", "process":
env, ok := resource["env"].(map[string]any) env, ok := resource["env"].(map[string]any)
if !ok { if !ok {
return nil return nil
@@ -78,8 +78,8 @@ func seatInto(resource map[string]any, module string, with Rendering) error {
continue continue
} }
value, err := seatsFilledInto(written, value, err := seatsFilledInto(written,
fmt.Sprintf("%s's container %s sets %s to something that", fmt.Sprintf("%s's %s %s sets %s to something that",
module, resource["name"], key), with) module, resource["type"], resource["name"], key), with)
if err != nil { if err != nil {
return err return err
} }
+43 -14
View File
@@ -42,9 +42,29 @@ type Build struct {
// Failed is the builder's own words, empty when it worked. // Failed is the builder's own words, empty when it worked.
Failed string Failed string
Made []Artifact Made []Artifact
At time.Time // Asked is when the build was requested, zero when that is not known (an id of another shape,
// or a build recorded before the mesh kept it). **What orders one build of a module against
// another** (novox/hq 04-ISSUES/219): builds in flight together finish in any order, and the
// one asked last stood on the newest bases.
Asked time.Time
// At is when the outcome was recorded — when it finished, not when it was asked.
At time.Time
} }
// AskedOrAt is when the build was asked, or when it was recorded when that is not known — the
// order the mesh had before it kept the request time.
func (b Build) AskedOrAt() time.Time {
if !b.Asked.IsZero() {
return b.Asked
}
return b.At
}
// newestRequestFirst is the ordering every "what a module currently is" question uses: the newest
// request wins, whenever it finished (novox/hq 04-ISSUES/219). A build whose request time is not
// known is placed at the moment it was recorded, which is the rule that held before.
const newestRequestFirst = `coalesce(asked, at) desc, at desc`
// ReadRepository is a repository a build read source from besides the module's own. // ReadRepository is a repository a build read source from besides the module's own.
type ReadRepository struct { type ReadRepository struct {
Repository string `json:"repository"` Repository string `json:"repository"`
@@ -83,13 +103,17 @@ func (i *Inventory) RecordBuild(ctx context.Context, b Build) error {
if b.Module != "" { if b.Module != "" {
module = &b.Module module = &b.Module
} }
var asked *time.Time
if !b.Asked.IsZero() {
asked = &b.Asked
}
_, err = i.store.Pool().Exec(ctx, _, err = i.store.Pool().Exec(ctx,
`insert into build (id, repository, ref, module, commit_hash, built_on, failed, made, `insert into build (id, repository, ref, module, commit_hash, built_on, failed, made,
source_path, manifest, built_against, built_contexts) source_path, manifest, built_against, built_contexts, asked)
values ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12) values ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13)
on conflict (id) do nothing`, on conflict (id) do nothing`,
b.ID, b.Repository, b.Ref, module, b.Commit, b.On, b.Failed, made, b.ID, b.Repository, b.Ref, module, b.Commit, b.On, b.Failed, made,
b.Path, manifestOrNil(b.Manifest), against, read) b.Path, manifestOrNil(b.Manifest), against, read, asked)
return err return err
} }
@@ -102,11 +126,11 @@ func (i *Inventory) Builds(ctx context.Context, module string, limit int) ([]Bui
if limit <= 0 { if limit <= 0 {
limit = 20 limit = 20
} }
query := `select id, repository, ref, coalesce(module,''), commit_hash, built_on, failed, made, at query := `select id, repository, ref, coalesce(module,''), commit_hash, built_on, failed, made, asked, at
from build order by at desc limit $1` from build order by at desc limit $1`
args := []any{limit} args := []any{limit}
if module != "" { if module != "" {
query = `select id, repository, ref, coalesce(module,''), commit_hash, built_on, failed, made, at query = `select id, repository, ref, coalesce(module,''), commit_hash, built_on, failed, made, asked, at
from build where module = $2 order by at desc limit $1` from build where module = $2 order by at desc limit $1`
args = append(args, module) args = append(args, module)
} }
@@ -121,10 +145,14 @@ func (i *Inventory) Builds(ctx context.Context, module string, limit int) ([]Bui
for rows.Next() { for rows.Next() {
var b Build var b Build
var made []byte var made []byte
var asked *time.Time
if err := rows.Scan(&b.ID, &b.Repository, &b.Ref, &b.Module, &b.Commit, if err := rows.Scan(&b.ID, &b.Repository, &b.Ref, &b.Module, &b.Commit,
&b.On, &b.Failed, &made, &b.At); err != nil { &b.On, &b.Failed, &made, &asked, &b.At); err != nil {
return nil, err return nil, err
} }
if asked != nil {
b.Asked = *asked
}
if err := json.Unmarshal(made, &b.Made); err != nil { if err := json.Unmarshal(made, &b.Made); err != nil {
return nil, err return nil, err
} }
@@ -135,8 +163,9 @@ func (i *Inventory) Builds(ctx context.Context, module string, limit int) ([]Bui
// Held is every artifact this mesh has built, keyed "<module>/<artifact>". // Held is every artifact this mesh has built, keyed "<module>/<artifact>".
// //
// **The newest successful build of each module wins**, which is the same rule the rest of the mesh // **The successful build of each module asked last wins**, which is the same rule the rest of the
// uses for what a module currently is. A module rebuilt to something broken and then rebuilt again // mesh uses for what a module currently is — asked last, not finished last (novox/hq
// 04-ISSUES/219): an older request that finishes later stood on older bases. A module rebuilt to something broken and then rebuilt again
// is at the second one; a module whose last build failed is at the last one that worked, because a // is at the second one; a module whose last build failed is at the last one that worked, because a
// failure published nothing and the thing it published before is still what exists. // failure published nothing and the thing it published before is still what exists.
// //
@@ -147,7 +176,7 @@ func (i *Inventory) Held(ctx context.Context) (map[string]string, error) {
`select distinct on (module) module, made `select distinct on (module) module, made
from build from build
where module is not null and module <> '' and failed = '' where module is not null and module <> '' and failed = ''
order by module, at desc`) order by module, `+newestRequestFirst)
if err != nil { if err != nil {
return nil, err return nil, err
} }
@@ -185,7 +214,7 @@ func (i *Inventory) BuiltAgainst(ctx context.Context) (map[string][]string, erro
`select distinct on (module) module, built_against `select distinct on (module) module, built_against
from build from build
where module is not null and module <> '' and failed = '' where module is not null and module <> '' and failed = ''
order by module, at desc`) order by module, `+newestRequestFirst)
if err != nil { if err != nil {
return nil, err return nil, err
} }
@@ -223,7 +252,7 @@ func (i *Inventory) ReadRepositories(ctx context.Context) (map[string][]ReadRepo
`select distinct on (module) module, built_contexts `select distinct on (module) module, built_contexts
from build from build
where module is not null and module <> '' and failed = '' where module is not null and module <> '' and failed = ''
order by module, at desc`) order by module, `+newestRequestFirst)
if err != nil { if err != nil {
return nil, err return nil, err
} }
@@ -274,7 +303,7 @@ func manifestOrNil(raw []byte) any {
// follows when it decides whether to announce at all. // follows when it decides whether to announce at all.
// //
// One row per module and commit: a module built twice at the same commit is one fact, and the // One row per module and commit: a module built twice at the same commit is one fact, and the
// latest row is the one whose artifacts are current. // row asked last is the one whose artifacts are current (novox/hq 04-ISSUES/219).
func (i *Inventory) Announceable(ctx context.Context) ([]Build, error) { func (i *Inventory) Announceable(ctx context.Context) ([]Build, error) {
rows, err := i.store.Pool().Query(ctx, rows, err := i.store.Pool().Query(ctx,
`select distinct on (module, commit_hash) `select distinct on (module, commit_hash)
@@ -282,7 +311,7 @@ func (i *Inventory) Announceable(ctx context.Context) ([]Build, error) {
source_path, manifest, built_against, at source_path, manifest, built_against, at
from build from build
where failed = '' and module is not null and module <> '' and commit_hash <> '' where failed = '' and module is not null and module <> '' and commit_hash <> ''
order by module, commit_hash, at desc`) order by module, commit_hash, `+newestRequestFirst)
if err != nil { if err != nil {
return nil, err return nil, err
} }
+39 -1
View File
@@ -49,6 +49,12 @@ func (i *Inventory) BusRecords(ctx context.Context) (broker.Records, error) {
} }
} }
// Who holds each seat held once for the mesh, where the mesh recorded it (novox/hq issue 218).
holdings, err := i.Holdings(ctx)
if err != nil {
return broker.Records{}, fmt.Errorf("cannot read who holds the mesh's seats: %w", err)
}
out := broker.Records{Assigned: map[string][]broker.Declared{}, People: map[string][]string{}, out := broker.Records{Assigned: map[string][]broker.Declared{}, People: map[string][]string{},
Interchangeable: map[string]bool{}} Interchangeable: map[string]bool{}}
for _, n := range nodes { for _, n := range nodes {
@@ -72,7 +78,9 @@ func (i *Inventory) BusRecords(ctx context.Context) (broker.Records, error) {
"%s is assigned to %s and is not in the catalogue, so what it may say cannot "+ "%s is assigned to %s and is not in the catalogue, so what it may say cannot "+
"be derived", module, n.Name) "be derived", module, n.Name)
} }
out.Assigned[n.Name] = append(out.Assigned[n.Name], declaredFor(m, seats)) d := declaredFor(m, seats)
d.Holds = heldHere(d.Holds, holdings, n.Name, module)
out.Assigned[n.Name] = append(out.Assigned[n.Name], d)
if m.Instances == catalogue.InstancesInterchangeable { if m.Instances == catalogue.InstancesInterchangeable {
out.Interchangeable[m.Module] = true out.Interchangeable[m.Module] = true
} }
@@ -183,3 +191,33 @@ func (i *Inventory) NodesWithALiveToken(ctx context.Context) ([]string, error) {
} }
return out, rows.Err() return out, rows.Err()
} }
// heldHere keeps of what a module claims only the seats it holds on this machine (novox/hq issue 218).
// A seat held once per machine is held by every assignment that claims it. A seat held once for the
// mesh is held by one assignment: where the mesh recorded who holds it, a claim on any other machine
// grants nothing and issues nothing — or the module would serve the role's verbs from a machine that
// is not the role's, and a question to the mesh's store would be answered from the wrong database. A
// mesh seat with no holder on record is left as it was derived.
func heldHere(claimed []broker.Seat, holdings []catalogue.Held, node, module string) []broker.Seat {
recorded := map[string][]catalogue.Held{}
for _, h := range holdings {
if h.Scope == catalogue.ScopeMesh {
recorded[h.Claim] = append(recorded[h.Claim], h)
}
}
var out []broker.Seat
for _, s := range claimed {
holders, onRecord := recorded[s.Name]
if s.Scope != catalogue.ScopeMesh || !onRecord {
out = append(out, s)
continue
}
for _, h := range holders {
if h.Node == node && h.Module == module {
out = append(out, s)
break
}
}
}
return out
}
+38 -6
View File
@@ -17,6 +17,10 @@ import (
// ErrNoSuchModule is what the mesh says about a module it has never been told about. // ErrNoSuchModule is what the mesh says about a module it has never been told about.
var ErrNoSuchModule = errors.New("no module of that name") var ErrNoSuchModule = errors.New("no module of that name")
// ErrSuperseded is a registration from a build asked before the one the module is already at
// (novox/hq 04-ISSUES/219). The build is recorded; what the module is does not change.
var ErrSuperseded = errors.New("a build asked later is already what the module is")
// ErrStillAssigned is why a module cannot be forgotten. // ErrStillAssigned is why a module cannot be forgotten.
// //
// Its own error because it is not a fault: it means a machine is running that module now, and // Its own error because it is not a fault: it means a machine is running that module now, and
@@ -47,6 +51,10 @@ type Source struct {
// itself no longer carries its build (novox/hq to-be 38 WP2.4). Empty for a manifest handed over // itself no longer carries its build (novox/hq to-be 38 WP2.4). Empty for a manifest handed over
// by hand, which carries its `build.on` itself. // by hand, which carries its `build.on` itself.
Against []string Against []string
// Asked is when the build this manifest came from was requested (novox/hq 04-ISSUES/219). Zero
// is a manifest handed over by hand, or a build whose request time is not known: either is
// taken as asked at the moment it is registered.
Asked time.Time
} }
// Current reports whether what the mesh holds is what the source last had. // Current reports whether what the mesh holds is what the source last had.
@@ -97,13 +105,23 @@ func (i *Inventory) RegisterModule(ctx context.Context, m catalogue.Manifest, fr
return err return err
} }
asked := from.Asked
if asked.IsZero() {
asked = time.Now()
}
// A module registered without provenance keeps whatever it had. Handing over a manifest by // A module registered without provenance keeps whatever it had. Handing over a manifest by
// hand is a legitimate way to fix something in a hurry, and it should not silently erase the // hand is a legitimate way to fix something in a hurry, and it should not silently erase the
// record of where the module normally comes from — which is the only thing that would say, // record of where the module normally comes from — which is the only thing that would say,
// afterwards, that the machine is running something nobody can rebuild. // afterwards, that the machine is running something nobody can rebuild.
_, err = i.store.Pool().Exec(ctx, //
`insert into module (name, manifest, version, source, source_path, source_seat, ref, built_from, source_head) // **An older request never replaces a newer one** (novox/hq 04-ISSUES/219). Builds of one
values ($1, $2, nullif($3,''), nullif($4,''), $7, $8, nullif($5,''), nullif($6,''), nullif($6,'')) // module in flight together finish in any order, and each stood on the bases the mesh held when
// it was asked; the one asked later is what the module is, whichever is heard last. An outcome
// of an earlier request is kept in the build records and changes nothing here.
tag, err := i.store.Pool().Exec(ctx,
`insert into module (name, manifest, version, source, source_path, source_seat, ref, built_from, source_head, built_asked)
values ($1, $2, nullif($3,''), nullif($4,''), $7, $8, nullif($5,''), nullif($6,''), nullif($6,''), $9)
on conflict (name) do update set on conflict (name) do update set
manifest = excluded.manifest, manifest = excluded.manifest,
version = excluded.version, version = excluded.version,
@@ -115,9 +133,23 @@ func (i *Inventory) RegisterModule(ctx context.Context, m catalogue.Manifest, fr
else excluded.source_seat end, else excluded.source_seat end,
ref = coalesce(excluded.ref, module.ref), ref = coalesce(excluded.ref, module.ref),
built_from = coalesce(excluded.built_from, module.built_from), built_from = coalesce(excluded.built_from, module.built_from),
source_head = coalesce(excluded.built_from, module.source_head)`, source_head = coalesce(excluded.built_from, module.source_head),
m.Module, raw, m.Version, from.Repository, from.Ref, from.BuiltFrom, from.Path, from.Seat) built_asked = excluded.built_asked
return err where module.built_asked is null or module.built_asked <= excluded.built_asked`,
m.Module, raw, m.Version, from.Repository, from.Ref, from.BuiltFrom, from.Path, from.Seat, asked)
if err != nil {
return err
}
if tag.RowsAffected() == 0 {
var current time.Time
if err := i.store.Pool().QueryRow(ctx,
`select built_asked from module where name = $1`, m.Module).Scan(&current); err != nil {
return err
}
return fmt.Errorf("%w: %s is at a build asked %s, and this one was asked %s",
ErrSuperseded, m.Module, current.UTC().Format(time.RFC3339), asked.UTC().Format(time.RFC3339))
}
return nil
} }
// registeredInThatShape is whether the catalogue already holds this module as a tools container on // registeredInThatShape is whether the catalogue already holds this module as a tools container on
+20
View File
@@ -100,3 +100,23 @@ func TestABundleStandsOnTheToolchainItIsCompiledIn(t *testing.T) {
} }
} }
} }
// novox/hq 04-ISSUES/212: a toolchain standing on the SDK's package is planned after the SDK, so a
// release of the SDK rebuilds the toolchain, and every bundle compiled in it after that.
func TestAToolchainStandingOnTheSDKFollowsIt(t *testing.T) {
entries := []Entry{
{Manifest: catalogue.Manifest{Module: "mesh-sdk"}},
{Manifest: catalogue.Manifest{Module: "mesh-tools", Build: &catalogue.Build{
On: []catalogue.BuildsOn{{Arg: "MESH_SDK", Module: "mesh-sdk", Artifact: "lib"}}}}},
}
edges := dependenciesOf(entries, nil, nil)
found := false
for _, e := range edges {
if e.From == "mesh-tools" && e.To == "mesh-sdk" {
found = true
}
}
if !found {
t.Errorf("no edge from the toolchain to the SDK: %v", edges)
}
}
+32
View File
@@ -0,0 +1,32 @@
package inventory
import (
"testing"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/catalogue"
)
// novox/hq issue 218: a seat held once for the mesh is granted and issued only to the holder on record;
// a node seat to every machine's claimant; a mesh seat with no holder on record as derived.
func TestOnlyTheRecordedHolderHoldsAMeshSeat(t *testing.T) {
claimed := []broker.Seat{
{Name: "mesh-store", Scope: catalogue.ScopeMesh},
{Name: "node-packet-filter", Scope: catalogue.ScopeNode},
{Name: "unrecorded", Scope: catalogue.ScopeMesh},
}
holdings := []catalogue.Held{{Claim: "mesh-store", Scope: catalogue.ScopeMesh, Node: "control", Module: "postgres"}}
names := func(ss []broker.Seat) (out []string) {
for _, s := range ss {
out = append(out, s.Name)
}
return
}
if got := names(heldHere(claimed, holdings, "control", "postgres")); len(got) != 3 {
t.Errorf("the holder lost a seat: %v", got)
}
got := names(heldHere(claimed, holdings, "other", "postgres"))
if len(got) != 2 || got[0] != "node-packet-filter" || got[1] != "unrecorded" {
t.Errorf("a claimant on another machine holds %v; want the node seat and the unrecorded one, not the store", got)
}
}
@@ -0,0 +1,23 @@
-- A build is ordered by when it was asked, not when it finished (novox/hq 04-ISSUES/219).
--
-- Two builds of one module can be in flight together — two merge plans a few minutes apart, each
-- asking for everything standing on what it changed — and they finish in any order. Each build
-- stands on the bases the mesh held when it was *asked*, so the one asked later is the newer one.
-- The mesh ordered builds by `at`, which is when the outcome was recorded, and registered whatever
-- it heard last: an older request that took longer replaced a newer one as what the module is, and
-- the next push sent machines an image built on a base the mesh had already replaced.
--
-- `build.asked` is when the build was requested, read from the correlation id the controller wrote
-- (`build-<unix nanoseconds>`). Nullable: an id of any other shape says no request time, and such a
-- build is placed where it was recorded, which is the order the mesh had before this.
alter table build add column asked timestamptz;
update build
set asked = to_timestamp((substring(id from '^build-([0-9]{19})$'))::numeric / 1000000000)
where id ~ '^build-[0-9]{19}$';
-- `module.built_asked` is when the build the module's registered manifest came from was asked, so
-- a later-heard outcome of an earlier request is recorded and not registered. A manifest handed over
-- by hand is a request made when it is handed over. Null for a module registered before this was
-- kept: its next registration, whichever it is, sets it.
alter table module add column built_asked timestamptz;
+124
View File
@@ -0,0 +1,124 @@
package inventory
import (
"context"
"errors"
"testing"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
)
// novox/hq 04-ISSUES/219: two builds of one module in flight together, the one asked first heard
// last. The newer request stood on the newer base; the older one's late outcome is recorded and is
// not what the module is.
func postgresBuild(id string, asked time.Time, image string) Build {
b := aBuild(id, "postgres", "")
b.Asked = asked
b.Against = []string{"mesh-tools/runtime@sha256:" + id}
b.Made = []Artifact{{Name: "server", Kind: "image", Reference: "postgres@sha256:" + image}}
return b
}
func TestAnOlderRequestFinishingLaterIsNotWhatTheModuleHolds(t *testing.T) {
inv := fresh(t)
ctx := context.Background()
older := time.Date(2026, 10, 3, 21, 33, 45, 0, time.UTC)
newer := time.Date(2026, 10, 3, 21, 51, 57, 0, time.UTC)
// The newer request finishes first, the older one last — recorded in that order.
if err := inv.RecordBuild(ctx, postgresBuild("newer", newer, "4bcd5f73")); err != nil {
t.Fatal(err)
}
if err := inv.RecordBuild(ctx, postgresBuild("older", older, "0ab07fa9")); err != nil {
t.Fatal(err)
}
held, err := inv.Held(ctx)
if err != nil {
t.Fatal(err)
}
if got := held["postgres/server"]; got != "postgres@sha256:4bcd5f73" {
t.Errorf("postgres holds %q; want the newer request's image 4bcd5f73", got)
}
against, err := inv.BuiltAgainst(ctx)
if err != nil {
t.Fatal(err)
}
if got := against["postgres"]; len(got) != 1 || got[0] != "mesh-tools/runtime@sha256:newer" {
t.Errorf("postgres stands on %v; want what the newer request stood on", got)
}
// Both are still recorded, the late one first as what happened lately.
builds, err := inv.Builds(ctx, "postgres", 5)
if err != nil {
t.Fatal(err)
}
if len(builds) != 2 || builds[0].ID != "older" || !builds[0].Asked.Equal(older) {
t.Fatalf("both builds, newest heard first, with when they were asked: %+v", builds)
}
}
func TestABuildWithNoKnownRequestTimeIsOrderedByWhenItWasRecorded(t *testing.T) {
// What the mesh did before it kept the request time, so a row from before still answers.
inv := fresh(t)
ctx := context.Background()
for _, id := range []string{"first", "second"} {
b := aBuild(id, "shell", "")
b.Made = []Artifact{{Name: "config", Kind: "archive", Reference: "…/" + id}}
if err := inv.RecordBuild(ctx, b); err != nil {
t.Fatal(err)
}
}
held, err := inv.Held(ctx)
if err != nil {
t.Fatal(err)
}
if got := held["shell/config"]; got != "…/second" {
t.Errorf("shell holds %q; want the one recorded last", got)
}
}
func TestARegistrationFromAnOlderRequestDoesNotReplaceANewerOne(t *testing.T) {
inv := fresh(t)
ctx := context.Background()
older := time.Date(2026, 10, 3, 21, 33, 45, 0, time.UTC)
newer := time.Date(2026, 10, 3, 21, 51, 57, 0, time.UTC)
from := func(asked time.Time) Source {
return Source{Repository: "novox/mesh-catalog", Seat: "git", Path: "modules/postgres",
BuiltFrom: "efff5415", Asked: asked}
}
if err := inv.RegisterModule(ctx, catalogue.Manifest{Module: "postgres", Version: "fixed"}, from(newer)); err != nil {
t.Fatal(err)
}
err := inv.RegisterModule(ctx, catalogue.Manifest{Module: "postgres", Version: "stale"}, from(older))
if !errors.Is(err, ErrSuperseded) {
t.Fatalf("an older request's registration was not refused as superseded: %v", err)
}
shelf, err := inv.Catalogue(ctx)
if err != nil {
t.Fatal(err)
}
if got := shelf["postgres"].Version; got != "fixed" {
t.Fatalf("postgres is %q; want the newer request's manifest", got)
}
// A later request, and a manifest handed over by hand — asked when it is handed over — both
// replace it as before.
if err := inv.RegisterModule(ctx, catalogue.Manifest{Module: "postgres", Version: "later"},
from(newer.Add(time.Minute))); err != nil {
t.Fatal(err)
}
if err := inv.RegisterModule(ctx, catalogue.Manifest{Module: "postgres", Version: "by-hand"}, Source{}); err != nil {
t.Fatal(err)
}
if shelf, _ := inv.Catalogue(ctx); shelf["postgres"].Version != "by-hand" {
t.Fatalf("postgres is %q; want the manifest handed over by hand", shelf["postgres"].Version)
}
src, err := inv.SourceOf(ctx, "postgres")
if err != nil || src.Repository != "novox/mesh-catalog" {
t.Fatalf("a hand registration erased the provenance: %+v %v", src, err)
}
}
+9 -2
View File
@@ -44,8 +44,15 @@ func TestAPersonMayCallToolsAndNothingElse(t *testing.T) {
} }
// The one tool, both ways it is addressed (novox/hq ADR 0159): to whichever instance // The one tool, both ways it is addressed (novox/hq ADR 0159): to whichever instance
// answers, and to the instance on one machine. Nothing else. // answers, and to the instance on one machine. Nothing else.
if len(perms.Publish) != 2 || perms.Publish[0] != "mesh.mod.mesh-catalog.tool.catalog_tools" || // And asking what answers (novox/hq ADR 0197), which claims nothing and calls nothing.
perms.Publish[1] != "mesh.mod.mesh-catalog.tool.catalog_tools.*" { var tools []string
for _, s := range perms.Publish {
if !strings.HasPrefix(s, "$SRV.") {
tools = append(tools, s)
}
}
if len(tools) != 2 || tools[0] != "mesh.mod.mesh-catalog.tool.catalog_tools" ||
tools[1] != "mesh.mod.mesh-catalog.tool.catalog_tools.*" {
t.Errorf("ada may publish %v, which should be the one tool, both ways addressed, and nothing else", perms.Publish) t.Errorf("ada may publish %v, which should be the one tool, both ways addressed, and nothing else", perms.Publish)
} }
for _, s := range perms.Publish { for _, s := range perms.Publish {
+29
View File
@@ -2,6 +2,9 @@ package link
import ( import (
"encoding/json" "encoding/json"
"strconv"
"strings"
"time"
) )
// Asking a machine to build a module, and hearing what came out. // Asking a machine to build a module, and hearing what came out.
@@ -17,6 +20,32 @@ import (
// holds no opinion about what they contain, and a host that also built things would be a host // holds no opinion about what they contain, and a host that also built things would be a host
// with a container runtime requirement and a git dependency (novox/hq ADR 0005). // with a container runtime requirement and a git dependency (novox/hq ADR 0005).
// NewBuildID is the correlation for a build asked at that moment: `build-<unix nanoseconds>`.
//
// **The id carries when the build was asked, and that is read back** (novox/hq 04-ISSUES/219). Builds
// of one module can be in flight together and finish in any order; what a module currently is must
// be the newest *request's* outcome, not the last one heard, and the id is the one thing every
// outcome echoes whichever builder answered it. One place writes the shape and one reads it.
func NewBuildID(asked time.Time) string {
return "build-" + strconv.FormatInt(asked.UnixNano(), 10)
}
// BuildAskedAt is when the build with this id was asked, as NewBuildID wrote it. False for an id
// of any other shape — one written before this was read, or by hand — whose request time the mesh
// does not know.
func BuildAskedAt(id string) (time.Time, bool) {
digits, ok := strings.CutPrefix(id, "build-")
if !ok || digits == "" {
return time.Time{}, false
}
nanos, err := strconv.ParseInt(digits, 10, 64)
// A number too small to be a moment this mesh could have asked at is a name, not a time.
if err != nil || nanos < time.Date(2020, 1, 1, 0, 0, 0, 0, time.UTC).UnixNano() {
return time.Time{}, false
}
return time.Unix(0, nanos).UTC(), true
}
// BuildRequest is one module to build. // BuildRequest is one module to build.
type BuildRequest struct { type BuildRequest struct {
// ID correlates the answer with the asking. Not the module name: two builds of one module can // ID correlates the answer with the asking. Not the module name: two builds of one module can