Compare commits

...
Author SHA1 Message Date
jschoubben 75b2676d32 The derived-value record is ADR 0202 (was 0201)
The key-value-buckets record took 0201 on main while this waited to merge.
Only the comments citing the derived-value work move; this repository's other
0201 citations are the buckets record's own and stay.
2026-10-04 04:33:56 +02:00
jschoubben 1363a2fe27 while-stopped names the container as the machine knows it (hq ADR 0189)
A module names its own resources locally; a declaration names them under the
module. restart-on and reload-on are rewritten for exactly that reason and
while-stopped was not, so the store's step said it held "store" still while
the machine's container is "distribution.store".

The host refuses a declaration naming a container it does not have — whole.
So novox took nothing at all, on every push, from 04:15 until this. The
machine was never damaged: refusing whole is what kept it serving.

Both sides' tests passed throughout. The controller's read manifests, the
host's read hand-written declarations with bare ids, and nothing composed one
and judged the result. That test now exists.
2026-10-04 04:18:22 +02:00
mesh-admin 17b8f14fe1 Merge pull request 'A module's state on the bus: buckets from the catalogue, grants, membership (hq ADR 0201)' (#257) from feat/module-state-on-the-bus into main 2026-10-04 01:43:36 +00:00
mesh-admin 42c394acc2 Merge pull request 'Group 8: a served value may name its consumer (hq ADR 0201), and the store keeps what the records name (hq ADR 0189)' (#227) from feat/the-store-keeps-what-the-records-name into main 2026-10-04 01:39:39 +00:00
jschoubben b9ad7a2948 Review before merge: refuse a silent disagreement, and bound the sweep
Three things found reading this back, each of which would have been quiet.

A consumer that keeps several holders of one provision (ADR 0094) gets a
login per holder, and a provider derives from the login — so it would make a
resource per holder while the consumer is told one value for the requirement.
That is issue 124's own failure one case to the side: authenticate, then be
refused on every object. Refused now, naming both ends.

The sweep runs inside somebody's build and was unbounded. At most two hundred
artifacts and sixty seconds, stopping at the first refusal because a store
that refuses one refuses all; the rest is offered again next build.

The citation and migration renumbers are in the commit before this one.
2026-10-04 03:27:32 +02:00
jochen cde22ff627 module check names a read of state its owner does not keep, and says what each module keeps and reads (novox/hq ADR 0201) 2026-10-04 02:50:02 +02:00
jschoubben 79993fb498 Rebased onto main: ADR 0188 renumbered to 0201, migration 0055 to 0056
The bundles refactor took ADR 0188 on main, so this work's record is 0201 and
every comment citing it moves with it. Main also took migration 0055 (an
older build never replaces a newer), so the store's collected-artifacts table
is 0056 — a number two migrations share is a schema nobody can trust.

make check passes except TestTheResolverIsToldEveryMachineOnTheNetworkAndToldAgainWhenOneLeaves,
which fails on main too and now for two stacked reasons (hq issues 203 and 202).
2026-10-04 02:45:06 +02:00
jochen aec55b7072 A module's state on the bus: buckets from the catalogue, grants, membership (novox/hq ADR 0201)
A manifest names the state it keeps (state) and reads (reads); the controller
asserts a key-value bucket per name on every raise, grants owners write and
readers read (measured against a running server), issues each assignment its
buckets in the membership, and reports buckets nothing declares without
removing them.
2026-10-04 02:40:49 +02:00
jschoubben c7884f5a72 The store keeps what the records name (hq ADR 0189)
The mesh names what may go from its own build records — a digest it did not
record making is never named, which is what keeps the sweep away from the
images genesis pushed. An artifact stays because a definition the mesh holds
names it, or because it belongs to one of the five most recent successful
builds of its module.

internal/artifacts asks the store to let go of one; internal/inventory
decides and remembers (migration 0055); the sweep runs after a build the mesh
recorded, which is when both the bytes and the keep set moved. Never fatal to
a build.

And the manifest side of while-stopped, refused from the definition alone:
no schedule, run-once, a container the module does not declare, itself.
2026-10-04 02:32:19 +02:00
jschoubben 580c4d66a7 A served value may name the consumer it is served to (hq ADR 0188)
${consumer:as} and ${consumer:as:dns} in a serves block are filled per
consumer at resolution, and the one filled value reaches both ends: the
consumer's binding and its ${bound:...} substitutions, and the provider's
contributions entry as `derived`. A fact or alphabet the mesh does not have
is refused at parse; a consumer whose own file already holds the derived
value is refused at resolution, naming the placeholder to write instead.
2026-10-04 02:32:19 +02:00
mesh-admin 63bfc5fda5 Merge pull request 'Refuse the tools-container shape for every module (to-be 38 WP4b's last step)' (#256) from feat/wp4b-the-gate-refuses-the-container-shape-for-all into main 2026-10-04 00:28:59 +00:00
jochen 02e3482eb5 Refuse the tools-container shape for every module (to-be 38 WP4b's last step)
While some thirty modules still stood in that shape, one already registered so was rebuilt without
complaint. Every module has moved since; the exception would only let one move back.
2026-10-04 02:28:54 +02:00
mesh-admin 294e83dab1 Merge pull request 'Run the controller as a Go bundle the host starts as a process (hq issue 213, 2 of 2)' (#253) from fix/issue-213-the-controller-is-a-process-manifest into main 2026-10-04 00:06:44 +00:00
jochen b5438bb331 Pin the image's Go base in the Dockerfile, which genesis builds as it stands (hq issue 223)
Genesis now raises a process-form controller as a container built from
this repository's Dockerfile with no build arguments (mesh-host
bootstrap, novox/hq issue 223); the manifest builds no image, so nothing
passes the base in. The default was a tag older than go.mod asks for.
It is now the digest the Makefile pins, and a test holds the two equal.
2026-10-04 01:49:05 +02:00
jochen c23be73d4d Run the controller as a Go bundle the host starts as a process (hq issue 213)
The controller is a Go program and was the one piece of the mesh's own Go
code still shipped and run as an image (novox/hq issue 213; ADR 0188 §1:
a module's own code is bundles; §3: a service bundle is a process).

The manifest now builds one Go bundle, `controller`, and runs it as the
process `mesh-controller` (`./mesh-controller serve`) under an account
the module declares. What the container gave it, replaced:

- host network: a process is on the host's network; nothing it reads
  names a container network
- user 65534: the account `mesh-controller`, which owns its secrets and
  its state directory
- the eight mounts: the env names the host paths the mesh already places
  (the store, broker and bus files under the state directory, the
  broker's certificate under /var/lib/mesh-broker-tls); the `broker`
  mount was read by nothing and is gone with the others
- `container-runtime` is no longer required on its machine

Its preparation is the same binary with `prepare`, as a run-once process,
and the process `replaces` the container `server`: the host keeps the
container answering until the process is running (mesh-host). Needs the
previous commit live in the running controller, and the host's
`replaces` on the controller's machine, before it is registered.

No image is built by the mesh any more. The Dockerfile stays for genesis
and the lab (`make image`, its Go base now pinned in the Makefile).
2026-10-04 01:45:25 +02:00
mesh-admin d2d171f2d2 Merge pull request 'Compose a module's Go service as a process the host runs (hq issue 213, 1 of 2)' (#252) from fix/issue-213-the-controller-is-a-process into main 2026-10-03 23:40:38 +00:00
mesh-admin 73fa64ea68 Merge pull request 'A TypeScript bundle installs its module's own packages before it is compiled (hq ADR 0198 §4)' (#255) from feat/a-bundle-installs-its-own-packages into main 2026-10-03 23:20:57 +00:00
jochen 1a13dbeb17 A TypeScript bundle installs its module's own packages before it is compiled
A bundle could import only what the toolchain image carried: the compiler and the bundler resolve an import from the module's directory and then the toolchain's node_modules, and nothing ever put anything in the first. So a module needing a database driver (pg, mongodb, mssql) could not be a bundle, and kept a container whose recipe installed it (hq ADR 0198 §4: the backend's own driver inside the bundle).

Now, when a module's package.json depends on anything beyond the SDK, the build installs its production dependencies into the module's directory, in the toolchain image, before the compile: npm ci from the lockfile when there is one, npm install from the ranges otherwise, the mesh's registry for the SDK's scope and the public one for the rest, install scripts off. esbuild then inlines them. A module depending only on the SDK runs exactly the commands it did before.

The SDK stays the toolchain's (hq issue 212): it is taken out of what is installed and any copy something pulls in is removed, so every import of it resolves past the module's node_modules to the one the toolchain carries; a module's own range never shadows it. npm's verified download cache is a named volume; nothing installed is kept between builds. Without a registry, a scoped package is refused rather than resolved on the public registry.
2026-10-04 01:17:49 +02:00
jochen e11caecdad Let two controllers overlap safely while one hands over to the other (hq issue 213)
The controller's machine moves it from the container to a process by
starting the process first and removing the container once the process
is up (mesh-host's `replaces`). For that moment two controllers share the
store and the bus. Checked what each does:

- the seat's verbs: a queue group per seat, each call answered once. Safe.
- the controller's consumers on CONTROL and EVENTS: push consumers with
  no delivery group, so the second bind is refused with "consumer is
  already bound" and serve exited. The process would restart for ever,
  the host would never see it up, and the container would never go. The
  second controller now stands by and binds when the first lets go
  (tested on a real bus; fails without the change).
- plans: read, changed and saved whole by the 30s timer, by build
  outcomes, by a merge and by `plans stop`. Two timers would each ask a
  tier the other had just asked. Working the plans now takes a
  session-level advisory lock on the inventory: the timer skips while
  another holds it, the other paths wait for it. Build asks happen only
  inside plan work and are covered by the same lock.
2026-10-04 01:11:26 +02:00
jochen 7bb9e55d0b Compose a module's Go service as a process the host runs (hq issue 213)
The controller is to be declared as a Go bundle run by a process instead of
an image (novox/hq issue 213, ADR 0188 §1, §3). The composer could not
express that honestly yet:

- a module declaring tools had every bundle served by the node's runtime,
  so the controller's own binary would have been launched a second time as
  an MCP child; a bundle one of the module's resources runs is now served
  only when it says `loads`
- a module's accounts went after the mesh-computed files, so secrets owned
  by the account a process runs as were refused on the first apply; a
  module's `user` resources now go first
- `prepares` derived its step only from a container; a process is now
  prepared by the same program with `prepare` as a run-once process
- a process may say what it `replaces` (a resource of its module it no
  longer declares), prefixed as the host records it, so the host keeps the
  old one running until the process is (needs mesh-host's `replaces`)

This lands before the controller's manifest uses any of it: the running
controller composes its own declaration, so the code that fills the new
shape must be live first.
2026-10-04 01:11:26 +02:00
mesh-admin 00037608ae Merge pull request 'The forge's tests compose its code as a bundle the node's runtime serves (hq ADR 0198, to-be 38 WP4c)' (#254) from feat/0198-waves-2-3-the-forges-code-is-a-bundle into main 2026-10-03 23:01:17 +00:00
jochen cea59428b1 The forge's tests compose its code as a bundle the node's runtime serves (hq ADR 0198)
gitea's own code moves out of its runtime container (mesh-catalog, to-be 38 WP4c waves 2-3), so the three tests that composed the forge from the catalogue beside this checkout resolve its build as the code bundle, compose it beside the node's runtime, and read the forge's address from the words the runtime hands the module rather than from a sidecar's env.
2026-10-04 00:50:06 +02:00
mesh-admin 5c832f2d19 Merge pull request 'Compose a process's environment as a container's' (#250) from feat/a-process-env-is-composed-like-a-containers into main 2026-10-03 22:29:11 +00:00
jochen cdebb7d1a5 Compose a process's environment as a container's
A module's own code moving out of its container (novox/hq to-be 38 WP4c)
becomes a process on the machine, and still has to be told what its
container was: the port this machine gave the module and where the
foundation's seats are. ${port:…} and ${seat:…} were filled only in a
file's content and a container's env, so in a process's env they reached
the machine as literals, and the modules that moved first (mesh-catalog
#245) wrote their run-once steps a 0600 env file instead. A process's env
now takes the same resolution and the same refusals; ${dir:…} and
${access:…} already did, and a bundle's env (ADR 0192) already resolves
${dir:…} and ${port:…}.
2026-10-04 00:27:42 +02:00
mesh-admin 6a803ea5b3 Merge pull request 'Issue 219: an older build request never replaces a newer one's artifact' (#249) from fix/issue-219-an-older-build-never-replaces-a-newer into main 2026-10-03 22:23:31 +00:00
jochen 9745c1ab31 An older build request never replaces a newer one's artifact
Builds of one module in flight together finish in any order, and the mesh
took whatever it heard last as what the module is: RegisterModule overwrote
the module's manifest unconditionally, and Held/BuiltAgainst/ReadRepositories
ordered builds by when they were recorded. A postgres build asked before the
mesh-tools runtime fix finished after the one asked after it, and the next
push deployed the stale image (novox/hq issue 219).

A build is now ordered by when it was asked, read from the build-<nanos> id
the controller writes: build.asked and module.built_asked (migration 0055).
A registration from an earlier request than the module's current one is
recorded and refused as superseded. A plan takes as its outcome only a build
asked at or after its own ask, so an earlier plan's leftover build cannot
settle a later plan. Ids of any other shape keep the old order.
2026-10-04 00:22:11 +02:00
mesh-admin c0c3c3fed4 Merge pull request 'A TypeScript bundle is one file per entrypoint, bundled in the toolchain (hq ADR 0193); a toolchain follows the SDK it stands on (hq issue 212)' (#247) from feat/a-typescript-bundle-is-one-file into main 2026-10-03 21:50:44 +00:00
mesh-admin c1449fffe9 Merge pull request 'Issue only the recorded holder a seat held once for the mesh (hq issue 218)' (#248) from fix/issue-218-only-the-holder-serves-a-mesh-seat into main 2026-10-03 21:30:50 +00:00
jochen f873c97db5 Issue only the recorded holder a seat held once for the mesh (novox/hq issue 218)
A module claiming a mesh-scoped seat was granted and issued the seat's subjects on every machine
it runs on, so the store's verbs answered from whichever postgres replied first. Where the mesh
records the seat's holder, only that (node, module) is now issued it; the module's own tools are
untouched everywhere.
2026-10-03 23:30:27 +02:00
jochen b0b3d87fe2 Run the toolchain's esbuild as itself: npm installs its native binary in place of the script 2026-10-03 23:26:08 +02:00
jochen ba189e6943 A TypeScript bundle is one file per entrypoint, bundled in the toolchain (hq ADR 0193); a toolchain follows the SDK it stands on (hq issue 212)
Every served bundle is its own process now, so each carries its own copy of what it imports: after
the compile and the launchers, the toolchain image's esbuild bundles every entrypoint in place and
every launcher under its own name into one ES module file, the SDK inlined, require provided to
inlined CommonJS, the launcher's shebang kept and its mode 0755. The toolchain's node_modules is
copied only for packages an artifact names external. An image without the bundler is refused by
name. Issue 212: build.on already passes a published package by its exact version and plans the
toolchain after it; tests say so.
2026-10-03 23:24:08 +02:00
64 changed files with 4289 additions and 197 deletions
+8 -2
View File
@@ -1,5 +1,11 @@
ARG GO_BASE=golang:1.25-alpine
# The control plane's image.
# The Go it builds with, pinned here because genesis builds this file with no arguments (novox/hq
# issue 223) — the Makefile passes the same digest. A tag older than go.mod asks for is how
# `make image` broke once before (issue 146).
ARG GO_BASE=golang@sha256:8ac98ca534ac3f51e1f420a1dd2c15e74c75cfa0f23f3ad27eb5d7236c349a0c
# The control plane's image — for genesis and the lab only. The mesh runs the controller as a Go
# bundle the host starts as a process (module.json; novox/hq issue 213), and builds no image of it.
# Genesis builds this file and raises it as the container the process replaces on the first push
# (mesh-host internal/bootstrap, novox/hq issue 223).
#
# novox/hq ADR 0006: this image is pinned by digest in the bundle the host carries, fetched on a
# machine where no mesh exists yet, and run before there is anything to check it against. So it
+8 -4
View File
@@ -27,17 +27,21 @@ build:
IMAGE ?= mesh-controller:$(VERSION)
DEV_TAG ?= mesh-controller:development
# The base the module declares, read from the manifest rather than written here twice.
# The Go base the image is built on.
#
# **`make image` was broken and stayed broken**, because the Dockerfile's fallback base was a Go
# older than go.mod asks for: every build died at `go mod download` with "go.mod requires go >=
# 1.26.0", and the pipeline never saw it because the pipeline passes the declared base in. Anybody
# building the image by hand hit it and had to find the digest themselves (novox/hq 04-ISSUES/146,
# what it cost).
GO_BASE ?= $(shell python3 -c "import json;print(next(o['image'] for o in json.load(open('module.json'))['build']['on'] if o['arg']=='GO_BASE'))" 2>/dev/null)
#
# **Pinned here since the manifest stopped building an image** (novox/hq issue 213): the mesh builds
# the controller as a Go bundle with its own toolchain, and only `make image` — genesis and the lab —
# still needs a Go base. The digest is the one the manifest declared until then.
GO_BASE ?= golang@sha256:8ac98ca534ac3f51e1f420a1dd2c15e74c75cfa0f23f3ad27eb5d7236c349a0c
image:
@test -n "$(GO_BASE)" || { echo "module.json declares no GO_BASE; pass GO_BASE=<image> or fix the manifest"; exit 1; }
@test -n "$(GO_BASE)" || { echo "no GO_BASE; pass GO_BASE=<image>"; exit 1; }
docker build --build-arg GO_BASE=$(GO_BASE) --build-arg VERSION=$(VERSION) -t $(IMAGE) -t $(DEV_TAG) .
@echo
@docker image inspect $(IMAGE) --format 'built {{.RepoTags}} {{.Size}} bytes'
@@ -48,7 +52,7 @@ BUILDER_IMAGE ?= mesh-builder:$(VERSION)
BUILDER_DEV_TAG ?= mesh-builder:development
builder-image:
@test -n "$(GO_BASE)" || { echo "module.json declares no GO_BASE; pass GO_BASE=<image> or fix the manifest"; exit 1; }
@test -n "$(GO_BASE)" || { echo "no GO_BASE; pass GO_BASE=<image>"; exit 1; }
docker build --build-arg GO_BASE=$(GO_BASE) -f cmd/mesh-builder/Dockerfile -t $(BUILDER_IMAGE) -t $(BUILDER_DEV_TAG) .
@echo
@docker image inspect $(BUILDER_IMAGE) --format 'built {{.RepoTags}} {{.Size}} bytes'
+7
View File
@@ -193,6 +193,13 @@ passes every check that only looks at the message.
## The image
**The mesh no longer runs the controller from it** (novox/hq issue 213). The module declares a Go
bundle, `controller`, which the host on the controller's machine unpacks and runs as the process
`mesh-controller` under the account of the same name (ADR 0188 §1, §3). The image stays for what
still runs a container of the controller: genesis, which raises the first controller from it and
installs the module from its manifest (mesh-host `internal/bootstrap`), and the lab. Neither is the
mesh's own build any more — `make image` builds it.
`FROM scratch`, holding one statically linked binary and nothing else — no shell, no package
manager, no libc, no CA certificates.
+6 -6
View File
@@ -145,7 +145,7 @@ func TestTheRegistryTrustAndEveryImageFollowThePortTheNodeGaveTheStore(t *testin
//
// Composed from the control plane's own manifest against a real inventory: the store's module is
// given 6852 on this node the way genesis or an operator gives it, and the control plane's
// container is told so beside the sealed connection genesis wrote.
// process is told so beside the sealed connection genesis wrote.
func TestTheControlPlaneIsToldWhereTheNodePutTheStoreAndTheBroker(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
@@ -157,8 +157,8 @@ func TestTheControlPlaneIsToldWhereTheNodePutTheStoreAndTheBroker(t *testing.T)
if err != nil {
t.Fatal(err)
}
control, err := withSeatPorts(m).Resolve([]catalogue.Built{{Name: "server", Kind: catalogue.ArtifactImage,
Reference: "registry.example/control@" + aDigest}})
control, err := withSeatPorts(m).Resolve([]catalogue.Built{{Name: "controller", Kind: catalogue.ArtifactBundle,
Reference: "https://registry.example/mesh-controller/controller.tar.gz", Digest: aDigest}})
if err != nil {
t.Fatal(err)
}
@@ -200,12 +200,12 @@ func TestTheControlPlaneIsToldWhereTheNodePutTheStoreAndTheBroker(t *testing.T)
var env map[string]any
for _, r := range composed(t, open, "anchor").Resources {
if r["id"] == "mesh-controller.server" {
if r["id"] == "mesh-controller.controller" {
env, _ = r["env"].(map[string]any)
}
}
if env == nil {
t.Fatal("the control plane's container is not in its own node's declaration")
t.Fatal("the control plane's process is not in its own node's declaration")
}
for key, want := range map[string]string{
"MESH_STORE_INVENTORY_PORT": "6852",
@@ -238,7 +238,7 @@ func withSeatPorts(m catalogue.Manifest) catalogue.Manifest {
out := m
out.Resources = nil
for _, r := range m.Resources {
if r["type"] != "container" {
if r["type"] != "container" && r["type"] != "process" {
out.Resources = append(out.Resources, r)
continue
}
+18 -2
View File
@@ -148,6 +148,11 @@ func buildFrom(result link.BuildResult) inventory.Build {
// rebuild the graph rather than a list of names.
Path: result.Path,
}
// When it was asked, which is what orders it against another build of the same module
// (novox/hq 04-ISSUES/219) — not when it was heard.
if asked, ok := link.BuildAskedAt(result.ID); ok {
kept.Asked = asked
}
for _, ref := range result.Against {
kept.Against = append(kept.Against, catalogue.Recorded(ref))
}
@@ -409,7 +414,7 @@ func buildOne(ctx context.Context, source buildSource, path, ref string, wait ti
// Correlated by something the control plane makes, not by the module's name: two builds of one
// module can be in flight, and the second answer is not the first one's.
request := link.BuildRequest{
ID: fmt.Sprintf("%s-%d", "build", time.Now().UnixNano()),
ID: link.NewBuildID(time.Now()),
Repository: repository,
Path: path,
Ref: ref,
@@ -526,6 +531,9 @@ func takeIn(ctx context.Context, inv *inventory.Inventory, result link.BuildResu
BuiltFrom: result.Commit, Head: result.Commit,
// What it stood on, so registration can judge a built manifest's base (to-be 38 WP2.4).
Against: kept.Against,
// When it was asked, so an older request heard later does not replace a newer one
// (novox/hq 04-ISSUES/219).
Asked: kept.Asked,
}
if result.Source != nil && result.Source.Seat != "" {
recorded.Repository, recorded.Seat = result.Source.Repository, result.Source.Seat
@@ -544,8 +552,16 @@ func takeIn(ctx context.Context, inv *inventory.Inventory, result link.BuildResu
result.On, result.Repository, short(result.Commit), err)
}
if err := inv.RegisterModule(ctx, manifest, recorded); err != nil {
if errors.Is(err, inventory.ErrSuperseded) {
return manifest, kept, fmt.Errorf("%s built %s (%s), recorded and not registered: %w",
result.On, manifest.Module, short(result.Commit), err)
}
return manifest, kept, err
}
// The keep set just moved, and new bytes just landed (novox/hq ADR 0189). Asked here rather
// than on a timer of its own: this is the only moment either is true. Never fatal — the build
// worked and the module is registered.
collect(ctx, inv)
return manifest, kept, nil
}
@@ -573,7 +589,7 @@ func buildAndShow(ctx context.Context, source buildSource, path, ref string, wai
defer ask.Close()
result, err := ask.Submit(ctx, link.BuildRequest{
ID: fmt.Sprintf("%s-%d", "build", time.Now().UnixNano()),
ID: link.NewBuildID(time.Now()),
Repository: repository, Path: path, Ref: ref,
Held: heldBy(ctx), Seats: seatBases(ctx),
}, wait)
+50
View File
@@ -2,9 +2,12 @@ package main
import (
"encoding/json"
"errors"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
@@ -100,3 +103,50 @@ func TestABuildAtACommitKeepsTheBranchTheModuleFollows(t *testing.T) {
t.Errorf("a module first built at a commit follows %q, want the default branch", src.Ref)
}
}
// novox/hq 04-ISSUES/219: an older request heard after a newer one is recorded and not registered,
// so a push sends what the newer request built.
func TestAnOlderBuildHeardLaterDoesNotReplaceTheNewer(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
older := time.Date(2026, 10, 3, 21, 33, 45, 0, time.UTC)
newer := time.Date(2026, 10, 3, 21, 51, 57, 0, time.UTC)
result := func(asked time.Time, image string) link.BuildResult {
manifest, _ := json.Marshal(map[string]any{"module": "postgres", "version": image})
return link.BuildResult{ID: link.NewBuildID(asked), Repository: "http://forge.internal:20000/novox/mesh-catalog.git",
Path: "modules/postgres", Ref: "main", On: "anchor", Commit: "efff5415", Manifest: manifest,
Source: &link.SourceOnSeat{Seat: "git", Repository: "novox/mesh-catalog"}}
}
if _, _, err := takeIn(ctx, open.inventory, result(newer, "4bcd5f73")); err != nil {
t.Fatal(err)
}
_, _, err := takeIn(ctx, open.inventory, result(older, "0ab07fa9"))
if !errors.Is(err, inventory.ErrSuperseded) {
t.Fatalf("the older request's outcome was taken in as current: %v", err)
}
shelf, err := open.inventory.Catalogue(ctx)
if err != nil {
t.Fatal(err)
}
if got := shelf["postgres"].Version; got != "4bcd5f73" {
t.Errorf("postgres is %q; want the newer request's 4bcd5f73", got)
}
if builds, _ := open.inventory.Builds(ctx, "postgres", 5); len(builds) != 2 {
t.Errorf("the late build was not recorded: %v", builds)
}
}
func TestABuildIDSaysWhenItWasAsked(t *testing.T) {
at := time.Date(2026, 10, 3, 21, 51, 57, 392539762, time.UTC)
if got, ok := link.BuildAskedAt(link.NewBuildID(at)); !ok || !got.Equal(at) {
t.Errorf("read back %v %v; want %v", got, ok, at)
}
if got, ok := link.BuildAskedAt("build-1791064317392539762"); !ok || got.Format(time.TimeOnly) != "21:51:57" {
t.Errorf("the incident's id reads as %v %v", got, ok)
}
for _, id := range []string{"b-1", "build-2", "build-", "build-x", ""} {
if _, ok := link.BuildAskedAt(id); ok {
t.Errorf("%q read as a request time", id)
}
}
}
+12
View File
@@ -7,6 +7,7 @@ import (
"os"
"path/filepath"
"sort"
"strings"
"github.com/novox/mesh-controller/internal/catalogue"
)
@@ -90,6 +91,17 @@ func moduleCheck(paths []string, out io.Writer) error {
if len(m.Invokes) > 0 {
fmt.Fprintf(out, ", invokes %s", joinInvokes(m.Invokes))
}
// The state it keeps and reads (novox/hq ADR 0201), so a reviewer sees what lands on the bus.
if len(m.State) > 0 {
kept := make([]string, 0, len(m.State))
for _, s := range m.State {
kept = append(kept, s.Name)
}
fmt.Fprintf(out, ", keeps state %s", strings.Join(kept, ", "))
}
if len(m.Reads) > 0 {
fmt.Fprintf(out, ", reads %s", strings.Join(m.Reads, ", "))
}
fmt.Fprintln(out)
}
if failed > 0 {
+108
View File
@@ -0,0 +1,108 @@
package main
import (
"context"
"errors"
"fmt"
"os"
"time"
"github.com/novox/mesh-controller/internal/artifacts"
"github.com/novox/mesh-controller/internal/inventory"
)
// Letting the artifact store go of what the mesh no longer keeps (novox/hq ADR 0189, issue 108).
//
// **Run where the records change.** A build is the moment new bytes landed in the store and the
// moment the keep set moved, so it is the moment to say what may go — and it needs no timer of
// its own. Reclaiming the bytes is the store's own nightly step; this only decides.
//
// Never fatal to a build. The build succeeded, the module is registered, and a store that could
// not be reached is a thing to say rather than a reason to undo any of that. The next build asks
// again, and the references it could not collect are still uncollected, so nothing is lost by
// having failed.
// collect asks the store to let go of everything the mesh made and no longer keeps, and records
// what it let go of. Says what it did and what it could not; returns nothing, because nothing
// upstream should branch on it.
func collect(ctx context.Context, inv *inventory.Inventory) {
references, err := inv.ToCollect(ctx)
if err != nil {
fmt.Fprintf(os.Stderr, "could not work out what the artifact store may let go of: %v\n", err)
return
}
if len(references) == 0 {
return
}
shelf, err := inv.Catalogue(ctx)
if err != nil {
fmt.Fprintf(os.Stderr, "could not read the catalogue to find the artifact store: %v\n", err)
return
}
// As the mesh reaches it from the network. Empty means the store is not on the network — on a
// mesh being raised it is not yet, and there the store holds one build of anything and has
// nothing to collect.
address, err := artifactStoreAddress(ctx, inv, shelf, "")
if err != nil || address == "" {
if err != nil {
fmt.Fprintf(os.Stderr, "could not find the artifact store to collect from: %v\n", err)
}
return
}
// **Bounded, because this runs inside somebody's build.** The first sweep of a mesh that has
// never collected has the whole history to get through, and a person waiting on `build` should
// not pay for it. Two bounds, and what is left over is simply offered again next time —
// builds are frequent, and the point is that the store stops growing, not that it empties
// tonight.
within, stop := context.WithTimeout(ctx, sweepBudget)
defer stop()
store := artifacts.Store{Address: address}
var done []string
var left int
for i, reference := range references {
if i >= mostPerSweep || within.Err() != nil {
left = len(references) - i
break
}
err := store.LetGo(within, reference)
if err == nil || errors.Is(err, artifacts.Gone) {
// Gone is the outcome wanted, already true. Recorded so the next sweep does not ask
// again for ever.
done = append(done, reference)
continue
}
// **Stopped at the first refusal, not pushed through.** A store that refuses one refuses
// all of them — deletion disabled, the store down, the network gone — so going on would
// be a hundred identical failures and a hundred identical log lines in front of whoever
// was building something.
fmt.Fprintf(os.Stderr, "the artifact store kept %s, so nothing more was asked of it: %v\n",
reference, err)
left = len(references) - i
break
}
if len(done) > 0 {
// Recorded outside `within`: the deletions happened, and losing the record of them because
// the sweep ran out of budget would mean asking about them again for ever.
if err := inv.MarkCollected(ctx, done); err != nil {
fmt.Fprintf(os.Stderr, "the store let go of %d artifact(s) and the record of it did not keep: %v\n",
len(done), err)
return
}
fmt.Fprintf(os.Stderr, "the artifact store let go of %d artifact(s) the mesh no longer keeps\n",
len(done))
}
if left > 0 {
fmt.Fprintf(os.Stderr, "%d more to collect; the next build asks again\n", left)
}
}
// mostPerSweep is how many artifacts one sweep will ask about. Enough that a mesh building
// several times a day converges within days of this landing; small enough that no single build
// waits on the whole backlog.
const mostPerSweep = 200
// sweepBudget is the longest a sweep will keep a build waiting.
const sweepBudget = 60 * time.Second
+13 -3
View File
@@ -8,6 +8,7 @@ package main
import (
"context"
"errors"
"flag"
"fmt"
"os"
@@ -253,25 +254,34 @@ func parseAround(set *flag.FlagSet, args []string) ([]string, error) {
// became of a build nobody was watching.
func (b builds) Built(ctx context.Context, result link.BuildResult) error {
manifest, _, err := takeIn(ctx, b.inv, result)
// When it was asked, so a plan takes as its outcome only a build asked for it or after it
// (novox/hq 04-ISSUES/219). Zero when the id does not say.
asked, _ := link.BuildAskedAt(result.ID)
switch {
case err != nil && result.Failed != "":
fmt.Printf("%s: %v\n", result.ID, err)
if result.Module != "" {
planBuilt(ctx, b.open, result.Module, result.Commit, result.Failed)
planBuilt(ctx, b.open, result.Module, result.Commit, result.Failed, asked)
} else {
planFailedBuild(ctx, b.open, result)
}
return nil
case errors.Is(err, inventory.ErrSuperseded):
// Not a failure: the module is already at what a later request built. A plan that asked
// before that later request is answered by it; one that asked after it ignores this.
fmt.Printf("%s: %v\n", result.ID, err)
planBuilt(ctx, b.open, manifest.Module, result.Commit, "", asked)
return nil
case err != nil:
fmt.Printf("%s: heard and recorded, and not registered: %v\n", result.ID, err)
if manifest.Module != "" {
planBuilt(ctx, b.open, manifest.Module, result.Commit, err.Error())
planBuilt(ctx, b.open, manifest.Module, result.Commit, err.Error(), asked)
}
return nil
}
fmt.Printf("%s: %s %s registered, built on %s from %s\n",
result.ID, manifest.Module, manifest.Version, result.On, short(result.Commit))
saysWhenThePolicyActs(ctx, b.inv, manifest.Module)
planBuilt(ctx, b.open, manifest.Module, result.Commit, "")
planBuilt(ctx, b.open, manifest.Module, result.Commit, "", asked)
return nil
}
@@ -0,0 +1,47 @@
package main
import (
"testing"
"time"
"github.com/novox/mesh-controller/internal/inventory"
)
// novox/hq issue 213: for the moment a machine hands its controller over, the container and the
// process both run the plan timer on one store. Only the one holding the plans moves them; the other
// leaves them alone, and moves them once they are let go.
func TestAControllerLeavesThePlansToTheOneHoldingThem(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
now := time.Now().UTC()
// Every tier done: the next step is the plan's last, and needs nothing but the store.
plan := inventory.Plan{ID: "plan-213", Repository: "r", Commit: "abc", Created: now, Updated: now,
State: inventory.PlanRolling, Tier: 1, Tiers: [][]string{{"app"}},
Modules: map[string]*inventory.PlanModule{"app": {State: "built"}}}
if err := open.inventory.SavePlan(ctx, plan); err != nil {
t.Fatal(err)
}
// The other controller: its own connections to the same store, holding the plans.
other, err := inventory.Open(ctx)
if err != nil {
t.Fatal(err)
}
t.Cleanup(other.Close)
release, err := other.HoldPlans(ctx, false)
if err != nil {
t.Fatal(err)
}
t.Cleanup(release) // before the close above: a pool waits for a connection still held
advancePlans(ctx, open)
if p, err := open.inventory.PlanByID(ctx, "plan-213"); err != nil || !p.Open() {
t.Fatalf("a controller moved a plan another held: %+v %v", p, err)
}
release()
advancePlans(ctx, open)
if p, err := open.inventory.PlanByID(ctx, "plan-213"); err != nil || p.State != inventory.PlanDone {
t.Fatalf("the plan did not move once it was let go: %+v %v", p, err)
}
}
+17 -2
View File
@@ -898,6 +898,21 @@ func raiseTheBus(ctx context.Context, inv *inventory.Inventory, address string)
if err := broker.RaiseSeats(js, inventory.MeshSeats(), holders); err != nil {
return err
}
// Every module's state (novox/hq ADR 0201), from the catalogue: a bucket exists from
// registration, so a module reading one may watch it before its owner runs anywhere. One that
// nothing declares any more is said and kept — what it holds is data.
buckets, err := inv.DeclaredBuckets(ctx)
if err != nil {
return err
}
undeclared, err := broker.RaiseBuckets(js, buckets)
if err != nil {
return err
}
if len(undeclared) > 0 {
fmt.Printf("the bus holds state nothing declares any more, kept because it is data: %s — "+
"removing it is a person's act\n", strings.Join(undeclared, ", "))
}
// And how every module hears what it consumes. Derived from the same records the user list is
// composed from, so a module the mesh grants a consumer's subjects has that consumer waiting.
// Done on every raise, not only when a credential is issued: every module moved onto this bus
@@ -921,8 +936,8 @@ func raiseTheBus(ctx context.Context, inv *inventory.Inventory, address string)
}
hearing++
}
fmt.Printf("the bus at %s has its streams, %d machine(s) can hear a declaration, and %d module(s) "+
"can hear what they consume\n", broker.BareAddress(address), len(names), hearing)
fmt.Printf("the bus at %s has its streams, %d machine(s) can hear a declaration, %d module(s) "+
"can hear what they consume, and %d bucket(s) of state\n", broker.BareAddress(address), len(names), hearing, len(buckets))
return nil
}
+64 -3
View File
@@ -2,6 +2,7 @@ package main
import (
"context"
"errors"
"flag"
"fmt"
"sort"
@@ -288,8 +289,23 @@ func askTier(ctx context.Context, inv *inventory.Inventory, p *inventory.Plan) e
// planBuilt marks a module built (or failed) in every open plan whose current tier holds it, and
// advances what that completes. Called from the daemon's take-in of every outcome.
func planBuilt(ctx context.Context, open *stores, module, commit, failed string) {
//
// **Only a build asked at or after the plan's ask is its outcome** (novox/hq 04-ISSUES/219). Two
// plans a few minutes apart both ask for a module; the earlier plan's build, finishing late, is not
// the later plan's answer — it stood on the bases from before the later plan's merge, and taking it
// would send machines, and the next tier, what the later merge replaced. asked is zero when the
// build's request time is not known, and such an outcome is taken as before.
func planBuilt(ctx context.Context, open *stores, module, commit, failed string, asked time.Time) {
inv := open.inventory
// One controller works the plans at a time (novox/hq issue 213); an outcome waits its turn rather
// than write over what the holder is about to save. Not taken, it is still in the build records,
// which the holder settles the plan from (issue 214).
release, err := inv.HoldPlans(ctx, true)
if err != nil {
fmt.Printf("plans: %s's outcome is left to the build records: %v\n", module, err)
return
}
defer release()
plans, err := inv.OpenPlans(ctx)
if err != nil {
fmt.Printf("plans: cannot read them: %v\n", err)
@@ -315,6 +331,9 @@ func planBuilt(ctx context.Context, open *stores, module, commit, failed string)
state = &inventory.PlanModule{}
p.Modules[module] = state
}
if askedBefore(asked, state.AskedAt) {
continue
}
if failed != "" {
state.State = "failed"
state.Why = failed
@@ -333,13 +352,30 @@ func planBuilt(ctx context.Context, open *stores, module, commit, failed string)
fmt.Printf("%s: %s; the tiers after it are not asked\n", p.ID, p.Note)
}
}
advancePlans(ctx, open)
advanceHeld(ctx, open)
}
// advancePlans moves every open plan as far as the facts allow: a tier whose modules are all built
// and whose gates are applied gives way to the next; the last tier done is the plan done. Called
// after every outcome and on a timer, so a plan waiting on a machine's report moves when it comes.
//
// **One controller at a time** (novox/hq issue 213). A plan is read, changed and saved whole; two
// controllers — the old and the new while a machine hands its controller over — would each ask a
// tier the other had just asked. Taken without waiting: whoever holds the plans is moving them.
func advancePlans(ctx context.Context, open *stores) {
release, err := open.inventory.HoldPlans(ctx, false)
if err != nil {
if !errors.Is(err, inventory.ErrPlansBusy) {
fmt.Printf("plans: cannot hold them: %v\n", err)
}
return
}
defer release()
advanceHeld(ctx, open)
}
// advanceHeld is advancePlans for a caller already holding the plans.
func advanceHeld(ctx context.Context, open *stores) {
inv := open.inventory
plans, err := inv.OpenPlans(ctx)
if err != nil {
@@ -549,7 +585,8 @@ func planFailedBuild(ctx context.Context, open *stores, result link.BuildResult)
}
for _, e := range entries {
if repositoryMatches(e.Source.Repository, result.Repository) && e.Source.Path == result.Path {
planBuilt(ctx, open, e.Manifest.Module, result.Commit, result.Failed)
asked, _ := link.BuildAskedAt(result.ID)
planBuilt(ctx, open, e.Manifest.Module, result.Commit, result.Failed, asked)
return
}
}
@@ -666,6 +703,19 @@ func plansCommand(ctx context.Context, args []string) error {
}
p.State = inventory.PlanFailed
p.Note = "stopped by hand at tier " + fmt.Sprint(p.Tier)
release, err := inv.HoldPlans(ctx, true)
if err != nil {
return err
}
defer release()
if p, err = inv.PlanByID(ctx, positionals[1]); err != nil {
return err
}
if !p.Open() {
return fmt.Errorf("%s is already %s", p.ID, p.State)
}
p.State = inventory.PlanFailed
p.Note = "stopped by hand at tier " + fmt.Sprint(p.Tier)
if err := inv.SavePlan(ctx, p); err != nil {
return err
}
@@ -791,6 +841,11 @@ func settleFromRecords(p *inventory.Plan, tier []string, recorded map[string][]i
if b.At.Before(*s.AskedAt) {
break
}
// Recorded after the ask and asked before it: an earlier ask's late outcome, not this
// one's (novox/hq 04-ISSUES/219).
if askedBefore(b.Asked, s.AskedAt) {
continue
}
outcome = &b
}
if outcome == nil {
@@ -812,3 +867,9 @@ func settleFromRecords(p *inventory.Plan, tier []string, recorded map[string][]i
}
return changed
}
// askedBefore is whether a build asked at asked was asked before a plan asked for its module — and
// so is not that plan's outcome (novox/hq 04-ISSUES/219). False when either time is not known.
func askedBefore(asked time.Time, planAsked *time.Time) bool {
return !asked.IsZero() && planAsked != nil && asked.Before(*planAsked)
}
+18
View File
@@ -155,6 +155,24 @@ func TestAPlanSettlesAnAskedBuildFromTheRecords(t *testing.T) {
t.Errorf("an ask with no record after it was settled: %+v", s)
}
// novox/hq 04-ISSUES/219: a build recorded after the ask but asked before it — an earlier
// plan's late outcome — is not this ask's, built or failed.
r := inventory.Plan{ID: "plan-3", Tiers: [][]string{{"postgres"}},
Modules: map[string]*inventory.PlanModule{"postgres": {State: "asked", AskedAt: &asked}}}
late := map[string][]inventory.Build{"postgres": {
{ID: "build-old", Commit: "efff5415", Asked: asked.Add(-18 * time.Minute), At: asked.Add(12 * time.Minute)},
}}
if settleFromRecords(&r, r.Tiers[0], late) || r.Modules["postgres"].State != "asked" {
t.Errorf("an earlier ask's late outcome settled this ask: %+v", r.Modules["postgres"])
}
// Newest heard first: the earlier ask's late outcome, then this ask's own, heard before it.
late["postgres"] = append(late["postgres"], inventory.Build{ID: "build-mine", Commit: "4bcd5f73",
Asked: asked.Add(time.Second), At: asked.Add(5 * time.Minute)})
if !settleFromRecords(&r, r.Tiers[0], late) || r.Modules["postgres"].State != "built" ||
r.Modules["postgres"].Commit != "4bcd5f73" {
t.Errorf("this ask's own outcome, heard before the earlier ask's, did not settle it: %+v", r.Modules["postgres"])
}
// A failure recorded after the ask fails the plan, as hearing it would have.
q := inventory.Plan{ID: "plan-2", Tiers: [][]string{{"x"}},
Modules: map[string]*inventory.PlanModule{"x": {State: "asked", AskedAt: &asked}}}
+7
View File
@@ -315,6 +315,13 @@ func (f following) SourceMoved(ctx context.Context, m link.SourceMoved) error {
for _, e := range moved {
movedNames = append(movedNames, e.Manifest.Module)
}
// Written and its first tier asked as one act on the plans (novox/hq issue 213): a timer on
// another controller reading it between the two would ask the tier again.
release, err := inv.HoldPlans(ctx, true)
if err != nil {
return notNow(err)
}
defer release()
plan := planOfMerge(m, movedNames, edges)
if hasCycle(plan.Tiers, edges) {
fmt.Printf(" the last tier depends on itself: %s — built together, in no order\n",
+99
View File
@@ -0,0 +1,99 @@
// Package artifacts speaks to the mesh's artifact store over its own door.
//
// Only what the mesh needs that nothing else does: letting go of something it put there
// (novox/hq ADR 0189, issue 108). Pushing is the builder's, through the container runtime; reading
// is every machine's, through its runtime. This is the one operation that belongs to the thing
// holding the records, because it is the only one that is a decision rather than a transfer.
package artifacts
import (
"context"
"fmt"
"net/http"
"strings"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
)
// Store is the artifact store at an address, as this machine reaches it.
type Store struct {
// Address is `host:port` — the store as the caller reaches it now, composed and never
// recorded (novox/hq 04-ISSUES/102).
Address string
// HTTP is the client used; nil is a client with a modest timeout.
HTTP *http.Client
}
// Gone is the answer when the store does not hold it: the outcome wanted, already true.
var Gone = fmt.Errorf("the store does not hold it")
// LetGo asks the store to drop one artifact the mesh recorded making.
//
// Takes a reference as the mesh records it — `artifact-store://<module>/<artifact>@sha256:…` for
// an image, `…/blobs/sha256:…` for an archive — because that is the identity every record uses,
// and composes the address here at the moment of use.
//
// Returns Gone when the store answers that it does not have it. That is not a failure: the sweep
// wants the artifact absent, and it is. It is distinguished from success only so a caller can say
// which of the two happened.
func (s Store) LetGo(ctx context.Context, reference string) error {
path, kept := catalogue.InArtifactStore(reference)
if !kept {
// Nothing the mesh put in its own store. Refused rather than attempted: composing a
// delete for a reference of unknown shape is how a sweep reaches something that is not
// the mesh's.
return fmt.Errorf("%s is not a reference into the mesh's artifact store", reference)
}
if s.Address == "" {
return fmt.Errorf("this mesh has no artifact store on its network to ask about %s", reference)
}
repository, kind, digest, err := split(path)
if err != nil {
return err
}
url := "http://" + s.Address + "/v2/" + repository + "/" + kind + "/" + digest
request, err := http.NewRequestWithContext(ctx, http.MethodDelete, url, nil)
if err != nil {
return err
}
client := s.HTTP
if client == nil {
client = &http.Client{Timeout: 30 * time.Second}
}
response, err := client.Do(request)
if err != nil {
return err
}
defer response.Body.Close()
switch response.StatusCode {
case http.StatusAccepted, http.StatusOK, http.StatusNoContent:
return nil
case http.StatusNotFound:
return Gone
case http.StatusMethodNotAllowed:
// The registry was started without deletion enabled. Said plainly, because the remedy is
// a setting on the store's module and not anything about this artifact.
return fmt.Errorf(
"the artifact store refuses deletion: its server was started without it enabled "+
"(REGISTRY_STORAGE_DELETE_ENABLED), so nothing can be collected until the store "+
"module is applied again (novox/hq ADR 0189). Asking about %s", reference)
default:
return fmt.Errorf("the artifact store answered %s for %s", response.Status, reference)
}
}
// split reads a recorded path into the repository, which endpoint names the thing, and the digest.
//
// Two shapes, which are the two the mesh records: `<repository>@sha256:<hex>` is a manifest, and
// `<repository>/blobs/sha256:<hex>` is a blob.
func split(path string) (repository, kind, digest string, err error) {
if before, after, ok := strings.Cut(path, "@sha256:"); ok {
return before, "manifests", "sha256:" + after, nil
}
if before, after, ok := strings.Cut(path, "/blobs/sha256:"); ok {
return before, "blobs", "sha256:" + after, nil
}
return "", "", "", fmt.Errorf("%q names nothing the store holds by digest", path)
}
+94
View File
@@ -0,0 +1,94 @@
package artifacts
import (
"context"
"errors"
"net/http"
"net/http/httptest"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
)
// Asking the store to let go of what the mesh no longer keeps (novox/hq ADR 0189, issue 108).
//
// A fake store records what it was asked to delete, so what is asserted is the mesh's decision
// and the shape of the request — not the registry's behaviour, which is the registry's to test.
func fakeStore(t *testing.T, answer int) (Store, *[]string) {
t.Helper()
var asked []string
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodDelete {
t.Errorf("the store was asked %s %s; collecting is a delete", r.Method, r.URL.Path)
}
asked = append(asked, r.URL.Path)
w.WriteHeader(answer)
}))
t.Cleanup(server.Close)
return Store{Address: strings.TrimPrefix(server.URL, "http://")}, &asked
}
func TestAnImageAndAnArchiveAreAskedForAtTheirOwnEndpoints(t *testing.T) {
// The two shapes the mesh records: a manifest by digest, and a blob by digest. They are
// different endpoints, and asking at the wrong one answers 404 — which this would then
// record as collected, leaving the bytes on disk for ever while the record says otherwise.
store, asked := fakeStore(t, http.StatusAccepted)
ctx := context.Background()
image := catalogue.ArtifactStoreScheme + "web/app@sha256:abc123"
archive := catalogue.ArtifactStoreScheme + "web/config/blobs/sha256:def456"
if err := store.LetGo(ctx, image); err != nil {
t.Fatal(err)
}
if err := store.LetGo(ctx, archive); err != nil {
t.Fatal(err)
}
want := []string{"/v2/web/app/manifests/sha256:abc123", "/v2/web/config/blobs/sha256:def456"}
if len(*asked) != 2 || (*asked)[0] != want[0] || (*asked)[1] != want[1] {
t.Fatalf("the store was asked %v; want %v", *asked, want)
}
}
func TestAStoreThatDoesNotHaveItAnswersGone(t *testing.T) {
// The outcome wanted, already true. Told apart from success only so the sweep can say which
// happened; both are recorded, because retrying for ever is the thing to avoid.
store, _ := fakeStore(t, http.StatusNotFound)
err := store.LetGo(context.Background(), catalogue.ArtifactStoreScheme+"web/app@sha256:abc123")
if !errors.Is(err, Gone) {
t.Fatalf("a store that does not hold it answered %v, want Gone", err)
}
}
func TestAStoreWithDeletionOffSaysSoAndNamesTheRemedy(t *testing.T) {
// The registry answers 405 when it was started without deletion enabled. The remedy is a
// setting on the store's module, and saying "405" would send somebody to the wrong place.
store, _ := fakeStore(t, http.StatusMethodNotAllowed)
err := store.LetGo(context.Background(), catalogue.ArtifactStoreScheme+"web/app@sha256:abc123")
if err == nil {
t.Fatal("a store that refuses deletion was read as success")
}
if !strings.Contains(err.Error(), "REGISTRY_STORAGE_DELETE_ENABLED") {
t.Fatalf("the refusal does not name the remedy: %v", err)
}
}
func TestAReferenceThatIsNotTheMeshsOwnIsNeverAsked(t *testing.T) {
// The whole safety of the sweep is that it names only what the mesh recorded putting there.
// A reference of another shape — a vendor's image, a package version — is refused rather
// than composed into a delete somewhere that is not the mesh's store.
store, asked := fakeStore(t, http.StatusAccepted)
for _, reference := range []string{
"docker.io/library/registry@sha256:abc123",
"registry@sha256:abc123",
"1.4.2",
} {
if err := store.LetGo(context.Background(), reference); err == nil {
t.Errorf("%s was asked about; it is not a reference into the mesh's store", reference)
}
}
if len(*asked) != 0 {
t.Fatalf("the store was asked about %v", *asked)
}
}
+49
View File
@@ -1,6 +1,7 @@
package broker
import (
"context"
"crypto/sha256"
"crypto/tls"
"crypto/x509"
@@ -12,6 +13,7 @@ import (
"time"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
)
// The JetStream side of the controller: the one place the mesh's streams and consumers are
@@ -316,3 +318,50 @@ func retentionOf(r Retention) nats.RetentionPolicy {
return nats.LimitsPolicy
}
}
// EnsureBucket creates a module's bucket if it is absent and brings its options to match if it is
// present (novox/hq ADR 0201).
//
// **An update, never a delete and recreate**, for the reason a stream is updated: recreating
// discards what the bucket holds, and what a module's state holds is data. The mesh's caps are
// asserted with the owner's options, so a bucket made by hand converges to them.
func (j *JetStream) EnsureBucket(b Bucket) error {
history := b.History
if history == 0 {
history = 1
}
js, err := jetstream.New(j.conn)
if err != nil {
return err
}
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
if _, err := js.CreateOrUpdateKeyValue(ctx, jetstream.KeyValueConfig{
Bucket: b.Bucket(),
Description: b.Why(),
History: uint8(history),
TTL: time.Duration(b.TTLSeconds) * time.Second,
MaxValueSize: StateMaxValueBytes,
MaxBytes: StateMaxBytes,
Storage: jetstream.FileStorage,
}); err != nil {
return fmt.Errorf("asserting bucket %s: %w", b.Bucket(), err)
}
return nil
}
// BucketNames is every key-value bucket on the server, the mesh's and anybody else's.
func (j *JetStream) BucketNames() ([]string, error) {
js, err := jetstream.New(j.conn)
if err != nil {
return nil, err
}
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
lister := js.KeyValueStoreNames(ctx)
var out []string
for name := range lister.Name() {
out = append(out, name)
}
return out, lister.Error()
}
+6
View File
@@ -45,6 +45,11 @@ type Membership struct {
// module that must tell the mesh from the world, the route proxy serving an internal name, reads
// it here rather than keeping a definition of its own.
Mesh []string `json:"mesh,omitempty"`
// State is every bucket this module's code may reach, by the name it uses for each, and whether
// it may write it (novox/hq ADR 0201): the runtime answers a bundle's state verbs from this list
// and refuses, with the reason, what is not on it — the bus enforces only the union over every
// module on the machine.
State []StateIssued `json:"state,omitempty"`
}
// Served is one address a tool is answered on.
@@ -103,6 +108,7 @@ func MembershipFor(node string, d Declared, where Placements) Membership {
m.Seats = append(m.Seats, SeatServed{Seat: s.Name, Verb: verb, Subject: seatToolSubject(s, verb, node)})
}
}
m.State = stateIssuedFor(d)
if len(d.Invokes) > 0 {
m.Reaches = map[string][]string{}
for _, t := range d.Invokes {
+17
View File
@@ -103,6 +103,12 @@ type Principal struct {
// permission and nothing beside it.
Invokes []string
// State is the local names of the state this principal's module keeps, and Reads the state of
// others it reads as `<module>.<name>` (novox/hq ADR 0201): a bucket each, kept by the owner's
// instances and read by whoever declares it.
State []string
Reads []string
// PasswordHash is the bcrypt hash the mesh minted. The plaintext is sealed to the principal
// and never appears here: this file is written to a node's disk and read by a server, and a
// secret that can be read from a configuration file is a secret with a wider blast radius
@@ -421,6 +427,10 @@ func PermissionsFor(p Principal) (Permissions, error) {
}
}
// 5. Its state, and the state of others it reads (novox/hq ADR 0201): every one read and
// watched, its own written too.
pub = append(pub, stateGrants(p.Module, p.State, p.Reads)...)
case KindNodeTools:
// **One process serves what every module on the machine would have served for itself**
// (novox/hq ADR 0175). Each carried module's whole tool namespace — the same grant that
@@ -487,6 +497,13 @@ func PermissionsFor(p Principal) (Permissions, error) {
"$JS.API.CONSUMER.MSG.NEXT."+stream+"."+durable,
"$JS.ACK."+stream+"."+durable+".>")
}
// **And it keeps and reads state for the modules it carries** (novox/hq ADR 0201): the union
// of what each may do with a bucket — an owner's write, a reader's read. That one module's code
// does not write another's bucket through it is the runtime's to keep, from the membership
// each assignment is issued, as it keeps each module's events under that module's own name.
for _, d := range p.Carries {
pub = append(pub, stateGrants(d.Module, stateNames(d.State), d.Reads)...)
}
sub = unique(sub)
pub = unique(pub)
}
+169
View File
@@ -0,0 +1,169 @@
package broker
import (
"fmt"
"sort"
"strings"
)
// A module's state on the bus (novox/hq ADR 0201, design 32 §4, design 25 §3).
//
// A module names the state it keeps (`state`) and the state of others it reads (`reads`), and each
// is a key-value bucket: the server's own last-per-subject stream with direct reads, delete markers
// and watches, which is the state relationship the mesh already uses for declarations, opened to
// modules. The controller creates every bucket from the catalogue — from registration, like a
// seat's stream, so a reader may watch before the owner runs anywhere — and no module can.
//
// Pure, like everything else in this package that decides what the bus holds; jetstream.go is the
// part that asks a server.
// The mesh's caps on a bucket, the same for every module: a value is a piece of state, not a file,
// and a bucket that grew without bound would be one module filling the bus's disk for everyone.
const (
StateMaxValueBytes = 256 * 1024
StateMaxBytes = 64 * 1024 * 1024
)
// A Bucket is one module's declared state as the bus holds it.
type Bucket struct {
Module string
Name string
// History is how many values a key keeps; zero is one.
History int
// TTLSeconds is how long a value lives; zero is until replaced or deleted.
TTLSeconds int
}
// BucketName is the bucket a module's state lives in: the module and the local name joined by an
// underscore, which neither may contain, so two modules can never derive one bucket.
func BucketName(module, name string) string { return module + "_" + name }
// Bucket is this bucket's name on the bus.
func (b Bucket) Bucket() string { return BucketName(b.Module, b.Name) }
// Why is carried into the server's description of the bucket, so somebody reading the server's
// own state finds whose it is and why it is kept.
func (b Bucket) Why() string {
return fmt.Sprintf("%s's state %q (novox/hq ADR 0201): its current value per key, written by %s, "+
"read by whatever declares it reads it; kept when %s is unassigned, because it is data",
b.Module, b.Name, b.Module, b.Module)
}
// bucketOfRead is the bucket a read names, `<module>.<name>`, or false when it names none.
func bucketOfRead(read string) (string, bool) {
at := strings.LastIndex(read, ".")
if at <= 0 || at == len(read)-1 {
return "", false
}
module, name := read[:at], read[at+1:]
if !safeSubject.MatchString(module) || !safeSubject.MatchString(name) {
return "", false
}
return BucketName(module, name), true
}
// stateGrants is what a principal publishes to reach the state its modules keep and read: for every
// bucket, binding to it, reading a key directly, and an ordered consumer for listing and watching,
// created and deleted on the bucket's own stream, with its flow control answered; for a bucket an
// owner keeps, writing under the bucket's own subjects too.
//
// **Measured against a running server, 2026-10-04** (novox/hq research 024), and each one is there
// because leaving it out failed: without STREAM.INFO nothing binds; without DIRECT.GET nothing is
// read; without CONSUMER.CREATE no key is listed and nothing is watched; without CONSUMER.DELETE a
// watch cannot be stopped and lingers on the server. A write outside these is refused by the server
// — and reaches the writer as a timeout, not a refusal, which is why the runtime refuses first.
func stateGrants(module string, keeps []string, reads []string) []string {
var out []string
read := func(bucket string) {
stream := "KV_" + bucket
out = append(out,
"$JS.API.STREAM.INFO."+stream,
"$JS.API.DIRECT.GET."+stream+".>",
"$JS.API.CONSUMER.CREATE."+stream+".>",
"$JS.API.CONSUMER.DELETE."+stream+".>",
"$JS.FC."+stream+".>")
}
for _, name := range keeps {
if !safeSubject.MatchString(name) {
continue
}
bucket := BucketName(module, name)
read(bucket)
out = append(out, "$KV."+bucket+".>")
}
for _, r := range reads {
if bucket, ok := bucketOfRead(r); ok {
read(bucket)
}
}
return out
}
// StateIssued is one bucket an assignment may reach, by the name its module uses for it: its own
// state by the local name, another's as `<module>.<name>` (novox/hq ADR 0201).
type StateIssued struct {
Name string `json:"name"`
Bucket string `json:"bucket"`
Writes bool `json:"writes,omitempty"`
}
// stateIssuedFor is every bucket a module's code may reach, as its membership lists them.
func stateIssuedFor(d Declared) []StateIssued {
var out []StateIssued
for _, b := range d.State {
out = append(out, StateIssued{Name: b.Name, Bucket: BucketName(d.Module, b.Name), Writes: true})
}
for _, r := range d.Reads {
if bucket, ok := bucketOfRead(r); ok {
out = append(out, StateIssued{Name: r, Bucket: bucket})
}
}
return out
}
// stateNames is the local names of a module's own buckets.
func stateNames(buckets []Bucket) []string {
out := make([]string, 0, len(buckets))
for _, b := range buckets {
out = append(out, b.Name)
}
return out
}
// A BucketAsserter is the part of a JetStream connection bucket assertion needs.
type BucketAsserter interface {
// EnsureBucket creates the bucket if absent and brings its options to match if present, never
// discarding what it holds.
EnsureBucket(b Bucket) error
// BucketNames is every key-value bucket on the server.
BucketNames() ([]string, error)
}
// RaiseBuckets asserts every declared bucket and answers the buckets on the server that nothing
// declares any more.
//
// **Those are reported, never removed** (novox/hq ADR 0201, ADR 0030): what a module stored is
// data, and a manifest edited, a module renamed or a catalogue entry dropped is an ordinary day's
// work that must not take data with it. Removing one is a person's act.
func RaiseBuckets(a BucketAsserter, buckets []Bucket) (undeclared []string, err error) {
sorted := append([]Bucket(nil), buckets...)
sort.Slice(sorted, func(i, j int) bool { return sorted[i].Bucket() < sorted[j].Bucket() })
declared := map[string]bool{}
for _, b := range sorted {
if err := a.EnsureBucket(b); err != nil {
return nil, fmt.Errorf("asserting %s's state %q: %w", b.Module, b.Name, err)
}
declared[b.Bucket()] = true
}
names, err := a.BucketNames()
if err != nil {
return nil, fmt.Errorf("listing the bus's state: %w", err)
}
for _, n := range names {
if !declared[n] {
undeclared = append(undeclared, n)
}
}
sort.Strings(undeclared)
return undeclared, nil
}
+166
View File
@@ -0,0 +1,166 @@
package broker
import (
"slices"
"strings"
"testing"
"github.com/nats-io/nats.go"
)
// The grants measured against a running server (novox/hq research 024): an owner reads and writes
// its bucket, a reader only reads, and neither reaches any other bucket.
func TestAnOwnerWritesItsStateAndAReaderOnlyReads(t *testing.T) {
owner, err := PermissionsFor(Principal{Kind: KindModule, Node: "one", Module: "claude-code",
State: []string{"servers"}, PasswordHash: "x"})
if err != nil {
t.Fatal(err)
}
for _, s := range []string{
"$KV.claude-code_servers.>",
"$JS.API.STREAM.INFO.KV_claude-code_servers",
"$JS.API.DIRECT.GET.KV_claude-code_servers.>",
"$JS.API.CONSUMER.CREATE.KV_claude-code_servers.>",
"$JS.API.CONSUMER.DELETE.KV_claude-code_servers.>",
"$JS.FC.KV_claude-code_servers.>",
} {
has(t, owner.Publish, s)
}
hasNot(t, owner.Publish, "$KV.>")
hasNot(t, owner.Publish, "$JS.API.>")
reader, err := PermissionsFor(Principal{Kind: KindModule, Node: "one", Module: "console",
Reads: []string{"claude-code.servers"}, PasswordHash: "x"})
if err != nil {
t.Fatal(err)
}
has(t, reader.Publish, "$JS.API.DIRECT.GET.KV_claude-code_servers.>")
has(t, reader.Publish, "$JS.API.CONSUMER.CREATE.KV_claude-code_servers.>")
hasNot(t, reader.Publish, "$KV.claude-code_servers.>")
for _, s := range reader.Subscribe {
if s == "$KV.claude-code_servers.>" {
t.Fatalf("a reader subscribes the bucket's subjects directly: %v", reader.Subscribe)
}
}
}
// One runtime carries every module on its machine, so its grant is the union: the owner's write
// where an owner is carried, a read where only a reader is.
func TestTheRuntimeKeepsAndReadsStateForItsModules(t *testing.T) {
perms, err := PermissionsFor(Principal{Kind: KindNodeTools, Node: "one", Module: RuntimeModule,
Carries: []Declared{
{Module: "claude-code", State: []Bucket{{Module: "claude-code", Name: "servers"}},
Reads: []string{"licence-manager.bindings"}},
{Module: "audit"},
}, PasswordHash: "x"})
if err != nil {
t.Fatal(err)
}
has(t, perms.Publish, "$KV.claude-code_servers.>")
has(t, perms.Publish, "$JS.API.DIRECT.GET.KV_licence-manager_bindings.>")
hasNot(t, perms.Publish, "$KV.licence-manager_bindings.>")
}
// A module with no state is granted nothing of any bucket — the composition of every module that
// existed before this is unchanged.
func TestAModuleWithNoStateReachesNoBucket(t *testing.T) {
perms, err := PermissionsFor(Principal{Kind: KindModule, Node: "one", Module: "billing",
Emits: []string{"order.placed"}, PasswordHash: "x"})
if err != nil {
t.Fatal(err)
}
for _, s := range perms.Publish {
if strings.HasPrefix(s, "$KV.") || strings.HasPrefix(s, "$JS.FC.") || strings.Contains(s, ".KV_") {
t.Fatalf("granted %q without declaring state", s)
}
}
}
// A read that names no bucket grants nothing rather than something that happens to parse.
func TestAReadThatNamesNoBucketGrantsNothing(t *testing.T) {
if got := stateGrants("a", nil, []string{"nodot", "x.", ".y", "a.b>"}); len(got) != 0 {
t.Fatalf("granted %v for reads that name no bucket", got)
}
}
// The membership lists every bucket the module's code may reach, by the name the module uses for
// it, and whether it may write it — the list the runtime refuses from.
func TestAMembershipListsTheStateItsModuleMayReach(t *testing.T) {
m := MembershipFor("one", Declared{Module: "claude-code",
State: []Bucket{{Module: "claude-code", Name: "servers"}},
Reads: []string{"licence-manager.bindings"}}, Placements{})
want := []StateIssued{
{Name: "servers", Bucket: "claude-code_servers", Writes: true},
{Name: "licence-manager.bindings", Bucket: "licence-manager_bindings"},
}
if !slices.Equal(m.State, want) {
t.Fatalf("issued %+v, want %+v", m.State, want)
}
if none := MembershipFor("one", Declared{Module: "audit"}, Placements{}); none.State != nil {
t.Fatalf("a module with no state was issued %+v", none.State)
}
}
type buckets struct {
ensured []string
on []string
}
func (b *buckets) EnsureBucket(x Bucket) error {
b.ensured = append(b.ensured, x.Bucket())
return nil
}
func (b *buckets) BucketNames() ([]string, error) { return b.on, nil }
// Every declared bucket is asserted; one on the server that nothing declares is said, not removed.
func TestRaisingStateReportsWhatNothingDeclares(t *testing.T) {
b := &buckets{on: []string{"claude-code_servers", "gone_old", "ours_by_hand"}}
undeclared, err := RaiseBuckets(b, []Bucket{{Module: "claude-code", Name: "servers"}, {Module: "a", Name: "b"}})
if err != nil {
t.Fatal(err)
}
if !slices.Equal(b.ensured, []string{"a_b", "claude-code_servers"}) {
t.Fatalf("asserted %v", b.ensured)
}
if !slices.Equal(undeclared, []string{"gone_old", "ours_by_hand"}) {
t.Fatalf("reported %v", undeclared)
}
}
// Against a real server: a bucket is created with the owner's options and the mesh's caps,
// asserting it again changes nothing and keeps what it holds, and a changed option is brought to
// match in place.
func TestABucketIsAssertedInPlace(t *testing.T) {
js := aLiveBus(t)
b := Bucket{Module: "statetest", Name: "servers"}
if _, err := RaiseBuckets(js, []Bucket{b}); err != nil {
t.Fatalf("a real server refused a module's bucket: %v", err)
}
kv, err := js.Context().KeyValue(b.Bucket())
if err != nil {
t.Fatal(err)
}
if _, err := kv.Put("all.one", []byte(`{"kept":true}`)); err != nil {
t.Fatal(err)
}
b.History = 3
if _, err := RaiseBuckets(js, []Bucket{b}); err != nil {
t.Fatalf("asserting the bucket again failed, so a restart would: %v", err)
}
got, err := kv.Get("all.one")
if err != nil || string(got.Value()) != `{"kept":true}` {
t.Fatalf("asserting again lost what the bucket held: %v %v", got, err)
}
status, err := kv.Status()
if err != nil {
t.Fatal(err)
}
if status.History() != 3 {
t.Fatalf("history is %d, the owner declared 3", status.History())
}
if s, ok := status.(*nats.KeyValueBucketStatus); ok {
if c := s.StreamInfo().Config; c.MaxMsgSize != StateMaxValueBytes || c.MaxBytes != StateMaxBytes {
t.Fatalf("the mesh's caps are not on the bucket: value %d, bucket %d", c.MaxMsgSize, c.MaxBytes)
}
}
}
+5
View File
@@ -33,6 +33,10 @@ type Declared struct {
Watches []Seat
// Invokes are the tools it calls, `<module>.<tool>` or `*` (novox/hq ADR 0152).
Invokes []string
// State is the state it keeps, each a bucket its instances write (novox/hq ADR 0201).
State []Bucket
// Reads are other modules' state it reads, each `<module>.<name>` (novox/hq ADR 0201).
Reads []string
}
// Records is what composing a user list needs to know about the mesh, and nothing more.
@@ -81,6 +85,7 @@ func Users(r Records) ([]Principal, error) {
Kind: KindModule, Node: node, Module: d.Module,
Emits: d.Emits, Consumes: d.Consumes, Serves: d.Serves,
Holds: d.Holds, Uses: d.Uses, Watches: d.Watches, Invokes: d.Invokes,
State: stateNames(d.State), Reads: d.Reads,
})
}
if runtimeHere {
+106 -3
View File
@@ -215,7 +215,7 @@ func Build(ctx context.Context, run Runner, publish Publisher,
sort.Slice(artifacts, func(i, j int) bool { return artifacts[i].Name < artifacts[j].Name })
for _, a := range artifacts {
say("artifact", "%s (%s%s) — starting", a.Name, a.Kind, langSuffix(a))
made, err := one(ctx, run, publish, manifest.Module, within, workspace, commit, credentials, a, args, held, npmrcPath, seatBases, say)
made, err := one(ctx, run, publish, manifest.Module, within, workspace, commit, credentials, a, args, held, npmrcPath, npmrc, seatBases, say)
if err != nil {
say("artifact", "%s FAILED: %v", a.Name, err)
return Result{}, err
@@ -443,7 +443,7 @@ func wantsPackages(manifest catalogue.Manifest, within string) bool {
func one(ctx context.Context, run Runner, publish Publisher,
module, tree, workspace, commit, credentials string, a catalogue.Artifact, args []string,
held map[string]string, npmrc string, seats map[string]string,
held map[string]string, npmrc string, registry Npmrc, seats map[string]string,
say func(step, format string, args ...any)) (catalogue.Built, error) {
switch a.Kind {
@@ -582,6 +582,11 @@ func one(ctx context.Context, run Runner, publish Publisher,
"holds no copy of it. Build %s first",
module, a.Name, chain.Language, chain.Base, chain.Artifact, chain.Base)
}
// The module's own packages first, where the compiler and the bundler resolve them from
// (dependencies.go); nothing at all for a module whose package.json names only the SDK.
if err := installOwn(ctx, run, tree, chain, base, registry, say); err != nil {
return catalogue.Built{}, fmt.Errorf("%s: %s: %w", module, a.Name, err)
}
say("bundle", "compiling %s in %s's toolchain", a.Language, chain.Base)
compiled, err := compile(ctx, run, tree, chain, base, a)
if err != nil {
@@ -593,6 +598,12 @@ func one(ctx context.Context, run Runner, publish Publisher,
if err != nil {
return catalogue.Built{}, fmt.Errorf("%s: writing %s's launchers failed: %w", module, a.Name, err)
}
if chain.Bundler != "" {
say("bundle", "bundling each entrypoint into one file")
if compiled, err = bundled(ctx, run, tree, chain, base, a, launchers); err != nil {
return catalogue.Built{}, fmt.Errorf("%s: bundling %s failed: %w", module, a.Name, err)
}
}
say("bundle", "compiled, packing")
body, err := pack(compiled)
if err != nil {
@@ -974,7 +985,7 @@ func compile(ctx context.Context, run Runner, tree string, chain Toolchain,
if _, err := run(ctx, tree, "docker", invocation...); err != nil {
return "", err
}
if chain.Dependencies != "" {
if chain.Dependencies != "" && chain.Bundler == "" {
// **What the bundle runs with, from the image it was compiled in** (Toolchain.Dependencies).
// A second run in the same image rather than a shell wrapped around the compiler: the
// compile line stays a plain command a reader can run by hand, and the copy is one more
@@ -1224,3 +1235,95 @@ func writeLaunchers(root string, chain Toolchain, a catalogue.Artifact) (map[str
}
return out, nil
}
// bundledSuffix is where a bundle's one-file output is written, beside what the compiler wrote.
const bundledSuffix = ".bundled"
// bundled makes every entrypoint and every launcher of a compiled bundle ONE file, in the toolchain
// image's bundler, and answers the directory to pack (novox/hq ADR 0193).
//
// **What a launched bundle runs is what it imports, and nothing else.** Every served bundle is its
// own process, so it carries its own copy of the SDK and its own dependencies inlined — the
// toolchain's whole node_modules no longer travels in every bundle. An entrypoint a process runs by
// name (`node daemon/index.js`) is bundled in place under its own name; a launcher keeps its name
// and its first line, and stays executable. A package the bundler cannot inline is named by the
// artifact (`external`), kept as an import, and only then is the toolchain's runtime directory
// copied beside the files. CommonJS inlined into an ES module still finds `require`.
func bundled(ctx context.Context, run Runner, tree string, chain Toolchain, base string,
a catalogue.Artifact, launchers map[string]string) (string, error) {
const within = "/app/modules/module"
out, final := Out(a.Name), Out(a.Name)+bundledSuffix
if err := os.RemoveAll(filepath.Join(tree, final)); err != nil {
return "", err
}
if err := os.MkdirAll(filepath.Join(tree, final), 0o755); err != nil {
return "", err
}
common := []string{"--bundle", "--platform=node", "--format=esm", "--target=node22",
"--outbase=" + out, "--outdir=" + final, "--log-level=warning",
"--banner:js=import { createRequire as __meshRequire } from 'node:module'; const require = __meshRequire(import.meta.url);"}
for _, x := range a.External {
common = append(common, "--external:"+x)
}
var plain []string
for _, e := range a.Entrypoints {
if strings.HasSuffix(e, ".js") {
plain = append(plain, out+"/"+e)
}
}
var launch []string
for _, l := range sortedValues(launchers) {
launch = append(launch, out+"/"+l)
}
// Refused by name in an image that predates the bundler, as the dependencies copy is: a bundle
// packed without it would carry nothing it imports. Run as itself: npm installs esbuild's native
// binary in place of its script, which `node` cannot run.
guard := `test -x "$0" || { echo "the toolchain image carries no bundler at $0: it predates one-file bundles, rebuild mesh-tools first" >&2; exit 1; }; exec "$0" "$@"`
step := func(entries []string, extra ...string) error {
if len(entries) == 0 {
return nil
}
invocation := []string{"run", "--rm", "--volume", tree + ":" + within, "--workdir", within, base,
"sh", "-c", guard, chain.Bundler}
invocation = append(invocation, entries...)
invocation = append(invocation, common...)
invocation = append(invocation, extra...)
_, err := run(ctx, tree, "docker", invocation...)
return err
}
if err := step(plain); err != nil {
return "", err
}
if err := step(launch, "--out-extension:.js=.mjs"); err != nil {
return "", err
}
// Plain `.js` output is an ES module; said once, as the runtime directory used to say it.
if err := os.WriteFile(filepath.Join(tree, final, "package.json"), []byte(`{"type":"module","private":true}`+"\n"), 0o644); err != nil {
return "", err
}
for _, l := range launchers {
path := filepath.Join(tree, final, filepath.FromSlash(l))
if _, err := os.Stat(path); err == nil {
if err := os.Chmod(path, 0o755); err != nil {
return "", err
}
}
}
if len(a.External) > 0 && chain.Dependencies != "" {
copying := []string{"run", "--rm", "--volume", tree + ":" + within, "--workdir", within, base,
"sh", "-c", `cp -a "$0/node_modules" "$1/"`, chain.Dependencies, final}
if _, err := run(ctx, tree, "docker", copying...); err != nil {
return "", fmt.Errorf("copying the packages %s keeps external: %w", a.Name, err)
}
}
return filepath.Join(tree, final), nil
}
func sortedValues(m map[string]string) []string {
out := make([]string, 0, len(m))
for _, v := range m {
out = append(out, v)
}
sort.Strings(out)
return out
}
+38 -14
View File
@@ -83,25 +83,49 @@ func TestABundleIsCompiledAndPackedWithNoDockerfile(t *testing.T) {
t.Fatalf("the bundle was not pinned: %v", got.Manifest.Resources[0])
}
// **And what it runs with, from the image it was compiled in** (novox/hq to-be 38 WP3). A
// second run in the same toolchain image copies the toolchain's runtime directory — the
// `"type": "module"` package.json and the pruned node_modules — into the output's root, and
// refuses by name when the image carries none rather than packing a bundle that starts nowhere.
var copied string
// **One file per entrypoint and launcher, in the toolchain's bundler** (novox/hq ADR 0193). A
// second run in the same toolchain image bundles each into the artifact's bundled output, the SDK
// inlined, refusing by name in an image that predates the bundler; and the toolchain's
// node_modules is no longer copied into a bundle that keeps nothing external.
var bundling []string
for _, line := range r.ran {
if strings.HasPrefix(line, "docker run") && strings.Contains(line, "/app/runtime") {
copied = line
if strings.HasPrefix(line, "docker run") && strings.Contains(line, "esbuild") {
bundling = append(bundling, line)
}
}
if copied == "" {
t.Fatalf("the bundle's dependencies were not copied in after the compile:\n%s", strings.Join(r.ran, "\n"))
if len(bundling) != 2 {
t.Fatalf("want one bundling run for the entrypoints and one for the launchers:\n%s", strings.Join(r.ran, "\n"))
}
if !strings.Contains(copied, "mesh-tools/build@sha256:") || !strings.Contains(copied, "predates") ||
!strings.Contains(copied, Out("code")) {
t.Fatalf("the copy does not run in the same toolchain, refuse an older image by name, or land in the artifact's output: %s", copied)
for _, want := range []string{"mesh-tools/build@sha256:", "predates one-file bundles", "--bundle", "--format=esm",
"--platform=node", "--outdir=" + Out("code") + ".bundled", Out("code") + "/index.js"} {
if !strings.Contains(bundling[0], want) {
t.Errorf("the entrypoints' bundling lacks %q: %s", want, bundling[0])
}
}
if strings.Index(strings.Join(r.ran, "\n"), "--outDir") > strings.Index(strings.Join(r.ran, "\n"), "/app/runtime") {
t.Fatal("the dependencies were copied before the compile wrote its output")
if !strings.Contains(bundling[1], Out("code")+"/index.serve.mjs") || !strings.Contains(bundling[1], "--out-extension:.js=.mjs") {
t.Errorf("the launcher is not bundled under its own name: %s", bundling[1])
}
if strings.Contains(strings.Join(r.ran, "\n"), "/app/runtime") {
t.Errorf("the toolchain's node_modules was copied into a bundle that keeps nothing external:\n%s", strings.Join(r.ran, "\n"))
}
if strings.Index(strings.Join(r.ran, "\n"), "--outDir") > strings.Index(strings.Join(r.ran, "\n"), "esbuild") {
t.Fatal("the bundler ran before the compile wrote its output")
}
}
// A bundle naming packages it keeps external is bundled with them as imports, and carries the
// toolchain's node_modules for them — the one case it still does.
func TestABundleKeepingAPackageExternalCarriesTheToolchainsModules(t *testing.T) {
manifest := strings.Replace(aBundle, `"entrypoints":["index.js"]`, `"entrypoints":["index.js"],"external":["sharp"]`, 1)
r, workspace := aRepository(t, manifest, map[string]string{"index.ts": "console.log(1)"})
held := map[string]string{"mesh-tools/build": "registry.invalid/mesh-tools/build@sha256:" + strings.Repeat("b", 64)}
if _, err := Build(context.Background(), compiling{r}.run, r,
"https://forge.invalid/greeter.git", "", "", workspace, held, Npmrc{}, GitCredential{}, nil); err != nil {
t.Fatal(err)
}
all := strings.Join(r.ran, "\n")
if !strings.Contains(all, "--external:sharp") || !strings.Contains(all, "/app/runtime") {
t.Errorf("an external package was not kept as an import with the toolchain's modules beside it:\n%s", all)
}
}
+147
View File
@@ -0,0 +1,147 @@
package builder
import (
"context"
"encoding/json"
"errors"
"fmt"
"os"
"path/filepath"
"sort"
"strings"
)
// A module's own packages, installed before its bundle is compiled, so the bundler inlines them.
//
// **A bundle could only import what the toolchain happened to carry.** The compiler and the bundler
// resolve an import by walking up from the module's source: the module's own directory first, then
// the toolchain image's node_modules. Nothing ever put anything in the first, so a module needing a
// database driver (`pg`, `mongodb`, `mssql`) could not be a bundle at all, and kept a container whose
// recipe installed it by hand (novox/hq ADR 0198 §4: "the backend's own driver inside the bundle").
// Now the module's `package.json` says what it depends on, as any Node package does, and the build
// installs exactly that into the module's own directory before compiling.
//
// **The SDK the toolchain carries is the one a bundle is built with, whatever the module says**
// (novox/hq issue 212: the toolchain is rebuilt on every SDK release and every bundle after it). A
// module's `package.json` names `@novox/mesh-sdk` with a range — it has to, to type-check on a
// workstation — and installing that range would shadow the toolchain's copy for this module alone:
// one module compiled against an older SDK than its neighbours, chosen by a caret nobody re-reads.
// So the SDK is taken out of what is installed (and never fetched), and any copy something else
// pulls in is removed afterwards; every import of it resolves past the module's node_modules to the
// toolchain's. A module therefore cannot pin a different SDK, by design: the toolchain is the pin.
//
// **Correctness before speed.** Every build installs afresh into a fresh clone, from the lockfile
// when the module has one (`npm ci`, exact) and from its ranges otherwise; nothing installed is kept
// between builds. What is shared is npm's own download cache, a named volume, which is
// content-addressed and verified by integrity on every read — it saves the network, never the
// install. Install scripts do not run: the build node runs nobody's postinstall, and what a script
// would build natively could not be inlined into one file anyway.
// sdkPackage is the package a TypeScript bundle's launcher serves through, and the one package a
// module's own dependencies never supply (above).
const sdkPackage = "@novox/mesh-sdk"
// npmCache is the named volume npm's download cache lives in across builds on one build node.
const npmCache = "mesh-builder-npm-cache"
// ownDependencies is what a module's package.json depends on beyond the SDK, sorted; nothing when
// the module has no package.json or depends on nothing else — which builds exactly as before.
func ownDependencies(tree string) ([]string, error) {
raw, err := os.ReadFile(filepath.Join(tree, "package.json"))
if errors.Is(err, os.ErrNotExist) {
return nil, nil
}
if err != nil {
return nil, err
}
var p struct {
Dependencies map[string]string `json:"dependencies"`
}
if err := json.Unmarshal(raw, &p); err != nil {
return nil, fmt.Errorf("the module's package.json is not JSON: %w", err)
}
var names []string
for name := range p.Dependencies {
if name != sdkPackage {
names = append(names, name)
}
}
sort.Strings(names)
return names, nil
}
// installSteps is the script run inside the toolchain image, from the module's own directory ($0).
// It works in a scratch copy so the module's package.json and lockfile are never rewritten, takes
// the SDK out of what is installed, installs production dependencies only, removes any copy of the
// SDK something pulled in, and puts the result at the module's node_modules.
const installSteps = `set -e
work="$(mktemp -d)"
cp "$0/package.json" "$work/"
if [ -f "$0/package-lock.json" ]; then cp "$0/package-lock.json" "$work/"; fi
cd "$work"
node -e '
const fs = require("fs"), sdk = process.argv[1];
const p = JSON.parse(fs.readFileSync("package.json", "utf8"));
for (const k of ["dependencies", "peerDependencies", "optionalDependencies"]) if (p[k]) delete p[k][sdk];
delete p.devDependencies; delete p.scripts;
fs.writeFileSync("package.json", JSON.stringify(p));
' "$1"
shift
if [ -f package-lock.json ]; then
npm ci --omit=dev --omit=peer --ignore-scripts --no-audit --no-fund "$@"
else
npm install --omit=dev --omit=peer --ignore-scripts --no-audit --no-fund --no-package-lock "$@"
fi
find node_modules -depth -type d -path "*/node_modules/@novox/mesh-sdk" -exec rm -rf {} +
rm -rf "$0/node_modules"
cp -a node_modules "$0/node_modules"
`
// installOwn installs a TypeScript module's own production dependencies into its directory, in the
// toolchain image, before the compile — or does nothing at all for a module that has none.
func installOwn(ctx context.Context, run Runner, tree string, chain Toolchain, base string,
registry Npmrc, say func(step, format string, args ...any)) error {
if chain.Language != "typescript" {
return nil
}
deps, err := ownDependencies(tree)
if err != nil || len(deps) == 0 {
return err
}
scoped := strings.TrimSpace(registry.Scope)
if !registry.Enabled() {
// **No registry, no scoped package.** Without the mesh's registry a scoped name resolves on
// the public one, where anybody may have published it: a dependency that installs is not
// the dependency the module meant.
for _, d := range deps {
if strings.HasPrefix(d, "@novox/") {
return fmt.Errorf("the module depends on %s, and this build knows no package registry "+
"for its scope; it would resolve from the public registry, which is not where the "+
"mesh publishes it", d)
}
}
}
const within = "/app/modules/module"
invocation := []string{"run", "--rm",
"--volume", tree + ":" + within,
"--volume", npmCache + ":/root/.npm",
"--workdir", within}
var flags []string
if registry.Enabled() {
// The registry is reached where the binding says it is, which may be this machine's own
// loopback — the reason an image build that resolves packages runs on the host network too.
invocation = append(invocation, "--network", "host")
reg := strings.TrimSpace(registry.Registry)
if !strings.HasSuffix(reg, "/") {
reg += "/"
}
flags = append(flags, "--"+scoped+":registry="+reg)
}
invocation = append(invocation, base, "sh", "-c", installSteps, within, sdkPackage)
invocation = append(invocation, flags...)
say("bundle", "installing the module's own packages: %s", strings.Join(deps, ", "))
if _, err := run(ctx, tree, "docker", invocation...); err != nil {
return fmt.Errorf("installing the module's own packages (%s): %w", strings.Join(deps, ", "), err)
}
return nil
}
+131
View File
@@ -0,0 +1,131 @@
package builder
import (
"context"
"strings"
"testing"
)
// A module's own packages (dependencies.go): installed into its own directory, in the toolchain,
// before the compile, so the bundler inlines them — the SDK always the toolchain's.
func buildWithPackageJSON(t *testing.T, pkg string, extra map[string]string, registry Npmrc) (*recorded, error) {
t.Helper()
files := map[string]string{"index.ts": "console.log(1)"}
if pkg != "" {
files["package.json"] = pkg
}
for k, v := range extra {
files[k] = v
}
r, workspace := aRepository(t, aBundle, files)
held := map[string]string{"mesh-tools/build": "registry.invalid/mesh-tools/build@sha256:" + strings.Repeat("b", 64)}
_, err := Build(context.Background(), compiling{r}.run, r,
"https://forge.invalid/greeter.git", "", "", workspace, held, registry, GitCredential{}, nil)
return r, err
}
func installs(r *recorded) []string {
var out []string
for _, line := range r.ran {
if strings.HasPrefix(line, "docker run") && strings.Contains(line, "npm ci") {
out = append(out, line)
}
}
return out
}
func compileIndex(r *recorded) int {
for i, line := range r.ran {
if strings.Contains(line, "--outDir") {
return i
}
}
return -1
}
func TestAModulesOwnPackagesAreInstalledInTheToolchainBeforeTheCompile(t *testing.T) {
r, err := buildWithPackageJSON(t, `{"type":"module","dependencies":{"@novox/mesh-sdk":"^0.1.0","pg":"^8"},"devDependencies":{"typescript":"^5"}}`,
nil, Npmrc{Scope: "@novox", Registry: "https://forge.invalid/api/packages/novox/npm"})
if err != nil {
t.Fatal(err)
}
got := installs(r)
if len(got) != 1 {
t.Fatalf("want one install of the module's own packages:\n%s", strings.Join(r.ran, "\n"))
}
line := got[0]
for _, want := range []string{
"mesh-tools/build@sha256:", // in the toolchain image
":/app/modules/module", // into the module's own directory
"--workdir /app/modules/module", //
npmCache + ":/root/.npm", // npm's verified download cache, and only that
"--omit=dev", "--ignore-scripts", // production packages, no build-node scripts
"npm ci", "npm install", "--no-package-lock", // the lockfile when there is one, else the ranges
"--@novox:registry=https://forge.invalid/api/packages/novox/npm/", // the scope from the mesh's registry
"--network host",
"@novox/mesh-sdk", // named, to be taken out of what is installed
} {
if !strings.Contains(line, want) {
t.Errorf("the install lacks %q:\n%s", want, line)
}
}
// The SDK is the toolchain's: never installed from the module's range, and any copy removed.
if !strings.Contains(line, `delete p[k][sdk]`) || !strings.Contains(line, `-path "*/node_modules/@novox/mesh-sdk" -exec rm -rf`) {
t.Errorf("the module's own SDK range could shadow the toolchain's SDK:\n%s", line)
}
if i, c := strings.Index(strings.Join(r.ran, "\n"), "npm ci"), compileIndex(r); c < 0 ||
i > strings.Index(strings.Join(r.ran, "\n"), "--outDir") {
t.Fatalf("the install did not run before the compile:\n%s", strings.Join(r.ran, "\n"))
}
}
// **A module with nothing beyond the SDK builds exactly as before**: the same commands, no install.
func TestAModuleDependingOnlyOnTheSDKBuildsExactlyAsBefore(t *testing.T) {
without, err := buildWithPackageJSON(t, "", nil, Npmrc{})
if err != nil {
t.Fatal(err)
}
for _, pkg := range []string{
`{"type":"module","dependencies":{"@novox/mesh-sdk":"^0.1.0"},"devDependencies":{"typescript":"^5"}}`,
`{"type":"module"}`,
} {
with, err := buildWithPackageJSON(t, pkg, map[string]string{"package-lock.json": "{}"}, Npmrc{Scope: "@novox", Registry: "https://forge.invalid/npm/"})
if err != nil {
t.Fatal(err)
}
if strings.Contains(strings.Join(with.ran, "\n"), "npm ") {
t.Fatalf("a module depending on nothing but the SDK ran npm:\n%s", strings.Join(with.ran, "\n"))
}
if len(with.ran) != len(without.ran) {
t.Fatalf("a module depending only on the SDK built differently from one with no package.json:\n%s\n---\n%s",
strings.Join(with.ran, "\n"), strings.Join(without.ran, "\n"))
}
}
}
// Without the mesh's registry a scoped package would resolve on the public one: refused by name.
func TestAScopedPackageWithNoRegistryIsRefused(t *testing.T) {
r, err := buildWithPackageJSON(t, `{"dependencies":{"@novox/mesh-sdk":"^0.1.0","@novox/other":"^1"}}`, nil, Npmrc{})
if err == nil || !strings.Contains(err.Error(), "@novox/other") {
t.Fatalf("a scoped package was installed with no registry for its scope: %v", err)
}
if strings.Contains(strings.Join(r.ran, "\n"), "--outDir") {
t.Fatal("the compile ran after the refusal")
}
// A public package installs without one, from the public registry and nothing else.
r, err = buildWithPackageJSON(t, `{"dependencies":{"mssql":"^11"}}`, nil, Npmrc{})
if err != nil {
t.Fatal(err)
}
if got := installs(r); len(got) != 1 || strings.Contains(got[0], ":registry=") || strings.Contains(got[0], "--network host") {
t.Fatalf("a public package's install: %v", got)
}
}
func TestAnUnreadablePackageJSONIsRefusedByName(t *testing.T) {
_, err := buildWithPackageJSON(t, `{"dependencies":`, nil, Npmrc{})
if err == nil || !strings.Contains(err.Error(), "package.json") {
t.Fatalf("a broken package.json was not refused by name: %v", err)
}
}
+20
View File
@@ -200,3 +200,23 @@ func TestWhatABuildReadIsTheRepositoriesItsRecipesName(t *testing.T) {
t.Fatal("a module whose recipes name no other repository read one")
}
}
// novox/hq 04-ISSUES/212: a toolchain stands on the SDK's published package, and is built with the
// exact version the mesh published — an argument that changes when the SDK does, so a rebuild after
// a release never reuses an install of the version before it.
func TestAPackageTheMeshPublishedIsPassedByItsExactVersion(t *testing.T) {
manifest := catalogue.Manifest{
Module: "mesh-tools",
Build: &catalogue.Build{
On: []catalogue.BuildsOn{{Arg: "MESH_SDK", Module: "mesh-sdk", Artifact: "lib"}},
},
}
held := map[string]string{"mesh-sdk/lib": "@novox/mesh-sdk@0.1.6"}
args, resolved, err := standingOn(context.Background(), manifest, held, noMirror)
if err != nil {
t.Fatal(err)
}
if fmt.Sprint(args) != "[--build-arg MESH_SDK=@novox/mesh-sdk@0.1.6]" || fmt.Sprint(resolved) != "[@novox/mesh-sdk@0.1.6]" {
t.Errorf("the package was passed as %v, recorded as %v", args, resolved)
}
}
+13 -2
View File
@@ -67,13 +67,23 @@ type Toolchain struct {
// `package.json` saying `"type": "module"` — Node reads a bare `.js` as CommonJS otherwise, so a
// bundle with its dependencies and without that line still fails to start — and the pruned,
// production-only node_modules the runtime itself ships with: the SDK's and the runtime's
// dependencies, and nothing module-specific yet (novox/hq ADR 0188 §5: a skeleton; a module's
// own npm dependencies are a later step). Empty for a language whose bundle carries its own —
// dependencies, and nothing module-specific (a module's own npm dependencies are installed into
// its own directory before the compile and inlined by the bundler: dependencies.go). Empty for a
// language whose bundle carries its own —
// a Go binary is static, a Python bundle is installed with its dependencies.
//
// A toolchain image without the directory fails the build by name rather than packing a bundle
// that starts nowhere: the image predates this and must be rebuilt first.
//
// *Since the bundler (below):* copied only for a bundle that names packages it keeps external,
// which cannot be inlined; a bundle with none carries no node_modules at all.
Dependencies string
// Bundler is the bundler inside the toolchain image that makes each compiled entrypoint and each
// launcher ONE self-contained file (novox/hq ADR 0193): every served bundle is its own process
// now, so each carries its own copy of what it imports — the SDK included — and nothing else.
// A bundle shrinks from the toolchain's whole node_modules to the code it runs. Empty for a
// language whose build is already one file.
Bundler string
// SystemStamp is the variable this language's linker fills with the artifact's declared system,
// for a language whose binaries are pinned to one at link time (novox/hq ADR 0005).
//
@@ -139,6 +149,7 @@ var toolchains = []Toolchain{
Unit: UnitSources,
SourceExt: ".ts",
Dependencies: "/app/runtime",
Bundler: "/app/node_modules/esbuild/bin/esbuild",
},
{
Language: "go",
@@ -0,0 +1,166 @@
package catalogue
import (
"encoding/json"
"strings"
"testing"
)
// novox/hq issue 213 (ADR 0188 §1, §3): a module's own Go service is a bundle the host runs as a
// process, not an image. Each test holds one thing that had to change in the composer for the
// controller to be declared that way.
const aServiceDigest = "sha256:dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd"
// aServiceModule is the controller's shape in miniature: it answers tools of its own, its code is a
// Go bundle a process runs as an account it declares, its secrets belong to that account, it
// prepares its state, and its process replaces the container it used to run as.
func aServiceModule(t *testing.T) Manifest {
t.Helper()
raw := `{
"module": "svc", "version": "1", "tools": ["status"], "prepares": true,
"own-secrets": {"store": "${dir:state}/store"},
"secrets-owner": "svc",
"resources": [
{"id": "state", "type": "directory", "mode": "0700", "place": "mesh", "owner": "svc"},
{"id": "service", "type": "process", "name": "svc", "artifact": "code",
"run": ["./svc", "serve"], "user": "svc", "replaces": ["server"],
"env": {"SVC_STORE_FILE": "${dir:state}/store", "SVC_STORE_PORT": "${seat:mesh-store:5432}"}},
{"id": "account", "type": "user", "name": "svc", "shell": "/usr/bin/nologin", "home": "/var/lib/svc"}
],
"build": {"artifacts": [{"name": "code", "kind": "bundle", "language": "go", "system": "arch",
"from": "cmd/svc", "binary": "svc"}]}
}`
m, err := ParseManifest([]byte(raw))
if err != nil {
t.Fatalf("the service's manifest is refused: %v", err)
}
resolved, err := m.Resolve([]Built{{Name: "code", Kind: ArtifactBundle,
Reference: ArtifactStoreScheme + "svc/code@" + aServiceDigest, Digest: aServiceDigest}})
if err != nil {
t.Fatal(err)
}
return resolved
}
func composeTheService(t *testing.T, with Rendering) []map[string]any {
t.Helper()
with.Needed = map[string]map[string]string{"svc": {"store": "sealed-store"}}
with.ArtifactStore = "anchor.internal:5100"
out, err := Resolution{Node: "anchor", Modules: []Manifest{aServiceModule(t)}}.Declaration(with)
if err != nil {
t.Fatalf("the service does not compose: %v", err)
}
return out
}
func indexOf(out []map[string]any, id string) int {
for i, r := range out {
if r["id"] == id {
return i
}
}
return -1
}
// A module that declares tools has every bundle served by the node's runtime unless it says
// otherwise — and the controller declares the verbs it answers as tools. Its service bundle is run
// by its own process; launched a second time by the runtime it would be a second controller
// pretending to be an MCP server.
func TestABundleItsOwnProcessRunsIsNotServedByTheRuntime(t *testing.T) {
m := aServiceModule(t)
if len(m.Bundles) != 1 {
t.Fatalf("the service's bundle was not kept: %+v", m.Bundles)
}
if loads := m.Bundles[0].Loads; len(loads) != 0 {
t.Fatalf("the runtime would launch the service's own bundle as tools: %v", loads)
}
// And a bundle no resource runs still is served, as a module declaring tools always had it.
tools := Manifest{Module: "t", Version: "1", Tools: []string{"x"},
Build: &Build{Artifacts: []Artifact{{Name: "tools", Kind: ArtifactBundle, Language: "go",
System: "arch", From: "cmd/t"}}}}
resolved, err := tools.Resolve([]Built{{Name: "tools", Kind: ArtifactBundle,
Reference: ArtifactStoreScheme + "t/tools@" + aServiceDigest, Digest: aServiceDigest}})
if err != nil {
t.Fatal(err)
}
if loads := resolved.Bundles[0].Loads; len(loads) != 1 || loads[0] != "t" {
t.Fatalf("a tools bundle nothing runs is no longer served: %v", loads)
}
}
// The account is created before anything is given to it. Its secrets are mesh-computed and so
// placed before the module's own resources; given to a user the machine did not have yet, they were
// refused on the first apply and the process started without them.
func TestAModulesAccountComesBeforeWhatBelongsToIt(t *testing.T) {
out := composeTheService(t, Rendering{})
account, secret := indexOf(out, "svc.account"), indexOf(out, "svc."+NeedID("store"))
if account < 0 || secret < 0 {
t.Fatalf("the account or the secret is missing: %v", out)
}
if account > secret {
t.Fatalf("the secret owned by svc is written before svc exists: account at %d, secret at %d",
account, secret)
}
if owner := out[secret]["owner"]; owner != "svc" {
t.Errorf("the secret belongs to %v, not the account its process runs as", owner)
}
}
// The process is the module's program; its preparation is the same program asked to prepare, as a
// step before it — with the same account and environment, and handing nothing over.
func TestAProcessIsPreparedByItsOwnProgram(t *testing.T) {
out := composeTheService(t, Rendering{})
step, process := indexOf(out, "svc.service-prepare"), indexOf(out, "svc.service")
if step < 0 || process < 0 || step > process {
t.Fatalf("the preparation is not a step before the process (%d, %d): %v", step, process, out)
}
s := out[step]
if s["type"] != "process" || s["run-once"] != true || s["name"] != "svc-prepare" {
t.Errorf("the preparation is not a run-once process: %v", s)
}
if run, _ := json.Marshal(s["run"]); string(run) != `["./svc","prepare"]` {
t.Errorf("the preparation runs %s", run)
}
if s["user"] != "svc" || s["source"] != out[process]["source"] {
t.Errorf("the preparation does not run the same bundle as the same account: %v", s)
}
if env, _ := s["env"].(map[string]any); env["SVC_STORE_FILE"] == nil {
t.Errorf("the preparation is not given the process's environment: %v", s["env"])
}
if _, has := s["replaces"]; has {
t.Errorf("the preparation would hand over what the process replaces: %v", s)
}
if _, has := s["args"]; has {
t.Errorf("the preparation carries a container's args: %v", s)
}
}
// What the process replaces is named as the host recorded it, `<module>.<id>`; unprefixed, the host
// matches nothing and removes the container first, as before.
func TestWhatAProcessReplacesIsNamedAsTheHostRecordedIt(t *testing.T) {
out := composeTheService(t, Rendering{})
p := out[indexOf(out, "svc.service")]
if got, _ := json.Marshal(p["replaces"]); string(got) != `["svc.server"]` {
t.Fatalf("the process replaces %s", got)
}
}
func TestWhatReplacesMayNameIsRefusedNearItsAuthor(t *testing.T) {
for what, resource := range map[string]string{
"a container": `{"id":"c","type":"container","name":"c","image":"x@` + aServiceDigest + `","replaces":["old"]}`,
"a step": `{"id":"p","type":"process","name":"p","run":["./p"],"run-once":true,"replaces":["old"]}`,
"something declared": `{"id":"p","type":"process","name":"p","run":["./p"],"replaces":["p"]}`,
"another module's": `{"id":"p","type":"process","name":"p","run":["./p"],"replaces":["other.old"]}`,
"not a list": `{"id":"p","type":"process","name":"p","run":["./p"],"replaces":"old"}`,
} {
raw := `{"module":"m","version":"1","resources":[` + resource + `]}`
if _, err := ParseManifest([]byte(raw)); err == nil || !strings.Contains(err.Error(), "replace") {
t.Errorf("replaces on %s was accepted: %v", what, err)
}
}
ok := `{"module":"m","version":"1","resources":[{"id":"p","type":"process","name":"p","run":["./p"],"replaces":["old"]}]}`
if _, err := ParseManifest([]byte(ok)); err != nil {
t.Errorf("a process replacing what its module no longer declares was refused: %v", err)
}
}
+12 -4
View File
@@ -46,7 +46,7 @@ func boundUsed(content string) [][2]string {
// Three facts the mesh states about any provision, plus whatever the provider said it serves. A
// module may not reach a binding it does not have — the same boundary as a secret, for the same
// reason.
func knownFor(m Manifest, needs []Needed, node string) map[string]map[string]string {
func knownFor(m Manifest, needs []Needed, node string) (map[string]map[string]string, error) {
out := map[string]map[string]string{}
for _, want := range m.Wants() {
for i := range needs {
@@ -54,12 +54,20 @@ func knownFor(m Manifest, needs []Needed, node string) map[string]map[string]str
if n.Name != want || n.For != m.Module {
continue
}
as := ConsumerIdentity(node, IdentitySource(m.Slug, m.Module))
values := map[string]string{
"at": n.At,
"from": n.From,
"as": ConsumerIdentity(node, IdentitySource(m.Slug, m.Module)),
"as": as,
}
for key, value := range n.Serves {
// What the provider derives for this consumer rather than for all of them
// (novox/hq ADR 0202). Filled here, the one place a provision and the module
// requiring it are both in hand.
served, err := ServedTo(n.Serves, as)
if err != nil {
return nil, fmt.Errorf("%s requires %s: %w", m.Module, want, err)
}
for key, value := range served {
// The provider's own vocabulary. Rendered plainly: a port is 5432, not 5432.000000,
// which is what a float would write and what a connection string would refuse.
values[key] = plainly(value)
@@ -67,7 +75,7 @@ func knownFor(m Manifest, needs []Needed, node string) map[string]map[string]str
out[want] = values
}
}
return out
return out, nil
}
// withOwnNames adds a module's own composed names to what it may name from one binding:
+25 -7
View File
@@ -77,6 +77,7 @@ func (m Manifest) Resolve(built []Built) (Manifest, error) {
// build compare equal.
out.Bundles = nil
if m.Build != nil {
run := runByAResource(m)
for _, a := range m.Build.Artifacts {
if a.Kind != ArtifactBundle {
continue
@@ -84,8 +85,13 @@ func (m Manifest) Resolve(built []Built) (Manifest, error) {
made := by[a.Name]
// What the runtime loads: what the artifact said, else every entrypoint of a module
// that declares tools, else nothing (the field's own rule; see Artifact.Loads).
//
// **Never, unasked, a bundle one of the module's own resources runs** (novox/hq issue 213).
// A process the host runs is the module's service, not its tools: the controller declares
// the verbs it answers as `tools` and serves them itself, and its bundle would otherwise
// have been launched a second time by the node's runtime, as an MCP child it is not.
loads := append([]string(nil), a.Loads...)
if a.Loads == nil && len(m.Tools) > 0 {
if a.Loads == nil && len(m.Tools) > 0 && !run[a.Name] {
loads = append([]string(nil), a.Entrypoints...)
// A bundle compiled to a binary has no entrypoints: the binary is what it is, and what
// the runtime starts to serve it (novox/hq ADR 0193). So a Go tools bundle is served
@@ -214,6 +220,11 @@ func (b *Build) problems(module string) []string {
// A bundle's source is the module's own directory by definition, and what it needs to say
// is which compiler — because the mesh chooses that, and cannot choose for a module that
// has not said.
if len(a.External) > 0 && (a.Kind != ArtifactBundle || a.Language != "typescript") {
problems = append(problems, fmt.Sprintf(
"%s: %q names packages it keeps external, and only a TypeScript bundle is bundled into "+
"one file with some kept out (novox/hq ADR 0193)", module, a.Name))
}
if len(a.Env) > 0 && a.Kind != ArtifactBundle {
problems = append(problems, fmt.Sprintf(
"%s: %q is a %q and says what it is given (env). Only a bundle the node's runtime "+
@@ -448,6 +459,18 @@ func BinaryOf(a Artifact) string {
return a.Name
}
// runByAResource is the artifacts one of a module's own resources names — a process that runs it,
// a step, an archive that unpacks it — by name.
func runByAResource(m Manifest) map[string]bool {
named := map[string]bool{}
for _, r := range m.Resources {
if a, ok := r["artifact"].(string); ok && a != "" {
named[a] = true
}
}
return named
}
// undeliveredBundles says which of a module's bundles nothing would ever put on a machine (novox/hq
// 04-ISSUES/216). A bundle reaches a machine three ways: the node's runtime serves it (it says
// `loads`, or its module declares `tools`), a resource names it (a process, a step, an archive), or
@@ -458,12 +481,7 @@ func undeliveredBundles(m Manifest) []string {
if m.Build == nil || m.Module == RuntimeModule {
return nil
}
named := map[string]bool{}
for _, r := range m.Resources {
if a, ok := r["artifact"].(string); ok && a != "" {
named[a] = true
}
}
named := runByAResource(m)
var problems []string
for _, a := range m.Build.Artifacts {
if a.Kind != ArtifactBundle || named[a.Name] || len(a.Loads) > 0 || len(m.Tools) > 0 {
+311
View File
@@ -0,0 +1,311 @@
package catalogue
import (
"fmt"
"regexp"
"sort"
"strings"
)
// What a provider derives for one consumer, said once in the provider's definition and delivered
// to both ends (novox/hq ADR 0202, issue 124).
//
// A `serves` block is otherwise literal: the same values for every consumer. Where the provider
// *names the resource* — a bucket, a database, a vhost — the name is derived from who is asking,
// and before this the mesh had no channel for it. The provider recomputed it in its own code and
// every consumer transcribed it into its own definition by hand, which is a copy of somebody
// else's rule kept in agreement by nobody. One of three transcriptions was wrong for months.
//
// **The mesh learns no protocol here; it spells its own name in an alphabet it already knows.**
// The only fact a served value may name is the identity the mesh itself minted for the consumer,
// in one of two alphabets: as it was minted, and as a DNS label. Everything a provider wants
// around it — a prefix, a suffix, a separator — it writes around the placeholder, because a
// served value is a string.
// consumerFact is `${consumer:<fact>}` or `${consumer:<fact>:<alphabet>}`.
var consumerFact = regexp.MustCompile(`\$\{consumer:([a-z][a-z0-9-]*)(?::([a-z][a-z0-9-]*))?\}`)
// consumerFacts are what a served value may name about the consumer it is being derived for.
// One entry, deliberately: the identity is the one thing about a consumer the mesh itself chose,
// so it is the one thing the mesh can hand to a provider without either end guessing.
var consumerFacts = []string{"as"}
// consumerAlphabets are the ways the mesh will write that identity. `dns` is the mesh's own
// identifier with its separator written `-` instead of `_` — the whole of the difference between
// the alphabet the mesh mints in and the one buckets, vhosts and hostnames accept.
var consumerAlphabets = []string{"dns"}
// ServedTo fills a provider's served values for one consumer.
//
// `as` is the identity the mesh minted for that consumer — the same string it is told to present
// as a login. Values with no placeholder are returned exactly as they were, and a block with no
// placeholder at all is returned unchanged, so this costs nothing for the providers that derive
// nothing.
//
// Only strings carry placeholders. A number, a boolean or a nested object is a value the provider
// stated outright, and is left alone.
func ServedTo(serves map[string]any, as string) (map[string]any, error) {
if len(serves) == 0 {
return serves, nil
}
var out map[string]any
for _, key := range sortedAnyKeys(serves) {
text, ok := serves[key].(string)
if !ok || !strings.Contains(text, "${consumer:") {
continue
}
filled, err := consumerInto(text, as)
if err != nil {
return nil, fmt.Errorf("the value served as %q: %w", key, err)
}
if out == nil {
// Copied only once something actually changes: the caller's map is the manifest's,
// and a provider that derives nothing must not have it rewritten underneath it.
out = make(map[string]any, len(serves))
for k, v := range serves {
out[k] = v
}
}
out[key] = filled
}
if out == nil {
return serves, nil
}
return out, nil
}
// consumerInto replaces every `${consumer:…}` in one value.
//
// **A fact or an alphabet the mesh does not have is refused, not left standing.** Written through,
// the literal `${consumer:as}` would reach a configuration file and be read as a bucket name,
// failing somewhere that names neither the module nor the mesh — the same reasoning `${bound:…}`
// is refused by (boundInto).
func consumerInto(value, as string) (string, error) {
var failed error
out := consumerFact.ReplaceAllStringFunc(value, func(match string) string {
parts := consumerFact.FindStringSubmatch(match)
fact, alphabet := parts[1], parts[2]
if fact != "as" {
if failed == nil {
failed = fmt.Errorf(
"says %s, and the mesh states %s about a consumer", match, orNothing(consumerFacts))
}
return match
}
switch alphabet {
case "":
return as
case "dns":
return asDNSLabel(as)
default:
if failed == nil {
failed = fmt.Errorf(
"says %s, and the mesh writes an identity as %s", match, orNothing(consumerAlphabets))
}
return match
}
})
if failed != nil {
return "", failed
}
return out, nil
}
// asDNSLabel writes a minted identity as a DNS label.
//
// The mesh's identities are already lower-case letters, digits and `_` (ConsumerIdentity), and
// already short enough for the tightest backend they reach (CheckIdentity, twenty characters). So
// this is the separator and nothing else — no lower-casing of what is already lower case, no
// truncation to a limit the identity is already inside, no padding of a name that is already long
// enough. Each of those would be the mesh guessing at a rule it has not been given.
func asDNSLabel(as string) string {
return strings.ReplaceAll(as, "_", "-")
}
// CheckServes refuses a `serves` block that names a consumer fact or an alphabet the mesh does not
// have, when the definition is parsed rather than when a consumer is resolved.
//
// A provision nobody consumes yet still has its rule read: a definition that would be refused the
// first time somebody required it is a definition that is wrong now.
func CheckServes(m Manifest) []string {
var problems []string
for _, provision := range sortedServes(m.Serves) {
for _, key := range sortedAnyKeys(m.Serves[provision]) {
text, ok := m.Serves[provision][key].(string)
if !ok {
continue
}
// A probe identity, because what is checked is the shape of the statement and not
// what any consumer is called.
if _, err := consumerInto(text, "mesh_node_module"); err != nil {
problems = append(problems, fmt.Sprintf(
"%s serves %s, and the value it serves as %q %s", m.Module, provision, key, err))
}
}
}
return problems
}
func sortedServes(serves map[string]map[string]any) []string {
out := make([]string, 0, len(serves))
for k := range serves {
out = append(out, k)
}
sort.Strings(out)
return out
}
func sortedAnyKeys(values map[string]any) []string {
out := make([]string, 0, len(values))
for k := range values {
out = append(out, k)
}
sort.Strings(out)
return out
}
// derivedFor is what the provider on this machine derives for one consumer of one provision
// (novox/hq ADR 0202).
//
// Settled first, then derived: an operator may set a prefix on what the provider serves and the
// mesh still fills the consumer's half of it ([ADR 0174]). Only the keys that actually name the
// consumer are returned — the rest of a `serves` block is the same for every consumer and is
// already in the provider's own definition, so repeating it here would be a second copy to go
// stale.
//
// The first module in the resolved order that says it serves the provision answers, which is the
// choice servedOnThisMachine makes for the consumer's half. Nothing serving it on this machine is
// not an error: a contribution can reach a machine whose provider is a record or an adapter, and
// then there is nothing derived to tell.
func (r Resolution) derivedFor(provision, as, consumer, local string, settings SettingsBy) (map[string]any, error) {
for _, m := range r.Modules {
serves, said := m.Serves[provision]
if !said {
continue
}
var names map[string]any
for key, value := range serves {
if text, ok := value.(string); ok && strings.Contains(text, "${consumer:") {
if names == nil {
names = map[string]any{}
}
names[key] = value
}
}
if names == nil {
return nil, nil
}
// **A consumer that keeps several holders of this provision is refused** — this is issue
// 124's own failure one case to the side, and it would be just as quiet.
//
// Each holder gets its own login, `…_<local>` (ADR 0094), and a provider derives from the
// login, so it would make one resource per holder. The consumer's side has no such
// dimension: one binding file per provision, one `${bound:<provision>:<key>}`, both
// derived from the un-suffixed identity. So the provider would create the holder's
// resource and the consumer would be configured against a name nothing made — it would
// authenticate successfully and be refused on every object, which reads like a credential
// fault and is not one.
//
// Lifting this means giving the consumer's side a local dimension. That is a decision,
// not an omission, and until it is taken the mesh says so rather than guessing.
if local != "" {
return nil, fmt.Errorf(
"%s keeps several holders of %s (this one is %q), and %s derives %s for each "+
"consumer from the login the mesh minted. Each holder has its own login, and a "+
"consumer is told one value per requirement — so the two ends would name "+
"different things and nothing would compare them (novox/hq ADR 0202)",
consumer, local, provision, m.Module, orNothing(sortedAnyKeys(names)))
}
settled, err := Settle(names, settings[m.Module])
if err != nil {
return nil, fmt.Errorf("%s serving %s: %w", m.Module, provision, err)
}
derived, err := ServedTo(settled, as)
if err != nil {
return nil, fmt.Errorf("%s serving %s to %s: %w", m.Module, provision, as, err)
}
return derived, nil
}
return nil, nil
}
// notTranscribed refuses a consumer's file that writes out the value its provider derives for it,
// instead of asking for it (novox/hq ADR 0202, issue 124).
//
// **What would have caught the one wrong instance.** The object store's three consumers each wrote
// their bucket into their own configuration by hand. One of them named a predecessor's bucket, and
// nothing compared it to what the provider would actually create: the module would have
// authenticated successfully and been refused on every object, which reads like a credential fault
// and is not one. It looked authoritative for months.
//
// The test is exact and costs one string search: a definition whose file already contains the
// value the mesh is about to derive for it has written down somebody else's rule. It cannot be a
// coincidence — a derived value carries the identity the mesh minted for this very consumer on
// this very machine, which nothing else would spell out — and it cannot be checked afterwards,
// because after substitution every consumer's file contains it legitimately.
//
// Only values that actually name the consumer are judged. A provider that serves a constant under
// the same key serves the same constant to everyone, and a consumer repeating it is redundant
// rather than wrong.
func notTranscribed(resource map[string]any, known map[string]map[string]string, module string) error {
if fmt.Sprint(resource["type"]) != "file" {
return nil
}
content, ok := resource["content"].(string)
if !ok || content == "" {
return nil
}
for _, provision := range sortedKnown(known) {
values := known[provision]
identity := values["as"]
if identity == "" {
continue
}
for _, key := range sortedStringKeys(values) {
if key == "as" {
// The login is not derived from itself, and a consumer that must present it in a
// connection string legitimately has it from `${bound:…}` — which is what it will
// be after substitution, so this would judge the substitution, not the module.
continue
}
value := values[key]
if value == "" || !namesTheConsumer(value, identity) {
continue
}
if !strings.Contains(content, value) {
continue
}
return fmt.Errorf(
"%s writes %q into %v, and that is exactly what %s derives for it — a definition "+
"keeping its own copy of somebody else's naming rule is one that can disagree "+
"with it, silently. Say ${bound:%s:%s} and be told",
module, value, resource["id"], provision, provision, key)
}
}
return nil
}
// namesTheConsumer is whether a derived value was built from this consumer's identity — in the
// alphabet it was minted in, or as a DNS label. A value that does not contain it was not derived
// from it, whatever else it may be.
func namesTheConsumer(value, identity string) bool {
return strings.Contains(value, identity) || strings.Contains(value, asDNSLabel(identity))
}
func sortedKnown(known map[string]map[string]string) []string {
out := make([]string, 0, len(known))
for k := range known {
out = append(out, k)
}
sort.Strings(out)
return out
}
func sortedStringKeys(values map[string]string) []string {
out := make([]string, 0, len(values))
for k := range values {
out = append(out, k)
}
sort.Strings(out)
return out
}
@@ -0,0 +1,131 @@
package catalogue
import (
"encoding/json"
"fmt"
"os"
"strings"
"testing"
)
// novox/hq issue 213: the controller is a Go program and was the one piece of the mesh's own Go
// code still shipped as an image (ADR 0188 §1). Its own manifest, composed for the machine that runs
// it, is a Go bundle run by the host as a process — and no container.
func TestTheControllerIsAProcessAndNoContainer(t *testing.T) {
raw, err := os.ReadFile("../../module.json")
if err != nil {
t.Fatal(err)
}
m, err := ParseManifest(raw)
if err != nil {
t.Fatalf("the controller's own manifest does not parse:\n%v", err)
}
if m.Build == nil || len(m.Build.Artifacts) != 1 {
t.Fatalf("the controller builds %+v; it is one bundle", m.Build)
}
a := m.Build.Artifacts[0]
if a.Kind != ArtifactBundle || a.Language != "go" || a.System == "" || BinaryOf(a) != "mesh-controller" {
t.Fatalf("the controller's artifact is %+v, not a Go bundle naming its system and binary", a)
}
for _, c := range m.Capabilities {
if c == "container-runtime" {
t.Error("the controller still requires a container runtime on its machine")
}
}
digest := "sha256:" + strings.Repeat("c", 64)
control, err := m.Resolve([]Built{{Name: a.Name, Kind: ArtifactBundle,
Reference: ArtifactStoreScheme + "mesh-controller/" + a.Name + "@" + digest, Digest: digest}})
if err != nil {
t.Fatal(err)
}
// The node's runtime does not launch it: it serves its seat's verbs itself.
if loads := control.Bundles[0].Loads; len(loads) != 0 {
t.Errorf("the node's runtime would launch the controller as a tools bundle: %v", loads)
}
needed := map[string]map[string]string{"mesh-controller": {}}
for name := range m.OwnSecrets {
needed["mesh-controller"][name] = "sealed-" + name
}
out, err := Resolution{Node: "anchor", Modules: []Manifest{control}}.Declaration(Rendering{
Needed: needed, ArtifactStore: "anchor.internal:5100",
Seats: map[string]map[int]int{"mesh-store": {5432: 6852}},
})
if err != nil {
t.Fatalf("the controller does not compose: %v", err)
}
var process, step map[string]any
account, firstSecret := -1, -1
for i, r := range out {
switch {
case r["type"] == "container":
t.Errorf("the controller's declaration still runs a container: %v", r)
case r["id"] == "mesh-controller.controller":
process = r
case r["id"] == "mesh-controller.controller-prepare":
step = r
if process != nil {
t.Error("the controller's preparation is placed after the process it prepares for")
}
case r["type"] == "user" && r["name"] == "mesh-controller":
account = i
case strings.HasPrefix(fmt.Sprint(r["id"]), "mesh-controller.needs-") && firstSecret < 0:
firstSecret = i
}
}
if process == nil {
t.Fatalf("the controller's process is not in its declaration: %v", out)
}
if run, _ := json.Marshal(process["run"]); string(run) != `["./mesh-controller","serve"]` {
t.Errorf("the controller is run as %s, not its own bundle's binary", run)
}
if process["source"] != "anchor.internal:5100/mesh-controller/"+a.Name+"@"+digest || process["digest"] != digest {
t.Errorf("the controller's bundle is fetched from %v (%v)", process["source"], process["digest"])
}
// The user: an account the host declares, which owns what the process reads.
if process["user"] != "mesh-controller" || account < 0 {
t.Errorf("the controller runs as %v, and the account declared is at %d", process["user"], account)
}
if firstSecret >= 0 && account > firstSecret {
t.Error("the controller's secrets are written before the account they belong to exists")
}
for _, r := range out {
if strings.HasPrefix(fmt.Sprint(r["id"]), "mesh-controller.needs-") && r["owner"] != "mesh-controller" {
t.Errorf("%v belongs to %v, which the controller's process cannot read", r["id"], r["owner"])
}
}
if dir := fileNamed(out, "mesh-controller.mesh-state"); dir == nil || dir["owner"] != "mesh-controller" {
t.Errorf("the controller's state directory is not its account's to enter: %v", dir)
}
// Each mount became a path the process reads: nothing it is told is a path inside a container.
state := fmt.Sprint(fileNamed(out, "mesh-controller.mesh-state")["path"])
env, _ := process["env"].(map[string]any)
for key, value := range env {
v := fmt.Sprint(value)
if strings.HasPrefix(v, "/run/secrets") || strings.HasPrefix(v, "/broker-tls") {
t.Errorf("%s=%s is a path inside the container the controller no longer runs in", key, v)
}
if strings.HasSuffix(key, "_FILE") && !strings.HasPrefix(v, state+"/") {
t.Errorf("%s=%s is not one of the files the mesh places for it", key, v)
}
}
if env["MESH_BROKER_CERTIFICATE"] != "/var/lib/mesh-broker-tls/tls.crt" {
t.Errorf("the controller reads the broker's certificate from %v", env["MESH_BROKER_CERTIFICATE"])
}
if env["MESH_STORE_INVENTORY_PORT"] != "6852" {
t.Errorf("the controller is told the store is on %v; the node put it on 6852", env["MESH_STORE_INVENTORY_PORT"])
}
// The handover: the container it ran as goes only once this is running.
if got, _ := json.Marshal(process["replaces"]); string(got) != `["mesh-controller.server"]` {
t.Errorf("the controller's process replaces %s, not the container it ran as", got)
}
// And its state is prepared first, by the same program as the same account.
if step == nil || step["run-once"] != true || step["user"] != "mesh-controller" {
t.Fatalf("the controller's preparation is %v", step)
}
if run, _ := json.Marshal(step["run"]); string(run) != `["./mesh-controller","prepare"]` {
t.Errorf("the controller's preparation runs %s", run)
}
}
+111 -9
View File
@@ -645,7 +645,16 @@ func (r Resolution) compose(with Rendering, owner map[string]string,
if err != nil {
return nil, err
}
file, err := boundFile(*found, m.Binds[to], ConsumerIdentity(r.Node, IdentitySource(m.Slug, m.Module)), own)
as := ConsumerIdentity(r.Node, IdentitySource(m.Slug, m.Module))
// What the provider derives for THIS consumer, filled here where the consumer is
// known (novox/hq ADR 0202). The same fill knownFor does below, so the binding file
// and the module's `${bound:…}` substitutions cannot say different things.
told := *found
told.Serves, err = ServedTo(told.Serves, as)
if err != nil {
return nil, fmt.Errorf("%s is told about %s: %w", m.Module, to, err)
}
file, err := boundFile(told, m.Binds[to], as, own)
if err != nil {
return nil, err
}
@@ -693,7 +702,15 @@ func (r Resolution) compose(with Rendering, owner map[string]string,
// Now, and not before: a module whose resources are computed replaces them wholesale, and
// merging earlier would throw away the files it still needs.
resources = append(append([]map[string]any{}, first...), resources...)
//
// **Except the module's own accounts, which go before even those** (novox/hq issue 213). What
// the mesh computes may belong to one: a module whose code runs as an account it declares has
// its secrets written owned by that account, and a file given to a user the machine does not
// have yet fails — so on the first apply the secrets were refused, the process started without
// them, and the second apply healed it, which is the fault the paragraph above describes.
// An account depends on nothing the mesh computes.
accounts, rest := accountsFirst(resources)
resources = append(append(accounts, first...), rest...)
// No container is given the mesh's names (novox/hq ADR 0148). It used to be: every
// container got the whole roster as `--add-host` entries at creation, and a name that
@@ -711,7 +728,10 @@ func (r Resolution) compose(with Rendering, owner map[string]string,
return nil, err
}
// And what its bindings say, for the half of a connection that is not secret.
known := knownFor(m, r.Needs, r.Node)
known, err := knownFor(m, r.Needs, r.Node)
if err != nil {
return nil, err
}
// A requirement answered on this same machine is not in r.Needs — its binding file is
// written from `here` (above) — and so `${bound:…}` could not name it, though the file
// beside it said the same facts. Filled from the same answer, so the two cannot disagree.
@@ -728,7 +748,11 @@ func (r Resolution) compose(with Rendering, owner map[string]string,
}
local := *answered
local.For = m.Module
for provision, values := range knownFor(m, []Needed{local}, r.Node) {
here, err := knownFor(m, []Needed{local}, r.Node)
if err != nil {
return nil, err
}
for provision, values := range here {
known[provision] = values
}
}
@@ -753,6 +777,17 @@ func (r Resolution) compose(with Rendering, owner map[string]string,
// And the machine underneath, which no binding of its own can tell it.
thisMachine := machineFacts(r, with.Names, with.MeshRange)
// **A definition that already holds the answer transcribed it** (novox/hq ADR 0202).
// Judged over what the module itself declares, and before anything is substituted: the
// mesh's own generated files — the binding, the contributions — legitimately carry the
// derived value, and after substitution so does every consumer's file, so this is the one
// moment the two can be told apart.
for _, own := range m.Resources {
if err := notTranscribed(own, known, m.Module); err != nil {
return nil, err
}
}
// Which of this module's files carry a secret, for the rule that a container may not read
// one of them as its environment without saying so (ADR 0086, issue 041).
secretFiles := secretFilesOf(resources)
@@ -872,6 +907,21 @@ func (r Resolution) compose(with Rendering, owner map[string]string,
if renamed := reflectsRenamed(m.Module, resource["reload-on"]); renamed != nil {
copied["reload-on"] = renamed
}
// And which of its module's containers a scheduled step holds still (novox/hq ADR 0189).
// **The loudest of the three when it is missed.** An unprefixed `restart-on` matches
// nothing and a service quietly never restarts; an unprefixed `while-stopped` names a
// container the declaration does not contain, and the host refuses the whole
// declaration — so the machine takes nothing at all, for every push, until this is
// right. That is what it did on the control node (2026-10-04).
if renamed := reflectsRenamed(m.Module, resource[WhileStopped]); renamed != nil {
copied[WhileStopped] = renamed
}
// And what a process replaces (novox/hq issue 213): a resource of this module's that it
// no longer declares, named as the host recorded it, or the host hands nothing over and
// removes it first.
if renamed := reflectsRenamed(m.Module, resource["replaces"]); renamed != nil {
copied["replaces"] = renamed
}
// **What reads one of this module's own secrets is restarted when it changes** (novox/hq
// issue 203, issue 206). A credential is re-issued by the mesh, and a container that
// mounted the old file keeps the old one open: the build machine ran for an hour on a
@@ -1161,6 +1211,19 @@ type Contribution struct {
// requirement's name — everything providing `reverse-proxy` understands the same shape, which
// is what makes swapping one for another cost nothing.
Values map[string]any `json:"values"`
// Derived is what this provider's own definition said it derives for this consumer, already
// derived (novox/hq ADR 0202).
//
// **The provider is told, rather than recomputing it.** A served value may name the consumer's
// identity — a bucket named for who is asking, a database prefixed with it — and before this
// the rule lived twice: once in the provisioner's code, once transcribed into every consumer's
// definition. The mesh fills the provider's own statement here and delivers the same filled
// value to the consumer, so the two cannot disagree: there is no second computation to
// disagree with.
//
// Only the keys that are per-consumer. The rest of what the provider serves is the same for
// everyone and is in its own definition, where it already is.
Derived map[string]any `json:"derived,omitempty"`
}
// grantPath is where one consumer's sealed credential lands on the providing machine.
@@ -1252,12 +1315,17 @@ func (r Resolution) contributions(settings SettingsBy, grants []Grant,
// told about it and withdraws the login on its next pass.
continue
}
as := holderAs(ConsumerIdentity(g.Consumer, IdentitySource(g.Slug, g.From)), g.Local)
derived, err := r.derivedFor(g.Provision, as, g.From, g.Local, settings)
if err != nil {
return nil, err
}
out[g.Provision] = append(out[g.Provision], Contribution{
From: g.From, Node: g.Consumer, At: g.At, Values: g.Values,
From: g.From, Node: g.Consumer, At: g.At, Values: g.Values, Derived: derived,
// One holder per local name: the identity the consumer is known by, and the local name
// after it where the module keeps several (ADR 0094). Not a login any backend checks —
// a secret is not a login — so the identity limit does not apply to the suffix.
As: holderAs(ConsumerIdentity(g.Consumer, IdentitySource(g.Slug, g.From)), g.Local),
As: as,
Secret: grantPath(directories[g.Provision], g.Consumer, holderAs(g.From, g.Local)),
})
if granted[g.Provision] == nil {
@@ -2010,12 +2078,18 @@ func preparationTarget(m Manifest) string {
return ""
}
for _, r := range m.Resources {
if fmt.Sprint(r["type"]) != "container" || !ownArtifact(r, m.Module) {
// A container, or a process the host runs from a bundle the module built (novox/hq issue
// 213): the same program in the same context, hosted as a unit rather than a container.
kind := fmt.Sprint(r["type"])
if (kind != "container" && kind != "process") || !ownArtifact(r, m.Module) {
continue
}
if once, _ := r["run-once"].(bool); once {
continue
}
if r["schedule"] != nil {
continue
}
return fmt.Sprint(r["id"])
}
return ""
@@ -2029,7 +2103,10 @@ func ownArtifact(resource map[string]any, module string) bool {
return true
}
image, _ := resource["image"].(string)
return strings.HasPrefix(image, ArtifactStoreScheme+module+"/")
// A process or an archive carries what was built as its source (novox/hq issue 213).
source, _ := resource["source"].(string)
return strings.HasPrefix(image, ArtifactStoreScheme+module+"/") ||
strings.HasPrefix(source, ArtifactStoreScheme+module+"/")
}
// prepared is the module's own resource as the step that prepares its state: the same image, the same
@@ -2051,7 +2128,19 @@ func prepared(from map[string]any) map[string]any {
step["id"] = fmt.Sprint(from["id"]) + "-prepare"
step["name"] = fmt.Sprint(from["name"]) + "-prepare"
step["run-once"] = true
step["args"] = []any{PreparationArgument}
if fmt.Sprint(from["type"]) == "process" {
// A process says its whole command: the program, then its arguments. The step is the same
// program asked to prepare (novox/hq issue 213). It replaces nothing — what the process
// replaces is handed over to the process, never to the step that runs before it — and a
// step is not restarted, it runs again when what it reads changed, which `restart-on` says.
run := stringsIn(from["run"])
if len(run) > 0 {
step["run"] = []any{run[0], PreparationArgument}
}
delete(step, "replaces")
} else {
step["args"] = []any{PreparationArgument}
}
delete(step, "ports")
delete(step, "ip")
delete(step, "schedule")
@@ -2196,3 +2285,16 @@ func withRestartOn(have any, add []string) []any {
}
return out
}
// accountsFirst splits a module's resources into its accounts and everything else, each in the order
// written.
func accountsFirst(resources []map[string]any) (accounts, rest []map[string]any) {
for _, r := range resources {
if fmt.Sprint(r["type"]) == "user" {
accounts = append(accounts, r)
continue
}
rest = append(rest, r)
}
return accounts, rest
}
@@ -0,0 +1,343 @@
package catalogue
import (
"encoding/json"
"strings"
"testing"
)
// What a provider derives for each consumer, said once and delivered to both ends
// (novox/hq ADR 0202, issue 124).
//
// The failure these are written against: the object store's provisioner derived each consumer's
// bucket from the login the mesh minted, in its own code, and the mesh had no channel to tell the
// consumer which bucket that was — so all three consumers wrote the answer into their own
// definitions by hand. Two were right. One named a predecessor's bucket and would have
// authenticated successfully and been refused on every object. Each of them also named the
// machine the module happens to run on, which a definition may not do.
// store is an object store in the shape minio has: it serves a region and a port to everyone, and
// a bucket named for whoever is asking.
func store() Manifest {
return Manifest{
Module: "store", Version: "1",
Provides: FromAnywhere("s3-bucket"),
Listens: []Listening{{Port: 9000, Protocol: "tcp", From: FromMesh}},
Serves: map[string]map[string]any{"s3-bucket": {
"region": "eu-west",
"bucket": "${consumer:as:dns}",
}},
Receives: map[string]string{"s3-bucket": "/var/lib/store/grants/mesh.json"},
Grants: map[string]string{"s3-bucket": "/var/lib/store/grants"},
Resources: []map[string]any{{
"id": "server", "type": "container", "name": "store", "ports": []any{"9000"},
}},
}
}
// files is a consumer that writes the bucket into its own configuration — which is the thing it
// could not do before, and had to transcribe.
func files() Manifest {
return Manifest{
Module: "files", Version: "1", Slug: "files",
Requires: []string{"s3-bucket"},
Binds: map[string]string{"s3-bucket": "/var/lib/files/store.json"},
Secrets: map[string]string{"s3-bucket": "/var/lib/files/store.secret"},
Resources: []map[string]any{{
"id": "env", "type": "file", "path": "/var/lib/files/env", "mode": "0600",
"content": "BUCKET=${bound:s3-bucket:bucket}\nREGION=${bound:s3-bucket:region}\n",
}},
}
}
// pics is a second consumer of the same provider on the same machine: two derivations, neither
// the other's.
func pics() Manifest {
return Manifest{
Module: "pics", Version: "1", Slug: "pics",
Requires: []string{"s3-bucket"},
Binds: map[string]string{"s3-bucket": "/var/lib/pics/store.json"},
Secrets: map[string]string{"s3-bucket": "/var/lib/pics/store.secret"},
Resources: []map[string]any{{
"id": "env", "type": "file", "path": "/var/lib/pics/env", "mode": "0600",
"content": "BUCKET=${bound:s3-bucket:bucket}\n",
}},
}
}
// The three places the derived value lands must agree, because agreeing is the whole point: the
// consumer's own file, the binding it reads as JSON, and the provider's contributions entry.
func TestADerivedValueReachesBothEndsAndAgrees(t *testing.T) {
r, err := Resolve(shelf(store(), files()), []string{"store", "files"}, reachable(), World{})
if err != nil {
t.Fatal(err)
}
out, err := r.Declaration(Rendering{Grants: []Grant{{
Provision: "s3-bucket", Consumer: "workstation", From: "files", Slug: "files",
Values: map[string]any{}, Sealed: "c2VhbGVk",
}}})
if err != nil {
t.Fatal(err)
}
// The mesh minted this identity for the consumer; the bucket is that identity as a DNS label.
// Derived here with the mesh's own function, so the test cannot agree with a wrong rule.
as := ConsumerIdentity("workstation", IdentitySource("files", "files"))
want := strings.ReplaceAll(as, "_", "-")
if want == as || !strings.Contains(as, "_") {
t.Fatalf("the mesh's identity %q has no separator to rewrite; this test proves nothing", as)
}
env := fileNamed(out, "files.env")
if env == nil {
t.Fatalf("the consumer was given no file: %v", out)
}
if got := env["content"].(string); !strings.Contains(got, "BUCKET="+want+"\n") {
t.Errorf("the consumer's own file was not told the bucket:\n%s\nwant BUCKET=%s", got, want)
}
binding := fileNamed(out, "files.bound-s3-bucket")
if binding == nil {
t.Fatalf("the consumer was given no binding: %v", out)
}
var said struct {
Serves map[string]any `json:"serves"`
}
if err := json.Unmarshal([]byte(binding["content"].(string)), &said); err != nil {
t.Fatal(err)
}
if said.Serves["bucket"] != want {
t.Errorf("the binding says the bucket is %q, want %q", said.Serves["bucket"], want)
}
// And what is the same for everybody is still the same for everybody.
if said.Serves["region"] != "eu-west" {
t.Errorf("the binding lost what the provider serves to all: %v", said.Serves)
}
given := storeGrants(t, out)
if len(given) != 1 {
t.Fatalf("the provider was told about %d consumer(s): %v", len(given), given)
}
if given[0].Derived["bucket"] != want {
t.Errorf("the provider was told the bucket is %v, and the consumer was told %q — "+
"the two ends disagree, which is the whole failure", given[0].Derived["bucket"], want)
}
// Only the per-consumer half. The region is the same for everyone and is already in the
// provider's own definition; repeating it here would be a copy to go stale.
if _, carried := given[0].Derived["region"]; carried {
t.Errorf("the provider was handed back what it already says for everyone: %v", given[0].Derived)
}
}
// Two consumers of one provider on one machine get two buckets, and neither gets the other's.
func TestTwoConsumersOfOneProviderGetTheirOwnDerivation(t *testing.T) {
r, err := Resolve(shelf(store(), files(), pics()),
[]string{"store", "files", "pics"}, reachable(), World{})
if err != nil {
t.Fatal(err)
}
out, err := r.Declaration(Rendering{Grants: []Grant{
{Provision: "s3-bucket", Consumer: "workstation", From: "files", Slug: "files",
Values: map[string]any{}, Sealed: "c2VhbGVk"},
{Provision: "s3-bucket", Consumer: "workstation", From: "pics", Slug: "pics",
Values: map[string]any{}, Sealed: "c2VhbGVk"},
}})
if err != nil {
t.Fatal(err)
}
forFiles := strings.ReplaceAll(ConsumerIdentity("workstation", IdentitySource("files", "files")), "_", "-")
forPics := strings.ReplaceAll(ConsumerIdentity("workstation", IdentitySource("pics", "pics")), "_", "-")
if forFiles == forPics {
t.Fatal("the two consumers were given the same identity; this test proves nothing")
}
if got := fileNamed(out, "files.env")["content"].(string); !strings.Contains(got, "BUCKET="+forFiles+"\n") {
t.Errorf("files was not given its own bucket:\n%s", got)
}
if got := fileNamed(out, "pics.env")["content"].(string); !strings.Contains(got, "BUCKET="+forPics+"\n") {
t.Errorf("pics was not given its own bucket:\n%s", got)
}
var buckets []any
for _, g := range storeGrants(t, out) {
buckets = append(buckets, g.Derived["bucket"])
}
if len(buckets) != 2 || buckets[0] == buckets[1] {
t.Errorf("the provider was told %v; it must be told one bucket per consumer", buckets)
}
}
// An operator may still set what the provider serves, and the mesh still derives the rest: the
// setting is laid on first, then the consumer's half is filled.
func TestASettingComposesWithADerivedValue(t *testing.T) {
r, err := Resolve(shelf(store(), files()), []string{"store", "files"}, reachable(), World{})
if err != nil {
t.Fatal(err)
}
out, err := r.Declaration(Rendering{
Settings: SettingsBy{"store": {{From: "the operator",
Values: map[string]any{"bucket": "team-${consumer:as:dns}"}}}},
Grants: []Grant{{Provision: "s3-bucket", Consumer: "workstation", From: "files", Slug: "files",
Values: map[string]any{}, Sealed: "c2VhbGVk"}},
})
if err != nil {
t.Fatal(err)
}
want := "team-" + strings.ReplaceAll(ConsumerIdentity("workstation", IdentitySource("files", "files")), "_", "-")
if got := fileNamed(out, "files.env")["content"].(string); !strings.Contains(got, "BUCKET="+want+"\n") {
t.Errorf("the operator's prefix did not survive the derivation:\n%s\nwant BUCKET=%s", got, want)
}
if given := storeGrants(t, out); given[0].Derived["bucket"] != want {
t.Errorf("the provider was told %v, the consumer %q", given[0].Derived["bucket"], want)
}
}
// A fact or an alphabet the mesh does not have is refused where the definition is, not where a
// consumer happens to be resolved — and the refusal says what may be said instead.
func TestAServedValueNamingSomethingTheMeshDoesNotHaveIsRefused(t *testing.T) {
for _, c := range []struct{ value, says string }{
{"${consumer:node}", "as"},
{"${consumer:as:punycode}", "dns"},
} {
m := store()
m.Serves["s3-bucket"]["bucket"] = c.value
raw, err := json.Marshal(m)
if err != nil {
t.Fatal(err)
}
_, err = ParseManifest(raw)
if err == nil {
t.Fatalf("%s was accepted", c.value)
}
if !strings.Contains(err.Error(), c.value) {
t.Errorf("the refusal of %s does not quote it: %v", c.value, err)
}
if !strings.Contains(err.Error(), c.says) {
t.Errorf("the refusal of %s does not say what may be said (%q): %v", c.value, c.says, err)
}
}
}
// `dns` is checked against an identity the mesh actually mints, not an invented string.
func TestTheDNSAlphabetIsTheMintedIdentityWithItsSeparatorRewritten(t *testing.T) {
as := ConsumerIdentity("anchor", IdentitySource("ncloud", "nextcloud"))
if err := CheckIdentity("anchor", IdentitySource("ncloud", "nextcloud")); err != nil {
t.Fatalf("the mesh would not mint this identity at all: %v", err)
}
label := asDNSLabel(as)
if strings.Contains(label, "_") {
t.Errorf("%q is not a DNS label", label)
}
if strings.ReplaceAll(label, "-", "_") != as {
t.Errorf("%q is not %q with its separator rewritten", label, as)
}
}
// The check that would have caught the one wrong instance: a consumer that writes the derived
// value into its own definition instead of asking for it is refused, whether it transcribed the
// right answer or a predecessor's.
func TestAConsumerThatTranscribesWhatItsProviderDerivesIsRefused(t *testing.T) {
as := ConsumerIdentity("workstation", IdentitySource("files", "files"))
transcribed := strings.ReplaceAll(as, "_", "-")
m := files()
m.Resources = []map[string]any{{
"id": "env", "type": "file", "path": "/var/lib/files/env", "mode": "0600",
// Exactly what the provider will create — correct today, and a copy of a rule that is
// not this module's.
"content": "BUCKET=" + transcribed + "\n",
}}
r, err := Resolve(shelf(store(), m), []string{"store", "files"}, reachable(), World{})
if err != nil {
t.Fatal(err)
}
_, err = r.Declaration(Rendering{Grants: []Grant{{
Provision: "s3-bucket", Consumer: "workstation", From: "files", Slug: "files",
Values: map[string]any{}, Sealed: "c2VhbGVk",
}}})
if err == nil {
t.Fatal("a definition holding its own copy of the provider's naming rule was accepted")
}
if !strings.Contains(err.Error(), "${bound:s3-bucket:bucket}") {
t.Errorf("the refusal does not say what to write instead: %v", err)
}
// And a constant the provider serves to everyone is not a transcription: repeating it is
// redundant, not wrong, and refusing it would be the mesh policing style.
m.Resources = []map[string]any{{
"id": "env", "type": "file", "path": "/var/lib/files/env", "mode": "0600",
"content": "REGION=eu-west\n",
}}
r, err = Resolve(shelf(store(), m), []string{"store", "files"}, reachable(), World{})
if err != nil {
t.Fatal(err)
}
if _, err := r.Declaration(Rendering{Grants: []Grant{{
Provision: "s3-bucket", Consumer: "workstation", From: "files", Slug: "files",
Values: map[string]any{}, Sealed: "c2VhbGVk",
}}}); err != nil {
t.Errorf("a value the provider serves to everyone was judged a transcription: %v", err)
}
}
func storeGrants(t *testing.T, out []map[string]any) []Contribution {
t.Helper()
for _, r := range out {
if r["path"] != "/var/lib/store/grants/mesh.json" {
continue
}
var parsed struct {
Given []Contribution `json:"given"`
}
if err := json.Unmarshal([]byte(r["content"].(string)), &parsed); err != nil {
t.Fatal(err)
}
return parsed.Given
}
t.Fatalf("the provider was given no contributions file: %v", out)
return nil
}
// A consumer that keeps SEVERAL holders of one provision is refused, rather than told one thing
// while its provider is told another.
//
// **This is issue 124's own failure, one case to the side.** The mesh gives each holder its own
// login — `mesh_node_mod_<local>` (ADR 0094) — and the provider derives from the login, so it
// would make one resource per holder. The consumer's side has no such dimension: there is one
// binding file per provision and one `${bound:<provision>:<key>}`, both derived from the
// un-suffixed identity. So the provider would create `…-mod-cold` and the consumer would be
// configured against `…-mod`: it would authenticate successfully and be refused on every object,
// which is exactly the fault this whole record exists to end.
//
// Refused, loudly, at the one place that can see both halves. Lifting it means giving the
// consumer's side a local dimension, which is a decision and not an omission.
func TestAConsumerWithSeveralHoldersOfADerivingProviderIsRefused(t *testing.T) {
m := files()
// Two holders of the one provision, the shape ADR 0094 gives a module that keeps several.
m.Secrets = nil
m.SecretsMany = map[string]map[string]string{"s3-bucket": {
"hot": "/var/lib/files/hot.secret",
"cold": "/var/lib/files/cold.secret",
}}
m.Resources = []map[string]any{{
"id": "env", "type": "file", "path": "/var/lib/files/env", "mode": "0600",
"content": "BUCKET=${bound:s3-bucket:bucket}\n",
}}
r, err := Resolve(shelf(store(), m), []string{"store", "files"}, reachable(), World{})
if err != nil {
t.Fatal(err)
}
_, err = r.Declaration(Rendering{Grants: []Grant{
{Provision: "s3-bucket", Consumer: "workstation", From: "files", Slug: "files",
Local: "hot", Values: map[string]any{}, Sealed: "c2VhbGVk"},
{Provision: "s3-bucket", Consumer: "workstation", From: "files", Slug: "files",
Local: "cold", Values: map[string]any{}, Sealed: "c2VhbGVk"},
}})
if err == nil {
t.Fatal("a consumer with several holders of a deriving provider was accepted; " +
"its two ends would have disagreed in silence")
}
for _, want := range []string{"files", "s3-bucket", "bucket"} {
if !strings.Contains(err.Error(), want) {
t.Errorf("the refusal does not name %q: %v", want, err)
}
}
}
+23 -17
View File
@@ -1,6 +1,7 @@
package catalogue
import (
"encoding/json"
"fmt"
"os"
"reflect"
@@ -170,7 +171,7 @@ func TestTheForgeHoldsTheNpmAndGitSeats(t *testing.T) {
// **And the forge's own address follows it**, composed from the manifest in the catalogue beside
// this checkout (novox/hq 04-ISSUES/088).
//
// The forge is reached a third way that neither test above covers: by its own sidecar, over the
// The forge is reached a third way that neither test above covers: by its own code, over the
// machine's loopback, told where to go in its environment. The `2999:3000` mapping that lets the
// forge go on binding 3000 does nothing for a caller dialling the machine — so a literal there is
// wrong on every node whose assignment differs, and wrong for a second reason on a node given the
@@ -178,13 +179,13 @@ func TestTheForgeHoldsTheNpmAndGitSeats(t *testing.T) {
// in an `env` at all is a declaration, not a manifest.
func TestTheForgesOwnAddressFollowsThePortTheNodeGaveIt(t *testing.T) {
forge, err := catalogueManifest(t, "gitea").Resolve([]Built{{
Name: "runtime", Kind: ArtifactImage,
Reference: "registry.example/gitea-runtime@sha256:" + strings.Repeat("a", 64),
Name: "code", Kind: ArtifactBundle,
Reference: ArtifactStoreScheme + "gitea/code/blobs/" + bundleDigest, Digest: bundleDigest,
}})
if err != nil {
t.Fatalf("the forge's manifest does not resolve against its own build: %v", err)
}
r := Resolution{Node: "anchor", Modules: []Manifest{forge}, Needs: []Needed{
r := Resolution{Node: "anchor", Modules: []Manifest{forge, theRuntime(t)}, Needs: []Needed{
{Name: "postgres-database", For: "gitea", From: "anchor", At: "127.0.0.1",
Serves: map[string]any{"port": float64(5432)}, Sealed: "sealed-db"},
{Name: "route", For: "gitea", From: "anchor"},
@@ -194,8 +195,8 @@ func TestTheForgesOwnAddressFollowsThePortTheNodeGaveIt(t *testing.T) {
// The number this node was given for the forge — the one the machine it is about to run on
// already publishes.
out, err := r.Declaration(Rendering{
Needed: map[string]map[string]string{"gitea": {"broker": "sealed-broker"}},
out, err := r.Declaration(Rendering{ArtifactStore: "anchor.internal:5101",
Needed: map[string]map[string]string{RuntimeModule: {"broker": "sealed-broker"}},
Given: map[string]map[int]int{"gitea": {3000: 2999}},
})
if err != nil {
@@ -210,14 +211,19 @@ func TestTheForgesOwnAddressFollowsThePortTheNodeGaveIt(t *testing.T) {
if published := fmt.Sprint(server["ports"]); !strings.Contains(published, "2999:3000") {
t.Fatalf("the forge is not published on the port this node gave it: %v", server["ports"])
}
runtime := fileNamed(out, "gitea.runtime")
// The forge's own code runs in the node's runtime (novox/hq ADR 0198), given its words there.
runtime := fileNamed(out, RuntimeModule+"."+RuntimeProcessID())
if runtime == nil {
t.Fatalf("the forge's sidecar is not in the declaration: %v", out)
t.Fatalf("the node's runtime is not in the declaration: %v", ids(out))
}
env, _ := runtime["env"].(map[string]any)
if env["MESH_GITEA_URL"] != "http://127.0.0.1:2999" {
t.Fatalf("the forge's sidecar dials %v while the machine publishes the forge on 2999 — "+
"whatever reads it dials a dead port", env["MESH_GITEA_URL"])
env, _ := runtime["env"].(map[string]string)
var given map[string]map[string]string
if err := json.Unmarshal([]byte(env[RuntimeToolEnv]), &given); err != nil {
t.Fatalf("the runtime's %s is not JSON: %q", RuntimeToolEnv, env[RuntimeToolEnv])
}
if given["gitea"]["MESH_GITEA_URL"] != "http://127.0.0.1:2999" {
t.Fatalf("the forge's code dials %v while the machine publishes the forge on 2999 — "+
"whatever reads it dials a dead port", given["gitea"]["MESH_GITEA_URL"])
}
}
@@ -229,13 +235,13 @@ func declaredGiteaSsh(t *testing.T, given map[int]int) map[string]any {
t.Helper()
forge := catalogueManifest(t, "gitea")
resolved, err := forge.Resolve([]Built{{
Name: "runtime", Kind: ArtifactImage,
Reference: "registry.example/gitea-runtime@sha256:" + strings.Repeat("a", 64),
Name: "code", Kind: ArtifactBundle,
Reference: ArtifactStoreScheme + "gitea/code/blobs/" + bundleDigest, Digest: bundleDigest,
}})
if err != nil {
t.Fatalf("the forge's manifest does not resolve against its own build: %v", err)
}
r := Resolution{Node: "anchor", Modules: []Manifest{resolved}, Needs: []Needed{
r := Resolution{Node: "anchor", Modules: []Manifest{resolved, theRuntime(t)}, Needs: []Needed{
{Name: "postgres-database", For: "gitea", From: "anchor", At: "127.0.0.1",
Serves: map[string]any{"port": float64(5432)}, Sealed: "sealed-db"},
{Name: "route", For: "gitea", From: "anchor"},
@@ -246,8 +252,8 @@ func declaredGiteaSsh(t *testing.T, given map[int]int) map[string]any {
for k, v := range given {
givenPorts[k] = v
}
out, err := r.Declaration(Rendering{
Needed: map[string]map[string]string{"gitea": {"broker": "sealed-broker"}},
out, err := r.Declaration(Rendering{ArtifactStore: "anchor.internal:5101",
Needed: map[string]map[string]string{RuntimeModule: {"broker": "sealed-broker"}},
Ports: map[string]map[int]int{"gitea": givenPorts},
Given: map[string]map[int]int{"gitea": given},
})
+29
View File
@@ -0,0 +1,29 @@
package catalogue
import (
"os"
"regexp"
"testing"
)
// novox/hq issue 223: genesis raises the controller as a container built from this repository's own
// Dockerfile, with no build arguments — the manifest no longer builds an image, so nothing passes a
// base in. The Dockerfile's own default must therefore be a Go that builds this module, pinned by
// digest, and the replacement the manifest's process names must be the container genesis raises.
func TestGenesisCanBuildTheControllersImageAsItStands(t *testing.T) {
raw, err := os.ReadFile("../../Dockerfile")
if err != nil {
t.Fatal(err)
}
if !regexp.MustCompile(`(?m)^ARG GO_BASE=golang@sha256:[0-9a-f]{64}$`).Match(raw) {
t.Fatal("the Dockerfile's default Go base is not pinned by digest; genesis builds it with no arguments")
}
makefile, err := os.ReadFile("../../Makefile")
if err != nil {
t.Fatal(err)
}
pin := regexp.MustCompile(`golang@sha256:[0-9a-f]{64}`)
if string(pin.Find(raw)) != string(pin.Find(makefile)) {
t.Errorf("the Dockerfile and the Makefile build on different Go: %s, %s", pin.Find(raw), pin.Find(makefile))
}
}
+143 -1
View File
@@ -325,6 +325,15 @@ type Manifest struct {
// person's account (design 25 §7) already had the same shape.
Invokes []string `json:"invokes,omitempty"`
// State is the current state this module keeps on the bus, by local name: each a key-value
// bucket the controller creates, which every instance of the module writes and reads
// (novox/hq ADR 0201). Not history — that is an event — and never a secret, sealed or not.
State []StateDeclaration `json:"state,omitempty"`
// Reads are other modules' state this module reads and watches, each `<module>.<name>`
// (novox/hq ADR 0201). Read-only: only the owner's instances write.
Reads []string `json:"reads,omitempty"`
// Capabilities the machine must have. A different field from Requires because the remedy
// differs: a missing module can be assigned, and a missing capability means the wrong
// machine.
@@ -757,6 +766,11 @@ type Artifact struct {
// list twice. A module declaring no tools has nothing the runtime loads, whatever it compiles.
Loads []string `json:"loads,omitempty"`
// External are packages a TypeScript bundle keeps as imports rather than inlining — a native
// addon, a package that reads its own files — and so carries the toolchain's node_modules for
// (novox/hq ADR 0193). Absent for nearly every bundle, which is then one file per entrypoint.
External []string `json:"external,omitempty"`
// Env is what a tools bundle is given on a machine (novox/hq ADR 0192): words and their values,
// paths and constants composed with ${dir:…} and ${port:…} exactly as a container's environment
// is, never a secret's content. The node's runtime hands it to this bundle and to no other.
@@ -1311,6 +1325,8 @@ func ParseManifest(raw []byte) (Manifest, error) {
// module whose event names are wrong installs, starts, connects and reacts to nothing, with
// every log line saying it is fine (novox/hq 04-ISSUES/127).
problems = append(problems, EventProblems(m)...)
// And what it may call its state, and whose it may read (state.go, novox/hq ADR 0201).
problems = append(problems, StateProblems(m)...)
wellFormed := true
for _, c := range m.Claims {
if !name.MatchString(c.Name) {
@@ -1423,6 +1439,10 @@ func ParseManifest(raw []byte) (Manifest, error) {
"%s serves %q to whoever requires it, and does not provide it", m.Module, to))
}
}
// A served value may be derived for the consumer it is served to (novox/hq ADR 0202). Read
// here, where the definition is, rather than when somebody first requires it: a rule that
// would be refused at the first consumer is wrong from the moment it is written.
problems = append(problems, CheckServes(m)...)
for to, where := range m.Binds {
if !placedOrAbsolute(where) {
problems = append(problems, fmt.Sprintf(
@@ -1510,6 +1530,53 @@ func ParseManifest(raw []byte) (Manifest, error) {
"program that reads what the mesh delivered and reconciles",
m.Module, r["id"]))
}
// **What a process replaces is something the module no longer declares** (novox/hq issue 213).
// The host keeps it running until the process is, then removes it: so it is named by the id the
// module used to give it, it is never a resource the module still declares — that would be
// applied and removed by one declaration — and only a process that stays up has anything to
// hand over to. Said here, near the author, as the host would refuse it far away.
ids := map[string]bool{}
for _, r := range m.Resources {
ids[fmt.Sprint(r["id"])] = true
}
for _, r := range m.Resources {
raw, present := r["replaces"]
if !present {
continue
}
if fmt.Sprint(r["type"]) != "process" {
problems = append(problems, fmt.Sprintf(
"%s: %v says what it replaces, and only a process does", m.Module, r["id"]))
continue
}
if once, _ := r["run-once"].(bool); once || r["schedule"] != nil {
problems = append(problems, fmt.Sprintf(
"%s: %v replaces something and runs once or on a schedule — only a process that stays "+
"up is there a moment later to hand over to", m.Module, r["id"]))
}
list, ok := raw.([]any)
if !ok {
problems = append(problems, fmt.Sprintf(
"%s: %v replaces %v; replaces is a list of the ids this module no longer declares",
m.Module, r["id"], raw))
continue
}
for _, item := range list {
id, ok := item.(string)
switch {
case !ok || strings.TrimSpace(id) == "":
problems = append(problems, fmt.Sprintf(
"%s: %v replaces %v, which is not an id", m.Module, r["id"], item))
case strings.Contains(id, "."):
problems = append(problems, fmt.Sprintf(
"%s: %v replaces %q; a process replaces only a resource of its own module, named "+
"by its own id", m.Module, r["id"], id))
case ids[id]:
problems = append(problems, fmt.Sprintf(
"%s: %v replaces %q, which this module still declares", m.Module, r["id"], id))
}
}
}
// **A module that prepares its state must have code the mesh can run** (novox/hq ADR 0135). The
// preparation is the module's own program in its preparation mode, so it is derived from the
// resource that runs that program — and a module declaring none has asked for something the mesh
@@ -1517,7 +1584,7 @@ func ParseManifest(raw []byte) (Manifest, error) {
// quietly prepares nothing.
if m.Prepares && preparationTarget(m) == "" {
problems = append(problems, fmt.Sprintf(
"%s says it prepares its state, and declares no container running an artifact it built — "+
"%s says it prepares its state, and declares no container or process running an artifact it built — "+
"the preparation is this module's own program, so there has to be one for the mesh to "+
"run it in", m.Module))
}
@@ -1570,6 +1637,13 @@ func ParseManifest(raw []byte) (Manifest, error) {
}
}
}
// **A scheduled step may hold this module's own containers still while it runs**
// (novox/hq ADR 0189). What the host judges is the declaration it receives — whether each
// id is a container placed on that machine; what belongs here is what only the definition
// shows: that the ids are this module's, that they are containers, and that the step is
// scheduled. A module naming a neighbour's container would be a module that can stop the
// mesh, and the manifest is where that is visible.
problems = append(problems, whileStoppedProblems(m, r, hasSchedule(r))...)
}
for name, own := range m.OwnSecrets {
if !placedOrAbsolute(own.Path) {
@@ -2101,3 +2175,71 @@ func (o OwnSecrets) Paths() map[string]string {
// InstancesInterchangeable is the one value of a definition's `instances`: the module is the same
// on every machine, so any instance may answer for the module.
const InstancesInterchangeable = "interchangeable"
// WhileStopped is the resource key naming the containers a scheduled step holds still while it
// runs (novox/hq ADR 0189). Carried to the host unchanged, like `schedule`.
const WhileStopped = "while-stopped"
// hasSchedule is whether a resource declares a cadence, as a string.
func hasSchedule(r map[string]any) bool {
s, _ := r["schedule"].(string)
return s != ""
}
// whileStoppedProblems judges one container's maintenance window against its own definition
// (novox/hq ADR 0189).
//
// Three things the manifest is the only place to see: that the step is scheduled (a one-time
// offline job says *before* rather than *instead of* — at apply the host already has a window,
// because the declaration is applied in order and a run-once step gates what follows); that every
// id it names is **this module's own** container; and that it does not name itself.
//
// The host checks the fourth — that the container is actually placed on that machine — because
// that is a fact about the declaration and not about the definition.
func whileStoppedProblems(m Manifest, r map[string]any, scheduled bool) []string {
raw, present := r[WhileStopped]
if !present {
return nil
}
ids, ok := raw.([]any)
if !ok {
return []string{fmt.Sprintf(
"%s declares %s on %v as a %T; it is a list of this module's container ids",
m.Module, WhileStopped, r["id"], raw)}
}
var problems []string
if len(ids) > 0 && !scheduled {
problems = append(problems, fmt.Sprintf(
"%s declares %s on %v, which has no schedule. A maintenance window is for a recurring "+
"step: at apply the mesh already has one, because a run-once step gates what is "+
"declared after it (novox/hq ADR 0189)", m.Module, WhileStopped, r["id"]))
}
containers := map[string]bool{}
for _, own := range m.Resources {
if fmt.Sprint(own["type"]) == "container" {
containers[fmt.Sprint(own["id"])] = true
}
}
for _, each := range ids {
id, ok := each.(string)
if !ok {
problems = append(problems, fmt.Sprintf(
"%s declares %s on %v naming a %T; each entry is a container's id",
m.Module, WhileStopped, r["id"], each))
continue
}
if id == fmt.Sprint(r["id"]) {
problems = append(problems, fmt.Sprintf(
"%s declares %s on %v naming itself", m.Module, WhileStopped, r["id"]))
continue
}
if !containers[id] {
problems = append(problems, fmt.Sprintf(
"%s declares %s on %v naming %q, which is not a container this module declares. "+
"A step may hold still its own module's containers and nobody else's — one "+
"that could quiesce a neighbour could stop the mesh",
m.Module, WhileStopped, r["id"], id))
}
}
return problems
}
+10 -5
View File
@@ -31,7 +31,7 @@ import (
//
// So a module asks. `${port:8080}` is "the machine-side port you gave me for the 8080 I said I
// listen on", and the module writes that where it would otherwise have written a literal — in a
// file's content, or in a value of a container's `env`.
// file's content, or in a value of a container's or a process's `env`.
//
// **The environment is filled by the control plane, exactly as a bound value is.** A port is not
// secret — the mesh holds it in the clear — so there is nothing for the host to be the only
@@ -64,7 +64,12 @@ func portsUsed(content string) []int {
}
// portInto replaces a resource's ${port:…} placeholders with what this machine assigned — in a
// file's content, and in a value of a container's environment.
// file's content, and in a value of a container's or a process's environment.
//
// **A process's environment is a container's** (novox/hq to-be 38 WP4c). A module's code moving out
// of its container becomes a process on the machine and still has to be told what the container
// was told; filled for one kind and not the other, the literal reached the process and was read as
// a port, and the modules that moved first wrote their run-once steps a 0600 env file instead.
//
// A port the module did not say it listens on is refused, for the same reason a binding's unknown
// key is: the module is asking about something it never declared, and the answer would be a guess.
@@ -84,7 +89,7 @@ func portInto(resource map[string]any, module string, listens []Listening, with
}
resource["content"] = filled
case "container":
case "container", "process":
env, ok := resource["env"].(map[string]any)
if !ok {
return nil
@@ -106,8 +111,8 @@ func portInto(resource map[string]any, module string, listens []Listening, with
continue
}
value, err := portsFilledInto(written,
fmt.Sprintf("%s's container %s sets %s to something that",
module, resource["name"], key), module, listens, with)
fmt.Sprintf("%s's %s %s sets %s to something that",
module, resource["type"], resource["name"], key), module, listens, with)
if err != nil {
return err
}
+80
View File
@@ -0,0 +1,80 @@
package catalogue
import (
"strings"
"testing"
)
// **A process's environment is composed as a container's is** (novox/hq to-be 38 WP4c).
//
// A module's code moving out of its container becomes a process on the machine, and what its
// container's environment asked for — the port this machine gave the module, the place it put the
// module's directory — it still has to be told. Filled for a container and not for a process, the
// literal `${port:8080}` reached the process as its environment and was read as a port; the modules
// that moved first wrote their run-once steps an env file instead.
func processModule(env map[string]any) Manifest {
return Manifest{
Module: "showcase",
Listens: []Listening{{Port: 8080, From: FromMesh}},
Resources: []map[string]any{
{"id": "data", "type": "directory", "mode": "0700"},
{"id": "setup", "type": "process", "name": "showcase-setup", "run-once": true,
"run": []any{"/usr/bin/showcase", "setup"}, "env": env},
},
}
}
func TestAProcessIsToldItsPortAndItsPlaceInItsEnvironment(t *testing.T) {
env := map[string]any{
"SHOWCASE_URL": "http://127.0.0.1:${port:8080}",
"SHOWCASE_DATA": "${dir:data}/objects",
"SHOWCASE_DB": "127.0.0.1:${seat:mesh-store:5432}",
"GREETING": "hello",
}
out, err := Resolution{Node: "anchor", Modules: []Manifest{processModule(env)}}.Declaration(Rendering{
Ports: map[string]map[int]int{"showcase": {8080: 21000}},
Seats: map[string]map[int]int{"mesh-store": {5432: 6852}},
})
if err != nil {
t.Fatalf("a process asking for its port and its place does not compose: %v", err)
}
setup := fileNamed(out, "showcase.setup")
if setup == nil {
t.Fatalf("the process is not in the declaration: %v", out)
}
got, _ := setup["env"].(map[string]any)
for key, want := range map[string]string{
"SHOWCASE_URL": "http://127.0.0.1:21000",
"SHOWCASE_DATA": "/var/lib/showcase/data/objects",
"SHOWCASE_DB": "127.0.0.1:6852",
"GREETING": "hello",
} {
if got[key] != want {
t.Errorf("the process is told %s=%v, want %q", key, got[key], want)
}
}
if env["SHOWCASE_URL"] != "http://127.0.0.1:${port:8080}" {
t.Fatalf("composing for one machine edited the module's own manifest: %v", env)
}
}
// An unknown reference in a process's environment is refused as a container's is, naming the
// process and the variable — left alone, it would reach the machine as a literal.
func TestAProcessAskingAboutAnUndeclaredPortIsRefused(t *testing.T) {
env := map[string]any{"SHOWCASE_URL": "http://127.0.0.1:${port:9999}"}
_, err := Resolution{Node: "anchor", Modules: []Manifest{processModule(env)}}.Declaration(Rendering{})
if err == nil {
t.Fatal("a process was told a port its module never said it listens on")
}
for _, said := range []string{"showcase-setup", "SHOWCASE_URL", "${port:9999}", "8080"} {
if !strings.Contains(err.Error(), said) {
t.Errorf("the refusal does not say %q: %v", said, err)
}
}
env = map[string]any{"SHOWCASE_DATA": "${dir:date}/objects"}
if _, err := (Resolution{Node: "anchor", Modules: []Manifest{processModule(env)}}).Declaration(Rendering{}); err == nil ||
!strings.Contains(err.Error(), "${dir:date}") {
t.Fatalf("a process naming no directory of its module was not refused: %v", err)
}
}
+5 -5
View File
@@ -43,8 +43,8 @@ import (
var ofSeat = regexp.MustCompile(`\$\{seat:([a-z0-9][a-z0-9-]*):([0-9]+)\}`)
// seatInto replaces a resource's ${seat:…} placeholders with where this machine put each seat's
// holder — in a file's content, and in a value of a container's environment. The same two places
// portInto fills, for the same reason: they are where a process reads a number from.
// holder — in a file's content, and in a value of a container's or a process's environment. The
// same places portInto fills, for the same reason: they are where a program reads a number from.
func seatInto(resource map[string]any, module string, with Rendering) error {
switch fmt.Sprint(resource["type"]) {
case "file":
@@ -58,7 +58,7 @@ func seatInto(resource map[string]any, module string, with Rendering) error {
}
resource["content"] = filled
case "container":
case "container", "process":
env, ok := resource["env"].(map[string]any)
if !ok {
return nil
@@ -78,8 +78,8 @@ func seatInto(resource map[string]any, module string, with Rendering) error {
continue
}
value, err := seatsFilledInto(written,
fmt.Sprintf("%s's container %s sets %s to something that",
module, resource["name"], key), with)
fmt.Sprintf("%s's %s %s sets %s to something that",
module, resource["type"], resource["name"], key), with)
if err != nil {
return err
}
+10 -9
View File
@@ -101,7 +101,7 @@ func TestTheControlPlanesOwnAddressesFollowTheNodesPorts(t *testing.T) {
}
// The manifest itself names them now; withSeatPorts is a no-op on it, and this holds it so.
for _, r := range m.Resources {
if r["type"] != "container" {
if r["type"] != "process" {
continue
}
env, _ := r["env"].(map[string]any)
@@ -113,8 +113,9 @@ func TestTheControlPlanesOwnAddressesFollowTheNodesPorts(t *testing.T) {
}
m = withSeatPorts(m)
control, err := m.Resolve([]Built{{
Name: "server", Kind: ArtifactImage,
Reference: ArtifactStoreScheme + "mesh-controller/server@sha256:" + strings.Repeat("c", 64),
Name: "controller", Kind: ArtifactBundle,
Reference: ArtifactStoreScheme + "mesh-controller/controller@sha256:" + strings.Repeat("c", 64),
Digest: "sha256:" + strings.Repeat("c", 64),
}})
if err != nil {
t.Fatal(err)
@@ -135,9 +136,9 @@ func TestTheControlPlanesOwnAddressesFollowTheNodesPorts(t *testing.T) {
if err != nil {
t.Fatalf("the control plane does not compose: %v", err)
}
server := fileNamed(out, "mesh-controller.server")
server := fileNamed(out, "mesh-controller.controller")
if server == nil {
t.Fatalf("the control plane's container is not in the declaration: %v", out)
t.Fatalf("the control plane's process is not in the declaration: %v", out)
}
env, _ := server["env"].(map[string]any)
for key, want := range map[string]string{
@@ -151,8 +152,8 @@ func TestTheControlPlanesOwnAddressesFollowTheNodesPorts(t *testing.T) {
t.Errorf("the control plane is told %s=%v; the node put it on %s", key, env[key], want)
}
}
if got := server["image"]; got != "anchor.internal:5100/mesh-controller/server@sha256:"+strings.Repeat("c", 64) {
t.Errorf("the control plane's own image is %v, not routed through the store", got)
if got := server["source"]; got != "anchor.internal:5100/mesh-controller/controller@sha256:"+strings.Repeat("c", 64) {
t.Errorf("the control plane's own bundle is fetched from %v, not routed through the store", got)
}
// And on a mesh where the foundation is where genesis raised it, nothing is added.
@@ -160,7 +161,7 @@ func TestTheControlPlanesOwnAddressesFollowTheNodesPorts(t *testing.T) {
if err != nil {
t.Fatal(err)
}
env, _ = fileNamed(out, "mesh-controller.server")["env"].(map[string]any)
env, _ = fileNamed(out, "mesh-controller.controller")["env"].(map[string]any)
if env["MESH_STORE_INVENTORY_PORT"] != "" {
t.Errorf("with no settings, the control plane is told %v", env)
}
@@ -186,7 +187,7 @@ func withSeatPorts(m Manifest) Manifest {
out := m
out.Resources = nil
for _, r := range m.Resources {
if r["type"] != "container" {
if r["type"] != "container" && r["type"] != "process" {
out.Resources = append(out.Resources, r)
continue
}
+7
View File
@@ -215,6 +215,13 @@ func CatalogueProblems(shelf Shelf) []string {
}
}
}
// A read of a module's state that module does not keep (novox/hq ADR 0201) — said only where the
// owner is on the shelf, as a consumer may be installed before its emitter.
var manifests []Manifest
for _, module := range shelfOrder(shelf) {
manifests = append(manifests, shelf[module])
}
problems = append(problems, StateReadsNothingDeclares(manifests)...)
sort.Strings(problems)
return problems
}
+150
View File
@@ -0,0 +1,150 @@
package catalogue
import (
"bytes"
"encoding/json"
"fmt"
"regexp"
"strings"
)
// What a module may call its state, and whose state it may ask to read (novox/hq ADR 0201).
//
// A module names its state **locally** — `servers`, never a bucket or a subject — and another
// module's as `<module>.<name>`, the way a consumed event names its emitter (design 32 §1). The
// mesh derives the bucket from the two names, so the module and the local name must each be one
// token: the bucket joins them with an underscore, which neither may contain, so two modules can
// never derive one bucket.
// stateName is one local name of a module's state: lower-case, no dot, no underscore.
var stateName = regexp.MustCompile(`^[a-z0-9][a-z0-9-]*$`)
// The mesh's caps on what a module may ask of a bucket's history.
const (
// StateMostHistory is the most past values a key may keep. The server's own limit.
StateMostHistory = 64
)
// StateDeclaration is one bucket a module owns: its local name, and the options that are the
// owner's to choose, as a seat chooses how long its backlog survives (design 32 §3).
type StateDeclaration struct {
Name string `json:"name"`
// History is how many values a key keeps, the current one included; zero is one.
History int `json:"history,omitempty"`
// TTLSeconds is how long a value lives once written; zero is until it is replaced or deleted.
TTLSeconds int `json:"ttl-seconds,omitempty"`
}
// UnmarshalJSON reads a bucket as its bare name, or as {name, history, ttl-seconds}.
func (s *StateDeclaration) UnmarshalJSON(raw []byte) error {
trimmed := bytes.TrimSpace(raw)
if len(trimmed) > 0 && trimmed[0] == '"' {
return json.Unmarshal(trimmed, &s.Name)
}
type plain StateDeclaration
var full plain
dec := json.NewDecoder(bytes.NewReader(trimmed))
dec.DisallowUnknownFields()
if err := dec.Decode(&full); err != nil {
return fmt.Errorf("a state is either a name or {name, history, ttl-seconds}: %w", err)
}
*s = StateDeclaration(full)
return nil
}
// MarshalJSON writes back the short form when there is nothing else to say.
func (s StateDeclaration) MarshalJSON() ([]byte, error) {
if s.History == 0 && s.TTLSeconds == 0 {
return json.Marshal(s.Name)
}
type plain StateDeclaration
return json.Marshal(plain(s))
}
// ReadState splits a read into the owning module and the local name, or says why it is not one.
func ReadState(read string) (module, local string, err error) {
at := strings.LastIndex(read, ".")
if at <= 0 || at == len(read)-1 {
return "", "", fmt.Errorf("%q does not name a module and its state: a read is <module>.<name>", read)
}
module, local = read[:at], read[at+1:]
if !stateName.MatchString(module) {
return "", "", fmt.Errorf("%q cannot own state: a module whose state is read is one plain name", module)
}
if !stateName.MatchString(local) {
return "", "", fmt.Errorf("%q is not a state name: lower-case letters, digits and hyphens", local)
}
return module, local, nil
}
// StateProblems is what is wrong with a manifest's state and reads.
//
// Refused at registration, because a bucket name the bus cannot hold is a module that installs,
// starts, and is refused on its first write with a reason about a bucket nobody named.
func StateProblems(m Manifest) []string {
var problems []string
if len(m.State) > 0 && !stateName.MatchString(m.Module) {
problems = append(problems, fmt.Sprintf(
"%s keeps state, and a module's name is part of its buckets' names, which take one plain "+
"name — no dot (novox/hq ADR 0201)", m.Module))
}
seen := map[string]bool{}
for _, s := range m.State {
switch {
case !stateName.MatchString(s.Name):
problems = append(problems, fmt.Sprintf(
"%s keeps state %q: a state is named locally — lower-case letters, digits and hyphens, "+
"no dot and no underscore; the mesh derives the bucket (novox/hq ADR 0201)", m.Module, s.Name))
case seen[s.Name]:
problems = append(problems, fmt.Sprintf("%s keeps state %q twice", m.Module, s.Name))
}
seen[s.Name] = true
if s.History < 0 || s.History > StateMostHistory {
problems = append(problems, fmt.Sprintf(
"%s keeps %d values of %q; a key keeps between 1 and %d", m.Module, s.History, s.Name, StateMostHistory))
}
if s.TTLSeconds < 0 {
problems = append(problems, fmt.Sprintf("%s gives %q a negative lifetime", m.Module, s.Name))
}
}
for _, r := range m.Reads {
module, _, err := ReadState(r)
if err != nil {
problems = append(problems, fmt.Sprintf("%s reads %v", m.Module, err))
continue
}
if module == m.Module {
problems = append(problems, fmt.Sprintf(
"%s reads %q, which is its own state: a module reads and writes what it keeps already", m.Module, r))
}
}
return problems
}
// StateReadsNothingDeclares is every read across a catalogue whose owner is present and declares no
// such state. An absent owner says nothing — a module may be installed long before the one whose
// state it reads, as a consumer may before its emitter (design 32 §1).
func StateReadsNothingDeclares(manifests []Manifest) []string {
declared := map[string]map[string]bool{}
for _, m := range manifests {
own := map[string]bool{}
for _, s := range m.State {
own[s.Name] = true
}
declared[m.Module] = own
}
var problems []string
for _, m := range manifests {
for _, r := range m.Reads {
module, local, err := ReadState(r)
if err != nil {
continue
}
if own, present := declared[module]; present && !own[local] {
problems = append(problems, fmt.Sprintf(
"%s reads %q, and %s keeps no state called %q", m.Module, r, module, local))
}
}
}
return problems
}
+94
View File
@@ -0,0 +1,94 @@
package catalogue
import (
"encoding/json"
"strings"
"testing"
)
// A module declares the state it keeps and the state it reads (novox/hq ADR 0201), a bucket by its
// bare name or with the owner's options.
func TestAManifestMaySayWhatStateItKeepsAndReads(t *testing.T) {
m, err := ParseManifest([]byte(`{"module":"claude-code","version":"1",` +
`"state":["servers",{"name":"seen","history":5,"ttl-seconds":3600}],` +
`"reads":["licence-manager.bindings"]}`))
if err != nil {
t.Fatal(err)
}
if len(m.State) != 2 || m.State[0].Name != "servers" || m.State[1].History != 5 || m.State[1].TTLSeconds != 3600 {
t.Fatalf("state not read: %+v", m.State)
}
if len(m.Reads) != 1 || m.Reads[0] != "licence-manager.bindings" {
t.Fatalf("reads not read: %v", m.Reads)
}
// Written back as it came in: the short form where nothing else is said.
out, _ := json.Marshal(m.State)
if string(out) != `["servers",{"name":"seen","history":5,"ttl-seconds":3600}]` {
t.Fatalf("written back as %s", out)
}
}
// A name the bus could not hold, or that would let two modules derive one bucket, is refused at
// registration in the manifest's words.
func TestAStateNameIsLocalAndOneToken(t *testing.T) {
for _, c := range []struct{ manifest, says string }{
{`{"module":"a","version":"1","state":["mesh.servers"]}`, `keeps state "mesh.servers": a state is named locally`},
{`{"module":"a","version":"1","state":["my_servers"]}`, `keeps state "my_servers"`},
{`{"module":"a","version":"1","state":["s","s"]}`, `keeps state "s" twice`},
{`{"module":"a","version":"1","state":[{"name":"s","history":65}]}`, `a key keeps between 1 and 64`},
{`{"module":"a.b","version":"1","state":["s"]}`, `no dot`},
{`{"module":"a","version":"1","reads":["bindings"]}`, `a read is <module>.<name>`},
{`{"module":"a","version":"1","reads":["a.s"]}`, `which is its own state`},
{`{"module":"a","version":"1","state":[{"name":"s","shared":true}]}`, `{name, history, ttl-seconds}`},
} {
_, err := ParseManifest([]byte(c.manifest))
if err == nil {
t.Errorf("%s was accepted", c.manifest)
continue
}
if !strings.Contains(err.Error(), c.says) {
t.Errorf("%s refused for the wrong reason: %v", c.manifest, err)
}
}
}
// A read whose owner is present must name a state that owner keeps; an absent owner says nothing,
// because a module may be installed before the one whose state it reads.
func TestAReadNamesStateItsOwnerKeeps(t *testing.T) {
owner := Manifest{Module: "licence-manager", State: []StateDeclaration{{Name: "bindings"}}}
good := Manifest{Module: "claude-code", Reads: []string{"licence-manager.bindings", "absent.anything"}}
bad := Manifest{Module: "other", Reads: []string{"licence-manager.tokens"}}
if p := StateReadsNothingDeclares([]Manifest{owner, good}); len(p) != 0 {
t.Fatalf("a read of declared state was refused: %v", p)
}
p := StateReadsNothingDeclares([]Manifest{owner, bad})
if len(p) != 1 || !strings.Contains(p[0], `licence-manager keeps no state called "tokens"`) {
t.Fatalf("a read of state nobody keeps was not named: %v", p)
}
}
// **Across the whole catalogue**: every state name is local, and every read whose owner is present
// names state that owner keeps.
func TestEveryManifestsStateIsLocalAndEveryReadIsKept(t *testing.T) {
manifests := theCatalogue(t)
var problems []string
for _, m := range manifests {
problems = append(problems, StateProblems(m)...)
}
problems = append(problems, StateReadsNothingDeclares(manifests)...)
if len(problems) > 0 {
t.Fatalf("the catalogue's state is not what ADR 0201 says:\n %s", strings.Join(problems, "\n "))
}
}
// `module check` says it too: the cross-catalogue pass names a read nothing on the shelf keeps.
func TestTheCataloguePassNamesAReadItsOwnerDoesNotKeep(t *testing.T) {
shelf := Shelf{
"licence-manager": {Module: "licence-manager", State: []StateDeclaration{{Name: "bindings"}}},
"claude-code": {Module: "claude-code", Reads: []string{"licence-manager.tokens"}},
}
problems := CatalogueProblems(shelf)
if len(problems) != 1 || !strings.Contains(problems[0], `keeps no state called "tokens"`) {
t.Fatalf("the catalogue pass said %v", problems)
}
}
+145
View File
@@ -0,0 +1,145 @@
package catalogue
import (
"encoding/json"
"fmt"
"strings"
"testing"
)
// A scheduled step may hold its module's own containers still while it runs (novox/hq ADR 0189).
//
// The host judges what it receives — whether each id is a container on that machine. What the
// definition is the only place to see is judged here, near whoever wrote it.
func aStoreManifest(step map[string]any) []byte {
m := map[string]any{
"module": "distribution", "version": "1",
"resources": []any{
map[string]any{"id": "store", "type": "container", "name": "mesh-registry",
"image": "registry@sha256:" + strings.Repeat("a", 64)},
step,
},
}
raw, _ := json.Marshal(m)
return raw
}
func TestAMaintenanceWindowOnItsOwnModulesContainerIsAccepted(t *testing.T) {
raw := aStoreManifest(map[string]any{
"id": "collect", "type": "container", "name": "mesh-registry-collect",
"image": "registry@sha256:" + strings.Repeat("a", 64),
"schedule": "30 3 * * *", "while-stopped": []any{"store"},
})
if _, err := ParseManifest(raw); err != nil {
t.Fatalf("a step holding its own module's container still was refused: %v", err)
}
}
func TestAMaintenanceWindowIsRefusedWhereTheDefinitionShowsItCannotMean(t *testing.T) {
for _, c := range []struct {
name string
step map[string]any
says string
}{
{
"on a step with no schedule",
map[string]any{"id": "collect", "type": "container", "name": "c",
"image": "registry@sha256:" + strings.Repeat("a", 64),
"while-stopped": []any{"store"}},
"gates what is declared after it",
},
{
"on a run-once step, which already has order",
map[string]any{"id": "collect", "type": "container", "name": "c",
"image": "registry@sha256:" + strings.Repeat("a", 64),
"run-once": true, "while-stopped": []any{"store"}},
"A maintenance window is for a recurring step",
},
{
"naming a container this module does not declare",
map[string]any{"id": "collect", "type": "container", "name": "c",
"image": "registry@sha256:" + strings.Repeat("a", 64),
"schedule": "30 3 * * *", "while-stopped": []any{"the-broker"}},
"could quiesce a neighbour could stop the mesh",
},
{
"naming itself",
map[string]any{"id": "collect", "type": "container", "name": "c",
"image": "registry@sha256:" + strings.Repeat("a", 64),
"schedule": "30 3 * * *", "while-stopped": []any{"collect"}},
"naming itself",
},
{
"written as something that is not a list",
map[string]any{"id": "collect", "type": "container", "name": "c",
"image": "registry@sha256:" + strings.Repeat("a", 64),
"schedule": "30 3 * * *", "while-stopped": "store"},
"a list of this module's container ids",
},
} {
_, err := ParseManifest(aStoreManifest(c.step))
if err == nil {
t.Errorf("%s was accepted", c.name)
continue
}
if !strings.Contains(err.Error(), c.says) {
t.Errorf("%s: the refusal does not say %q:\n%v", c.name, c.says, err)
}
}
}
// A composed declaration names the step's held containers the way the machine knows them.
//
// **The gap that let a bug through to the control node.** The manifest says `while-stopped:
// ["store"]`, because a module names its own resources locally; the declaration a machine
// receives calls that container `distribution.store`, because every resource is composed under
// its module. `restart-on` and `reload-on` are rewritten for exactly this reason, and
// `while-stopped` was not — so the host found no container by that id and refused the whole
// declaration, every push, until it was fixed.
//
// It passed every test on both sides: the controller's tests read manifests, the host's read
// hand-written declarations with bare ids. Only composing one and judging the result catches it.
func TestAComposedWindowNamesTheContainerAsTheMachineKnowsIt(t *testing.T) {
store := Manifest{
Module: "distribution", Version: "1",
Provides: FromAnywhere("artifact-store"),
Listens: []Listening{{Port: 5000, Protocol: "tcp", From: FromMesh}},
Serves: map[string]map[string]any{"artifact-store": {"port": 5000}},
Resources: []map[string]any{
{"id": "store", "type": "container", "name": "mesh-registry",
"image": "registry@sha256:" + strings.Repeat("a", 64), "ports": []any{"5000"}},
{"id": "collect", "type": "container", "name": "mesh-registry-collect",
"image": "registry@sha256:" + strings.Repeat("a", 64),
"schedule": "30 3 * * *", WhileStopped: []any{"store"}},
},
}
r, err := Resolve(shelf(store), []string{"distribution"}, reachable(), World{})
if err != nil {
t.Fatal(err)
}
out, err := r.Declaration(Rendering{})
if err != nil {
t.Fatal(err)
}
collect := fileNamed(out, "distribution.collect")
if collect == nil {
for _, res := range out {
if res["id"] == "distribution.collect" {
collect = res
}
}
}
if collect == nil {
t.Fatalf("the step was not composed at all: %v", out)
}
held, _ := collect[WhileStopped].([]any)
if len(held) != 1 {
t.Fatalf("the composed step holds %v still; want one container", collect[WhileStopped])
}
if got := fmt.Sprint(held[0]); got != "distribution.store" {
t.Fatalf("the composed step says it holds %q still, and the machine's container is "+
"called %q — the host refuses a declaration naming a container it does not have, "+
"whole, so the machine would take nothing at all", got, "distribution.store")
}
}
+43 -14
View File
@@ -42,9 +42,29 @@ type Build struct {
// Failed is the builder's own words, empty when it worked.
Failed string
Made []Artifact
At time.Time
// Asked is when the build was requested, zero when that is not known (an id of another shape,
// or a build recorded before the mesh kept it). **What orders one build of a module against
// another** (novox/hq 04-ISSUES/219): builds in flight together finish in any order, and the
// one asked last stood on the newest bases.
Asked time.Time
// At is when the outcome was recorded — when it finished, not when it was asked.
At time.Time
}
// AskedOrAt is when the build was asked, or when it was recorded when that is not known — the
// order the mesh had before it kept the request time.
func (b Build) AskedOrAt() time.Time {
if !b.Asked.IsZero() {
return b.Asked
}
return b.At
}
// newestRequestFirst is the ordering every "what a module currently is" question uses: the newest
// request wins, whenever it finished (novox/hq 04-ISSUES/219). A build whose request time is not
// known is placed at the moment it was recorded, which is the rule that held before.
const newestRequestFirst = `coalesce(asked, at) desc, at desc`
// ReadRepository is a repository a build read source from besides the module's own.
type ReadRepository struct {
Repository string `json:"repository"`
@@ -83,13 +103,17 @@ func (i *Inventory) RecordBuild(ctx context.Context, b Build) error {
if b.Module != "" {
module = &b.Module
}
var asked *time.Time
if !b.Asked.IsZero() {
asked = &b.Asked
}
_, err = i.store.Pool().Exec(ctx,
`insert into build (id, repository, ref, module, commit_hash, built_on, failed, made,
source_path, manifest, built_against, built_contexts)
values ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12)
source_path, manifest, built_against, built_contexts, asked)
values ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13)
on conflict (id) do nothing`,
b.ID, b.Repository, b.Ref, module, b.Commit, b.On, b.Failed, made,
b.Path, manifestOrNil(b.Manifest), against, read)
b.Path, manifestOrNil(b.Manifest), against, read, asked)
return err
}
@@ -102,11 +126,11 @@ func (i *Inventory) Builds(ctx context.Context, module string, limit int) ([]Bui
if limit <= 0 {
limit = 20
}
query := `select id, repository, ref, coalesce(module,''), commit_hash, built_on, failed, made, at
query := `select id, repository, ref, coalesce(module,''), commit_hash, built_on, failed, made, asked, at
from build order by at desc limit $1`
args := []any{limit}
if module != "" {
query = `select id, repository, ref, coalesce(module,''), commit_hash, built_on, failed, made, at
query = `select id, repository, ref, coalesce(module,''), commit_hash, built_on, failed, made, asked, at
from build where module = $2 order by at desc limit $1`
args = append(args, module)
}
@@ -121,10 +145,14 @@ func (i *Inventory) Builds(ctx context.Context, module string, limit int) ([]Bui
for rows.Next() {
var b Build
var made []byte
var asked *time.Time
if err := rows.Scan(&b.ID, &b.Repository, &b.Ref, &b.Module, &b.Commit,
&b.On, &b.Failed, &made, &b.At); err != nil {
&b.On, &b.Failed, &made, &asked, &b.At); err != nil {
return nil, err
}
if asked != nil {
b.Asked = *asked
}
if err := json.Unmarshal(made, &b.Made); err != nil {
return nil, err
}
@@ -135,8 +163,9 @@ func (i *Inventory) Builds(ctx context.Context, module string, limit int) ([]Bui
// Held is every artifact this mesh has built, keyed "<module>/<artifact>".
//
// **The newest successful build of each module wins**, which is the same rule the rest of the mesh
// uses for what a module currently is. A module rebuilt to something broken and then rebuilt again
// **The successful build of each module asked last wins**, which is the same rule the rest of the
// mesh uses for what a module currently is — asked last, not finished last (novox/hq
// 04-ISSUES/219): an older request that finishes later stood on older bases. A module rebuilt to something broken and then rebuilt again
// is at the second one; a module whose last build failed is at the last one that worked, because a
// failure published nothing and the thing it published before is still what exists.
//
@@ -147,7 +176,7 @@ func (i *Inventory) Held(ctx context.Context) (map[string]string, error) {
`select distinct on (module) module, made
from build
where module is not null and module <> '' and failed = ''
order by module, at desc`)
order by module, `+newestRequestFirst)
if err != nil {
return nil, err
}
@@ -185,7 +214,7 @@ func (i *Inventory) BuiltAgainst(ctx context.Context) (map[string][]string, erro
`select distinct on (module) module, built_against
from build
where module is not null and module <> '' and failed = ''
order by module, at desc`)
order by module, `+newestRequestFirst)
if err != nil {
return nil, err
}
@@ -223,7 +252,7 @@ func (i *Inventory) ReadRepositories(ctx context.Context) (map[string][]ReadRepo
`select distinct on (module) module, built_contexts
from build
where module is not null and module <> '' and failed = ''
order by module, at desc`)
order by module, `+newestRequestFirst)
if err != nil {
return nil, err
}
@@ -274,7 +303,7 @@ func manifestOrNil(raw []byte) any {
// follows when it decides whether to announce at all.
//
// One row per module and commit: a module built twice at the same commit is one fact, and the
// latest row is the one whose artifacts are current.
// row asked last is the one whose artifacts are current (novox/hq 04-ISSUES/219).
func (i *Inventory) Announceable(ctx context.Context) ([]Build, error) {
rows, err := i.store.Pool().Query(ctx,
`select distinct on (module, commit_hash)
@@ -282,7 +311,7 @@ func (i *Inventory) Announceable(ctx context.Context) ([]Build, error) {
source_path, manifest, built_against, at
from build
where failed = '' and module is not null and module <> '' and commit_hash <> ''
order by module, commit_hash, at desc`)
order by module, commit_hash, `+newestRequestFirst)
if err != nil {
return nil, err
}
+66 -1
View File
@@ -49,6 +49,12 @@ func (i *Inventory) BusRecords(ctx context.Context) (broker.Records, error) {
}
}
// Who holds each seat held once for the mesh, where the mesh recorded it (novox/hq issue 218).
holdings, err := i.Holdings(ctx)
if err != nil {
return broker.Records{}, fmt.Errorf("cannot read who holds the mesh's seats: %w", err)
}
out := broker.Records{Assigned: map[string][]broker.Declared{}, People: map[string][]string{},
Interchangeable: map[string]bool{}}
for _, n := range nodes {
@@ -72,7 +78,9 @@ func (i *Inventory) BusRecords(ctx context.Context) (broker.Records, error) {
"%s is assigned to %s and is not in the catalogue, so what it may say cannot "+
"be derived", module, n.Name)
}
out.Assigned[n.Name] = append(out.Assigned[n.Name], declaredFor(m, seats))
d := declaredFor(m, seats)
d.Holds = heldHere(d.Holds, holdings, n.Name, module)
out.Assigned[n.Name] = append(out.Assigned[n.Name], d)
if m.Instances == catalogue.InstancesInterchangeable {
out.Interchangeable[m.Module] = true
}
@@ -124,6 +132,9 @@ func declaredFor(m catalogue.Manifest, seats map[string]catalogue.SeatDeclaratio
Serves: m.Tools,
// And what it calls (novox/hq ADR 0152) — the console's `*`, nothing else's.
Invokes: m.Invokes,
// And the state it keeps and reads (novox/hq ADR 0201).
State: bucketsOf(m),
Reads: m.Reads,
}
for _, c := range m.Claims {
// Every seat with a protocol, the mesh's own included. One that says only who does a job is
@@ -140,6 +151,30 @@ func declaredFor(m catalogue.Manifest, seats map[string]catalogue.SeatDeclaratio
return d
}
// bucketsOf is the state a module keeps, as the bus holds it.
func bucketsOf(m catalogue.Manifest) []broker.Bucket {
var out []broker.Bucket
for _, s := range m.State {
out = append(out, broker.Bucket{Module: m.Module, Name: s.Name, History: s.History, TTLSeconds: s.TTLSeconds})
}
return out
}
// DeclaredBuckets is every bucket the catalogue declares, registered modules assigned or not: a
// bucket exists from registration, like a seat's stream, so a module reading it may watch before its
// owner runs anywhere (novox/hq ADR 0201).
func (i *Inventory) DeclaredBuckets(ctx context.Context) ([]broker.Bucket, error) {
declared, err := i.Catalogue(ctx)
if err != nil {
return nil, fmt.Errorf("cannot read the catalogue: %w", err)
}
var out []broker.Bucket
for _, m := range declared {
out = append(out, bucketsOf(m)...)
}
return out, nil
}
func asSeat(s catalogue.SeatDeclaration) broker.Seat {
return broker.Seat{Name: s.Name, Scope: s.Scope, Accepts: s.Accepts, Emits: s.Emits,
Serves: catalogue.VerbNames(s.Serves)}
@@ -183,3 +218,33 @@ func (i *Inventory) NodesWithALiveToken(ctx context.Context) ([]string, error) {
}
return out, rows.Err()
}
// heldHere keeps of what a module claims only the seats it holds on this machine (novox/hq issue 218).
// A seat held once per machine is held by every assignment that claims it. A seat held once for the
// mesh is held by one assignment: where the mesh recorded who holds it, a claim on any other machine
// grants nothing and issues nothing — or the module would serve the role's verbs from a machine that
// is not the role's, and a question to the mesh's store would be answered from the wrong database. A
// mesh seat with no holder on record is left as it was derived.
func heldHere(claimed []broker.Seat, holdings []catalogue.Held, node, module string) []broker.Seat {
recorded := map[string][]catalogue.Held{}
for _, h := range holdings {
if h.Scope == catalogue.ScopeMesh {
recorded[h.Claim] = append(recorded[h.Claim], h)
}
}
var out []broker.Seat
for _, s := range claimed {
holders, onRecord := recorded[s.Name]
if s.Scope != catalogue.ScopeMesh || !onRecord {
out = append(out, s)
continue
}
for _, h := range holders {
if h.Node == node && h.Module == module {
out = append(out, s)
break
}
}
}
return out
}
+42 -35
View File
@@ -17,6 +17,10 @@ import (
// ErrNoSuchModule is what the mesh says about a module it has never been told about.
var ErrNoSuchModule = errors.New("no module of that name")
// ErrSuperseded is a registration from a build asked before the one the module is already at
// (novox/hq 04-ISSUES/219). The build is recorded; what the module is does not change.
var ErrSuperseded = errors.New("a build asked later is already what the module is")
// ErrStillAssigned is why a module cannot be forgotten.
//
// Its own error because it is not a fault: it means a machine is running that module now, and
@@ -47,6 +51,10 @@ type Source struct {
// itself no longer carries its build (novox/hq to-be 38 WP2.4). Empty for a manifest handed over
// by hand, which carries its `build.on` itself.
Against []string
// Asked is when the build this manifest came from was requested (novox/hq 04-ISSUES/219). Zero
// is a manifest handed over by hand, or a build whose request time is not known: either is
// taken as asked at the moment it is registered.
Asked time.Time
}
// Current reports whether what the mesh holds is what the source last had.
@@ -70,11 +78,12 @@ func (i *Inventory) RegisterModule(ctx context.Context, m catalogue.Manifest, fr
// (novox/hq ADR 0175, to-be 38 WP2.4): a module serving its tools from a container built on the
// runtime's image. Refused at registration, by name, for a module that is new to the catalogue
// or that was registered in another shape — the mechanism that keeps the old pattern from
// returning by habit. **Not refused for a module already registered in that shape**: the
// catalogue holds some thirty of them the day the runtime arrives, each moves to a bundle in
// its own change (to-be 38 WP4 onward), and a gate that refused every rebuild of every unmoved
// module in the meantime would stop the whole pipeline to make a point the record already makes.
// Before the runtime exists the pattern is accepted as it always was.
// returning by habit. Before the runtime exists the pattern is accepted as it always was.
//
// *Since 2026-10-04 (to-be 38 WP4b's last step):* refused for **every** module. While some
// thirty modules still stood in that shape, one already registered so was rebuilt without
// complaint, so the pipeline kept running while each moved; every module has moved since, and
// the exception would only let one move back.
if m.Module != catalogue.RuntimeModule {
if why := catalogue.ToolContainerOnTheRuntime(m, from.Against); why != "" {
runtime, err := i.hasModule(ctx, catalogue.RuntimeModule)
@@ -82,13 +91,7 @@ func (i *Inventory) RegisterModule(ctx context.Context, m catalogue.Manifest, fr
return err
}
if runtime {
already, err := i.registeredInThatShape(ctx, m.Module)
if err != nil {
return err
}
if !already {
return fmt.Errorf("%s is not registered: %s", m.Module, why)
}
return fmt.Errorf("%s is not registered: %s", m.Module, why)
}
}
}
@@ -97,13 +100,23 @@ func (i *Inventory) RegisterModule(ctx context.Context, m catalogue.Manifest, fr
return err
}
asked := from.Asked
if asked.IsZero() {
asked = time.Now()
}
// A module registered without provenance keeps whatever it had. Handing over a manifest by
// hand is a legitimate way to fix something in a hurry, and it should not silently erase the
// record of where the module normally comes from — which is the only thing that would say,
// afterwards, that the machine is running something nobody can rebuild.
_, err = i.store.Pool().Exec(ctx,
`insert into module (name, manifest, version, source, source_path, source_seat, ref, built_from, source_head)
values ($1, $2, nullif($3,''), nullif($4,''), $7, $8, nullif($5,''), nullif($6,''), nullif($6,''))
//
// **An older request never replaces a newer one** (novox/hq 04-ISSUES/219). Builds of one
// module in flight together finish in any order, and each stood on the bases the mesh held when
// it was asked; the one asked later is what the module is, whichever is heard last. An outcome
// of an earlier request is kept in the build records and changes nothing here.
tag, err := i.store.Pool().Exec(ctx,
`insert into module (name, manifest, version, source, source_path, source_seat, ref, built_from, source_head, built_asked)
values ($1, $2, nullif($3,''), nullif($4,''), $7, $8, nullif($5,''), nullif($6,''), nullif($6,''), $9)
on conflict (name) do update set
manifest = excluded.manifest,
version = excluded.version,
@@ -115,29 +128,23 @@ func (i *Inventory) RegisterModule(ctx context.Context, m catalogue.Manifest, fr
else excluded.source_seat end,
ref = coalesce(excluded.ref, module.ref),
built_from = coalesce(excluded.built_from, module.built_from),
source_head = coalesce(excluded.built_from, module.source_head)`,
m.Module, raw, m.Version, from.Repository, from.Ref, from.BuiltFrom, from.Path, from.Seat)
return err
}
// registeredInThatShape is whether the catalogue already holds this module as a tools container on
// the runtime's image — judged from the manifest it holds and what that module's newest build stood
// on, the same two things the gate judges a new registration by. False for a module the catalogue
// does not hold.
func (i *Inventory) registeredInThatShape(ctx context.Context, name string) (bool, error) {
held, err := i.Catalogue(ctx)
source_head = coalesce(excluded.built_from, module.source_head),
built_asked = excluded.built_asked
where module.built_asked is null or module.built_asked <= excluded.built_asked`,
m.Module, raw, m.Version, from.Repository, from.Ref, from.BuiltFrom, from.Path, from.Seat, asked)
if err != nil {
return false, err
return err
}
stored, has := held[name]
if !has {
return false, nil
if tag.RowsAffected() == 0 {
var current time.Time
if err := i.store.Pool().QueryRow(ctx,
`select built_asked from module where name = $1`, m.Module).Scan(&current); err != nil {
return err
}
return fmt.Errorf("%w: %s is at a build asked %s, and this one was asked %s",
ErrSuperseded, m.Module, current.UTC().Format(time.RFC3339), asked.UTC().Format(time.RFC3339))
}
against, err := i.BuiltAgainst(ctx)
if err != nil {
return false, err
}
return catalogue.ToolContainerOnTheRuntime(stored, against[name]) != "", nil
return nil
}
// hasModule is whether the catalogue holds a module of that name.
+8 -8
View File
@@ -688,9 +688,9 @@ func TestRegisteringWithoutProvenanceKeepsTheSeat(t *testing.T) {
// Once the node's tool runtime is in the catalogue, a module serving its tools from a container
// built on the runtime's image is refused at registration, naming the record (novox/hq ADR 0175,
// to-be 38 WP2.4) — for a module new to the catalogue or one that had moved away from it; a module
// already standing in that shape is rebuilt as before, so the catalogue's pipeline keeps running
// while each moves (WP3's amendment). Before the runtime, it is accepted as it always was — so a
// to-be 38 WP2.4) — for every module, since every module has moved (WP4b's last step; WP3's
// amendment let one already standing in that shape be rebuilt while each moved). Before the
// runtime, it is accepted as it always was — so a
// mesh converts in the order the design says and nothing is refused before there is anything to
// move to.
func TestAToolContainerIsRefusedOnceTheRuntimeIsRegistered(t *testing.T) {
@@ -715,11 +715,11 @@ func TestAToolContainerIsRefusedOnceTheRuntimeIsRegistered(t *testing.T) {
t.Fatal(err)
}
// **A module already registered in that shape is rebuilt without complaint** (to-be 38 WP2.4 as
// amended by WP3): some thirty of them stand the day the runtime arrives, and each moves in its
// own change. The gate is against the pattern spreading, not against the pipeline running.
if err := inv.RegisterModule(ctx, filter, Source{Repository: "/r", Against: stoodOn}); err != nil {
t.Fatalf("a rebuild of a module that already had the pattern was refused: %v", err)
// **A module already registered in that shape is refused too** (WP4b's last step): every module
// has moved, and a rebuild in the old shape is one moving back.
if err := inv.RegisterModule(ctx, filter, Source{Repository: "/r", Against: stoodOn}); err == nil ||
!strings.Contains(err.Error(), "ADR 0175") {
t.Fatalf("a rebuild of a module in the old pattern was registered beside the runtime: %v", err)
}
// A module new to the catalogue in that shape is refused, naming the record.
newcomer := filter
+242
View File
@@ -0,0 +1,242 @@
package inventory
import (
"context"
"encoding/json"
"strings"
)
// What the artifact store keeps, and what it may let go (novox/hq ADR 0189, issue 108).
//
// The store has never collected anything: every build pushes another layer set and nothing has
// ever removed one. The registry's own answer — collect what no tag names — is wrong here, because
// the mesh pushes each artifact under one moving tag and pins machines by digest, so every build
// but the newest is untagged and some machine may still be running it.
//
// **So the mesh decides, from its own records, and it never has to look in the store to do it.**
// It has never put anything there it did not record, which means every digest it could remove is
// already in a build row. A digest the mesh did not record making is therefore never named here —
// not as a safety margin but as the rule restated, and it is what keeps the sweep away from the
// images genesis pushed before any record existed (04-ISSUES/102, F4).
// KeptBuilds is how many successful builds of each module keep their artifacts, counting the
// newest. The newest is what the mesh hands a machine now; the four behind it are how far back a
// release that turns out wrong can be taken.
const KeptBuilds = 5
// ToCollect is every artifact the mesh made, no longer keeps, and has not already collected.
//
// Three reasons an artifact stays, and nothing else is a reason:
//
// - **a definition names it** — the reference appears in a module's recorded manifest, which is
// what the mesh would hand a machine now. No age limit: this is the floor;
// - **the mesh can still go back to it** — it is an artifact of one of the KeptBuilds most
// recent successful builds of its module;
// - it was already collected, in which case there is nothing left to do.
//
// Returned in a stated order so two runs over the same records ask for the same things in the
// same sequence, which is what makes a failed sweep safe to simply run again.
func (i *Inventory) ToCollect(ctx context.Context) ([]string, error) {
keep, err := i.keptReferences(ctx)
if err != nil {
return nil, err
}
rows, err := i.store.Pool().Query(ctx,
// Every artifact of every successful build, oldest first, minus what has already been
// collected. A failed build published nothing, so it names nothing to remove.
`select b.made
from build b
where b.failed = '' and b.module is not null and b.module <> ''
order by b.at asc, b.id asc`)
if err != nil {
return nil, err
}
defer rows.Close()
collected, err := i.alreadyCollected(ctx)
if err != nil {
return nil, err
}
seen := map[string]bool{}
var out []string
for rows.Next() {
var raw []byte
if err := rows.Scan(&raw); err != nil {
return nil, err
}
var made []Artifact
if err := json.Unmarshal(raw, &made); err != nil {
// One unreadable record must not stop the rest being collected — and an artifact this
// row named is simply not offered, which errs toward keeping.
continue
}
for _, a := range made {
if a.Reference == "" || keep[a.Reference] || collected[a.Reference] || seen[a.Reference] {
continue
}
seen[a.Reference] = true
out = append(out, a.Reference)
}
}
return out, rows.Err()
}
// keptReferences is every artifact reference the mesh still keeps, for either of the two reasons.
func (i *Inventory) keptReferences(ctx context.Context) (map[string]bool, error) {
keep := map[string]bool{}
// **Whatever a definition the mesh holds names.** Read as text rather than by walking the
// resource shapes: a reference may be a container's image, a bundle's source, or a field some
// later kind of resource grows, and what matters is only whether the mesh could hand this
// string to a machine. A manifest that mentions it is a manifest that might.
manifests, err := i.store.Pool().Query(ctx, `select manifest::text from module where manifest is not null`)
if err != nil {
return nil, err
}
defer manifests.Close()
var named []string
for manifests.Next() {
var text string
if err := manifests.Scan(&text); err != nil {
return nil, err
}
named = append(named, text)
}
if err := manifests.Err(); err != nil {
return nil, err
}
// The KeptBuilds most recent successful builds of each module, whole.
recent, err := i.store.Pool().Query(ctx,
`select made from (
select made, row_number() over (partition by module order by at desc, id desc) as back
from build
where failed = '' and module is not null and module <> ''
) ranked where back <= $1`, KeptBuilds)
if err != nil {
return nil, err
}
defer recent.Close()
for recent.Next() {
var raw []byte
if err := recent.Scan(&raw); err != nil {
return nil, err
}
var made []Artifact
if err := json.Unmarshal(raw, &made); err != nil {
continue
}
for _, a := range made {
if a.Reference != "" {
keep[a.Reference] = true
}
}
}
if err := recent.Err(); err != nil {
return nil, err
}
// And anything a manifest mentions. Done after the recent set so the scan runs over the
// candidates rather than over every reference ever recorded: a manifest holds a reference
// composed with the store's address or kept bare, so the search is for the digest within it.
if len(named) > 0 {
all, err := i.everyReferenceMade(ctx)
if err != nil {
return nil, err
}
for _, reference := range all {
if keep[reference] {
continue
}
digest := digestIn(reference)
if digest == "" {
// Not something the store holds by digest; nothing here can speak for it, so it
// is kept rather than guessed about.
keep[reference] = true
continue
}
for _, text := range named {
if strings.Contains(text, digest) {
keep[reference] = true
break
}
}
}
}
return keep, nil
}
// everyReferenceMade is every artifact reference any successful build recorded.
func (i *Inventory) everyReferenceMade(ctx context.Context) ([]string, error) {
rows, err := i.store.Pool().Query(ctx,
`select made from build where failed = '' and module is not null and module <> ''`)
if err != nil {
return nil, err
}
defer rows.Close()
seen := map[string]bool{}
var out []string
for rows.Next() {
var raw []byte
if err := rows.Scan(&raw); err != nil {
return nil, err
}
var made []Artifact
if err := json.Unmarshal(raw, &made); err != nil {
continue
}
for _, a := range made {
if a.Reference == "" || seen[a.Reference] {
continue
}
seen[a.Reference] = true
out = append(out, a.Reference)
}
}
return out, rows.Err()
}
// digestIn is the `sha256:<hex>` a reference names, empty when it names none.
func digestIn(reference string) string {
for _, marker := range []string{"@sha256:", "/sha256:"} {
if _, after, ok := strings.Cut(reference, marker); ok {
return "sha256:" + after
}
}
return ""
}
// alreadyCollected is what the store has already been asked to let go.
func (i *Inventory) alreadyCollected(ctx context.Context) (map[string]bool, error) {
rows, err := i.store.Pool().Query(ctx, `select reference from artifact_collected`)
if err != nil {
return nil, err
}
defer rows.Close()
out := map[string]bool{}
for rows.Next() {
var reference string
if err := rows.Scan(&reference); err != nil {
return nil, err
}
out[reference] = true
}
return out, rows.Err()
}
// MarkCollected records that the store no longer holds these.
//
// **A store that answered "not found" is recorded too.** The outcome wanted is that the artifact
// is gone, and it is; retrying it every sweep for ever is the failure this table exists to
// prevent. Only a store that could not be reached, or refused, leaves a reference unmarked — and
// then the next sweep asks again, which is what should happen.
func (i *Inventory) MarkCollected(ctx context.Context, references []string) error {
for _, reference := range references {
if _, err := i.store.Pool().Exec(ctx,
`insert into artifact_collected (reference) values ($1) on conflict (reference) do nothing`,
reference); err != nil {
return err
}
}
return nil
}
+147
View File
@@ -0,0 +1,147 @@
package inventory
import (
"context"
"fmt"
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
)
// What the store keeps, and what it may let go (novox/hq ADR 0189, issue 108).
//
// The store has collected nothing since it was raised, and the registry's own answer — collect
// what no tag names — would delete images machines are running, because the mesh pushes under one
// moving tag and pins by digest. So the rule is the mesh's, read from its own records, and these
// are the three reasons an artifact stays and the one reason it goes.
// ref is an artifact reference as the mesh records one.
func ref(module, artifact string, n int) string {
return fmt.Sprintf("%s%s/%s@sha256:%064x", catalogue.ArtifactStoreScheme, module, artifact, n)
}
// built records one successful build of a module publishing one image.
func built(t *testing.T, inv *Inventory, id, module string, n int) string {
t.Helper()
reference := ref(module, "app", n)
b := aBuild(id, module, "")
b.Made = []Artifact{{Name: "app", Kind: "image", Reference: reference}}
if err := inv.RecordBuild(context.Background(), b); err != nil {
t.Fatal(err)
}
return reference
}
func TestTheStoreKeepsTheRecentBuildsAndLetsGoOfTheRest(t *testing.T) {
inv := fresh(t)
ctx := context.Background()
// Eight builds of one module, oldest first. Five are kept — the newest, and the four a
// release that turns out wrong can be taken back to.
var made []string
for i := 1; i <= 8; i++ {
made = append(made, built(t, inv, fmt.Sprintf("b%02d", i), "web", i))
}
go_, err := inv.ToCollect(ctx)
if err != nil {
t.Fatal(err)
}
want := made[:3] // the three oldest
if len(go_) != len(want) {
t.Fatalf("offered %v to collect; want the %d oldest of %d", go_, len(want), len(made))
}
for i := range want {
if go_[i] != want[i] {
t.Fatalf("offered %v; want %v — and in that order, so a failed sweep is safe to run again",
go_, want)
}
}
}
func TestADefinitionNamingAnArtifactKeepsItHoweverOldItIs(t *testing.T) {
// The floor: no age limit. A module recorded at an older commit still names what the mesh
// would hand a machine now, and that is what must not be collected out from under it.
inv := fresh(t)
ctx := context.Background()
var made []string
for i := 1; i <= 8; i++ {
made = append(made, built(t, inv, fmt.Sprintf("b%02d", i), "web", i))
}
oldest := made[0]
// A definition the mesh holds, whose container runs that oldest image.
m := catalogue.Manifest{Module: "web", Version: "1", Resources: []map[string]any{{
"id": "app", "type": "container", "name": "web", "image": oldest,
}}}
if err := inv.RegisterModule(ctx, m, Source{Repository: "https://forge.invalid/web.git"}); err != nil {
t.Fatal(err)
}
go_, err := inv.ToCollect(ctx)
if err != nil {
t.Fatal(err)
}
for _, reference := range go_ {
if reference == oldest {
t.Fatalf("the mesh offered to collect %s, which a definition it holds names", oldest)
}
}
if len(go_) != 2 {
t.Fatalf("offered %v; want the two oldest that nothing names", go_)
}
}
func TestWhatHasBeenCollectedIsNotOfferedAgain(t *testing.T) {
// Without this the sweep reissues a delete for every artifact it has ever collected, every
// time it runs, for ever — a number of requests that grows with the mesh's whole history.
inv := fresh(t)
ctx := context.Background()
for i := 1; i <= 7; i++ {
built(t, inv, fmt.Sprintf("b%02d", i), "web", i)
}
first, err := inv.ToCollect(ctx)
if err != nil {
t.Fatal(err)
}
if len(first) != 2 {
t.Fatalf("offered %v, want two", first)
}
if err := inv.MarkCollected(ctx, first); err != nil {
t.Fatal(err)
}
again, err := inv.ToCollect(ctx)
if err != nil {
t.Fatal(err)
}
if len(again) != 0 {
t.Fatalf("offered %v again after collecting it", again)
}
}
func TestAFailedBuildNamesNothingToCollectAndEachModuleIsCountedOnItsOwn(t *testing.T) {
inv := fresh(t)
ctx := context.Background()
// A failed build published nothing, so it is neither kept nor collected — and it must not
// count against the module's five.
for i := 1; i <= 6; i++ {
built(t, inv, fmt.Sprintf("w%02d", i), "web", i)
}
if err := inv.RecordBuild(ctx, aBuild("w99", "web", "the recipe would not build")); err != nil {
t.Fatal(err)
}
// And a second module with three builds keeps all three: five each, not five between them.
for i := 1; i <= 3; i++ {
built(t, inv, fmt.Sprintf("d%02d", i), "db", 100+i)
}
go_, err := inv.ToCollect(ctx)
if err != nil {
t.Fatal(err)
}
if len(go_) != 1 || go_[0] != ref("web", "app", 1) {
t.Fatalf("offered %v; want only web's oldest — db's three are all within its five", go_)
}
}
+20
View File
@@ -100,3 +100,23 @@ func TestABundleStandsOnTheToolchainItIsCompiledIn(t *testing.T) {
}
}
}
// novox/hq 04-ISSUES/212: a toolchain standing on the SDK's package is planned after the SDK, so a
// release of the SDK rebuilds the toolchain, and every bundle compiled in it after that.
func TestAToolchainStandingOnTheSDKFollowsIt(t *testing.T) {
entries := []Entry{
{Manifest: catalogue.Manifest{Module: "mesh-sdk"}},
{Manifest: catalogue.Manifest{Module: "mesh-tools", Build: &catalogue.Build{
On: []catalogue.BuildsOn{{Arg: "MESH_SDK", Module: "mesh-sdk", Artifact: "lib"}}}}},
}
edges := dependenciesOf(entries, nil, nil)
found := false
for _, e := range edges {
if e.From == "mesh-tools" && e.To == "mesh-sdk" {
found = true
}
}
if !found {
t.Errorf("no edge from the toolchain to the SDK: %v", edges)
}
}
+32
View File
@@ -0,0 +1,32 @@
package inventory
import (
"testing"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/catalogue"
)
// novox/hq issue 218: a seat held once for the mesh is granted and issued only to the holder on record;
// a node seat to every machine's claimant; a mesh seat with no holder on record as derived.
func TestOnlyTheRecordedHolderHoldsAMeshSeat(t *testing.T) {
claimed := []broker.Seat{
{Name: "mesh-store", Scope: catalogue.ScopeMesh},
{Name: "node-packet-filter", Scope: catalogue.ScopeNode},
{Name: "unrecorded", Scope: catalogue.ScopeMesh},
}
holdings := []catalogue.Held{{Claim: "mesh-store", Scope: catalogue.ScopeMesh, Node: "control", Module: "postgres"}}
names := func(ss []broker.Seat) (out []string) {
for _, s := range ss {
out = append(out, s.Name)
}
return
}
if got := names(heldHere(claimed, holdings, "control", "postgres")); len(got) != 3 {
t.Errorf("the holder lost a seat: %v", got)
}
got := names(heldHere(claimed, holdings, "other", "postgres"))
if len(got) != 2 || got[0] != "node-packet-filter" || got[1] != "unrecorded" {
t.Errorf("a claimant on another machine holds %v; want the node seat and the unrecorded one, not the store", got)
}
}
+59
View File
@@ -87,3 +87,62 @@ func (i *Inventory) tryHold(ctx context.Context, sorted []string) (func(), strin
}
return release, "", nil
}
// ErrPlansBusy is the plans held by another act — on a machine replacing its controller, the other
// controller — for longer than a caller waits, or at all for one that does not wait.
var ErrPlansBusy = errors.New("another controller is working the plans")
// HoldPlans makes working the plans one act at a time, across every controller on the store
// (novox/hq issue 213). A plan is read, changed and written whole; two controllers doing that at
// once — the old and the new for the moment a machine hands its controller over, or a controller
// and a person's `plans stop` — each act on what the other has not saved yet: a tier asked twice,
// an outcome written over. A session-level advisory lock on one connection, released by the
// returned function and by the session ending, so a controller that dies holding it holds nothing.
//
// wait false gives ErrPlansBusy at once when another holds them — the timer's way: the holder is
// moving the plans already. wait true looks again every HoldPoll for up to HoldWaitFor — an
// outcome's or a merge's way, which must be written.
func (i *Inventory) HoldPlans(ctx context.Context, wait bool) (func(), error) {
deadline := time.Now().Add(HoldWaitFor)
for {
release, took, err := i.tryLock(ctx, "mesh-plans")
if err != nil || took {
return release, err
}
if !wait || time.Now().After(deadline) {
return nil, ErrPlansBusy
}
select {
case <-ctx.Done():
return nil, ctx.Err()
case <-time.After(HoldPoll):
}
}
}
// tryLock takes one named advisory lock on a connection of its own, or gives the connection back.
func (i *Inventory) tryLock(ctx context.Context, key string) (func(), bool, error) {
conn, err := i.store.Pool().Acquire(ctx)
if err != nil {
return nil, false, err
}
var once sync.Once
release := func() {
once.Do(func() {
if _, err := conn.Exec(context.WithoutCancel(ctx), `select pg_advisory_unlock_all()`); err != nil {
_ = conn.Conn().Close(context.WithoutCancel(ctx))
}
conn.Release()
})
}
var took bool
if err := conn.QueryRow(ctx, `select pg_try_advisory_lock(hashtext($1)::bigint)`, key).Scan(&took); err != nil {
release()
return nil, false, err
}
if !took {
release()
return nil, false, nil
}
return release, true, nil
}
+38
View File
@@ -0,0 +1,38 @@
package inventory
import (
"errors"
"testing"
"time"
)
// novox/hq issue 213: while a machine hands its controller over from the container to the process,
// two controllers run on one store for a moment. Working the plans is one act at a time across them.
func TestThePlansAreWorkedByOneControllerAtATime(t *testing.T) {
first := ForTest(t)
// A second controller: its own connections to the same store.
second, err := Open(t.Context())
if err != nil {
t.Fatal(err)
}
t.Cleanup(second.Close)
release, err := first.HoldPlans(t.Context(), false)
if err != nil {
t.Fatalf("the plans could not be held when nobody held them: %v", err)
}
t.Cleanup(release) // a pool closing waits for a connection still held; release is idempotent
if _, err := second.HoldPlans(t.Context(), false); !errors.Is(err, ErrPlansBusy) {
t.Fatalf("a second controller held the plans while the first did: %v", err)
}
// A waiter gets them once they are let go.
was := HoldPoll
HoldPoll = 10 * time.Millisecond
defer func() { HoldPoll = was }()
go func() { time.Sleep(50 * time.Millisecond); release() }()
again, err := second.HoldPlans(t.Context(), true)
if err != nil {
t.Fatalf("a waiting controller never got the plans once they were let go: %v", err)
}
again()
}
@@ -0,0 +1,23 @@
-- A build is ordered by when it was asked, not when it finished (novox/hq 04-ISSUES/219).
--
-- Two builds of one module can be in flight together — two merge plans a few minutes apart, each
-- asking for everything standing on what it changed — and they finish in any order. Each build
-- stands on the bases the mesh held when it was *asked*, so the one asked later is the newer one.
-- The mesh ordered builds by `at`, which is when the outcome was recorded, and registered whatever
-- it heard last: an older request that took longer replaced a newer one as what the module is, and
-- the next push sent machines an image built on a base the mesh had already replaced.
--
-- `build.asked` is when the build was requested, read from the correlation id the controller wrote
-- (`build-<unix nanoseconds>`). Nullable: an id of any other shape says no request time, and such a
-- build is placed where it was recorded, which is the order the mesh had before this.
alter table build add column asked timestamptz;
update build
set asked = to_timestamp((substring(id from '^build-([0-9]{19})$'))::numeric / 1000000000)
where id ~ '^build-[0-9]{19}$';
-- `module.built_asked` is when the build the module's registered manifest came from was asked, so
-- a later-heard outcome of an earlier request is recorded and not registered. A manifest handed over
-- by hand is a request made when it is handed over. Null for a module registered before this was
-- kept: its next registration, whichever it is, sets it.
alter table module add column built_asked timestamptz;
@@ -0,0 +1,23 @@
-- What the artifact store no longer keeps (novox/hq ADR 0189, issue 108).
--
-- The mesh removes from its store only what it put there and can account for: every digest it
-- could remove is already in a build record, so the sweep reads its own records rather than
-- enumerating the store. What it does not get from those records is whether it has already
-- removed something -- `build.made` says what that build published, for ever, which is history
-- and not an index of what is on disk.
--
-- Without this the sweep would reissue a delete for every artifact it has ever collected, every
-- time it runs, and each one would answer 404 -- a number of requests that grows with the mesh's
-- whole history and never shrinks.
--
-- Keyed by the reference as the mesh records it (`artifact-store://<module>/<artifact>@sha256:…`),
-- because that is the identity the record uses everywhere else. Not a foreign key to build: two
-- builds can publish the same digest (the same source built twice produces the same bytes), and
-- what is collected is the artifact, not the attempt that made it.
create table artifact_collected (
reference text primary key,
-- When the store answered. Kept so a reader of an old build record can tell "this artifact is
-- gone" from "this artifact was never there", which are different kinds of surprise.
at timestamptz not null default now()
);
+124
View File
@@ -0,0 +1,124 @@
package inventory
import (
"context"
"errors"
"testing"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
)
// novox/hq 04-ISSUES/219: two builds of one module in flight together, the one asked first heard
// last. The newer request stood on the newer base; the older one's late outcome is recorded and is
// not what the module is.
func postgresBuild(id string, asked time.Time, image string) Build {
b := aBuild(id, "postgres", "")
b.Asked = asked
b.Against = []string{"mesh-tools/runtime@sha256:" + id}
b.Made = []Artifact{{Name: "server", Kind: "image", Reference: "postgres@sha256:" + image}}
return b
}
func TestAnOlderRequestFinishingLaterIsNotWhatTheModuleHolds(t *testing.T) {
inv := fresh(t)
ctx := context.Background()
older := time.Date(2026, 10, 3, 21, 33, 45, 0, time.UTC)
newer := time.Date(2026, 10, 3, 21, 51, 57, 0, time.UTC)
// The newer request finishes first, the older one last — recorded in that order.
if err := inv.RecordBuild(ctx, postgresBuild("newer", newer, "4bcd5f73")); err != nil {
t.Fatal(err)
}
if err := inv.RecordBuild(ctx, postgresBuild("older", older, "0ab07fa9")); err != nil {
t.Fatal(err)
}
held, err := inv.Held(ctx)
if err != nil {
t.Fatal(err)
}
if got := held["postgres/server"]; got != "postgres@sha256:4bcd5f73" {
t.Errorf("postgres holds %q; want the newer request's image 4bcd5f73", got)
}
against, err := inv.BuiltAgainst(ctx)
if err != nil {
t.Fatal(err)
}
if got := against["postgres"]; len(got) != 1 || got[0] != "mesh-tools/runtime@sha256:newer" {
t.Errorf("postgres stands on %v; want what the newer request stood on", got)
}
// Both are still recorded, the late one first as what happened lately.
builds, err := inv.Builds(ctx, "postgres", 5)
if err != nil {
t.Fatal(err)
}
if len(builds) != 2 || builds[0].ID != "older" || !builds[0].Asked.Equal(older) {
t.Fatalf("both builds, newest heard first, with when they were asked: %+v", builds)
}
}
func TestABuildWithNoKnownRequestTimeIsOrderedByWhenItWasRecorded(t *testing.T) {
// What the mesh did before it kept the request time, so a row from before still answers.
inv := fresh(t)
ctx := context.Background()
for _, id := range []string{"first", "second"} {
b := aBuild(id, "shell", "")
b.Made = []Artifact{{Name: "config", Kind: "archive", Reference: "…/" + id}}
if err := inv.RecordBuild(ctx, b); err != nil {
t.Fatal(err)
}
}
held, err := inv.Held(ctx)
if err != nil {
t.Fatal(err)
}
if got := held["shell/config"]; got != "…/second" {
t.Errorf("shell holds %q; want the one recorded last", got)
}
}
func TestARegistrationFromAnOlderRequestDoesNotReplaceANewerOne(t *testing.T) {
inv := fresh(t)
ctx := context.Background()
older := time.Date(2026, 10, 3, 21, 33, 45, 0, time.UTC)
newer := time.Date(2026, 10, 3, 21, 51, 57, 0, time.UTC)
from := func(asked time.Time) Source {
return Source{Repository: "novox/mesh-catalog", Seat: "git", Path: "modules/postgres",
BuiltFrom: "efff5415", Asked: asked}
}
if err := inv.RegisterModule(ctx, catalogue.Manifest{Module: "postgres", Version: "fixed"}, from(newer)); err != nil {
t.Fatal(err)
}
err := inv.RegisterModule(ctx, catalogue.Manifest{Module: "postgres", Version: "stale"}, from(older))
if !errors.Is(err, ErrSuperseded) {
t.Fatalf("an older request's registration was not refused as superseded: %v", err)
}
shelf, err := inv.Catalogue(ctx)
if err != nil {
t.Fatal(err)
}
if got := shelf["postgres"].Version; got != "fixed" {
t.Fatalf("postgres is %q; want the newer request's manifest", got)
}
// A later request, and a manifest handed over by hand — asked when it is handed over — both
// replace it as before.
if err := inv.RegisterModule(ctx, catalogue.Manifest{Module: "postgres", Version: "later"},
from(newer.Add(time.Minute))); err != nil {
t.Fatal(err)
}
if err := inv.RegisterModule(ctx, catalogue.Manifest{Module: "postgres", Version: "by-hand"}, Source{}); err != nil {
t.Fatal(err)
}
if shelf, _ := inv.Catalogue(ctx); shelf["postgres"].Version != "by-hand" {
t.Fatalf("postgres is %q; want the manifest handed over by hand", shelf["postgres"].Version)
}
src, err := inv.SourceOf(ctx, "postgres")
if err != nil || src.Repository != "novox/mesh-catalog" {
t.Fatalf("a hand registration erased the provenance: %+v %v", src, err)
}
}
+29
View File
@@ -2,6 +2,9 @@ package link
import (
"encoding/json"
"strconv"
"strings"
"time"
)
// Asking a machine to build a module, and hearing what came out.
@@ -17,6 +20,32 @@ import (
// holds no opinion about what they contain, and a host that also built things would be a host
// with a container runtime requirement and a git dependency (novox/hq ADR 0005).
// NewBuildID is the correlation for a build asked at that moment: `build-<unix nanoseconds>`.
//
// **The id carries when the build was asked, and that is read back** (novox/hq 04-ISSUES/219). Builds
// of one module can be in flight together and finish in any order; what a module currently is must
// be the newest *request's* outcome, not the last one heard, and the id is the one thing every
// outcome echoes whichever builder answered it. One place writes the shape and one reads it.
func NewBuildID(asked time.Time) string {
return "build-" + strconv.FormatInt(asked.UnixNano(), 10)
}
// BuildAskedAt is when the build with this id was asked, as NewBuildID wrote it. False for an id
// of any other shape — one written before this was read, or by hand — whose request time the mesh
// does not know.
func BuildAskedAt(id string) (time.Time, bool) {
digits, ok := strings.CutPrefix(id, "build-")
if !ok || digits == "" {
return time.Time{}, false
}
nanos, err := strconv.ParseInt(digits, 10, 64)
// A number too small to be a moment this mesh could have asked at is a name, not a time.
if err != nil || nanos < time.Date(2020, 1, 1, 0, 0, 0, 0, time.UTC).UnixNano() {
return time.Time{}, false
}
return time.Unix(0, nanos).UTC(), true
}
// BuildRequest is one module to build.
type BuildRequest struct {
// ID correlates the answer with the asking. Not the module name: two builds of one module can
+54 -2
View File
@@ -5,6 +5,7 @@ import (
"encoding/json"
"errors"
"fmt"
"log"
"strings"
"time"
@@ -78,10 +79,15 @@ func (n *natsInbound) Receive(ctx context.Context, act func(context.Context, Con
// than creating one here: the consumer is an object with a configuration — ack policy, ack
// wait, redelivery — and a client that creates its own would be a second opinion about it.
control := make(chan *nats.Msg, Prefetch)
said, err := js.ChanSubscribe("", control, nats.Bind("CONTROL", broker.ControllerName))
said, err := standingBy(ctx, log.Default(), "CONTROL", func() (*nats.Subscription, error) {
return js.ChanSubscribe("", control, nats.Bind("CONTROL", broker.ControllerName))
})
if err != nil {
return fmt.Errorf("subscribing to what nodes say: %w", err)
}
if said == nil {
return nil // stopped while standing by
}
defer func() { _ = said.Unsubscribe() }()
// Heartbeats, on core NATS and off any stream (design 25 §3). Their own subscription because
@@ -97,10 +103,15 @@ func (n *natsInbound) Receive(ctx context.Context, act func(context.Context, Con
var events chan *nats.Msg
if len(n.follows) > 0 {
events = make(chan *nats.Msg, Prefetch)
followed, err := js.ChanSubscribe("", events, nats.Bind("EVENTS", broker.ControllerName))
followed, err := standingBy(ctx, log.Default(), "EVENTS", func() (*nats.Subscription, error) {
return js.ChanSubscribe("", events, nats.Bind("EVENTS", broker.ControllerName))
})
if err != nil {
return fmt.Errorf("subscribing to what the catalogue says: %w", err)
}
if followed == nil {
return nil
}
defer func() { _ = followed.Unsubscribe() }()
}
@@ -324,3 +335,44 @@ func (m *natsControl) forget() {
type replyAddressed struct {
ReplyTo string `json:"reply_to,omitempty"`
}
// StandbyPoll is how often a controller standing by looks again for its consumers. A variable so a
// test need not wait.
var StandbyPoll = 2 * time.Second
// standingBy binds one of the controller's consumers, waiting while another controller holds it.
//
// **Two controllers, one consumer** (novox/hq issue 213). The controller's consumers are push
// consumers with no delivery group, so the server lets one subscription bind each — on purpose:
// two would each act on every message (issue 146). When a machine hands its controller over from
// the container to the process, the host starts the process first and removes the container only
// once the process is up; the process then finds the consumers bound. Exiting on that would never
// be up, so the container would never go. It stands by instead — the seat's verbs are already
// served from a queue group, and the plans wait on their lock — and binds as soon as the other lets
// go. Nil and no error is ctx ending while it waited.
func standingBy(ctx context.Context, logger interface{ Printf(string, ...any) }, stream string,
bind func() (*nats.Subscription, error)) (*nats.Subscription, error) {
said := false
for {
sub, err := bind()
if err == nil {
if said {
logger.Printf("took the controller's consumer on %s: the controller that held it let go", stream)
}
return sub, nil
}
if !strings.Contains(err.Error(), "already bound") {
return nil, err
}
if !said {
logger.Printf("another controller holds the controller's consumer on %s; standing by "+
"until it lets go", stream)
said = true
}
select {
case <-ctx.Done():
return nil, nil
case <-time.After(StandbyPoll):
}
}
}
+69
View File
@@ -0,0 +1,69 @@
package link
import (
"context"
"encoding/json"
"os"
"testing"
"time"
"github.com/novox/mesh-controller/internal/broker"
)
// novox/hq issue 213: while a machine hands its controller over, the new controller (the process)
// is started while the old one (the container) still holds the controller's consumers. It must not
// exit — the host would read that as a replacement that did not come up and never remove the
// container — and must not act on what the old one is handed. It stands by, and takes the consumers
// when the old one lets go.
func TestNatsASecondControllerStandsByAndTakesOverWhenTheFirstLetsGo(t *testing.T) {
js := aBus(t)
was := StandbyPoll
StandbyPoll = 50 * time.Millisecond
defer func() { StandbyPoll = was }()
old := &counted{}
_, stopOld := servingOn(t, js, old)
eventually(t, "the first controller binding its consumer", func() bool {
info, err := js.Context().ConsumerInfo("CONTROL", broker.ControllerName)
return err == nil && info.PushBound
})
// The new one, on a connection of its own as the process would have.
second, err := broker.Dial(os.Getenv("MESH_TEST_NATS"))
if err != nil {
t.Fatal(err)
}
t.Cleanup(second.Close)
fresh := &counted{}
s := &Server{inbound: Nats(second), bus: OverNATS{Conn: second.Conn(), JS: second.Context()},
listener: fresh, log: quiet()}
ctx, stopNew := context.WithCancel(context.Background())
defer stopNew()
ended := make(chan error, 1)
go func() { ended <- s.Serve(ctx) }()
select {
case err := <-ended:
t.Fatalf("the second controller stopped instead of standing by: %v", err)
case <-time.After(500 * time.Millisecond):
}
report := func(declared string) {
body, _ := json.Marshal(Report{Node: "anchor", Declared: declared, Applied: []string{"store"}})
if _, err := js.Context().Publish(ReportSubject("anchor"), body); err != nil {
t.Fatal(err)
}
}
report("d1")
eventually(t, "the holding controller hearing the report", func() bool { return old.count() == 1 })
if fresh.count() != 0 {
t.Fatal("the controller standing by acted on a report the holder was handed")
}
stopOld()
report("d2")
eventually(t, "the second controller taking over once the first let go", func() bool { return fresh.count() == 1 })
if old.count() != 1 {
t.Errorf("the first controller heard %d reports", old.count())
}
}
+1 -1
View File
@@ -30,7 +30,7 @@ func TestTheManifestsOwnPlaceholderUnfilledLeavesTheStoreWhereTheFileSays(t *tes
}
var written string
for _, r := range m.Resources {
if r.Type == "container" {
if r.Type == "process" {
written = r.Env["MESH_STORE_INVENTORY_PORT"]
}
}
+31 -38
View File
@@ -2,9 +2,6 @@
"module": "mesh-controller",
"version": "1",
"slug": "control",
"capabilities": [
"container-runtime"
],
"claims": [
{
"name": "mesh-controller",
@@ -26,7 +23,7 @@
"broker-address": "${dir:mesh-state}/broker-address",
"bus": "${dir:mesh-state}/bus"
},
"secrets-owner": "65534:65534",
"secrets-owner": "mesh-controller",
"prepares": true,
"tools": [
"tools",
@@ -43,62 +40,58 @@
"build"
],
"resources": [
{
"id": "account",
"type": "user",
"name": "mesh-controller",
"shell": "/usr/bin/nologin",
"home": "/var/lib/mesh-controller"
},
{
"id": "mesh-state",
"type": "directory",
"mode": "0700",
"place": "mesh"
"place": "mesh",
"owner": "mesh-controller"
},
{
"id": "server",
"type": "container",
"id": "controller",
"type": "process",
"name": "mesh-controller",
"network": "host",
"args": [
"artifact": "controller",
"run": [
"./mesh-controller",
"serve"
],
"user": "mesh-controller",
"env": {
"MESH_BROKER_CERTIFICATE": "/broker-tls/tls.crt",
"MESH_STORE_INVENTORY_FILE": "/run/secrets/inventory",
"MESH_STORE_IDENTITY_FILE": "/run/secrets/identity",
"MESH_STORE_LICENCES_FILE": "/run/secrets/licences",
"MESH_BROKER_MANAGEMENT_FILE": "/run/secrets/broker-management",
"MESH_BROKER_ADDRESS_FILE": "/run/secrets/broker-address",
"MESH_BROKER_CERTIFICATE": "/var/lib/mesh-broker-tls/tls.crt",
"MESH_STORE_INVENTORY_FILE": "${dir:mesh-state}/inventory",
"MESH_STORE_IDENTITY_FILE": "${dir:mesh-state}/identity",
"MESH_STORE_LICENCES_FILE": "${dir:mesh-state}/licences",
"MESH_BROKER_MANAGEMENT_FILE": "${dir:mesh-state}/broker-management",
"MESH_BROKER_ADDRESS_FILE": "${dir:mesh-state}/broker-address",
"MESH_STORE_INVENTORY_PORT": "${seat:mesh-store:5432}",
"MESH_STORE_IDENTITY_PORT": "${seat:mesh-store:5432}",
"MESH_STORE_LICENCES_PORT": "${seat:mesh-store:5432}",
"MESH_BROKER_MANAGEMENT_PORT": "${seat:mesh-broker:15672}",
"MESH_BROKER_ADDRESS_PORT": "${seat:mesh-broker:5671}",
"MESH_BUS_NATS_FILE": "/run/secrets/bus"
"MESH_BUS_NATS_FILE": "${dir:mesh-state}/bus"
},
"volumes": [
"/var/lib/mesh-broker-tls:/broker-tls:ro",
"${dir:mesh-state}/inventory:/run/secrets/inventory:ro",
"${dir:mesh-state}/identity:/run/secrets/identity:ro",
"${dir:mesh-state}/licences:/run/secrets/licences:ro",
"${dir:mesh-state}/broker:/run/secrets/broker:ro",
"${dir:mesh-state}/bus:/run/secrets/bus:ro",
"${dir:mesh-state}/broker-management:/run/secrets/broker-management:ro",
"${dir:mesh-state}/broker-address:/run/secrets/broker-address:ro"
],
"artifact": "server",
"restart-on": [
"control-env"
"replaces": [
"server"
]
}
],
"build": {
"artifacts": [
{
"name": "server",
"kind": "image",
"from": "Dockerfile"
}
],
"on": [
{
"arg": "GO_BASE",
"image": "golang@sha256:8ac98ca534ac3f51e1f420a1dd2c15e74c75cfa0f23f3ad27eb5d7236c349a0c"
"name": "controller",
"kind": "bundle",
"language": "go",
"system": "arch",
"from": "cmd/mesh-controller",
"binary": "mesh-controller"
}
]
}