Compare commits

...
Author SHA1 Message Date
jschoubben 28a278c146 Take mesh-sdk at bd7731f: a machine that does not resolve keeps its name and its address in the mesh as well as its tunnel (issue 1833)
mesh/delivery-group group fix/1833-one-machine-alone checking: 0 of 2 member(s) ready
mesh/repo-check head 28a278c1462a merged into trunk 8f2d22db0caa — pass: its merge-check.sh passed
mesh/delivery checking
mesh/merge-gate building on ace since 16:03
mesh/check failed: unit: ..test: node:internal/modules/esm/resolve:241 | 28a278c1462a on b8f77577fed2 | unit 3353 run, 0 kept, 7 skipped | integration…
2026-10-22 15:27:27 +02:00
jschoubben c34ab0e43a Take mesh-sdk at c154bde: a machine that does not resolve keeps its place on the private network, so it alone is refused (issue 1833)
One machine whose modules could not be resolved left the private network uncomposable when it was the hub, and every
send to every machine was held. The SDK now keeps a refused machine at the place the controller already holds for it.
2026-10-22 15:04:47 +02:00
mesh-tools 8f2d22db0c Merge pull request 'Move route-proxy's program into its module and run it on the pinned alpine image (issues 1633, 1634, 1635, ADR 0480 §1-§4, §11)' (#433) from feat/r4d into main 2026-10-22 12:19:07 +00:00
mesh-tools aa9da94260 Merge pull request 'A build node leaves its builds' images behind no more, and takes no work under a floor of free space (issue 1834)' (#435) from fix/1834-build-node-disk into main 2026-10-22 12:05:14 +00:00
jschoubben 01f615aa0a Read the bus release from the nats module's upstream artifact, so the tests' bus is still held to the mesh's once the Dockerfile is gone (issue 1640)
mesh/merge-gate head 01f615aa0ad6 merged into trunk aa9da942606d — pass: group feat/r4d composed: every machine composes with the change as it did without …
mesh/delivery delivered
mesh/repo-check head 01f615aa0ad6 merged into trunk aa9da942606d — pass: its merge-check.sh passed
mesh/delivery-group group feat/r4d delivered: every member is delivered
mesh/check failed: unit: ..test: node:internal/modules/esm/resolve:241 | 01f615aa0ad6 on aa9da942606d | unit 3345 run, 0 kept, 7 skipped | integration…
mesh-catalog #434 runs the nats server as the upstream image named in module.json (ADR 0497) and
deletes modules/nats/Dockerfile, so the check failed opening it. The manifest's upstream nats
artifact is read first; the Dockerfile only for a catalogue from before the move (the captured copy).
2026-10-22 13:38:00 +02:00
jschoubben f501eaca43 Remove an image's pushed names with its build tag, and take no work under the free-space floor (issue 1834)
mesh/delivery covered: a later merge that contains it was delivered: novox/mesh-controller@01f615aa0ad6 (merged as 8f2d22db into main, walk plan-17926707…
mesh/merge-gate head f501eaca4305 merged into trunk 4463991e0609 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head f501eaca4305 merged into trunk 4463991e0609 — pass: its merge-check.sh passed
mesh/check failed: integration: github.com/novox/mesh-controller/internal/identity.TestACertificateChainsToTheMeshsOwnAuthority: authority_test.go:45:…
The registry push tags the image a second time, so removing only the build tag freed nothing.
The held image a comparison pulls goes after it is read, and a node under the floor leaves the ask
to another build node instead of taking it and failing.
2026-10-22 13:19:58 +02:00
jschoubben 995f8237ce Free a build node's disk after every build and refuse work under a floor of free space (issue 1834)
A full / made every check on shanks end 'store never became ready' with the reason erased by --rm.
Builds now remove their local image tag, narrowed source and context clone; a build or check
refuses under 20 GiB free on / and the workspace, saying so; the throwaway store is no longer
started with --rm and a store that does not come up reports its exit code and last 20 log lines.
2026-10-22 13:14:00 +02:00
jschoubben 707bfba4ea Check an image's Dockerfile place instead of refusing the image kind (ADR 0487 clause 13)
mesh/delivery-group group feat/r4d rejected: a member's own check failed
mesh/merge-gate waits for its group's composed check
mesh/repo-check head 707bfba4ea36 merged into trunk 4463991e0609 beside mesh-catalog#434@68183ee4 — fail: its merge-check.sh failed: --- FAIL: TestTheBusTh…
mesh/delivery superseded: a newer head of the same pull request
mesh/check failed: unit: i3/cmd/i3-tools.TestTheMainFileAndEveryModulesContributionLoadTogether: manifest_test.go:247: expected the launcher, clipboar…
2026-10-22 11:45:06 +02:00
mesh-tools 4463991e06 Merge pull request 'Compose a pull request's check with the config values held for it (issue 1826, ADR 0502)' (#434) from fix/1826-held-config into main 2026-10-22 09:38:35 +00:00
jschoubben 4357b3688f Tell mesh-config what a head declares at every head, so a head that reverts a key drops its held value (issue 1826)
mesh/check failed: integration: github.com/novox/mesh-controller/internal/identity.TestACertificateChainsToTheMeshsOwnAuthority: authority_test.go:45:…
mesh/repo-check head 4357b3688f58 merged into trunk 3d4ad6fefb7c — pass: its merge-check.sh passed
mesh/merge-gate head 4357b3688f58 merged into trunk 3d4ad6fefb7c — pass: group fix/1826-held-config composed: every machine composes with the change as it …
mesh/delivery delivered
mesh/delivery-group group fix/1826-held-config delivered: every member is delivered
2026-10-22 11:07:34 +02:00
jschoubben 03927a6f6c Carry a config value held for a pull request's new key into its composed check (issue 1826)
The controller tells mesh-config what a pull request declares, asks the approved held values with each check, and the build agent and merge gate lay them over the change's composition only: a key the trunk does not declare yet can then be approved before the merge (ADR 0502).
2026-10-22 11:01:12 +02:00
jschoubben 499ff5c218 Move the route proxy into its module and run it from the trust image, on mesh-sdk's public membership (issues 1633, 1634, 1635)
mesh/repo-check out of date: judged beside mesh-catalog#434 at c948f3a61c74, it is now at 7451e7edf4e5; judged again on a recheck or when it is merged
mesh/merge-gate group feat/r4d: composed again for the group's new heads
mesh/delivery superseded: a newer head of the same pull request
mesh/delivery-group group feat/r4d rejected: a member's own check failed
mesh/check failed: unit: i3/cmd/i3-tools.TestTheMainFileAndEveryModulesContributionLoadTogether: manifest_test.go:247: expected the launcher, clipboar…
A bundle is built only from its module's directory (ADR 0480 clause 4), so the proxy leaves
examples/ and the controller's internal/broker; the broker keeps no second copy of what the
proxy shares with mesh-sdk. The image and its Dockerfile go, and a test holds that only
build-agent declares one (clause 11).
2026-10-22 09:53:56 +02:00
mesh-tools 3d4ad6fefb Merge pull request 'Limit what a build or check starts on a build node: a per-node build slice (issue 1798)' (#432) from fix/1798-limit-build-resources into main 2026-10-21 23:20:57 +00:00
jschoubben 6bf38072ca Say in the build-agent settings that changing one restarts the build slice (issue 1798)
mesh/merge-gate head 6bf38072cacf merged into trunk a929df9b0ec3 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head 6bf38072cacf merged into trunk a929df9b0ec3 — pass: its merge-check.sh passed
mesh/delivery delivered
mesh/check failed: integration: github.com/novox/mesh-controller/internal/identity.TestACertificateChainsToTheMeshsOwnAuthority: authority_test.go:45:…
2026-10-22 01:01:56 +02:00
jschoubben b174d743c0 Name the build slice without a dash so it sits beside user.slice, and restart it when its limits change (issue 1798)
mesh/merge-gate building on ace since 00:57
mesh/delivery superseded: a newer head of the same pull request
mesh/check failed: integration: github.com/novox/mesh-controller/internal/identity.TestACertificateChainsToTheMeshsOwnAuthority: authority_test.go:45:…
2026-10-22 00:56:48 +02:00
jschoubben 37119ea31f Fill the build slice's limits as the mesh fills a module's template today (issue 1798)
mesh/merge-gate building on ace since 00:53
mesh/delivery superseded: a newer head of the same pull request
mesh/check failed: integration: github.com/novox/mesh-controller/internal/identity.TestACertificateChainsToTheMeshsOwnAuthority: authority_test.go:45:…
2026-10-22 00:38:02 +02:00
jschoubben af2c6b5e94 Run builds and checks in a per-node build slice with CPU, IO and memory limits (issue 1798) 2026-10-22 00:24:45 +02:00
mesh-tools a929df9b0e Merge pull request 'The checker runs beside today's check: checks wait while a delivery builds; a group's SDK member merges; a flaky test fixed' (#430) from feat/the-checker into main 2026-10-21 21:46:26 +00:00
mesh-tools a3c965b9a0 Merge pull request 'Remove a check's throwaway containers with their anonymous volumes, so a build node's disk does not fill (issue 1772)' (#431) from fix/1772-check-volumes into main 2026-10-21 21:15:07 +00:00
jschoubben b9ec1919b8 Name plans-building among the controller's readable states and pin the SDK that does (issue 1687, ADR 0483 clause 2)
mesh/repo-check head b9ec1919b865 merged into trunk a3c965b9a0d5 beside mesh-catalog#423@12ff1cfb — pass: its merge-check.sh passed
mesh/merge-gate head b9ec1919b865 merged into trunk a3c965b9a0d5 — pass: group feat/the-checker composed: every machine composes with the change as it did …
mesh/delivery delivered
mesh/delivery-group group feat/the-checker delivered: every member is delivered
2026-10-21 22:47:04 +02:00
jschoubben d491356f59 Remove a build's throwaway containers with their anonymous volumes, so a build node's disk does not fill (issue 1772)
mesh/merge-gate head d491356f59dd merged into trunk 1f3fdf311167 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head d491356f59dd merged into trunk 1f3fdf311167 — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-21 22:40:16 +02:00
jschoubben 47e1819101 Merge remote-tracking branch 'origin/fix/1106-sdk-member-merges' into feat/the-checker
mesh/repo-check out of date: judged beside mesh-sdk#66 at 78364e318bdc, it is now at 580da61a9107; judged again on a recheck or when it is merged
mesh/delivery-group group feat/the-checker checking: 4 of 4 member(s) ready
mesh/delivery superseded: a newer head of the same pull request
mesh/merge-gate head 47e1819101a1 merged into trunk 1f3fdf311167 — fail: group feat/the-checker composed: 1 problem(s) the change brings; the first: mesh-d…
2026-10-21 22:01:42 +02:00
jschoubben 0e25ed7c5a Merge remote-tracking branch 'origin/fix/1399-person-word-test' into feat/the-checker 2026-10-21 22:01:42 +02:00
jschoubben 79e2923dc8 Merge remote-tracking branch 'origin/fix/1652-checks-wait' into feat/the-checker
# Conflicts:
#	internal/broker/testdata/composed.conf
#	internal/link/builds_nats.go
2026-10-21 22:01:42 +02:00
mesh-tools 1f3fdf3111 Merge pull request 'Judge a duplicate hand-out before marking a build in hand, and never hand every build back over a stale mark (issue 1768)' (#429) from fix/1768-in-hand-marker into main 2026-10-21 19:59:30 +00:00
jschoubben 9902a3f16c Name every group member's trunk in the composed verdict, the announced one when the seat names none (issue 1106) 2026-10-21 21:57:30 +02:00
jschoubben 11815c0fb3 Have the delivery test's view hear a plan a verb let go, so a loaded machine cannot read it still waiting (issue 1399) 2026-10-21 21:45:58 +02:00
mesh-tools f973cefd7f Merge pull request 'Ask the operator for a held move and a failed release, and send no image the store lacks (issues 1732, 1320, 1742)' (#428) from fix/1732-held-moves-asked into main 2026-10-21 19:07:51 +00:00
jschoubben 696300b9e2 Judge a duplicate hand-out before the in-hand claim, and treat a stale mark as stale (issue 1768)
mesh/merge-gate head 696300b9e280 merged into trunk f973cefd7fd3 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head 696300b9e280 merged into trunk f973cefd7fd3 — pass: its merge-check.sh passed
mesh/delivery delivered
A dropped duplicate left the in-hand marker, so every later claim failed and every ask spent its deliveries.
2026-10-21 20:57:12 +02:00
mesh-tools 98155e327b Merge pull request 'Take mesh-sdk at 07a6592: an item's text places a config value (issue 1658), and fit the unit level in its bound (issue 1763)' (#420) from fix/1656-xterm-config into main 2026-10-21 18:36:39 +00:00
jschoubben d6db334840 Offer no Release on a machine that also holds a person's own hold (issue 1732, review round 1)
mesh/repo-check head d6db334840a5 merged into trunk 98155e327b6d beside mesh-host#129@09c77d57 — pass: its merge-check.sh passed
mesh/merge-gate head d6db334840a5 merged into trunk 98155e327b6d — pass: group fix/1732-held-moves-asked composed: every machine composes with the change a…
mesh/delivery delivered
mesh/delivery-group group fix/1732-held-moves-asked delivered: every member is delivered
2026-10-21 20:15:58 +02:00
jschoubben 69d016bfdf Merge remote-tracking branch 'origin/main' into fix/1732-held-moves-asked 2026-10-21 20:15:13 +02:00
jschoubben a58b38cee3 Keep the missing image out of the condition's summary (issue 1742) 2026-10-21 20:06:10 +02:00
jschoubben 9d260412b3 Merge remote-tracking branch 'origin/main' into fix/1656-xterm-config
mesh/repo-check head 9d260412b330 merged into trunk 44033f4b6d72 beside mesh-catalog#416@b9dcf19c — pass: its merge-check.sh passed
mesh/merge-gate head 9d260412b330 merged into trunk 44033f4b6d72 — pass: group fix/1656-xterm-config composed: every machine composes with the change as it…
mesh/delivery-group group fix/1656-xterm-config failed: a member failed
mesh/delivery covered after its failure: a later walk that contains it was delivered: plan-1792609673527907956 (novox/mesh-controller at f973cefd into ma…
2026-10-21 19:59:59 +02:00
jschoubben 99610e9569 Cut the controller's slowest unit tests (a source scan that regexed every file, resolver stand-in timeouts, a wait bound) so mesh-controller's unit level fits its bound (issue 1763) 2026-10-21 19:59:58 +02:00
jschoubben dc06af1f5a Leave a module out of a send when the store lacks its image, and say it (issue 1742) 2026-10-21 19:43:36 +02:00
jschoubben b62fbeabbf Ask a held move and the builds held after a failed release, and say them on the plan's line (issue 1732, 1320) 2026-10-21 19:43:36 +02:00
jschoubben 373aefd14e Take mesh-sdk at 07a6592, which counts an item's text as a placement of a config value (issue 1658)
mesh/delivery-group group fix/1656-xterm-config checking: composed check out of date: novox/mesh-sdk@c8e35097e861 was judged against trunk 85b5a2687e6a, the tr…
mesh/merge-gate building on ace since 19:52
mesh/repo-check out of date: judged against trunk 22c9815f6522, the trunk is now 44033f4b6d72; judged again on a recheck or when it is merged
mesh/delivery superseded: a newer head of the same pull request
2026-10-21 19:43:32 +02:00
jschoubben f66458fe25 Merge remote-tracking branch 'origin/main' into fix/1656-xterm-config 2026-10-21 19:43:02 +02:00
jschoubben 99da867a98 Merge origin/main into fix/1656-xterm-config 2026-10-21 19:16:11 +02:00
jschoubben 69a67876c2 Take mesh-sdk at 6f63275, which counts an item's text as a placement of a config value (issue 1658)
mesh/delivery-group group fix/1656-xterm-config rejected: a member's own check failed
mesh/merge-gate waits for its group's composed check
mesh/repo-check head 69a67876c269 merged into trunk d96c9f56a3f1 beside mesh-catalog#416@f178aab6 — fail: its merge-check.sh failed: --- FAIL: TestModuleCh…
mesh/delivery superseded: a newer head of the same pull request
2026-10-21 17:03:35 +02:00
380 changed files with 92648 additions and 505 deletions
+3 -12
View File
@@ -90,15 +90,6 @@ redis-provisioner-image:
@echo
@docker image inspect $(REDIS_PROVISIONER_IMAGE) --format 'built {{.RepoTags}} {{.Size}} bytes'
# The proxy that turns a route grant into traffic reaching a workload.
PROXY_IMAGE ?= mesh-route-proxy:$(VERSION)
PROXY_DEV_TAG ?= mesh-route-proxy:development
proxy-image:
docker build --build-arg GO_BASE=$(GO_BASE) -f modules/route-proxy/Dockerfile -t $(PROXY_IMAGE) -t $(PROXY_DEV_TAG) .
@echo
@docker image inspect $(PROXY_IMAGE) --format 'built {{.RepoTags}} {{.Size}} bytes'
# The whole check. Raises a database, runs the repository's own check against it -- merge-check.sh,
# the very script `mesh/repo-check` runs -- and takes the database down again, including when the
# check fails, which is why the teardown is not conditional and the script's exit status is the
@@ -117,7 +108,7 @@ proxy-image:
# as the script runs them: a bus per test, of the release the mesh runs, makes the suite the same in any
# order, and the timeout bounds a hang to a failure with a stack, never a stalled gate.
check: postgres
@sh merge-check.sh ; status=$$? ; docker rm -f $(PG_CONTAINER) >/dev/null 2>&1 || true ; exit $$status
@sh merge-check.sh ; status=$$? ; docker rm -f -v $(PG_CONTAINER) >/dev/null 2>&1 || true ; exit $$status
# The unit level alone (untagged: no bus, no store, no other process), which a developer runs continuously. The tests
# that need a bus or the store carry `//go:build integration` and run in `check`, not here; without a database the
@@ -135,7 +126,7 @@ fmt:
if [ -n "$$unformatted" ] ; then echo "not gofmt'd:" ; echo "$$unformatted" ; exit 1 ; fi
postgres:
@docker rm -f $(PG_CONTAINER) >/dev/null 2>&1 || true
@docker rm -f -v $(PG_CONTAINER) >/dev/null 2>&1 || true
@docker run -d --name $(PG_CONTAINER) -e POSTGRES_PASSWORD=check \
-p 127.0.0.1:$(PG_PORT):5432 $(PG_IMAGE) >/dev/null
@printf 'waiting for postgres'
@@ -147,7 +138,7 @@ postgres:
echo ' — never came up' ; docker logs $(PG_CONTAINER) | tail -20 ; exit 1
postgres-stop:
@docker rm -f $(PG_CONTAINER) >/dev/null 2>&1 || true
@docker rm -f -v $(PG_CONTAINER) >/dev/null 2>&1 || true
clean:
rm -rf build/
+91 -2
View File
@@ -6,7 +6,11 @@ import (
"os"
"path/filepath"
"strings"
"sync"
"sync/atomic"
"time"
"github.com/novox/mesh-controller/internal/builder"
)
// **The build agent says whether it is busy, and being asked drains it** (novox/hq ADR 0335 §1, §5). Its
@@ -73,14 +77,77 @@ func drainedTooLong(now time.Time) error {
"within %s (novox/hq ADR 0335)", began.UTC().Format(time.RFC3339), drainTooLong)
}
// inHand is true while this process has claimed a build and not yet released it. The mark in the temporary directory
// is only believed while it is: a mark with no build of this process behind it is stale (a crash, a fault in an
// earlier path) and is removed, because left alone it hands back every ask for ever (issue 1768).
var inHand atomic.Bool
// markRetry is how long an agent that cannot mark a build in hand takes no work, and markFault when it last failed to.
// Taking nothing is the point: an ask fetched and handed back at once, many times a second, spends all its deliveries.
const markRetry = 30 * time.Second
var (
markMu sync.Mutex
markFault time.Time
markSaid string
)
// markFaulty says the agent could not mark a build in hand and has not yet been able to since: it takes no work. Once
// markRetry has passed it tries the mark itself, with no ask in hand, so that an ask is fetched only when marking works.
func markFaulty(now time.Time) bool {
markMu.Lock()
failed := markFault
markMu.Unlock()
if failed.IsZero() {
return false
}
if now.Sub(failed) < markRetry {
return true
}
f, err := os.OpenFile(inHandMark(), os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0o644)
if err != nil && errors.Is(err, os.ErrExist) && !inHand.Load() {
_ = os.Remove(inHandMark())
f, err = os.OpenFile(inHandMark(), os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0o644)
}
if err != nil {
markFailed(err)
return true
}
_ = f.Close()
_ = os.Remove(inHandMark())
markMu.Lock()
markFault, markSaid = time.Time{}, ""
markMu.Unlock()
fmt.Fprintf(os.Stderr, "the mark of a build in hand can be made again (%s): taking builds\n", inHandMark())
return false
}
// markFailed notes that a build could not be marked in hand, and says it when it is a new reason.
func markFailed(err error) {
markMu.Lock()
defer markMu.Unlock()
markFault = time.Now()
if said := err.Error(); said != markSaid {
markSaid = said
fmt.Fprintf(os.Stderr, "cannot mark a build in hand (%s), so this agent takes no build for %s and tries again: %v\n",
inHandMark(), markRetry, err)
}
}
// claim marks a request in hand, unless the agent is drained: then nothing is in hand, and the request goes
// back to the queue.
func claim() bool {
if !inHand.Load() {
// Nothing of this process is in hand, so a mark already there is stale.
if err := os.Remove(inHandMark()); err == nil {
fmt.Fprintf(os.Stderr, "removed the mark of a build in hand (%s): no build of this agent is in hand\n", inHandMark())
}
}
// Made, never written over: one build at a time, and a mark already there is a build in hand.
f, err := os.OpenFile(inHandMark(), os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0o644)
if err != nil {
// Unmarked, an ask would say idle while this build runs: the build is not taken.
fmt.Fprintf(os.Stderr, "cannot mark a build in hand (%s), so it is handed back: %v\n", inHandMark(), err)
markFailed(err)
return false
}
_, err = f.WriteString(time.Now().UTC().Format(time.RFC3339) + "\n")
@@ -89,7 +156,7 @@ func claim() bool {
}
if err != nil {
_ = os.Remove(inHandMark())
fmt.Fprintf(os.Stderr, "cannot mark a build in hand (%s), so it is handed back: %v\n", inHandMark(), err)
markFailed(err)
return false
}
if drained() {
@@ -97,6 +164,7 @@ func claim() bool {
fmt.Fprintln(os.Stderr, "asked whether busy, so this agent takes no new build: handed back")
return false
}
inHand.Store(true)
return true
}
@@ -107,6 +175,7 @@ func holding(what string) {
// released ends the build in hand.
func released() {
inHand.Store(false)
if err := os.Remove(inHandMark()); err != nil && !errors.Is(err, os.ErrNotExist) {
fmt.Fprintf(os.Stderr, "cannot remove the mark of the build in hand (%s): %v\n", inHandMark(), err)
}
@@ -144,3 +213,23 @@ func busy() (string, error) {
}
return fmt.Sprintf("busy: %s, in hand since %s; taking no new build", what, since), nil
}
var (
roomMu sync.Mutex
roomSaid bool
)
// noRoom says this node is under the floor of free space (novox/hq issue 1834): it takes no work, so another build node
// takes the ask instead of this one taking it and ending it in an error. Said once when it starts and again when it ends.
func noRoom(workspace string) bool {
err := builder.RoomToWork("/", workspace)
roomMu.Lock()
defer roomMu.Unlock()
if err != nil && !roomSaid {
fmt.Fprintf(os.Stderr, "taking no builds: %v\n", err)
} else if err == nil && roomSaid {
fmt.Fprintf(os.Stderr, "there is room again: taking builds\n")
}
roomSaid = err != nil
return err != nil
}
+77
View File
@@ -1,6 +1,7 @@
package main
import (
"github.com/novox/mesh-controller/internal/builder"
"os"
"strings"
"sync"
@@ -160,3 +161,79 @@ func TestAnUnknownCommandFails(t *testing.T) {
t.Fatalf("help failed: %v", err)
}
}
// 1768/AC2: a mark naming no build of this agent (left by a crash or a fault) is stale: the next ask removes it and is
// run, never fetched and handed back in a loop.
func TestAStaleInHandMarkIsRemovedAndTheAskRuns(t *testing.T) {
t.Setenv("TMPDIR", t.TempDir())
inHand.Store(false)
if err := undrained(); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(inHandMark(), []byte("2026-10-21T18:45:06Z\n"), 0o644); err != nil {
t.Fatal(err)
}
if !claim() {
t.Fatal("a stale in-hand mark made the agent hand back an ask")
}
// A live one — this agent's own build in hand — is never taken over.
if claim() {
t.Fatal("a second build was taken while one is in hand")
}
if _, err := busy(); err != nil {
t.Fatalf("the build in hand was lost: %v", err)
}
released()
}
// 1768/AC3: an agent that cannot mark a build in hand for another reason takes no work, instead of fetching asks and
// handing them back; it takes work again once marking works.
func TestAnAgentThatCannotMarkABuildTakesNoWorkUntilItCan(t *testing.T) {
dir := t.TempDir()
t.Setenv("TMPDIR", dir)
inHand.Store(false)
markFault, markSaid = time.Time{}, ""
t.Cleanup(func() { markFault, markSaid = time.Time{}, "" })
if err := undrained(); err != nil {
t.Fatal(err)
}
// The mark's place is a directory that cannot be removed or made over: a directory named as the mark.
if err := os.MkdirAll(inHandMark()+"/x", 0o755); err != nil {
t.Fatal(err)
}
if claim() {
t.Fatal("an agent that cannot mark a build took it")
}
if !markFaulty(time.Now()) {
t.Fatal("an agent that cannot mark a build still takes work")
}
// Still failing after the retry time: still takes nothing, and no ask was needed to find out.
if !markFaulty(time.Now().Add(2 * markRetry)) {
t.Fatal("an agent whose mark still cannot be made takes work")
}
if err := os.RemoveAll(inHandMark()); err != nil {
t.Fatal(err)
}
if markFaulty(time.Now().Add(4 * markRetry)) {
t.Fatal("an agent whose mark can be made again takes no work")
}
if !claim() {
t.Fatal("an agent that can mark takes no build")
}
released()
}
// 1834/AC4: given a node under the floor of free space, when the taking loop asks, then it takes nothing, and takes again
// once there is room.
func TestANodeUnderTheFloorOfFreeSpaceTakesNoWork(t *testing.T) {
was := builder.FreeBytes
t.Cleanup(func() { builder.FreeBytes = was })
builder.FreeBytes = func(string) (uint64, error) { return 2 << 30, nil }
if !noRoom(t.TempDir()) {
t.Error("a node with 2 GiB free took work")
}
builder.FreeBytes = func(string) (uint64, error) { return 200 << 30, nil }
if noRoom(t.TempDir()) {
t.Error("a node with 200 GiB free refused work")
}
}
+14
View File
@@ -36,3 +36,17 @@ func TestTheSiblingsAnAskNamesReachTheSpecAndTheOutcome(t *testing.T) {
t.Fatalf("a request that names none has siblings %+v", plain.Siblings)
}
}
// 1826/AC2: the held values a check request carries reach the spec; a request that carries none leaves it empty.
func TestTheHeldValuesAnAskCarriesReachTheSpec(t *testing.T) {
spec := checkSpecOf(link.BuildRequest{Check: &link.CheckRequest{Owner: "novox", Repo: "mesh-catalog",
HeldConfig: []link.HeldConfig{{Pull: "novox/mesh-catalog#434", Head: "a", Node: "ace", Module: "n8n", Key: "media-gid",
Value: []byte("2000"), Approved: "the operator"}}}})
if len(spec.Held) != 1 || spec.Held[0].Node != "ace" || spec.Held[0].Key != "media-gid" || string(spec.Held[0].Value) != "2000" ||
spec.Held[0].Approved != "the operator" {
t.Fatalf("the spec reads %+v", spec.Held)
}
if plain := checkSpecOf(link.BuildRequest{Check: &link.CheckRequest{Owner: "o", Repo: "r"}}); plain.Held != nil {
t.Fatalf("a request that carries none has %+v", plain.Held)
}
}
+2 -2
View File
@@ -91,7 +91,7 @@ func TestKillEndsTheBuildRunningHereAndNoOther(t *testing.T) {
t.Errorf("kill said %q", said)
}
// Removed at the kill and again once the build had ended: a container made in between is caught.
if len(removed) != 4 || !strings.Contains(removed[0], "label=mesh.build=build-1") || !strings.Contains(removed[2], "label=mesh.build=build-1") {
if len(removed) != 6 || !strings.Contains(removed[0], "label=mesh.build=build-1") || !strings.Contains(removed[3], "label=mesh.build=build-1") {
t.Errorf("removed %v", removed)
}
if c := h.current(); c.Running != nil {
@@ -160,7 +160,7 @@ func TestAHolderStartingRemovesWhatEarlierBuildsLeftHere(t *testing.T) {
}
return "", nil
}
if n := h.removeLeftBehind(); n != 2 || len(removed) != 2 || removed[1] != "docker rm -f c1 c2" {
if n := h.removeLeftBehind(); n != 2 || len(removed) != 3 || removed[2] != "docker rm -f -v c1 c2" {
t.Fatalf("removed %d: %v", n, removed)
}
h.remove = func(context.Context, string, string, ...string) (string, error) {
+11 -1
View File
@@ -64,6 +64,8 @@ is dialled except the broker.
MESH_NPM_TOKEN the token for it, likewise
MESH_NPM_SCOPE the scope it answers for (default: @novox)
MESH_WORKSPACE where to clone and build (default: a temporary directory)
MESH_BUILD_CGROUP_PARENT the slice every container a build or check starts joins, which carries the
machine's limits on builds (novox/hq issue 1798; default: the runtime's own)
It also builds one module and stops, which is how a mesh is raised — before there is a
broker to take work from or a registry to publish into:
@@ -137,6 +139,10 @@ func run() error {
if err != nil {
return err
}
// A build and a check run below the machine's own work (novox/hq issue 1798): the agent itself at the
// lowest priority, which its git and go children inherit, and every container in the build slice.
_ = syscall.Setpriority(syscall.PRIO_PROCESS, 0, 19)
builder.CgroupParent = strings.TrimSpace(os.Getenv("MESH_BUILD_CGROUP_PARENT"))
workspace := os.Getenv("MESH_WORKSPACE")
if workspace == "" {
workspace = os.TempDir() + "/mesh-builder"
@@ -184,7 +190,7 @@ func run() error {
// Paused or drained, it takes nothing new (novox/hq ADR 0219, ADR 0335): a drained agent finishes the build
// in hand and then says idle, and its container is replaced between builds.
machine := link.MachineOverNATSWith(js, on, seat, link.MachineOptions{
Paused: func() bool { return h.Paused() || drained() },
Paused: func() bool { return h.Paused() || drained() || markFaulty(time.Now()) || noRoom(workspace) },
Taking: func() { taking(seat) },
Claim: claim,
})
@@ -499,6 +505,10 @@ func checkSpecOf(request link.BuildRequest) builder.CheckSpec {
Modules: c.Modules, New: c.New, Manifests: c.Manifests, Judge: c.Judge, Base: c.Base,
Toolchain: builder.ToolchainOf(request.Held), Toolchains: builder.ToolchainsOf(request.Held),
Tests: c.Tests, StoreOnly: c.StoreOnly, SiblingsUnknown: c.SiblingsUnknown}
for _, h := range c.HeldConfig {
spec.Held = append(spec.Held, builder.HeldValue{Pull: h.Pull, Head: h.Head, Node: h.Node, Module: h.Module, Key: h.Key,
Value: h.Value, Approved: h.Approved, By: h.By, At: h.At})
}
for _, s := range c.Siblings {
spec.Siblings = append(spec.Siblings, builder.Sibling{Repository: s.Repository, Number: s.Number, Head: s.Head,
Base: s.Base, Dir: s.Dir})
+4
View File
@@ -645,6 +645,10 @@ func doesWords(act conditions.Action) string {
return "the delivery is stopped"
case act.Verb == broker.ControllerSeat+".plans" && act.Arguments["retry"] != "":
return "the failed builds are asked again"
case act.Verb == broker.ControllerSeat+".push" && act.Arguments["node"] != "":
return "the held builds are sent to " + act.Arguments["node"]
case act.Verb == broker.ControllerSeat+".upgrade" && act.Arguments["release-backlog"] != "":
return "the builds held after the failed delivery are released"
case strings.HasSuffix(act.Verb, ".restart"):
return "its service is restarted on " + act.Machine
}
+27 -2
View File
@@ -191,6 +191,16 @@ func (f following) pullUpdated(ctx context.Context, p link.PullUpdated, w io.Wri
scope.Modules, scope.Dependents, scope.New, scope.Manifests = nil, nil, nil, nil
composedGate(&direct)
}
// **Mesh-config is told what the head declares at every announced head of a pull request in a repository that holds
// modules** (novox/hq ADR 0502 clauses 2 and 6, issue 1826), also when the head touches no module: a head that reverts
// the change must drop the value held for it. Asked of nothing else here.
this := pullAtOf(p)
if this.Truncated {
this.Paths, this.Truncated = truncatedPaths(entries, p.Owner, p.Repo, scope.Modules), false
}
if scope.gated() || holdsModules(entries, p.Owner, p.Repo) {
tellPullDeclares(ctx, w, []pullAt{this})
}
switch {
case !scope.gated() && !scope.Mesh:
fmt.Fprintf(w, "%s/%s#%d (%.8s): %s, and the repository is not the mesh's: said, nothing run\n",
@@ -209,6 +219,9 @@ func (f following) pullUpdated(ctx context.Context, p link.PullUpdated, w io.Wri
}
request.Check.Plan = &plan
request.Check.Composed = composed
if !composed {
carryHeld(ctx, w, &request, []pullAt{this})
}
// **The group's other open pull requests are read beside it** (novox/hq ADR 0382 §1).
siblingsFrom(&request, p, entries, lines, groupsErr)
seat := buildSeatHeld(ctx)
@@ -594,11 +607,13 @@ func checkedOf(result link.BuildResult) link.Checked {
// A delivery group's composed check (novox/hq ADR 0239): every head it judged, this one first.
if g := result.Checked; g.Group != "" {
c.Group = g.Group
// Every member is named with its trunk commit (novox/hq issue 1106): the seat's word, else the one the forge
// announced with the head, so the forge's merge tool never meets a member whose verdict names none.
c.Members = append(c.Members, link.CheckedMember{Owner: g.Owner, Repo: g.Repo, Number: g.Number,
Commit: result.Ref, Trunk: c.Trunk})
Commit: result.Ref, Trunk: firstTrunk(c.Trunk, g.Trunk)})
for _, m := range g.Members {
c.Members = append(c.Members, link.CheckedMember{Owner: m.Owner, Repo: m.Repo, Number: m.Number,
Commit: m.Ref, Trunk: memberTrunk(result.Check, m)})
Commit: m.Ref, Trunk: firstTrunk(memberTrunk(result.Check, m), m.Trunk)})
}
}
}
@@ -627,6 +642,16 @@ func checkedOf(result link.BuildResult) link.Checked {
return c
}
// firstTrunk is the first trunk commit named, as twelve characters; empty when none is.
func firstTrunk(named ...string) string {
for _, n := range named {
if n != "" {
return twelve(n)
}
}
return ""
}
// memberTrunk is the trunk commit the build seat merged one of a group's heads into (novox/hq ADR 0323 §6), empty
// when it said none.
func memberTrunk(outcome *link.CheckOutcome, m link.GroupMember) string {
+25
View File
@@ -250,6 +250,31 @@ func TestAVerdictNamesTheTrunkCommitItJudged(t *testing.T) {
}
}
// **A group's composed verdict names every member's trunk commit** (novox/hq issue 1106): a member the build seat named no
// trunk for (an SDK member whose base the seat did not merge, or a seat from before the rule) is named by the trunk the forge
// announced with its head, and the primary the same, so the merge tool never reads "names no trunk commit" on a member
// whose trunk did not move.
func TestAGroupVerdictNamesEveryMembersTrunkTheSDKMemberToo(t *testing.T) {
group := &link.CheckRequest{Owner: "novox", Repo: "mesh-catalog", Number: 7, Base: "main", Group: "feat/x",
Trunk: "dddddddddddd0000",
Members: []link.GroupMember{
{Owner: "novox", Repo: "mesh-sdk", Number: 8, Ref: "cccccccccccc0000", Base: "main", Trunk: "eeeeeeeeeeee0000"},
{Owner: "novox", Repo: "mesh-host", Number: 9, Ref: "bbbbbbbbbbbb0000", Base: "main", Trunk: "ffffffffffff0000"}}}
c := checkedOf(link.BuildResult{ID: "g", Ref: "aaaaaaaaaaaa0000", Checked: group, Check: &link.CheckOutcome{
Verdict: "pass", Head: "aaaaaaaaaaaa", Gate: &link.CheckLayer{Verdict: "pass"},
Members: []link.CheckedMember{{Owner: "novox", Repo: "mesh-host", Commit: "bbbbbbbbbbbb", Trunk: "222222222222"}}}})
want := []string{"dddddddddddd", "eeeeeeeeeeee", "222222222222"}
if len(c.Members) != 3 {
t.Fatalf("the composed check names its members %+v", c.Members)
}
for i, m := range c.Members {
if m.Trunk != want[i] {
t.Fatalf("member %d (%s/%s) is named with trunk %q, want %q (the seat's word first, else the announced one): %+v",
i, m.Owner, m.Repo, m.Trunk, want[i], c.Members)
}
}
}
// **The merge made for a check is never registered** (novox/hq ADR 0323 §2): a build of it — a commit on no branch
// of the forge — is refused, naming the trunk. The rule of ADR 0238 §1 stands unchanged; this holds it for the
// merge commit a check makes.
+5 -1
View File
@@ -239,7 +239,11 @@ func markUnresolvable(err error) error {
// still names anything foreseen is refused, said, and never sent.
func composeSend(ctx context.Context, open *stores, node string) (sendable, error) {
declared, _, err := composeSendValidated(ctx, open, node)
return declared, err
if err != nil {
return declared, err
}
// Never a declaration naming an image the store does not hold (novox/hq issue 1742): that module is left out.
return withoutMissingImages(ctx, open, node, declared), nil
}
// composeSendValidated is composeSend and what the node-engine's validator refuses of the declaration it composed, for the
+7 -1
View File
@@ -1177,6 +1177,12 @@ func askGroupCheck(ctx context.Context, open *stores, group string, members []or
return "", err
}
request.Check.Group = group
// **The values held for the keys the members introduce are told and carried** (novox/hq ADR 0502, issue 1826).
var pulls []pullAt
for i, s := range changeSourcesOf(members, entries, reach) {
pulls = append(pulls, pullAt{changeSource: s, Number: members[i].pull().Number})
}
withHeldConfig(ctx, os.Stderr, &request, pulls, true)
// A group's check is queued as ready when any member's pull request is marked ready.
request.Check.Ready = slices.ContainsFunc(members, func(m orderMember) bool { return m.Ready })
world, err := restOfTheMesh(ctx, open, shelfOf(entries), "")
@@ -1204,7 +1210,7 @@ func askGroupCheck(ctx context.Context, open *stores, group string, members []or
return "", fmt.Errorf("%s cannot be cloned for the group's check: %w", m.ID, err)
}
request.Check.Members = append(request.Check.Members, link.GroupMember{Owner: mp.Owner, Repo: mp.Repo,
Number: mp.Number, Repository: url, Ref: mp.Commit, Paths: mp.Paths, Base: mp.Base})
Number: mp.Number, Repository: url, Ref: mp.Commit, Paths: mp.Paths, Base: mp.Base, Trunk: mp.BaseSHA})
}
seat := buildSeatHeld(ctx)
ask, err := askOverOn(seat)
@@ -59,6 +59,18 @@ func TestAMergeWaitsForItsDeliverysWordAndThePersonsWordWorksWithoutIt(t *testin
}
}
// heardOf has this mesh's own view of the plans take in the plan as the bucket holds it. A verb of the
// controller (deliveryCommand, plansCommand) opens an inventory of its own and writes the plan there; this
// mesh's inventory learns of the write from the bucket's watch, a moment later, and a loaded machine makes the
// moment long enough for advanceHeld to read the plan as it was, still waiting for its word, and ask nothing
// (issue 1399). Reading the plan from the bucket puts it into the view at once, with no wait to bound.
heardOf := func(id string) {
t.Helper()
if _, err := inv.PlanByID(ctx, id); err != nil {
t.Fatal(err)
}
}
// No holder on record: the merge starts its walk, as before.
p := merge("c1aaaaaaaa", "modules/app/index.ts")
if p.Waiting() || len(*asked) != 1 {
@@ -90,6 +102,7 @@ func TestAMergeWaitsForItsDeliverysWordAndThePersonsWordWorksWithoutIt(t *testin
if err := deliveryCommand(ctx, []string{"go", p.ID}); err == nil {
t.Fatal("a walk was let go twice")
}
heardOf(p.ID)
advanceHeld(ctx, open)
if len(*asked) != 2 {
t.Fatalf("the word did not start the walk: %v", *asked)
@@ -119,6 +132,7 @@ func TestAMergeWaitsForItsDeliverysWordAndThePersonsWordWorksWithoutIt(t *testin
if err := plansCommand(ctx, []string{"go", p.ID, "--why", "mesh-delivery is down"}); err != nil {
t.Fatal(err)
}
heardOf(p.ID)
advanceHeld(ctx, open)
got, _ = inv.PlanByID(ctx, p.ID)
if got.Waiting() || !strings.HasPrefix(got.Delivery.By, "a person") || len(*asked) < 4 {
+1 -1
View File
@@ -68,7 +68,7 @@ func TestTheForgeNotAnsweringInTimeIsNotAskedAgain(t *testing.T) {
// An ask that hangs past the bound is cut off at the bound, not at its own wait, and the failure still says it
// waited: the receive loop is held no longer than the bound (novox/hq issue 486).
func TestAnAskHangingPastTheBoundIsCutOffAtIt(t *testing.T) {
w := waitOut{bound: 200 * time.Millisecond, first: 10 * time.Millisecond, most: 50 * time.Millisecond,
w := waitOut{bound: 100 * time.Millisecond, first: 10 * time.Millisecond, most: 50 * time.Millisecond,
now: time.Now, sleep: sleepOrDone}
asked := 0
began := time.Now()
+198
View File
@@ -0,0 +1,198 @@
package main
import (
"context"
"fmt"
"sort"
"strings"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
)
// A move held for a person is asked of the person (novox/hq issue 1732, issue 1320; ADR 0377, ADR 0259).
//
// **A wait only a person can end is said to that person at once.** ADR 0377 holds a recorded module's move that
// recreates a container or touches its data until a person pushes the machine. The walk said so in its note and
// ended "done", `status` named the machine behind, and nobody was asked: on 2026-10-21 mailu's new build waited on
// novox for 40 minutes behind a plan that read "done, 3 tier(s)", and on 2026-10-19 mssql waited on ace the same way.
// Row S24 raises one condition per machine that has such a move, for the operator, with the answer **Release**: the
// push of that machine, which the controller's own `push` verb performs on the operator's warrant (ADR 0259). It ends
// when the machine no longer waits, whoever pushed it.
//
// A hold a person chose is not asked again — a module they set to `record` with a why, and the bus, whose step has
// its own row (S17).
// kindMovesHeld is S24's kind: machine.<node>.moves-held.
const kindMovesHeld = "moves-held"
// heldFacts is one machine and the moves of recorded modules that wait there for a person.
type heldFacts struct {
node string
moves []heldMove
}
// gatherHeldMoves is, per machine that holds at least one move no person chose to hold, all its held moves, by machine.
func gatherHeldMoves(ctx context.Context, inv *inventory.Inventory) ([]heldFacts, error) {
nodes, err := inv.Nodes(ctx)
if err != nil {
return nil, fmt.Errorf("the machines cannot be read: %w", err)
}
f, err := readMoveFacts(ctx, inv)
if err != nil {
return nil, err
}
var out []heldFacts
for _, n := range nodes {
moves, err := heldMovesOn(ctx, inv, f, n.Name)
if err != nil {
return nil, err
}
h := heldFacts{node: n.Name, moves: moves}
if len(h.asked()) > 0 {
out = append(out, h)
}
}
return out, nil
}
// asked are the moves no person chose to hold, and chosen the ones a person did (a module set to record with a why, the bus).
func (h heldFacts) asked() []heldMove { return h.byChoice(false) }
func (h heldFacts) chosen() []heldMove { return h.byChoice(true) }
func (h heldFacts) byChoice(chosen bool) []heldMove {
var out []heldMove
for _, m := range h.moves {
if m.Chosen == chosen {
out = append(out, m)
}
}
return out
}
// modules are the held moves' modules, in order, of all of them.
func (h heldFacts) modules() []string { return moduleNames(h.moves) }
func moduleNames(moves []heldMove) []string {
out := make([]string, 0, len(moves))
for _, m := range moves {
out = append(out, m.Module)
}
return out
}
// releaseAction is the one answer a held move offers: the push of the machine, approved by the operator. The push
// names every module whose move waits there as one that may move its data: the operator's Approve is the word
// ADR 0217 asks for, and the ask says so. Offered only where no person's own hold is on the machine too: a push
// composes every recorded module at its current build, so it would send that move as well, which the ask never named
// (ADR 0259 §6), and hold the whole machine if that module's data would move.
func (h heldFacts) releaseAction() conditions.Action {
return conditions.Action{Label: "Release", Verb: "mesh-controller.push", Level: conditions.LevelApprove,
Arguments: map[string]string{"node": h.node, "move": strings.Join(moduleNames(h.asked()), ","), "why": "",
"cause": conditions.CauseOperatorAnswer}}
}
// heldWords are the plain words of a machine's held moves: which modules wait, why in general, and what an Approve does.
func heldWords(h heldFacts) (headline, explanation, needs string, resolved string) {
names := namesWords(moduleNames(h.asked()), 3)
headline = clipWords(conditions.Capital(names)+" waits for you on "+h.node, conditions.HeadlineMax)
recreates, data := false, false
for _, m := range h.asked() {
switch {
case strings.Contains(m.Why, "recreates"):
recreates = true
default:
data = true
}
}
why := "would change what it runs"
switch {
case recreates && data:
why = "would restart a container or touch its data"
case recreates:
why = "would restart one of its containers"
case data:
why = "would touch its files or data"
}
explanation = clipWords(fmt.Sprintf("The new version of %s %s on %s, so it does not go out with its delivery; "+
"it waits for you. Release sends %s everything it is owed, these among it, and lets their data move where it "+
"has to. Nothing is sent until you answer.", names, why, h.node, h.node), conditions.ExplanationMax)
needs = "release them " + FromMeshMCPServer
if chosen := h.chosen(); len(chosen) > 0 {
// A push of the machine would send the move a person chose to hold as well: no answer is offered here.
explanation = clipWords(fmt.Sprintf("The new version of %s %s on %s, so it does not go out with its delivery; "+
"it waits for you. %s is also held there by your own choice, and a push of %s would send that too, so "+
"nothing is offered here: the push is yours at the controller's terminal.",
names, why, h.node, conditions.Capital(namesWords(moduleNames(chosen), 3)), h.node), conditions.ExplanationMax)
needs = fmt.Sprintf("push %s at the controller's terminal, once you have decided about %s.", h.node,
namesWords(moduleNames(chosen), 3))
}
resolved = "Resolved: " + names + " is no longer waiting on " + h.node
return
}
// watchMovesHeld is S24: a machine that has moves of recorded modules waiting for a person, said to the operator with the
// answer that sends them. The condition ends when the machine no longer waits: it is no longer among the facts.
func watchMovesHeld(f *signalFacts) []conditions.Observation {
var out []conditions.Observation
for _, h := range f.held {
var said []string
for _, m := range h.moves {
said = append(said, fmt.Sprintf("%s %s → %s %s", m.Module, short(m.From), short(m.To), m.Why))
}
headline, explanation, needs, resolved := heldWords(h)
if c := h.chosen(); len(c) > 0 {
said = append(said, "held by a person's own choice: "+strings.Join(moduleNames(c), ", "))
}
var actions []conditions.Action
if len(h.chosen()) == 0 {
actions = []conditions.Action{h.releaseAction()}
}
out = append(out, conditions.Observation{Scope: conditions.ScopeMachine, ID: h.node, Kind: kindMovesHeld,
Machine: h.node, Severity: conditions.Warning, Resolver: conditions.ResolverOperator,
Summary: fmt.Sprintf("%s has %d move(s) of recorded modules waiting for a person's push: %s — `push %s` sends them",
h.node, len(h.moves), strings.Join(h.modules(), ", "), h.node),
Said: strings.Join(said, "; "),
Headline: headline, Explanation: explanation, Needs: needs, Resolved: resolved,
Actions: actions})
}
return out
}
// planHeld is, for a plan that ended or is running, the moves its modules carried that a machine still holds for a
// person: "module on machine", sorted. The plan's own record says the move waited when it was judged
// (PlanModule.Recorded); the machines say whether it still does, and for the build this plan carried, so a plan older
// than the build now held is not named.
func planHeld(p inventory.Plan, held []heldFacts) []string {
var out []string
for _, h := range held {
for _, m := range h.moves {
s := p.Modules[m.Module]
if s == nil || !strings.HasPrefix(s.Recorded[h.node], "waits:") || !sameCommit(s.Commit, m.To) {
continue
}
out = append(out, m.Module+" on "+h.node)
}
}
sort.Strings(out)
return out
}
// heldSaid is what a plan's line adds when it carries a move that a machine holds for a person (novox/hq issue 1732): a
// plan whose module waits for a push is not plainly "done".
func heldSaid(p inventory.Plan, held []heldFacts) string {
waiting := planHeld(p, held)
if len(waiting) == 0 {
return ""
}
return fmt.Sprintf(" — %d held for a person: %s (the operator is asked)", len(waiting), strings.Join(waiting, ", "))
}
// heldNow reads the machines' held moves for a plan's line, nil when they cannot be read: the line then reads as before.
func heldNow(ctx context.Context, inv *inventory.Inventory) []heldFacts {
held, err := gatherHeldMoves(ctx, inv)
if err != nil {
return nil // empty-on-error: a plan's line then reads as it did before; the watchdog (S24) says it cannot read
}
return held
}
@@ -0,0 +1,92 @@
//go:build integration
package main
import (
"context"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/inventory"
)
func inventoryUpgrade(why string) inventory.Upgrade { return inventory.Upgrade{Why: why, By: "jochen"} }
func mustMoveFacts(t *testing.T, rm *recordedMesh) moveFacts {
t.Helper()
f, err := readMoveFacts(t.Context(), rm.open.inventory)
if err != nil {
t.Fatal(err)
}
return f
}
// 1732/AC8 (integration): a mesh running mailu at c1 on two machines, whose registered c2 recreates a container, is read
// as two machines each holding mailu's move for a person; a bundle-only c3 on top changes nothing of it; after a machine
// is sent c2 (as a push records it) it is no longer among them, and plans that carried c2 say nothing more of it.
func TestMachinesHoldingAMoveForAPersonAreReadFromTheRecords(t *testing.T) {
rm := aRecordedMesh(t, aMailu(true))
c2 := aMailu(true)
resourceOf(&c2, "imap")["image"] = "registry.invalid/imap@sha256:2"
rm.register(t, c2, "c2", time.Now().Add(-time.Minute))
ctx := t.Context()
held, err := gatherHeldMoves(ctx, rm.open.inventory)
if err != nil {
t.Fatal(err)
}
if len(held) != 2 || held[0].node != "anchor" || held[1].node != "laptop" || len(held[0].moves) != 1 ||
held[0].moves[0].Module != "mailu" || held[0].moves[0].To != "c2" || !strings.Contains(held[0].moves[0].Why, "recreates") {
t.Fatalf("held %+v", held)
}
if err := rm.open.inventory.RecordSent(ctx, nodeID(t, rm.open, "anchor"), "d-anchor-c2", map[string]string{"mailu": "c2"}); err != nil {
t.Fatal(err)
}
held, err = gatherHeldMoves(ctx, rm.open.inventory)
if err != nil || len(held) != 1 || held[0].node != "laptop" {
t.Fatalf("after anchor was sent c2: %+v (%v)", held, err)
}
}
// 1732/AC9 (integration): a module a person set to record is no ask: they chose the hold.
func TestAHoldAPersonChoseIsNotAsked(t *testing.T) {
rm := aRecordedMesh(t, aMailu(false))
c2 := aMailu(false)
c2.Bundles[0].Digest = "sha256:bbb"
rm.register(t, c2, "c2", time.Now().Add(-time.Minute))
if err := rm.open.inventory.SetUpgradeOf(t.Context(), "mailu", inventoryUpgrade("the migration week")); err != nil {
t.Fatal(err)
}
held, err := gatherHeldMoves(t.Context(), rm.open.inventory)
if err != nil || len(held) != 0 {
t.Fatalf("a machine holding only a person's own hold is asked: %+v (%v)", held, err)
}
if moves, _ := heldMovesOn(t.Context(), rm.open.inventory, mustMoveFacts(t, rm), "anchor"); len(moves) != 1 || !moves[0].Chosen {
t.Fatalf("status still names it, marked as chosen: %+v", moves)
}
}
// 1742/AC4 (integration): a registered build naming an image of the store that the store lacks is read as a missing
// image on each machine that runs the module; one the store holds is not; and what the store holds is not asked twice.
func TestARegisteredBuildNamingAnImageTheStoreLacksIsRead(t *testing.T) {
m := aMailu(true)
resourceOf(&m, "imap")["image"] = "artifact-store://mailu/imap@sha256:" + strings.Repeat("e", 64)
rm := aRecordedMesh(t, m)
asked := 0
lacking := func(_ context.Context, reference string) (bool, error) { asked++; return false, nil }
imagesHeld.Lock()
imagesHeld.held = map[string]bool{}
imagesHeld.Unlock()
got, err := gatherMissingImages(t.Context(), rm.open.inventory, lacking)
if err != nil || len(got) != 2 || got[0].Module != "mailu" || got[0].Node != "anchor" || got[1].Node != "laptop" {
t.Fatalf("missing %+v (%v)", got, err)
}
holding := func(_ context.Context, reference string) (bool, error) { asked++; return true, nil }
if got, err := gatherMissingImages(t.Context(), rm.open.inventory, holding); err != nil || len(got) != 0 {
t.Fatalf("an image the store holds is missing: %+v (%v)", got, err)
}
before := asked
if _, err := gatherMissingImages(t.Context(), rm.open.inventory, lacking); err != nil || asked != before {
t.Fatalf("an image the store was found to hold was asked about again (%d, %v)", asked-before, err)
}
}
+237
View File
@@ -0,0 +1,237 @@
package main
import (
"context"
"errors"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/asks"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
)
// **A move held for a person is asked of the person** (novox/hq issue 1732, issue 1320; ADR 0377, ADR 0259).
// aHeldMailu is novox holding mailu's new build because it would recreate a container: the walk's own words.
func aHeldMailu() heldFacts {
return heldFacts{node: "novox", moves: []heldMove{{Module: "mailu", From: "b2f5ebb2aaaa", To: "3259765dbbbb",
Why: "waits: recreates 1 of mailu's 12 container(s) (with a new image: automx)"}}}
}
func heldConditionOn(f *signalFacts) conditions.Condition {
o := watchMovesHeld(f)[0]
return conditions.Condition{Key: o.Key(), Kind: o.Kind, Severity: o.Severity, Headline: o.Headline,
Explanation: conditions.Verdict(o.Needs, o.Explanation), Needs: o.Needs, Actions: o.Actions}
}
// 1732/AC1: a machine holding a move of a recorded module for a person raises one condition for the operator, with
// Release as its only answer: the machine's push, approved, naming the modules whose moves wait there.
func TestAHeldMoveIsRaisedWithReleaseAsItsOnlyAction(t *testing.T) {
f := calm(time.Date(2026, 10, 21, 17, 30, 0, 0, time.UTC))
f.held = []heldFacts{aHeldMailu()}
got := watchMovesHeld(f)
if len(got) != 1 {
t.Fatalf("%+v", got)
}
o := got[0]
if o.Kind != "moves-held" || o.Scope != conditions.ScopeMachine || o.ID != "novox" || o.Severity != conditions.Warning ||
o.Resolver != conditions.ResolverOperator || o.Key() != "machine.novox.moves-held" {
t.Fatalf("kind, scope, subject, severity or resolver: %+v (%s)", o, o.Key())
}
if len(o.Actions) != 1 {
t.Fatalf("actions: %+v", o.Actions)
}
a := o.Actions[0]
if a.Label != "Release" || a.Verb != "mesh-controller.push" || a.Level != conditions.LevelApprove ||
a.Arguments["node"] != "novox" || a.Arguments["move"] != "mailu" || a.Arguments["cause"] != conditions.CauseOperatorAnswer {
t.Fatalf("the action is not the push of novox: %+v", a)
}
if _, takes := a.Arguments["why"]; !takes {
t.Fatal("the push takes a why, which the asker fills in with the operator's answer")
}
for _, want := range []string{"mailu", "novox"} {
if !strings.Contains(o.Headline+o.Explanation, want) {
t.Errorf("the words do not name %q: %q / %q", want, o.Headline, o.Explanation)
}
}
if !strings.Contains(o.Said, "recreates 1 of mailu's 12") {
t.Errorf("the evidence does not carry the walk's own words: %q", o.Said)
}
if w := (conditions.Observation{Headline: o.Headline, Explanation: o.Explanation, Needs: o.Needs, Resolved: o.Resolved}); w.Headline == "" || len(o.Headline) > conditions.HeadlineMax {
t.Errorf("headline: %q", o.Headline)
}
}
// 1732/AC2: a machine with nothing held raises nothing, so the condition ends the look after the move is sent.
func TestAMachineHoldingNothingRaisesNoHeldMoveCondition(t *testing.T) {
f := calm(time.Date(2026, 10, 21, 17, 30, 0, 0, time.UTC))
if got := watchMovesHeld(f); len(got) != 0 {
t.Fatalf("%+v", got)
}
}
// 1732/AC3: the condition is asked on a channel at the approve level in words that say what Release does, and the
// operator's warrant calls the controller's own push with the machine and the modules, once.
func TestAHeldMovesReleaseIsAskedAndAWarrantPushesTheMachine(t *testing.T) {
f := calm(time.Date(2026, 10, 21, 17, 30, 0, 0, time.UTC))
f.held = []heldFacts{aHeldMailu()}
c := heldConditionOn(f)
r := newAskerRig(t)
r.open = []conditions.Condition{c}
if err := r.a.reconcile(context.Background()); err != nil {
t.Fatal(err)
}
sent := r.asksSent(t)
if len(sent) != 1 || len(sent[0].Options) != 1 {
t.Fatalf("asked %+v", sent)
}
o := sent[0].Options[0]
if o.Label != "Release" || o.Level != asks.Approve || o.Does != "the held builds are sent to novox" {
t.Errorf("the option is %+v", o)
}
if err := sent[0].Check(r.now); err != nil {
t.Errorf("the ask is refused as made: %v", err)
}
w := r.warrantFor(t, c.Key, "Release")
answerWith(t, r, w)
answerWith(t, r, w)
if len(r.called) != 1 || !strings.HasPrefix(r.called[0], "mesh-controller.push@ ") ||
!strings.Contains(r.called[0], `"node":"novox"`) || !strings.Contains(r.called[0], `"move":"mailu"`) ||
!strings.Contains(r.called[0], `"cause":"`+conditions.CauseOperatorAnswer+`"`) ||
!strings.Contains(r.called[0], "chose Release") {
t.Fatalf("called %v", r.called)
}
}
// 1732/AC4: the controller may call its own push and upgrade verbs on a warrant: the grant names them.
func TestTheControllerMayActOnAWarrantForPushAndUpgrade(t *testing.T) {
for _, verb := range []string{"push", "upgrade"} {
granted := false
for _, v := range broker.VerbsTheControllerActsOnAWarrant {
granted = granted || (v.Seat == broker.ControllerSeat && v.Verb == verb)
}
if !granted {
t.Errorf("the controller may not act on a warrant for %s", verb)
}
}
}
func aDoneMailuPlan() inventory.Plan {
return inventory.Plan{ID: "plan-1792585970446304628", Repository: "novox/mesh-catalog", State: inventory.PlanDone,
Tiers: [][]string{{"a"}, {"b"}, {"mailu"}},
Modules: map[string]*inventory.PlanModule{"mailu": {State: "built", Commit: "3259765dbbbbcccc",
Recorded: map[string]string{"novox": "waits: recreates 1 of mailu's 12 container(s)"}}}}
}
// 1732/AC5: a done plan that carries a move a machine still holds for a person says so on its line; once the machine
// no longer holds it, or holds another build of the module, the line says nothing more.
func TestADonePlanSaysWhatItHoldsForAPerson(t *testing.T) {
p := aDoneMailuPlan()
line := planLine(p, time.Date(2026, 10, 21, 16, 0, 0, 0, time.UTC)) + heldSaid(p, []heldFacts{aHeldMailu()})
if !strings.Contains(line, "done, 3 tier(s)") || !strings.Contains(line, "1 held for a person: mailu on novox") {
t.Errorf("the line: %q", line)
}
if said := heldSaid(p, nil); said != "" {
t.Errorf("a plan whose moves were sent says %q", said)
}
other := aHeldMailu()
other.moves[0].To = "9999999ffff"
if said := heldSaid(p, []heldFacts{other}); said != "" {
t.Errorf("a plan older than the build now held says %q", said)
}
p.Modules["mailu"].Recorded = map[string]string{"novox": recordedRollsWhy}
if said := heldSaid(p, []heldFacts{aHeldMailu()}); said != "" {
t.Errorf("a move the plan sent says %q", said)
}
}
// 1732/AC6: the builds held after a failed release are asked of the operator: the held-release condition offers
// Release, which calls the controller's own `upgrade` with release-backlog and nothing else, once, on a warrant.
func TestTheBuildsHeldAfterAFailedReleaseAreAskedAndReleasedOnAWarrant(t *testing.T) {
saved := backlogNow
t.Cleanup(func() { backlogNow = saved })
backlogNow.held = "release-1792595552499982311 failed (failed on novox); what waits is released again by a person"
backlogNow.waiting = map[string][]inventory.CarriedMove{"novox": {{Module: "indexer"}, {Module: "nextcloud"}}}
got := backlogObservation()
if len(got) != 1 || len(got[0].Actions) != 1 {
t.Fatalf("raised %+v", got)
}
a := got[0].Actions[0]
if a.Label != "Release" || a.Verb != "mesh-controller.upgrade" || a.Level != conditions.LevelApprove ||
a.Arguments["release-backlog"] != "true" || a.Arguments["cause"] != conditions.CauseOperatorAnswer {
t.Fatalf("the action: %+v", a)
}
o := got[0]
c := conditions.Condition{Key: o.Key(), Kind: o.Kind, Severity: o.Severity, Headline: o.Headline,
Explanation: conditions.Verdict(o.Needs, o.Explanation), Needs: o.Needs, Actions: o.Actions}
r := newAskerRig(t)
r.open = []conditions.Condition{c}
if err := r.a.reconcile(context.Background()); err != nil {
t.Fatal(err)
}
sent := r.asksSent(t)
if len(sent) != 1 || len(sent[0].Options) != 1 || sent[0].Options[0].Does != "the builds held after the failed delivery are released" {
t.Fatalf("asked %+v", sent)
}
w := r.warrantFor(t, c.Key, "Release")
answerWith(t, r, w)
answerWith(t, r, w)
if len(r.called) != 1 || !strings.HasPrefix(r.called[0], "mesh-controller.upgrade@ ") ||
!strings.Contains(r.called[0], `"release-backlog":"true"`) || !strings.Contains(r.called[0], "chose Release") {
t.Fatalf("called %v", r.called)
}
}
// 1732/AC7: a plan whose send waits for a gate while the builds are held after a failed release reads as waiting for
// a person to release them, not "building ... LATE"; with nothing held it still reads as the wait it is.
func TestAPlanWaitingForAGateBehindAFailedReleaseIsNotLate(t *testing.T) {
saved := backlogNow
t.Cleanup(func() { backlogNow = saved })
now := time.Date(2026, 10, 21, 18, 50, 0, 0, time.UTC)
p := inventory.Plan{ID: "plan-1", Repository: "novox/mesh-sdk", Commit: "c0ffee00", State: inventory.PlanBuilding,
Tiers: [][]string{{"indexer"}}, TierEntered: now.Add(-100 * time.Minute), Created: now.Add(-110 * time.Minute),
Modules: map[string]*inventory.PlanModule{"indexer": {State: "built"}},
Note: refusalNote(0, errors.New("a build waits there for its gate: novox"))}
backlogNow.held = ""
if line := planLine(p, now); !strings.Contains(line, "a build waits there for its gate: novox") {
t.Errorf("the wait is not said: %q", line)
}
backlogNow.held = "release-1 failed (failed on novox); what waits is released again by a person"
line := planLine(p, now)
if strings.Contains(line, "LATE") || strings.Contains(line, "building") ||
!strings.Contains(line, "until a person releases it") {
t.Errorf("a plan held behind a failed release reads %q", line)
}
}
// 1732/AC10: a machine that also holds a move a person chose to hold gets no Release, since a push would send that move
// too; the condition names both and says the push is the controller's terminal. A machine with only asked moves keeps it.
func TestAMachineAlsoHoldingAPersonsOwnHoldIsOfferedNoRelease(t *testing.T) {
f := calm(time.Date(2026, 10, 21, 17, 30, 0, 0, time.UTC))
h := aHeldMailu()
h.moves = append(h.moves, heldMove{Module: "nftables", From: "n1aaaa", To: "n2bbbb",
Why: "waits: a person set it to record, and holds every move of it until they push", Chosen: true})
f.held = []heldFacts{h}
got := watchMovesHeld(f)
if len(got) != 1 || len(got[0].Actions) != 0 {
t.Fatalf("%+v", got)
}
o := got[0]
text := o.Headline + o.Explanation + o.Needs + o.Summary + o.Said
for _, want := range []string{"mailu", "nftables", "terminal", "push novox"} {
if !strings.Contains(text, want) {
t.Errorf("does not say %q: %q", want, text)
}
}
w := conditions.Words{Headline: o.Headline, Explanation: o.Explanation, Resolved: o.Resolved, Needs: o.Needs}
if why, ok := conditions.PlainWords(w, "novox"); !ok {
t.Errorf("not plain: %s", why)
}
f.held = []heldFacts{aHeldMailu()}
if got := watchMovesHeld(f); len(got) != 1 || len(got[0].Actions) != 1 || got[0].Actions[0].Arguments["move"] != "mailu" {
t.Fatalf("a machine with only asked moves: %+v", got)
}
}
+245
View File
@@ -0,0 +1,245 @@
package main
import (
"context"
"fmt"
"slices"
"sort"
"strings"
"sync"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/artifacts"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
)
// No declaration is sent that names an image the artifact store does not hold (novox/hq issue 1742).
//
// On 2026-10-21 at 17:13 a send to novox named `mailu/automx@sha256:c62150b8…`; the build record keeps that digest
// ("recent-build") and the store did not hold it. The node-engine removed the running container, then could not pull
// the replacement, and every later apply on novox failed for the next hour while the mail clients' autoconfiguration
// had no backend. The node-engine's half (pull first, replace after) is its own change; this is the controller's: **a
// module whose image the store lacks is left out of the declaration**, as a module whose setting nobody gave is
// (ADR 0163, rule 6) — the node-engine keeps what it has of it and touches none of its containers — and **it is said**,
// row S25: a condition naming the module, the machine and the image, until the store holds the image again (a new build
// of the module, or the copy put back).
//
// Judged only for an image the store's own address names, from the send's own resources; an image of another registry
// is no business of the store's. A store that cannot be asked judges nothing: a send is not refused for a question
// that did not answer.
// kindImageMissing is S25's kind: module.<module>.<machine>.image-missing.
const kindImageMissing = "image-missing"
// missingImage is a module held out of a machine's declaration because the store lacks an image it names.
type missingImage struct {
Node, Module, Image string
}
// storeHolds is whether the store holds the manifest an image reference at its address names.
type storeHolds func(ctx context.Context, reference string) (bool, error)
// imageInStore is the artifact-store reference of an image the store's address names, "" for any other.
func imageInStore(image, address string) string {
if address == "" || !strings.HasPrefix(image, address+"/") || !strings.Contains(image, "@sha256:") {
return ""
}
reference := catalogue.Recorded(image)
if _, ours := catalogue.InArtifactStore(reference); !ours {
return ""
}
return reference
}
// holdBackMissingImages leaves out of the declaration every module that names an image the store does not hold,
// and answers what it left out. The module's resources go, it joins LeftOut with the reason, and its build stays the
// one the machine was last sent (`was`).
func holdBackMissingImages(ctx context.Context, node string, s *sendable, address string, holds storeHolds,
was map[string]string) []missingImage {
lacks := map[string]bool{} // reference → the store does not hold it; absent: not asked, or no answer
asked := map[string]bool{}
lacking := map[string]string{} // module → the image
for _, r := range s.Resources {
if fmt.Sprint(r["type"]) != "container" {
continue
}
image, _ := r["image"].(string)
reference := imageInStore(image, address)
module := s.moduleOf[fmt.Sprint(r["id"])]
if reference == "" || module == "" || lacking[module] != "" {
continue
}
if !asked[reference] {
asked[reference] = true
held, err := holds(ctx, reference)
// empty-on-error: the store did not answer, so nothing is held back for a question unanswered
lacks[reference] = err == nil && !held
}
if lacks[reference] {
lacking[module] = image
}
}
if len(lacking) == 0 {
return nil
}
kept := s.Resources[:0:0]
for _, r := range s.Resources {
if _, out := lacking[s.moduleOf[fmt.Sprint(r["id"])]]; !out {
kept = append(kept, r)
}
}
s.Resources = kept
if s.leftOutWhy == nil {
s.leftOutWhy = map[string]string{}
}
var found []missingImage
for module, image := range lacking {
s.leftOutWhy[module] = "its image " + image + " is not in the artifact store"
if !slices.Contains(s.LeftOut, module) {
s.LeftOut = append(s.LeftOut, module)
}
if s.Builds != nil {
if prior, sent := was[module]; sent {
s.Builds[module] = prior
} else {
delete(s.Builds, module)
}
}
found = append(found, missingImage{Node: node, Module: module, Image: image})
}
sort.Strings(s.LeftOut)
sort.Slice(found, func(i, j int) bool { return found[i].Module < found[j].Module })
return found
}
// withoutMissingImages is composeSend's last step: the store asked about the images the declaration names, the modules
// whose image it lacks left out and kept in the ledger S25 reads. A store that cannot be found or asked leaves the
// declaration as composed.
func withoutMissingImages(ctx context.Context, open *stores, node string, s sendable) sendable {
inv := open.inventory
shelf, err := inv.Catalogue(ctx)
if err != nil {
return s
}
address, err := artifactStoreAddress(ctx, inv, shelf, node)
if err != nil || address == "" {
return s
}
sent, known, err := inv.SentBuilds(ctx, node)
if err != nil || !known {
sent = nil
}
store := artifacts.Store{Address: address}
found := holdBackMissingImages(ctx, node, &s, address, store.HoldsManifest, sent)
for _, m := range found {
fmt.Printf("%s: %s is left out of the declaration: its image %s is not in the artifact store\n", node, m.Module, m.Image)
}
return s
}
// imageFacts are the images S25 raises: a module whose registered build names an image the store lacks, on each machine
// that runs it.
type imageFacts struct {
missingImage
}
// imagesHeld remembers the images the store was found to hold: a stored image is named by its digest and stays what it
// is, so it is asked about once; one it lacks is asked again at every look.
var imagesHeld = struct {
sync.Mutex
held map[string]bool
}{held: map[string]bool{}}
// gatherMissingImages is every machine's module whose registered build — the one a send would carry — names an image of
// the artifact store that the store does not hold. Read from the catalogue and the store, never from what a send did:
// a push is a command of its own, and the serving controller's watchdog hears nothing of it.
func gatherMissingImages(ctx context.Context, inv *inventory.Inventory, holds storeHolds) ([]imageFacts, error) {
shelf, err := inv.Catalogue(ctx)
if err != nil {
return nil, fmt.Errorf("the catalogue cannot be read: %w", err)
}
var out []imageFacts
for _, module := range sortedKeysOf(modulesOf(shelf)) {
for _, r := range shelf[module].Resources {
if fmt.Sprint(r["type"]) != "container" {
continue
}
image, _ := r["image"].(string)
if _, ours := catalogue.InArtifactStore(image); !ours || !strings.Contains(image, "@sha256:") {
continue
}
imagesHeld.Lock()
known := imagesHeld.held[image]
imagesHeld.Unlock()
if !known {
held, err := holds(ctx, image)
if err != nil {
return nil, fmt.Errorf("the artifact store cannot be asked about %s: %w", image, err)
}
if held {
imagesHeld.Lock()
imagesHeld.held[image] = true
imagesHeld.Unlock()
continue
}
} else {
continue
}
running, err := inv.Running(ctx, module)
if err != nil {
return nil, err
}
for _, node := range running {
out = append(out, imageFacts{missingImage{Node: node, Module: module, Image: image}})
}
}
}
return out, nil
}
// modulesOf is the shelf's modules as a map of names, for sortedKeysOf.
func modulesOf(shelf map[string]catalogue.Manifest) map[string]string {
out := make(map[string]string, len(shelf))
for m := range shelf {
out[m] = m
}
return out
}
// watchMissingImages is S25: a module left out of a machine's declaration because the store lacks its image, said at
// once. The condition ends when the store holds the image, or a send names another.
func watchMissingImages(f *signalFacts) []conditions.Observation {
var out []conditions.Observation
for _, m := range f.images {
out = append(out, conditions.Observation{Scope: conditions.ScopeModule, ID: m.Module + "." + m.Node, Kind: kindImageMissing,
Machine: m.Node, Severity: conditions.Warning, Resolver: conditions.ResolverOperator,
Summary: fmt.Sprintf("%s is left out of every send to %s: its declaration names an image that the artifact store does "+
"not hold (the evidence names it); the running container is untouched", m.Module, m.Node),
Said: "image " + m.Image + " is not in the store",
Headline: clipWords(conditions.Capital(m.Module)+" is not updated on "+m.Node, conditions.HeadlineMax),
Explanation: clipWords(fmt.Sprintf("The new version of %s names a container image that the mesh's store does not "+
"hold, so it was not sent to %s. What runs there is untouched, and the rest of %s is sent as usual.",
m.Module, m.Node, m.Node), conditions.ExplanationMax),
Needs: fmt.Sprintf("build %s again; the details name the image.", m.Module),
Resolved: "Resolved: the store holds the image of " + m.Module + " again"})
}
return out
}
// gatherMissingImagesHere is gatherMissingImages asking the mesh's own artifact store; nothing to ask where the mesh has
// none on its network yet.
func gatherMissingImagesHere(ctx context.Context, inv *inventory.Inventory) ([]imageFacts, error) {
shelf, err := inv.Catalogue(ctx)
if err != nil {
return nil, fmt.Errorf("the catalogue cannot be read: %w", err)
}
address, err := artifactStoreAddress(ctx, inv, shelf, "")
if err != nil {
return nil, err
}
if address == "" {
return nil, nil
}
return gatherMissingImages(ctx, inv, artifacts.Store{Address: address}.HoldsManifest)
}
+125
View File
@@ -0,0 +1,125 @@
package main
import (
"context"
"errors"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/conditions"
)
// **No declaration is sent that names an image the artifact store does not hold** (novox/hq issue 1742).
const storeAt = "novox.internal:5100"
// automx is the image the send of 2026-10-21 17:13 named and the store lacked; the others are held.
const (
automxImage = storeAt + "/mailu/automx@sha256:c62150b8e2cbcc18cb911b098c2877f928761a1e6251667f39ea18251ea35591"
imapImage = storeAt + "/mailu/imap@sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
)
func aMailSend() sendable {
return sendable{
Resources: []map[string]any{
{"id": "mailu.automx", "type": "container", "image": automxImage},
{"id": "mailu.automx-config", "type": "file", "path": "/etc/automx.conf"},
{"id": "mailu.imap", "type": "container", "image": imapImage},
{"id": "other.web", "type": "container", "image": "docker.io/library/nginx@sha256:" + strings.Repeat("b", 64)},
{"id": "dns.dnsmasq", "type": "container", "image": storeAt + "/dnsmasq/main@sha256:" + strings.Repeat("c", 64)},
},
moduleOf: map[string]string{"mailu.automx": "automx", "mailu.automx-config": "automx", "mailu.imap": "mailu",
"other.web": "other", "dns.dnsmasq": "dnsmasq"},
Builds: map[string]string{"automx": "b2f5ebb2new", "mailu": "m2", "other": "o1", "dnsmasq": "d1"},
}
}
func storeHolding(missing ...string) (storeHolds, *[]string) {
var asked []string
return func(_ context.Context, reference string) (bool, error) {
asked = append(asked, reference)
for _, m := range missing {
if strings.Contains(reference, m) {
return false, nil
}
}
return true, nil
}, &asked
}
// 1742/AC1: given a send naming automx's image, which the store does not hold, when the declaration is judged, then
// every resource of automx is out of it, automx is among the modules left out with the reason, its build is the one the
// machine was last sent, and every other module — an image of another registry among them — is as composed.
func TestAModuleWhoseImageTheStoreLacksIsLeftOutOfTheSend(t *testing.T) {
s := aMailSend()
holds, asked := storeHolding("mailu/automx@")
found := holdBackMissingImages(t.Context(), "novox", &s, storeAt, holds, map[string]string{"automx": "b2f5ebb2old"})
if len(found) != 1 || found[0].Module != "automx" || found[0].Node != "novox" || found[0].Image != automxImage {
t.Fatalf("found %+v", found)
}
var ids []string
for _, r := range s.Resources {
ids = append(ids, r["id"].(string))
}
if got := strings.Join(ids, ","); got != "mailu.imap,other.web,dns.dnsmasq" {
t.Errorf("the declaration keeps %s", got)
}
if len(s.LeftOut) != 1 || s.LeftOut[0] != "automx" || !strings.Contains(s.leftOutWhy["automx"], "not in the artifact store") {
t.Errorf("left out %v because %v", s.LeftOut, s.leftOutWhy)
}
if s.Builds["automx"] != "b2f5ebb2old" || s.Builds["mailu"] != "m2" || s.Builds["other"] != "o1" {
t.Errorf("builds %v", s.Builds)
}
for _, a := range *asked {
if strings.Contains(a, "docker.io") {
t.Errorf("the store was asked about an image of another registry: %s", a)
}
}
}
// 1742/AC2: given a module never sent to the machine, when its image is missing, then it has no build in the record of the
// send; given the store does not answer, then nothing is held back for a question unanswered.
func TestAnUnansweredStoreHoldsNothingBackAndANewModuleHasNoBuild(t *testing.T) {
s := aMailSend()
holds, _ := storeHolding("mailu/automx@")
found := holdBackMissingImages(t.Context(), "novox", &s, storeAt, holds, nil)
if len(found) != 1 {
t.Fatalf("%+v", found)
}
if _, has := s.Builds["automx"]; has {
t.Errorf("a module never sent is recorded as sent: %v", s.Builds)
}
s = aMailSend()
silent := func(context.Context, string) (bool, error) { return false, errors.New("registry: connection refused") }
if found := holdBackMissingImages(t.Context(), "novox", &s, storeAt, silent, nil); len(found) != 0 || len(s.LeftOut) != 0 ||
len(s.Resources) != 5 {
t.Errorf("an unanswered store held back %+v, left out %v", found, s.LeftOut)
}
}
// 1742/AC3: a module left out for its image raises a condition for the operator naming the module, the machine and the
// image in the evidence, plain words, and no answer to give; a mesh whose images are all held raises nothing.
func TestAMissingImageIsRaisedForTheOperator(t *testing.T) {
f := calm(time.Date(2026, 10, 21, 17, 40, 0, 0, time.UTC))
if got := watchMissingImages(f); len(got) != 0 {
t.Fatalf("%+v", got)
}
f.images = []imageFacts{{missingImage{Node: "novox", Module: "automx", Image: automxImage}}}
got := watchMissingImages(f)
if len(got) != 1 {
t.Fatalf("%+v", got)
}
o := got[0]
if o.Kind != "image-missing" || o.Key() != "module.automx.novox.image-missing" || o.Resolver != conditions.ResolverOperator ||
o.Severity != conditions.Warning || o.Machine != "novox" {
t.Fatalf("%+v (%s)", o, o.Key())
}
if !strings.Contains(o.Said, "c62150b8") || !strings.Contains(o.Headline+o.Explanation, "automx") {
t.Errorf("words: %q / %q / %q", o.Headline, o.Explanation, o.Said)
}
w := conditions.Words{Headline: o.Headline, Explanation: o.Explanation, Resolved: o.Resolved, Needs: o.Needs}
if why, ok := conditions.PlainWords(w, "novox"); !ok {
t.Errorf("not plain: %s", why)
}
}
+1 -1
View File
@@ -35,7 +35,7 @@ func TestAMergeSaysTheModulesWhoseBuildSourceItLeftUnchanged(t *testing.T) {
{Own: true, Paths: []string{"modules/build-agent/Dockerfile", "modules/build-agent/module.json"}},
},
"route-proxy": {
{Repository: "novox/mesh-controller", Ref: "main", Paths: []string{"go.mod", "examples/route-proxy/"}},
{Repository: "novox/mesh-controller", Ref: "main", Paths: []string{"go.mod", "modules/route-proxy/cmd/route-proxy/"}},
{Own: true, Paths: []string{"modules/route-proxy/Dockerfile", "modules/route-proxy/module.json"}},
},
}
+20 -2
View File
@@ -65,6 +65,9 @@ type mergeCheckInput struct {
// group are the other heads of a delivery group composed with this one (novox/hq ADR 0239), each laid
// over the mesh after the one before: the group judged as one future state.
group []groupChange
// held are the approved config values held for the keys the pull requests introduce (novox/hq ADR 0502 clause 4),
// laid over the composition with the change only.
held []heldValue
// admin is a PostgreSQL the gate may create and drop databases in.
admin string
// beside is the directory holding the core clones beside the check (MESH_CHECK_BESIDE), for finding moves.
@@ -189,6 +192,8 @@ func mergeGateCommand(ctx context.Context, args []string) error {
"--changed, for a list too long for an argument (novox/hq issue 1247)")
group := set.String("group", "", "a delivery group's other heads, as JSON [{repository, tree, changed}], composed "+
"with this one as one future state (novox/hq ADR 0239)")
heldFile := set.String("held", "", "a file of the config values approved for keys the change's pull requests introduce, as JSON "+
"[{pull, node, module, key, value, approved_by}], laid over the composition with the change (novox/hq ADR 0502)")
asJSON := set.Bool("json", false, "print the verdict as JSON")
if _, err := parseAround(set, args); err != nil {
return err
@@ -208,6 +213,10 @@ func mergeGateCommand(ctx context.Context, args []string) error {
if err != nil {
return err
}
held, err := readHeld(*heldFile)
if err != nil {
return err
}
out := io.Writer(os.Stdout)
stdout := os.Stdout
if *asJSON {
@@ -223,7 +232,7 @@ func mergeGateCommand(ctx context.Context, args []string) error {
return err
}
v, err := judgeChange(ctx, mergeCheckInput{facts: f, repository: *repository, tree: *tree, changed: changedList,
group: others, admin: *admin, say: out, beside: os.Getenv("MESH_CHECK_BESIDE")})
group: others, held: held, admin: *admin, say: out, beside: os.Getenv("MESH_CHECK_BESIDE")})
os.Stdout = stdout
if err != nil {
return err
@@ -387,7 +396,16 @@ func judgeChange(ctx context.Context, in mergeCheckInput) (mergeVerdict, error)
}
}
sort.Strings(added)
after, tried, notes, err := composeEveryMachine(ctx, in, change, changeSources, added)
// **The held values are laid in this composition only** (novox/hq ADR 0502 clause 4): the mesh as it is does not
// declare their keys, so the base composition would refuse them.
withHeld := in
var heldNotes []string
withHeld.facts, heldNotes = layHeld(in.facts, in.held, change)
for _, n := range heldNotes {
say("%s", n)
}
v.Notes = append(v.Notes, heldNotes...)
after, tried, notes, err := composeEveryMachine(ctx, withHeld, change, changeSources, added)
if err != nil {
return v, fmt.Errorf("the mesh with the change could not be raised: %w", err)
}
+135
View File
@@ -0,0 +1,135 @@
package main
import (
"encoding/json"
"fmt"
"os"
"sort"
"strings"
"git.novox.be/novox/mesh-sdk/go/catalogue"
snapshot "git.novox.be/novox/mesh-sdk/go/facts"
)
// Config values held for the keys an open pull request introduces (novox/hq ADR 0502 clause 4, issue 1826): the gate
// lays each approved one over its module's values on its node in the composition with the change, and in that one
// only. In the mesh as it is the key is not declared, so the value would be refused there.
// heldValue is one approved held value as the build agent writes it (--held): the value of Key for Module on Node,
// a machine's real name, which the snapshot holds only as a pseudonym.
type heldValue struct {
Pull string `json:"pull"`
Head string `json:"head,omitempty"`
Node string `json:"node"`
Module string `json:"module"`
Key string `json:"key"`
Value json.RawMessage `json:"value"`
Approved string `json:"approved,omitempty"`
By string `json:"by,omitempty"`
At string `json:"at,omitempty"`
}
// readHeld reads the file --held names; none named, none held.
func readHeld(path string) ([]heldValue, error) {
if path == "" {
return nil, nil
}
raw, err := os.ReadFile(path)
if err != nil {
return nil, err
}
var held []heldValue
if err := json.Unmarshal(raw, &held); err != nil {
return nil, fmt.Errorf("the held values are not readable: %w", err)
}
for _, h := range held {
if h.Node == "" || h.Module == "" || h.Key == "" || len(h.Value) == 0 {
return nil, fmt.Errorf("a held value of %s names no node, module, key or value", h.Pull)
}
}
return held, nil
}
// layHeld is the snapshot with each held value laid as the value of its key in its node's layer of its module, and what
// it said of them. The snapshot given is not changed. **A held value for a machine the snapshot does not hold, or a
// module the change does not define, is said and not laid**, never dropped silently; one for a module not assigned to
// its machine is laid and said. The notes name machines as the snapshot does (roles and pseudonym), not by their names.
func layHeld(f snapshot.Facts, held []heldValue, shelf map[string]catalogue.Manifest) (snapshot.Facts, []string) {
if len(held) == 0 {
return f, nil
}
out := f
out.Machines = make([]snapshot.Machine, len(f.Machines))
copy(out.Machines, f.Machines)
var notes, laid []string
pulls := map[string]bool{}
for _, h := range held {
at := -1
pseudonym := snapshot.Pseudonym("machine", h.Node)
for i, m := range out.Machines {
if m.Name == pseudonym {
at = i
}
}
if at < 0 {
notes = append(notes, fmt.Sprintf("the value held for %s of %s (%s=%s) is not composed: the snapshot holds no machine %s",
h.Module, h.Pull, h.Key, h.Value, pseudonym))
continue
}
if _, defined := shelf[h.Module]; !defined {
notes = append(notes, fmt.Sprintf("the value held for %s of %s (%s=%s) is not composed: the change defines no module %s",
h.Module, h.Pull, h.Key, h.Value, h.Module))
continue
}
var value any
if err := json.Unmarshal(h.Value, &value); err != nil {
notes = append(notes, fmt.Sprintf("the value held for %s of %s (%s) is not composed: it is not a value: %v",
h.Module, h.Pull, h.Key, err))
continue
}
m := &out.Machines[at]
layers := make([]snapshot.Settings, len(m.Settings))
copy(layers, m.Settings)
found := false
for i := range layers {
if layers[i].Module == h.Module {
values := make(map[string]any, len(layers[i].Values)+1)
for k, v := range layers[i].Values {
values[k] = v
}
values[h.Key] = value
layers[i].Values = values
found = true
}
}
if !found {
layers = append(layers, snapshot.Settings{Module: h.Module, Values: map[string]any{h.Key: value}})
sort.Slice(layers, func(a, b int) bool { return layers[a].Module < layers[b].Module })
}
m.Settings = layers
assigned := false
for _, a := range m.Assigned {
assigned = assigned || a == h.Module
}
by := ""
if h.Approved != "" {
by = ", approved by " + h.Approved
}
line := fmt.Sprintf("%s on %s %s=%s%s", h.Module, m.Described(), h.Key, h.Value, by)
if !assigned {
line += " (not assigned there in the snapshot)"
}
laid = append(laid, line)
pulls[h.Pull] = true
}
if len(laid) > 0 {
var names []string
for p := range pulls {
names = append(names, p)
}
sort.Strings(names)
notes = append([]string{fmt.Sprintf("composed with %d value(s) held for %s: %s", len(laid), strings.Join(names, ", "),
strings.Join(laid, "; "))}, notes...)
}
return out, notes
}
@@ -0,0 +1,85 @@
package main
import (
"os"
"path/filepath"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
snapshot "git.novox.be/novox/mesh-sdk/go/facts"
)
func heldGateFacts() snapshot.Facts {
return snapshot.Facts{Machines: []snapshot.Machine{
{Name: snapshot.Pseudonym("machine", "ace"), Roles: []string{"the build node"}, Assigned: []string{"n8n"},
Settings: []snapshot.Settings{{Module: "n8n", Values: map[string]any{"port": 5678.0}}}},
{Name: snapshot.Pseudonym("machine", "g14")},
}}
}
var heldShelf = map[string]catalogue.Manifest{"n8n": {Module: "n8n"}}
// 1826/AC3: a held value is laid in its node's layer of its module, beside what the layer holds, and said.
func TestAHeldValueIsLaidOverItsNodesLayerAndSaid(t *testing.T) {
f := heldGateFacts()
got, notes := layHeld(f, []heldValue{{Pull: "novox/mesh-catalog#434", Node: "ace", Module: "n8n", Key: "media-gid",
Value: []byte("2000"), Approved: "the operator"}}, heldShelf)
layer := got.Machines[0].Settings[0].Values
if layer["media-gid"] != 2000.0 || layer["port"] != 5678.0 {
t.Fatalf("the layer holds %v", layer)
}
if _, has := f.Machines[0].Settings[0].Values["media-gid"]; has {
t.Fatal("the snapshot given was changed")
}
if len(notes) != 1 || !strings.Contains(notes[0], "composed with 1 value(s) held for novox/mesh-catalog#434") ||
!strings.Contains(notes[0], "media-gid=2000") || !strings.Contains(notes[0], "approved by the operator") ||
strings.Contains(notes[0], "ace") {
t.Fatalf("said %q", notes)
}
}
// 1826/AC3: a machine with no layer of the module gets one.
func TestAHeldValueMakesTheLayerAMachineLacks(t *testing.T) {
got, _ := layHeld(heldGateFacts(), []heldValue{{Pull: "p", Node: "g14", Module: "n8n", Key: "k", Value: []byte(`"x"`)}}, heldShelf)
s := got.Machines[1].Settings
if len(s) != 1 || s[0].Module != "n8n" || s[0].Values["k"] != "x" {
t.Fatalf("layers %+v", s)
}
}
// 1826/AC3: a held value for a machine or a module the snapshot or the change does not hold is said, never laid.
func TestAHeldValueForWhatIsNotHeldIsSaidNotLaid(t *testing.T) {
f := heldGateFacts()
got, notes := layHeld(f, []heldValue{
{Pull: "p", Node: "nowhere", Module: "n8n", Key: "k", Value: []byte("1")},
{Pull: "p", Node: "ace", Module: "ghost", Key: "k", Value: []byte("1")},
}, heldShelf)
if len(notes) != 2 || !strings.Contains(notes[0], "holds no machine") || !strings.Contains(notes[1], "defines no module ghost") {
t.Fatalf("said %q", notes)
}
if _, has := got.Machines[0].Settings[0].Values["k"]; has {
t.Fatal("laid")
}
}
// 1826/AC3: none held lays nothing and says nothing; the file is read and refuses a value without a node.
func TestNoHeldValuesLayNothingAndTheFileIsRead(t *testing.T) {
if got, notes := layHeld(heldGateFacts(), nil, heldShelf); len(notes) != 0 || len(got.Machines) != 2 {
t.Fatalf("%v %v", got, notes)
}
dir := t.TempDir()
good := filepath.Join(dir, "good.json")
os.WriteFile(good, []byte(`[{"pull":"p","node":"ace","module":"n8n","key":"k","value":1}]`), 0o644)
if h, err := readHeld(good); err != nil || len(h) != 1 || h[0].Key != "k" {
t.Fatalf("%v %v", h, err)
}
bad := filepath.Join(dir, "bad.json")
os.WriteFile(bad, []byte(`[{"pull":"p","module":"n8n","key":"k","value":1}]`), 0o644)
if _, err := readHeld(bad); err == nil {
t.Fatal("a value without a node was read")
}
if h, err := readHeld(""); h != nil || err != nil {
t.Fatalf("%v %v", h, err)
}
}
@@ -526,3 +526,57 @@ func TestAMemberJudgedAloneSeesAMoveThroughTheClonesBeside(t *testing.T) {
t.Errorf("nothing beside, yet moves %v", got)
}
}
// 1826/AC3 (integration level, a throwaway store): a change whose container says ${config:media-gid} fails the gate for
// want of the value, passes it with the value the operator approved held for its pull request, and the mesh as it is
// composes the same either way.
func TestAChangeNeedingAHeldValueFailsWithoutItAndPassesWithIt(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
trunk := `{"module":"n8n","version":"1","resources":[{"id":"server","type":"container","name":"n8n",
"image":"n8nio/n8n@sha256:4846eb2f4b874ab04cde7fc1e249d2ddaec66e9aea64439beb2972cfea88e3c0"}]}`
m, err := catalogue.ParseManifest([]byte(trunk))
if err != nil {
t.Fatal(err)
}
if err := open.inventory.RegisterModule(ctx, m, inventory.Source{Repository: "novox/mesh-catalog",
Path: "modules/n8n", BuiltFrom: "c0ffee"}); err != nil {
t.Fatal(err)
}
if _, err := assign(ctx, open, "laptop", "n8n"); err != nil {
t.Fatal(err)
}
f, err := gatherFacts(ctx, open, "2.11.17")
if err != nil {
t.Fatal(err)
}
for i := range f.Machines {
f.Machines[i].Capabilities = append(f.Machines[i].Capabilities, snapshot.Capability{Name: "container-groups", Present: true})
}
tree := aTree(t, map[string]string{"n8n": n8nWithAGroup})
judge := func(held []heldValue) mergeVerdict {
v, err := judgeChange(t.Context(), mergeCheckInput{facts: f, admin: os.Getenv("MESH_TEST_POSTGRES"),
repository: "novox/mesh-catalog", tree: tree, changed: []string{"modules/n8n/module.json"}, held: held})
if err != nil {
t.Fatal(err)
}
return v
}
without := judge(nil)
if without.Verdict != "fail" || !strings.Contains(without.Report(), "media-gid") {
t.Fatalf("a required value nothing sets passed, or failed for another reason:\n%s", without.Report())
}
with := judge([]heldValue{{Pull: "novox/mesh-catalog#434", Node: "laptop", Module: "n8n", Key: "media-gid",
Value: []byte("2000"), Approved: "the operator"}})
if with.Verdict == "fail" || with.Verdict == "error" {
t.Fatalf("the held value did not make it compose:\n%s", with.Report())
}
if !strings.Contains(with.Report(), "composed with 1 value(s) held for novox/mesh-catalog#434") {
t.Fatalf("the value laid is not said:\n%s", with.Report())
}
for i := range with.Machines {
if with.Machines[i].Base.Composes != without.Machines[i].Base.Composes {
t.Fatalf("the mesh as it is composes differently with a held value: %+v / %+v", with.Machines[i].Base, without.Machines[i].Base)
}
}
}
+12
View File
@@ -457,6 +457,18 @@ var plainWordings = map[string]func(conditions.Observation) words{
Explanation: "A delivery did not finish. Nothing is tried again until you say so.",
Resolved: "Resolved: the delivery is no longer failed"}
}),
kindMovesHeld: worded(func(o conditions.Observation) words {
return words{Headline: "An update waits for you",
Explanation: "A new version of a module would restart a container or touch its data, so it waits for you.",
Resolved: "Resolved: the update is no longer waiting"}
}),
kindImageMissing: worded(func(o conditions.Observation) words {
return words{Headline: "A module is not updated",
Explanation: "The new version of a module names a container image that the mesh's store does not hold, so it " +
"was not sent. What runs is untouched.",
Needs: "build the module again; the details name the image.",
Resolved: "Resolved: the store holds the image again"}
}),
"ask-lost": worded(func(o conditions.Observation) words {
return words{Headline: "A build has no result",
Explanation: "A build was asked for and never answered. It is asked again with the next delivery.",
+1 -1
View File
@@ -260,7 +260,7 @@ func TestUpdatesHeldNameWhatWaitsAndWhereItIsReleased(t *testing.T) {
plainExample(t, got[0], "Updates of openrazer and sensors wait for your release",
"Needs you: release them from the mesh MCP server; this notification cannot do it. Updates of openrazer and "+
"sensors on g14 and shanks wait for a person to release them, because the last walk failed. "+
"They are not delivered until then, and stay held if you leave them.")
"They are not delivered until then, and stay held if you leave them.", "Release")
}
// **What held them is said in the glossary's words** (2026-10-08 review): the backlog is held after any
@@ -31,7 +31,8 @@ func TestNoPlanIsWrittenButThroughSavePlansMoving(t *testing.T) {
return err
}
text := string(raw)
if write.MatchString(text) {
// The regular expression is the costly part of reading every source: only a file that names the table can match.
if strings.Contains(strings.ToLower(text), "release_plan") && write.MatchString(text) {
found = append(found, path+" writes the plan table")
}
// A plan record is written by the bucket's raw create and update, which only the plans' own file calls.
+2 -2
View File
@@ -52,7 +52,7 @@ func TestASharedRepositoryIsPlannedFromTheRecordedBuildSources(t *testing.T) {
{Repository: "novox/mesh-controller", Ref: "main", Paths: append([]string{"cmd/mesh-builder/", "internal/broker/"}, gomod...)}}
sourcesOf["route-proxy"] = []inventory.BuildSource{
{Paths: []string{"modules/route-proxy/Dockerfile", "modules/route-proxy/module.json"}},
{Repository: "novox/mesh-controller", Ref: "main", Paths: append([]string{"examples/route-proxy/", "internal/broker/"}, gomod...)}}
{Repository: "novox/mesh-controller", Ref: "main", Paths: append([]string{"modules/route-proxy/cmd/route-proxy/", "internal/broker/"}, gomod...)}}
register(catalogue.Manifest{Module: "mesh-controller", Version: "1"}, "novox/mesh-controller", "", nil, nil)
register(agent, "novox/mesh-catalog", "modules/build-agent", nil, controllerRead)
register(catalogue.Manifest{Module: "route-proxy", Version: "1"}, "novox/mesh-catalog", "modules/route-proxy", nil, controllerRead)
@@ -133,7 +133,7 @@ func TestASharedRepositoryIsPlannedFromTheRecordedBuildSources(t *testing.T) {
{"a package only the controller builds from: the controller alone", "mesh-controller",
[]string{"internal/conditions/condition.go"}, "mesh-controller", true},
{"the route proxy's program: the route proxy alone", "mesh-controller",
[]string{"examples/route-proxy/main.go"}, "route-proxy", true},
[]string{"modules/route-proxy/cmd/route-proxy/main.go"}, "route-proxy", true},
{"a package all three build from: all three", "mesh-controller",
[]string{"internal/broker/broker.go"}, "mesh-controller,route-proxy | build-agent", false},
{"the route proxy's directory in the catalogue: it alone", "mesh-catalog",
+11 -11
View File
@@ -259,7 +259,7 @@ func TestASharedRepositoryMovesOnlyWhatItsBuildSourceHolds(t *testing.T) {
{Own: true, Paths: []string{"modules/build-agent/Dockerfile", "modules/build-agent/module.json"}},
},
"route-proxy": {
{Repository: "novox/mesh-controller", Ref: "main", Paths: with("examples/route-proxy/", "internal/broker/",
{Repository: "novox/mesh-controller", Ref: "main", Paths: with("modules/route-proxy/cmd/route-proxy/", "internal/broker/",
"vendor/github.com/nats-io/nats.go/")},
{Own: true, Paths: []string{"modules/route-proxy/Dockerfile", "modules/route-proxy/module.json"}},
},
@@ -288,7 +288,7 @@ func TestASharedRepositoryMovesOnlyWhatItsBuildSourceHolds(t *testing.T) {
{"a README of the controller's repository: no module (338)", "mesh-controller",
[]string{"README.md"}, read, "", "README.md"},
{"the route proxy's program alone: the route proxy alone (338)", "mesh-controller",
[]string{"examples/route-proxy/main.go"}, read, "route-proxy", ""},
[]string{"modules/route-proxy/cmd/route-proxy/main.go"}, read, "route-proxy", ""},
{"the build seat's program alone: its holder alone", "mesh-controller",
[]string{"cmd/mesh-builder/main.go"}, read, "build-agent", ""},
{"a package the build seat's program and the controller build from: both", "mesh-controller",
@@ -305,7 +305,7 @@ func TestASharedRepositoryMovesOnlyWhatItsBuildSourceHolds(t *testing.T) {
{"go.sum: all three", "mesh-controller",
[]string{"go.sum"}, read, "mesh-controller,route-proxy | build-agent", ""},
{"a file added to the proxy's package: the proxy", "mesh-controller",
[]string{"examples/route-proxy/new.go"}, read, "route-proxy", ""},
[]string{"modules/route-proxy/cmd/route-proxy/new.go"}, read, "route-proxy", ""},
// Before any build source is said: as before.
{"no build source said: a README moves all three, as before", "mesh-controller",
[]string{"README.md"}, unsaid, "mesh-controller,route-proxy | build-agent", ""},
@@ -345,7 +345,7 @@ func TestAModuleAPlanOvertookIsReadWhole(t *testing.T) {
built := time.Date(2026, 10, 10, 1, 0, 0, 0, time.UTC)
read := map[string][]inventory.ReadRepository{
"route-proxy": {
{Repository: "novox/mesh-controller", Ref: "main", Paths: []string{"examples/route-proxy/", "go.mod"}, Built: built},
{Repository: "novox/mesh-controller", Ref: "main", Paths: []string{"modules/route-proxy/cmd/route-proxy/", "go.mod"}, Built: built},
{Own: true, Paths: []string{"modules/route-proxy/module.json"}, Built: built},
},
"mesh-controller": {{Own: true, Paths: []string{"module.json", "cmd/mesh-controller/"}, Built: built}},
@@ -396,11 +396,11 @@ func TestAMissedMergeMovingOnlyAPackagingModuleIsActedOnOnce(t *testing.T) {
}
read := map[string][]inventory.ReadRepository{
"mesh-controller": {{Own: true, Paths: []string{"module.json", "cmd/mesh-controller/"}, Built: merged.Add(-time.Hour)}},
"route-proxy": {{Repository: "novox/mesh-controller", Ref: "main", Paths: []string{"examples/route-proxy/"},
"route-proxy": {{Repository: "novox/mesh-controller", Ref: "main", Paths: []string{"modules/route-proxy/cmd/route-proxy/"},
Built: merged.Add(-time.Hour), Looked: merged.Add(-time.Hour)}},
}
m := link.SourceMoved{Owner: "novox", Repo: "mesh-controller", Base: "main", Commit: "c1",
MergedAt: merged.Format(time.RFC3339), Paths: []string{"examples/route-proxy/main.go"}}
MergedAt: merged.Format(time.RFC3339), Paths: []string{"modules/route-proxy/cmd/route-proxy/main.go"}}
if got := wouldMove(m, entries, planningView(read, nil)); len(got) != 1 || got[0].Manifest.Module != "route-proxy" {
t.Fatalf("a missed merge of the proxy's program would move %v", got)
}
@@ -599,14 +599,14 @@ func TestTheGatePlansFromTheBuildSourcesTheSnapshotCarries(t *testing.T) {
Sources: []snapshot.BuildSource{{Own: true, Paths: []string{"module.json", "cmd/mesh-controller/", "internal/broker/"}}}},
{Name: "route-proxy", Repository: "novox/mesh-catalog", Path: "modules/route-proxy", Manifest: manifest("route-proxy"),
Reads: []string{"novox/mesh-controller"},
Sources: []snapshot.BuildSource{{Repository: "novox/mesh-controller", Paths: []string{"examples/route-proxy/", "internal/broker/"}},
Sources: []snapshot.BuildSource{{Repository: "novox/mesh-controller", Paths: []string{"modules/route-proxy/cmd/route-proxy/", "internal/broker/"}},
{Own: true, Paths: []string{"modules/route-proxy/module.json"}}}},
}}
for paths, want := range map[string]string{
"cmd/mesh-controller/main.go": "mesh-controller",
"examples/route-proxy/main.go": "route-proxy",
"internal/broker/broker.go": "mesh-controller,route-proxy",
"README.md": "",
"cmd/mesh-controller/main.go": "mesh-controller",
"modules/route-proxy/cmd/route-proxy/main.go": "route-proxy",
"internal/broker/broker.go": "mesh-controller,route-proxy",
"README.md": "",
} {
r, err := reachOfChange(facts, "novox/mesh-controller", []string{paths}, "")
if err != nil {
+1 -1
View File
@@ -153,7 +153,7 @@ func planWaitCommand(ctx context.Context, inv *inventory.Inventory, id, untilAsk
line := planLineSaid(p, now, pause, bounds.of(p.Repository), func(module string) []string {
on, _ := inv.Running(ctx, module)
return on
}, waitsOf(ctx, inv, one))
}, waitsOf(ctx, inv, one)) + heldSaid(p, heldNow(ctx, inv))
doc := planWaitedDoc(p, planStatuses(one, now, pause, bounds)[0], until, came, now.Sub(began), line)
return json.NewEncoder(os.Stdout).Encode(doc)
}
+211
View File
@@ -0,0 +1,211 @@
package main
import (
"context"
"encoding/json"
"fmt"
"io"
"path"
"reflect"
"slices"
"time"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// What the controller tells mesh-config of an open pull request, and asks it back (novox/hq ADR 0502 clauses 2 and 4,
// issue 1826): a config key a pull request introduces is not declared on the trunk, so a value for it can be judged
// only against the pull request's own declaration, which the core reads from the forge and mesh-config never does
// (ADR 0495 §1). The call is mesh-config's `config` verb in the controller's own name, actions `pull-declares` and
// `held`; the shapes are those of the interface the two sides agreed.
// meshConfigWait is how long the controller waits for mesh-config before it goes on without: a check is never held up
// by a module that does not answer.
const meshConfigWait = 10 * time.Second
// meshConfigAsk is one call of mesh-config's `config` verb in the controller's own name, answering its result. A
// variable so a test needs no bus.
var meshConfigAsk = func(ctx context.Context, args map[string]any) ([]byte, error) {
var result []byte
err := onTheBus(func(conn *nats.Conn) error {
answer, err := link.AskMeshSeatToolAs(ctx, conn, broker.ControllerName, "mesh-config", "config", args, meshConfigWait)
if err != nil {
return err
}
if answer.Error != "" {
return fmt.Errorf("%s", answer.Error)
}
result = answer.Result
return nil
})
return result, err
}
// pullAt is an open pull request at a head: the change to read, and its number.
type pullAt struct {
changeSource
Number int
}
// pullAtOf is a pull request announced, as the change it makes over what it merges into.
func pullAtOf(p link.PullUpdated) pullAt {
old := p.BaseSHA
if old == "" {
old = p.Base
}
return pullAt{Number: p.Number, changeSource: changeSource{Owner: p.Owner, Repo: p.Repo, New: p.Commit, Old: old,
Paths: p.Paths, Truncated: p.PathsTruncated, Removed: p.Removed}}
}
func pullKey(owner, repo string, number int) string {
return fmt.Sprintf("%s/%s#%d", owner, repo, number)
}
func (p pullAt) key() string { return pullKey(p.Owner, p.Repo, p.Number) }
// pullDeclarations are the config declarations a pull request's head adds or changes over what it merges into, by
// module, in the shape of the declared-config mirror. The bool says the head could be read: false for a pull request whose
// paths are cut (the caller cuts them to the manifests of the modules it moves) or whose base is unknown, which is then
// told nothing, never an empty set.
func pullDeclarations(ctx context.Context, p pullAt) ([]inventory.DeclaredConfigMirrored, bool, error) {
var out []inventory.DeclaredConfigMirrored
if p.Truncated || p.Old == "" {
return nil, false, nil
}
for _, file := range p.Paths {
if path.Base(file) != moduleManifestFile || slices.Contains(p.Removed, file) {
continue
}
now, exists, err := manifestCached(ctx, p.Owner, p.Repo, file, p.New)
if err != nil {
return nil, false, err
}
if !exists || now.Module == "" {
continue
}
was, _, err := manifestCached(ctx, p.Owner, p.Repo, file, p.Old)
if err != nil {
return nil, false, err
}
have, err := inventory.DeclaredOf(now.Settings)
if err != nil {
return nil, false, fmt.Errorf("%s at %.8s: %w", file, p.New, err)
}
had, err := inventory.DeclaredOf(was.Settings)
if err != nil {
return nil, false, fmt.Errorf("%s at %.8s: %w", file, p.Old, err)
}
changed := map[string]inventory.DeclaredMirrored{}
for key, d := range have {
if before, there := had[key]; !there || !reflect.DeepEqual(before, d) {
changed[key] = d
}
}
if len(changed) > 0 {
out = append(out, inventory.DeclaredConfigMirrored{Module: now.Module, Settings: changed})
}
}
return out, true, nil
}
// tellPullDeclares tells mesh-config, for each pull request at every announced head, what the head declares, an empty set
// when it declares nothing new: mesh-config re-judges the values held at every head, so a head that reverts the change
// drops them (ADR 0502 clause 6). A
// failure is said to w and the check goes on: the held values are then none (ADR 0502 clause 2).
func tellPullDeclares(ctx context.Context, w io.Writer, pulls []pullAt) {
for _, p := range pulls {
mods, readable, err := pullDeclarations(ctx, p)
if err != nil {
fmt.Fprintf(w, "%s: what its head declares cannot be read, so mesh-config is told nothing: %v\n", p.key(), err)
continue
}
if !readable {
continue
}
declares := map[string]map[string]inventory.DeclaredMirrored{}
for _, m := range mods {
declares[m.Module] = m.Settings
}
if _, err := meshConfigAsk(ctx, map[string]any{"action": "pull-declares", "pull": p.key(), "head": p.New,
"declares": declares}); err != nil {
fmt.Fprintf(w, "%s: mesh-config cannot be told what its head declares (%v); the check goes on without held values\n",
p.key(), err)
}
}
}
// heldFor asks mesh-config for the approved held values of these pull requests at their heads (ADR 0502 clause 4). Only
// those of the pull requests asked about are carried. A failure is said to w and answers none.
func heldFor(ctx context.Context, w io.Writer, pulls []pullAt) []link.HeldConfig {
if len(pulls) == 0 {
return nil
}
type pullHead struct {
Pull string `json:"pull"`
Head string `json:"head"`
}
asked := make([]pullHead, 0, len(pulls))
for _, p := range pulls {
asked = append(asked, pullHead{p.key(), p.New})
}
raw, err := meshConfigAsk(ctx, map[string]any{"action": "held", "pulls": asked})
if err != nil {
fmt.Fprintf(w, "mesh-config cannot say what it holds for these pull requests (%v); the check goes on without held values\n", err)
return nil // empty-on-error: said above; a check without held values fails for want of the value and says so, it never passes on a guess
}
var answer struct {
Held []link.HeldConfig `json:"held"`
Skipped []struct {
Pull string `json:"pull"`
Why string `json:"why"`
} `json:"skipped"`
}
if len(raw) == 0 {
return nil
}
if err := json.Unmarshal(raw, &answer); err != nil {
fmt.Fprintf(w, "mesh-config's held values cannot be read (%v); the check goes on without them\n", err)
return nil // empty-on-error: said above; a check without held values fails for want of the value and says so, it never passes on a guess
}
for _, s := range answer.Skipped {
fmt.Fprintf(w, "%s: mesh-config holds a value it does not carry: %s\n", s.Pull, s.Why)
}
var out []link.HeldConfig
for _, h := range answer.Held {
for _, p := range pulls {
if h.Pull == p.key() {
out = append(out, h)
break
}
}
}
return out
}
// withHeldConfig tells mesh-config what these pull requests declare and puts the approved held values of them on the
// check request (ADR 0502 clause 4). With carry false, a member of a group whose composed check composes, only the
// declarations are told.
func withHeldConfig(ctx context.Context, w io.Writer, request *link.BuildRequest, pulls []pullAt, carry bool) {
tellPullDeclares(ctx, w, pulls)
if carry {
carryHeld(ctx, w, request, pulls)
}
}
// carryHeld puts the approved held values of these pull requests on the check request.
func carryHeld(ctx context.Context, w io.Writer, request *link.BuildRequest, pulls []pullAt) {
if request.Check != nil {
request.Check.HeldConfig = heldFor(ctx, w, pulls)
}
}
// holdsModules is whether a module of the mesh is built from this repository.
func holdsModules(entries []inventory.Entry, owner, repo string) bool {
return slices.ContainsFunc(entries, func(e inventory.Entry) bool {
return sameRepository(e.Source.Repository, link.SourceMoved{Owner: owner, Repo: repo})
})
}
+209
View File
@@ -0,0 +1,209 @@
package main
import (
"context"
"encoding/json"
"io"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
const (
headSHA = "2222222222222222222222222222222222222222"
baseSHA = "1111111111111111111111111111111111111111"
)
// fakeManifests serves manifests by ref, and counts what the forge was asked. Issue 1826/AC1 (unit level).
func fakeManifests(t *testing.T, at map[string]*catalogue.Manifest) {
t.Helper()
was := manifestAt
t.Cleanup(func() { manifestAt = was })
manifestsReadMu.Lock()
clear(manifestsRead)
manifestsReadMu.Unlock()
manifestAt = func(_ context.Context, _, _, _, ref string) (catalogue.Manifest, bool, error) {
if m := at[ref]; m != nil {
return *m, true, nil
}
return catalogue.Manifest{}, false, nil
}
}
// fakeConfig records the calls to mesh-config and answers each action from answers.
func fakeConfig(t *testing.T, answers map[string]string) *[]map[string]any {
t.Helper()
var calls []map[string]any
was := meshConfigAsk
t.Cleanup(func() { meshConfigAsk = was })
meshConfigAsk = func(_ context.Context, args map[string]any) ([]byte, error) {
calls = append(calls, args)
return []byte(answers[args["action"].(string)]), nil
}
return &calls
}
func manifest(module string, keys map[string]catalogue.SettingDeclaration) *catalogue.Manifest {
return &catalogue.Manifest{Module: module, Settings: keys}
}
func thePull() pullAt {
return pullAt{Number: 434, changeSource: changeSource{Owner: "novox", Repo: "mesh-catalog", New: headSHA, Old: baseSHA,
Paths: []string{"modules/n8n/module.json", "modules/n8n/README.md"}}}
}
// 1826/AC1: a key the head adds is sent with its declaration, under the pull request and the head.
func TestAKeyThePullRequestAddsIsToldToMeshConfig(t *testing.T) {
fakeManifests(t, map[string]*catalogue.Manifest{
baseSHA: manifest("n8n", map[string]catalogue.SettingDeclaration{"port": {Type: "number"}}),
headSHA: manifest("n8n", map[string]catalogue.SettingDeclaration{"port": {Type: "number"},
"media-gid": {Type: "number", Required: true}}),
})
calls := fakeConfig(t, map[string]string{"pull-declares": `{"kept":1}`})
tellPullDeclares(context.Background(), io.Discard, []pullAt{thePull()})
if len(*calls) != 1 {
t.Fatalf("mesh-config was asked %d times: %v", len(*calls), *calls)
}
c := (*calls)[0]
if c["action"] != "pull-declares" || c["pull"] != "novox/mesh-catalog#434" || c["head"] != headSHA {
t.Fatalf("the call is %v", c)
}
body, _ := json.Marshal(c["declares"])
var declares map[string]map[string]inventory.DeclaredMirrored
if err := json.Unmarshal(body, &declares); err != nil || len(declares) != 1 || declares["n8n"] == nil {
t.Fatalf("declares %s: %v", body, err)
}
mods := []inventory.DeclaredConfigMirrored{{Module: "n8n", Settings: declares["n8n"]}}
if _, has := mods[0].Settings["port"]; has || len(mods[0].Settings) != 1 {
t.Fatalf("only the added key is told, got %v", mods[0].Settings)
}
d := mods[0].Settings["media-gid"]
if d.Type != "number" || !d.Required {
t.Fatalf("the declaration is %+v", d)
}
}
// 1826/AC1: a key whose declaration the head changes is sent; one it leaves as it was is not.
func TestAnUnchangedKeyIsNotToldAndAChangedOneIs(t *testing.T) {
fakeManifests(t, map[string]*catalogue.Manifest{
baseSHA: manifest("n8n", map[string]catalogue.SettingDeclaration{"port": {Type: "number"}, "x": {Type: "string"}}),
headSHA: manifest("n8n", map[string]catalogue.SettingDeclaration{"port": {Type: "number"}, "x": {Type: "string", Pattern: "a+"}}),
})
mods, touched, err := pullDeclarations(context.Background(), thePull())
if err != nil || !touched || len(mods) != 1 || len(mods[0].Settings) != 1 || mods[0].Settings["x"].Pattern != "a+" {
t.Fatalf("%+v %v %v", mods, touched, err)
}
}
// 1826/AC1: a head that changes no module.json, or removes it, is still told, with declares {}: mesh-config re-judges at every
// head, so a head that reverts the change drops the value held for it (review round 1, finding 1).
func TestAHeadThatDeclaresNothingIsToldAsEmpty(t *testing.T) {
fakeManifests(t, map[string]*catalogue.Manifest{baseSHA: manifest("n8n", nil), headSHA: manifest("n8n", nil)})
calls := fakeConfig(t, nil)
p := thePull()
p.Paths = []string{"modules/n8n/README.md"}
tellPullDeclares(context.Background(), io.Discard, []pullAt{p})
p = thePull()
p.Removed = p.Paths
tellPullDeclares(context.Background(), io.Discard, []pullAt{p})
if len(*calls) != 2 {
t.Fatalf("mesh-config was told %v", *calls)
}
for _, c := range *calls {
if d, _ := json.Marshal(c["declares"]); string(d) != "{}" || c["head"] != headSHA {
t.Fatalf("told %v", c)
}
}
}
// 1826/AC1: a head whose paths are cut, or whose base is unknown, cannot be read: nothing is told, never an empty set.
func TestAHeadThatCannotBeReadIsNotToldAsEmpty(t *testing.T) {
calls := fakeConfig(t, nil)
p := thePull()
p.Truncated = true
tellPullDeclares(context.Background(), io.Discard, []pullAt{p})
if len(*calls) != 0 {
t.Fatalf("told %v", *calls)
}
}
// 1826/AC1: a module.json changed without a declaration changing is told as an empty set, so mesh-config re-judges what it holds.
func TestAChangedManifestWithNoNewDeclarationIsToldAsEmpty(t *testing.T) {
m := manifest("n8n", map[string]catalogue.SettingDeclaration{"port": {Type: "number"}})
fakeManifests(t, map[string]*catalogue.Manifest{baseSHA: m, headSHA: m})
calls := fakeConfig(t, nil)
tellPullDeclares(context.Background(), io.Discard, []pullAt{thePull()})
if len(*calls) != 1 {
t.Fatalf("calls %v", *calls)
}
if d, _ := json.Marshal((*calls)[0]["declares"]); string(d) != "{}" {
t.Fatalf("declares %s", d)
}
}
// 1826/AC1: mesh-config not answering is said and goes on; the check is not stopped.
func TestMeshConfigNotAnsweringIsSaidAndGoneOn(t *testing.T) {
fakeManifests(t, map[string]*catalogue.Manifest{baseSHA: manifest("n8n", nil),
headSHA: manifest("n8n", map[string]catalogue.SettingDeclaration{"k": {Type: "string"}})})
was := meshConfigAsk
t.Cleanup(func() { meshConfigAsk = was })
meshConfigAsk = func(context.Context, map[string]any) ([]byte, error) { return nil, io.ErrUnexpectedEOF }
var said strings.Builder
tellPullDeclares(context.Background(), &said, []pullAt{thePull()})
got := heldFor(context.Background(), &said, []pullAt{thePull()})
if len(got) != 0 || !strings.Contains(said.String(), "mesh-config") {
t.Fatalf("held %v, said %q", got, said.String())
}
}
// 1826/AC2: the held values mesh-config answers for the check's pull requests are what the check request carries.
func TestTheHeldValuesOfTheChecksPullRequestsAreAsked(t *testing.T) {
calls := fakeConfig(t, map[string]string{"held": `{"held":[{"pull":"novox/mesh-catalog#434","head":"` + headSHA +
`","node":"ace","module":"n8n","key":"media-gid","value":2000,"approved":"the operator"}]}`})
other := pullAt{Number: 7, changeSource: changeSource{Owner: "novox", Repo: "mesh-controller", New: baseSHA}}
got := heldFor(context.Background(), io.Discard, []pullAt{thePull(), other})
if len(got) != 1 || got[0].Node != "ace" || got[0].Key != "media-gid" || string(got[0].Value) != "2000" {
t.Fatalf("got %+v", got)
}
c := (*calls)[0]
pulls, _ := json.Marshal(c["pulls"])
if c["action"] != "held" || string(pulls) != `[{"pull":"novox/mesh-catalog#434","head":"`+headSHA+
`"},{"pull":"novox/mesh-controller#7","head":"`+baseSHA+`"}]` {
t.Fatalf("asked %v %s", c, pulls)
}
}
// 1826/AC2: a value held for a pull request the check is not of is never carried.
func TestAHeldValueOfAnotherPullRequestIsNotCarried(t *testing.T) {
fakeConfig(t, map[string]string{"held": `{"held":[{"pull":"novox/mesh-catalog#999","head":"x","node":"ace","module":"n8n","key":"k","value":1}]}`})
if got := heldFor(context.Background(), io.Discard, []pullAt{thePull()}); len(got) != 0 {
t.Fatalf("carried %+v", got)
}
}
// 1826/AC2: the check request carries what mesh-config answered; a member of a composed group asks none.
func TestTheCheckRequestCarriesTheHeldValues(t *testing.T) {
fakeManifests(t, map[string]*catalogue.Manifest{baseSHA: manifest("n8n", nil), headSHA: manifest("n8n", nil)})
fakeConfig(t, map[string]string{"held": `{"held":[{"pull":"novox/mesh-catalog#434","node":"ace","module":"n8n","key":"media-gid","value":2000}]}`})
request := link.BuildRequest{Check: &link.CheckRequest{}}
withHeldConfig(context.Background(), io.Discard, &request, []pullAt{thePull()}, true)
if len(request.Check.HeldConfig) != 1 || request.Check.HeldConfig[0].Key != "media-gid" {
t.Fatalf("the request carries %+v", request.Check.HeldConfig)
}
request = link.BuildRequest{Check: &link.CheckRequest{}}
withHeldConfig(context.Background(), io.Discard, &request, []pullAt{thePull()}, false)
if len(request.Check.HeldConfig) != 0 {
t.Fatalf("a composed member carries %+v", request.Check.HeldConfig)
}
}
// 1826/AC1: a repository that builds a module of the mesh holds modules; another does not.
func TestARepositoryThatBuildsAModuleHoldsModules(t *testing.T) {
entries := []inventory.Entry{{Manifest: catalogue.Manifest{Module: "a"}, Source: inventory.Source{Repository: "novox/mesh-catalog", Path: "modules/a"}}}
if !holdsModules(entries, "novox", "mesh-catalog") || holdsModules(entries, "novox", "hq") {
t.Fatal("wrong")
}
}
+28 -6
View File
@@ -181,15 +181,24 @@ func planModuleNames(p inventory.Plan) map[string]string {
return out
}
// recordedWaiting is, on one machine, every move of a recorded module that waits for a person, each said with why
// (ADR 0377 §6): what `status` names the machine behind for, with `push <node>` as the remedy. A move under a
// module's own `record` that recreates nothing and moves no data is not among them: it goes with its walk.
func recordedWaiting(ctx context.Context, inv *inventory.Inventory, f moveFacts, node string) ([]string, error) {
// heldMove is one move of a recorded module that waits for a person on one machine (ADR 0377 §6).
type heldMove struct {
Module string
From, To string
// Why is the walk's own word for it: "waits: " and the recreate words, the data change, or the hold.
Why string
// Chosen is whether a person's own choice holds it — a module they set to record, or the bus's step — which the
// operator is not asked about again: they chose that hold, and the bus has its own condition (S17).
Chosen bool
}
// heldMovesOn is, on one machine, every move of a recorded module that waits for a person, in module order.
func heldMovesOn(ctx context.Context, inv *inventory.Inventory, f moveFacts, node string) ([]heldMove, error) {
sent, known, err := inv.SentBuilds(ctx, node)
if err != nil || !known {
return nil, err
}
var out []string
var out []heldMove
for _, m := range sortedKeysOf(sent) {
was := sent[m]
now, held := f.current[m]
@@ -197,12 +206,25 @@ func recordedWaiting(ctx context.Context, inv *inventory.Inventory, f moveFacts,
continue
}
if v := f.recordedMove(m, was); !v.rolls {
out = append(out, fmt.Sprintf("%s %s → %s %s", m, short(was), short(now.Commit), v.said))
out = append(out, heldMove{Module: m, From: was, To: now.Commit, Why: v.said,
Chosen: now.From == catalogue.FromPerson || now.From == catalogue.FromBus})
}
}
return out, nil
}
// recordedWaiting is, on one machine, every move of a recorded module that waits for a person, each said with why
// (ADR 0377 §6): what `status` names the machine behind for, with `push <node>` as the remedy. A move under a
// module's own `record` that recreates nothing and moves no data is not among them: it goes with its walk.
func recordedWaiting(ctx context.Context, inv *inventory.Inventory, f moveFacts, node string) ([]string, error) {
moves, err := heldMovesOn(ctx, inv, f, node)
var out []string
for _, h := range moves {
out = append(out, fmt.Sprintf("%s %s → %s %s", h.Module, short(h.From), short(h.To), h.Why))
}
return out, err
}
// stringsKeys is a map of lists as a map of its keys, for sortedKeysOf.
func stringsKeys(m map[string][]string) map[string]string {
out := map[string]string{}
+4
View File
@@ -1371,6 +1371,10 @@ func backlogObservation() []conditions.Observation {
"them, `upgrade release-backlog --why …` releases them", n, strings.Join(machines, ", "), backlogNow.held)}
w := releaseHeldWords(modules, machines)
o.Headline, o.Explanation, o.Resolved, o.Needs = w.Headline, w.Explanation, w.Resolved, w.Needs
// Asked of the operator, who releases them with an Approve (novox/hq issue 1732): the controller's own
// `upgrade release-backlog`, which opens the release plan that walks them through the machines, each judged.
o.Actions = []conditions.Action{{Label: "Release", Verb: "mesh-controller.upgrade", Level: conditions.LevelApprove,
Arguments: map[string]string{"release-backlog": "true", "why": "", "cause": conditions.CauseOperatorAnswer}}}
return []conditions.Observation{o}
}
+9 -2
View File
@@ -2064,6 +2064,12 @@ func planLineSaid(p inventory.Plan, now time.Time, pause pauseView, bound time.D
// refused ("a build waits there for its gate: <machine> (...)") is kept there. "building" with LATE
// hid that reason from `plans`, `status` and the stalled condition (novox/hq issue 920).
what = p.Note
// And when the builds are held after a failed release, they wait for a person, not for a build, and the
// operator is asked to release them (novox/hq issue 1732): said so, and not late.
if strings.Contains(p.Note, errUngated.Error()) && backlogNow.held != "" {
what += " — held after a failed delivery until a person releases it (the operator is asked)"
late = ""
}
return fmt.Sprintf("%s · %s, held: %s, for %s%s", name, where, what, since, late)
}
return fmt.Sprintf("%s · %s, %s for %s%s", name, where, what, since, late)
@@ -2271,7 +2277,7 @@ func plansCommand(ctx context.Context, args []string) error {
bounds.of(p.Repository), func(module string) []string {
on, _ := inv.Running(ctx, module)
return on
}, waitsOf(ctx, inv, []inventory.Plan{p})))
}, waitsOf(ctx, inv, []inventory.Plan{p}))+heldSaid(p, heldNow(ctx, inv)))
if r := p.Release; r != nil {
// A release plan's walk (ADR 0236): machines done, the one judged, those to come.
fmt.Printf(" machines in order: %s; done: %s; skipped: %s\n", strings.Join(r.Order, ", "),
@@ -2466,11 +2472,12 @@ func plansCommand(ctx context.Context, args []string) error {
return on
}
waits := waitsOf(ctx, inv, plans)
held := heldNow(ctx, inv)
for _, p := range plans {
if p.Batch() {
continue
}
fmt.Printf("%-28s %s\n", p.ID, planLineSaid(p, now, pause, bounds.of(p.Repository), running, waits))
fmt.Printf("%-28s %s%s\n", p.ID, planLineSaid(p, now, pause, bounds.of(p.Repository), running, waits), heldSaid(p, held))
}
return nil
}
+1 -1
View File
@@ -152,7 +152,7 @@ func answersRightly(q dnsmessage.Question) *dnsmessage.Message {
func quickResolvers(t *testing.T, port string) {
t.Helper()
before, within, pause := resolverPort, resolverWithin, resolverPause
resolverPort, resolverWithin, resolverPause = port, 150*time.Millisecond, 10*time.Millisecond
resolverPort, resolverWithin, resolverPause = port, 60*time.Millisecond, 5*time.Millisecond
t.Cleanup(func() { resolverPort, resolverWithin, resolverPause = before, within, pause })
}
+16
View File
@@ -244,6 +244,22 @@ var signalsTable = []signalRow{
newest: func(f *signalFacts) time.Time {
return newestOf(f.failed, func(p failedPlanFacts) time.Time { return p.since })
}},
{Row: "S24", Signal: "a move of a recorded module that waits for a person's push is asked of the operator",
Emitter: "controller's plan", Trigger: "each machine with a move of a recorded module held for a person (novox/hq ADR 0377, issue 1732)",
Bound: "none: raised on the first look that finds the machine holding such a move, naming the modules and the " +
"machine, with Release (the machine's push) as the answer; ended when the machine no longer waits, whoever " +
"pushed it. A hold a person chose (a module set to record with a why, the bus) is not asked.",
Kind: kindMovesHeld, Severity: conditions.Warning, Phase: 3,
needs: func(f *signalFacts) error { return f.heldErr }, watch: watchMovesHeld,
newest: func(f *signalFacts) time.Time { return time.Time{} }},
{Row: "S25", Signal: "a module left out of a send because the artifact store lacks an image it names",
Emitter: "controller's send", Trigger: "each send whose declaration names an image of the store that the store does not hold (novox/hq issue 1742)",
Bound: "none: raised on the first look at a registered build that names such an image, naming the module, each " +
"machine that runs it and the image; every send leaves that module out; the store is asked again at each " +
"look and the condition ends when it holds the image",
Kind: kindImageMissing, Severity: conditions.Warning, Phase: 3,
needs: func(f *signalFacts) error { return f.imagesErr }, watch: watchMissingImages,
newest: func(f *signalFacts) time.Time { return time.Time{} }},
{Row: "S20", Signal: "a send refused for the assignment generation it was composed from",
Emitter: "node-engine", Trigger: "each refusal (novox/hq issue 234)",
Bound: "none: raised at the first refusal, naming its sender, the generation it came from and the one the " +
+13
View File
@@ -171,6 +171,19 @@ var suppressions = map[string]suppression{
since: f.now.Add(-time.Second)}}
},
},
// A machine holding a move of a recorded module for a person is asked at its first look (novox/hq issue 1732).
// Inside: nothing waits there.
"S24": {
inside: func(f *signalFacts) { f.held = nil },
past: func(f *signalFacts) { f.held = []heldFacts{aHeldMailu()} },
},
// A registered build naming an image the artifact store lacks is raised at the first look (novox/hq issue 1742).
"S25": {
inside: func(f *signalFacts) { f.images = nil },
past: func(f *signalFacts) {
f.images = []imageFacts{{missingImage{Node: "anchor", Module: "automx", Image: automxImage}}}
},
},
// A batch queued behind an open walk past that walk's bound, named.
"S19": {
inside: func(f *signalFacts) {
+8
View File
@@ -58,6 +58,12 @@ type signalFacts struct {
// failed are the delivery plans that failed and are asked of the operator (S23, novox/hq ADR 0449).
failed []failedPlanFacts
failedErr error
// held are the machines with moves of recorded modules waiting for a person's push (S24, novox/hq issue 1732).
held []heldFacts
heldErr error
// images are the modules left out of a send because the artifact store lacks an image they name (S25, issue 1742).
images []imageFacts
imagesErr error
// refusedSends are the sends a machine refused within the hour for the generation they were composed
// from, each naming its sender (S20, novox/hq issue 234).
refusedSends []inventory.Send
@@ -357,6 +363,8 @@ func (w *watchdogs) gather(ctx context.Context) *signalFacts {
f.batches, f.plansErr = gatherBatches(ctx, inv, now)
}
f.failed, f.failedErr = gatherFailedPlans(ctx, inv, now)
f.held, f.heldErr = gatherHeldMoves(ctx, inv)
f.images, f.imagesErr = gatherMissingImagesHere(ctx, inv)
f.loop, f.loopErr = w.gatherLoop()
f.asks, f.asksErr = w.gatherAsks(ctx, inv)
if open, err := w.keeper.Open(ctx); err != nil {
+1 -1
View File
@@ -3,7 +3,7 @@ module github.com/novox/mesh-controller
go 1.26.0
require (
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261021153908-725590e06e75
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261022132648-bd7731f3588c
github.com/jackc/pgx/v5 v5.10.0
github.com/nats-io/nats-server/v2 v2.11.17
github.com/nats-io/nats.go v1.54.0
+2 -2
View File
@@ -1,7 +1,7 @@
git.novox.be/novox/mesh-host v0.0.0-20261018171808-9d9e3a46c4f5 h1:E5stfIeCN6uYgA9TCv+4K8JgaURg3zQppvUgJLaDMP4=
git.novox.be/novox/mesh-host v0.0.0-20261018171808-9d9e3a46c4f5/go.mod h1:4wyoCr6IIIIarEUuROThWDj7DAvJvIidqMjWcpSnOwE=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261021153908-725590e06e75 h1:AcYfyJpYaUufhI93VG76+b001msXY4XX7kuRjnf33HY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261021153908-725590e06e75/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261022132648-bd7731f3588c h1:UKlwMw6fmesDqe0aL0lfI/ivhorsGjAvh9XefFjhNI4=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261022132648-bd7731f3588c/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
+1 -1
View File
@@ -153,7 +153,7 @@ func ChainBuckets() []string { return []string{BuildsBucket, RolloutsBucket, Ass
// ControllerMirrors are the local names of the controller's buckets a module may read: the record mirrors and the
// chain's buckets.
func ControllerMirrors() []string {
return []string{MirrorAssignments, MirrorDeclaredConfig, MirrorPredefined, ChainBuilds, ChainRollouts, ChecksRunning}
return []string{MirrorAssignments, MirrorDeclaredConfig, MirrorPredefined, ChainBuilds, ChainRollouts, ChecksRunning, PlansBuilding}
}
// MirrorsWrittenByAnother are the controller's mirrors whose writer is not the controller, each with its reason: the rule
-22
View File
@@ -2,10 +2,7 @@ package broker
import (
"context"
"crypto/sha256"
"crypto/tls"
"crypto/x509"
"encoding/hex"
"errors"
"fmt"
"os"
@@ -112,25 +109,6 @@ func DialPinned(url, fingerprint string, opts ...nats.Option) (*JetStream, error
return Dial(url, opts...)
}
// PinnedToFingerprint accepts exactly the certificate with this SHA-256 and no other.
func PinnedToFingerprint(want string) *tls.Config {
return &tls.Config{
InsecureSkipVerify: true, //nolint:gosec // replaced by the pin below, which is stricter
MinVersion: tls.VersionTLS12,
VerifyPeerCertificate: func(rawCerts [][]byte, _ [][]*x509.Certificate) error {
if len(rawCerts) == 0 {
return errors.New("the bus presented no certificate")
}
sum := sha256.Sum256(rawCerts[0])
got := "sha256:" + hex.EncodeToString(sum[:])
if got != want {
return fmt.Errorf("the bus presented a certificate this mesh does not know (%s…), expected %s…", got[:23], want[:23])
}
return nil
},
}
}
// Conn is the connection itself, for what the mesh keeps off JetStream on purpose — a heartbeat,
// a tool call — where a lost message is answered by the next one or by a timeout the caller
// already handles (design 25 §3).
-11
View File
@@ -98,17 +98,6 @@ func (l Log) Config() jetstream.StreamConfig {
}
}
// LogIssued is one log an assignment may reach, by the name its module uses for it (novox/hq ADR
// 0297): its stream, the subject its entries go under, and whether it may append. Only the owner's
// instances are issued a log, so Writes is always true today; it is said so the runtime need not
// assume it.
type LogIssued struct {
Name string `json:"name"`
Stream string `json:"stream"`
Subject string `json:"subject"`
Writes bool `json:"writes"`
}
// logsIssuedFor is every log a module's code may reach, as its membership lists them: its own.
func logsIssuedFor(d Declared) []LogIssued {
var out []LogIssued
-76
View File
@@ -1,7 +1,6 @@
package broker
import (
"encoding/json"
"slices"
"sort"
"strings"
@@ -16,81 +15,6 @@ import (
// grant is the same composition read the other way. The shape issued today is the shape the mesh
// already had, so nothing moves when a membership first arrives; only who decides it moves.
// Membership is one assignment's subjects: what this instance of a module on this machine serves,
// and what it may reach.
type Membership struct {
Node string `json:"node"`
Module string `json:"module"`
// Serves is every address a tool of this instance answers on. `{tool}` stands for the tool's
// own name, which the module knows and the mesh does not need to: the mesh issues the address,
// the runtime fills the name. An address with a queue is shared with the module's other
// instances, and the bus hands each call to one of them; an address without is this instance's.
Serves []Served `json:"serves"`
// Seats is every verb of a seat this instance holds, at the subject the seat's callers use.
Seats []SeatServed `json:"seats,omitempty"`
// Emits is where an event of this module lands; `{event}` stands for the event's name.
Emits string `json:"emits"`
// Reaches is each tool this module may call, `<module>.<tool>`, to the subjects that reach it:
// the first is whichever instance answers, when the mesh issued one; the rest name a machine.
Reaches map[string][]string `json:"reaches,omitempty"`
// Tools is where this instance answers what it serves — the runtime's one verb of its own.
Tools string `json:"tools"`
// Receives is what this assignment is given for each requirement it receives, by requirement:
// the contributions of every module that asked for it, as the catalogue composed them (novox/hq
// ADR 0167). The same list its received file is written from, so the two cannot disagree; a
// requirement nobody contributed to is an empty list, never absent. Kept as JSON because the
// catalogue owns the shape of a contribution and the bus only carries it.
Receives map[string]json.RawMessage `json:"receives,omitempty"`
// Mesh is every machine's address on the private network — what a rule saying "from the mesh"
// resolves to in the packet filter, issued here from the same list (novox/hq ADR 0167). A
// module that must tell the mesh from the world, the route proxy serving an internal name, reads
// it here rather than keeping a definition of its own.
Mesh []string `json:"mesh,omitempty"`
// State is every bucket this module's code may reach, by the name it uses for each, and whether
// it may write it (novox/hq ADR 0201): the runtime answers a bundle's state verbs from this list
// and refuses, with the reason, what is not on it — the bus enforces only the union over every
// module on the machine.
State []StateIssued `json:"state,omitempty"`
// Logs is every log this module's code may reach, by the name it uses for each (novox/hq ADR 0297):
// its own and no other's, since a log is its owner's alone. The runtime answers a bundle's log verbs
// from this list and refuses, with the reason, a log not on it.
Logs []LogIssued `json:"logs,omitempty"`
// SeatTraffic is what this module's code may submit, say, hear, take, ask, answer and read on seats
// that name their caller or their kind (novox/hq ADR 0259 §3). The runtime carrying the module
// publishes, takes and answers for it only what is listed here: the bus enforces only the union
// over every module on the machine, so one module's code reaching another's name or kind through
// the runtime is the runtime's to refuse.
SeatTraffic *SeatTraffic `json:"seat-traffic,omitempty"`
// Streams are this module's own streams (novox/hq ADR 0363 §2): an event one of them carries is published
// on that stream's subject, never on Emits. Hears are the module streams it hears, each through a durable
// consumer on that stream the runtime binds beside the one on EVENTS (§8).
Streams []StreamIssued `json:"streams,omitempty"`
Hears []HeardIssued `json:"hears,omitempty"`
// Waits are the event families a session may wait on through the mesh MCP server, the tool runner's
// alone (novox/hq ADR 0367 §2): the subjects its account is granted, under the names a session asks for.
// The tool runner refuses any family not listed here.
Waits []WaitIssued `json:"waits,omitempty"`
}
// Served is one address a tool is answered on.
type Served struct {
Subject string `json:"subject"`
Queue string `json:"queue,omitempty"`
}
// SeatServed is one verb of a held seat, where its callers ask.
type SeatServed struct {
Seat string `json:"seat"`
Verb string `json:"verb"`
Subject string `json:"subject"`
}
// MembershipSubject is the one address a runtime derives for itself: where its own membership is
// published, from the two names its credential carries. Everything else is in the membership.
func MembershipSubject(node, module string) string {
return "mesh.assignment." + node + "." + module
}
// Placements is where every module runs, for deciding which instance answers for the module.
type Placements struct {
// Nodes is each module's machines.
-15
View File
@@ -260,21 +260,6 @@ func ownStreamGrants(module string, streams []ModuleStream) []string {
return out
}
// StreamIssued is one of a module's own streams as its membership names it (ADR 0363 §2): the events it
// carries, as the bus spells them, and the subject an event of them is published on, `{event}` the event.
type StreamIssued struct {
Stream string `json:"stream"`
Events []string `json:"events"`
Subject string `json:"subject"`
}
// HeardIssued is one module stream an assignment hears, as its membership names it (ADR 0363 §8): the
// stream and the durable consumer on it the runtime binds.
type HeardIssued struct {
Stream string `json:"stream"`
Consumer string `json:"consumer"`
}
// streamsIssuedFor is every stream of its own a module's membership names.
func streamsIssuedFor(d Declared) []StreamIssued {
var out []StreamIssued
+17 -5
View File
@@ -361,9 +361,11 @@ var VaultNodeKey = SeatVerb{Seat: "mesh-vault", Verb: "node-key"}
// VerbsTheControllerActsOnAWarrant are the other seat verbs the controller calls when the operator's warrant
// chooses them (novox/hq ADR 0259): a machine's service restarted, and a walk started or stopped through the
// controller's own `plans`. Named one by one; a node seat's on any machine, a mesh seat's flat.
// controller's own `plans`, a machine's held builds sent by its own `push` and the builds held after a failed release
// sent by its own `upgrade release-backlog` (novox/hq issue 1732). Named one by one; a node seat's on any machine, a
// mesh seat's flat.
var VerbsTheControllerActsOnAWarrant = []SeatVerb{{Seat: "node-service-manager", Verb: "restart"},
{Seat: ControllerSeat, Verb: "plans"}}
{Seat: ControllerSeat, Verb: "plans"}, {Seat: ControllerSeat, Verb: "push"}, {Seat: ControllerSeat, Verb: "upgrade"}}
// VerbsTheControllerAsksTheOperatorChannel are the operator channel's verbs the controller calls (novox/hq
// ADR 0380): `tell`, one plain message with no condition behind it — a module's own secret given, named,
@@ -387,6 +389,16 @@ type DegradableGrant struct {
// works. Every other grant stays required — missing one of those is a controller that comes up and is
// refused on the first thing it tries, with no fallback that covers it.
var DegradableGrants = []DegradableGrant{
{
Subject: "mesh.seat." + ControllerSeat + ".tool.push",
Reason: "refused: the ask of a held move (Release) is answered, and the refusal told, while the operator's own " +
"push at the terminal still sends the machine (issue 1732; ADR 0259)",
},
{
Subject: "mesh.seat." + ControllerSeat + ".tool.upgrade",
Reason: "refused: the ask of the builds held after a failed release is answered, and the refusal told, while the " +
"operator's own `upgrade release-backlog` at the terminal still releases them (issue 1732; ADR 0259)",
},
{
Subject: "mesh.seat." + VerbsTheControllerAsksTheOperatorChannel[0].Seat + ".tool." + VerbsTheControllerAsksTheOperatorChannel[0].Verb,
Reason: "refused: the controller raises the secret-given condition as before (ADR 0380 §9)",
@@ -702,7 +714,7 @@ func PermissionsFor(p Principal) (Permissions, error) {
// And asking the operator (novox/hq ADR 0259): an ask and its cancel under its own name, its warrants
// heard under its own name, the record of its asks read under its own name — as any user of the seat,
// derived the same way, from the seat its holder declares.
tp, ts := SeatTrafficOf(ControllerSeat, nil, p.Uses, nil).grants()
tp, ts := seatTrafficGrants(SeatTrafficOf(ControllerSeat, nil, p.Uses, nil))
pub = append(pub, tp...)
sub = append(sub, ts...)
// And asks who answers (novox/hq to-be 45 §4, D3): the self-check finds every seat's holder by
@@ -1035,7 +1047,7 @@ func PermissionsFor(p Principal) (Permissions, error) {
// 6. Its traffic on seats that name their caller or their kind, ask proofs or keep records
// (novox/hq ADR 0259 §3).
tp, ts := SeatTrafficOf(p.Module, p.Holds, p.Uses, p.Watches).grants()
tp, ts := seatTrafficGrants(SeatTrafficOf(p.Module, p.Holds, p.Uses, p.Watches))
pub = append(pub, tp...)
sub = append(sub, ts...)
@@ -1159,7 +1171,7 @@ func PermissionsFor(p Principal) (Permissions, error) {
// not publish under another's name or kind through it is the runtime's to keep, from the seat
// traffic each module's membership lists.
for _, d := range p.Carries {
tp, ts := SeatTrafficOf(d.Module, d.Holds, d.Uses, d.Watches).grants()
tp, ts := seatTrafficGrants(SeatTrafficOf(d.Module, d.Holds, d.Uses, d.Watches))
pub = append(pub, tp...)
sub = append(sub, ts...)
}
-33
View File
@@ -35,39 +35,6 @@ func OnNATS() (address string, on bool, err error) {
return address, true, nil
}
// CredentialIn reads the user and password out of a bus address, and the address without them.
//
// The controller's own credential arrives in its address, the way the old bus's does. Split out so the
// controller can record a hash of what it is actually using: its user is created by the installer at a
// bootstrap password, before the controller exists to mint one, and a composition that left itself out
// would produce a bus the writer cannot connect to.
func CredentialIn(address string) (user, password, bare string) {
at := strings.LastIndex(address, "@")
if at < 0 {
return "", "", address
}
scheme := ""
rest := address[:at]
if i := strings.Index(rest, "://"); i >= 0 {
scheme, rest = rest[:i+3], rest[i+3:]
}
user, password, _ = strings.Cut(rest, ":")
return user, password, scheme + address[at+1:]
}
// BareAddress is a bus address with any credential stripped, for something that only needs to know
// whether a server is answering there.
func BareAddress(address string) string {
_, _, bare := CredentialIn(address)
if bare == "" {
return address
}
if strings.Contains(bare, "://") {
return bare
}
return "nats://" + bare
}
// BusAddress is where the mesh's bus is, with this process's credential, and refuses to be empty:
// there is one bus, and a control plane without it can hold records and answer nothing (novox/hq
// ADR 0131, design 28 task 5.5).
-28
View File
@@ -1,28 +0,0 @@
package broker
import (
"testing"
)
// Which bus the mesh is on is one fact, and being told about both is refused.
//
// The controller's own credential arrives in its address, and has to be readable out of it — its user
// is created by the installer at a bootstrap password, before the controller exists to mint one.
func TestACredentialIsReadOutOfABusAddress(t *testing.T) {
for _, c := range []struct{ in, user, password, bare string }{
{"nats://controller:secret@127.0.0.1:4222", "controller", "secret", "nats://127.0.0.1:4222"},
{"controller:secret@127.0.0.1:4222", "controller", "secret", "127.0.0.1:4222"},
{"nats://127.0.0.1:4222", "", "", "nats://127.0.0.1:4222"},
{"127.0.0.1:4222", "", "", "127.0.0.1:4222"},
// A password containing an at-sign: split on the last one, or the address becomes part of the
// credential and the connection goes somewhere nobody named.
{"nats://controller:a@b@127.0.0.1:4222", "controller", "a@b", "nats://127.0.0.1:4222"},
} {
user, password, bare := CredentialIn(c.in)
if user != c.user || password != c.password || bare != c.bare {
t.Errorf("%q read as %q/%q at %q; wanted %q/%q at %q",
c.in, user, password, bare, c.user, c.password, c.bare)
}
}
}
@@ -0,0 +1,30 @@
package broker
import (
"slices"
"testing"
)
// Issue 1687 (ADR 0483 clause 2): a module whose manifest reads mesh-controller.plans-building may name it in `reads`, and
// reads the bucket and writes none; the controller writes it, so it is no mirror written by another.
func TestAModuleReadingPlansBuildingReadsAndWritesNone(t *testing.T) {
if !slices.Contains(ControllerMirrors(), "plans-building") || BucketName(ControllerSeat, PlansBuilding) != PlansBuildingBucket {
t.Fatalf("a module cannot name %s in `reads`: %v", PlansBuildingBucket, ControllerMirrors())
}
if _, other := MirrorsWrittenByAnother[PlansBuilding]; other {
t.Error("the controller writes the plans building itself")
}
perms, err := PermissionsFor(Principal{Kind: KindModule, Node: "one", Module: "mesh-delivery-checker", PasswordHash: "x",
Reads: []string{"mesh-controller.plans-building"}})
if err != nil {
t.Fatal(err)
}
stream := "KV_" + PlansBuildingBucket
has(t, perms.Publish, "$JS.API.STREAM.INFO."+stream)
has(t, perms.Publish, "$JS.API.DIRECT.GET."+stream+".>")
for _, p := range perms.Publish {
if SubjectsOverlap(p, "$KV."+PlansBuildingBucket+".plan-1") {
t.Errorf("the reading module may write %s through %q", PlansBuildingBucket, p)
}
}
}
+38
View File
@@ -0,0 +1,38 @@
package broker
import (
"git.novox.be/novox/mesh-sdk/go/membership"
"git.novox.be/novox/mesh-sdk/go/process"
)
// The membership the mesh issues an assignment (novox/hq ADR 0160), its subject, its stream, the bare address and the
// pin are mesh-sdk's, public so that a module's program outside this Go module, the route proxy, reads them too
// (novox/hq ADR 0480 clause 4, issue 1633). This package keeps no second copy: what follows only names them where the
// controller already did.
type (
Membership = membership.Membership
Served = membership.Served
SeatServed = membership.SeatServed
StateIssued = membership.StateIssued
LogIssued = membership.LogIssued
Worker = membership.Worker
SeatTraffic = membership.SeatTraffic
KindHeld = membership.KindHeld
StreamIssued = membership.StreamIssued
HeardIssued = membership.HeardIssued
WaitIssued = membership.WaitIssued
)
// AssignmentsStream holds every assignment's membership, the newest per subject.
const AssignmentsStream = membership.AssignmentsStream
var (
// MembershipSubject is where an assignment's own membership is published.
MembershipSubject = membership.Subject
// CredentialIn reads the user and password out of a bus address, and the address without them.
CredentialIn = membership.CredentialIn
// BareAddress is a bus address with any credential stripped.
BareAddress = membership.BareAddress
// PinnedToFingerprint accepts exactly the certificate with this SHA-256 and no other.
PinnedToFingerprint = process.PinnedToFingerprint
)
+2 -40
View File
@@ -24,44 +24,6 @@ import "sort"
// And one read: **a holder's records**, a bucket the seat names, read by each user under its own name only
// (`$KV.<bucket>.<module>.>`), so an asker reads the state of its own asks and no other asker's.
// Worker is one durable consumer a holder pulls a seat's work from.
type Worker struct {
Stream string
Consumer string
Filter string
}
// SeatTraffic is one module's seat traffic beyond tools. Publish and Subscribe are subject patterns, in the
// server's wildcards; the runtime that carries the module checks a bundle's request against them, since
// the runtime's own principal holds the union of every module it carries.
type SeatTraffic struct {
// Publish is what it submits (accepts of seats it uses), says (events of seats it holds) and asks
// (proofs of seats it holds a kind of).
Publish []string `json:"publish,omitempty"`
// Subscribe is what it hears: events of seats it uses that are named by caller, events of seats it
// watches, and accepts of seats it holds.
Subscribe []string `json:"subscribe,omitempty"`
// Answers is the proof subjects it answers, as a watcher of a kinded seat.
Answers []string `json:"answers,omitempty"`
// Workers are the work queues it takes from, as a holder.
Workers []Worker `json:"workers,omitempty"`
// Records is the direct-get subjects of the records it reads under its own name.
Records []string `json:"records,omitempty"`
// Kinds are the holders of every kinded bench it uses or watches, with what each promises: the one
// account of which channel is which, and what it can carry, that the router judges an answer by. The
// controller's, from the claims, never a channel's word (novox/hq ADR 0259 §5).
Kinds []KindHeld `json:"kinds,omitempty"`
}
// KindHeld is one kind of a kinded bench and who holds it.
type KindHeld struct {
Seat string `json:"seat"`
Kind string `json:"kind"`
Module string `json:"module"`
Node string `json:"node"`
Capabilities []string `json:"capabilities,omitempty"`
}
// KindedBenches are the seats that may be kinded (ADR 0234 §2): making another is a decision, recorded.
var KindedBenches = map[string]bool{"channel": true, "intake": true}
@@ -175,9 +137,9 @@ func SeatTrafficOf(module string, holds, uses, watches []Seat) SeatTraffic {
return t
}
// grants is the bus permissions seat traffic needs: the subjects themselves, and the JetStream API a
// seatTrafficGrants is the bus permissions seat traffic needs: the subjects themselves, and the JetStream API a
// worker is pulled and acknowledged through and a record is read through.
func (t SeatTraffic) grants() (pub, sub []string) {
func seatTrafficGrants(t SeatTraffic) (pub, sub []string) {
pub = append(pub, t.Publish...)
sub = append(sub, t.Subscribe...)
sub = append(sub, t.Answers...)
-10
View File
@@ -186,16 +186,6 @@ func perMachineNames(buckets []Bucket) []string {
return out
}
// StateIssued is one bucket an assignment may reach, by the name its module uses for it: its own
// state by the local name, another's as `<module>.<name>` (novox/hq ADR 0201). Key, when set, is the
// one key it may reach (novox/hq ADR 0260): this machine's.
type StateIssued struct {
Name string `json:"name"`
Bucket string `json:"bucket"`
Writes bool `json:"writes,omitempty"`
Key string `json:"key,omitempty"`
}
// stateIssuedFor is every bucket a module's code may reach on a machine, as its membership lists them.
func stateIssuedFor(d Declared, node string) []StateIssued {
var out []StateIssued
-3
View File
@@ -80,9 +80,6 @@ const RestrictedMetadata = "mesh-restricted"
// it under: never set by a mesh-internal work-queue stream (a seat's, a check's), only by Stream.WorkQueue.
const WorkQueueMetadata = "mesh-work-queue"
// AssignmentsStream holds every assignment's membership, the newest per subject.
const AssignmentsStream = "ASSIGNMENTS"
// What a durable consumer gave up on is kept (novox/hq issue 330, design 25 §3).
//
// **The server says it and keeps it; the controller copies it.** A consumer that handed a message over
+1 -1
View File
@@ -29,7 +29,7 @@ accounts {
]
users = [
{ user: "controller", password: "$2a$11$cccccccccccccccccccccc", permissions: {
publish: { allow: ["$JS.ACK.CONTROL.controller.>", "$JS.ACK.DEAD_LETTER_NOTICES.controller.>", "$JS.ACK.EVENTS.controller-merges.>", "$JS.ACK.EVENTS.controller.>", "$JS.API.>", "$KV.SEAT_MESH_BUILD_MACHINE_cancelled.>", "$KV.SEAT_NODE_BUILD_AGENT_cancelled.>", "$KV.mesh-controller_asked.>", "$KV.mesh-controller_assignments.>", "$KV.mesh-controller_builds.>", "$KV.mesh-controller_calls.>", "$KV.mesh-controller_checks-running.>", "$KV.mesh-controller_condition-history.>", "$KV.mesh-controller_conditions.>", "$KV.mesh-controller_declared-config.>", "$KV.mesh-controller_delivery-merges.>", "$KV.mesh-controller_hand-acts.>", "$KV.mesh-controller_lease.>", "$KV.mesh-controller_merge-window.>", "$KV.mesh-controller_plans-building.>", "$KV.mesh-controller_predefined.>", "$KV.mesh-controller_rollout-silent.>", "$KV.mesh-controller_rollouts.>", "$SRV.INFO", "_INBOX.enrol.>", "mesh.again.>", "mesh.assignment.>", "mesh.events.dead.>", "mesh.mod.*.tool.>", "mesh.node.>", "mesh.seat.mesh-build-machine.accept.>", "mesh.seat.mesh-build-machine.tool.>", "mesh.seat.mesh-controller.event.applied", "mesh.seat.mesh-controller.event.assignment-changed", "mesh.seat.mesh-controller.event.built-before", "mesh.seat.mesh-controller.event.checked", "mesh.seat.mesh-controller.event.condition-changed", "mesh.seat.mesh-controller.event.condition-cleared", "mesh.seat.mesh-controller.event.condition-raised", "mesh.seat.mesh-controller.event.declared-config-changed", "mesh.seat.mesh-controller.event.doctor-heartbeat", "mesh.seat.mesh-controller.event.healer-acted", "mesh.seat.mesh-controller.event.plan-moved", "mesh.seat.mesh-controller.event.predefined-changed", "mesh.seat.mesh-controller.event.refused", "mesh.seat.mesh-controller.event.rolled-back", "mesh.seat.mesh-controller.event.secret-replaced", "mesh.seat.mesh-controller.event.sent", "mesh.seat.mesh-controller.tool.plans", "mesh.seat.mesh-delivery.tool.close", "mesh.seat.mesh-delivery.tool.groups", "mesh.seat.mesh-delivery.tool.release", "mesh.seat.mesh-delivery.tool.stalled", "mesh.seat.mesh-delivery.tool.stop", "mesh.seat.mesh-delivery.tool.times", "mesh.seat.node-backup.tool.backed-up.*", "mesh.seat.node-backup.tool.now.*", "mesh.seat.node-build-agent.accept.>", "mesh.seat.node-build-agent.check.>", "mesh.seat.node-build-agent.tool.>", "mesh.seat.node-intrusion-prevention.tool.banned.*", "mesh.seat.node-launcher.tool.secret.*", "mesh.seat.node-service-manager.tool.restart.*", "mesh.seat.operator-channel.tool.tell"] }
publish: { allow: ["$JS.ACK.CONTROL.controller.>", "$JS.ACK.DEAD_LETTER_NOTICES.controller.>", "$JS.ACK.EVENTS.controller-merges.>", "$JS.ACK.EVENTS.controller.>", "$JS.API.>", "$KV.SEAT_MESH_BUILD_MACHINE_cancelled.>", "$KV.SEAT_NODE_BUILD_AGENT_cancelled.>", "$KV.mesh-controller_asked.>", "$KV.mesh-controller_assignments.>", "$KV.mesh-controller_builds.>", "$KV.mesh-controller_calls.>", "$KV.mesh-controller_checks-running.>", "$KV.mesh-controller_condition-history.>", "$KV.mesh-controller_conditions.>", "$KV.mesh-controller_declared-config.>", "$KV.mesh-controller_delivery-merges.>", "$KV.mesh-controller_hand-acts.>", "$KV.mesh-controller_lease.>", "$KV.mesh-controller_merge-window.>", "$KV.mesh-controller_plans-building.>", "$KV.mesh-controller_predefined.>", "$KV.mesh-controller_rollout-silent.>", "$KV.mesh-controller_rollouts.>", "$SRV.INFO", "_INBOX.enrol.>", "mesh.again.>", "mesh.assignment.>", "mesh.events.dead.>", "mesh.mod.*.tool.>", "mesh.node.>", "mesh.seat.mesh-build-machine.accept.>", "mesh.seat.mesh-build-machine.tool.>", "mesh.seat.mesh-controller.event.applied", "mesh.seat.mesh-controller.event.assignment-changed", "mesh.seat.mesh-controller.event.built-before", "mesh.seat.mesh-controller.event.checked", "mesh.seat.mesh-controller.event.condition-changed", "mesh.seat.mesh-controller.event.condition-cleared", "mesh.seat.mesh-controller.event.condition-raised", "mesh.seat.mesh-controller.event.declared-config-changed", "mesh.seat.mesh-controller.event.doctor-heartbeat", "mesh.seat.mesh-controller.event.healer-acted", "mesh.seat.mesh-controller.event.plan-moved", "mesh.seat.mesh-controller.event.predefined-changed", "mesh.seat.mesh-controller.event.refused", "mesh.seat.mesh-controller.event.rolled-back", "mesh.seat.mesh-controller.event.secret-replaced", "mesh.seat.mesh-controller.event.sent", "mesh.seat.mesh-controller.tool.plans", "mesh.seat.mesh-controller.tool.push", "mesh.seat.mesh-controller.tool.upgrade", "mesh.seat.mesh-delivery.tool.close", "mesh.seat.mesh-delivery.tool.groups", "mesh.seat.mesh-delivery.tool.release", "mesh.seat.mesh-delivery.tool.stalled", "mesh.seat.mesh-delivery.tool.stop", "mesh.seat.mesh-delivery.tool.times", "mesh.seat.node-backup.tool.backed-up.*", "mesh.seat.node-backup.tool.now.*", "mesh.seat.node-build-agent.accept.>", "mesh.seat.node-build-agent.check.>", "mesh.seat.node-build-agent.tool.>", "mesh.seat.node-intrusion-prevention.tool.banned.*", "mesh.seat.node-launcher.tool.secret.*", "mesh.seat.node-service-manager.tool.restart.*", "mesh.seat.operator-channel.tool.tell"] }
subscribe: { allow: ["$JS.API.>", "$JS.EVENT.ADVISORY.CONSUMER.DELETED.>", "$JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES.>", "$SRV.INFO", "$SRV.INFO.mesh-controller", "$SRV.INFO.mesh-controller.>", "$SRV.PING", "$SRV.PING.mesh-controller", "$SRV.PING.mesh-controller.>", "$SRV.STATS", "$SRV.STATS.mesh-controller", "$SRV.STATS.mesh-controller.>", "_DELIVER.controller", "_DELIVER.controller.>", "_INBOX.controller.>", "mesh.control.>", "mesh.mod.*.event.provisioner.failing", "mesh.mod.*.event.provisioner.recovered", "mesh.mod.*.event.provisioner.retirement", "mesh.mod.gitea.event.pull.merged", "mesh.mod.gitea.event.pull.updated", "mesh.mod.mesh-catalog.event.catching-up", "mesh.mod.mesh-catalog.event.upgraded", "mesh.seat.mesh-build-machine.event.built", "mesh.seat.mesh-controller.tool.>", "mesh.seat.node-build-agent.event.built", "mesh.seat.operator-channel.event.decided.mesh-controller"] }
allow_responses: { max: 1, ttl: "1m" }
} }
-6
View File
@@ -31,12 +31,6 @@ type Wait struct {
Last string
}
// WaitIssued is one family as a membership issues it: the name a session asks for, and its subject.
type WaitIssued struct {
Event string `json:"event"`
Subject string `json:"subject"`
}
// WaitSubject is where a family's events land. Each token is checked before it becomes a subject: a
// wildcard, a dot inside a name or an empty token would widen the grant.
func WaitSubject(w Wait) (string, error) {
+1 -1
View File
@@ -177,7 +177,7 @@ var WritersTable = []WriterRow{
KeptIn: "key-value " + CheckRunningBucket + ", aged out when not renewed", Others: "the build seat's holders read it to drop a duplicate request",
Subjects: kvOf(CheckRunningBucket), Writes: holdsTheBuildSeat,
Shared: "every machine holding the build seat puts the key of the check it runs and deletes it when the check ends"},
{State: "a delivery plan in its build phase", Writer: "controller (lease holder), at the first build's ask and at each refresh",
{State: "the plans-building mirror", Writer: "controller (lease holder), at the first build's ask and at each refresh",
KeptIn: "key-value " + PlansBuildingBucket + ", aged out when not refreshed",
Others: "the build seat's holders read it and take no new check while a key is there (ADR 0483)",
Subjects: kvOf(PlansBuildingBucket), Writes: isController},
+1 -1
View File
@@ -30,7 +30,7 @@ var designRows = []string{
"a pull request's check asked",
"a pull request's check running",
// The plans in their build phase, a bucket the build seat's holders read (ADR 0483 clause 2, issue 1687).
"a delivery plan in its build phase",
"the plans-building mirror",
"a merge announced",
"a pull request's head announced",
"a pull request's merge check",
+6 -3
View File
@@ -86,11 +86,14 @@ func TestAnImageCompilingGoIsHandedItsBuildSourceAndSaysIt(t *testing.T) {
}
for file, want := range map[string]bool{"proxy/main.go": true, "internal/shared/s.go": true, "go.mod": true,
"cmd/ctl/main.go": false, "internal/only/o.go": false, "README.md": false} {
_, err := os.Stat(filepath.Join(workspace, "narrow-server", filepath.FromSlash(file)))
if (err == nil) != want {
t.Errorf("%s handed to the recipe: %v, wanted %v", file, err == nil, want)
held := slices.Contains(r.builtFrom, file)
if held != want {
t.Errorf("%s handed to the recipe: %v, wanted %v", file, held, want)
}
}
if _, err := os.Stat(filepath.Join(workspace, "narrow-server")); err == nil {
t.Error("the narrowed build source was left in the workspace")
}
// Said per repository: its own, the manifest and the recipe; the context's, the closure.
if len(got.Sources) != 2 {
+55 -1
View File
@@ -219,6 +219,11 @@ func build(ctx context.Context, run Runner, publish Publisher,
if err := os.MkdirAll(workspace, 0o755); err != nil {
return Result{}, err
}
// **A node with no room takes no build** (novox/hq issue 1834).
if err := RoomToWork("/", workspace); err != nil {
say("clone", "refused: %v", err)
return Result{}, err
}
// The credential is a file git reads, never an argument: a URL carrying a password in argv
// would be readable by anything that can list processes for as long as a clone runs.
credentials, forget, err := storeCredential(workspace, forge)
@@ -738,11 +743,41 @@ func logging(log Log) func(step, format string, args ...any) {
}
}
// keepsImageLocally says the publisher leaves the image in the machine's runtime as the artifact itself.
func keepsImageLocally(publish Publisher) bool {
switch publish.(type) {
case Local, *Local:
return true
}
return false
}
// PushedNames is implemented by a publisher that tags the image under other names to push it: the names it leaves in the
// machine's runtime for a repository, which go with the build's own tag (novox/hq issue 1834).
type PushedNames interface {
PushedNames(repository string) []string
}
// releaseImage removes the names an image was given on the node, in one command: its layers go only with the last. It runs
// with a context of its own, the build's may be cancelled. A name never made (the build failed before it) is no fault;
// one that cannot be removed is said, and removal is said only when the image is gone.
func releaseImage(ctx context.Context, run Runner, names []string, say func(step, format string, args ...any)) {
removing, done := context.WithTimeout(context.WithoutCancel(ctx), time.Minute)
defer done()
if _, err := run(removing, "", "docker", append([]string{"rmi"}, names...)...); err != nil {
if !strings.Contains(err.Error(), "No such image") {
say("image", "the local image %s could not be removed, and goes with the week's sweep: %v", strings.Join(names, ", "), err)
}
return
}
say("image", "the local image %s is removed: the registry holds it", strings.Join(names, ", "))
}
// contextFrom clones an image artifact's own build context, when it names one apart from this
// module's own repository — a fresh tree, the same way the module's own is, keyed by artifact
// name so two artifacts of one module naming different contexts do not collide.
func contextFrom(ctx context.Context, run Runner, workspace, artifact, credentials string,
from catalogue.ArtifactContext, seats map[string]string, say func(step, format string, args ...any)) (string, error) {
from catalogue.ArtifactContext, seats map[string]string, say func(step, format string, args ...any)) (_ string, err error) {
url, err := contextURL(from, seats)
if err != nil {
return "", err
@@ -752,6 +787,12 @@ func contextFrom(ctx context.Context, run Runner, workspace, artifact, credentia
if err := os.RemoveAll(dir); err != nil {
return "", err
}
// A clone that failed half-way is no context to leave behind (novox/hq issue 1834).
defer func() {
if err != nil {
os.RemoveAll(dir)
}
}()
if _, err := run(ctx, workspace, "git", cloneWith(credentials, "clone", "--quiet", url, dir)...); err != nil {
return "", fmt.Errorf("cannot clone %s: %w", url, err)
}
@@ -1143,6 +1184,8 @@ func one(ctx context.Context, run Runner, publish Publisher,
src.contexts[a.Name] = t
}
buildDir = cloned
// Let go of when the image is built, whatever became of it (novox/hq issue 1834).
defer os.RemoveAll(cloned)
if t, _ := trunkOf(ctx, run, cloned, "HEAD"); t == "" || !atTrunkHead(ctx, run, cloned, "HEAD", t) {
src.contextOffHead()
}
@@ -1180,6 +1223,7 @@ func one(ctx context.Context, run Runner, publish Publisher,
src.ownHas(paths...)
}
buildDir = narrowed
defer os.RemoveAll(narrowed)
} else if a.Context != nil {
src.readWhole(*a.Context)
}
@@ -1205,6 +1249,16 @@ func one(ctx context.Context, run Runner, publish Publisher,
}
invocation = append(invocation, ".")
say("image", "docker build -f %s", recipePath)
// **The local tag goes with the build** (novox/hq issue 1834): an image published to a registry is held there, and
// a tag per module and commit left on the node was the 96 GB of a full disk. Only a node publishing nowhere (a
// mesh being raised) keeps it, since the local image is then the artifact.
if !keepsImageLocally(publish) {
names := []string{local}
if p, ok := publish.(PushedNames); ok {
names = append(names, p.PushedNames(module+"/"+a.Name)...)
}
defer releaseImage(ctx, run, names, say)
}
if _, err := run(ctx, buildDir, "docker", invocation...); err != nil {
return catalogue.Built{}, fmt.Errorf("%s: building %s failed: %w", module, a.Name, err)
}
+13
View File
@@ -38,6 +38,9 @@ type recorded struct {
stamped time.Time
// tree is what git names as the tree of the module's directory; empty answers as for a commit.
tree string
// builtFrom is the files (slash-separated, relative) of the folder the last `docker build` ran in, read as it ran:
// the build removes that folder when it ends (novox/hq issue 1834).
builtFrom []string
}
func (r *recorded) run(_ context.Context, dir, name string, args ...string) (string, error) {
@@ -53,6 +56,16 @@ func (r *recorded) run(_ context.Context, dir, name string, args ...string) (str
break
}
}
if name == "docker" && len(args) > 0 && args[0] == "build" {
r.builtFrom = nil
_ = filepath.WalkDir(dir, func(p string, d os.DirEntry, err error) error {
if err == nil && !d.IsDir() {
rel, _ := filepath.Rel(dir, p)
r.builtFrom = append(r.builtFrom, filepath.ToSlash(rel))
}
return nil
})
}
switch {
case name == "git" && isClone:
repository := args[len(args)-2]
+85 -5
View File
@@ -123,8 +123,41 @@ type CheckSpec struct {
// controller could not learn them, and the verdict says so.
Siblings []Sibling
SiblingsUnknown string
// Held are the approved held values of the check's pull requests (novox/hq ADR 0502 clause 4): written to a file and
// handed to the gate, which lays them over the change's composition only. None: no flag, no file.
Held []HeldValue
}
// HeldValue is one approved config value held for a key an open pull request introduces: the value of Key for Module on
// Node (a machine's real name; the gate finds it in the snapshot by its pseudonym). The gate reads this document.
type HeldValue struct {
Pull string `json:"pull"`
Head string `json:"head,omitempty"`
Node string `json:"node"`
Module string `json:"module"`
Key string `json:"key"`
Value json.RawMessage `json:"value"`
Approved string `json:"approved,omitempty"`
By string `json:"by,omitempty"`
At string `json:"at,omitempty"`
}
// writeHeld writes the held values to held.json under root and answers its path; empty, and nothing written, for none.
func writeHeld(root string, held []HeldValue) (string, error) {
if len(held) == 0 {
return "", nil
}
raw, err := json.Marshal(held)
if err != nil {
return "", err
}
file := filepath.Join(root, "held.json")
return file, os.WriteFile(file, raw, 0o644)
}
// heldFlag is how the gate is given the held values: only when there are some, so a judge from before --held still runs.
const heldFlag = `${MESH_CHECK_HELD:+--held "$MESH_CHECK_HELD"}`
// Sibling is one other open pull request of a delivery group (novox/hq ADR 0382): the repository it is in as
// owner/repo, its number, its whole head, the branch it merges into, and Dir, the directory the check finds that
// repository under beside it (an entry of Beside).
@@ -272,6 +305,8 @@ const (
// EnvGroup is a delivery group's other heads, as JSON, for the gate (novox/hq ADR 0239); empty for a
// pull request checked alone.
EnvGroup = "MESH_CHECK_GROUP"
// EnvHeld is the file of the approved held values of the check's pull requests (novox/hq ADR 0502); empty for none.
EnvHeld = "MESH_CHECK_HELD"
// EnvTests is a `go test -run` pattern and EnvStoreOnly is 1 when a branch tried before its pull request
// asks for only those tests, or only the store tests (novox/hq issue 686). Never set for a pull request's
// check. A script that reads neither runs whole.
@@ -357,6 +392,12 @@ func Check(ctx context.Context, run Runner, spec CheckSpec, workspace, registry
ctx, stop := context.WithTimeout(ctx, CheckTimeout)
defer stop()
// **A node with no room takes no check** (novox/hq issue 1834): said as the check's verdict, never a throwaway store
// that dies at once on a full disk.
if err := RoomToWork("/", workspace); err != nil {
say("check", "refused: %v", err)
return CheckVerdict{}, err
}
root := filepath.Join(workspace, "check")
if err := os.RemoveAll(root); err != nil {
return CheckVerdict{}, err
@@ -572,6 +613,15 @@ func Check(ctx context.Context, run Runner, spec CheckSpec, workspace, registry
}
}
// The approved held values of the check's pull requests, for the gate (novox/hq ADR 0502 clause 4).
heldFile, err := writeHeld(root, spec.Held)
if err != nil {
return CheckVerdict{}, err
}
if heldFile != "" {
say("check", "with %d config value(s) held for its pull request(s)", len(spec.Held))
}
// Every clone is made: the credential is gone before anything of the check runs (novox/hq issue 462).
forget()
@@ -733,7 +783,7 @@ func Check(ctx context.Context, run Runner, spec CheckSpec, workspace, registry
EnvTestStore + "=" + storeURL, EnvTestBus + "=nats://" + bus, EnvRepository + "=" + spec.Owner + "/" + spec.Repo,
EnvBeside + "=" + root,
EnvModules + "=" + strings.Join(append(append([]string{}, spec.Modules...), spec.New...), ","),
EnvGroup + "=" + groupFile,
EnvGroup + "=" + groupFile, EnvHeld + "=" + heldFile,
"GOCACHE=" + filepath.Join(workspace, "go-cache"), "GOMODCACHE=" + filepath.Join(workspace, "go-modules")}
env = append(env, changedEnv...)
env = append(env, selection(spec)...)
@@ -1100,7 +1150,7 @@ func gateLayer(ctx context.Context, spec CheckSpec, tree, root, gate, verdictFil
if err := running(inToolchain(tree, append(env, EnvVerdict+"="+verdictFile), "sh", "-c",
`"$MESH_GATE" merge-gate --facts "$MESH_FACTS" --store "$MESH_GATE_POSTGRES" `+
`--repository "$MESH_CHECK_REPOSITORY" --tree . `+gateChangedFlag(env)+` `+
`${MESH_CHECK_GROUP:+--group "$MESH_CHECK_GROUP"} --json > "$MESH_CHECK_VERDICT"`)); err != nil {
`${MESH_CHECK_GROUP:+--group "$MESH_CHECK_GROUP"} `+heldFlag+` --json > "$MESH_CHECK_VERDICT"`)); err != nil {
if ctx.Err() != nil {
return "error", "the check was ended during the merge gate"
}
@@ -1587,7 +1637,9 @@ func BusImage(version string) string {
// throwaway starts a container publishing one port on loopback, and answers where it is reached.
func throwaway(ctx context.Context, run, plain Runner, name, image string, port int, opts, args []string) (string, error) {
invocation := []string{"run", "-d", "--rm", "--name", name, "-p", fmt.Sprintf("127.0.0.1::%d", port)}
// Never with --rm (novox/hq issue 1834): a store that dies at once took its log with it. The check removes what it
// raised, with its volumes, after it has read why one stopped (RemoveContainersOf).
invocation := []string{"run", "-d", "--name", name, "-p", fmt.Sprintf("127.0.0.1::%d", port)}
invocation = append(invocation, opts...)
invocation = append(invocation, image)
invocation = append(invocation, args...)
@@ -1608,17 +1660,45 @@ func throwaway(ctx context.Context, run, plain Runner, name, image string, port
// waitFor runs a readiness command in a container until it answers, for a minute.
func waitFor(ctx context.Context, run Runner, name string, ready []string) error {
for i := 0; i < 60; i++ {
return waitReady(ctx, run, name, ready, 60)
}
// waitReady is waitFor for a number of tries, a second apart. A container that never answers is not left a mystery
// (novox/hq issue 1834): its state and exit code and the last 20 lines of its log are in the error, read before the
// check removes it.
func waitReady(ctx context.Context, run Runner, name string, ready []string, attempts int) error {
for i := 0; i < attempts; i++ {
if _, err := run(ctx, "", "docker", append([]string{"exec", name}, ready...)...); err == nil {
return nil
}
if i == attempts-1 {
break
}
select {
case <-ctx.Done():
return ctx.Err()
case <-time.After(time.Second):
}
}
return fmt.Errorf("%s never became ready", name)
return fmt.Errorf("%s never became ready%s", name, whyItStopped(ctx, run, name))
}
// whyItStopped is ": <state>, exit code N; its last log lines: …" for a container, or "" when docker says nothing.
func whyItStopped(ctx context.Context, run Runner, name string) string {
reading, done := context.WithTimeout(context.WithoutCancel(ctx), 30*time.Second)
defer done()
var said []string
if out, err := run(reading, "", "docker", "inspect", "--format",
"{{.State.Status}}, exit code {{.State.ExitCode}}", name); err == nil && strings.TrimSpace(out) != "" {
said = append(said, strings.TrimSpace(out))
}
if out, err := run(reading, "", "docker", "logs", "--tail", "20", name); err == nil && strings.TrimSpace(out) != "" {
said = append(said, "its last log lines: "+strings.TrimSpace(out))
}
if len(said) == 0 {
return ""
}
return ": " + strings.Join(said, "; ")
}
// dialable waits for an address to take a connection, for a minute.
+33
View File
@@ -0,0 +1,33 @@
package builder
import (
"os"
"path/filepath"
"strings"
"testing"
)
// 1826/AC2: the held values are written whole to held.json, and the gate is given the flag only when there are some.
func TestHeldValuesAreWrittenForTheGateAndNoneWriteNothing(t *testing.T) {
root := t.TempDir()
if file, err := writeHeld(root, nil); err != nil || file != "" {
t.Fatalf("none wrote %q %v", file, err)
}
if _, err := os.Stat(filepath.Join(root, "held.json")); err == nil {
t.Fatal("none wrote a file")
}
file, err := writeHeld(root, []HeldValue{{Pull: "novox/mesh-catalog#434", Node: "ace", Module: "n8n", Key: "media-gid",
Value: []byte("2000"), Approved: "the operator"}})
if err != nil || file != filepath.Join(root, "held.json") {
t.Fatalf("wrote %q %v", file, err)
}
raw, _ := os.ReadFile(file)
for _, want := range []string{`"node":"ace"`, `"module":"n8n"`, `"key":"media-gid"`, `"value":2000`} {
if !strings.Contains(string(raw), want) {
t.Fatalf("%s lacks %s", raw, want)
}
}
if !strings.Contains(heldFlag, `--held "$MESH_CHECK_HELD"`) || !strings.HasPrefix(heldFlag, "${MESH_CHECK_HELD:+") {
t.Fatalf("the flag is %s", heldFlag)
}
}
+216
View File
@@ -0,0 +1,216 @@
package builder
import (
"context"
"errors"
"os"
"path/filepath"
"strings"
"testing"
)
// Issue 1834: a build node's root disk filled and every check there ended "the store never became ready". A node
// frees what each build and check leaves, refuses to take work under a floor of free space, and a store that does
// not come up says why.
func init() {
// The disk the tests run on is none of their business; the tests of the floor say their own.
FreeBytes = func(string) (uint64, error) { return 500 << 30, nil }
}
func withFree(t *testing.T, bytes uint64) {
t.Helper()
was := FreeBytes
FreeBytes = func(string) (uint64, error) { return bytes, nil }
t.Cleanup(func() { FreeBytes = was })
}
// 1834/AC2: given 3 GiB free, when a check or a build is asked, then it refuses at once, saying so, running nothing.
func TestACheckUnderTheFloorOfFreeSpaceRefusesSayingSo(t *testing.T) {
withFree(t, 3<<30)
var ran []string
run := func(_ context.Context, _ string, name string, args ...string) (string, error) {
ran = append(ran, name+" "+strings.Join(args, " "))
return "", nil
}
_, err := Check(context.Background(), run, CheckSpec{ID: "build-1"}, t.TempDir(), "registry.invalid", GitCredential{}, nil)
if err == nil || !strings.Contains(err.Error(), "the build node has 3 GiB free on /, under the floor of 20 GiB") {
t.Fatalf("a check on a full disk said %v", err)
}
if len(ran) != 0 {
t.Errorf("a refused check ran %v", ran)
}
}
func TestABuildUnderTheFloorOfFreeSpaceRefusesSayingSo(t *testing.T) {
withFree(t, 3<<30)
r := &recorded{}
_, err := Build(context.Background(), r.run, r, "https://forge.invalid/x.git", "", "", t.TempDir(), nil, Npmrc{}, GitCredential{}, nil)
if err == nil || !strings.Contains(err.Error(), "under the floor of 20 GiB") {
t.Fatalf("a build on a full disk said %v", err)
}
if len(r.ran) != 0 {
t.Errorf("a refused build ran %v", r.ran)
}
}
// The workspace's filesystem is read too, and a workspace not made yet is read at its nearest folder.
func TestTheWorkspacesFilesystemIsReadToo(t *testing.T) {
was := FreeBytes
t.Cleanup(func() { FreeBytes = was })
missing := filepath.Join(t.TempDir(), "not", "yet")
FreeBytes = func(p string) (uint64, error) {
if p == "/" {
return 100 << 30, nil
}
if _, err := os.Stat(p); err != nil {
t.Errorf("read the free space of %s, which is not there", p)
}
return 1 << 30, nil
}
err := RoomToWork("/", missing)
if err == nil || !strings.Contains(err.Error(), "1 GiB free on "+missing) {
t.Errorf("the workspace's disk, under the floor, said %v", err)
}
FreeBytes = func(string) (uint64, error) { return 0, errors.New("no") }
if RoomToWork("/") == nil {
t.Error("a disk that cannot be read was taken as having room")
}
}
// 1834/AC3: given a store that stops at once, when the check waits for it, then its exit code and last log lines are
// in the error and the container is removed afterwards, not by --rm before anyone could read it.
func TestAStoreThatDiesSaysWhyItDied(t *testing.T) {
var ran []string
run := func(_ context.Context, _ string, name string, args ...string) (string, error) {
line := name + " " + strings.Join(args, " ")
ran = append(ran, line)
switch {
case args[0] == "exec":
return "", errors.New("Error response from daemon: container is not running")
case args[0] == "inspect":
return "exited exit code 1\n", nil
case args[0] == "logs":
return "initdb: error: could not create directory: No space left on device\n", nil
}
return "", nil
}
err := waitReady(context.Background(), run, "build-1-store", []string{"pg_isready"}, 1)
for _, want := range []string{"build-1-store never became ready", "exit code 1", "No space left on device"} {
if err == nil || !strings.Contains(err.Error(), want) {
t.Errorf("the verdict %q lacks %q", err, want)
}
}
logs := ""
for _, l := range ran {
if strings.HasPrefix(l, "docker logs") {
logs = l
}
}
if logs != "docker logs --tail 20 build-1-store" {
t.Errorf("the log was read as %q", logs)
}
}
func TestTheThrowawayStoreIsNotStartedWithRm(t *testing.T) {
var started []string
run := func(_ context.Context, _ string, name string, args ...string) (string, error) {
if args[0] == "run" {
started = args
}
return "127.0.0.1:40000\n", nil
}
if _, err := throwaway(context.Background(), run, run, "build-1-store", "postgres:17-alpine", 5432, nil, nil); err != nil {
t.Fatal(err)
}
for _, a := range started {
if a == "--rm" {
t.Errorf("started with --rm, which erases why it died: %v", started)
}
}
}
// 1834/AC1: given an image built for a check or a walk and published to the registry, when the build ends, then the
// local image tag, the narrowed source and the context clone it left are gone; an image published nowhere (a mesh
// being raised) is kept, since it is the artifact.
// pushedNamed is a publisher that tags the image as <registry>/<repository> to push it, as the registry one does.
type pushedNamed struct{ *recorded }
func (pushedNamed) PushedNames(repository string) []string {
return []string{"registry.invalid/" + repository}
}
func TestABuildRemovesWhatItLeftOnTheNode(t *testing.T) {
r := &recorded{
contents: map[string]string{"modules/m/" + ManifestName: anImageWithAContext, "modules/m/Dockerfile": "FROM scratch\n"},
secondary: map[string]map[string]string{"https://forge.invalid/ctl.git": {"go.mod": "module x\n"}},
}
workspace := t.TempDir()
if _, err := Build(context.Background(), r.run, pushedNamed{r}, "https://forge.invalid/c.git", "modules/m", "", workspace, nil, Npmrc{},
GitCredential{}, nil); err != nil {
t.Fatal(err)
}
// Both names go in one rmi: the layers stay while either is left (the push tagged the image with the registry's).
if !strings.Contains(strings.Join(r.ran, "\n"), "docker rmi m-server:c0ffeec0 registry.invalid/m/server") {
t.Errorf("the image built for it and its pushed name were not removed: %v", r.ran)
}
left, _ := filepath.Glob(filepath.Join(workspace, "*"))
if len(left) != 0 {
t.Errorf("the build left %v in its workspace", left)
}
}
const anImageWithAContext = `{"module":"m","version":"1",
"build":{"artifacts":[{"name":"server","kind":"image","from":"Dockerfile",
"context":{"repository":"https://forge.invalid/ctl.git","ref":"main"}}]}}`
func TestAnImageKeptLocallyIsNotRemoved(t *testing.T) {
r := &recorded{contents: map[string]string{"modules/m/" + ManifestName: anImageWithAContext, "modules/m/Dockerfile": "FROM scratch\n"}}
local := Local{Run: func(ctx context.Context, dir, name string, args ...string) (string, error) {
if args[0] == "image" {
return "sha256:" + strings.Repeat("a", 64) + "\n", nil
}
return r.run(ctx, dir, name, args...)
}}
if _, err := Build(context.Background(), local.Run, local, "https://forge.invalid/c.git", "modules/m", "", t.TempDir(), nil, Npmrc{},
GitCredential{}, nil); err != nil {
t.Fatal(err)
}
if strings.Contains(strings.Join(r.ran, "\n"), "docker rmi") {
t.Errorf("the artifact of a mesh being raised was removed: %v", r.ran)
}
}
// The held image the comparison pulls to read is let go of once read.
func TestTheHeldImageAComparisonPullsIsRemovedAfterReading(t *testing.T) {
var ran []string
run := func(_ context.Context, _ string, name string, args ...string) (string, error) {
ran = append(ran, name+" "+strings.Join(args, " "))
if args[0] == "create" {
return "abcdef0123456789\n", nil
}
if args[0] == "image" {
return "", errors.New("stop here")
}
return "", nil
}
_, _ = imageContent(context.Background(), run, t.TempDir(), "reg.invalid/m/server@sha256:"+strings.Repeat("a", 64))
if got := ran[len(ran)-1]; !strings.HasPrefix(got, "docker rmi reg.invalid/m/server@sha256:") {
t.Errorf("the last command was %q, not the removal of the pulled image: %v", got, ran)
}
}
func TestAnImageThatCannotBeRemovedIsNotSaidRemoved(t *testing.T) {
var said []string
run := func(context.Context, string, string, ...string) (string, error) {
return "", errors.New("image is being used")
}
releaseImage(context.Background(), run, []string{"a:1"}, func(step, format string, args ...any) {
said = append(said, format)
})
for _, s := range said {
if strings.Contains(s, "is removed") {
t.Errorf("said removed: %q", s)
}
}
}
+46
View File
@@ -0,0 +1,46 @@
package builder
import (
"fmt"
"os"
"path/filepath"
"syscall"
)
// FreeSpaceFloor is the free space a build node must have, on / and on its workspace's filesystem, before it takes
// a build or a check (novox/hq issue 1834): under it, a throwaway store cannot initialise its data directory and
// dies at once, and the check ended "never became ready" instead of saying the disk was full.
const FreeSpaceFloor = 20 << 30
// FreeBytes reads the bytes an unprivileged writer can still use on the filesystem holding path. A variable so a
// test answers for a disk it does not have.
var FreeBytes = func(path string) (uint64, error) {
var s syscall.Statfs_t
if err := syscall.Statfs(path, &s); err != nil {
return 0, err
}
return s.Bavail * uint64(s.Bsize), nil
}
// RoomToWork refuses, saying how much is free, when any of the paths' filesystems is under FreeSpaceFloor. A path
// that does not exist yet (a workspace not made) is read at its nearest folder that does.
func RoomToWork(paths ...string) error {
for _, p := range paths {
at := p
for {
if _, err := os.Stat(at); err == nil || filepath.Dir(at) == at {
break
}
at = filepath.Dir(at)
}
free, err := FreeBytes(at)
if err != nil {
return fmt.Errorf("the free space on %s cannot be read: %w", p, err)
}
if free < FreeSpaceFloor {
return fmt.Errorf("the build node has %d GiB free on %s, under the floor of %d GiB",
free>>30, p, FreeSpaceFloor>>30)
}
}
return nil
}
+8 -8
View File
@@ -172,11 +172,11 @@ func TestANarrowedTreeHoldsTheBuildSourceAndNothingElse(t *testing.T) {
}
}
// This repository's own programs: the route proxy's build source is not the controller's, and neither
// holds the other's command.
// This repository's own programs: the route proxy lives in its module's own Go module, so its build source is
// not the controller's, and neither holds the other's command.
func TestThisRepositorysProgramsHaveBuildSourcesOfTheirOwn(t *testing.T) {
root := filepath.Join("..", "..")
proxy, err := GoBuildSource(root, "examples/route-proxy")
proxy, err := GoBuildSource(filepath.Join(root, "modules", "route-proxy"), "cmd/route-proxy")
if err != nil {
t.Fatal(err)
}
@@ -194,15 +194,15 @@ func TestThisRepositorysProgramsHaveBuildSourcesOfTheirOwn(t *testing.T) {
file string
held bool
}{
{proxy, "the route proxy", "examples/route-proxy/main.go", true},
{proxy, "the route proxy", "internal/broker/broker.go", true},
{proxy, "the route proxy", "cmd/mesh-controller/main.go", false},
{proxy, "the route proxy", "internal/conditions/condition.go", false},
{proxy, "the route proxy", "cmd/route-proxy/main.go", true},
{proxy, "the route proxy", "cmd/route-proxy/bus.go", true},
{proxy, "the route proxy", "go.mod", true},
{proxy, "the route proxy", "cmd/route-provider/main.go", false},
{proxy, "the route proxy", "README.md", false},
{controller, "the controller", "cmd/mesh-controller/main.go", true},
{controller, "the controller", "internal/conditions/condition.go", true},
{controller, "the controller", "internal/inventory/migrations/0001-nodes.sql", true},
{controller, "the controller", "examples/route-proxy/main.go", false},
{controller, "the controller", "modules/route-proxy/cmd/route-proxy/main.go", false},
{controller, "the controller", "cmd/mesh-builder/main.go", false},
{controller, "the controller", "README.md", false},
{builder, "the build seat's program", "cmd/mesh-builder/main.go", true},
+45 -3
View File
@@ -3,6 +3,7 @@ package builder
import (
"context"
"os/exec"
"regexp"
"strings"
"syscall"
"time"
@@ -41,6 +42,11 @@ func inItsOwnGroup(cmd *exec.Cmd) {
cmd.WaitDelay = KillWait
}
// CgroupParent is the slice every container a build or check starts joins (novox/hq issue 1798), so the node's own
// weights and cap bind it: the build-agent module declares the slice with the machine's CPU weight, IO weight and
// memory limits, and names it in MESH_BUILD_CGROUP_PARENT. Empty, a container joins the runtime's default.
var CgroupParent string
// Labelled is a Runner that marks every container a build starts with the build's id, so a kill
// finds what outlived the docker client (novox/hq ADR 0219). Applied at the one place every command
// passes rather than at each `docker run` the builder composes: a run added later is labelled too.
@@ -55,8 +61,11 @@ func LabelledArgs(name string, args []string, id string) []string {
if name != "docker" || len(args) == 0 || args[0] != "run" || id == "" {
return args
}
out := make([]string, 0, len(args)+2)
out := make([]string, 0, len(args)+4)
out = append(out, "run", "--label", BuildLabel+"="+id)
if CgroupParent != "" {
out = append(out, "--cgroup-parent", CgroupParent)
}
return append(out, args[1:]...)
}
@@ -71,12 +80,45 @@ func RemoveContainersOf(ctx context.Context, run Runner, id string) (int, error)
if len(ids) == 0 {
return 0, nil
}
if _, err := run(ctx, "", "docker", append([]string{"rm", "-f"}, ids...)...); err != nil {
if err := removeWithVolumes(ctx, run, ids); err != nil {
return 0, err
}
return len(ids), nil
}
// MountedVolumes is the template `docker inspect --format` reads the names of a container's volumes with.
const MountedVolumes = `{{range .Mounts}}{{if eq .Type "volume"}}{{println .Name}}{{end}}{{end}}`
// anonymousVolume is the name docker gives a volume it makes for an image's VOLUME line: 64 hex digits.
var anonymousVolume = regexp.MustCompile(`^[0-9a-f]{64}$`)
// removeWithVolumes removes containers together with their anonymous volumes (novox/hq issue 1772). A throwaway
// store declares its data directory a VOLUME, docker makes an anonymous volume for it, and `docker rm -f`
// without -v leaves that volume for ever: 1,629 of them, 107 GB, on one build node. So the volumes the
// containers mount are read first, the containers are removed with -v, and an anonymous volume still there
// afterwards is removed by name. Only a volume one of these containers mounted, and only one with an anonymous
// volume's name, is touched: a named volume — an operator's own data — never is. A volume already gone is
// no failure.
func removeWithVolumes(ctx context.Context, run Runner, ids []string) error {
var volumes []string
if out, err := run(ctx, "", "docker", append([]string{"inspect", "--format", MountedVolumes}, ids...)...); err == nil {
seen := map[string]bool{}
for _, name := range strings.Fields(out) {
if anonymousVolume.MatchString(name) && !seen[name] {
seen[name] = true
volumes = append(volumes, name)
}
}
}
if _, err := run(ctx, "", "docker", append([]string{"rm", "-f", "-v"}, ids...)...); err != nil {
return err
}
if len(volumes) > 0 {
_, _ = run(ctx, "", "docker", append([]string{"volume", "rm"}, volumes...)...)
}
return nil
}
// A holder that stops mid-build leaves its containers running (novox/hq issue 306).
//
// A build's containers are removed by the build itself, by a kill, or by the next delivery of the same
@@ -106,7 +148,7 @@ func RemoveLeftBehind(ctx context.Context, run Runner) (int, error) {
if len(ids) == 0 {
return 0, nil
}
if _, err := run(ctx, "", "docker", append([]string{"rm", "-f"}, ids...)...); err != nil {
if err := removeWithVolumes(ctx, run, ids); err != nil {
return 0, err
}
return len(ids), nil
+76 -2
View File
@@ -2,6 +2,7 @@ package builder
import (
"context"
"errors"
"reflect"
"strings"
"testing"
@@ -49,7 +50,8 @@ func TestAKillRemovesTheContainersLabelledWithTheBuild(t *testing.T) {
if err != nil || n != 2 {
t.Fatalf("%d %v", n, err)
}
if want := []string{"docker ps -aq --filter label=mesh.build=build-3", "docker rm -f c1 c2"}; !reflect.DeepEqual(ran, want) {
if want := []string{"docker ps -aq --filter label=mesh.build=build-3", "docker inspect --format " + MountedVolumes + " c1 c2",
"docker rm -f -v c1 c2"}; !reflect.DeepEqual(ran, want) {
t.Errorf("ran %v", ran)
}
}
@@ -73,7 +75,7 @@ func TestAStartingHolderRemovesWhatEarlierHoldersLeftAndNothingElse(t *testing.T
if n != 3 {
t.Errorf("removed %d", n)
}
if len(ran) != 2 || !reflect.DeepEqual(ran[1], []string{"docker", "rm", "-f", "c1", "c2", "c7"}) {
if len(ran) != 3 || !reflect.DeepEqual(ran[2], []string{"docker", "rm", "-f", "-v", "c1", "c2", "c7"}) {
t.Fatalf("ran %v", ran)
}
if !strings.Contains(strings.Join(ran[0], " "), "--filter label=mesh.build") {
@@ -90,3 +92,75 @@ func TestAStartingHolderRemovesWhatEarlierHoldersLeftAndNothingElse(t *testing.T
t.Errorf("with nothing of the seat's: %d %v %v", n, err, ran)
}
}
// **Issue 1772**: a throwaway store declares its data directory a VOLUME, so docker makes an anonymous volume for
// it, and `docker rm -f` without -v leaves that volume behind (1,629 of them, 107 GB, on one build node). Every
// container a build or check removes is removed with its anonymous volumes (-v), and any volume the container
// mounted that is still there afterwards is removed by name — those, and only those, never a named volume.
func TestARemovedContainerTakesItsAnonymousVolumesWithIt(t *testing.T) {
anon := strings.Repeat("a", 64)
other := strings.Repeat("b", 64)
var ran []string
run := func(_ context.Context, _ string, name string, args ...string) (string, error) {
ran = append(ran, name+" "+strings.Join(args, " "))
switch args[0] {
case "ps":
return "c1\n", nil
case "inspect":
return anon + "\n" + "compose_data\n" + other + "\n", nil
}
return "", nil
}
if n, err := RemoveContainersOf(context.Background(), run, "build-4"); err != nil || n != 1 {
t.Fatalf("%d %v", n, err)
}
last := ran[len(ran)-1]
if want := "docker volume rm " + anon + " " + other; last != want {
t.Errorf("last command %q, want %q", last, want)
}
if !strings.Contains(strings.Join(ran, "\n"), "docker rm -f -v c1") {
t.Errorf("ran %v", ran)
}
if strings.Contains(last, "compose_data") {
t.Errorf("a named volume was removed: %q", last)
}
}
// A volume already gone when the sweep looks (rm -v took it) is no failure.
func TestAVolumeAlreadyGoneIsNoFailure(t *testing.T) {
run := func(_ context.Context, _ string, name string, args ...string) (string, error) {
switch args[0] {
case "ps":
return "c1\n", nil
case "inspect":
return strings.Repeat("c", 64) + "\n", nil
case "volume":
return "", errors.New("no such volume")
}
return "", nil
}
if n, err := RemoveContainersOf(context.Background(), run, "build-5"); err != nil || n != 1 {
t.Fatalf("%d %v", n, err)
}
}
// A node's build slice limits everything a build or check starts (novox/hq issue 1798): with a parent
// set, every `docker run` joins it, so the node's CPU and IO weights and memory cap bind the container;
// with none set the arguments are as they always were.
func TestEveryContainerJoinsTheNodesBuildSlice(t *testing.T) {
was := CgroupParent
defer func() { CgroupParent = was }()
CgroupParent = "meshbuild.slice"
got := LabelledArgs("docker", []string{"run", "--rm", "img", "sh"}, "build-1")
want := []string{"run", "--label", "mesh.build=build-1", "--cgroup-parent", "meshbuild.slice", "--rm", "img", "sh"}
if !reflect.DeepEqual(got, want) {
t.Errorf("1798/AC1: docker run became %v", got)
}
if got := LabelledArgs("docker", []string{"build", "."}, "build-1"); !reflect.DeepEqual(got, []string{"build", "."}) {
t.Errorf("a docker build became %v", got)
}
CgroupParent = ""
if got := LabelledArgs("docker", []string{"run", "img"}, "build-1"); !reflect.DeepEqual(got, []string{"run", "--label", "mesh.build=build-1", "img"}) {
t.Errorf("1798/AC1: with no slice, docker run became %v", got)
}
}
+29 -4
View File
@@ -43,6 +43,11 @@ type Registry struct {
HTTP *http.Client
}
// PushedNames are the names PublishImage leaves the image under on the node.
func (r Registry) PushedNames(repository string) []string {
return []string{r.Address + "/" + repository}
}
// PublishImage pushes a locally built image and returns a reference pinned by digest.
//
// The digest is read back from the registry's own answer rather than computed here. What matters
@@ -59,15 +64,35 @@ func (r Registry) PublishImage(ctx context.Context, localTag, repository string)
}); err != nil {
return "", err
}
out, err := r.Run(ctx, "", "docker", "inspect", "--format", "{{index .RepoDigests 0}}", remote)
// Every digest the daemon lists for the image, one per line: the image may have been pushed under other names before
// (a check's scratch copy, another registry), and the first of them is not the one this push made (novox/hq issue
// 1742: a build recorded a digest the store never held).
out, err := r.Run(ctx, "", "docker", "inspect", "--format", "{{range .RepoDigests}}{{println .}}{{end}}", remote)
if err != nil {
return "", fmt.Errorf("pushed %s and cannot read back what it was pinned as: %w", remote, err)
}
pinned := strings.TrimSpace(out)
if !strings.Contains(pinned, "@sha256:") {
first, pinned := "", ""
for _, line := range strings.Split(out, "\n") {
line = strings.TrimSpace(line)
if line == "" {
continue
}
if first == "" {
first = line
}
if strings.HasPrefix(line, remote+"@sha256:") {
pinned = line
break
}
}
if pinned == "" {
if strings.Contains(first, "@sha256:") {
return "", fmt.Errorf("pushed %s, but the daemon lists no digest for that name (only %q): the digest of another "+
"repository is not what this registry serves", remote, first)
}
// A tag is not a pin. It can be made to point at something else, and this file is applied
// on machines with no mesh to ask about anything (novox/hq ADR 0006).
return "", fmt.Errorf("%s came back as %q, which is not pinned by digest", remote, pinned)
return "", fmt.Errorf("%s came back as %q, which is not pinned by digest", remote, first)
}
return pinned, nil
}
+29
View File
@@ -252,3 +252,32 @@ func TestAPushedImageComesBackPinned(t *testing.T) {
t.Fatalf("it did not tag and then push: %v", ran)
}
}
// 1742: an image pushed under other names before lists their digests too, and the digest recorded is the one of the name
// this push made — never the first listed, which may be another repository's.
func TestThePinnedDigestIsTheOneOfTheNamePushed(t *testing.T) {
want := "registry.invalid/shell/server@sha256:" + strings.Repeat("b", 64)
earlier := "registry.invalid/scratch/shell/server@sha256:" + strings.Repeat("a", 64)
r := Registry{Address: "registry.invalid", Run: func(
_ context.Context, _, name string, args ...string) (string, error) {
if name == "docker" && len(args) > 0 && args[0] == "inspect" {
return earlier + "\n" + want + "\n", nil
}
return "", nil
}}
got, err := r.PublishImage(context.Background(), "local", "shell/server")
if err != nil || got != want {
t.Fatalf("pinned %q (%v); want %q", got, err, want)
}
// Only another repository's digest is listed: the push is not taken for pinned.
r.Run = func(_ context.Context, _, name string, args ...string) (string, error) {
if name == "docker" && len(args) > 0 && args[0] == "inspect" {
return earlier + "\n", nil
}
return "", nil
}
if _, err := r.PublishImage(context.Background(), "local", "shell/server"); err == nil ||
!strings.Contains(err.Error(), "no digest for that name") {
t.Fatalf("a digest of another repository was taken for the push's own: %v", err)
}
}
+7
View File
@@ -253,6 +253,13 @@ func (s scratchPublisher) PublishImage(ctx context.Context, localTag, repository
return s.to.PublishImage(ctx, localTag, ScratchPrefix+repository)
}
func (s scratchPublisher) PushedNames(repository string) []string {
if p, ok := s.to.(PushedNames); ok {
return p.PushedNames(ScratchPrefix + repository)
}
return nil
}
func (s scratchPublisher) PublishArchive(ctx context.Context, repository string, body []byte, digest string) (string, error) {
return s.to.PublishArchive(ctx, ScratchPrefix+repository, body, digest)
}
+4 -1
View File
@@ -88,6 +88,9 @@ func imageContent(ctx context.Context, run Runner, dir, ref string) (string, err
if _, err := run(ctx, dir, "docker", "pull", "--quiet", ref); err != nil {
return "", fmt.Errorf("pulling %s: %w", ref, err)
}
// The image is on the node only for the reading (novox/hq issue 1834); one a container still runs is refused by
// docker and stays.
defer func() { _, _ = run(context.WithoutCancel(ctx), dir, "docker", "rmi", ref) }()
config, err := run(ctx, dir, "docker", "image", "inspect", "--format",
"{{.Os}}/{{.Architecture}}/{{.Variant}} {{json .Config}}", ref)
if err != nil {
@@ -105,7 +108,7 @@ func imageContent(ctx context.Context, run Runner, dir, ref string) (string, err
if id == "" {
return "", errors.New("the container engine named no container for " + ref)
}
defer func() { _, _ = run(context.WithoutCancel(ctx), dir, "docker", "rm", "--force", id) }()
defer func() { _, _ = run(context.WithoutCancel(ctx), dir, "docker", "rm", "--force", "--volumes", id) }()
if err := os.MkdirAll(dir, 0o700); err != nil {
return "", err
}
+3 -3
View File
@@ -185,7 +185,7 @@ func TestABuildsBuildSourceComesBackOverWhatItRead(t *testing.T) {
said.Read = []ReadRepository{{Repository: "novox/mesh-controller", Ref: "main"}, {Repository: "novox/other"}}
said.Sources = []BuildSource{
{Paths: []string{"modules/route-proxy/Dockerfile", "modules/route-proxy/module.json"}},
{Repository: "novox/mesh-controller", Ref: "main", Paths: []string{"examples/route-proxy/", "go.mod"}},
{Repository: "novox/mesh-controller", Ref: "main", Paths: []string{"modules/route-proxy/cmd/route-proxy/", "go.mod"}},
{Repository: "novox/unread", Paths: []string{"x"}},
}
unsaid := aBuild("unsaid", "build-agent", "")
@@ -200,7 +200,7 @@ func TestABuildsBuildSourceComesBackOverWhatItRead(t *testing.T) {
t.Fatal(err)
}
want := []ReadRepository{
{Repository: "novox/mesh-controller", Ref: "main", Paths: []string{"examples/route-proxy/", "go.mod"}},
{Repository: "novox/mesh-controller", Ref: "main", Paths: []string{"modules/route-proxy/cmd/route-proxy/", "go.mod"}},
{Repository: "novox/other"},
{Paths: []string{"modules/route-proxy/Dockerfile", "modules/route-proxy/module.json"}, Own: true},
}
@@ -238,7 +238,7 @@ func TestAFailedNewerBuildLeavesTheBuildSourceStale(t *testing.T) {
worked.Asked = time.Now().Add(-time.Hour)
worked.Read = []ReadRepository{{Repository: "novox/mesh-controller", Ref: "main"}}
worked.Sources = []BuildSource{{Paths: []string{"modules/route-proxy/module.json"}},
{Repository: "novox/mesh-controller", Ref: "main", Paths: []string{"examples/route-proxy/"}}}
{Repository: "novox/mesh-controller", Ref: "main", Paths: []string{"modules/route-proxy/cmd/route-proxy/"}}}
failed := aBuild("failed", "route-proxy", "compile error")
failed.Asked = time.Now()
for _, b := range []Build{worked, failed} {
+12
View File
@@ -6,6 +6,8 @@ import (
"encoding/json"
"fmt"
"time"
"git.novox.be/novox/mesh-sdk/go/catalogue"
)
// The values of the controller's record mirrors, built from the store (novox/hq ADR 0289, design 54 §1,
@@ -179,3 +181,13 @@ func (i *Inventory) PredefinedMirror(ctx context.Context) (map[string][]byte, er
}
return map[string][]byte{PredefinedMirrorKey: value}, nil
}
// DeclaredOf is a manifest's config declarations as the mirror holds them (DeclaredMirrored), so a pull request's
// declarations are told to mesh-config in the one shape it reads the trunk's in (novox/hq ADR 0502 clause 2).
func DeclaredOf(settings map[string]catalogue.SettingDeclaration) (map[string]DeclaredMirrored, error) {
raw, err := json.Marshal(settings)
if err != nil {
return nil, err
}
return declaredMirrored(raw)
}
+28
View File
@@ -0,0 +1,28 @@
package link
import (
"context"
"testing"
"github.com/nats-io/nats.go"
)
// 1768/AC1: a request dropped as a duplicate of one another builder runs marks nothing in hand: the claim is not made,
// so no mark is left behind to fail every later claim.
func TestADuplicateDropIsJudgedBeforeTheClaimSoLeavesNoMark(t *testing.T) {
claims := 0
m := &natsMachine{
opts: MachineOptions{Claim: func() bool { claims++; return true }},
duplicate: func(context.Context, BuildRequest) string { return "ace is running it" },
}
if m.admit(context.Background(), &nats.Msg{}, BuildRequest{ID: "build-1"}) {
t.Fatal("a duplicate was admitted")
}
if claims != 0 {
t.Fatalf("a dropped duplicate was claimed %d times, which leaves the in-hand mark behind", claims)
}
m.duplicate = func(context.Context, BuildRequest) string { return "" }
if !m.admit(context.Background(), &nats.Msg{}, BuildRequest{ID: "build-2"}) || claims != 1 {
t.Fatalf("a request that is no duplicate was not claimed once: claims=%d", claims)
}
}
+23
View File
@@ -218,6 +218,25 @@ type CheckRequest struct {
Try bool `json:"try,omitempty"`
Tests string `json:"tests,omitempty"`
StoreOnly bool `json:"store-only,omitempty"`
// HeldConfig are the approved held values of the pull requests this check is of (novox/hq ADR 0502 clause 4): each
// a config value the operator approved for a key an open pull request introduces. The merge gate lays them over
// their module's values on their node in the composition with the change only, never in the mesh as it is. Empty
// from a controller from before it, or when mesh-config held none or did not answer.
HeldConfig []HeldConfig `json:"held_config,omitempty"`
}
// HeldConfig is one approved held value (novox/hq ADR 0502), as mesh-config's `held` action answers it.
type HeldConfig struct {
Pull string `json:"pull"`
Head string `json:"head,omitempty"`
Node string `json:"node"`
Module string `json:"module"`
Key string `json:"key"`
Value json.RawMessage `json:"value"`
// Approved says who approved it, in words, as mesh-config states it; By and At are who held it and when.
Approved string `json:"approved,omitempty"`
By string `json:"by,omitempty"`
At string `json:"at,omitempty"`
}
// StepTried is the step a build seat says a try's outcome under, in the build's log: one line, a Tried as JSON
@@ -292,6 +311,10 @@ type GroupMember struct {
// Base is the branch it merges into: the build seat merges the head into that trunk's head before the
// group is composed (novox/hq ADR 0323 §6). A build seat from before it composes the head alone.
Base string `json:"base,omitempty"`
// Trunk is the base branch's head the forge announced with the head (novox/hq issue 1106): the trunk commit the
// composed verdict names for this member when the build seat named none, so a member whose trunk did not move is
// never merged-refused for a verdict that names no trunk.
Trunk string `json:"trunk,omitempty"`
}
// Who judges a merge check's gate.
+30 -12
View File
@@ -163,6 +163,8 @@ type natsMachine struct {
// build waits, then the checks of pull requests marked ready, then every other check (novox/hq ADR 0406 §3).
subs []*takeQueue
opts MachineOptions
// duplicate stands in for runningElsewhere in tests; nil is the real one.
duplicate func(context.Context, BuildRequest) string
// gate is what the loop asks of the bus to hold checks while a delivery builds; nil is the bus's (a test sets a fake).
gate checkGate
// said is the faults of the hold already said, letKey whether this holder created the let-through key for the check it
@@ -405,18 +407,7 @@ func (m *natsMachine) handOver(ctx context.Context, msg *nats.Msg, q *takeQueue,
return
}
}
// **Drained in the moment it was fetched** (novox/hq ADR 0335): handed back at once, as a pause
// landing then is.
if m.opts.Claim != nil && !m.opts.Claim() {
_ = msg.Nak()
return
}
// **A duplicate of a check another builder is running is dropped** (issue 1649): an ask redelivered to this machine
// because its first holder did not say, in time, that it had it. The running key names that builder and this same
// ask; running it a second time would spend a second place on one check.
if why := m.runningElsewhere(ctx, request); why != "" {
fmt.Fprintf(os.Stderr, "%s is handed out again while another builder runs it and is dropped: %s\n", request.ID, why)
_ = msg.Ack()
if !m.admit(ctx, msg, request) {
return
}
// A build outlives the acknowledgement window many times over; said while it runs,
@@ -460,6 +451,33 @@ func (m *natsMachine) quiesce(keep *takeQueue) {
}
}
// admit decides whether a fetched request is run here, and answers the message when it is not.
//
// **The duplicate is judged before the request is claimed** (issue 1768): a claim marks the build in hand, and a request
// dropped after it left the mark behind, so every later claim failed and every ask was handed back at once until all
// had spent their deliveries. A request dropped here has marked nothing.
//
// **Drained in the moment it was fetched** (novox/hq ADR 0335): handed back at once, as a pause landing then is.
func (m *natsMachine) admit(ctx context.Context, msg *nats.Msg, request BuildRequest) bool {
// **A duplicate of a check another builder is running is dropped** (issue 1649): an ask redelivered to this machine
// because its first holder did not say, in time, that it had it. The running key names that builder and this same
// ask; running it a second time would spend a second place on one check.
elsewhere := m.runningElsewhere
if m.duplicate != nil {
elsewhere = m.duplicate
}
if why := elsewhere(ctx, request); why != "" {
fmt.Fprintf(os.Stderr, "%s is handed out again while another builder runs it and is dropped: %s\n", request.ID, why)
_ = msg.Ack()
return false
}
if m.opts.Claim != nil && !m.opts.Claim() {
_ = msg.Nak()
return false
}
return true
}
// runningElsewhere says why an ask need not run here, or nothing: the bucket of the checks running says another builder
// runs this very ask (its id, its head and its trunk commit). A key that cannot be read runs the check.
func (m *natsMachine) runningElsewhere(ctx context.Context, r BuildRequest) string {
+92
View File
@@ -0,0 +1,92 @@
package ownmanifest
import (
"encoding/json"
"os"
"path/filepath"
"strings"
"testing"
)
// imagesOutOfPlace lists the image artifacts among manifests (module name to its manifest's text) whose Dockerfile
// is not a file inside that module's own directory, root/<module> (novox/hq ADR 0487 clause 13, which replaced the
// refusal of the image kind of ADR 0480 clause 11). The image builder builds a module from its directory alone.
// It reads the manifest as plain JSON, since the catalogue parser refuses some of these itself. It mirrors mesh-catalog's checks/imageplace: an empty from, an absolute one, one leaving the directory with ..,
// or a name that is no file there, is refused.
func imagesOutOfPlace(t *testing.T, root string, manifests map[string][]byte) []string {
t.Helper()
var out []string
for module, raw := range manifests {
var m struct {
Build struct {
Artifacts []struct{ Name, Kind, From string } `json:"artifacts"`
} `json:"build"`
}
if err := json.Unmarshal(raw, &m); err != nil {
t.Fatalf("%s does not parse: %v", module, err)
}
for _, a := range m.Build.Artifacts {
if a.Kind != "image" {
continue
}
clean := filepath.Clean(a.From)
bad := a.From == "" || filepath.IsAbs(clean) || clean == ".." || strings.HasPrefix(clean, ".."+string(filepath.Separator))
if !bad {
info, err := os.Stat(filepath.Join(root, module, clean))
bad = err != nil || info.IsDir()
}
if bad {
out = append(out, module+" ("+a.Name+" from "+a.From+")")
}
}
}
return out
}
// Every image a module of this repository declares is built from a Dockerfile in the module's own directory
// (novox/hq ADR 0487 clause 13). The catalogue's own modules are held to the same by checks/imageplace in mesh-catalog.
func TestEveryImageArtifactBuildsFromADockerfileInItsOwnModuleDirectory(t *testing.T) {
root := filepath.Join("..", "..", "modules")
paths, err := filepath.Glob(filepath.Join(root, "*", "module.json"))
if err != nil || len(paths) == 0 {
t.Fatalf("no module manifests found in this repository's modules: %v", err)
}
manifests := map[string][]byte{}
for _, p := range paths {
raw, err := os.ReadFile(p)
if err != nil {
t.Fatal(err)
}
manifests[filepath.Base(filepath.Dir(p))] = raw
}
if got := imagesOutOfPlace(t, root, manifests); len(got) != 0 {
t.Errorf("these image artifacts do not build from a Dockerfile in their own module's directory "+
"(ADR 0487 clause 13): %v", got)
}
}
// The check fails on each bad case and passes on a good one, so it holds something.
func TestAnImageWhoseDockerfileIsOutOfPlaceIsFoundAndOneInPlaceIsNot(t *testing.T) {
root := t.TempDir()
if err := os.MkdirAll(filepath.Join(root, "x", "sub"), 0o755); err != nil {
t.Fatal(err)
}
for _, f := range []string{"x/Dockerfile", "x/sub/Dockerfile", "outside"} {
if err := os.WriteFile(filepath.Join(root, f), nil, 0o644); err != nil {
t.Fatal(err)
}
}
manifest := func(from string) map[string][]byte {
return map[string][]byte{"x": []byte(`{"module":"x","version":"1","build":{"artifacts":[{"name":"server","kind":"image","from":"` + from + `"}]}}`)}
}
for _, from := range []string{"Dockerfile", "sub/Dockerfile", "./Dockerfile"} {
if got := imagesOutOfPlace(t, root, manifest(from)); len(got) != 0 {
t.Errorf("from %q: found %v, want none", from, got)
}
}
for _, from := range []string{"", "/etc/Dockerfile", "../outside", "..", "sub/../../outside", "Missing", "sub"} {
if got := imagesOutOfPlace(t, root, manifest(from)); len(got) != 1 {
t.Errorf("from %q: found %v, want it refused", from, got)
}
}
}
+59 -2
View File
@@ -11,7 +11,7 @@ import (
//
// route-proxy keeps each ACME authority's account and certificates in a directory named after a
// digest of the directory URL and of the root bundle its `trust` container copies in
// (examples/route-proxy, forThisAuthority). Until ADR 0226 the URL was rendered from a binding to
// (modules/route-proxy/cmd/route-proxy, forThisAuthority). Until ADR 0226 the URL was rendered from a binding to
// `public-acme`'s `acme-ca`, spelled with the port. A URL spelled any other way — even the same
// authority without `:443` — or a root bundle from another image is a new authority to the proxy:
// a new account, and every routed name ordered again, on two machines at once, against Let's
@@ -36,8 +36,11 @@ func TestRouteProxyKeepsItsPublicAccountDirectory(t *testing.T) {
}
// As a build would leave it: each artifact a container names resolved to an image. Which image
// does not matter to the file checked here.
// does not matter to the file checked here. An archive resource carries no image: it unpacks a bundle.
for _, res := range m.Resources {
if res["type"] == "archive" {
continue
}
if artifact, ok := res["artifact"].(string); ok {
delete(res, "artifact")
res["image"] = "registry.invalid/route-proxy/" + artifact +
@@ -86,3 +89,57 @@ func TestRouteProxyKeepsItsPublicAccountDirectory(t *testing.T) {
"plan for the account (ADR 0226)", trust, trustImage)
}
}
// The proxy's program is mounted into the pinned trust image, not baked into an image (novox/hq ADR 0480 clauses 1,
// 2 and 4): a Go bundle of the module's own Go module stamped for alpine, unpacked by an archive resource, mounted
// read-only at /opt/mesh, called through args, and named under restart-on so a new program never leaves the old
// one running. The container's name, network and logging are what the jail and the routes rely on.
func TestRouteProxyRunsItsMountedProgramOnTheTrustImage(t *testing.T) {
m := catalogueManifest(t, "route-proxy")
var bundle *Artifact
for i, a := range m.Build.Artifacts {
if a.Name == "server" {
bundle = &m.Build.Artifacts[i]
}
if a.Kind == "image" {
t.Errorf("artifact %s is an image built from a Dockerfile (ADR 0452 §1)", a.Name)
}
}
if bundle == nil || bundle.Kind != "bundle" || bundle.Language != "go" || bundle.System != "alpine" ||
bundle.From != "cmd/route-proxy" || bundle.Binary != "route-proxy" {
t.Fatalf("the server artifact is not a Go bundle of cmd/route-proxy stamped for alpine: %+v", bundle)
}
var archive, server map[string]any
for _, r := range m.Resources {
switch {
case r["type"] == "archive" && r["artifact"] == "server":
archive = r
case r["id"] == "server":
server = r
}
}
if archive == nil || server == nil {
t.Fatalf("archive %v or server container %v is missing", archive, server)
}
if server["artifact"] != "trust" || server["name"] != "route-proxy" || server["network"] != "host" || server["logging"] != "journald" {
t.Errorf("the server container moved: %v", server)
}
mount := archive["path"].(string) + ":/opt/mesh:ro"
mounted := false
for _, v := range server["volumes"].([]any) {
mounted = mounted || v == mount
}
if !mounted {
t.Errorf("the archive is not mounted read-only at /opt/mesh (%s): %v", mount, server["volumes"])
}
if args, _ := server["args"].([]any); len(args) != 1 || args[0] != "/opt/mesh/route-proxy" {
t.Errorf("args are %v, want the mounted program", server["args"])
}
restarts := false
for _, v := range server["restart-on"].([]any) {
restarts = restarts || v == archive["id"]
}
if !restarts {
t.Errorf("restart-on %v does not name the archive %v", server["restart-on"], archive["id"])
}
}
+52 -12
View File
@@ -19,18 +19,10 @@ import (
// bus upgrade moves the catalogue's pin; this then fails in the merge check until the tests' pin moves
// with it, so the controller is never tested against a bus the mesh no longer runs, or not yet.
func TestTheBusTheTestsRunIsTheBusTheMeshRuns(t *testing.T) {
dockerfile := filepath.Join(beside.Catalogue(t), "nats", "Dockerfile")
raw, err := os.ReadFile(dockerfile)
if err != nil {
t.Fatal(err)
}
pinned := regexp.MustCompile(`upstream: nats ([0-9.]+)-alpine`).FindSubmatch(raw)
if pinned == nil {
t.Fatalf("%s says no release its digest is", dockerfile)
}
if string(pinned[1]) != Version {
t.Errorf("the tests run bus %s and the catalogue's bus image is %s: move go.mod's nats-server pin "+
"(and `go mod vendor`) with the image", Version, pinned[1])
release, from := catalogueBusRelease(t, filepath.Join(beside.Catalogue(t), "nats"))
if release != Version {
t.Errorf("the tests run bus %s and the catalogue's bus image is %s (%s): move go.mod's nats-server pin "+
"(and `go mod vendor`) with the image", Version, release, from)
}
path := os.Getenv("MESH_FACTS")
if path == "" {
@@ -57,3 +49,51 @@ func TestTheBusTheTestsRunIsTheBusTheMeshRuns(t *testing.T) {
}
// A test's bus is its own: what one test makes, the next never sees.
// catalogueBusRelease is the server release the catalogue's nats module runs, and where it was read. Since
// ADR 0497 the server is an upstream artifact in the module's manifest (`nats:<release>-alpine@sha256:…`);
// before it, an image built from the module's Dockerfile, whose `upstream: nats <release>-alpine` line said
// the release its digest is. The manifest is read first; the Dockerfile only where the manifest names no
// upstream nats image (a catalogue from before that move, such as the captured copy). Neither is a failure.
func catalogueBusRelease(t *testing.T, module string) (release, from string) {
t.Helper()
manifest := filepath.Join(module, "module.json")
raw, err := os.ReadFile(manifest)
if err != nil {
t.Fatal(err)
}
var m struct {
Build struct {
Artifacts []struct {
Name string `json:"name"`
Kind string `json:"kind"`
From string `json:"from"`
} `json:"artifacts"`
} `json:"build"`
}
if err := json.Unmarshal(raw, &m); err != nil {
t.Fatalf("%s: %v", manifest, err)
}
upstream := regexp.MustCompile(`^(?:[^/]+/)*nats:([0-9]+\.[0-9]+\.[0-9]+)-alpine(?:@sha256:[0-9a-f]{64})?$`)
for _, a := range m.Build.Artifacts {
if a.Kind != "upstream" || !strings.HasPrefix(a.From[strings.LastIndex(a.From, "/")+1:], "nats:") {
continue
}
pinned := upstream.FindStringSubmatch(a.From)
if pinned == nil {
t.Fatalf("%s: the upstream artifact %q is %q, which says no release in the form nats:<x.y.z>-alpine",
manifest, a.Name, a.From)
}
return pinned[1], manifest
}
dockerfile := filepath.Join(module, "Dockerfile")
raw, err = os.ReadFile(dockerfile)
if err != nil {
t.Fatalf("%s names no upstream nats image, and the image built before it is not there either: %v", manifest, err)
}
pinned := regexp.MustCompile(`upstream: nats ([0-9.]+)-alpine`).FindSubmatch(raw)
if pinned == nil {
t.Fatalf("%s says no release its digest is", dockerfile)
}
return string(pinned[1]), dockerfile
}
@@ -6,3 +6,4 @@ MESH_NPM_TOKEN_FILE=/run/mesh/package-registry.secret
MESH_GENESIS_PACKAGE_REGISTRY=/run/mesh/genesis-package-registry.json
MESH_WORKSPACE=${dir:workspace}
MESH_JUDGE_CACHE=${dir:judge-cache}
MESH_BUILD_CGROUP_PARENT=meshbuild.slice
+10
View File
@@ -0,0 +1,10 @@
# Managed by the mesh (module build-agent). Replaced on every push; edit the module's settings instead.
#
# Everything a build or check starts on this machine runs in this slice (novox/hq issue 1798): the build
# agent starts every container with --cgroup-parent=meshbuild.slice. The weights only bind while the machine
# is busy, so an idle machine still builds at full speed; the memory limits are the part that always binds.
[Slice]
CPUWeight=${setting:cpu-weight}
IOWeight=${setting:io-weight}
MemoryHigh=${setting:memory-high}
MemoryMax=${setting:memory-max}
+58
View File
@@ -187,3 +187,61 @@ func TestTheAgentReadsTheGatesPullKeysAndTodaysBoth(t *testing.T) {
}
}
}
// Builds and checks run below the machine's own work (novox/hq issue 1798): the module declares the slice
// every build container joins with its limits as settings that have defaults, and tells the agent its name.
func TestTheBuildSliceCarriesTheMachinesLimits(t *testing.T) {
raw, err := os.ReadFile("module.json")
if err != nil {
t.Fatal(err)
}
var m struct {
Config map[string]struct {
Default any `json:"default"`
} `json:"config"`
}
if err := json.Unmarshal(raw, &m); err != nil {
t.Fatal(err)
}
slice, err := os.ReadFile("files/meshbuild.slice")
if err != nil {
t.Fatal(err)
}
for key, property := range map[string]string{"cpu-weight": "CPUWeight", "io-weight": "IOWeight", "memory-high": "MemoryHigh", "memory-max": "MemoryMax"} {
if c, ok := m.Config[key]; !ok || c.Default == nil {
t.Errorf("1798/AC2: setting %s is missing or has no default", key)
}
if !strings.Contains(string(slice), property+"=${setting:"+key+"}") {
t.Errorf("1798/AC2: the slice does not set %s from %s", property, key)
}
}
var placed, running bool
for _, r := range read(t).Resources {
placed = placed || (r["type"] == "file" && r["path"] == "/etc/systemd/system/meshbuild.slice")
running = running || (r["type"] == "service" && r["unit"] == "meshbuild.slice" && r["state"] == "running")
}
if !placed || !running {
t.Errorf("1798/AC2: the slice must be placed (%v) and running (%v)", placed, running)
}
// 1798/AC3: a dash in a slice's name nests it in the slice named before the dash, and cgroup v2 weights only
// compare siblings, so the name has none and the slice sits at the top level beside user.slice.
for _, r := range read(t).Resources {
if r["type"] == "service" && r["unit"] == "meshbuild.slice" {
if strings.Contains(strings.TrimSuffix(r["unit"].(string), ".slice"), "-") {
t.Errorf("1798/AC3: slice name %v nests", r["unit"])
}
// 1798/AC4: a changed limit reaches systemd only if the unit restarts when its file changes.
restart, _ := r["restart-on"].([]any)
if len(restart) != 1 || restart[0] != "build-slice" {
t.Errorf("1798/AC4: the slice unit does not restart on its file: %v", restart)
}
}
}
env, err := os.ReadFile("files/build-agent.env")
if err != nil {
t.Fatal(err)
}
if !strings.Contains(string(env), "MESH_BUILD_CGROUP_PARENT=meshbuild.slice") {
t.Error("1798/AC1: the agent is not told the slice's name")
}
}
+52 -2
View File
@@ -2,6 +2,34 @@
"module": "build-agent",
"version": "1",
"slug": "agent",
"config": {
"cpu-weight": {
"type": "number",
"description": "The CPU weight of everything a build or check starts on this machine, from 1 to 10000 against 100 for an ordinary service or a login session: under load the builds get their share of that, on an idle machine they take what is free (novox/hq issue 1798). The default 20 is about a sixth of the CPU against one competing workload; a desktop may take 10. Changing it restarts the build slice, which stops every build and check running on that machine at that moment; they fail and must be retried.",
"minimum": 1,
"maximum": 10000,
"default": 20
},
"io-weight": {
"type": "number",
"description": "The disk IO weight of everything a build or check starts, from 1 to 10000 against 100 for an ordinary service: the default 10 lets the machine's own work go first (novox/hq issue 1798). Changing it restarts the build slice, which stops every build and check running on that machine at that moment; they fail and must be retried.",
"minimum": 1,
"maximum": 10000,
"default": 10
},
"memory-high": {
"type": "string",
"description": "The memory above which builds and checks are slowed and made to give memory back, a percentage of the machine's memory or a size (systemd's MemoryHigh): the default 50% leaves half the machine to what it runs (novox/hq issue 1798). Changing it restarts the build slice, which stops every build and check running on that machine at that moment; they fail and must be retried.",
"pattern": "([1-9][0-9]?|100)%|[1-9][0-9]*[KMGT]",
"default": "50%"
},
"memory-max": {
"type": "string",
"description": "The memory builds and checks together may never exceed, a percentage or a size (systemd's MemoryMax); a build past it is killed by the kernel and fails, so keep it above memory-high: the default 70% (novox/hq issue 1798). Changing it restarts the build slice, which stops every build and check running on that machine at that moment; they fail and must be retried.",
"pattern": "([1-9][0-9]?|100)%|[1-9][0-9]*[KMGT]",
"default": "70%"
}
},
"capabilities": [
"container-runtime"
],
@@ -9,7 +37,12 @@
{
"name": "node-build-agent",
"scope": "node",
"serves": ["current", "kill", "pause", "resume"]
"serves": [
"current",
"kill",
"pause",
"resume"
]
}
],
"reads": [
@@ -48,7 +81,7 @@
"id": "judge-cache",
"path": "${dir:judge-cache}",
"class": "cache",
"why": "the Go cache every merge check's judge is built with and the group judges kept per head; mounted into judge builds only, never into a check's containers (novox/hq ADR 0358 §6)"
"why": "the Go cache every merge check's judge is built with and the group judges kept per head; mounted into judge builds only, never into a check's containers (novox/hq ADR 0358 \u00a76)"
}
]
},
@@ -69,6 +102,23 @@
"type": "directory",
"mode": "0700"
},
{
"id": "build-slice",
"type": "file",
"path": "/etc/systemd/system/meshbuild.slice",
"mode": "0644",
"trusted": false,
"template": "files/meshbuild.slice"
},
{
"id": "build-slice-unit",
"type": "service",
"unit": "meshbuild.slice",
"state": "running",
"restart-on": [
"build-slice"
]
},
{
"id": "agent-env",
"type": "file",
-29
View File
@@ -1,29 +0,0 @@
ARG ALPINE_BASE=alpine:3.20
ARG GO_BASE=golang:1.25
# The route-proxy module's runtime image: the reference reverse proxy compiled into a container.
#
# **The proxy source sits beside this module.** The canonical proxy, the contract written as something
# that runs, lives in this same repository, mesh-controller, at examples/route-proxy (novox/hq 08-connectivity
# §3), and the repository is the build context (a module lives whole in one repository, novox/hq ADR 0420),
# so this Dockerfile compiles ./examples/route-proxy from the repository root. The build seat hands it only
# that program's build source, its import closure, go.mod, go.sum and the vendored packages it uses
# (novox/hq ADR 0267, the artifact's `compiles`), so a merge elsewhere in the repository does not
# move this image, and a COPY of anything outside it fails by name.
#
# docker build -f modules/route-proxy/Dockerfile -t mesh-route-proxy:development .
#
# The mesh pins the digest of what this produces; the committed module.json carries the placeholder
# digest every mesh-built image does, replaced at publish.
FROM ${GO_BASE} AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -trimpath -buildvcs=false -ldflags=-buildid= -o /mesh-route-proxy ./examples/route-proxy
# A small runtime with the public CA roots the ACME client needs to reach a real authority, and run
# as root so it can bind :80 and :443 — the two privileged ports a public front door listens on.
FROM ${ALPINE_BASE}
RUN apk add --no-cache ca-certificates
COPY --from=build /mesh-route-proxy /usr/local/bin/mesh-route-proxy
ENTRYPOINT ["/usr/local/bin/mesh-route-proxy"]
+8 -8
View File
@@ -45,15 +45,15 @@ pinned image is not moved without a decision; mesh-controller's
## How it ships the Go proxy
The proxy is a Go program, unlike the TypeScript tool-runtime modules. Its source sits in this same
repository, mesh-controller, at `examples/route-proxy`, the contract written as something that runs. This
module ships the packaging: a multi-stage [`Dockerfile`](Dockerfile) whose build context is the repository
root and which compiles `./examples/route-proxy` into `mesh-route-proxy`. The committed `module.json` carries the
placeholder digest every mesh-built image does (`@sha256:0000…`); the mesh pins the real digest at
publish.
The proxy is a Go program, unlike the TypeScript tool-runtime modules. Its source sits in this module, at
`cmd/route-proxy`, in the module's own Go module (`go.mod`, with a `vendor/` directory), the contract written as
something that runs. No image is built for it (novox/hq ADR 0480): the `server` artifact is a Go bundle compiled
for alpine, an `archive` resource unpacks it onto the machine, and the `route-proxy` container runs the pinned
`trust` image (the same alpine digest whose system roots the public account directory is named after) with the
archive mounted read-only at `/opt/mesh` and `/opt/mesh/route-proxy` as its program. A new build of the program
restarts the container, because the archive is in its `restart-on`.
In the lab, `mesh-lab/scripts/build-route-proxy-image.sh` builds this into the local daemon as
`mesh-route-proxy:development`, and a scenario stocks it and serves it by digest.
In the lab, a scenario stocks the same alpine image and serves it by digest.
## Certificates, and the endpoint (resolves 04-ISSUES/004)
@@ -77,7 +77,8 @@ func TestAConsumerOfARouteIsToldItsNamesFromItsPlanEntry(t *testing.T) {
// route-proxy's own files and resources stay as they were (issue 792): the module answers its route through the
// helper but is not marked moved, so its acme.env, its broker secret and its containers are composed as today and
// the public ingress does not change. Everything of the manifest but its build, what it provides and the state it
// keeps is pinned by this digest; changing the ingress means changing it here, on purpose.
// keeps is pinned by this digest; changing the ingress means changing it here, on purpose. Re-pinned in R4d (novox/hq
// ADR 0480): the program is an archive mounted into the trust image, the container and its files are otherwise as before.
func TestRouteProxyIsNotMovedAndKeepsItsOwnFiles(t *testing.T) {
raw, err := os.ReadFile("../../module.json")
if err != nil {
@@ -102,7 +103,7 @@ func TestRouteProxyIsNotMovedAndKeepsItsOwnFiles(t *testing.T) {
t.Fatal(err)
}
sum := sha256.Sum256(pinned)
if got := hex.EncodeToString(sum[:]); got != "57a39de0943d8b33b9db6cb998ce97f8793602cca6cdfa5849c1c392390454a3" {
if got := hex.EncodeToString(sum[:]); got != "85cb955e5a4b2d8826f50aa76433064a576e8b15b71537032eb93c7388c0ee28" {
t.Fatalf("route-proxy's own manifest changed: digest %s", got)
}
// The two acme templates are pinned the same way (issue 849): they are what the proxy's certificates are asked
@@ -4,14 +4,14 @@ import (
"encoding/json"
"fmt"
"log"
"os"
"strings"
"sync/atomic"
"time"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-controller/internal/broker"
"git.novox.be/novox/mesh-sdk/go/membership"
"git.novox.be/novox/mesh-sdk/go/process"
)
// What the mesh issued this proxy, read on the bus (novox/hq ADR 0160, ADR 0167).
@@ -21,16 +21,6 @@ import (
// network, which is who may be served an internal name. Read once at connect and followed, so a
// route added or a machine joining reaches a running proxy without a restart.
// credential is the bus account the mesh delivered as this module's own secret named broker.
type credential struct {
URL string `json:"url"`
Fingerprint string `json:"fingerprint"`
Node string `json:"node"`
Module string `json:"module"`
User string `json:"user"`
Password string `json:"password"`
}
// followMembership connects with the credential in path and applies every membership the mesh
// issues this proxy. It retries the first connection for as long as it takes: a proxy that started
// before the bus keeps serving the file, and takes the bus when it answers.
@@ -46,60 +36,23 @@ func followMembership(path string, held *table, fromBus *atomic.Bool) {
}
func followOnce(path string, held *table, fromBus *atomic.Bool) error {
raw, err := os.ReadFile(path)
account, err := process.ReadAccount(path)
if err != nil {
return err
}
var cred credential
if err := json.Unmarshal(raw, &cred); err != nil {
return fmt.Errorf("the broker credential is not one: %w", err)
}
if cred.Node == "" || cred.Module == "" {
return fmt.Errorf("the broker credential names no node or module, so it has no membership")
}
opts := []nats.Option{
nats.Name(cred.Node + "." + cred.Module),
nats.UserInfo(cred.User, cred.Password),
// Its own inbox, and nothing wider: every principal is granted `_INBOX.<its user>.>` alone.
nats.CustomInboxPrefix("_INBOX." + cred.User),
// The bus being restarted is an upgrade, not a reason to stop following.
nats.MaxReconnects(-1),
}
if strings.TrimSpace(cred.Fingerprint) != "" {
opts = append(opts, nats.Secure(broker.PinnedToFingerprint(cred.Fingerprint)))
}
conn, err := nats.Connect(cred.URL, opts...)
conn, err := process.Connect(account, nats.Name(account.Node+"."+account.Module))
if err != nil {
return fmt.Errorf("connecting to the bus at %s: %w", broker.BareAddress(cred.URL), err)
return fmt.Errorf("connecting to the bus at %s: %w", membership.BareAddress(account.URL), err)
}
subject := broker.MembershipSubject(cred.Node, cred.Module)
apply := func(body []byte) {
var issued broker.Membership
if err := json.Unmarshal(body, &issued); err != nil {
log.Printf("a membership arrived that is not one: %v", err)
return
}
subject := membership.Subject(account.Node, account.Module)
err = conn.FollowMembership(func(issued membership.Membership) {
if took := applyMembership(issued, held); took && !fromBus.Swap(true) {
log.Printf("routes now come from this proxy's membership on %s", subject)
}
}
// Followed first, read second: an issue landing between the two is applied, not missed.
if _, err := conn.Subscribe(subject, func(m *nats.Msg) { apply(m.Data) }); err != nil {
}, func(err error) { log.Print(err) })
if err != nil {
conn.Close()
return fmt.Errorf("cannot follow %s: %w", subject, err)
}
// The subject-addressed direct get: the one request this account may make of the stream.
got, err := conn.Request("$JS.API.DIRECT.GET."+broker.AssignmentsStream+"."+subject, nil, 5*time.Second)
switch {
case err != nil:
log.Printf("cannot read the membership issued on %s yet (%v); following it", subject, err)
case got.Header.Get("Status") != "" || len(got.Data) == 0:
log.Printf("no membership issued on %s yet; serving the file until one is", subject)
default:
apply(got.Data)
return err
}
return nil
}
@@ -108,7 +61,7 @@ func followOnce(path string, held *table, fromBus *atomic.Bool) error {
//
// A membership with no routes in it is one from a controller older than ADR 0167, and the file stays
// the source rather than every route being withdrawn because a field was absent.
func applyMembership(issued broker.Membership, held *table) bool {
func applyMembership(issued membership.Membership, held *table) bool {
raw, carries := issued.Receives["route"]
if !carries {
return false
@@ -12,7 +12,7 @@ import (
"strings"
"testing"
"github.com/novox/mesh-controller/internal/broker"
"git.novox.be/novox/mesh-sdk/go/membership"
)
// behind is a workload the proxy can send to, and a table routing one public name and one
@@ -169,7 +169,7 @@ func TestTheMeshIsReadAsAddressesAndRanges(t *testing.T) {
// machines it names (novox/hq ADR 0167).
func TestAMembershipIsServedAsIssued(t *testing.T) {
held := newTable()
took := applyMembership(broker.Membership{
took := applyMembership(membership.Membership{
Receives: map[string]json.RawMessage{"route": json.RawMessage(`[
{"from":"admin","node":"anchor","at":"anchor.internal",
"values":{"internal-name":"admin.anchor.internal","port":8080}}]`)},
@@ -191,13 +191,13 @@ func TestAMembershipIsServedAsIssued(t *testing.T) {
func TestAMembershipWithoutRoutesLeavesTheFileServing(t *testing.T) {
held := behind(t, "10.10.0.7")
before := held.names()
if applyMembership(broker.Membership{Mesh: []string{"10.10.0.7"}}, held) {
if applyMembership(membership.Membership{Mesh: []string{"10.10.0.7"}}, held) {
t.Error("a membership without routes was taken as the source of routes")
}
if got := held.names(); strings.Join(got, ",") != strings.Join(before, ",") {
t.Errorf("a membership without routes changed what is served: %v, was %v", got, before)
}
if applyMembership(broker.Membership{
if applyMembership(membership.Membership{
Receives: map[string]json.RawMessage{"route": json.RawMessage(`[]`)},
Mesh: []string{"not-an-address"},
}, held) {
+14 -1
View File
@@ -2,4 +2,17 @@ module route-proxy
go 1.26.0
require git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261021114014-66e8fdaa62e2
require (
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261022075101-c61212eba77c
github.com/nats-io/nats.go v1.54.0
golang.org/x/crypto v0.57.0
)
require (
github.com/klauspost/compress v1.20.0 // indirect
github.com/nats-io/nkeys v0.4.16 // indirect
github.com/nats-io/nuid v1.0.1 // indirect
golang.org/x/net v0.58.0 // indirect
golang.org/x/sys v0.48.0 // indirect
golang.org/x/text v0.42.0 // indirect
)
+30 -2
View File
@@ -1,2 +1,30 @@
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261021114014-66e8fdaa62e2 h1:Cdl5N56QPfmFBoLHrDfvvyc5Q+PUnTjf3fgYD7imcco=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261021114014-66e8fdaa62e2/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261022075101-c61212eba77c h1:YdSbkI42rlVmClybH8BS717ut0vouKIMavO691fpdps=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261022075101-c61212eba77c/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI=
github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo=
github.com/google/go-tpm v0.9.8/go.mod h1:h9jEsEECg7gtLis0upRBQU+GhYVH6jMjrFxI8u6bVUY=
github.com/klauspost/compress v1.20.0 h1:a3C1ke2ohxFymNlb2HWAHjDeKCI90scRskErZkR0ezA=
github.com/klauspost/compress v1.20.0/go.mod h1:LUdAzn7YLVvxLpc7y3V1m40wESHTgc1422pwwBSKYuI=
github.com/minio/highwayhash v1.0.4 h1:asJizugGgchQod2ja9NJlGOWq4s7KsAWr5XUc9Clgl4=
github.com/minio/highwayhash v1.0.4/go.mod h1:GGYsuwP/fPD6Y9hMiXuapVvlIUEhFhMTh0rxU3ik1LQ=
github.com/nats-io/jwt/v2 v2.8.1 h1:V0xpGuD/N8Mi+fQNDynXohVvp7ZztevW5io8CUWlPmU=
github.com/nats-io/jwt/v2 v2.8.1/go.mod h1:nWnOEEiVMiKHQpnAy4eXlizVEtSfzacZ1Q43LIRavZg=
github.com/nats-io/nats-server/v2 v2.11.17 h1:GKEghcFK6A+aFx11Yf1LjgLC3txAwvyhnYzhBIQZA8I=
github.com/nats-io/nats-server/v2 v2.11.17/go.mod h1:B1sFVz4StNosQ903ak4N1G01Fl/9f8e06mXpFIE2K24=
github.com/nats-io/nats.go v1.54.0 h1:vsXoOxjHp/GmPUN+EcI7uOf/uB+iAP+kEsAFNQN0yzA=
github.com/nats-io/nats.go v1.54.0/go.mod h1:y+DZoD1oBOYfZTU681eTUiUjI0vbqYGixNVFHcjHJ0k=
github.com/nats-io/nkeys v0.4.16 h1:rd5oAuLOb8mnAycB0xleuEBNS1pVVnN0fv/FF34Eypg=
github.com/nats-io/nkeys v0.4.16/go.mod h1:llLgWoI0o4z/Q57q2R1kHfmocyhGV6VG/U18Glg1Afs=
github.com/nats-io/nuid v1.0.1 h1:5iA8DT8V7q8WK2EScv2padNa/rTESc1KdnPw4TC2paw=
github.com/nats-io/nuid v1.0.1/go.mod h1:19wcPz3Ph3q0Jbyiqsd0kePYG7A95tJPxeL+1OSON2c=
golang.org/x/crypto v0.57.0 h1:3ZVCjf8Ggz7zneR/EHRVx68Ctf+2pmIMP2UFhh9cC6M=
golang.org/x/crypto v0.57.0/go.mod h1:Fdz0i5U6CoizGwLda9DttjSk6qlZo25zYNtR+ycvuZA=
golang.org/x/net v0.58.0 h1:ynWG7rqYi4ccpTEuPZ2QGWHktVEM9DMCj9yzDE0Q7To=
golang.org/x/net v0.58.0/go.mod h1:YwCddHnFlT7eLQqVprV19OnhLGtc5xOKgE0RyqgfWAU=
golang.org/x/sys v0.48.0 h1:bbX/i/6MgT9BVLM9RT1thmxL04yeTAhbEz4SyadbXoo=
golang.org/x/sys v0.48.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
golang.org/x/text v0.42.0 h1:JbOZXgfeCPU9gacVtYliJqOhD+zhrEqK4LfdpmlUZqI=
golang.org/x/text v0.42.0/go.mod h1:ojzP1Z+2QtioaF8DTtO8K5q7JWVVYwZKenzujK0Zd0E=
golang.org/x/time v0.15.0 h1:bbrp8t3bGUeFOx08pvsMYRTCVSMk89u4tKbNOZbp88U=
golang.org/x/time v0.15.0/go.mod h1:Y4YMaQmXwGQZoFaVFk4YpCt4FLQMYKZe9oeV/f4MSno=
+20 -21
View File
@@ -149,11 +149,18 @@
"internal-acme-env"
]
},
{
"id": "program-route-proxy",
"type": "archive",
"path": "${dir:state}/route-proxy",
"artifact": "server",
"unknown": "program"
},
{
"id": "server",
"type": "container",
"name": "route-proxy",
"artifact": "server",
"artifact": "trust",
"network": "host",
"env-file": [
"${dir:state}/acme.env",
@@ -163,7 +170,8 @@
"${dir:routes-dir}:/routes:ro",
"${dir:acme-cache}:/acme",
"${dir:ca-dir}:/ca:ro",
"${dir:mesh-state}/broker:/run/secrets/broker:ro"
"${dir:mesh-state}/broker:/run/secrets/broker:ro",
"${dir:state}/route-proxy:/opt/mesh:ro"
],
"env": {
"ROUTES": "/routes/mesh.json",
@@ -174,11 +182,15 @@
"INTERNAL_ACME_CA_BUNDLE": "/ca/internal-root.crt",
"MESH_BROKER_FILE": "/run/secrets/broker"
},
"args": [
"/opt/mesh/route-proxy"
],
"restart-on": [
"trust",
"acme-env",
"internal-trust",
"internal-acme-env"
"internal-acme-env",
"program-route-proxy"
],
"logging": "journald",
"health": {
@@ -195,14 +207,11 @@
"artifacts": [
{
"name": "server",
"kind": "image",
"from": "Dockerfile",
"compiles": "examples/route-proxy",
"context": {
"seat": "git",
"repository": "novox/mesh-controller",
"ref": "main"
}
"kind": "bundle",
"language": "go",
"system": "alpine",
"from": "cmd/route-proxy",
"binary": "route-proxy"
},
{
"name": "trust",
@@ -225,16 +234,6 @@
"kind": "archive",
"from": "files"
}
],
"on": [
{
"arg": "GO_BASE",
"image": "golang@sha256:6c2a5538f964f1c82f97ad14988bf05de100d922d159d0e398b54c7b0ca0c6c9"
},
{
"arg": "ALPINE_BASE",
"image": "alpine@sha256:d9e853e87e55526f6b2917df91a2115c36dd7c696a35be12163d44e6e2a4b6bc"
}
]
},
"jails": [
@@ -0,0 +1,165 @@
// Package membership is what the mesh issues an assignment to serve and to reach (novox/hq ADR 0160), as a
// process reads it from the bus: the Membership shape, the one subject it is published on, the stream that
// holds it, and the bus address helpers a reader needs. The controller composes it and the route proxy reads
// it; both use this one definition, so there is no second copy to drift (novox/hq issue 1633).
package membership
import (
"encoding/json"
"strings"
)
// AssignmentsStream holds every assignment's membership, the newest per subject.
const AssignmentsStream = "ASSIGNMENTS"
// Subject is the one address a runtime derives for itself: where its own membership is published, from
// the two names its credential carries. Everything else is in the membership.
func Subject(node, module string) string {
return "mesh.assignment." + node + "." + module
}
// DirectGetSubject is the subject-addressed direct get of a membership from its stream: the one request an
// account may make of the stream.
func DirectGetSubject(node, module string) string {
return "$JS.API.DIRECT.GET." + AssignmentsStream + "." + Subject(node, module)
}
// Membership is one assignment's subjects: what this instance of a module on this machine serves,
// and what it may reach.
type Membership struct {
Node string `json:"node"`
Module string `json:"module"`
// Serves is every address a tool of this instance answers on. `{tool}` stands for the tool's
// own name. An address with a queue is shared with the module's other instances.
Serves []Served `json:"serves"`
// Seats is every verb of a seat this instance holds, at the subject the seat's callers use.
Seats []SeatServed `json:"seats,omitempty"`
// Emits is where an event of this module lands; `{event}` stands for the event's name.
Emits string `json:"emits"`
// Reaches is each tool this module may call, `<module>.<tool>`, to the subjects that reach it.
Reaches map[string][]string `json:"reaches,omitempty"`
// Tools is where this instance answers what it serves.
Tools string `json:"tools"`
// Receives is what this assignment is given for each requirement it receives, by requirement (novox/hq
// ADR 0167): kept as JSON because the catalogue owns the shape of a contribution.
Receives map[string]json.RawMessage `json:"receives,omitempty"`
// Mesh is every machine's address on the private network (novox/hq ADR 0167).
Mesh []string `json:"mesh,omitempty"`
// State is every bucket this module's code may reach, and whether it may write it (novox/hq ADR 0201).
State []StateIssued `json:"state,omitempty"`
// Logs is every log this module's code may reach (novox/hq ADR 0297).
Logs []LogIssued `json:"logs,omitempty"`
// SeatTraffic is what this module's code may submit, say, hear, take, ask, answer and read on seats
// that name their caller or their kind (novox/hq ADR 0259 §3).
SeatTraffic *SeatTraffic `json:"seat-traffic,omitempty"`
// Streams are this module's own streams, Hears the module streams it hears (novox/hq ADR 0363).
Streams []StreamIssued `json:"streams,omitempty"`
Hears []HeardIssued `json:"hears,omitempty"`
// Waits are the event families a session may wait on through the mesh MCP server (novox/hq ADR 0367 §2).
Waits []WaitIssued `json:"waits,omitempty"`
}
// Served is one address a tool is answered on.
type Served struct {
Subject string `json:"subject"`
Queue string `json:"queue,omitempty"`
}
// SeatServed is one verb of a held seat, where its callers ask.
type SeatServed struct {
Seat string `json:"seat"`
Verb string `json:"verb"`
Subject string `json:"subject"`
}
// StateIssued is one bucket an assignment may reach, by the name its module uses for it (novox/hq ADR 0201).
// Key, when set, is the one key it may reach (novox/hq ADR 0260): this machine's.
type StateIssued struct {
Name string `json:"name"`
Bucket string `json:"bucket"`
Writes bool `json:"writes,omitempty"`
Key string `json:"key,omitempty"`
}
// LogIssued is one log an assignment may reach, by the name its module uses for it (novox/hq ADR 0297).
type LogIssued struct {
Name string `json:"name"`
Stream string `json:"stream"`
Subject string `json:"subject"`
Writes bool `json:"writes"`
}
// Worker is one durable consumer a holder pulls a seat's work from.
type Worker struct {
Stream string
Consumer string
Filter string
}
// SeatTraffic is one module's seat traffic beyond tools. Publish and Subscribe are subject patterns, in the
// server's wildcards.
type SeatTraffic struct {
Publish []string `json:"publish,omitempty"`
Subscribe []string `json:"subscribe,omitempty"`
Answers []string `json:"answers,omitempty"`
Workers []Worker `json:"workers,omitempty"`
Records []string `json:"records,omitempty"`
Kinds []KindHeld `json:"kinds,omitempty"`
}
// KindHeld is one kind of a kinded bench and who holds it.
type KindHeld struct {
Seat string `json:"seat"`
Kind string `json:"kind"`
Module string `json:"module"`
Node string `json:"node"`
Capabilities []string `json:"capabilities,omitempty"`
}
// StreamIssued is one of a module's own streams as its membership names it (ADR 0363 §2).
type StreamIssued struct {
Stream string `json:"stream"`
Events []string `json:"events"`
Subject string `json:"subject"`
}
// HeardIssued is one module stream an assignment hears (ADR 0363 §8).
type HeardIssued struct {
Stream string `json:"stream"`
Consumer string `json:"consumer"`
}
// WaitIssued is one family as a membership issues it: the name a session asks for, and its subject.
type WaitIssued struct {
Event string `json:"event"`
Subject string `json:"subject"`
}
// CredentialIn reads the credential out of a bus address, `[scheme://]user:password@host:port`, splitting on
// the last at-sign so a password containing one stays whole, and returns the address without it.
func CredentialIn(address string) (user, password, bare string) {
at := strings.LastIndex(address, "@")
if at < 0 {
return "", "", address
}
scheme := ""
rest := address[:at]
if i := strings.Index(rest, "://"); i >= 0 {
scheme, rest = rest[:i+3], rest[i+3:]
}
user, password, _ = strings.Cut(rest, ":")
return user, password, scheme + address[at+1:]
}
// BareAddress is a bus address with any credential stripped, for something that only needs to know
// whether a server is answering there.
func BareAddress(address string) string {
_, _, bare := CredentialIn(address)
if bare == "" {
return address
}
if strings.Contains(bare, "://") {
return bare
}
return "nats://" + bare
}
@@ -0,0 +1,54 @@
// Package process is a module's own process's connection to the bus (novox/hq ADR 0436 clause 4): it reads
// the account the controller delivers, connects as it with its pinned fingerprint and its inbox prefix, and
// offers emitting an event, the module's states with compare-and-set, consuming a work queue, reading the
// router's records of the module's own asks, and calling a tool or verb — each only as the account's grants
// allow. It offers no Serve and no way to serve a tool: a process never answers a call. A sibling of stdio,
// which it imports only for stdio's pure types (ErrChanged, CodeChanged, StateEntry, EmitOptions); it never
// touches stdio's current, channel or live(), which stay the bundle's own way to the bus.
package process
import (
"encoding/json"
"fmt"
"os"
)
// Account is a process's own bus account, exactly as the controller delivers it as the module's own secret
// broker-<process id> (mesh-controller cmd/mesh-controller/modules.go issueProcess, heldProcessAccount, and
// issueWith for a module's own broker account, which also carries claims). A process account carries no
// claims at all; this type reads either file without error.
type Account struct {
URL string `json:"url"`
Fingerprint string `json:"fingerprint,omitempty"`
Node string `json:"node"`
Module string `json:"module"`
User string `json:"user"`
Password string `json:"password"`
// Claims rides on a module's own broker account only; a process holds no seat, so its own account
// carries none (novox/hq ADR 0436 §2, §3). Kept raw: this package never reads it.
Claims json.RawMessage `json:"claims,omitempty"`
}
// InboxPrefix is this account's own inbox, and nothing wider: `_INBOX.<user>`.
func (a Account) InboxPrefix() string { return "_INBOX." + a.User }
// ReadAccount reads and validates the account file the controller delivers: it refuses a file missing url,
// user or password, naming the field — never the password, which does not appear in an error.
func ReadAccount(path string) (Account, error) {
data, err := os.ReadFile(path)
if err != nil {
return Account{}, fmt.Errorf("reading the process account at %s: %w", path, err)
}
var a Account
if err := json.Unmarshal(data, &a); err != nil {
return Account{}, fmt.Errorf("the process account at %s is not JSON: %w", path, err)
}
for _, f := range []struct{ name, value string }{
{"url", a.URL}, {"user", a.User}, {"password", a.Password},
} {
if f.value == "" {
return Account{}, fmt.Errorf("the process account at %s has no %s", path, f.name)
}
}
return a, nil
}
@@ -0,0 +1,68 @@
package process
import (
"context"
"encoding/json"
"fmt"
"strings"
stdio "git.novox.be/novox/mesh-sdk/go"
)
// Ask reads the router's record of one of this process's module's own asks, under its own module and no
// other's (novox/hq ADR 0436 clause 4, ADR 0259 §3): by direct get on bucket, the same subject the tool
// runner reads (mesh-tools node-tools/internal/bus/seat.go ~300-345):
//
// $JS.API.DIRECT.GET.KV_<bucket>.$KV.<bucket>.<module>.<id>
//
// bucket is the router's records bucket as the caller's own manifest names it (operator-channel's is
// messenger_asks in the controller's own test fixtures) — never hard-coded to one bucket here, since a
// process may read records of more than one seat's router. A missing, deleted or purged key is nil, not
// an error.
func (c *Conn) Ask(bucket, id string) (*stdio.StateEntry, error) {
if err := checkAskID(id); err != nil {
return nil, err
}
subject := "$JS.API.DIRECT.GET.KV_" + bucket + ".$KV." + bucket + "." + c.account.Module + "." + id
ctx, cancel := context.WithTimeout(context.Background(), stateTimeout)
defer cancel()
reply, err := c.nc.RequestWithContext(ctx, subject, nil)
if err != nil {
return nil, fmt.Errorf("reading %s's ask %s from %s: %w", c.account.Module, id, bucket, err)
}
if status := reply.Header.Get("Status"); status == "404" {
return nil, nil
} else if status != "" {
return nil, fmt.Errorf("the record %s.%s could not be read: %s %s", c.account.Module, id, status,
reply.Header.Get("Description"))
}
if op := reply.Header.Get("KV-Operation"); op == "DEL" || op == "PURGE" {
return nil, nil
}
var revision uint64
if _, err := fmt.Sscan(reply.Header.Get("Nats-Sequence"), &revision); err != nil {
return nil, fmt.Errorf("the record %s.%s carries no readable revision: %w", c.account.Module, id, err)
}
return &stdio.StateEntry{Key: c.account.Module + "." + id, Value: askValueOf(reply.Data), Revision: revision}, nil
}
// checkAskID refuses an id holding a dot or a wildcard before anything is sent: either would name a
// different key under the direct-get subject's own dot-joined path, never this one (novox/hq issue 1189
// AC2).
func checkAskID(id string) error {
if id == "" || strings.ContainsAny(id, ".*>") {
return fmt.Errorf("%q is not an ask id: an ask id holds no dot and no wildcard, naming one key and "+
"no other", id)
}
return nil
}
// askValueOf is a record's raw bytes as JSON, quoting them when they are not already valid JSON
// (mesh-tools node-tools/internal/bus/state.go:351-357, valueOf).
func askValueOf(raw []byte) json.RawMessage {
if len(raw) == 0 || !json.Valid(raw) {
b, _ := json.Marshal(string(raw))
return b
}
return json.RawMessage(append([]byte(nil), raw...))
}

Some files were not shown because too many files have changed in this diff Show More