Author SHA1 Message Date
mesh-tools 64dfa90c3a Report whether an unprivileged user namespace can be made (issue 1308)
A build module's sandbox needs one, and a module can require only a capability the node-engine detects (ADR 0448 section 5). The probe drops to an unprivileged uid first when run as root, since root always can.
2026-10-20 01:12:24 +00:00
mesh-tools 2474bf2833 Merge pull request 'The node-engine sends no alive message: presence replaces it (design 47 R2d, issue 1158)' (#120) from feat/alive-gone into main 2026-10-19 07:07:56 +00:00
jschoubben 671f284004 Delete the node-engine's alive message, presence replaces it (issue 1158)
mesh/merge-gate head 671f2840046f merged into trunk 89025ad004d4 — pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes wi…
mesh/repo-check head 671f2840046f merged into trunk 89025ad004d4 — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-19 02:15:00 +02:00
mesh-tools 89025ad004 Merge pull request 'Judge a filled write with the roots the last apply read, asking no runtime per file (issue 1138)' (#119) from fix/1138-filled-write-asks-no-runtime into main 2026-10-18 22:20:54 +00:00
jochen 3798673821 Judge a filled write with the roots the last apply read, asking no runtime per file (issue 1138)
mesh/merge-gate out of date: judged against trunk 8119c470d393, the trunk is now 89025ad004d4; judged again on a recheck or when it is merged
mesh/repo-check out of date: judged against trunk 8119c470d393, the trunk is now 89025ad004d4; judged again on a recheck or when it is merged
mesh/delivery delivered
2026-10-19 00:11:58 +02:00
mesh-tools 8119c470d3 Merge pull request 'Write each module's have and the node's own facts beside the report (issue 1132); the check reads build-agent from the controller (issue 1142)' (#118) from feat/delivery-have into main 2026-10-18 18:36:14 +00:00
jschoubben aab65b81f6 Read the build agent's manifest from the controller's repository beside the check, where it moved (issue 1151)
mesh/repo-check head aab65b81f65c merged into trunk 6469e8238443 — pass: its merge-check.sh passed
mesh/merge-gate head aab65b81f65c merged into trunk 6469e8238443 — pass: group feat/delivery-have composed: every machine composes with the change as it di…
mesh/delivery delivered
mesh/delivery-group group feat/delivery-have delivered: every member is delivered
2026-10-18 19:28:07 +02:00
jschoubben 9d9e3a46c4 Bind the have bucket once per pass, bound by the queue's timeout, and stop the pass at the first error (issue 1132) 2026-10-18 19:18:08 +02:00
jschoubben 0b391f49a2 Write each module's have and the node's own facts beside the report, grouped per module and keyed by the module the controller names (issue 1132)
The node-engine reads a resource's module and a declaration's builds, says it does in every report (reads_modules), groups the apply's outcome per module with every failing resource named, and writes have.<node>.<module> and have.<node>.node to the bucket only when a value differs, never ahead of or in place of the report (novox/hq ADR 0429 section 4).
2026-10-18 19:10:05 +02:00
mesh-tools 6469e82384 Merge pull request 'A directory says how it reacts to files it does not know; the node-engine moves out what a move-out directory does not know (issue 1024)' (#117) from feat/owned-paths into main 2026-10-17 16:20:06 +00:00
mesh-tools dae79c7643 Prove move-out on a machine that is not Arch (issue 1024)
mesh/merge-gate head dae79c76434c merged into trunk cd13f7726ef0 — pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes wi…
mesh/repo-check head dae79c76434c merged into trunk cd13f7726ef0 — pass: its merge-check.sh passed
mesh/delivery delivered
The merge check's build container has no pacman, so the acceptance test now stubs it and names the host it was built for, as other command tests do; nothing in the node-engine itself changes.
2026-10-17 16:16:02 +00:00
mesh-tools d237dee4c9 Let a directory say how it reacts to files it does not know, and move out what a move-out directory does not know (issue 1024, ADR 0422)
mesh/merge-gate head d237dee4c9e3 merged into trunk cd13f7726ef0 — pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes wi…
mesh/repo-check head d237dee4c9e3 merged into trunk cd13f7726ef0 — fail: its merge-check.sh failed: --- FAIL: TestAMoveOutDirectoryKeepsOnlyWhatItKnows (0.…
mesh/delivery superseded: a newer head of the same pull request
A directory or archive resource carries `unknown` (move-out, module, program) and `knows`. For move-out under a home, every look moves each path the declaration does not know into ~/.local/state/mesh-moved/<module>/<time>/, the list written before each move, without following a link, a young path waiting, a socket or pipe left and reported, nothing made when nothing moved, and another file system refused. Reviewed in two rounds.
2026-10-17 16:09:33 +00:00
mesh-tools cd13f7726e Merge pull request 'Feature 57 (config and provisioning): release mesh-host' (#116) from release/57-config-and-provisioning into main 2026-10-16 09:32:15 +00:00
mesh-tools bff1ac14a1 Feature 57 (config and provisioning): release mesh-host, the reviewed release branch on top of main
mesh/merge-gate head bff1ac14a198 merged into trunk 49c9002c04d3 beside mesh-controller#355@8c381172 — pass: builds mesh-host → ace, g14, novox, shanks; no…
mesh/repo-check head bff1ac14a198 merged into trunk 49c9002c04d3 beside mesh-controller#355@8c381172 — pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group release/57-config-and-provisioning delivered: every member is delivered
2026-10-16 08:24:13 +00:00
mesh-tools 49c9002c04 Merge pull request 'Grant the genesis controller user operator-channel.tell (issue 835)' (#113) from feat/mail-seats into main 2026-10-15 17:09:30 +00:00
jochen c3b2fce0a2 Grant the controller's new operator-channel.tell publish to the genesis user (issue 835)
mesh/merge-gate head c3b2fce0a29f merged into trunk 198fd34ec497 — pass: the change touches no module of the mesh's graph
mesh/repo-check head c3b2fce0a29f merged into trunk 198fd34ec497 — pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group feat/mail-seats delivered: every member is delivered
ADR 0380's secret-given announcement asks operator-channel.tell over the
bus; the genesis bundle's first user list must already carry that
publish subject, in the same place the controller composes it
(internal/broker, mesh-controller ms-835 435468c1), or a first node's
controller starts without the grant until its own first composition.
2026-10-15 17:49:51 +02:00
mesh-admin 198fd34ec4 Merge pull request 'Start no account's manager to look at its units (hq issue 764)' (#112) from fix/764-a-look-starts-no-user-manager into main 2026-10-14 22:38:02 +00:00
jschoubben 62af89f85d Start no account's manager to look at its units (hq issue 764)
mesh/merge-gate head 62af89f85d0d merged into trunk 1a2677882380 — pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes wi…
mesh/repo-check head 62af89f85d0d merged into trunk 1a2677882380 — pass: its merge-check.sh passed
mesh/delivery delivered
Every 15 s the failed-units look read the own manager of every account
the declaration creates, and reading one is a login that starts it: six
module accounts' managers on the control node started and stopped all
day, most of its journal. A look only reads (ADR 0240 rule 6) and never
asks an account's manager (ADR 0177).

An account's manager is read now only when the declaration places a
user-scoped unit in it, and only after the machine's own manager says
user@<uid>.service is active, as the apply asks before it. A manager not
running is said away in the units statement, with no failed units; in
liveness its units read unknown with the apply's words, never down. The
operator's account on a desktop is still read while logged in, since a
module places a user unit there (issue 315's Razer daemon stays said).
2026-10-15 00:33:34 +02:00
mesh-admin 1a26778823 Merge pull request 'A busy container is left running until it says idle, within 15 minutes (issue 594, hq ADR 0335)' (#98) from feat/335-a-busy-container-is-replaced-when-idle into main 2026-10-13 22:20:54 +00:00
jschoubben b9a293f8c8 Merge remote-tracking branch 'origin/main' into feat/335-a-busy-container-is-replaced-when-idle
mesh/merge-gate head b9a293f8c8bc merged into trunk 0e92ef3d1b21 — pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes wi…
mesh/repo-check head b9a293f8c8bc merged into trunk 0e92ef3d1b21 — pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group feat/335-a-busy-container-is-replaced-when-idle delivered: every member is delivered
# Conflicts:
#	cmd/mesh-host/main.go
2026-10-14 00:00:09 +02:00
jschoubben 695e022991 Keep a replacement owed after a busy wait, and clamp a wait from a later clock
A restart-on trigger that is no file counts as changed only in its own pass,
so after a busy wait the container read unchanged and was never replaced,
while the report stopped listing it (review of #98).
2026-10-13 23:59:58 +02:00
mesh-admin 0e92ef3d1b Merge pull request 'Report the networks the outward links are on, for the local network (issue 154, ADR 0340)' (#107) from fix/154-local-network into main 2026-10-13 13:07:08 +00:00
mesh-admin 12bc6a56a3 Merge pull request 'Adopt genesis's bus through the bus's planned step, its streams snapshotted first (issue 590, hq ADR 0333 §3)' (#108) from fix/590-adopt-the-bus-as-its-planned-step into main 2026-10-13 11:11:41 +00:00
jochen cdcfcd9380 Adopt genesis's bus through the bus's planned step, its streams snapshotted first (hq issue 590, ADR 0333 §3)
mesh/merge-gate head cdcfcd938016 merged into trunk 130c5d98a601 — pass: the change touches no module of the mesh's graph
mesh/repo-check head cdcfcd938016 merged into trunk 130c5d98a601 — pass: its merge-check.sh passed
mesh/delivery delivered
The broker step sent the nats module with a plain push, and the controller refuses a push that would
replace the bus: that is a planned step, bus upgrade, which snapshots the streams first and checks
them after (ADR 0236). Genesis stopped at step 18.

The step now snapshots every stream with the module's own snapshot program (no backup holder exists
yet to be asked), asks bus upgrade with that snapshot and why, says the one restart, and waits until
the bus runs the module's image, answers, and the controller ends the step done. The bundle's bus runs
the release the module's image is built on (2.11.17, by the same digest, where it ran 2.10.29), so the
step changes no server release and is said reversible.
2026-10-13 11:47:10 +02:00
mesh-admin 130c5d98a6 Merge pull request 'Build the Go toolchain at genesis's base step, beside the base (issue 616)' (#106) from fix/616-the-base-builds-every-genesis-toolchain into main 2026-10-13 09:23:04 +00:00
jochen cb7b71dcc5 Report the networks the outward links are on, for the local network (issue 154, ADR 0340)
mesh/merge-gate head cb7b71dcc568 merged into trunk 12bc6a56a38a — pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes wi…
mesh/repo-check head cb7b71dcc568 merged into trunk 12bc6a56a38a — pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group fix/154-local-network delivered: every member is delivered
A port reached from local-network is admitted from the private networks the machine is attached to, and only the machine can say which those are. Every network on a link facing outside is said among its facts, and a change is said at the next reconcile.
2026-10-13 11:20:21 +02:00
jochen 8108dd5c23 Build the Go toolchain at genesis's base step, beside the base (hq issue 616)
mesh/merge-gate head 8108dd5c23c6 merged into trunk f0d39b4a4ab0 — pass: the change touches no module of the mesh's graph
mesh/repo-check head 8108dd5c23c6 merged into trunk f0d39b4a4ab0 — pass: its merge-check.sh passed
mesh/delivery delivered
Genesis's base step built only mesh-tools, the TypeScript toolchain. The store's and the bus's modules
carry Go bundles, compiled in mesh-tools-go, so genesis stopped at the store step: "this mesh holds no
copy of it. Build mesh-tools-go first". The base step now builds mesh-tools-go too, from
--tools-go-source or the repository beside --tools-source, and a test holds every toolchain a genesis
module's bundles compile in, read from the catalogue, to the ones the step builds.
2026-10-13 11:13:44 +02:00
mesh-admin f0d39b4a4a Merge pull request 'Wait until the node applied the genesis document before the sdk step asks for a build (issue 615)' (#105) from fix/615-wait-for-the-genesis-document into main 2026-10-13 09:08:59 +00:00
jochen 89fd0c06e1 Wait until the node applied the genesis document before the sdk step asks for a build (hq issue 615)
mesh/merge-gate head 89fd0c06e102 merged into trunk 17038b2a074b — pass: the change touches no module of the mesh's graph
mesh/repo-check head 89fd0c06e102 merged into trunk 17038b2a074b — pass: its merge-check.sh passed
mesh/delivery delivered
The packages step sealed the build agent its genesis document and pushed, and the push returned as soon
as it was sent. The sdk step then asked for the SDK's build at once, and the agent reads the document
from the file the node writes when it applies that push: one genesis found it in time, the next was
refused for having neither a binding nor the document. That one push now waits until the node reports
it applied it (push --wait); every other push still returns on sending.
2026-10-13 10:59:45 +02:00
mesh-admin 17038b2a07 Merge pull request 'Raise genesis's store as the postgres module's server, so the store step adopts it as it runs (issue 609, hq ADR 0338)' (#104) from fix/609-genesis-raises-the-store-as-postgres into main 2026-10-13 08:55:41 +00:00
jochen 4dde21f0ab Raise genesis's store as the postgres module's server, so the store step adopts it as it runs (hq issue 609, ADR 0338)
mesh/merge-gate head 4dde21f0ab20 merged into trunk 7d1d6bffe30c — pass: the change touches no module of the mesh's graph
mesh/repo-check head 4dde21f0ab20 merged into trunk 7d1d6bffe30c — pass: its merge-check.sh passed
mesh/delivery delivered
The bundle raised the store as mesh-store, on upstream postgres, in a docker volume; the postgres
module keeps it as postgres, on its pgvector image, in /var/lib/mesh-store owned 999:70, reading its
superuser password from a file. The store step refused, and genesis stopped at step 17 of 24.

The bundle now raises the store with the module's name, image, data directory and password file, and
every step of the bundle and the installer reaches it by that name. The refusal to raise a store over
data it cannot open looks for a cluster in the store's directory, no longer for the old volume.
2026-10-13 10:47:53 +02:00
mesh-admin 7d1d6bffe3 Merge pull request 'Place the bus's users after every push until the bus is adopted (issue 608)' (#103) from fix/608-place-the-bus-users-after-every-push into main 2026-10-13 05:00:42 +00:00
jochen 7283209d1a Place the bus's users after every push until the bus is adopted (hq issue 608)
mesh/merge-gate head 7283209d1abc merged into trunk 64f0ab405101 — pass: the change touches no module of the mesh's graph
mesh/repo-check head 7283209d1abc merged into trunk 64f0ab405101 — pass: its merge-check.sh passed
mesh/delivery delivered
The bus reads its user list from a file that, until the broker step adopts it as the nats module,
only the installer writes, and the installer wrote it only at enrolment. The build agent's account,
minted with step 13's push, never reached the bus; the agent was refused (Authorization Violation)
and the sdk step's build waited until it was killed. Every push genesis makes now places the composed
list and has the bus re-read it, until the nats module is registered (ADR 0331 §5, ADR 0333 §4).
2026-10-13 06:58:39 +02:00
mesh-admin 64f0ab4051 Merge pull request 'genesis: seal the build agent the forge it raised, in place of the builder's npm-password (hq issue 121, ADR 0336)' (#102) from fix/121-genesis-package-registry into main 2026-10-13 03:37:15 +00:00
jochen 5f147fe6b9 Seal the build agent the forge genesis raised, in place of the builder's npm-password (hq issue 121, ADR 0336)
mesh/merge-gate head 5f147fe6b9c7 merged into trunk 0d11b4fc930d — pass: the change touches no module of the mesh's graph
mesh/repo-check head 5f147fe6b9c7 merged into trunk 0d11b4fc930d — pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group fix/121-genesis-package-registry delivered: every member is delivered
The build agent declares neither the retired builder's npm-password own-secret
nor a serves setting, so the packages step handed its credential to nothing
and genesis's sdk step had no way to publish. The packages step now seals the
genesis-package-registry own-secret to build-agent: the forge on this node's
loopback, the port given for the package registry, its npm path, and the
mesh-builder account with its password. setFoundationSettings no longer gives
the builder a serves.port.
2026-10-13 03:01:35 +02:00
jschoubben bb60e7b584 Leave a busy container running until it says idle, within 15 minutes
mesh/delivery superseded: a newer head of the same pull request
mesh/delivery-group group feat/335-a-busy-container-is-replaced-when-idle rejected: its order contradicts itself: novox/mesh-catalog@190608193764, novox/mesh-c…
mesh/merge-gate out of date: judged against trunk 7a41e120635b, the trunk is now 8431ac96562e; judged again on a recheck or when it is merged
mesh/repo-check out of date: judged against trunk 7a41e120635b, the trunk is now 8431ac96562e; judged again on a recheck or when it is merged
A send that replaced the build agent killed the check it ran, and the queue
started it again elsewhere (issue 594). A container may now declare busy; the
node-engine asks it before replacing it, applies everything else, says the
wait, and replaces it on the first idle pass or at the bound, kept across
restarts (novox/hq ADR 0335).
2026-10-13 00:41:32 +02:00
102 changed files with 9858 additions and 367 deletions
+13
View File
@@ -102,6 +102,19 @@ what it did not understand would apply most of a declaration and report success.
]}
```
**A directory or archive says how it treats files it did not write** (issue 1024,
[`novox/hq` ADR 0422](https://git.novox.be/novox/hq)). `unknown` is `move-out`, `module` or `program`;
absent, nothing about unknown paths is done. `knows` lists paths inside it, relative to its `path`, that
are not declared but belong there; a trailing `/` names a folder and everything in it. `knows` without
`unknown` is refused. For `move-out` the node-engine, after every apply and so on every reconcile, moves
each path inside the directory that nothing declared, no archive unpacked and `knows` does not name,
whole, into `<home>/.local/state/mesh-moved/<module>/<UTC time>/` beside a `<UTC time>.json` list. The
list names each move before it is made. Nothing is deleted. A path younger than a minute waits for the
next look; a socket, pipe or device is left and said. It is only for a directory under an account's
home on one file system with the kept folder, and otherwise the resource fails and nothing moves.
`module` and `program` are read by the module and the program that owns the path, and the node-engine
does nothing for them.
**It converges rather than executes.** Applying twice changes nothing the second time; applying
to a drifted machine returns it. A mode is *maintained*, not merely set — a permission applied
at creation is not a permission held.
+6
View File
@@ -0,0 +1,6 @@
# Release 57: config and provisioning
This branch carries the work of hq design 57 (ADR 0348, ADR 0349) in this repository. Each package
merges here through its own reviewed pull request; the branch reaches main as one pull request, in one
delivery group with the branch of the same name in mesh-controller, mesh-host, mesh-sdk, mesh-catalog
and hq, after a rehearsal in the lab. This file is deleted by the last commit before that pull request.
+6
View File
@@ -116,6 +116,8 @@ const usage = `mesh-bootstrap — make a bare machine into a mesh
--tools-source the repository the shared base is built from
--tools-ref what of it to build (default main)
--tools-go-source the repository the Go toolchain is built from (default: mesh-tools-go beside --tools-source)
--tools-go-ref what of it to build (default main)
--catalog-source the catalogue REPOSITORY, for building its modules;
--catalog is the checkout that says what they are
--catalog-ref what of it to build (default main)
@@ -308,6 +310,10 @@ func newFlagSet(opts *bootstrap.Options, jsonOut *bool) *flag.FlagSet {
"the repository the shared base is built from")
set.StringVar(&opts.ToolsSource.Ref, "tools-ref", opts.ToolsSource.Ref,
"what of it to build (default main)")
set.StringVar(&opts.ToolsGoSource.Repository, "tools-go-source", opts.ToolsGoSource.Repository,
"the repository the Go toolchain is built from (default: mesh-tools-go beside --tools-source)")
set.StringVar(&opts.ToolsGoSource.Ref, "tools-go-ref", opts.ToolsGoSource.Ref,
"what of it to build (default main)")
set.StringVar(&opts.CatalogSource.Repository, "catalog-source", opts.CatalogSource.Repository,
"the catalogue REPOSITORY, for building its modules — --catalog is the checkout that says what they are")
set.StringVar(&opts.CatalogSource.Ref, "catalog-ref", opts.CatalogSource.Ref,
+66
View File
@@ -0,0 +1,66 @@
package main
import (
"context"
"sync/atomic"
"testing"
"time"
"github.com/novox/mesh-host/internal/link"
)
// A container waiting because it said busy is news when it starts waiting and when it is replaced, never on
// its output alone (novox/hq ADR 0335): the walk waiting on this machine starts its gate at the report that
// says it replaced, and a reconcile is otherwise silent.
func TestABusyWaitIsSaidWhenItStartsAndWhenItEnds(t *testing.T) {
quiet := link.Report{Outward: []string{"eth0"}}
waiting := link.Report{Outward: []string{"eth0"}, Busy: []link.Busy{{ID: "build-agent.server",
Container: "mesh-build-agent", State: "waiting", Said: "building a"}}}
still := link.Report{Outward: []string{"eth0"}, Busy: []link.Busy{{ID: "build-agent.server",
Container: "mesh-build-agent", State: "waiting", Said: "building b, a minute on"}}}
replaced := link.Report{Outward: []string{"eth0"}, Busy: []link.Busy{{ID: "build-agent.server",
Container: "mesh-build-agent", State: "replaced-when-idle"}}}
if adoptionFingerprint(waiting) == adoptionFingerprint(quiet) {
t.Error("a container starting to wait is not news")
}
if adoptionFingerprint(still) != adoptionFingerprint(waiting) {
t.Error("a wait whose output changed reads as news on every ask")
}
if adoptionFingerprint(replaced) == adoptionFingerprint(waiting) {
t.Error("a container replaced after it waited is not news")
}
if !worthSaying(link.Report{Busy: replaced.Busy}) {
t.Error("a replacement after a wait is not worth saying")
}
}
// While a container waits, a pass is asked for every so often, not at the reconcile's five minutes; with
// nothing waiting, none (novox/hq ADR 0335 §2).
func TestAWaitIsAskedAgainSoonerThanTheReconcile(t *testing.T) {
if BusyAskEvery >= ReconcileEvery || BusyAskEvery > time.Minute {
t.Fatalf("a waiting container is asked every %s", BusyAskEvery)
}
var reconciles atomic.Int32
queue := &link.Queue{Reconcile: func(context.Context) (link.Report, bool) {
reconciles.Add(1)
return link.Report{}, false
}}
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
go queue.Run(ctx)
busyWaiting.Store(false)
t.Cleanup(func() { busyWaiting.Store(false) })
go busyPasses(ctx, queue, 10*time.Millisecond)
time.Sleep(100 * time.Millisecond)
if n := reconciles.Load(); n != 0 {
t.Fatalf("%d passes were asked with nothing waiting", n)
}
busyWaiting.Store(true)
deadline := time.Now().Add(2 * time.Second)
for reconciles.Load() == 0 && time.Now().Before(deadline) {
time.Sleep(10 * time.Millisecond)
}
if reconciles.Load() == 0 {
t.Fatal("no pass was asked while a container waits")
}
}
+74
View File
@@ -0,0 +1,74 @@
package main
import (
"context"
"crypto/ed25519"
"errors"
"path/filepath"
"github.com/novox/mesh-host/internal/apply"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/filling"
"github.com/novox/mesh-host/internal/store"
"github.com/novox/mesh-host/internal/system"
)
// serveFilling writes filled files on the provisioner client's request for as long as this engine serves (novox/hq
// ADR 0349 §4). A socket that cannot be served is said once, and this engine goes on applying.
func serveFilling(ctx context.Context, opts options, signer ed25519.PublicKey, say func(string)) {
if err := filling.Serve(ctx, filling.Socket, fillingEngine(opts, signer, say), say); err != nil {
say("filled files are not written on request on this machine: " + err.Error())
}
}
// fillingEngine is the write of a filled file as this engine does it: the declaration it keeps, verified, read under
// the apply's lock — the same two locks every apply takes, the process's and the one beside the state — and the file
// written through the apply (apply.WriteFilled), its outcome recorded in the node's state as an apply's is.
func fillingEngine(opts options, signer ed25519.PublicKey, say func(string)) *filling.Engine {
return &filling.Engine{
Declared: func() (*declaration.Declaration, error) {
raw, err := store.LoadDeclared(store.DeclaredPath(opts.state), signer)
if err != nil {
return nil, err
}
return declaration.Parse(raw)
},
UIDOf: filling.UIDOf,
Lock: func() (func(), error) {
applying.Lock()
unlock, err := store.Lock(opts.state, nil)
if err != nil {
applying.Unlock()
return nil, err
}
return func() { unlock(); applying.Unlock() }, nil
},
Place: func(ctx context.Context, d *declaration.Declaration, path string, content []byte) (string, string, error) {
built, err := system.For(builtFor)
if err != nil {
return "", "", err
}
known, err := store.Load(opts.state)
if err != nil {
return "", "", err
}
report, updated, writeErr := apply.WriteFilled(ctx, built, d, known, path, content, apply.ExecRunner,
announceOr(say), apply.KeepIn(filepath.Dir(opts.state)), apply.WindowsIn(filepath.Dir(opts.state)))
// Saved whichever way it went, as an apply's: what was written is on the machine either way.
if err := store.Save(opts.state, updated); err != nil {
return "", "", errors.Join(writeErr, errors.New("written, and the node's state could not be saved: "+
err.Error()))
}
action, digest := "", ""
for _, o := range report.Outcomes {
if o.Target == path && o.Type == string(declaration.TypeFile) {
action = o.Action
}
}
if rec, ok := updated.At(string(declaration.TypeFile), path); ok {
digest = rec.Wrote
}
return action, digest, writeErr
},
}
}
+105
View File
@@ -0,0 +1,105 @@
package main
import (
"context"
"crypto/ed25519"
"encoding/json"
"os"
"os/user"
"path/filepath"
"testing"
"time"
"github.com/novox/mesh-host/internal/filling"
"github.com/novox/mesh-host/internal/store"
)
// Defends novox/hq ADR 0349 §4 (issue 668): a write of a filled file and an apply are one at a time — the write
// takes the lock the apply holds, and writes only once the apply let it go — and the write is recorded in the
// node's state as an apply's outcome is.
func TestAWriteAndAnApplyOfTheSameFileSerialise(t *testing.T) {
was := builtFor
builtFor = "arch"
t.Cleanup(func() { builtFor = was })
me, err := user.Current()
if err != nil {
t.Fatal(err)
}
if me.Uid == "0" {
t.Skip("run as root, whose account is never the provisioner's")
}
dir := t.TempDir()
opts := options{state: filepath.Join(dir, "state.json")}
public, private, err := ed25519.GenerateKey(nil)
if err != nil {
t.Fatal(err)
}
path := filepath.Join(dir, "m.conf")
raw, err := json.Marshal(map[string]any{"declaration": 1, "provisioner_account": me.Username,
"filled_files": []string{path}, "resources": []any{map[string]any{"id": "m.conf", "type": "file",
"path": path, "content": "from the declaration\n", "owner-at-runtime": "owned", "owner": me.Username}}})
if err != nil {
t.Fatal(err)
}
if err := store.SaveDeclared(store.DeclaredPath(opts.state),
store.Declared{Declaration: raw, Signature: ed25519.Sign(private, raw)}); err != nil {
t.Fatal(err)
}
e := fillingEngine(opts, public, func(string) {})
uid := uint32(os.Getuid())
if err := e.Accepts(uid); err != nil {
t.Fatal(err)
}
// An apply holds the lock.
applying.Lock()
done := make(chan filling.Answer)
go func() {
done <- e.Write(context.Background(), uid, filling.Request{Path: path, Content: []byte("filled\n")})
}()
select {
case a := <-done:
applying.Unlock()
t.Fatalf("written while an apply held the lock: %+v", a)
case <-time.After(200 * time.Millisecond):
}
if _, err := os.Stat(path); err == nil {
applying.Unlock()
t.Fatal("the file was written while an apply held the lock")
}
applying.Unlock()
a := <-done
if a.Refused != "" || a.Failed != "" || a.Action != "created" {
t.Fatalf("the write after the apply: %+v", a)
}
got, err := os.ReadFile(path)
if err != nil || string(got) != "filled\n" {
t.Fatalf("the file holds %q (%v)", got, err)
}
known, err := store.Load(opts.state)
if err != nil {
t.Fatal(err)
}
if rec, ok := known.Find("m.conf"); !ok || rec.Wrote != a.Digest || rec.Target != path {
t.Errorf("the write is not recorded in the node's state: %+v, answered %q", rec, a.Digest)
}
// And the cross-process lock beside the state too: a write waits for a `reconcile` run by hand.
unlock, err := store.Lock(opts.state, nil)
if err != nil {
t.Fatal(err)
}
go func() {
done <- e.Write(context.Background(), uid, filling.Request{Path: path, Content: []byte("again\n")})
}()
select {
case a := <-done:
unlock()
t.Fatalf("written while another process held the apply's lock: %+v", a)
case <-time.After(200 * time.Millisecond):
}
unlock()
if a := <-done; a.Action != "updated" {
t.Fatalf("the second write: %+v", a)
}
}
+29
View File
@@ -0,0 +1,29 @@
package main
import (
"testing"
"github.com/novox/mesh-host/internal/link"
"github.com/novox/mesh-host/internal/outward"
"github.com/novox/mesh-host/internal/profile"
)
func reportOn(networks ...outward.Network) link.Report {
return link.Report{Outward: []string{"eth0"},
Profile: profileAsReported(profile.Profile{Machine: &profile.Machine{Networks: networks}})}
}
// **A change of the networks a machine is attached to is said unasked** (novox/hq ADR 0340 §7): a port
// reached from the local network is admitted from them, so a laptop that joins another network is said at
// the next reconcile, not at the next push — and the same networks again are not news.
func TestAChangedNetworkIsSaidUnasked(t *testing.T) {
w := &adoptionWatch{}
home := reportOn(outward.Network{Link: "eth0", Network: "192.168.1.0/24"})
w.said(home)
if w.differs(reportOn(outward.Network{Link: "eth0", Network: "192.168.1.0/24"})) {
t.Fatal("the same networks read as a change, so the machine would speak on every reconcile")
}
if !w.differs(reportOn(outward.Network{Link: "eth0", Network: "10.20.0.0/16"})) {
t.Fatal("a machine that moved to another network does not say so until it is pushed to")
}
}
+130 -8
View File
@@ -24,6 +24,7 @@ import (
"sort"
"strings"
"sync"
"sync/atomic"
"syscall"
"text/tabwriter"
"time"
@@ -1129,7 +1130,7 @@ func runLink(ctx context.Context, opts options) error {
// The one liveness judge on this machine (novox/hq ADR 0240), made before anything applies: every
// report this host makes carries what it says.
if j, err := liveness.Open(filepath.Join(filepath.Dir(opts.state), liveness.FileName),
&liveness.Exec{Run: apply.ExecRunner}); j != nil {
&liveness.Exec{Run: apply.ExecRunner, System: builtFor}); j != nil {
if err != nil {
say(err.Error())
}
@@ -1241,6 +1242,16 @@ func runLink(ctx context.Context, opts options) error {
Say: say,
Timeout: opts.timeout,
}
// And the move of a value sealed to this machine into the vault, asked by the controller (design 57's V3,
// issue 676): the vault is asked over this queue's link, so the queue is named once it exists.
queue.MoveSecret = moveSecretFrom(opts.state, queue)
// And this machine's sealing key stated to the vault in its own name, at start and whenever it changes (ADR 0372,
// issue 786), whether or not any value was ever moved in.
go keepNodeKeyStated(ctx, func() (identity.SealingKey, error) {
return identity.LoadSealingKey(identity.SealingKeyPath(opts.state))
}, func(verb string, args any) (json.RawMessage, error) {
return queue.AskVault(ctx, verb, args)
}, say, nodeKeyRetryFrom, nodeKeyRetryUpTo)
worker := make(chan struct{})
go func() {
defer close(worker)
@@ -1259,6 +1270,7 @@ func runLink(ctx context.Context, opts options) error {
queue.ReconcileDue()
}
go holdTheMachine(ctx, queue)
go busyPasses(ctx, queue, BusyAskEvery)
// And whether what it runs stays up, looked at on its own clock and said when it changes (novox/hq
// ADR 0240) — between applies, which is when a container crash-loops.
if judging != nil {
@@ -1278,6 +1290,9 @@ func runLink(ctx context.Context, opts options) error {
// And mesh-cli is answered on this machine: the account from the kernel, the line asked of the controller
// (novox/hq ADR 0272).
go serveMeshCLI(aside, queue, say)
// And a filled file is written on the provisioner client's request, one at a time with the apply (novox/hq ADR
// 0349 §4).
go serveFilling(aside, opts, mine.Membership.Signer, say)
held := link.HoldRoused(aside, membership, queue, say, opts.timeout, rousedBySignal(ctx))
// The act in hand finishes and is kept before this process exits: an apply that stood aside with
@@ -1397,6 +1412,18 @@ func adoptionFingerprint(r link.Report) string {
// it could only speak when a declaration arrived, and a declaration cannot be composed until it
// has spoken: a machine waiting for a push that is waiting for the machine.
parts := []string{"firewall=" + r.Firewall, "outward=" + strings.Join(r.Outward, ",")}
// **And the networks those links are on** (novox/hq ADR 0340 §7): a port reached from the local
// network is admitted from them, so a laptop that joins another network, or a lease that moves, is
// said at the next reconcile rather than at the next push or the next day.
if networks := attachedNetworksOf(r); networks != "" {
parts = append(parts, "networks="+networks)
}
// And a container waiting because it said busy, or replaced after it waited (novox/hq ADR 0335): the walk
// waiting on this machine starts its gate at the report that says it replaced. Not its output, which
// may change on every ask.
for _, b := range r.Busy {
parts = append(parts, "busy "+b.ID+"="+b.State)
}
for _, h := range r.Held {
parts = append(parts, "held "+h.ID+"="+h.Changed)
}
@@ -1428,6 +1455,26 @@ func adoptionFingerprint(r link.Report) string {
return strings.Join(parts, "\n")
}
// attachedNetworksOf is the networks a report's profile says the machine is attached to, as one line: the
// part of what the machine is made of that the filter is composed from (novox/hq ADR 0340).
func attachedNetworksOf(r link.Report) string {
machine, ok := r.Profile["machine"].(map[string]any)
if !ok {
return ""
}
networks, ok := machine["networks"].([]any)
if !ok {
return ""
}
said := make([]string, 0, len(networks))
for _, n := range networks {
if entry, ok := n.(map[string]any); ok {
said = append(said, fmt.Sprintf("%v %v", entry["link"], entry["network"]))
}
}
return strings.Join(said, ",")
}
// applyState is how an apply went, as the fingerprint keeps it: refused, clean, or failed with what it did not
// apply and which keys its failure is said under.
func applyState(r link.Report) string {
@@ -1613,10 +1660,10 @@ func declaredResolvConf(d *declaration.Declaration) (string, string, bool) {
}
for _, r := range d.Resources {
f, ok := r.(*declaration.File)
if !ok || f.Path != network.ResolvConf || f.CreateOnce || f.Into != "" {
if !ok || f.Path != network.ResolvConf || f.Seeded() || f.Into != "" {
continue
}
module, ok := liveness.ModuleOf(f.ID)
module, ok := d.ModuleOfID(f.ID)
if !ok {
return "", "", false
}
@@ -1735,8 +1782,13 @@ func judgeWhatRuns(ctx context.Context, j *liveness.Judge, queue *link.Queue, sa
func healthAsReported(st liveness.Statement, ns *network.Statement) *link.Health {
// LibraryContract: this engine reads a resource's declared `health` and judges it (ADR 0240 Phase B), judges a
// user's declared `root` (novox/hq ADR 0266), and reads a declaration's identities, groups and libraries
// (novox/hq ADR 0328), which is what tells the controller it may be sent any of those fields.
h := &link.Health{Contract: link.OwnGroupContract, At: st.At.UTC(), Resources: []link.ResourceHealth{}}
// (novox/hq ADR 0328), a container's busy (ADR 0335) and a file's owner-at-runtime and the filled files (ADR
// 0349 §5) and the provisioner client's account (ADR 0349 §4), which is what tells the controller it may be sent
// any of those fields; and it writes a filled file only on the client's word (HandedContract, design 57 P7),
// which is what tells it it may hand a moved module's files to the provisioner here; and it opens the vault's box
// for a core credential (VaultBoxContract, ADR 0349 §16), which is what tells it it may send that box here; and it states its sealing key to the vault (NodeKeyContract, ADR
// 0372, issue 786).
h := &link.Health{Contract: link.NodeKeyContract, At: st.At.UTC(), Resources: []link.ResourceHealth{}}
for _, r := range st.Resources {
h.Resources = append(h.Resources, link.ResourceHealth{Module: r.Module, Resource: r.ID, Kind: r.Kind,
Target: r.Target, State: r.State, Reason: r.Reason, Since: r.Since.UTC(), Streak: r.Streak,
@@ -1760,7 +1812,8 @@ func withUnits(h *link.Health, us *units.Statement) *link.Health {
if us == nil || us.State == "" {
return h
}
h.Units = &link.UnitsHealth{State: us.State, Failed: []link.FailedUnit{}, Unread: us.Unread}
h.Units = &link.UnitsHealth{State: us.State, Failed: []link.FailedUnit{}, Unread: us.Unread,
Away: us.Away}
for _, f := range us.Failed {
if f.Module == "" {
h.Units.Failed = append(h.Units.Failed, link.FailedUnit{Unit: f.Unit, Scope: f.Scope, Load: f.Load,
@@ -1868,6 +1921,32 @@ func holdTheMachine(ctx context.Context, queue *link.Queue) {
}
}
// busyWaiting is whether the last apply left a container running because it said busy (novox/hq ADR 0335).
var busyWaiting atomic.Bool
// BusyAskEvery is how often a container waiting because it said busy is asked again (novox/hq ADR 0335 §2):
// each ask is a pass of its own, so a build that ends is followed by its replacement within this, not
// within the reconcile's five minutes — and the bound of 15 minutes is met within this of its end.
const BusyAskEvery = 20 * time.Second
// busyPasses asks for a pass every so often while a container waits because it said busy. **It asks; it
// does not apply**, as holdTheMachine: the queue's worker does, when its turn comes, so a pass never holds
// the machine for the wait, and a delivery waiting is that pass's apply.
func busyPasses(ctx context.Context, queue *link.Queue, every time.Duration) {
ticker := time.NewTicker(every)
defer ticker.Stop()
for {
select {
case <-ctx.Done():
return
case <-ticker.C:
if busyWaiting.Load() {
queue.ReconcileDue()
}
}
}
}
// reconcileKept is the reconcile's act, as the queue's worker runs it: hold the machine to what it
// was last told, read once it is this act's turn, and say what did not go.
//
@@ -1919,7 +1998,7 @@ func worthSaying(report link.Report) bool {
}
return len(report.Held) > 0 || report.Firewall != "" || len(report.Outward) > 0 ||
len(report.Filters) > 0 || report.FoundFirewall != nil || len(report.Windows) > 0 ||
len(report.Rollbacks) > 0
len(report.Rollbacks) > 0 || len(report.Busy) > 0
}
// announceOr is what the apply writes its detail with, given what the caller has to say things with.
@@ -2168,6 +2247,20 @@ func applyAndKeepHeld(ctx context.Context, opts options, raw []byte, signed *sto
for _, w := range outcome.Windows {
report.Windows = append(report.Windows, link.Window{Step: w.Step, Holds: w.Holds, Since: w.Opened, Until: w.Until})
}
// And a container left running because it said busy, or replaced after it waited (novox/hq ADR 0335):
// the walk waiting on this machine reads it. While one waits, this machine asks it again sooner than its
// reconcile (busyPasses).
for _, b := range outcome.Busy {
report.Busy = append(report.Busy, link.Busy{ID: b.ID, Container: b.Container, State: b.State,
Since: b.Since, Until: b.Until, Said: b.Said})
}
for _, m := range outcome.MovedOut {
report.MovedOut = append(report.MovedOut, link.MovedOut{Resource: m.Resource, From: m.From, To: m.To})
}
for _, l := range outcome.Left {
report.Left = append(report.Left, link.Left{Resource: l.Resource, Path: l.Path})
}
busyWaiting.Store(outcome.Waiting())
// And what runs here that nobody asked for (novox/hq ADR 0163).
if strays, err := apply.Strays(ctx, apply.ExecRunner, updated); err != nil {
fmt.Fprintf(os.Stderr, "mesh-host: applied, and could not list what else runs here: %v\n", err)
@@ -2230,6 +2323,9 @@ func applyAndKeepHeld(ctx context.Context, opts options, raw []byte, signed *sto
report.Failed = map[string]string{"apply": applyErr.Error()}
}
report.Failed = withUnanswered(report.Failed, probe)
// And how each module of the declaration came out, for the have keys written beside this report (novox/hq ADR 0429
// §4, issue 1132): not on the wire.
report.Modules = moduleFacts(declared, outcome, applyErr)
// Which resources did not apply, for whether a reconcile's account is news (issue 233): not said on the wire.
report.Unapplied = unapplied(declared, outcome, applyErr)
return report
@@ -2280,7 +2376,7 @@ func sealOpener(statePath string) apply.Unseal {
if err != nil {
return nil, err
}
return key.Unseal(sealed)
return key.Open(sealed)
}
}
@@ -2482,3 +2578,29 @@ func unhealthyWords(r liveness.State) string {
return fmt.Sprintf("%s (%s %s) is unhealthy: %s, %d automatic restart(s) since this engine began watching it",
r.ID, r.Kind, r.Target, r.Reason, r.Restarts)
}
// moduleFacts is how each module of a declaration came out of an apply, with the build the controller sent for it
// (novox/hq ADR 0429 §4, issue 1132): sorted by module so the facts do not move between applies. A module an adopted
// node has not taken is held as found and not applied, so it is no fact.
func moduleFacts(d *declaration.Declaration, outcome apply.Report, applyErr error) []link.ModuleFact {
if d == nil {
return nil
}
results := apply.ModuleResults(d, outcome, applyErr)
facts := make([]link.ModuleFact, 0, len(results))
for module, result := range results {
if _, untaken := untakenModules(d)[module]; untaken {
continue
}
facts = append(facts, link.ModuleFact{Module: module, Build: d.Builds[module], Failed: result.Failed, Detail: result.Detail})
}
sort.Slice(facts, func(i, j int) bool { return facts[i].Module < facts[j].Module })
return facts
}
func untakenModules(d *declaration.Declaration) map[string][]string {
if d.Adoption == nil {
return nil
}
return d.Adoption.Untaken
}
+38
View File
@@ -0,0 +1,38 @@
package main
import (
"errors"
"testing"
"github.com/novox/mesh-host/internal/apply"
"github.com/novox/mesh-host/internal/declaration"
)
// 1132/AC1, AC3: the report carries one fact per module the declaration names, with the build the controller sent for
// it; a module not yet taken on an adopted node is not one.
func TestTheReportCarriesOneFactPerModuleWithItsBuild(t *testing.T) {
d, err := declaration.Parse([]byte(`{"declaration":1,"builds":{"a":"b-1","b":"b-2"},"resources":[
{"id":"a.dir","type":"directory","path":"/srv/a","module":"a"},
{"id":"b.dir","type":"directory","path":"/srv/b","module":"b"}]}`))
if err != nil {
t.Fatal(err)
}
failure := &apply.Error{Resource: "b.dir", Err: errors.New("no space"), Failures: []apply.Failure{{Resource: "b.dir", Err: errors.New("no space")}}}
facts := moduleFacts(d, apply.Report{}, failure)
if len(facts) != 2 || facts[0].Module != "a" || facts[0].Build != "b-1" || facts[0].Failed ||
facts[1].Module != "b" || facts[1].Build != "b-2" || !facts[1].Failed || facts[1].Detail == "" {
t.Fatalf("facts %+v", facts)
}
adopted, err := declaration.Parse([]byte(`{"declaration":1,"adoption":{"taken":["a"],"untaken":{"b":["b.dir"]}},"resources":[
{"id":"a.dir","type":"directory","path":"/srv/a","module":"a"},
{"id":"b.dir","type":"directory","path":"/srv/b","module":"b"}]}`))
if err != nil {
t.Fatal(err)
}
if facts := moduleFacts(adopted, apply.Report{}, nil); len(facts) != 1 || facts[0].Module != "a" {
t.Fatalf("an untaken module is a fact: %+v", facts)
}
if moduleFacts(nil, apply.Report{}, nil) != nil {
t.Fatal("facts without a declaration")
}
}
+113
View File
@@ -0,0 +1,113 @@
package main
import (
"context"
"encoding/json"
"fmt"
"os"
osuser "os/user"
"path/filepath"
"strconv"
"strings"
"syscall"
"testing"
"time"
"github.com/novox/mesh-host/internal/apply"
)
// TestAMoveOutDirectoryKeepsOnlyWhatItKnows drives the node-engine whole through its command (issue 1024, novox/hq
// ADR 0422): a directory under an account's home that says move-out keeps the file the mesh declared and moves a
// file nobody declared into the kept folder, named in a list, and the run says it moved it. It needs no bus: the
// apply is the one a delivery and the five-minute reconcile both run.
func TestAMoveOutDirectoryKeepsOnlyWhatItKnows(t *testing.T) {
me, err := osuser.Current()
if err != nil {
t.Fatal(err)
}
// The user database lists an account as a person's only at uid 1000 or more; root, as the forge's merge check
// runs, is listed under a number of its own (it can give files to it), by the same name.
listedUID := me.Uid
if uid, _ := strconv.Atoi(me.Uid); uid < 1000 || uid == 65534 {
listedUID = "4242"
}
root := t.TempDir()
home := filepath.Join(root, "home", "agent")
dir := filepath.Join(home, ".ssh")
if err := os.MkdirAll(dir, 0o755); err != nil {
t.Fatal(err)
}
passwd := filepath.Join(root, "passwd")
line := fmt.Sprintf("%s:x:%s:%s::%s:/bin/sh\n", me.Username, listedUID, me.Gid, home)
if err := os.WriteFile(passwd, []byte(line), 0o644); err != nil {
t.Fatal(err)
}
t.Cleanup(apply.UseAccountsFileForTest(passwd))
// Independent of the machine: an arch host with a pacman on PATH that answers and holds nothing, as
// TestReplay233 puts one there (it answers with an error); nothing here declares a package.
bin := t.TempDir()
if err := os.WriteFile(filepath.Join(bin, "pacman"), []byte("#!/bin/sh\nexit 0\n"), 0o755); err != nil {
t.Fatal(err)
}
t.Setenv("PATH", bin+string(os.PathListSeparator)+os.Getenv("PATH"))
wasBuilt := builtFor
builtFor = "arch"
t.Cleanup(func() { builtFor = wasBuilt })
stray := filepath.Join(dir, "id_stray")
if err := os.WriteFile(stray, []byte("somebody else's key"), 0o600); err != nil {
t.Fatal(err)
}
old := time.Now().Add(-time.Hour)
if err := os.Chtimes(stray, old, old); err != nil {
t.Fatal(err)
}
pipe := filepath.Join(dir, "agent.sock")
if err := syscall.Mkfifo(pipe, 0o644); err != nil {
t.Fatal(err)
}
if err := os.Chtimes(pipe, old, old); err != nil {
t.Fatal(err)
}
opts := options{state: filepath.Join(root, "state.json")}
raw := []byte(`{"declaration":1,"resources":[` +
`{"id":"ssh.dir","type":"directory","path":"` + dir + `","owner":"` + me.Username + `","unknown":"move-out"},` +
`{"id":"ssh.config","type":"file","path":"` + dir + `/config","content":"Host *\n","owner":"` + me.Username + `"}]}`)
var said []string
report := applyAndKeep(context.Background(), opts, raw, nil, nil, func(s string) { said = append(said, s) })
if report.Refused != "" || len(report.Failed) > 0 {
t.Fatalf("refused or failed: %q %v", report.Refused, report.Failed)
}
if _, err := os.Stat(filepath.Join(dir, "config")); err != nil {
t.Errorf("the declared file is not there: %v", err)
}
if _, err := os.Lstat(stray); err == nil {
t.Error("the file nobody declared stayed in the directory")
}
lists, _ := filepath.Glob(filepath.Join(home, ".local", "state", "mesh-moved", "ssh", "*.json"))
if len(lists) != 1 {
t.Fatalf("one list expected: %v", lists)
}
rawList, err := os.ReadFile(lists[0])
if err != nil {
t.Fatal(err)
}
var entries []struct{ From, To string }
if err := json.Unmarshal(rawList, &entries); err != nil || len(entries) != 1 {
t.Fatalf("the list: %s %v", rawList, err)
}
if got, err := os.ReadFile(entries[0].To); err != nil || string(got) != "somebody else's key" {
t.Errorf("the moved file is not whole in the kept folder: %q %v", got, err)
}
if len(report.MovedOut) != 1 || report.MovedOut[0].From != stray || report.MovedOut[0].To != entries[0].To {
t.Errorf("the report does not carry the move: %+v", report.MovedOut)
}
if len(report.Left) != 1 || report.Left[0].Path != pipe {
t.Errorf("the report does not list the pipe left in place: %+v", report.Left)
}
if !strings.Contains(strings.Join(said, "\n"), "moved "+stray) {
t.Errorf("the run does not say it moved the file: %v", said)
}
}
+131
View File
@@ -0,0 +1,131 @@
package main
import (
"context"
"crypto/rand"
"crypto/subtle"
"encoding/base64"
"encoding/json"
"errors"
"fmt"
"os"
"time"
"golang.org/x/crypto/nacl/box"
"github.com/novox/mesh-host/internal/identity"
"github.com/novox/mesh-host/internal/link"
)
// The move of a value sealed to this machine into the vault (design 57's V3, issue 676; internal/link
// movesecret.go). The controller cannot open what it holds; this machine can open what was sealed to it, and so
// hands it to the vault, sealed to a key the vault made for the call. The value is in this process for that one
// call and written nowhere.
// vaultAsk asks one of the vault's verbs in this engine's own name and answers its result.
type vaultAsk func(verb string, args any) (json.RawMessage, error)
// moveSecretFrom is the engine's move: this machine's sealing key, read for each ask, and the vault asked over
// the link open now.
func moveSecretFrom(statePath string, queue *link.Queue) link.MoveSecretFunc {
return func(m link.MoveSecret) (link.MovedSecret, error) {
key, err := identity.LoadSealingKey(identity.SealingKeyPath(statePath))
if err != nil {
return link.MovedSecret{}, err
}
ask := func(verb string, args any) (json.RawMessage, error) {
return queue.AskVault(context.Background(), verb, args)
}
moved, err := moveIntoTheVault(ask, key, m)
if err != nil {
fmt.Fprintf(os.Stderr, "mesh-host: a value was not moved into the vault: %v\n", err)
}
return moved, err
}
}
// moveIntoTheVault opens the row's value with this machine's sealing key, hands it to the vault's move-in in its two
// steps, and answers what the vault kept once its read-back — sealed to a key made here for this one call — opens
// to the same value.
func moveIntoTheVault(ask vaultAsk, key identity.SealingKey, m link.MoveSecret) (link.MovedSecret, error) {
if m.SealedTo != "" && m.SealedTo != key.Public {
return link.MovedSecret{}, errors.New("the controller's store says this value was sealed to a sealing key this " +
"machine no longer has, so it cannot be opened here")
}
value, err := key.Unseal(m.Sealed)
if err != nil {
return link.MovedSecret{}, fmt.Errorf("the value does not open with this machine's sealing key: %w", err)
}
defer wipe(value)
// This machine's sealing key, stated in its own name: the key the vault seals the core's credentials to for the
// controller to carry here (novox/hq ADR 0349 §16, issue 763), never a key the controller names.
first := map[string]any{"origin": m.Origin, "generation": m.Generation, "sealing_key": key.Public}
if !m.MadeAt.IsZero() {
first["made_at"] = m.MadeAt.UTC().Format(time.RFC3339Nano)
}
if m.Own() {
first["node"], first["module"], first["name"] = m.Node, m.Module, m.Name
} else {
first["provider"], first["provision"] = m.Provider, m.Provision
first["consumer"], first["consumer_module"] = m.Consumer, m.ConsumerModule
if m.Local != "" {
first["local"] = m.Local
}
}
raw, err := ask("move-in", first)
if err != nil {
return link.MovedSecret{}, fmt.Errorf("the vault did not open the move: %w", err)
}
var opened struct {
Call string `json:"call"`
SealTo string `json:"seal_to"`
Secret string `json:"secret"`
}
if err := json.Unmarshal(raw, &opened); err != nil || opened.Call == "" || opened.SealTo == "" {
return link.MovedSecret{}, errors.New("the vault answered no call and no key to seal to")
}
sealed, err := identity.Seal(opened.SealTo, value)
if err != nil {
return link.MovedSecret{}, fmt.Errorf("the vault's key for the call is not a sealing key: %w", err)
}
backPub, backPriv, err := box.GenerateKey(rand.Reader)
if err != nil {
return link.MovedSecret{}, err
}
defer wipe(backPriv[:])
raw, err = ask("move-in", map[string]any{"call": opened.Call, "sealed": sealed,
"read_back_to": base64.StdEncoding.EncodeToString(backPub[:])})
if err != nil {
return link.MovedSecret{}, fmt.Errorf("the vault did not take %s: %w", opened.Secret, err)
}
var kept struct {
Name string `json:"name"`
Generation int `json:"generation"`
Unchanged bool `json:"unchanged"`
ReadBack string `json:"read_back"`
}
if err := json.Unmarshal(raw, &kept); err != nil || kept.Name == "" || kept.ReadBack == "" {
return link.MovedSecret{}, fmt.Errorf("the vault answered %s with no read-back", opened.Secret)
}
blob, err := base64.StdEncoding.DecodeString(kept.ReadBack)
if err != nil {
return link.MovedSecret{}, fmt.Errorf("the vault's read-back of %s is not sealed", kept.Name)
}
back, ok := box.OpenAnonymous(nil, blob, backPub, backPriv)
if !ok {
return link.MovedSecret{}, fmt.Errorf("the vault's read-back of %s does not open with the key made for it", kept.Name)
}
defer wipe(back)
if subtle.ConstantTimeCompare(back, value) != 1 {
return link.MovedSecret{}, fmt.Errorf("the vault keeps %s, but it read back another value than the one handed "+
"in: the move of it did not hold", kept.Name)
}
return link.MovedSecret{Secret: kept.Name, Generation: kept.Generation, Unchanged: kept.Unchanged}, nil
}
func wipe(b []byte) {
for i := range b {
b[i] = 0
}
}
+134
View File
@@ -0,0 +1,134 @@
package main
import (
"crypto/rand"
"encoding/base64"
"encoding/json"
"errors"
"strings"
"testing"
"golang.org/x/crypto/nacl/box"
"github.com/novox/mesh-host/internal/identity"
"github.com/novox/mesh-host/internal/link"
)
// standInVault answers move-in's two steps as the vault does: a key made for the call, the value opened with it,
// and the value kept read back sealed to the key the second step sent. lie makes its read-back another value.
type standInVault struct {
t *testing.T
pub *[32]byte
priv *[32]byte
opened map[string]any
kept string
lie string
refuse string
carried []string
}
func (v *standInVault) ask(verb string, args any) (json.RawMessage, error) {
raw, _ := json.Marshal(args)
v.carried = append(v.carried, string(raw))
if verb != "move-in" {
v.t.Fatalf("asked %s", verb)
}
if v.refuse != "" {
return nil, errors.New(v.refuse)
}
var a map[string]any
_ = json.Unmarshal(raw, &a)
if a["call"] == nil {
v.opened = a
pub, priv, _ := box.GenerateKey(rand.Reader)
v.pub, v.priv = pub, priv
return json.Marshal(map[string]string{"call": "c1", "seal_to": base64.StdEncoding.EncodeToString(pub[:]),
"secret": "own/anchor/postgres/superuser-password"})
}
blob, _ := base64.StdEncoding.DecodeString(a["sealed"].(string))
value, ok := box.OpenAnonymous(nil, blob, v.pub, v.priv)
if !ok {
v.t.Fatal("what the engine sealed does not open with the call's key")
}
v.kept = string(value)
back := v.kept
if v.lie != "" {
back = v.lie
}
key, _ := base64.StdEncoding.DecodeString(a["read_back_to"].(string))
var to [32]byte
copy(to[:], key)
boxed, _ := box.SealAnonymous(nil, []byte(back), &to, rand.Reader)
return json.Marshal(map[string]any{"name": "own/anchor/postgres/superuser-password", "generation": 1,
"read_back": base64.StdEncoding.EncodeToString(boxed)})
}
const theValue = "the-superusers-password"
func aMoveOf(t *testing.T, key identity.SealingKey) link.MoveSecret {
t.Helper()
sealed, err := identity.Seal(key.Public, []byte(theValue))
if err != nil {
t.Fatal(err)
}
return link.MoveSecret{Node: "anchor", By: "the move", MoveSecretAsk: link.MoveSecretAsk{Module: "postgres",
Name: "superuser-password", Origin: "made", Generation: "g-1", Sealed: sealed, SealedTo: key.Public}}
}
// The engine opens what was sealed to its machine, hands it to the vault sealed to the call's key — never in the
// clear — and answers what the vault kept only once the read-back opens to the same value.
func TestTheEngineMovesAValueSealedToItAndChecksTheReadBack(t *testing.T) {
key, err := identity.GenerateSealingKey()
if err != nil {
t.Fatal(err)
}
vault := &standInVault{t: t}
moved, err := moveIntoTheVault(vault.ask, key, aMoveOf(t, key))
if err != nil {
t.Fatal(err)
}
if moved.Secret != "own/anchor/postgres/superuser-password" || moved.Generation != 1 || vault.kept != theValue {
t.Fatalf("moved %+v, the vault kept %q", moved, vault.kept)
}
if vault.opened["module"] != "postgres" || vault.opened["name"] != "superuser-password" || vault.opened["node"] != "anchor" ||
vault.opened["origin"] != "made" || vault.opened["generation"] != "g-1" ||
vault.opened["sealing_key"] != key.Public {
t.Errorf("the first step named %v", vault.opened)
}
if strings.Contains(strings.Join(vault.carried, " "), theValue) {
t.Fatal("the value crossed the bus in the clear")
}
}
// A read-back that is not the value handed in is a failed move, said; so is a value this machine cannot open, and
// the vault's refusal.
func TestTheEngineSaysAMoveThatDidNotHold(t *testing.T) {
key, _ := identity.GenerateSealingKey()
vault := &standInVault{t: t, lie: "something-else"}
if _, err := moveIntoTheVault(vault.ask, key, aMoveOf(t, key)); err == nil ||
!strings.Contains(err.Error(), "read back") {
t.Fatalf("a read-back of another value: %v", err)
}
other, _ := identity.GenerateSealingKey()
if _, err := moveIntoTheVault((&standInVault{t: t}).ask, key, aMoveOf(t, other)); err == nil ||
!strings.Contains(err.Error(), "sealing key") {
t.Fatalf("a value sealed to another key: %v", err)
}
refusing := &standInVault{t: t, refuse: "the move never replaces a value the vault holds"}
if _, err := moveIntoTheVault(refusing.ask, key, aMoveOf(t, key)); err == nil ||
!strings.Contains(err.Error(), "never replaces") {
t.Fatalf("the vault's refusal: %v", err)
}
// A pair credential names both its ends.
pair := aMoveOf(t, key)
pair.Module, pair.Name = "", ""
pair.Provider, pair.Provision, pair.Consumer, pair.ConsumerModule = "anchor", "database", "shanks", "gitea"
ends := &standInVault{t: t}
if _, err := moveIntoTheVault(ends.ask, key, pair); err != nil {
t.Fatal(err)
}
if ends.opened["provider"] != "anchor" || ends.opened["consumer"] != "shanks" || ends.opened["consumer_module"] != "gitea" ||
ends.opened["node"] != nil {
t.Errorf("the pair's first step named %v", ends.opened)
}
}
+62
View File
@@ -0,0 +1,62 @@
package main
import (
"context"
"fmt"
"time"
"github.com/novox/mesh-host/internal/identity"
)
// This machine's sealing key, stated to the vault in the engine's own name (novox/hq ADR 0372, issue 786). The vault
// records it from the machine's own account alone and, when it is a first or a changed key, says the machine's core
// secrets changed so the controller asks for them again sealed to it. Without this a machine that never moved a
// secret in (adopted after feature 57) has no key at the vault. Nothing but the public half crosses.
const (
// nodeKeyRetryFrom and nodeKeyRetryUpTo bound the wait between tries while the vault does not answer: it starts
// after the bus and may not serve the verb yet.
nodeKeyRetryFrom = 5 * time.Second
nodeKeyRetryUpTo = 5 * time.Minute
// nodeKeyLook is how often the key file is read to see whether it changed.
nodeKeyLook = 30 * time.Second
)
// keepNodeKeyStated states the sealing key load answers to the vault at start, retrying with a growing wait from
// retryFrom to retryUpTo while it does not answer, and states it again whenever the key changes. It says one line
// when a key is stated and one when a try fails for another reason than the last, and returns when ctx ends.
func keepNodeKeyStated(ctx context.Context, load func() (identity.SealingKey, error), ask vaultAsk, say func(string),
retryFrom, retryUpTo time.Duration) {
stated, lastFailure := "", ""
wait := retryFrom
fail := func(why string) {
if why != lastFailure {
say("mesh-host: the sealing key was not stated to the vault, trying again: " + why)
lastFailure = why
}
wait = min(wait*2, retryUpTo)
}
look := min(nodeKeyLook, retryFrom)
for {
delay := look
key, err := load()
switch {
case err != nil:
fail(err.Error())
delay = wait
case key.Public != stated:
if _, err := ask("node-key", map[string]string{"key": key.Public}); err != nil {
fail(err.Error())
delay = wait
} else {
stated, lastFailure, wait = key.Public, "", retryFrom
say(fmt.Sprintf("mesh-host: this machine's sealing key %s is stated to the vault", key.Public))
}
}
select {
case <-ctx.Done():
return
case <-time.After(delay):
}
}
}
+112
View File
@@ -0,0 +1,112 @@
package main
import (
"context"
"encoding/json"
"errors"
"sync"
"testing"
"time"
"github.com/novox/mesh-host/internal/identity"
"github.com/novox/mesh-host/internal/link"
)
// Issue 786 (novox/hq ADR 0372): the engine states its own sealing key to the vault's node-key at start, retries
// while the vault does not answer, and states it again when the key changes; the same key is not stated twice.
type keyVault struct {
mu sync.Mutex
stated []string
fail int
verbs []string
}
func (v *keyVault) ask(verb string, args any) (json.RawMessage, error) {
v.mu.Lock()
defer v.mu.Unlock()
v.verbs = append(v.verbs, verb)
if v.fail > 0 {
v.fail--
return nil, errors.New("the vault does not answer")
}
raw, _ := json.Marshal(args)
var a struct {
Key string `json:"key"`
}
_ = json.Unmarshal(raw, &a)
v.stated = append(v.stated, a.Key)
return json.RawMessage(`{"recorded":true}`), nil
}
func (v *keyVault) said() []string {
v.mu.Lock()
defer v.mu.Unlock()
return append([]string(nil), v.stated...)
}
func waitFor(t *testing.T, what string, ok func() bool) {
t.Helper()
for i := 0; i < 400; i++ {
if ok() {
return
}
time.Sleep(5 * time.Millisecond)
}
t.Fatalf("timed out: %s", what)
}
func TestTheEngineStatesItsSealingKeyToTheVaultAtStartRetriesAndRestatesAChange(t *testing.T) {
first, _ := identity.GenerateSealingKey()
second, _ := identity.GenerateSealingKey()
var mu sync.Mutex
current := first
load := func() (identity.SealingKey, error) {
mu.Lock()
defer mu.Unlock()
return current, nil
}
vault := &keyVault{fail: 2}
var lines []string
var lmu sync.Mutex
say := func(l string) { lmu.Lock(); lines = append(lines, l); lmu.Unlock() }
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
go keepNodeKeyStated(ctx, load, vault.ask, say, time.Millisecond, 5*time.Millisecond)
waitFor(t, "the first key stated after two refusals", func() bool { return len(vault.said()) == 1 })
if vault.said()[0] != first.Public {
t.Errorf("stated %q, want the engine's own key %q", vault.said()[0], first.Public)
}
vault.mu.Lock()
verbs := append([]string(nil), vault.verbs...)
vault.mu.Unlock()
for _, v := range verbs {
if v != "node-key" {
t.Errorf("asked %s", v)
}
}
// The same key again is not stated again.
time.Sleep(60 * time.Millisecond)
if n := len(vault.said()); n != 1 {
t.Errorf("the same key was stated %d times", n)
}
mu.Lock()
current = second
mu.Unlock()
waitFor(t, "the changed key stated", func() bool { return len(vault.said()) == 2 })
if vault.said()[1] != second.Public {
t.Errorf("stated %q after the change, want %q", vault.said()[1], second.Public)
}
lmu.Lock()
defer lmu.Unlock()
if len(lines) == 0 {
t.Error("the engine said nothing")
}
}
// The statement a host makes names this contract (issue 786), one above the vault's box.
func TestTheNodeKeyContractIsThirteen(t *testing.T) {
if link.NodeKeyContract != 13 {
t.Errorf("NodeKeyContract %d", link.NodeKeyContract)
}
}
+7 -4
View File
@@ -87,10 +87,13 @@ func TestTheStatementSaysAnAccountJudgedForRoot(t *testing.T) {
Groups: []string{"openrazer"}}, State: accounts.Healthy, Since: at},
}}
h := withAccounts(healthAsReported(liveness.Statement{At: at}, nil), as)
// At RootContract or later: an engine that also reads an identity's runner says OwnGroupContract (novox/hq ADR
// 0334), one above ADR 0328's LibraryContract.
if h.Contract < link.RootContract || link.RootContract != 3 || h.Contract != link.OwnGroupContract ||
link.OwnGroupContract != link.LibraryContract+1 {
// At RootContract or later: an engine that states its sealing key to the vault says NodeKeyContract (ADR 0372, issue 786), one above
// VaultBoxContract (it opens the vault's box for the core's credentials, ADR 0349 §16, issue 755), one above HandedContract (design 57 P7), one above ProvisionerContract (novox/hq ADR
// 0349 §4), one above ADR 0349 §5's OwnershipContract.
if h.Contract < link.RootContract || link.RootContract != 3 || h.Contract != link.NodeKeyContract ||
link.NodeKeyContract != link.VaultBoxContract+1 || link.VaultBoxContract != link.HandedContract+1 ||
link.HandedContract != link.ProvisionerContract+1 || link.ProvisionerContract != link.OwnershipContract+1 || link.OwnershipContract != link.BusyContract+1 ||
link.BusyContract != link.OwnGroupContract+1 {
t.Fatalf("contract %d", h.Contract)
}
got := map[string]string{}
+154
View File
@@ -0,0 +1,154 @@
package main
// Issue 755's proof on a real bus (novox/hq ADR 0349 §16): a core credential the vault sealed to this machine crosses
// the bus inside the declaration, which the controller carries and cannot open, and this engine opens it with its
// sealing key and writes the value. A listener on every subject keeps every byte that crossed; the value is never
// among them. Run against the throwaway bus merge-check.sh starts (MESH_TEST_NATS).
import (
"context"
"encoding/base64"
"encoding/json"
"os"
"path/filepath"
"strings"
"sync"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/seal"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-host/internal/apply"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/identity"
"github.com/novox/mesh-host/internal/link"
"github.com/novox/mesh-host/internal/store"
"github.com/novox/mesh-host/internal/system"
)
const coreCredential = "the-provisioner-clients-bus-password-2025"
func TestNatsTheEngineFillsACoreCredentialFromTheVaultsBoxWithNoValueInTheClear(t *testing.T) {
url := os.Getenv("MESH_TEST_NATS")
if url == "" {
t.Skip("MESH_TEST_NATS unset")
}
// This machine's sealing key, where the engine reads it.
state := filepath.Join(t.TempDir(), "state.json")
key, err := identity.GenerateSealingKey()
if err != nil {
t.Fatal(err)
}
if err := os.WriteFile(identity.SealingKeyPath(state), []byte(key.Private), 0o600); err != nil {
t.Fatal(err)
}
// Every byte that crosses.
tap, err := nats.Connect(url)
if err != nil {
t.Fatal(err)
}
defer tap.Close()
var mu sync.Mutex
var crossed []string
if _, err := tap.Subscribe(">", func(m *nats.Msg) {
mu.Lock()
crossed = append(crossed, m.Subject+" "+string(m.Data))
mu.Unlock()
}); err != nil {
t.Fatal(err)
}
_ = tap.Flush()
// The engine hears its declaration on its own subject.
const node = "vaultbox"
engine, err := nats.Connect(url)
if err != nil {
t.Fatal(err)
}
defer engine.Close()
heard, err := engine.SubscribeSync(link.DeclareSubject(node))
if err != nil {
t.Fatal(err)
}
_ = engine.Flush()
// The vault's answer to the controller: the value in the ADR 0206 box, sealed to this machine's key. The
// controller carries it in the declaration as it is.
pub, err := key.PublicPEM()
if err != nil {
t.Fatal(err)
}
box, err := seal.Seal(coreCredential, pub)
if err != nil {
t.Fatal(err)
}
path := filepath.Join(t.TempDir(), "broker")
body, _ := json.Marshal(map[string]any{"declaration": 1, "resources": []map[string]any{
{"id": "need-provisioner-broker", "type": "file", "path": path, "vault-sealed": box}}})
controller, err := nats.Connect(url)
if err != nil {
t.Fatal(err)
}
defer controller.Close()
if err := controller.Publish(link.DeclareSubject(node), body); err != nil {
t.Fatal(err)
}
_ = controller.Flush()
msg, err := heard.NextMsg(5 * time.Second)
if err != nil {
t.Fatalf("the declaration did not reach the engine: %v", err)
}
d, err := declaration.Parse(msg.Data)
if err != nil {
t.Fatal(err)
}
sys, err := system.For("arch")
if err != nil {
t.Fatal(err)
}
if _, _, err := apply.Apply(context.Background(), sys, d, store.State{}, store.OriginCarried,
func(context.Context, string, ...string) (string, error) { return "", nil }, nil, sealOpener(state)); err != nil {
t.Fatal(err)
}
on, err := os.ReadFile(path)
if err != nil || string(on) != coreCredential {
t.Fatalf("the engine wrote %q, %v", on, err)
}
_ = tap.Flush()
time.Sleep(100 * time.Millisecond)
mu.Lock()
all := strings.Join(crossed, "\n")
mu.Unlock()
if !strings.Contains(all, link.DeclareSubject(node)) || !strings.Contains(all, "vault-sealed") {
t.Fatalf("the listener heard nothing of the declaration:\n%s", all)
}
for _, form := range []string{coreCredential, base64.StdEncoding.EncodeToString([]byte(coreCredential))} {
if strings.Contains(all, form) {
t.Fatal("the core credential crossed the bus in the clear")
}
}
}
// The engine opens a value the controller sealed (the NaCl box) as before, and the vault's box, with one key.
func TestTheEnginesOpenerOpensBothBoxes(t *testing.T) {
state := filepath.Join(t.TempDir(), "state.json")
key, _ := identity.GenerateSealingKey()
if err := os.WriteFile(identity.SealingKeyPath(state), []byte(key.Private), 0o600); err != nil {
t.Fatal(err)
}
open := sealOpener(state)
nacl, _ := identity.Seal(key.Public, []byte("old"))
if got, err := open(nacl); err != nil || string(got) != "old" {
t.Errorf("the controller's box opened to %q, %v", got, err)
}
pub, _ := key.PublicPEM()
box, _ := seal.Seal("new", pub)
raw, _ := json.Marshal(box)
if got, err := open(string(raw)); err != nil || string(got) != "new" {
t.Errorf("the vault's box opened to %q, %v", got, err)
}
}
+1 -1
View File
@@ -49,7 +49,7 @@ func TestTheBundleCarriesTheFoundationAndTheControlPlaneAndNothingElse(t *testin
}
sort.Strings(images)
want := []string{"mesh-controller", "nats", "postgres"}
want := []string{"mesh-controller", "nats", "pgvector"}
if strings.Join(images, ",") != strings.Join(want, ",") {
t.Fatalf("the bundle carries %v; expected exactly %v.\n\n"+
"Adding one is a change to what every first node becomes, and to ADR 0006's "+
+34 -19
View File
@@ -28,11 +28,12 @@
// image, and this file is applied on a machine with no mesh to ask about anything (ADR 0006). The
// registry prefix is the lab's; the digests are upstream's.
//
// The store's data is a NAMED VOLUME, not a directory on the machine. A directory the host
// creates is owned by root, and the database runs as somebody else inside the container — so it
// could not write, and the container crash-looped. A named volume lets the image set up its own
// ownership, and outlives the container, which is what you want for the thing holding the mesh's
// state.
// **The store is raised where the `postgres` module keeps it** (novox/hq ADR 0338): the container
// `postgres`, on the module's pgvector image, its data in the directory /var/lib/mesh-store owned by
// the image's postgres user, its superuser password read from the module's own-secret file — so the
// store step adopts it as it runs. Pinned upstream by the module's own digest, not through the lab's
// registry prefix, because the two must be the same reference (mesh-host internal/bootstrap holds
// them to each other against the module's manifest).
{
"declaration": 1,
@@ -83,17 +84,28 @@
"boot": "enabled",
"restart-on": ["base-filter"]
},
// The store's data is a directory owned by the image's postgres user (999), made before the
// container starts: a directory the runtime made would be root's, and the database could not
// write it. Measured on the module's image 2026-10-13: it initialises a new store on a directory
// made so (novox/hq ADR 0338 §5).
{
"id": "store-data",
"type": "directory",
"path": "/var/lib/mesh-store",
"mode": "0700",
"owner": "999:70"
},
{
"id": "store",
"type": "container",
"name": "mesh-store",
"image": "192.0.2.250:5000/postgres@sha256:7abf537131b66ed5af448d90653abf1679b0c7e9a1f07efdd4c3108a401b259a",
"name": "postgres",
"image": "pgvector/pgvector@sha256:cf134a767f474095eeba57e0117be8e568e011a63f33fbf252f14c9b760f8e6f",
"env": {
"POSTGRES_PASSWORD": "bootstrap",
"PGDATA": "/var/lib/postgresql/data/pgdata"
},
"ports": ["5432:5432"],
"volumes": ["mesh-store-data:/var/lib/postgresql/data"]
"volumes": ["/var/lib/mesh-store:/var/lib/postgresql/data"]
},
// Over TCP, not the socket. While the store initialises it runs a temporary server on the
// socket ONLY, then stops it and starts the real one — so a socket check passes, the action
@@ -102,21 +114,21 @@
{
"id": "store-ready",
"type": "action",
"in": "mesh-store",
"in": "postgres",
"command": ["sh", "-c", "for i in $(seq 1 180); do pg_isready -h 127.0.0.1 -U postgres >/dev/null 2>&1 && exit 0; sleep 1; done; echo 'the store did not answer within 180s; its own last words follow'; pg_isready -h 127.0.0.1 -U postgres; tail -n 20 /var/lib/postgresql/data/log/*.log 2>/dev/null; exit 1"],
"verify": ["pg_isready", "-h", "127.0.0.1", "-U", "postgres"]
},
{
"id": "inventory-database",
"type": "action",
"in": "mesh-store",
"in": "postgres",
"command": ["sh", "-c", "psql -U postgres -c 'CREATE DATABASE inventory'"],
"verify": ["sh", "-c", "psql -U postgres -lqt | cut -d'|' -f1 | grep -qw inventory"]
},
{
"id": "identity-database",
"type": "action",
"in": "mesh-store",
"in": "postgres",
"command": ["sh", "-c", "psql -U postgres -c 'CREATE DATABASE identity'"],
"verify": ["sh", "-c", "psql -U postgres -lqt | cut -d'|' -f1 | grep -qw identity"]
},
@@ -126,20 +138,20 @@
{
"id": "licences-database",
"type": "action",
"in": "mesh-store",
"in": "postgres",
"command": ["sh", "-c", "psql -U postgres -c 'CREATE DATABASE licences'"],
"verify": ["sh", "-c", "psql -U postgres -lqt | cut -d'|' -f1 | grep -qw licences"]
},
{
"id": "context-schemas",
"type": "action",
"command": ["docker", "run", "--rm", "--network", "container:mesh-store",
"command": ["docker", "run", "--rm", "--network", "container:postgres",
"-e", "MESH_STORE_INVENTORY=postgres://postgres:bootstrap@127.0.0.1:5432/inventory?sslmode=disable",
"-e", "MESH_STORE_IDENTITY=postgres://postgres:bootstrap@127.0.0.1:5432/identity?sslmode=disable",
"-e", "MESH_STORE_LICENCES=postgres://postgres:bootstrap@127.0.0.1:5432/licences?sslmode=disable",
"192.0.2.250:5000/mesh-controller@sha256:c67db38439ff0aee242b467486765467bb95801f52175fc5727cc4e437338ace",
"migrate"],
"verify": ["sh", "-c", "docker exec mesh-store psql -U postgres -d inventory -tAc \"select to_regclass('public.node')\" | grep -qx node && docker exec mesh-store psql -U postgres -d identity -tAc \"select to_regclass('public.signing_key')\" | grep -qx signing_key && docker exec mesh-store psql -U postgres -d licences -tAc \"select to_regclass('public.licence')\" | grep -qx licence"]
"verify": ["sh", "-c", "docker exec postgres psql -U postgres -d inventory -tAc \"select to_regclass('public.node')\" | grep -qx node && docker exec postgres psql -U postgres -d identity -tAc \"select to_regclass('public.signing_key')\" | grep -qx signing_key && docker exec postgres psql -U postgres -d licences -tAc \"select to_regclass('public.licence')\" | grep -qx licence"]
},
// **The bus is raised where the `nats` module keeps it** (novox/hq ADR 0333 §1): the container
// `mesh-broker-nats`, its configuration and user list in /var/lib/nats-module/conf, its streams in
@@ -209,17 +221,20 @@
"path": "/var/lib/nats-module/conf/accounts.conf",
"mode": "0600",
"create-once": true,
"content": "// The first user list, carried by the installer because at genesis there is no mesh to\n// compose one. A bootstrap credential, rotated with the store's and replaced by the\n// controller's own composition from its first start onward.\naccounts {\n MESH {\n jetstream: enabled\n users = [\n { user: \"controller\", password: \"$2a$10$AHqJgOifIVbU41KmATiMhuXFs8xa7Wl2HuN4UVBCXdN2jIQzjqApy\", permissions: {\n publish: { allow: [\"$JS.ACK.CONTROL.controller.>\", \"$JS.ACK.EVENTS.controller.>\", \"$JS.ACK.DEAD_LETTER_NOTICES.controller.>\", \"$JS.API.>\", \"$KV.mesh-controller_asked.>\", \"$KV.mesh-controller_calls.>\", \"$KV.mesh-controller_hand-acts.>\", \"$KV.mesh-controller_conditions.>\", \"$KV.mesh-controller_condition-history.>\", \"$KV.mesh-controller_lease.>\", \"$KV.SEAT_MESH_BUILD_MACHINE_cancelled.>\", \"$KV.SEAT_NODE_BUILD_AGENT_cancelled.>\", \"_INBOX.enrol.>\", \"mesh.assignment.>\", \"mesh.mod.*.tool.>\", \"mesh.node.>\", \"mesh.seat.mesh-build-machine.accept.>\", \"mesh.seat.node-build-agent.accept.>\", \"mesh.seat.mesh-build-machine.tool.>\", \"mesh.seat.node-build-agent.tool.>\", \"mesh.seat.mesh-controller.event.applied\", \"mesh.seat.mesh-controller.event.built-before\", \"mesh.seat.mesh-controller.event.refused\", \"mesh.seat.mesh-controller.event.rolled-back\", \"mesh.seat.mesh-controller.event.checked\", \"mesh.seat.mesh-controller.event.plan-moved\", \"mesh.seat.mesh-controller.event.condition-raised\", \"mesh.seat.mesh-controller.event.condition-changed\", \"mesh.seat.mesh-controller.event.condition-cleared\", \"mesh.seat.mesh-controller.event.doctor-heartbeat\", \"mesh.seat.mesh-controller.event.secret-replaced\", \"mesh.seat.mesh-controller.event.healer-acted\", \"$SRV.INFO\", \"mesh.seat.node-intrusion-prevention.tool.banned.*\", \"mesh.seat.node-backup.tool.backed-up.*\", \"mesh.seat.node-backup.tool.now.*\", \"mesh.seat.mesh-delivery.tool.stalled\", \"mesh.seat.mesh-delivery.tool.close\", \"mesh.seat.node-launcher.tool.secret.*\", \"mesh.seat.mesh-delivery.tool.release\", \"mesh.seat.mesh-delivery.tool.stop\", \"mesh.seat.mesh-delivery.tool.times\", \"mesh.seat.mesh-delivery.tool.groups\", \"mesh.seat.node-service-manager.tool.restart.*\", \"mesh.seat.mesh-controller.tool.plans\", \"mesh.events.dead.>\", \"mesh.again.>\"] }\n subscribe: { allow: [\"$JS.API.>\", \"$JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES.>\", \"$JS.EVENT.ADVISORY.CONSUMER.DELETED.>\", \"_DELIVER.controller\", \"_DELIVER.controller.>\", \"_INBOX.controller.>\", \"mesh.control.>\", \"mesh.mod.*.event.provisioner.failing\", \"mesh.mod.*.event.provisioner.recovered\", \"mesh.mod.*.event.provisioner.retirement\", \"mesh.mod.gitea.event.pull.merged\", \"mesh.mod.gitea.event.pull.updated\", \"mesh.mod.mesh-catalog.event.catching-up\", \"mesh.mod.mesh-catalog.event.upgraded\", \"mesh.seat.mesh-build-machine.event.built\", \"mesh.seat.node-build-agent.event.built\", \"mesh.seat.operator-channel.event.decided.mesh-controller\", \"mesh.seat.mesh-controller.tool.>\", \"$SRV.PING\", \"$SRV.INFO\", \"$SRV.PING.mesh-controller\", \"$SRV.PING.mesh-controller.>\", \"$SRV.INFO.mesh-controller\", \"$SRV.INFO.mesh-controller.>\", \"$SRV.STATS\", \"$SRV.STATS.mesh-controller\", \"$SRV.STATS.mesh-controller.>\"] }\n allow_responses: { max: 1, ttl: \"1m\" }\n } }\n ]\n }\n}\n"
"content": "// The first user list, carried by the installer because at genesis there is no mesh to\n// compose one. A bootstrap credential, rotated with the store's and replaced by the\n// controller's own composition from its first start onward.\naccounts {\n MESH {\n jetstream: enabled\n users = [\n { user: \"controller\", password: \"$2a$10$AHqJgOifIVbU41KmATiMhuXFs8xa7Wl2HuN4UVBCXdN2jIQzjqApy\", permissions: {\n publish: { allow: [\"$JS.ACK.CONTROL.controller.>\", \"$JS.ACK.EVENTS.controller.>\", \"$JS.ACK.DEAD_LETTER_NOTICES.controller.>\", \"$JS.API.>\", \"$KV.mesh-controller_asked.>\", \"$KV.mesh-controller_assignments.>\", \"$KV.mesh-controller_calls.>\", \"$KV.mesh-controller_hand-acts.>\", \"$KV.mesh-controller_conditions.>\", \"$KV.mesh-controller_condition-history.>\", \"$KV.mesh-controller_declared-config.>\", \"$KV.mesh-controller_lease.>\", \"$KV.mesh-controller_predefined.>\", \"$KV.SEAT_MESH_BUILD_MACHINE_cancelled.>\", \"$KV.SEAT_NODE_BUILD_AGENT_cancelled.>\", \"_INBOX.enrol.>\", \"mesh.assignment.>\", \"mesh.mod.*.tool.>\", \"mesh.node.>\", \"mesh.seat.mesh-build-machine.accept.>\", \"mesh.seat.node-build-agent.accept.>\", \"mesh.seat.mesh-build-machine.tool.>\", \"mesh.seat.node-build-agent.tool.>\", \"mesh.seat.mesh-controller.event.applied\", \"mesh.seat.mesh-controller.event.built-before\", \"mesh.seat.mesh-controller.event.refused\", \"mesh.seat.mesh-controller.event.rolled-back\", \"mesh.seat.mesh-controller.event.checked\", \"mesh.seat.mesh-controller.event.plan-moved\", \"mesh.seat.mesh-controller.event.assignment-changed\", \"mesh.seat.mesh-controller.event.declared-config-changed\", \"mesh.seat.mesh-controller.event.predefined-changed\", \"mesh.seat.mesh-controller.event.condition-raised\", \"mesh.seat.mesh-controller.event.condition-changed\", \"mesh.seat.mesh-controller.event.condition-cleared\", \"mesh.seat.mesh-controller.event.doctor-heartbeat\", \"mesh.seat.mesh-controller.event.secret-replaced\", \"mesh.seat.mesh-controller.event.healer-acted\", \"$SRV.INFO\", \"mesh.seat.node-intrusion-prevention.tool.banned.*\", \"mesh.seat.node-backup.tool.backed-up.*\", \"mesh.seat.node-backup.tool.now.*\", \"mesh.seat.mesh-delivery.tool.stalled\", \"mesh.seat.mesh-delivery.tool.close\", \"mesh.seat.node-launcher.tool.secret.*\", \"mesh.seat.mesh-delivery.tool.release\", \"mesh.seat.mesh-delivery.tool.stop\", \"mesh.seat.mesh-delivery.tool.times\", \"mesh.seat.mesh-delivery.tool.groups\", \"mesh.seat.node-service-manager.tool.restart.*\", \"mesh.seat.operator-channel.tool.tell\", \"mesh.seat.mesh-controller.tool.plans\", \"mesh.events.dead.>\", \"mesh.again.>\"] }\n subscribe: { allow: [\"$JS.API.>\", \"$JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES.>\", \"$JS.EVENT.ADVISORY.CONSUMER.DELETED.>\", \"_DELIVER.controller\", \"_DELIVER.controller.>\", \"_INBOX.controller.>\", \"mesh.control.>\", \"mesh.mod.*.event.provisioner.failing\", \"mesh.mod.*.event.provisioner.recovered\", \"mesh.mod.*.event.provisioner.retirement\", \"mesh.mod.gitea.event.pull.merged\", \"mesh.mod.gitea.event.pull.updated\", \"mesh.mod.mesh-catalog.event.catching-up\", \"mesh.mod.mesh-catalog.event.upgraded\", \"mesh.seat.mesh-build-machine.event.built\", \"mesh.seat.node-build-agent.event.built\", \"mesh.seat.operator-channel.event.decided.mesh-controller\", \"mesh.seat.mesh-controller.tool.>\", \"$SRV.PING\", \"$SRV.INFO\", \"$SRV.PING.mesh-controller\", \"$SRV.PING.mesh-controller.>\", \"$SRV.INFO.mesh-controller\", \"$SRV.INFO.mesh-controller.>\", \"$SRV.STATS\", \"$SRV.STATS.mesh-controller\", \"$SRV.STATS.mesh-controller.>\"] }\n allow_responses: { max: 1, ttl: \"1m\" }\n } }\n ]\n }\n}\n"
},
// The upstream server, until the broker step adopts it: the `nats` module's own image, built by the
// mesh, takes its place there once, on these same directories (ADR 0333 §3). The published port is the
// mesh, takes its place there once, on these same directories (ADR 0333 §3), as the bus's planned
// step (ADR 0236). The release is the one the module's image is built on (its NATS_BASE, 2.11.17),
// pinned upstream by that same digest: the adoption changes no server release, so it can be undone
// by putting the old image back and is said reversible. The published port is the
// node's bus port, 5671 unless given, which every enrolment token names; the installer gives the
// module the setting that maps its declared 4222 to it (ADR 0333 §2).
{
"id": "broker",
"type": "container",
"name": "mesh-broker-nats",
"image": "192.0.2.250:5000/nats@sha256:b83efabe3e7def1e0a4a31ec6e078999bb17c80363f881df35edc70fcb6bb927",
"image": "nats@sha256:e4bf19f15fd3218814a4e3c9e0064e1334bd8aa20d5984b9f1a0afd084f8cc00",
"ports": ["5671:4222", "127.0.0.1:8222:8222"],
"volumes": ["/var/lib/mesh-broker-nats:/data", "/var/lib/nats-module/conf:/etc/nats:ro", "/var/lib/mesh-broker-tls:/tls:ro"],
"args": ["-c", "/etc/nats/nats.conf", "-js"]
@@ -227,8 +242,8 @@
{
"id": "broker-ready",
"type": "action",
"command": ["sh", "-c", "for i in $(seq 1 60); do docker run --rm --network host --entrypoint sh 192.0.2.250:5000/nats@sha256:b83efabe3e7def1e0a4a31ec6e078999bb17c80363f881df35edc70fcb6bb927 -c 'nc -z 127.0.0.1 5671' >/dev/null 2>&1 && exit 0; sleep 1; done; exit 1"],
"verify": ["sh", "-c", "docker run --rm --network host --entrypoint sh 192.0.2.250:5000/nats@sha256:b83efabe3e7def1e0a4a31ec6e078999bb17c80363f881df35edc70fcb6bb927 -c 'nc -z 127.0.0.1 5671'"]
"command": ["sh", "-c", "for i in $(seq 1 60); do docker run --rm --network host --entrypoint sh nats@sha256:e4bf19f15fd3218814a4e3c9e0064e1334bd8aa20d5984b9f1a0afd084f8cc00 -c 'nc -z 127.0.0.1 5671' >/dev/null 2>&1 && exit 0; sleep 1; done; exit 1"],
"verify": ["sh", "-c", "docker run --rm --network host --entrypoint sh nats@sha256:e4bf19f15fd3218814a4e3c9e0064e1334bd8aa20d5984b9f1a0afd084f8cc00 -c 'nc -z 127.0.0.1 5671'"]
},
{
"id": "control-plane",
+1 -1
View File
@@ -3,7 +3,7 @@ module github.com/novox/mesh-host
go 1.26.0
require (
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261010191001-33917f91ac3c
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261018164924-2b420603fe83
github.com/nats-io/nats.go v1.54.0
golang.org/x/crypto v0.57.0
golang.org/x/sys v0.48.0
+2 -6
View File
@@ -1,9 +1,5 @@
git.novox.be/novox/mesh-sdk/go v0.1.11-0.20261009143344-f047d0a4a970 h1:9tFDQsgmI+4X7/BpZGXIr+HemPKE7YddYGqWV0lINAI=
git.novox.be/novox/mesh-sdk/go v0.1.11-0.20261009143344-f047d0a4a970/go.mod h1:GFuZUElBZ9A++mxgIKo97aXXo+kV0uJ/UkbhQPPIbrY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261010190644-622b6b4f556d h1:BR8uwv4BqyLaBwITuLylrd9DRUnu0D5DjJw0RuQSXbk=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261010190644-622b6b4f556d/go.mod h1:GFuZUElBZ9A++mxgIKo97aXXo+kV0uJ/UkbhQPPIbrY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261010191001-33917f91ac3c h1:l5onJwoIeH8yE/PjVlEeoPyXBsHp2NQiWLllz2fwX7s=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261010191001-33917f91ac3c/go.mod h1:GFuZUElBZ9A++mxgIKo97aXXo+kV0uJ/UkbhQPPIbrY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261018164924-2b420603fe83 h1:BrHyE3ifFMUQLIza6I/iNlEgdcysB9/Y+1FO6Q9xws8=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261018164924-2b420603fe83/go.mod h1:GFuZUElBZ9A++mxgIKo97aXXo+kV0uJ/UkbhQPPIbrY=
github.com/klauspost/compress v1.20.0 h1:a3C1ke2ohxFymNlb2HWAHjDeKCI90scRskErZkR0ezA=
github.com/klauspost/compress v1.20.0/go.mod h1:LUdAzn7YLVvxLpc7y3V1m40wESHTgc1422pwwBSKYuI=
github.com/nats-io/nats.go v1.54.0 h1:vsXoOxjHp/GmPUN+EcI7uOf/uB+iAP+kEsAFNQN0yzA=
+1 -1
View File
@@ -119,7 +119,7 @@ func secretsOf(d *declaration.Declaration, account string) []string {
var out []string
for _, r := range d.Resources {
f, ok := r.(*declaration.File)
if !ok || f.Path == "" || (f.Sealed == "" && len(f.Secrets) == 0) || f.Owner == account {
if !ok || f.Path == "" || (!f.Secret() && len(f.Secrets) == 0) || f.Owner == account {
continue
}
out = append(out, f.Path)
+158 -46
View File
@@ -15,6 +15,7 @@ import (
"crypto/sha256"
"encoding/base64"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"os"
@@ -92,6 +93,11 @@ type Outcome struct {
// reads is, for a container, the digest of each file it was created reading, by path — so
// the next apply can say which one changed (novox/hq 04-ISSUES/103).
reads map[string]string
// busySince is, for a container left running because it said busy, when the first apply that would
// have replaced it ran (novox/hq ADR 0335): kept in its record, so the bound holds across passes and
// across a restart of the node-engine. busy is what the report says of it.
busySince *time.Time
busy *BusyWait
}
// Report is what an apply did, in the order it did it.
@@ -108,6 +114,24 @@ type Report struct {
// 224): a machine whose server is stopped at 03:31 because its collector is running is
// working, not broken, and the report is where that difference is said.
Windows []Window `json:"windows,omitempty"`
// Busy is every container this apply left running because it said busy, or replaced after it had
// waited (novox/hq ADR 0335): a machine whose walk waits for it says why, and how it ended.
Busy []BusyWait `json:"busy,omitempty"`
// MovedOut is every path a move-out directory moved into its kept folder (issue 1024, novox/hq ADR 0422).
MovedOut []MovedOut `json:"moved-out,omitempty"`
// Left is every socket, pipe or device a move-out directory holds and did not move (ADR 0422 §4).
Left []LeftInPlace `json:"left,omitempty"`
}
// Waiting says whether a container of this apply waits to be replaced because it is busy: what makes the
// node-engine ask again sooner than its reconcile (novox/hq ADR 0335 §2).
func (r Report) Waiting() bool {
for _, b := range r.Busy {
if b.State == BusyWaiting {
return true
}
}
return false
}
// Changed reports whether anything about the machine actually moved. An apply that changed
@@ -118,7 +142,10 @@ func (r Report) Changed() bool {
// whose account's manager is not running, which nothing moved either (novox/hq ADR 0177);
// held still is a container a maintenance window stopped, which the apply did not touch
// (novox/hq issue 224).
if o.Action != "unchanged" && o.Action != "held" && o.Action != "waiting" && o.Action != heldStill {
// waits busy is a container left running because it said busy (novox/hq ADR 0335), which nothing
// moved either.
if o.Action != "unchanged" && o.Action != "held" && o.Action != "waiting" && o.Action != heldStill &&
o.Action != waitsBusy {
return true
}
}
@@ -152,6 +179,23 @@ type Error struct {
// (novox/hq ADR 0136) or a directory the engine refused (issue 495). Named, because "everything
// was attempted" is then untrue, and a report that says so after skipping a module lies.
Modules []string
// Failures is every resource that failed, the first among them, each with its error (issue 1132, novox/hq ADR
// 0429 §4): Others counts the rest and names none, and a module's state on the bus names what failed in it.
Failures []Failure
}
// Failure is one resource that failed in an apply, and why.
type Failure struct {
Resource string
Err error
}
// ModuleResult is how one module's part of an apply ended (issue 1132).
type ModuleResult struct {
Failed bool
// Detail names each failing resource with its error, or why the module was not attempted.
Detail string
}
func (e *Error) Error() string {
@@ -566,6 +610,8 @@ func ApplyMindingWindows(
// is a different thing — one is "this machine could not do it", the other is "this was never
// a declaration", and they are fixed in different places.
var failures []*Error
// Which resources applied without error in this run, for the move-out pass after the loop (issue 1024).
appliedOK := map[string]bool{}
// Modules whose own step did not complete. What follows *within such a module* is not attempted;
// the rest of the machine is (novox/hq ADR 0136).
gated := map[string]bool{}
@@ -718,6 +764,7 @@ func ApplyMindingWindows(
first := failures[0]
first.Done = report
first.Others = len(failures) - 1
first.Failures = failuresOf(failures)
first.Modules = gatedModules()
log(" kept " + guardPrefix + "*: the guard is not up, so what it replaces was left in force")
return report, known, first
@@ -729,7 +776,7 @@ func ApplyMindingWindows(
// **A module whose step did not complete is skipped from there on** (novox/hq ADR 0136).
// Reported rather than passed over in silence: "not attempted" and "nothing to do" are
// different answers, and only one of them is somebody's to fix.
if module, ours := moduleOf(resource.Identity()); ours && gated[module] {
if module, ours := d.ModuleOfID(resource.Identity()); ours && gated[module] {
report.Outcomes = append(report.Outcomes, Outcome{
ID: resource.Identity(), Type: string(resource.Kind()), Target: resource.Target(),
Action: "skipped", Detail: gatedWhy[module],
@@ -759,6 +806,16 @@ func ApplyMindingWindows(
}
}
// **A filled file is the provisioner client's to write** (novox/hq ADR 0349 §5, design 57 P7): one writer per
// file. The declaration says where it goes and how it is written, and only the write verb (WriteFilled)
// writes it, on the client's word. This apply — the controller's send and the five-minute re-apply alike —
// neither writes it nor records it, and nothing that reads it is moved for it; so only what the verb placed
// is recorded, and only that is removed as an orphan when the file leaves the declaration.
if f, isFile := resource.(*declaration.File); isFile && d.Fills(f.Path) {
report.Outcomes = append(report.Outcomes, leftToTheClient(f, &known, handedFiles, &report, log))
continue
}
// The private network takes over the tunnel it found, ahead of the service that replaces
// it (novox/hq ADR 0105): its configuration kept, its unit stopped and disabled, never
// flushed. A failure here fails the service too — the mesh's interface is not started on a
@@ -881,7 +938,7 @@ func ApplyMindingWindows(
in.refused[filepath.Clean(dir.Path)] = err
}
if dir, ok := resource.(*declaration.Directory); ok && in.refused[filepath.Clean(dir.Path)] != nil {
if module, ours := moduleOf(resource.Identity()); ours {
if module, ours := d.ModuleOfID(resource.Identity()); ours {
gated[module] = true
gatedWhy[module] = fmt.Sprintf("its directory %s was refused (novox/hq issue 495), so "+
"nothing more of this module was attempted", filepath.Clean(dir.Path))
@@ -930,7 +987,7 @@ func ApplyMindingWindows(
//
// An action still gates the machine: the bootstrap is a row of them, each making the
// next possible, and they belong to no module.
if module, ours := moduleOf(resource.Identity()); ours {
if module, ours := d.ModuleOfID(resource.Identity()); ours {
gated[module] = true
gatedWhy[module] = "a step this module declares did not complete"
log(fmt.Sprintf(" gated %s.*: a step it declares did not complete, so the rest "+
@@ -940,6 +997,7 @@ func ApplyMindingWindows(
failed.Stopped = true
failed.Done = report
failed.Others = len(failures) - 1
failed.Failures = failuresOf(failures)
failed.Modules = gatedModules()
return report, known, failed
}
@@ -977,33 +1035,8 @@ func ApplyMindingWindows(
kept, keptMode, keptOwner = from.Kept, from.KeptMode, from.KeptOwner
}
// Only now. The record follows the fact, never leads it.
known.Record(store.Applied{
Origin: origin,
ID: resource.Identity(), Type: string(resource.Kind()),
Target: outcome.Target, AppliedAt: time.Now().UTC(),
Wrote: outcome.wrote,
Into: outcome.into,
Kept: kept,
KeptMode: keptMode,
KeptOwner: keptOwner,
Reads: outcome.reads,
Stateless: outcome.stateless,
Scope: outcome.scope,
User: outcome.user,
Found: outcome.found,
DeclaredUnit: outcome.declared,
Shell: outcome.shell,
Linger: outcome.linger,
Groups: outcome.groups,
Unpacked: outcome.unpacked,
AsFound: outcome.asFound,
Declared: outcome.asFoundDeclared,
FoundAt: outcome.asFoundAt,
Holds: holds(resource),
})
if outcome.found != nil {
known.DropFound(resource.Identity())
}
recordApplied(&known, origin, resource, outcome, kept, keptMode, keptOwner)
appliedOK[resource.Identity()] = true
// Its module has been taken, and what was held for it is now the mesh's. A file written
// into, or a service whose lifecycle is the machine's, replaced nothing that was found, so
// its outcome says what the apply did, not that a cutover happened; a hold from when it was
@@ -1046,10 +1079,13 @@ func ApplyMindingWindows(
Action: "forgotten", Detail: detail})
log(fmt.Sprintf(" forgotten %s (%s): %s", from.ID, from.Target, detail))
}
if outcome.Action == heldStill {
if outcome.busy != nil {
report.Busy = append(report.Busy, *outcome.busy)
}
if outcome.Action == heldStill || outcome.Action == waitsBusy {
// Not changed: nothing about it moved, so nothing that restarts on it restarts.
log(fmt.Sprintf(" %s %s (%s): %s", outcome.Action, outcome.ID, outcome.Target, outcome.Detail))
} else if outcome.Action != "unchanged" {
} else if moved(outcome) {
changed[resource.Identity()] = true
// With the detail, when there is one: "updated app" says a container was replaced;
// which file made that happen is what somebody reading the log at the time needs
@@ -1072,6 +1108,15 @@ func ApplyMindingWindows(
failures = append(failures, identityFailures...)
// A directory or archive that says move-out keeps only what it knows (issue 1024, novox/hq ADR 0422).
movedOut, leftIn, moveFailures := moveOutPass(d, known, appliedOK, log)
report.MovedOut = append(report.MovedOut, movedOut...)
report.Left = append(report.Left, leftIn...)
for _, f := range moveFailures {
f.Done = report
failures = append(failures, f)
}
// A replacement that did not apply — failed, skipped behind its module's step, held — leaves
// what it replaces running and recorded, said, for the next apply to hand over.
unhanded := make([]string, 0, len(handover))
@@ -1136,12 +1181,54 @@ func ApplyMindingWindows(
first := failures[0]
first.Done = report
first.Others = len(failures) - 1
first.Failures = failuresOf(failures)
first.Modules = gatedModules()
return report, known, first
}
return report, known, nil
}
// recordApplied records what applying a resource did, once it is done: the record follows the fact, never leads it.
// kept, keptMode and keptOwner are where the original of what it replaced was kept, and how it was found. What an
// earlier apply found its unit as goes with it.
func recordApplied(known *store.State, origin string, resource declaration.Resource, outcome Outcome,
kept, keptMode, keptOwner string) {
known.Record(store.Applied{
Origin: origin,
ID: resource.Identity(), Type: string(resource.Kind()),
Target: outcome.Target, AppliedAt: time.Now().UTC(),
Wrote: outcome.wrote,
Into: outcome.into,
Kept: kept,
KeptMode: keptMode,
KeptOwner: keptOwner,
Reads: outcome.reads,
BusySince: outcome.busySince,
Stateless: outcome.stateless,
Scope: outcome.scope,
User: outcome.user,
Found: outcome.found,
DeclaredUnit: outcome.declared,
Shell: outcome.shell,
Linger: outcome.linger,
Groups: outcome.groups,
Unpacked: outcome.unpacked,
AsFound: outcome.asFound,
Declared: outcome.asFoundDeclared,
FoundAt: outcome.asFoundAt,
Holds: holds(resource),
})
if outcome.found != nil {
known.DropFound(resource.Identity())
}
}
// moved reports whether applying a resource moved it, so what restarts or reloads on it does: anything but
// unchanged, and not a container held still or waiting for its busy one to say idle.
func moved(outcome Outcome) bool {
return outcome.Action != "unchanged" && outcome.Action != heldStill && outcome.Action != waitsBusy
}
// afterWhatTheyRead is the resources in declared order, except that **a service comes after every
// resource it names under restart-on or reload-on** (novox/hq issue 260).
//
@@ -1210,7 +1297,9 @@ func afterWhatTheyRead(resources []declaration.Resource) []declaration.Resource
const guardPrefix = declaration.AdoptionPrefix + "guard"
// Unseal opens a value the mesh sealed to this node. Nil when the node has no sealing key, which
// makes every sealed file an error rather than a silently skipped one.
// makes every sealed file an error rather than a silently skipped one. A value the vault sealed (a core
// credential, novox/hq ADR 0349 §16) is given as its box's JSON, which begins with `{`; one the controller sealed
// is base64, which never does (identity.SealingKey.Open).
type Unseal func(sealed string) ([]byte, error)
func applyOne(ctx context.Context, sys system.System, r declaration.Resource, run Runner,
@@ -1550,7 +1639,16 @@ func applyFile(r *declaration.File, previous store.Applied, unseal Unseal, keepF
return out, fmt.Errorf(
"%s is sealed to this node and this node has no sealing key", r.Path)
}
opened, err := unseal(r.Sealed)
sealed := r.Sealed
if r.VaultSealed != nil {
// The vault's box, handed to the opener as its JSON (Unseal).
raw, err := json.Marshal(r.VaultSealed)
if err != nil {
return out, err
}
sealed = string(raw)
}
opened, err := unseal(sealed)
if err != nil {
return out, fmt.Errorf("cannot open %s: %w", r.Path, err)
}
@@ -1591,7 +1689,7 @@ func applyFile(r *declaration.File, previous store.Applied, unseal Unseal, keepF
// A seed that is already there is left exactly as it is — whatever has grown in it since is
// not the mesh's to put back (novox/hq issue 035). What this host records is what it once
// wrote, so a later declaration that changes the seed is not mistaken for drift either.
if r.CreateOnce && existed {
if r.Seeded() && existed {
out.wrote = previous.Wrote
if out.wrote == "" {
out.wrote = digestOf(string(existing))
@@ -2642,6 +2740,12 @@ func applyContainer(ctx context.Context, r *declaration.Container, run Runner,
// (novox/hq 04-ISSUES/009). Recreating is how a container gets restart-on, which a service
// already has.
reasons := restartedBy(r.RestartOn, changed)
// **A replacement owed since a busy wait began is still owed** (novox/hq ADR 0335): a restart-on trigger
// counts as changed only in the pass it changed, so a container left running then would otherwise read
// unchanged at the next pass, never be replaced, and drop from the report as if it had been.
if owed := owedSinceBusy(r, previous); owed != "" {
reasons = append(reasons, owed)
}
// What this container was created reading, as last recorded. By its declared id first, and by
// its NAME when that id has no record: the bundle's `store` becomes the postgres module's
@@ -2692,6 +2796,8 @@ func applyContainer(ctx context.Context, r *declaration.Container, run Runner,
// stop, so a container this apply found stopped by a window is one whose window it can see.
if win, held := in.windows.holding(r.Name, time.Now()); held {
out.reads = wasReading
// A replacement owed since a busy wait stays owed through the window (novox/hq ADR 0335).
out.busySince = previous.BusySince
out.Action = heldStill
out.Detail = fmt.Sprintf("held still by the maintenance window of %s since %s; left as it is, "+
"and converged by the first apply after the window closes (novox/hq issue 224)",
@@ -2707,6 +2813,15 @@ func applyContainer(ctx context.Context, r *declaration.Container, run Runner,
out.Action = "unchanged"
return out, nil
case existed:
// **A container that says it is busy is asked before it is replaced** (novox/hq ADR 0335 §2): left
// running while it says busy, within the bound, and replaced on the first pass it says idle. Only a
// running one: a stopped container holds no work in hand.
if r.Busy != nil && before.Running {
if waits := askBeforeReplacing(ctx, r, run, cri, previous, &out); waits {
out.reads = wasReading
return out, nil
}
}
if _, err := run(ctx, cri, "rm", "-f", r.Name); err != nil {
return out, fmt.Errorf("replacing container %s: %w", r.Name, err)
}
@@ -2807,6 +2922,9 @@ func applyContainer(ctx context.Context, r *declaration.Container, run Runner,
default:
out.Detail = "replaced; a container's configuration is fixed when it is created"
}
if out.busy != nil {
out.Detail += "; " + out.busy.Words()
}
}
return out, nil
}
@@ -3598,16 +3716,10 @@ func unitKey(scope, user, unit string) string {
// False for what the mesh declares in its own right: the foundation's resources carry no dot at all,
// and the adoption's are named for the mesh rather than for a module. Both belong to no module, and
// their gate is therefore the machine's.
func moduleOf(identity string) (string, bool) {
if strings.HasPrefix(identity, declaration.AdoptionPrefix) {
return "", false
}
at := strings.LastIndex(identity, ".")
if at <= 0 {
return "", false
}
return identity[:at], true
}
//
// This is the guess from an id alone, for a caller with no declaration in hand; a caller with one asks
// Declaration.ModuleOfID, which reads the module the controller said (issue 1132).
func moduleOf(identity string) (string, bool) { return declaration.ModuleGuess(identity) }
// handOver removes what a process replaces, once the process is running and still is a moment later
// (novox/hq issue 213, ADR 0184). A replacement that is not up keeps what it replaces in place
+153
View File
@@ -0,0 +1,153 @@
package apply
import (
"context"
"fmt"
"strings"
"time"
"unicode/utf8"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/store"
)
// A container that says it is busy is replaced once it is idle, within a bound (novox/hq ADR 0335).
//
// **Asked, never waited on.** The apply asks the container once, and while it says busy leaves it running
// as it is and goes on with everything else of the declaration: the machine's lock is held for one ask,
// never for the wait. The wait is carried between passes in the container's record (store.Applied's
// BusySince), so the next pass — the node-engine asks again sooner than its reconcile while anything waits
// — asks again, and so does a node-engine started again in between: the bound runs from the first apply
// that would have replaced the container, whoever applied it.
// waitsBusy is the action of a container left running because it said busy (novox/hq ADR 0335 §2). Its own
// word rather than "unchanged": its declaration moved, and the machine is not yet what was declared.
const waitsBusy = "waits-busy"
// What a report says of a busy container (novox/hq ADR 0335 §2, §3).
const (
// BusyWaiting is a container left running because it said busy, waiting to be replaced.
BusyWaiting = "waiting"
// BusyReplacedIdle is a container replaced on the first pass it said idle, after it had waited.
BusyReplacedIdle = "replaced-when-idle"
// BusyReplacedAtBound is a container still busy at the bound, replaced anyway.
BusyReplacedAtBound = "replaced-at-bound"
)
// BusyWait is one container a busy command held, as a report says it.
type BusyWait struct {
// ID is the resource's id (`<module>.<resource>`), Container its runtime name.
ID string `json:"id"`
Container string `json:"container"`
// State is BusyWaiting, BusyReplacedIdle or BusyReplacedAtBound.
State string `json:"state"`
// Since is when the first apply that would have replaced it ran; Until when the bound replaces it.
Since time.Time `json:"since"`
Until time.Time `json:"until"`
// Said is the last output its busy command gave.
Said string `json:"said,omitempty"`
}
// Words is the wait in a sentence, for an outcome's detail and the console.
func (b BusyWait) Words() string {
since := b.Since.UTC().Format(time.RFC3339)
said := ""
if b.Said != "" {
said = "; its busy command last said: " + b.Said
}
switch b.State {
case BusyReplacedAtBound:
return fmt.Sprintf("replaced while still busy: it said busy since %s and the bound of %s ran out "+
"(novox/hq ADR 0335 §3)%s", since, declaration.BusyBound, said)
case BusyReplacedIdle:
return fmt.Sprintf("replaced once it said idle, after waiting since %s (novox/hq ADR 0335 §2)%s", since, said)
}
return fmt.Sprintf("busy: left running and replaced on the first pass it says idle, or at %s at the latest; "+
"waiting since %s (novox/hq ADR 0335 §2)%s", b.Until.UTC().Format(time.RFC3339), since, said)
}
// owedSinceBusy is the reason a container still owes the replacement it was left running for, or empty: its
// record keeps when its busy wait began, under the same name.
func owedSinceBusy(r *declaration.Container, previous store.Applied) string {
if previous.BusySince == nil || previous.Target != r.Name {
return ""
}
return "the replacement owed since its busy wait began at " + previous.BusySince.UTC().Format(time.RFC3339)
}
// busyClock is the clock the bound is read on. A test moves it.
var busyClock = time.Now
// busySaidMost is how much of a busy command's output a report carries: its last lines, enough to say what
// it held.
const busySaidMost = 400
// askBeforeReplacing asks a running container that declares busy whether it may be replaced now, and says
// in out what came of it. True when it waits: out is then the whole outcome of this pass. False when the
// caller replaces it — idle, or past the bound — with out carrying the wait it ended, if there was one.
func askBeforeReplacing(ctx context.Context, r *declaration.Container, run Runner, cri string,
previous store.Applied, out *Outcome) bool {
now := busyClock().UTC()
busy, said := askBusy(ctx, r, run, cri)
var since time.Time
waited := previous.BusySince != nil && previous.Target == r.Name
if waited {
since = previous.BusySince.UTC()
// A wait kept from a clock that has since gone back reads as starting now, never as later: the bound
// holds from no later than this pass.
if since.After(now) {
since = now
}
}
wait := BusyWait{ID: r.ID, Container: r.Name, Since: since, Said: said}
switch {
case !busy && !waited:
// Idle at the first ask: replaced at once, as a container that declares nothing is.
return false
case !busy:
wait.State, wait.Until = BusyReplacedIdle, since.Add(declaration.BusyBound)
out.busy = &wait
return false
}
if !waited {
since = now
wait.Since = now
}
wait.Until = since.Add(declaration.BusyBound)
if !now.Before(wait.Until) {
wait.State = BusyReplacedAtBound
out.busy = &wait
return false
}
wait.State = BusyWaiting
out.Action, out.Detail = waitsBusy, wait.Words()
out.busySince, out.busy = &since, &wait
return true
}
// askBusy runs a container's busy command inside it, by its shell, as an exec health check is run: busy is
// an exit of 0 and nothing else. An ask that could not be made, or did not answer within its timeout, is
// not busy: the bound is what keeps a container whose command never answers from being held.
func askBusy(ctx context.Context, r *declaration.Container, run Runner, cri string) (bool, string) {
ask, cancel := context.WithTimeout(ctx, r.Busy.Within())
defer cancel()
output, err := run(ask, cri, "exec", r.Name, "/bin/sh", "-c", r.Busy.Command)
said := strings.TrimSpace(output)
if err != nil && said == "" {
said = err.Error()
}
return err == nil, lastOf(said, busySaidMost)
}
// lastOf is the end of a text, at most n bytes, on one line.
func lastOf(s string, n int) string {
s = strings.Join(strings.Fields(s), " ")
if len(s) <= n {
return s
}
from := len(s) - n
for from < len(s) && !utf8.RuneStart(s[from]) {
from++
}
return "…" + s[from:]
}
+433
View File
@@ -0,0 +1,433 @@
package apply
import (
"context"
"errors"
"path/filepath"
"slices"
"strings"
"sync"
"testing"
"time"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/store"
)
// A container that says it is busy is replaced once it is idle, within a bound (novox/hq ADR 0335). The
// machine of these tests runs two containers, a build agent that declares `busy` and a web server that
// does not; every call that replaces a container is counted.
type aBusyMachine struct {
mu sync.Mutex
// containers is each container's name and the spec it was created with; running whether it runs.
containers map[string]string
busy bool // what the agent's busy command answers
said string // what it prints
asked int
calls []string
}
func newBusyMachine() *aBusyMachine {
return &aBusyMachine{containers: map[string]string{}}
}
func (m *aBusyMachine) run(_ context.Context, _ string, args ...string) (string, error) {
m.mu.Lock()
defer m.mu.Unlock()
switch args[0] {
case "info", "image":
return "27.0\n", nil
case "container": // inspect
name := args[len(args)-1]
spec, there := m.containers[name]
if !there {
return "", errors.New("no such container")
}
return "true\t" + spec, nil
case "exec":
m.asked++
m.calls = append(m.calls, "ask "+args[1])
if strings.Join(args[2:], " ") != "/bin/sh -c mesh-builder busy" {
return "", errors.New("asked something else: " + strings.Join(args[2:], " "))
}
if m.busy {
return m.said + "\n", nil
}
return m.said + "\n", errors.New("exited 1")
case "rm":
name := args[len(args)-1]
m.calls = append(m.calls, "rm "+name)
delete(m.containers, name)
case "run":
name := args[slices.Index(args, "--name")+1]
m.calls = append(m.calls, "create "+name)
for i, a := range args {
if a == "--label" && strings.HasPrefix(args[i+1], specLabel+"=") {
m.containers[name] = strings.TrimPrefix(args[i+1], specLabel+"=")
}
}
}
return "", nil
}
func (m *aBusyMachine) mark() int {
m.mu.Lock()
defer m.mu.Unlock()
return len(m.calls)
}
// since is every call since mark that replaced a container, or asked one.
func (m *aBusyMachine) since(mark int) []string {
m.mu.Lock()
defer m.mu.Unlock()
return append([]string(nil), m.calls[mark:]...)
}
func (m *aBusyMachine) answer(busy bool, said string) {
m.mu.Lock()
defer m.mu.Unlock()
m.busy, m.said = busy, said
}
// twoContainers is the declaration at a version: v moves the agent's image tag through its env, w the web's.
func twoContainers(t *testing.T, agent, web string) *declaration.Declaration {
t.Helper()
return parse(t, `{"declaration":1,"resources":[
{"id":"build-agent.server","type":"container","name":"mesh-build-agent","image":"`+pinned+`",
"env":{"V":"`+agent+`"},"busy":{"command":"mesh-builder busy","timeout":"5s"}},
{"id":"web.server","type":"container","name":"web","image":"`+pinned+`","env":{"V":"`+web+`"}}
]}`)
}
// atClock moves the bound's clock for one test.
func atClock(t *testing.T, at *time.Time) {
t.Helper()
was := busyClock
busyClock = func() time.Time { return *at }
t.Cleanup(func() { busyClock = was })
}
func busyOf(r Report, id string) (BusyWait, bool) {
for _, b := range r.Busy {
if b.ID == id {
return b, true
}
}
return BusyWait{}, false
}
func applyBusy(t *testing.T, m *aBusyMachine, d *declaration.Declaration, known store.State) (Report, store.State) {
t.Helper()
report, state, err := Apply(context.Background(), archHost(t), d, known, store.OriginDeclared, m.run, nil, nil)
if err != nil {
t.Fatalf("the apply failed: %v", err)
}
return report, state
}
// The decision's first check: an apply that would replace a container whose busy command exits 0 leaves it
// running, applies the rest, and replaces it on the first pass the command exits non-zero.
func TestABusyContainerIsLeftRunningAndReplacedOnTheFirstPassItSaysIdle(t *testing.T) {
now := time.Date(2026, 10, 13, 1, 0, 0, 0, time.UTC)
atClock(t, &now)
m := newBusyMachine()
_, state := applyBusy(t, m, twoContainers(t, "1", "1"), store.State{})
// A merge moves both: the agent holds a build, the web server holds nothing.
m.answer(true, "building novox/mesh-controller (check of #280) since 00:58")
mark := m.mark()
report, state := applyBusy(t, m, twoContainers(t, "2", "2"), state)
if got := m.since(mark); !slices.Equal(got, []string{"ask mesh-build-agent", "rm web", "create web"}) {
t.Fatalf("a busy agent was not left running while the rest applied: %v", got)
}
o := outcomeOf(report, "build-agent.server")
if o.Action != waitsBusy {
t.Fatalf("a busy container was reported %q, want %q: %+v", o.Action, waitsBusy, o)
}
for _, says := range []string{"busy", "first pass it says idle", "check of #280", "ADR 0335"} {
if !strings.Contains(o.Detail, says) {
t.Errorf("the outcome does not say %q: %s", says, o.Detail)
}
}
if (Report{Outcomes: []Outcome{o}}).Changed() {
t.Errorf("a container left running reads as a change: %+v", o)
}
b, said := busyOf(report, "build-agent.server")
if !said || b.State != BusyWaiting || !b.Since.Equal(now) || !b.Until.Equal(now.Add(15*time.Minute)) ||
b.Container != "mesh-build-agent" || !strings.Contains(b.Said, "check of #280") {
t.Fatalf("the report does not say the agent waits, since when and until when: %+v", report.Busy)
}
if !report.Waiting() {
t.Error("a report with a container waiting does not say so")
}
if rec, _ := state.Find("build-agent.server"); rec.BusySince == nil || !rec.BusySince.Equal(now) {
t.Fatalf("the wait is not kept in the container's record: %+v", rec)
}
// Asked again a minute later, still busy: still left running, the wait counted from the first apply.
now = now.Add(time.Minute)
mark = m.mark()
report, state = applyBusy(t, m, twoContainers(t, "2", "2"), state)
if got := m.since(mark); !slices.Equal(got, []string{"ask mesh-build-agent"}) {
t.Fatalf("a second pass while busy did more than ask: %v", got)
}
if b, _ := busyOf(report, "build-agent.server"); b.State != BusyWaiting || !b.Since.Equal(now.Add(-time.Minute)) {
t.Fatalf("the second pass does not keep the wait from the first apply: %+v", report.Busy)
}
// The build ends: the next pass replaces it, and says it waited.
now = now.Add(time.Minute)
m.answer(false, "idle")
mark = m.mark()
report, state = applyBusy(t, m, twoContainers(t, "2", "2"), state)
if got := m.since(mark); !slices.Equal(got, []string{"ask mesh-build-agent", "rm mesh-build-agent", "create mesh-build-agent"}) {
t.Fatalf("the agent was not replaced on the first pass it said idle: %v", got)
}
if o := outcomeOf(report, "build-agent.server"); o.Action != "updated" || !strings.Contains(o.Detail, "once it said idle") {
t.Errorf("the replacement does not say it waited: %+v", o)
}
if b, _ := busyOf(report, "build-agent.server"); b.State != BusyReplacedIdle {
t.Errorf("the report does not say the agent was replaced when idle: %+v", report.Busy)
}
if report.Waiting() {
t.Error("a report with nothing waiting says something waits")
}
if rec, _ := state.Find("build-agent.server"); rec.BusySince != nil {
t.Errorf("the wait is still kept after the replacement: %+v", rec)
}
// And the pass after that has nothing to say about it.
report, _ = applyBusy(t, m, twoContainers(t, "2", "2"), state)
if len(report.Busy) != 0 {
t.Errorf("a container replaced is still said: %+v", report.Busy)
}
}
// An idle container is replaced at once, as one that declares nothing is: asked once, nothing waits.
func TestAnIdleContainerIsReplacedAtOnce(t *testing.T) {
m := newBusyMachine()
_, state := applyBusy(t, m, twoContainers(t, "1", "1"), store.State{})
m.answer(false, "")
mark := m.mark()
report, _ := applyBusy(t, m, twoContainers(t, "2", "1"), state)
if got := m.since(mark); !slices.Equal(got, []string{"ask mesh-build-agent", "rm mesh-build-agent", "create mesh-build-agent"}) {
t.Fatalf("an idle agent was not replaced at once: %v", got)
}
if len(report.Busy) != 0 {
t.Errorf("an idle container replaced at once is said as a wait: %+v", report.Busy)
}
// And a container whose declaration did not move is never asked.
mark = m.mark()
applyBusy(t, m, twoContainers(t, "2", "1"), state)
if got := m.since(mark); slices.Contains(got, "ask mesh-build-agent") {
t.Errorf("a container nothing replaces was asked: %v", got)
}
}
// Past the bound it is replaced anyway, the report carrying the command's last output — and the bound holds
// across a restart of the node-engine: the wait is read back from the state on disk.
func TestABusyContainerIsReplacedAtTheBoundEvenAfterARestart(t *testing.T) {
now := time.Date(2026, 10, 13, 1, 0, 0, 0, time.UTC)
atClock(t, &now)
m := newBusyMachine()
_, state := applyBusy(t, m, twoContainers(t, "1", "1"), store.State{})
m.answer(true, "building novox/mesh-host (a check that never ends)")
_, state = applyBusy(t, m, twoContainers(t, "2", "1"), state)
// The node-engine stops and starts again: what it knows is what it saved.
path := filepath.Join(t.TempDir(), "state.json")
if err := store.Save(path, state); err != nil {
t.Fatal(err)
}
reloaded, err := store.Load(path)
if err != nil {
t.Fatal(err)
}
now = now.Add(14 * time.Minute)
mark := m.mark()
report, reloaded := applyBusy(t, m, twoContainers(t, "2", "1"), reloaded)
if got := m.since(mark); slices.Contains(got, "rm mesh-build-agent") {
t.Fatalf("a container busy within the bound was replaced after a restart: %v", got)
}
if b, _ := busyOf(report, "build-agent.server"); b.State != BusyWaiting || !b.Since.Equal(now.Add(-14*time.Minute)) {
t.Fatalf("a restart started the wait again: %+v", report.Busy)
}
now = now.Add(time.Minute)
mark = m.mark()
report, _ = applyBusy(t, m, twoContainers(t, "2", "1"), reloaded)
if got := m.since(mark); !slices.Equal(got, []string{"ask mesh-build-agent", "rm mesh-build-agent", "create mesh-build-agent"}) {
t.Fatalf("a container still busy at the bound was not replaced: %v", got)
}
b, _ := busyOf(report, "build-agent.server")
if b.State != BusyReplacedAtBound || !strings.Contains(b.Said, "never ends") {
t.Fatalf("the report does not say it replaced a busy container, with its last output: %+v", report.Busy)
}
if o := outcomeOf(report, "build-agent.server"); !strings.Contains(o.Detail, "still busy") ||
!strings.Contains(o.Detail, "never ends") {
t.Errorf("the outcome does not say a busy container was replaced, with what it said: %s", o.Detail)
}
}
// While one container waits, a send that changes another module on the same machine is applied at once: the
// apply returns after one ask, and holds nothing for the wait.
func TestASendToAnotherModuleIsAppliedWhileOneWaits(t *testing.T) {
now := time.Date(2026, 10, 13, 1, 0, 0, 0, time.UTC)
atClock(t, &now)
m := newBusyMachine()
_, state := applyBusy(t, m, twoContainers(t, "1", "1"), store.State{})
m.answer(true, "building")
_, state = applyBusy(t, m, twoContainers(t, "2", "1"), state)
// The next send moves only the web server.
mark := m.mark()
done := make(chan struct{})
var report Report
go func() {
defer close(done)
report, _ = applyBusy(t, m, twoContainers(t, "2", "2"), state)
}()
select {
case <-done:
case <-time.After(5 * time.Second):
t.Fatal("an apply while a container waits did not return")
}
got := m.since(mark)
if !slices.Contains(got, "create web") || slices.Contains(got, "rm mesh-build-agent") {
t.Fatalf("the other module's send was not applied at once beside the wait: %v", got)
}
if o := outcomeOf(report, "web.server"); o.Action != "updated" {
t.Errorf("the web server was reported %q: %+v", o.Action, o)
}
}
// A container's busy is held to its shape where it arrives.
func TestBusyIsReadAndHeldToItsShape(t *testing.T) {
d := twoContainers(t, "1", "1")
c := d.Resources[0].(*declaration.Container)
if c.Busy == nil || c.Busy.Command != "mesh-builder busy" || c.Busy.Within() != 5*time.Second {
t.Fatalf("busy was not read: %+v", c.Busy)
}
for name, raw := range map[string]string{
"no command": `"busy":{"command":" "}`,
"a long timeout": `"busy":{"command":"x","timeout":"5m"}`,
"not a duration": `"busy":{"command":"x","timeout":"soon"}`,
"an unknown key": `"busy":{"command":"x","every":"1m"}`,
"on a step": `"run-once":true,"busy":{"command":"x"}`,
} {
_, err := declaration.Parse([]byte(`{"declaration":1,"resources":[{"id":"a.b","type":"container",` +
`"name":"b","image":"` + pinned + `",` + raw + `}]}`))
if err == nil {
t.Errorf("%s: a busy of that shape was accepted", name)
}
}
}
// A replacement owed to a restart-on trigger is still owed after the wait: a trigger that is no file or
// directory (a service, an identity's or a library's change) is in no container's spec and counts as changed only
// in the pass it changed, so the container must not read unchanged at the next pass and drop from the report as if
// replaced (novox/hq ADR 0335).
func TestAReplacementOwedToARestartTriggerIsMadeOnceIdle(t *testing.T) {
now := time.Date(2026, 10, 13, 1, 0, 0, 0, time.UTC)
atClock(t, &now)
d := parse(t, `{"declaration":1,"resources":[
{"id":"build-agent.server","type":"container","name":"mesh-build-agent","image":"`+pinned+`",
"restart-on":["identity:build-agent"],"busy":{"command":"mesh-builder busy"}}
]}`)
r := d.Resources[0].(*declaration.Container)
in := inputs{declares: map[string]string{}}
m := newBusyMachine()
m.containers["mesh-build-agent"] = containerSpec(r, in)
ctx := context.Background()
// The trigger moved this pass, and the agent holds a build: left running.
m.answer(true, "building")
out, err := applyContainer(ctx, r, m.run, map[string]bool{"identity:build-agent": true}, in, store.Applied{})
if err != nil || out.Action != waitsBusy || out.busySince == nil {
t.Fatalf("a busy container was not left running for its trigger: %+v %v", out, err)
}
record := store.Applied{ID: r.ID, Target: r.Name, BusySince: out.busySince}
// The next pass: nothing triggers now, and the replacement is still owed.
now = now.Add(time.Minute)
m.answer(false, "idle")
mark := m.mark()
out, err = applyContainer(ctx, r, m.run, map[string]bool{}, in, record)
if err != nil {
t.Fatal(err)
}
if got := m.since(mark); !slices.Equal(got, []string{"ask mesh-build-agent", "rm mesh-build-agent", "create mesh-build-agent"}) {
t.Fatalf("the replacement owed to the trigger was dropped: %v (%+v)", got, out)
}
if out.busy == nil || out.busy.State != BusyReplacedIdle || out.busySince != nil {
t.Fatalf("the replacement does not say it was made once idle: %+v", out)
}
}
// A wait kept from a clock that later went back is never longer than the bound from now.
func TestAWaitFromAClockThatWentBackIsClampedToNow(t *testing.T) {
now := time.Date(2026, 10, 13, 1, 0, 0, 0, time.UTC)
atClock(t, &now)
m := newBusyMachine()
_, state := applyBusy(t, m, twoContainers(t, "1", "1"), store.State{})
m.answer(true, "building")
_, state = applyBusy(t, m, twoContainers(t, "2", "1"), state)
now = now.Add(-time.Hour) // the clock went back an hour
report, _ := applyBusy(t, m, twoContainers(t, "2", "1"), state)
b, _ := busyOf(report, "build-agent.server")
if b.State != BusyWaiting || b.Since.After(now) || !b.Until.Equal(now.Add(15*time.Minute)) {
t.Fatalf("a wait from a later clock is not clamped to now: %+v (now %s)", b, now)
}
}
// An ask that fails — no shell in the image, a timeout, a paused container — is not busy: replaced at once.
func TestAnAskThatFailsIsIdle(t *testing.T) {
m := newBusyMachine()
_, state := applyBusy(t, m, twoContainers(t, "1", "1"), store.State{})
failing := func(ctx context.Context, name string, args ...string) (string, error) {
if args[0] == "exec" {
m.mu.Lock()
m.calls = append(m.calls, "ask "+args[1])
m.mu.Unlock()
return "", errors.New("OCI runtime exec failed: exec: \"/bin/sh\": no such file or directory")
}
return m.run(ctx, name, args...)
}
mark := m.mark()
report, _, err := Apply(context.Background(), archHost(t), twoContainers(t, "2", "1"), state,
store.OriginDeclared, failing, nil, nil)
if err != nil {
t.Fatal(err)
}
if got := m.since(mark); !slices.Equal(got, []string{"ask mesh-build-agent", "rm mesh-build-agent", "create mesh-build-agent"}) {
t.Fatalf("an ask that failed held the container: %v", got)
}
if len(report.Busy) != 0 {
t.Errorf("an ask that failed is said as a wait: %+v", report.Busy)
}
}
// A stopped container holds no work: it is replaced without being asked.
func TestAStoppedContainerIsNotAsked(t *testing.T) {
m := newBusyMachine()
_, state := applyBusy(t, m, twoContainers(t, "1", "1"), store.State{})
m.answer(true, "would say busy")
stopped := func(ctx context.Context, name string, args ...string) (string, error) {
if args[0] == "container" && args[len(args)-1] == "mesh-build-agent" {
out, err := m.run(ctx, name, args...)
return strings.Replace(out, "true\t", "false\t", 1), err
}
return m.run(ctx, name, args...)
}
mark := m.mark()
if _, _, err := Apply(context.Background(), archHost(t), twoContainers(t, "2", "1"), state,
store.OriginDeclared, stopped, nil, nil); err != nil && !strings.Contains(err.Error(), "not running") {
t.Fatal(err)
}
got := m.since(mark)
if slices.Contains(got, "ask mesh-build-agent") || !slices.Contains(got, "rm mesh-build-agent") {
t.Fatalf("a stopped container was asked, or not replaced: %v", got)
}
}
+234
View File
@@ -0,0 +1,234 @@
package apply
import (
"context"
"fmt"
"slices"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/store"
"github.com/novox/mesh-host/internal/system"
)
// WriteFilled writes what the provisioner's client filled into one file the declaration names as filled (novox/hq
// ADR 0349 §4), and makes what reads the file take the change, exactly as an apply that changed it would.
//
// **The declared file says everything but the content.** Its path, mode, owner and who owns it at runtime are the
// declaration's; the client's request carries the content alone. So the file is applied as the declared one with
// that content in place of the declaration's: an owned file whole through applyFile, a seed only when absent, a
// managed file's keys through applyInto or its block through applyBlock — the same guards on where nothing is
// placed, the same Keep of an original found there, the same record. What the declaration seals or fills with
// `${secret:}` the client has already filled, so the content is written as it came; a file that carried a secret
// keeps the mode a secret is written with when the declaration gives none.
//
// **Then what reads it, in the apply's order** (afterWhatTheyRead): a service naming it under restart-on is
// restarted and under reload-on reloaded, a process or container naming it under restart-on is restarted, and a
// running container that reads it at creation — an env-file, a file mounted into it — is recreated, because what
// it read is recorded by digest and this write changed the digest. Each of those is applied and recorded as an
// apply does, and what moves goes on moving what names it. Nothing else of the declaration is touched.
//
// The caller holds the apply's lock: a write and an apply never write one file at once. log is said the path, the
// action and the digest, never the content.
//
// **It does not refresh the runtimes' data roots.** A filled write is one file, not an apply; the placement guard
// judges it with the roots the last apply read (runtimeDataRoots), falling back to the runtimes' configuration
// files when no apply has read them yet — the runtimes themselves are asked once, at an apply's start, not once
// per file (novox/hq issue 1138).
func WriteFilled(
ctx context.Context,
sys system.System,
d *declaration.Declaration,
known store.State,
path string,
content []byte,
run Runner,
log func(string),
keep Keep,
windows *Windows,
) (report Report, _ store.State, _ error) {
if log == nil {
log = func(string) {}
}
if !d.Fills(path) {
return report, known, &Error{Resource: path, Err: fmt.Errorf("this node's declaration does not name %s "+
"as a file the provisioner fills, so nothing was written", path)}
}
var declared []*declaration.File
for _, r := range d.Resources {
if f, ok := r.(*declaration.File); ok && f.Path == path {
declared = append(declared, f)
}
}
if len(declared) != 1 {
// The parser refuses a filled path no file declares; two at one path would need the request to say whose
// keys or block it carries, and it carries the content only.
return report, known, &Error{Resource: path, Err: fmt.Errorf("%d resources declare %s, and a filled file "+
"is written for exactly one, so nothing was written", len(declared), path)}
}
file := filledAs(declared[0], content)
if held, isHeld := known.HeldAt(file.ID); isHeld {
return report, known, &Error{Resource: file.ID, Err: fmt.Errorf("%s is held as found for a module not "+
"yet taken (%s), so nothing was written", path, held.Why)}
}
in := inputs{declares: map[string]string{}, known: &known, windows: windows, away: newStoreAway(windows, log),
groups: groupsWanted(d.Resources), handed: Handed{}}
in.refused = refusedPlaces(d.Resources)
in.agentHomes = rootNeverHomes(d)
in.dirsBefore = map[string]bool{}
// A container that reads the file and mounts a library whose disk or mount is not there is held, as a full apply
// holds it: not created over the missing disk (novox/hq ADR 0328 §9).
in.held = heldLibraries(ctx, d.Libraries)
for _, r := range d.Resources {
in.declares[r.Identity()] = declaredDigest(r)
}
changed := map[string]bool{}
// The file, as the apply loop applies it.
was, _ := known.Find(file.ID)
var keepFound Keep
if (was.ID == "" || was.Target != file.Path) && !known.Recorded(string(declaration.TypeFile), file.Path) {
keepFound = keep
}
outcome, err := applyOne(ctx, sys, file, run, changed, in, was, nil, keepFound)
if err != nil {
log(fmt.Sprintf(" failed %s (%s): %v", file.ID, file.Path, err))
return report, known, &Error{Resource: file.ID, Err: err, Done: report}
}
recordKeeping(&known, file, outcome, was)
report.Outcomes = append(report.Outcomes, outcome)
if !moved(outcome) {
return report, known, nil
}
changed[file.ID] = true
line := fmt.Sprintf(" %s %s (%s), filled: %s", outcome.Action, file.ID, file.Path, outcome.wrote)
if outcome.Detail != "" {
line += ": " + outcome.Detail
}
log(line)
// What reads it, in the apply's order, each moving what names it in turn.
var failures []*Error
for _, r := range afterWhatTheyRead(d.Resources) {
if r.Identity() == file.ID || !readsChange(r, path, changed) {
continue
}
if _, isHeld := known.HeldAt(r.Identity()); isHeld {
log(fmt.Sprintf(" held %s (%s): held as found for a module not yet taken, so it was not restarted "+
"for %s", r.Identity(), r.Target(), file.ID))
continue
}
was, _ := known.Find(r.Identity())
o, err := applyOne(ctx, sys, r, run, changed, in, was, nil, nil)
if err != nil {
if o.found != nil {
known.KeepFound(r.Identity(), store.OriginDeclared, *o.found)
}
failures = append(failures, &Error{Resource: r.Identity(), Err: err})
log(fmt.Sprintf(" failed %s (%s): %v", r.Identity(), o.Target, err))
continue
}
if o.Action == "waiting" {
if o.found != nil {
known.KeepFound(r.Identity(), store.OriginDeclared, *o.found)
}
report.Outcomes = append(report.Outcomes, o)
log(fmt.Sprintf(" waiting %s (%s): %s", o.ID, o.Target, o.Detail))
continue
}
recordKeeping(&known, r, o, was)
report.Outcomes = append(report.Outcomes, o)
if o.busy != nil {
report.Busy = append(report.Busy, *o.busy)
}
if moved(o) {
changed[r.Identity()] = true
line := fmt.Sprintf(" %s %s (%s)", o.Action, o.ID, o.Target)
if o.Detail != "" {
line += ": " + o.Detail
}
log(line)
} else if o.Action != "unchanged" {
log(fmt.Sprintf(" %s %s (%s): %s", o.Action, o.ID, o.Target, o.Detail))
}
}
if len(failures) > 0 {
first := failures[0]
first.Done = report
first.Others = len(failures) - 1
return report, known, first
}
return report, known, nil
}
// filledAs is the declared file with the filled content in place of the declaration's.
func filledAs(declared *declaration.File, content []byte) *declaration.File {
f := *declared
f.Content, f.Sealed, f.VaultSealed, f.Bytes, f.Secrets = string(content), "", nil, "", nil
if f.Mode == "" && (declared.Secret() || len(declared.Secrets) > 0) {
// Written as the declaration would have written it: a file carrying a credential is not world-readable.
f.Mode = "0600"
}
return &f
}
// readsChange reports whether applying r is what makes it take a change of what changed: a service's restart-on or
// reload-on, a process's or container's restart-on, or a running container that reads path at creation.
func readsChange(r declaration.Resource, path string, changed map[string]bool) bool {
switch res := r.(type) {
case *declaration.Service:
return len(restartedBy(res.RestartOn, changed)) > 0 || len(restartedBy(res.ReloadOn, changed)) > 0
case *declaration.Process:
return len(restartedBy(res.RestartOn, changed)) > 0
case *declaration.Container:
if len(restartedBy(res.RestartOn, changed)) > 0 {
return true
}
if res.RunOnce || res.Schedule != "" {
return false
}
if slices.Contains(res.EnvFile, path) {
return true
}
for _, v := range res.Volumes {
if mountSource(v) == path {
return true
}
}
}
return false
}
// recordKeeping records r as applied, carrying where the original it replaced was kept from its record when this
// write kept none — for as long as the resource is recorded at the same target, as the apply loop carries it.
func recordKeeping(known *store.State, r declaration.Resource, o Outcome, was store.Applied) {
kept, keptMode, keptOwner := o.kept, o.keptMode, o.keptOwner
if kept == "" && was.Target == o.Target {
kept, keptMode, keptOwner = was.Kept, was.KeptMode, was.KeptOwner
}
recordApplied(known, store.OriginDeclared, r, o, kept, keptMode, keptOwner)
}
// leftToTheClient is a filled file as an apply leaves it (novox/hq ADR 0349 §5, design 57 P7): untouched, and said
// so. A file the mesh wrote at the same path under another id, going in this apply, is handed to it as an apply hands
// a whole file (novox/hq ADR 0223): its record — what the mesh wrote there, and the original it kept — is the filled
// file's now, so the verb judges what is there against it, and the file is neither removed nor written.
func leftToTheClient(f *declaration.File, known *store.State, handedFiles map[string]store.Applied, report *Report,
log func(string)) Outcome {
if from, handed := handedFiles[f.ID]; handed {
if _, recorded := known.Find(f.ID); !recorded {
rec := from
rec.ID = f.ID
known.Record(rec)
}
delete(handedFiles, f.ID)
known.Forget(from.ID)
detail := "no longer declared; " + f.ID + " declares the file now, filled by the provisioner's client, so " +
"it was handed to it, not removed"
report.Outcomes = append(report.Outcomes, Outcome{ID: from.ID, Type: from.Type, Target: from.Target,
Action: "forgotten", Detail: detail})
log(fmt.Sprintf(" forgotten %s (%s): %s", from.ID, from.Target, detail))
}
return Outcome{ID: f.ID, Type: string(declaration.TypeFile), Target: f.Path, Action: "unchanged",
Detail: "filled by the provisioner's client, which alone has it written"}
}
+218
View File
@@ -0,0 +1,218 @@
package apply
import (
"context"
"fmt"
"os"
"path/filepath"
"strings"
"testing"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/store"
)
// Defends novox/hq ADR 0349 §5, one writer per file (design 57 P7, issue 673): a file the declaration names as
// filled is written only through the write verb (WriteFilled), on the provisioner client's word. An apply — the
// controller's send and the five-minute re-apply alike — never writes it, never records it, and removes it as an
// orphan only when the verb placed it.
// An apply leaves a filled file alone, whatever who owns it at runtime says: absent stays absent and is not
// recorded, and what the verb wrote stays as it wrote it through every re-apply.
func TestAnApplyNeverWritesAFilledFile(t *testing.T) {
for _, c := range []struct{ name, fields, content string }{
{"owned", `"content":"","mode":"0640","owner-at-runtime":"owned"`, "port = 8080\n"},
{"seed", `"content":"","mode":"0600","owner-at-runtime":"seed"`, "user default on\n"},
{"managed block", `"into":"block","content":"","owner-at-runtime":"managed"`, "10.42.0.1 ace\n"},
} {
t.Run(c.name, func(t *testing.T) {
path := filepath.Join(t.TempDir(), "the-file")
d := filledDecl(t, path, c.fields, "")
report, state, err := Apply(context.Background(), archHost(t), d, store.State{}, store.OriginDeclared,
noServices, nil, nil)
if err != nil {
t.Fatal(err)
}
if _, err := os.Stat(path); err == nil {
t.Fatalf("an apply wrote a filled file the client has not filled: %q", readText(t, path))
}
if _, ok := state.Find("m.conf"); ok {
t.Errorf("an apply recorded a filled file it did not write")
}
if o := outcomeOf(report, "m.conf"); o.Action != "unchanged" || !strings.Contains(o.Detail, "provisioner") {
t.Errorf("an apply did not say it left the file to the provisioner's client: %+v", o)
}
_, state, err = WriteFilled(context.Background(), archHost(t), d, state, path, []byte(c.content),
noServices, nil, nil, nil)
if err != nil {
t.Fatal(err)
}
placed, _ := state.Find("m.conf")
want, wantMode := readText(t, path), modeAt(t, path)
// The re-apply, twice over: nothing written, the verb's record kept as it wrote it.
for i := 0; i < 2; i++ {
var said []string
report, state, err = Apply(context.Background(), archHost(t), d, state, store.OriginDeclared,
noServices, func(line string) { said = append(said, line) }, nil)
if err != nil {
t.Fatal(err)
}
if got := readText(t, path); got != want || modeAt(t, path) != wantMode {
t.Fatalf("re-apply %d wrote the filled file again: %q, want %q", i+1, got, want)
}
if o := outcomeOf(report, "m.conf"); o.Action != "unchanged" {
t.Errorf("re-apply %d: %+v", i+1, o)
}
if rec, _ := state.Find("m.conf"); rec.Wrote != placed.Wrote || rec.Target != path {
t.Errorf("re-apply %d changed the verb's record: %+v, was %+v", i+1, rec, placed)
}
if len(said) != 0 {
t.Errorf("re-apply %d said something of an untouched file: %q", i+1, said)
}
}
})
}
}
// What reads a filled file is moved by the verb's write, once, never again by an apply: a service under
// restart-on is not restarted and a container reading it is not recreated by the re-apply.
func TestAnApplyMovesNothingForAFilledFile(t *testing.T) {
env := filepath.Join(t.TempDir(), "forge.env")
d := filledDecl(t, env, `"content":"","mode":"0600","owner-at-runtime":"owned"`,
`{"id":"forge.server","type":"container","name":"forge","image":"`+pinned+`","env-file":["`+env+`"]}`)
m := &machine{containers: map[string]*fakeContainer{}}
_, state, err := WriteFilled(context.Background(), archHost(t), d, store.State{}, env, []byte("PORT=5432\n"),
m.run, nil, nil, nil)
if err != nil {
t.Fatal(err)
}
m.asked = nil
if _, _, err := Apply(context.Background(), archHost(t), d, state, store.OriginDeclared, m.run, nil,
nil); err != nil {
t.Fatal(err)
}
if m.removed("forge") || m.did("docker run") {
t.Errorf("the re-apply recreated the container reading the filled file: %v", m.asked)
}
if got := readText(t, env); got != "PORT=5432\n" {
t.Errorf("the re-apply wrote the filled file: %q", got)
}
conf := filepath.Join(t.TempDir(), "daemon.conf")
d = filledDecl(t, conf, `"content":"","owner-at-runtime":"owned"`,
`{"id":"m.daemon","type":"service","unit":"m.service","state":"running","restart-on":["m.conf"]}`)
var commands []string
_, state, err = WriteFilled(context.Background(), archHost(t), d, store.State{}, conf, []byte("level = 1\n"),
recordingServices(&commands), nil, nil, nil)
if err != nil {
t.Fatal(err)
}
commands = nil
if _, _, err := Apply(context.Background(), archHost(t), d, state, store.OriginDeclared,
recordingServices(&commands), nil, nil); err != nil {
t.Fatal(err)
}
if n := count(commands, "systemctl stop m.service"); n != 0 {
t.Errorf("the re-apply restarted what reads the filled file: %v", commands)
}
}
// A filled file the verb never placed is not the mesh's to remove: one found there, or put there by hand, stays
// when the file leaves the declaration.
func TestAFilledFileTheVerbNeverPlacedIsNotRemoved(t *testing.T) {
path := filepath.Join(t.TempDir(), "the-file")
if err := os.WriteFile(path, []byte("the machine's own\n"), 0o644); err != nil {
t.Fatal(err)
}
d := filledDecl(t, path, `"content":"","owner-at-runtime":"owned"`, "")
_, state, err := Apply(context.Background(), archHost(t), d, store.State{}, store.OriginDeclared, noServices,
nil, nil)
if err != nil {
t.Fatal(err)
}
gone := elsewhere(t)
report, state, err := Apply(context.Background(), archHost(t), gone, state, store.OriginDeclared, noServices,
nil, nil)
if err != nil {
t.Fatal(err)
}
if got := readText(t, path); got != "the machine's own\n" {
t.Errorf("a filled file the verb never placed was touched when it left: %q", got)
}
if o := outcomeOf(report, "m.conf"); o.Action == "removed" {
t.Errorf("a filled file the verb never placed was removed: %+v", o)
}
if _, ok := state.Find("m.conf"); ok {
t.Errorf("a record of the filled file was left")
}
}
// What the verb placed is the mesh's: when the file leaves the declaration it is removed as an orphan, as a file
// an apply wrote is.
func TestAFilledFileTheVerbPlacedIsRemovedWhenItLeaves(t *testing.T) {
path := filepath.Join(t.TempDir(), "the-file")
d := filledDecl(t, path, `"content":"","owner-at-runtime":"owned"`, "")
_, state, err := Apply(context.Background(), archHost(t), d, store.State{}, store.OriginDeclared, noServices,
nil, nil)
if err != nil {
t.Fatal(err)
}
_, state, err = WriteFilled(context.Background(), archHost(t), d, state, path, []byte("port = 8080\n"),
noServices, nil, nil, nil)
if err != nil {
t.Fatal(err)
}
gone := elsewhere(t)
report, state, err := Apply(context.Background(), archHost(t), gone, state, store.OriginDeclared, noServices,
nil, nil)
if err != nil {
t.Fatal(err)
}
if _, err := os.Stat(path); err == nil {
t.Errorf("a filled file the verb placed stayed when it left the declaration: %q", readText(t, path))
}
if o := outcomeOf(report, "m.conf"); o.Action != "removed" {
t.Errorf("not removed: %+v", o)
}
if _, ok := state.Find("m.conf"); ok {
t.Errorf("its record stayed")
}
}
// A file the mesh wrote the old way and that is now filled keeps its record for the verb to judge against, and is
// not written over by the apply that hands it to the client; a file at the same path recorded under another id
// is handed to the filled one, not removed.
func TestAFileThatMovesToTheClientIsHandedNotRewritten(t *testing.T) {
path := filepath.Join(t.TempDir(), "the-file")
old := parseTrusted(t, fmt.Sprintf(`{"declaration":1,"resources":[{"id":"m.old","type":"file","path":%q,`+
`"content":"port = 1\n"}]}`, path))
_, state, err := Apply(context.Background(), archHost(t), old, store.State{}, store.OriginDeclared, noServices,
nil, nil)
if err != nil {
t.Fatal(err)
}
was, _ := state.Find("m.old")
d := filledDecl(t, path, `"content":"","owner-at-runtime":"owned"`, "")
_, state, err = Apply(context.Background(), archHost(t), d, state, store.OriginDeclared, noServices, nil, nil)
if err != nil {
t.Fatal(err)
}
if got := readText(t, path); got != "port = 1\n" {
t.Errorf("the file handed to the client was rewritten or removed: %q", got)
}
if _, ok := state.Find("m.old"); ok {
t.Errorf("the record going was kept beside the filled file's")
}
if rec, ok := state.Find("m.conf"); !ok || rec.Wrote != was.Wrote || rec.Target != path {
t.Errorf("the filled file did not take the record of what the mesh wrote there: %+v", rec)
}
}
// elsewhere is a declaration with nothing at the filled path: one unrelated file.
func elsewhere(t *testing.T) *declaration.Declaration {
t.Helper()
return parseTrusted(t, fmt.Sprintf(`{"declaration":1,"resources":[{"id":"n.other","type":"file","path":%q,`+
`"content":"x\n"}]}`, filepath.Join(t.TempDir(), "other")))
}
+57
View File
@@ -0,0 +1,57 @@
//go:build linux
package apply
import (
"context"
"fmt"
"path/filepath"
"strings"
"testing"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/store"
)
// A container that reads a filled file and mounts a library whose disk is not there is held by the write verb as a
// full apply holds it (novox/hq ADR 0328 §9, issue 668): never created or recreated over the missing disk.
func TestAWriteDoesNotRecreateAContainerHeldForItsLibrary(t *testing.T) {
// No folder is given the group: the library's root is absent, and nothing is made.
gid := 61000
libraries(t, nil, nil)
base := t.TempDir()
root := filepath.Join(base, "gone", "media")
env := filepath.Join(base, "radarr.env")
raw := fmt.Sprintf(`{"declaration":1,"provisioner_account":"mesh-node-provisioner","filled_files":[%q],
"groups":[{"name":"mesh-media","gid":%d}],"libraries":[%s],"resources":[
{"id":"radarr.env","type":"file","path":%q,"content":"","mode":"0600","owner-at-runtime":"owned"},
{"id":"radarr.app","type":"container","name":"radarr","image":%q,"env-file":[%q],"volumes":[%q]}]}`,
env, gid, libJSON("media", root, gid, true, false, "movies"), env, pinned, env,
filepath.Join(root, "movies")+":/movies")
d, err := declaration.Parse([]byte(raw))
if err != nil {
t.Fatal(err)
}
var ran []string
run := func(_ context.Context, name string, args ...string) (string, error) {
ran = append(ran, name+" "+strings.Join(args, " "))
return "", nil
}
var state store.State
for _, content := range []string{"PORT=1\n", "PORT=2\n"} {
var report Report
report, state, err = WriteFilled(context.Background(), archHost(t), d, state, env, []byte(content), run,
nil, nil, nil)
if err != nil {
t.Fatal(err)
}
if o := outcomeOf(report, "radarr.app"); o.Action != "waiting" {
t.Errorf("after writing %q the container is %+v, not held for its library", content, o)
}
}
for _, line := range ran {
if strings.HasPrefix(line, "docker run") || strings.HasPrefix(line, "docker rm") {
t.Errorf("the container was created or removed over a missing disk: %v", ran)
}
}
}
+168
View File
@@ -0,0 +1,168 @@
package apply
import (
"context"
"errors"
"fmt"
"os"
"path/filepath"
"strings"
"testing"
"github.com/novox/mesh-host/internal/store"
)
// A filled write is one file, not an apply: it asks no container runtime, and judges where nothing is placed
// (issue 339) with the roots the last apply read, falling back to the runtimes' configuration files when no apply
// has read them yet (novox/hq issue 1138). These tests set the package's runtimeRoots and AskRuntimes, so none
// runs in parallel with another that reads or sets them, and each restores what it changed.
// setRuntimeRoots makes runtimeDataRoots answer roots, as if an apply had just read them (or, with nil, as if none
// ever had), restoring what a test before it left there once this one is done.
func setRuntimeRoots(t *testing.T, roots []string) {
t.Helper()
runtimeRootsMu.Lock()
before := runtimeRoots
runtimeRoots = roots
runtimeRootsMu.Unlock()
t.Cleanup(func() {
runtimeRootsMu.Lock()
runtimeRoots = before
runtimeRootsMu.Unlock()
})
}
// askNoRuntime makes AskRuntimes record the name of every runtime it is asked for, answering as one that is not
// running does (nothing, no error): were it consulted, that is what would say so. Restored once the test is done.
func askNoRuntime(t *testing.T, asked *[]string) {
t.Helper()
before := AskRuntimes
AskRuntimes = func(_ context.Context, name string, _ ...string) (string, error) {
*asked = append(*asked, name)
return "", nil
}
t.Cleanup(func() { AskRuntimes = before })
}
// aRuntimeConfig points the runtimes' configuration at a fresh, unmounted root below a temp directory, restoring
// the engine's real configuration once the test is done.
func aRuntimeConfig(t *testing.T) (dir, root string) {
t.Helper()
dir = t.TempDir()
root = filepath.Join(dir, "docker")
daemon := filepath.Join(dir, "daemon.json")
if err := os.WriteFile(daemon, []byte(`{"data-root":"`+root+`"}`), 0o644); err != nil {
t.Fatal(err)
}
before := runtimeConfigs
runtimeConfigs = runtimeFiles{daemonJSON: daemon, storageConf: filepath.Join(dir, "none.conf")}
t.Cleanup(func() { runtimeConfigs = before })
return dir, root
}
// refusedAsPlacement fails the test unless err is a PlacementRefusedError.
func refusedAsPlacement(t *testing.T, err error) {
t.Helper()
var refusal *PlacementRefusedError
if err == nil || !errors.As(err, &refusal) {
t.Fatalf("a filled file below a refused runtime root was not refused: %v", err)
}
}
// 1138/AC1: a filled write asks no runtime, and judges with the roots the last apply read.
func TestAFilledWriteAsksNoRuntimeAndUsesTheLastApplysRoots(t *testing.T) {
root := filepath.Join(t.TempDir(), "docker-root")
setRuntimeRoots(t, []string{root}) // as if an apply had just read it
var asked []string
askNoRuntime(t, &asked)
path := filepath.Join(root, "app.conf")
d := filledDecl(t, path, `"content":"","owner-at-runtime":"owned"`, "")
_, _, err := WriteFilled(context.Background(), archHost(t), d, store.State{}, path, []byte("x\n"), noServices,
nil, nil, nil)
refusedAsPlacement(t, err)
if len(asked) != 0 {
t.Errorf("a filled write asked the container runtimes: %v", asked)
}
}
// 1138/AC2: given a real apply that read a runtime data root, a filled write below it is refused as today, with
// no apply run again in between.
func TestAFilledWriteBelowARootARealApplyReadIsRefused(t *testing.T) {
dir, root := aRuntimeConfig(t)
t.Cleanup(func() {
runtimeRootsMu.Lock()
runtimeRoots = nil
runtimeRootsMu.Unlock()
})
// A full apply of something unrelated reads the runtimes' configuration once, at its start.
elsewhere := filepath.Join(dir, "elsewhere.conf")
if _, _, err := Apply(context.Background(), archHost(t), declare(t,
fmt.Sprintf(`{"id":"m.other","type":"file","path":%q,"content":"x\n"}`, elsewhere)),
store.State{}, store.OriginDeclared, noServices, nil, nil); err != nil {
t.Fatal(err)
}
// A filled write below the runtime's root, with no apply run since, is still refused.
path := filepath.Join(root, "sub", "app.conf")
d := filledDecl(t, path, `"content":"","owner-at-runtime":"owned"`, "")
_, _, err := WriteFilled(context.Background(), archHost(t), d, store.State{}, path, []byte("x\n"), noServices,
nil, nil, nil)
refusedAsPlacement(t, err)
}
// 1138/AC3: with no apply having read the roots yet, a filled write judges by the runtimes' configuration files,
// asking no runtime.
func TestAFilledWriteWithNoApplyYetUsesTheRuntimesConfigFiles(t *testing.T) {
setRuntimeRoots(t, nil) // no apply has read the roots yet
_, root := aRuntimeConfig(t)
var asked []string
askNoRuntime(t, &asked)
path := filepath.Join(root, "sub", "app.conf")
d := filledDecl(t, path, `"content":"","owner-at-runtime":"owned"`, "")
_, _, err := WriteFilled(context.Background(), archHost(t), d, store.State{}, path, []byte("x\n"), noServices,
nil, nil, nil)
refusedAsPlacement(t, err)
if len(asked) != 0 {
t.Errorf("a filled write with no apply yet asked the container runtimes: %v", asked)
}
}
// 1138/AC4: a full apply still asks each runtime once, at its start, whatever the number of files it writes.
func TestAFullApplyAsksEachRuntimeOnceRegardlessOfFileCount(t *testing.T) {
dir := t.TempDir()
var asked []string
before := AskRuntimes
AskRuntimes = func(_ context.Context, name string, _ ...string) (string, error) {
asked = append(asked, name)
return "", errors.New("not running in this test")
}
t.Cleanup(func() {
AskRuntimes = before
runtimeRootsMu.Lock()
runtimeRoots = nil
runtimeRootsMu.Unlock()
})
var resources []string
for i := 0; i < 3; i++ {
resources = append(resources, fmt.Sprintf(`{"id":"m.f%d","type":"file","path":%q,"content":"x\n"}`, i,
filepath.Join(dir, fmt.Sprintf("f%d.conf", i))))
}
if _, _, err := Apply(context.Background(), archHost(t), declare(t, strings.Join(resources, ",")),
store.State{}, store.OriginDeclared, noServices, nil, nil); err != nil {
t.Fatal(err)
}
if got := count(asked, "docker"); got != 1 {
t.Errorf("docker was asked %d times for one apply of 3 files; want 1: %v", got, asked)
}
if got := count(asked, "podman"); got != 1 {
t.Errorf("podman was asked %d times for one apply of 3 files; want 1: %v", got, asked)
}
}
+271
View File
@@ -0,0 +1,271 @@
package apply
import (
"context"
"encoding/json"
"fmt"
"os"
"path/filepath"
"strings"
"testing"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/store"
)
// Defends novox/hq ADR 0349 §4 (issue 668): the provisioner's client has the node-engine write a file it filled,
// and the node-engine writes it as an apply of the declared file with that content would — through applyFile,
// applyInto or applyBlock by who owns it at runtime — and makes what reads it take the change as an apply does.
// filledDecl is a declaration naming path as filled, with the file's fields after id, type and path, and more
// resources after it.
func filledDecl(t *testing.T, path, fields, more string) *declaration.Declaration {
t.Helper()
if more != "" {
more = "," + more
}
return parseTrusted(t, fmt.Sprintf(`{"declaration":1,"provisioner_account":"mesh-node-provisioner",
"filled_files":[%q],"resources":[{"id":"m.conf","type":"file","path":%q,%s}%s]}`, path, path, fields, more))
}
func TestAWriteOfAPathNotFilledIsRefused(t *testing.T) {
dir := t.TempDir()
filled := filepath.Join(dir, "filled.conf")
other := filepath.Join(dir, "other.conf")
d := parseTrusted(t, fmt.Sprintf(`{"declaration":1,"filled_files":[%q],"resources":[
{"id":"m.conf","type":"file","path":%q,"content":"x\n","owner-at-runtime":"owned"},
{"id":"m.other","type":"file","path":%q,"content":"y\n"}
]}`, filled, filled, other))
for _, path := range []string{other, filepath.Join(dir, "nothing.conf")} {
_, state, err := WriteFilled(context.Background(), archHost(t), d, store.State{}, path, []byte("evil\n"),
noServices, nil, nil, nil)
if err == nil || !strings.Contains(err.Error(), "does not name") {
t.Errorf("%s: a write of a path the declaration does not fill was not refused: %v", path, err)
}
if _, err := os.Stat(path); err == nil && path != other {
t.Errorf("%s was written", path)
}
if len(state.Resources) != 0 {
t.Errorf("a refused write recorded something: %+v", state.Resources)
}
}
if _, err := os.Stat(other); err == nil {
t.Errorf("%s was written", other)
}
}
// Each ownership is written by the write verb exactly as an apply of the declared file with the same content: the
// same action, the same bytes on disk, the same mode.
func TestAFilledFileIsWrittenAsItsOwnershipSays(t *testing.T) {
for _, c := range []struct {
name, fields, content, before, action string
}{
{"owned, absent", `"content":"","mode":"0640","owner-at-runtime":"owned"`, "port = 8080\n", "", "created"},
{"owned, over what is there", `"content":"","mode":"0640","owner-at-runtime":"owned"`, "port = 8080\n",
"port = 1\n", "updated"},
{"seed, absent", `"content":"","mode":"0600","owner-at-runtime":"seed"`, "user default on\n", "", "created"},
{"seed, present", `"content":"","mode":"0600","owner-at-runtime":"seed"`, "user default on\n",
"user default on\nuser app-one on\n", "kept"},
{"managed json", `"into":"json","content":"{}","owner-at-runtime":"managed"`,
`{"insecure-registries":["10.42.0.1:5000"]}`, machinesOwn, "updated"},
{"managed block", `"into":"block","content":"x\n","owner-at-runtime":"managed"`, "10.42.0.1 ace\n",
workstationHosts, "updated"},
} {
t.Run(c.name, func(t *testing.T) {
at := func() string {
path := filepath.Join(t.TempDir(), "the-file")
if c.before != "" {
if err := os.WriteFile(path, []byte(c.before), 0o644); err != nil {
t.Fatal(err)
}
}
return path
}
written := at()
report, state, err := WriteFilled(context.Background(), archHost(t), filledDecl(t, written, c.fields, ""),
store.State{}, written, []byte(c.content), noServices, nil, nil, nil)
if err != nil {
t.Fatal(err)
}
applied := at()
withContent := strings.Replace(c.fields, fmt.Sprintf(`"content":%q`, contentOf(t, c.fields)),
fmt.Sprintf(`"content":%q`, c.content), 1)
// The same file declared the old way: an apply writes a filled file only on the client's word (P7).
plain := parseTrusted(t, fmt.Sprintf(`{"declaration":1,"resources":[{"id":"m.conf","type":"file",`+
`"path":%q,%s}]}`, applied, withContent))
want, _, err := Apply(context.Background(), archHost(t), plain, store.State{}, store.OriginDeclared,
noServices, nil, nil)
if err != nil {
t.Fatal(err)
}
if got := report.Outcomes[0].Action; got != c.action || got != want.Outcomes[0].Action {
t.Errorf("written %s; an apply of the same content %s; want %s", got, want.Outcomes[0].Action, c.action)
}
if got, apply := readText(t, written), readText(t, applied); got != apply {
t.Errorf("written, the file holds\n%q\nan apply of the same content leaves\n%q", got, apply)
}
if a, b := modeAt(t, written), modeAt(t, applied); a != b {
t.Errorf("written mode %o, an apply's %o", a, b)
}
if rec, ok := state.Find("m.conf"); !ok || rec.Target != written || rec.Origin != store.OriginDeclared {
t.Errorf("the write is not recorded as the file's: %+v", rec)
}
})
}
}
func contentOf(t *testing.T, fields string) string {
t.Helper()
var holder struct {
Content string `json:"content"`
}
if err := json.Unmarshal([]byte("{"+fields+"}"), &holder); err != nil {
t.Fatal(err)
}
return holder.Content
}
func modeAt(t *testing.T, path string) os.FileMode {
t.Helper()
info, err := os.Stat(path)
if err != nil {
t.Fatal(err)
}
return info.Mode().Perm()
}
// A file that carried a secret keeps the mode a secret is written with: the client filled the value, and the file
// is no more readable for it.
func TestAFilledFileThatCarriedASecretIsNotWorldReadable(t *testing.T) {
path := filepath.Join(t.TempDir(), "token.env")
d := filledDecl(t, path, `"content":"TOKEN=${secret:token}\n","secrets":{"token":"c2VhbGVk"},"owner-at-runtime":"owned"`, "")
if _, _, err := WriteFilled(context.Background(), archHost(t), d, store.State{}, path, []byte("TOKEN=abc\n"),
noServices, nil, nil, nil); err != nil {
t.Fatal(err)
}
if m := modeAt(t, path); m != 0o600 {
t.Errorf("a filled file that carried a secret is mode %o", m)
}
}
func TestAWriteUnderReloadOnReloadsItsProgramOnce(t *testing.T) {
path := filepath.Join(t.TempDir(), "daemon.conf")
d := filledDecl(t, path, `"content":"","owner-at-runtime":"owned"`,
`{"id":"m.daemon","type":"service","unit":"m.service","state":"running","reload-on":["m.conf"]},
{"id":"m.other","type":"service","unit":"other.service","state":"running"}`)
var commands []string
var said []string
_, state, err := WriteFilled(context.Background(), archHost(t), d, store.State{}, path, []byte("level = 1\n"),
recordingServices(&commands), func(line string) { said = append(said, line) }, nil, nil)
if err != nil {
t.Fatal(err)
}
if n := count(commands, "systemctl reload m.service"); n != 1 {
t.Errorf("the program was reloaded %d times; commands were %v", n, commands)
}
joined := strings.Join(commands, "\n")
if strings.Contains(joined, "restart m.service") || strings.Contains(joined, "stop m.service") {
t.Errorf("the program was restarted, not reloaded: %v", commands)
}
if strings.Contains(joined, "other.service") {
t.Errorf("a service that does not read the file was touched: %v", commands)
}
if _, ok := state.Find("m.daemon"); !ok {
t.Errorf("the reloaded service was not recorded")
}
// The log says the path and the digest, never the content.
for _, line := range said {
if strings.Contains(line, "level = 1") {
t.Errorf("the content was said: %q", line)
}
}
if !strings.Contains(strings.Join(said, "\n"), digestOf("level = 1\n")) {
t.Errorf("the digest was not said: %q", said)
}
// The same content again moves nothing, and reloads nothing.
commands = nil
report, _, err := WriteFilled(context.Background(), archHost(t), d, state, path, []byte("level = 1\n"),
recordingServices(&commands), nil, nil, nil)
if err != nil {
t.Fatal(err)
}
if report.Outcomes[0].Action != "unchanged" || count(commands, "reload") != 0 {
t.Errorf("an unchanged write reloaded: %v %+v", commands, report.Outcomes)
}
}
func TestAWriteUnderRestartOnRestartsItsProgram(t *testing.T) {
path := filepath.Join(t.TempDir(), "daemon.conf")
d := filledDecl(t, path, `"content":"","owner-at-runtime":"owned"`,
`{"id":"m.daemon","type":"service","unit":"m.service","state":"running","restart-on":["m.conf"]}`)
var commands []string
if _, _, err := WriteFilled(context.Background(), archHost(t), d, store.State{}, path, []byte("level = 1\n"),
recordingServices(&commands), nil, nil, nil); err != nil {
t.Fatal(err)
}
// A restart is a stop and a start, once.
stops, starts := count(commands, "systemctl stop m.service"), count(commands, "systemctl start m.service")
if stops != 1 || starts != 1 {
t.Errorf("the program was stopped %d and started %d times; commands were %v", stops, starts, commands)
}
}
// A running container that reads the file at creation is recreated, as an apply recreates it for a changed file.
func TestAWriteOfAFileAContainerReadsRecreatesIt(t *testing.T) {
env := filepath.Join(t.TempDir(), "forge.env")
d := filledDecl(t, env, `"content":"","mode":"0600","owner-at-runtime":"owned"`,
`{"id":"forge.server","type":"container","name":"forge","image":"`+pinned+`","env-file":["`+env+`"]}`)
m := &machine{containers: map[string]*fakeContainer{}}
_, state, err := WriteFilled(context.Background(), archHost(t), d, store.State{}, env, []byte("PORT=5432\n"),
m.run, nil, nil, nil)
if err != nil {
t.Fatal(err)
}
if !m.did("docker run") {
t.Fatalf("the container was not created: %v", m.asked)
}
m.asked = nil
report, _, err := WriteFilled(context.Background(), archHost(t), d, state, env, []byte("PORT=5433\n"), m.run,
nil, nil, nil)
if err != nil {
t.Fatal(err)
}
if !m.removed("forge") || !m.did("docker run") {
t.Fatalf("the container kept what it read before the write: %v", m.asked)
}
if o := outcomeOf(report, "forge.server"); o.Action != "updated" || !strings.Contains(o.Detail, env) {
t.Errorf("the recreate did not say which file changed: %+v", o)
}
}
func count(lines []string, prefix string) int {
n := 0
for _, l := range lines {
if strings.Contains(l, prefix) {
n++
}
}
return n
}
// Two resources declaring one filled path — two modules' blocks in one file — are refused: the request carries the
// content only, so it cannot say whose block it is, and nothing is written.
func TestAFilledPathTwoResourcesDeclareIsRefused(t *testing.T) {
path := filepath.Join(t.TempDir(), "hosts")
if err := os.WriteFile(path, []byte(workstationHosts), 0o644); err != nil {
t.Fatal(err)
}
d := parseTrusted(t, fmt.Sprintf(`{"declaration":1,"filled_files":[%q],"resources":[
{"id":"a.hosts","type":"file","path":%q,"into":"block","content":"10.42.0.1 ace\n","owner-at-runtime":"managed"},
{"id":"b.hosts","type":"file","path":%q,"into":"block","content":"10.42.0.2 g14\n","owner-at-runtime":"managed"}
]}`, path, path, path))
_, state, err := WriteFilled(context.Background(), archHost(t), d, store.State{}, path, []byte("6.6.6.6 evil\n"),
noServices, nil, nil, nil)
if err == nil || !strings.Contains(err.Error(), "2 resources declare") {
t.Fatalf("a path two resources declare was not refused: %v", err)
}
if got := readText(t, path); got != workstationHosts || len(state.Resources) != 0 {
t.Errorf("something was written or recorded: %q %+v", got, state.Resources)
}
}
+12 -6
View File
@@ -318,6 +318,17 @@ func heldRoots(verdicts map[string]libraryVerdict, libs []declaration.Library) m
return out
}
// heldLibraries is the root of every library whose disk or mount is not there, with its name, judged now and read
// only: what a full apply holds its containers for (ADR 0328 §9), for one that judges without applying libraries.
func heldLibraries(ctx context.Context, libs []declaration.Library) map[string]string {
verdicts := map[string]libraryVerdict{}
for _, l := range libs {
rec, _ := libraryRecord(l.Library)
verdicts[l.Library] = assessLibrary(ctx, l, rec)
}
return heldRoots(verdicts, libs)
}
// mountsHeld is the library a container waits for: one whose root it mounts, or mounts below, is held.
func mountsHeld(c *declaration.Container, held map[string]string) string {
for _, v := range c.Volumes {
@@ -341,12 +352,7 @@ func WaitingOnLibrary(d *declaration.Declaration) map[string]bool {
if d == nil || len(d.Libraries) == 0 {
return out
}
verdicts := map[string]libraryVerdict{}
for _, l := range d.Libraries {
rec, _ := libraryRecord(l.Library)
verdicts[l.Library] = assessLibrary(context.Background(), l, rec)
}
held := heldRoots(verdicts, d.Libraries)
held := heldLibraries(context.Background(), d.Libraries)
for _, r := range d.Resources {
if c, ok := r.(*declaration.Container); ok && mountsHeld(c, held) != "" {
out[c.ID] = true
+99
View File
@@ -0,0 +1,99 @@
package apply
import (
"errors"
"fmt"
"sort"
"strings"
"github.com/novox/mesh-host/internal/declaration"
)
// failuresOf is every failure of an apply as the error carries it: the first among them, then the rest, so a module
// whose resource failed second is still named (issue 1132, and issue 500, which counted the rest and named none).
func failuresOf(failures []*Error) []Failure {
out := make([]Failure, 0, len(failures))
for _, f := range failures {
out = append(out, Failure{Resource: f.Resource, Err: f.Err})
}
return out
}
// ModuleResults groups an apply's outcome per module the declaration names (novox/hq ADR 0429 §4, issue 1132): the
// module's resources all applied, or it failed and Detail says which resource failed with its error, which resources
// were skipped behind a step that did not complete, or that its resources were not attempted because a step of no
// module stopped the apply. A module is named by what the declaration says of each resource (ModuleOfID), so one
// whose id would guess another is still its own.
//
// A failure that is no resource's (an error that is not an Error, or a resource of no module) belongs to no module and
// changes none: the node says it in its report. The report is the outcome of an apply that did not fail; an apply that
// failed carries its own account in the Error, and that is read in its place.
func ModuleResults(d *declaration.Declaration, report Report, err error) map[string]ModuleResult {
if d == nil {
return nil
}
out := map[string]ModuleResult{}
attempted := map[string]bool{}
var failed *Error
errors.As(err, &failed)
outcomes := report.Outcomes
if failed != nil {
outcomes = append(append([]Outcome(nil), outcomes...), failed.Done.Outcomes...)
}
skipped := map[string]string{}
for _, o := range outcomes {
module, ok := d.ModuleOfID(o.ID)
if !ok {
continue
}
if o.Action == "skipped" {
skipped[module] = o.Detail
continue
}
attempted[o.ID] = true
}
var reasons = map[string][]string{}
for _, r := range d.Resources {
if module, ok := d.ModuleOfID(r.Identity()); ok {
out[module] = ModuleResult{}
if failed != nil && failed.Stopped && !attempted[r.Identity()] {
reasons[module] = append(reasons[module], fmt.Sprintf("%s was not attempted: %s stopped the apply", r.Identity(), failed.Resource))
}
}
}
if failed != nil {
for _, f := range failed.Failures {
if module, ok := d.ModuleOfID(f.Resource); ok {
if _, named := out[module]; named {
reasons[module] = append(reasons[module], fmt.Sprintf("%s failed: %v", f.Resource, f.Err))
}
}
}
if len(failed.Failures) == 0 {
if module, ok := d.ModuleOfID(failed.Resource); ok {
if _, named := out[module]; named {
reasons[module] = append(reasons[module], fmt.Sprintf("%s failed: %v", failed.Resource, failed.Err))
}
}
}
for _, module := range failed.Modules {
if _, named := out[module]; named {
why := skipped[module]
if why == "" {
why = "a step it declares did not complete"
}
reasons[module] = append(reasons[module], "gated: "+why)
}
}
}
for module, why := range skipped {
if _, named := out[module]; named && len(reasons[module]) == 0 {
reasons[module] = append(reasons[module], "gated: "+why)
}
}
for module, why := range reasons {
sort.Strings(why)
out[module] = ModuleResult{Failed: true, Detail: strings.Join(why, "; ")}
}
return out
}
+76
View File
@@ -0,0 +1,76 @@
package apply
import (
"errors"
"strings"
"testing"
"github.com/novox/mesh-host/internal/declaration"
)
func twoModules(t *testing.T) *declaration.Declaration {
t.Helper()
d, err := declaration.Parse([]byte(`{"declaration":1,"builds":{"a":"b-1","b":"b-2"},"resources":[
{"id":"a.dir","type":"directory","path":"/srv/a","module":"a"},
{"id":"b.dir","type":"directory","path":"/srv/b","module":"b"},
{"id":"b.conf","type":"file","path":"/srv/b/c","content":"x","module":"b"},
{"id":"c.dir","type":"directory","path":"/srv/c","module":"c"}]}`))
if err != nil {
t.Fatal(err)
}
return d
}
// 1132/AC3: every resource of module A applied and one of module B failed: A is a success, B failed with the failing
// resource and its error named; a module gated by another's failure is failed too.
func TestAnApplyIsGroupedPerModule(t *testing.T) {
d := twoModules(t)
done := Report{Outcomes: []Outcome{
{ID: "a.dir", Action: "created"},
{ID: "c.dir", Action: "skipped", Detail: "a step this module declares did not complete"},
}}
err := &Error{Resource: "b.dir", Err: errors.New("no space left"), Done: done, Others: 1,
Failures: []Failure{{Resource: "b.dir", Err: errors.New("no space left")}, {Resource: "b.conf", Err: errors.New("denied")}},
Modules: []string{"c"}}
got := ModuleResults(d, Report{}, err)
if len(got) != 3 {
t.Fatalf("modules %v, want a, b and c", got)
}
if got["a"].Failed || got["a"].Detail != "" {
t.Errorf("a: %+v, want a clean success", got["a"])
}
b := got["b"]
if !b.Failed || !strings.Contains(b.Detail, "b.dir") || !strings.Contains(b.Detail, "no space left") ||
!strings.Contains(b.Detail, "b.conf") || !strings.Contains(b.Detail, "denied") {
t.Errorf("b: %+v, want both failing resources and their errors", b)
}
c := got["c"]
if !c.Failed || !strings.Contains(c.Detail, "did not complete") {
t.Errorf("c: %+v, want it failed as gated, with why", c)
}
}
// 1132/AC3: an apply that did not fail is a success for every module the declaration names.
func TestACleanApplyIsASuccessForEveryModule(t *testing.T) {
got := ModuleResults(twoModules(t), Report{}, nil)
for _, m := range []string{"a", "b", "c"} {
if r, ok := got[m]; !ok || r.Failed {
t.Errorf("%s: %+v (%v), want a success", m, r, ok)
}
}
}
// 1132/AC3: a step that belongs to no module stops the apply there; a module with a resource not attempted is not a
// success.
func TestAModuleNotAttemptedIsNotASuccess(t *testing.T) {
d := twoModules(t)
err := &Error{Resource: "boot", Err: errors.New("boom"), Stopped: true, Failures: []Failure{{Resource: "boot", Err: errors.New("boom")}},
Done: Report{Outcomes: []Outcome{{ID: "a.dir", Action: "created"}}}}
got := ModuleResults(d, Report{}, err)
if got["a"].Failed {
t.Errorf("a was done: %+v", got["a"])
}
if !got["b"].Failed || !strings.Contains(got["b"].Detail, "boot") {
t.Errorf("b was not attempted: %+v", got["b"])
}
}
+371
View File
@@ -0,0 +1,371 @@
package apply
// A directory or archive that says unknown "move-out" keeps only what it knows (issue 1024, novox/hq ADR 0422).
//
// Everything inside it that nothing declared, no archive unpacked and `knows` does not name is moved, whole, into
// `<home>/.local/state/mesh-moved/<module>/<UTC time>/`, with a list beside it that names each move before it is
// made (ADR 0417 §7). Nothing is deleted: the owner of the path reacts to a file it did not write by keeping it
// where its writer can find it. The pass runs after the resources of an apply, so a reconcile (every five minutes)
// runs it too.
import (
"encoding/json"
"errors"
"fmt"
"io/fs"
"os"
"path/filepath"
"sort"
"strings"
"time"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/store"
)
// movedOutWait is how old a path is before it is moved: a younger one may be mid-write by whatever made it, and
// waits for the next look (ADR 0417 §4).
const movedOutWait = time.Minute
// MovedOut is one path a move-out directory moved.
type MovedOut struct {
Resource string `json:"resource"`
From string `json:"from"`
To string `json:"to"`
}
// LeftInPlace is a socket, pipe or device a move-out directory did not move (ADR 0422 §4).
type LeftInPlace struct {
Resource string `json:"resource"`
Path string `json:"path"`
}
// renamePath and deviceOf are the machine's calls, variables so a test can fail a move or name another file system.
// renamePath renames from to to without following a link in either parent below home (move_out_linux.go).
var renamePath = renameUnder
type movedEntry struct {
From string `json:"from"`
To string `json:"to"`
}
// moveOutOwner is the user and group the kept folders and the list are given: the declared owner, or the home's
// account.
func moveOutOwner(home, owner string) (uid, gid int, err error) {
if owner != "" {
return idsOf(owner)
}
acc, ok := accountsOfHomes()[home]
if !ok {
return 0, 0, fmt.Errorf("%s is not the home of an account", home)
}
if u, g, err := idsOf(acc.Name); err == nil {
return u, g, nil
}
return acc.UID, acc.UID, nil
}
// knownUnder is what a directory knows of its own contents: the declared targets, the files an archive unpacked,
// `knows` entries, and every folder above one of them.
type knownUnder struct {
exact map[string]bool
prefixes []string
// own are nested directories and archives that say what they do with unknown paths themselves.
own map[string]bool
}
func (k *knownUnder) add(root, path string) {
path = filepath.Clean(path)
for path != root && strings.HasPrefix(path, root+string(os.PathSeparator)) {
k.exact[path] = true
path = filepath.Dir(path)
}
}
func (k *knownUnder) knows(path string) bool {
if k.exact[path] {
return true
}
for _, p := range k.prefixes {
if path == p || strings.HasPrefix(path, p+string(os.PathSeparator)) {
return true
}
}
return false
}
func knownOf(d *declaration.Declaration, known store.State, id, root string, knows []string) *knownUnder {
k := &knownUnder{exact: map[string]bool{}, own: map[string]bool{}}
for _, r := range d.Resources {
target := filepath.Clean(r.Target())
if target == "" || target == "." {
continue
}
if r.Identity() != id && strings.HasPrefix(target, root+string(os.PathSeparator)) {
k.add(root, target)
switch o := r.(type) {
case *declaration.Directory:
if o.Unknown != "" {
k.own[target] = true
}
case *declaration.Archive:
if o.Unknown != "" {
k.own[target] = true
}
}
}
if a, ok := r.(*declaration.Archive); ok {
if rec, found := known.Find(a.ID); found && rec.Unpacked != nil {
base := filepath.Clean(a.Path)
for _, f := range append(append([]string{}, rec.Unpacked.Files...), rec.Unpacked.Dirs...) {
p := filepath.Join(base, filepath.FromSlash(f))
if p == root || strings.HasPrefix(p, root+string(os.PathSeparator)) {
k.add(root, p)
}
}
}
}
}
for _, e := range knows {
p := filepath.Join(root, filepath.FromSlash(strings.TrimSuffix(e, "/")))
if strings.HasSuffix(e, "/") {
k.prefixes = append(k.prefixes, p)
} else {
k.exact[p] = true
}
k.add(root, p)
}
return k
}
// moveOut moves what the directory at dir does not know, for the resource id. It answers the paths moved.
func moveOut(d *declaration.Declaration, known store.State, id, module, dir, owner string, knows []string,
log func(string)) ([]MovedOut, []LeftInPlace, error) {
dir = filepath.Clean(dir)
home := homeAbove(dir)
if home == "" {
return nil, nil, fmt.Errorf("%s: move-out is only for a directory under an account's home, and %s is under none "+
"(ADR 0422 §5); nothing was moved", id, dir)
}
k := knownOf(d, known, id, dir, knows)
skip := filepath.Join(keptRoot(home), module)
var candidates []string
var left []string
err := filepath.WalkDir(dir, func(path string, e fs.DirEntry, err error) error {
if err != nil {
if errors.Is(err, fs.ErrNotExist) {
return nil
}
return err
}
if path == dir {
return nil
}
if path == skip || path == keptRoot(home) {
return fs.SkipDir
}
if k.own[path] {
if e.IsDir() {
return fs.SkipDir
}
return nil
}
if k.knows(path) {
if e.IsDir() && !k.exact[path] {
return fs.SkipDir
}
return nil
}
// Unknown. Moved whole, so never descended into.
info, err := e.Info()
if err != nil {
if errors.Is(err, fs.ErrNotExist) {
return nil
}
return err
}
switch {
case info.Mode()&(os.ModeSocket|os.ModeNamedPipe|os.ModeDevice|os.ModeCharDevice) != 0:
left = append(left, path)
case time.Since(info.ModTime()) < movedOutWait:
default:
candidates = append(candidates, path)
}
if e.IsDir() {
return fs.SkipDir
}
return nil
})
if err != nil {
return nil, nil, fmt.Errorf("%s: cannot look through %s: %w", id, dir, err)
}
for _, p := range left {
log(fmt.Sprintf(" left %s in %s: a socket, pipe or device is not moved (ADR 0422)", p, dir))
}
var stayed []LeftInPlace
for _, p := range left {
stayed = append(stayed, LeftInPlace{Resource: id, Path: p})
}
if len(candidates) == 0 {
return nil, stayed, nil
}
sort.Strings(candidates)
uid, gid, err := moveOutOwner(home, owner)
if err != nil {
return nil, nil, fmt.Errorf("%s: %w", id, err)
}
// The same file system first, so a move is a rename and nothing is copied (ADR 0417 §5).
ancestor := skip
for {
if _, err := os.Lstat(ancestor); err == nil {
break
}
ancestor = filepath.Dir(ancestor)
}
devDir, err1 := deviceOf(dir)
devKept, err2 := deviceOf(ancestor)
if err1 != nil || err2 != nil {
return nil, nil, fmt.Errorf("%s: cannot read the file system of %s or %s: %v %v", id, dir, ancestor, err1, err2)
}
if devDir != devKept {
return nil, nil, fmt.Errorf("%s: %s and %s are on different file systems (devices %d and %d), so a move would "+
"copy; nothing was moved (ADR 0417 §5)", id, dir, ancestor, devDir, devKept)
}
// The kept folder and the list, named by the second; a clash in the same second gets -2, -3.
stamp := time.Now().UTC().Format("20060102T150405Z")
kept, listPath := "", ""
for n := 1; ; n++ {
name := stamp
if n > 1 {
name = fmt.Sprintf("%s-%d", stamp, n)
}
kept, listPath = filepath.Join(skip, name), filepath.Join(skip, name+".json")
if !pathExists(kept) && !pathExists(listPath) {
break
}
}
var made []string
cleanup := func() {
_ = os.Remove(listPath)
// In the reverse of the order they were made, so a folder goes after what is in it.
for i := len(made) - 1; i >= 0; i-- {
_ = os.Remove(made[i])
}
}
chown := func(paths []string) error {
for _, p := range paths {
if err := chownIfNeeded(home, p, uid, gid); err != nil {
return err
}
}
return nil
}
madeNow, err := mkdirAllUnder(home, kept, 0o700)
made = append(made, reversed(madeNow)...)
if err == nil {
err = chown(madeNow)
}
if err != nil {
cleanup()
return nil, nil, fmt.Errorf("%s: cannot make the kept folder %s: %w", id, kept, err)
}
var list []movedEntry
var moved []MovedOut
var failures []string
for _, from := range candidates {
rel, _ := filepath.Rel(dir, from)
to := filepath.Join(kept, rel)
// Named before it is moved: a move that stops half way leaves its entry (ADR 0417 §7).
list = append(list, movedEntry{From: from, To: to})
raw, _ := json.MarshalIndent(list, "", " ")
if err := writeUnder(home, listPath, append(raw, '\n'), 0o600); err != nil {
failures = append(failures, fmt.Sprintf("cannot write the list %s: %v", listPath, err))
break
}
if err := chownIfNeeded(home, listPath, uid, gid); err != nil {
failures = append(failures, err.Error())
break
}
parents, err := mkdirAllUnder(home, filepath.Dir(to), 0o700)
made = append(made, reversed(parents)...)
if err == nil {
err = chown(parents)
}
if err == nil {
err = renamePath(home, from, to)
}
if err != nil {
failures = append(failures, fmt.Sprintf("cannot move %s to %s: %v", from, to, err))
continue
}
log(fmt.Sprintf(" moved %s to %s (ADR 0422)", from, to))
moved = append(moved, MovedOut{Resource: id, From: from, To: to})
}
if len(moved) == 0 {
cleanup()
}
if len(failures) > 0 {
return moved, stayed, fmt.Errorf("%s: %s", id, strings.Join(failures, "; "))
}
return moved, stayed, nil
}
// reversed is paths shallowest first: mkdirAllUnder answers deepest first.
func reversed(in []string) []string {
out := make([]string, len(in))
for i, p := range in {
out[len(in)-1-i] = p
}
return out
}
func keptRoot(home string) string { return filepath.Join(home, ".local", "state", "mesh-moved") }
// chownIfNeeded gives a path to uid and gid through the home-safe call, unless it is theirs already.
func chownIfNeeded(home, path string, uid, gid int) error {
if u, g, ok := ownerIDs(path); ok && u == uid && g == gid {
return nil
}
return chownUnder(home, path, uid, gid)
}
// moveOutPass runs move-out for every directory and archive of d that says so and applied in this run.
func moveOutPass(d *declaration.Declaration, known store.State, applied map[string]bool, log func(string)) (
moved []MovedOut, left []LeftInPlace, failures []*Error) {
for _, r := range d.Resources {
var unknown, owner string
var knows []string
switch o := r.(type) {
case *declaration.Directory:
unknown, owner, knows = o.Unknown, o.Owner, o.Knows
case *declaration.Archive:
unknown, owner, knows = o.Unknown, o.Owner, o.Knows
default:
continue
}
if unknown != declaration.UnknownMoveOut || !applied[r.Identity()] {
continue
}
module, _ := d.ModuleOfID(r.Identity())
if module == "" {
module = r.Identity()
}
m, l, err := moveOut(d, known, r.Identity(), module, r.Target(), owner, knows, log)
moved = append(moved, m...)
left = append(left, l...)
if err != nil {
log(fmt.Sprintf(" failed %s (%s): %v", r.Identity(), r.Target(), err))
failures = append(failures, &Error{Resource: r.Identity(), Err: err})
}
}
return moved, left, failures
}
func pathExists(path string) bool {
_, err := os.Lstat(path)
return err == nil
}
+61
View File
@@ -0,0 +1,61 @@
//go:build linux
package apply
import (
"fmt"
"path/filepath"
"syscall"
"golang.org/x/sys/unix"
)
// deviceOf is the device a path lives on; a variable so a test can name another file system. Below a home it is
// read through a descriptor opened without following a link (issue 1024).
var deviceOf = func(path string) (uint64, error) {
var st unix.Stat_t
if home := homeAbove(path); home != "" {
fd, err := openDirUnder(home, path)
if err != nil {
return 0, err
}
defer unix.Close(fd)
if err := unix.Fstat(fd, &st); err != nil {
return 0, err
}
return uint64(st.Dev), nil
}
if err := unix.Stat(path, &st); err != nil {
return 0, err
}
return uint64(st.Dev), nil
}
// renameUnder moves from to to, both below home, through descriptors of their parents opened without following a
// link, and never over something that is there: an account that owns the folders cannot point either at another
// place between the look and the move (issue 1024, novox/hq ADR 0266).
func renameUnder(home, from, to string) error {
fromDir, err := openDirUnder(home, filepath.Dir(from))
if err != nil {
return err
}
defer unix.Close(fromDir)
toDir, err := openDirUnder(home, filepath.Dir(to))
if err != nil {
return err
}
defer unix.Close(toDir)
if err := unix.Renameat2(fromDir, filepath.Base(from), toDir, filepath.Base(to), unix.RENAME_NOREPLACE); err != nil {
return fmt.Errorf("rename %s to %s: %w", from, to, err)
}
return nil
}
// ownerIDs is the user and group of a path, without following a link.
func ownerIDs(path string) (uid, gid int, ok bool) {
var st syscall.Stat_t
if err := syscall.Lstat(path, &st); err != nil {
return 0, 0, false
}
return int(st.Uid), int(st.Gid), true
}
+19
View File
@@ -0,0 +1,19 @@
//go:build !linux
package apply
import "os"
var deviceOf = func(string) (uint64, error) { return 0, nil }
func ownerIDs(string) (int, int, bool) { return 0, 0, false }
func renameUnder(home, from, to string) error {
if err := refuseLinksUnderHome(from); err != nil {
return err
}
if err := refuseLinksUnderHome(to); err != nil {
return err
}
return os.Rename(from, to)
}
+355
View File
@@ -0,0 +1,355 @@
package apply
// Issue 1024, novox/hq ADR 0422: a directory that says unknown "move-out" keeps only what it knows, and moves every
// other path into a kept folder under the account's home, listed before it is moved.
import (
"context"
"encoding/json"
"errors"
"os"
osuser "os/user"
"path/filepath"
"strings"
"syscall"
"testing"
"time"
"github.com/novox/mesh-host/internal/store"
"golang.org/x/sys/unix"
)
// me is the account running the test, which owns the home the test names.
func me() string {
u, err := osuser.Current()
if err != nil {
panic(err)
}
return u.Username
}
// aMoveOutHome is a home the test names, of the account running the test, so no chown needs privilege.
func aMoveOutHome(t *testing.T) string {
t.Helper()
home := filepath.Join(t.TempDir(), "home", "agent")
if err := os.MkdirAll(home, 0o755); err != nil {
t.Fatal(err)
}
wasHomes, wasAccounts := homes, accountsOfHomes
homes = func() []string { return []string{home} }
accountsOfHomes = func() map[string]homeAccount {
return map[string]homeAccount{home: {Name: me(), UID: os.Getuid()}}
}
t.Cleanup(func() { homes, accountsOfHomes = wasHomes, wasAccounts })
return home
}
func put(t *testing.T, path, content string, age time.Duration) {
t.Helper()
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(path, []byte(content), 0o644); err != nil {
t.Fatal(err)
}
old := time.Now().Add(-age)
if err := os.Chtimes(path, old, old); err != nil {
t.Fatal(err)
}
}
func keptLists(t *testing.T, home string) []string {
t.Helper()
out, _ := filepath.Glob(filepath.Join(keptRoot(home), "*", "*.json"))
return out
}
func readList(t *testing.T, path string) []movedEntry {
t.Helper()
raw, err := os.ReadFile(path)
if err != nil {
t.Fatal(err)
}
var list []movedEntry
if err := json.Unmarshal(raw, &list); err != nil {
t.Fatal(err)
}
return list
}
func TestAMoveOutDirectoryMovesOnlyWhatIsNotKnown(t *testing.T) {
home := aMoveOutHome(t)
dir := filepath.Join(home, ".ssh")
outside := filepath.Join(t.TempDir(), "target")
put(t, outside, "elsewhere", time.Hour)
put(t, filepath.Join(dir, "config"), "declared", time.Hour)
put(t, filepath.Join(dir, "known_hosts"), "knows", time.Hour)
put(t, filepath.Join(dir, "x", "inside"), "in x", time.Hour)
put(t, filepath.Join(dir, "stray"), "stray content", time.Hour)
put(t, filepath.Join(dir, "oldfolder", "deep", "f"), "deep", time.Hour)
put(t, filepath.Join(dir, "young"), "just made", 5*time.Second)
if err := os.Symlink(outside, filepath.Join(dir, "alink")); err != nil {
t.Fatal(err)
}
old := time.Now().Add(-time.Hour)
_ = os.Chtimes(filepath.Join(dir, "oldfolder"), old, old)
_ = os.Chtimes(filepath.Join(dir, "x"), old, old)
// A named pipe stands for the socket: a socket's path is too long under a test directory to bind.
if err := syscall.Mkfifo(filepath.Join(dir, "agent.sock"), 0o644); err != nil {
t.Fatal(err)
}
_ = os.Chtimes(filepath.Join(dir, "agent.sock"), old, old)
// The directory itself and its declared file are old enough too.
// The symbolic link is older than a minute as well.
ts := []unix.Timespec{unix.NsecToTimespec(old.UnixNano()), unix.NsecToTimespec(old.UnixNano())}
if err := unix.UtimesNanoAt(unix.AT_FDCWD, filepath.Join(dir, "alink"), ts, unix.AT_SYMLINK_NOFOLLOW); err != nil {
t.Fatal(err)
}
var logged []string
report, _, err := Apply(context.Background(), archHost(t), declare(t,
`{"id":"mod.dir","type":"directory","path":"`+dir+`","owner":"`+me()+`","unknown":"move-out","knows":["known_hosts","x/"]},`+
`{"id":"mod.config","type":"file","path":"`+dir+`/config","content":"declared"}`),
store.State{}, store.OriginDeclared, noServices, func(s string) { logged = append(logged, s) }, nil)
if err != nil {
t.Fatal(err)
}
_ = report
for _, kept := range []string{"config", "known_hosts", "x/inside", "young", "agent.sock"} {
if !exists(filepath.Join(dir, kept)) {
t.Errorf("%s should stay", kept)
}
}
for _, gone := range []string{"stray", "oldfolder", "alink"} {
if exists(filepath.Join(dir, gone)) {
t.Errorf("%s should have moved", gone)
}
}
lists := keptLists(t, home)
if len(lists) != 1 {
t.Fatalf("one list expected, got %v", lists)
}
if !strings.Contains(lists[0], filepath.Join("mesh-moved", "mod")) {
t.Errorf("the module's name is not in the kept path: %s", lists[0])
}
list := readList(t, lists[0])
if len(list) != 3 {
t.Fatalf("the list names three paths: %+v", list)
}
kept := strings.TrimSuffix(lists[0], ".json")
for _, e := range list {
if !strings.HasPrefix(e.To, kept+string(os.PathSeparator)) || !exists(e.To) {
t.Errorf("not at %s: %+v", kept, e)
}
}
if b, _ := os.ReadFile(filepath.Join(kept, "stray")); string(b) != "stray content" {
t.Errorf("content changed: %q", b)
}
if b, _ := os.ReadFile(filepath.Join(kept, "oldfolder", "deep", "f")); string(b) != "deep" {
t.Errorf("the folder was not moved whole: %q", b)
}
if info, err := os.Lstat(filepath.Join(kept, "alink")); err != nil || info.Mode()&os.ModeSymlink == 0 {
t.Errorf("the link is not still a link: %v", err)
}
if !strings.Contains(strings.Join(logged, "\n"), "(ADR 0422)") {
t.Errorf("the moves were not logged: %v", logged)
}
if len(report.Left) != 1 || filepath.Base(report.Left[0].Path) != "agent.sock" {
t.Errorf("the report does not list the pipe it left: %+v", report.Left)
}
if len(report.MovedOut) != 3 {
t.Errorf("the report does not count the moves: %+v", report.MovedOut)
}
}
func TestAMoveOutListIsWrittenBeforeTheMove(t *testing.T) {
home := aMoveOutHome(t)
dir := filepath.Join(home, "d")
put(t, filepath.Join(dir, "a"), "a", time.Hour)
put(t, filepath.Join(dir, "b"), "b", time.Hour)
was := renamePath
renamePath = func(h, from, to string) error {
if filepath.Base(from) == "b" {
return errors.New("injected")
}
return was(h, from, to)
}
t.Cleanup(func() { renamePath = was })
_, _, err := applyOnce(t, `{"id":"mod.dir","type":"directory","path":"`+dir+`","owner":"`+me()+`","unknown":"move-out"}`, nil, store.State{})
if err == nil {
t.Fatal("a failed move must fail the resource")
}
if !exists(filepath.Join(dir, "b")) || exists(filepath.Join(dir, "a")) {
t.Error("b stays, a moved")
}
lists := keptLists(t, home)
if len(lists) != 1 {
t.Fatalf("lists: %v", lists)
}
names := map[string]bool{}
for _, e := range readList(t, lists[0]) {
names[filepath.Base(e.From)] = true
}
if !names["a"] || !names["b"] {
t.Errorf("the entry for the failed move was not listed first: %v", names)
}
}
func TestAMoveOutThatMovedNothingMakesNothing(t *testing.T) {
home := aMoveOutHome(t)
dir := filepath.Join(home, "d")
put(t, filepath.Join(dir, "known"), "k", time.Hour)
put(t, filepath.Join(dir, "young"), "y", time.Second)
if _, _, err := applyOnce(t, `{"id":"mod.dir","type":"directory","path":"`+dir+`","owner":"`+me()+`","unknown":"move-out","knows":["known"]}`, nil, store.State{}); err != nil {
t.Fatal(err)
}
if exists(keptRoot(home)) {
t.Error("a kept folder was made with nothing to move")
}
// And when every move fails, what was made goes.
put(t, filepath.Join(dir, "stray"), "s", time.Hour)
was := renamePath
renamePath = func(string, string, string) error { return errors.New("injected") }
t.Cleanup(func() { renamePath = was })
if _, _, err := applyOnce(t, `{"id":"mod.dir","type":"directory","path":"`+dir+`","owner":"`+me()+`","unknown":"move-out","knows":["known"]}`, nil, store.State{}); err == nil {
t.Fatal("expected a failure")
}
if len(keptLists(t, home)) != 0 {
t.Error("a list stayed though nothing moved")
}
if matches, _ := filepath.Glob(filepath.Join(keptRoot(home), "*", "*")); len(matches) != 0 {
t.Errorf("a kept folder stayed though nothing moved: %v", matches)
}
}
func TestAMoveOutOfANestedStrayThatFailsLeavesNoKeptFolder(t *testing.T) {
home := aMoveOutHome(t)
dir := filepath.Join(home, "d")
put(t, filepath.Join(dir, "x", "f"), "declared", time.Hour)
put(t, filepath.Join(dir, "x", "stray"), "s", time.Hour)
was := renamePath
renamePath = func(string, string, string) error { return errors.New("injected") }
t.Cleanup(func() { renamePath = was })
_, _, err := applyOnce(t, `{"id":"mod.dir","type":"directory","path":"`+dir+`","owner":"`+me()+`","unknown":"move-out"},`+
`{"id":"mod.f","type":"file","path":"`+dir+`/x/f","content":"declared","owner":"`+me()+`"}`, nil, store.State{})
if err == nil {
t.Fatal("expected a failure")
}
if exists(keptRoot(home)) {
matches, _ := filepath.Glob(filepath.Join(keptRoot(home), "*", "*"))
t.Errorf("something was left behind though nothing moved: %v", matches)
}
}
// A link swapped into a parent folder, on either side, between the look and the move, is not followed.
func TestAMoveOutDoesNotFollowALinkSwappedIntoAParent(t *testing.T) {
for _, side := range []string{"from", "to"} {
t.Run(side, func(t *testing.T) {
home := aMoveOutHome(t)
dir := filepath.Join(home, "d")
outside := filepath.Join(t.TempDir(), "etc")
put(t, filepath.Join(outside, "stray"), "root's", time.Hour)
put(t, filepath.Join(dir, "x", "f"), "declared", time.Hour)
put(t, filepath.Join(dir, "x", "stray"), "s", time.Hour)
was := renamePath
renamePath = func(h, from, to string) error {
victim := filepath.Dir(from)
if side == "to" {
victim = filepath.Dir(to)
_ = os.Remove(filepath.Join(victim, "nothing"))
}
if err := os.RemoveAll(victim); err != nil {
t.Fatal(err)
}
if side == "from" {
put(t, filepath.Join(outside, "other"), "o", time.Hour)
}
if err := os.Symlink(outside, victim); err != nil {
t.Fatal(err)
}
return was(h, from, to)
}
t.Cleanup(func() { renamePath = was })
_, _, err := applyOnce(t, `{"id":"mod.dir","type":"directory","path":"`+dir+`","owner":"`+me()+`","unknown":"move-out"},`+
`{"id":"mod.f","type":"file","path":"`+dir+`/x/f","content":"declared","owner":"`+me()+`"}`, nil, store.State{})
if err == nil {
t.Fatal("a move through a swapped link must fail")
}
if b, e := os.ReadFile(filepath.Join(outside, "stray")); e != nil || string(b) != "root's" {
t.Errorf("the file outside the home was touched: %q %v", b, e)
}
entries, _ := os.ReadDir(outside)
if side == "to" && len(entries) != 1 || side == "from" && len(entries) != 2 {
t.Errorf("something was moved into or out of the folder outside the home: %v", entries)
}
})
}
}
func TestOtherUnknownValuesAndNoValueMoveNothing(t *testing.T) {
home := aMoveOutHome(t)
for _, v := range []string{`,"unknown":"program"`, `,"unknown":"module"`, ``} {
dir := filepath.Join(home, "d")
put(t, filepath.Join(dir, "stray"), "s", time.Hour)
if _, _, err := applyOnce(t, `{"id":"mod.dir","type":"directory","path":"`+dir+`","owner":"`+me()+`"`+v+`}`, nil, store.State{}); err != nil {
t.Fatal(err)
}
if !exists(filepath.Join(dir, "stray")) || exists(keptRoot(home)) {
t.Errorf("%q moved something", v)
}
}
}
func TestAMoveOutArchiveKeepsWhatItUnpackedAndMovesAStray(t *testing.T) {
home := aMoveOutHome(t)
dir := filepath.Join(home, "theme")
put(t, filepath.Join(dir, "stray"), "s", time.Hour)
res, _ := archiveDecl(t, "mod.theme", dir, map[string]string{"one.txt": "1", "sub/two.txt": "2"}, "")
res = strings.Replace(res, `"path":`, `"owner":"`+me()+`","unknown":"move-out","path":`, 1)
if _, _, err := Apply(context.Background(), archHost(t), declare(t, res), store.State{},
store.OriginDeclared, noServices, nil, nil); err != nil {
t.Fatal(err)
}
for _, f := range []string{"one.txt", "sub/two.txt"} {
if !exists(filepath.Join(dir, f)) {
t.Errorf("%s was moved", f)
}
}
if exists(filepath.Join(dir, "stray")) || len(keptLists(t, home)) != 1 {
t.Error("the stray file did not move")
}
}
func TestMoveOutNeedsAHome(t *testing.T) {
aMoveOutHome(t)
dir := filepath.Join(t.TempDir(), "elsewhere")
put(t, filepath.Join(dir, "stray"), "s", time.Hour)
_, _, err := applyOnce(t, `{"id":"mod.dir","type":"directory","path":"`+dir+`","owner":"`+me()+`","unknown":"move-out"}`, nil, store.State{})
if err == nil || !strings.Contains(err.Error(), "home") {
t.Fatalf("expected a refusal naming the home: %v", err)
}
if !exists(filepath.Join(dir, "stray")) {
t.Error("something moved")
}
}
func TestMoveOutRefusesAnotherFileSystem(t *testing.T) {
home := aMoveOutHome(t)
dir := filepath.Join(home, "d")
put(t, filepath.Join(dir, "stray"), "s", time.Hour)
was := deviceOf
deviceOf = func(p string) (uint64, error) {
if p == dir {
return 1, nil
}
return 2, nil
}
t.Cleanup(func() { deviceOf = was })
_, _, err := applyOnce(t, `{"id":"mod.dir","type":"directory","path":"`+dir+`","owner":"`+me()+`","unknown":"move-out"}`, nil, store.State{})
if err == nil || !strings.Contains(err.Error(), "file system") {
t.Fatalf("expected a refusal naming the file systems: %v", err)
}
if !exists(filepath.Join(dir, "stray")) || exists(keptRoot(home)) {
t.Error("something moved or was made")
}
}
+129
View File
@@ -0,0 +1,129 @@
package apply
import (
"context"
"fmt"
"os"
"path/filepath"
"testing"
"github.com/novox/mesh-host/internal/store"
)
// Defends novox/hq ADR 0349 §5 (issue 667): a file that says who owns it at runtime is written as the
// form it names always was — owned as a whole file whose drift is written back, seed as create-once,
// managed as only its keys or its marked block. Each is applied beside its old form, through the same
// drift, and the two must end the same: the same actions, the same bytes on disk.
// ownershipCase is one ownership word and the old form it maps onto.
type ownershipCase struct {
name string
// said and old are the resource's fields after id, type and path: with the word, and in the old form.
said, old string
// before is what is at the path before the first apply; "" leaves the path absent.
before string
// drift is what the machine writes there between the two applies.
drift string
// first and second are the actions the old form is known to report.
first, second string
}
func TestEachOwnershipBehavesAsItsOldForm(t *testing.T) {
for _, c := range []ownershipCase{
{
name: "owned is a whole file whose drift is written back",
said: `"content":"correct\n","mode":"0644","owner-at-runtime":"owned"`,
old: `"content":"correct\n","mode":"0644"`,
drift: "someone edited this\n",
first: "created", second: "corrected",
},
{
name: "seed is written once and then the program's",
said: `"content":"user default on\n","mode":"0600","owner-at-runtime":"seed"`,
old: `"content":"user default on\n","mode":"0600","create-once":true`,
drift: "user default on\nuser app-one on\n",
first: "created", second: "kept",
},
{
name: "managed json sets only its keys",
said: `"into":"json","content":"{\"insecure-registries\":[\"10.42.0.1:5000\"]}","owner-at-runtime":"managed"`,
old: `"into":"json","content":"{\"insecure-registries\":[\"10.42.0.1:5000\"]}"`,
before: machinesOwn,
drift: `{"data-root":"/srv/other","log-opts":{"max-size":"10m"},"insecure-registries":["192.0.2.7:5000"]}`,
first: "updated", second: "corrected",
},
{
name: "managed block holds only its region",
said: `"into":"block","content":"10.42.0.1 ace\n","owner-at-runtime":"managed"`,
old: `"into":"block","content":"10.42.0.1 ace\n"`,
before: workstationHosts,
drift: workstationHosts + "\n" + marked("f", "6.6.6.6 evil\n") + "192.0.2.9 the-machines-own\n",
first: "updated", second: "corrected",
},
} {
t.Run(c.name, func(t *testing.T) {
saidActions, saidBytes := applyThroughDrift(t, c, c.said)
oldActions, oldBytes := applyThroughDrift(t, c, c.old)
if saidActions != oldActions {
t.Errorf("with the word the actions were %s; the old form's were %s", saidActions, oldActions)
}
if want := c.first + " " + c.second; oldActions != want {
t.Errorf("the old form's actions were %s; known to be %s", oldActions, want)
}
if saidBytes != oldBytes {
t.Errorf("with the word the file ends\n%q\nthe old form's ends\n%q", saidBytes, oldBytes)
}
})
}
}
// applyThroughDrift applies one file, lets the machine write the case's drift into it, applies again,
// and says the two actions and what the file holds at the end.
func applyThroughDrift(t *testing.T, c ownershipCase, fields string) (string, string) {
t.Helper()
path := filepath.Join(t.TempDir(), "the-file")
if c.before != "" {
if err := os.WriteFile(path, []byte(c.before), 0o644); err != nil {
t.Fatal(err)
}
}
d := parse(t, fmt.Sprintf(`{"declaration":1,"resources":[{"id":"f","type":"file","path":%q,%s}]}`, path, fields))
first, state, err := Apply(context.Background(), archHost(t), d, store.State{}, store.OriginDeclared, noServices, nil, nil)
if err != nil {
t.Fatal(err)
}
if err := os.WriteFile(path, []byte(c.drift), 0o644); err != nil {
t.Fatal(err)
}
second, _, err := Apply(context.Background(), archHost(t), d, state, store.OriginDeclared, noServices, nil, nil)
if err != nil {
t.Fatal(err)
}
return first.Outcomes[0].Action + " " + second.Outcomes[0].Action, readText(t, path)
}
// A seed said by the word alone, with no create-once beside it, is still kept once present: the word is
// enough, and the file the program grew is not written back.
func TestASeedByTheWordAloneKeepsWhatGrew(t *testing.T) {
path := filepath.Join(t.TempDir(), "acl.conf")
d := parse(t, fmt.Sprintf(`{"declaration":1,"resources":[
{"id":"acl","type":"file","path":%q,"content":"user default on\n","owner-at-runtime":"seed"}
]}`, path))
_, state, err := Apply(context.Background(), archHost(t), d, store.State{}, store.OriginDeclared, noServices, nil, nil)
if err != nil {
t.Fatal(err)
}
if err := os.WriteFile(path, []byte("grown\n"), 0o600); err != nil {
t.Fatal(err)
}
report, _, err := Apply(context.Background(), archHost(t), d, state, store.OriginDeclared, noServices, nil, nil)
if err != nil {
t.Fatal(err)
}
if got := report.Outcomes[0].Action; got != "kept" {
t.Errorf("a seed was %q", got)
}
if got := readText(t, path); got != "grown\n" {
t.Errorf("what grew in the seed was written over: %q", got)
}
}
+10
View File
@@ -0,0 +1,10 @@
package apply
// UseAccountsFileForTest makes the accounts and homes this package knows come from the user database at path, until
// the returned function is called. For the node-engine's own integration tests, which cannot name a home any other
// way; nothing in the node-engine calls it (issue 1024).
func UseAccountsFileForTest(path string) (restore func()) {
was := passwdFile
passwdFile = path
return func() { passwdFile = was }
}
+112
View File
@@ -0,0 +1,112 @@
package apply
// A core credential the vault sealed to this machine (novox/hq ADR 0349 §16, issue 755): the controller carries the
// vault's box (ADR 0206) in the declaration and cannot open it; this machine opens it with its sealing key and writes
// the value, as it writes a value the controller sealed.
import (
"context"
"encoding/json"
"os"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/seal"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/identity"
"github.com/novox/mesh-host/internal/store"
)
// vaultSealedTo is what the vault answers the controller: the value in the ADR 0206 box, sealed to the machine's
// sealing key as SPKI PEM.
func vaultSealedTo(t *testing.T, key identity.SealingKey, value string) seal.Box {
t.Helper()
pub, err := key.PublicPEM()
if err != nil {
t.Fatal(err)
}
box, err := seal.Seal(value, pub)
if err != nil {
t.Fatal(err)
}
return box
}
func vaultSealedFile(t *testing.T, path string, box seal.Box) *declaration.Declaration {
t.Helper()
raw := map[string]any{"declaration": 1, "resources": []map[string]any{
{"id": "broker", "type": "file", "path": path, "vault-sealed": box},
}}
body, _ := json.Marshal(raw)
d, err := declaration.Parse(body)
if err != nil {
t.Fatal(err)
}
return d
}
func boxOpener(key identity.SealingKey) Unseal {
return func(sealed string) ([]byte, error) { return key.Open(sealed) }
}
func TestAFileTheVaultSealedToThisMachineIsOpenedAndWrittenUnreadableToOthers(t *testing.T) {
key, err := identity.GenerateSealingKey()
if err != nil {
t.Fatal(err)
}
path := t.TempDir() + "/broker"
d := vaultSealedFile(t, path, vaultSealedTo(t, key, "the-provisioners-bus-password"))
report, state, err := Apply(context.Background(), archHost(t), d, store.State{},
store.OriginCarried, noServices, nil, boxOpener(key))
if err != nil {
t.Fatal(err)
}
on, err := os.ReadFile(path)
if err != nil || string(on) != "the-provisioners-bus-password" {
t.Fatalf("the file holds %q, %v", on, err)
}
if info, _ := os.Stat(path); info.Mode().Perm() != 0o600 {
t.Errorf("a credential landed mode %o", info.Mode().Perm())
}
said, _ := json.Marshal(report)
kept, _ := json.Marshal(state)
for what, blob := range map[string][]byte{"the report": said, "the node's state": kept} {
if strings.Contains(string(blob), "the-provisioners-bus-password") {
t.Fatalf("%s carries the value in the clear", what)
}
}
}
func TestAVaultBoxSealedToAnotherMachineIsRefusedAndNothingWritten(t *testing.T) {
mine, _ := identity.GenerateSealingKey()
theirs, _ := identity.GenerateSealingKey()
path := t.TempDir() + "/broker"
d := vaultSealedFile(t, path, vaultSealedTo(t, theirs, "not for you"))
if _, _, err := Apply(context.Background(), archHost(t), d, store.State{},
store.OriginCarried, noServices, nil, boxOpener(mine)); err == nil {
t.Fatal("a box sealed to another machine was applied")
}
if _, err := os.Stat(path); err == nil {
t.Fatal("something was written before the failure")
}
}
func TestAVaultSealedFileSaysWhatIsInItOnce(t *testing.T) {
key, _ := identity.GenerateSealingKey()
box, _ := json.Marshal(vaultSealedTo(t, key, "x"))
for what, extra := range map[string]string{
"content": `"content":"a"`, "sealed": `"sealed":"b"`, "bytes": `"bytes":"Yw=="`,
"secrets": `"content":"${secret:x}","secrets":{"x":"c"}`, "into": `"into":"json","content":"{}"`,
} {
_, err := declaration.Parse([]byte(`{"declaration":1,"resources":[{"id":"f","type":"file","path":"/etc/x",` +
`"vault-sealed":` + string(box) + `,` + extra + `}]}`))
if err == nil {
t.Errorf("a vault-sealed file that also says %s was accepted", what)
}
}
if _, err := declaration.Parse([]byte(`{"declaration":1,"resources":[{"id":"f","type":"file","path":"/etc/x",` +
`"vault-sealed":{"v":2,"eph":"a","iv":"b","tag":"c","ct":"d"}}]}`)); err == nil {
t.Error("a box of a version the SDK does not open was accepted")
}
}
+60 -26
View File
@@ -36,6 +36,7 @@ import (
"time"
"github.com/novox/mesh-host/internal/firewall"
"github.com/novox/mesh-host/internal/system"
"github.com/novox/mesh-host/internal/tunnel"
)
@@ -173,6 +174,10 @@ type Options struct {
// ToolsSource is where the shared base is built from. Everything with code of its own
// compiles against it, so it is the first thing the mesh builds for itself.
ToolsSource Source
// ToolsGoSource is where the Go toolchain is built from (`mesh-tools-go`, ADR 0142): every Go
// bundle compiles in it, and genesis's own modules have Go code (issue 616). Empty means the
// repository beside ToolsSource named mesh-tools-go, as the mesh keeps it.
ToolsGoSource Source
// CatalogSource is where the catalogue REPOSITORY is, for building its modules. The catalogue
// CHECKOUT (Catalogue, above) says what a module is; this is where a builder clones it.
CatalogSource Source
@@ -292,12 +297,14 @@ type Result struct {
Agent string `json:"host-agent,omitempty"`
// Registry is the mesh's own artifact store, once it answers.
Registry string `json:"registry,omitempty"`
RegistryReplied int `json:"registry-replied,omitempty"`
RegistryKnown bool `json:"registry-already-registered,omitempty"`
RegistryRunning string `json:"registry-container,omitempty"`
PublishedAs string `json:"control-plane-image,omitempty"`
PublishedAlready bool `json:"control-plane-image-already-published,omitempty"`
Registry string `json:"registry,omitempty"`
RegistryReplied int `json:"registry-replied,omitempty"`
RegistryKnown bool `json:"registry-already-registered,omitempty"`
RegistryRunning string `json:"registry-container,omitempty"`
// RegistryTrusted are the names the runtime was told to trust the registry by, before the push.
RegistryTrusted []string `json:"registry-trusted,omitempty"`
PublishedAs string `json:"control-plane-image,omitempty"`
PublishedAlready bool `json:"control-plane-image-already-published,omitempty"`
// Permanent is the control plane as an ordinary module.
Permanent string `json:"permanent-container,omitempty"`
@@ -555,7 +562,9 @@ func Run(ctx context.Context, o Options, d Deps, say func(string)) (Result, erro
// Named rather than attempted. Everything from step 6 on is a conversation with a
// control plane that a dry run has not raised, so there is nothing to ask and nothing
// honest to report about the answers.
say(" would then enrol " + o.Node + ", install the registry from " +
say(" would then enrol " + o.Node + ", have the runtime trust the registry as " +
strings.Join(RegistryNames(o), ", ") + ",")
say(" install the registry from " +
o.Catalogue + ", push the control plane's image into it,")
say(" reinstall the control plane as a module, and drop " +
rewritten.TempName)
@@ -654,26 +663,10 @@ func Run(ctx context.Context, o Options, d Deps, say func(string)) (Result, erro
return result, failed(StepOperator, err)
}
// ---- 7. registry ----------------------------------------------------------------------
say("registry — somewhere for this mesh to keep its own images")
registry, err := InstallRegistry(ctx, o, d, temporary, say)
result.Registry, result.RegistryRunning = registry.Address, registry.Container
result.RegistryKnown, result.RegistryReplied = registry.Known, registry.Answered
// ---- 7. registry, 8. publish -----------------------------------------------------------
published, err := RegistryAndPublish(ctx, o, d, sys, temporary, controlPlaneImage, &result, say)
if err != nil {
return result, failed(StepRegistry, err)
}
// ---- 8. publish -----------------------------------------------------------------------
say("publish — the control plane's image gets its first manifest digest")
// **The image this mesh built, not the one the installer carried.** The carried one is the
// builder; publishing it here would put a builder in the registry under the control plane's
// name and install it as the control plane — which is exactly what happened the first time
// this ran, and presented as a control plane that started, printed a builder's usage, and
// exited cleanly over and over.
published, err := PublishControlPlane(ctx, o, d, controlPlaneImage, say)
result.PublishedAs, result.PublishedAlready = published.Reference, published.Already
if err != nil {
return result, failed(StepPublish, err)
return result, err
}
// ---- 9. control plane -----------------------------------------------------------------
@@ -803,3 +796,44 @@ func Run(ctx context.Context, o Options, d Deps, say func(string)) (Result, erro
return result, nil
}
// RegistryAndPublish is steps 7 and 8: the runtime trusts the mesh's registry, the registry is
// installed and answers, and the control plane's image is pushed into it.
//
// **One function so the order is held by a test** (novox/hq issue 92). The runtime refuses plain
// HTTP to a registry it has not been told to trust, and the module that tells it is applied long
// after the push — so the trust is written and confirmed here, before anything is tagged or pushed.
// A refusal names the step it happened in.
func RegistryAndPublish(ctx context.Context, o Options, d Deps, sys system.System, temporary controlPlane,
controlPlaneImage string, result *Result, say func(string)) (Published, error) {
// ---- 7. registry ----------------------------------------------------------------------
say("registry — somewhere for this mesh to keep its own images")
// Trusted by the runtime before anything is pushed to it, by every name the mesh will use for
// it here: the module that writes this trust is applied long after the push (novox/hq issue 92).
trusted, err := TrustTheRegistry(ctx, o, sys, d.Run, say)
result.RegistryTrusted = trusted
if err != nil {
return Published{}, failed(StepRegistry, err)
}
registry, err := InstallRegistry(ctx, o, d, temporary, say)
result.Registry, result.RegistryRunning = registry.Address, registry.Container
result.RegistryKnown, result.RegistryReplied = registry.Known, registry.Answered
if err != nil {
return Published{}, failed(StepRegistry, err)
}
// ---- 8. publish -----------------------------------------------------------------------
say("publish — the control plane's image gets its first manifest digest")
// **The image this mesh built, not the one the installer carried.** The carried one is the
// builder; publishing it here would put a builder in the registry under the control plane's
// name and install it as the control plane — which is exactly what happened the first time
// this ran, and presented as a control plane that started, printed a builder's usage, and
// exited cleanly over and over.
published, err := PublishControlPlane(ctx, o, d, controlPlaneImage, say)
result.PublishedAs, result.PublishedAlready = published.Reference, published.Already
if err != nil {
return published, failed(StepPublish, err)
}
return published, nil
}
+36 -26
View File
@@ -38,27 +38,6 @@ func recording(t *testing.T) (*controlRecorder, controlPlane) {
return c, controlPlane{container: "temp-mesh-controller", run: c.run, timeout: time.Second}
}
func TestTheBuildersPackageRegistryPortIsASettingAndNotTheManifest(t *testing.T) {
c, control := recording(t)
o := Options{Node: "anchor", Ports: FoundationPorts{Packages: 3100}, Wait: time.Second}
if _, err := installModule(context.Background(), o, control, BuilderModule,
[]byte(theBuildersBinding), quietly); err != nil {
t.Fatal(err)
}
if got := c.settings[BuilderModule+"-settings.json"]; got != `{"serves":{"port":3100}}` {
t.Errorf("the builder was told %q about the package registry's port", got)
}
// The manifest reaches the mesh as the catalogue wrote it. A port rewritten into it here is a
// port the next registration from the catalogue silently takes back.
if got := c.settings[BuilderModule+"-module.json"]; got != theBuildersBinding {
t.Errorf("the installer changed the builder's manifest:\n%s", got)
}
set, push := c.index("settings set "+BuilderModule), c.index("push anchor")
if set < 0 || push < 0 || set > push {
t.Fatalf("the port was not set before the push that raises the builder: %v", c.told)
}
}
func TestTheBuilderOnTheDefaultPackagesPortIsToldNothing(t *testing.T) {
c, control := recording(t)
o := Options{Node: "anchor", Wait: time.Second}
@@ -169,21 +148,26 @@ func TestAForgeManifestTheCatalogueDoesNotHaveIsNamed(t *testing.T) {
}
}
// theCataloguesModule is the catalogue's own manifest of a module, never a copy: the clone beside a merge
// check (MESH_CHECK_BESIDE), where it must be; else a checkout beside this one; else the test skips aloud.
// theCataloguesModule is a module's own manifest, never a copy: the clone beside a merge check (MESH_CHECK_BESIDE), where it
// must be; else a checkout beside this one; else the test skips aloud. The catalogue holds it, but for the build agent,
// which moved whole into the controller's repository (ADR 0420, issue 1019, issue 1151) and is read from there.
func theCataloguesModule(t *testing.T, module string) []byte {
t.Helper()
inside := filepath.Join("mesh-catalog", "modules", module, "module.json")
repository := "mesh-catalog"
if module == BuilderModule {
repository = "mesh-controller"
}
inside := filepath.Join(repository, "modules", module, "module.json")
if root := os.Getenv("MESH_CHECK_BESIDE"); root != "" {
raw, err := os.ReadFile(filepath.Join(root, inside))
if err != nil {
t.Fatalf("the catalogue beside this check (MESH_CHECK_BESIDE=%s) has no %s: %v", root, inside, err)
t.Fatalf("%s beside this check (MESH_CHECK_BESIDE=%s) has no %s: %v", repository, root, inside, err)
}
return raw
}
raw, err := os.ReadFile(filepath.Join("..", "..", "..", inside))
if err != nil {
t.Skipf("NOT JUDGED: no catalogue beside this checkout and no MESH_CHECK_BESIDE; the merge check judges it")
t.Skipf("NOT JUDGED: no %s beside this checkout and no MESH_CHECK_BESIDE; the merge check judges it", repository)
}
return raw
}
@@ -238,3 +222,29 @@ func TestABuildAgentManifestWithNowhereForTheCarriedBuilderIsRefused(t *testing.
t.Fatalf("a manifest with no image artifact was pinned: %v", err)
}
}
// 1151: the build agent's manifest lives in the controller's repository now (ADR 0420, issue 1019), and a merge check of
// this repository has the controller cloned beside it, so a tree shaped like that check's, whose catalogue no longer holds
// the build agent, is read from the controller.
func TestTheBuildAgentsManifestIsReadFromTheControllerBesideTheCheck(t *testing.T) {
beside := t.TempDir()
for path, body := range map[string]string{
"mesh-catalog/modules/nats/module.json": `{"module":"nats"}`,
"mesh-controller/modules/build-agent/module.json": `{"module":"build-agent"}`,
} {
full := filepath.Join(beside, path)
if err := os.MkdirAll(filepath.Dir(full), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(full, []byte(body), 0o644); err != nil {
t.Fatal(err)
}
}
t.Setenv("MESH_CHECK_BESIDE", beside)
if got := string(theCataloguesModule(t, BuilderModule)); got != `{"module":"build-agent"}` {
t.Errorf("read %s", got)
}
if got := string(theCataloguesModule(t, "nats")); got != `{"module":"nats"}` {
t.Errorf("a module the catalogue holds was read as %s", got)
}
}
+179
View File
@@ -0,0 +1,179 @@
package bootstrap
import (
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"encoding/pem"
"errors"
"fmt"
"os"
"regexp"
"strconv"
"strings"
"time"
"github.com/novox/mesh-host/internal/declaration"
)
// Adopting the bus is the mesh's planned step for replacing it (novox/hq ADR 0333 §3, ADR 0236).
//
// The adoption replaces the bus's image once: the module's built image takes the place of the upstream
// one the bundle raised, on the same streams, users and certificate. The controller refuses a push that
// would replace the bus — "a planned step: `bus upgrade --why …` snapshots its streams first and checks
// them after" — so genesis stopped at step 18 (issue 590). The installer takes that step: it snapshots the
// streams itself, because at genesis no backup holder exists to be asked, and hands the snapshot to
// `bus upgrade`, which sends the machine, opens `bus-maintenance`, and ends the step done only once the
// bus answers healthy again — every stream, every durable consumer, a round trip (H-bus). The installer
// waits for that end.
// adoptionWhy is what the step is recorded as, for the record of hand acts and the bus's step.
const adoptionWhy = "genesis adopts its bus as the nats module (ADR 0333)"
// Where the installer keeps the streams' snapshot before the adoption, and the credential the snapshot
// program reads while it runs. Beside genesis's own credentials; variables so a test writes elsewhere.
var (
busSnapshotFile = "/var/lib/mesh-genesis/bus-before-adoption.tar"
busSnapshotCredential = "/var/lib/mesh-genesis/bus-snapshot.credential"
busCertificateFile = "/var/lib/mesh-broker-tls/tls.crt"
busControllerSecret = BusControllerFile
// busStepWithin bounds the wait for the controller to say the step ended: its own bound (15 min,
// mesh-controller busStepBound), and its doctor's next look after that.
busStepWithin = 17 * time.Minute
)
// adoptTheBus sends the nats module to this machine through the bus's planned step. A bus already on the
// module's image is the module's: it is sent like any module, and nothing is replaced.
func adoptTheBus(ctx context.Context, o Options, control controlPlane, bus *declaration.Container,
say func(string)) error {
if running := runningImage(ctx, control, bus.Name); running != "" && running != bus.Image {
say(" already adopted " + bus.Name + " — it runs the module's image")
_, err := pushNode(ctx, o, control, say)
return err
}
image, err := theModulesBusImage(ctx, o, control)
if err != nil {
return err
}
if err := snapshotTheBus(ctx, o, control, image, say); err != nil {
return err
}
say(" restarting " + bus.Name + " — once, onto the " + BusModule + " module's image, as the bus's " +
"planned step; its streams, users and certificate stay where they are (novox/hq ADR 0333 §3, ADR 0236)")
if _, err := control.within(2*o.Wait).tell(ctx, "bus", "upgrade", "--why", adoptionWhy,
"--reversible", "--snapshot-taken", busSnapshotFile); err != nil {
return fmt.Errorf("the bus's planned step was refused, so the bus is not replaced: %w", err)
}
return nil
}
// meshBuiltImage is an image the mesh built, as it records it: `artifact-store://<module>/<artifact>@sha256:…`.
var meshBuiltImage = regexp.MustCompile(`artifact-store://([a-z0-9][a-z0-9-]*/[a-z0-9][a-z0-9-]*@sha256:[0-9a-f]{64})`)
// theModulesBusImage is the nats module's server image, as the build just made it, pulled from this
// mesh's registry: the image that carries the snapshot program.
func theModulesBusImage(ctx context.Context, o Options, control controlPlane) (string, error) {
said, err := control.tell(ctx, "builds", BusModule, "-n", "1")
if err != nil {
return "", err
}
for _, m := range meshBuiltImage.FindAllStringSubmatch(said, -1) {
if strings.HasPrefix(m[1], BusModule+"/") {
return strings.TrimSuffix(o.Registry, "/") + "/" + m[1], nil
}
}
return "", fmt.Errorf("the mesh says no image it built for the %s module, so its snapshot program cannot "+
"be run before the bus is replaced:\n%s", BusModule, indent(strings.TrimSpace(said)))
}
// snapshotTheBus writes every stream of the running bus to busSnapshotFile with the module's own snapshot
// program (mesh-catalog modules/nats, ADR 0235), as the controller's account, pinned to the bus's
// certificate. The credential is written for the program's one run and removed after it.
func snapshotTheBus(ctx context.Context, o Options, control controlPlane, image string, say func(string)) error {
password, err := readCredentialFile(busControllerSecret)
if err != nil {
return fmt.Errorf("the controller's bus credential is not at %s, so the streams cannot be snapshotted "+
"before the bus is replaced: %w", busControllerSecret, err)
}
fingerprint, err := certificateFingerprint(busCertificateFile)
if err != nil {
return err
}
port := o.Ports.orDefaults().Bus
server := "tls://127.0.0.1:" + strconv.Itoa(port)
credential, err := json.Marshal(map[string]string{"url": server, "fingerprint": fingerprint,
"user": busControllerUser, "password": password})
if err != nil {
return err
}
if err := os.WriteFile(busSnapshotCredential, credential, 0o600); err != nil {
return fmt.Errorf("the snapshot's credential could not be written: %w", err)
}
defer os.Remove(busSnapshotCredential)
script := "docker run --rm -i --network host --entrypoint /usr/local/bin/mesh-nats-snapshot " + image +
" snapshot --server " + server + " --credential - < " + busSnapshotCredential + " > " + busSnapshotFile +
".partial && mv " + busSnapshotFile + ".partial " + busSnapshotFile
running, cancel := context.WithTimeout(ctx, busStepWithin)
defer cancel()
if out, err := control.run(running, "sh", "-c", script); err != nil {
_, _ = control.run(ctx, "rm", "-f", busSnapshotFile+".partial")
return fmt.Errorf("the bus's streams could not be snapshotted, so the bus is not replaced: %w\n%s",
err, indent(strings.TrimSpace(out)))
}
say(" snapshotted every stream of the bus, at " + busSnapshotFile + " (its module's snapshot program)")
return nil
}
// certificateFingerprint is the pin of the bus's certificate: sha256 of its DER, as every machine pins it.
func certificateFingerprint(path string) (string, error) {
raw, err := os.ReadFile(path)
if err != nil {
return "", fmt.Errorf("the bus's certificate is not at %s, so it cannot be pinned for the snapshot: %w", path, err)
}
block, _ := pem.Decode(raw)
if block == nil {
return "", fmt.Errorf("%s holds no PEM certificate", path)
}
sum := sha256.Sum256(block.Bytes)
return "sha256:" + hex.EncodeToString(sum[:]), nil
}
// waitForTheBusStep waits until the controller says the bus's planned step ended: done once the bus
// answered healthy — every stream, every durable consumer, a round trip — or failed, with its snapshot as
// the way back. The step's own bound is the controller's; this waits a little past it.
func waitForTheBusStep(ctx context.Context, o Options, control controlPlane, say func(string)) error {
deadline := time.Now().Add(busStepWithin)
var last string
for {
said, err := control.tell(ctx, "bus")
if err == nil {
for _, line := range strings.Split(said, "\n") {
if strings.HasPrefix(strings.TrimSpace(line), "last step ") {
last = strings.TrimSpace(line)
}
}
switch {
case strings.Contains(last, ": done at "):
say(" bus step done " + last)
return nil
case strings.Contains(last, ": failed at "):
return fmt.Errorf("the bus's planned step ended failed: %s\n%s", last, indent(strings.TrimSpace(said)))
}
}
if !time.Now().Before(deadline) {
break
}
select {
case <-ctx.Done():
return ctx.Err()
case <-time.After(answerEvery):
}
}
if last == "" {
return errors.New("the controller says no bus step after the adoption was sent; `mesh-controller bus` says how it stands")
}
return fmt.Errorf("the bus's planned step did not end within %s: %s", busStepWithin, last)
}
+135
View File
@@ -0,0 +1,135 @@
package bootstrap
import (
"context"
"encoding/json"
"os"
"strings"
"testing"
"time"
)
// Defends novox/hq issue 590 at its broker step (ADR 0333 §3, ADR 0236): the adoption replaces the bus's
// image, and the mesh replaces its bus only as a planned step. A plain push was refused — "sending anchor
// would replace the bus … a planned step: `bus upgrade --why …`" — and genesis stopped at step 18.
func TestTheBrokerStepAdoptsTheBusThroughItsPlannedStep(t *testing.T) {
catalogue := aCatalogue(t, map[string]string{BusModule: aBusModule(theModulesLayout)})
rewritten, err := Rewrite(theRealBundle(t), held)
if err != nil {
t.Fatal(err)
}
bundleImage := containerNamed(rewritten.Declaration, "mesh-broker-nats").Image
upgraded := false
var credential []byte
runtime := &asked{}
agrees := aMeshThatAgrees(nil)
runtime.answer = func(name string, args []string) (string, error) {
joined := strings.Join(args, " ")
switch {
case name == "sh" && strings.Contains(joined, "mesh-nats-snapshot"):
credential, _ = os.ReadFile(busSnapshotCredential) // as the program reads it
case strings.Contains(joined, "bus upgrade --why"):
upgraded = true
case strings.Contains(joined, "inspect --format {{.Config.Image}} mesh-broker-nats"):
if upgraded {
return theModulesImage + "\n", nil
}
return bundleImage + "\n", nil
}
return agrees(name, args)
}
_, fetch := aBusThatAdopts()
o := installing(t, catalogue)
o.CatalogSource = Source{Repository: "git://forge/mesh-catalog.git"}
o.Wait = 5 * time.Second
said, err := busStepWith(t, o, runtime, fetch)
if err != nil {
t.Fatal(err)
}
snapshot, upgrade := -1, -1
for i, c := range runtime.commands {
switch {
case strings.HasPrefix(c, "sh -c") && strings.Contains(c, "mesh-nats-snapshot"):
snapshot = i
for _, want := range []string{"127.0.0.1:5000/nats/server@sha256:" + strings.Repeat("4", 64),
"snapshot --server tls://127.0.0.1:5671 --credential - < " + busSnapshotCredential,
"> " + busSnapshotFile + ".partial && mv " + busSnapshotFile + ".partial " + busSnapshotFile} {
if !strings.Contains(c, want) {
t.Errorf("the snapshot is not taken with %q: %s", want, c)
}
}
case strings.Contains(c, "/mesh-controller bus upgrade"):
upgrade = i
if want := "bus upgrade --why " + adoptionWhy + " --reversible --snapshot-taken " + busSnapshotFile; !strings.HasSuffix(c, want) {
t.Errorf("the planned step is asked as %q; want it to end %q", c, want)
}
case strings.HasSuffix(c, "/mesh-controller push anchor"):
t.Errorf("the bus's module was sent by a plain push, which the mesh refuses for the bus: %v", runtime.commands)
}
}
if snapshot < 0 || upgrade < 0 || snapshot > upgrade {
t.Fatalf("the streams were not snapshotted before the bus's planned step (snapshot %d, upgrade %d): %v",
snapshot, upgrade, runtime.commands)
}
var cred map[string]string
if err := json.Unmarshal(credential, &cred); err != nil {
t.Fatalf("the snapshot program was given no credential: %v", err)
}
fingerprint, _ := certificateFingerprint(busCertificateFile)
if cred["user"] != "controller" || cred["password"] != "BUS-PW" || cred["fingerprint"] != fingerprint ||
cred["url"] != "tls://127.0.0.1:5671" {
t.Errorf("the snapshot's credential is %v", cred)
}
if _, err := os.Stat(busSnapshotCredential); !os.IsNotExist(err) {
t.Error("the snapshot's credential was left on the machine")
}
text := strings.Join(said, "\n")
for _, want := range []string{"snapshotted every stream", "restarting mesh-broker-nats — once",
"bus answers mesh-broker-nats", "bus step done last step 1", "adopted mesh-broker-nats"} {
if !strings.Contains(text, want) {
t.Errorf("the step did not say %q:\n%s", want, text)
}
}
}
// A planned step the controller ends failed is the broker step failing, with the controller's words.
func TestABusStepThatEndsFailedIsRefused(t *testing.T) {
aGenesisBusOnDisk(t)
runtime := &asked{answer: func(name string, args []string) (string, error) {
return "last step 1: not known → 53e51864 on anchor by installer (x): failed at 2026-10-13 11:59; snapshot s\n" +
" stream KV_x has no messages\n", nil
}}
control := controlPlane{container: "mesh-controller", run: runtime.run, timeout: time.Second}
err := waitForTheBusStep(context.Background(), Options{}, control, quietly)
if err == nil || !strings.Contains(err.Error(), "failed at") || !strings.Contains(err.Error(), "KV_x") {
t.Fatalf("a failed bus step was not refused with its words: %v", err)
}
}
// The bundle's bus runs the release the module's image is built on, pinned by the same digest, so the
// adoption changes no server release and is reversible as the step says (ADR 0333 §3).
func TestTheBundlesBusRunsTheReleaseTheModulesImageIsBuiltOn(t *testing.T) {
var m struct {
Build struct {
On []struct{ Arg, Image string } `json:"on"`
} `json:"build"`
}
if err := json.Unmarshal(theCataloguesModule(t, BusModule), &m); err != nil {
t.Fatal(err)
}
base := ""
for _, on := range m.Build.On {
if on.Arg == "NATS_BASE" {
base = on.Image
}
}
rewritten, err := Rewrite(theRealBundle(t), held)
if err != nil {
t.Fatal(err)
}
if got := containerNamed(rewritten.Declaration, "mesh-broker-nats").Image; got != base {
t.Errorf("the bundle raises the bus on %s; the nats module's image is built on %s", got, base)
}
}
@@ -0,0 +1,65 @@
package bootstrap
import (
"context"
"encoding/json"
"strings"
"testing"
"time"
)
// Genesis seals the build agent the forge it raised (novox/hq ADR 0336 §3, issue 121): the agent's
// npm-package-registry requirement is optional and binds only once the gitea module is assigned, after
// the base, so until then the packages step's genesis document is how the sdk and base steps reach the
// forge.
func TestThePackagesStepSealsTheGenesisDocumentToTheBuildAgent(t *testing.T) {
if BuilderModule != "build-agent" {
t.Fatalf("genesis installs %q; the document is the build agent's", BuilderModule)
}
c, control := recording(t)
o := Options{Node: "anchor", Ports: FoundationPorts{Packages: 3100}, Wait: time.Second}
doc := genesisRegistry(o.Ports.orDefaults(), "builder-password")
if err := deliverGenesisRegistry(context.Background(), o, control, doc, quietly); err != nil {
t.Fatal(err)
}
carried := c.settings["accepting-"+GenesisRegistrySecret]
var got GenesisRegistry
if err := json.Unmarshal([]byte(carried), &got); err != nil {
t.Fatalf("the genesis document carried is not JSON: %q", carried)
}
want := GenesisRegistry{Scheme: "http", At: "127.0.0.1", Port: 3100, NpmPath: "/api/packages/novox/npm/",
As: "mesh-builder", Password: "builder-password"}
if got != want {
t.Fatalf("the genesis document says %+v, want %+v", got, want)
}
accept := c.index("secret accept anchor build-agent genesis-package-registry --from /accepting-genesis-package-registry")
push := c.index("push anchor")
if accept < 0 || push < 0 || accept > push {
t.Fatalf("the genesis document was not sealed to the build agent before the push: %v", c.told)
}
}
// What the retired builder was given and the build agent declares neither of: an `npm-password`
// own-secret and a `serves` setting for its carried binding (ADR 0336 §3).
func TestNoStepGivesTheBuildAgentAServesSettingOrAnNpmPassword(t *testing.T) {
for _, module := range []string{BuilderModule} {
c, control := recording(t)
o := Options{Node: "anchor", Ports: FoundationPorts{Packages: 3100}, Wait: time.Second}
if _, err := installModule(context.Background(), o, control, module, []byte(`{"module":"`+module+`"}`), quietly); err != nil {
t.Fatal(err)
}
if got, set := c.settings[module+"-settings.json"]; set {
t.Errorf("%s was given a setting on a node with a non-default packages port: %s", module, got)
}
if err := deliverGenesisRegistry(context.Background(), o, control,
genesisRegistry(o.Ports.orDefaults(), "p"), quietly); err != nil {
t.Fatal(err)
}
for _, told := range c.told {
if strings.Contains(told, "npm-password") || strings.Contains(told, "serves") {
t.Errorf("genesis told the controller %q", told)
}
}
}
}
@@ -0,0 +1,89 @@
package bootstrap
import (
"context"
"encoding/json"
"strings"
"testing"
"time"
)
// Defends novox/hq issue 616: genesis's base step builds the toolchain of every language a module
// genesis installs is compiled in. It built only mesh-tools, and the store's module, with Go code,
// stopped genesis at step 17: "this mesh holds no copy of it. Build mesh-tools-go first".
// genesisModules is every catalogue module genesis installs on its own, read from the steps that
// install them: the registry, the build agent, the store, the bus, the vault, the catalogue and the
// filter (extras are the operator's choice and come after the base too, but are not genesis's own).
var genesisModules = []string{RegistryModule, BuilderModule, "postgres", BusModule, "mesh-vault", "mesh-catalog", "nftables"}
// Every genesis module's bundles are compiled in a toolchain the base step builds, read from the real
// catalogue's manifests: a module that gains a language genesis does not build fails here, not at a
// genesis.
func TestTheBaseStepBuildsEveryToolchainAGenesisModuleIsCompiledIn(t *testing.T) {
built := map[string]bool{}
for _, tc := range genesisToolchains {
built[tc.Language] = true
}
for _, module := range genesisModules {
var m struct {
Build struct {
Artifacts []struct {
Name, Kind, Language string
} `json:"artifacts"`
} `json:"build"`
}
if err := json.Unmarshal(theCataloguesModule(t, module), &m); err != nil {
t.Fatalf("%s: %v", module, err)
}
for _, a := range m.Build.Artifacts {
if a.Kind != "bundle" {
continue // an image builds on the bases its Dockerfile names, pinned upstream
}
if !built[a.Language] {
t.Errorf("%s's %q is a %s bundle, and genesis's base step builds no %s toolchain",
module, a.Name, a.Language, a.Language)
}
}
}
}
// And the base step asks for each toolchain's build: the base, then the Go toolchain beside it.
func TestTheBaseStepBuildsTheGoToolchainBesideTheBase(t *testing.T) {
runtime := &asked{answer: aMeshThatAgrees(nil)}
control := controlPlane{container: "mesh-controller", run: runtime.run, timeout: time.Second}
o := Options{ToolsSource: Source{Repository: "git://192.0.2.30/mesh-tools.git"}}
if err := BuildBase(context.Background(), o, control, quietly); err != nil {
t.Fatal(err)
}
var builds []string
for _, c := range runtime.commands {
if strings.Contains(c, "/mesh-controller build ") {
builds = append(builds, c[strings.Index(c, "build "):])
}
}
want := []string{
"build git://192.0.2.30/mesh-tools.git --ref main --wait 1200s",
"build git://192.0.2.30/mesh-tools-go.git --ref main --wait 1200s",
}
if strings.Join(builds, "\n") != strings.Join(want, "\n") {
t.Fatalf("the base step built\n%s\nand not\n%s", strings.Join(builds, "\n"), strings.Join(want, "\n"))
}
}
func TestTheGoToolchainIsAsGivenOrBesideTheBase(t *testing.T) {
for _, c := range []struct{ tools, given, want string }{
{"http://forge/novox/mesh-tools.git", "", "http://forge/novox/mesh-tools-go.git"},
{"http://forge/novox/mesh-tools", "", "http://forge/novox/mesh-tools-go"},
{"http://forge/other/base.git", "http://forge/other/go.git", "http://forge/other/go.git"},
} {
got, err := toolsGoSource(Options{ToolsSource: Source{Repository: c.tools}, ToolsGoSource: Source{Repository: c.given}})
if err != nil || got.Repository != c.want {
t.Errorf("tools %s, given %q: %q %v; want %s", c.tools, c.given, got.Repository, err, c.want)
}
}
if _, err := toolsGoSource(Options{ToolsSource: Source{Repository: "http://forge/other/base.git"}}); err == nil ||
!strings.Contains(err.Error(), "--tools-go-source") {
t.Errorf("a base not named mesh-tools, and no Go toolchain given, was not refused naming the flag: %v", err)
}
}
+34 -2
View File
@@ -137,7 +137,26 @@ func registerAndAssign(ctx context.Context, o Options, control controlPlane, mod
// should hold, seals every secret in them and publishes them, and on a first node that is the
// slowest thing the mesh does.
func pushNode(ctx context.Context, o Options, control controlPlane, say func(string)) (string, error) {
said, err := control.within(o.Wait).tell(ctx, "push", o.Node)
return pushing(ctx, o, control, false, say)
}
// pushNodeApplied is pushNode that returns only once the node reports it applied exactly what it was
// sent (`push <node> --wait`, novox/hq ADR 0010): for a push whose content the very next step reads
// on the machine. A plain push returns as soon as the declaration is sent, and the node applies it a
// moment later (issue 615).
func pushNodeApplied(ctx context.Context, o Options, control controlPlane, say func(string)) (string, error) {
return pushing(ctx, o, control, true, say)
}
func pushing(ctx context.Context, o Options, control controlPlane, applied bool, say func(string)) (string, error) {
args := []string{"push", o.Node}
patience := o.Wait
if applied {
args = append(args, "--wait", o.Wait.String())
// The controller's own wait, and time to compose and send before it starts.
patience = 2 * o.Wait
}
said, err := control.within(patience).tell(ctx, args...)
if err != nil {
return "", fmt.Errorf(
"%w\n\nWhat was registered and assigned is registered and assigned, and this node has "+
@@ -145,7 +164,20 @@ func pushNode(ctx context.Context, o Options, control controlPlane, say func(str
"at all. Fix what it named and run this installer again: it will find the module "+
"already registered and try the push again", err)
}
say(" pushed " + o.Node)
if applied {
say(" pushed " + o.Node + ", and it applied it")
} else {
say(" pushed " + o.Node)
}
// **Every push may carry a new bus user, and until the adoption the bus hears of it only from here**
// (novox/hq issue 608, ADR 0331 §5, ADR 0333 §4). A module's account is minted into the mesh's
// records and sealed to this node with the push; the bus reads its user list from a file, and until
// the broker step adopts the bus as the nats module nothing but this installer writes that file. The
// list was placed only when the node enrolled, so the build agent's account (step 13) never reached
// the bus and the agent was refused ("Authorization Violation") until the sdk step gave up.
if err := placeUntilAdopted(ctx, control, say); err != nil {
return "", err
}
return strings.TrimSpace(said), nil
}
+47 -5
View File
@@ -25,20 +25,62 @@ import (
// buildWait bounds one module build. Generous, because the first build compiles a toolchain.
const buildWait = 20 * time.Minute
// BuildBase asks the mesh to build the shared base every module with code of its own stands on.
// BuildBase asks the mesh to build the shared base every module with code of its own stands on, and
// the toolchain of every language genesis's own modules are written in.
//
// **First, because until it exists nothing else with code can be built.** Not registered as a
// module here: it is never assigned — it runs nowhere — and the build itself records what was
// made, which is all anything downstream reads.
//
// **Every toolchain a genesis module compiles in** (novox/hq issue 616). A bundle is compiled in its
// language's base (mesh-controller internal/builder toolchain.go): TypeScript in mesh-tools, Go in
// mesh-tools-go. The store's and the bus's modules carry Go code, so a genesis that built only
// mesh-tools stopped at the store step: "this mesh holds no copy of it. Build mesh-tools-go first".
// The bases are the ones genesisToolchains names, and a test holds them to the languages of every
// module genesis installs, read from the catalogue.
func BuildBase(ctx context.Context, o Options, control controlPlane, say func(string)) error {
if o.ToolsSource.Repository == "" {
return fmt.Errorf("phase two needs --tools-source: the shared base is built from its " +
"own repository, and an installer told nothing cannot know where that is")
}
say(" building " + o.ToolsSource.Repository + " at " + refOr(o.ToolsSource.Ref))
_, err := control.within(buildWait).tell(ctx,
"build", o.ToolsSource.Repository, "--ref", refOr(o.ToolsSource.Ref), "--wait", "1200s")
return err
goSource, err := toolsGoSource(o)
if err != nil {
return err
}
for _, src := range []Source{o.ToolsSource, goSource} {
say(" building " + src.Repository + " at " + refOr(src.Ref))
if _, err := control.within(buildWait).tell(ctx,
"build", src.Repository, "--ref", refOr(src.Ref), "--wait", "1200s"); err != nil {
return err
}
}
return nil
}
// genesisToolchains is the base each language's bundles compile in, for the languages genesis's own
// modules are written in — the same pairs as mesh-controller internal/builder toolchain.go — in the
// order BuildBase builds them.
var genesisToolchains = []struct{ Language, Base string }{
{"typescript", "mesh-tools"},
{"go", "mesh-tools-go"},
}
// toolsGoSource is where the Go toolchain is built from: as given, or the repository beside the base's
// named mesh-tools-go, which is how the mesh keeps its repositories side by side.
func toolsGoSource(o Options) (Source, error) {
if o.ToolsGoSource.Repository != "" {
return o.ToolsGoSource, nil
}
repo := strings.TrimSuffix(o.ToolsSource.Repository, "/")
suffix := ""
if strings.HasSuffix(repo, ".git") {
repo, suffix = strings.TrimSuffix(repo, ".git"), ".git"
}
if !strings.HasSuffix(repo, "mesh-tools") {
return Source{}, fmt.Errorf("the Go toolchain is built from mesh-tools-go, and --tools-source (%s) is not a "+
"repository named mesh-tools to find it beside; give --tools-go-source", o.ToolsSource.Repository)
}
return Source{Repository: repo + "-go" + suffix, Ref: o.ToolsGoSource.Ref}, nil
}
// InstallFromCatalogue builds a catalogue module and installs it on this machine.
+16 -22
View File
@@ -84,7 +84,7 @@ func InstallStore(ctx context.Context, o Options, control controlPlane,
say); err != nil {
return err
}
say(" adopted mesh-store — the foundation's store is now the " + module + " module")
say(" adopted " + store.Name + " — the foundation's store is now the " + module + " module")
return nil
}
@@ -93,7 +93,8 @@ func InstallStore(ctx context.Context, o Options, control controlPlane,
// be in it. InstallFromCatalogue is the same sequence without that moment; the store, the broker
// and the vault each need it or need the shape, and three copies of it drifted.
func installProvider(ctx context.Context, o Options, control controlPlane, module string,
manifest []byte, buildNote string, beforePush func() error, say func(string)) error {
manifest []byte, buildNote string, beforePush func() error, say func(string),
send ...func() error) error {
remote := "/" + module + "-module.json"
if err := control.carrying(ctx, module+"-module.json", manifest, remote); err != nil {
@@ -132,6 +133,11 @@ func installProvider(ctx context.Context, o Options, control controlPlane, modul
return err
}
}
// A module whose first send is not a plain push says how it is sent: the bus, whose replacement is a
// planned step (ADR 0236), is sent by that step (adoptTheBus).
if len(send) > 0 && send[0] != nil {
return send[0]()
}
_, err := pushNode(ctx, o, control, say)
return err
}
@@ -176,9 +182,10 @@ const BusModule = "nats"
// **The image is the one thing that changes, once** (ADR 0333 §3). The store's module pins the upstream
// postgres the foundation runs; the bus's module builds its server image (it carries the mesh's snapshot
// tool), so no pinned reference can be equal to the foundation's. The applier recreates the container on
// the module's image, on the same streams, users and certificate. This says so before the push and waits
// after it until the container runs the module's image and the bus answers, so nothing after this step
// dials a bus that is restarting.
// the module's image, on the same streams, users and certificate. The mesh replaces its bus only as a
// planned step (ADR 0236), so the module is sent by that step: its streams snapshotted first, the restart
// said, and the installer waits until the container runs the module's image, the bus answers, and the
// controller ends the step done (bus_adoption.go), so nothing after this step dials a bus that is restarting.
func InstallBroker(ctx context.Context, o Options, control controlPlane,
foundation *declaration.Declaration, fetch func(context.Context, string) (int, string, error),
say func(string)) error {
@@ -197,20 +204,15 @@ func InstallBroker(ctx context.Context, o Options, control controlPlane,
}
say(" adopting " + bus.Name + " — the bus the foundation raised, its streams kept")
if err := installProvider(ctx, o, control, module, manifest, "(the server, adopting the foundation's bus)",
func() error {
if running := runningImage(ctx, control, bus.Name); running != "" && running != bus.Image {
say(" already restarted " + bus.Name + " — it runs the module's image")
return nil
}
say(" restarting " + bus.Name + " — once, onto the " + module + " module's image; its " +
"streams, users and certificate stay where they are (novox/hq ADR 0333 §3)")
return nil
}, say); err != nil {
nil, say, func() error { return adoptTheBus(ctx, o, control, bus, say) }); err != nil {
return err
}
if err := waitForTheAdoptedBus(ctx, o, control, bus, fetch, say); err != nil {
return err
}
if err := waitForTheBusStep(ctx, o, control, say); err != nil {
return err
}
say(" adopted " + bus.Name + " — the foundation's bus is now the " + module +
" module, and its user list is the module's from here on")
return nil
@@ -478,11 +480,3 @@ func serverMatchesFoundation(manifest []byte, store *declaration.Container, modu
"store with. Its server container has to carry the name the foundation raised",
module, store.Name)
}
// orUnknown is s, or a word for nothing: trust.go held it, and this branch takes trust.go out.
func orUnknown(s string) string {
if strings.TrimSpace(s) == "" {
return "(nothing)"
}
return s
}
+30 -2
View File
@@ -69,6 +69,7 @@ func aBusThatAdopts() (*asked, func(context.Context, string) (int, string, error
func busStepWith(t *testing.T, o Options, runtime *asked,
fetch func(context.Context, string) (int, string, error)) ([]string, error) {
t.Helper()
aGenesisBusOnDisk(t)
rewritten, err := Rewrite(theRealBundle(t), held)
if err != nil {
t.Fatal(err)
@@ -222,7 +223,7 @@ func TestTheBrokerStepSaysTheRestartAndWaitsForTheBusToAnswer(t *testing.T) {
runtime.answer = func(name string, args []string) (string, error) {
joined := strings.Join(args, " ")
switch {
case strings.Contains(joined, "push anchor"):
case strings.Contains(joined, "bus upgrade --why"):
pushed = true
case strings.Contains(joined, "inspect --format {{.Config.Image}} mesh-broker-nats"):
if !pushed {
@@ -304,7 +305,7 @@ func TestARerunAfterTheAdoptionSaysTheBusIsTheModulesAlready(t *testing.T) {
t.Fatal(err)
}
text := strings.Join(said, "\n")
if strings.Contains(text, "restarting") || !strings.Contains(text, "already restarted mesh-broker-nats") {
if strings.Contains(text, "restarting") || !strings.Contains(text, "already adopted mesh-broker-nats") {
t.Errorf("a re-run announced a restart, or did not say the bus is the module's already:\n%s", text)
}
}
@@ -417,3 +418,30 @@ func TestTheBundlesBusFilesAreSeeds(t *testing.T) {
}
}
}
// aGenesisBusOnDisk puts what the adoption reads on the machine in a test's own directory: the bus's
// certificate, the controller's bus credential, and where the snapshot goes. The step's wait is short.
func aGenesisBusOnDisk(t *testing.T) {
t.Helper()
dir := t.TempDir()
cert := filepath.Join(dir, "tls.crt")
if err := os.WriteFile(cert, []byte(aCertificatePEM), 0o644); err != nil {
t.Fatal(err)
}
secret := filepath.Join(dir, "bus-controller.secret")
if err := os.WriteFile(secret, []byte("BUS-PW"), 0o600); err != nil {
t.Fatal(err)
}
was := []string{busSnapshotFile, busSnapshotCredential, busCertificateFile, busControllerSecret}
wasWithin := busStepWithin
busSnapshotFile, busSnapshotCredential = filepath.Join(dir, "bus.tar"), filepath.Join(dir, "bus.credential")
busCertificateFile, busControllerSecret = cert, secret
busStepWithin = 50 * time.Millisecond
t.Cleanup(func() {
busSnapshotFile, busSnapshotCredential, busCertificateFile, busControllerSecret = was[0], was[1], was[2], was[3]
busStepWithin = wasWithin
})
}
// aCertificatePEM is a PEM block standing for the bus's certificate; its pin is the sha256 of these bytes.
const aCertificatePEM = "-----BEGIN CERTIFICATE-----\nAAECAwQFBgcICQ==\n-----END CERTIFICATE-----\n"
+67 -24
View File
@@ -2,6 +2,7 @@ package bootstrap
import (
"context"
"encoding/json"
"fmt"
"net/http"
"os"
@@ -24,7 +25,7 @@ import (
const (
// foundationStore is the foundation's postgres container — the mesh's own memory, raised from the
// bundle. gitea's bootstrap database lives here too, so a mesh runs one postgres (issue 051).
foundationStore = "mesh-store"
foundationStore = "postgres"
// giteaBootstrap is the gitea server raised directly at genesis, before gitea is a module —
// under the name the gitea MODULE declares for its container, so the module finds it and holds
// it rather than raising a second forge beside it (novox/hq ADR 0163, rule 7; issue 090).
@@ -49,8 +50,8 @@ const (
packagesTeam = "packages"
// giteaAdminUser is the admin the bootstrap creates and the provisioner later authenticates as.
giteaAdminUser = "mesh-admin"
// builderGiteaUser is the gitea account the builder publishes and pulls with at genesis. It is
// the `as` the builder's static package binding names.
// builderGiteaUser is the gitea account the build agent publishes and pulls with at genesis: the
// `as` of the genesis document (novox/hq ADR 0336 §3). Retiring it is issue 602.
builderGiteaUser = "mesh-builder"
// giteaDBRole/giteaDBName is gitea's own database in the foundation store.
giteaDBRole = "mesh_gitea"
@@ -133,8 +134,8 @@ func RaisePackageRegistry(ctx context.Context, o Options, d Deps, control contro
return err
}
say(" delivering the builder its registry credential")
if err := deliverBuilderNpm(ctx, o, control, builderPassword, say); err != nil {
say(" sealing the build agent the genesis document for the forge it raised")
if err := deliverGenesisRegistry(ctx, o, control, genesisRegistry(ports, builderPassword), say); err != nil {
return err
}
@@ -362,27 +363,69 @@ func packagePasswords() (db, admin, builder string, err error) {
return db, admin, builder, nil
}
// deliverBuilderNpm seals the builder's registry password to this node as its `npm-password`
// own-secret, the same way the control plane's store connections are delivered — carry the value in,
// `secret accept`, and the next push writes it sealed where the builder reads it.
func deliverBuilderNpm(ctx context.Context, o Options, control controlPlane, password string,
say func(string)) error {
at := "/accepting-npm-password"
if err := control.carryingSecret(ctx, "mesh-accepting-npm-password", []byte(password), at); err != nil {
return err
}
if _, err := control.tell(ctx, "secret", "accept", o.Node, BuilderModule, "npm-password", "--from", at); err != nil {
return err
}
// Push so the sealed secret reaches the builder, which restarts on it and comes back credentialed.
if _, err := control.tell(ctx, "push", o.Node); err != nil {
return err
}
return nil
// GenesisRegistrySecret is the build agent's own-secret the genesis document is sealed as (novox/hq
// ADR 0336 §3). The build-agent manifest declares it, issued outside the mesh: only this installer
// gives one, on genesis's node.
const GenesisRegistrySecret = "genesis-package-registry"
// GenesisRegistry is the genesis document (novox/hq ADR 0336 §3): the forge this step raised, the port
// this node gave the package registry, its npm path, and the account this step made with its password.
// mesh-controller's cmd/mesh-builder reads it under the same field names while the build agent's
// npm-package-registry requirement is unbound, and prefers the binding once the gitea module is
// assigned and it binds.
type GenesisRegistry struct {
Scheme string `json:"scheme"`
At string `json:"at"`
Port int `json:"port"`
NpmPath string `json:"npm-path"`
As string `json:"as"`
Password string `json:"password"`
}
// PublishTheSDK dispatches a build of the SDK to the builder. The builder has its registry
// credential by now, so the build's `npm publish` authenticates; the artifact is a `package`, built
// genesisRegistry is the document for the forge raised here: on the machine's own loopback, where it
// answers on the host network and where its ROOT_URL says it is (raiseGiteaServer), on the port this
// node gave the package registry.
func genesisRegistry(ports FoundationPorts, password string) GenesisRegistry {
return GenesisRegistry{Scheme: "http", At: "127.0.0.1", Port: ports.Packages,
NpmPath: "/api/packages/" + packagesOrg + "/npm/", As: builderGiteaUser, Password: password}
}
// deliverGenesisRegistry seals the genesis document to the build agent on this node as its
// `genesis-package-registry` own-secret (novox/hq ADR 0336 §3), the same way the control plane's store
// connections are delivered — carry the value in, `secret accept`, and the next push writes it sealed
// where the build agent reads it.
//
// **The build agent's one way to the forge until the registry binds.** Its npm-package-registry
// requirement is optional (ADR 0336 §1), and nothing provides it until the gitea module is assigned
// after the base, so the sdk step (15) publishes and the base step (16) resolves with the account in
// this document. It replaces the retired builder's `npm-password` own-secret and the `serves.port`
// setting its carried binding took, neither of which the build agent declares.
func deliverGenesisRegistry(ctx context.Context, o Options, control controlPlane, doc GenesisRegistry,
say func(string)) error {
raw, err := json.Marshal(doc)
if err != nil {
return err
}
at := "/accepting-" + GenesisRegistrySecret
if err := control.carryingSecret(ctx, "mesh-accepting-"+GenesisRegistrySecret, raw, at); err != nil {
return err
}
if _, err := control.tell(ctx, "secret", "accept", o.Node, BuilderModule, GenesisRegistrySecret, "--from", at); err != nil {
return err
}
// Push so the sealed document reaches the build agent, which reads it at its next build — and,
// like every push before the adoption, the bus hears the users it carries (issue 608).
//
// **Waited for until the node applied it** (issue 615). The next step asks the build agent to
// publish the SDK, and the agent reads the document from the file the node writes when it applies
// this push. A push that returned on sending let the ask race the apply: one genesis found the
// document in time, the next was refused, "neither its binding nor the genesis document".
_, err = pushNodeApplied(ctx, o, control, say)
return err
}
// PublishTheSDK dispatches a build of the SDK to the build agent. The agent has the genesis document by
// now (novox/hq ADR 0336 §7), so the build's `npm publish` authenticates; the artifact is a `package`, built
// on a public base, so this needs no toolchain — which is the whole point of doing it before the base.
func PublishTheSDK(ctx context.Context, o Options, control controlPlane, say func(string)) error {
if o.SDKSource.Repository == "" {
+2 -25
View File
@@ -117,24 +117,6 @@ const busModulePort = 4222
// PortsSetting is the controller's settings key for a module's given ports.
const PortsSetting = "ports"
// ServesSetting is the settings key that carries what a provider serves, inside the one binding a
// module carries rather than resolves — the builder's package binding (novox/hq 04-ISSUES/085).
//
// **A port given at genesis has to be a setting wherever it is read, or something puts it back.**
// The builder reaches the package registry through a binding written in its own manifest, because
// at genesis no module yet provides `package-registry` and so there is nothing for the controller
// to resolve it from. That binding used to be rewritten, as text, when the installer registered the
// builder — which a later registration from the catalogue undid silently. Set as a node's setting
// instead, it is held by the controller rather than by the manifest, so re-registering the builder
// leaves it where it was.
//
// **A settings layer is the module's, not one resource's.** The controller lays it over every file
// of that module which merges as JSON, so `serves` lands in the package binding only because that
// binding is the builder's one mergeable file. A second mergeable file added to the builder would
// be given a `serves` key too, meaning nothing to whatever reads it. Worth knowing before adding
// one.
const ServesSetting = "serves"
// setFoundationSettings tells the controller the ports this node gave a foundation module — and,
// on an adopted node, that the registry is reached from anywhere, as a node pulls from it before it
// has a private-network address (novox/hq ADR 0100). Done after the module is registered and before
@@ -148,13 +130,8 @@ func setFoundationSettings(ctx context.Context, o Options, control controlPlane,
if o.Adopted && module == RegistryModule {
values["expose"] = map[string]string{strconv.Itoa(DefaultPorts().Registry): "anywhere"}
}
if packages := o.Ports.orDefaults().Packages; module == BuilderModule &&
packages != DefaultPorts().Packages {
// The one binding nothing resolves: the builder dials the forge by a number it carries.
// The rest of the binding — the provision, the forge it is from, the account it presents —
// is the manifest's and stays there.
values[ServesSetting] = map[string]int{"port": packages}
}
// No `serves` setting for the build agent: the port given for the package registry reaches it in
// the genesis document (novox/hq ADR 0336 §3), and from the binding once the registry binds.
if len(values) == 0 {
return nil
}
+7 -1
View File
@@ -68,7 +68,7 @@ func TestGivenPortsMoveOnlyTheMachinesSide(t *testing.T) {
if got.Places == 0 {
t.Fatal("nothing was rewritten")
}
storeC := containerNamed(r.Declaration, "mesh-store")
storeC := containerNamed(r.Declaration, "postgres")
if len(storeC.Ports) != 1 || storeC.Ports[0] != "5433:5432" {
t.Errorf("the store publishes %v", storeC.Ports)
}
@@ -239,6 +239,12 @@ func (c *controlRecorder) run(_ context.Context, name string, args ...string) (s
}
if name == "docker" && args[0] == "exec" {
c.told = append(c.told, strings.Join(args[3:], " "))
if strings.Join(args[3:], " ") == "broker accounts" {
return "accounts {}\n", nil // the composed user list, placed after every push (issue 608)
}
}
if name == "sh" {
c.told = append(c.told, "placed the bus's users")
}
return "", nil
}
+7
View File
@@ -127,6 +127,13 @@ func Preflight(ctx context.Context, o Options, d Deps, say func(string)) ([]byte
if err := RefuseAMachineInUse(ctx, o, d.Run, say); err != nil {
return nil, err
}
// The registry's trust is written at step 7, before the push (novox/hq issue 92). What can be
// known about whether it will take, without writing anything, is asked now.
if o.pivots() {
if err := RuntimeWillTakeTheTrust(ctx, o, d.Run, say); err != nil {
return nil, err
}
}
// 4. Can this machine reach what the bundle's images come from?
//
@@ -0,0 +1,134 @@
package bootstrap
import (
"context"
"strings"
"testing"
"time"
)
// Defends novox/hq issue 608 (ADR 0331 §5, ADR 0333 §4): until the bus is adopted as the nats module,
// every push genesis makes is followed by the installer placing the composed user list where the bus
// reads it, and making the bus re-read it. The build agent's account was minted at step 13 and sealed
// with that push, and the list was placed only at enrolment, so the bus refused the agent
// ("Authorization Violation") and genesis stopped at step 15.
// placing answers the controller as a mesh whose bus is or is not the nats module yet, and records
// what was run, in order.
func placing(adopted bool) (*asked, controlPlane) {
listed := "build-agent 1\n"
if adopted {
listed += "nats 1\n"
}
runtime := &asked{}
runtime.answer = func(name string, args []string) (string, error) {
joined := strings.Join(args, " ")
switch {
case name == "sh":
return "", nil
case strings.HasSuffix(joined, "module list"):
return listed, nil
case strings.HasSuffix(joined, "broker accounts"):
return "accounts {\n MESH { users = [ { user: \"anchor.build-agent\" } ] }\n}\n", nil
}
return "", nil
}
return runtime, controlPlane{container: "mesh-controller", run: runtime.run, timeout: time.Second}
}
// placedAfterThePush is whether the bus's users were placed — written at the module's bus-users path,
// the bus told to re-read them — after the push, and how often.
func placedAfterThePush(t *testing.T, commands []string) int {
t.Helper()
pushed, placed := -1, 0
for i, c := range commands {
if strings.Contains(c, "/mesh-controller push anchor") && pushed < 0 {
pushed = i
}
if strings.HasPrefix(c, "sh -c") && strings.Contains(c, busAccounts) &&
strings.Contains(c, "kill -s HUP "+busContainer) {
if pushed < 0 || i < pushed {
t.Fatalf("the bus's users were placed before the push that carries the account: %v", commands)
}
placed++
}
}
if pushed < 0 {
t.Fatalf("nothing was pushed: %v", commands)
}
return placed
}
func TestTheBusHearsTheBuildAgentsAccountAfterItsPush(t *testing.T) {
t.Setenv("TMPDIR", t.TempDir())
runtime, control := placing(false)
o := Options{Node: "anchor", Wait: time.Second}
if _, err := installModule(context.Background(), o, control, BuilderModule, []byte(`{}`), quietly); err != nil {
t.Fatal(err)
}
if n := placedAfterThePush(t, runtime.commands); n != 1 {
t.Fatalf("the bus's users were placed %d time(s) after the build agent's push; once: %v", n, runtime.commands)
}
}
func TestTheGenesisDocumentsPushIsHeardByTheBusToo(t *testing.T) {
t.Setenv("TMPDIR", t.TempDir())
runtime, control := placing(false)
o := Options{Node: "anchor", Wait: time.Second}
if err := deliverGenesisRegistry(context.Background(), o, control, genesisRegistry(DefaultPorts(), "pw"), quietly); err != nil {
t.Fatal(err)
}
if n := placedAfterThePush(t, runtime.commands); n != 1 {
t.Fatalf("the bus's users were placed %d time(s) after the genesis document's push; once: %v", n, runtime.commands)
}
}
// From the adoption on the nats module writes the list, and the installer does not (ADR 0333 §4).
func TestOnceTheBusIsTheNatsModulesThePushPlacesNothing(t *testing.T) {
t.Setenv("TMPDIR", t.TempDir())
runtime, control := placing(true)
o := Options{Node: "anchor", Wait: time.Second}
if _, err := installModule(context.Background(), o, control, BuilderModule, []byte(`{}`), quietly); err != nil {
t.Fatal(err)
}
if n := placedAfterThePush(t, runtime.commands); n != 0 {
t.Fatalf("the installer placed the bus's users over the nats module's: %v", runtime.commands)
}
}
// Defends novox/hq issue 615: the packages step's push is waited for until the node applied it, because
// the sdk step right after asks the build agent for a build that reads the genesis document from the
// file that apply writes. A push that returned on sending let the ask race the apply.
func TestThePackagesStepWaitsUntilTheNodeAppliedTheGenesisDocument(t *testing.T) {
t.Setenv("TMPDIR", t.TempDir())
runtime, control := placing(false)
o := Options{Node: "anchor", Wait: 3 * time.Minute}
if err := deliverGenesisRegistry(context.Background(), o, control, genesisRegistry(DefaultPorts(), "pw"), quietly); err != nil {
t.Fatal(err)
}
for _, c := range runtime.commands {
if strings.Contains(c, "/mesh-controller push anchor") {
if !strings.HasSuffix(c, "push anchor --wait 3m0s") {
t.Fatalf("the genesis document's push does not wait for the node to apply it: %s", c)
}
return
}
}
t.Fatalf("the genesis document was never pushed: %v", runtime.commands)
}
// Every other push of genesis still returns on sending: only a push whose content the next step reads on
// the machine waits.
func TestAnOrdinaryPushDoesNotWait(t *testing.T) {
t.Setenv("TMPDIR", t.TempDir())
runtime, control := placing(false)
o := Options{Node: "anchor", Wait: time.Second}
if _, err := installModule(context.Background(), o, control, BuilderModule, []byte(`{}`), quietly); err != nil {
t.Fatal(err)
}
for _, c := range runtime.commands {
if strings.Contains(c, "push anchor --wait") {
t.Fatalf("an ordinary push waits: %s", c)
}
}
}
+12
View File
@@ -58,8 +58,20 @@ func aMeshThatAgrees(answers map[string]string) func(string, []string) (string,
}
}
switch {
case name == "sh" && len(args) == 2 && strings.Contains(args[1], "accounts.conf"):
return "", nil // the bus's users placed after a push (issue 608)
case name == "sh" && len(args) == 2 && strings.Contains(args[1], "mesh-nats-snapshot"):
return "", nil // the streams snapshotted before the bus's planned step (issue 590)
case name == "docker" && strings.HasSuffix(joined, "/mesh-controller bus"):
return "the bus is nats, built 53e51864, on anchor\nlast step 1: not known → 53e51864 on anchor by installer (" +
adoptionWhy + "): done at 2026-10-13 11:40; snapshot " + busSnapshotFile + "\n", nil
case name == "docker" && strings.Contains(joined, "/mesh-controller builds nats -n 1"):
return "nats built 53e51864 anchor\n image artifact-store://nats/server@sha256:" +
strings.Repeat("4", 64) + "\n bundle artifact-store://nats/tools/blobs/sha256:" + strings.Repeat("5", 64) + "\n", nil
case name != "docker":
return "", fmt.Errorf("unexpected program %q", name)
case strings.HasSuffix(joined, "broker accounts"):
return "accounts {}\n", nil
case args[0] == "cp":
return "", nil
case args[0] == "container":
+13 -7
View File
@@ -47,6 +47,9 @@ const (
// StoreSuperuserFile is where the store's superuser password lives on the machine — the
// postgres module's own-secret path, so genesis and adoption write the same file.
StoreSuperuserFile = "/var/lib/postgres/superuser.secret"
// StoreDataDir is where the store keeps its data — the postgres module's data directory, so
// genesis raises the store where the module adopts it (novox/hq ADR 0338 §1).
StoreDataDir = "/var/lib/mesh-store"
// BusControllerFile is where the controller's password on the bus lives on the machine. Not a
// module's path: no module reads it — the permanent controller is handed the address the
// temporary one was given — and genesis keeps it only so a second run puts the same account
@@ -62,7 +65,7 @@ const (
// What the template says, matched exactly. A template that says something else is a template
// this installer does not know how to make safe, and it says so rather than guessing.
templateStorePassword = `"POSTGRES_PASSWORD": "bootstrap"`
templateStoreVolumes = `"volumes": ["mesh-store-data:/var/lib/postgresql/data"]`
templateStoreVolumes = `"volumes": ["` + StoreDataDir + `:/var/lib/postgresql/data"]`
templateStoreURL = "postgres:bootstrap@"
// BusURLPlaceholder is the controller's account in its bus address, as the template names it.
@@ -265,8 +268,8 @@ func Masking(say func(string), c RootCredentials) func(string) {
// RefuseExistingServers stops a run that would put a made credential in front of a store raised
// by an earlier installer with the template's.
//
// The store's password is set by initdb, once, on an empty volume. A machine that already holds
// `mesh-store-data` and has no credential file was raised with `bootstrap`, and minting a new
// The store's password is set by initdb, once, on an empty data directory. A machine that already holds
// a cluster in the store's data directory and has no credential file was raised with `bootstrap`, and minting a new
// value here would make a bundle that dials with a password the server does not have — failing
// three steps later, in the schemas, with nothing pointing back here. Refused by name instead,
// with the way forward.
@@ -278,11 +281,14 @@ func RefuseExistingServers(ctx context.Context, run Runner, c RootCredentials) e
if !c.StoreMade {
return nil
}
if _, err := run(ctx, "docker", "volume", "inspect", "mesh-store-data"); err != nil {
return nil // no such volume: a fresh machine, which is the case this installer makes
// Where the store keeps its data, as the postgres module does (novox/hq ADR 0338 §2): a cluster
// there is one initdb already set a password for. The docker volume genesis used before is not
// the store's data any more, and a machine holding only that is not refused.
if _, err := run(ctx, "test", "-e", StoreDataDir+"/pgdata/PG_VERSION"); err != nil {
return nil // no cluster there: a fresh machine, which is the case this installer makes
}
return fmt.Errorf(
"this machine already holds the store's data (volume mesh-store-data) and no credential at %s, so it was raised "+
"this machine already holds the store's data (in "+StoreDataDir+") and no credential at %s, so it was raised "+
"by an earlier installer with the template's password. A new one made here would not open it. "+
"Put the password the store has into %s (0600, the value alone) and run again; then change it "+
"on the server and accept the new value — this installer does not rotate a running store",
@@ -311,7 +317,7 @@ func RewriteRoot(r *Rewritten, c RootCredentials) (RootRewrite, error) {
return out, err
}
if bundle, err = replaceOnce(bundle, templateStoreVolumes,
`"volumes": ["mesh-store-data:/var/lib/postgresql/data", "`+StoreSuperuserFile+":"+storeSuperuserMount+`:ro"]`,
`"volumes": ["`+StoreDataDir+`:/var/lib/postgresql/data", "`+StoreSuperuserFile+":"+storeSuperuserMount+`:ro"]`,
"the store's volumes"); err != nil {
return out, err
}
+1 -1
View File
@@ -59,7 +59,7 @@ func TestTheProducedBundleCarriesNoWellKnownCredential(t *testing.T) {
switch x := res.(type) {
case *declaration.Container:
switch x.Name {
case "mesh-store":
case "postgres":
store = true
if _, has := x.Env["POSTGRES_PASSWORD"]; has {
t.Error("the store still takes its password from its environment")
+163
View File
@@ -0,0 +1,163 @@
package bootstrap
import (
"context"
"encoding/json"
"errors"
"strings"
"testing"
"github.com/novox/mesh-host/internal/declaration"
)
// Defends novox/hq ADR 0338 (issue 609): genesis raises its store as the postgres module's server, so
// the store step adopts it as it runs. Until then the bundle raised `mesh-store`, on another image, in a
// docker volume, and genesis stopped at step 17 of 24.
// theStoreGenesisRaises is the store container of the bundle as the installer applies it: rewritten for
// this machine's credentials, so its password is the file the module reads too.
func theStoreGenesisRaises(t *testing.T) *declaration.Container {
t.Helper()
r := producedBundle(t)
store, err := storeIn(r.Declaration)
if err != nil {
t.Fatal(err)
}
return store
}
// The bundle's store is exactly the catalogue's postgres module's server, read from the module's own
// manifest (ADR 0338 §1): its name, its image (so a drift of either pin fails here), where its data is,
// and the file its superuser password is read from, with the environment that says so.
func TestTheBundleRaisesTheStoreAsTheCataloguesPostgresModuleDeclaresIt(t *testing.T) {
manifest := theCataloguesModule(t, "postgres")
var m struct {
Resources []struct {
ID string `json:"id"`
Type string `json:"type"`
Name string `json:"name"`
Image string `json:"image"`
Env map[string]string `json:"env"`
Ports []string `json:"ports"`
Volumes []string `json:"volumes"`
} `json:"resources"`
}
if err := json.Unmarshal(manifest, &m); err != nil {
t.Fatal(err)
}
ours := theStoreGenesisRaises(t)
found := false
for _, r := range m.Resources {
if r.Type != "container" || r.ID != "server" {
continue
}
found = true
if ours.Name != r.Name {
t.Errorf("the bundle raises the store as %q; the module's server is %q", ours.Name, r.Name)
}
if ours.Image != r.Image {
t.Errorf("the bundle raises the store on %s; the module pins %s — the two pins move together", ours.Image, r.Image)
}
for _, key := range []string{"POSTGRES_PASSWORD_FILE", "PGDATA"} {
if ours.Env[key] != r.Env[key] {
t.Errorf("the bundle's store has %s=%q; the module's %q", key, ours.Env[key], r.Env[key])
}
}
if _, has := ours.Env["POSTGRES_PASSWORD"]; has {
t.Error("the bundle's store takes its password from its environment; the module reads a file")
}
// The module's own directory for its state is <data root>/<module>, where genesis keeps the
// superuser's file (StoreSuperuserFile).
var theirs []string
for _, v := range r.Volumes {
theirs = append(theirs, strings.ReplaceAll(v, "${dir:state}", "/var/lib/postgres"))
}
if strings.Join(ours.Volumes, " ") != strings.Join(theirs, " ") {
t.Errorf("the bundle's store mounts %v; the module's %v", ours.Volumes, theirs)
}
if strings.Join(ours.Ports, " ") != strings.Join(r.Ports, " ") {
t.Errorf("the bundle publishes the store on %v; the module on %v", ours.Ports, r.Ports)
}
// And the store step's own check agrees: by name and image, before anything is registered.
if err := serverMatchesFoundation(manifest, ours, "postgres"); err != nil {
t.Errorf("the store step would refuse the store the bundle raises: %v", err)
}
}
if !found {
t.Fatal("the postgres module declares no server container")
}
}
// The store's data directory is made by the bundle before the store starts, owned by the image's
// postgres user, as the module declares it (ADR 0338 §1, and the measurement under its §5).
func TestTheBundleMakesTheStoresDataDirectoryAsTheModuleDoes(t *testing.T) {
r := producedBundle(t)
at, store := -1, -1
for i, res := range r.Declaration.Resources {
if d, ok := res.(*declaration.Directory); ok && d.Path == StoreDataDir {
at = i
if d.Mode != "0700" || d.Owner != "999:70" {
t.Errorf("the store's data directory is made %s %s; the module makes it 0700 999:70", d.Mode, d.Owner)
}
}
if res.Identity() == StoreID {
store = i
}
}
if at < 0 || store < 0 || at > store {
t.Fatalf("the store's data directory %s is not made before the store (directory at %d, store at %d)", StoreDataDir, at, store)
}
}
// No step of the bundle and no step of the installer names `mesh-store`, which is the seat's name only
// (ADR 0338 §2): every step that reaches the store reaches the container the bundle raised.
func TestNoStepReachesTheStoreByTheSeatsName(t *testing.T) {
text := string(theRealBundle(t))
for _, by := range []string{`"in": "mesh-store"`, `container:mesh-store`, `exec mesh-store`, `"name": "mesh-store"`, `mesh-store-data`} {
if strings.Contains(text, by) {
t.Errorf("the bundle still reaches the store as %s", by)
}
}
store := theStoreGenesisRaises(t)
if foundationStore != store.Name {
t.Errorf("the packages step reaches the store as %q; the bundle raises it as %q", foundationStore, store.Name)
}
r := producedBundle(t)
for _, res := range r.Declaration.Resources {
if a, ok := res.(*declaration.Action); ok && a.In != "" && a.In != store.Name && strings.Contains(a.ID, "database") {
t.Errorf("the step %s reaches the store in %q", a.ID, a.In)
}
}
}
// The installer's refusal to raise a store over data it cannot open looks where the store keeps its data
// (ADR 0338 §2): a cluster there and no credential is refused; the old docker volume alone is not.
func TestTheRefusalOverExistingDataLooksInTheStoresDirectory(t *testing.T) {
made := RootCredentials{Store: "s", Bus: "b", StoreMade: true}
machine := func(cluster, volume bool) Runner {
return func(_ context.Context, name string, args ...string) (string, error) {
joined := name + " " + strings.Join(args, " ")
switch {
case joined == "test -e "+StoreDataDir+"/pgdata/PG_VERSION":
if cluster {
return "", nil
}
case strings.HasPrefix(joined, "docker volume inspect"):
if volume {
return "[]", nil
}
}
return "", errors.New("exit status 1")
}
}
err := RefuseExistingServers(context.Background(), machine(true, false), made)
if err == nil || !strings.Contains(err.Error(), StoreDataDir) {
t.Fatalf("a cluster in %s and no credential was not refused naming it: %v", StoreDataDir, err)
}
if err := RefuseExistingServers(context.Background(), machine(false, true), made); err != nil {
t.Fatalf("a machine holding only the old volume was refused: %v", err)
}
if err := RefuseExistingServers(context.Background(), machine(true, false), RootCredentials{Store: "s"}); err != nil {
t.Fatalf("a kept credential was refused: %v", err)
}
}
+317
View File
@@ -0,0 +1,317 @@
package bootstrap
import (
"context"
"encoding/json"
"fmt"
"net"
"os"
"strconv"
"strings"
"time"
"github.com/novox/mesh-host/internal/apply"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/store"
"github.com/novox/mesh-host/internal/system"
)
// The registry's trust, written by genesis before anything is pushed to it (novox/hq issue 92).
//
// The mesh's registry speaks plain HTTP on purpose: every path to it is inside the private
// network's encryption (novox/hq ADR 0082). The container runtime refuses plain HTTP to a registry
// it has not been told to trust, and the module that tells it — `docker`, which writes the
// artifact store's reach into the runtime's configuration — is applied long after genesis pushes
// the control plane's image. So genesis writes that trust itself, first, with the same writer the
// module uses: into the runtime's file beside what is there (ADR 0102), the runtime reloaded and
// never restarted. It then asks the runtime whether it will speak to every name, and refuses with
// the runtime's answer before anything is pushed.
//
// **The handover.** What genesis writes is kept in no record of its own: no record of the foundation
// could hold it, because the retire step re-applies the bundle and takes away a carried record the
// bundle does not name. The `docker` module, applied later, declares the artifact store's reach —
// the mesh name on the node's port — finds it already in the file it has no record of, and takes it
// as its own (ADR 0102: a declared member found in an unrecorded file is the mesh's). That member
// does not change at the handover. The module's first apply usually still changes the file and
// reloads the runtime once, for the keys it writes beside it (`live-restore`, `log-driver`,
// `log-opts`); a reload stops nothing. The address the operator gave, when it is not the mesh name,
// stays as part of the machine's own configuration, as the hand-written trust it replaces would have.
//
// **This rests on an assumption about order.** The module takes the mesh name's member over only
// because its FIRST apply on this node already declares it: genesis assigns no `docker` before the
// network step (16), and once the node is on the network the controller composes the reach as
// `<node>.<suffix>:<port>`.
// Were `docker` applied earlier — while the reach is empty or `127.0.0.1:<port>`, as it is before the
// network — its record would exist without the mesh name, and the member would stay the machine's
// for good: undeclaring the module would never take it out. Installing the node's seat holders
// earlier in genesis must revisit this.
//
// **The suffix** is read from this process's environment, and the controller reads its own: a mesh
// with a custom `MESH_INTERNAL_SUFFIX` sets it for both.
// runtimeConfig is the container runtime's configuration, which the trust is written into. A
// variable so a test writes into a file of its own.
var runtimeConfig = "/etc/docker/daemon.json"
// runtimeUnit is the runtime's service, reloaded when the file changed.
const runtimeUnit = "docker.service"
// internalSuffixVar and defaultInternalSuffix are what a node's mesh name ends in: the same variable
// and default the controller composes `<node>.<suffix>` from (mesh-controller internal/overlay),
// which is the name the artifact store's reach — and so every push by the mesh's builder — uses.
const (
internalSuffixVar = "MESH_INTERNAL_SUFFIX"
defaultInternalSuffix = "internal"
)
// RegistryNames are the names the mesh will use for its registry on this node, each as host:port.
//
// The address the operator gave, and the node's mesh name: the controller's reach for the artifact
// store, which the builder pushes to without being told. Both on the port the registry module
// serves on this node — the node's setting, never the module's default (novox/hq issue 088). A
// loopback name is left out: the runtime trusts loopback without being told, and the mesh never
// names it to another node.
func RegistryNames(o Options) []string {
port := strconv.Itoa(o.Ports.orDefaults().Registry)
var names []string
add := func(hostport string) {
host, _, err := net.SplitHostPort(hostport)
if err != nil || host == "" || isLoopback(host) {
return
}
for _, n := range names {
if n == hostport {
return
}
}
names = append(names, hostport)
}
if host, _, err := net.SplitHostPort(o.Registry); err == nil {
add(net.JoinHostPort(strings.Trim(host, "[]"), port))
}
if node := strings.TrimSpace(o.Node); node != "" {
add(net.JoinHostPort(node+"."+internalSuffix(), port))
}
return names
}
func internalSuffix() string {
if v := strings.TrimSpace(os.Getenv(internalSuffixVar)); v != "" {
return strings.TrimPrefix(v, ".")
}
return defaultInternalSuffix
}
func isLoopback(host string) bool {
if strings.EqualFold(host, "localhost") {
return true
}
ip := net.ParseIP(strings.Trim(host, "[]"))
return ip != nil && ip.IsLoopback()
}
// trustDeclaration is the declaration genesis applies to write the trust: the same two resources
// the `docker` module declares for it — the runtime's file written into, the runtime reloaded on it.
func trustDeclaration(names []string) (*declaration.Declaration, error) {
keys, err := json.Marshal(map[string][]string{"insecure-registries": names})
if err != nil {
return nil, err
}
raw, err := json.Marshal(map[string]any{
"declaration": 1,
"resources": []map[string]any{
{"id": "genesis.registry-trust", "type": "file", "path": runtimeConfig, "mode": "0644",
"into": "json", "content": string(keys) + "\n"},
{"id": "genesis.registry-trust-reload", "type": "service", "unit": runtimeUnit,
"state": "running", "reload-on": []string{"genesis.registry-trust"}},
},
})
if err != nil {
return nil, err
}
return declaration.ParseTrusted(raw)
}
// TrustTheRegistry writes the runtime's trust of every name the mesh serves its registry on here,
// then refuses unless the runtime says it will speak plain HTTP to each.
func TrustTheRegistry(ctx context.Context, o Options, sys system.System, run Runner,
say func(string)) ([]string, error) {
names := RegistryNames(o)
if len(names) == 0 {
say(" trusted only loopback names this registry, which the runtime trusts already")
return nil, nil
}
d, err := trustDeclaration(names)
if err != nil {
return names, err
}
// One apply at a time on this machine, as ApplyBundle does: a host already running here may be
// writing the same file.
unlock, err := store.Lock(o.State, func() { say(" waiting another apply holds this node's state") })
if err != nil {
return names, err
}
report, _, err := apply.Apply(ctx, sys, d, store.State{}, store.OriginCarried, run,
func(line string) { say(" " + strings.TrimPrefix(line, " ")) }, refuseSealed)
unlock()
if err != nil {
return names, fmt.Errorf("the registry's trust could not be written into %s: %w\n"+
"The registry is plain HTTP over the private network (novox/hq ADR 0082), and the container "+
"runtime refuses to push to it until it trusts it. Nothing has been pushed", runtimeConfig, err)
}
if report.Changed() {
say(" trusted " + strings.Join(names, ", ") + " — written into " + runtimeConfig +
", the runtime reloaded")
} else {
say(" trusted " + strings.Join(names, ", ") + " — " + runtimeConfig + " already says so")
}
return names, runtimeWillSpeakTo(ctx, o, run, names)
}
// runtimeWillSpeakTo asks the runtime, until it says it trusts every name or the wait runs out.
//
// The file being right is not the runtime trusting it: a reload the runtime refused, or a flag on
// its command line that contradicts the file, leaves the push to fail at step 8 with an HTTPS error.
func runtimeWillSpeakTo(ctx context.Context, o Options, run Runner, names []string) error {
deadline := time.Now().Add(o.Wait)
var missing []string
var last error
for {
asking, cancel := context.WithTimeout(ctx, o.Timeout)
out, err := run(asking, "docker", "info", "--format", "{{json .RegistryConfig}}")
cancel()
last = err
if err == nil {
missing, last = untrusted(out, names)
if last == nil && len(missing) == 0 {
return nil
}
}
if time.Now().After(deadline) {
break
}
select {
case <-ctx.Done():
return ctx.Err()
case <-time.After(answerEvery):
}
}
if last != nil {
return fmt.Errorf("the container runtime could not be asked whether it trusts the registry: %w", last)
}
return fmt.Errorf(
"the container runtime does not trust %s after %s says it should, so it would refuse to push to "+
"it.\nThe file was written and the runtime reloaded; a runtime that still refuses was started "+
"with its own registry options, or did not take the reload — `journalctl -u %s` says which. "+
"Nothing has been pushed", strings.Join(missing, ", "), runtimeConfig, runtimeUnit)
}
// untrusted reads the runtime's registry configuration and gives back the names it would speak
// HTTPS to: a name is trusted when the runtime lists it as not secure, or when its address falls in
// a range the runtime trusts.
func untrusted(registryConfig string, names []string) ([]string, error) {
var c struct {
InsecureRegistryCIDRs []string `json:"InsecureRegistryCIDRs"`
IndexConfigs map[string]struct {
Secure bool `json:"Secure"`
} `json:"IndexConfigs"`
}
if err := json.Unmarshal([]byte(strings.TrimSpace(registryConfig)), &c); err != nil {
return nil, fmt.Errorf("the runtime's registry configuration is not readable: %w", err)
}
var missing []string
for _, name := range names {
if index, ok := c.IndexConfigs[name]; ok && !index.Secure {
continue
}
host, _, _ := net.SplitHostPort(name)
if ip := net.ParseIP(strings.Trim(host, "[]")); ip != nil && inAny(ip, c.InsecureRegistryCIDRs) {
continue
}
missing = append(missing, name)
}
return missing, nil
}
func inAny(ip net.IP, cidrs []string) bool {
for _, cidr := range cidrs {
if _, n, err := net.ParseCIDR(cidr); err == nil && n.Contains(ip) {
return true
}
}
return false
}
// procRoot is where a process's command line is read from. A variable so a test gives its own.
var procRoot = "/proc"
// RuntimeWillTakeTheTrust is preflight's half of the registry's trust: everything that can be known
// about it without writing anything, asked at step 1 so a machine that cannot take the trust is
// refused before the foundation is raised, rather than at step 7 after it (novox/hq issue 92).
//
// - The runtime's file is absent or a JSON object: the mesh writes into an object, and refuses to
// write over anything else (ADR 0102).
// - The runtime's service is known to systemd and active: the trust is taken by reloading it.
// - The runtime was not started with `--insecure-registry`: dockerd refuses a directive given both
// as a flag and in its file, so it would refuse the reload and keep distrusting the registry.
func RuntimeWillTakeTheTrust(ctx context.Context, o Options, run Runner, say func(string)) error {
names := RegistryNames(o)
if len(names) == 0 {
return nil
}
if raw, err := os.ReadFile(runtimeConfig); err == nil {
var object map[string]json.RawMessage
if len(strings.TrimSpace(string(raw))) > 0 {
if err := json.Unmarshal(raw, &object); err != nil || object == nil {
return fmt.Errorf("%s is not a JSON object, and the mesh writes the trust of its registry "+
"(%s) into it beside what is there, never over it (novox/hq ADR 0102). Make it an object "+
"or remove it; nothing has been changed", runtimeConfig, strings.Join(names, ", "))
}
}
} else if !os.IsNotExist(err) {
return fmt.Errorf("the runtime's configuration %s cannot be read: %w", runtimeConfig, err)
}
asking, cancel := context.WithTimeout(ctx, o.Timeout)
out, err := run(asking, "systemctl", "show", runtimeUnit, "--property=LoadState,ActiveState,MainPID")
cancel()
if err != nil {
return fmt.Errorf("systemd could not be asked about %s: %w\nThe registry's trust is taken by "+
"reloading it; nothing has been changed", runtimeUnit, err)
}
props := map[string]string{}
for _, line := range strings.Split(out, "\n") {
if k, v, ok := strings.Cut(strings.TrimSpace(line), "="); ok {
props[k] = v
}
}
if props["LoadState"] != "loaded" || props["ActiveState"] != "active" {
return fmt.Errorf("%s is %s and %s, and the registry's trust (%s) is taken by reloading it. "+
"Genesis needs the runtime run by systemd as %s; nothing has been changed",
runtimeUnit, orUnknown(props["LoadState"]), orUnknown(props["ActiveState"]),
strings.Join(names, ", "), runtimeUnit)
}
if pid := props["MainPID"]; pid != "" && pid != "0" {
if cmdline, err := os.ReadFile(procRoot + "/" + pid + "/cmdline"); err == nil {
for _, arg := range strings.Split(string(cmdline), "\x00") {
if strings.HasPrefix(arg, "--insecure-registry") {
return fmt.Errorf("the runtime was started with %s, and dockerd refuses insecure registries "+
"given both as a flag and in %s — so it would refuse the reload that makes it trust "+
"%s. Move the flag's registries into the file and restart the runtime once; nothing "+
"has been changed", arg, runtimeConfig, strings.Join(names, ", "))
}
}
}
}
say(" registry trust " + runtimeConfig + " can be written into and " + runtimeUnit + " reloaded")
return nil
}
func orUnknown(s string) string {
if s == "" {
return "unknown"
}
return s
}
+347
View File
@@ -0,0 +1,347 @@
package bootstrap
import (
"context"
"encoding/json"
"fmt"
"net/http"
"os"
"path/filepath"
"strings"
"testing"
"time"
"github.com/novox/mesh-host/internal/apply"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/store"
"github.com/novox/mesh-host/internal/system"
)
// novox/hq issue 92: genesis pushed the control plane's image to its registry before anything had
// told the container runtime to trust that registry, so the push failed with "server gave HTTP
// response to HTTPS client" — first by the address given, then by the mesh name the builder used,
// and the trust the mesh wrote later named the registry's default port rather than the node's.
// aRuntime is a container runtime on systemd whose registry configuration is what its file said at
// its last reload: what a real dockerd does, and why a written file is not yet a trusted registry.
type aRuntime struct {
path string
trusted []string
commands []string
// ignoresReload is a runtime started with options of its own, which a reload does not change.
ignoresReload bool
}
func (r *aRuntime) run(_ context.Context, name string, args ...string) (string, error) {
line := name + " " + strings.Join(args, " ")
r.commands = append(r.commands, line)
switch {
case name == "docker" && len(args) > 0 && args[0] == "info":
index := map[string]map[string]any{"docker.io": {"Secure": true}}
for _, n := range r.trusted {
index[n] = map[string]any{"Name": n, "Secure": false}
}
out, _ := json.Marshal(map[string]any{"InsecureRegistryCIDRs": []string{"127.0.0.0/8"}, "IndexConfigs": index})
return string(out), nil
case name == "systemctl" && strings.Contains(line, "reload"):
if !r.ignoresReload {
r.trusted = r.trusted[:0]
raw, _ := os.ReadFile(r.path)
var c struct {
Insecure []string `json:"insecure-registries"`
}
_ = json.Unmarshal(raw, &c)
r.trusted = append(r.trusted, c.Insecure...)
}
case strings.Contains(line, "is-enabled"):
return "enabled", nil
case strings.Contains(line, "show") && strings.Contains(line, "ActiveState"):
return "LoadState=loaded\nActiveState=active\nSubState=running", nil
}
return "", nil
}
func (r *aRuntime) did(word string) int {
n := 0
for _, c := range r.commands {
if strings.Contains(c, "systemctl") && strings.Contains(c, " "+word+" ") || strings.HasSuffix(c, " "+word) {
n++
}
}
return n
}
func trustSetup(t *testing.T, existing string) (Options, *aRuntime, system.System) {
t.Helper()
restore := apply.ForTests(t.TempDir(), t.TempDir())
t.Cleanup(restore)
dir := t.TempDir()
was := runtimeConfig
runtimeConfig = filepath.Join(dir, "daemon.json")
t.Cleanup(func() { runtimeConfig = was })
if existing != "" {
if err := os.WriteFile(runtimeConfig, []byte(existing), 0o644); err != nil {
t.Fatal(err)
}
}
t.Setenv(internalSuffixVar, "")
sys, err := system.For("arch")
if err != nil {
t.Fatal(err)
}
o := Options{Node: "anchor", Registry: "192.0.2.10:5100", Ports: FoundationPorts{Registry: 5100},
State: filepath.Join(dir, "state", "state.json"), Timeout: time.Second, Wait: 0}
return o, &aRuntime{path: runtimeConfig}, sys
}
func insecureIn(t *testing.T) (string, map[string]json.RawMessage) {
t.Helper()
raw, err := os.ReadFile(runtimeConfig)
if err != nil {
t.Fatal(err)
}
var o map[string]json.RawMessage
if err := json.Unmarshal(raw, &o); err != nil {
t.Fatal(err)
}
var list []string
_ = json.Unmarshal(o["insecure-registries"], &list)
return fmt.Sprint(list), o
}
func TestGenesisTrustsTheRegistryByEveryNameBeforeItPushes(t *testing.T) {
o, runtime, sys := trustSetup(t, `{"data-root":"/srv/docker","insecure-registries":["198.51.100.7:5000"]}`)
names, err := TrustTheRegistry(context.Background(), o, sys, runtime.run, func(string) {})
if err != nil {
t.Fatal(err)
}
// The address given and the mesh name the builder pushes by, both on the node's port.
if fmt.Sprint(names) != "[192.0.2.10:5100 anchor.internal:5100]" {
t.Errorf("trusted by %v", names)
}
list, object := insecureIn(t)
if list != "[198.51.100.7:5000 192.0.2.10:5100 anchor.internal:5100]" {
t.Errorf("the runtime's list is %s", list)
}
if string(object["data-root"]) != `"/srv/docker"` {
t.Errorf("the machine's own key was not kept: %s", object["data-root"])
}
if runtime.did("reload") != 1 || runtime.did("restart") != 0 || runtime.did("stop") != 0 {
t.Errorf("the runtime was not reloaded exactly once and never restarted: %v", runtime.commands)
}
// Re-run: nothing changes, nothing is reloaded.
runtime.commands = nil
if _, err := TrustTheRegistry(context.Background(), o, sys, runtime.run, func(string) {}); err != nil {
t.Fatal(err)
}
if runtime.did("reload") != 0 {
t.Errorf("a re-run reloaded the runtime: %v", runtime.commands)
}
}
// The port is the node's setting for the registry, not the module's default, whatever the operator
// typed the address with — and loopback, which the runtime trusts already, is not written.
func TestTheRegistrysTrustNamesThePortTheNodeServesItOn(t *testing.T) {
o := Options{Node: "anchor", Registry: "127.0.0.1:5100", Ports: FoundationPorts{Registry: 5100}}
t.Setenv(internalSuffixVar, "")
if got := fmt.Sprint(RegistryNames(o)); got != "[anchor.internal:5100]" {
t.Errorf("names %s", got)
}
o.Ports = FoundationPorts{}
o.Registry = "192.0.2.10:5000"
t.Setenv(internalSuffixVar, ".mesh.example")
if got := fmt.Sprint(RegistryNames(o)); got != "[192.0.2.10:5000 anchor.mesh.example:5000]" {
t.Errorf("names %s", got)
}
}
// A runtime that still refuses after the write is refused here, with the names, before any push.
func TestGenesisRefusesARuntimeThatStillDoesNotTrustTheRegistry(t *testing.T) {
o, runtime, sys := trustSetup(t, "")
runtime.ignoresReload = true
_, err := TrustTheRegistry(context.Background(), o, sys, runtime.run, func(string) {})
if err == nil {
t.Fatal("a runtime that does not trust the registry was let through to the push")
}
if !strings.Contains(err.Error(), "192.0.2.10:5100, anchor.internal:5100") ||
!strings.Contains(err.Error(), "Nothing has been pushed") {
t.Errorf("the refusal does not say what: %v", err)
}
}
// The handover: the `docker` module, applied later, writes the reach the controller composes for
// the artifact store — the mesh name and the node's port. It finds it there: the file does not
// change, the runtime is not reloaded, and the member it declares is recorded as the module's from
// then on (a member found in a file with no record of it is the mesh's, ADR 0102), so the module
// owns the trust it writes and taking the module away takes that member out.
func TestTheDockerModuleTakesTheTrustOverWithoutAChange(t *testing.T) {
o, runtime, sys := trustSetup(t, `{"live-restore":true}`)
if _, err := TrustTheRegistry(context.Background(), o, sys, runtime.run, func(string) {}); err != nil {
t.Fatal(err)
}
before, _ := os.ReadFile(runtimeConfig)
module, err := declaration.ParseTrusted([]byte(fmt.Sprintf(`{"declaration":1,"resources":[
{"id":"docker.daemon","type":"file","path":%q,"mode":"0644","into":"json",
"content":"{\"live-restore\": true, \"insecure-registries\": [\"anchor.internal:5100\"]}\n"},
{"id":"docker.runtime","type":"service","unit":"docker.service","state":"running","boot":"enabled",
"reload-on":["docker.daemon"]}]}`, runtimeConfig)))
if err != nil {
t.Fatal(err)
}
runtime.commands = nil
_, state, err := apply.Apply(context.Background(), sys, module, store.State{}, store.OriginDeclared,
runtime.run, nil, nil)
if err != nil {
t.Fatal(err)
}
after, _ := os.ReadFile(runtimeConfig)
if string(before) != string(after) {
t.Errorf("the handover changed the file:\n%s\n%s", before, after)
}
if runtime.did("reload") != 0 || runtime.did("restart") != 0 {
t.Errorf("the handover touched the runtime: %v", runtime.commands)
}
if rec, _ := state.Find("docker.daemon"); rec.Into == nil || len(rec.Into.Added["insecure-registries"]) != 1 {
t.Errorf("the module did not take the mesh name's member over: %+v", rec.Into)
}
}
func TestARangeTheRuntimeTrustsCoversAnAddressInIt(t *testing.T) {
missing, err := untrusted(`{"InsecureRegistryCIDRs":["10.42.0.0/16"],"IndexConfigs":{}}`,
[]string{"10.42.0.1:5000", "anchor.internal:5000"})
if err != nil {
t.Fatal(err)
}
if fmt.Sprint(missing) != "[anchor.internal:5000]" {
t.Errorf("missing %v", missing)
}
}
// **The order, which is the fault issue 92 is about.** Steps 7 and 8 against a runtime that behaves
// as dockerd does: a push to a registry it has not been told to trust is refused with the error the
// first genesis met. The trust must be written and confirmed before anything is tagged or pushed;
// moving the trust after the push, or dropping it, fails this test.
func TestNothingIsPushedToTheRegistryBeforeTheRuntimeTrustsIt(t *testing.T) {
o, runtime, sys := trustSetup(t, `{"data-root":"/srv/docker"}`)
o.Catalogue = catalogueWith(t, RegistryModule, upstreamRegistryManifest)
o.Registry = "192.0.2.10:5100"
pushed := false
mesh := aMeshThatAgrees(nil)
run := func(ctx context.Context, name string, args ...string) (string, error) {
if name == "systemctl" || (name == "docker" && args[0] == "info") {
return runtime.run(ctx, name, args...)
}
runtime.commands = append(runtime.commands, name+" "+strings.Join(args, " "))
if name == "docker" {
switch args[0] {
case "tag":
return "", nil
case "push":
trusted := false
for _, n := range runtime.trusted {
trusted = trusted || strings.HasPrefix(args[1], n+"/")
}
if !trusted {
return "", fmt.Errorf(`Get "https://%s/v2/": http: server gave HTTP response to HTTPS client`, o.Registry)
}
pushed = true
return "", nil
case "image":
return `["` + o.Registry + `/mesh-controller@sha256:` + strings.Repeat("a", 64) + `"]`, nil
}
}
return mesh(name, args)
}
d := Deps{Run: run, Fetch: func(_ context.Context, url string) (int, string, error) {
if strings.HasSuffix(url, "/v2/") || pushed {
return http.StatusOK, `{"name":"mesh-controller","tags":["genesis"]}`, nil
}
return http.StatusNotFound, "", nil
}}
var result Result
published, err := RegistryAndPublish(context.Background(), o, d, sys,
controlPlane{container: "temp-mesh-controller", run: run, timeout: time.Second}, held, &result, func(string) {})
if err != nil {
t.Fatalf("genesis did not get past the push: %v\ncommands: %v", err, runtime.commands)
}
if !strings.HasPrefix(published.Reference, o.Registry+"/mesh-controller@sha256:") {
t.Errorf("published as %q", published.Reference)
}
firstPush, reloaded, confirmed := -1, -1, -1
for i, c := range runtime.commands {
switch {
case firstPush < 0 && (strings.HasPrefix(c, "docker tag") || strings.HasPrefix(c, "docker push")):
firstPush = i
case strings.HasPrefix(c, "systemctl") && strings.Contains(c, "reload"):
reloaded = i
case strings.HasPrefix(c, "docker info"):
confirmed = i
}
}
if firstPush < 0 || reloaded < 0 || confirmed < 0 || !(reloaded < confirmed && confirmed < firstPush) {
t.Errorf("the trust was not written (%d) and confirmed (%d) before the first tag or push (%d): %v",
reloaded, confirmed, firstPush, runtime.commands)
}
if fmt.Sprint(result.RegistryTrusted) != "[192.0.2.10:5100 anchor.internal:5100]" {
t.Errorf("the result says it trusted %v", result.RegistryTrusted)
}
}
// Preflight asks, without writing, what would stop the trust from taking at step 7.
func TestPreflightRefusesARuntimeThatCouldNotTakeTheTrust(t *testing.T) {
proc := t.TempDir()
was := procRoot
procRoot = proc
t.Cleanup(func() { procRoot = was })
if err := os.MkdirAll(filepath.Join(proc, "42"), 0o755); err != nil {
t.Fatal(err)
}
unit := func(load, active string) Runner {
return func(_ context.Context, name string, args ...string) (string, error) {
return "LoadState=" + load + "\nActiveState=" + active + "\nMainPID=42\n", nil
}
}
cmdline := func(args ...string) {
if err := os.WriteFile(filepath.Join(proc, "42", "cmdline"), []byte(strings.Join(args, "\x00")+"\x00"), 0o644); err != nil {
t.Fatal(err)
}
}
cmdline("/usr/bin/dockerd", "-H", "fd://")
for _, c := range []struct {
name, file string
run Runner
flag bool
refused string
}{
{"a runtime that can take it", `{"data-root":"/srv/docker"}`, unit("loaded", "active"), false, ""},
{"no file yet", "", unit("loaded", "active"), false, ""},
{"a file that is not an object", `["not", "an", "object"]`, unit("loaded", "active"), false, "not a JSON object"},
{"a runtime systemd does not run", `{}`, unit("not-found", "inactive"), false, "not-found and inactive"},
{"a runtime with the flag", `{}`, unit("loaded", "active"), true, "--insecure-registry=10.0.0.1:5000"},
} {
t.Run(c.name, func(t *testing.T) {
o, _, _ := trustSetup(t, c.file)
if c.flag {
cmdline("/usr/bin/dockerd", "--insecure-registry=10.0.0.1:5000")
defer cmdline("/usr/bin/dockerd", "-H", "fd://")
}
before, _ := os.ReadFile(runtimeConfig)
err := RuntimeWillTakeTheTrust(context.Background(), o, c.run, func(string) {})
switch {
case c.refused == "" && err != nil:
t.Errorf("refused: %v", err)
case c.refused != "" && (err == nil || !strings.Contains(err.Error(), c.refused)):
t.Errorf("wanted a refusal naming %q, got %v", c.refused, err)
}
if after, _ := os.ReadFile(runtimeConfig); string(after) != string(before) {
t.Error("preflight wrote the runtime's file")
}
})
}
}
+1 -1
View File
@@ -89,7 +89,7 @@ func TestAFoundationThatIsUpAndAnsweringIsAccepted(t *testing.T) {
// run-once and scheduled shapes are excluded on purpose — a step that has exited is not a
// fault. The name is `temp-mesh-controller` because the permanent one is a module and takes the
// plain name (novox/hq ADR 0067), which is what makes the two of them coexist at all.
want := []string{"mesh-store", "mesh-broker-nats", "temp-mesh-controller"}
want := []string{"postgres", "mesh-broker-nats", "temp-mesh-controller"}
if len(verified.Running) != len(want) {
t.Fatalf("confirmed %v running, want %v", verified.Running, want)
}
+87
View File
@@ -0,0 +1,87 @@
package declaration
import (
"bytes"
"encoding/json"
"fmt"
"strings"
"time"
)
// Busy is how a long-running container says it holds work that would be lost if it were stopped now
// (novox/hq ADR 0335 §1): a command run inside it, by its shell, as an exec health check is, that exits 0
// while it is busy and non-zero when it is idle. Being asked is also the container's notice that it is
// about to be replaced: from the first ask it takes no new work, so its answer ends once the work in hand
// ends.
//
// **Not part of what the container is.** Busy is not in the container's spec: a changed command changes
// when the container is replaced, never whether — a container is not recreated for its busy command alone.
type Busy struct {
Command string `json:"command"`
// Timeout is how long one ask may take; empty is BusyTimeoutDefault.
Timeout string `json:"timeout,omitempty"`
}
// The bounds of an ask (novox/hq ADR 0335).
const (
BusyTimeoutDefault = 10 * time.Second
// BusyTimeoutCeiling is the longest one ask may take: an ask runs inside an apply, and an apply waits
// for nothing longer than a look.
BusyTimeoutCeiling = time.Minute
// BusyBound is how long a container that says busy is left running after the first apply that would
// have replaced it (ADR 0335 §3). The node-engine's, not the module's: no module holds its own
// replacement for ever.
BusyBound = 15 * time.Minute
)
// UnmarshalJSON reads a busy strictly, as everything a declaration carries is read.
func (b *Busy) UnmarshalJSON(raw []byte) error {
type plain Busy
var p plain
dec := json.NewDecoder(bytes.NewReader(raw))
dec.DisallowUnknownFields()
if err := dec.Decode(&p); err != nil {
return fmt.Errorf("busy: %w", err)
}
*b = Busy(p)
return nil
}
// Within is how long one ask may take. Validated on arrival, so an unreadable timeout is the default.
func (b *Busy) Within() time.Duration {
if d, err := time.ParseDuration(b.Timeout); err == nil && d > 0 {
return d
}
return BusyTimeoutDefault
}
// problems holds a container's busy to its shape: a one-line command, a timeout within its bounds, and only
// on a container that stays up.
func (b *Busy) problems(where string, longRunning bool) []string {
if b == nil {
return nil
}
var problems []string
say := func(format string, args ...any) {
problems = append(problems, where+": "+fmt.Sprintf(format, args...))
}
if !longRunning {
say("busy is asked of a container that stays up; a step or a scheduled run is never replaced mid-run")
}
if strings.TrimSpace(b.Command) == "" {
say("busy needs a command")
}
if strings.ContainsAny(b.Command, "\n\r") {
say("busy's command is one line")
}
if b.Timeout != "" {
d, err := time.ParseDuration(b.Timeout)
switch {
case err != nil:
say("busy's timeout is a duration, not %q", b.Timeout)
case d <= 0 || d > BusyTimeoutCeiling:
say("an ask of busy takes more than nothing and at most %s, not %s", BusyTimeoutCeiling, d)
}
}
return problems
}
+359 -18
View File
@@ -19,6 +19,8 @@ import (
"sort"
"strconv"
"strings"
"git.novox.be/novox/mesh-sdk/go/seal"
)
// Version is the vocabulary this host speaks. A declaration naming any other version is
@@ -101,6 +103,8 @@ type Resource interface {
// and not a hash of the content: it is what lets the store say *this is the same resource
// I applied last time*, which is what makes removal possible at all.
Identity() string
// DeclaredModule is the module the controller says the resource belongs to, or "" (issue 1132).
DeclaredModule() string
// Kind is the resource's type, for the store and for reporting.
Kind() Type
// Target is what the resource acts on, for a person reading a report.
@@ -113,15 +117,87 @@ type Resource interface {
// strict decoder would otherwise refuse. `Kind()` returns the constant and is what anything
// else should read.
// DeclaredModule is on every kind: the module the controller says the resource belongs to, or "" when it does not say
// (an older controller). Read through Declaration.ModuleOfID, which falls back to the guess.
func (d *Directory) DeclaredModule() string { return d.Module }
func (f *File) DeclaredModule() string { return f.Module }
func (u *User) DeclaredModule() string { return u.Module }
func (n *Network) DeclaredModule() string { return n.Module }
func (a *Access) DeclaredModule() string { return a.Module }
func (a *Archive) DeclaredModule() string { return a.Module }
func (d *Process) DeclaredModule() string { return d.Module }
func (s *Service) DeclaredModule() string { return s.Module }
func (o *Opening) DeclaredModule() string { return o.Module }
func (p *Package) DeclaredModule() string { return p.Module }
func (c *Container) DeclaredModule() string { return c.Module }
func (a *Action) DeclaredModule() string { return a.Module }
// Directory is a directory that should exist, with a mode.
type Directory struct {
ID string `json:"id"`
Type Type `json:"type"`
Path string `json:"path"`
Mode string `json:"mode,omitempty"`
// Module is the module this resource belongs to, as the controller says it (novox/hq ADR 0429 §4, issue 1132). Absent
// is an older controller, and the module is then guessed from the id (ModuleGuess).
Module string `json:"module,omitempty"`
Path string `json:"path"`
Mode string `json:"mode,omitempty"`
// Owner is the user this belongs to, by name. Absent means root, which is what everything
// managed was until users existed.
Owner string `json:"owner,omitempty"`
// Unknown says how the directory treats a path inside it that nothing declared (issue 1024, novox/hq ADR 0422):
// move-out, module or program. Empty means not said, and nothing moves.
Unknown string `json:"unknown,omitempty"`
// Knows names paths relative to Path that are not declared but belong here; a trailing "/" names a folder and
// everything in it.
Knows []string `json:"knows,omitempty"`
}
// How a directory or archive treats what it did not write (issue 1024, ADR 0422).
const (
UnknownMoveOut = "move-out"
UnknownModule = "module"
UnknownProgram = "program"
)
// checkUnknown judges the unknown and knows fields of a directory or an archive.
func checkUnknown(where, kind, unknown string, knows []string) []string {
var problems []string
switch unknown {
case "", UnknownMoveOut, UnknownModule, UnknownProgram:
default:
problems = append(problems, fmt.Sprintf(
"%s: a %s's unknown is %q; it is %q, %q or %q (issue 1024)", where, kind, unknown,
UnknownMoveOut, UnknownModule, UnknownProgram))
}
if len(knows) > 0 && unknown == "" {
problems = append(problems, where+": knows is given without unknown, and says nothing without it")
}
seen := map[string]bool{}
for _, k := range knows {
switch {
case k == "":
problems = append(problems, where+": knows holds an empty entry")
continue
case strings.HasPrefix(k, "/"):
problems = append(problems, fmt.Sprintf("%s: knows entry %q is absolute; it is relative to the path", where, k))
continue
}
bad := false
for _, c := range strings.Split(strings.TrimSuffix(k, "/"), "/") {
if c == "" || c == "." || c == ".." {
bad = true
}
}
if bad {
problems = append(problems, fmt.Sprintf("%s: knows entry %q holds an empty, . or .. component", where, k))
continue
}
if seen[k] {
problems = append(problems, fmt.Sprintf("%s: knows entry %q is repeated", where, k))
}
seen[k] = true
}
return problems
}
func (d *Directory) Identity() string { return d.ID }
@@ -139,13 +215,17 @@ func (d *Directory) validate(where string, _ bool) []string {
if len(mode) == 4 && mode[0] == '2' {
mode = "0" + mode[1:]
}
problems = append(problems, checkUnknown(where, "directory", d.Unknown, d.Knows)...)
return append(problems, checkMode(where, mode)...)
}
// File is a file with literal content. The host renders nothing.
type File struct {
ID string `json:"id"`
Type Type `json:"type"`
ID string `json:"id"`
Type Type `json:"type"`
// Module is the module this resource belongs to, as the controller says it (novox/hq ADR 0429 §4, issue 1132). Absent
// is an older controller, and the module is then guessed from the id (ModuleGuess).
Module string `json:"module,omitempty"`
Path string `json:"path"`
Content string `json:"content"`
Mode string `json:"mode,omitempty"`
@@ -205,6 +285,12 @@ type File struct {
// by looking rather than by knowing which field won.
Sealed string `json:"sealed,omitempty"`
// VaultSealed is content the vault sealed to this node's sealing key, in the mesh's sealed box (novox/hq ADR
// 0206): a core module's own credential, which the controller carries in the declaration and cannot open
// (ADR 0349 §16, issue 755). The same promise as Sealed, from the vault rather than the controller; sent only to
// an engine that states VaultBoxContract. Exclusive with Content, Sealed and Bytes.
VaultSealed *seal.Box `json:"vault-sealed,omitempty"`
// Trusted is the catalogue's word on whether the settings this file's content took are trusted (novox/hq
// issue 339). It is the controller's to act on, and a controller takes it out before it sends a declaration.
// Accepted here, and nothing is done with it, so that a controller that passes it on — an older one, or one
@@ -237,11 +323,48 @@ type File struct {
// Owner is the user this belongs to, by name. Absent means root.
Owner string `json:"owner,omitempty"`
// OwnerAtRuntime says who owns the file once it is placed (novox/hq ADR 0349 §5): "owned", the mesh's
// whole, written back whenever it differs; "seed", written once and then the program's; or "managed",
// only its named keys or its marked block the mesh's, the program keeping the rest.
//
// **Three words for what the forms above already do.** Owned is an ordinary file, seed is create-once
// and managed is a file written into, so the word changes no write: a file that says it is applied
// exactly as its old form is, and one that does not say is owned by its old form (Ownership). The word
// is what the provisioner's files say of themselves, which the old forms only implied. Seed is enough
// alone; managed still needs `into`, which says the format. A word that contradicts the form is refused.
OwnerAtRuntime string `json:"owner-at-runtime,omitempty"`
}
// The words a file says who owns it at runtime with (novox/hq ADR 0349 §5).
const (
RuntimeOwned = "owned"
RuntimeSeed = "seed"
RuntimeManaged = "managed"
)
// Ownership is who owns the file at runtime: the word it says, or the one its old form implies — a file
// written into is managed, one created once is a seed, and any other is owned.
func (f *File) Ownership() string {
switch {
case f.OwnerAtRuntime != "":
return f.OwnerAtRuntime
case f.Into != "":
return RuntimeManaged
case f.CreateOnce:
return RuntimeSeed
default:
return RuntimeOwned
}
}
// Seeded reports whether the file is written only when absent: created once, by create-once or by the
// word seed.
func (f *File) Seeded() bool { return f.CreateOnce || f.OwnerAtRuntime == RuntimeSeed }
// Secret reports whether this file arrived sealed, which is what decides both that it must be
// opened before writing and that its contents must never appear in a report.
func (f *File) Secret() bool { return f.Sealed != "" }
func (f *File) Secret() bool { return f.Sealed != "" || f.VaultSealed != nil }
func (f *File) Identity() string { return f.ID }
func (f *File) Kind() Type { return TypeFile }
@@ -314,12 +437,13 @@ func (f *File) validate(where string, _ bool) []string {
where, f.At))
}
if f.Into == IntoJSON || f.Into == IntoBlock {
if f.Sealed != "" || f.Bytes != "" || len(f.Secrets) > 0 || f.CreateOnce {
if f.Secret() || f.Bytes != "" || len(f.Secrets) > 0 || f.CreateOnce {
problems = append(problems, where+
": a file written into says only its part, in content — not sealed, bytes, "+
"secrets or create-once")
}
}
problems = append(problems, f.checkOwnership(where)...)
var said []string
for name, value := range map[string]string{
"content": f.Content, "sealed": f.Sealed, "bytes": f.Bytes,
@@ -328,6 +452,16 @@ func (f *File) validate(where string, _ bool) []string {
said = append(said, name)
}
}
if f.VaultSealed != nil {
said = append(said, "vault-sealed")
if f.VaultSealed.V != seal.Version {
problems = append(problems, fmt.Sprintf("%s: vault-sealed is a sealed box of version %d, and this "+
"node-engine opens version %d", where, f.VaultSealed.V, seal.Version))
}
if len(f.Secrets) > 0 {
problems = append(problems, where+": a file the vault sealed whole has no holes for secrets")
}
}
if len(said) > 1 {
sort.Strings(said)
problems = append(problems, where+
@@ -361,6 +495,42 @@ func (f *File) validate(where string, _ bool) []string {
return append(problems, checkMode(where, f.Mode)...)
}
// checkOwnership refuses a word that contradicts the file's form (novox/hq ADR 0349 §5): owned is a whole
// file written back, so not created once nor written into; a seed is a whole file created once, so not
// written into; managed is a file written into, which says its format. Managed and created once together
// is already refused by the rule that a file written into is not created once.
func (f *File) checkOwnership(where string) []string {
switch f.OwnerAtRuntime {
case "":
case RuntimeOwned:
var also []string
if f.CreateOnce {
also = append(also, "create-once")
}
if f.Into != "" {
also = append(also, "into")
}
if len(also) > 0 {
return []string{fmt.Sprintf("%s: owned at runtime is the mesh's whole file, written back whenever it "+
"differs, and this also says %s", where, strings.Join(also, " and "))}
}
case RuntimeSeed:
if f.Into != "" {
return []string{where + ": a seed is a whole file written once, and this says into; a file the " +
"program shares with the mesh is managed"}
}
case RuntimeManaged:
if f.Into == "" {
return []string{where + ": a managed file names the format its keys or block are written into, " +
"with into \"json\" or \"block\""}
}
default:
return []string{fmt.Sprintf("%s: owner-at-runtime %q; a placed file is %q, %q or %q, or omits it to be "+
"owned by its form", where, f.OwnerAtRuntime, RuntimeOwned, RuntimeSeed, RuntimeManaged)}
}
return nil
}
// User is a login on the machine.
//
// The thing that makes a shell, a chat client or a desktop expressible at all: each is a package
@@ -373,7 +543,10 @@ func (f *File) validate(where string, _ bool) []string {
type User struct {
ID string `json:"id"`
Type Type `json:"type"`
Name string `json:"name"`
// Module is the module this resource belongs to, as the controller says it (novox/hq ADR 0429 §4, issue 1132). Absent
// is an older controller, and the module is then guessed from the id (ModuleGuess).
Module string `json:"module,omitempty"`
Name string `json:"name"`
// Shell this user logs in with. Absent means the host asserts nothing and leaves whatever is
// there — the same rule Service.Boot follows, for the same reason: a field that always
@@ -424,7 +597,10 @@ const RootNever = "never"
type Network struct {
ID string `json:"id"`
Type Type `json:"type"`
Name string `json:"name"`
// Module is the module this resource belongs to, as the controller says it (novox/hq ADR 0429 §4, issue 1132). Absent
// is an older controller, and the module is then guessed from the id (ModuleGuess).
Module string `json:"module,omitempty"`
Name string `json:"name"`
}
func (n *Network) Identity() string { return n.ID }
@@ -469,7 +645,10 @@ const (
type Access struct {
ID string `json:"id"`
Type Type `json:"type"`
Path string `json:"path"`
// Module is the module this resource belongs to, as the controller says it (novox/hq ADR 0429 §4, issue 1132). Absent
// is an older controller, and the module is then guessed from the id (ModuleGuess).
Module string `json:"module,omitempty"`
Path string `json:"path"`
// Mode is how this module reaches the path: read or read-write. Absent narrows to read.
Mode string `json:"mode,omitempty"`
@@ -596,6 +775,9 @@ func (u *User) validate(where string, _ bool) []string {
type Archive struct {
ID string `json:"id"`
Type Type `json:"type"`
// Module is the module this resource belongs to, as the controller says it (novox/hq ADR 0429 §4, issue 1132). Absent
// is an older controller, and the module is then guessed from the id (ModuleGuess).
Module string `json:"module,omitempty"`
// Source is where to fetch it from.
Source string `json:"source"`
// Digest is sha256 of the archive, as "sha256:<hex>".
@@ -604,6 +786,9 @@ type Archive struct {
Path string `json:"path"`
// Owner is the user the unpacked files belong to. Absent means root.
Owner string `json:"owner,omitempty"`
// Unknown and Knows are as a directory's (issue 1024, ADR 0422).
Unknown string `json:"unknown,omitempty"`
Knows []string `json:"knows,omitempty"`
}
func (a *Archive) Identity() string { return a.ID }
@@ -624,6 +809,7 @@ func (a *Archive) validate(where string, _ bool) []string {
problems = append(problems, where+
": an archive is pinned by digest, as sha256:<64 hex characters>")
}
problems = append(problems, checkUnknown(where, "archive", a.Unknown, a.Knows)...)
return problems
}
@@ -645,6 +831,9 @@ func (a *Archive) validate(where string, _ bool) []string {
type Process struct {
ID string `json:"id"`
Type Type `json:"type"`
// Module is the module this resource belongs to, as the controller says it (novox/hq ADR 0429 §4, issue 1132). Absent
// is an older controller, and the module is then guessed from the id (ModuleGuess).
Module string `json:"module,omitempty"`
// Name is what the unit is called, and what an operator will see in the process table.
Name string `json:"name"`
// Source is where to fetch the bundle from, and Digest is what it must hash to. The same
@@ -834,7 +1023,10 @@ func (d *Process) validate(where string, _ bool) []string {
type Service struct {
ID string `json:"id"`
Type Type `json:"type"`
Unit string `json:"unit"`
// Module is the module this resource belongs to, as the controller says it (novox/hq ADR 0429 §4, issue 1132). Absent
// is an older controller, and the module is then guessed from the id (ModuleGuess).
Module string `json:"module,omitempty"`
Unit string `json:"unit"`
// State is "running" or "stopped" — or absent, and then **the unit's lifecycle is the
// machine's; the mesh only reflects its triggers** (novox/hq ADR 0117). The uplink modules
// declare the machine's own network manager this way: the mesh writes into its configuration
@@ -1006,8 +1198,11 @@ func BlockMarkers(id string) (begin, end string) {
// firewall sees a published port after the runtime has translated it, so a forwarded opening names
// the container's own port in To as well as the machine's in Port.
type Opening struct {
ID string `json:"id"`
Type Type `json:"type"`
ID string `json:"id"`
Type Type `json:"type"`
// Module is the module this resource belongs to, as the controller says it (novox/hq ADR 0429 §4, issue 1132). Absent
// is an older controller, and the module is then guessed from the id (ModuleGuess).
Module string `json:"module,omitempty"`
Port int `json:"port"`
Protocol string `json:"protocol"`
From string `json:"from"`
@@ -1072,8 +1267,11 @@ func (o *Opening) validate(where string, _ bool) []string {
// Present is the whole of what it asserts, never a version: version is the package manager's
// business and the mesh does not hold a second opinion about it.
type Package struct {
ID string `json:"id"`
Type Type `json:"type"`
ID string `json:"id"`
Type Type `json:"type"`
// Module is the module this resource belongs to, as the controller says it (novox/hq ADR 0429 §4, issue 1132). Absent
// is an older controller, and the module is then guessed from the id (ModuleGuess).
Module string `json:"module,omitempty"`
Package string `json:"package"`
// Absent declares that the package is NOT installed (novox/hq ADR 0180): the host removes it
// when it is, and leaves a machine that never had it alone. For the one case a module replaces
@@ -1098,7 +1296,10 @@ func (p *Package) validate(where string, _ bool) []string {
type Container struct {
ID string `json:"id"`
Type Type `json:"type"`
Name string `json:"name"`
// Module is the module this resource belongs to, as the controller says it (novox/hq ADR 0429 §4, issue 1132). Absent
// is an older controller, and the module is then guessed from the id (ModuleGuess).
Module string `json:"module,omitempty"`
Name string `json:"name"`
// Image is pinned by digest (novox/hq ADR 0006) — a tag moves and a digest does not.
Image string `json:"image"`
Env map[string]string `json:"env,omitempty"`
@@ -1239,6 +1440,11 @@ type Container struct {
// timing, judged by the node-engine beside liveness. Absent: judged alive or not, and nothing more.
Health *Health `json:"health,omitempty"`
// Busy is how this container says it holds work a replacement would lose (novox/hq ADR 0335): asked
// before it is replaced, and while it says busy it is left running, within the bound. Absent: never
// busy, and replaced at once, as before.
Busy *Busy `json:"busy,omitempty"`
// RunAs is who the container runs as, resolved by the controller from the module's identity and the
// image's declared interface (novox/hq ADR 0325 §2, §3): told by environment (the image reads PUID and
// PGID, say) or by the runtime's own user option. Part of the spec, so a change recreates it. Absent: the
@@ -1318,6 +1524,7 @@ func (c *Container) Target() string { return c.Name }
func (c *Container) validate(where string, _ bool) []string {
problems := c.Health.problems(where, true, !c.RunOnce && c.Schedule == "")
problems = append(problems, c.Busy.problems(where, !c.RunOnce && c.Schedule == "")...)
if c.Name == "" {
problems = append(problems, where+": a container needs a name")
}
@@ -1398,8 +1605,11 @@ func (c *Container) validate(where string, _ bool) []string {
// Action runs something the bundle declared, and the host never learns what it means.
type Action struct {
ID string `json:"id"`
Type Type `json:"type"`
ID string `json:"id"`
Type Type `json:"type"`
// Module is the module this resource belongs to, as the controller says it (novox/hq ADR 0429 §4, issue 1132). Absent
// is an older controller, and the module is then guessed from the id (ModuleGuess).
Module string `json:"module,omitempty"`
Command []string `json:"command"`
// Verify is not optional and is not a courtesy. It is the read-back AND the idempotency
// check: the host does not know what a database is, so "is it already there" is a question
@@ -1559,6 +1769,123 @@ type Declaration struct {
// this is its home, whether the operator's warrant to take it was carried out, the sub-paths its modules ask for,
// and the tool runner's account to put in its group (§11). Absent is a controller older than ADR 0328.
Libraries []Library
// Filled is the paths of the files the provisioner fills on this machine (novox/hq ADR 0349 §4, §5): each a
// file this declaration declares, which says who owns it at runtime. The provisioner's client has the
// node-engine write only these. Absent is a controller that hands no file to the provisioner, and every
// file is written from the declaration as before.
Filled []string
// Provisioner is the account the provisioner's client runs as on this machine, by name (novox/hq ADR 0349 §4):
// the only account whose write of a filled file the node-engine's local write socket accepts. Absent is a
// controller that names none, and every such write is refused.
Provisioner string
// Builds is the build id of every module this declaration names, by module (novox/hq ADR 0429 §4, issue 1132).
Builds map[string]string
// declared is what each resource says of its module, by id; nil on a declaration built by hand.
declared map[string]string
}
// ModuleGuess is the module a resource id names when the declaration does not say: everything before its last dot.
//
// The mesh composes a module's resource ids as `<module>.<its own id>`, and **a module's name may contain a dot**
// (`novox.be` is one on this mesh) while a resource's own id never does, so the owner is everything before the *last*
// dot. False for what the mesh declares in its own right: the foundation's resources carry no dot, and the
// adoption's are named for the mesh. This is the guess an older controller's declaration is read by; a declaration
// that says `module` is read by ModuleOfID, which never guesses for a resource that says (issue 1132).
func ModuleGuess(id string) (string, bool) {
if strings.HasPrefix(id, AdoptionPrefix) {
return "", false
}
at := strings.LastIndex(id, ".")
if at <= 0 {
return "", false
}
return id[:at], true
}
// ModuleOfID is the module a resource of this declaration belongs to: the module it says (novox/hq ADR 0429 §4,
// issue 1132), and otherwise the guess from its id (ModuleGuess), so a declaration from an older controller reads as
// it always did. False for what belongs to no module.
func (d *Declaration) ModuleOfID(id string) (string, bool) {
if d != nil {
if d.declared == nil {
for _, r := range d.Resources {
if r.Identity() == id && r.DeclaredModule() != "" {
return r.DeclaredModule(), true
}
}
} else if m, ok := d.declared[id]; ok {
return m, true
}
}
return ModuleGuess(id)
}
// indexModules keeps what each resource says of its module, so ModuleOfID is a lookup.
func (d *Declaration) indexModules() {
d.declared = map[string]string{}
for _, r := range d.Resources {
if m := r.DeclaredModule(); m != "" {
d.declared[r.Identity()] = m
}
}
}
// checkProvisioner refuses a provisioner client's account that is no account name, or is root: the account whose
// writes the node-engine accepts must be one that could not write the file without it.
func checkProvisioner(name string) []string {
switch {
case name == "":
return nil
case !accountName.MatchString(name):
return []string{fmt.Sprintf("provisioner_account %q is not an account name", name)}
case name == "root":
return []string{"provisioner_account is root; the provisioner's client runs as an account of its own"}
}
return nil
}
// Fills reports whether the provisioner fills the file at path on this machine.
func (d *Declaration) Fills(path string) bool { return slices.Contains(d.Filled, path) }
// checkFilled refuses a filled path that is not a declared file saying who owns it at runtime, or that is named
// twice or empty. Every file declared at the path must say it: the provisioner writes the path, and a file there
// the mesh writes some other way would be a second writer of it (ADR 0349 §5).
func checkFilled(filled []string, resources []Resource) []string {
var problems []string
files := map[string][]*File{}
for _, r := range resources {
if f, ok := r.(*File); ok {
files[f.Path] = append(files[f.Path], f)
}
}
seen := map[string]bool{}
for _, path := range filled {
switch {
case path == "":
problems = append(problems, "filled_files names a file with no path")
continue
case seen[path]:
problems = append(problems, fmt.Sprintf("filled_files names %q twice", path))
continue
}
seen[path] = true
if len(files[path]) == 0 {
problems = append(problems, fmt.Sprintf("filled_files names %q, and this declaration declares no "+
"file there", path))
continue
}
for _, f := range files[path] {
if f.OwnerAtRuntime == "" {
problems = append(problems, fmt.Sprintf("resource %q: the provisioner fills %q, and a file it "+
"fills says who owns it at runtime with owner-at-runtime", f.ID, path))
}
}
}
return problems
}
// Identity is one module's identity, as the controller allocated it (ADR 0328 §4): its user `mesh-<module>` with
@@ -1898,6 +2225,16 @@ type envelope struct {
// sends them only to a host whose health says link.LibraryContract.
Groups []Group `json:"groups,omitempty"`
Libraries []Library `json:"libraries,omitempty"`
// Filled is optional too (novox/hq ADR 0349 §5), as is a file's `owner-at-runtime`. **An older host refuses
// these keys**; a controller sends them only to a host whose health says link.OwnershipContract.
Filled []string `json:"filled_files,omitempty"`
// Provisioner is optional too (novox/hq ADR 0349 §4). **An older host refuses this key**; a controller sends it
// only to a host whose health says link.ProvisionerContract.
Provisioner string `json:"provisioner_account,omitempty"`
// Builds is optional too (novox/hq ADR 0429 §4, issue 1132): the build id of each module the declaration names, and
// every resource's `module`. **An older host refuses these keys**; a controller sends them only to a host whose
// reports say `reads_modules`.
Builds map[string]string `json:"builds,omitempty"`
}
func parse(raw []byte, allowActions bool) (*Declaration, error) {
@@ -1916,8 +2253,10 @@ func parse(raw []byte, allowActions bool) (*Declaration, error) {
d := &Declaration{Version: env.Version, For: env.For, Adoption: env.Adoption, Sequence: env.Sequence,
Epoch: env.Epoch, LeftOut: env.LeftOut, Generation: env.Generation, PutBack: env.PutBack,
Identities: env.Identities, Groups: env.Groups, Libraries: env.Libraries}
Identities: env.Identities, Groups: env.Groups, Libraries: env.Libraries, Filled: env.Filled,
Provisioner: env.Provisioner, Builds: env.Builds}
problems := checkIdentities(env.Identities, env.Groups)
problems = append(problems, checkProvisioner(env.Provisioner)...)
problems = append(problems, checkLibraries(env.Groups, env.Libraries)...)
if env.Generation < 0 {
// Below zero would read as "none claimed" and pass every refusal of an older generation.
@@ -2009,6 +2348,7 @@ func parse(raw []byte, allowActions bool) (*Declaration, error) {
problems = append(problems, checkWhileStopped(d.Resources)...)
problems = append(problems, checkAdoption(env.Adoption, d.Resources, allowActions)...)
problems = append(problems, checkReplaces(d.Resources)...)
problems = append(problems, checkFilled(env.Filled, d.Resources)...)
if env.Adoption == nil {
for _, r := range d.Resources {
if r.Kind() == TypeOpening {
@@ -2033,6 +2373,7 @@ func parse(raw []byte, allowActions bool) (*Declaration, error) {
if len(problems) > 0 {
return nil, &RefusalError{Problems: problems}
}
d.indexModules()
return d, nil
}
+48
View File
@@ -545,3 +545,51 @@ func TestAContainerMayLogToTheJournalAndNowhereElse(t *testing.T) {
}
}
}
// Issue 1024, ADR 0422: a directory or archive says how it treats files it did not write.
func TestADirectoryAndAnArchiveSayWhatTheyDoWithUnknownFiles(t *testing.T) {
d, err := Parse([]byte(`{"declaration":1,"resources":[
{"id":"d","type":"directory","path":"/home/a/x","unknown":"move-out","knows":["keep","sub/"]},
{"id":"r","type":"archive","source":"https://x/y","digest":"sha256:` + strings.Repeat("a", 64) + `","path":"/home/a/y","unknown":"module"}
]}`))
if err != nil {
t.Fatalf("refused: %v", err)
}
dir := d.Resources[0].(*Directory)
if dir.Unknown != UnknownMoveOut || len(dir.Knows) != 2 {
t.Errorf("directory: %+v", dir)
}
if ar := d.Resources[1].(*Archive); ar.Unknown != UnknownModule {
t.Errorf("archive: %+v", ar)
}
for _, v := range []string{UnknownMoveOut, UnknownModule, UnknownProgram} {
if _, err := Parse([]byte(`{"declaration":1,"resources":[{"id":"d","type":"directory","path":"/x","unknown":"` + v + `"}]}`)); err != nil {
t.Errorf("%s refused: %v", v, err)
}
}
}
func TestBadUnknownAndKnowsFormsAreRefusedNamingTheResource(t *testing.T) {
digest := `"digest":"sha256:` + strings.Repeat("a", 64) + `"`
cases := map[string]string{
"bad value": `{"id":"d","type":"directory","path":"/x","unknown":"delete"}`,
"empty entry": `{"id":"d","type":"directory","path":"/x","unknown":"move-out","knows":[""]}`,
"absolute": `{"id":"d","type":"directory","path":"/x","unknown":"move-out","knows":["/etc"]}`,
"empty comp": `{"id":"d","type":"directory","path":"/x","unknown":"move-out","knows":["a//b"]}`,
"dot": `{"id":"d","type":"directory","path":"/x","unknown":"move-out","knows":["./a"]}`,
"dotdot": `{"id":"d","type":"directory","path":"/x","unknown":"move-out","knows":["a/../b"]}`,
"repeat": `{"id":"d","type":"directory","path":"/x","unknown":"move-out","knows":["a","a"]}`,
"no unknown": `{"id":"d","type":"directory","path":"/x","knows":["a"]}`,
"archive bad": `{"id":"d","type":"archive","source":"s",` + digest + `,"path":"/x","unknown":"nope"}`,
"archive knws": `{"id":"d","type":"archive","source":"s",` + digest + `,"path":"/x","knows":["a"]}`,
}
for name, res := range cases {
t.Run(name, func(t *testing.T) {
refusal := refusalFor(t, `{"declaration":1,"resources":[`+res+`]}`)
if !strings.Contains(strings.Join(refusal.Problems, "\n"), `resource "d"`) {
t.Errorf("resource not named: %v", refusal.Problems)
}
})
}
}
+60
View File
@@ -0,0 +1,60 @@
package declaration
import "testing"
// 1132/AC1: a resource says its module, and the declaration says each module's build; a resource that does
// not say gets today's guess, so an older controller still works.
func TestAResourceSaysItsModuleAndTheDeclarationItsBuilds(t *testing.T) {
d, err := Parse([]byte(`{"declaration":1,
"builds":{"novox.be":"b-9","web":"b-4"},
"resources":[
{"id":"novox.be.data","type":"directory","path":"/srv/a","module":"novox.be"},
{"id":"web.conf","type":"file","path":"/etc/w","content":"x","module":"web"},
{"id":"old.dir","type":"directory","path":"/srv/old"},
{"id":"base","type":"directory","path":"/srv/base"}]}`))
if err != nil {
t.Fatal(err)
}
for id, want := range map[string]string{"novox.be.data": "novox.be", "web.conf": "web", "old.dir": "old"} {
got, ok := d.ModuleOfID(id)
if !ok || got != want {
t.Errorf("%s belongs to %q (%v), want %q", id, got, ok, want)
}
}
if m, ok := d.ModuleOfID("base"); ok {
t.Errorf("a resource with no module and no dot belongs to %q, want none", m)
}
if d.Builds["web"] != "b-4" || d.Builds["novox.be"] != "b-9" || len(d.Builds) != 2 {
t.Errorf("builds read as %v", d.Builds)
}
}
// 1132/AC1: the module the declaration gives wins over the guess, even when the id would guess another.
func TestTheDeclaredModuleWinsOverTheGuess(t *testing.T) {
d, err := Parse([]byte(`{"declaration":1,"resources":[
{"id":"a.b.c","type":"directory","path":"/srv/a","module":"a.b"},
{"id":"x.y","type":"directory","path":"/srv/x","module":"other"}]}`))
if err != nil {
t.Fatal(err)
}
if m, _ := d.ModuleOfID("x.y"); m != "other" {
t.Errorf("x.y belongs to %q, want the declared %q", m, "other")
}
}
// 1132/AC1: every kind accepts the field.
func TestEveryKindAcceptsAModule(t *testing.T) {
for _, typ := range []Type{TypeDirectory, TypeFile, TypeUser, TypeNetwork, TypeAccess, TypeArchive, TypeProcess,
TypeService, TypeOpening, TypePackage, TypeContainer, TypeAction} {
r := newOf(typ)
if r == nil {
t.Fatalf("no kind %s", typ)
}
if _, ok := r.(interface{ DeclaredModule() string }); !ok {
t.Errorf("%s does not say its module", typ)
}
if unknown := unknownFields([]byte(`{"id":"m.x","type":"`+string(typ)+`","module":"m"}`), r); len(unknown) != 0 {
t.Errorf("%s refuses %v", typ, unknown)
}
}
}
+130
View File
@@ -0,0 +1,130 @@
package declaration
import (
"strings"
"testing"
)
// Defends novox/hq ADR 0349 §5 (issue 667): a file the provisioner places says who owns it at runtime
// — owned, seed or managed — and the declaration names the files the provisioner fills. The words are
// read and judged here before any controller sends them.
func TestAFileSaysWhoOwnsItAtRuntime(t *testing.T) {
d, err := Parse([]byte(`{"declaration":1,"resources":[
{"id":"o","type":"file","path":"/etc/o","content":"a","owner-at-runtime":"owned"},
{"id":"s","type":"file","path":"/etc/s","content":"a","owner-at-runtime":"seed"},
{"id":"s2","type":"file","path":"/etc/s2","content":"a","owner-at-runtime":"seed","create-once":true},
{"id":"mj","type":"file","path":"/etc/mj","into":"json","content":"{\"k\":1}","owner-at-runtime":"managed"},
{"id":"mb","type":"file","path":"/etc/mb","into":"block","content":"a\n","owner-at-runtime":"managed"}
]}`))
if err != nil {
t.Fatal(err)
}
want := map[string]string{"o": RuntimeOwned, "s": RuntimeSeed, "s2": RuntimeSeed, "mj": RuntimeManaged,
"mb": RuntimeManaged}
for _, r := range d.Resources {
f := r.(*File)
if f.OwnerAtRuntime != want[f.ID] || f.Ownership() != want[f.ID] {
t.Errorf("%s: read as %q, ownership %q; want %q", f.ID, f.OwnerAtRuntime, f.Ownership(), want[f.ID])
}
}
}
// A file that does not say is owned by its old form, so the word means exactly what that form does.
func TestAFileThatDoesNotSayIsOwnedByItsOldForm(t *testing.T) {
d, err := Parse([]byte(`{"declaration":1,"resources":[
{"id":"o","type":"file","path":"/etc/o","content":"a"},
{"id":"s","type":"file","path":"/etc/s","content":"a","create-once":true},
{"id":"mj","type":"file","path":"/etc/mj","into":"json","content":"{}"},
{"id":"mb","type":"file","path":"/etc/mb","into":"block","content":"a\n"}
]}`))
if err != nil {
t.Fatal(err)
}
want := map[string]string{"o": RuntimeOwned, "s": RuntimeSeed, "mj": RuntimeManaged, "mb": RuntimeManaged}
for _, r := range d.Resources {
f := r.(*File)
if f.OwnerAtRuntime != "" || f.Ownership() != want[f.ID] {
t.Errorf("%s: said %q, ownership %q; want unsaid and %q", f.ID, f.OwnerAtRuntime, f.Ownership(), want[f.ID])
}
if f.Seeded() != (want[f.ID] == RuntimeSeed) {
t.Errorf("%s: seeded %v", f.ID, f.Seeded())
}
}
}
func TestAnOwnershipThatContradictsTheFileIsRefused(t *testing.T) {
for name, c := range map[string]struct{ resource, refusal string }{
"another word": {`{"id":"f","type":"file","path":"/etc/x","content":"a","owner-at-runtime":"mine"}`,
`owner-at-runtime "mine"`},
"owned and create-once": {`{"id":"f","type":"file","path":"/etc/x","content":"a","owner-at-runtime":"owned","create-once":true}`,
"create-once"},
"owned and into": {`{"id":"f","type":"file","path":"/etc/x","into":"json","content":"{}","owner-at-runtime":"owned"}`,
"into"},
"seed and into": {`{"id":"f","type":"file","path":"/etc/x","into":"block","content":"a","owner-at-runtime":"seed"}`,
"into"},
"managed and whole": {`{"id":"f","type":"file","path":"/etc/x","content":"a","owner-at-runtime":"managed"}`,
"into"},
"managed and create-once": {`{"id":"f","type":"file","path":"/etc/x","into":"json","content":"{}","owner-at-runtime":"managed","create-once":true}`,
"create-once"},
// The rule that was there before stays: a file written into is never sealed or created once.
"managed and sealed": {`{"id":"f","type":"file","path":"/etc/x","into":"json","content":"{}","sealed":"x","owner-at-runtime":"managed"}`,
"sealed"},
} {
_, err := Parse([]byte(`{"declaration":1,"resources":[` + c.resource + `]}`))
if err == nil || !strings.Contains(err.Error(), c.refusal) {
t.Errorf("%s: want a refusal naming %q, got %v", name, c.refusal, err)
}
}
}
func TestTheDeclarationNamesTheFilesTheProvisionerFills(t *testing.T) {
d, err := Parse([]byte(`{"declaration":1,"filled_files":["/etc/a","/etc/b"],"resources":[
{"id":"a","type":"file","path":"/etc/a","content":"a","owner-at-runtime":"owned"},
{"id":"b","type":"file","path":"/etc/b","into":"block","content":"b\n","owner-at-runtime":"managed"},
{"id":"c","type":"file","path":"/etc/c","content":"c"}
]}`))
if err != nil {
t.Fatal(err)
}
if len(d.Filled) != 2 || d.Filled[0] != "/etc/a" || d.Filled[1] != "/etc/b" {
t.Fatalf("filled files read as %v", d.Filled)
}
if !d.Fills("/etc/a") || !d.Fills("/etc/b") || d.Fills("/etc/c") {
t.Errorf("Fills: a %v, b %v, c %v", d.Fills("/etc/a"), d.Fills("/etc/b"), d.Fills("/etc/c"))
}
// A declaration that names none fills none, as every declaration before this did.
d, err = Parse([]byte(`{"declaration":1,"resources":[{"id":"c","type":"file","path":"/etc/c","content":"c"}]}`))
if err != nil {
t.Fatal(err)
}
if d.Filled != nil || d.Fills("/etc/c") {
t.Errorf("a declaration naming no filled files fills %v", d.Filled)
}
}
func TestAFilledFileMustBeADeclaredFileThatSaysItsOwner(t *testing.T) {
for name, c := range map[string]struct{ filled, resources, refusal string }{
"not declared": {`["/etc/none"]`,
`{"id":"a","type":"file","path":"/etc/a","content":"a","owner-at-runtime":"owned"}`,
`"/etc/none"`},
"a directory, not a file": {`["/etc/d"]`,
`{"id":"d","type":"directory","path":"/etc/d"}`,
`"/etc/d"`},
"says no owner": {`["/etc/a"]`,
`{"id":"a","type":"file","path":"/etc/a","content":"a"}`,
"owner-at-runtime"},
"named twice": {`["/etc/a","/etc/a"]`,
`{"id":"a","type":"file","path":"/etc/a","content":"a","owner-at-runtime":"owned"}`,
"twice"},
"no path": {`[""]`,
`{"id":"a","type":"file","path":"/etc/a","content":"a","owner-at-runtime":"owned"}`,
"no path"},
} {
_, err := Parse([]byte(`{"declaration":1,"filled_files":` + c.filled + `,"resources":[` + c.resources + `]}`))
if err == nil || !strings.Contains(err.Error(), c.refusal) {
t.Errorf("%s: want a refusal naming %q, got %v", name, c.refusal, err)
}
}
}
+37
View File
@@ -0,0 +1,37 @@
package declaration
import (
"strings"
"testing"
)
// Defends novox/hq ADR 0349 §4 (issue 668): the declaration names the account the provisioner's client runs as,
// the only one whose write of a filled file the node-engine accepts. Read here before any controller sends it.
func TestTheDeclarationNamesTheProvisionersAccount(t *testing.T) {
d, err := Parse([]byte(`{"declaration":1,"provisioner_account":"mesh-node-provisioner","filled_files":["/etc/a"],
"resources":[{"id":"a","type":"file","path":"/etc/a","content":"a","owner-at-runtime":"owned"}]}`))
if err != nil {
t.Fatal(err)
}
if d.Provisioner != "mesh-node-provisioner" {
t.Errorf("the provisioner's account read as %q", d.Provisioner)
}
d, err = Parse([]byte(`{"declaration":1,"resources":[{"id":"a","type":"file","path":"/etc/a","content":"a"}]}`))
if err != nil {
t.Fatal(err)
}
if d.Provisioner != "" {
t.Errorf("a declaration naming no account names %q", d.Provisioner)
}
}
func TestAProvisionersAccountThatIsNoAccountIsRefused(t *testing.T) {
for name, account := range map[string]string{"root": "root", "not a name": "a b", "a path": "../x"} {
_, err := Parse([]byte(`{"declaration":1,"provisioner_account":"` + account + `",
"resources":[{"id":"a","type":"file","path":"/etc/a","content":"a"}]}`))
if err == nil || !strings.Contains(err.Error(), "provisioner_account") {
t.Errorf("%s: want a refusal naming provisioner_account, got %v", name, err)
}
}
}
+512
View File
@@ -0,0 +1,512 @@
// Package filling is the node-engine's narrow local write verb (novox/hq ADR 0349 §4, design 57 P2): a socket on
// this machine through which the provisioner's client has a file it filled written, and nothing else.
//
// **The client fills; the node-engine writes.** The client renders a module's template with the values each domain
// owns and hands the node-engine the content; the node-engine writes it as the declared file, with the declared
// file's path, owner, mode and ownership at runtime, and makes what reads it take the change. So the one process that
// writes the machine's files is still the node-engine, and the client needs no right to write any of them.
//
// **Narrow by three checks, each from something the client cannot say for itself.** The account asking, from the
// kernel (SO_PEERCRED), in this machine's own namespaces, must be the account the node's current declaration names
// as the provisioner's client's; the path, clean and absolute, must be one the declaration names as filled; and the
// write takes the apply's lock, so a write and an apply never write one file at once. While the declaration names no
// account — every declaration before the provisioner's client has one — every write is refused.
//
// **No listening control surface** (novox/hq ADR 0004): the socket has no network address. And the content is never
// said: the journal has the path, the account and the content's digest.
package filling
import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"io"
"net"
"os"
"os/user"
"path/filepath"
"sort"
"strconv"
"sync"
"sync/atomic"
"time"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/meshcli"
)
// Socket is where the node-engine accepts a write of a filled file, beside mesh-cli's socket.
const Socket = "/run/nox-mesh-host/fill.sock"
// MaxRequest bounds one request, its content encoded in it: a configuration file, never an archive.
const MaxRequest = 8 << 20
// Request is one write, sent as one JSON object on one line: the path of a filled file and what the client filled
// into it (base64 in the JSON). The content only: the declaration says everything else about the file.
type Request struct {
Path string `json:"path"`
Content []byte `json:"content"`
}
// Answer is what came of one request. Refused is a write that was not attempted, with why; Failed one that was
// attempted and did not complete, with why — the file, or something that reads it. Otherwise Action is what the
// write did to the file (created, updated, corrected, unchanged, kept) and Digest what the file holds now, as the
// node-engine records it.
type Answer struct {
Path string `json:"path,omitempty"`
Action string `json:"action,omitempty"`
Digest string `json:"digest,omitempty"`
Refused string `json:"refused,omitempty"`
Failed string `json:"failed,omitempty"`
}
// Refusal is an answer that nothing was written, and why.
func Refusal(path, why string) Answer { return Answer{Path: path, Refused: why} }
// Writer is the node-engine's side of the socket.
type Writer interface {
// Accepts judges the asking account before its request is read: nil when the current declaration names it as
// the provisioner's client's.
Accepts(uid uint32) error
// Write judges the request and writes it, holding the apply's lock.
Write(ctx context.Context, uid uint32, req Request) Answer
}
// Engine is the Writer the node-engine serves: the declaration it applies now, read again under the apply's lock.
type Engine struct {
// Declared reads the declaration this node applies now, verified.
Declared func() (*declaration.Declaration, error)
// UIDOf is the user id of an account, by name, from this machine's user database.
UIDOf func(account string) (uint32, error)
// Lock takes the apply's lock and returns what releases it: the same the apply takes, so a write and an apply
// are one at a time.
Lock func() (func(), error)
// Place writes the content into the filled file at path, as the declared file, and makes what reads it take the
// change. Called holding the lock, with the declaration read under it.
Place func(ctx context.Context, d *declaration.Declaration, path string, content []byte) (action, digest string,
err error)
}
// Accepts reads the declaration and judges the account.
func (e *Engine) Accepts(uid uint32) error {
d, err := e.Declared()
if err != nil {
return fmt.Errorf("this node's declaration cannot be read, so no write is accepted: %w", err)
}
return JudgeAccount(d, uid, e.UIDOf)
}
// Write takes the apply's lock, reads the declaration under it, judges the account and the path again — the
// declaration may have changed since the request was accepted — and writes.
func (e *Engine) Write(ctx context.Context, uid uint32, req Request) Answer {
if err := JudgeShape(req.Path); err != nil {
return Refusal(req.Path, err.Error())
}
unlock, err := e.Lock()
if err != nil {
return Refusal(req.Path, "the apply's lock could not be taken, so nothing was written: "+err.Error())
}
defer unlock()
d, err := e.Declared()
if err != nil {
return Refusal(req.Path, "this node's declaration cannot be read, so nothing was written: "+err.Error())
}
if err := JudgeAccount(d, uid, e.UIDOf); err != nil {
return Refusal(req.Path, err.Error())
}
if err := JudgePath(d, req.Path); err != nil {
return Refusal(req.Path, err.Error())
}
action, digest, err := e.Place(ctx, d, req.Path, req.Content)
if err != nil {
return Answer{Path: req.Path, Action: action, Digest: digest, Failed: err.Error()}
}
return Answer{Path: req.Path, Action: action, Digest: digest}
}
// JudgeAccount refuses every account but the one the declaration names as the provisioner's client's, by its user
// id here — and every account while the declaration names none.
func JudgeAccount(d *declaration.Declaration, uid uint32, uidOf func(string) (uint32, error)) error {
if d.Provisioner == "" {
return errors.New("this node's declaration names no provisioner client's account, so no write is accepted")
}
want, err := uidOf(d.Provisioner)
if err != nil {
return fmt.Errorf("the provisioner client's account %s is not on this machine, so no write is accepted: %w",
d.Provisioner, err)
}
if want == 0 {
return fmt.Errorf("the provisioner client's account %s is root here, so no write is accepted", d.Provisioner)
}
if uid != want {
return fmt.Errorf("user id %d is not the provisioner client's account %s (user id %d), so nothing was written",
uid, d.Provisioner, want)
}
return nil
}
// JudgeShape refuses a path that is not clean and absolute: a relative path, `..`, a doubled or trailing slash.
// Compared as written with the declaration's, a path that is not clean could name a file the declaration does not.
func JudgeShape(path string) error {
switch {
case path == "":
return errors.New("the request names no path, so nothing was written")
case !filepath.IsAbs(path):
return fmt.Errorf("%q is not an absolute path, so nothing was written", path)
case filepath.Clean(path) != path:
return fmt.Errorf("%q is not a clean path (it is %q cleaned), so nothing was written", path,
filepath.Clean(path))
}
return nil
}
// JudgePath refuses a path that is not clean and absolute, or that the declaration does not name as filled. Where it
// leads — a link on the way, a place nothing is written — is judged by the write, by the apply's own rules.
func JudgePath(d *declaration.Declaration, path string) error {
if err := JudgeShape(path); err != nil {
return err
}
if !d.Fills(path) {
return fmt.Errorf("%s is not a file this node's declaration names as filled, so nothing was written", path)
}
return nil
}
// UIDOf is the user id of an account by name, from this machine's user database.
func UIDOf(account string) (uint32, error) {
u, err := user.Lookup(account)
if err != nil {
return 0, err
}
id, err := strconv.ParseUint(u.Uid, 10, 32)
if err != nil {
return 0, fmt.Errorf("user id %q: %w", u.Uid, err)
}
return uint32(id), nil
}
// peerOf and sameNamespaces are the kernel's word on the asking process, as mesh-cli's socket reads it. Variables so
// a test can say.
var (
peerOf = meshcli.PeerCredentials
sameNamespaces = meshcli.SameNamespaces
)
// within bounds how long an accepted connection may take to say its request, and refusedWithin how long a busy one
// is read and dropped before its refusal is written (so the client is not answered with a broken pipe). Variables so
// a test need not wait.
var (
within = 30 * time.Second
refusedWithin = 1 * time.Second
)
// atOnce bounds the accepted connections served at the same time: one client, writing one file at a time. One over
// is refused busy. busyWaitingAtOnce bounds how many busy refusals wait for their request at once, per Serve; over it
// a busy refusal is written at once, as mesh-cli's socket does. judgingAtOnce bounds the callers judged at once, and
// judgeWaiting the connections waiting to be judged — each an open file, so far below the engine's open-file limit;
// one over is closed at once, unread. Variables so a test can say.
var (
atOnce = 4
busyWaitingAtOnce = 8
judgingAtOnce = 16
judgeWaiting = 256
)
// refusalsSaidEvery is how often the refusals counted and not yet said are said: each kind of refusal of each account
// is said at once the first time, and what follows it within this time is counted and said at its end, or when the
// socket stops. A variable so a test need not wait.
var refusalsSaidEvery = time.Minute
// acceptBackoff is how long the socket waits after an accept that failed before it accepts again, at first and at
// most: a socket that stopped at the first such failure (too many open files, say) would stop the client's writes
// until the engine restarted. Variables so a test need not wait.
var (
acceptBackoff = 50 * time.Millisecond
acceptBackoffMax = 5 * time.Second
)
// acceptFrom accepts one connection. A variable so a test can make it fail.
var acceptFrom = func(l *net.UnixListener) (*net.UnixConn, error) { return l.AcceptUnix() }
// Serve listens on socket until ctx ends, and answers each connection: one request, one answer.
//
// The directory is made 0755 and the socket 0666, as mesh-cli's: who may write is judged per connection from the
// kernel's word, against the current declaration, so the socket needs no account that may not exist yet. **The
// caller is judged before it holds anything**: one the kernel does not say is the provisioner client's account is
// answered and closed at once, and only the client's connections are served or wait as busy, so no other account can
// hold the socket from it. A file at the socket's path that is not a socket is never removed: the engine says so and
// serves nothing.
func Serve(ctx context.Context, socket string, w Writer, say func(string)) error {
if say == nil {
say = func(string) {}
}
dir := filepath.Dir(socket)
if err := os.MkdirAll(dir, 0o755); err != nil {
return fmt.Errorf("the filled files' socket directory: %w", err)
}
if problem := meshcli.NotOwnDirectory(dir); problem != "" {
return fmt.Errorf("filled files are not written on request on this machine: %s", problem)
}
if err := os.Chmod(dir, 0o755); err != nil {
return fmt.Errorf("the filled files' socket directory: %w", err)
}
if info, err := os.Lstat(socket); err == nil {
if info.Mode()&os.ModeSocket == 0 {
return fmt.Errorf("%s is there and is not a socket, and the node-engine did not put it there; filled "+
"files are not written on request until it is moved", socket)
}
if err := os.Remove(socket); err != nil {
return fmt.Errorf("the socket a previous engine left at %s: %w", socket, err)
}
}
l, err := net.ListenUnix("unix", &net.UnixAddr{Name: socket, Net: "unix"})
if err != nil {
return fmt.Errorf("listening for filled files at %s: %w", socket, err)
}
if err := os.Chmod(socket, 0o666); err != nil {
l.Close()
return fmt.Errorf("opening the filled files' socket: %w", err)
}
go func() {
<-ctx.Done()
l.Close()
}()
slots := make(chan struct{}, atOnce)
// Judging reads the declaration: at most judging callers are judged at once, and the rest wait their turn — a
// few milliseconds each — rather than being refused, so a flood delays the client's write and never stops it.
judging := make(chan struct{}, judgingAtOnce)
var waiting atomic.Int64
busyWaiting := make(chan struct{}, busyWaitingAtOnce)
refusals := &refusalWords{say: say}
// Said once everything has stopped: what was counted is never lost with the socket.
defer refusals.flush()
var wg sync.WaitGroup
defer wg.Wait()
wg.Add(1)
go func() {
defer wg.Done()
tick := time.NewTicker(refusalsSaidEvery)
defer tick.Stop()
for {
select {
case <-ctx.Done():
return
case <-tick.C:
refusals.flush()
}
}
}()
backoff := time.Duration(0)
for {
conn, err := acceptFrom(l)
if err != nil {
if ctx.Err() != nil {
return nil
}
if errors.Is(err, net.ErrClosed) {
return fmt.Errorf("the filled files' socket was closed: %w", err)
}
// Said, waited for, and accepted again: the failure is the machine's for a moment, not the socket's.
backoff = min(max(2*backoff, acceptBackoff), acceptBackoffMax)
say(fmt.Sprintf("the filled files' socket could not accept a connection (%v); accepting again in %s",
err, backoff))
select {
case <-ctx.Done():
return nil
case <-time.After(backoff):
}
continue
}
backoff = 0
if waiting.Add(1) > int64(judgeWaiting) {
// Too many waiting to be judged already: this one is closed unread, and counted.
waiting.Add(-1)
refusals.said(^uint32(0), "overflow", "too many connections are waiting to be judged; closed unread")
conn.Close()
continue
}
wg.Add(1)
go func() {
defer wg.Done()
// Judged before it holds a slot or is waited for.
judging <- struct{}{}
uid, ok := judgePeer(conn, w, refusals)
<-judging
waiting.Add(-1)
if !ok {
return
}
readBy := time.Now().Add(within)
_ = conn.SetDeadline(readBy)
select {
case slots <- struct{}{}:
defer func() { <-slots }()
answer(ctx, conn, uid, w, say)
default:
busy(conn, readBy, busyWaiting, say)
}
}()
}
}
// judgePeer reads the asking account from the kernel and judges it. A caller that is not the provisioner client's
// account is answered with why and closed at once, its request never read: false.
func judgePeer(conn *net.UnixConn, w Writer, refusals *refusalWords) (uint32, bool) {
closeWith := func(uid uint32, kind, why string) (uint32, bool) {
refusals.said(uid, kind, why)
_ = conn.SetWriteDeadline(time.Now().Add(refusedWithin))
_ = json.NewEncoder(conn).Encode(Refusal("", why))
conn.Close()
return uid, false
}
uid, pid, err := peerOf(conn)
if err != nil {
return closeWith(^uint32(0), "unread", "the node-engine could not read which account is asking, so nothing was "+
"written: "+err.Error())
}
if err := sameNamespaces(pid); err != nil {
return closeWith(uid, "namespace", "the asking process does not run in this machine's own namespaces, so nothing was "+
"written: "+err.Error())
}
if err := w.Accepts(uid); err != nil {
return closeWith(uid, "account", err.Error())
}
return uid, true
}
// refusalWords says refused callers in the journal without letting a flood flood it: the first refusal of each kind
// for each account at once, and those that follow it counted and said by flush — on a timer and when the socket stops
// — so an account that stops still has its count said, and a refusal of another kind never hides behind one said.
type refusalWords struct {
mu sync.Mutex
say func(string)
pending map[refusalKey]*refusalCount
}
type refusalKey struct {
uid uint32
kind string
}
type refusalCount struct {
n int
why string
}
func (r *refusalWords) said(uid uint32, kind, why string) {
r.mu.Lock()
defer r.mu.Unlock()
if r.pending == nil {
r.pending = map[refusalKey]*refusalCount{}
}
k := refusalKey{uid, kind}
if c, ok := r.pending[k]; ok {
c.n++
c.why = why
return
}
r.pending[k] = &refusalCount{}
r.say(fmt.Sprintf("a filled file's write from %s was refused (%s): %s", who(k), kind, why))
}
// flush says what was counted since each was last said, and forgets every key, so the next refusal of each is said at
// once again.
func (r *refusalWords) flush() {
r.mu.Lock()
defer r.mu.Unlock()
keys := make([]refusalKey, 0, len(r.pending))
for k := range r.pending {
keys = append(keys, k)
}
sort.Slice(keys, func(i, j int) bool {
if keys[i].uid != keys[j].uid {
return keys[i].uid < keys[j].uid
}
return keys[i].kind < keys[j].kind
})
for _, k := range keys {
if c := r.pending[k]; c.n > 0 {
r.say(fmt.Sprintf("%d more writes of a filled file from %s were refused (%s) since it was last said; the last: %s",
c.n, who(k), k.kind, c.why))
}
}
r.pending = nil
}
// who names a caller by its user id, or says why it has none.
func who(k refusalKey) string {
switch {
case k.kind == "overflow":
return "a caller not yet judged"
case k.uid == ^uint32(0):
return "an account that could not be read"
}
return fmt.Sprintf("user id %d", k.uid)
}
// answer serves one connection from the provisioner client's account.
func answer(ctx context.Context, conn *net.UnixConn, uid uint32, w Writer, say func(string)) {
var req Request
dec := json.NewDecoder(io.LimitReader(conn, MaxRequest))
dec.DisallowUnknownFields()
if err := dec.Decode(&req); err != nil {
// Answered even when the deadline to say it passed: the deadline covered the write too.
_ = conn.SetWriteDeadline(time.Now().Add(refusedWithin))
reply(conn, Refusal("", "the request could not be read, so nothing was written: "+err.Error()), say)
return
}
// The write waits for the apply's lock as long as an apply takes, and the client waits with it.
_ = conn.SetDeadline(time.Time{})
a := w.Write(ctx, uid, req)
switch {
case a.Refused != "":
say(fmt.Sprintf("a write of %s from user id %d was refused: %s", req.Path, uid, a.Refused))
case a.Failed != "":
say(fmt.Sprintf("a write of %s from user id %d failed: %s", req.Path, uid, a.Failed))
default:
say(fmt.Sprintf("wrote %s for the provisioner's client (user id %d): %s, %s", req.Path, uid, a.Action,
a.Digest))
}
_ = conn.SetWriteDeadline(time.Now().Add(within))
reply(conn, a, say)
}
// busy answers the client's connection over atOnce: its request read and dropped first, briefly, so the client reads
// the answer rather than a broken pipe — while no more than busyWaiting such refusals wait at once; over that, at once.
func busy(conn *net.UnixConn, readBy time.Time, busyWaiting chan struct{}, say func(string)) {
select {
case busyWaiting <- struct{}{}:
by := time.Now().Add(refusedWithin)
if readBy.Before(by) {
by = readBy
}
_ = conn.SetReadDeadline(by)
drain(conn)
<-busyWaiting
default:
}
_ = conn.SetWriteDeadline(time.Now().Add(refusedWithin))
reply(conn, Refusal("", "busy: other writes are being served; ask again"), say)
}
// drain reads and drops a request up to its end — the newline after it, the end of the stream or the deadline —
// and at most MaxRequest bytes, keeping none of it.
func drain(r io.Reader) {
var buf [4096]byte
for read := 0; read < MaxRequest; {
n, err := r.Read(buf[:])
read += n
if bytes.IndexByte(buf[:n], '\n') >= 0 || err != nil {
return
}
}
}
func reply(conn *net.UnixConn, a Answer, say func(string)) {
defer conn.Close()
if err := json.NewEncoder(conn).Encode(a); err != nil {
say("a filled file's answer could not be written back: " + err.Error())
}
}
+561
View File
@@ -0,0 +1,561 @@
package filling
import (
"context"
"encoding/json"
"errors"
"fmt"
"net"
"os"
"path/filepath"
"strings"
"sync"
"testing"
"time"
"github.com/novox/mesh-host/internal/declaration"
)
// Defends novox/hq ADR 0349 §4 (issue 668): the node-engine writes a filled file on request only for the provisioner
// client's account, only for a clean path the current declaration names as filled, and one at a time with the apply.
const provisionerUID = 4242
func declared(t *testing.T, raw string) *declaration.Declaration {
t.Helper()
d, err := declaration.Parse([]byte(raw))
if err != nil {
t.Fatal(err)
}
return d
}
const withAccount = `{"declaration":1,"provisioner_account":"mesh-node-provisioner","filled_files":["/etc/m/m.conf"],
"resources":[{"id":"m.conf","type":"file","path":"/etc/m/m.conf","content":"x","owner-at-runtime":"owned"},
{"id":"m.other","type":"file","path":"/etc/m/other.conf","content":"y"}]}`
const withoutAccount = `{"declaration":1,"filled_files":["/etc/m/m.conf"],
"resources":[{"id":"m.conf","type":"file","path":"/etc/m/m.conf","content":"x","owner-at-runtime":"owned"}]}`
// engine is an Engine over a fixed declaration, recording what it was asked to place.
type engine struct {
Engine
mu sync.Mutex
placed []string
}
func newEngine(t *testing.T, raw string, lock sync.Locker) *engine {
d := declared(t, raw)
e := &engine{}
e.Engine = Engine{
Declared: func() (*declaration.Declaration, error) { return d, nil },
UIDOf: func(name string) (uint32, error) {
if name == "mesh-node-provisioner" {
return provisionerUID, nil
}
return 0, errors.New("no such account")
},
Lock: func() (func(), error) { lock.Lock(); return lock.Unlock, nil },
Place: func(_ context.Context, _ *declaration.Declaration, path string, content []byte) (string, string, error) {
e.mu.Lock()
defer e.mu.Unlock()
e.placed = append(e.placed, path)
return "updated", "digest", nil
},
}
return e
}
func TestAWriteFromTheProvisionersAccountOfAFilledPathIsPlaced(t *testing.T) {
e := newEngine(t, withAccount, &sync.Mutex{})
if err := e.Accepts(provisionerUID); err != nil {
t.Fatal(err)
}
a := e.Write(context.Background(), provisionerUID, Request{Path: "/etc/m/m.conf", Content: []byte("x")})
if a.Refused != "" || a.Failed != "" || a.Action != "updated" || len(e.placed) != 1 {
t.Fatalf("answer %+v, placed %v", a, e.placed)
}
}
func TestAnUndeclaredPathIsRefused(t *testing.T) {
e := newEngine(t, withAccount, &sync.Mutex{})
for _, path := range []string{"/etc/m/other.conf", "/etc/m/none.conf", "/etc/passwd"} {
a := e.Write(context.Background(), provisionerUID, Request{Path: path, Content: []byte("evil")})
if a.Refused == "" || !strings.Contains(a.Refused, "not a file this node's declaration names as filled") {
t.Errorf("%s: %+v", path, a)
}
}
if len(e.placed) != 0 {
t.Errorf("placed %v", e.placed)
}
}
func TestAnotherAccountIsRefused(t *testing.T) {
e := newEngine(t, withAccount, &sync.Mutex{})
for _, uid := range []uint32{0, 1000, provisionerUID + 1} {
if err := e.Accepts(uid); err == nil {
t.Errorf("user id %d was accepted", uid)
}
a := e.Write(context.Background(), uid, Request{Path: "/etc/m/m.conf", Content: []byte("evil")})
if a.Refused == "" || !strings.Contains(a.Refused, "is not the provisioner client's account") {
t.Errorf("user id %d: %+v", uid, a)
}
}
if len(e.placed) != 0 {
t.Errorf("placed %v", e.placed)
}
}
func TestEveryWriteIsRefusedWhileTheDeclarationNamesNoAccount(t *testing.T) {
e := newEngine(t, withoutAccount, &sync.Mutex{})
if err := e.Accepts(provisionerUID); err == nil || !strings.Contains(err.Error(), "names no provisioner") {
t.Errorf("accepted with no account named: %v", err)
}
a := e.Write(context.Background(), provisionerUID, Request{Path: "/etc/m/m.conf", Content: []byte("x")})
if a.Refused == "" || len(e.placed) != 0 {
t.Errorf("written with no account named: %+v", a)
}
}
func TestAnAccountThatIsRootHereIsRefused(t *testing.T) {
e := newEngine(t, withAccount, &sync.Mutex{})
e.UIDOf = func(string) (uint32, error) { return 0, nil }
if err := e.Accepts(0); err == nil || !strings.Contains(err.Error(), "root") {
t.Errorf("an account that is root here was accepted: %v", err)
}
}
func TestAnUncleanPathIsRefused(t *testing.T) {
e := newEngine(t, withAccount, &sync.Mutex{})
for _, path := range []string{"", "etc/m/m.conf", "m.conf", "/etc//m/m.conf", "/etc/m/../m/m.conf",
"/etc/m/./m.conf", "/etc/m/m.conf/"} {
a := e.Write(context.Background(), provisionerUID, Request{Path: path, Content: []byte("x")})
if a.Refused == "" {
t.Errorf("%q was not refused: %+v", path, a)
}
}
if len(e.placed) != 0 {
t.Errorf("placed %v", e.placed)
}
}
// A write waits for an apply holding the apply's lock, and writes only once it is let go.
func TestAWriteWaitsForTheApply(t *testing.T) {
var applying sync.Mutex
e := newEngine(t, withAccount, &applying)
applying.Lock()
done := make(chan Answer)
go func() {
done <- e.Write(context.Background(), provisionerUID, Request{Path: "/etc/m/m.conf", Content: []byte("x")})
}()
select {
case a := <-done:
t.Fatalf("written while the apply held the lock: %+v", a)
case <-time.After(100 * time.Millisecond):
}
e.mu.Lock()
n := len(e.placed)
e.mu.Unlock()
if n != 0 {
t.Fatalf("placed while the apply held the lock")
}
applying.Unlock()
if a := <-done; a.Action != "updated" {
t.Fatalf("after the apply: %+v", a)
}
}
// fixed is a Writer that accepts one user id and answers every write.
type fixed struct {
uid uint32
mu sync.Mutex
writes []Request
}
func (f *fixed) Accepts(uid uint32) error {
if uid != f.uid {
return errors.New("not the provisioner client's account")
}
return nil
}
func (f *fixed) Write(_ context.Context, _ uint32, req Request) Answer {
f.mu.Lock()
defer f.mu.Unlock()
f.writes = append(f.writes, req)
return Answer{Path: req.Path, Action: "updated", Digest: "d"}
}
// serve serves w on a socket in a fresh directory and returns its path, and what was said.
func serve(t *testing.T, w Writer) (string, func() []string) {
t.Helper()
dir, err := os.MkdirTemp("", "fill")
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { os.RemoveAll(dir) })
socket := filepath.Join(dir, "fill.sock")
ctx, cancel := context.WithCancel(context.Background())
var mu sync.Mutex
var said []string
stopped := make(chan error)
go func() {
stopped <- Serve(ctx, socket, w, func(s string) { mu.Lock(); said = append(said, s); mu.Unlock() })
}()
t.Cleanup(func() {
cancel()
if err := <-stopped; err != nil {
t.Error(err)
}
})
for i := 0; i < 100; i++ {
if _, err := os.Stat(socket); err == nil {
break
}
time.Sleep(10 * time.Millisecond)
}
return socket, func() []string { mu.Lock(); defer mu.Unlock(); return append([]string(nil), said...) }
}
func ask(t *testing.T, socket string, req Request) Answer {
t.Helper()
conn, err := net.Dial("unix", socket)
if err != nil {
t.Fatal(err)
}
defer conn.Close()
// Bounded: a socket that never answers fails the test instead of hanging it.
_ = conn.SetDeadline(time.Now().Add(5 * time.Second))
if err := json.NewEncoder(conn).Encode(req); err != nil {
t.Fatal(err)
}
var a Answer
if err := json.NewDecoder(conn).Decode(&a); err != nil {
t.Fatal(err)
}
return a
}
// Over a real socket, the kernel says who is asking: this test's own account is the provisioner's here.
func TestTheSocketWritesForTheAccountTheKernelSays(t *testing.T) {
w := &fixed{uid: uint32(os.Getuid())}
socket, said := serve(t, w)
a := ask(t, socket, Request{Path: "/etc/m/m.conf", Content: []byte("secret value")})
if a.Refused != "" || a.Action != "updated" || len(w.writes) != 1 || string(w.writes[0].Content) != "secret value" {
t.Fatalf("answer %+v, writes %+v", a, w.writes)
}
for _, line := range said() {
if strings.Contains(line, "secret value") {
t.Errorf("the content was said: %q", line)
}
}
}
func TestTheSocketRefusesAnotherAccountBeforeReadingItsRequest(t *testing.T) {
w := &fixed{uid: uint32(os.Getuid()) + 1}
socket, said := serve(t, w)
conn, err := net.Dial("unix", socket)
if err != nil {
t.Fatal(err)
}
defer conn.Close()
// Closed at once, its request never read: the write may meet a closed socket, and the refusal is there to read.
_ = json.NewEncoder(conn).Encode(Request{Path: "/etc/m/m.conf", Content: []byte("secret value")})
var a Answer
if err := json.NewDecoder(conn).Decode(&a); err != nil {
t.Fatal(err)
}
if a.Refused == "" || len(w.writes) != 0 {
t.Fatalf("another account was written for: %+v", a)
}
for _, line := range said() {
if strings.Contains(line, "secret value") {
t.Errorf("the content was said: %q", line)
}
}
}
// floodersAreAnother has the kernel's word say another account for every connection a test binds to a name holding
// "flood": the test runs as one account, and the socket must tell the client from everyone else.
func floodersAreAnother(t *testing.T) {
t.Helper()
was := peerOf
peerOf = func(conn *net.UnixConn) (uint32, int32, error) {
uid, pid, err := was(conn)
if a, ok := conn.RemoteAddr().(*net.UnixAddr); ok && a != nil && strings.Contains(a.Name, "flood") {
uid++
}
return uid, pid, err
}
t.Cleanup(func() { peerOf = was })
}
// A flood of connections from another account, each saying nothing and held open, does not stop the client's write:
// each is refused and closed before it holds anything.
func TestAFloodFromAnotherAccountDoesNotStopTheClient(t *testing.T) {
floodersAreAnother(t)
w := &fixed{uid: uint32(os.Getuid())}
socket, _ := serve(t, w)
from := t.TempDir()
var flood []*net.UnixConn
t.Cleanup(func() {
for _, c := range flood {
c.Close()
}
})
for i := 0; i < 3*atOnce+busyWaitingCap(); i++ {
c, err := net.DialUnix("unix", &net.UnixAddr{Name: filepath.Join(from, fmt.Sprintf("flood-%d", i)), Net: "unix"},
&net.UnixAddr{Name: socket, Net: "unix"})
if err != nil {
t.Fatal(err)
}
flood = append(flood, c)
}
start := time.Now()
a := ask(t, socket, Request{Path: "/etc/m/m.conf", Content: []byte("x")})
if a.Refused != "" || a.Action != "updated" {
t.Fatalf("the client's write during a flood: %+v", a)
}
if took := time.Since(start); took > refusedWithin/2 {
t.Errorf("the client's write waited %s behind the flood", took)
}
// And every flooder was closed: it reads its refusal and then the end.
for _, c := range flood {
_ = c.SetReadDeadline(time.Now().Add(2 * time.Second))
var got Answer
if err := json.NewDecoder(c).Decode(&got); err != nil || got.Refused == "" {
t.Fatalf("a flooder was not refused at once: %+v %v", got, err)
}
}
}
func busyWaitingCap() int { return busyWaitingAtOnce }
// A client that connects and says nothing holds one slot only until within, and another write goes on meanwhile.
func TestASilentClientDoesNotHoldTheSocket(t *testing.T) {
was := within
within = 300 * time.Millisecond
t.Cleanup(func() { within = was })
w := &fixed{uid: uint32(os.Getuid())}
socket, _ := serve(t, w)
silent, err := net.Dial("unix", socket)
if err != nil {
t.Fatal(err)
}
defer silent.Close()
if a := ask(t, socket, Request{Path: "/etc/m/m.conf", Content: []byte("x")}); a.Action != "updated" {
t.Fatalf("a write beside a silent client: %+v", a)
}
_ = silent.SetReadDeadline(time.Now().Add(3 * time.Second))
var got Answer
if err := json.NewDecoder(silent).Decode(&got); err != nil || got.Refused == "" {
t.Errorf("the silent client was not let go after within: %+v %v", got, err)
}
}
// A request carrying more than the path and the content is refused: it says what the declaration says.
func TestARequestWithAnUnknownFieldIsRefused(t *testing.T) {
w := &fixed{uid: uint32(os.Getuid())}
socket, _ := serve(t, w)
conn, err := net.Dial("unix", socket)
if err != nil {
t.Fatal(err)
}
defer conn.Close()
if _, err := conn.Write([]byte(`{"path":"/etc/m/m.conf","content":"eA==","mode":"0777"}` + "\n")); err != nil {
t.Fatal(err)
}
var a Answer
if err := json.NewDecoder(conn).Decode(&a); err != nil {
t.Fatal(err)
}
if a.Refused == "" || !strings.Contains(a.Refused, "mode") || len(w.writes) != 0 {
t.Errorf("a request with an unknown field: %+v, writes %v", a, w.writes)
}
}
// An accept that fails is said, waited for and tried again: the socket goes on serving.
func TestTheSocketGoesOnAfterAnAcceptFails(t *testing.T) {
wasAccept, wasBackoff := acceptFrom, acceptBackoff
failures := 3
var mu sync.Mutex
acceptFrom = func(l *net.UnixListener) (*net.UnixConn, error) {
mu.Lock()
defer mu.Unlock()
if failures > 0 {
failures--
return nil, errors.New("accept: too many open files")
}
return l.AcceptUnix()
}
acceptBackoff = time.Millisecond
t.Cleanup(func() { acceptFrom, acceptBackoff = wasAccept, wasBackoff })
w := &fixed{uid: uint32(os.Getuid())}
socket, said := serve(t, w)
if a := ask(t, socket, Request{Path: "/etc/m/m.conf", Content: []byte("x")}); a.Action != "updated" {
t.Fatalf("after failed accepts: %+v", a)
}
if !strings.Contains(strings.Join(said(), "\n"), "too many open files") {
t.Errorf("the failed accepts were not said: %v", said())
}
}
// The socket's directory is given the mode the socket needs, as mesh-cli's is.
func TestTheSocketsDirectoryIsOpenedToEveryAccount(t *testing.T) {
dir, err := os.MkdirTemp("", "fill")
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { os.RemoveAll(dir) })
if err := os.Chmod(dir, 0o700); err != nil {
t.Fatal(err)
}
ctx, cancel := context.WithCancel(context.Background())
stopped := make(chan error)
go func() { stopped <- Serve(ctx, filepath.Join(dir, "fill.sock"), &fixed{}, nil) }()
for i := 0; i < 100; i++ {
if _, err := os.Stat(filepath.Join(dir, "fill.sock")); err == nil {
break
}
time.Sleep(10 * time.Millisecond)
}
cancel()
if err := <-stopped; err != nil {
t.Fatal(err)
}
info, err := os.Stat(dir)
if err != nil {
t.Fatal(err)
}
if info.Mode().Perm() != 0o755 {
t.Errorf("the socket's directory is %o", info.Mode().Perm())
}
}
// The first refusal of each kind for each account is said at once; what follows within the time is counted, and the
// count is said when the time is up, even when the account has stopped.
func TestRefusalsAreSaidOnceAndTheirCountIsNeverLost(t *testing.T) {
var mu sync.Mutex
var said []string
r := &refusalWords{say: func(s string) { mu.Lock(); said = append(said, s); mu.Unlock() }}
for i := 0; i < 3; i++ {
r.said(1000, "account", "not the provisioner client's account")
}
// Another kind from the same account is said at once, not hidden behind the first.
r.said(1000, "namespace", "another namespace")
if len(said) != 2 || !strings.Contains(said[0], "(account)") || !strings.Contains(said[1], "(namespace)") {
t.Fatalf("said at once: %q", said)
}
// The account stops; the time is up; the two counted are said.
r.flush()
if len(said) != 3 || !strings.Contains(said[2], "2 more") || !strings.Contains(said[2], "user id 1000") {
t.Fatalf("after the time: %q", said)
}
// And the next refusal is said at once again.
r.said(1000, "account", "not the provisioner client's account")
if len(said) != 4 {
t.Fatalf("after the count was said: %q", said)
}
}
// refusingSocket serves a socket that refuses this test's account, with counted refusals said every every. It returns
// how to be refused n times, everything said so far, and how to stop it.
func refusingSocket(t *testing.T, every time.Duration) (func(int), func() string, func()) {
t.Helper()
was := refusalsSaidEvery
refusalsSaidEvery = every
dir, err := os.MkdirTemp("", "fill")
if err != nil {
t.Fatal(err)
}
socket := filepath.Join(dir, "fill.sock")
ctx, cancel := context.WithCancel(context.Background())
var mu sync.Mutex
var said []string
all := func() string { mu.Lock(); defer mu.Unlock(); return strings.Join(said, "\n") }
stopped := make(chan error)
go func() {
stopped <- Serve(ctx, socket, &fixed{uid: uint32(os.Getuid()) + 1},
func(s string) { mu.Lock(); said = append(said, s); mu.Unlock() })
}()
for i := 0; i < 100; i++ {
if _, err := os.Stat(socket); err == nil {
break
}
time.Sleep(10 * time.Millisecond)
}
refuse := func(n int) {
for i := 0; i < n; i++ {
conn, err := net.Dial("unix", socket)
if err != nil {
t.Fatal(err)
}
_ = conn.SetDeadline(time.Now().Add(5 * time.Second))
var a Answer
_ = json.NewDecoder(conn).Decode(&a)
conn.Close()
}
}
var once sync.Once
stop := func() {
once.Do(func() {
cancel()
if err := <-stopped; err != nil {
t.Error(err)
}
refusalsSaidEvery = was
os.RemoveAll(dir)
})
}
t.Cleanup(stop)
return refuse, all, stop
}
// Refusals counted while the socket serves are said on its timer, though the account has stopped asking.
func TestTheSocketSaysItsCountedRefusalsOnATimer(t *testing.T) {
refuse, all, _ := refusingSocket(t, 200*time.Millisecond)
refuse(3)
// "more writes", not a number: a tick may fall between two refusals and split the count.
deadline := time.Now().Add(3 * time.Second)
for !strings.Contains(all(), "more writes") && time.Now().Before(deadline) {
time.Sleep(20 * time.Millisecond)
}
if !strings.Contains(all(), "more writes") {
t.Fatalf("the counted refusals were not said on the timer: %s", all())
}
}
// Refusals counted when the socket stops are said as it stops.
func TestTheSocketSaysItsCountedRefusalsWhenItStops(t *testing.T) {
refuse, all, stop := refusingSocket(t, time.Hour)
refuse(5)
if strings.Contains(all(), "4 more") {
t.Fatalf("said before the socket stopped: %s", all())
}
stop()
if !strings.Contains(all(), "4 more") {
t.Errorf("the refusals counted when the socket stopped were lost: %s", all())
}
}
// A connection over the bound of those waiting to be judged is closed unread, and said.
func TestAConnectionOverTheWaitingBoundIsClosedUnread(t *testing.T) {
was := judgeWaiting
judgeWaiting = 0
t.Cleanup(func() { judgeWaiting = was })
w := &fixed{uid: uint32(os.Getuid())}
socket, said := serve(t, w)
conn, err := net.Dial("unix", socket)
if err != nil {
t.Fatal(err)
}
defer conn.Close()
_ = conn.SetDeadline(time.Now().Add(5 * time.Second))
var a Answer
if err := json.NewDecoder(conn).Decode(&a); err == nil {
t.Fatalf("answered over the bound: %+v", a)
}
if len(w.writes) != 0 || !strings.Contains(strings.Join(said(), "\n"), "(overflow)") {
t.Errorf("writes %v, said %q", w.writes, said())
}
}
+64
View File
@@ -3,11 +3,15 @@ package identity
import (
"crypto/ecdh"
"crypto/rand"
"crypto/x509"
"encoding/base64"
"encoding/json"
"encoding/pem"
"fmt"
"os"
"strings"
"git.novox.be/novox/mesh-sdk/go/seal"
"golang.org/x/crypto/nacl/box"
)
@@ -141,3 +145,63 @@ func Seal(publicKey string, value []byte) (string, error) {
}
return base64.StdEncoding.EncodeToString(sealed), nil
}
// **The vault's box, opened with the same key** (novox/hq ADR 0349 §16, ADR 0206, issue 755). The core's own
// credentials travel in the declaration as values the vault sealed to this machine, which the controller carries and
// cannot open. The vault seals in the mesh's sealed box (mesh-sdk go/seal), whose keys travel as PEM: the public half
// as SPKI, the private as PKCS #8. They are this key's two halves in another wrapping, never another key.
// PublicPEM is this machine's sealing key's public half as SPKI PEM, the form the vault seals to.
func (s SealingKey) PublicPEM() (string, error) {
raw, err := base64.StdEncoding.DecodeString(s.Public)
if err != nil || len(raw) != 32 {
return "", fmt.Errorf("this node's sealing key is unusable")
}
pub, err := ecdh.X25519().NewPublicKey(raw)
if err != nil {
return "", fmt.Errorf("this node's sealing key is unusable: %w", err)
}
der, err := x509.MarshalPKIXPublicKey(pub)
if err != nil {
return "", err
}
return string(pem.EncodeToMemory(&pem.Block{Type: "PUBLIC KEY", Bytes: der})), nil
}
// OpenVaultBox opens a value the vault sealed to this machine. It fails for a box sealed to another key, or changed.
func (s SealingKey) OpenVaultBox(b seal.Box) ([]byte, error) {
raw, err := base64.StdEncoding.DecodeString(s.Private)
if err != nil || len(raw) != 32 {
return nil, fmt.Errorf("this node's sealing key is unusable")
}
priv, err := ecdh.X25519().NewPrivateKey(raw)
if err != nil {
return nil, fmt.Errorf("this node's sealing key is unusable: %w", err)
}
der, err := x509.MarshalPKCS8PrivateKey(priv)
if err != nil {
return nil, err
}
block := &pem.Block{Type: "PRIVATE KEY", Bytes: der}
opened, err := seal.Open(b, string(pem.EncodeToMemory(block)))
for i := range block.Bytes {
block.Bytes[i] = 0
}
if err != nil {
return nil, fmt.Errorf("this was not sealed by the vault to this node's current key")
}
return []byte(opened), nil
}
// Open opens anything the mesh sealed to this machine: the vault's box, given as its JSON (it begins with `{`), or
// the controller's sealed value, base64 (which never does).
func (s SealingKey) Open(sealed string) ([]byte, error) {
if strings.HasPrefix(strings.TrimSpace(sealed), "{") {
var b seal.Box
if err := json.Unmarshal([]byte(sealed), &b); err != nil {
return nil, fmt.Errorf("this is not the vault's sealed box: %w", err)
}
return s.OpenVaultBox(b)
}
return s.Unseal(sealed)
}
+52 -16
View File
@@ -2,9 +2,11 @@ package link
import (
"context"
"errors"
"fmt"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/nats-io/nats.go"
)
@@ -24,11 +26,6 @@ type Bus interface {
// disagreeing with nothing anywhere saying so — the shape of fault this project keeps
// finding. Returns false when it could not be delivered, so the caller can say so.
Report(ctx context.Context, node string, body []byte) error
// Alive says this node is here, and nothing else. **Losing one is nothing**: the next is a
// minute away and the mesh reads a gap rather than counting arrivals. Insisting on delivery
// would turn a harmless miss into a logged failure every minute.
Alive(ctx context.Context, node string, body []byte) error
}
// --- The bus the mesh runs on today -----------------------------------------------------------
@@ -40,11 +37,10 @@ type OverNATS struct {
JS nats.JetStreamContext
}
// ReportSubject and AliveSubject are this node's own, and no other node's: a host's account may
// ReportSubject is this node's own, and no other node's: a host's account may
// publish `mesh.control.<its own node>.>` and nothing wider, so the subject is the authority on
// which node a report is about.
func ReportSubject(node string) string { return "mesh.control." + node + ".report" }
func AliveSubject(node string) string { return "mesh.control." + node + ".alive" }
// HealthSubject is where this node says its long-running resources' health between reports (novox/hq
// ADR 0240): its own, inside the grant every host already has, and on core NATS like the heartbeat — a
@@ -103,15 +99,55 @@ func (b OverNATS) Report(ctx context.Context, node string, body []byte) error {
return nil
}
func (b OverNATS) Alive(ctx context.Context, node string, body []byte) error {
// Core, deliberately: a heartbeat in a stream is the mesh's least valuable message competing
// for retention with its most valuable, and a lost one is the next one.
if err := b.Conn.Publish(AliveSubject(node), body); err != nil {
// HaveBind binds the have bucket the controller made (novox/hq ADR 0429 §4, issue 1132), once for a pass of writes.
// **Only binds**: a node-engine's account reaches no part of the JetStream API beyond what it is granted, and the bucket
// is the controller's to create and to configure, so a bucket that is not there is an error said as what it is. Every
// call on the bucket waits no longer than ctx has left: a refused publish is answered by nothing, and the client's own
// wait is five seconds.
func (b OverNATS) HaveBind(ctx context.Context) (HaveBucket, error) {
js := b.JS
wait := 5 * time.Second
if at, ok := ctx.Deadline(); ok {
wait = time.Until(at)
if wait <= 0 {
return nil, context.DeadlineExceeded
}
}
if b.Conn != nil {
bounded, err := b.Conn.JetStream(nats.MaxWait(wait))
if err != nil {
return nil, err
}
js = bounded
}
kv, err := js.KeyValue(delivery.HaveBucket)
if err != nil {
return nil, fmt.Errorf("the have bucket %s: %w", delivery.HaveBucket, err)
}
return natsHaveBucket{kv}, nil
}
type natsHaveBucket struct{ kv nats.KeyValue }
func (n natsHaveBucket) Get(_ context.Context, key string) ([]byte, uint64, error) {
entry, err := n.kv.Get(key)
if errors.Is(err, nats.ErrKeyNotFound) {
return nil, 0, ErrHaveAbsent
}
if err != nil {
return nil, 0, err
}
return entry.Value(), entry.Revision(), nil
}
func (n natsHaveBucket) Set(_ context.Context, key string, value []byte, revision uint64) error {
var err error
if revision == 0 {
_, err = n.kv.Create(key, value)
return err
}
// Flushed rather than fired and forgotten, so "could not tell the mesh" means the write
// failed rather than that nobody has looked yet.
flush, cancel := context.WithTimeout(ctx, 2*time.Second)
defer cancel()
return b.Conn.FlushWithContext(flush)
_, err = n.kv.Update(key, value, revision)
return err
}
var _ HaveBus = OverNATS{}
+8 -1
View File
@@ -143,7 +143,14 @@ func believed(check HandOverCheck, context string, withPath bool, body []byte) (
// believedBytes is believed with the ask's bytes as signed, for an ask that carries more than a hand-over's
// fields: read out of them only once they are believed.
func believedBytes(check HandOverCheck, context string, withPath bool, body []byte) (HandOverAsk, []byte, string) {
if len(body) > MaxHandOverAsk {
return believedBytesWithin(check, context, withPath, body, MaxHandOverAsk)
}
// believedBytesWithin is believedBytes for an ask bounded at max bytes: a move carries a sealed value, more than
// a hand-over's few hundred bytes (movesecret.go).
func believedBytesWithin(check HandOverCheck, context string, withPath bool, body []byte, max int) (HandOverAsk,
[]byte, string) {
if len(body) > max {
return HandOverAsk{}, nil, fmt.Sprintf("the ask is %d bytes, more than an ask", len(body))
}
var signed SignedHandOver
+242
View File
@@ -0,0 +1,242 @@
package link
import (
"context"
"encoding/json"
"errors"
"fmt"
"sort"
"sync"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// ErrHaveAbsent is what HaveGet returns for a key the bucket does not hold.
var ErrHaveAbsent = errors.New("link: no such key")
// HaveBus is a link that can bind the have bucket (novox/hq ADR 0429 §4). Separate from Bus, as HealthBus is: a link that
// cannot is still a link, and the write waits for the next apply or look.
type HaveBus interface {
// HaveBind binds the bucket the controller made, once for a pass of writes, within ctx. A node whose grant is not there
// yet gets an error here (or the context's end), and writes nothing.
HaveBind(ctx context.Context) (HaveBucket, error)
}
// HaveBucket is the bound bucket.
type HaveBucket interface {
// Get is a key's value and last revision, or ErrHaveAbsent.
Get(ctx context.Context, key string) (value []byte, revision uint64, err error)
// Set writes a key over the revision read: zero creates it, and a key that moved since is refused.
Set(ctx context.Context, key string, value []byte, revision uint64) error
}
// haveState is what the node-engine last learned of each module of the declaration it applied, and of each module's health
// as the last look found it — kept so a look that changes only a health can write only that module's have (issue 1132).
type haveState struct {
mu sync.Mutex
modules map[string]ModuleFact
health map[string]delivery.Health
node *delivery.HaveNode
}
// moduleHealths is the health of each module a look names, from the states of its resources: failed when any is
// unhealthy, waits when any waits (waiting, starting, held), ok when all are healthy, unknown otherwise.
func moduleHealths(h *Health) map[string]delivery.Health {
out := map[string]delivery.Health{}
if h == nil {
return out
}
rank := map[string]int{}
for _, r := range h.Resources {
if r.Module == "" {
continue
}
var this int
switch r.State {
case StateUnhealthy:
this = 3
case StateWaiting, StateStarting, StateHeld:
this = 2
case StateHealthy:
this = 0
default:
this = 1
}
if was, seen := rank[r.Module]; !seen || this > was {
rank[r.Module] = this
}
}
for module, r := range rank {
out[module] = [...]delivery.Health{delivery.HealthOK, delivery.HealthUnknown, delivery.HealthWaits, delivery.HealthFailed}[r]
}
return out
}
// nodeFacts is the node's own facts of a report, each as the JSON the report carries it in.
func nodeFacts(r Report) delivery.HaveNode {
n := delivery.HaveNode{V: delivery.Version}
carry := func(into *json.RawMessage, empty bool, v any) {
if empty {
return
}
if raw, err := json.Marshal(v); err == nil {
*into = raw
}
}
carry(&n.Windows, len(r.Windows) == 0, r.Windows)
carry(&n.Busy, len(r.Busy) == 0, r.Busy)
carry(&n.Strays, len(r.Strays) == 0, r.Strays)
carry(&n.Filters, len(r.Filters) == 0, r.Filters)
carry(&n.Profile, len(r.Profile) == 0, r.Profile)
return n
}
// noteApply takes what an apply's report says of the modules and the node.
func (q *Queue) noteApply(r Report) bool {
if r.Modules == nil {
return false
}
s := &q.haves
s.mu.Lock()
defer s.mu.Unlock()
s.modules = map[string]ModuleFact{}
for _, m := range r.Modules {
s.modules[m.Module] = m
}
if r.Health != nil {
s.health = moduleHealths(r.Health)
}
n := nodeFacts(r)
s.node = &n
return true
}
// noteHealth takes a look's health, and says whether any module's changed.
func (q *Queue) noteHealth(h Health) bool {
s := &q.haves
s.mu.Lock()
defer s.mu.Unlock()
next := moduleHealths(&h)
changed := false
for module := range s.modules {
want, ok := next[module]
if !ok {
want = delivery.HealthUnknown
}
have, known := s.health[module]
if !known {
have = delivery.HealthUnknown
}
if have != want {
changed = true
}
}
s.health = next
return changed
}
// sayHave is a line in the node-engine's log, if it has one.
func (q *Queue) sayHave(s string) {
if q.Say != nil {
q.Say(s)
}
}
// writeHaves writes this node's keys in the have bucket where they differ from what the bucket holds (novox/hq ADR 0429
// §4, issue 1132): each module's have, and the node's own facts. It reads the key first and writes nothing when the value
// is the same, otherwise writes over the revision it read, and creates the key when there is none. A write the bucket
// refuses is said and left: the next apply or look writes again, and the report was sent before this ran.
func (q *Queue) writeHaves(ctx context.Context, bus Bus) {
hb, ok := bus.(HaveBus)
if !ok || bus == nil {
return
}
s := &q.haves
s.mu.Lock()
defer s.mu.Unlock()
if s.modules == nil && s.node == nil {
return
}
node := q.Membership.Node
// The whole pass is bounded by the queue's timeout, and it stops at the first error: a node whose grant is not there
// yet would otherwise make the worker wait the bound again for every key (issue 1132).
write, cancel := context.WithTimeout(context.WithoutCancel(ctx), q.timeout())
defer cancel()
bucket, err := hb.HaveBind(write)
if err != nil {
q.sayHave(fmt.Sprintf("could not bind the have bucket, and wrote nothing there; the next apply or look writes again: %v", err))
return
}
modules := make([]string, 0, len(s.modules))
for module := range s.modules {
modules = append(modules, module)
}
sort.Strings(modules)
for _, module := range modules {
f := s.modules[module]
h := delivery.Have{V: delivery.Version, Build: f.Build, Phase: delivery.PhaseRunning,
Health: delivery.HealthUnknown, Detail: f.Detail}
if f.Failed {
h.Phase = delivery.PhaseFailed
}
if got, known := s.health[module]; known {
h.Health = got
}
key, err := delivery.HaveKey(node, module)
if err != nil {
q.sayHave(fmt.Sprintf("module %q has no key in the have bucket: %v", module, err))
continue
}
raw, err := delivery.EncodeHave(h)
if err != nil {
q.sayHave(fmt.Sprintf("cannot encode %s: %v", key, err))
continue
}
if !q.putHave(write, bucket, key, raw, func(current []byte) bool {
c, err := delivery.DecodeHave(current)
return err == nil && delivery.SameHave(c, h)
}) {
return
}
}
if s.node != nil {
key, err := delivery.HaveNodeKey(node)
if err == nil {
var raw []byte
if raw, err = delivery.EncodeHaveNode(*s.node); err == nil {
want := *s.node
q.putHave(write, bucket, key, raw, func(current []byte) bool {
c, err := delivery.DecodeHaveNode(current)
return err == nil && delivery.SameHaveNode(c, want)
})
}
}
if err != nil {
q.sayHave(fmt.Sprintf("cannot write this node's own facts: %v", err))
}
}
}
// putHave reads a key and writes it unless same says it already holds the value. False when the pass should stop: the
// bucket failed, the error is said once, and the rest wait for the next pass.
func (q *Queue) putHave(ctx context.Context, b HaveBucket, key string, value []byte, same func([]byte) bool) bool {
if ctx.Err() != nil {
q.sayHave(fmt.Sprintf("the time for writing the have bucket ran out before %s; the next apply or look writes it", key))
return false
}
current, revision, err := b.Get(ctx, key)
switch {
case errors.Is(err, ErrHaveAbsent):
revision = 0
case err != nil:
q.sayHave(fmt.Sprintf("could not read %s in the have bucket, and wrote nothing more there: %v", key, err))
return false
case same(current):
return true
}
if err := b.Set(ctx, key, value, revision); err != nil {
q.sayHave(fmt.Sprintf("could not write %s in the have bucket; the next apply or look writes it again: %v", key, err))
return false
}
return true
}
+104
View File
@@ -0,0 +1,104 @@
//go:build integration
package link
import (
"context"
"encoding/json"
"sync/atomic"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/nats-io/nats.go"
)
// 1132/AC8: against a real bus with the bucket node-engine_have, a node-engine that applies a declaration of two
// modules writes <node>.<a>, <node>.<b> and <node>.node, and a second apply with nothing changed adds no revision.
//
// MESH_TEST_NATS=nats://127.0.0.1:4222 go test -tags integration ./internal/link/ -run TestNatsHavesAreWrittenOnceAndOnlyOnChange
//
// The bucket is made here by the test: in the mesh the controller makes it, and the node-engine only binds to it.
func TestNatsHavesAreWrittenOnceAndOnlyOnChange(t *testing.T) {
conn, js := aBus(t)
const node = "haveok"
kv, err := js.KeyValue(delivery.HaveBucket)
if err != nil {
if kv, err = js.CreateKeyValue(&nats.KeyValueConfig{Bucket: delivery.HaveBucket, History: delivery.HaveHistory}); err != nil {
t.Fatal(err)
}
}
for _, name := range []string{"a", "b", "node"} {
_ = kv.Purge(node + "." + name)
}
m, key := aMember(t)
m.Node = node
var failB atomic.Bool
var applied atomic.Int32
failB.Store(true)
apply := func(_ context.Context, raw, _ []byte) Report {
applied.Add(1)
var d struct{ Order }
_ = json.Unmarshal(raw, &d)
return Report{Declared: digest(raw), Order: d.Order, Applied: []string{"a.x", "b.x"},
Modules: []ModuleFact{{Module: "a", Build: "b-1"},
{Module: "b", Build: "b-2", Failed: failB.Load(), Detail: "b.x failed: no space"}},
Health: healthOf(map[string]string{"a": StateHealthy, "b": StateHealthy}),
Windows: []Window{{Step: "backup", Holds: []string{"db"}}}}
}
q := aQueue(m, apply, &keptInMemory{})
q.attach(context.Background(), OverNATS{Conn: conn, JS: js})
ctx, stop := context.WithCancel(context.Background())
worker := runWorker(ctx, q)
defer func() { stop(); <-worker }()
revisions := func() map[string]uint64 {
out := map[string]uint64{}
for _, name := range []string{"a", "b", "node"} {
if e, err := kv.Get(node + "." + name); err == nil {
out[name] = e.Revision()
}
}
return out
}
arrives := func(what string, id string, seq int64) {
want := applied.Load() + 1
q.Deliver(ordered(t, key, id, 7, seq))
eventually(t, what, func() bool { return applied.Load() >= want })
time.Sleep(300 * time.Millisecond)
}
arrives("the first apply", "one", 1)
eventually(t, "the three keys were never written", func() bool { return len(revisions()) == 3 })
first := revisions()
a, _ := kv.Get(node + ".a")
haveA, err := delivery.DecodeHave(a.Value())
if err != nil || haveA.Phase != delivery.PhaseRunning || haveA.Build != "b-1" || haveA.Health != delivery.HealthOK {
t.Fatalf("a holds %+v (%v)", haveA, err)
}
b, _ := kv.Get(node + ".b")
haveB, err := delivery.DecodeHave(b.Value())
if err != nil || haveB.Phase != delivery.PhaseFailed || haveB.Detail == "" {
t.Fatalf("b holds %+v (%v)", haveB, err)
}
n, _ := kv.Get(node + ".node")
if hn, err := delivery.DecodeHaveNode(n.Value()); err != nil || len(hn.Windows) == 0 {
t.Fatalf("the node key holds %+v (%v)", hn, err)
}
arrives("the second apply", "two", 2)
time.Sleep(200 * time.Millisecond)
for name, rev := range revisions() {
if rev != first[name] {
t.Errorf("%s moved from revision %d to %d with nothing changed", name, first[name], rev)
}
}
failB.Store(false)
arrives("the third apply", "three", 3)
eventually(t, "b was never updated", func() bool { return revisions()["b"] > first["b"] })
if now := revisions(); now["a"] != first["a"] || now["node"] != first["node"] {
t.Errorf("a key that did not change moved: %v from %v", now, first)
}
}
+341
View File
@@ -0,0 +1,341 @@
package link
import (
"context"
"encoding/json"
"errors"
"strings"
"sync"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// What the node-engine writes beside its report: a module's have, and the node's own facts (novox/hq ADR 0429 §4,
// issue 1132).
// haveLink is a link with a bucket behind it that keeps each key's revision as the bus does.
type haveLink struct {
*quietLink
hmu sync.Mutex
kv map[string]haveEntry
sets []string // keys written, in order
events []string // "report" and "have <key>", in the order they happened
failing bool
binds int
setTries int
bindFails, bindStuck bool
}
type haveEntry struct {
value []byte
rev uint64
}
func newHaveLink() *haveLink { return &haveLink{quietLink: newQuietLink(), kv: map[string]haveEntry{}} }
func (l *haveLink) Report(ctx context.Context, node string, body []byte) error {
err := l.quietLink.Report(ctx, node, body)
if err == nil {
l.hmu.Lock()
l.events = append(l.events, "report")
l.hmu.Unlock()
}
return err
}
func (l *haveLink) Health(context.Context, string, []byte) error { return nil }
func (l *haveLink) HaveBind(ctx context.Context) (HaveBucket, error) {
l.hmu.Lock()
l.binds++
stuck, refuse := l.bindStuck, l.bindFails
l.hmu.Unlock()
if stuck {
<-ctx.Done()
return nil, ctx.Err()
}
if refuse {
return nil, errors.New("no grant for the have bucket")
}
return l, nil
}
func (l *haveLink) Get(_ context.Context, key string) ([]byte, uint64, error) {
l.hmu.Lock()
defer l.hmu.Unlock()
e, ok := l.kv[key]
if !ok {
return nil, 0, ErrHaveAbsent
}
return e.value, e.rev, nil
}
func (l *haveLink) Set(_ context.Context, key string, value []byte, revision uint64) error {
l.hmu.Lock()
defer l.hmu.Unlock()
l.setTries++
if l.failing {
return errors.New("the bucket refused")
}
e, ok := l.kv[key]
if (revision == 0) == ok || (ok && e.rev != revision) {
return errors.New("wrong last sequence")
}
l.kv[key] = haveEntry{value: value, rev: e.rev + 1}
l.sets = append(l.sets, key)
l.events = append(l.events, "have "+key)
return nil
}
func (l *haveLink) have(t *testing.T, key string) delivery.Have {
t.Helper()
l.hmu.Lock()
defer l.hmu.Unlock()
h, err := delivery.DecodeHave(l.kv[key].value)
if err != nil {
t.Fatalf("%s holds no have: %v", key, err)
}
return h
}
func (l *haveLink) setCount() int {
l.hmu.Lock()
defer l.hmu.Unlock()
return len(l.sets)
}
func (l *haveLink) put(t *testing.T, key string, h delivery.Have) {
t.Helper()
raw, err := delivery.EncodeHave(h)
if err != nil {
t.Fatal(err)
}
l.hmu.Lock()
defer l.hmu.Unlock()
l.kv[key] = haveEntry{value: raw, rev: l.kv[key].rev + 1}
}
func healthOf(states map[string]string) *Health {
h := &Health{Contract: LivenessContract, At: time.Now().UTC()}
for module, state := range states {
h.Resources = append(h.Resources, ResourceHealth{Module: module, Resource: module + ".server", State: state})
}
return h
}
func appliedModules() Report {
return Report{Declared: "d1", Applied: []string{"a.x", "b.x"},
Modules: []ModuleFact{{Module: "a", Build: "b-1"}, {Module: "b", Build: "b-2", Failed: true, Detail: "b.x failed: no space"}},
Health: healthOf(map[string]string{"a": StateHealthy, "b": StateUnhealthy})}
}
func aHaveQueue(l *haveLink) (*Queue, *[]string) {
var said []string
var mu sync.Mutex
q := &Queue{Membership: Membership{Node: "n1"}, Timeout: time.Second, Say: func(s string) {
mu.Lock()
said = append(said, s)
mu.Unlock()
}}
q.attach(context.Background(), l)
return q, &said
}
// 1132/AC3, AC5: a module that applied is running with its build and a module that failed is failed with the detail;
// the key does not exist yet, so it is created.
func TestAnAppliedModuleIsWrittenRunningAndAFailedOneFailed(t *testing.T) {
l := newHaveLink()
q, _ := aHaveQueue(l)
q.tell(context.Background(), appliedModules(), applied)
a, b := l.have(t, "n1.a"), l.have(t, "n1.b")
if a.Phase != delivery.PhaseRunning || a.Build != "b-1" || a.Health != delivery.HealthOK {
t.Errorf("a: %+v", a)
}
if b.Phase != delivery.PhaseFailed || b.Build != "b-2" || !strings.Contains(b.Detail, "no space") ||
b.Health != delivery.HealthFailed {
t.Errorf("b: %+v", b)
}
if l.kv["n1.a"].rev != 1 {
t.Errorf("a was created at revision %d", l.kv["n1.a"].rev)
}
}
// 1132/AC5: a have equal to the bucket's value is not written; a different one is written over the last revision.
func TestAHaveIsWrittenOnlyWhenItDiffers(t *testing.T) {
l := newHaveLink()
l.put(t, "n1.a", delivery.Have{V: delivery.Version, Build: "b-1", Phase: delivery.PhaseRunning, Health: delivery.HealthOK})
l.put(t, "n1.b", delivery.Have{V: delivery.Version, Build: "b-0", Phase: delivery.PhaseRunning, Health: delivery.HealthOK})
q, _ := aHaveQueue(l)
q.tell(context.Background(), appliedModules(), applied)
if l.kv["n1.a"].rev != 1 {
t.Errorf("a equal to the bucket was written again, to revision %d", l.kv["n1.a"].rev)
}
if l.kv["n1.b"].rev != 2 || l.have(t, "n1.b").Phase != delivery.PhaseFailed {
t.Errorf("b was not updated over its last revision: %+v at %d", l.have(t, "n1.b"), l.kv["n1.b"].rev)
}
before := l.setCount()
q.tell(context.Background(), appliedModules(), applied)
if l.setCount() != before {
t.Errorf("an apply that changed nothing wrote %d keys", l.setCount()-before)
}
}
// 1132/AC4: a look's health reaches the module's have: ok, failed, waits or unknown, and only a change is written.
func TestAModulesHealthFollowsTheLook(t *testing.T) {
l := newHaveLink()
q, _ := aHaveQueue(l)
rep := Report{Declared: "d1", Modules: []ModuleFact{{Module: "a", Build: "b"}, {Module: "b", Build: "b"},
{Module: "c", Build: "b"}, {Module: "d", Build: "b"}}}
q.tell(context.Background(), rep, applied)
for _, m := range []string{"a", "b", "c", "d"} {
if h := l.have(t, "n1."+m).Health; h != delivery.HealthUnknown {
t.Errorf("%s with no look is %s, want unknown", m, h)
}
}
h := healthOf(map[string]string{"a": StateHealthy, "b": StateUnhealthy, "c": StateStarting})
h.Resources = append(h.Resources, ResourceHealth{Module: "a", Resource: "a.two", State: StateWaiting},
ResourceHealth{Module: "b", Resource: "b.two", State: StateHealthy}, ResourceHealth{Module: "d", Resource: "d.x", State: StateUnknown})
if !q.SayHealth(context.Background(), *h) {
t.Fatal("not said")
}
for m, want := range map[string]delivery.Health{"a": delivery.HealthWaits, "b": delivery.HealthFailed,
"c": delivery.HealthWaits, "d": delivery.HealthUnknown} {
if got := l.have(t, "n1."+m).Health; got != want {
t.Errorf("%s is %s, want %s", m, got, want)
}
}
before := l.setCount()
h2 := *h
q.SayHealth(context.Background(), h2)
if l.setCount() != before {
t.Errorf("the same look wrote %d keys", l.setCount()-before)
}
h2.Resources = []ResourceHealth{{Module: "a", Resource: "a.server", State: StateHealthy}}
q.SayHealth(context.Background(), h2)
if got := l.have(t, "n1.a").Health; got != delivery.HealthOK {
t.Errorf("a after a healthy look is %s", got)
}
}
// 1132/AC6: the report is sent first and carries no new key; the node's key holds its windows, busy, strays, filters
// and profile.
func TestTheReportIsSentFirstAndTheNodeKeyCarriesTheNodesFacts(t *testing.T) {
l := newHaveLink()
q, _ := aHaveQueue(l)
rep := appliedModules()
rep.Windows = []Window{{Step: "backup", Holds: []string{"db"}}}
rep.Busy = []Busy{{ID: "x", Container: "x", State: "waiting"}}
rep.Strays = []Stray{{Kind: "container", Name: "old"}}
rep.Filters = []Filter{{Where: "inet mesh", Owner: "mesh"}}
rep.Profile = map[string]any{"cores": 8}
q.tell(context.Background(), rep, applied)
if len(l.events) < 2 || l.events[0] != "report" {
t.Fatalf("the report was not first: %v", l.events)
}
raw, _ := json.Marshal(l.said()[0])
if strings.Contains(string(raw), "Modules") || strings.Contains(string(raw), "\"modules\"") {
t.Errorf("the report carries the module facts: %s", raw)
}
got, err := delivery.DecodeHaveNode(l.kv["n1.node"].value)
if err != nil {
t.Fatal(err)
}
enc := func(v any) json.RawMessage { b, _ := json.Marshal(v); return b }
want := delivery.HaveNode{V: delivery.Version, Windows: enc(rep.Windows), Busy: enc(rep.Busy), Strays: enc(rep.Strays),
Filters: enc(rep.Filters), Profile: enc(rep.Profile)}
if !delivery.SameHaveNode(got, want) {
t.Errorf("the node key holds %+v, want %+v", got, want)
}
}
// 1132/AC7: a write the bucket refuses is said in the log; the report went out, and the next apply writes again.
func TestAFailedHaveWriteIsSaidAndNeverBlocksTheReport(t *testing.T) {
l := newHaveLink()
l.failing = true
q, said := aHaveQueue(l)
if !q.tell(context.Background(), appliedModules(), applied) {
t.Fatal("the report did not go out")
}
if len(l.said()) != 1 {
t.Fatalf("reports sent: %d", len(l.said()))
}
if !strings.Contains(strings.Join(*said, "\n"), "n1.a") || !strings.Contains(strings.Join(*said, "\n"), "refused") {
t.Errorf("the failure is not in the log: %v", *said)
}
l.hmu.Lock()
l.failing = false
l.hmu.Unlock()
q.tell(context.Background(), appliedModules(), applied)
if l.have(t, "n1.a").Phase != delivery.PhaseRunning {
t.Error("the next apply did not write again")
}
}
// A link without a bucket, and a report with no module facts (a refusal), write nothing and say nothing wrong.
func TestNoBucketAndNoFactsWriteNothing(t *testing.T) {
plain := newQuietLink()
q := &Queue{Membership: Membership{Node: "n1"}, Timeout: time.Second}
q.attach(context.Background(), plain)
if !q.tell(context.Background(), appliedModules(), applied) {
t.Fatal("the report did not go out")
}
l := newHaveLink()
q2, _ := aHaveQueue(l)
q2.tell(context.Background(), Report{Refused: "no"}, applied)
if l.setCount() != 0 {
t.Errorf("a refusal wrote %d keys", l.setCount())
}
}
// 1132/AC5: a reconcile whose report is not news still writes the have keys that differ, so a node-engine that
// restarted writes them again; and writes nothing when they are the same.
func TestASilentReconcileStillWritesTheHaves(t *testing.T) {
l := newHaveLink()
q, _ := aHaveQueue(l)
q.Reconcile = func(context.Context) (Report, bool) { return appliedModules(), true }
q.reconcile(context.Background(), false)
if len(l.said()) != 0 {
t.Fatalf("a reconcile that is not news said %d reports", len(l.said()))
}
if l.have(t, "n1.a").Phase != delivery.PhaseRunning {
t.Error("the reconcile wrote no have")
}
before := l.setCount()
q.reconcile(context.Background(), false)
if l.setCount() != before {
t.Errorf("an unchanged reconcile wrote %d keys", l.setCount()-before)
}
}
// 1132/AC7: a bind that fails, or never answers, writes nothing and returns within the queue's timeout; a write that
// fails ends the pass, and is said once.
func TestAFailedBindOrWriteEndsThePassAndReturnsInTime(t *testing.T) {
for name, set := range map[string]func(*haveLink){
"a bind refused": func(l *haveLink) { l.bindFails = true },
"a bind that never answers": func(l *haveLink) { l.bindStuck = true },
} {
l := newHaveLink()
set(l)
q, said := aHaveQueue(l)
q.Timeout = 100 * time.Millisecond
start := time.Now()
if !q.tell(context.Background(), appliedModules(), applied) {
t.Fatalf("%s: the report did not go out", name)
}
if took := time.Since(start); took > time.Second {
t.Errorf("%s: the pass took %s", name, took)
}
if l.binds != 1 || l.setTries != 0 || len(*said) != 1 {
t.Errorf("%s: binds %d, writes tried %d, said %v; want one bind, no write, one line", name, l.binds, l.setTries, *said)
}
}
l := newHaveLink()
l.failing = true
q, said := aHaveQueue(l)
q.tell(context.Background(), appliedModules(), applied)
if l.binds != 1 || l.setTries != 1 || len(*said) != 1 {
t.Errorf("a failed write: binds %d, writes tried %d, said %v; want one of each and one line", l.binds, l.setTries, *said)
}
}
+28 -4
View File
@@ -68,6 +68,8 @@ type natsLink struct {
judging *nats.Subscription
// acting hears the controller asking for the take act on a library (novox/hq ADR 0328 §7).
acting *nats.Subscription
// moving hears the controller asking to move a value sealed to this machine into the vault (issue 676).
moving *nats.Subscription
node string
// signer is the mesh's key, which a hand-over ask is verified with as a declaration is (issue 356).
signer ed25519.PublicKey
@@ -279,6 +281,24 @@ func (l *natsLink) HearAccessActs(handle AccessActFunc) {
}
}
// HearMoveSecrets listens for the controller asking this node to move one value sealed to it into the vault (design
// 57's V3, issue 676), on this machine's own subject, verified as a hand-over is (AnswerMoveSecret). A bus whose
// user list is older than the grant refuses the subscription; the controller then hears no answer.
func (l *natsLink) HearMoveSecrets(handle MoveSecretFunc) {
if handle == nil {
return
}
check := HandOverCheck{Signer: l.signer, Node: l.node}
moving, err := l.conn.Subscribe(AskMoveSecretSubject(l.node), func(msg *nats.Msg) {
if body, answer := MoveSecretReply(check, handle, msg.Reply, msg.Data); answer {
_ = msg.Respond(body)
}
})
if err == nil {
l.moving = moving
}
}
func (l *natsLink) Declarations() <-chan Declaration { return l.arrived }
func (l *natsLink) AskedToReport() <-chan struct{} { return l.asked }
func (l *natsLink) Lost() <-chan error { return l.lost }
@@ -299,6 +319,9 @@ func (l *natsLink) Close() {
if l.acting != nil {
_ = l.acting.Unsubscribe()
}
if l.moving != nil {
_ = l.moving.Unsubscribe()
}
if l.judging != nil {
_ = l.judging.Unsubscribe()
}
@@ -311,14 +334,14 @@ func (l *natsLink) Report(ctx context.Context, node string, body []byte) error {
return OverNATS{Conn: l.conn, JS: l.js}.Report(ctx, node, body)
}
func (l *natsLink) Alive(ctx context.Context, node string, body []byte) error {
return OverNATS{Conn: l.conn, JS: l.js}.Alive(ctx, node, body)
}
func (l *natsLink) Health(ctx context.Context, node string, body []byte) error {
return OverNATS{Conn: l.conn, JS: l.js}.Health(ctx, node, body)
}
func (l *natsLink) HaveBind(ctx context.Context) (HaveBucket, error) {
return OverNATS{Conn: l.conn, JS: l.js}.HaveBind(ctx)
}
// Ask asks a module's tool for a declared tool check (novox/hq ADR 0240, to-be 48 §3). The link the
// node-engine holds is this one, not OverNATS: without it here the queue's ToolBus assertion failed on
// every live link and each tool check read "no link to the bus is open" while the link was up.
@@ -347,6 +370,7 @@ func (l *natsLink) Ask(ctx context.Context, subject string, body []byte) ([]byte
var (
_ Bus = (*natsLink)(nil)
_ HealthBus = (*natsLink)(nil)
_ HaveBus = (*natsLink)(nil)
_ ToolBus = (*natsLink)(nil)
_ Asker = (*natsLink)(nil)
_ Asked = (*natsLink)(nil)
+3 -8
View File
@@ -203,9 +203,8 @@ func TestNatsANodeThatWasAwayGetsOnlyTheNewest(t *testing.T) {
}
}
// A report goes through the stream and a heartbeat does not: the one that must survive the
// controller's store restarting is kept, and the one that must not is not.
func TestNatsAReportIsKeptAndAHeartbeatIsNot(t *testing.T) {
// A report goes through the stream: it must survive the controller's store restarting.
func TestNatsAReportIsKept(t *testing.T) {
conn, js := aBus(t)
bus := OverNATS{Conn: conn, JS: js}
ctx := context.Background()
@@ -214,17 +213,13 @@ func TestNatsAReportIsKeptAndAHeartbeatIsNot(t *testing.T) {
if err := bus.Report(ctx, "anchor", body); err != nil {
t.Fatal(err)
}
beat, _ := json.Marshal(Alive{Node: "anchor"})
if err := bus.Alive(ctx, "anchor", beat); err != nil {
t.Fatal(err)
}
info, err := js.StreamInfo("CONTROL")
if err != nil {
t.Fatal(err)
}
if info.State.Msgs != 1 {
t.Fatalf("%d messages were kept; a report must be and a heartbeat must not", info.State.Msgs)
t.Fatalf("%d messages were kept; a report must be, once", info.State.Msgs)
}
}
+98 -18
View File
@@ -16,26 +16,8 @@ import (
// queue, so it can say these things and nothing else.
const (
KeyReport = "report"
KeyAlive = "alive"
)
// Alive is a node saying nothing except that it is there.
//
// novox/hq 09-the-node-lifecycle: *how long it has been disconnected is a fact the mesh must
// hold, and nothing holds it today. Without it, a node running last month's assignments looks
// exactly like one that is current.*
//
// Separate from a report because the two happen at completely different rates: a node is alive
// constantly and applies something rarely, and reading one as the other would make a quiet node
// look like a stale one.
type Alive struct {
Node string `json:"node"`
// IntervalSeconds is how often this node says it is there (novox/hq to-be 45 §3, S1): the
// controller's watchdog is bound to three of them, so the bound moves with the interval rather
// than with a number the controller guessed. A controller older than that ignores it.
IntervalSeconds int `json:"interval_seconds"`
}
// Signed is a declaration and the signature over it.
//
// novox/hq ADR 0004: the transport is verified once at connect, and **each declaration is
@@ -134,6 +116,11 @@ type Report struct {
// know, whole. A node-engine rolled back says so no longer.
ReadsGeneration bool `json:"reads_generation,omitempty"`
// ReadsModules says this node-engine reads a resource's `module` and a declaration's `builds` (novox/hq ADR 0429
// §4, issue 1132), on every report it makes, beside ReadsGeneration: the controller sends them only to a machine
// that said so, for the same reason.
ReadsModules bool `json:"reads_modules,omitempty"`
// OlderGeneration is set on a report refusing a declaration composed from an older assignment
// generation than this machine applied (novox/hq issue 234). The refused declaration is the report's
// own Declared and Order, so the controller can name who sent it from its record of sends.
@@ -174,6 +161,19 @@ type Report struct {
// broken.
Windows []Window `json:"windows,omitempty"`
// Busy is every container the apply that reports left running because it said busy, and every one it
// replaced after it had waited, with how (novox/hq ADR 0335): `waiting`, `replaced-when-idle` or
// `replaced-at-bound`, since when, until when at the latest, and the last output of its busy command.
// A walk waiting on this machine for a module whose container is listed `waiting` waits for the report
// that no longer lists it so: its first-machine gate starts then (ADR 0335 §4). Absent from an engine
// older than BusyContract, which never waits.
Busy []Busy `json:"busy,omitempty"`
// MovedOut is every path a move-out directory moved into its kept folder, and Left every socket, pipe or
// device it held and did not move (issue 1024, novox/hq ADR 0422).
MovedOut []MovedOut `json:"moved-out,omitempty"`
Left []Left `json:"left,omitempty"`
// Strays is what runs on the machine that the mesh neither wrote nor holds (novox/hq ADR
// 0163): containers nobody declared and nobody holds, the ones a cutover leaves behind.
Strays []Stray `json:"strays,omitempty"`
@@ -183,6 +183,11 @@ type Report struct {
// a network manager switched, reaches the mesh at the next push rather than never.
Profile map[string]any `json:"profile,omitempty"`
// Modules is how each module of the declaration this report is about came out (novox/hq ADR 0429 §4, issue 1132). **Not
// on the wire**: the report is sent as it always was, and the node-engine writes these beside it as the module's
// have. Nil on a report that is not an apply's.
Modules []ModuleFact `json:"-"`
// Host is the version of the host that produced this report (novox/hq ADR 0141).
//
// Without it nothing can say a machine is behind, so "every machine current with its source"
@@ -280,6 +285,46 @@ const LibraryContract = 6
// before it, its presence is what tells the controller this host may be sent `runner` on an identity.
const OwnGroupContract = 7
// BusyContract is the statement of a host that also reads a container's `busy` (novox/hq ADR 0335): it asks a
// container that declares it before replacing it, leaves it running while it says busy, within a bound of
// 15 minutes, and says the wait in its report's `busy`. Like the fields before it, its presence is what tells
// the controller this host may be sent `busy` on a container; an older host refuses the whole declaration
// for it.
const BusyContract = 8
// OwnershipContract is the statement of a host that also reads a file's `owner-at-runtime` and a declaration's
// `filled_files` (novox/hq ADR 0349 §5): it writes an owned file back on drift, a seed once and a managed file's
// keys or block only, exactly as their old forms, and refuses a filled path that is not a declared file saying
// its owner. Like the fields before it, its presence is what tells the controller this host may be sent those
// fields; an older host refuses the whole declaration for them.
const OwnershipContract = 9
// ProvisionerContract is the statement of a host that also reads a declaration's `provisioner_account` (novox/hq ADR
// 0349 §4) and serves the local write socket: it accepts a write of a filled file from that account alone. Its
// presence is what tells the controller this host may be sent the field; an older host refuses the whole
// declaration for it.
const ProvisionerContract = 10
// HandedContract is the statement of a host that writes a filled file only on the provisioner client's word (novox/hq
// ADR 0349 §5, design 57 P7): an apply neither writes nor records a file the declaration names as filled, and removes
// one as an orphan only when the write verb placed it. Its presence is what tells the controller it may hand a moved
// module's files to the provisioner here (mesh-controller internal/link/protocol.go's HandedContract): to an older
// host a filled file would have two writers.
const HandedContract = 11
// VaultBoxContract is the statement of a host that also reads a file's `vault-sealed` (novox/hq ADR 0349 §16, issue
// 755): a core module's own credential the vault sealed to this machine in the mesh's sealed box (ADR 0206), which
// the controller carries and cannot open, opened here with this machine's sealing key and written as a sealed file
// is. Its presence is what tells the controller it may send the vault's box here (mesh-controller internal/link
// protocol.go's VaultBoxContract); an older host refuses the whole declaration for the field, and is sent the
// controller's sealed value as before.
const VaultBoxContract = 12
// NodeKeyContract is the statement of a host that states its own sealing key to the vault's node-key, in its own name,
// at its start and whenever the key changes (novox/hq ADR 0372, issue 786). The same number as mesh-controller
// internal/link/protocol.go's NodeKeyContract.
const NodeKeyContract = 13
// Health is one statement of every long-running resource's state on this machine (to-be 48 §4). Said in
// every report, as an event on each change, and again every minute while anything is not healthy — so a
// lost statement is not a lost fault.
@@ -344,6 +389,8 @@ type UnitsHealth struct {
Failed []FailedUnit `json:"failed"`
// Unread names a manager that could not be read, with why.
Unread []string `json:"unread,omitempty"`
// Away names an account's own manager not running, so not read: nothing in it is failed (issue 764).
Away []string `json:"away,omitempty"`
}
// FailedUnit is one failed unit no module places.
@@ -731,6 +778,30 @@ type Window struct {
Until time.Time `json:"until"`
}
// Busy is one container a busy command held (novox/hq ADR 0335), as a report says it. ID is the resource's
// id, `<module>.<resource>`.
type Busy struct {
ID string `json:"id"`
Container string `json:"container"`
State string `json:"state"`
Since time.Time `json:"since"`
Until time.Time `json:"until"`
Said string `json:"said,omitempty"`
}
// MovedOut is one path a move-out directory moved (issue 1024).
type MovedOut struct {
Resource string `json:"resource"`
From string `json:"from"`
To string `json:"to"`
}
// Left is one socket, pipe or device a move-out directory did not move (issue 1024).
type Left struct {
Resource string `json:"resource"`
Path string `json:"path"`
}
// A Stray is a container the mesh neither wrote nor holds (ADR 0163).
type Stray struct {
Kind string `json:"kind"`
@@ -772,3 +843,12 @@ type GenerationRefusal struct {
Generation int64 `json:"generation"`
Applied int64 `json:"applied"`
}
// ModuleFact is how one module's part of an apply came out: the build the node was sent for it, and whether every
// resource of it applied. Detail names the failing resource and its error (issue 1132).
type ModuleFact struct {
Module string
Build string
Failed bool
Detail string
}
+209
View File
@@ -0,0 +1,209 @@
package link
import (
"context"
"encoding/json"
"errors"
"fmt"
"strings"
"time"
)
// The move of a secret into the vault, asked over the bus (design 57's V3, issue 676; the operator's decision of
// 2026-10-14 on that issue).
//
// **Only the machine a value was sealed to can open it**, so the controller, which cannot, asks each machine's
// engine to move what was sealed to it: one row of its store at a time, on AskMoveSecretSubject, signed as a
// hand-over is and held to the same checks (believed), under a signing context of its own, naming the row — a
// module's own secret on this machine, or a pair credential with this machine at one end — its origin, the
// controller's generation, when it was made, and the value as the store holds it, sealed to this machine. The
// engine opens it with its sealing key, hands it to the vault's `move-in` in the vault's two steps, and answers
// what the vault kept only once the vault's read-back opens to the same value. Nothing is rotated; the engine
// writes nothing on this machine. The controller holds the same shapes in its own link code, with a test on each
// side holding the field names.
// AskMoveSecretSubject is where the controller asks this node's engine to move one value sealed to it into the vault.
func AskMoveSecretSubject(node string) string { return "mesh.node." + node + ".ask.move-secret" }
// MoveSecretContext is prefixed to a move ask's bytes before signing: never another ask's, nor a declaration's.
const MoveSecretContext = "novox-mesh move-secret v1\n"
// MoveSecretAsk is a move's own fields, beside a hand-over's (node, by, expires, nonce; no path): the row — module
// and name for a module's own secret on the ask's node; provider, provision, consumer, consumer module and local
// name for a pair credential — where its value came from, the controller's generation, when it was made, the value
// sealed to this machine as the store holds it, and the sealing key the store says it was sealed to.
type MoveSecretAsk struct {
Module string `json:"module,omitempty"`
Name string `json:"name,omitempty"`
Provider string `json:"provider,omitempty"`
Provision string `json:"provision,omitempty"`
Consumer string `json:"consumer,omitempty"`
ConsumerModule string `json:"consumer_module,omitempty"`
Local string `json:"local,omitempty"`
Origin string `json:"origin"`
Generation string `json:"generation,omitempty"`
MadeAt time.Time `json:"made_at"`
Sealed string `json:"sealed"`
SealedTo string `json:"sealed_to,omitempty"`
}
// MoveSecretAnswer is the engine's answer: what the vault kept, or why nothing was moved — never both.
type MoveSecretAnswer struct {
Said string `json:"said,omitempty"`
Refused string `json:"refused,omitempty"`
Secret string `json:"secret,omitempty"`
Generation int `json:"generation,omitempty"`
Unchanged bool `json:"unchanged,omitempty"`
}
// MoveSecret is one move asked and believed, as the handler is given it: the row, this machine, and who asked.
type MoveSecret struct {
MoveSecretAsk
Node, By string
}
// Own says the row is a module's own secret rather than a pair credential.
func (m MoveSecret) Own() bool { return m.Module != "" || m.Name != "" }
// MovedSecret is what the vault kept: its name, its generation, and whether it held the value already.
type MovedSecret struct {
Secret string
Generation int
Unchanged bool
}
// MoveSecretFunc moves one value into the vault: what it kept, or why not.
type MoveSecretFunc func(MoveSecret) (MovedSecret, error)
// MaxMoveSecretAsk bounds a move's request: a row's names, a time, a value sealed and a signature.
const MaxMoveSecretAsk = 64 << 10
// movedOrigins are the origins the controller's store holds.
var movedOrigins = map[string]bool{"made": true, "accepted": true, "issued": true}
func refusedMove(why string) MoveSecretAnswer {
return MoveSecretAnswer{Refused: why + "; nothing was moved"}
}
// oneName is a short name of one line.
func oneName(s string) bool {
return s != "" && len(s) <= MaxActField && !strings.ContainsAny(s, "\r\n\x00/ ")
}
// AnswerMoveSecret is the engine's judgement of one move ask's bytes: believed as a hand-over is, then the row —
// one shape, this machine's, a value to open and an origin — and only then the handler.
func AnswerMoveSecret(check HandOverCheck, handle MoveSecretFunc, body []byte) MoveSecretAnswer {
if handle == nil {
return refusedMove("this engine moves no secret into the vault")
}
ask, signed, refused := believedBytesWithin(check, MoveSecretContext, false, body, MaxMoveSecretAsk)
if refused != "" {
return refusedMove(refused)
}
var move MoveSecretAsk
if err := json.Unmarshal(signed, &move); err != nil {
return refusedMove("the signed ask is not a move's ask: " + err.Error())
}
own := move.Module != "" || move.Name != ""
pair := move.Provider != "" || move.Provision != "" || move.Consumer != "" || move.ConsumerModule != "" ||
move.Local != ""
switch {
case own == pair:
return refusedMove("the ask names neither one module's own secret nor one pair credential")
case own && (!oneName(move.Module) || !oneName(move.Name)):
return refusedMove("the ask's module and name are not short names")
case pair && (!oneName(move.Provider) || !oneName(move.Provision) || !oneName(move.Consumer) ||
!oneName(move.ConsumerModule) || (move.Local != "" && !oneName(move.Local))):
return refusedMove("the ask's provider, provision, consumer, consuming module and local name are not short names")
case pair && move.Provider != ask.Node && move.Consumer != ask.Node:
return refusedMove(fmt.Sprintf("the pair is between %s and %s, and this is %s", move.Provider, move.Consumer,
ask.Node))
case !movedOrigins[move.Origin]:
return refusedMove(fmt.Sprintf("the origin %q is not made, accepted or issued", move.Origin))
case strings.TrimSpace(move.Sealed) == "":
return refusedMove("the ask carries no value sealed to this machine")
case len(move.Generation) > MaxActField || strings.ContainsAny(move.Generation, "\r\n\x00"):
return refusedMove("the ask's generation is not one short line")
}
moved, err := handle(MoveSecret{MoveSecretAsk: move, Node: ask.Node, By: ask.By})
if err != nil {
return refusedMove(err.Error())
}
said := fmt.Sprintf("%s is in the vault at generation %d, read back the same", moved.Secret, moved.Generation)
if moved.Unchanged {
said = fmt.Sprintf("the vault held %s already, the same value (generation %d)", moved.Secret, moved.Generation)
}
return MoveSecretAnswer{Said: said, Secret: moved.Secret, Generation: moved.Generation, Unchanged: moved.Unchanged}
}
// MoveSecretReply is what a move request is answered with, and whether to answer, as HandOverReply.
func MoveSecretReply(check HandOverCheck, handle MoveSecretFunc, reply string, body []byte) ([]byte, bool) {
if reply == "" {
return nil, false
}
answer, err := json.Marshal(AnswerMoveSecret(check, handle, body))
if err != nil {
return nil, false
}
return answer, true
}
// MoveSecrets is a link that hears the controller asking this node to move a value into the vault. Optional.
type MoveSecrets interface {
HearMoveSecrets(MoveSecretFunc)
}
// ---- asking the vault ------------------------------------------------------------------------------------------
// VaultSeat is the vault's seat; its verbs are asked on `mesh.seat.<seat>.call.<verb>.<caller>`, the subject that
// names its caller (novox/hq issue 365), which the bus lets only that user publish.
const VaultSeat = "mesh-vault"
// NodeEngineUser is the bus user of a machine's node-engine, as the controller's broker names it.
func NodeEngineUser(node string) string { return "node." + node }
// CalledVaultSubject is where this node's engine asks one of the vault's verbs in its own name: each dot of its
// user's name written `~`, as the controller's broker writes a caller's token.
func CalledVaultSubject(verb, node string) string {
return "mesh.seat." + VaultSeat + ".call." + verb + "." + strings.ReplaceAll(NodeEngineUser(node), ".", "~")
}
// VaultWithin is how long the engine gives the vault to answer one step.
const VaultWithin = 15 * time.Second
// AskVault asks one of the vault's verbs in this node's engine's own name over the link open now, and answers its
// result, or its refusal as an error.
func (q *Queue) AskVault(ctx context.Context, verb string, args any) (json.RawMessage, error) {
q.init()
q.mu.Lock()
bus := q.bus
q.mu.Unlock()
if bus == nil {
return nil, fmt.Errorf("%w: this machine has no link to the bus open now", ErrNotAsked)
}
asker, ok := bus.(ToolBus)
if !ok {
return nil, fmt.Errorf("%w: the link to the bus open now cannot ask the vault", ErrNotAsked)
}
body, err := json.Marshal(args)
if err != nil {
return nil, err
}
asking, cancel := context.WithTimeout(ctx, VaultWithin)
defer cancel()
reply, err := asker.Ask(asking, CalledVaultSubject(verb, q.Membership.Node), body)
if err != nil {
return nil, fmt.Errorf("the vault's %s could not be asked: %w", verb, err)
}
var env struct {
Result json.RawMessage `json:"result"`
Error string `json:"error"`
}
if err := json.Unmarshal(reply, &env); err != nil {
return nil, fmt.Errorf("the vault answered %s with something that is not a tool's answer", verb)
}
if env.Error != "" {
return nil, errors.New(env.Error)
}
return env.Result, nil
}
+141
View File
@@ -0,0 +1,141 @@
package link
import (
"crypto/ed25519"
"encoding/json"
"errors"
"fmt"
"strings"
"testing"
"time"
"github.com/nats-io/nats.go"
)
// The move's ask (design 57's V3, issue 676): a hand-over's fields and the row's own in one signed object, under a
// signing context of its own. The controller's side holds the same names and words.
func TestTheMovesAskKeepsItsFieldNamesAndContext(t *testing.T) {
body, _ := json.Marshal(MoveSecretAsk{Module: "m", Name: "n", Provider: "p", Provision: "q", Consumer: "c",
ConsumerModule: "cm", Local: "l", Origin: "made", Generation: "g", MadeAt: time.Unix(1, 0).UTC(),
Sealed: "s", SealedTo: "k"})
if got := keysOf(t, body); got != "consumer consumer_module generation local made_at module name origin provider "+
"provision sealed sealed_to" {
t.Fatalf("the move's fields are %q", got)
}
if AskMoveSecretSubject("ace") != "mesh.node.ace.ask.move-secret" {
t.Fatalf("the subject is %q", AskMoveSecretSubject("ace"))
}
if MoveSecretContext != "novox-mesh move-secret v1\n" {
t.Fatalf("the signing context is %q", MoveSecretContext)
}
answer, _ := json.Marshal(MoveSecretAnswer{Said: "s", Refused: "r", Secret: "own/a/b/c", Generation: 1, Unchanged: true})
if got := keysOf(t, answer); got != "generation refused said secret unchanged" {
t.Fatalf("the answer's fields are %q", got)
}
}
// signedMove is a move's ask as the controller makes it.
func signedMove(t *testing.T, key ed25519.PrivateKey, ask HandOverAsk, move MoveSecretAsk) []byte {
t.Helper()
raw, _ := json.Marshal(struct {
HandOverAsk
MoveSecretAsk
}{ask, move})
body, _ := json.Marshal(SignedHandOver{Ask: raw, Signature: ed25519.Sign(key, append([]byte(MoveSecretContext), raw...))})
return body
}
// A move is believed as every signed ask is, then names one row this machine holds — its own secret, or a pair
// credential with this machine at one end — with a value sealed to it and an origin the store has; only then is the
// handler given it. A hand-over's signature is not a move's.
func TestAMoveIsTakenOnlyForARowThisMachineHolds(t *testing.T) {
m, key := aMember(t)
now := time.Now()
var given []MoveSecret
handle := func(s MoveSecret) (MovedSecret, error) {
given = append(given, s)
return MovedSecret{Secret: "own/" + m.Node + "/postgres/superuser-password", Generation: 1}, nil
}
check := HandOverCheck{Signer: m.Signer, Node: m.Node, Nonces: &Nonces{}}
ask := func(nonce string) HandOverAsk {
return HandOverAsk{Node: m.Node, By: "the move", Expires: now.Add(HandOverGood), Nonce: nonce}
}
own := MoveSecretAsk{Module: "postgres", Name: "superuser-password", Origin: "made", Sealed: "c2VhbGVk"}
for why, body := range map[string][]byte{
"a hand-over's": signedAsk(t, key, ask(aNonce+"01")),
"no value": signedMove(t, key, ask(aNonce+"02"), MoveSecretAsk{Module: "postgres", Name: "x", Origin: "made"}),
"no origin": signedMove(t, key, ask(aNonce+"03"), MoveSecretAsk{Module: "postgres", Name: "x", Sealed: "c2Vh"}),
"both shapes": signedMove(t, key, ask(aNonce+"04"), MoveSecretAsk{Module: "postgres", Name: "x", Provider: m.Node,
Provision: "database", Consumer: "shanks", ConsumerModule: "gitea", Origin: "made", Sealed: "c2Vh"}),
"neither end": signedMove(t, key, ask(aNonce+"05"), MoveSecretAsk{Provider: "anchor", Provision: "database",
Consumer: "shanks", ConsumerModule: "gitea", Origin: "made", Sealed: "c2Vh"}),
"a path": signedMove(t, key, HandOverAsk{Node: m.Node, Path: "/etc", By: "x", Expires: now.Add(HandOverGood),
Nonce: aNonce + "06"}, own),
"two lines": signedMove(t, key, ask(aNonce+"07"), MoveSecretAsk{Module: "postgres", Name: "x\ny", Origin: "made",
Sealed: "c2Vh"}),
} {
if a := AnswerMoveSecret(check, handle, body); a.Refused == "" || !strings.HasSuffix(a.Refused, "nothing was moved") {
t.Errorf("%s: not refused, or refused without saying nothing was moved: %+v", why, a)
}
}
if len(given) != 0 {
t.Fatalf("a refused ask reached the move: %+v", given)
}
a := AnswerMoveSecret(check, handle, signedMove(t, key, ask(aNonce+"08"), own))
if a.Refused != "" || a.Secret != "own/"+m.Node+"/postgres/superuser-password" || a.Generation != 1 || a.Said == "" ||
len(given) != 1 || given[0].Name != "superuser-password" || given[0].By != "the move" {
t.Fatalf("a good ask: %+v %+v", a, given)
}
pair := MoveSecretAsk{Provider: "anchor", Provision: "database", Consumer: m.Node, ConsumerModule: "gitea",
Origin: "accepted", Sealed: "c2Vh"}
if a := AnswerMoveSecret(check, handle, signedMove(t, key, ask(aNonce+"09"), pair)); a.Refused != "" {
t.Fatalf("a pair this machine consumes was refused: %+v", a)
}
// What the move refuses is said, and nothing else.
refusing := func(MoveSecret) (MovedSecret, error) { return MovedSecret{}, errors.New("the vault said no") }
if a := AnswerMoveSecret(check, refusing, signedMove(t, key, ask(aNonce+"10"), own)); !strings.Contains(a.Refused,
"the vault said no") || a.Secret != "" {
t.Fatalf("a refused move: %+v", a)
}
}
// **Against a real server**: the move's signed ask on the node's own subject is answered with what the handler
// moved, and the engine asks the vault on the subject that names it, `node~<machine>` last.
func TestNatsAMoveIsAskedOfTheNodeAndTheVaultIsAskedInItsName(t *testing.T) {
conn, js := aBus(t)
m, key := aMember(t)
vault, err := conn.Subscribe("mesh.seat.mesh-vault.call.move-in.node~"+m.Node, func(msg *nats.Msg) {
_ = msg.Respond([]byte(`{"result":{"call":"c1","seal_to":"k"}}`))
})
if err != nil {
t.Fatal(err)
}
defer vault.Unsubscribe()
l := &natsLink{conn: conn, js: js, node: m.Node, signer: m.Signer}
q := &Queue{Membership: m}
q.attach(t.Context(), l)
l.HearMoveSecrets(func(s MoveSecret) (MovedSecret, error) {
raw, err := q.AskVault(t.Context(), "move-in", map[string]string{"module": s.Module})
if err != nil || !strings.Contains(string(raw), `"call":"c1"`) {
return MovedSecret{}, fmt.Errorf("the vault answered %s, %v", raw, err)
}
return MovedSecret{Secret: "own/" + s.Node + "/" + s.Module + "/" + s.Name, Generation: 1}, nil
})
defer l.Close()
if err := conn.Flush(); err != nil {
t.Fatal(err)
}
ask := HandOverAsk{Node: m.Node, By: "the move", Expires: time.Now().Add(HandOverGood), Nonce: strings.Repeat("ab", 16)}
reply, err := conn.Request(AskMoveSecretSubject(m.Node), signedMove(t, key, ask,
MoveSecretAsk{Module: "postgres", Name: "superuser-password", Origin: "made", Sealed: "c2Vh"}), 5*time.Second)
if err != nil {
t.Fatal(err)
}
var said MoveSecretAnswer
if err := json.Unmarshal(reply.Data, &said); err != nil || said.Secret != "own/"+m.Node+"/postgres/superuser-password" {
t.Fatalf("answered %s, %v", reply.Data, err)
}
if _, err := (&Queue{Membership: Membership{Node: "nowhere"}}).AskVault(t.Context(), "move-in", nil); !errors.Is(err, ErrNotAsked) {
t.Fatalf("a queue with no link asked the vault: %v", err)
}
}
+53
View File
@@ -0,0 +1,53 @@
package link
import (
"context"
"sync"
"testing"
"time"
)
// listeningForAlive is a quiet link that also records anything said on an alive subject. The method
// is deliberately kept here: once the bus has no such method, this link can never be asked for it,
// and the test still holds the loop to saying nothing.
type listeningForAlive struct {
*quietLink
mu sync.Mutex
heard int
}
func (l *listeningForAlive) Alive(context.Context, string, []byte) error {
l.mu.Lock()
defer l.mu.Unlock()
l.heard++
return nil
}
func (l *listeningForAlive) count() int {
l.mu.Lock()
defer l.mu.Unlock()
return l.heard
}
// 1158/AC1: the node-engine publishes nothing on its alive subject (novox/hq ADR 0429 §7, ADR 0433
// §3: presence replaces the alive message). The loop runs over a fake bus and no alive is published,
// neither at the start nor on a tick.
func TestTheLinkLoopPublishesNoAlive(t *testing.T) {
m, _ := aMember(t)
l := &listeningForAlive{quietLink: newQuietLink()}
q := aQueue(m, nil, &keptInMemory{})
ctx, stop := context.WithCancel(context.Background())
done := make(chan error, 1)
go func() { done <- serve(ctx, l, m, q, nil, time.Second) }()
time.Sleep(300 * time.Millisecond)
stop()
select {
case <-done:
case <-time.After(5 * time.Second):
t.Fatal("the loop did not end")
}
if n := l.count(); n != 0 {
t.Fatalf("the node published %d alive messages (1158/AC1)", n)
}
}
+26 -2
View File
@@ -65,8 +65,11 @@ type Queue struct {
// AccessAct judges and runs the take act on an access's folder, asked by the controller on an approved
// warrant (novox/hq ADR 0328 §7). Nil answers every such ask with a refusal.
AccessAct AccessActFunc
Say Announce
Timeout time.Duration
// MoveSecret moves one value sealed to this machine into the vault, asked by the controller during the move of
// the secrets (design 57's V3, issue 676). Nil answers every such ask with a refusal.
MoveSecret MoveSecretFunc
Say Announce
Timeout time.Duration
once sync.Once
wake chan struct{}
@@ -90,6 +93,9 @@ type Queue struct {
saying sync.Mutex
counted bool
numbers numbered
// haves is what this node-engine writes beside its reports (novox/hq ADR 0429 §4, issue 1132).
haves haveState
}
// Reconcile is the reconcile's act, run by the queue's worker. See Queue.Reconcile.
@@ -262,6 +268,14 @@ func (q *Queue) reconcile(ctx context.Context, asked bool) {
return
}
if !asked && (q.News == nil || !q.News(report)) {
// Not worth a report, but a reconcile found how each module is now: the have keys say it (issue 1132), so a
// node-engine that restarted writes them again without waiting for a change.
if q.noteApply(report) {
q.mu.Lock()
bus := q.bus
q.mu.Unlock()
q.writeHaves(ctx, bus)
}
return
}
if asked {
@@ -321,6 +335,10 @@ func (q *Queue) tell(ctx context.Context, r Report, kind kindOf, settle ...Decla
q.unasked = &r
}
if published {
// After the report, which is sent first and unchanged; a write the bucket refuses never undoes it (issue 1132).
if q.noteApply(r) {
q.writeHaves(ctx, bus)
}
if keep && q.Unsaid != nil {
if err := q.Unsaid.Said(r.Declared); err != nil {
q.Say("said this apply's report, and cannot forget it: " + err.Error())
@@ -351,6 +369,8 @@ func (q *Queue) stamp(r Report) Report {
// Every report of a node-engine that orders its reports says it reads a generation (novox/hq issue
// 234): this build does, and the controller sends one only to a machine that said so.
r.ReadsGeneration = true
// And the module of each resource and the build of each module (novox/hq ADR 0429 §4, issue 1132), the same way.
r.ReadsModules = true
return r
}
@@ -535,6 +555,10 @@ func (q *Queue) SayHealth(ctx context.Context, h Health) bool {
q.Say("could not say this machine's health: " + err.Error())
return false
}
// A look that changed a module's health changes its have (issue 1132).
if q.noteHealth(h) {
q.writeHaves(ctx, bus)
}
return true
}
+23
View File
@@ -563,3 +563,26 @@ func TestEveryReportSaysItReadsAGeneration(t *testing.T) {
t.Fatalf("the report does not say it reads a generation: %s", raw)
}
}
// 1132/AC2: every report says this node-engine reads a resource's module and a declaration's builds, beside the
// generation: the controller sends them only to a machine that said so, because an older node-engine refuses the keys.
func TestEveryReportSaysItReadsModules(t *testing.T) {
m, key := aMember(t)
l := newQuietLink()
a := &applies{}
q := aQueue(m, a.apply, &keptInMemory{})
q.attach(context.Background(), l)
q.Deliver(ordered(t, key, "new", 7, 12))
ctx, stop := context.WithCancel(context.Background())
worker := runWorker(ctx, q)
eventually(t, "the delivery was never reported", func() bool { return len(l.said()) == 1 })
stop()
<-worker
raw, err := json.Marshal(l.said()[0])
if err != nil {
t.Fatal(err)
}
if !strings.Contains(string(raw), `"reads_modules":true`) || !strings.Contains(string(raw), `"reads_generation":true`) {
t.Fatalf("the report does not say it reads modules beside the generation: %s", raw)
}
}
+5 -34
View File
@@ -18,13 +18,6 @@ import (
// the first means somebody is trying, the second means something is broken.
var ErrForged = errors.New("this declaration was not signed by the mesh this node joined")
// AliveEvery is how often a node says it is there.
//
// Often enough that "no word for five minutes" means something, rarely enough that a hundred
// nodes are not a hundred messages a second. The mesh reads absence rather than presence, so what
// matters is the interval being known and steady.
const AliveEvery = 60 * time.Second
// Membership is what a node needs to reach its mesh again, held by the caller.
type Membership struct {
Node string
@@ -218,13 +211,6 @@ func serve(ctx context.Context, link Link, m Membership, queue *Queue, say Annou
// still broken when it means the opposite.
say("in the mesh, hearing what this node should be")
// A word every so often, so the mesh can tell a node that is quiet from one that is gone.
// Cheap on purpose: it carries a name and nothing else, because anything more would be a
// report, and reports are rare where this is constant.
beat := time.NewTicker(AliveEvery)
defer beat.Stop()
publishAlive(ctx, link, m, say, timeout)
// **What the last apply did, if the mesh never heard it**, is said before anything newly
// delivered can be: the queue says it as the link is handed to it (novox/hq issue 264). And the
// link is not let go while the queue's worker is mid-act: an apply that ends the link — the host
@@ -256,6 +242,10 @@ func serve(ctx context.Context, link Link, m Membership, queue *Queue, say Annou
if s, ok := link.(AccessActs); ok {
s.HearAccessActs(queue.AccessAct)
}
// And asking to move a value sealed to this machine into the vault (design 57's V3, issue 676).
if s, ok := link.(MoveSecrets); ok {
s.HearMoveSecrets(queue.MoveSecret)
}
for {
// Asked to stop — by standing aside, say — nothing further is taken in.
@@ -265,8 +255,6 @@ func serve(ctx context.Context, link Link, m Membership, queue *Queue, say Annou
select {
case <-ctx.Done():
return nil
case <-beat.C:
publishAlive(ctx, link, m, say, timeout)
case reason := <-link.Lost():
return reason
case <-asks:
@@ -284,7 +272,7 @@ func serve(ctx context.Context, link Link, m Membership, queue *Queue, say Annou
}
// Whatever else is already waiting goes with it. **Enqueued, not applied**: the queue
// applies the newest of everything delivered and not yet taken, and reports the rest as
// set aside (novox/hq to-be 45 §6). The link goes on beating meanwhile.
// set aside (novox/hq to-be 45 §6).
queue.Deliver(gather(declarations, declaration, drainWindow)...)
}
}
@@ -392,23 +380,6 @@ func publishReport(ctx context.Context, bus Bus, m Membership, report Report,
return true
}
// publishAlive says this node is here, and nothing else.
func publishAlive(ctx context.Context, bus Bus, m Membership, say Announce,
timeout time.Duration) {
body, err := json.Marshal(Alive{Node: m.Node, IntervalSeconds: int(AliveEvery / time.Second)})
if err != nil {
return
}
publish, cancel := context.WithTimeout(ctx, timeout)
defer cancel()
// Not mandatory, unlike a report. Losing one is nothing: the next is a minute away, and the
// mesh is reading a gap rather than counting arrivals. Insisting on delivery would turn a
// harmless miss into a logged failure every minute.
if err := bus.Alive(publish, m.Node, body); err != nil {
say("could not tell the mesh this node is here: " + err.Error())
}
}
// heldNote is what this apply did NOT do, for the line that says what it did.
//
// **A count that does not add up is the only symptom a held resource had** (novox/hq 04-ISSUES/125).
+3 -4
View File
@@ -48,10 +48,9 @@ func (l *quietLink) Report(ctx context.Context, _ string, body []byte) error {
l.reports = append(l.reports, r)
return nil
}
func (l *quietLink) Alive(context.Context, string, []byte) error { return nil }
func (l *quietLink) Declarations() <-chan Declaration { return l.delivered }
func (l *quietLink) Lost() <-chan error { return l.lost }
func (l *quietLink) Close() {}
func (l *quietLink) Declarations() <-chan Declaration { return l.delivered }
func (l *quietLink) Lost() <-chan error { return l.lost }
func (l *quietLink) Close() {}
func (l *quietLink) said() []Report {
l.mu.Lock()
+2 -11
View File
@@ -100,7 +100,7 @@ func LongRunning(d *declaration.Declaration, held map[string]bool) []Resource {
return nil
}
for _, r := range d.Resources {
module, ok := ModuleOf(r.Identity())
module, ok := d.ModuleOfID(r.Identity())
if !ok || held[r.Identity()] {
continue
}
@@ -133,16 +133,7 @@ func LongRunning(d *declaration.Declaration, held map[string]bool) []Resource {
// ModuleOf is the module a resource's id belongs to: everything before its last dot — a module's name may
// carry a dot, its resources' own ids never do (the apply's moduleOf, for the same reason). False for
// what the mesh declares in its own right.
func ModuleOf(id string) (string, bool) {
if strings.HasPrefix(id, declaration.AdoptionPrefix) {
return "", false
}
at := strings.LastIndex(id, ".")
if at <= 0 {
return "", false
}
return id[:at], true
}
func ModuleOf(id string) (string, bool) { return declaration.ModuleGuess(id) }
// Observed is one resource as one read found it.
type Observed struct {
+53
View File
@@ -330,3 +330,56 @@ func TestOneLookIsOneReadOfEveryContainer(t *testing.T) {
t.Fatalf("a container the inspect found, beside one it did not: %+v", s)
}
}
// **A look starts no account's manager** (novox/hq issue 764, ADR 0177): a user-scoped unit's manager is
// asked of the machine's first; one not running is not read, nothing is started, and its unit is said
// unknown with why — away, never down; one running is read as before.
func TestAUserUnitsManagerNotRunningIsNotReadAndNotDown(t *testing.T) {
var asked []string
manager := "inactive"
run := func(_ context.Context, name string, args ...string) (string, error) {
asked = append(asked, name+" "+strings.Join(args, " "))
switch {
case name == "getent":
return "operator:x:1000:1000::/home/operator:/bin/zsh\n", nil
case name == "systemctl" && len(args) > 0 && args[0] == "is-active":
if manager != "active" {
return manager + "\n", errors.New("systemctl exited 3")
}
return "active\n", nil
case name == "systemctl":
return "Id=dunst.service\nLoadState=loaded\nActiveState=active\nSubState=running\nNRestarts=0\nInvocationID=i1\n", nil
}
return "", errors.New("not a command the judge may run")
}
c := &clock{now: t0}
j := aJudge(t, &Exec{Run: run, System: "arch"}, c, "")
j.Set([]Resource{{Module: "dunst", ID: "dunst.daemon", Kind: KindService, Target: "dunst.service",
Scope: "user", User: "operator"}})
var st Statement
for i := 0; i < 4; i++ {
c.now = t0.Add(time.Duration(i) * time.Minute)
st, _ = j.Look(t.Context())
}
if s := stateOf(t, st, "dunst.daemon"); s.State != Unknown || !strings.Contains(s.Reason, "not logged in") {
t.Fatalf("a unit in a manager not running is away, not down: %+v", s)
}
for _, a := range asked {
if strings.Contains(a, "--machine=") {
t.Fatalf("the account's own manager was asked while it was not running: %q", a)
}
if a != "getent passwd operator" && a != "systemctl is-active user@1000.service" {
t.Errorf("asked something but whether the manager runs: %q", a)
}
}
// Logged in: read in the account's own manager, as before.
manager, asked = "active", nil
c.now = t0.Add(5 * time.Minute)
st, _ = j.Look(t.Context())
if s := stateOf(t, st, "dunst.daemon"); s.State == Unknown || s.State == Unhealthy {
t.Fatalf("a unit in a running manager is read: %+v", s)
}
if last := asked[len(asked)-1]; !strings.HasPrefix(last, "systemctl --user --machine=operator@ show ") {
t.Fatalf("the running manager is read in the account's own: %q", last)
}
}
+20 -1
View File
@@ -8,6 +8,8 @@ import (
"strconv"
"strings"
"sync"
"github.com/novox/mesh-host/internal/system"
)
// Runner runs a command and answers what it printed — the apply's own (apply.ExecRunner), so a look
@@ -16,9 +18,13 @@ type Runner func(ctx context.Context, name string, args ...string) (string, erro
// Exec reads the container runtime and the service manager through their command lines: one inspect of
// every container, one show of every unit per manager. **Reads only** — the whole of what this file may
// ask either of them is `container inspect`, `version` and `show` (ADR 0240 rule 6, and a test holds it).
// ask either of them is `container inspect`, `version` and `show`, and of the machine's manager whether an
// account's own is active (ADR 0240 rule 6, and a test holds it).
type Exec struct {
Run Runner
// System is what the host was built for: it says how the machine's manager is asked whether an
// account's own runs (system.ManagerRunning).
System string
mu sync.Mutex
// cri is the runtime that answered, once one has.
@@ -146,6 +152,19 @@ func (e *Exec) Units(ctx context.Context, scope, user string, units []string) (m
}
args := []string{"show", "--property=" + unitProperties, "--"}
if scope == "user" && user != "" {
// **A look starts no account's manager** (novox/hq issue 764, ADR 0177): reading it is a login,
// which starts it when none runs. The machine's manager is asked first; a manager not running is
// away — its units unread, said unknown with why, never down — as the apply says them waiting.
running, exists, err := system.ManagerRunning(ctx, e.System, system.Runner(e.Run), user)
switch {
case err != nil:
return nil, fmt.Errorf("whether %s's own service manager runs could not be read: %w", user, err)
case !exists:
return nil, fmt.Errorf("the machine has no account %q, whose own manager these units are in", user)
case !running:
return nil, fmt.Errorf("%s's own service manager is not running: the account is not logged in "+
"and does not linger", user)
}
args = append([]string{"--user", "--machine=" + user + "@"}, args...)
}
said, err := e.Run(ctx, "systemctl", append(args, units...)...)
+19
View File
@@ -0,0 +1,19 @@
package meshcli
import "net"
// The kernel's word on who is at the other end of a socket, shared with the node-engine's other local socket, the
// write of a filled file (internal/filling, novox/hq ADR 0349 §4): one reading of it, so the two never judge a peer
// differently.
// PeerCredentials is the user id and process id of the process at the other end, as the kernel recorded them when
// it connected (SO_PEERCRED). Nothing the process says is read.
func PeerCredentials(conn *net.UnixConn) (uint32, int32, error) { return peerCredentials(conn) }
// SameNamespaces refuses a process whose mount or user namespace is not the engine's: a user id is this machine's
// account only in this machine's namespaces.
func SameNamespaces(pid int32) error { return namespacesOf(pid) }
// NotOwnDirectory says what is wrong with a socket's directory, or nothing: it must be a directory, not a link,
// owned by the account the engine runs as.
func NotOwnDirectory(dir string) string { return notOwnDirectory(dir) }
+92
View File
@@ -0,0 +1,92 @@
package outward
import (
"errors"
"fmt"
"net"
"net/netip"
"sort"
)
// Network is one network this machine is directly attached to, and the link it is on (novox/hq ADR
// 0340 §7): the address and prefix of the network, not of the machine's own address on it.
//
// **Every one is said, and the machine judges none of them.** Which count as the machine's local
// network — private ranges only, never a public subnet, never a link-local one — is the controller's
// judgement (ADR 0340 §2), made where the filter is composed and shown with why; a machine that left a
// network out would make that judgement silently.
type Network struct {
Link string `json:"link"`
Network string `json:"network"`
}
// AddressesOf is the addresses on one link, as address and prefix. A parameter so a test can hold a
// machine's links without one.
type AddressesOf func(link string) ([]netip.Prefix, error)
// RealAddresses reads one link's addresses from the kernel.
func RealAddresses(link string) ([]netip.Prefix, error) {
iface, err := net.InterfaceByName(link)
if err != nil {
return nil, err
}
addrs, err := iface.Addrs()
if err != nil {
return nil, err
}
var out []netip.Prefix
for _, a := range addrs {
ipnet, ok := a.(*net.IPNet)
if !ok {
continue
}
addr, ok := netip.AddrFromSlice(ipnet.IP)
if !ok {
continue
}
bits, _ := ipnet.Mask.Size()
out = append(out, netip.PrefixFrom(addr.Unmap(), bits))
}
return out, nil
}
// Networks is the networks on each of these links — the links that face outside (Links) — sorted by link
// and network, without repeats. A link that has gone since Links read it has no networks rather than
// failing the rest; a link whose addresses cannot be read fails, so the machine says the fact is unknown
// rather than reporting a partial account as the whole.
func Networks(links []string, addressesOf AddressesOf) ([]Network, error) {
if addressesOf == nil {
addressesOf = RealAddresses
}
seen := map[Network]bool{}
var out []Network
for _, link := range links {
prefixes, err := addressesOf(link)
if err != nil {
if isNoSuchInterface(err) {
continue
}
return nil, fmt.Errorf("cannot read the addresses on %s: %w", link, err)
}
for _, p := range prefixes {
n := Network{Link: link, Network: p.Masked().String()}
if !seen[n] {
seen[n] = true
out = append(out, n)
}
}
}
sort.Slice(out, func(i, j int) bool {
if out[i].Link != out[j].Link {
return out[i].Link < out[j].Link
}
return out[i].Network < out[j].Network
})
return out, nil
}
// isNoSuchInterface is the error net.InterfaceByName gives for a link that is not there.
func isNoSuchInterface(err error) bool {
var op *net.OpError
return errors.As(err, &op) && op.Err != nil && op.Err.Error() == "no such network interface"
}
+59
View File
@@ -0,0 +1,59 @@
package outward
import (
"errors"
"net/netip"
"reflect"
"testing"
)
func prefixes(s ...string) []netip.Prefix {
var out []netip.Prefix
for _, p := range s {
out = append(out, netip.MustParsePrefix(p))
}
return out
}
// **Each link facing outside says the networks it is attached to** (novox/hq ADR 0340 §7), as the
// network's address and prefix, never the machine's own address on it — and every one, public and
// link-local included, because which count is the controller's judgement and said with why.
func TestEachOutwardLinkSaysTheNetworksItIsOn(t *testing.T) {
addrs := map[string][]netip.Prefix{
"eth0": prefixes("192.168.1.10/24", "fe80::1/64", "fd12:3456:789a::5/64"),
"eth1": prefixes("203.0.113.7/24"),
// Not asked: the bridge is not a link facing outside.
"docker0": prefixes("172.17.0.1/16"),
}
got, err := Networks([]string{"eth1", "eth0"}, func(link string) ([]netip.Prefix, error) { return addrs[link], nil })
if err != nil {
t.Fatal(err)
}
want := []Network{
{Link: "eth0", Network: "192.168.1.0/24"},
{Link: "eth0", Network: "fd12:3456:789a::/64"},
{Link: "eth0", Network: "fe80::/64"},
{Link: "eth1", Network: "203.0.113.0/24"},
}
if !reflect.DeepEqual(got, want) {
t.Fatalf("got %+v\nwant %+v", got, want)
}
}
// A link whose addresses cannot be read fails the whole, so the machine says the fact is unknown rather
// than reporting part of its networks as all of them.
func TestAnUnreadableLinkIsNotAPartialAccount(t *testing.T) {
_, err := Networks([]string{"eth0"}, func(string) ([]netip.Prefix, error) { return nil, errors.New("denied") })
if err == nil {
t.Fatal("an unreadable link was reported as having no networks")
}
}
// A link gone between reading the routing table and reading its addresses has no networks, and the
// rest are still said.
func TestALinkGoneSinceItWasListedHasNoNetworks(t *testing.T) {
got, err := Networks([]string{"gone0"}, RealAddresses)
if err != nil || len(got) != 0 {
t.Fatalf("got %v, %v", got, err)
}
}
+6
View File
@@ -24,6 +24,11 @@ const (
CapSeat = "seat"
CapPrivileged = "privileged"
// CapUserNamespaces is whether an unprivileged user can make a user namespace: what a build
// module's sandbox (bubblewrap) needs (novox/hq ADR 0448 section 5). Proven by making one, as
// an unprivileged user, never by a version string or a sysctl alone.
CapUserNamespaces = "user-namespaces"
// The network manager this machine runs, one capability per dialect (novox/hq ADR 0161): the
// uplink seat's holder declares its own, so the holder for a manager the machine does not run
// is refused the way any missing capability is, naming it. Active, not installed — a machine
@@ -189,6 +194,7 @@ func Default(runner Runner) []Detector {
seat{},
battery{},
powerMeter{},
userNamespaces(runner),
commandCapability{
name: CapContainerRuntime, command: "docker", args: []string{"info", "--format", "{{.ServerVersion}}"},
why: "asks the daemon for its version — a running daemon, not an installed client",
+35 -2
View File
@@ -10,6 +10,8 @@ import (
"sync"
"syscall"
"time"
"github.com/novox/mesh-host/internal/outward"
)
// Machine is what this machine is made of, as it says itself (novox/hq issue 160): what somebody
@@ -50,6 +52,11 @@ type Machine struct {
// Filesystems is each local filesystem mounted, by where, with its size and what is free.
Filesystems []Filesystem `json:"filesystems,omitempty"`
// Networks is every network the machine is directly attached to on a link facing outside it, as the
// link and the network's address and prefix (novox/hq ADR 0340 §7): what a port reached from
// `local-network` is resolved from. Every one is said; which count is the controller's judgement.
Networks []outward.Network `json:"networks,omitempty"`
// Unknown names each fact that could not be determined, and why.
Unknown map[string]string `json:"unknown,omitempty"`
}
@@ -79,11 +86,12 @@ const (
FactTimezone = "timezone"
FactFilesystems = "filesystems"
FactPublicAddress = "public address"
FactNetworks = "networks"
)
// MachineFacts is every fact Unknown may name, in the order a reader shows them.
var MachineFacts = []string{FactKernelRelease, FactDistribution, FactVersion, FactProcessor, FactCores,
FactThreads, FactMemory, FactVirtual, FactUptime, FactTimezone, FactFilesystems, FactPublicAddress}
FactThreads, FactMemory, FactVirtual, FactUptime, FactTimezone, FactFilesystems, FactNetworks, FactPublicAddress}
// publicAddressUnknown is why the public uplink address is not said. A machine behind a NAT cannot see
// its public address from itself, and asking a service outside the mesh is a dependency the mesh does not
@@ -101,11 +109,24 @@ type Sources struct {
// Statfs is (size, free) in bytes of the filesystem mounted at a path.
Statfs func(path string) (size, free int64, err error)
Now func() time.Time
// Networks is the networks on the links facing outside this machine (novox/hq ADR 0340).
Networks func() ([]outward.Network, error)
}
// RealSources reads this machine.
func RealSources() Sources {
return Sources{Read: os.ReadFile, Readlink: os.Readlink, Run: ExecRunner, Statfs: statfs, Now: time.Now}
return Sources{Read: os.ReadFile, Readlink: os.Readlink, Run: ExecRunner, Statfs: statfs, Now: time.Now,
Networks: attachedNetworks}
}
// attachedNetworks is this machine's networks on the links that face outside it, read the way the report
// reads those links (novox/hq ADR 0140), so the two never disagree about which links those are.
func attachedNetworks() ([]outward.Network, error) {
links, err := outward.Links("", "")
if err != nil {
return nil, err
}
return outward.Networks(links, nil)
}
func statfs(path string) (int64, int64, error) {
@@ -193,6 +214,18 @@ func DescribeMachine(ctx context.Context, src Sources) Machine {
}
}
// The networks it is attached to, for the local network (novox/hq ADR 0340): every one on a link facing
// outside, or why none is said — never an empty list that reads as a machine on no network.
if src.Networks == nil {
unknown(FactNetworks, "this node-engine was given no way to read its links")
} else if networks, err := src.Networks(); err != nil {
unknown(FactNetworks, err.Error())
} else if len(networks) == 0 {
unknown(FactNetworks, "no link facing outside this machine carries an address")
} else {
m.Networks = networks
}
unknown(FactPublicAddress, publicAddressUnknown)
return m
}
+29
View File
@@ -10,6 +10,8 @@ import (
"sync/atomic"
"testing"
"time"
"github.com/novox/mesh-host/internal/outward"
)
// The parsing layer, with the machine's files given (novox/hq ADR 0017 allows a fake here and only here);
@@ -40,6 +42,9 @@ func given(read map[string]string) Sources {
Run: func(context.Context, string, ...string) (string, error) { return "none\n", errors.New("exited 1") },
Statfs: func(p string) (int64, int64, error) { return 1000, 400, nil },
Now: func() time.Time { return time.Date(2026, 10, 12, 15, 0, 0, 0, time.UTC) },
Networks: func() ([]outward.Network, error) {
return []outward.Network{{Link: "eth0", Network: "192.168.1.0/24"}, {Link: "eth1", Network: "203.0.113.0/24"}}, nil
},
}
}
@@ -92,6 +97,7 @@ func TestMachine_aFactItCannotReadIsSaidUnknownNeverZero(t *testing.T) {
src := given(map[string]string{})
src.Readlink = func(string) (string, error) { return "", os.ErrNotExist }
src.Run = func(context.Context, string, ...string) (string, error) { return "", errors.New("not found") }
src.Networks = func() ([]outward.Network, error) { return nil, errors.New("cannot read the routing table") }
m := DescribeMachine(context.Background(), src)
raw, _ := json.Marshal(m)
@@ -183,3 +189,26 @@ func TestAgainstThisMachine_machineSaysWhatItIs(t *testing.T) {
raw, _ := json.MarshalIndent(m, "", " ")
t.Logf("%s", raw)
}
// **The networks it is attached to are among its facts** (novox/hq ADR 0340 §7), every one — the public
// one too, which the controller judges and never counts — and a machine with none on a link facing
// outside says why rather than reporting an empty list.
func TestMachine_saysTheNetworksItIsAttachedTo(t *testing.T) {
m := DescribeMachine(context.Background(), given(files))
raw, _ := json.Marshal(m)
var said struct {
Networks []map[string]string `json:"networks"`
}
_ = json.Unmarshal(raw, &said)
if len(said.Networks) != 2 || said.Networks[0]["link"] != "eth0" || said.Networks[0]["network"] != "192.168.1.0/24" ||
said.Networks[1]["network"] != "203.0.113.0/24" {
t.Fatalf("networks said as %s", raw)
}
src := given(files)
src.Networks = func() ([]outward.Network, error) { return nil, nil }
m = DescribeMachine(context.Background(), src)
if len(m.Networks) != 0 || !strings.Contains(m.Unknown[FactNetworks], "no link facing outside") {
t.Fatalf("a machine with no network said %v, unknown %q", m.Networks, m.Unknown[FactNetworks])
}
}
+61
View File
@@ -0,0 +1,61 @@
package profile
import (
"fmt"
"os"
"strings"
)
// effectiveUID and readProcFile are replaceable so the unit tests stay off the machine.
var (
effectiveUID = os.Geteuid
readProcFile = func(path string) (string, error) {
b, err := os.ReadFile(path)
return string(b), err
}
)
// unprivilegedID is the uid and gid the probe drops to (nobody on every mesh node).
const unprivilegedID = "65534"
// userNamespaces builds the detector for CapUserNamespaces: it makes a user namespace as an
// unprivileged user. The node-engine runs as root, and root can always make one, so as root the
// probe drops first with setpriv (util-linux). Go's SysProcAttr Cloneflags with a Credential
// would clone before the uid drops and prove nothing, so it is not used.
func userNamespaces(runner Runner) Detector {
command, args := "unshare", []string{"--user", "true"}
if effectiveUID() == 0 {
command = "setpriv"
args = []string{"--reuid=" + unprivilegedID, "--regid=" + unprivilegedID, "--clear-groups", "unshare", "--user", "true"}
}
read := readProcFile // bound now, so a probe cut by the timeout never reads a later replacement
return commandCapability{
name: CapUserNamespaces, command: command, args: args,
why: "makes a user namespace as an unprivileged user — what bubblewrap needs, not a kernel setting read",
interpret: func(_ string, err error) (bool, string) {
if err == nil {
return true, "an unprivileged user can make a user namespace"
}
return false, err.Error() + userNamespaceSettings(read)
},
runner: runner,
}
}
// userNamespaceSettings names the two kernel settings that switch user namespaces off, when
// they can be read. It is added to the reason of an absent verdict only.
func userNamespaceSettings(read func(string) (string, error)) string {
var parts []string
for _, s := range []struct{ path, name string }{
{"/proc/sys/kernel/unprivileged_userns_clone", "unprivileged_userns_clone"},
{"/proc/sys/user/max_user_namespaces", "max_user_namespaces"},
} {
if v, err := read(s.path); err == nil {
parts = append(parts, fmt.Sprintf("%s=%s", s.name, strings.TrimSpace(v)))
}
}
if len(parts) == 0 {
return ""
}
return " (" + strings.Join(parts, ", ") + ")"
}
+155
View File
@@ -0,0 +1,155 @@
package profile
import (
"context"
"errors"
"strings"
"sync"
"testing"
"time"
)
// recorded is a runner that remembers every command it was given and answers the user-namespace
// probe with the given output and error. Other detectors get "not here".
type recorded struct {
mu sync.Mutex
calls [][]string
out string
err error
}
func (r *recorded) run(_ context.Context, name string, args ...string) (string, error) {
if name != "setpriv" && name != "unshare" {
return "", errors.New("not here")
}
r.mu.Lock()
defer r.mu.Unlock()
r.calls = append(r.calls, append([]string{name}, args...))
return r.out, r.err
}
// asUID makes the detector believe it runs as uid, and reads no files from the machine.
func asUID(t *testing.T, uid int, procs map[string]string) {
t.Helper()
oldUID, oldRead := effectiveUID, readProcFile
effectiveUID = func() int { return uid }
readProcFile = func(path string) (string, error) {
if v, ok := procs[path]; ok {
return v, nil
}
return "", errors.New("no such file")
}
t.Cleanup(func() { effectiveUID, readProcFile = oldUID, oldRead })
}
func verdictOf(t *testing.T, p Profile, name string) Verdict {
t.Helper()
for _, v := range p.Capabilities {
if v.Name == name {
return v
}
}
t.Fatalf("no verdict named %s in the profile", name)
return Verdict{}
}
// unshare's own words when user namespaces are off, from util-linux's unshare.c:
// err(EXIT_FAILURE, _("unshare failed")), which prints "unshare: unshare failed: " and strerror(EPERM).
// ExecRunner wraps stderr as "<name> exited <code>: <stderr>".
const unshareRefused = "unshare exited 1: unshare: unshare failed: Operation not permitted"
func TestUserNamespacesAsRootDropsToAnUnprivilegedUIDFirst(t *testing.T) {
// 1308/AC1. Root can always make a user namespace, so a probe run as root proves nothing: the
// command must drop to an unprivileged uid before unshare runs.
asUID(t, 0, nil)
r := &recorded{}
got := Detect(context.Background(), Default(r.run), time.Second)
v := verdictOf(t, got, CapUserNamespaces)
if !v.Present {
t.Fatalf("a successful probe was reported absent: %+v", v)
}
want := "setpriv --reuid=65534 --regid=65534 --clear-groups unshare --user true"
if len(r.calls) != 1 || strings.Join(r.calls[0], " ") != want {
t.Fatalf("the probe ran %v, want one call: %s", r.calls, want)
}
}
func TestUserNamespacesAbsentCarriesTheCommandsWords(t *testing.T) {
// 1308/AC2. The failing probe gives absent, and the detail is what unshare said.
asUID(t, 0, nil)
r := &recorded{err: errors.New(unshareRefused)}
got := Detect(context.Background(), Default(r.run), time.Second)
v := verdictOf(t, got, CapUserNamespaces)
if v.Present {
t.Fatal("a probe that failed was reported present")
}
if !strings.Contains(v.Detail, "unshare failed: Operation not permitted") {
t.Fatalf("the command's words were lost; detail was %q", v.Detail)
}
if strings.TrimSpace(v.How) == "" {
t.Fatal("an absent verdict with no stated method")
}
}
func TestUserNamespacesAbsentSaysWhyFromTheKernelSettings(t *testing.T) {
// 1308/AC2, the reasons: the two kernel settings that switch user namespaces off are read
// through a replaceable function and named in the detail.
asUID(t, 0, map[string]string{
"/proc/sys/user/max_user_namespaces": "0\n",
"/proc/sys/kernel/unprivileged_userns_clone": "0\n",
})
r := &recorded{err: errors.New(unshareRefused)}
v := verdictOf(t, Detect(context.Background(), Default(r.run), time.Second), CapUserNamespaces)
if v.Present {
t.Fatal("reported present")
}
// The detail is one line (firstLine), so the settings must be on that line.
if !strings.Contains(v.Detail, "max_user_namespaces=0") || !strings.Contains(v.Detail, "unprivileged_userns_clone=0") {
t.Fatalf("the kernel settings are not in the detail: %q", v.Detail)
}
}
func TestUserNamespacesNotAsRootUnsharesDirectly(t *testing.T) {
// 1308/AC3. An unprivileged node-engine is already the unprivileged user: no setpriv.
asUID(t, 1000, nil)
r := &recorded{}
v := verdictOf(t, Detect(context.Background(), Default(r.run), time.Second), CapUserNamespaces)
if !v.Present {
t.Fatalf("a successful probe was reported absent: %+v", v)
}
if len(r.calls) != 1 || strings.Join(r.calls[0], " ") != "unshare --user true" {
t.Fatalf("the probe ran %v, want: unshare --user true", r.calls)
}
if strings.TrimSpace(v.Detail) == "" || strings.TrimSpace(v.How) == "" {
t.Fatalf("present verdict lacks detail or how: %+v", v)
}
}
func TestUserNamespacesProbeThatHangsIsCutAndTheRestStillReport(t *testing.T) {
// 1308/AC4. A wedged unshare is cut by Detect's timeout; the other detectors still answer.
asUID(t, 0, nil)
blocking := func(ctx context.Context, name string, args ...string) (string, error) {
if name == "setpriv" {
<-ctx.Done()
return "", ctx.Err()
}
return "ok", nil
}
done := make(chan Profile, 1)
go func() { done <- Detect(context.Background(), Default(blocking), 50*time.Millisecond) }()
select {
case got := <-done:
if verdictOf(t, got, CapUserNamespaces).Present {
t.Error("a probe that never answered was reported present")
}
if !got.Has(CapPackageManager) {
t.Error("the other capabilities were lost behind the wedged probe")
}
case <-time.After(5 * time.Second):
t.Fatal("detection did not return")
}
}
+5
View File
@@ -155,6 +155,11 @@ type Applied struct {
// container's spec label; this is the same information kept per file, so that when the spec
// no longer matches the host can say WHICH file changed rather than only that something did.
Reads map[string]string `json:"reads,omitempty"`
// BusySince is, for a container left running because it said busy, when the first apply that would
// have replaced it ran (novox/hq ADR 0335 §3): kept here, beside the machine's state, so the bound runs
// from then across every pass and across a restart of the node-engine. Absent while nothing waits.
BusySince *time.Time `json:"busy_since,omitempty"`
}
// Into is what a file written into held before the mesh's keys, or before the mesh's block.
+21
View File
@@ -304,6 +304,27 @@ func For(name string) (System, error) {
name, strings.Join(names, ", "))
}
// userManagers is a service manager that runs a manager per account (novox/hq ADR 0177).
type userManagers interface {
UserManagerRunning(ctx context.Context, run Runner, account string) (running, exists bool, err error)
}
// ManagerRunning is whether account's own manager runs on a machine built for name, asked of the
// machine's manager through run — never of the account's, which a question would start (see
// arch.UserManagerRunning). A look asks it before it reads anything in an account's manager (novox/hq
// issue 764). A system with no manager per account is an error: nothing can be read there.
func ManagerRunning(ctx context.Context, name string, run Runner, account string) (running, exists bool, err error) {
sys, err := For(name)
if err != nil {
return false, false, err
}
um, ok := sys.(userManagers)
if !ok {
return false, false, fmt.Errorf("this machine's service manager (%s) has no manager per account", sys.Name())
}
return um.UserManagerRunning(ctx, run, account)
}
// All is every system the host can be built for.
func All() []System {
return []System{arch{}, alpine{}, android{}}
+12 -2
View File
@@ -5,6 +5,8 @@ import (
"errors"
"fmt"
"strings"
"github.com/novox/mesh-host/internal/system"
)
// Runner runs a command and answers what it printed — the apply's own (apply.ExecRunner), so a look reads
@@ -12,8 +14,9 @@ import (
type Runner func(ctx context.Context, name string, args ...string) (string, error)
// Exec reads the service managers through systemctl and the package database through the system's own
// owner query. **Reads only**: `list-units`, `show` and the owner query are the whole of what it asks
// (ADR 0240 rule 6, and a test holds it).
// owner query. **Reads only**: `list-units`, `show`, the owner query, and — of the machine's manager and
// user database — whether an account's manager is active, are the whole of what it asks (ADR 0240 rule 6,
// and a test holds it).
type Exec struct {
Run Runner
// System is what the host was built for; it says how a file's package is asked. A system with no
@@ -103,3 +106,10 @@ func (e Exec) PackageOwning(ctx context.Context, path string) (string, bool, err
}
return "", false, nil
}
// UserManagerRunning asks the machine's manager whether an account's own runs, as the apply does before it
// touches a user-scoped unit (system.UserManagerRunning; novox/hq ADR 0177, issue 764). A system with no
// manager per account cannot say, and is an error: that manager is said unread.
func (e Exec) UserManagerRunning(ctx context.Context, account string) (running, exists bool, err error) {
return system.ManagerRunning(ctx, e.System, system.Runner(e.Run), account)
}
+48 -12
View File
@@ -9,8 +9,8 @@
// own mount table, and a unit a removed package left behind, failed beside it, said by nobody.
//
// On every look the engine asks each service manager the mesh places units in — the machine's, and the
// account manager of every account the declaration names — which units failed, and says each one's
// owner:
// account manager of every account the declaration places a user-scoped unit for — which units failed,
// and says each one's owner:
//
// 1. a service or process the declaration states, by its unit and manager;
// 2. a file the declaration writes, when the unit's file is that file;
@@ -23,6 +23,12 @@
//
// **It reads; it never acts** (ADR 0240 rule 6): `list-units`, `show` and the package manager's owner
// query are the whole of what it asks. It neither resets nor restarts anything.
//
// **A look starts no account's manager** (novox/hq issue 764, ADR 0177). Reading an account's manager is
// a login, and a login starts the manager when none runs; so the machine's own manager is asked first
// whether user@<uid>.service is active, and the account's is read only then. A manager that is not
// running is away: no unit in it is failed, none is read, and the statement says it away — the apply's
// word for the same absence ("waiting" for its manager).
package units
import (
@@ -72,7 +78,9 @@ type Owned struct {
// Files is keyed by path; Packages by package name, only those declared present.
Files map[string]owner
Packages map[string]owner
// Accounts are the accounts whose own managers are read, sorted.
// Accounts are the accounts whose own managers are read, sorted: each one the declaration places a
// user-scoped unit for, and no other (novox/hq issue 764). An account the declaration only creates
// has no unit of the mesh's in its manager to fail, and asking that manager starts it.
Accounts []string
}
@@ -123,8 +131,6 @@ func OwnedBy(d *declaration.Declaration, held map[string]bool) Owned {
if !v.Absent {
o.Packages[v.Package] = own
}
case *declaration.User:
accounts[v.Name] = true
}
}
for a := range accounts {
@@ -169,6 +175,10 @@ type Reader interface {
Show(ctx context.Context, scope, user string, units []string) (map[string]Shown, error)
// PackageOwning is the package a file belongs to; false when none does or the machine cannot say.
PackageOwning(ctx context.Context, path string) (string, bool, error)
// UserManagerRunning is whether an account's own manager runs now, asked of the machine's manager
// and never of the account's, which a question would start (novox/hq issue 764); exists is false
// for an account the machine does not have.
UserManagerRunning(ctx context.Context, account string) (running, exists bool, err error)
}
// Failed is one failed unit as the statement says it.
@@ -198,6 +208,9 @@ type Statement struct {
Failed []Failed
// Unread names each manager that could not be read, with why.
Unread []string
// Away names each account manager not running, so not read: its account is not logged in and does
// not linger. Nothing in it is failed (novox/hq issue 764).
Away []string
}
// Owned answers the failures a module owns; Unowned those no module does.
@@ -290,16 +303,39 @@ func (j *Judge) Look(ctx context.Context) (Statement, bool) {
st := Statement{At: now, State: Unknown}
read := 0
failedNow := map[string]bool{}
keep := func(m manager) {
// What it held is neither cleared nor counted while it cannot be read.
for k := range j.seen {
if strings.HasPrefix(k, key(m.scope, m.user, "")) {
failedNow[k] = true
}
}
}
for _, m := range managers {
if m.scope == ScopeUser {
// Asked of the machine's manager first: reading the account's own would start it.
running, exists, err := j.reader.UserManagerRunning(ctx, m.user)
switch {
case err != nil:
st.Unread = append(st.Unread, fmt.Sprintf("%s could not be read: %s", m, firstLine(err.Error())))
keep(m)
continue
case !exists:
st.Unread = append(st.Unread, fmt.Sprintf("%s could not be read: the machine has no account %q",
m, m.user))
keep(m)
continue
case !running:
// Away: nothing in it runs, so nothing in it is failed; what it held is cleared.
st.Away = append(st.Away, m.user+"'s own service manager is not running: the account is not "+
"logged in and does not linger, so its units are not read")
continue
}
}
listed, err := j.reader.Failed(ctx, m.scope, m.user)
if err != nil {
st.Unread = append(st.Unread, fmt.Sprintf("%s could not be read: %s", m, firstLine(err.Error())))
// What it held is neither cleared nor counted while it cannot be read.
for k := range j.seen {
if strings.HasPrefix(k, key(m.scope, m.user, "")) {
failedNow[k] = true
}
}
keep(m)
continue
}
read++
@@ -353,7 +389,7 @@ func (j *Judge) Look(ctx context.Context) (Statement, bool) {
if read == 0 {
st.State = Unknown
}
word := st.State
word := st.State + "|away:" + strings.Join(st.Away, ",")
for _, f := range st.Failed {
word += "|" + f.Scope + "/" + f.Unit + "/" + f.Module
}

Some files were not shown because too many files have changed in this diff Show More