Compare commits

..
Author SHA1 Message Date
jschoubben 11e4bc0ba1 The roster is the machines: each node's internal domain covers its routes (hq ADR 0191)
The roster published routed names — public ones first, then (in this PR's first take) internal ones
told apart by suffix. Neither is needed: a node has one internal domain and every route on it is a
name under it, answered by the resolver's per-node wildcard; a node's public domains are public
DNS's. routeNamesInTheMesh and NamesServed are removed, and a test pins .Names to the machines.
2026-10-03 16:10:16 +02:00
jschoubben e56f3aa1cb The roster publishes a route's internal name, never its public one (hq ADR 0191)
NamesServed read a route's public `name` and plan.go then filtered by suffix — telling the mesh's
names from public ones by their spelling, when the mesh composed both itself. It now publishes the
`internal-name` it composed under the serving node (ADR 0151); the suffix filter is gone.
2026-10-03 15:39:05 +02:00
mesh-admin f966693583 Merge pull request 'A file a tools bundle's words name restarts the runtime when it changes (hq ADR 0192)' (#237) from feat/0192-a-named-file-restarts-the-runtime into main 2026-10-03 13:33:57 +00:00
jochen 5acc763992 A file a tools bundle's words name restarts the runtime when it changes (hq ADR 0192)
The tool containers were restarted when their configuration file changed; the runtime now is
too, for every file a module's words name exactly — configuration and own secret alike.
2026-10-03 15:33:44 +02:00
mesh-admin 4f009fff83 Merge pull request 'A tools bundle is given its words, composed per machine; what they name is the account's to read (hq ADR 0192)' (#236) from feat/0192-a-bundles-env into main 2026-10-03 13:32:38 +00:00
jochen f27e31954f A tools bundle is given its words, composed per machine; what they name is the account's to read (hq ADR 0192)
build.artifacts[].env on a bundle: words and values written with ${dir:…} and ${port:…} only,
refused when a value carries any other reference (a secret's content, a binding) or names a word
the runtime sets for itself, and on any artifact that is not a bundle. Resolved per machine like a
container's environment and handed to the runtime as MESH_TOOL_ENV, module by module, in the unit
so a change restarts it. Every file and directory of the module a word names, or that holds one, is
owned by the account the runtime runs as where it says no owner, since a tool reads as that account.
2026-10-03 15:32:03 +02:00
jschoubben 62650cd48c Merge pull request 'The mesh answers only its own names privately; a public name resolves publicly (hq ADR 0191)' (#235) from fix/the-mesh-resolves-only-its-own-names into main 2026-10-03 13:16:14 +00:00
jschoubben 408f6dbad9 The mesh answers only its own names privately; a public name resolves publicly (hq ADR 0191)
Every routed public name was published into each machine's hosts region at its serving node's
private address. ace's resolver also answers its LAN, so a phone there got the control-node's
tunnel address for the mail server and could not connect. Routes have internal names under the
serving node (ADR 0151), so only names under the mesh suffix are published now.
2026-10-03 15:14:01 +02:00
mesh-admin eae0577567 Merge pull request 'Issues 203 and 206: an assignment issues its credential; the controller owns a worker's shape; the build seat's holder follows the controller' (#233) from fix/issues-203-206 into main 2026-10-03 09:49:54 +00:00
mesh-admin de26918c52 Merge pull request 'A declaration is numbered when it is composed, and a send is recorded even by a sender being replaced (hq issue 204)' (#232) from fix/issue-204 into main 2026-10-03 09:44:42 +00:00
mesh-admin e01d18e548 Merge pull request 'An idle build machine's empty fetch is asked again, not read as the end (hq ADR 0190)' (#234) from fix/an-empty-fetch-is-not-the-end into main 2026-10-03 09:41:52 +00:00
jochen 59166b1031 An idle build machine's empty fetch is asked again, not read as the end (hq ADR 0190)
A fetch on a context without a deadline waits the client's own while and reports the deadline
passed — the client's, not ours — and the loop read it as "stop": every idle build agent exited
clean every half minute and was restarted by its supervisor, a crash loop with nothing in the log
to say why. Only our own context ending ends the machine; an empty fetch, however it is reported,
is asked again.
2026-10-03 11:41:23 +02:00
jochen c294949f2a A worker of the wrong type on a history-keeping stream is re-made to deliver from now on, never from the start (hq issue 207)
Left for a hand, the hand re-made it with the server's default — everything the stream holds — and
on 2026-10-03 that replayed every build ask since 1 October into the catalogue. Re-made with
deliver-new instead: nothing acknowledged comes back; what was in flight is said and asked again.
2026-10-03 11:07:29 +02:00
jochen 28853a251b The build seat's holder follows the controller that defines its worker (hq issue 206)
A plan is ordered by artifacts and says nothing about what must be running before what (ADR 0162);
on 2026-10-03 that put the build machine in tier 0 and the controller in tier 1, and the new build
machine could not bind the worker the old controller had defined. One running order enters the
graph, named as its own edge: a module claiming the build seat follows the control plane, and the
built-by edge from the control plane to that holder yields to it — the controller is built by
whichever build machine is running, as the runtime image always was. The edge orders a plan and
never widens it, like built-by.
2026-10-03 04:04:41 +02:00
jochen 76a8b8df9e The controller owns a worker's shape, type included: one of the wrong type is re-made on a work queue (hq issue 206)
A holder built for a pull worker cannot bind a push one — `cannot pull subscribe to push based
consumer` — and on 2026-10-03 the build machine rolled before the controller that would have
redefined its worker, restarted on that for an hour, and nothing could build the controller that
would have ended it. The server cannot change a consumer's type in place, so the assertion re-makes
one of the wrong type: on a work queue nothing is lost, because what was acknowledged is gone from the
stream and what was not is delivered again from the start. On a stream that keeps its history it is
said and left, since a re-made consumer replays what this one acknowledged (issue 156), and that is a
person's call. Proven against a real bus: a push worker with one ask acknowledged and two pending is
re-made as pull, a pull subscription binds, and takes exactly the two.
2026-10-03 04:04:41 +02:00
jochen f86f6a74f0 A declaration is numbered when it is composed, and a send is recorded even by a sender being replaced (hq issue 204)
On 2026-10-02 a runtime assigned and applied on two machines was undone two seconds later by a
declaration that had the assignments of a minute earlier. Every path composes from the records at
compose time and holds the machines it sends — but the number went on at SEND time, after
composing, so a declaration composed before an assignment changed and sent after a newer one
carried the higher number, and the host, which rightly refuses a lower number, took the older
content as the mesh's newest word. The record of that send was never written either: it is written
after the declaration is away, on the sender's context, and the controller sending it was being
replaced in that very second — status read "applied, current" over a machine just told otherwise.

Now the number is taken before the composition reads anything, in every path, so what was composed
earlier is numbered lower however late it goes out and the host's refusal does what it is for; and
what was sent is written down on a context that outlives the sender, bounded, so a dying controller
still records what it told a machine. The `declare` command — a declaration a person sends by hand —
records its send too. Proven: compositions in one order and sends in the other keep the numbers in
composition order; a send is recorded after the sender's context is cancelled.
2026-10-03 04:02:36 +02:00
jochen a3e8a4185b An assignment issues its bus credential, a push refuses one nobody issued, and what reads a secret restarts on it (hq issue 203)
`assign` recorded a module and `push` sealed a random own secret where its bus credential belongs;
the process crash-looped until a person ran `module issue` and pushed again, and the only warning was
one line in a list printed on every push. Now assigning a module that declares a broker secret issues
the credential in the same act — kept when one exists, so re-assigning rotates nothing — and when the
bus cannot be reached from here the assignment says which verb to run. A push never seals a
placeholder in a credential's place: a module whose bus user is unminted is refused by name, with the
verb. The control plane's own user is the installer's, seeded at genesis, which the test now says.

And what reads one of a module's own secrets is restarted when it changes — composed for a container
or daemon that names the secret's path in its volumes, environment or env-files, so a manifest need
not say it: the build machine ran on an hour-old credential because its manifest restarted it on its
environment file alone (issue 206). A scheduled or run-once process is left alone; it reads afresh.
2026-10-03 04:01:17 +02:00
mesh-admin 78de54381b Merge pull request 'A seat's work is shared by its holders: node-build-agent, pulled one ask at a time (hq ADR 0190)' (#228) from feat/a-seats-work-is-shared-by-its-holders into main 2026-10-03 00:53:42 +00:00
jochen ff5ef0ab60 The controller asks the build role that has a holder, and hears both roles' outcomes (hq ADR 0190, the handover)
A controller that asked node-build-agent from its first run would queue every build where nothing
pulls, and the build that registers build-agent — the first holder — would be among them. So the
role is chosen at ask time from the catalogue: the current role when any assigned module claims it,
the retired one while only the builder does, the current one when neither. Outcomes are followed on
both seats, the controller may publish to both, and a build's log is read under whichever role did
it; a machine on the retired role is proven on the bus to take that role's asks. The switch order
is written where the role is named, and the retired half is marked for removal with the seat row.
2026-10-03 02:51:03 +02:00
jochen a5d6a1187c A build machine serves the seat its credential claims (hq ADR 0190, the handover)
After the build role moved to node-build-agent, nothing would hold it until build-agent is
registered — and registering build-agent needs a build outcome that only the running builder
could produce, bound as it was to the old seat by name. One binary, two roles: the seat a machine
serves is the first its credential claims, as the mesh writes the claims beside the credential it
issues (ADR 0159); the old builder keeps draining mesh-build-machine, a build-agent takes
node-build-agent, and what each says about a build goes out as that seat's events, so an outcome
is heard where the asker of that seat listens. A credential naming no claim serves the current role.
2026-10-03 02:47:49 +02:00
mesh-admin 06d0a4bfe8 Merge pull request 'A changed jail filter restarts fail2ban' (#231) from jschoubben/jail-filter-restart into main 2026-10-02 21:28:32 +00:00
jschoubben d293a0deaf A changed jail filter restarts fail2ban
fail2ban restarts when the composed jail file changes, and each filter is
a file of its own, so a module that changed only its failregex left the
running jail on the old pattern. The jail file now names each filter's
digest.
2026-10-02 23:28:25 +02:00
mesh-admin 11a44e1ec4 Merge pull request 'A resolved manifest keeps a built reference in the store's own form, never the address it was reached by (hq ADR 0155)' (#230) from fix/a-resolved-reference-is-kept-not-routed into main 2026-10-02 21:14:06 +00:00
jochen 28c7f05b39 A resolved manifest keeps a built reference in the store's own form, never the address it was reached by (hq ADR 0155)
The first bundle resolved on the mesh carried the store's host in bundles[0].source, and registration
refused node-tools as naming an installation — rightly. The build record already keeps the
store-relative form; the resolved manifest now keeps the same for bundles and archive resources,
and composition routes it through the store a machine reaches, as it already did for a kept one.
2026-10-02 23:13:22 +02:00
mesh-admin 93a0c6202e Merge pull request 'The bus is never public: the broker port is no longer a foundation opening (hq ADR 0169)' (#229) from jschoubben/the-bus-is-never-public into main 2026-10-02 20:52:28 +00:00
jschoubben 7d82751862 The bus is never public: the broker port is no longer a foundation opening
The controller widened the bus's from-mesh port to from-anywhere on the
broker's host so a machine could enrol before it had a tunnel. ADR 0169
has machines join through the tunnel and decides the bus is never public;
every live bus connection already comes from the mesh.
2026-10-02 22:52:22 +02:00
jochen 905f3363c9 Two machines holding the build role share one queue, and neither is handed an ask while busy (hq ADR 0190)
Against a real bus: three asks, two machines; each takes one, the third waits until one is free
and then goes to that one; a machine that stops leaves nothing taken twice. The redelivery of an
ask a dead machine held is the ack wait's, proven by the hand-back test beside this one.

And the order a live mesh switches over in, written where the role is named: queued builds first,
then this controller, then build-agent assigned where machines build, then the builder and the old
seat's stream forgotten.
2026-10-02 22:35:39 +02:00
jochen 9f9d9b3b25 A seat's holders pull one ask at a time from one shared worker (hq ADR 0190, issue 186)
The worker a holder bound was a push consumer in a queue group with one ask in flight: right for
one holder, and with two it would still be a queue of one — the server hands a pushed ask to
whichever subscriber it picks, busy or not, and the in-flight cap is per consumer, not per holder.
Now the worker is pulled: every machine holding the seat binds the same durable and fetches one
ask when it has finished the last, so an idle machine is the one that takes the next, the asks in
flight are bounded by the holders working, and nothing is delivered that nobody asked for — which
is also what ended the race issue 186 describes. A holder's grants trade the delivery subject for
MSG.NEXT on the worker; the ack grant and the heartbeat that keeps a long build alive stay.

Proven against a real bus: the build round trip, a backlog taken by a machine that arrives later,
and work handed back by one machine coming round again.
2026-10-02 22:34:44 +02:00
jochen bde4b61b3b The build role is the node-scoped seat node-build-agent, and its work is shared by every holder (hq ADR 0190)
One build machine built everything, in a queue of one, because the seat was mesh-scoped and a
mesh seat has one holder. ADR 0190 makes building a node role: node-build-agent, held on every
machine that builds, with the work asked of the role and taken by whichever holder is idle. The
work subject of a node-scoped seat carries no node — that token is for a seat's tools, asked of
one machine (design 33 §4) — so holders on several machines read one queue; a test now says so.

The retired mesh-build-machine row stays while the builder module's registered manifest claims
it: a claim to a seat the mesh no longer defines is refused, and the machine holding it would be
unresolvable until build-agent replaces it. Removed once no manifest claims it.

The installer's genesis template (in the host's repository) still grants the controller the old
seat's subjects; its test here says so until that template names node-build-agent.
2026-10-02 22:31:55 +02:00
mesh-admin d07018f3c5 Merge pull request 'WP3: a TypeScript bundle carries what it runs with, the runtime's credential belongs to its account, and the gate refuses spreading not standing (hq to-be 38)' (#226) from feat/wp3-node-tools into main 2026-10-02 19:57:21 +00:00
jochen 729a5f9e6c The gate refuses the old tool-container pattern spreading, not a rebuild of what already stands (hq to-be 38 WP2.4, amended by WP3)
Once the runtime module is registered, a module serving tools from a container built on the
runtime's image is refused at registration — as written, including every rebuild of the thirty-odd
modules already in that shape, from the day the runtime arrives until WP4 onward moves each one.
That would stop the catalogue's whole pipeline to make a point the record already makes. Now a
module already registered in that shape — judged from the manifest the catalogue holds and what
its newest build stood on, the same two things a new registration is judged by — is rebuilt as
before; a module new to the catalogue in that shape, or one that had moved to a bundle and comes
back, is refused naming the record.
2026-10-02 21:44:44 +02:00
jochen 773b561f5e The runtime's credential belongs to the account it runs as (hq to-be 38 WP3)
The runtime's process is composed `user: <account>` where the node has one, and its broker file was
root's at 0600: a credential the process could not read. Composed in the declaration rather than
said in the manifest, because a manifest cannot say ${machine:account} safely — a node with no
account has nothing to resolve it to — and there the runtime runs as root and the file stays root's.
2026-10-02 21:44:44 +02:00
jochen ca7e81e964 A TypeScript bundle carries what it runs with: the toolchain's runtime directory is copied into it (hq to-be 38 WP3, ADR 0188)
A bundle that compiled was not yet a bundle that ran. The compiler resolved `import "nats"` from
the toolchain image's own node_modules and the pack took only what the compiler wrote, so what a
machine unpacked could not find a single dependency — and Node would have read the bare `.js` as
CommonJS besides. No TypeScript bundle had run live to show it; the runtime's own is the first that
must. A toolchain now names a Dependencies directory in its image, copied whole into the output's
root after the compile by a second run in the same image: for TypeScript /app/runtime, which the
runtime's image puts a `"type": "module"` package.json and its pruned node_modules at. An older
image without it fails the build by name rather than packing a bundle that starts nowhere. The
SDK's and the runtime's dependencies, nothing module-specific yet: a skeleton, by ADR 0188 §5.
2026-10-02 21:44:44 +02:00
mesh-admin 08bb56f1d3 Merge pull request 'The controller composes one tool runtime per node: its principal, the bundles, its process, and the gate (hq ADR 0175, to-be 38 WP2)' (#223) from feat/the-operators-machine into main 2026-10-02 19:27:52 +00:00
mesh-admin 1a44d281c2 Merge pull request 'node show cites ADR 0180 for a removed front end (hq ADR 0186)' (#224) from fix/a-ban-list-never-holds-a-neighbour into main 2026-10-02 16:47:25 +00:00
jschoubben 1c8fe65601 node show cites ADR 0180 for a removed front end (hq ADR 0186)
Another session took 0175 while that record was in review; the line printed on every converged
machine was pointing at an unrelated decision.
2026-10-02 18:42:16 +02:00
jochen 8eb6c3e94a A tool container on the runtime's image is refused once the runtime is registered (hq ADR 0175, to-be 38 WP2.4)
A module declaring tools, a container, and a build on mesh-tools' runtime image is a container whose
purpose is serving tools — the pattern the node's tool runtime retires. Once node-tools is in the
catalogue, registering one is refused by name, with the record that says why; before, it is accepted
as it always was, so a mesh converts in the design's order and nothing is refused before there is
anything to move to. This is the mechanism that keeps the old pattern from returning by habit.

Judged from a repository manifest's own build.on, and for a built manifest — which carries no build
— from what its build stood on, now recorded beside the commit as part of a module's provenance.
2026-10-02 18:30:14 +02:00
jochen 9d4d847dc4 The machine runs one tool runtime, loading every delivered bundle (hq ADR 0175, to-be 38 WP2.3)
Where node-tools is in a node's set, the declaration ends with one process: the runtime module's own
bundle, run from its one entrypoint by its language's interpreter, told in MESH_TOOL_MODULES every
<module>=<file> the machine's bundles load, where its credential is (the module's own broker secret
as this node places it), and — on a machine with an operator account — who the operator is, running
as that account so a tool that needs root can escalate as the operator would. Restarted when any
bundle it loads or the credential changes. A machine with no account runs it as root without the two
operator words; a machine without the runtime is sent nothing new.

A bundle says which of its entrypoints the runtime LOADS (`loads`), because one bundle may carry a
daemon beside its tools and importing the daemon into the runtime would start it there; absent, a
module declaring tools has every entrypoint loaded. And the TypeScript toolchain is rooted at the
module, so an entrypoint lands at the path it is named by — the runtime loading bundles by their
declared paths is what made the compiler's common-directory default visible.
2026-10-02 18:26:54 +02:00
mesh-admin bea1a1c513 Merge pull request 'A control plane behind its seat's row serves what it can (hq ADR 0185)' (#222) from fix/a-service-asked-to-run-is-still-running into main 2026-10-02 16:21:55 +00:00
jochen b1df688c62 A module's tools bundle reaches the machine as an archive where the runtime runs (hq ADR 0175, to-be 38 WP2.2)
The resolved manifest now carries what the build compiled — each bundle's source, digest, language
and entrypoints — because a tools bundle is named by no resource of the module's own: the node's
runtime loads it, and until this the mesh held no trace of the one artifact that runtime needs. A
repository manifest that writes `bundles` beside its build is refused: the mesh derives it.

Where the runtime module is in a node's set, every assigned module's bundle with entrypoints is
composed as an archive under the mesh's own directory, routed through the artifact store like any
image or archive the mesh built. A bundle without entrypoints is run rather than loaded and is
delivered by the process that runs it. A node without the runtime is sent exactly what it was.
2026-10-02 18:19:10 +02:00
jschoubben 21d38c9b0e A control plane behind its seat's row serves what it can (hq ADR 0185)
One verb in the row that this binary cannot run aborted the start, and a stale push that put an
older control plane back took the whole mesh off the bus for ten minutes — recoverable only by a
person running the binary outside its service, because the push that repairs it is one of the verbs
that had stopped being served. Now the verbs it knows are served, the ones it does not answer the
reason, and the start names them once.
2026-10-02 18:16:24 +02:00
jochen 1f86ed4135 One runtime principal per node carries every assigned module's tools (hq ADR 0175, to-be 38 WP2.1)
Where the node-tools module is assigned, the machine's bus user list gains one principal of kind
node-tools in place of that module's own: it may subscribe every carried module's tool namespace
and every held seat's verbs on its node, read and follow every membership on its node, call any
tool anywhere, answer what it is asked — and consume nothing, because tools are what it runs.
Every other module keeps its own principal, so a module still serving tools from its container
holds its own credential until it moves.

Named exactly as the module it stands for, so `module issue` and `rollout mint` deliver its
credential through the path a module's already takes, into node-tools' own `broker` secret. The
runtime module's name is one constant in each of the broker and catalogue packages, held to one
string by the agreement test, because a rule turns on it.
2026-10-02 18:16:14 +02:00
mesh-admin 689dd060b0 Merge pull request 'A jail's pattern names &lt;HOST&gt; once, and is refused by name when it does not (hq ADR 0179)' (#221) from fix/a-jails-pattern-names-the-host-once into main 2026-10-02 15:32:14 +00:00
jschoubben 99c4c4ef04 A jail's pattern names <HOST> once, and is refused by name when it does not (hq ADR 0179)
fail2ban expands <HOST> into a named capture group, so a pattern naming it twice is a duplicate
group name: the daemon refuses its whole configuration and exits, and the machine keeps no bans at
all — for every jail, not the one at fault. Hit live on the control node the day the jails shipped.
A jail with no name, no pattern, or a name another of the module's jails took is refused too.
2026-10-02 17:25:34 +02:00
mesh-admin e5e1666f91 Merge pull request 'The intrusion-prevention seat serves status, banned, ban and unban; the proxy logs a refused name (hq ADR 0179)' (#219) from feat/the-intrusion-seat-serves-its-verbs into main 2026-10-02 15:15:41 +00:00
jschoubben bd58d1c3ef The intrusion-prevention seat serves status, banned, ban and unban; the proxy logs a refused name (hq ADR 0179)
Every holder of node-intrusion-prevention owes the four verbs, as the packet filter's holder owes
its three (ADR 0170). The proxy says in its log when a name this mesh does not serve is asked for,
with the asking address last, so its own jail can read it.
2026-10-02 17:02:49 +02:00
mesh-admin d134c89a7c Merge pull request 'The service manager is a node seat with the unit verbs, and a user shape names the account (hq ADR 0176, 0177)' (#218) from feat/the-operators-machine into main 2026-10-02 14:58:50 +00:00
jochen a9307d9f33 The controller seat gains a generic verb: command runs one line of the binary and answers what it printed
Beside the named verbs, `command` takes a command line as the controller's
own shell would — `node account g14 jochen`, `node show ace`, `module list` —
splits it as a shell does (quotes group, backslash escapes, nothing expanded)
and runs it in this binary like every other verb. The named verbs keep their
schemas; this is the whole binary, added because the operator decided any
node may call any tool (hq ADR 0175) and a verb per command was the only
thing keeping the rest behind a shell on the control node. ADR 0154 carries
the dated note. Tests: a plain line, quoted words, an empty line and an
unclosed quote refused.
2026-10-02 16:53:27 +02:00
jochen 5eb1c9c2b7 The service manager is a node seat with the unit verbs, and a user shape names the account (hq ADR 0176, 0177)
node-service-manager joins the mesh's own seats, node-scoped, serving eight
verbs — units, status, start, stop, restart, enable, disable, journal — each
with a schema that takes an optional scope, "system" or the operator
account's "user" manager. Sixteen seats now; the count test says so and names
the record.

${machine:account} resolves in a resource's `name` and `user` as it already
did in path, owner and content: the shell module makes the operator's account
its holder's login shell through the `user` shape, and the desktop's watchers
run as that account through a user-scoped unit (host change alongside).
Neither can name the person. A machine with no account refuses by name.
2026-10-02 16:48:16 +02:00
mesh-admin 37b8edeec6 Merge pull request 'node show says a found firewall that was uninstalled is removed (hq ADR 0175)' (#217) from feat/the-found-front-end-is-uninstalled into main 2026-10-02 14:29:02 +00:00
jschoubben 424406b2d6 node show says a found firewall that was uninstalled is removed (hq ADR 0175) 2026-10-02 16:27:39 +02:00
mesh-admin 90455c61f6 Merge pull request 'The example images build from the Go the manifest pins' (#216) from jschoubben/example-images-go-base into main 2026-10-02 13:56:22 +00:00
jschoubben 1f6793cf0c The example images build from the Go the manifest pins
Four example Dockerfiles named golang:1.25 while go.mod requires 1.26;
the control plane's image takes GO_BASE from the manifest and these did
not, so a build that could not fetch a newer toolchain failed at go mod
download (found running the lab through the mesh).
2026-10-02 15:52:25 +02:00
mesh-admin 6ef522fbda Merge pull request 'Cite hq ADR 0170, not 0169: the firewall seat's record was renumbered' (#215) from fix/adr-0170-cited into main 2026-10-02 12:53:13 +00:00
jschoubben 46f65c12a0 Cite hq ADR 0170, not 0169: the firewall seat's record was renumbered after a collision on hq main 2026-10-02 14:52:24 +02:00
mesh-admin 8f7c02d77a Merge pull request 'The virtualisation capability grants the lab its daemon's socket (hq ADR 0172)' (#214) from jschoubben/the-lab-is-a-module into main 2026-10-02 12:47:45 +00:00
jschoubben a637df01ea The virtualisation capability grants the lab its daemon's socket
The lab raises machines on the virtualisation daemon, and a module may
mount a machine's socket only through the capability that grants it
(novox/hq ADR 0172). Also brings the resolver's tests to the setting
dnsmasq's listen addresses now come from, and to a module left out
rather than refused.
2026-10-02 14:46:17 +02:00
mesh-admin 252eb786a7 Merge pull request 'A filter module's own filter file counts as declared for a mount (hq ADR 0169)' (#213) from feat/the-firewall-seat-serves-its-verbs into main 2026-10-02 12:05:09 +00:00
jschoubben 9d13b0593b A filter module's own filter file counts as declared for a mount (hq ADR 0169)
The nftables module's runtime mounts the file filtering.into names, to reload
the mesh's table; the mount check knew every other declaration of a path and
not this one, and the module's first build was refused for it.
2026-10-02 14:04:35 +02:00
mesh-admin 30d548a762 Merge pull request 'The node-packet-filter seat serves rules, reload and remove (hq ADR 0169)' (#212) from feat/the-firewall-seat-serves-its-verbs into main 2026-10-02 11:28:57 +00:00
jschoubben 550e4c6acb The node-packet-filter seat serves rules, reload and remove (hq ADR 0169)
What a person asks a machine's packet filter whatever filter answers: the
rules as enforced, reload the mesh's own, remove one rule set the mesh did
not write — named as the host reports it under ADR 0168. Every holder serves
all three; a running mesh widens its seat row at the next controller start.
2026-10-02 13:27:04 +02:00
mesh-admin 1587fd97f9 Merge pull request 'The mesh says what filters a converged machine: filters kept per node, shown, named by status, previewed with fates (hq ADR 0168)' (#211) from feat/one-thing-filters-a-converged-machine into main 2026-10-02 10:03:01 +00:00
jschoubben b5df244096 The mesh says what filters a converged machine: filters kept per node, shown by node show, named by status, and previewed with their fates (hq ADR 0168)
A host reports every table and chain that refuses traffic with its owner,
and a converged machine's found firewall's state. The controller keeps both
on the node's record (migration 0054), shows them on node show, names every
converged machine something other than the mesh filters in status — text
and JSON, and such a machine is not well — and the converge preview lists
what filters the machine with the fate of each: retired with the front end,
left as the runtime's, left as a ban, or left in force and not the mesh's.
What was invisible for eleven hours (issues 144, 145) is said by name.
2026-10-02 12:00:12 +02:00
mesh-admin db47bb68e9 Merge pull request 'The controller's tools can set an assignment's settings (hq issue 198)' (#210) from jschoubben/settings-verb into main 2026-10-02 09:42:08 +00:00
jschoubben db84142d38 The controller's tools can set an assignment's settings
Per-machine and mesh-wide settings could be set only from the
controller's command line. The settings verb runs settings set|clear,
passing the values inline, which the command now accepts as well as a
file (novox/hq issue 198).
2026-10-02 11:41:57 +02:00
mesh-admin c9204591bf Merge pull request 'The hub relays the mesh passing through it (hq issue 196)' (#209) from jschoubben/the-hub-relays-the-mesh into main 2026-10-02 09:17:17 +00:00
jschoubben e8e707e013 The hub relays the mesh passing through it
The forward chain judged a packet relayed from one machine of the mesh
to another by this machine's own published ports, so two machines behind
the hub reached each other only on ports the hub published for itself
(novox/hq issue 196). In and out on the tunnel is now accepted, and the
machine it is for filters it.
2026-10-02 11:17:09 +02:00
mesh-admin 0c003774ba Merge pull request 'A take acts on the preview it showed; a setting is judged where it is stored; a kept network and a minted secret are said (hq ADR 0163)' (#206) from feat/a-take-is-a-comparison-the-rest into main 2026-10-02 09:04:05 +00:00
jschoubben fbc3d320ea A take acts on the preview it showed; a setting is judged where it is stored; a kept network and a minted secret are said (hq ADR 0163)
take ends its preview with a digest and --yes names it, as the flip does; a
changed preview or an account older than the flip allows is refused. A module
the machine holds nothing for has nothing to compare, and --yes suffices. A
published port's reach is said as the machine reported it. Every secret the
module holds on the machine is listed with where it came from, and one the
mesh minted for a service whose data was found refuses unless --mint names it.

One judgement of a module's settings against its definition, in the catalogue:
settings set refuses what cannot compose or reaches nothing, naming node,
module, layer and key; Compose leaves out a module whose definition moved
under a stored setting, the envelope says so (left_out), plan and push say it
by name, and the machine is told everything else. A stray setting no longer
refuses the whole machine where it is read (issue 096).

The per-machine setting networks keeps a found network for a taken container,
on an adopted machine only; the container's declaration carries it and the
preview names it (rule 4).
2026-10-02 11:01:09 +02:00
mesh-admin cf495e315f Merge pull request 'The proxy is told its routes and the mesh on the bus, and serves internal names to the mesh only (hq issue 191, ADR 0167)' (#207) from jschoubben/an-internal-only-route into main 2026-10-02 07:42:45 +00:00
mesh-admin 86bc1fad2c Merge pull request 'The controller's tools can issue a module its bus account (hq issue 191)' (#208) from jschoubben/the-controller-issues-a-module-account into main 2026-10-02 00:22:23 +00:00
jschoubben c9eba5f3b8 The controller's tools can issue a module its bus account
A module's account was mintable only from the controller's command line,
so a rollout that gave a module one could not be finished through the
mesh's own tools (novox/hq issue 191). The issue verb runs module issue
for a module on a machine; the caller pushes the machine after.
2026-10-02 01:55:06 +02:00
100 changed files with 5004 additions and 690 deletions
+4 -4
View File
@@ -59,7 +59,7 @@ PROVISIONER_IMAGE ?= mesh-provision-postgres:$(VERSION)
PROVISIONER_DEV_TAG ?= mesh-provision-postgres:development
provisioner-image:
docker build -f examples/postgres-provisioner/Dockerfile \
docker build --build-arg GO_BASE=$(GO_BASE) -f examples/postgres-provisioner/Dockerfile \
-t $(PROVISIONER_IMAGE) -t $(PROVISIONER_DEV_TAG) .
@echo
@docker image inspect $(PROVISIONER_IMAGE) --format 'built {{.RepoTags}} {{.Size}} bytes'
@@ -70,7 +70,7 @@ OBJECTSTORE_IMAGE ?= mesh-provision-objectstore:$(VERSION)
OBJECTSTORE_DEV_TAG ?= mesh-provision-objectstore:development
objectstore-image:
docker build -f examples/objectstore-provisioner/Dockerfile \
docker build --build-arg GO_BASE=$(GO_BASE) -f examples/objectstore-provisioner/Dockerfile \
-t $(OBJECTSTORE_IMAGE) -t $(OBJECTSTORE_DEV_TAG) .
@echo
@docker image inspect $(OBJECTSTORE_IMAGE) --format 'built {{.RepoTags}} {{.Size}} bytes'
@@ -81,7 +81,7 @@ REDIS_PROVISIONER_IMAGE ?= mesh-provision-redis:$(VERSION)
REDIS_PROVISIONER_DEV_TAG ?= mesh-provision-redis:development
redis-provisioner-image:
docker build -f examples/redis-provisioner/Dockerfile \
docker build --build-arg GO_BASE=$(GO_BASE) -f examples/redis-provisioner/Dockerfile \
-t $(REDIS_PROVISIONER_IMAGE) -t $(REDIS_PROVISIONER_DEV_TAG) .
@echo
@docker image inspect $(REDIS_PROVISIONER_IMAGE) --format 'built {{.RepoTags}} {{.Size}} bytes'
@@ -91,7 +91,7 @@ PROXY_IMAGE ?= mesh-route-proxy:$(VERSION)
PROXY_DEV_TAG ?= mesh-route-proxy:development
proxy-image:
docker build -f examples/route-proxy/Dockerfile -t $(PROXY_IMAGE) -t $(PROXY_DEV_TAG) .
docker build --build-arg GO_BASE=$(GO_BASE) -f examples/route-proxy/Dockerfile -t $(PROXY_IMAGE) -t $(PROXY_DEV_TAG) .
@echo
@docker image inspect $(PROXY_IMAGE) --format 'built {{.RepoTags}} {{.Size}} bytes'
+21 -1
View File
@@ -137,7 +137,12 @@ func takeWorkFrom(credential Credential, on string) (link.BuildMachine, error) {
if err != nil {
return nil, err
}
return link.MachineOverNATS(js, on), nil
// **The seat this machine serves is the one its credential claims** (novox/hq ADR 0190, the
// handover): the mesh issues a build machine's credential naming the seat its module claims,
// and one binary serves the old role as `builder` and the new as `build-agent` from that alone.
seat := link.BuildSeatClaimed(credential.seatsClaimed())
fmt.Fprintf(os.Stderr, "taking build work as a holder of %s\n", seat)
return link.MachineOverNATSOn(js, on, seat), nil
}
// answer does one build and says what happened, whichever way it went.
@@ -453,6 +458,21 @@ type Credential struct {
// as two fields and this machine joins them once, here, to dial.
User string `json:"user,omitempty"`
Password string `json:"password,omitempty"`
// Claims are the seats the module this credential was issued for claims, as the mesh writes
// them beside the credential (novox/hq ADR 0159). The first is the build role this machine
// serves; a credential naming none is from before claims travelled in it.
Claims []struct {
Seat string `json:"seat"`
} `json:"claims,omitempty"`
}
// seatsClaimed is the seats the credential names, in order.
func (c Credential) seatsClaimed() []string {
out := make([]string, 0, len(c.Claims))
for _, claim := range c.Claims {
out = append(out, claim.Seat)
}
return out
}
// onTheNewBus is whether a credential is for the bus being built: its address says so, and the
+27
View File
@@ -0,0 +1,27 @@
package main
import (
"encoding/json"
"testing"
"github.com/novox/mesh-controller/internal/link"
)
// The seat a build machine serves comes from its credential (novox/hq ADR 0190 handover).
func TestTheCredentialSaysWhichBuildRoleThisMachineServes(t *testing.T) {
var held Credential
if err := json.Unmarshal([]byte(`{"url":"nats://bus:4222","user":"anchor.builder","password":"x",
"claims":[{"seat":"mesh-build-machine","scope":"mesh","serves":[]}]}`), &held); err != nil {
t.Fatal(err)
}
if got := link.BuildSeatClaimed(held.seatsClaimed()); got != "mesh-build-machine" {
t.Errorf("the old builder's credential serves %q", got)
}
var bare Credential
if err := json.Unmarshal([]byte(`{"url":"nats://bus:4222","user":"anchor.build-agent","password":"x"}`), &bare); err != nil {
t.Fatal(err)
}
if got := link.BuildSeatClaimed(bare.seatsClaimed()); got != link.TheBuildMachine {
t.Errorf("a credential without claims serves %q, want %s", got, link.TheBuildMachine)
}
}
+38
View File
@@ -3,6 +3,7 @@ package main
import (
"context"
"fmt"
"github.com/novox/mesh-controller/internal/broker"
"sort"
"strings"
@@ -67,6 +68,13 @@ func assign(ctx context.Context, open *stores, node, module string) (string, err
for _, line := range settled {
said += "\n " + line
}
// Its bus credential, in the same act (novox/hq issue 203): an assignment pushed before its
// credential exists delivers a process that cannot authenticate and crash-loops until somebody
// runs a second verb and a second push. Issued here when the module speaks on the bus and has
// no credential yet; kept when it has one, so re-assigning rotates nothing.
if line := issueOnAssign(ctx, open, node, module); line != "" {
said += "\n " + line
}
plan, _, err := planFor(ctx, open, node)
if err != nil {
// Kept, and still refused. Both halves are the answer, and the rest of the mesh is still
@@ -152,3 +160,33 @@ func blockedElsewhere(ctx context.Context, open *stores, except string) string {
out.WriteString("\nThis may or may not be what just changed — it is what is true now.")
return out.String()
}
// issueOnAssign gives a newly assigned module its bus credential, the way `module issue` does, and
// says what it did in one line. Nothing for a module that declares no broker secret; nothing for one
// whose user is already minted (a credential is rotated on purpose, never by re-assigning); and when
// the bus cannot be reached from here, the line names the verb and the push that would refuse the
// module until it is run — never a silent placeholder (novox/hq issue 203).
func issueOnAssign(ctx context.Context, open *stores, node, module string) string {
inv := open.inventory
shelf, err := inv.Catalogue(ctx)
if err != nil {
return ""
}
m, known := shelf[module]
if !known || mayIssue(m) != nil {
return ""
}
user := broker.Principal{Kind: broker.KindModule, Node: node, Module: module}.Username()
if _, minted, err := inv.BusUserHash(ctx, user); err != nil || minted {
return ""
}
busAddress, err := broker.BusAddress()
if err == nil {
err = issueOnTheNewBus(ctx, inv, m, node, busAddress)
}
if err != nil {
return fmt.Sprintf("its bus credential is not issued (%v): `module issue %s --node %s` first — "+
"`push %s` refuses to send %s until it is", err, module, node, node, module)
}
return fmt.Sprintf("its bus credential is issued and sealed to %s, and arrives with the push", node)
}
+6
View File
@@ -175,6 +175,12 @@ func TestTheControlPlaneIsToldWhereTheNodePutTheStoreAndTheBroker(t *testing.T)
Guards: []int{15672},
Resources: []map[string]any{{"id": "server", "type": "container", "name": "mesh-broker",
"ports": []any{"5671:5671", "5672:5672", "127.0.0.1:15672:15672"}, "image": "mq@" + aDigest}}})
// The control plane's own bus user is the installer's, seeded at genesis before the controller
// runs (SeedBusUser); without it a push now refuses the credential nobody issued (issue 203).
if err := open.inventory.SeedBusUser(ctx, inventory.BusUser{Username: "anchor.mesh-controller",
Kind: inventory.BusController, Node: "anchor", Module: "mesh-controller"}, "bootstrap"); err != nil {
t.Fatal(err)
}
if _, err := assign(ctx, open, "anchor", "mesh-controller"); err != nil {
t.Fatal(err)
}
+104 -1
View File
@@ -95,11 +95,22 @@ func reportsReaching(t *testing.T, open *stores, reachable []link.Reach, held ..
// filter is not sent to one that has not. The anchor reports one, as a real host does; this
// fixture lacked it from 2026-09-28 and nothing ran the test (issue 177).
Outward: []string{"eth0"},
// And what filters it (ADR 0168): its front end, the runtime's own, and a chain a
// predecessor left in the runtime's user chain.
Filters: anchorFilters,
}); err != nil {
t.Fatal(err)
}
}
// anchorFilters is what the adopted anchor says filters it: ufw's chains, the runtime's, and a
// predecessor's chain the mesh did not write.
var anchorFilters = []link.Filter{
{Where: "table ip filter, chain ufw-reject-input", Owner: "found-firewall", Refuses: "reject"},
{Where: "table ip filter, chain DOCKER", Owner: "runtime", Refuses: `iifname != "docker0" oifname "docker0" drop`},
{Where: "table ip filter, chain DOCKER-USER", Owner: "other", Refuses: `iifname "eth0" tcp dport 6000 drop`},
}
var (
heldContainer = link.Held{ID: "hello-web.server", Module: "hello-web", Kind: "container",
Target: "hello-web", Since: time.Now()}
@@ -143,7 +154,24 @@ func TestTheFlipIsRefusedWhileAFoundContainerIsHeld(t *testing.T) {
func TestTakingNamesWhatItReplaces(t *testing.T) {
open, _ := anAdoptedAnchor(t)
reportsHolding(t, open, heldContainer, heldFile)
said, err := take(t.Context(), open, "anchor", "hello-web", takeOptions{Yes: true})
ctx := t.Context()
// The machine holds something for the module, so the take acts on the preview the operator
// saw and names its digest (novox/hq ADR 0163).
preview, err := take(ctx, open, "anchor", "hello-web", takeOptions{})
if err != nil {
t.Fatal(err)
}
saw := takeDigestIn(t, preview)
if !strings.Contains(preview, "nothing taken; `take anchor hello-web --yes "+saw+"`") {
t.Fatalf("the preview does not say how to act on it:\n%s", preview)
}
if taken, _ := open.inventory.Taken(ctx, "anchor"); len(taken) != 0 {
t.Fatal("the preview took something")
}
if _, err := take(ctx, open, "anchor", "hello-web", takeOptions{Yes: true}); err == nil || !strings.Contains(err.Error(), "name its digest") {
t.Fatalf("--yes without the digest was not refused: %v", err)
}
said, err := take(ctx, open, "anchor", "hello-web", takeOptions{Yes: true, Digest: saw})
if err != nil {
t.Fatal(err)
}
@@ -153,6 +181,67 @@ func TestTakingNamesWhatItReplaces(t *testing.T) {
}
}
// takeDigestIn is the digest a take's preview printed.
func takeDigestIn(t *testing.T, preview string) string {
t.Helper()
for _, line := range strings.Split(preview, "\n") {
if fields := strings.Fields(line); len(fields) == 2 && fields[0] == "preview" {
return fields[1]
}
}
t.Fatalf("the preview printed no digest:\n%s", preview)
return ""
}
// A take acts on the preview the operator saw, and on an account of the machine that is still the
// machine: a changed preview and a stale account refuse (novox/hq ADR 0163, rule 1).
func TestATakeIsRefusedOnAChangedPreviewOrAStaleAccount(t *testing.T) {
open, _ := anAdoptedAnchor(t)
ctx := t.Context()
reportsHolding(t, open, heldContainer, heldFile)
preview, err := take(ctx, open, "anchor", "hello-web", takeOptions{})
if err != nil {
t.Fatal(err)
}
saw := takeDigestIn(t, preview)
// The machine reports again, and what it holds has changed: the found container now carries
// facts the preview never showed.
changed := heldContainer
changed.Facts = map[string]any{"image": "hello:2", "declared_image": "registry.example/hello"}
reportsHolding(t, open, changed, heldFile)
_, err = take(ctx, open, "anchor", "hello-web", takeOptions{Yes: true, Digest: saw})
if err == nil || !strings.Contains(err.Error(), "has changed since preview "+saw) {
t.Fatalf("a changed preview was acted on: %v", err)
}
if taken, _ := open.inventory.Taken(ctx, "anchor"); len(taken) != 0 {
t.Fatal("a refused take took something")
}
// And an account older than the flip allows.
preview, err = take(ctx, open, "anchor", "hello-web", takeOptions{})
if err != nil {
t.Fatal(err)
}
saw = takeDigestIn(t, preview)
saved := reportFreshFor
reportFreshFor = -time.Second
defer func() { reportFreshFor = saved }()
_, err = take(ctx, open, "anchor", "hello-web", takeOptions{Yes: true, Digest: saw})
if err == nil || !strings.Contains(err.Error(), "a take acts only on an account newer than") {
t.Fatalf("a stale account was acted on: %v", err)
}
reportFreshFor = saved
if _, err := take(ctx, open, "anchor", "hello-web", takeOptions{Yes: true, Digest: saw}); err != nil {
t.Fatal(err)
}
// A module the machine holds nothing for has nothing to compare: --yes alone suffices.
if _, err := take(ctx, open, "anchor", "notes", takeOptions{Yes: true}); err == nil {
// notes holds a file, so this one needs the digest too.
t.Fatal("notes holds a found file and was taken without a digest")
}
}
func TestConvergingPreviewsThenChangesAndAdoptingKeepsWhatWasTaken(t *testing.T) {
open, sent := anAdoptedAnchor(t)
ctx := t.Context()
@@ -186,6 +275,20 @@ func TestConvergingPreviewsThenChangesAndAdoptingKeepsWhatWasTaken(t *testing.T)
if strings.Contains(preview, "15672") {
t.Errorf("a loopback listener is in the preview:\n%s", preview)
}
// What filters the machine now, and the fate of each (novox/hq ADR 0168): the predecessor's
// chain is named as not the mesh's and left, so the reader knows before the flip.
for _, want := range []string{
"table ip filter, chain DOCKER-USER",
"NOT THE MESH'S; left in force",
`iifname "eth0" tcp dport 6000 drop`,
"table ip filter, chain ufw-reject-input",
"the found firewall's; retired with it",
"the container runtime's own; left",
} {
if !strings.Contains(preview, want) {
t.Errorf("the preview does not say %q:\n%s", want, preview)
}
}
for _, line := range strings.Split(preview, "\n") {
if strings.Contains(line, "5000") && !strings.Contains(line, "WILL CLOSE") {
t.Errorf("an undeclared published port is not said to close: %s", line)
+299 -23
View File
@@ -29,6 +29,10 @@ func showMode(ctx context.Context, inv *inventory.Inventory, node inventory.Node
if said, err := inv.AdoptionOf(ctx, node.Name); err == nil && len(said.Strays) > 0 {
showStrays(said.Strays)
}
// And what filters it, truthfully (novox/hq ADR 0168): the mesh alone, or not.
if filtering, err := inv.FilteringOf(ctx, node.Name); err == nil {
showFiltering(filtering, false)
}
return nil
}
fmt.Printf(" mode adopted since %s\n",
@@ -72,10 +76,65 @@ func showMode(ctx context.Context, inv *inventory.Inventory, node inventory.Node
}
}
showStrays(said.Strays)
if filtering, err := inv.FilteringOf(ctx, node.Name); err == nil {
showFiltering(filtering, true)
}
fmt.Printf(" as of %s\n", said.At.Local().Format(time.DateTime))
return nil
}
// showFiltering says what filters a machine, with owners (novox/hq ADR 0168), and for a converged
// machine the state of the firewall it was found with. A machine that has not said is not said to
// be filtered by anything.
func showFiltering(f inventory.Filtering, adopted bool) {
if len(f.Filters) == 0 && f.FoundFirewall == nil {
return
}
if fw := f.FoundFirewall; fw != nil && !adopted {
switch {
case fw.Active:
fmt.Printf(" found firewall %s is ACTIVE on this converged machine; the next apply retires it again\n", fw.Kind)
case fw.RetiredBy == "removed":
fmt.Printf(" found firewall %s, removed: the mesh's filter is what filters this machine (novox/hq ADR 0180)\n", fw.Kind)
case fw.RetiredBy == inventory.FilterMesh || fw.RetiredBy == "mesh":
fmt.Printf(" found firewall %s, retired by the mesh; its configuration stays on disk\n", fw.Kind)
case fw.RetiredBy != "":
fmt.Printf(" found firewall %s, found inactive — not by the mesh\n", fw.Kind)
default:
fmt.Printf(" found firewall %s, inactive\n", fw.Kind)
}
}
if len(f.Filters) == 0 {
return
}
if f.Alone() {
fmt.Printf(" filtered by the mesh alone (%s)\n", filterSummary(f.Filters))
return
}
fmt.Printf(" filtered by NOT the mesh alone: %d rule set(s) the mesh did not write refuse traffic here\n", len(f.Others()))
for _, x := range f.Filters {
if x.Owner == inventory.FilterOther || x.Owner == inventory.FilterFoundFirewall {
fmt.Printf(" %-17s %s — %s: %s\n", "", x.Where, x.Owner, x.Refuses)
}
}
fmt.Printf(" %-17s and its own: %s\n", "", filterSummary(f.Filters))
}
// filterSummary counts a machine's filters by owner: "mesh 2, runtime 3, ban 1".
func filterSummary(filters []inventory.Filter) string {
counts := map[string]int{}
for _, x := range filters {
counts[x.Owner]++
}
var parts []string
for _, owner := range []string{inventory.FilterMesh, inventory.FilterRuntime, inventory.FilterBan, inventory.FilterFoundFirewall, inventory.FilterOther} {
if n := counts[owner]; n > 0 {
parts = append(parts, fmt.Sprintf("%s %d", owner, n))
}
}
return strings.Join(parts, ", ")
}
// showStrays says what a machine runs that the mesh neither wrote nor holds (ADR 0163).
func showStrays(strays []inventory.Stray) {
if len(strays) == 0 {
@@ -156,11 +215,25 @@ const DefaultFilter = "nftables"
// take is a module's cutover on an adopted node: the operator's act, done when that module's data
// has moved. From the next push its resources converge there like any other, replacing what the
// node found and holds for it.
//
// **Previewed, and the preview is a comparison** (novox/hq ADR 0163): for every held thing the
// module would replace, what runs beside what the module declares, and the difference; the
// module's secrets on the machine and where each came from; its settings on the machine. Without
// --yes the comparison is printed and nothing changes. `--yes <digest>` cuts over exactly what was
// previewed, the way the flip is confirmed: the preview ends with a digest of what it said, and a
// take naming an older one, or acting on an account of the machine older than the flip allows, is
// refused. A module the machine holds nothing for has nothing to compare, and `--yes` suffices.
// takeOptions is what a take was told about the differences it may pass (novox/hq ADR 0163).
type takeOptions struct {
Yes bool
Yes bool
// Digest is the preview's, named with --yes; required whenever the machine holds something
// for the module.
Digest string
Downgrade bool
Replace map[string]bool
// Mint names the secrets the service shall take a new value for, although the mesh minted
// one and the service already has its own (rule 2).
Mint map[string]bool
}
func take(ctx context.Context, open *stores, node, module string, opts takeOptions) (string, error) {
@@ -179,45 +252,128 @@ func take(ctx context.Context, open *stores, node, module string, opts takeOptio
}
// The comparison first (novox/hq ADR 0163): every held thing the module would replace, beside
// what the module declares, and the differences that refuse unless named.
reported, err := inv.AdoptionOf(ctx, node)
c, err := comparisonFor(ctx, open, node, module)
if err != nil {
return "", err
}
preview, refusals := comparisonOf(reported.Held, module, opts)
preview, refusals, saw := comparisonOf(module, c, opts)
if len(refusals) > 0 {
return "", fmt.Errorf("taking %s on %s is refused:\n %s\n%s", module, node,
strings.Join(refusals, "\n "), preview)
}
holds := len(heldOf(c.reported, module)) > 0
if holds {
preview += "\n preview " + saw
}
if !opts.Yes {
return preview + fmt.Sprintf("\nnothing taken; `take %s %s --yes` cuts it over as previewed", node, module), nil
if !holds {
return preview + fmt.Sprintf("\nnothing taken; `take %s %s --yes` declares it as the mesh's own", node, module), nil
}
return preview + fmt.Sprintf("\nnothing taken; `take %s %s --yes %s` cuts it over as previewed", node, module, saw), nil
}
if holds {
// The take acts on the preview the operator saw, and on an account of the machine that
// is still the machine: the same two refusals the flip makes.
if age := time.Since(c.reported.At); age > reportFreshFor {
return preview, fmt.Errorf("%s last said what it holds %s ago, and a take acts only on "+
"an account newer than %s: run `push %s --wait 2m`, then preview again",
node, age.Round(time.Second), reportFreshFor, node)
}
if opts.Digest == "" {
return preview, fmt.Errorf("taking %s on %s acts on the preview you saw: name its digest, "+
"`take %s %s --yes %s`, once you have read it", module, node, node, module, saw)
}
if opts.Digest != saw {
return preview, fmt.Errorf("what taking %s on %s would replace has changed since preview %s "+
"(it is now %s): read the preview above, and run `take %s %s --yes %s` if it is "+
"what you want", module, node, opts.Digest, saw, node, module, saw)
}
}
if err := inv.Take(ctx, node, module); err != nil {
return "", err
}
said := fmt.Sprintf("%s is taken on %s", module, node)
if preview != "" {
if holds {
said += "; the next push replaces what the node found and holds for it:\n" + preview
}
return said + fmt.Sprintf("\n run `push %s` to cut it over", node), nil
}
// comparison is everything a take puts beside what the module declares: the machine's account of
// what it holds and what is reachable on it, the module's secrets on the machine, its settings
// there, and which found networks a setting keeps for each of its containers (by held id).
type comparison struct {
reported inventory.Adoption
secrets []inventory.SecretState
layers []catalogue.Layer
keeps map[string][]string
// settingsRefused is why the module's settings cannot compose with its definition, when
// they cannot — the module would be left out of the declaration (rule 6).
settingsRefused string
}
func comparisonFor(ctx context.Context, open *stores, node, module string) (comparison, error) {
inv := open.inventory
var c comparison
var err error
if c.reported, err = inv.AdoptionOf(ctx, node); err != nil {
return c, err
}
if c.secrets, err = inv.SecretsOf(ctx, node, module); err != nil {
return c, err
}
if c.layers, err = inv.SettingsFor(ctx, node, module); err != nil {
return c, err
}
shelf, err := inv.Catalogue(ctx)
if err != nil {
return c, err
}
if m, known := shelf[module]; known && len(c.layers) > 0 {
if err := catalogue.JudgeSettings(m, c.layers, true); err != nil {
c.settingsRefused = err.Error()
}
if kept, err := catalogue.KeptNetworks(m, c.layers, true); err == nil && len(kept) > 0 {
c.keeps = map[string][]string{}
for id, networks := range kept {
c.keeps[module+"."+id] = networks
}
}
}
return c, nil
}
// heldOf is what a node holds for one module.
func heldOf(reported inventory.Adoption, module string) []inventory.Held {
var out []inventory.Held
for _, h := range reported.Held {
if h.Module == module {
out = append(out, h)
}
}
return out
}
// comparisonOf is a take's preview: for every held thing of the module, what runs beside what the
// module declares, and the refusals the differences earn unless the take named them
// (novox/hq ADR 0163): an image older than the one running, a declared file that differs from the
// found one. A narrowed port and a shared network are said and not refused.
func comparisonOf(held []inventory.Held, module string, opts takeOptions) (string, []string) {
// module declares; its secrets and its settings on the machine; and the refusals the differences
// earn unless the take named them (novox/hq ADR 0163): an image older than the one running, a
// declared file that differs from the found one, a secret the mesh minted for a service whose data
// was found. A narrowed port and a shared network are said and not refused. The digest is of what
// the preview says, so anything in it changing changes the digest.
func comparisonOf(module string, c comparison, opts takeOptions) (preview string, refusals []string, digest string) {
var b strings.Builder
var refusals []string
held := heldOf(c.reported, module)
foundData := false
for _, h := range held {
if h.Module != module {
continue
if h.Kind == "container" || h.Kind == "directory" {
foundData = true
}
fmt.Fprintf(&b, " %s", heldLine(h))
if h.Kept != "" {
fmt.Fprintf(&b, ", original kept at %s", h.Kept)
}
b.WriteString("\n")
for _, line := range comparisonLines(h) {
for _, line := range comparisonLinesWith(h, c.keeps[h.ID], c.reported) {
fmt.Fprintf(&b, " %s\n", line)
}
f := factsOf(h)
@@ -232,7 +388,52 @@ func comparisonOf(held []inventory.Held, module string, opts takeOptions) (strin
h.Target, h.Target))
}
}
return b.String(), refusals
// The module's secrets on the machine (rule 2 and 3): a service whose data was found already
// has a value for each, so one the mesh minted and nobody accepted refuses unless --mint says
// the service shall take a new one.
for _, sec := range c.secrets {
name := sec.Name
if sec.Local != "" {
name += " (" + sec.Local + ")"
}
what := "own secret"
accept := fmt.Sprintf("`secret accept <node> %s %s`", module, sec.Name)
if !sec.Own() {
what = "secret from " + sec.Provider
accept = fmt.Sprintf("`secret accept <node> %s %s --provider %s`", module, sec.Name, sec.Provider)
if sec.Local != "" {
accept = strings.TrimSuffix(accept, "`") + " --local " + sec.Local + "`"
}
}
switch {
case sec.Origin == inventory.OriginAccepted:
fmt.Fprintf(&b, " %s %s: accepted from a person, carried in as it is\n", what, name)
case opts.Mint[sec.Name]:
fmt.Fprintf(&b, " %s %s: minted by the mesh; the service takes the new value, as --mint said\n", what, name)
case foundData:
fmt.Fprintf(&b, " %s %s: MINTED by the mesh and not accepted — the running service already has one\n", what, name)
refusals = append(refusals, fmt.Sprintf("%s: the mesh minted a value and the service whose data was found "+
"already uses its own; %s carries the existing value in, or `--mint %s` says the service shall take "+
"the new one", name, accept, sec.Name))
default:
fmt.Fprintf(&b, " %s %s: minted by the mesh\n", what, name)
}
}
// And its settings on this machine, composed against its definition (rule 1, rule 6).
for _, layer := range c.layers {
keys := make([]string, 0, len(layer.Values))
for k := range layer.Values {
keys = append(keys, k)
}
sort.Strings(keys)
fmt.Fprintf(&b, " settings from %s: %s\n", layer.From, strings.Join(keys, ", "))
}
if c.settingsRefused != "" {
fmt.Fprintf(&b, " SETTINGS DO NOT COMPOSE with the module's definition, so the push leaves it out: %s\n", c.settingsRefused)
}
preview = strings.TrimRight(b.String(), "\n")
sum := sha256.Sum256([]byte(preview))
return preview, refusals, hex.EncodeToString(sum[:])[:12]
}
// facts is a held thing's facts as the preview reads them.
@@ -286,6 +487,13 @@ func factsOf(h inventory.Held) facts {
// comparisonLines says a held thing's facts the way a person weighs them.
func comparisonLines(h inventory.Held) []string {
return comparisonLinesWith(h, nil, inventory.Adoption{})
}
// comparisonLinesWith is comparisonLines knowing which found networks this machine's setting keeps
// for the container (rule 4) and what the machine reports reachable, so a published port's reach
// is said beside the port (rule 1).
func comparisonLinesWith(h inventory.Held, keeps []string, reported inventory.Adoption) []string {
f := factsOf(h)
var out []string
if f.image != "" || f.declaredImage != "" {
@@ -311,14 +519,46 @@ func comparisonLines(h inventory.Held) []string {
}
sort.Strings(names)
for _, n := range names {
if members := f.networks[n]; len(members) > 0 {
out = append(out, fmt.Sprintf("on the network %s with %s, which may reach it by name and will not once it moves to the module's own network",
members := f.networks[n]
if len(members) == 0 {
continue
}
if slices.Contains(keeps, n) {
out = append(out, fmt.Sprintf("on the network %s with %s — kept by this machine's setting, so they still reach it by name once taken",
n, strings.Join(members, ", ")))
continue
}
out = append(out, fmt.Sprintf("on the network %s with %s, which may reach it by name and will not once it moves to the module's own network"+
" (`settings set %s --node <node>` with {%q: {<container>: [%q]}} keeps it)",
n, strings.Join(members, ", "), h.Module, catalogue.NetworksSetting, n))
}
for _, n := range keeps {
if _, found := f.networks[n]; !found {
out = append(out, fmt.Sprintf("keeps the network %s by this machine's setting, which the found container is not on", n))
}
}
if len(f.ports) > 0 || len(f.declaredPorts) > 0 {
out = append(out, fmt.Sprintf("publishes %s; the module declares %s",
orNone(strings.Join(f.ports, " ")), orNone(strings.Join(f.declaredPorts, " "))))
// How far each published port reaches now, as the machine reported it: the listener the
// runtime publishes for this container. The found firewall's and the guard's rules are
// not read; what they let through is said as what was reported reachable.
var reach []string
for _, r := range reported.Reachable {
if r.By == h.Target && r.Published {
reach = append(reach, fmt.Sprintf("%s:%d (%s, container port %d)", r.Address, r.Port, r.Protocol, r.ContainerPort))
}
}
switch {
case len(reach) > 0:
line := "reachable now at " + strings.Join(reach, ", ")
if reported.Firewall != "" && reported.Firewall != "none" {
line += ", behind the found firewall (" + reported.Firewall + "), whose rules are not read"
}
out = append(out, line)
case len(f.ports) > 0 && len(reported.Reachable) > 0:
out = append(out, "not reported reachable on the machine")
}
}
if len(f.mounts) > 0 || len(f.declaredVolumes) > 0 {
out = append(out, fmt.Sprintf("mounts %s; the module declares %s",
@@ -480,7 +720,11 @@ func converge(ctx context.Context, open *stores, node string, yes bool, digest s
}
derived := derivedFilter{rules: rules, foundation: with.Foundation, mesh: with.Mesh,
outward: plan.PublicDomain != "", outwardLinks: with.OutwardLinks}
preview, saw := previewOf(node, reported, derived, plan, taken, filter, runs[filter])
filtering, err := inv.FilteringOf(ctx, node)
if err != nil {
return "", err
}
preview, saw := previewOf(node, reported, filtering, derived, plan, taken, filter, runs[filter])
preview += "\n\n preview " + saw
if !yes {
return preview + fmt.Sprintf("\n\nNothing has changed. Run `converge %s --yes %s` to do "+
@@ -550,7 +794,7 @@ func converge(ctx context.Context, open *stores, node string, yes bool, digest s
// previewOf is what converging a node will change, before it changes it, and a short digest of
// what it said: every reachable thing and its fate, the modules the flip takes and the filter. The
// digest is what the flip is asked to act on, so it changes whenever any of those would.
func previewOf(node string, reported inventory.Adoption, derived derivedFilter,
func previewOf(node string, reported inventory.Adoption, filtering inventory.Filtering, derived derivedFilter,
plan catalogue.Resolution, taken []string, filter string, filterAssigned bool) (string, string) {
var said []string
var b strings.Builder
@@ -642,6 +886,30 @@ func previewOf(node string, reported inventory.Adoption, derived derivedFilter,
fmt.Fprintf(&b, " the found firewall (%s) is disabled, never flushed: its configuration stays on disk\n", fw)
}
said = append(said, fmt.Sprintf("filter %s assigned=%t firewall=%s", filter, filterAssigned, fw))
// What filters the machine now, and the fate of each (novox/hq ADR 0168): the found firewall
// retired, the runtime's own and bans left, and what the mesh did not write left and named —
// so the reader knows before the flip that the machine will not be filtered by the mesh alone.
if len(filtering.Filters) > 0 {
b.WriteString("\n what filters the machine now, and what the flip does to each:\n")
for _, x := range filtering.Filters {
fate := "left: " + x.Owner + "'s"
switch x.Owner {
case inventory.FilterMesh:
fate = "the mesh's guard; replaced by its filter"
case inventory.FilterFoundFirewall:
fate = "the found firewall's; retired with it"
case inventory.FilterRuntime:
fate = "the container runtime's own; left"
case inventory.FilterBan:
fate = "a ban list; left"
case inventory.FilterOther:
fate = "NOT THE MESH'S; left in force — the machine is not filtered by the mesh alone until you remove it"
}
fmt.Fprintf(&b, " %-50s %s\n", x.Where, fate)
fmt.Fprintf(&b, " %-50s %s\n", "", x.Refuses)
said = append(said, "filter "+x.Owner+" "+x.Where)
}
}
// Sorted: the same account, reported in another order, is the same preview.
sort.Strings(said)
sum := sha256.Sum256([]byte(strings.Join(said, "\n")))
@@ -767,21 +1035,29 @@ func adopt(ctx context.Context, open *stores, node string) (string, error) {
// takeCommand, convergeCommand and adoptCommand are the command line's adapters to the acts above.
func takeCommand(ctx context.Context, args []string) error {
set := flag.NewFlagSet("take", flag.ContinueOnError)
yes := set.Bool("yes", false, "cut over as previewed; without it the comparison is printed and nothing is taken")
yes := set.Bool("yes", false, "cut over as previewed, naming the digest the preview printed after it; "+
"without it the comparison is printed and nothing is taken")
downgrade := set.Bool("downgrade", false, "take it although the module's image is older than the one running")
var replace stringList
var replace, mint stringList
set.Var(&replace, "replace", "a found file's path whose content the module may replace although it differs (repeatable; * for every one)")
set.Var(&mint, "mint", "a secret the service shall take the mesh's minted value for, although it already has its own (repeatable)")
positionals, err := parseAround(set, args)
if err != nil {
return err
}
if len(positionals) != 2 {
return errors.New("take <node> <module> [--yes] [--downgrade] [--replace <path>]...")
if len(positionals) < 2 || len(positionals) > 3 || (len(positionals) == 3 && !*yes) {
return errors.New("take <node> <module> [--yes <digest>] [--downgrade] [--replace <path>]... [--mint <secret>]...")
}
opts := takeOptions{Yes: *yes, Downgrade: *downgrade, Replace: map[string]bool{}, Mint: map[string]bool{}}
if len(positionals) == 3 {
opts.Digest = positionals[2]
}
opts := takeOptions{Yes: *yes, Downgrade: *downgrade, Replace: map[string]bool{}}
for _, r := range replace {
opts.Replace[r] = true
}
for _, m := range mint {
opts.Mint[m] = true
}
return runAct(ctx, func(open *stores) (string, error) { return take(ctx, open, positionals[0], positionals[1], opts) })
}
+3 -3
View File
@@ -102,7 +102,7 @@ func commands(who Authenticator) http.Handler {
}))
// Adoption (novox/hq ADR 0100): the same acts as `take`, `converge` and `adopt`.
mux.HandleFunc("POST /take", acting(who, true, func(ctx context.Context, open *stores, in request) (string, error) {
return take(ctx, open, in.Node, in.Module, takeOptions{Yes: true})
return take(ctx, open, in.Node, in.Module, takeOptions{Yes: in.Yes, Digest: in.Digest})
}))
mux.HandleFunc("POST /converge", acting(who, false, func(ctx context.Context, open *stores, in request) (string, error) {
return converge(ctx, open, in.Node, in.Yes, in.Digest, in.Filter)
@@ -124,8 +124,8 @@ func commands(who Authenticator) http.Handler {
type request struct {
Node string `json:"node"`
Module string `json:"module"`
// Yes, Digest and Filter are converge's: do it rather than preview it, the digest of the
// preview it acts on, and which module loads the mesh's filter.
// Yes, Digest and Filter are converge's and take's: do it rather than preview it, the digest
// of the preview it acts on, and (converge) which module loads the mesh's filter.
Yes bool `json:"yes,omitempty"`
Digest string `json:"digest,omitempty"`
Filter string `json:"filter,omitempty"`
+61 -6
View File
@@ -430,11 +430,13 @@ func buildOne(ctx context.Context, source buildSource, path, ref string, wait ti
}
fmt.Println()
ask, err := askOver(server)
seat := buildSeatHeld(ctx)
ask, err := askOverOn(seat)
if err != nil {
return err
}
defer ask.Close()
fmt.Printf(" of %s\n", seat)
if wait == 0 {
// Asked and not waited for (novox/hq issue 176): the outcome is the role's event, and the
@@ -522,6 +524,8 @@ func takeIn(ctx context.Context, inv *inventory.Inventory, result link.BuildResu
recorded := inventory.Source{
Repository: result.Repository, Path: result.Path, Ref: result.Ref,
BuiltFrom: result.Commit, Head: result.Commit,
// What it stood on, so registration can judge a built manifest's base (to-be 38 WP2.4).
Against: kept.Against,
}
if result.Source != nil && result.Source.Seat != "" {
recorded.Repository, recorded.Seat = result.Source.Repository, result.Source.Seat
@@ -553,7 +557,7 @@ func buildAndShow(ctx context.Context, source buildSource, path, ref string, wai
}
defer server.Close()
ask, err := askOver(server)
ask, err := askOverOn(buildSeatHeld(ctx))
if err != nil {
return err
}
@@ -607,6 +611,10 @@ type answers struct {
// a consequence of the refusals above: a node that does not resolve is not on the network, and
// a mesh whose hub is that node has no hub.
network string
// filtered is every converged machine that is not filtered by the mesh alone (novox/hq ADR
// 0168): what filters it beyond the mesh's own, the runtime's plumbing and bans, by name — a
// predecessor's chain, a found firewall in force again. Such a machine is not "all well".
filtered map[string]inventory.Filtering
// untaken is, per machine, each assigned module whose resources the machine is holding as it
// found them, and how many — a module that was assigned, sent, and is running none of what it
// declares because nothing has taken it (novox/hq ADR 0100, 04-ISSUES/125).
@@ -662,12 +670,56 @@ func heldBy(ctx context.Context) map[string]string {
// **One place chooses**, as everywhere else the bus change went (novox/hq ADR 0116 step 5). On the bus
// the mesh runs on today this needs the controller's own connection, so it is handed one; on the bus
// being built it dials, because a build request is a one-shot and holds nothing else.
func askOver(_ *link.Server) (link.Builders, error) {
func askOverOn(seat string) (link.Builders, error) {
address, err := broker.BusAddress()
if err != nil {
return nil, err
}
return link.BuildsOverNATS(address)
return link.BuildsOverNATSOn(address, seat)
}
// buildSeatHeld is the build role to ask: the one some assigned module claims (novox/hq ADR 0190,
// the handover). Read from the catalogue at ask time, because the answer changes exactly once, the
// moment the first build-agent is assigned — and a controller that asked the new role before then
// would queue work nothing takes, while the outcome that registers build-agent itself has to come
// from the old builder. When the catalogue cannot be read the current role is asked, said aloud.
func buildSeatHeld(ctx context.Context) string {
open, err := openStores(ctx)
if err != nil {
fmt.Fprintf(os.Stderr, "could not read what is assigned, so the build is asked of %s: %v\n",
link.TheBuildMachine, err)
return link.TheBuildMachine
}
defer open.Close()
entries, err := open.inventory.Catalogued(ctx)
if err != nil {
fmt.Fprintf(os.Stderr, "could not read the catalogue, so the build is asked of %s: %v\n",
link.TheBuildMachine, err)
return link.TheBuildMachine
}
return buildSeatAmong(entries)
}
// buildSeatAmong is the rule, over what the catalogue holds: the current build role when any
// assigned module claims it; else the retired role while an assigned module still claims that; else
// the current role, which is where every ask goes once the handover is done.
func buildSeatAmong(entries []inventory.Entry) string {
heldBefore := false
for _, e := range entries {
if len(e.On) == 0 {
continue
}
if e.Manifest.ClaimsSeat(link.TheBuildMachine) {
return link.TheBuildMachine
}
if e.Manifest.ClaimsSeat(link.TheBuildMachineBefore) {
heldBefore = true
}
}
if heldBefore {
return link.TheBuildMachineBefore
}
return link.TheBuildMachine
}
// buildLog prints everything a build machine said about one build, read back from the bus.
@@ -687,10 +739,13 @@ func buildLog(ctx context.Context, id string) error {
}
defer js.Close()
sub, err := js.Context().PullSubscribe(link.BuildLog(id), "",
// Under whichever build role did it: a build asked of the retired role during the handover
// (ADR 0190) said its lines as that role's events, and a reader should not have to know which.
lines := link.BuildLogOf("*", id)
sub, err := js.Context().PullSubscribe(lines, "",
nats.BindStream(broker.EventsStream), nats.DeliverAll(), nats.AckNone())
if err != nil {
return fmt.Errorf("cannot read %s from the bus: %w", link.BuildLog(id), err)
return fmt.Errorf("cannot read %s from the bus: %w", lines, err)
}
defer func() { _ = sub.Unsubscribe() }()
+43
View File
@@ -0,0 +1,43 @@
package main
import (
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
func claiming(module, seat string, on ...string) inventory.Entry {
return inventory.Entry{
Manifest: catalogue.Manifest{Module: module, Claims: []catalogue.Claim{{Name: seat}}},
On: on,
}
}
// The controller asks the build role that has a holder (novox/hq ADR 0190 handover): the retired
// one while only the builder is assigned, the current one from the first build-agent on, and the
// current one when nothing holds either — where every ask goes once the handover is done.
func TestTheControllerAsksTheBuildRoleThatHasAHolder(t *testing.T) {
onlyTheBuilder := []inventory.Entry{
claiming("builder", link.TheBuildMachineBefore, "anchor"),
claiming("build-agent", link.TheBuildMachine), // registered, assigned nowhere yet
}
if got := buildSeatAmong(onlyTheBuilder); got != link.TheBuildMachineBefore {
t.Errorf("with only the builder assigned, asked %q", got)
}
bothHeld := []inventory.Entry{
claiming("builder", link.TheBuildMachineBefore, "anchor"),
claiming("build-agent", link.TheBuildMachine, "home-server"),
}
if got := buildSeatAmong(bothHeld); got != link.TheBuildMachine {
t.Errorf("with a build-agent assigned anywhere, asked %q", got)
}
neither := []inventory.Entry{claiming("builder", link.TheBuildMachineBefore)}
if got := buildSeatAmong(neither); got != link.TheBuildMachine {
t.Errorf("with no holder of either, asked %q, want the current role", got)
}
if got := buildSeatAmong(nil); got != link.TheBuildMachine {
t.Errorf("an empty catalogue asks %q", got)
}
}
@@ -0,0 +1,90 @@
package main
import (
"strings"
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
)
// A fresh assignment is pushed before its credential exists (novox/hq issue 203): `assign` recorded
// the module, `push` sealed a random own secret where the bus credential belongs, and the process
// crash-looped until a person ran `module issue` and pushed again. Now assigning a module that speaks
// on the bus issues its credential in the same act — or, when the bus cannot be reached from here,
// says which verb to run — and a push never seals a placeholder in a credential's place.
func aTalker() catalogue.Manifest {
return catalogue.Manifest{Module: "talker", Version: "1",
OwnSecrets: catalogue.OwnSecrets{"broker": {Path: "/var/lib/mesh/talker/broker"}},
Resources: []map[string]any{
{"id": "state", "type": "directory", "path": "/var/lib/mesh/talker", "mode": "0700"},
}}
}
func TestAssigningAModuleThatSpeaksOnTheBusNamesItsCredential(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
register(t, open, aTalker())
// No bus is known to this process, so the credential cannot be issued here: the assignment
// stands and says exactly what must happen before a push — never silently.
said, err := assign(ctx, open, "laptop", "talker")
if err != nil {
t.Fatal(err)
}
if !strings.Contains(said, "module issue talker --node laptop") {
t.Fatalf("an assignment whose credential could not be issued does not name the verb:\n%s", said)
}
// And the push refuses to send it, naming the same verb, rather than sealing a placeholder.
plan, settings, err := planFor(ctx, open, "laptop")
if err != nil {
t.Fatal(err)
}
_, err = declarationFor(ctx, open, "laptop", plan, settings)
if err == nil {
t.Fatal("a push sealed a placeholder where talker's bus credential belongs")
}
if !strings.Contains(err.Error(), "module issue talker --node laptop") || !strings.Contains(err.Error(), "issue 203") {
t.Fatalf("the refusal does not say what to run: %v", err)
}
// Once the user is minted, the push goes on to the credential the mesh sealed, and re-assigning
// does not mint again: a credential rotates on purpose, never by habit.
if _, err := open.inventory.MintBusPassword(ctx, inventory.BusUser{
Username: "laptop.talker", Kind: inventory.BusModule, Node: "laptop", Module: "talker"}); err != nil {
t.Fatal(err)
}
hash, _, err := open.inventory.BusUserHash(ctx, "laptop.talker")
if err != nil {
t.Fatal(err)
}
said, err = assign(ctx, open, "laptop", "talker")
if err != nil {
t.Fatal(err)
}
if strings.Contains(said, "module issue") {
t.Fatalf("a module with a minted credential was told to issue one:\n%s", said)
}
again, _, err := open.inventory.BusUserHash(ctx, "laptop.talker")
if err != nil {
t.Fatal(err)
}
if again != hash {
t.Fatal("re-assigning rotated the credential")
}
}
// A module that declares no broker secret is left alone: nothing to issue, nothing said.
func TestAssigningAModuleThatDoesNotSpeakSaysNothingOfCredentials(t *testing.T) {
open := aMesh(t)
register(t, open, helloWeb())
said, err := assign(t.Context(), open, "laptop", "hello-web")
if err != nil {
t.Fatal(err)
}
if strings.Contains(said, "credential") {
t.Fatalf("a module without a broker secret was told about credentials:\n%s", said)
}
}
@@ -1,33 +0,0 @@
package main
// The broker opening belongs only on the node that listens on it (novox/hq: it leaked onto
// every enrolled node's declaration, opening a from-anywhere hole for a port nothing there
// serves). foundationPortsFor is the scope.
import (
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
)
func TestTheBrokerHostGetsTheFoundationOpening(t *testing.T) {
broker := catalogue.Manifest{Module: "lavinmq", Listens: []catalogue.Listening{
{Port: 5671, Protocol: "tcp", From: "mesh"},
{Port: 5672, Protocol: "tcp", From: "mesh"},
}}
got := foundationPortsFor(5671, []catalogue.Manifest{broker})
if len(got) != 1 || got[0] != 5671 {
t.Fatalf("the node that listens on the broker port keeps it; got %v", got)
}
}
func TestANodeThatOnlyDialsTheBrokerGetsNoOpening(t *testing.T) {
// ace's set: things that reach the broker as a client, none listening on 5671.
ace := []catalogue.Manifest{
{Module: "plex", Listens: []catalogue.Listening{{Port: 32400, Protocol: "tcp", From: "anywhere"}}},
{Module: "postgres", Listens: []catalogue.Listening{{Port: 5432, Protocol: "tcp", From: "mesh"}}},
}
if got := foundationPortsFor(5671, ace); got != nil {
t.Fatalf("a node that only dials out opens nothing for the broker; got %v", got)
}
}
+93
View File
@@ -0,0 +1,93 @@
package main
import (
"context"
"testing"
"github.com/novox/mesh-controller/internal/inventory"
)
// A declaration composed earlier is numbered lower than one composed later, whatever order the two
// are sent in (novox/hq issue 204). The number used to be taken at send time, after composing, so a
// declaration composed before an assignment changed and sent after a newer one carried the higher
// number — and the machine, which refuses a lower number, took the older content as the mesh's
// newest word. Taken before the composition reads anything, the order of numbers is the order of
// compositions, and the host's refusal does what it is for.
func TestADeclarationComposedEarlierIsNumberedLowerWhateverOrderItIsSent(t *testing.T) {
allot := numbered()
var composed []string
compose := func(stamp string) func(string) (sendable, error) {
return func(node string) (sendable, error) {
composed = append(composed, stamp)
return sendable{Resources: []map[string]any{{"id": node + "." + stamp}}}, nil
}
}
// Composed first — before an assignment changed — and sent last.
stale, _ := composeEach([]string{"anchor"}, allot, compose("before"))
// Composed after the change, sent first.
fresh, _ := composeEach([]string{"anchor"}, allot, compose("after"))
if stale[0].declared.Sequence != 1 || fresh[0].declared.Sequence != 2 {
t.Fatalf("the numbers do not follow the compositions: before=%d after=%d",
stale[0].declared.Sequence, fresh[0].declared.Sequence)
}
// Sent in the other order, the numbers do not change — so the machine that has applied the
// fresh one (2) refuses the stale one (1) when it arrives late.
if !(stale[0].declared.Sequence < fresh[0].declared.Sequence) {
t.Fatal("a declaration composed earlier must carry the lower number, however late it is sent")
}
if len(composed) != 2 || composed[0] != "before" {
t.Fatalf("compositions happened in an unexpected order: %v", composed)
}
}
// The number is taken before the first read of the composition, not after it: an allotter that
// fails leaves nothing composed for that machine, and the others are still composed.
func TestTheNumberIsTakenBeforeComposingAndItsFailureIsARefusal(t *testing.T) {
calls := 0
allot := func(node string) (int64, error) {
if node == "anchor" {
return 0, context.DeadlineExceeded
}
return 7, nil
}
sending, refusals := composeEach([]string{"anchor", "laptop"}, allot, func(node string) (sendable, error) {
calls++
if node == "anchor" {
t.Fatal("anchor was composed although its number could not be taken")
}
return sendable{}, nil
})
if calls != 1 || len(sending) != 1 || sending[0].node != "laptop" || sending[0].declared.Sequence != 7 {
t.Fatalf("laptop should be composed with its number and anchor refused: %v / %v", sending, refusals)
}
if len(refusals) != 1 {
t.Fatalf("anchor's failed number should be a refusal naming it: %v", refusals)
}
}
// What was sent is written down even when the sender's context is already cancelled (issue 204): a
// controller replaced mid-send had told the machine and never recorded it, so status read "applied,
// current" over a machine that had just been sent something else.
func TestASendIsRecordedEvenWhenTheSenderIsBeingCancelled(t *testing.T) {
inv := inventory.ForTest(t)
ctx, cancel := context.WithCancel(t.Context())
if _, err := inv.AddNode(ctx, "anchor"); err != nil {
t.Fatal(err)
}
cancel() // the sender is going away: its context is cancelled between the send and the record
body := []byte(`{"declaration":1,"resources":[]}`)
digest, err := recordSent(ctx, inv, "anchor", body)
if err != nil {
// NodeByName on the cancelled context may itself refuse; the record must still be possible
// through the detached context, so look the node up again on a live one.
t.Fatalf("recording a send after cancellation failed: %v", err)
}
outstanding, err := inv.Outstanding(t.Context(), "anchor")
if err != nil {
t.Fatal(err)
}
if outstanding != digest || digest != digestOf(body) {
t.Fatalf("the send was not recorded: outstanding %q, sent %q", outstanding, digest)
}
}
+2 -1
View File
@@ -180,7 +180,8 @@ func usage() {
api --issuer URL [--listen A] assign and unassign over http, for a surface that is not here
assign <node> <module> put a module on a node
unassign <node> <module> take it off
take <node> <module> cut a module over on an adopted node, once its data has moved
take <node> <module> preview a module's cutover on an adopted node: what runs beside
what it declares; --yes <digest> cuts it over as previewed
converge <node> [--yes <digest>] [--filter nftables] preview, then make, an adopted node converged
adopt <node> return a converged node to adopted; what was taken stays taken
settings set <module> <file> what a module's config should say, for the whole mesh
+18 -4
View File
@@ -352,10 +352,14 @@ func settingsCommand(ctx context.Context, args []string) error {
switch args[0] {
case "set":
if len(positionals) != 2 {
return errors.New("settings set <module> <settings.json> [--node <node>]")
return errors.New("settings set <module> <settings.json | {…}> [--node <node>]")
}
raw, err := os.ReadFile(positionals[1])
if err != nil {
// A file, or the values themselves when they begin with `{` — which is how the mesh's own
// `settings` tool passes them, having no file to hand over (novox/hq issue 198).
var raw []byte
if strings.HasPrefix(strings.TrimSpace(positionals[1]), "{") {
raw = []byte(positionals[1])
} else if raw, err = os.ReadFile(positionals[1]); err != nil {
return err
}
var values map[string]any
@@ -553,7 +557,7 @@ func issueOnTheNewBus(ctx context.Context, inv *inventory.Inventory, m catalogue
user := broker.Principal{Kind: broker.KindModule, Node: node, Module: m.Module}.Username()
password, err := inv.MintBusPassword(ctx, inventory.BusUser{
Username: user, Kind: inventory.BusModule, Node: node, Module: m.Module,
Username: user, Kind: busKindOf(m.Module), Node: node, Module: m.Module,
})
if err != nil {
return err
@@ -573,6 +577,16 @@ func issueOnTheNewBus(ctx context.Context, inv *inventory.Inventory, m catalogue
return issueWith(ctx, inv, m, node, busAddress, known, reachable, user, password)
}
// busKindOf is what a module's bus user is recorded as: the node's tool runtime where the module is
// the runtime (novox/hq ADR 0175), a module otherwise. The username is the same either way — the
// runtime is issued through this same path — and the kind is what a reader of the records sees.
func busKindOf(module string) string {
if module == catalogue.RuntimeModule {
return inventory.BusNodeTools
}
return inventory.BusModule
}
// issueWith is the delivery half: the minted password sealed to the machine as the module's broker
// secret, and the module's consumer created where the bus can be reached. Split from the minting
// so the move can issue every module against a bus whose address it worked out itself
+26
View File
@@ -3,6 +3,7 @@ package main
import (
"context"
"os"
"reflect"
"strings"
"testing"
@@ -303,3 +304,28 @@ func TestTheResolverIsToldEveryMachineOnTheNetworkAndToldAgainWhenOneLeaves(t *t
t.Fatalf("a machine that left the network is still a wildcard, or the one that stayed is not:\n%s", after)
}
}
// The roster is the machines and nothing else (novox/hq ADR 0191): each node's internal domain covers
// every route on it, and a node's public domains are public DNS's. A routed name in `.Names` was a
// private answer for a public name, handed by a resolver serving a LAN to a phone that could not use it.
func TestTheRosterNamesOnlyTheMachines(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
gens, err := generators(ctx, open)
if err != nil {
t.Fatal(err)
}
for _, node := range []string{"anchor", "laptop"} {
plan, settings, err := planFor(ctx, open, node)
if err != nil {
t.Fatal(err)
}
with, _, err := renderingFor(ctx, open, node, plan, settings, gens, Reading)
if err != nil {
t.Fatal(err)
}
if !reflect.DeepEqual(with.Names, with.Machines) {
t.Fatalf("%s's roster names more than the machines:\n names %v\n machines %v", node, with.Names, with.Machines)
}
}
}
+2 -1
View File
@@ -1,6 +1,7 @@
package main
import (
"reflect"
"strings"
"testing"
"time"
@@ -191,7 +192,7 @@ func TestWhatAHandedOverModuleRecordsAboutItsSource(t *testing.T) {
t.Fatalf("the source records as %+v", from)
}
// A manifest with no provenance at all is legitimate: fixing something in a hurry.
if from, err := whereItComesFrom("", "", "", "", false); err != nil || from != (inventory.Source{}) {
if from, err := whereItComesFrom("", "", "", "", false); err != nil || !reflect.DeepEqual(from, inventory.Source{}) {
t.Fatalf("a manifest handed over with no provenance was refused: %+v, %v", from, err)
}
for _, c := range []struct {
+78 -135
View File
@@ -16,8 +16,6 @@ import (
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/licences"
"github.com/novox/mesh-controller/internal/overlay"
"net"
"strconv"
)
// working out what one machine should be.
@@ -186,8 +184,12 @@ func planFor(ctx context.Context, open *stores, nodeName string) (catalogue.Reso
// Settings for everything that resolved, including modules nobody assigned directly: a
// requirement pulled in by something else is still configurable, and finding out that it is
// not only when you try would be an arbitrary line nobody could predict.
//
// A setting that reaches nothing, or cannot compose with the definition it was stored for,
// no longer refuses the machine here: it is judged where it is stored, and a definition that
// moved under it costs that module its place in the declaration, said by name (novox/hq ADR
// 0163, rule 6 — see Compose).
settings := catalogue.SettingsBy{}
var stray []string
for _, m := range resolved.Modules {
layers, err := inv.SettingsFor(ctx, nodeName, m.Module)
if err != nil {
@@ -197,18 +199,6 @@ func planFor(ctx context.Context, open *stores, nodeName string) (catalogue.Reso
continue
}
settings[m.Module] = layers
stray = append(stray, catalogue.UnusedSettings(m, layers)...)
}
if len(stray) > 0 {
// Somebody set something that reaches no file. Said here rather than discovered by the
// machine not behaving differently, which is the slowest way there is.
//
// Marked like a set that will not compose, and for the same reason: it is a standing fact
// about this node's own configuration, not a question the mesh could not answer. A gatherer
// passes over it as it always did — one node's stray setting must not stop every other node
// being described (novox/hq 04-ISSUES/152).
return catalogue.Resolution{}, nil, notResolvable{fmt.Errorf(
"these settings reach nothing:\n - %s", strings.Join(stray, "\n - "))}
}
return resolved, settings, nil
}
@@ -405,7 +395,32 @@ func declarationWith(ctx context.Context, open *stores, node string,
return sendable{}, err
}
return sendable{Resources: composed.Resources, Adoption: adoption,
Received: composed.Received, Mesh: with.Mesh}, nil
Received: composed.Received, Mesh: with.Mesh,
LeftOut: sortedKeysOf(composed.LeftOut), leftOutWhy: composed.LeftOut}, nil
}
// sortedKeysOf is a map's keys, sorted — so what a declaration says it left out does not move
// for a reordering nobody made.
func sortedKeysOf(m map[string]string) []string {
if len(m) == 0 {
return nil
}
out := make([]string, 0, len(m))
for k := range m {
out = append(out, k)
}
sort.Strings(out)
return out
}
// reportLeftOut says which of a machine's modules its declaration leaves out and why (novox/hq ADR
// 0163, rule 6), one line each: the machine is told everything else, and is told it was left out.
func reportLeftOut(node string, declared sendable) {
for _, m := range declared.LeftOut {
fmt.Printf("%s: %s left out — a setting stored for it cannot compose with its definition; "+
"what the machine holds for it is kept and its containers are untouched. %s\n",
node, m, declared.leftOutWhy[m])
}
}
// renderingFor is everything a node's declaration is composed with, and the node's record.
@@ -445,7 +460,10 @@ func renderingFor(ctx context.Context, open *stores, node string,
for _, m := range plan.Modules {
g, err := catalogue.GivenPorts(m, settings[m.Module])
if err != nil {
return catalogue.Rendering{}, inventory.Node{}, err
// A given port its definition no longer publishes: the module is left out of the
// declaration, by name, when it is composed (novox/hq ADR 0163, rule 6) — never the
// machine refused here for it.
continue
}
if g != nil {
given[m.Module] = g
@@ -515,6 +533,23 @@ func renderingFor(ctx context.Context, open *stores, node string,
var sealed string
var err error
if choosing == Allocating {
// **The broker credential is never invented here** (novox/hq issue 203). Every other
// own secret is the mesh's to make — a password nobody else knows — but this one
// is an account on the bus, minted by `module issue` and sealed by it; a push that
// made a random one would deliver a file the process cannot read and report the
// machine applied. Refused by name, with the verb.
if name == "broker" {
user := broker.Principal{Kind: broker.KindModule, Node: node, Module: m.Module}.Username()
if _, minted, err := inv.BusUserHash(ctx, user); err != nil {
return catalogue.Rendering{}, inventory.Node{}, err
} else if !minted {
return catalogue.Rendering{}, inventory.Node{}, fmt.Errorf(
"%s on %s has no bus credential: nothing was issued for %s, and a push "+
"would seal a placeholder its process cannot read (novox/hq issue 203). "+
"`module issue %s --node %s`, then push again",
m.Module, node, user, m.Module, node)
}
}
sealed, err = inv.SecretForModule(ctx, node, m.Module, name)
} else {
var held bool
@@ -610,43 +645,24 @@ func renderingFor(ctx context.Context, open *stores, node string,
}
}
// And every routed name → the node that serves it (novox/hq ADR 0066). Alongside the
// `<node>.internal` names above, so a container — or an internal ACME validator — resolves a
// routed name to the proxy that serves it, mesh-wide. The mesh publishes the names it was told
// to serve and knows nothing about what they mean.
// Kept apart from the machines, because a fact about the machines must not be handed the names
// the mesh merely serves (novox/hq 04-ISSUES/111).
// **The roster is the machines and nothing else** (novox/hq ADR 0191). Each node has one internal
// domain, `<node>.internal`, and every route on it is a name under that domain (ADR 0151), which
// the resolver answers with one wildcard per machine — so no route needs a line of its own. A
// node's public domains are the operator's and public DNS answers them; the mesh gives no private
// answer for any of them. The roster once carried every routed name, public ones included, and a
// resolver that also serves a LAN handed a phone a tunnel address for the mail server.
// `.Names` and `.Machines` stay two fields so a module's template keeps rendering (issue 111).
machines := make(map[string]string, len(names))
for name, at := range names {
machines[name] = at
}
routes, err := routeNamesInTheMesh(ctx, open)
if err != nil {
return catalogue.Rendering{}, inventory.Node{}, err
}
for name, at := range routes {
names[name] = at
}
// The ports the mesh itself needs open, which no module declares. Read from the broker this
// control plane was told about rather than written down twice: the address a node is handed in
// its token and the port its machine must accept on are the same fact.
//
// **Only on the node that listens on it** (novox/hq issue: the broker opening leaked onto
// every node). The opening exists to WIDEN the broker's port to from-anywhere — a machine
// enrolling is not on the mesh yet, so the broker's own `from: mesh` listen would refuse its
// first dial. That widening belongs on the broker's host and nowhere else: a node that only
// dials out needs no incoming rule, and an opening for a port nothing here listens on is a
// from-anywhere hole for a dead port. So the foundation port is kept only when a module
// resolved onto THIS node actually listens on it.
// **The bus is never public** (novox/hq ADR 0169). It was a foundation port — widened from the
// broker's own `from: mesh` to from-anywhere on the broker's host, so a machine could enrol
// before it had an address on the private network. A machine joins through the tunnel now, and
// every link to the bus crosses it, so its reach is what the `nats` module declares: the mesh.
// Nothing the mesh itself needs is opened beyond what a module declares.
var foundation []int
if b, err := broker.FromEnvironment(); err == nil {
if _, port, err := net.SplitHostPort(b.Address); err == nil {
if n, err := strconv.Atoi(port); err == nil {
foundation = foundationPortsFor(n, plan.Modules)
}
}
}
// And, for a module that keeps them, every operator-sealed secret in the mesh — the vault's
// copy, outside the store (novox/hq ADR 0085, amended). Read only; nothing here mints. The
@@ -718,76 +734,6 @@ func renderingFor(ctx context.Context, open *stores, node string,
}, record, nil
}
// routeNamesInTheMesh is every routed name and the address of the node that serves it (novox/hq
// ADR 0066).
//
// **Mesh-wide, so any container resolves any routed name to its proxy** — including an internal
// ACME validator, which cannot complete a challenge for a name it cannot reach. A routed name is
// composed on the consumer's node (from its label and that node's public domain) and served by the
// node answering the consumer's route requirement; this gathers both.
//
// It reads route names off resolutions rather than a table because there is no table: a route is a
// contribution, computed from what each node runs. Name-agnostic — a contribution counts as a
// routed name only because it carried a label the mesh composed, never because the mesh knows what
// "route" means. A node that does not resolve is skipped, so one machine's broken set does not cost
// the rest their names.
//
// **A node that could not be READ is a different matter and is raised.** Skipping one states, to
// every machine at once, that its names do not exist — and since the roster is part of every
// container's identity, that withdraws them and replaces every container (novox/hq 04-ISSUES/152,
// 151). So every failure here says which machine and which read, because the alternative is a
// mesh-wide refusal with nothing named in it.
func routeNamesInTheMesh(ctx context.Context, open *stores) (map[string]string, error) {
inv := open.inventory
places, err := inv.Overlays(ctx)
if err != nil {
return nil, fmt.Errorf("where the machines are cannot be read: %w", err)
}
address := map[string]string{}
for _, p := range places {
if strings.TrimSpace(p.Address) != "" {
address[p.Name] = p.Address
}
}
nodes, err := inv.Nodes(ctx)
if err != nil {
return nil, fmt.Errorf("which machines the mesh has cannot be read: %w", err)
}
// Every machine's resolution first, then the names across them at once: which node serves a
// name is a question about the graph — the consumer on one machine, the provider on another —
// and answered wrongly by looking at one contribution at a time (novox/hq issue 178).
plans := map[string]catalogue.Resolution{}
settings := map[string]catalogue.SettingsBy{}
for _, n := range nodes {
plan, layers, err := planFor(ctx, open, n.Name)
switch {
case unresolvable(err):
// Their set does not compose, so they serve no names. Passed over, so one machine's
// broken set does not cost the rest theirs.
continue
case err != nil:
// The mesh could not be asked. Returning the roster without this machine's names would
// state that they do not exist — to every machine, and indistinguishably from the
// operator having withdrawn them (novox/hq 04-ISSUES/152).
return nil, fmt.Errorf("the names %s serves cannot be read: %w", n.Name, err)
}
plans[n.Name], settings[n.Name] = plan, layers
}
served, err := catalogue.NamesServed(plans, settings)
if err != nil {
return nil, err
}
out := map[string]string{}
for name, node := range served {
if at := address[node]; at != "" {
out[name] = at
}
}
return out, nil
}
// certificateFor is what the mesh certifies about one machine's internal name.
//
// It reaches across two contexts and reads neither one's store from the other: `inventory` knows
@@ -1000,6 +946,20 @@ func planCommand(ctx context.Context, args []string) error {
return nil
}
// Which modules a push would leave out, and why — said before the plan, since the plan is of
// what the machine would be told (novox/hq ADR 0163, rule 6). Judged, never composed: `plan`
// without --json allocates nothing.
if record, err := open.inventory.NodeByName(ctx, args[0]); err == nil {
left := plan.LeftOut(settings, record.Adopted)
reportLeftOut(args[0], sendable{LeftOut: sortedKeysOf(left), leftOutWhy: left})
}
// And a setting that reaches nothing — refused where it is stored, and said here for one
// stored before its definition moved from under it.
for _, m := range plan.Modules {
for _, stray := range catalogue.UnusedSettings(m, settings[m.Module]) {
fmt.Printf("%s: a setting reaches nothing — %s\n", args[0], stray)
}
}
fmt.Printf("%s would run:\n", args[0])
for _, m := range plan.Modules {
fmt.Printf(" %-20s %s\n", m.Module, plan.Because[m.Module])
@@ -1346,23 +1306,6 @@ func composeBusUsers(ctx context.Context, inv *inventory.Inventory,
return broker.ComposeAccounts(filled)
}
// foundationPortsFor is the broker port, kept only when a module resolved onto this node listens
// on it (novox/hq issue: the broker opening leaked onto every node). The foundation opening
// exists to WIDEN the broker's `from: mesh` port to from-anywhere, because a machine enrolling is
// not on the mesh yet and its first dial would be refused. That widening belongs on the broker's
// host alone: a node that only dials out needs no incoming rule, and an opening for a port
// nothing here listens on is a from-anywhere hole for a dead port.
func foundationPortsFor(brokerPort int, modules []catalogue.Manifest) []int {
for _, m := range modules {
for _, l := range m.Listens {
if l.Port == brokerPort {
return []int{brokerPort}
}
}
}
return nil
}
// providerModuleOf is which module answers a need on the providing node: the one in this node's
// own set when the provider is here, else the one the catalogue says offers it.
func providerModuleOf(resolved catalogue.Resolution, open *stores, ctx context.Context, n catalogue.Needed) string {
+84 -40
View File
@@ -131,10 +131,17 @@ func serve(ctx context.Context) error {
// And the mesh's own verbs, as the seat this control plane holds (novox/hq ADR 0154). Served
// from the store's row, so what the seat declares is what is answered.
handlers, err := seatToolHandlers()
handlers, behind, err := seatToolHandlers()
if err != nil {
return err
}
if len(behind) > 0 {
// Said once, loudly, and then served anyway (novox/hq ADR 0185): the mesh keeps answering
// while whatever put an older control plane here is undone.
fmt.Printf("this control plane is behind the %s row: it cannot run %s. "+
"Those answer the reason when called; everything else is served as usual\n",
catalogue.ControllerSeatName, strings.Join(behind, ", "))
}
bus, isNATS := server.Bus().(link.OverNATS)
if !isNATS {
return errors.New("the mesh's verbs are served over the bus, and this control plane is not on it")
@@ -198,6 +205,12 @@ func declare(ctx context.Context, args []string) error {
if err := link.Declare(ctx, server.Bus(), ident, node, raw, 15*time.Second); err != nil {
return err
}
// Written down like every other send (novox/hq issue 204): a declaration a person sent by hand
// is still what the machine was last told, and status must not read it as current for the one
// the mesh would compose.
if _, err := recordSent(ctx, inv, node, raw); err != nil {
return err
}
fmt.Printf("sent %s a signed declaration (%d bytes)\n", node, len(raw))
return nil
}
@@ -341,7 +354,7 @@ func pushCommand(ctx context.Context, args []string) error {
if err != nil {
return err
}
sending, refusals := composeEach(asked, func(node string) (sendable, error) {
sending, refusals := composeEach(asked, allotting(held, inv), func(node string) (sendable, error) {
plan, settings, err := planFor(held, open, node)
if err != nil {
return sendable{}, err
@@ -353,18 +366,18 @@ func pushCommand(ctx context.Context, args []string) error {
// The private network is in here with everything else. It used to be composed separately
// and prepended, which meant every machine with an address was on it and no machine could
// be kept off. It is a module now, so it arrives the way a module does.
return declarationWith(held, open, node, plan, settings, gens, Allocating)
declared, err := declarationWith(held, open, node, plan, settings, gens, Allocating)
if err == nil {
reportLeftOut(node, declared)
}
return declared, err
})
sentDigest := map[string]string{}
defer release()
for _, s := range sending {
// Numbered under the hold, one higher than the last, before the body exists — the number is
// inside the signed bytes, so a replayed older declaration cannot borrow a newer one's
// (novox/hq 04-ISSUES/107).
if err := number(ctx, inv, &s); err != nil {
return err
}
// The number is inside the signed bytes, so a replayed older declaration cannot borrow a
// newer one's (novox/hq 04-ISSUES/107); it was taken when the composition began (issue 204).
body, err := s.declared.Body()
if err != nil {
return err
@@ -374,14 +387,10 @@ func pushCommand(ctx context.Context, args []string) error {
}
// After it is away, not before. A digest recorded for something that failed to send would
// make the machine look current for a declaration it never received.
record, err := inv.NodeByName(ctx, s.node)
digest, err := recordSent(ctx, inv, s.node, body)
if err != nil {
return err
}
digest := digestOf(body)
if err := inv.RecordSent(ctx, record.ID, digest); err != nil {
return err
}
sentDigest[s.node] = digest
fmt.Printf("sent %s %d resource(s)\n", s.node, len(s.declared.Resources))
}
@@ -454,18 +463,18 @@ func pushCommand(ctx context.Context, args []string) error {
return sendable{}, err
}
reportUnhostable(node, plan)
return declarationWith(held, open, node, plan, settings, gens, Allocating)
declared, err := declarationWith(held, open, node, plan, settings, gens, Allocating)
if err == nil {
reportLeftOut(node, declared)
}
return declared, err
},
func(s readyNode, body []byte) error {
if err := link.Declare(ctx, server.Bus(), ident, s.node, body,
15*time.Second); err != nil {
return err
}
record, err := inv.NodeByName(ctx, s.node)
if err != nil {
return err
}
if err := inv.RecordSent(ctx, record.ID, digestOf(body)); err != nil {
if _, err := recordSent(ctx, inv, s.node, body); err != nil {
return err
}
fmt.Printf("sent %s %d resource(s)\n", s.node, len(s.declared.Resources))
@@ -554,17 +563,31 @@ type readyNode struct {
//
// The all-or-nothing rule is kept where it means something — sendTo, which rotates a credential
// across two machines that must agree — and dropped here, where it never did.
func composeEach(names []string,
func composeEach(names []string, allot func(node string) (int64, error),
compose func(node string) (sendable, error)) ([]readyNode, []string) {
var sending []readyNode
var refusals []string
for _, name := range names {
// **Numbered before it is composed, not before it is sent** (novox/hq issue 204). The
// number says where this declaration stands against every other the mesh composed for the
// machine, and the host refuses one lower than the last it applied. Taken at send time, as
// it was, a declaration composed a minute ago — before an assignment changed — went out with
// a number higher than one composed after the change and sent before it, and the machine
// took the older content as the newer word: on 2026-10-02 a runtime assigned and applied on
// two machines was undone two seconds later by exactly that. Taken here, before the first
// read, what was composed earlier is numbered lower whatever order the sends happen in.
seq, err := allot(name)
if err != nil {
refusals = append(refusals, fmt.Sprintf("%s:\n%v", name, err))
continue
}
declared, err := compose(name)
if err != nil {
refusals = append(refusals, fmt.Sprintf("%s:\n%v", name, err))
continue
}
declared.Sequence = seq
if len(declared.Resources) == 0 {
// Sent, not skipped (novox/hq issue 127). A node whose declaration composes to
// nothing may have HELD something before — the broker opening a placement gave it,
@@ -592,13 +615,10 @@ func sendRound(ctx context.Context, open *stores, names []string,
return nil, err
}
defer release()
sending, refused := composeEach(names, func(node string) (sendable, error) {
sending, refused := composeEach(names, allotting(held, open.inventory), func(node string) (sendable, error) {
return compose(held, node)
})
for _, s := range sending {
if err := number(ctx, open.inventory, &s); err != nil {
return refused, err
}
body, err := s.declared.Body()
if err != nil {
return refused, err
@@ -655,6 +675,12 @@ func sendTo(ctx context.Context, open *stores, names []string) error {
var sending []readyNode
var refusals []string
for _, name := range names {
// Numbered before composing, for the reason composeEach gives (novox/hq issue 204).
seq, err := allot(ctx, inv, name)
if err != nil {
refusals = append(refusals, fmt.Sprintf("%s:\n%v", name, err))
continue
}
plan, settings, err := planFor(ctx, open, name)
if err != nil {
refusals = append(refusals, fmt.Sprintf("%s:\n%v", name, err))
@@ -666,6 +692,8 @@ func sendTo(ctx context.Context, open *stores, names []string) error {
refusals = append(refusals, fmt.Sprintf("%s:\n%v", name, err))
continue
}
declared.Sequence = seq
reportLeftOut(name, declared)
sending = append(sending, readyNode{name, declared})
}
if len(refusals) > 0 {
@@ -680,9 +708,6 @@ func sendTo(ctx context.Context, open *stores, names []string) error {
defer server.Close()
for _, s := range sending {
if err := number(ctx, inv, &s); err != nil {
return err
}
body, err := s.declared.Body()
if err != nil {
return err
@@ -690,11 +715,7 @@ func sendTo(ctx context.Context, open *stores, names []string) error {
if err := link.Declare(ctx, server.Bus(), ident, s.node, body, 15*time.Second); err != nil {
return err
}
record, err := inv.NodeByName(ctx, s.node)
if err != nil {
return err
}
if err := inv.RecordSent(ctx, record.ID, digestOf(body)); err != nil {
if _, err := recordSent(ctx, inv, s.node, body); err != nil {
return err
}
fmt.Printf(" sent %s %d resource(s)\n", s.node, len(s.declared.Resources))
@@ -935,15 +956,38 @@ func seatHolders(ctx context.Context, inv *inventory.Inventory) (map[string]brok
}
// number gives one send the next sequence for its node (novox/hq 04-ISSUES/107).
func number(ctx context.Context, inv *inventory.Inventory, s *readyNode) error {
record, err := inv.NodeByName(ctx, s.node)
// allotting is allot over one inventory, in the shape composeEach takes.
func allotting(ctx context.Context, inv *inventory.Inventory) func(node string) (int64, error) {
return func(node string) (int64, error) { return allot(ctx, inv, node) }
}
// allot takes the next sequence for a machine — the number its next declaration carries.
func allot(ctx context.Context, inv *inventory.Inventory, node string) (int64, error) {
record, err := inv.NodeByName(ctx, node)
if err != nil {
return err
return 0, err
}
seq, err := inv.NextSequence(ctx, record.ID)
return inv.NextSequence(ctx, record.ID)
}
// recordSent writes down what a machine was just sent, and returns the digest.
//
// **On a context that outlives the caller's** (novox/hq issue 204). The record is written after the
// declaration is away, so a send that failed is never recorded as current — and a controller being
// replaced mid-send had its context cancelled between the two, so the machine was told and the mesh
// never wrote it down: status read "applied, current" over a machine that had just been sent
// something else. What was sent was sent; the record of it must not depend on the sender living
// another second. Bounded, so a store that is away does not hold a dying process open for ever.
func recordSent(ctx context.Context, inv *inventory.Inventory, node string, body []byte) (string, error) {
kept, cancel := context.WithTimeout(context.WithoutCancel(ctx), 10*time.Second)
defer cancel()
record, err := inv.NodeByName(kept, node)
if err != nil {
return err
return "", err
}
s.declared.Sequence = seq
return nil
digest := digestOf(body)
if err := inv.RecordSent(kept, record.ID, digest); err != nil {
return "", err
}
return digest, nil
}
+8 -2
View File
@@ -17,7 +17,7 @@ import (
// the wrong machine no longer refuses the whole node), applied one level up.
func TestOneUnresolvableNodeStillLetsTheRestBeSent(t *testing.T) {
sending, refusals := composeEach(
[]string{"anchor", "home-server", "laptop"},
[]string{"anchor", "home-server", "laptop"}, numbered(),
func(node string) (sendable, error) {
if node == "anchor" {
return sendable{}, errors.New(`nothing provides "acme-ca", wanted by route-proxy`)
@@ -43,7 +43,7 @@ func TestOneUnresolvableNodeStillLetsTheRestBeSent(t *testing.T) {
// (novox/hq issue 127): it may have held something before, and only sending the empty
// declaration tells it to drop what the mesh owned. It is never a refusal.
func TestAnEmptyDeclarationIsSentSoTheNodeDropsWhatItHeld(t *testing.T) {
sending, refusals := composeEach([]string{"spare"},
sending, refusals := composeEach([]string{"spare"}, numbered(),
func(string) (sendable, error) { return sendable{}, nil })
if len(sending) != 1 || len(refusals) != 0 {
t.Errorf("an empty declaration must be sent, not skipped or refused: %v / %v", sending, refusals)
@@ -74,3 +74,9 @@ func TestASkippedMachineIsStillAnError(t *testing.T) {
}
}
}
// numbered is an allotter for tests: one higher per call, as the inventory's is per machine.
func numbered() func(string) (int64, error) {
var n int64
return func(string) (int64, error) { n++; return n, nil }
}
+31
View File
@@ -3,6 +3,7 @@ package main
import (
"encoding/json"
"fmt"
"github.com/novox/mesh-controller/internal/inventory"
"sort"
"time"
)
@@ -66,6 +67,21 @@ type meshStatus struct {
// **A document without this said an outage was a well mesh.** Read from what each machine
// reported, so it is the machine's account and not the mesh's take-time listing.
Untaken []machineUntaken `json:"untaken,omitempty"`
// Filtered is every converged machine that is not filtered by the mesh alone (novox/hq ADR
// 0168), one entry per rule set the mesh did not write — the found firewall in force again,
// or a chain nobody speaks for. Absent when every converged machine is filtered by the mesh
// alone. A document without this called a machine well while a predecessor's chain refused
// what the mesh declared open.
Filtered []machineFiltered `json:"filtered,omitempty"`
}
// machineFiltered is one rule set on a converged machine that the mesh did not write and that
// refuses traffic: where it is, whose the host reads it as, and what it refuses.
type machineFiltered struct {
Node string `json:"node"`
Where string `json:"where"`
Owner string `json:"owner"`
Refuses string `json:"refuses"`
}
// machineUntaken is one module a machine is holding rather than running, and how many resources of
@@ -173,6 +189,21 @@ func statusAsJSON(asked answers) ([]byte, error) {
machineUntaken{Node: name, Module: m, Held: asked.untaken[name][m]})
}
}
filteredNodes := make([]string, 0, len(asked.filtered))
for name := range asked.filtered {
filteredNodes = append(filteredNodes, name)
}
sort.Strings(filteredNodes)
for _, name := range filteredNodes {
f := asked.filtered[name]
if fw := f.FoundFirewall; fw != nil && fw.Active {
out.Filtered = append(out.Filtered, machineFiltered{Node: name, Where: "the found firewall",
Owner: inventory.FilterFoundFirewall, Refuses: fw.Kind + " is in force again"})
}
for _, x := range f.Others() {
out.Filtered = append(out.Filtered, machineFiltered{Node: name, Where: x.Where, Owner: x.Owner, Refuses: x.Refuses})
}
}
for name := range asked.refused {
out.Unresolved = append(out.Unresolved, machineUnresolved{
Node: name, Problem: asked.refused[name]})
+43
View File
@@ -167,3 +167,46 @@ func TestAMachineFailingTheSameWayIsSaidToBeStuck(t *testing.T) {
t.Fatalf("one failure is not stuck: %v", once)
}
}
// A converged machine something other than the mesh filters is named, per rule set, and is not
// well (novox/hq ADR 0168); one filtered by the mesh alone is not in the list.
func TestAMachineNotFilteredByTheMeshAloneIsNamedAndNotWell(t *testing.T) {
alone := inventory.Filtering{Filters: []inventory.Filter{
{Where: "table inet mesh, chain input", Owner: inventory.FilterMesh, Refuses: "policy drop"},
{Where: "table ip filter, chain DOCKER", Owner: inventory.FilterRuntime, Refuses: "drop"},
{Where: "table ip filter, chain f2b-sshd", Owner: inventory.FilterBan, Refuses: "ip saddr 192.0.2.1 reject"},
}}
if !alone.Alone() {
t.Fatal("the mesh's own, the runtime's and a ban are not the mesh alone")
}
notAlone := inventory.Filtering{
Filters: append(alone.Filters, inventory.Filter{Where: "chain HAL-MESH-ONLY (iptables-legacy)",
Owner: inventory.FilterOther, Refuses: `-A HAL-MESH-ONLY -m comment --comment "not public" -j DROP`}),
FoundFirewall: &inventory.FoundFirewall{Kind: "ufw", Active: true},
}
asked := answers{nodes: []inventory.Node{{Name: "home-server"}, {Name: "laptop"}},
filtered: map[string]inventory.Filtering{"home-server": notAlone}}
if asked.well() {
t.Fatal("a machine not filtered by the mesh alone reads as well")
}
body, err := statusAsJSON(asked)
if err != nil {
t.Fatal(err)
}
var parsed struct {
Filtered []map[string]string `json:"filtered"`
}
if err := json.Unmarshal(body, &parsed); err != nil {
t.Fatal(err)
}
if len(parsed.Filtered) != 2 {
t.Fatalf("filtered: %v", parsed.Filtered)
}
if parsed.Filtered[0]["node"] != "home-server" || parsed.Filtered[0]["owner"] != inventory.FilterFoundFirewall ||
parsed.Filtered[1]["where"] != "chain HAL-MESH-ONLY (iptables-legacy)" || parsed.Filtered[1]["owner"] != inventory.FilterOther {
t.Fatalf("filtered: %v", parsed.Filtered)
}
if body, _ := statusAsJSON(answers{nodes: asked.nodes}); strings.Contains(string(body), `"filtered"`) {
t.Fatal("a mesh filtered by itself alone carries a filtered list")
}
}
+20 -5
View File
@@ -36,17 +36,29 @@ func tiersOf(set []string, edges []inventory.Edge) [][]string {
for _, m := range set {
deps[m] = map[string]bool{}
}
// The build seat's holders follow the controller that defines their worker (EdgeWorkerOf,
// novox/hq issue 206), so the built-by edge from that controller to such a holder yields: the
// controller is built by whichever build machine is running, as the runtime image always was.
worker := map[string]map[string]bool{}
for _, e := range edges {
if e.Kind == inventory.EdgeWorkerOf && in[e.From] && in[e.To] {
if worker[e.To] == nil {
worker[e.To] = map[string]bool{}
}
worker[e.To][e.From] = true
}
}
for _, e := range edges {
// A code dependency — B packages A's source — rebuilds B with A, in the same tier: B's
// build needs nothing of A's first. The other kinds order: stands-on and declared after
// the base is built, built-by after the build machine is built and running — except for
// what the build machine itself stands on. The runtime image is built by the builder and
// the builder is built on the runtime image; the image comes first, built by the builder
// that is running, which is the only one there could be.
// what the build machine itself stands on, and for the controller whose worker the build
// machine binds. The runtime image is built by the builder and the builder is built on the
// runtime image; the image comes first, built by the builder that is running.
if !in[e.From] || !in[e.To] || e.From == e.To || e.Kind == inventory.EdgePackages {
continue
}
if e.Kind == inventory.EdgeBuiltBy && isBaseOf(e.From, e.To, edges, in) {
if e.Kind == inventory.EdgeBuiltBy && (isBaseOf(e.From, e.To, edges, in) || worker[e.From][e.To]) {
continue
}
deps[e.From][e.To] = true
@@ -122,7 +134,10 @@ func reachableFrom(moved []string, edges []inventory.Edge) []string {
for grew := true; grew; {
grew = false
for _, e := range edges {
if e.Kind == inventory.EdgeBuiltBy {
// Built-by and worker-of order a plan; neither widens it. A new build machine changes
// nothing it builds, and a new controller changes nothing about the holder it orders —
// what packages the controller's source is already a code edge.
if e.Kind == inventory.EdgeBuiltBy || e.Kind == inventory.EdgeWorkerOf {
continue
}
if in[e.To] && !in[e.From] {
+4 -2
View File
@@ -346,7 +346,9 @@ func rolloutMint(ctx context.Context, again bool) error {
}
machines++
case broker.KindModule:
case broker.KindModule, broker.KindNodeTools:
// The runtime is minted and delivered exactly as a module is (novox/hq ADR 0175): it is
// issued as the module it stands for, to that module's `broker` secret.
if p.Module == "mesh-controller" {
// The control plane is a module too, and its `broker` secret is the old bus's
// credential it is still using while this runs. Writing the new bus's blob there
@@ -365,7 +367,7 @@ func rolloutMint(ctx context.Context, again bool) error {
skipped++
continue
}
password, err := inv.MintBusPassword(ctx, inventory.BusUser{Username: p.Username(), Kind: inventory.BusModule, Node: p.Node, Module: p.Module})
password, err := inv.MintBusPassword(ctx, inventory.BusUser{Username: p.Username(), Kind: busKindOf(p.Module), Node: p.Node, Module: p.Module})
if err != nil {
return err
}
+1 -54
View File
@@ -2,13 +2,12 @@ package main
import (
"context"
"strings"
"testing"
)
// A node's own set failing to compose, and the mesh being unable to answer at all, are different
// things, and only the first may be passed over when something is gathered across every machine
// (novox/hq 04-ISSUES/152). These pin that distinction where the three gatherers rely on it.
// (novox/hq 04-ISSUES/152). These pin that distinction where the gatherers rely on it.
func TestASetThatDoesNotComposeIsMarkedAsTheNodesOwnProblem(t *testing.T) {
open := aMesh(t)
@@ -45,55 +44,3 @@ func TestAStoreThatCannotBeReadIsNotANodeThatDoesNotCompose(t *testing.T) {
t.Fatalf("a question the mesh could not answer was read as a node that runs nothing: %v", err)
}
}
func TestOneIncoherentNodeDoesNotCostTheRestTheirNames(t *testing.T) {
open := aMesh(t)
one, two := rivals()
register(t, open, one)
register(t, open, two)
for _, m := range []string{one.Module, two.Module} {
if _, err := open.inventory.Assign(t.Context(), "laptop", m); err != nil {
t.Fatal(err)
}
}
// laptop cannot compose. That is laptop's problem and nobody else's: the roster is still
// answerable, and anchor keeps whatever it serves.
if _, err := routeNamesInTheMesh(t.Context(), open); err != nil {
t.Fatalf("one node's broken set cost the whole mesh its roster: %v", err)
}
}
func TestARosterIsNeverReturnedWithNamesItCouldNotRead(t *testing.T) {
open := aMesh(t)
stopped, cancel := context.WithCancel(t.Context())
cancel()
names, err := routeNamesInTheMesh(stopped, open)
if err == nil {
t.Fatalf("a roster was composed from a store that could not be read: %v", names)
}
// The failure must be raised, not turned into an absence. A roster missing a machine's names
// is indistinguishable, on every machine that receives it, from the operator withdrawing them —
// and because the roster is part of every container's identity, it replaces all of them.
if names != nil {
t.Fatalf("a partial roster was returned beside the error: %v", names)
}
}
// Kept so the reason survives the next person reading it: the message the gatherer raises must say
// which machine could not be read, or the operator is left with a mesh-wide failure and no name.
func TestTheRaisedFailureNamesTheMachineItCouldNotRead(t *testing.T) {
open := aMesh(t)
stopped, cancel := context.WithCancel(t.Context())
cancel()
_, err := routeNamesInTheMesh(stopped, open)
if err == nil {
t.Fatal("no failure was raised")
}
if !strings.Contains(err.Error(), "cannot be read") {
t.Fatalf("the failure does not say the mesh could not be read: %v", err)
}
}
+124 -7
View File
@@ -48,6 +48,21 @@ func argvFor(verb string, args map[string]any) ([]string, error) {
return nil
}
switch verb {
case "command":
// The generic verb: the command line as given, split as a shell would split it, with
// nothing added — the named verbs add flags a caller cannot reach; this one is the whole
// binary and says so in its description (novox/hq ADR 0154, 0175).
if err := need("command"); err != nil {
return nil, err
}
argv, err := splitCommandLine(str("command"))
if err != nil {
return nil, err
}
if len(argv) == 0 {
return nil, errors.New("command names no command")
}
return argv, nil
case "status":
return []string{"status", "--json"}, nil
case "nodes":
@@ -129,6 +144,33 @@ func argvFor(verb string, args map[string]any) ([]string, error) {
// Half of either shape: the command says its usage, which names both shapes, and that is
// the answer the caller needs.
return []string{"rotate"}, nil
case "settings":
// `settings set|clear` at a shell (novox/hq issue 198). The values travel as an argument
// because a tool has no file to hand the command; the command reads either.
if err := need("module"); err != nil {
return nil, err
}
argv := []string{"settings", "set", str("module")}
switch {
case str("clear") == "true":
argv = []string{"settings", "clear", str("module")}
case str("values") != "":
argv = append(argv, str("values"))
}
// Neither values nor clear: the command says its usage, which names both, and that is the
// answer the caller needs — the same as `rotate` given half of either shape.
if n := str("node"); n != "" {
argv = append(argv, "--node", n)
}
return argv, nil
case "issue":
// The same act as `module issue` at a shell (novox/hq design 25 §4): the account is minted
// into the mesh's records and delivered at the machine's next push, which is the caller's to
// ask for — so the mesh is never pushed as a side effect of a credential.
if err := need("node", "module"); err != nil {
return nil, err
}
return []string{"module", "issue", str("module"), "--node", str("node")}, nil
case "build":
if err := need("repository"); err != nil {
return nil, err
@@ -188,13 +230,15 @@ func runVerb(ctx context.Context, argv []string) (verbAnswer, error) {
}
// seatToolHandlers are the handlers for every verb the mesh-controller seat declares, from the
// store's row, so a verb the row does not carry is not served and a verb it carries that this binary
// cannot run is said at start rather than at the first call.
func seatToolHandlers() (map[string]link.ToolHandler, error) {
// store's row, so a verb the row does not carry is not served. A verb it carries that this binary
// cannot run is named at start and answers the reason when called — never a refusal to serve, which
// would take the whole control plane down for one word (novox/hq ADR 0185).
func seatToolHandlers() (map[string]link.ToolHandler, []string, error) {
seat, known := catalogue.SeatNamed(catalogue.ControllerSeatName)
if !known {
return nil, fmt.Errorf("this mesh defines no %s seat", catalogue.ControllerSeatName)
return nil, nil, fmt.Errorf("this mesh defines no %s seat", catalogue.ControllerSeatName)
}
var behind []string
handlers := map[string]link.ToolHandler{}
for _, v := range seat.Serves {
verb := v.Name
@@ -205,8 +249,27 @@ func seatToolHandlers() (map[string]link.ToolHandler, error) {
continue
}
if _, err := argvFor(verb, sampleArguments(v)); err != nil {
return nil, fmt.Errorf("the %s seat's row declares %q, which this control plane cannot run: %w",
catalogue.ControllerSeatName, verb, err)
// **A row ahead of this binary is not a reason to go silent.**
//
// The row is the store's and a control plane follows it (novox/hq ADR 0154), so a verb
// this build does not know means the row was widened by a newer one — the ordinary
// state of a roll-out, and of a push that put an older control plane back. Refusing to
// serve at all made that transient fatal: on 2026-10-02 one unknown verb took the whole
// mesh off the bus for ten minutes, and the way back was a human running the binary by
// hand, because the thing that would have repaired it is the thing that was down
// (novox/hq 04-ISSUES/201, ADR 0185).
//
// So the verbs this binary knows are served, and this one answers the reason instead of
// nothing: a caller gets a sentence naming the fault, and everything else keeps working
// — including the push that replaces this binary with the one whose verb it is.
behind = append(behind, verb)
reason := err
handlers[verb] = func(context.Context, json.RawMessage) (any, error) {
return nil, fmt.Errorf("%s is in this mesh's %s row and the control plane running "+
"here cannot run it: %w. It is a verb of a newer build; this one is behind",
verb, catalogue.ControllerSeatName, reason)
}
continue
}
handlers[verb] = func(ctx context.Context, raw json.RawMessage) (any, error) {
args := map[string]any{}
@@ -222,7 +285,7 @@ func seatToolHandlers() (map[string]link.ToolHandler, error) {
return runVerb(ctx, argv)
}
}
return handlers, nil
return handlers, behind, nil
}
// seatTools is what `tools` answers: every seat with a protocol, and the tools each serves, from the
@@ -262,3 +325,57 @@ func sampleArguments(v catalogue.Verb) map[string]any {
}
return sample
}
// splitCommandLine splits a command line into words the way a POSIX shell does for the simple
// cases a controller command needs: spaces separate, single or double quotes group, a backslash
// escapes the next character inside double quotes or outside any. No expansion of anything.
func splitCommandLine(line string) ([]string, error) {
var words []string
var cur strings.Builder
inWord := false
quote := rune(0)
runes := []rune(line)
for i := 0; i < len(runes); i++ {
r := runes[i]
switch {
case quote == '\'':
if r == '\'' {
quote = 0
} else {
cur.WriteRune(r)
}
case quote == '"':
if r == '"' {
quote = 0
} else if r == '\\' && i+1 < len(runes) {
i++
cur.WriteRune(runes[i])
} else {
cur.WriteRune(r)
}
case r == '\'' || r == '"':
quote = r
inWord = true
case r == '\\' && i+1 < len(runes):
i++
cur.WriteRune(runes[i])
inWord = true
case r == ' ' || r == '\t' || r == '\n':
if inWord {
words = append(words, cur.String())
cur.Reset()
inWord = false
}
default:
cur.WriteRune(r)
inWord = true
}
}
if quote != 0 {
return nil, fmt.Errorf("command has an unclosed %c quote", quote)
}
if inWord {
words = append(words, cur.String())
}
return words, nil
}
+111 -1
View File
@@ -1,6 +1,7 @@
package main
import (
"context"
"strings"
"testing"
@@ -73,6 +74,38 @@ func TestRotateTakesAProvisionOrAnOwnSecret(t *testing.T) {
}
}
// `settings` is `settings set|clear` at a shell, with the values passed inline (novox/hq issue 198).
func TestSettingsSetsOrClearsALayer(t *testing.T) {
argv, err := argvFor("settings", map[string]any{"module": "dnsmasq", "values": `{"a":1}`, "node": "ace"})
if err != nil || strings.Join(argv, " ") != `settings set dnsmasq {"a":1} --node ace` {
t.Fatalf("set on a machine: %v %v", argv, err)
}
argv, _ = argvFor("settings", map[string]any{"module": "dnsmasq", "clear": "true"})
if strings.Join(argv, " ") != "settings clear dnsmasq" {
t.Fatalf("clear for the mesh: %v", argv)
}
argv, _ = argvFor("settings", map[string]any{"module": "dnsmasq"})
if strings.Join(argv, " ") != "settings set dnsmasq" {
t.Fatalf("a set with no values falls to the command's usage: %v", argv)
}
}
// `issue` is `module issue` at a shell: the module and the machine, and nothing that would push. A
// module's bus account was mintable only from the controller's command line, so an agent working
// through the tools could not finish a rollout that gave a module one (novox/hq issue 191).
func TestIssueGivesAModuleItsAccountOnAMachine(t *testing.T) {
argv, err := argvFor("issue", map[string]any{"node": "ace", "module": "route-proxy"})
if err != nil {
t.Fatal(err)
}
if strings.Join(argv, " ") != "module issue route-proxy --node ace" {
t.Fatalf("issue runs %v", argv)
}
if _, err := argvFor("issue", map[string]any{"module": "route-proxy"}); err == nil {
t.Error("an account was issued without saying which machine reads it")
}
}
// A required argument missing is refused in the verb's own words, before anything runs.
func TestAVerbMissingWhatItNeedsIsRefused(t *testing.T) {
if _, err := argvFor("node", map[string]any{}); err == nil || !strings.Contains(err.Error(), `node needs "node"`) {
@@ -98,10 +131,13 @@ func TestActsDoNotBlockTheCall(t *testing.T) {
// What `tools` answers is the seats' records, with each verb's schema.
func TestToolsAnswersTheSeatsRecords(t *testing.T) {
handlers, err := seatToolHandlers()
handlers, behind, err := seatToolHandlers()
if err != nil {
t.Fatal(err)
}
if len(behind) != 0 {
t.Fatalf("this build cannot run %v of its own seat's verbs", behind)
}
if len(handlers) != len(catalogue.ControllerVerbs) {
t.Fatalf("%d handlers for %d verbs", len(handlers), len(catalogue.ControllerVerbs))
}
@@ -139,3 +175,77 @@ func TestAJSONVerbsAnswerIsItsStandardOutput(t *testing.T) {
t.Fatalf("stderr and stdout are both what the command said: %s", answer.Output)
}
}
// `command` is the generic verb: the command line as given, split as a shell would, nothing added —
// so an operator's `node account g14 jochen` is one call through the console rather than a shell on
// the control node (novox/hq ADR 0154, ADR 0175).
func TestCommandRunsTheLineAsGiven(t *testing.T) {
argv, err := argvFor("command", map[string]any{"command": "node account g14 jochen"})
if err != nil || strings.Join(argv, " ") != "node account g14 jochen" {
t.Fatalf("a plain line: %v %v", argv, err)
}
argv, err = argvFor("command", map[string]any{"command": `settings set dnsmasq '{"a": "b c"}' --node ace`})
if err != nil || len(argv) != 6 || argv[3] != `{"a": "b c"}` {
t.Fatalf("a quoted word stays one word: %q %v", argv, err)
}
argv, err = argvFor("command", map[string]any{"command": `node add "the box" --adopted`})
if err != nil || len(argv) != 4 || argv[2] != "the box" {
t.Fatalf("double quotes group: %q %v", argv, err)
}
if _, err := argvFor("command", map[string]any{"command": " "}); err == nil {
t.Fatal("an empty line was accepted")
}
if _, err := argvFor("command", map[string]any{"command": `node "unclosed`}); err == nil {
t.Fatal("an unclosed quote was accepted")
}
}
// A verb in the row that this binary cannot run does not take the control plane off the bus: the
// rest are served, the unknown one answers the reason, and the start-up names it (novox/hq ADR
// 0185). One unknown word cost the mesh ten minutes of silence on 2026-10-02, recoverable only by
// a person running the binary by hand — the push that would have repaired it needs the control
// plane that was down.
func TestARowAheadOfThisBuildIsServedAnyway(t *testing.T) {
seat, known := catalogue.SeatNamed(catalogue.ControllerSeatName)
if !known {
t.Fatal("no controller seat")
}
// The row as a newer control plane would have written it: every verb this build knows, and one
// it does not.
widened := seat
widened.Serves = append(append([]catalogue.Verb{}, seat.Serves...),
catalogue.Verb{Name: "teleport", Description: "a verb from a build that does not exist yet"})
rows := catalogue.DefaultSeats()
for i := range rows {
if rows[i].Name == catalogue.ControllerSeatName {
rows[i] = widened
}
}
catalogue.UseSeats(rows)
t.Cleanup(func() { catalogue.UseSeats(catalogue.DefaultSeats()) })
handlers, behind, err := seatToolHandlers()
if err != nil {
t.Fatalf("a row with one unknown verb refused to serve at all: %v", err)
}
if len(behind) != 1 || behind[0] != "teleport" {
t.Fatalf("the verbs this build cannot run were reported as %v", behind)
}
if len(handlers) != len(widened.Serves) {
t.Fatalf("%d handlers for %d verbs in the row", len(handlers), len(widened.Serves))
}
for _, known := range []string{"status", "nodes", "push"} {
if handlers[known] == nil {
t.Errorf("%s is not served although this build knows it", known)
}
}
_, err = handlers["teleport"](context.Background(), nil)
if err == nil {
t.Fatal("the unknown verb answered as though it had run")
}
for _, want := range []string{"teleport", "cannot run it", "behind"} {
if !strings.Contains(err.Error(), want) {
t.Errorf("the answer does not say %q: %v", want, err)
}
}
}
+11
View File
@@ -31,6 +31,14 @@ type sendable struct {
// the same composition as its received files, and every machine's private-network address.
Received map[string]map[string][]catalogue.Contribution
Mesh []string
// LeftOut is every module of the machine's set left out of this declaration because a stored
// setting cannot compose with its definition (novox/hq ADR 0163, rule 6), sorted. The host
// keeps that module's held things and touches none of its containers; a machine is told
// everything or nothing about what it IS told, and what it is not told is said. Absent from
// the body when empty, so a declaration that leaves nothing out is byte for byte what it was.
LeftOut []string
// leftOutWhy is why each was, for push and plan to say; never on the wire.
leftOutWhy map[string]string
}
// adoptionEnvelope is what an adopted node is told about its mode. Taken is every module taken on
@@ -51,6 +59,9 @@ func (s sendable) Body() ([]byte, error) {
if s.Sequence > 0 {
envelope["sequence"] = s.Sequence
}
if len(s.LeftOut) > 0 {
envelope["left_out"] = s.LeftOut
}
// An empty declaration is deliberate here — the node owns nothing the mesh put there
// (novox/hq issue 127) — and the host refuses an empty body unless it is told the emptiness
// is meant, so a truncated or mis-composed body is never mistaken for "own nothing".
+59
View File
@@ -382,3 +382,62 @@ func TestAnEmptyDeclarationSaysOwnsNothing(t *testing.T) {
t.Fatalf("a non-empty declaration must not mark owns_nothing; got %v", env)
}
}
// A setting is judged where it is stored, and an impossible one costs a module, not a machine
// (novox/hq ADR 0163, rule 6): stored while it composed, a setting whose definition then moved from
// under it leaves that module out of the declaration — said in the envelope, so the host keeps the
// module's things — and the machine is told everything else.
func TestADefinitionMovingUnderAStoredSettingLeavesThatModuleOutNotTheMachine(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
web := helloWeb()
web.Resources[1]["ports"] = []any{"8080"}
register(t, open, web)
register(t, open, catalogue.Manifest{Module: "notes", Version: "1",
Resources: []map[string]any{{"id": "conf", "type": "file", "path": "/etc/notes.conf", "content": "x"}}})
for _, m := range []string{"hello-web", "notes"} {
if _, err := assign(ctx, open, "laptop", m); err != nil {
t.Fatal(err)
}
}
// Judged where it is stored: a port the module does not publish is refused by name.
err := open.inventory.SetSettings(ctx, "laptop", "hello-web",
map[string]any{catalogue.PortsSetting: map[string]any{"9999": 10000}})
if err == nil || !strings.Contains(err.Error(), "hello-web on laptop") || !strings.Contains(err.Error(), "9999") {
t.Fatalf("an impossible setting was stored: %v", err)
}
if err := open.inventory.SetSettings(ctx, "laptop", "hello-web",
map[string]any{catalogue.PortsSetting: map[string]any{"8080": 10000}}); err != nil {
t.Fatal(err)
}
if declared := composed(t, open, "laptop"); len(declared.LeftOut) != 0 {
t.Fatalf("a setting that composes left a module out: %v", declared.LeftOut)
}
// The definition moves: the container publishes another port now.
web.Version = "2"
web.Resources[1]["ports"] = []any{"9090"}
register(t, open, web)
declared := composed(t, open, "laptop")
if len(declared.LeftOut) != 1 || declared.LeftOut[0] != "hello-web" {
t.Fatalf("hello-web is not left out: %v", declared.LeftOut)
}
if !strings.Contains(declared.leftOutWhy["hello-web"], "no container of its publishes 8080") {
t.Fatalf("why it was left out is not said: %v", declared.leftOutWhy)
}
if hasID(declared.Resources, "hello-web.server") || !hasID(declared.Resources, "notes.conf") {
t.Fatalf("the machine was not told everything else: %v", declared.Resources)
}
body, err := declared.Body()
if err != nil {
t.Fatal(err)
}
var env map[string]any
if err := json.Unmarshal(body, &env); err != nil {
t.Fatal(err)
}
left, _ := env["left_out"].([]any)
if len(left) != 1 || left[0] != "hello-web" {
t.Fatalf("the envelope does not say what was left out: %v", env)
}
}
+55 -1
View File
@@ -227,6 +227,28 @@ func printStatus(asked answers) error {
" not readable from a commit; that needs a version the host reports as ordered\n\n")
}
if len(asked.filtered) > 0 {
// A converged machine is filtered by the mesh alone, and the mesh says truthfully which
// (novox/hq ADR 0168). One that is not — a predecessor's chain still refusing, a found
// firewall in force again — is said here, and is not well.
machines := make([]string, 0, len(asked.filtered))
for name := range asked.filtered {
machines = append(machines, name)
}
sort.Strings(machines)
fmt.Printf("%d converged machine(s) are not filtered by the mesh alone:\n", len(machines))
for _, name := range machines {
f := asked.filtered[name]
if fw := f.FoundFirewall; fw != nil && fw.Active {
fmt.Printf(" %-12s the found firewall (%s) is in force again; the next apply retires it\n", name, fw.Kind)
}
for _, x := range f.Others() {
fmt.Printf(" %-12s %s (%s): %s\n", name, x.Where, x.Owner, x.Refuses)
}
}
fmt.Printf("\n the mesh wrote none of these and removes none; `node show <node>` lists every filter with its owner\n\n")
}
if len(asked.untaken) > 0 {
// **Before the adopted line, and it breaks "all well".** An adopted machine is a state
// somebody chose and can leave alone; a module assigned to one and never taken is work
@@ -357,6 +379,13 @@ func theThreeQuestions(ctx context.Context, open *stores) (answers, error) {
if err != nil {
return answers{}, err
}
// And which converged machines something other than the mesh filters (novox/hq ADR 0168), as
// each last reported — the account that was missing when a predecessor's chain refused what the
// mesh declared open for eleven hours (04-ISSUES/144, 145).
out.filtered, err = filteredMachines(ctx, inv, out.nodes)
if err != nil {
return answers{}, err
}
out.plans, err = inv.RecentPlans(ctx, 5)
if err != nil {
return answers{}, err
@@ -406,6 +435,30 @@ func theThreeQuestions(ctx context.Context, open *stores) (answers, error) {
//
// A machine that reports no holds contributes nothing, so a converged mesh answers an empty map and
// the caller prints nothing.
// filteredMachines is every converged machine not filtered by the mesh alone, with what it last said
// filters it (novox/hq ADR 0168). An adopted machine keeps its found firewall by design and is not
// counted; a machine that has not said is not said to be filtered by anything.
func filteredMachines(ctx context.Context, inv *inventory.Inventory, nodes []inventory.Node) (
map[string]inventory.Filtering, error) {
out := map[string]inventory.Filtering{}
for _, n := range nodes {
if n.Adopted {
continue
}
f, err := inv.FilteringOf(ctx, n.Name)
if err != nil {
return nil, fmt.Errorf("what filters %s cannot be read: %w", n.Name, err)
}
if len(f.Filters) == 0 && f.FoundFirewall == nil {
continue
}
if !f.Alone() {
out[n.Name] = f
}
}
return out, nil
}
func untakenModules(ctx context.Context, inv *inventory.Inventory, nodes []inventory.Node) (
map[string]map[string]int, error) {
@@ -442,7 +495,8 @@ func untakenModules(ctx context.Context, inv *inventory.Inventory, nodes []inven
// read as success for the whole of the edge cut-over outage (novox/hq 04-ISSUES/125).
func (a answers) well() bool {
return len(a.wrong) == 0 && len(a.quiet) == 0 && len(a.behind) == 0 &&
len(a.waiting) == 0 && len(a.refused) == 0 && a.network == "" && len(a.untaken) == 0
len(a.waiting) == 0 && len(a.refused) == 0 && a.network == "" && len(a.untaken) == 0 &&
len(a.filtered) == 0
}
// hostSplit is which machines report which host version, for every version more than one machine
+110 -16
View File
@@ -4,26 +4,42 @@ import (
"strings"
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
)
// What the forge's take compares, as a machine would report it.
func aForgeComparison() comparison {
return comparison{reported: inventory.Adoption{
Firewall: "ufw",
Held: []inventory.Held{
{ID: "forge.server", Module: "forge", Kind: "container", Target: "forge", Facts: map[string]any{
"image": "forge:1.27.3", "image_created": "2026-09-17T10:00:00Z",
"declared_image": "forge:1.22.6", "declared_image_created": "2026-08-20T10:00:00Z", "downgrade": true,
"networks": map[string]any{"predecessor_default": []any{"office", "db"}},
"ports": []any{"3000/tcp>0.0.0.0:3000"}, "declared_ports": []any{"3000:3000"},
}},
{ID: "forge.config", Module: "forge", Kind: "file", Target: "/etc/forge/app.ini", Kept: "/var/lib/mesh/kept/app.ini",
Facts: map[string]any{"differs": true, "difference": []any{"- private scope: local", "+ upstream: public"}}},
{ID: "other.server", Module: "other", Kind: "container", Target: "other"},
},
Reachable: []inventory.Reach{
{Protocol: "tcp", Address: "0.0.0.0", Port: 3000, By: "forge", Published: true, ContainerPort: 3000},
{Protocol: "tcp", Address: "0.0.0.0", Port: 22, By: "sshd"},
},
}}
}
// A take is a comparison (novox/hq ADR 0163): the preview puts what runs beside what the module
// declares, and an older image or a differing file refuses unless named.
func TestATakePreviewsTheComparisonAndRefusesWhatIsNotNamed(t *testing.T) {
held := []inventory.Held{
{ID: "forge.server", Module: "forge", Kind: "container", Target: "forge", Facts: map[string]any{
"image": "forge:1.27.3", "image_created": "2026-09-17T10:00:00Z",
"declared_image": "forge:1.22.6", "declared_image_created": "2026-08-20T10:00:00Z", "downgrade": true,
"networks": map[string]any{"predecessor_default": []any{"office", "db"}},
"ports": []any{"3000/tcp>0.0.0.0:3000"}, "declared_ports": []any{"3000:3000"},
}},
{ID: "forge.config", Module: "forge", Kind: "file", Target: "/etc/forge/app.ini", Kept: "/var/lib/mesh/kept/app.ini",
Facts: map[string]any{"differs": true, "difference": []any{"- private scope: local", "+ upstream: public"}}},
{ID: "other.server", Module: "other", Kind: "container", Target: "other"},
}
preview, refusals := comparisonOf(held, "forge", takeOptions{})
c := aForgeComparison()
preview, refusals, saw := comparisonOf("forge", c, takeOptions{})
for _, want := range []string{"runs forge:1.27.3 (made 2026-09-17)", "declares forge:1.22.6 (made 2026-08-20)", "DOWNGRADE",
"on the network predecessor_default with office, db", "publishes 3000/tcp>0.0.0.0:3000; the module declares 3000:3000",
"on the network predecessor_default with office, db", "will not once it moves to the module's own network",
"publishes 3000/tcp>0.0.0.0:3000; the module declares 3000:3000",
// How far the port reaches now, as the machine reported it (rule 1).
"reachable now at 0.0.0.0:3000 (tcp, container port 3000), behind the found firewall (ufw)",
"- private scope: local", "original kept at /var/lib/mesh/kept/app.ini"} {
if !strings.Contains(preview, want) {
t.Errorf("the preview lacks %q:\n%s", want, preview)
@@ -35,15 +51,93 @@ func TestATakePreviewsTheComparisonAndRefusesWhatIsNotNamed(t *testing.T) {
if len(refusals) != 2 || !strings.Contains(refusals[0], "--downgrade") || !strings.Contains(refusals[1], "--replace /etc/forge/app.ini") {
t.Fatalf("the downgrade and the differing file refuse, each naming its override: %v", refusals)
}
if len(saw) != 12 {
t.Fatalf("the preview's digest is %q", saw)
}
// Named, they pass.
if _, refusals := comparisonOf(held, "forge", takeOptions{Downgrade: true, Replace: map[string]bool{"/etc/forge/app.ini": true}}); len(refusals) != 0 {
if _, refusals, _ := comparisonOf("forge", c, takeOptions{Downgrade: true, Replace: map[string]bool{"/etc/forge/app.ini": true}}); len(refusals) != 0 {
t.Fatalf("named differences still refused: %v", refusals)
}
if _, refusals := comparisonOf(held, "forge", takeOptions{Downgrade: true, Replace: map[string]bool{"*": true}}); len(refusals) != 0 {
if _, refusals, _ := comparisonOf("forge", c, takeOptions{Downgrade: true, Replace: map[string]bool{"*": true}}); len(refusals) != 0 {
t.Fatalf("replace * did not cover the file: %v", refusals)
}
// A held thing with no facts yet — a host older than this — refuses nothing and says what it can.
if preview, refusals := comparisonOf(held, "other", takeOptions{}); len(refusals) != 0 || !strings.Contains(preview, "container other") {
if preview, refusals, _ := comparisonOf("other", c, takeOptions{}); len(refusals) != 0 || !strings.Contains(preview, "container other") {
t.Fatalf("a factless hold: %q %v", preview, refusals)
}
// The digest is of what the preview says: a fact changing changes it.
c.reported.Held[0].Facts["image"] = "forge:1.27.4"
if _, _, again := comparisonOf("forge", c, takeOptions{}); again == saw {
t.Fatal("the found image changed and the digest did not")
}
}
// A secret the mesh minted for a service whose data was found refuses: the running service already
// has a value (rule 2). Accepted, it is carried in; `--mint` says the service shall take the new one.
func TestAMintedSecretForFoundDataRefusesUnlessAcceptedOrMinted(t *testing.T) {
c := aForgeComparison()
c.secrets = []inventory.SecretState{
{Name: "admin", Origin: inventory.OriginMade},
{Name: "postgres-database", Origin: inventory.OriginMade, Provider: "anchor"},
{Name: "broker", Origin: inventory.OriginAccepted},
}
preview, refusals, _ := comparisonOf("forge", c, takeOptions{Downgrade: true, Replace: map[string]bool{"*": true}})
for _, want := range []string{
"own secret admin: MINTED by the mesh and not accepted",
"secret from anchor postgres-database: MINTED by the mesh and not accepted",
"own secret broker: accepted from a person, carried in as it is",
} {
if !strings.Contains(preview, want) {
t.Errorf("the preview lacks %q:\n%s", want, preview)
}
}
if len(refusals) != 2 {
t.Fatalf("two minted secrets refuse: %v", refusals)
}
if !strings.Contains(refusals[0], "`secret accept <node> forge admin`") || !strings.Contains(refusals[0], "`--mint admin`") {
t.Errorf("the own secret's refusal names accepting it and minting it: %s", refusals[0])
}
if !strings.Contains(refusals[1], "`secret accept <node> forge postgres-database --provider anchor`") {
t.Errorf("the required secret's refusal names its provider: %s", refusals[1])
}
preview, refusals, _ = comparisonOf("forge", c, takeOptions{Downgrade: true, Replace: map[string]bool{"*": true},
Mint: map[string]bool{"admin": true, "postgres-database": true}})
if len(refusals) != 0 || !strings.Contains(preview, "admin: minted by the mesh; the service takes the new value, as --mint said") {
t.Fatalf("--mint did not pass the minted secrets: %v\n%s", refusals, preview)
}
// With no found data — only a file held — the service has no value of its own, and a minted
// secret is simply said.
c.reported.Held = c.reported.Held[1:2]
if _, refusals, _ := comparisonOf("forge", c, takeOptions{Replace: map[string]bool{"*": true}}); len(refusals) != 0 {
t.Fatalf("a minted secret refused with no data found: %v", refusals)
}
}
// A found network a per-machine setting keeps is named in the preview (rule 4), and the module's
// settings are said with where each came from, composed or not (rules 1 and 6).
func TestTheKeptNetworkAndTheSettingsAreInThePreview(t *testing.T) {
c := aForgeComparison()
c.keeps = map[string][]string{"forge.server": {"predecessor_default"}}
c.layers = []catalogue.Layer{
{From: catalogue.MeshWideLayer, Values: map[string]any{"site": "x"}},
{From: "anchor", Values: map[string]any{catalogue.NetworksSetting: map[string]any{"server": []any{"predecessor_default"}}}},
}
preview, _, _ := comparisonOf("forge", c, takeOptions{Downgrade: true, Replace: map[string]bool{"*": true}})
for _, want := range []string{
"on the network predecessor_default with office, db — kept by this machine's setting, so they still reach it by name once taken",
"settings from the mesh: site",
"settings from anchor: networks",
} {
if !strings.Contains(preview, want) {
t.Errorf("the preview lacks %q:\n%s", want, preview)
}
}
if strings.Contains(preview, "will not once it moves") {
t.Errorf("a kept network is still said to be lost:\n%s", preview)
}
c.settingsRefused = "forge: ports is a { port: machine-port } map"
preview, _, _ = comparisonOf("forge", c, takeOptions{Downgrade: true, Replace: map[string]bool{"*": true}})
if !strings.Contains(preview, "SETTINGS DO NOT COMPOSE with the module's definition, so the push leaves it out: forge: ports") {
t.Errorf("settings that cannot compose are not said:\n%s", preview)
}
}
+45
View File
@@ -0,0 +1,45 @@
package main
import (
"testing"
"github.com/novox/mesh-controller/internal/inventory"
)
// The holder of the build seat follows the controller that defines its worker (novox/hq issue 206).
// On 2026-10-03 a plan put the build machine in tier 0 and the controller in tier 1; the new build
// machine could not bind the worker the old controller had defined, and nothing could build the
// controller that would have redefined it. The built-by edge from the controller to its build
// machine yields to that order: the controller is built by whichever build machine is running.
func TestTheBuildSeatsHolderFollowsTheControllerThatDefinesItsWorker(t *testing.T) {
edges := []inventory.Edge{
{From: "build-agent", To: "mesh-controller", Kind: inventory.EdgePackages},
{From: "build-agent", To: "mesh-controller", Kind: inventory.EdgeWorkerOf},
{From: "mesh-controller", To: "build-agent", Kind: inventory.EdgeBuiltBy},
{From: "route-proxy", To: "mesh-controller", Kind: inventory.EdgePackages},
{From: "route-proxy", To: "build-agent", Kind: inventory.EdgeBuiltBy},
}
set := reachableFrom([]string{"mesh-controller"}, edges)
if len(set) != 3 {
t.Fatalf("the controller, what packages it, and nothing more: %v", set)
}
tiers := tiersOf(set, edges)
pos := map[string]int{}
for i, tier := range tiers {
for _, m := range tier {
pos[m] = i
}
}
if pos["mesh-controller"] != 0 {
t.Fatalf("the controller first, built by the build machine that is running: %v", tiers)
}
if pos["build-agent"] <= pos["mesh-controller"] {
t.Fatalf("the build machine after the controller that defines its worker: %v", tiers)
}
if pos["route-proxy"] <= pos["build-agent"] {
t.Fatalf("what the build machine builds comes after it: %v", tiers)
}
if hasCycle(tiers, edges) {
t.Fatalf("no cycle here: %v", tiers)
}
}
+4 -1
View File
@@ -10,7 +10,10 @@
# The client is copied from the vendor's own image rather than installed from a distribution:
# `apk add mc` on Alpine installs Midnight Commander, which is a different program with the same
# name, and the failure would be a provisioner that starts cleanly and cannot do anything.
FROM golang:1.25-alpine AS build
# The Go it builds with is the one the manifest pins (build.on GO_BASE), passed by the Makefile and the
# build machine alike; the default only serves a hand build, and matches go.mod.
ARG GO_BASE=golang:1.26-alpine
FROM ${GO_BASE} AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
+4 -1
View File
@@ -3,7 +3,10 @@
# Built here so a machine can be given it by the mesh rather than by somebody putting a binary on
# it. Static and FROM scratch for the same reason the control plane's image is: it is fetched by
# digest and run on a machine, and everything in it is something a person would have to audit.
FROM golang:1.25-alpine AS build
# The Go it builds with is the one the manifest pins (build.on GO_BASE), passed by the Makefile and the
# build machine alike; the default only serves a hand build, and matches go.mod.
ARG GO_BASE=golang:1.26-alpine
FROM ${GO_BASE} AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
+4 -1
View File
@@ -2,7 +2,10 @@
#
# FROM scratch, like the postgres one and unlike the bucket one: it speaks the store's own wire
# protocol directly and needs no client in the image.
FROM golang:1.25-alpine AS build
# The Go it builds with is the one the manifest pins (build.on GO_BASE), passed by the Makefile and the
# build machine alike; the default only serves a hand build, and matches go.mod.
ARG GO_BASE=golang:1.26-alpine
FROM ${GO_BASE} AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
+4 -1
View File
@@ -2,7 +2,10 @@
#
# Static and FROM scratch like the control plane's image, and for the same reason: it is fetched
# by digest and run on a machine, so everything in it is something a person would have to audit.
FROM golang:1.25-alpine AS build
# The Go it builds with is the one the manifest pins (build.on GO_BASE), passed by the Makefile and the
# build machine alike; the default only serves a hand build, and matches go.mod.
ARG GO_BASE=golang:1.26-alpine
FROM ${GO_BASE} AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
+5
View File
@@ -732,6 +732,11 @@ func handler(held *table) http.Handler {
// And since a host may now be routed only on some paths, those are a third thing:
// saying "no route for this name" while listing that very name as served is a
// contradiction an operator would have to disbelieve the proxy to get past.
// **Said in the log as well as to the client.** A name this mesh does not serve, asked
// for from outside, is what a scanner does, and the machine's intrusion prevention reads
// this proxy's log for exactly that line (novox/hq ADR 0179): the address last, as the
// jail's filter expects it.
log.Printf("refused: no route for %q, asked from %s", r.Host, r.RemoteAddr)
w.Header().Set("Content-Type", "text/plain; charset=utf-8")
w.WriteHeader(http.StatusNotFound)
if !hidden && held.routed(r.Host) {
@@ -234,3 +234,13 @@ func admitsSubject(pattern, subject []string) bool {
}
return len(pattern) == len(subject)
}
// The two packages name the runtime module separately — the broker's types stay free of the
// catalogue's on purpose — so this is what holds them to one string. A rename that reached only one
// side would compose a runtime principal for a module nobody assigns, silently, and leave the one
// that is assigned with a module's own grants.
func TestTheBrokerAndTheCatalogueAgreeOnTheRuntimeModule(t *testing.T) {
if RuntimeModule != catalogue.RuntimeModule {
t.Fatalf("the broker calls the runtime %q and the catalogue %q", RuntimeModule, catalogue.RuntimeModule)
}
}
+16 -15
View File
@@ -144,12 +144,18 @@ func ConsumerFor(p Principal) (Consumer, bool) {
}, true
}
// HolderConsumerFor is the worker a seat's holder gets on that seat's work queue.
// HolderConsumerFor is the worker a seat's holders share on that seat's work queue.
//
// **A queue group even though the seat guarantees one holder.** The seat is *authority* — who may
// be the telegram sender — and the queue group is *delivery*. Tie delivery to the seat and the
// day somebody allows two holders for throughput, every message is processed twice with nothing
// reporting it. Kept separate, relaxing one changes nothing about the other.
// **One worker for every holder, and each holder pulls one ask when it is idle** (novox/hq ADR
// 0190). The seat is *authority* — who may be the telegram sender — and the worker is *delivery*,
// kept separate so that relaxing one changes nothing about the other: a node-scoped seat has a
// holder per machine, and all of them take from this one consumer, so the work is shared without
// any holder knowing about the others. Pulled rather than pushed because a push consumer hands the
// next ask to whichever subscriber the server picks, busy or not, and a pulled one is asked for by
// a holder that has just become free. Which is also what ends the race issue 186 describes — asks
// delivered behind the one being worked, expiring unacknowledged and dropped after the fifth
// redelivery: nothing is delivered that nobody asked for. A long build keeps its own ask alive
// (stillWorking); the ack wait is for a holder that died.
func HolderConsumerFor(node, module string, seat DeclaredSeat) (Consumer, bool) {
if len(seat.Accepts) == 0 {
return Consumer{}, false
@@ -158,18 +164,13 @@ func HolderConsumerFor(node, module string, seat DeclaredSeat) (Consumer, bool)
Name: "SEAT_" + upperSnake(seat.Name) + "_worker",
Stream: seatStreamName(seat.Name),
Filters: []string{"mesh.seat." + seat.Name + ".accept.>"},
Queue: "holders",
AckWaitSeconds: 60,
MaxDeliver: 5,
// **One in flight.** A holder works one ask at a time, so the server hands it one at a
// time: with the default of many, every ask behind the one being worked was delivered,
// left unacknowledged for the length of the work, redelivered after the ack wait, and
// after the fifth time dropped — on 2026-10-01 twenty-six of forty-three builds asked in
// two minutes were never built, and the queue read as empty (novox/hq issue 186).
MaxAckPending: 1,
Why: fmt.Sprintf("%s on %s holds %s; it acknowledges after the work is done, so a "+
"crash mid-work redelivers rather than loses; one in flight, so a queue of asks is a "+
"queue and not a race against the ack wait", module, node, seat.Name),
// As many in flight as there are holders working, which pulling bounds by itself: a holder
// fetches one and fetches again only after it acknowledged. The server's default stands.
Why: fmt.Sprintf("%s on %s holds %s; every holder pulls one ask at a time from this worker "+
"and acknowledges after the work is done, so a crash mid-work redelivers rather than "+
"loses and an idle holder is the one that takes the next ask", module, node, seat.Name),
}, true
}
+25 -12
View File
@@ -88,15 +88,20 @@ func TestAModuleThatConsumesNothingGetsNoConsumer(t *testing.T) {
}
}
// The seat is authority and the queue group is delivery. Tie them together and the day somebody
// allows two holders, every message is processed twice with nothing reporting it.
func TestAHoldersWorkerUsesAQueueGroupAnyway(t *testing.T) {
// The seat is authority and the worker is delivery (novox/hq ADR 0190): one worker per seat, shared
// by every holder and pulled from, so a second holder takes the next ask rather than a copy of the
// same one — which is what a queue group used to guard, and what pulling one durable gives outright.
func TestAHoldersWorkerIsOneSharedByItsHolders(t *testing.T) {
c, ok := HolderConsumerFor("one", "telegram", telegramSeat())
if !ok {
t.Fatal("the holder of a seat with inbound work got no worker")
}
if c.Queue == "" {
t.Fatal("the worker is not in a queue group, so a second holder would double-process")
two, _ := HolderConsumerFor("two", "telegram", telegramSeat())
if c.Name != two.Name || c.Stream != two.Stream {
t.Fatal("two holders got two workers, so each would process every ask")
}
if c.Push || c.Queue != "" {
t.Fatal("the worker is pushed, so the server would hand an ask to a busy holder")
}
if c.Stream != "SEAT_TELEGRAM_SENDER" {
t.Fatalf("the worker reads %q, not the seat's own stream", c.Stream)
@@ -154,15 +159,23 @@ func TestANodesDeclarationConsumerIsWhatItsOwnGrantAllows(t *testing.T) {
has(t, perms.Subscribe, c.Filters[0])
}
// A holder works one ask at a time, so the server hands it one at a time (novox/hq issue 186):
// asks queued behind the one being worked wait in the stream rather than being delivered,
// left to expire and dropped after the fifth redelivery.
func TestAHoldersWorkerTakesOneAskAtATime(t *testing.T) {
c, found := HolderConsumerFor("anchor", "builder", DeclaredSeat{Name: "mesh-build-machine", Accepts: []string{"build"}})
// Every holder of a seat shares one worker and pulls from it (novox/hq ADR 0190): no queue group
// and no delivery subject, because a push consumer hands the next ask to whichever subscriber the
// server picks, busy or not; and no cap of one in flight, because pulling bounds the asks in flight
// by the holders that are free — which is what ended the race of issue 186, where asks delivered
// behind the one being worked expired and were dropped.
func TestAHoldersWorkerIsPulledByEveryHolder(t *testing.T) {
c, found := HolderConsumerFor("anchor", "build-agent", DeclaredSeat{Name: "node-build-agent", Accepts: []string{"build"}})
if !found {
t.Fatal("a seat that accepts work has no worker")
}
if c.MaxAckPending != 1 {
t.Fatalf("the worker may have %d asks in flight; one, so a queue is a queue", c.MaxAckPending)
if c.Queue != "" || c.Push {
t.Fatalf("the worker is pushed (queue %q, push %v); a holder pulls when it is free", c.Queue, c.Push)
}
if c.MaxAckPending != 0 {
t.Fatalf("the worker caps asks in flight at %d; pulling bounds them by the holders working", c.MaxAckPending)
}
if c.Name != "SEAT_NODE_BUILD_AGENT_worker" || c.Stream != "SEAT_NODE_BUILD_AGENT" {
t.Fatalf("the worker is %s on %s; one per seat, shared by its holders", c.Name, c.Stream)
}
}
+45
View File
@@ -214,6 +214,51 @@ func (j *JetStream) EnsureConsumer(c Consumer) error {
switch have, err := j.js.ConsumerInfo(c.Stream, c.Name); {
case err == nil:
// **The controller owns the worker's shape, type included** (novox/hq issue 206). A holder
// built for a pull worker cannot bind a push one — `cannot pull subscribe to push based
// consumer` — and on 2026-10-03 the build machine rolled before the controller that would
// have redefined its worker, restarted on that for an hour, and nothing could build the
// controller that would have ended it. The server cannot change a consumer's type in place,
// so one of the wrong type is re-made: on a work queue nothing is lost, because what was
// acknowledged is gone from the stream and what was not is delivered again from the start.
// On any other stream a re-made consumer would replay what this one acknowledged (issue
// 156), so there it is said and left, and the person re-makes it knowing the cost.
if havePush, wantPush := have.Config.DeliverSubject != "", want.DeliverSubject != ""; havePush != wantPush {
shape := func(push bool) string {
if push {
return "push"
}
return "pull"
}
info, err := j.js.StreamInfo(c.Stream)
if err != nil {
return fmt.Errorf("asking about stream %s to re-make consumer %s: %w", c.Stream, c.Name, err)
}
if info.Config.Retention != nats.WorkQueuePolicy {
// **A stream that keeps its history is re-made from now on, never from the start.**
// Left for a hand, the hand re-makes it with the server's default — everything the
// stream holds — which on 2026-10-03 replayed every build ask since 1 October and
// re-registered nine modules from the past (novox/hq issue 207). What this consumer
// had not yet acknowledged is lost with it, and said: on a history stream that is
// the smaller cost, and the asks in flight are visible to whoever asked.
j.note("consumer %s on %s changes from %s to %s delivery on a stream that keeps its history: "+
"re-made to deliver from now on, so nothing this one acknowledged comes back (novox/hq issue "+
"207); %d ask(s) it had not acknowledged are not carried over and must be asked again",
c.Name, c.Stream, shape(havePush), shape(wantPush), have.NumPending+uint64(have.NumAckPending))
want.DeliverPolicy = nats.DeliverNewPolicy
} else {
j.note("consumer %s on %s changes from %s to %s delivery: re-made where it left off, nothing "+
"acknowledged comes back and nothing pending is lost (novox/hq issue 206); a holder bound to "+
"the old shape binds again", c.Name, c.Stream, shape(havePush), shape(wantPush))
}
if err := j.js.DeleteConsumer(c.Stream, c.Name); err != nil {
return fmt.Errorf("re-making consumer %s on %s as %s: %w", c.Name, c.Stream, shape(wantPush), err)
}
if _, err := j.js.AddConsumer(c.Stream, want); err != nil {
return fmt.Errorf("re-making consumer %s on %s as %s: %w", c.Name, c.Stream, shape(wantPush), err)
}
return nil
}
// Where an existing consumer starts is its history, not something an assertion may move:
// the server refuses a changed deliver policy outright. Carried across, so asserting twice
// is the no-op a restart depends on.
+34
View File
@@ -73,3 +73,37 @@ func TestAnAccountMayReadItsOwnMembershipAndNoOthers(t *testing.T) {
has(t, perms.Publish, "$JS.API.DIRECT.GET.ASSIGNMENTS.mesh.assignment.anchor.postgres")
hasNot(t, perms.Subscribe, "mesh.assignment.>")
}
// The runtime arriving on a machine changes nothing about what each module is issued (to-be 38 WP2):
// the memberships are composed as before and the runtime reads several of them. What the machine's
// user list gains is one runtime principal, and loses nothing but the runtime module's own.
func TestTheRuntimeArrivingLeavesEveryMembershipAsItWas(t *testing.T) {
filter := Seat{Name: "node-packet-filter", Scope: "node", Serves: []string{"rules", "reload"}}
three := []Declared{
{Module: "nftables", Holds: []Seat{filter}, Serves: []string{"firewall_rules"}},
{Module: "zsh", Serves: []string{"execute"}},
{Module: "systemd", Serves: []string{"units"}},
}
before := Records{Nodes: []string{"anchor"}, Assigned: map[string][]Declared{"anchor": three}}
after := Records{Nodes: []string{"anchor"}, Assigned: map[string][]Declared{
"anchor": append(append([]Declared{}, three...), Declared{Module: RuntimeModule}),
}}
for _, d := range three {
was := MembershipFor("anchor", d, PlacementsOf(before, nil))
is := MembershipFor("anchor", d, PlacementsOf(after, nil))
if !reflect.DeepEqual(was, is) {
t.Errorf("%s's membership changed when the runtime arrived:\n%+v\n%+v", d.Module, was, is)
}
}
users, err := Users(after)
if err != nil {
t.Fatal(err)
}
kinds := map[Kind]int{}
for _, p := range users {
kinds[p.Kind]++
}
if kinds[KindNodeTools] != 1 || kinds[KindModule] != 3 || kinds[KindNode] != 1 || kinds[KindController] != 1 {
t.Errorf("the machine's users are %v; one runtime, the three modules, the host and the controller", kinds)
}
}
+102 -10
View File
@@ -34,8 +34,20 @@ const (
// authority is a list of tools and nothing else — not control, not declarations, not builds,
// and no ability to answer anything, because a person asks.
KindPerson Kind = "person"
// KindNodeTools is a machine's tool runtime (novox/hq ADR 0175, to-be 38): one process per
// node, on the host side, serving every assigned module's tools and every held seat's verbs.
// Its authority is the union of what the modules it carries would each have had for their
// tools — and nothing of what they consume, because tools are what it runs, not reactions.
KindNodeTools Kind = "node-tools"
)
// RuntimeModule is the module that IS the node's tool runtime (novox/hq ADR 0175). Where it is
// assigned, the mesh composes one runtime principal for the machine in place of that module's own,
// and the per-module containers that served tools until then stop being the way tools reach a node.
// Mirrored in the catalogue package, which the agreement test holds to the same string; one
// constant, so a rename is one edit and the two packages cannot drift.
const RuntimeModule = "node-tools"
// Seat is a role on the bus as a principal relates to it: the subjects it accepts, and those it
// emits (novox/hq ADR 0118, design 29 §5).
type Seat struct {
@@ -74,6 +86,13 @@ type Principal struct {
// a namespace no such module owns. Every service started and the graph stayed empty.
Watches []Seat
// Carries are the modules whose tools this principal serves, for a KindNodeTools principal
// (novox/hq ADR 0175): every module assigned to its node, as each declares itself. Its
// serving authority is the union of theirs — each module's own tool namespace and each held
// seat's verbs on this node — derived from the same declarations the modules' own principals
// are, so the runtime can serve nothing a module could not have served for itself.
Carries []Declared
// Invokes are the tools this principal may call, as `<module>.<tool>`; a single `*` is every
// tool. A person's whole authority (design 25 §7), and a module's only if its manifest says so
// (novox/hq ADR 0152) — the console's does, and nothing else's.
@@ -91,10 +110,13 @@ type Principal struct {
PasswordHash string
}
// meshSeatsTheControllerUses are the roles the mesh's own flows submit work to. Named rather than
// seatsTheControllerAsks are the roles the mesh's own flows submit work to. Named rather than
// derived from the seat set: the controller is not a module and declares no `uses`, so its side of a
// seat has to be stated, and a list is what makes "which roles does the mesh itself talk to" answerable.
var meshSeatsTheControllerUses = []string{"mesh-build-machine"}
// Both build roles while the handover runs (novox/hq ADR 0190): the controller asks whichever has a
// holder, and the retired one has one until build-agent replaces the builder. The second entry
// goes with the retired seat row.
var seatsTheControllerAsks = []string{"node-build-agent", "mesh-build-machine"}
// enrolmentPrefix is the space every enrolling node's user and inbox live under, so the one place the
// controller may answer an enrolment is derived from the same constant the user is named from.
@@ -112,7 +134,10 @@ func (p Principal) Username() string {
switch p.Kind {
case KindPerson:
return "person." + p.Module
case KindModule:
case KindModule, KindNodeTools:
// The runtime is named exactly as the module it stands for would have been: the mesh
// issues its credential through the same path a module's takes (`module issue`), and
// that path knows the node and the module, not the kind.
return p.Node + "." + p.Module
case KindNode:
return "node." + p.Node
@@ -186,7 +211,9 @@ func PermissionsFor(p Principal) (Permissions, error) {
// Work the mesh's own flows submit to a role, and the outcomes they wait on (ADR 0121). A
// build is the one today: the controller asks, and reads the answer from the seat's event
// like the catalogue does — which is why no holder needs to publish into anybody's inbox.
for _, seat := range meshSeatsTheControllerUses {
// A node-scoped seat's work subject carries no node (novox/hq ADR 0190): the ask goes to
// the role, and whichever machine holding it is idle takes it.
for _, seat := range seatsTheControllerAsks {
pub = append(pub, "mesh.seat."+seat+".accept.>")
}
// **And what the mesh says it did** (novox/hq ADR 0134). The control plane states its own
@@ -346,13 +373,17 @@ func PermissionsFor(p Principal) (Permissions, error) {
// 3. Seats it holds: full participation.
for _, s := range p.Holds {
// Taking work from the role's queue: the worker consumer it binds (asked about,
// delivered on, acknowledged), each on the seat's own stream. The first machine to
// take work over the new bus was refused the asking (2026-09-28).
// Taking work from the role's queue: the worker consumer every holder shares (asked
// about, pulled from, acknowledged), on the seat's own stream (novox/hq ADR 0190). A
// holder pulls — asks the consumer for its next message, answered on its own inbox —
// so what it needs is MSG.NEXT on that worker and nothing delivered to it. The first
// machine to take work over the new bus was refused the asking (2026-09-28).
worker := "SEAT_" + upperSnake(s.Name) + "_worker"
stream := seatStreamName(s.Name)
sub = append(sub, "_DELIVER."+worker, "_DELIVER."+worker+".>")
pub = append(pub, "$JS.API.CONSUMER.INFO."+stream+"."+worker, "$JS.ACK."+stream+"."+worker+".>")
pub = append(pub,
"$JS.API.CONSUMER.INFO."+stream+"."+worker,
"$JS.API.CONSUMER.MSG.NEXT."+stream+"."+worker,
"$JS.ACK."+stream+"."+worker+".>")
for _, a := range s.Accepts {
sub = append(sub, seatSubject(s, "accept", a))
}
@@ -375,6 +406,48 @@ func PermissionsFor(p Principal) (Permissions, error) {
pub = append(pub, seatToolSubject(s, t, "*"))
}
}
case KindNodeTools:
// **One process serves what every module on the machine would have served for itself**
// (novox/hq ADR 0175). Each carried module's whole tool namespace — the same grant that
// module's own principal has, for the same reason: the tools a module serves are what its
// code answers, and a list here would be a second copy of it. Each held seat's verbs on
// this node, as the holder's own principal would be granted them.
for _, d := range p.Carries {
if !safeSubject.MatchString(d.Module) {
return Permissions{}, fmt.Errorf(
"%q cannot be part of a subject: a permission is a subject pattern, and this would widen it", d.Module)
}
own := "mesh.mod." + d.Module
sub = append(sub, own+".tool.>")
// A tool that emits an event is the module's code and emits under the module's name
// (ADR 0042); the runtime carrying that code may publish what the module declared it
// emits, and nothing it did not.
for _, e := range d.Emits {
pub = append(pub, own+".event."+e)
}
for _, s := range d.Holds {
for _, t := range s.Serves {
sub = append(sub, seatToolSubject(s, t, p.Node))
}
}
}
// Every assigned module's membership on this node (ADR 0160): one per module, read
// directly from the stream and followed live. This node's and no other's — the one token
// that varies is the module, so the pattern is the machine's own assignments.
sub = append(sub, "mesh.assignment."+p.Node+".*")
pub = append(pub, "$JS.API.DIRECT.GET."+AssignmentsStream+".mesh.assignment."+p.Node+".*")
// And every tool on the mesh (ADR 0175, decision 5): any node may call any tool on any
// node, as the console already could — the runtime is the console's serving mode.
invoked, err := invokedSubjects([]string{"*"})
if err != nil {
return Permissions{}, err
}
pub = append(pub, invoked...)
// Nothing about consumers: it consumes nothing. A module's reactions to events are its
// own long-lived process, which ADR 0175 leaves where it is; what moves here is tools.
sub = unique(sub)
pub = unique(pub)
}
if p.Kind == KindPerson {
@@ -382,6 +455,11 @@ func PermissionsFor(p Principal) (Permissions, error) {
// consumer, because nothing is delivered to a person — they ask and are answered.
sub = append(sub, p.inbox())
}
if p.Kind == KindNodeTools {
// Its reply space, so the answers to what its tools call come back to it. No ack subject
// for the same reason a person has none: nothing is delivered to it.
sub = append(sub, p.inbox())
}
if p.Kind == KindModule || p.Kind == KindNode || p.Kind == KindController {
// Its own reply space, and nothing wider.
@@ -403,7 +481,7 @@ func PermissionsFor(p Principal) (Permissions, error) {
// A module answers what it was asked — a tool call reaches it on its own namespace, so the
// authority is bounded by having been asked — and so does the controller. A node and a
// person are never asked anything, and are granted nothing here.
AllowResponses: p.Kind == KindModule || p.Kind == KindController,
AllowResponses: p.Kind == KindModule || p.Kind == KindController || p.Kind == KindNodeTools,
}, nil
}
@@ -625,6 +703,20 @@ func ComposeAccounts(principals []Principal) (string, error) {
return b.String(), nil
}
// unique is a sorted list with each subject once. Two carried modules holding seats with the same
// verb, or the runtime module itself carried beside the others, would otherwise write a grant twice
// — harmless to the server, and noise in a file that is read as the mesh's authority model.
func unique(values []string) []string {
sort.Strings(values)
out := values[:0]
for i, v := range values {
if i == 0 || v != values[i-1] {
out = append(out, v)
}
}
return out
}
func quoted(values []string) string {
if len(values) == 0 {
return ""
+91
View File
@@ -371,3 +371,94 @@ func TestAModulePullsItsOwnConsumerAndNoOthers(t *testing.T) {
}
}
}
// The runtime's authority is the union of what the modules it carries would have been granted for
// their tools (novox/hq ADR 0175): every carried module's tool namespace, every held seat's verbs
// on this node, every module's membership on this node, and a call to anything. Nothing it
// consumes, because it reacts to nothing.
func TestTheRuntimeServesTheUnionAndConsumesNothing(t *testing.T) {
filter := Seat{Name: "node-packet-filter", Scope: "node", Serves: []string{"rules", "reload"}}
p := Principal{Kind: KindNodeTools, Node: "anchor", Module: RuntimeModule, Carries: []Declared{
{Module: "nftables", Holds: []Seat{filter}, Serves: []string{"firewall_rules"}},
{Module: "zsh", Emits: []string{"shell.opened"}, Consumes: []string{"shop.order.placed"}},
{Module: RuntimeModule},
}}
perms, err := PermissionsFor(p)
if err != nil {
t.Fatal(err)
}
for _, want := range []string{
"mesh.mod.nftables.tool.>", "mesh.mod.zsh.tool.>", "mesh.mod." + RuntimeModule + ".tool.>",
"mesh.seat.node-packet-filter.tool.rules.anchor", "mesh.seat.node-packet-filter.tool.reload.anchor",
"mesh.assignment.anchor.*",
"_INBOX.anchor." + RuntimeModule + ".>",
} {
if !contains(perms.Subscribe, want) {
t.Errorf("the runtime may not subscribe %s: %v", want, perms.Subscribe)
}
}
for _, want := range []string{
"mesh.mod.*.tool.>", "mesh.seat.*.tool.>",
"$JS.API.DIRECT.GET.ASSIGNMENTS.mesh.assignment.anchor.*",
"mesh.mod.zsh.event.shell.opened",
} {
if !contains(perms.Publish, want) {
t.Errorf("the runtime may not publish %s: %v", want, perms.Publish)
}
}
// Nothing of what a carried module consumes, and no consumer of its own to ack.
for _, s := range perms.Subscribe {
if strings.Contains(s, ".event.") || strings.HasPrefix(s, "_DELIVER.") {
t.Errorf("the runtime was granted a delivery it has no consumer for: %s", s)
}
}
for _, s := range perms.Publish {
if strings.HasPrefix(s, "$JS.ACK.") || strings.Contains(s, "CONSUMER") {
t.Errorf("the runtime was granted a consumer's subject and has no consumer: %s", s)
}
}
if !perms.AllowResponses {
t.Error("the runtime answers what it is asked, and may not reply")
}
if _, needed := ConsumerFor(p); needed {
t.Error("a consumer would be made for the runtime, which consumes nothing")
}
// Each subject once: the file is read as the mesh's authority model.
seen := map[string]bool{}
for _, s := range append(append([]string{}, perms.Subscribe...), perms.Publish...) {
if seen[s] {
t.Errorf("%s is granted twice", s)
}
seen[s] = true
}
}
func contains(list []string, want string) bool {
for _, s := range list {
if s == want {
return true
}
}
return false
}
// A node-scoped seat's work is shared (novox/hq ADR 0190): its holder on any machine subscribes the
// seat's one work subject, with no node in it, so holders on several machines read one queue. The
// node token belongs to a seat's tools, which are asked of one machine (design 33 §4), not to its work.
func TestANodeSeatsWorkSubjectCarriesNoNode(t *testing.T) {
seat := Seat{Name: "node-build-agent", Scope: "node", Accepts: []string{"build"}, Serves: []string{"status"}}
perms, err := PermissionsFor(Principal{Kind: KindModule, Node: "anchor", Module: "build-agent", Holds: []Seat{seat}})
if err != nil {
t.Fatal(err)
}
has(t, perms.Subscribe, "mesh.seat.node-build-agent.accept.build")
hasNot(t, perms.Subscribe, "mesh.seat.node-build-agent.accept.build.anchor")
// And its tools still carry the machine.
has(t, perms.Subscribe, "mesh.seat.node-build-agent.tool.status.anchor")
// The controller asks the role, not a machine.
controller, err := PermissionsFor(Principal{Kind: KindController})
if err != nil {
t.Fatal(err)
}
has(t, controller.Publish, "mesh.seat.node-build-agent.accept.>")
}
+6 -1
View File
@@ -217,10 +217,15 @@ var ControllerFollows = []string{
// A build's outcome, which is the build-machine role's own event now (ADR 0121) rather than a
// message on the control branch. Same three audiences, one publish: whoever asked, this, and the
// catalogue.
seatEventSubject("mesh-build-machine", "built"),
seatEventSubject("node-build-agent", "built"),
// The forge's merges: what moved a source, so the mesh builds what that source produces
// without anybody telling it (novox/hq 04-ISSUES/131). Appended, because the index is a name.
moduleEventSubject("gitea", "pull.merged"),
// The retired build role's outcome too, while the handover runs (novox/hq ADR 0190): the one
// build machine keeps answering on its seat until build-agent replaces it, and the outcome that
// registers build-agent itself comes from there. Appended, for the same reason as above; goes
// with the retired seat row.
seatEventSubject("mesh-build-machine", "built"),
}
// moduleEventSubject is where one module's event lands. The same derivation PermissionsFor uses, so
+4 -4
View File
@@ -24,8 +24,8 @@ accounts {
jetstream: enabled
users = [
{ user: "controller", password: "$2a$11$cccccccccccccccccccccc", permissions: {
publish: { allow: ["$JS.ACK.CONTROL.controller.>", "$JS.ACK.EVENTS.controller.>", "$JS.API.>", "_INBOX.enrol.>", "mesh.assignment.>", "mesh.control.>", "mesh.mod.*.tool.>", "mesh.node.>", "mesh.seat.mesh-build-machine.accept.>", "mesh.seat.mesh-controller.event.applied", "mesh.seat.mesh-controller.event.built-before", "mesh.seat.mesh-controller.event.refused"] }
subscribe: { allow: ["$JS.API.>", "_DELIVER.controller", "_DELIVER.controller.>", "_INBOX.controller.>", "mesh.control.>", "mesh.mod.gitea.event.pull.merged", "mesh.mod.mesh-catalog.event.catching-up", "mesh.mod.mesh-catalog.event.upgraded", "mesh.seat.mesh-build-machine.event.built", "mesh.seat.mesh-controller.tool.>"] }
publish: { allow: ["$JS.ACK.CONTROL.controller.>", "$JS.ACK.EVENTS.controller.>", "$JS.API.>", "_INBOX.enrol.>", "mesh.assignment.>", "mesh.control.>", "mesh.mod.*.tool.>", "mesh.node.>", "mesh.seat.mesh-build-machine.accept.>", "mesh.seat.mesh-controller.event.applied", "mesh.seat.mesh-controller.event.built-before", "mesh.seat.mesh-controller.event.refused", "mesh.seat.node-build-agent.accept.>"] }
subscribe: { allow: ["$JS.API.>", "_DELIVER.controller", "_DELIVER.controller.>", "_INBOX.controller.>", "mesh.control.>", "mesh.mod.gitea.event.pull.merged", "mesh.mod.mesh-catalog.event.catching-up", "mesh.mod.mesh-catalog.event.upgraded", "mesh.seat.mesh-build-machine.event.built", "mesh.seat.mesh-controller.tool.>", "mesh.seat.node-build-agent.event.built"] }
allow_responses: { max: 1, ttl: "1m" }
} }
{ user: "enrol.one", password: "$2a$11$eeeeeeeeeeeeeeeeeeeeee", permissions: {
@@ -37,8 +37,8 @@ accounts {
subscribe: { allow: ["_DELIVER.one", "_DELIVER.one.>", "_INBOX.node.one.>", "mesh.node.one.declare"] }
} }
{ user: "one.telegram", password: "$2a$11$tttttttttttttttttttttt", permissions: {
publish: { allow: ["$JS.ACK.EVENTS.one_telegram.>", "$JS.ACK.SEAT_TELEGRAM_SENDER.SEAT_TELEGRAM_SENDER_worker.>", "$JS.API.CONSUMER.INFO.EVENTS.one_telegram", "$JS.API.CONSUMER.INFO.SEAT_TELEGRAM_SENDER.SEAT_TELEGRAM_SENDER_worker", "$JS.API.CONSUMER.MSG.NEXT.EVENTS.one_telegram", "$JS.API.DIRECT.GET.ASSIGNMENTS.mesh.assignment.one.telegram", "mesh.seat.telegram-sender.event.delivered", "mesh.seat.telegram-sender.event.failed"] }
subscribe: { allow: ["_DELIVER.SEAT_TELEGRAM_SENDER_worker", "_DELIVER.SEAT_TELEGRAM_SENDER_worker.>", "_INBOX.one.telegram.>", "mesh.assignment.one.telegram", "mesh.mod.telegram.tool.>", "mesh.seat.telegram-sender.accept.send"] }
publish: { allow: ["$JS.ACK.EVENTS.one_telegram.>", "$JS.ACK.SEAT_TELEGRAM_SENDER.SEAT_TELEGRAM_SENDER_worker.>", "$JS.API.CONSUMER.INFO.EVENTS.one_telegram", "$JS.API.CONSUMER.INFO.SEAT_TELEGRAM_SENDER.SEAT_TELEGRAM_SENDER_worker", "$JS.API.CONSUMER.MSG.NEXT.EVENTS.one_telegram", "$JS.API.CONSUMER.MSG.NEXT.SEAT_TELEGRAM_SENDER.SEAT_TELEGRAM_SENDER_worker", "$JS.API.DIRECT.GET.ASSIGNMENTS.mesh.assignment.one.telegram", "mesh.seat.telegram-sender.event.delivered", "mesh.seat.telegram-sender.event.failed"] }
subscribe: { allow: ["_INBOX.one.telegram.>", "mesh.assignment.one.telegram", "mesh.mod.telegram.tool.>", "mesh.seat.telegram-sender.accept.send"] }
allow_responses: { max: 1, ttl: "1m" }
} }
{ user: "two.audit", password: "$2a$11$aaaaaaaaaaaaaaaaaaaaaa", permissions: {
+20
View File
@@ -62,13 +62,33 @@ func Users(r Records) ([]Principal, error) {
for _, node := range sortedCopy(r.Nodes) {
out = append(out, Principal{Kind: KindNode, Node: node})
// **Where the runtime is assigned, the machine gets one runtime principal in place of the
// runtime module's own** (novox/hq ADR 0175, to-be 38). It carries every module on the
// node: its serving grants are the union of theirs. Every other module keeps its own
// principal — a module still serving tools from its own container holds its own
// credential until it moves, and the two serve side by side in the meantime.
runtimeHere := false
for _, d := range r.Assigned[node] {
if d.Module == RuntimeModule {
runtimeHere = true
}
}
for _, d := range r.Assigned[node] {
if runtimeHere && d.Module == RuntimeModule {
continue
}
out = append(out, Principal{
Kind: KindModule, Node: node, Module: d.Module,
Emits: d.Emits, Consumes: d.Consumes, Serves: d.Serves,
Holds: d.Holds, Uses: d.Uses, Watches: d.Watches, Invokes: d.Invokes,
})
}
if runtimeHere {
out = append(out, Principal{
Kind: KindNodeTools, Node: node, Module: RuntimeModule,
Carries: append([]Declared(nil), r.Assigned[node]...),
})
}
}
for _, node := range sortedCopy(r.Enrolling) {
out = append(out, Principal{Kind: KindEnrolment, Node: node})
+51
View File
@@ -245,3 +245,54 @@ func TestAUserListIsComposedBeforeAnythingMovesOntoTheBus(t *testing.T) {
t.Errorf("the composed list does not contain the machine running the bus")
}
}
// Where the runtime module is assigned, the machine gets one runtime principal in place of the
// runtime module's own (novox/hq ADR 0175, to-be 38). Every other module keeps its own: a module
// still serving tools from its own container holds its own credential until it moves.
func TestTheRuntimeModuleBecomesTheMachinesRuntimePrincipal(t *testing.T) {
r := someRecords()
r.Assigned["one"] = append(r.Assigned["one"], Declared{Module: RuntimeModule})
users, err := Users(r)
if err != nil {
t.Fatal(err)
}
var runtime *Principal
for i := range users {
p := &users[i]
if p.Node == "one" && p.Module == RuntimeModule {
if p.Kind == KindModule {
t.Fatalf("%s on one was composed as an ordinary module beside the runtime", RuntimeModule)
}
runtime = p
}
}
if runtime == nil || runtime.Kind != KindNodeTools {
t.Fatalf("one runs %s and got no runtime principal: %v", RuntimeModule, namesOf(t, r))
}
if runtime.Username() != "one."+RuntimeModule {
t.Errorf("the runtime is named %q; `module issue` names it as the module it stands for", runtime.Username())
}
carried := map[string]bool{}
for _, d := range runtime.Carries {
carried[d.Module] = true
}
if !carried["telegram"] || !carried[RuntimeModule] {
t.Errorf("the runtime carries %v; it carries every module on its node", carried)
}
// And the other node, where the runtime is not assigned, is exactly as before.
for _, p := range users {
if p.Node == "two" && p.Kind == KindNodeTools {
t.Fatal("two runs no runtime and was given a runtime principal")
}
}
// A module serving its own tools beside the runtime keeps its own principal.
found := false
for _, p := range users {
if p.Kind == KindModule && p.Node == "one" && p.Module == "telegram" {
found = true
}
}
if !found {
t.Error("telegram lost its own principal when the runtime arrived on its node")
}
}
+167
View File
@@ -0,0 +1,167 @@
package broker
import (
"os"
"testing"
"time"
"github.com/nats-io/nats.go"
)
// A seat's worker that changed from push to pull delivery strands a holder built for the new shape
// (novox/hq issue 206): the server refuses a pull subscription on a push consumer, and the controller
// that would redefine it was the build that nobody could take. The controller owns the worker's
// shape, type included: on a work queue it re-makes one of the wrong type, losing nothing, and a
// pull subscription then binds and takes what was pending.
//
// docker run -d --rm --name t -p 14231:4222 nats:2.10-alpine -js
// MESH_TEST_NATS=nats://127.0.0.1:14231 go test ./internal/broker/ -run TestAWorker
func TestAWorkerOfTheWrongTypeIsRemadeOnAWorkQueueAndAPullThenBinds(t *testing.T) {
url := os.Getenv("MESH_TEST_NATS")
if url == "" {
t.Skip("MESH_TEST_NATS unset")
}
js, err := Dial(url)
if err != nil {
t.Fatal(err)
}
defer js.Close()
const stream, worker, filter = "SEAT_T_SHELF", "SEAT_T_SHELF_worker", "mesh.seat.t-shelf.accept.>"
_ = js.js.DeleteStream(stream)
if _, err := js.js.AddStream(&nats.StreamConfig{
Name: stream, Subjects: []string{filter}, Retention: nats.WorkQueuePolicy, Storage: nats.MemoryStorage,
}); err != nil {
t.Fatal(err)
}
defer func() { _ = js.js.DeleteStream(stream) }()
// The worker as the previous controller defined it: push, in a queue group.
if _, err := js.js.AddConsumer(stream, &nats.ConsumerConfig{
Durable: worker, AckPolicy: nats.AckExplicitPolicy, AckWait: 60 * time.Second, MaxDeliver: 5,
FilterSubject: filter, DeliverSubject: "_DELIVER." + worker, DeliverGroup: "holders",
}); err != nil {
t.Fatal(err)
}
for _, body := range []string{"one", "two", "three"} {
if _, err := js.js.Publish("mesh.seat.t-shelf.accept.build", []byte(body)); err != nil {
t.Fatal(err)
}
}
// The old holder took and acknowledged the first ask, then went away.
old, err := js.js.QueueSubscribeSync(filter, "holders", nats.Bind(stream, worker))
if err != nil {
t.Fatal(err)
}
m, err := old.NextMsg(twoSeconds)
if err != nil {
t.Fatal(err)
}
if string(m.Data) != "one" {
t.Fatalf("the first ask is %q", m.Data)
}
if err := m.AckSync(); err != nil {
t.Fatal(err)
}
if err := old.Unsubscribe(); err != nil {
t.Fatal(err)
}
// The new controller asserts the worker as the mesh derives it now: pull.
if err := js.EnsureConsumer(Consumer{
Name: worker, Stream: stream, Filters: []string{filter}, AckWaitSeconds: 60, MaxDeliver: 5,
Why: "the test's worker",
}); err != nil {
t.Fatal(err)
}
have, err := js.js.ConsumerInfo(stream, worker)
if err != nil {
t.Fatal(err)
}
if have.Config.DeliverSubject != "" || have.Config.DeliverGroup != "" {
t.Fatalf("the worker is still push: %+v", have.Config)
}
// A holder built for the new shape binds, and takes exactly what the old one left.
sub, err := js.js.PullSubscribe(filter, worker, nats.Bind(stream, worker), nats.ManualAck())
if err != nil {
t.Fatalf("a pull subscription does not bind the re-made worker: %v", err)
}
got, err := sub.Fetch(3, nats.MaxWait(twoSeconds))
if err != nil && len(got) == 0 {
t.Fatalf("nothing pending was delivered: %v", err)
}
var bodies []string
for _, g := range got {
bodies = append(bodies, string(g.Data))
_ = g.Ack()
}
if len(bodies) != 2 || bodies[0] != "two" || bodies[1] != "three" {
t.Fatalf("the pending asks after the acknowledged one, in order: %v", bodies)
}
// Asserted again, the pull worker is the no-op a restart depends on.
if err := js.EnsureConsumer(Consumer{
Name: worker, Stream: stream, Filters: []string{filter}, AckWaitSeconds: 60, MaxDeliver: 5,
}); err != nil {
t.Fatal(err)
}
}
// On a stream that keeps its history, a worker of the wrong type is re-made to deliver from now on:
// re-making it from the start would replay what it acknowledged (novox/hq issue 156), and leaving it
// for a hand re-made it exactly that way on 2026-10-03 (issue 207).
func TestAWorkerOfTheWrongTypeOnAHistoryStreamIsRemadeFromNowOn(t *testing.T) {
url := os.Getenv("MESH_TEST_NATS")
if url == "" {
t.Skip("MESH_TEST_NATS unset")
}
js, err := Dial(url)
if err != nil {
t.Fatal(err)
}
defer js.Close()
const stream, worker, filter = "EVENTS_T", "EVENTS_T_reader", "mesh.t.event.>"
_ = js.js.DeleteStream(stream)
if _, err := js.js.AddStream(&nats.StreamConfig{Name: stream, Subjects: []string{filter}, Storage: nats.MemoryStorage}); err != nil {
t.Fatal(err)
}
defer func() { _ = js.js.DeleteStream(stream) }()
if _, err := js.js.AddConsumer(stream, &nats.ConsumerConfig{
Durable: worker, AckPolicy: nats.AckExplicitPolicy, AckWait: 60 * time.Second,
FilterSubject: filter, DeliverSubject: "_DELIVER." + worker,
}); err != nil {
t.Fatal(err)
}
// History the old consumer would have acknowledged long ago, and must not come back.
for i := 0; i < 3; i++ {
if _, err := js.js.Publish("mesh.t.event.old", []byte("old")); err != nil {
t.Fatal(err)
}
}
if err := js.EnsureConsumer(Consumer{Name: worker, Stream: stream, Filters: []string{filter}, AckWaitSeconds: 60}); err != nil {
t.Fatal(err)
}
have, err := js.js.ConsumerInfo(stream, worker)
if err != nil {
t.Fatal(err)
}
if have.Config.DeliverSubject != "" {
t.Fatal("a history stream's consumer of the wrong type was left as it was")
}
if have.Config.DeliverPolicy != nats.DeliverNewPolicy || have.NumPending != 0 {
t.Fatalf("re-made consumer delivers %v with %d pending; it must deliver from now on with nothing of the past", have.Config.DeliverPolicy, have.NumPending)
}
// And what arrives from now on is delivered.
if _, err := js.js.Publish("mesh.t.event.new", []byte("new")); err != nil {
t.Fatal(err)
}
sub, err := js.js.PullSubscribe(filter, worker, nats.Bind(stream, worker))
if err != nil {
t.Fatal(err)
}
got, err := sub.Fetch(1, nats.MaxWait(3*time.Second))
if err != nil || len(got) != 1 || string(got[0].Data) != "new" {
t.Fatalf("the re-made consumer delivered %v, %v; want the one new message", got, err)
}
}
+20
View File
@@ -968,6 +968,26 @@ func compile(ctx context.Context, run Runner, tree string, chain Toolchain,
if _, err := run(ctx, tree, "docker", invocation...); err != nil {
return "", err
}
if chain.Dependencies != "" {
// **What the bundle runs with, from the image it was compiled in** (Toolchain.Dependencies).
// A second run in the same image rather than a shell wrapped around the compiler: the
// compile line stays a plain command a reader can run by hand, and the copy is one more
// plain command beside it. Refused by name when the image carries no such directory — an
// older toolchain image — because a bundle packed without its dependencies starts nowhere
// and says so three layers away from here.
copying := []string{
"run", "--rm",
"--volume", tree + ":" + within,
"--workdir", within,
base,
"sh", "-c",
`test -d "$1" || { echo "the toolchain image carries no $1: it predates the mesh shipping a bundle's dependencies, rebuild $2 first" >&2; exit 1; }; cp -a "$1/." "$3/"`,
"dependencies", chain.Dependencies, chain.Base, out,
}
if _, err := run(ctx, tree, "docker", copying...); err != nil {
return "", fmt.Errorf("copying the %s dependencies a bundle runs with: %w", chain.Language, err)
}
}
return filepath.Join(tree, out), nil
}
+40 -1
View File
@@ -82,6 +82,41 @@ func TestABundleIsCompiledAndPackedWithNoDockerfile(t *testing.T) {
if !strings.HasPrefix(digest, "sha256:") {
t.Fatalf("the bundle was not pinned: %v", got.Manifest.Resources[0])
}
// **And what it runs with, from the image it was compiled in** (novox/hq to-be 38 WP3). A
// second run in the same toolchain image copies the toolchain's runtime directory — the
// `"type": "module"` package.json and the pruned node_modules — into the output's root, and
// refuses by name when the image carries none rather than packing a bundle that starts nowhere.
var copied string
for _, line := range r.ran {
if strings.HasPrefix(line, "docker run") && strings.Contains(line, "/app/runtime") {
copied = line
}
}
if copied == "" {
t.Fatalf("the bundle's dependencies were not copied in after the compile:\n%s", strings.Join(r.ran, "\n"))
}
if !strings.Contains(copied, "mesh-tools/build@sha256:") || !strings.Contains(copied, "predates") ||
!strings.Contains(copied, Out("code")) {
t.Fatalf("the copy does not run in the same toolchain, refuse an older image by name, or land in the artifact's output: %s", copied)
}
if strings.Index(strings.Join(r.ran, "\n"), "--outDir") > strings.Index(strings.Join(r.ran, "\n"), "/app/runtime") {
t.Fatal("the dependencies were copied before the compile wrote its output")
}
}
// A language whose bundle carries its own dependencies copies nothing in: a Go binary is static.
func TestOnlyALanguageWithARuntimeDirectoryCopiesDependenciesIn(t *testing.T) {
ts, _ := ToolchainFor("typescript")
if ts.Dependencies != "/app/runtime" {
t.Fatalf("typescript bundles run with %q", ts.Dependencies)
}
for _, language := range []string{"go", "python"} {
chain, _ := ToolchainFor(language)
if chain.Dependencies != "" {
t.Fatalf("%s copies %q into every bundle, and its bundles carry their own", language, chain.Dependencies)
}
}
}
// **Refused before anything is built, naming what to build first.** A base the mesh has not built
@@ -145,9 +180,13 @@ func TestTwoBundlesInOneModuleArePackedSeparately(t *testing.T) {
t.Fatalf("a module with two bundles did not build: %v", err)
}
// Compiled into two different places.
// Compiled into two different places. Only the compile lines: the copy of each bundle's
// dependencies names the same directory again, deliberately.
var outputs []string
for _, line := range r.ran {
if !strings.Contains(line, "--outDir") {
continue
}
for _, part := range strings.Fields(line) {
if strings.HasPrefix(part, ".mesh-build/") {
outputs = append(outputs, part)
+28 -4
View File
@@ -57,6 +57,23 @@ type Toolchain struct {
// carrying its debug info. The mistake was believing a comment rather than reading the file it
// produced (novox/hq 04-ISSUES/161).
LinkerFlags []string
// Dependencies is a directory inside the toolchain image whose contents a bundle in this
// language runs with, copied whole into the compiled output's root after the compile.
//
// **A bundle that compiles is not yet a bundle that runs.** The compiler resolves `import
// "nats"` from the toolchain image's own node_modules and the pack takes only what the compiler
// wrote, so what a machine unpacked could not find a single dependency — and no TypeScript bundle
// had ever run live to show it (novox/hq to-be 38 WP3). For TypeScript the directory holds a
// `package.json` saying `"type": "module"` — Node reads a bare `.js` as CommonJS otherwise, so a
// bundle with its dependencies and without that line still fails to start — and the pruned,
// production-only node_modules the runtime itself ships with: the SDK's and the runtime's
// dependencies, and nothing module-specific yet (novox/hq ADR 0188 §5: a skeleton; a module's
// own npm dependencies are a later step). Empty for a language whose bundle carries its own —
// a Go binary is static, a Python bundle is installed with its dependencies.
//
// A toolchain image without the directory fails the build by name rather than packing a bundle
// that starts nowhere: the image predates this and must be rebuilt first.
Dependencies string
// SystemStamp is the variable this language's linker fills with the artifact's declared system,
// for a language whose binaries are pinned to one at link time (novox/hq ADR 0005).
//
@@ -107,14 +124,21 @@ var toolchains = []Toolchain{
// symlinks to a launcher that requires its library relatively — and the base image's own
// assembly resolves them away, leaving a launcher whose relative require points nowhere.
// Every module's hand-written Dockerfile had to know this. Now none of them does.
// **Rooted at the module, so an entrypoint lands where it is named.** Without a root the
// compiler takes the common directory of the files it is given: a module compiling only
// `tools/index.ts` had its output at `index.js`, and the entrypoint it declared —
// `tools/index.js`, "named as it will be found" — named a file the bundle did not
// contain. The runtime that loads bundles by their declared entrypoints (novox/hq ADR
// 0175) is what made this visible.
Compile: []string{
"node", "/app/node_modules/typescript/bin/tsc",
"--module", "NodeNext", "--moduleResolution", "NodeNext",
"--target", "ES2022",
"--target", "ES2022", "--rootDir", ".",
},
OutputFlag: "--outDir",
Unit: UnitSources,
SourceExt: ".ts",
OutputFlag: "--outDir",
Unit: UnitSources,
SourceExt: ".ts",
Dependencies: "/app/runtime",
},
{
Language: "go",
+100 -1
View File
@@ -67,6 +67,36 @@ func (m Manifest) Resolve(built []Built) (Manifest, error) {
out := m
out.Build = nil
out.Resources = nil
// What the build compiled, kept on the resolved manifest (novox/hq ADR 0175): a tools bundle is
// named by no resource of the module's own — the node's runtime loads it — so this is the only
// place the mesh would otherwise not have it. In artifact order, so two resolutions of one
// build compare equal.
out.Bundles = nil
if m.Build != nil {
for _, a := range m.Build.Artifacts {
if a.Kind != ArtifactBundle {
continue
}
made := by[a.Name]
// What the runtime loads: what the artifact said, else every entrypoint of a module
// that declares tools, else nothing (the field's own rule; see Artifact.Loads).
loads := append([]string(nil), a.Loads...)
if a.Loads == nil && len(m.Tools) > 0 {
loads = append([]string(nil), a.Entrypoints...)
}
// **Kept, never routed** (ADR 0155): the builder publishes to the store at the address
// it reached it by, and a manifest carrying that address names an installation —
// registration refused node-tools for exactly this on 2026-10-02. The build record
// already keeps the store-relative form; the resolved manifest keeps the same, and
// composition routes it through the store a machine reaches (Routed).
out.Bundles = append(out.Bundles, Bundle{
Name: a.Name, Source: Recorded(made.Reference), Digest: made.Digest,
Language: a.Language, Entrypoints: append([]string(nil), a.Entrypoints...),
Loads: loads, Env: copyWords(a.Env),
})
}
sort.Slice(out.Bundles, func(i, j int) bool { return out.Bundles[i].Name < out.Bundles[j].Name })
}
for _, r := range m.Resources {
named, _ := r["artifact"].(string)
if named == "" {
@@ -110,7 +140,8 @@ func (m Manifest) Resolve(built []Built) (Manifest, error) {
// The same on the wire: both are bytes fetched by digest and unpacked. They differ in
// how they were made — one packed as it stood, the other compiled first — and a
// machine has no reason to care which.
filled["source"] = artifact.Reference
// Kept, not routed, for the reason the bundles above are (ADR 0155).
filled["source"] = Recorded(artifact.Reference)
filled["digest"] = artifact.Digest
// **And `${version}`, so a resource can name a place that is this build's alone**
// (novox/hq ADR 0141, 04-ISSUES/142). A component is unpacked into a directory named
@@ -172,6 +203,12 @@ func (b *Build) problems(module string) []string {
// A bundle's source is the module's own directory by definition, and what it needs to say
// is which compiler — because the mesh chooses that, and cannot choose for a module that
// has not said.
if len(a.Env) > 0 && a.Kind != ArtifactBundle {
problems = append(problems, fmt.Sprintf(
"%s: %q is a %q and says what it is given (env). Only a bundle the node's runtime "+
"serves is given words (novox/hq ADR 0192); a container says its own environment",
module, a.Name, a.Kind))
}
if a.Kind == ArtifactBundle || a.Kind == ArtifactPackage {
// **Except for a language that compiles to a binary, where it names which one**
// (novox/hq 04-ISSUES/142). A bundle in an interpreted language is the module's own
@@ -191,6 +228,21 @@ func (b *Build) problems(module string) []string {
"%s: %q is a bundle and says no language, so nothing can choose a compiler "+
"for it", module, a.Name))
}
problems = append(problems, bundleEnvProblems(module, a)...)
// What the runtime loads is among what was compiled (ADR 0175): a name here that is
// not an entrypoint is a file the bundle does not contain, and the runtime would
// fail to import it on every machine rather than here.
for _, load := range a.Loads {
found := false
for _, e := range a.Entrypoints {
found = found || e == load
}
if !found {
problems = append(problems, fmt.Sprintf(
"%s: %q says the runtime loads %q, which is not among its entrypoints — "+
"what is loaded is compiled, so it is named there too", module, a.Name, load))
}
}
// **A system, for a language that compiles to a binary** (novox/hq ADR 0142). A binary
// is pinned to one operating system at link time so a host refuses to touch a machine
// it was not built for (novox/hq ADR 0005); an artifact that says nothing would be
@@ -315,3 +367,50 @@ func versionOf(digest string) string {
}
return hex
}
// bundleEnvWords are the words the runtime sets for itself; a bundle that named one would be
// telling the runtime what it is, which is the mesh's to say (novox/hq ADR 0192).
var bundleEnvWords = map[string]bool{
RuntimeToolModules: true, RuntimeBrokerFile: true, RuntimeOperatorAccount: true,
RuntimeOperatorHome: true, RuntimeToolEnv: true,
}
// bundleEnvProblems says what is wrong with what a bundle says it is given (novox/hq ADR 0192):
// a value is a path or a constant written with the references a container's environment may use
// for a place or a port, and never a secret's content or another module's binding — a secret
// reaches a tool as a file whose path is named.
func bundleEnvProblems(module string, a Artifact) []string {
if len(a.Env) == 0 {
return nil
}
var problems []string
for _, word := range sortedKeys(a.Env) {
value := a.Env[word]
if bundleEnvWords[word] {
problems = append(problems, fmt.Sprintf(
"%s: %q gives itself %s, which the node's runtime sets for itself; a bundle is "+
"given its own words beside the runtime's, never in place of them (novox/hq ADR 0192)",
module, a.Name, word))
}
rest := ofPort.ReplaceAllString(dirRef.ReplaceAllString(value, ""), "")
if strings.Contains(rest, "${") {
problems = append(problems, fmt.Sprintf(
"%s: %q gives %s the value %q. A bundle's word is a path or a constant, written with "+
"${dir:…} and ${port:…} only; a secret reaches a tool as a file the mesh places, "+
"named by its path, never as its content (novox/hq ADR 0192)",
module, a.Name, word, value))
}
}
return problems
}
func copyWords(in map[string]string) map[string]string {
if len(in) == 0 {
return nil
}
out := make(map[string]string, len(in))
for k, v := range in {
out[k] = v
}
return out
}
+207 -4
View File
@@ -249,13 +249,32 @@ type Composed struct {
Resources []map[string]any
Owner map[string]string
Received map[string]map[string][]Contribution
// LeftOut is every module of this machine's set that was left out of its declaration, and
// why (novox/hq ADR 0163, rule 6): a setting stored for it that its definition can no longer
// compose. Its held things are kept and its containers untouched — the machine is told so —
// and it is told everything else.
LeftOut map[string]string
}
// LeftOut is which of this machine's modules a declaration composed with these settings leaves
// out, and why (novox/hq ADR 0163, rule 6): each whose stored settings its definition can no longer
// compose. Empty when every module composes. The same judgement SetSettings makes before storing.
func (r Resolution) LeftOut(settings SettingsBy, adopted bool) map[string]string {
out := map[string]string{}
for _, m := range r.Modules {
if err := JudgeSettings(m, settings[m.Module], adopted); err != nil {
out[m.Module] = err.Error()
}
}
return out
}
// Compose is Declaration with the owner of every resource said.
func (r Resolution) Compose(with Rendering) (Composed, error) {
owner := map[string]string{}
received := map[string]map[string][]Contribution{}
resources, err := r.compose(with, owner, received)
leftOut := map[string]string{}
resources, err := r.compose(with, owner, received, leftOut)
if err != nil {
return Composed{}, err
}
@@ -267,7 +286,7 @@ func (r Resolution) Compose(with Rendering) (Composed, error) {
"sealed": with.BusMembership, "mode": "0600",
})
}
return Composed{Resources: resources, Owner: owner, Received: received}, nil
return Composed{Resources: resources, Owner: owner, Received: received, LeftOut: leftOut}, nil
}
// BusMembershipID names the resource carrying a machine's membership for the new bus, and
@@ -277,7 +296,25 @@ func BusMembershipID() string { return "bus-membership" }
const BusMembershipPath = "/var/lib/mesh/membership-next.json"
func (r Resolution) compose(with Rendering, owner map[string]string,
received map[string]map[string][]Contribution) ([]map[string]any, error) {
received map[string]map[string][]Contribution, leftOut map[string]string) ([]map[string]any, error) {
// **A setting is judged where it is stored, and an impossible one costs a module, not a
// machine** (novox/hq ADR 0163, rule 6). A definition that moved under a stored setting makes
// this module uncomposable; it is left out of the declaration — its held things kept, its
// containers untouched, the machine told so by name — and the machine is told everything else.
// Before placing, because a placement is a setting too.
left := r.LeftOut(with.Settings, with.Adopted)
kept := make([]Manifest, 0, len(r.Modules))
for _, m := range r.Modules {
if why, isLeft := left[m.Module]; isLeft {
if leftOut != nil {
leftOut[m.Module] = why
}
continue
}
kept = append(kept, m)
}
r.Modules = kept
// Every manifest is placed first (novox/hq ADR 0112): the maps naming where its bindings,
// credentials and contributions land are resolved against this node's directories, so every
// reader below — the binding files, the sealed secrets, the grant paths a contribution
@@ -471,10 +508,27 @@ func (r Resolution) compose(with Rendering, owner map[string]string,
return nil, fmt.Errorf(
"%s needs a secret called %q and none was made for it", m.Module, name)
}
first = append(first, ownedBy(m.SecretsOwner, map[string]any{
// The runtime's credential belongs to the account the runtime runs as (novox/hq ADR 0175,
// to-be 38 WP3): its process is composed `user: <account>` where the node has one, and a
// root-owned 0600 file is one that process cannot read. Composed here rather than said in
// the manifest, because a manifest cannot say ${machine:account} safely — a node with no
// account has nothing to resolve it to, and then the runtime runs as root and the file
// stays root's.
owner := m.SecretsOwner
if m.Module == RuntimeModule && r.Account != "" {
owner = r.Account
}
first = append(first, ownedBy(owner, map[string]any{
"id": NeedID(name), "type": "file", "path": m.OwnSecrets[name].Path, "sealed": sealed,
}))
}
// This module's tools bundles, where the machine runs the node's tool runtime (novox/hq
// ADR 0175, to-be 38 WP2). Mesh-computed like everything above it, and before the module's
// own resources for the same reason: the runtime's process names the files inside these
// and is restarted when one changes, so they are on the machine before it is.
if r.runtimeHere() {
first = append(first, bundleArchives(m)...)
}
// Operator-owned paths this module is granted use of (novox/hq ADR 0051). Written before
// the module's own resources, and so before the container that mounts them: the host must
// find each present — refusing clearly if the operator has not provided it — before it
@@ -706,6 +760,10 @@ func (r Resolution) compose(with Rendering, owner map[string]string,
// Which of this module's resources its preparation runs before, if it prepares anything.
prepareBefore := preparationTarget(m)
// Which found networks this machine's setting keeps for each of its containers (novox/hq
// ADR 0163, rule 4); judged above, so an invalid one is not here.
keptNetworks, _ := KeptNetworks(m, with.Settings[m.Module], with.Adopted)
for _, unsettled := range resources {
resource, err := ApplySettings(unsettled, with.Settings[m.Module])
if err != nil {
@@ -715,6 +773,16 @@ func (r Resolution) compose(with Rendering, owner map[string]string,
for k, v := range resource {
copied[k] = v
}
if networks, keeps := keptNetworks[fmt.Sprint(copied["id"])]; keeps {
// The container also joins the found network the setting names, so a neighbour
// that resolves it there keeps resolving it. Passed to the host as its own field,
// which it joins after the container is made.
joins := make([]any, 0, len(networks))
for _, n := range networks {
joins = append(joins, n)
}
copied["networks"] = joins
}
if err := refuseSecretsInEnvironment(copied, secretFiles, m.Module); err != nil {
return nil, err
}
@@ -804,6 +872,15 @@ func (r Resolution) compose(with Rendering, owner map[string]string,
if renamed := reflectsRenamed(m.Module, resource["reload-on"]); renamed != nil {
copied["reload-on"] = renamed
}
// **What reads one of this module's own secrets is restarted when it changes** (novox/hq
// issue 203, issue 206). A credential is re-issued by the mesh, and a container that
// mounted the old file keeps the old one open: the build machine ran for an hour on a
// credential the mesh had replaced, because its manifest restarted it on its
// environment file and nobody had thought to name the credential too. Composed here so
// no manifest has to say it, for a container or a daemon that names the secret's path.
if reads := secretsReadBy(copied, m); len(reads) > 0 {
copied["restart-on"] = withRestartOn(copied["restart-on"], reads)
}
// **A version prepares its state before it runs** (novox/hq ADR 0135). Derived from the
// module's own resource rather than declared beside it: what prepares the state is the
// module's own code, so what it is given has to be what that code is given — and a
@@ -834,6 +911,41 @@ func (r Resolution) compose(with Rendering, owner map[string]string,
out = append(out, fact)
}
}
// The node's tool runtime, last (novox/hq ADR 0175, to-be 38 WP2.3): one process loading every
// bundle delivered above and holding the credential sealed above, so both exist before it starts
// — the order written here is the order the machine applies.
if r.runtimeHere() {
process, err := r.runtimeProcess(with)
if err != nil {
return nil, err
}
owner[fmt.Sprint(process["id"])] = RuntimeModule
out = append(out, process)
// What each module's bundles are given is read as the account the runtime runs as.
words := map[string]map[string]string{}
for _, m := range r.Modules {
w, err := bundleWords(m, with)
if err != nil {
return nil, err
}
words[m.Module] = w
}
// And a file a module's words name is one the runtime is restarted for when it changes.
if named := givenTo(out, owner, words, r.Account); len(named) > 0 {
restarts, _ := process["restart-on"].([]any)
seen := map[string]bool{}
for _, id := range restarts {
seen[fmt.Sprint(id)] = true
}
for _, id := range named {
if !seen[id] {
restarts = append(restarts, id)
seen[id] = true
}
}
process["restart-on"] = restarts
}
}
if with.Adopted {
// First, before anything a module declares: what the mesh needs reachable, then its guard.
// The order a machine applies is the order written here.
@@ -960,8 +1072,15 @@ func (r Resolution) filtersHere() string {
// computed for this machine, and each module's per-node exposure. The same answer whether the node
// is adopted or converged — the one loads it as a filter, the other declares it as openings.
func (r Resolution) Rules(with Rendering) ([]Rule, error) {
// A module whose settings cannot compose is left out of the declaration (novox/hq ADR 0163,
// rule 6), and out of the filter with it: nothing of it is declared, so nothing of it is let
// through.
left := r.LeftOut(with.Settings, with.Adopted)
exposure := map[string]map[int]string{}
for _, m := range r.Modules {
if _, isLeft := left[m.Module]; isLeft {
continue
}
e, err := Exposure(m, with.Settings[m.Module])
if err != nil {
return nil, err
@@ -1993,3 +2112,87 @@ func portOfEndpoint(values map[string]any, ports map[string]int) {
values["port"] = port
}
}
// secretsReadBy is the file resources of this module's own secrets that a container or a daemon reads
// — named in its volumes, its environment or its env-files by the secret's placed path — as
// restart-on ids. Nothing for other shapes, and nothing for a scheduled or run-once process, which
// the host refuses a restart-on for (it runs again anyway, and reads the file afresh).
func secretsReadBy(resource map[string]any, m Manifest) []string {
kind := fmt.Sprint(resource["type"])
if kind != "container" && kind != "process" {
return nil
}
if resource["schedule"] != nil || resource["run-once"] == true {
return nil
}
var mentioned []string
for _, key := range []string{"volumes", "env", "env-file"} {
mentioned = append(mentioned, stringsIn(resource[key])...)
}
var out []string
for _, name := range sortedKeys(m.OwnSecrets) {
path := m.OwnSecrets[name].Path
if path == "" {
continue
}
for _, s := range mentioned {
// A volume is `source:destination[:mode]`; an env value or an env-file is the path itself.
if s == path || strings.HasPrefix(s, path+":") {
out = append(out, m.Module+"."+NeedID(name))
break
}
}
}
return out
}
// stringsIn is every string in a list or a map's values; nothing for anything else.
func stringsIn(v any) []string {
switch x := v.(type) {
case []any:
var out []string
for _, item := range x {
if s, ok := item.(string); ok {
out = append(out, s)
}
}
return out
case []string:
return x
case map[string]any:
var out []string
for _, k := range sortedKeys(x) {
if s, ok := x[k].(string); ok {
out = append(out, s)
}
}
return out
case map[string]string:
var out []string
for _, k := range sortedKeys(x) {
out = append(out, x[k])
}
return out
}
return nil
}
// withRestartOn is a resource's restart-on list with these ids added once each.
func withRestartOn(have any, add []string) []any {
var out []any
seen := map[string]bool{}
for _, id := range reflectsRenamed("", have) {
s := fmt.Sprint(id)
if !seen[s] {
seen[s] = true
out = append(out, s)
}
}
for _, id := range add {
if !seen[id] {
seen[id] = true
out = append(out, id)
}
}
return out
}
+12
View File
@@ -444,6 +444,18 @@ func AsNftables(rules []Rule, mesh []string, outward bool, foundation []int,
b.WriteString("\t\t# this machine's own guests reaching outward: not a port opened to anybody\n")
b.WriteString(fmt.Sprintf("\t\tiifname != { %s } accept\n", inward))
}
// **The mesh passing through, not arriving.** A machine the mesh routes through — the hub, for
// every path between machines that are not co-located (novox/hq ADR 0007) — relays a packet that
// came in on the tunnel and leaves on it again, addressed to another machine of the mesh. That is
// no port of this machine's: the machine it is for filters it against its own rules. Without
// this, the chain below judged a relayed packet by this machine's own published ports, so two
// machines behind the hub reached each other only on ports the hub happened to publish for itself
// (novox/hq issue 196). In and out on the tunnel both: a packet off the tunnel for this machine's
// own containers leaves by a bridge, and still meets the rules below.
if tunnel != "" {
b.WriteString("\t\t# the mesh passing through to another of its machines, which filters it itself\n")
b.WriteString(fmt.Sprintf("\t\tiifname %q oifname %q accept\n", tunnel, tunnel))
}
if len(rules) > 0 {
b.WriteString("\n")
+31
View File
@@ -887,3 +887,34 @@ func TestAPublicPortNeedsNoGuestLine(t *testing.T) {
t.Fatalf("a public port was given a guest line it does not need:\n%s", nft)
}
}
// **The hub relays the mesh** (novox/hq ADR 0007, issue 196). Two machines that are not co-located
// reach each other through the hub, so the hub forwards a packet that arrives on the tunnel and
// leaves on it. The forward chain judged that packet by the hub's own published ports, and two
// machines behind the hub reached each other only on the ports the hub happened to publish.
//
// Measured: from one home machine to another through the hub, 17 of 55 ports answered, and they
// were exactly the hub's own; the SYN for the rest never left the hub.
func TestTheMeshPassingThroughIsRelayedNotJudgedAsThisMachines(t *testing.T) {
nft := AsNftables(nil, []string{"10.42.0.1", "10.42.0.2"}, false, nil, []string{"eth0"}, "mesh0")
relay := `iifname "mesh0" oifname "mesh0" accept`
if !strings.Contains(chainBody(t, nft, "forward"), relay) {
t.Errorf("the forward chain does not relay the mesh through this machine:\n%s", chainBody(t, nft, "forward"))
}
// Relaying is not receiving: nothing in the input chain opens because of it.
if strings.Contains(chainBody(t, nft, "input"), "oifname") {
t.Errorf("the input chain names an outgoing interface, which no packet for this machine has:\n%s",
chainBody(t, nft, "input"))
}
// And off the tunnel into this machine's own containers is still judged: the tunnel is not
// accepted wholesale, only in and out on it.
if strings.Contains(chainBody(t, nft, "forward"), `iifname "mesh0" accept`) {
t.Errorf("the forward chain accepts everything off the tunnel:\n%s", chainBody(t, nft, "forward"))
}
// A machine with no tunnel relays nothing, and names no interface it does not have.
alone := AsNftables(nil, nil, false, nil, []string{"eth0"}, "")
if strings.Contains(alone, "oifname") {
t.Errorf("a machine with no tunnel was given a relay rule:\n%s", alone)
}
}
+12 -2
View File
@@ -1,6 +1,8 @@
package catalogue
import (
"crypto/sha256"
"encoding/hex"
"fmt"
"sort"
"strings"
@@ -43,8 +45,16 @@ func jailsInto(modules []Manifest, j *Jailing) []map[string]any {
out := make([]map[string]any, 0, len(jails)+1)
for _, d := range jails {
fmt.Fprintf(&composed, "\n# from %s\n[%s]\nenabled = true\nfilter = %s\n%s\n",
d.module, d.jail.Name, d.jail.Name, strings.TrimRight(d.jail.Jail, "\n"))
// **The filter's digest rides in the jail file.** fail2ban is restarted when this file
// changes, and the filter is a file of its own: a module that changed only what a failure
// looks like rewrote the filter on disk and left the running jail on the old pattern, with
// nothing said (novox/hq issue 191's rollout found it on gitea's sshd). Naming the filter's
// digest here makes a changed pattern a changed jail file, so the restart the service
// already takes on it covers the filter too.
sum := sha256.Sum256([]byte(d.jail.Failregex))
fmt.Fprintf(&composed, "\n# from %s, filter %s\n[%s]\nenabled = true\nfilter = %s\n%s\n",
d.module, hex.EncodeToString(sum[:])[:12], d.jail.Name, d.jail.Name,
strings.TrimRight(d.jail.Jail, "\n"))
// The filter is a file of its own, named as the jail's filter= references it.
out = append(out, map[string]any{
"id": "filter-" + d.jail.Name,
+26
View File
@@ -44,3 +44,29 @@ func TestTheComposedJailFileIsWrittenEvenWhenEmpty(t *testing.T) {
t.Fatalf("the empty composed jail file was not written alone: %v", files)
}
}
// A changed pattern restarts fail2ban (novox/hq issue 191's rollout): the service restarts when the
// composed jail file changes, and the filter is a file of its own, so the jail file names the
// filter's digest. Changing only the failregex must change the jail file; the same pattern must not.
func TestAChangedFilterChangesTheJailFile(t *testing.T) {
jailFile := func(failregex string) string {
modules := []Manifest{
{Module: "fail2ban", Jailing: &Jailing{Into: "/etc/fail2ban/jail.d/mesh.conf", FilterInto: "/etc/fail2ban/filter.d"}},
{Module: "gitea", Jails: []Jail{{Name: "gitea", Failregex: failregex, Jail: "port = 222"}}},
}
for _, f := range jailsInto(modules, modules[0].Jailing) {
if f["id"] == ComposedJailsID() {
return f["content"].(string)
}
}
t.Fatal("no composed jail file")
return ""
}
before := jailFile("web login failed from <HOST>")
if again := jailFile("web login failed from <HOST>"); again != before {
t.Errorf("the same pattern composed a different jail file, which would restart fail2ban for nothing")
}
if after := jailFile("web login failed from <HOST>\n Invalid user .* from <HOST>"); after == before {
t.Errorf("a changed pattern left the jail file as it was, so fail2ban keeps the old filter:\n%s", after)
}
}
+5 -2
View File
@@ -102,8 +102,11 @@ func accountHomeOf(account, home string) string {
func machineInto(resource map[string]any, facts map[string]string, module string) error {
// Content, and now the path and owner too: a module that writes into a person's home names it
// with ${machine:account-home} and ${machine:account}, which it cannot know until assigned
// (novox/hq to-be 29), the same reason its content names ${machine:address}.
for _, field := range []string{"path", "owner", "content"} {
// (novox/hq to-be 29), the same reason its content names ${machine:address}. And the name a
// `user` shape sets the login shell of, and the user a user-scoped unit or a process runs as:
// the shell module makes the operator's account its holder's login shell, and the desktop's
// watchers run as that account (novox/hq ADR 0176, ADR 0177) — neither can name the person.
for _, field := range []string{"path", "owner", "content", "name", "user"} {
s, ok := resource[field].(string)
if !ok {
continue
+103
View File
@@ -572,6 +572,38 @@ type Manifest struct {
// a module that could ask for it could read every credential on the bus — and the claim on
// `mesh-broker` is what authorises it, checked from this manifest alone.
BusUsers string `json:"bus-users,omitempty"`
// Bundles are this module's compiled bundles as the build produced them: what each is called,
// where it is, what it hashes to, what language it is in and which files a tool runtime loads
// from it (novox/hq ADR 0175, to-be 38).
//
// **Derived, never written.** The manifest in a repository says `build.artifacts`; the manifest
// the mesh holds says what came out, the way a resource naming an artifact comes to name a
// digest. Kept here because a tools bundle is referenced by no resource of the module's own —
// the node's runtime loads it, and the runtime is composed by the mesh — so without this the
// resolved manifest would carry no trace of the one artifact the runtime needs. A repository
// manifest that writes this beside a build is refused: it would be stating the build's output
// by hand.
Bundles []Bundle `json:"bundles,omitempty"`
}
// Bundle is one compiled bundle after it exists, as the resolved manifest carries it.
type Bundle struct {
Name string `json:"name"`
// Source is where a machine fetches it, kept without the store's address like every reference
// the mesh records (artifacts.go); Digest is what it must hash to.
Source string `json:"source"`
Digest string `json:"digest"`
// Language is what it was compiled from, which is what says how it is run.
Language string `json:"language,omitempty"`
// Entrypoints are the compiled files it was built around, relative to its root.
Entrypoints []string `json:"entrypoints,omitempty"`
// Loads are the entrypoints a node's tool runtime imports from it: what the artifact said, or
// every entrypoint for a module declaring tools that said nothing. Empty for a bundle that is
// run rather than loaded.
Loads []string `json:"loads,omitempty"`
// Env is what the artifact said it is given (ADR 0192), as written; composed per machine.
Env map[string]string `json:"env,omitempty"`
}
// Build says how to produce this module's artifacts from its source.
@@ -708,6 +740,21 @@ type Artifact struct {
// somebody adds a helper. An empty list is a bundle that is run rather than loaded — a
// provisioner or a step, named by whatever runs it.
Entrypoints []string `json:"entrypoints,omitempty"`
// Loads are the entrypoints of this bundle the node's tool runtime loads (novox/hq ADR 0175,
// to-be 38): the module's tool code, each file registering its tools as it is imported. A
// subset of Entrypoints, for a bundle that also carries things that are RUN — a daemon, a
// step, a report — and must not have them imported into the runtime.
//
// Absent means every entrypoint, for a module that declares `tools`: a bundle holding the
// module's tools and nothing else is the ordinary case and should not have to say the same
// list twice. A module declaring no tools has nothing the runtime loads, whatever it compiles.
Loads []string `json:"loads,omitempty"`
// Env is what a tools bundle is given on a machine (novox/hq ADR 0192): words and their values,
// paths and constants composed with ${dir:…} and ${port:…} exactly as a container's environment
// is, never a secret's content. The node's runtime hands it to this bundle and to no other.
Env map[string]string `json:"env,omitempty"`
}
// Kinds an artifact may be.
@@ -1333,6 +1380,15 @@ func ParseManifest(raw []byte) (Manifest, error) {
//
// Refused here because the alternative is a build that never returns, on a mesh new enough
// that nobody is watching it yet.
if m.Build != nil && len(m.Bundles) > 0 {
// The output of a build, written beside the build that produces it (ADR 0175). A resource
// naming a digest beside an `artifact` would be the same mistake, and is caught the same way:
// what the mesh derives, a repository does not state.
problems = append(problems, fmt.Sprintf(
"%s writes `bundles` beside its build. The mesh derives that from what the build "+
"produced; a manifest states `build.artifacts` and nothing about what came out",
m.Module))
}
if m.Build != nil && len(m.Build.Artifacts) > 0 {
for _, o := range m.Offers() {
if o != ArtifactStoreProvision {
@@ -1667,6 +1723,7 @@ func ParseManifest(raw []byte) (Manifest, error) {
problems = append(problems, m.undeclaredMounts()...)
problems = append(problems, m.unknownDirRefs()...)
problems = append(problems, m.unknownAccessRefs()...)
problems = append(problems, m.jailProblems()...)
for i, r := range m.Resources {
id, _ := r["id"].(string)
@@ -1769,6 +1826,46 @@ func (m Manifest) MachineSide(port int) (at int, mayAssign bool) {
var facilitiesOf = map[string][]string{
// Both spellings: /var/run is a link to /run on every machine the mesh runs on.
"container-runtime": {"/var/run/docker.sock", "/run/docker.sock"},
// The virtualisation daemon's socket, for the lab (novox/hq ADR 0172): it raises machines there.
"virtualisation": {"/var/lib/incus/unix.socket"},
}
// jailProblems is every jail this module declares that the machine's intrusion prevention would
// refuse (novox/hq ADR 0179).
//
// **Because one bad pattern stops every jail, not its own.** fail2ban expands `<HOST>` into a named
// capture group, so a pattern naming it twice is a duplicate group name, and the daemon refuses the
// whole configuration and exits — the machine keeps no bans at all, for any jail, including the one
// watching its ssh. Caught live on the control node the day this was built, where a proxy's pattern
// matched two shapes of refusal in one line. A pattern matches one shape; several shapes are several
// patterns, one per line, as fail2ban's own filters are written.
func (m Manifest) jailProblems() []string {
var problems []string
seen := map[string]bool{}
for _, j := range m.Jails {
switch {
case strings.TrimSpace(j.Name) == "":
problems = append(problems, m.Module+" declares a jail with no name")
case seen[j.Name]:
problems = append(problems, m.Module+" declares two jails called "+strconv.Quote(j.Name))
}
seen[j.Name] = true
if strings.TrimSpace(j.Failregex) == "" {
problems = append(problems, m.Module+"'s jail "+strconv.Quote(j.Name)+" says nothing a failed attempt looks like")
}
for _, line := range strings.Split(j.Failregex, "\n") {
if strings.TrimSpace(line) == "" {
continue
}
if n := strings.Count(line, "<HOST>"); n > 1 {
problems = append(problems, fmt.Sprintf("%s's jail %s names <HOST> %d times in one pattern; "+
"fail2ban reads it as one capture group and refuses the whole configuration, so the machine "+
"keeps no bans at all — write one pattern per shape, each naming <HOST> once",
m.Module, strconv.Quote(j.Name), n))
}
}
}
return problems
}
// undeclaredMounts is every bind-mount source no declaration covers — see the check above.
@@ -1810,6 +1907,12 @@ func (m Manifest) undeclaredMounts() []string {
claim(p)
}
}
// The file a filter module's rule set is written to is declared by `filtering.into`: the mesh
// writes it, the module loads it, and the module's runtime may read it back to reload the
// mesh's own table (novox/hq ADR 0170).
if m.Filtering != nil {
claim(m.Filtering.Into)
}
// Under a declared directory is declared: a module that says where its data lives has said so
// for what it puts inside.
covers := func(path string) bool {
+10
View File
@@ -98,3 +98,13 @@ func TestAMountOfABoundFactIsAccepted(t *testing.T) {
t.Fatalf("a mount of the file the mesh writes a binding to was refused: %v", err)
}
}
// The file a filter module's rule set is written to is declared by `filtering.into` (novox/hq ADR
// 0169): the module's runtime mounts it to reload the mesh's own table, and nothing else declares it.
func TestAMountOfTheFilterFileIsDeclaredByFilteringInto(t *testing.T) {
_, err := ParseManifest([]byte(`{"module":"nftables","filtering":{"into":"/etc/nftables.conf"},` +
`"resources":[` + strings.Replace(aContainerMounting, "%s", "/etc/nftables.conf", 1) + `]}`))
if err != nil {
t.Fatalf("a filter module mounting its own filter file was refused: %v", err)
}
}
-124
View File
@@ -1,124 +0,0 @@
package catalogue
import (
"sort"
"strings"
)
// Which machine serves each routed name (novox/hq ADR 0066, issue 178).
//
// A routed name is a label the mesh composed for a consumer's endpoint, and it is *served* by the
// provider that answers requests for it — the proxy the consumer's route reaches. The same name is
// composed into every labelled contribution the consumer makes, because a provider that must know
// the consumer's public name (an identity provider composing a redirect) is told it the same way
// (04-ISSUES/122). Attributing the name to whichever of those providers a map happened to yield
// last sent a public name to the identity provider's machine on one plan and to the proxy's on the
// next (forge issue 227), and the whole names region flipped with it.
//
// **The terminus serves the name.** Among the providers a name reaches, the one that serves it is
// the one that is not itself routed: a provider that contributes a labelled name of its own to some
// requirement is published through another provider, and is a consumer of names, not their end.
// Name-agnostic — nothing here knows what "route" means — and structural: it reads the graph the
// modules declared. Deterministic: names, requirements and nodes are walked in order, so two
// plans of one mesh yield one region.
// NamesServed is every routed name across the mesh and the node that serves it, from every node's
// resolution and settings. A name several termini claim goes to the first node in name order, so
// the answer is stable; a name nothing terminal claims is left out.
func NamesServed(plans map[string]Resolution, settings map[string]SettingsBy) (map[string]string, error) {
nodes := make([]string, 0, len(plans))
for n := range plans {
nodes = append(nodes, n)
}
sort.Strings(nodes)
out := map[string]string{}
for _, node := range nodes {
plan := plans[node]
all, err := plan.contributions(settings[node], nil, nil)
if err != nil {
return nil, err
}
requirements := make([]string, 0, len(all))
for to := range all {
requirements = append(requirements, to)
}
sort.Strings(requirements)
for _, to := range requirements {
for _, given := range all[to] {
if given.Node != "" {
// Said from another machine; that machine's own resolution carries it.
continue
}
// A routed name, and only that: a contribution the mesh composed a name for from a
// label it was given. A grant that happens to carry a `name` of its own — a database
// name — carries no label and is left alone.
if _, labelled := given.Values["label"]; !labelled {
continue
}
name, _ := given.Values["name"].(string)
if name == "" {
continue
}
serving := servingNodeOf(plan, to, given.From, node)
if !servesNames(plans[serving], to) {
continue
}
name = strings.ToLower(name)
if held, taken := out[name]; !taken || serving < held {
out[name] = serving
}
}
}
}
return out, nil
}
// servingNodeOf is the node answering one consumer's requirement: whoever the plan needs it from,
// or this same node when the provider is beside the consumer.
func servingNodeOf(plan Resolution, requirement, consumer, self string) string {
for _, need := range plan.Needs {
if need.Name == requirement && need.For == consumer && need.From != "" {
return need.From
}
}
return self
}
// servesNames says whether the module providing a requirement on a node is a terminus: it is not
// itself published under a labelled name through some other provider. A node whose plan is not
// known (it did not resolve) serves nothing.
func servesNames(plan Resolution, requirement string) bool {
for _, m := range plan.Modules {
if !offers(m, requirement) {
continue
}
return !contributesALabel(m)
}
return false
}
func offers(m Manifest, requirement string) bool {
for _, o := range m.Offers() {
if o == requirement {
return true
}
}
return false
}
func contributesALabel(m Manifest) bool {
for _, values := range m.Contributes {
if _, labelled := values["label"]; labelled {
return true
}
}
for _, locals := range m.ContributesMany {
for _, values := range locals {
if _, labelled := values["label"]; labelled {
return true
}
}
}
return false
}
-99
View File
@@ -1,99 +0,0 @@
package catalogue
import (
"testing"
)
// The mesh of forge issue 227 (novox/hq issue 178): a dashboard on the home server contributes its
// label to the route its proxy serves AND to the identity provider on the control node, which must
// know the dashboard's public name to compose a redirect. Both contributions carry the composed
// name; only the proxy serves it.
func twoNodesOneName(t *testing.T) (map[string]Resolution, map[string]SettingsBy) {
t.Helper()
catalogue := shelf(
Manifest{Module: "route-adapter", Version: "1", Provides: Offers("route"),
Serves: map[string]map[string]any{"route": {}}, Receives: map[string]string{"route": "/etc/adapter/mesh.json"}},
Manifest{Module: "route-proxy", Version: "1", Provides: Offers("route"),
Serves: map[string]map[string]any{"route": {}}, Receives: map[string]string{"route": "/etc/proxy/mesh.json"}},
Manifest{Module: "keycloak", Version: "1", Provides: FromAnywhere("oidc-client"),
Serves: map[string]map[string]any{"oidc-client": {"token-path": "/token"}},
Receives: map[string]string{"oidc-client": "/etc/keycloak/clients.json"},
Listens: []Listening{{Port: 8080, From: FromMesh, Why: "the login page"}},
// Published through the proxy itself: the identity provider is routed, not a router.
Contributes: map[string]map[string]any{"route": {"label": "login", "endpoint": "web", "port": 8080}}},
Manifest{Module: "grafana", Version: "1",
Listens: []Listening{{Port: 3000, From: FromMesh, Why: "dashboards"}},
Contributes: map[string]map[string]any{
"route": {"label": "grafana", "endpoint": "web", "port": 3000},
"oidc-client": {"label": "grafana", "endpoint": "web", "port": 3000, "callback": "/login"},
}},
)
home := withDomain("home.example")
home.Name, home.At = "home-server", "home-server.internal"
control := withDomain("control.example")
control.Name, control.At = "anchor", "anchor.internal"
onHome, err := Resolve(catalogue, []string{"grafana", "route-adapter"}, home, World{
Offered: map[string][]Provider{"oidc-client": {{Node: "anchor", At: "anchor.internal", Module: "keycloak"}}},
})
if err != nil {
t.Fatal(err)
}
onControl, err := Resolve(catalogue, []string{"keycloak", "route-proxy"}, control, World{})
if err != nil {
t.Fatal(err)
}
return map[string]Resolution{"home-server": onHome, "anchor": onControl},
map[string]SettingsBy{"home-server": {}, "anchor": {}}
}
func TestANameResolvesToTheNodeWhoseProxyServesIt(t *testing.T) {
plans, settings := twoNodesOneName(t)
// Many times, because the fault was map order: one plan said one node, the next the other.
for i := 0; i < 25; i++ {
served, err := NamesServed(plans, settings)
if err != nil {
t.Fatal(err)
}
if served["grafana.home.example"] != "home-server" {
t.Fatalf("run %d: the dashboard's name is served by %q, and its proxy is on the home server: %v",
i, served["grafana.home.example"], served)
}
if served["login.control.example"] != "anchor" {
t.Fatalf("run %d: the identity provider's own name is served by its proxy on the control node: %v", i, served)
}
if _, leaked := served["grafana.control.example"]; leaked {
t.Fatalf("a name composed for the identity provider's benefit is not one it serves: %v", served)
}
}
}
// A module that is routed several times names each route (ADR 0094's sibling for contributes);
// every one of them is a name the mesh must resolve, and none reached the names region before.
func TestEveryRouteOfAModuleWithSeveralIsANameServed(t *testing.T) {
catalogue := shelf(
Manifest{Module: "route-proxy", Version: "1", Provides: Offers("route"),
Serves: map[string]map[string]any{"route": {}}, Receives: map[string]string{"route": "/etc/proxy/mesh.json"}},
Manifest{Module: "photos", Version: "1",
Listens: []Listening{{Port: 8102, From: FromMesh, Why: "web"}, {Port: 9102, From: FromMesh, Why: "api"}},
ContributesMany: map[string]map[string]map[string]any{"route": {
"site": {"label": "photos", "endpoint": "web", "port": 8102},
"api": {"label": "photos-api", "endpoint": "api", "port": 9102},
}}},
)
node := withDomain("control.example")
node.Name, node.At = "anchor", "anchor.internal"
plan, err := Resolve(catalogue, []string{"photos", "route-proxy"}, node, World{})
if err != nil {
t.Fatal(err)
}
served, err := NamesServed(map[string]Resolution{"anchor": plan}, map[string]SettingsBy{"anchor": {}})
if err != nil {
t.Fatal(err)
}
for _, name := range []string{"photos.control.example", "photos-api.control.example"} {
if served[name] != "anchor" {
t.Fatalf("%s is not served by its proxy: %v", name, served)
}
}
}
@@ -0,0 +1,60 @@
package catalogue
import (
"strings"
"testing"
)
// A `user` shape and a user-scoped unit name the operator account the way a home file does
// (novox/hq ADR 0176, ADR 0177): with ${machine:account}, resolved when the module is assigned.
func TestAUserShapeAndAUserScopedUnitNameTheAccount(t *testing.T) {
facts := map[string]string{"account": "ops", "account-home": "/home/ops"}
login := map[string]any{"type": "user", "id": "login", "name": "${machine:account}", "shell": "/usr/bin/zsh"}
if err := machineInto(login, facts, "zsh"); err != nil {
t.Fatal(err)
}
if login["name"] != "ops" {
t.Fatalf("the user shape did not learn the account: %v", login["name"])
}
watcher := map[string]any{"type": "service", "id": "watcher", "unit": "i3-reload-watcher.service",
"scope": "user", "user": "${machine:account}"}
if err := machineInto(watcher, facts, "i3"); err != nil {
t.Fatal(err)
}
if watcher["user"] != "ops" {
t.Fatalf("the user-scoped unit did not learn the account: %v", watcher["user"])
}
// A machine with no operator account refuses rather than writing the literal.
err := machineInto(map[string]any{"type": "user", "id": "login", "name": "${machine:account}"},
map[string]string{"address": "10.0.0.1"}, "zsh")
if err == nil || !strings.Contains(err.Error(), "${machine:account}") {
t.Fatalf("a user shape on a machine with no account was not refused by name: %v", err)
}
}
// The service manager is a seat of the mesh's own with the unit verbs as its contract (novox/hq
// ADR 0177): every verb described, with a schema, taking a scope.
func TestTheServiceManagerSeatServesTheUnitVerbs(t *testing.T) {
seat, ok := SeatNamed("node-service-manager")
if !ok {
t.Fatal("node-service-manager is not a seat the mesh defines")
}
if seat.Scope != ScopeNode {
t.Fatalf("the service manager is a role each machine has once, and the seat is %s-scoped", seat.Scope)
}
want := []string{"units", "status", "start", "stop", "restart", "enable", "disable", "journal"}
var got []string
for _, v := range seat.Serves {
got = append(got, v.Name)
if v.Description == "" || v.Input == nil {
t.Fatalf("%s is promised without a description or a schema", v.Name)
}
props, _ := v.Input["properties"].(map[string]any)
if _, has := props["scope"]; !has {
t.Fatalf("%s takes no scope, and a user unit could not be asked for", v.Name)
}
}
if strings.Join(got, ",") != strings.Join(want, ",") {
t.Fatalf("the seat serves %v, not %v", got, want)
}
}
+14 -5
View File
@@ -75,17 +75,26 @@ func TestAnAssignmentPlacesDirectoriesAndAccesses(t *testing.T) {
}
// An access declared by id and placed by nobody resolves to nowhere, and that is refused with the
// setting to write — not mounted as the literal, not skipped.
// setting to write — not mounted as the literal, not skipped. The refusal costs the module its
// place in the declaration, not the machine its declaration (novox/hq ADR 0163, rule 6).
func TestAnUnplacedAccessIsRefusedByName(t *testing.T) {
got, err := Resolve(shelf(placeable()), []string{"arr"}, workstation(), World{})
if err != nil {
t.Fatal(err)
}
_, err = got.Declaration(placedBy(map[string]any{
with := placedBy(map[string]any{
AccessesSetting: map[string]any{"series": "/storage/media/series"},
}))
if err == nil || !strings.Contains(err.Error(), `"spool"`) || !strings.Contains(err.Error(), AccessesSetting) {
t.Fatalf("an access nobody placed was not refused by name: %v", err)
})
composed, err := got.Compose(with)
if err != nil {
t.Fatal(err)
}
why := composed.LeftOut["arr"]
if why == "" || !strings.Contains(why, `"spool"`) || !strings.Contains(why, AccessesSetting) {
t.Fatalf("an access nobody placed was not refused by name: %v", composed.LeftOut)
}
if _, declared := byID(composed.Resources)["arr.server"]; declared {
t.Fatal("the module with the unplaced access was declared anyway")
}
}
@@ -0,0 +1,42 @@
package catalogue
import (
"strings"
"testing"
)
// A resolved manifest keeps a built artifact's reference in the store-relative form, never the
// address the builder reached the store by (novox/hq ADR 0155): on 2026-10-02 the first bundle
// resolved on the mesh carried the store's host in bundles[0].source and registration refused it
// as naming an installation. Archive resources are the same kind of reference and get the same.
func TestAResolvedReferenceIsKeptNotRouted(t *testing.T) {
m, err := ParseManifest([]byte(`{
"module": "sample", "version": "1",
"tools": ["one"],
"build": {"artifacts": [
{"name": "code", "kind": "bundle", "language": "typescript", "entrypoints": ["tools/index.js"]},
{"name": "files", "kind": "archive", "from": "files"}
]},
"resources": [{"id": "packed", "type": "archive", "path": "/opt/sample", "artifact": "files"}]
}`))
if err != nil {
t.Fatal(err)
}
digest := "sha256:" + strings.Repeat("ab", 32)
resolved, err := m.Resolve([]Built{
{Name: "code", Kind: ArtifactBundle, Reference: "http://store.example:5100/v2/sample/code/blobs/" + digest, Digest: digest},
{Name: "files", Kind: ArtifactArchive, Reference: "http://store.example:5100/v2/sample/files/blobs/" + digest, Digest: digest},
})
if err != nil {
t.Fatal(err)
}
if got, want := resolved.Bundles[0].Source, ArtifactStoreScheme+"sample/code/blobs/"+digest; got != want {
t.Errorf("bundle source %q, want the kept form %q", got, want)
}
if got, want := resolved.Resources[0]["source"], ArtifactStoreScheme+"sample/files/blobs/"+digest; got != want {
t.Errorf("archive source %q, want the kept form %q", got, want)
}
if problems := InstallationProblems(resolved); len(problems) != 0 {
t.Errorf("a resolved manifest names an installation: %v", problems)
}
}
+16 -6
View File
@@ -48,7 +48,9 @@ func TestTheResolverForwardsToFixedUpstreamsAndNeverReadsResolvConf(t *testing.T
}
for _, want := range []string{
"\nno-resolv\n", "\nserver=1.1.1.1\n", "\nserver=8.8.8.8\n",
"\nlisten-address=127.0.0.1\n", "\nlisten-address=${machine:address}\n", "\nbind-dynamic\n",
// Loopback is the mesh-wide setting's default; a machine answering its own LAN adds its
// address there (novox/hq issue 198).
"\nlisten-address=${setting:listen-addresses}\n", "\nlisten-address=${machine:address}\n", "\nbind-dynamic\n",
"\ndomain-needed\n", "\nbogus-priv\n",
"\nconf-file=" + m.Facts["node-zones"].Path + "\n",
} {
@@ -113,7 +115,8 @@ func TestTheResolverAndWhatAsksItComposeOnOneMachine(t *testing.T) {
// issue 111) — the resolver's zones read only the second, and in this scenario the two
// happen to be the same map, since nothing routed is part of it.
Names: twoMachines, Machines: twoMachines, Suffix: "internal",
Needed: map[string]map[string]string{"dnsmasq": {"broker": "sealed"}},
Needed: map[string]map[string]string{"dnsmasq": {"broker": "sealed"}},
Settings: SettingsBy{"dnsmasq": {{From: "the mesh", Values: map[string]any{"listen-addresses": "127.0.0.1"}}}},
})
if err != nil {
t.Fatal(err)
@@ -207,9 +210,16 @@ func TestTheResolverOnAMachineOffTheNetworkIsRefused(t *testing.T) {
if err != nil {
t.Fatal(err)
}
_, err = got.Declaration(Rendering{Names: twoMachines, Suffix: "internal",
Needed: map[string]map[string]string{"dnsmasq": {"broker": "sealed"}}})
if err == nil || !strings.Contains(err.Error(), "${machine:address}") {
t.Fatalf("a machine off the network was composed a resolver, or refused for another reason: %v", err)
// Left out of the declaration and said, rather than composed listening nowhere: a module that
// cannot compose on a machine is kept as it is there, with the reason (hq ADR 0163).
composed, err := got.Compose(Rendering{Names: twoMachines, Suffix: "internal",
Needed: map[string]map[string]string{"dnsmasq": {"broker": "sealed"}},
Settings: SettingsBy{"dnsmasq": {{From: "the mesh", Values: map[string]any{"listen-addresses": "127.0.0.1"}}}}})
if err == nil && !strings.Contains(composed.LeftOut["dnsmasq"], "${machine:address}") {
t.Fatalf("a machine off the network was composed a resolver, or left out for another reason: %v",
composed.LeftOut)
}
if err != nil && !strings.Contains(err.Error(), "${machine:address}") {
t.Fatalf("a machine off the network was refused for another reason: %v", err)
}
}
+4 -4
View File
@@ -28,10 +28,10 @@ import (
// A RosterFile is a file the mesh renders from the roster of machines, in the format the module
// gives as a Go text/template. The template sees a rosterView: `.Node` (this machine's bare name),
// `.Suffix` (what its mesh name ends in), and two sets of `{Name, FQDN, Address}` — `.Names`, every
// name the mesh serves, and `.Machines`, only the nodes of the mesh. Which set a template ranges is
// how the hq issue 111 distinction is drawn: a container's hosts wants every name; a resolver told
// the suffix is its own wants only the machines.
// `.Suffix` (what its mesh name ends in), and two sets of `{Name, FQDN, Address}` — `.Names` and
// `.Machines`. Both are the nodes of the mesh: routed names were once in `.Names` too, and are not
// since every route became a name under its node's internal domain (novox/hq ADR 0191) — the hq
// issue 111 distinction is kept as two fields so the templates that range either keep rendering.
type RosterFile struct {
// Path is where on the machine the rendered file goes. Absolute, or it is refused here rather
// than discovered as a daemon that reads nothing.
+352
View File
@@ -0,0 +1,352 @@
package catalogue
import (
"encoding/json"
"fmt"
"sort"
"strings"
)
// The node's tool runtime, as the catalogue knows it (novox/hq ADR 0175, to-be 38).
//
// **One module is the runtime.** Where it is assigned, one process per machine serves every assigned
// module's tools and every held seat's verbs, on the host side, from the bundles each module's build
// produced — and no module needs a container to reach the bus with its tools. The name is a constant
// rather than a manifest field because a rule turns on it: the composer places the runtime's process
// where this module is, and registration refuses the old pattern once this module exists.
// RuntimeModule is the module that is the node's tool runtime. Mirrored in the broker package,
// which composes a principal of its own for it; the agreement test there holds the two to one string.
const RuntimeModule = "node-tools"
// BundleRoot is where a machine keeps the tools bundles the mesh delivers to it: under the mesh's
// own directory, beside the daemons the host unpacks there, and never where a package manager also
// writes. One directory per module, one per bundle beneath it, at a path that does not move with
// the version — so the runtime's process names each entrypoint once and is restarted, not
// recomposed, when a bundle changes.
const BundleRoot = "/var/lib/mesh/bundles"
// BundleID names the archive resource that delivers one of a module's bundles; prefixed with the
// module like every resource of its own.
func BundleID(bundle string) string { return "bundle-" + bundle }
// BundlePath is where one module's bundle is unpacked on a machine.
func BundlePath(module, bundle string) string { return BundleRoot + "/" + module + "/" + bundle }
// runtimeHere says whether this node's set includes the runtime module, which is what decides
// whether anything about tools changes on the machine (to-be 38 WP2): until the runtime is assigned,
// a node is sent exactly what it was sent before, bundles included, because a bundle nothing loads
// is bytes nobody reads.
func (r Resolution) runtimeHere() bool {
for _, m := range r.Modules {
if m.Module == RuntimeModule {
return true
}
}
return false
}
// bundleArchives is one archive per tools bundle of a module — a bundle the runtime LOADS something
// from — as the host fetches and unpacks any artifact (novox/hq ADR 0175 §3: a module brings its
// tools as a bundle, delivered by the host like any artifact, never an image). A bundle it loads
// nothing from is run rather than loaded: a daemon, a step, the runtime itself — delivered by the
// process that runs it, and not again here.
//
// The source is the kept reference; the per-resource pass that follows routes it through the
// artifact store as this network reaches it now, as it does every image and archive the mesh built.
func bundleArchives(m Manifest) []map[string]any {
var out []map[string]any
for _, b := range m.Bundles {
if len(b.Loads) == 0 {
continue
}
out = append(out, map[string]any{
"id": BundleID(b.Name), "type": "archive",
"source": b.Source, "digest": b.Digest,
"path": BundlePath(m.Module, b.Name),
})
}
return out
}
// RuntimeProcessID names the one process the mesh composes for a machine's runtime; prefixed with
// the runtime module like a resource of its own, because that module is what the host sees it as.
func RuntimeProcessID() string { return "runtime" }
// RuntimeToolModules is the variable the runtime reads the modules it serves from: one
// `<module>=<entrypoint>` per file it loads, comma-separated — several entries may name one module.
// RuntimeBrokerFile is where it reads the node's credential; RuntimeOperatorAccount and
// RuntimeOperatorHome are the machine's operator account and home, handed to every tool's
// environment (to-be 38 WP1), and absent on a machine with no account.
const (
RuntimeToolModules = "MESH_TOOL_MODULES"
RuntimeBrokerFile = "MESH_BROKER_FILE"
RuntimeOperatorAccount = "MESH_OPERATOR_ACCOUNT"
RuntimeOperatorHome = "MESH_OPERATOR_HOME"
// RuntimeToolEnv is every served module's composed environment, as JSON (novox/hq ADR 0192):
// {"<module>": {"<word>": "<value>"}}. The runtime takes it at start, removes it from its own
// environment and hands each module's words to that module's bundles alone. In the unit, so a
// change to any module's words changes the process and restarts it.
RuntimeToolEnv = "MESH_TOOL_ENV"
)
// interpreterFor is how a bundle in a language is run: the program the host's unit starts, with the
// bundle's entrypoint after it. The one thing the composer takes from a language, and said here
// rather than in a manifest because the runtime's process is the mesh's to compose (to-be 38 WP3).
func interpreterFor(language string) (string, error) {
switch language {
case "typescript":
return "node", nil
}
return "", fmt.Errorf(
"%s is written in %q, and the mesh knows no interpreter to run a %q bundle with",
RuntimeModule, language, language)
}
// runtimeProcess is the one process a machine runs the node's tool runtime as (novox/hq ADR 0175,
// to-be 38 WP2.3): the runtime module's own bundle, run by its language's interpreter, told which
// modules it serves and from which files, where its credential is, and who the machine's operator
// is — and restarted when any bundle it loads or the credential it holds changes.
//
// Composed from the placed manifests, so the credential's path is where this node puts it. The
// runtime runs as the operator's account when the machine has one, which is what lets a tool that
// needs root escalate as the operator would (ADR 0175 §4); on a machine with no account it runs as
// root, and the two operator words are not set.
func (r Resolution) runtimeProcess(with Rendering) (map[string]any, error) {
var runtime *Manifest
for i := range r.Modules {
if r.Modules[i].Module == RuntimeModule {
runtime = &r.Modules[i]
}
}
if runtime == nil {
return nil, nil
}
if len(runtime.Bundles) != 1 {
return nil, fmt.Errorf(
"%s is assigned to %s and its build produced %d bundle(s); the runtime is one bundle "+
"the mesh runs, so the module declares exactly one (novox/hq to-be 38)",
RuntimeModule, r.Node, len(runtime.Bundles))
}
bundle := runtime.Bundles[0]
if len(bundle.Entrypoints) != 1 {
return nil, fmt.Errorf(
"%s's bundle %q names %d entrypoint(s); the runtime is run from one, so the module "+
"declares exactly one (novox/hq to-be 38)", RuntimeModule, bundle.Name, len(bundle.Entrypoints))
}
interpreter, err := interpreterFor(bundle.Language)
if err != nil {
return nil, err
}
credential, declared := runtime.OwnSecrets["broker"]
if !declared {
return nil, fmt.Errorf(
"%s declares no own secret named broker, and the node's credential is delivered there: "+
"a module that speaks on the bus declares \"own-secrets\": {\"broker\": <path>}",
RuntimeModule)
}
// What it serves, and from which files: every module on this machine that composes here, in
// name order, each bundle it loads from in the order the manifest gave. A module left out of
// the declaration — a filter on an adopted machine — is left out of this too, or the runtime
// would be told to load files that were never delivered.
var served []string
var restartOn []string
given := map[string]map[string]string{}
for _, m := range r.Modules {
if with.Adopted && m.Filtering != nil {
continue
}
words, err := bundleWords(m, with)
if err != nil {
return nil, err
}
if len(words) > 0 {
given[m.Module] = words
}
for _, b := range m.Bundles {
if len(b.Loads) == 0 {
continue
}
for _, load := range b.Loads {
served = append(served, m.Module+"="+BundlePath(m.Module, b.Name)+"/"+load)
}
restartOn = append(restartOn, m.Module+"."+BundleID(b.Name))
}
}
sort.Strings(served)
restartOn = append(restartOn, RuntimeModule+"."+NeedID("broker"))
sort.Strings(restartOn)
env := map[string]string{
RuntimeToolModules: strings.Join(served, ","),
RuntimeBrokerFile: credential.Path,
}
if len(given) > 0 {
// Marshalled from maps, whose keys encoding/json sorts: the same words, the same unit.
body, err := json.Marshal(given)
if err != nil {
return nil, err
}
env[RuntimeToolEnv] = string(body)
}
process := map[string]any{
"id": RuntimeModule + "." + RuntimeProcessID(), "type": "process", "name": RuntimeModule,
"source": bundle.Source, "digest": bundle.Digest,
"run": []any{interpreter, bundle.Entrypoints[0]},
"env": env,
"restart-on": toAny(restartOn),
}
if r.Account != "" {
env[RuntimeOperatorAccount] = r.Account
env[RuntimeOperatorHome] = accountHomeOf(r.Account, r.AccountHome)
process["user"] = r.Account
}
// Routed through the artifact store as this network reaches it now, like everything the mesh
// built; refused with the same words when there is no store to route through.
if err := artifactsInto(process, RuntimeModule, with); err != nil {
return nil, err
}
return process, nil
}
func toAny(in []string) []any {
out := make([]any, 0, len(in))
for _, s := range in {
out = append(out, s)
}
return out
}
// RuntimeImageModule and RuntimeImageArtifact name the image every per-module tool container was
// built on: the tool runtime's own runtime image. With the runtime a module of its own, that image
// stays the way a module's SERVICE may be built and stops being the way tools reach a node (ADR 0175).
const (
RuntimeImageModule = "mesh-tools"
RuntimeImageArtifact = "runtime"
)
// ToolContainerOnTheRuntime says why a manifest is the pattern ADR 0175 retires — a module whose tools
// are served from a container built on the tool runtime's image — or nothing when it is not. Judged
// from the manifest's own `build.on` when it is a repository manifest, and from what its build stood
// on when it is a built one, because a resolved manifest carries no build. The gate itself is
// registration's (to-be 38 WP2.4): once the runtime module is in the catalogue, this is refused.
//
// Three things must hold, and each alone is fine: declaring tools (a bundle does that); a container
// (a module's service may well be one); building on the runtime's image (a service written against
// the SDK may). All three is a container whose purpose is tools, which the runtime now serves.
func ToolContainerOnTheRuntime(m Manifest, against []string) string {
if len(m.Tools) == 0 {
return ""
}
container := false
for _, r := range m.Resources {
if fmt.Sprint(r["type"]) == "container" {
container = true
}
}
if !container {
return ""
}
onTheRuntime := false
if m.Build != nil {
for _, on := range m.Build.On {
if on.Module == RuntimeImageModule && on.Artifact == RuntimeImageArtifact {
onTheRuntime = true
}
}
}
for _, ref := range against {
path, kept := InArtifactStore(Recorded(ref))
if kept && strings.HasPrefix(path, RuntimeImageModule+"/"+RuntimeImageArtifact+"@") {
onTheRuntime = true
}
}
if !onTheRuntime {
return ""
}
return fmt.Sprintf(
"%s declares tools and a container built on %s's %s image — a container whose purpose is "+
"serving tools. The node's tool runtime (%s) serves every module's tools from its bundle "+
"now (novox/hq ADR 0175, to-be 38); declare the tools as a bundle and drop the container",
m.Module, RuntimeImageModule, RuntimeImageArtifact, RuntimeModule)
}
// bundleWords is what one module's tools bundles are given on this machine (novox/hq ADR 0192):
// each loaded bundle's env, its ${dir:…} resolved to where this machine places the module's
// directories and its ${port:…} to the port this machine gave it — the same resolution a
// container's environment gets. Two bundles of one module naming one word differently is refused:
// the runtime hands a module's words to all its bundles.
func bundleWords(m Manifest, with Rendering) (map[string]string, error) {
var out map[string]string
dirs := dirsFor(m, with)
for _, b := range m.Bundles {
if len(b.Loads) == 0 || len(b.Env) == 0 {
continue
}
for _, word := range sortedKeys(b.Env) {
value, err := dirFill(b.Env[word], dirs, m.Module)
if err != nil {
return nil, err
}
if value, err = portsFilledInto(value, m.Module+"'s bundle "+b.Name+" ("+word+")", m.Module, m.Listens, with); err != nil {
return nil, err
}
if out == nil {
out = map[string]string{}
}
if was, had := out[word]; had && was != value {
return nil, fmt.Errorf("%s's bundles give %s two values (%q, %q); a module's words are "+
"handed to all its bundles, so they agree (novox/hq ADR 0192)", m.Module, word, was, value)
}
out[word] = value
}
}
return out, nil
}
// givenTo makes what a bundle's words name readable by the account the runtime runs as (novox/hq
// ADR 0192): every file and directory of the module whose path a word names, or that holds one,
// is owned by the account — a tool reads its configuration and its secret as the account, and a
// root-owned 0600 file or a 0700 directory is one it cannot. Only where it says no owner already:
// a module that named one knew why. Nothing on a machine with no account, where the runtime is root.
//
// It answers the files a word names exactly: what a tool reads, whose change the runtime must be
// restarted for, as the container the tools came from was restarted when its configuration changed.
func givenTo(out []map[string]any, owner map[string]string, words map[string]map[string]string, account string) []string {
var named []string
if len(words) == 0 {
return nil
}
for _, resource := range out {
module := owner[fmt.Sprint(resource["id"])]
mine := words[module]
if len(mine) == 0 {
continue
}
kind := fmt.Sprint(resource["type"])
if kind != "file" && kind != "directory" {
continue
}
path, _ := resource["path"].(string)
if path == "" {
continue
}
for _, value := range mine {
if kind == "file" && value == path {
named = append(named, fmt.Sprint(resource["id"]))
}
}
if _, said := resource["owner"]; said || account == "" {
continue
}
for _, value := range mine {
if value == path || strings.HasPrefix(value, strings.TrimRight(path, "/")+"/") {
resource["owner"] = account
break
}
}
}
sort.Strings(named)
return named
}
+88
View File
@@ -0,0 +1,88 @@
package catalogue
import (
"strings"
"testing"
)
// The packet-filter manifest as it was the day the runtime was decided (novox/hq ADR 0175): tools,
// served from a container built on the tool runtime's image, with NET_ADMIN so the container could
// reach the filter. The exact pattern to-be 38 WP4 moves it off, and the one the gate refuses.
const thePacketFilterAsItWas = `{
"module": "nftables",
"version": "1",
"capabilities": ["firewall", "container-runtime"],
"claims": [{"name": "node-packet-filter", "scope": "node", "serves": ["rules", "reload", "remove"]}],
"filtering": {"into": "/etc/nftables.conf"},
"resources": [
{"id": "mesh-state", "type": "directory", "mode": "0700", "place": "mesh"},
{"id": "package", "type": "package", "package": "nftables"},
{"id": "unit", "type": "file", "path": "/etc/systemd/system/mesh-filter.service",
"content": "[Unit]\nDescription=The mesh's packet filter\n[Service]\nType=oneshot\nExecStart=nft -f /etc/nftables.conf\n", "mode": "0644"},
{"id": "load", "type": "service", "unit": "mesh-filter.service", "state": "running", "boot": "enabled",
"restart-on": ["unit"], "reload-on": ["filtering"]},
{"id": "runtime", "type": "container", "name": "mesh-nftables", "network": "host",
"capabilities": ["NET_ADMIN"],
"volumes": ["${dir:mesh-state}/broker:/run/secrets/broker:ro", "/etc/nftables.conf:/etc/nftables.conf:ro"],
"env": {"MESH_BROKER_FILE": "/run/secrets/broker", "MESH_FILTER_FILE": "/etc/nftables.conf"},
"artifact": "runtime"}
],
"tools": ["firewall_rules"],
"own-secrets": {"broker": "${dir:mesh-state}/broker"},
"build": {
"on": [
{"arg": "BUILD_BASE", "module": "mesh-tools", "artifact": "build"},
{"arg": "RUNTIME_BASE", "module": "mesh-tools", "artifact": "runtime"}
],
"artifacts": [{"name": "runtime", "kind": "image", "from": "Dockerfile"}]
}
}`
func TestAToolContainerOnTheRuntimeImageIsNamedForWhatItIs(t *testing.T) {
m, err := ParseManifest([]byte(thePacketFilterAsItWas))
if err != nil {
t.Fatal(err)
}
// From the repository: the manifest says what it builds on.
why := ToolContainerOnTheRuntime(m, nil)
if why == "" {
t.Fatal("the packet filter's tool container was not recognised from its build")
}
for _, word := range []string{"nftables", "mesh-tools", "runtime", "ADR 0175", "bundle"} {
if !strings.Contains(why, word) {
t.Errorf("the refusal does not say %q: %s", word, why)
}
}
// Built: the manifest carries no build, and what it stood on says the same.
built, err := m.Resolve([]Built{{Name: "runtime", Kind: ArtifactImage,
Reference: ArtifactStoreScheme + "nftables/runtime@" + digest}})
if err != nil {
t.Fatal(err)
}
stoodOn := []string{"anchor.internal:5100/mesh-tools/build@" + digest, "anchor.internal:5100/mesh-tools/runtime@" + digest}
if ToolContainerOnTheRuntime(built, stoodOn) == "" {
t.Error("the packet filter's tool container was not recognised from what its build stood on")
}
if ToolContainerOnTheRuntime(built, nil) != "" {
t.Error("a built manifest with no record of its base was judged to be on the runtime")
}
// Each of the three alone is an ordinary module.
bundle := m
bundle.Resources = m.Resources[:len(m.Resources)-1]
if ToolContainerOnTheRuntime(bundle, nil) != "" {
t.Error("a module with tools and no container is the pattern the runtime serves, and was refused")
}
service := m
service.Tools = nil
if ToolContainerOnTheRuntime(service, nil) != "" {
t.Error("a service built against the SDK, declaring no tools, was refused")
}
elsewhere := m
elsewhere.Build = &Build{On: []BuildsOn{{Arg: "NODE_BASE", Image: "node@" + digest}},
Artifacts: m.Build.Artifacts}
if ToolContainerOnTheRuntime(elsewhere, nil) != "" {
t.Error("a tool container on a public base was refused as though it were on the runtime's")
}
}
+338
View File
@@ -0,0 +1,338 @@
package catalogue
import (
"encoding/json"
"fmt"
"strings"
"testing"
)
// The node's tool runtime (novox/hq ADR 0175, to-be 38): where the runtime module is assigned, a
// machine is sent every assigned module's tools bundle as an archive, and the runtime's own process
// loading them. Where it is not, the machine is sent exactly what it was sent before.
var bundleDigest = "sha256:" + strings.Repeat("b", 64)
// aToolsModule is a module whose tools come as a compiled bundle and nothing else — the shape every
// module takes once its tool container goes (to-be 38 WP4).
func aToolsModule(t *testing.T, name string, entrypoints ...string) Manifest {
t.Helper()
m := Manifest{Module: name, Version: "1", Tools: []string{"status"},
Build: &Build{Artifacts: []Artifact{
{Name: "tools", Kind: ArtifactBundle, Language: "typescript", Entrypoints: entrypoints},
}}}
resolved, err := m.Resolve([]Built{{Name: "tools", Kind: ArtifactBundle,
Reference: ArtifactStoreScheme + name + "/tools/blobs/" + bundleDigest, Digest: bundleDigest}})
if err != nil {
t.Fatal(err)
}
return resolved
}
// theRuntime is the runtime module as the catalogue holds it: its own bundle, run rather than
// loaded, and its broker secret to receive the node's credential in.
func theRuntime(t *testing.T) Manifest {
t.Helper()
m := Manifest{Module: RuntimeModule, Version: "1",
OwnSecrets: OwnSecrets{"broker": {Path: "/var/lib/mesh/" + RuntimeModule + "/broker"}},
Build: &Build{Artifacts: []Artifact{{Name: "runtime", Kind: ArtifactBundle, Language: "typescript",
Entrypoints: []string{"src/main.js"}}}}}
resolved, err := m.Resolve([]Built{{Name: "runtime", Kind: ArtifactBundle,
Reference: ArtifactStoreScheme + RuntimeModule + "/runtime/blobs/" + bundleDigest, Digest: bundleDigest}})
if err != nil {
t.Fatal(err)
}
return resolved
}
func TestABuildsBundlesAreCarriedOnTheResolvedManifest(t *testing.T) {
m := aToolsModule(t, "nftables", "tools/index.js")
if len(m.Bundles) != 1 {
t.Fatalf("the resolved manifest carries %d bundle(s), not the one the build made", len(m.Bundles))
}
b := m.Bundles[0]
if b.Name != "tools" || b.Digest != bundleDigest || b.Language != "typescript" ||
b.Source != ArtifactStoreScheme+"nftables/tools/blobs/"+bundleDigest ||
len(b.Entrypoints) != 1 || b.Entrypoints[0] != "tools/index.js" {
t.Errorf("the bundle is carried as %+v", b)
}
// A repository manifest may not write what the build derives.
raw := `{"module":"x","version":"1","build":{"artifacts":[{"name":"t","kind":"bundle","language":"typescript"}]},` +
`"bundles":[{"name":"t","source":"s","digest":"` + bundleDigest + `"}]}`
if _, err := ParseManifest([]byte(raw)); err == nil || !strings.Contains(err.Error(), "bundles") {
t.Errorf("a manifest stating its build's output by hand was accepted: %v", err)
}
}
func TestEveryToolsBundleIsDeliveredWhereTheRuntimeRuns(t *testing.T) {
store := Rendering{ArtifactStore: "anchor.internal:5101",
Needed: map[string]map[string]string{RuntimeModule: {"broker": "sealed-credential"}}}
nftables := aToolsModule(t, "nftables", "tools/index.js")
zsh := aToolsModule(t, "zsh", "tools/index.js", "tools/more.js")
t.Run("with the runtime, one archive per tools bundle", func(t *testing.T) {
r := Resolution{Node: "anchor", Modules: []Manifest{nftables, zsh, theRuntime(t)}}
out, err := r.Declaration(store)
if err != nil {
t.Fatal(err)
}
archive := fileNamed(out, "nftables."+BundleID("tools"))
if archive == nil {
t.Fatalf("nftables' tools bundle was not delivered: %v", ids(out))
}
if archive["type"] != "archive" || archive["digest"] != bundleDigest ||
archive["path"] != BundleRoot+"/nftables/tools" {
t.Errorf("delivered as %v", archive)
}
if archive["source"] != "http://anchor.internal:5101/v2/nftables/tools/blobs/"+bundleDigest {
t.Errorf("fetched from %v, not through the store as this network reaches it", archive["source"])
}
if fileNamed(out, "zsh."+BundleID("tools")) == nil {
t.Errorf("zsh's tools bundle was not delivered: %v", ids(out))
}
// The runtime's own bundle is run, not loaded: its process delivers it, not an archive.
if fileNamed(out, RuntimeModule+"."+BundleID("runtime")) != nil {
t.Error("the runtime's own bundle was delivered as an archive beside its process")
}
})
t.Run("without the runtime, nothing changes", func(t *testing.T) {
r := Resolution{Node: "anchor", Modules: []Manifest{nftables, zsh}}
out, err := r.Declaration(store)
if err != nil {
t.Fatal(err)
}
for _, id := range ids(out) {
if strings.Contains(id, BundleID("")) {
t.Errorf("%s was delivered to a machine running no runtime to load it", id)
}
}
})
}
func ids(out []map[string]any) []string {
var names []string
for _, r := range out {
names = append(names, r["id"].(string))
}
return names
}
// One process per machine runs the runtime from its own bundle, told what it serves and from where,
// where its credential is, and who the operator is — restarted when any of that changes.
func TestTheMachineRunsOneRuntimeLoadingEveryDeliveredBundle(t *testing.T) {
with := Rendering{ArtifactStore: "anchor.internal:5101",
Needed: map[string]map[string]string{RuntimeModule: {"broker": "sealed-credential"}}}
nftables := aToolsModule(t, "nftables", "tools/index.js")
// A bundle carrying a daemon beside its tools says which files the runtime loads.
showcase := Manifest{Module: "showcase", Version: "1", Tools: []string{"greet"},
Build: &Build{Artifacts: []Artifact{{Name: "code", Kind: ArtifactBundle, Language: "typescript",
Entrypoints: []string{"daemon/index.js", "tools/index.js"}, Loads: []string{"tools/index.js"}}}}}
showcase, err := showcase.Resolve([]Built{{Name: "code", Kind: ArtifactBundle,
Reference: ArtifactStoreScheme + "showcase/code/blobs/" + bundleDigest, Digest: bundleDigest}})
if err != nil {
t.Fatal(err)
}
r := Resolution{Node: "anchor", Account: "ops", Modules: []Manifest{nftables, showcase, theRuntime(t)}}
out, err := r.Declaration(with)
if err != nil {
t.Fatal(err)
}
process := fileNamed(out, RuntimeModule+"."+RuntimeProcessID())
if process == nil {
t.Fatalf("no runtime process was composed: %v", ids(out))
}
if process["type"] != "process" || process["name"] != RuntimeModule || process["digest"] != bundleDigest ||
process["source"] != "http://anchor.internal:5101/v2/"+RuntimeModule+"/runtime/blobs/"+bundleDigest {
t.Errorf("the runtime's process is %v", process)
}
if fmt.Sprint(process["run"]) != "[node src/main.js]" {
t.Errorf("the runtime is run as %v; its bundle's one entrypoint, by its language's interpreter", process["run"])
}
env := process["env"].(map[string]string)
if env[RuntimeToolModules] != "nftables="+BundleRoot+"/nftables/tools/tools/index.js,"+
"showcase="+BundleRoot+"/showcase/code/tools/index.js" {
t.Errorf("the runtime is told to serve %q: every loaded file, by module, and nothing a bundle runs", env[RuntimeToolModules])
}
if env[RuntimeBrokerFile] != "/var/lib/mesh/"+RuntimeModule+"/broker" {
t.Errorf("the runtime reads its credential at %q, not where the module's own secret is placed", env[RuntimeBrokerFile])
}
if env[RuntimeOperatorAccount] != "ops" || env[RuntimeOperatorHome] != "/home/ops" || process["user"] != "ops" {
t.Errorf("the operator is not handed to the runtime: %v as %v", env, process["user"])
}
// The credential the process reads belongs to the account it runs as, or it could not read it
// (to-be 38 WP3); other modules' secrets are left as their manifests say.
if credential := fileNamed(out, RuntimeModule+"."+NeedID("broker")); credential == nil || credential["owner"] != "ops" {
t.Errorf("the runtime's credential is not the account's to read: %v", credential)
}
restarts := fmt.Sprint(process["restart-on"])
for _, want := range []string{"nftables." + BundleID("tools"), "showcase." + BundleID("code"), RuntimeModule + "." + NeedID("broker")} {
if !strings.Contains(restarts, want) {
t.Errorf("the runtime is not restarted when %s changes: %s", want, restarts)
}
}
// After every bundle and the credential, so both exist before it starts.
names := ids(out)
if names[len(names)-1] != RuntimeModule+"."+RuntimeProcessID() {
t.Errorf("the runtime's process is not last: %v", names)
}
t.Run("a machine with no account runs it as root without the operator words", func(t *testing.T) {
out, err := Resolution{Node: "anchor", Modules: []Manifest{nftables, theRuntime(t)}}.Declaration(with)
if err != nil {
t.Fatal(err)
}
process := fileNamed(out, RuntimeModule+"."+RuntimeProcessID())
env := process["env"].(map[string]string)
if _, set := env[RuntimeOperatorAccount]; set {
t.Error("an operator account was named on a machine that has none")
}
if _, set := process["user"]; set {
t.Error("a user was set on a machine with no account")
}
if credential := fileNamed(out, RuntimeModule+"."+NeedID("broker")); credential == nil || credential["owner"] != nil {
t.Errorf("the runtime's credential was given an owner on a machine with no account: %v", credential)
}
})
t.Run("a runtime module built wrong is refused by name", func(t *testing.T) {
two := Manifest{Module: RuntimeModule, Version: "1", OwnSecrets: OwnSecrets{"broker": {Path: "/b"}},
Build: &Build{Artifacts: []Artifact{{Name: "runtime", Kind: ArtifactBundle, Language: "typescript",
Entrypoints: []string{"a.js", "b.js"}}}}}
resolved, err := two.Resolve([]Built{{Name: "runtime", Kind: ArtifactBundle,
Reference: ArtifactStoreScheme + "x/runtime/blobs/" + bundleDigest, Digest: bundleDigest}})
if err != nil {
t.Fatal(err)
}
_, err = Resolution{Node: "anchor", Modules: []Manifest{resolved}}.Declaration(with)
if err == nil || !strings.Contains(err.Error(), "entrypoint") {
t.Errorf("a runtime bundle with two entrypoints was composed: %v", err)
}
})
}
// novox/hq ADR 0192: a tools bundle says what it is given; the composer resolves it per machine as
// a container's environment, hands it to the runtime as the module's words, and makes what the
// words name readable by the account the runtime runs as.
func TestABundleIsGivenItsWordsResolvedForThisMachine(t *testing.T) {
with := Rendering{ArtifactStore: "anchor.internal:5101",
Needed: map[string]map[string]string{
RuntimeModule: {"broker": "sealed-credential"},
"dash": {"token": "sealed-token"},
}}
dash := Manifest{Module: "dash", Version: "1", Tools: []string{"status"},
Listens: []Listening{{Name: "web", Port: 3000, Protocol: "tcp"}},
OwnSecrets: OwnSecrets{"token": {Path: "${dir:mesh-state}/token"}},
Resources: []map[string]any{
{"id": "mesh-state", "type": "directory", "mode": "0700", "place": "mesh"},
{"id": "config", "type": "file", "path": "${dir:mesh-state}/config.json", "mode": "0600", "content": "{}\n"},
{"id": "unrelated", "type": "file", "path": "/etc/dash.conf", "content": "x\n"},
},
Build: &Build{Artifacts: []Artifact{{Name: "tools", Kind: ArtifactBundle, Language: "typescript",
Entrypoints: []string{"tools/index.js"},
Env: map[string]string{
"DASH_CONFIG_FILE": "${dir:mesh-state}/config.json",
"DASH_TOKEN_FILE": "${dir:mesh-state}/token",
"DASH_URL": "http://127.0.0.1:${port:3000}",
"DASH_ADMIN": "mesh-admin",
}}}}}
if problems := dash.Build.problems(dash.Module); len(problems) > 0 {
t.Fatalf("a bundle's words written with ${dir:…} and ${port:…} were refused: %v", problems)
}
dash, err := dash.Resolve([]Built{{Name: "tools", Kind: ArtifactBundle,
Reference: ArtifactStoreScheme + "dash/tools/blobs/" + bundleDigest, Digest: bundleDigest}})
if err != nil {
t.Fatal(err)
}
other := aToolsModule(t, "nftables", "tools/index.js")
out, err := Resolution{Node: "anchor", Account: "ops", Modules: []Manifest{dash, other, theRuntime(t)}}.Declaration(with)
if err != nil {
t.Fatal(err)
}
dir := fileNamed(out, "dash.mesh-state")
if dir == nil {
t.Fatalf("no directory: %v", ids(out))
}
at := fmt.Sprint(dir["path"])
process := fileNamed(out, RuntimeModule+"."+RuntimeProcessID())
env := process["env"].(map[string]string)
var given map[string]map[string]string
if err := json.Unmarshal([]byte(env[RuntimeToolEnv]), &given); err != nil {
t.Fatalf("the runtime's %s is not JSON: %q", RuntimeToolEnv, env[RuntimeToolEnv])
}
want := map[string]string{
"DASH_CONFIG_FILE": at + "/config.json",
"DASH_TOKEN_FILE": at + "/token",
"DASH_URL": "http://127.0.0.1:3000",
"DASH_ADMIN": "mesh-admin",
}
if fmt.Sprint(given["dash"]) != fmt.Sprint(want) {
t.Errorf("dash is given %v, want %v", given["dash"], want)
}
if _, has := given["nftables"]; has {
t.Errorf("a module that declares no words was given some: %v", given)
}
// What the words name is the account's to read; nothing else of the module's is touched.
for _, id := range []string{"dash.mesh-state", "dash.config", "dash." + NeedID("token")} {
if r := fileNamed(out, id); r == nil || r["owner"] != "ops" {
t.Errorf("%s is not the account's to read: %v", id, r)
}
}
if r := fileNamed(out, "dash.unrelated"); r == nil || r["owner"] != nil {
t.Errorf("a file no word names was given an owner: %v", r)
}
// A file a word names restarts the runtime when it changes, as it restarted the tool container.
restarts := fmt.Sprint(process["restart-on"])
for _, want := range []string{"dash.config", "dash." + NeedID("token")} {
if !strings.Contains(restarts, want) {
t.Errorf("the runtime is not restarted when %s changes: %s", want, restarts)
}
}
if strings.Contains(restarts, "dash.unrelated") || strings.Contains(restarts, "dash.mesh-state") {
t.Errorf("the runtime restarts for something no word names as a file: %s", restarts)
}
t.Run("on a machine with no account the runtime is root and nothing is re-owned", func(t *testing.T) {
out, err := Resolution{Node: "anchor", Modules: []Manifest{dash, theRuntime(t)}}.Declaration(with)
if err != nil {
t.Fatal(err)
}
if r := fileNamed(out, "dash.config"); r["owner"] != nil {
t.Errorf("re-owned with no account: %v", r)
}
})
t.Run("a change to a module's words changes the runtime's process", func(t *testing.T) {
changed := dash
changed.Bundles = append([]Bundle(nil), dash.Bundles...)
changed.Bundles[0].Env = map[string]string{"DASH_ADMIN": "somebody-else"}
out2, err := Resolution{Node: "anchor", Account: "ops", Modules: []Manifest{changed, theRuntime(t)}}.Declaration(with)
if err != nil {
t.Fatal(err)
}
if fmt.Sprint(fileNamed(out2, RuntimeModule+"."+RuntimeProcessID())["env"]) == fmt.Sprint(env) {
t.Error("the runtime's process is the same after a module's words changed, so it would not restart")
}
})
}
func TestABundlesWordsAreRefusedWhenTheyAreNotPathsOrConstants(t *testing.T) {
m := Manifest{Module: "dash", Version: "1", Build: &Build{Artifacts: []Artifact{
{Name: "tools", Kind: ArtifactBundle, Language: "typescript", Entrypoints: []string{"tools/index.js"},
Env: map[string]string{"DASH_TOKEN": "${secret:token}", RuntimeBrokerFile: "/x", "DASH_PEER": "${bound:db:url}"}},
{Name: "runtime", Kind: ArtifactImage, From: "Dockerfile", Env: map[string]string{"X": "y"}},
}}}
said := strings.Join(m.Build.problems(m.Module), "\n")
for _, want := range []string{
`"tools" gives DASH_TOKEN the value "${secret:token}"`,
`"tools" gives DASH_PEER the value "${bound:db:url}"`,
`"tools" gives itself ` + RuntimeBrokerFile,
`"runtime" is a "image" and says what it is given`,
} {
if !strings.Contains(said, want) {
t.Errorf("not refused: %s\nsaid:\n%s", want, said)
}
}
}
+85 -3
View File
@@ -96,11 +96,60 @@ var defaultSeats = []Seat{
// A build says what it does as it does it (novox/hq ADR 0157): `started` when work is taken,
// `log.<build id>` for every line, `built` for the outcome. The log's tail token is the build's
// id, so a reader follows one build by subject alone.
// **Node-scoped, and every holder takes from one queue** (novox/hq ADR 0190): a build is asked of
// the role, and whichever machine holding the seat is idle pulls it. One holder per machine is
// what the scope says; sharing the work is what a seat's queue has always done.
{Name: "node-build-agent", Scope: ScopeNode,
Accepts: []string{"build"}, Emits: []string{"started", "built", "log.*"}, Decision: "novox/hq ADR 0190"},
// **Retired by ADR 0190, kept while a manifest still claims it.** The one build machine's seat.
// A claim to a seat the mesh no longer defines is refused, and the module holding this one is
// assigned on a live machine until build-agent replaces it — removing the row first would make
// that machine unresolvable in the meantime. Deleted once no registered manifest claims it.
{Name: "mesh-build-machine", Scope: ScopeMesh,
Accepts: []string{"build"}, Emits: []string{"started", "built", "log.*"}, Decision: "novox/hq ADR 0121"},
Accepts: []string{"build"}, Emits: []string{"started", "built", "log.*"}, Decision: "novox/hq ADR 0190"},
{Name: "node-dns-resolver", Scope: ScopeNode, Decision: "novox/hq ADR 0121"},
{Name: "node-intrusion-prevention", Scope: ScopeNode, Decision: "novox/hq ADR 0121"},
{Name: "node-packet-filter", Scope: ScopeNode, Decision: "novox/hq ADR 0121"},
// The intrusion prevention's verbs (novox/hq ADR 0179): what a person asks a machine's ban list
// whatever keeps it — who is banned and why, ban one address, let one go. Every holder serves all
// four; the jails themselves are composed from the modules the machine runs (to-be 31).
{Name: "node-intrusion-prevention", Scope: ScopeNode, Decision: "novox/hq ADR 0121",
Serves: []Verb{
{Name: "status", Description: "Every jail on this machine with how many it is watching and " +
"holding now, and the totals since the jail started; one jail's detail when named.",
Input: schema(map[string]string{"jail": "one jail (optional)"}, nil)},
{Name: "banned", Description: "Every address banned on this machine right now, with the jail " +
"that holds it and when the ban ends.",
Input: schema(map[string]string{"jail": "one jail (optional)"}, nil)},
{Name: "ban", Description: "Ban one address in one jail now, for the jail's ban time — an " +
"operator's act on the live ban list, which the mesh never writes itself.",
Input: schema(map[string]string{"ip": "the address", "jail": "the jail to hold it"}, []string{"ip", "jail"})},
{Name: "unban", Description: "Let one address go, from one jail or from every jail when none is named.",
Input: schema(map[string]string{"ip": "the address", "jail": "one jail (optional)"}, []string{"ip"})},
}},
// The packet filter's verbs (novox/hq ADR 0170): what a person asks a machine's filter whatever
// filter answers — the rules as enforced, reload the mesh's own, remove one thing the mesh did
// not write. Every holder serves all three; what differs by filter is the holder's own tools.
{Name: "node-packet-filter", Scope: ScopeNode, Decision: "novox/hq ADR 0121",
Serves: []Verb{
{Name: "rules", Description: "The packet filter as this machine enforces it now: the nftables " +
"ruleset and, where the tool exists, the legacy filter's listings. Narrowed to one table or " +
"chain when asked.",
Input: schema(map[string]string{"table": "one nftables table, as `family name` (optional)",
"chain": "one chain of that table (optional)"}, nil)},
{Name: "reload", Description: "Load the mesh's own filter again from the file the mesh writes, " +
"and answer with the mesh's table as loaded.",
Input: schema(map[string]string{}, nil)},
{Name: "remove", Description: "Remove one rule set the mesh did not write, named exactly as the " +
"host reports it (novox/hq ADR 0168) — `chain X (iptables-legacy)` or `table ip6 filter, chain " +
"DOCKER-USER`. Refuses the mesh's tables, the runtime's own chains, a built-in chain and an " +
"active found firewall's chains. An operator's act, by name, never a flush.",
Input: schema(map[string]string{"where": "the rule set, as `node show` lists it"}, []string{"where"})},
}},
// The machine's service manager (novox/hq ADR 0177). The host applies every declared unit,
// system or user scope; the holder answers questions and operator acts about them, each verb
// taking the unit and an optional scope. The holder runs nothing of its own: its verbs are
// served by the node tools runtime (ADR 0175).
{Name: "node-service-manager", Scope: ScopeNode, Decision: "novox/hq ADR 0177",
Serves: serviceManagerVerbs()},
// Deferred (novox/hq ADR 0121): renaming to mesh-private-network is a scope + server/client
// model change, not a rename, so it stays until that is built.
{Name: "the-private-network", Scope: ScopeNode, Decision: "novox/hq ADR 0110"},
@@ -374,3 +423,36 @@ func SeatsWithAProtocol() []Seat {
}
return out
}
// serviceManagerVerbs is the contract every holder of node-service-manager serves (novox/hq ADR
// 0177): the units on the machine in both scopes, read and acted on by name. Every verb takes an
// optional scope — "system" when absent, "user" for the operator account's own manager — so a
// caller asks for a user unit the way it asks for a system one.
func serviceManagerVerbs() []Verb {
scoped := func(more map[string]string, required []string) map[string]any {
props := map[string]string{"scope": "\"system\" (the default) or \"user\": the operator account's own manager"}
for k, v := range more {
props[k] = v
}
return schema(props, required)
}
unit := map[string]string{"unit": "the unit's name, as the service manager knows it"}
return []Verb{
{Name: "units", Description: "The units the service manager knows in a scope, each with its load, active and sub state; narrowed to a pattern when asked.",
Input: scoped(map[string]string{"pattern": "a glob the unit's name must match (optional)"}, nil)},
{Name: "status", Description: "One unit as the service manager sees it now: its states, whether it starts at boot, its main process, and whether the mesh declares it.",
Input: scoped(unit, []string{"unit"})},
{Name: "start", Description: "Start one unit. For a unit the mesh declares, the answer says the host will restore what its declaration says at the next apply.",
Input: scoped(unit, []string{"unit"})},
{Name: "stop", Description: "Stop one unit; for a mesh-declared unit the answer says the host will restore its declared state.",
Input: scoped(unit, []string{"unit"})},
{Name: "restart", Description: "Restart one unit.",
Input: scoped(unit, []string{"unit"})},
{Name: "enable", Description: "Make one unit start at boot (or at the account's login, in user scope).",
Input: scoped(unit, []string{"unit"})},
{Name: "disable", Description: "Stop one unit starting at boot (or at login, in user scope).",
Input: scoped(unit, []string{"unit"})},
{Name: "journal", Description: "The last lines of one unit's journal.",
Input: scoped(map[string]string{"unit": unit["unit"], "lines": "how many lines from the end (default 100)"}, []string{"unit"})},
}
}
+4 -2
View File
@@ -44,8 +44,10 @@ func TestTheSeatsAreAClosedSetAndEachNamesItsDecision(t *testing.T) {
delivered[s.Delivers] = s.Name
}
}
if len(Seats()) != 15 {
t.Errorf("the mesh defines %d seats rather than 14; the set is closed, so a change here is "+
// Seventeen since node-build-agent (novox/hq ADR 0190) — sixteen once the retired
// mesh-build-machine row goes, when no registered manifest claims it any more.
if len(Seats()) != 17 {
t.Errorf("the mesh defines %d seats rather than 17; the set is closed, so a change here is "+
"a decision (novox/hq ADR 0110): %s", len(Seats()), seatNames())
}
}
+52
View File
@@ -0,0 +1,52 @@
package catalogue
import (
"reflect"
"strings"
"testing"
)
// What reads one of a module's own secrets is restarted when the secret changes (novox/hq issue 203,
// issue 206): the build machine kept an hour-old credential open because its manifest restarted it
// on its environment file alone. Composed, so a manifest need not say it; a scheduled process is
// left alone, because the host refuses a restart-on for one and it reads the file afresh each run.
func TestAContainerReadingAnOwnSecretIsRestartedWhenItChanges(t *testing.T) {
m := Manifest{Module: "agent", Version: "1",
OwnSecrets: OwnSecrets{"broker": {Path: "/var/lib/mesh/agent/broker"}},
Resources: []map[string]any{
{"id": "mesh-state", "type": "directory", "path": "/var/lib/mesh/agent", "mode": "0700"},
{"id": "settings", "type": "file", "path": "/var/lib/mesh/agent/agent.env", "mode": "0600", "content": "A=1\n"},
{"id": "server", "type": "container", "name": "agent", "network": "host",
"image": "registry.example/agent@sha256:" + strings.Repeat("a", 64),
"volumes": []any{"/var/lib/mesh/agent:/run/mesh:ro", "/var/lib/mesh/agent/broker:/run/mesh/broker:ro"},
"env-file": []any{"/var/lib/mesh/agent/agent.env"},
"restart-on": []any{"settings"}},
{"id": "nightly", "type": "container", "name": "agent-nightly", "schedule": "0 3 * * *",
"image": "registry.example/agent@sha256:" + strings.Repeat("a", 64),
"volumes": []any{"/var/lib/mesh/agent/broker:/run/mesh/broker:ro"}},
{"id": "other", "type": "container", "name": "agent-other",
"image": "registry.example/agent@sha256:" + strings.Repeat("a", 64)},
}}
got, err := Resolve(shelf(m), []string{m.Module},
Node{Name: "anchor", At: "10.0.0.1", Capabilities: map[string]bool{"container-runtime": true}}, World{})
if err != nil {
t.Fatal(err)
}
out, err := got.Declaration(Rendering{Needed: map[string]map[string]string{"agent": {"broker": "SEALED"}}})
if err != nil {
t.Fatal(err)
}
by := map[string]map[string]any{}
for _, r := range out {
by[r["id"].(string)] = r
}
if want := []any{"agent.settings", "agent.needs-broker"}; !reflect.DeepEqual(by["agent.server"]["restart-on"], want) {
t.Fatalf("the server reads the credential and is not restarted on it: %v", by["agent.server"]["restart-on"])
}
if _, has := by["agent.nightly"]["restart-on"]; has {
t.Fatalf("a scheduled container was given a restart-on, which the host refuses: %v", by["agent.nightly"]["restart-on"])
}
if _, has := by["agent.other"]["restart-on"]; has {
t.Fatalf("a container that reads no secret was given one to restart on: %v", by["agent.other"]["restart-on"])
}
}
+128
View File
@@ -3,6 +3,7 @@ package catalogue
import (
"encoding/json"
"fmt"
"regexp"
"sort"
"strings"
)
@@ -275,6 +276,11 @@ func UnusedSettings(m Manifest, layers []Layer) []string {
if key == AccessesSetting && len(m.Accesses) > 0 {
continue
}
// `networks` keeps a found network for a taken container on one adopted machine
// (novox/hq ADR 0163). Validated in KeptNetworks, so not stray.
if key == NetworksSetting {
continue
}
unused = append(unused, fmt.Sprintf(
"%s sets %q, and %s has no file that merges it, asks for no ${setting:%s}, and "+
"declares no %q in what it contributes or serves",
@@ -298,3 +304,125 @@ func stringsOf(v any) []string {
}
return out
}
// NetworksSetting is the settings key that keeps a found network for a taken container, on one
// adopted machine (novox/hq ADR 0163, rule 4):
//
// {"networks": {"server": ["predecessor_default"]}}
//
// has the module's container `server` also join `predecessor_default` once taken, so a neighbour
// that resolves it by name on that network keeps resolving it. Migration scaffolding in the sense
// of ADR 0104: assigned only on an adopted machine, reported while it stands, removed when the
// neighbours are taken. Keyed by the container's resource id; the value is the networks it keeps.
const NetworksSetting = "networks"
// KeptNetworks reads which found networks each of a module's containers keeps, by container id.
//
// Refused from a mesh-wide layer — a found network is a fact about one machine — for an id the
// module declares no container under, for a name that is not a network's, and on a machine that
// is not adopted: the setting exists so neighbours the mesh has not taken yet keep reaching the
// container, and a converged machine has no such neighbours.
func KeptNetworks(m Manifest, layers []Layer, adopted bool) (map[string][]string, error) {
containers := map[string]bool{}
for _, r := range m.Resources {
if fmt.Sprint(r["type"]) == "container" {
containers[fmt.Sprint(r["id"])] = true
}
}
out := map[string][]string{}
for _, layer := range layers {
raw, ok := layer.Values[NetworksSetting]
if !ok {
continue
}
if layer.From == MeshWideLayer {
return nil, fmt.Errorf("%s: %s is given per node — a found network is a fact about one "+
"machine; set it with --node", m.Module, NetworksSetting)
}
if !adopted {
return nil, fmt.Errorf("%s: %s keeps a found network for neighbours the mesh has not taken "+
"yet, and %s is converged — nothing on it is found; clear the setting", m.Module,
NetworksSetting, layer.From)
}
blocks, ok := raw.(map[string]any)
if !ok {
return nil, fmt.Errorf("%s: %s is a { container: [network, …] } map, and %q set it to "+
"something else", m.Module, NetworksSetting, layer.From)
}
for id, body := range blocks {
if !containers[id] {
return nil, fmt.Errorf("%s: %s names the container %q, which it does not declare — "+
"the setting reaches nothing; it declares %s", m.Module, NetworksSetting, id,
orNothing(sortedKeys(containers)))
}
names := stringsOf(body)
if len(names) == 0 {
return nil, fmt.Errorf("%s: %s for %q is a list of network names, and %q set it to %v",
m.Module, NetworksSetting, id, layer.From, body)
}
for _, n := range names {
if !networkName.MatchString(n) {
return nil, fmt.Errorf("%s: %s for %q names %q, which is not a network name",
m.Module, NetworksSetting, id, n)
}
}
sort.Strings(names)
out[id] = names
}
}
if len(out) == 0 {
return nil, nil
}
return out, nil
}
// networkName is what a container runtime accepts as a network's name.
var networkName = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9_.-]*$`)
// JudgeSettings composes a module's settings against its definition and refuses the first thing
// that cannot work, naming the module, the layer and the key (novox/hq ADR 0163, rule 6).
//
// **The same judgement where a setting is stored and where a machine is declared.** Stored, a
// setting that cannot compose is refused before it is kept; composed later, a definition that has
// moved under a stored setting leaves that module out of the machine's declaration rather than
// the machine without one. Every reader of settings runs here: a port given, an exposure, a reach,
// an endpoint, a placement, an access, a kept network, a mergeable file's keys and a file's
// `${setting:…}`. A key that reaches nothing is not here: it cannot break a composition, so it is
// refused where it is stored (SetSettings, with UnusedSettings) and said where a plan is read,
// and never costs a module its place.
func JudgeSettings(m Manifest, layers []Layer, adopted bool) error {
// With no layers too: a definition may ask for a setting nobody made — an access placed by
// nobody, a file's ${setting:…} nothing sets — and that is the same statement, missing.
if _, err := GivenPorts(m, layers); err != nil {
return err
}
if _, err := Reaches(m, layers); err != nil {
return err
}
if _, err := Endpoints(m, layers); err != nil {
return err
}
if _, err := Places(m, layers); err != nil {
return err
}
if _, _, err := accessesFor(m, layers); err != nil {
return err
}
if _, err := KeptNetworks(m, layers, adopted); err != nil {
return err
}
for _, r := range m.Resources {
settled, err := ApplySettings(r, layers)
if err != nil {
return err
}
copied := map[string]any{}
for k, v := range settled {
copied[k] = v
}
if err := settingInto(copied, layers, m.Module); err != nil {
return err
}
}
return nil
}
+127
View File
@@ -0,0 +1,127 @@
package catalogue
import (
"strings"
"testing"
)
// A setting is judged where it is stored, and an impossible one costs a module, not a machine
// (novox/hq ADR 0163, rule 6): the one judgement, used by SetSettings before storing and by
// Compose when a definition has moved under a stored setting.
func TestASettingThatCannotComposeIsRefusedByNameAndLeavesOnlyItsModuleOut(t *testing.T) {
web := Manifest{Module: "hello-web",
Listens: []Listening{{Port: 8080, From: FromEverywhere}},
Resources: []map[string]any{{"id": "server", "type": "container", "name": "hello-web",
"ports": []any{"8080"}}}}
for _, c := range []struct {
name string
layer map[string]any
refuse string
}{
{"a port the module does not publish", map[string]any{PortsSetting: map[string]any{"9999": 10000}},
"hello-web gives port 9999 a machine port, and no container of its publishes 9999"},
{"a mesh-wide port", map[string]any{PortsSetting: map[string]any{"8080": 10000}},
"a port is a fact about one machine"},
} {
from := "anchor"
if c.name == "a mesh-wide port" {
from = MeshWideLayer
}
err := JudgeSettings(web, []Layer{{From: from, Values: c.layer}}, true)
if err == nil || !strings.Contains(err.Error(), c.refuse) {
t.Errorf("%s: judged %v, want %q", c.name, err, c.refuse)
}
}
if err := JudgeSettings(web, []Layer{{From: "anchor", Values: map[string]any{PortsSetting: map[string]any{"8080": 10000}}}}, true); err != nil {
t.Fatalf("a port the module publishes was refused: %v", err)
}
// Composed, a module whose stored setting no longer works is left out by name; the rest of
// the machine is declared.
r := anAdoptedAnchor()
with := anchorRendering(false)
with.Settings = SettingsBy{"hello-web": {{From: "anchor", Values: map[string]any{PortsSetting: map[string]any{"9999": 10000}}}}}
composed, err := r.Compose(with)
if err != nil {
t.Fatal(err)
}
why, left := composed.LeftOut["hello-web"]
if !left || !strings.Contains(why, "no container of its publishes 9999") {
t.Fatalf("hello-web is not left out by name: %v", composed.LeftOut)
}
if len(composed.LeftOut) != 1 {
t.Fatalf("more than hello-web is left out: %v", composed.LeftOut)
}
got := byID(composed.Resources)
if _, declared := got["hello-web.server"]; declared {
t.Fatal("the left-out module's container is still declared")
}
if _, declared := got["distribution.store"]; !declared {
t.Fatal("the rest of the machine was not declared")
}
if left := r.LeftOut(with.Settings, false); len(left) != 1 || left["hello-web"] == "" {
t.Fatalf("the judgement a plan reads differs from what compose did: %v", left)
}
}
// A taken container keeps a found network by a per-machine setting (novox/hq ADR 0163, rule 4):
// on an adopted machine only, for a container the module declares, and it reaches the container's
// declaration as the networks it also joins.
func TestAKeptNetworkReachesTheContainerOnAnAdoptedMachineOnly(t *testing.T) {
r := anAdoptedAnchor()
keep := SettingsBy{"hello-web": {{From: "anchor",
Values: map[string]any{NetworksSetting: map[string]any{"server": []any{"predecessor_default"}}}}}}
with := anchorRendering(true)
with.Settings = keep
composed, err := r.Compose(with)
if err != nil {
t.Fatal(err)
}
if len(composed.LeftOut) != 0 {
t.Fatalf("a kept network left a module out: %v", composed.LeftOut)
}
server := byID(composed.Resources)["hello-web.server"]
networks, _ := server["networks"].([]any)
if len(networks) != 1 || networks[0] != "predecessor_default" {
t.Fatalf("the container does not join the kept network: %v", server)
}
if _, has := byID(composed.Resources)["distribution.store"]["networks"]; has {
t.Fatal("another container joins a network nobody kept for it")
}
// Converged, the setting reaches nothing it was for, and the module is left out saying so.
with = anchorRendering(false)
with.Settings = keep
composed, err = r.Compose(with)
if err != nil {
t.Fatal(err)
}
if why := composed.LeftOut["hello-web"]; !strings.Contains(why, "anchor is converged") {
t.Fatalf("a kept network on a converged machine: %v", composed.LeftOut)
}
web := r.Modules[4]
for _, c := range []struct {
name string
layer Layer
want string
}{
{"mesh-wide", Layer{From: MeshWideLayer, Values: map[string]any{NetworksSetting: map[string]any{"server": []any{"x"}}}},
"a found network is a fact about one machine"},
{"an unknown container", Layer{From: "anchor", Values: map[string]any{NetworksSetting: map[string]any{"db": []any{"x"}}}},
`names the container "db", which it does not declare`},
{"not a list", Layer{From: "anchor", Values: map[string]any{NetworksSetting: map[string]any{"server": "x"}}},
"is a list of network names"},
{"not a network name", Layer{From: "anchor", Values: map[string]any{NetworksSetting: map[string]any{"server": []any{"a/b"}}}},
"which is not a network name"},
} {
_, err := KeptNetworks(web, []Layer{c.layer}, true)
if err == nil || !strings.Contains(err.Error(), c.want) {
t.Errorf("%s: %v, want %q", c.name, err, c.want)
}
}
if kept, err := KeptNetworks(web, nil, false); err != nil || kept != nil {
t.Fatalf("no setting: %v %v", kept, err)
}
}
@@ -0,0 +1,23 @@
package catalogue
import (
"regexp"
"testing"
)
// The bus is never public (novox/hq ADR 0169). Its port is what the bus module declares, the mesh,
// and the control plane adds no opening of its own: a machine joins through the tunnel, so the
// broker's host is filtered like any other. Before this, the broker port was a foundation port and
// rendered from anywhere beside its from-the-mesh rule.
func TestTheBusPortIsReachedFromTheMeshAlone(t *testing.T) {
rules := []Rule{{Port: 4222, Protocol: "tcp", From: FromMesh, Because: []string{"nats"},
Why: []string{"the mesh bus"}}}
out := AsNftables(rules, []string{"10.10.0.1", "10.10.0.2"}, true, nil, []string{"eth0"}, "mesh0")
if !regexp.MustCompile(`ip saddr \{ 10\.10\.0\.1, 10\.10\.0\.2 \} tcp dport 4222 accept`).MatchString(out) {
t.Fatalf("the bus is not reachable from the mesh:\n%s", out)
}
if regexp.MustCompile(`(?m)^\s*tcp dport 4222 accept`).MatchString(out) {
t.Fatalf("the bus is reachable from anywhere:\n%s", out)
}
}
+23
View File
@@ -129,6 +129,29 @@ var ControllerVerbs = []Verb{
"module": "an own secret: the module",
"secret": "an own secret: its name in the module's definition",
}, nil)},
{Name: "issue", Description: "Give a module on a machine its account on the bus: minted, and sealed to the " +
"machine as the module's own secret named broker, read at the next push of that machine. For a module " +
"whose definition declares that secret; refused with the reason otherwise. Issued again, it replaces the account.",
Input: schema(map[string]string{
"node": "the machine that runs the module",
"module": "the module's name",
}, []string{"node", "module"})},
{Name: "settings", Description: "Set what an assignment is configured with: a module's settings for the whole mesh, " +
"or for one machine. Replaces that layer whole — what it does not name, it no longer sets — and takes effect " +
"at the next push. With clear, removes the layer and the module is back to what its definition says.",
Input: schema(map[string]string{
"module": "the module's name",
"values": "the settings as a JSON object, for set",
"node": "one machine; the whole mesh when absent",
"clear": "\"true\" to remove the layer instead of setting it",
}, []string{"module"})},
{Name: "command", Description: "Run one command line of the controller's own, as you would type it at its " +
"shell — `node account g14 jochen`, `node show ace`, `module list` — and answer what it printed. The " +
"generic verb beside the named ones (novox/hq ADR 0154): everything the binary can do, without a verb " +
"per command. Any node may call any tool (ADR 0175), so nothing is held back here.",
Input: schema(map[string]string{
"command": "the command line, as the controller's binary takes it; quotes group a word with spaces",
}, []string{"command"})},
{Name: "build", Description: "Have the build machine build a repository. Answers at once with the build's id: " +
"`builds` with that id follows it line by line, and the module is registered when the outcome comes.",
Input: schema(map[string]string{
+97
View File
@@ -178,6 +178,103 @@ type Stray struct {
Detail string `json:"detail,omitempty"`
}
// A Filter is one place on a machine that refuses traffic, with its owner (novox/hq ADR 0168).
type Filter struct {
Where string `json:"where"`
Owner string `json:"owner"`
Refuses string `json:"refuses"`
}
// Owners of a filter, as the host names them (ADR 0168).
const (
FilterMesh = "mesh"
FilterFoundFirewall = "found-firewall"
FilterRuntime = "runtime"
FilterBan = "ban"
FilterOther = "other"
)
// FoundFirewall is the state of a converged machine's found firewall (ADR 0168): in force now or
// not, and how it came to be inactive.
type FoundFirewall struct {
Kind string `json:"kind"`
Active bool `json:"active"`
RetiredBy string `json:"retired_by,omitempty"`
}
// Filtering is what a machine last said filters it (ADR 0168).
type Filtering struct {
Filters []Filter
FoundFirewall *FoundFirewall
}
// Alone is whether the machine is filtered by the mesh alone: nothing in its list but the mesh's
// own, the runtime's plumbing and bans, and no found firewall in force.
func (f Filtering) Alone() bool {
for _, x := range f.Filters {
if x.Owner == FilterOther || x.Owner == FilterFoundFirewall {
return false
}
}
return f.FoundFirewall == nil || !f.FoundFirewall.Active
}
// Others is every filter that is neither the mesh's, the runtime's nor a ban.
func (f Filtering) Others() []Filter {
var out []Filter
for _, x := range f.Filters {
if x.Owner == FilterOther || x.Owner == FilterFoundFirewall {
out = append(out, x)
}
}
return out
}
// RecordFiltering keeps what a machine last said filters it, replacing what was there (ADR 0168).
func (i *Inventory) RecordFiltering(ctx context.Context, nodeID string, filters []Filter, found *FoundFirewall) error {
raw, err := json.Marshal(nonNil(filters))
if err != nil {
return err
}
var foundRaw any
if found != nil {
b, err := json.Marshal(found)
if err != nil {
return err
}
foundRaw = string(b)
}
_, err = i.store.Pool().Exec(ctx,
`update node set filters = $2, found_firewall = $3 where id = $1`, nodeID, raw, foundRaw)
return err
}
// FilteringOf is what a machine last said filters it; empty for a machine that never said.
func (i *Inventory) FilteringOf(ctx context.Context, name string) (Filtering, error) {
var filtersRaw, foundRaw []byte
err := i.store.Pool().QueryRow(ctx,
`select filters, found_firewall from node where name = $1`, name).Scan(&filtersRaw, &foundRaw)
if errors.Is(err, pgx.ErrNoRows) {
return Filtering{}, fmt.Errorf("%w: %s", ErrNoSuchNode, name)
}
if err != nil {
return Filtering{}, err
}
var out Filtering
if len(filtersRaw) > 0 {
if err := json.Unmarshal(filtersRaw, &out.Filters); err != nil {
return Filtering{}, err
}
}
if len(foundRaw) > 0 {
out.FoundFirewall = &FoundFirewall{}
if err := json.Unmarshal(foundRaw, out.FoundFirewall); err != nil {
return Filtering{}, err
}
}
return out, nil
}
// Reach is one thing reachable on an adopted node: a listening socket or a published port.
type Reach struct {
Protocol string `json:"protocol"`
+3
View File
@@ -42,6 +42,9 @@ const (
BusModule = "module"
BusEnrolment = "enrolment"
BusPerson = "person"
// BusNodeTools is a machine's tool runtime (novox/hq ADR 0175): named like the module it
// stands for, recorded as what it is.
BusNodeTools = "node-tools"
)
// MintBusPassword makes a bus password and records its hash under a username, replacing whatever was
+111
View File
@@ -42,6 +42,11 @@ type Source struct {
// Seen is when the source was last looked at — by a build, by hand, or by the forge saying it
// moved. What a late report of an older move is judged against.
Seen time.Time
// Against is every artifact the build this manifest came from stood on, as recorded. Part of a
// module's provenance like the commit is, and what tells a built manifest's base when the manifest
// itself no longer carries its build (novox/hq to-be 38 WP2.4). Empty for a manifest handed over
// by hand, which carries its `build.on` itself.
Against []string
}
// Current reports whether what the mesh holds is what the source last had.
@@ -61,6 +66,32 @@ func (s Source) Current() bool {
// gains a requirement, a claim, a resource. What matters is that the change is visible the next
// time a node is resolved, which it is.
func (i *Inventory) RegisterModule(ctx context.Context, m catalogue.Manifest, from Source) error {
// **Once the node's tool runtime is in the catalogue, the pattern it retires may not spread**
// (novox/hq ADR 0175, to-be 38 WP2.4): a module serving its tools from a container built on the
// runtime's image. Refused at registration, by name, for a module that is new to the catalogue
// or that was registered in another shape — the mechanism that keeps the old pattern from
// returning by habit. **Not refused for a module already registered in that shape**: the
// catalogue holds some thirty of them the day the runtime arrives, each moves to a bundle in
// its own change (to-be 38 WP4 onward), and a gate that refused every rebuild of every unmoved
// module in the meantime would stop the whole pipeline to make a point the record already makes.
// Before the runtime exists the pattern is accepted as it always was.
if m.Module != catalogue.RuntimeModule {
if why := catalogue.ToolContainerOnTheRuntime(m, from.Against); why != "" {
runtime, err := i.hasModule(ctx, catalogue.RuntimeModule)
if err != nil {
return err
}
if runtime {
already, err := i.registeredInThatShape(ctx, m.Module)
if err != nil {
return err
}
if !already {
return fmt.Errorf("%s is not registered: %s", m.Module, why)
}
}
}
}
raw, err := json.Marshal(m)
if err != nil {
return err
@@ -89,6 +120,36 @@ func (i *Inventory) RegisterModule(ctx context.Context, m catalogue.Manifest, fr
return err
}
// registeredInThatShape is whether the catalogue already holds this module as a tools container on
// the runtime's image — judged from the manifest it holds and what that module's newest build stood
// on, the same two things the gate judges a new registration by. False for a module the catalogue
// does not hold.
func (i *Inventory) registeredInThatShape(ctx context.Context, name string) (bool, error) {
held, err := i.Catalogue(ctx)
if err != nil {
return false, err
}
stored, has := held[name]
if !has {
return false, nil
}
against, err := i.BuiltAgainst(ctx)
if err != nil {
return false, err
}
return catalogue.ToolContainerOnTheRuntime(stored, against[name]) != "", nil
}
// hasModule is whether the catalogue holds a module of that name.
func (i *Inventory) hasModule(ctx context.Context, name string) (bool, error) {
var one int
err := i.store.Pool().QueryRow(ctx, `select 1 from module where name = $1`, name).Scan(&one)
if errors.Is(err, pgx.ErrNoRows) {
return false, nil
}
return err == nil, err
}
// SourceMoved records that a module's source has a newer commit than the mesh has built.
//
// This is the whole of noticing. Nothing here builds anything — it writes down that the two
@@ -605,6 +666,12 @@ func (i *Inventory) SetSettings(ctx context.Context, nodeName, module string, va
if err != nil {
return err
}
// Judged here, against the module's current definition, before it is kept (novox/hq ADR 0163,
// rule 6): a setting that cannot compose is refused where it is set, naming the node, the
// module, the layer and the key — never stored to refuse the whole machine where it is read.
if err := i.judgeSettings(ctx, nodeName, module, values); err != nil {
return err
}
if nodeName == "" {
// A port is a fact about one machine (novox/hq ADR 0100). Refused here, in composition's
// words: stored, it refuses every node running the module at composition, and the mesh
@@ -661,6 +728,50 @@ func (i *Inventory) SetSettings(ctx context.Context, nodeName, module string, va
return tx.Commit(ctx)
}
// judgeSettings composes a layer somebody is about to store against the module's definition, with
// the mesh-wide layer under it when the layer is one node's, and refuses the first thing that
// cannot work (ADR 0163, rule 6). The same judgement composition makes; what passes here composes.
func (i *Inventory) judgeSettings(ctx context.Context, nodeName, module string, values map[string]any) error {
m, err := i.declared(ctx, module)
if err != nil {
return fmt.Errorf("%w: %s", ErrNoSuchModule, module)
}
where, from := "the mesh", catalogue.MeshWideLayer
adopted := false
var layers []catalogue.Layer
if nodeName != "" {
node, err := i.NodeByName(ctx, nodeName)
if err != nil {
return err
}
where, from, adopted = nodeName, nodeName, node.Adopted
var meshWide []byte
err = i.store.Pool().QueryRow(ctx,
`select values from settings where module = $1 and node is null`, module).Scan(&meshWide)
if err != nil && !errors.Is(err, pgx.ErrNoRows) {
return err
}
if len(meshWide) > 0 {
var under map[string]any
if err := json.Unmarshal(meshWide, &under); err != nil {
return err
}
layers = append(layers, catalogue.Layer{From: catalogue.MeshWideLayer, Values: under})
}
}
layers = append(layers, catalogue.Layer{From: from, Values: values})
if err := catalogue.JudgeSettings(m, layers, adopted); err != nil {
return fmt.Errorf("refused: %s on %s cannot compose with the layer %q — %w", module, where, from, err)
}
// And a key that reaches nothing, refused here where somebody can still fix the spelling:
// stored, it would be a setting somebody believes they made.
if stray := catalogue.UnusedSettings(m, layers[len(layers)-1:]); len(stray) > 0 {
return fmt.Errorf("refused: %s on %s — these settings reach nothing:\n - %s", module, where,
strings.Join(stray, "\n - "))
}
return nil
}
// givenIn is the machine ports a node-level settings layer gives a module, software port →
// machine port (novox/hq ADR 0100). Nothing when the layer gives none; what is not a port is left
// for composition to refuse in its own words.
+58
View File
@@ -685,3 +685,61 @@ func TestRegisteringWithoutProvenanceKeepsTheSeat(t *testing.T) {
t.Fatalf("a hand-registered manifest erased where the module comes from: %+v", got)
}
}
// Once the node's tool runtime is in the catalogue, a module serving its tools from a container
// built on the runtime's image is refused at registration, naming the record (novox/hq ADR 0175,
// to-be 38 WP2.4) — for a module new to the catalogue or one that had moved away from it; a module
// already standing in that shape is rebuilt as before, so the catalogue's pipeline keeps running
// while each moves (WP3's amendment). Before the runtime, it is accepted as it always was — so a
// mesh converts in the order the design says and nothing is refused before there is anything to
// move to.
func TestAToolContainerIsRefusedOnceTheRuntimeIsRegistered(t *testing.T) {
inv := fresh(t)
ctx := t.Context()
filter := catalogue.Manifest{Module: "nftables", Version: "1", Tools: []string{"firewall_rules"},
Resources: []map[string]any{{"id": "runtime", "type": "container", "name": "mesh-nftables"}}}
stoodOn := []string{catalogue.ArtifactStoreScheme + "mesh-tools/runtime@sha256:" + strings.Repeat("d", 64)}
// Before the runtime exists the old pattern is accepted as it always was — and built, which is
// how the catalogue comes to know what the module stood on.
if err := inv.RegisterModule(ctx, filter, Source{Repository: "/r", Against: stoodOn}); err != nil {
t.Fatalf("before the runtime exists the old pattern is accepted: %v", err)
}
built := aBuild("nf1", "nftables", "")
built.Against = stoodOn
if err := inv.RecordBuild(ctx, built); err != nil {
t.Fatal(err)
}
runtime := catalogue.Manifest{Module: catalogue.RuntimeModule, Version: "1"}
if err := inv.RegisterModule(ctx, runtime, Source{Repository: "/r"}); err != nil {
t.Fatal(err)
}
// **A module already registered in that shape is rebuilt without complaint** (to-be 38 WP2.4 as
// amended by WP3): some thirty of them stand the day the runtime arrives, and each moves in its
// own change. The gate is against the pattern spreading, not against the pipeline running.
if err := inv.RegisterModule(ctx, filter, Source{Repository: "/r", Against: stoodOn}); err != nil {
t.Fatalf("a rebuild of a module that already had the pattern was refused: %v", err)
}
// A module new to the catalogue in that shape is refused, naming the record.
newcomer := filter
newcomer.Module = "lamp"
err := inv.RegisterModule(ctx, newcomer, Source{Repository: "/r", Against: stoodOn})
if err == nil || !strings.Contains(err.Error(), "ADR 0175") {
t.Fatalf("a new module in the old pattern was registered beside the runtime: %v", err)
}
// And a module that had moved its tools to a bundle may not come back to a container.
moved := filter
moved.Resources = nil
if err := inv.RegisterModule(ctx, moved, Source{Repository: "/r", Against: stoodOn}); err != nil {
t.Fatalf("a module whose tools are a bundle was refused: %v", err)
}
unbuilt := aBuild("nf2", "nftables", "")
if err := inv.RecordBuild(ctx, unbuilt); err != nil {
t.Fatal(err)
}
err = inv.RegisterModule(ctx, filter, Source{Repository: "/r", Against: stoodOn})
if err == nil || !strings.Contains(err.Error(), "ADR 0175") {
t.Fatalf("a module that had moved returned to the old pattern unrefused: %v", err)
}
}
+17 -1
View File
@@ -18,8 +18,17 @@ const (
EdgeBuiltBy = "built-by"
// EdgeDeclared: the manifest's own `build.on`.
EdgeDeclared = "declared"
// EdgeWorkerOf: the module holds the build seat, whose worker the control plane defines
// (novox/hq issue 206). The one place a *running* order enters the graph: a build machine rolled
// before the controller that redefines its worker cannot bind it, and nothing can then build the
// controller that would end that — so the holder of the build seat follows the controller, and
// the controller is built by whichever build machine is running, as it always was.
EdgeWorkerOf = "worker-of"
)
// TheControlPlane is the module that defines every seat's worker on the bus.
const TheControlPlane = "mesh-controller"
// Edge is one dependency: From depends on To, in the way Kind says.
type Edge struct {
From string `json:"from"`
@@ -63,7 +72,7 @@ func dependenciesOf(entries []Entry, against map[string][]string, read map[strin
if r := repositoryKey(e.Source.Repository); r != "" {
byRepository[r] = append(byRepository[r], name)
}
if e.Manifest.ClaimsSeat("mesh-build-machine") {
if e.Manifest.ClaimsSeat("node-build-agent") || e.Manifest.ClaimsSeat("mesh-build-machine") {
builders = append(builders, name)
}
}
@@ -106,6 +115,13 @@ func dependenciesOf(entries []Entry, against map[string][]string, read map[strin
}
}
}
if known[TheControlPlane] {
for _, b := range builders {
if b != TheControlPlane {
add(b, TheControlPlane, EdgeWorkerOf)
}
}
}
sort.Slice(out, func(a, b int) bool {
if out[a].From != out[b].From {
return out[a].From < out[b].From
+1 -1
View File
@@ -12,7 +12,7 @@ func TestDependenciesAreOneRelationWithTheirKinds(t *testing.T) {
return Entry{Manifest: catalogue.Manifest{Module: name}, Source: Source{Repository: repository}}
}
builder := entry("builder", "http://forge/novox/mesh-catalog.git")
builder.Manifest.Claims = []catalogue.Claim{{Name: "mesh-build-machine", Scope: catalogue.ScopeMesh}}
builder.Manifest.Claims = []catalogue.Claim{{Name: "node-build-agent", Scope: catalogue.ScopeNode}}
plugin := entry("shop-plugin", "http://forge/novox/mesh-catalog.git")
plugin.Manifest.Build = &catalogue.Build{On: []catalogue.BuildsOn{{Arg: "BASE", Module: "shop"}}}
entries := []Entry{
+8 -1
View File
@@ -31,8 +31,11 @@ func TestRegisteringAModuleAgainKeepsWhatTheMeshHoldsForIt(t *testing.T) {
if err := inv.RecordSealingKey(ctx, node.ID, key); err != nil {
t.Fatal(err)
}
// A mergeable file, so any setting composes (novox/hq ADR 0163, rule 6: a setting is judged
// where it is stored).
m := catalogue.Manifest{Module: "step-ca", Version: "1",
Provides: catalogue.Offers("acme-ca"), OwnSecrets: catalogue.OwnSecrets{"password": {Path: "/run/password"}}}
Provides: catalogue.Offers("acme-ca"), OwnSecrets: catalogue.OwnSecrets{"password": {Path: "/run/password"}},
Resources: []map[string]any{{"id": "conf", "type": "file", "path": "/etc/step-ca.json", "content": "{}", "merge": "json"}}}
if err := inv.RegisterModule(ctx, m, Source{}); err != nil {
t.Fatal(err)
}
@@ -229,5 +232,9 @@ func TestAModuleStillAssignedRefusesBeforeAnythingAboutWhatItHolds(t *testing.T)
// declares (novox/hq 04-ISSUES/078).
func withOwnSecret(m catalogue.Manifest, name string) catalogue.Manifest {
m.OwnSecrets = catalogue.OwnSecrets{name: {Path: "/run/" + name}}
// And a mergeable file, so any setting these tests store composes (novox/hq ADR 0163, rule 6:
// a setting is judged where it is stored).
m.Resources = append(m.Resources, map[string]any{"id": "conf", "type": "file",
"path": "/etc/" + m.Module + ".json", "content": "{}", "merge": "json"})
return m
}
@@ -0,0 +1,7 @@
-- What filters a machine, with owners, as the host reports it with every apply (novox/hq ADR 0168):
-- every table and chain that refuses traffic — the mesh's, the found firewall's, the runtime's own,
-- a ban, or other — so the mesh says truthfully what filters a converged machine and names what it
-- did not write. And the state of the firewall a converged machine was found with: in force now or
-- not, and who retired it.
alter table node add column filters jsonb;
alter table node add column found_firewall jsonb;
+9 -2
View File
@@ -15,9 +15,16 @@ func aNodeWithModules(t *testing.T, modules ...string) (*Inventory, string) {
if _, err := inv.AddNode(ctx, "anchor"); err != nil {
t.Fatal(err)
}
// Each publishes the port these tests give it a machine port for: a port given for one the
// module does not publish is refused where it is stored (novox/hq ADR 0163, rule 6).
publishes := map[string]string{"postgres": "5432", "another-database": "5432", "cache": "6379", "web": "8080"}
for _, m := range modules {
if err := inv.RegisterModule(ctx,
catalogue.Manifest{Module: m, Version: "1"}, Source{}); err != nil {
manifest := catalogue.Manifest{Module: m, Version: "1"}
if port, known := publishes[m]; known {
manifest.Resources = []map[string]any{{"id": "server", "type": "container", "name": m,
"image": "x", "ports": []any{port}}}
}
if err := inv.RegisterModule(ctx, manifest, Source{}); err != nil {
t.Fatal(err)
}
}
+49
View File
@@ -820,3 +820,52 @@ func (i *Inventory) SharedHolders(ctx context.Context, provider, providerModule,
sort.Strings(out)
return out, nil
}
// SecretState is one secret a module holds on a machine, as a take compares it (novox/hq ADR
// 0163): its name, where it came from — made by the mesh or accepted from a person — and, for a
// credential the module requires from a provider, which node provides it and the local name it
// goes by where the module keeps several.
type SecretState struct {
Name string
// Local is the credential's name inside the module (ADR 0094); empty for an own secret or the
// ordinary one.
Local string
// Origin is OriginMade or OriginAccepted.
Origin string
// Provider is the node providing a required secret; empty for the module's own.
Provider string
}
// Own says the secret is the module's own rather than one it requires from a provider.
func (s SecretState) Own() bool { return s.Provider == "" }
// SecretsOf is every secret a module holds on a machine: its own, and each credential it requires
// from a provider — with where each value came from. What a take reads to refuse minting over a
// service that already has one (ADR 0163, rule 2).
func (i *Inventory) SecretsOf(ctx context.Context, node, module string) ([]SecretState, error) {
record, err := i.NodeByName(ctx, node)
if err != nil {
return nil, err
}
rows, err := i.store.Pool().Query(ctx,
`select name, '' as local, origin, '' as provider from module_secret
where node = $1 and module = $2
union all
select s.name, s.local, s.origin, p.name from secret s
join node p on p.id = s.provider
where s.consumer = $1 and s.consumer_module = $2
order by 4, 1, 2`, record.ID, module)
if err != nil {
return nil, err
}
defer rows.Close()
var out []SecretState
for rows.Next() {
var s SecretState
if err := rows.Scan(&s.Name, &s.Local, &s.Origin, &s.Provider); err != nil {
return nil, err
}
out = append(out, s)
}
return out, rows.Err()
}
+44
View File
@@ -934,3 +934,47 @@ func TestASharedCredentialIsOneValueSealedToEveryHolder(t *testing.T) {
t.Fatalf("a consumer binding after an acceptance must be refused with the way out: %v", err)
}
}
// SecretsOf is every secret a module holds on a machine with where each came from — what a take
// reads to refuse minting over a service that already has a value (novox/hq ADR 0163, rule 2).
func TestSecretsOfSaysEachSecretsOriginAndProvider(t *testing.T) {
inv, ctx := twoNodesWithKeys(t)
if err := inv.RegisterModule(ctx, catalogue.Manifest{Module: "forge", Version: "1",
Requires: []string{"secret", "postgres-database"},
Secrets: map[string]string{"secret": "/run/secret", "postgres-database": "/run/pg"},
OwnSecrets: catalogue.OwnSecrets{"admin": {Path: "/run/admin"}}}, Source{}); err != nil {
t.Fatal(err)
}
if _, err := inv.SecretForModule(ctx, "consumer", "forge", "admin"); err != nil {
t.Fatal(err)
}
if _, err := inv.SecretFor(ctx, "postgres-database", "consumer", "forge", "provider", ""); err != nil {
t.Fatal(err)
}
if err := inv.AcceptSecretForPair(ctx, "secret", "consumer", "forge", "provider", "", "hunter2"); err != nil {
t.Fatal(err)
}
got, err := inv.SecretsOf(ctx, "consumer", "forge")
if err != nil {
t.Fatal(err)
}
want := []SecretState{
{Name: "admin", Origin: OriginMade},
{Name: "postgres-database", Origin: OriginMade, Provider: "provider"},
{Name: "secret", Origin: OriginAccepted, Provider: "provider"},
}
if len(got) != len(want) {
t.Fatalf("got %+v", got)
}
for i := range want {
if got[i] != want[i] {
t.Errorf("secret %d: got %+v, want %+v", i, got[i], want[i])
}
}
if !got[0].Own() || got[1].Own() {
t.Error("own and required are not told apart")
}
if other, _ := inv.SecretsOf(ctx, "consumer", "gitea"); len(other) != 0 {
t.Fatalf("another module's secrets: %+v", other)
}
}
@@ -0,0 +1,69 @@
package inventory
import (
"strings"
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
)
// A setting is judged where it is stored (novox/hq ADR 0163, rule 6): one that cannot compose with
// the module's definition is refused naming the node, the module, the layer and the key, and is not
// kept; one that reaches nothing is refused the same way.
func TestASettingIsJudgedWhereItIsStored(t *testing.T) {
inv := fresh(t)
ctx := t.Context()
if _, err := inv.AddNode(ctx, "anchor"); err != nil {
t.Fatal(err)
}
web := catalogue.Manifest{Module: "web", Version: "1",
Resources: []map[string]any{
{"id": "server", "type": "container", "name": "web", "image": "x", "ports": []any{"8080"}},
{"id": "conf", "type": "file", "path": "/etc/web.json", "content": "{}", "merge": "json"},
}}
plain := catalogue.Manifest{Module: "plain", Version: "1",
Resources: []map[string]any{{"id": "server", "type": "container", "name": "plain", "image": "x"}}}
for _, m := range []catalogue.Manifest{web, plain} {
if err := inv.RegisterModule(ctx, m, Source{}); err != nil {
t.Fatal(err)
}
}
err := inv.SetSettings(ctx, "anchor", "web", map[string]any{catalogue.PortsSetting: map[string]any{"9999": 10000}})
for _, want := range []string{"refused: web on anchor", `layer "anchor"`, "9999"} {
if err == nil || !strings.Contains(err.Error(), want) {
t.Errorf("a port the module does not publish: %v, want %q", err, want)
}
}
if layers, _ := inv.SettingsFor(ctx, "anchor", "web"); len(layers) != 0 {
t.Fatalf("the refused layer was stored: %v", layers)
}
// A key that reaches nothing is refused too, where the spelling can still be fixed; a module
// with a mergeable file takes any key.
err = inv.SetSettings(ctx, "", "plain", map[string]any{"colour": "blue"})
if err == nil || !strings.Contains(err.Error(), "reach nothing") || !strings.Contains(err.Error(), `"colour"`) {
t.Fatalf("a stray key was stored: %v", err)
}
if err := inv.SetSettings(ctx, "", "web", map[string]any{"colour": "blue"}); err != nil {
t.Fatal(err)
}
// The mesh-wide layer is under the node's when the node's is judged.
if err := inv.SetSettings(ctx, "anchor", "web", map[string]any{catalogue.PortsSetting: map[string]any{"8080": 10000}}); err != nil {
t.Fatal(err)
}
// A kept network is for an adopted machine only (rule 4).
keep := map[string]any{catalogue.NetworksSetting: map[string]any{"server": []any{"predecessor_default"}}}
err = inv.SetSettings(ctx, "anchor", "plain", keep)
if err == nil || !strings.Contains(err.Error(), "anchor is converged") {
t.Fatalf("a kept network on a converged machine was stored: %v", err)
}
if err := inv.SetAdopted(ctx, "anchor", true); err != nil {
t.Fatal(err)
}
if err := inv.SetSettings(ctx, "anchor", "plain", keep); err != nil {
t.Fatal(err)
}
if err := inv.SetSettings(ctx, "anchor", "nothing", keep); err == nil {
t.Fatal("a setting for a module the mesh does not know was stored")
}
}
+32
View File
@@ -0,0 +1,32 @@
package link
import "testing"
// A build machine serves the seat its credential claims (novox/hq ADR 0190 handover): the old
// `builder` keeps the old role, a `build-agent` takes the new, from one binary and no flag.
func TestABuildMachineServesTheSeatItsCredentialClaims(t *testing.T) {
if got := BuildSeatClaimed([]string{"mesh-build-machine"}); got != "mesh-build-machine" {
t.Errorf("a credential claiming the old role serves %q", got)
}
if got := BuildSeatClaimed([]string{"node-build-agent"}); got != TheBuildMachine {
t.Errorf("a credential claiming the new role serves %q", got)
}
if got := BuildSeatClaimed(nil); got != TheBuildMachine {
t.Errorf("a credential claiming nothing serves %q, want the current role", got)
}
if got := BuildSeatClaimed([]string{"", "node-build-agent"}); got != TheBuildMachine {
t.Errorf("an empty claim is skipped; got %q", got)
}
}
// What a machine says about a build is the event of the seat it took the build from, so an outcome
// is heard where the asker of that seat listens.
func TestABuildsEventsAreItsSeats(t *testing.T) {
if BuildOutcomeOf(TheBuildMachineBefore) != "mesh.seat.mesh-build-machine.event.built" {
t.Error(BuildOutcomeOf(TheBuildMachineBefore))
}
if BuildWorkOf(TheBuildMachine) != BuildWork() || BuildOutcomeOf(TheBuildMachine) != BuildOutcome() ||
BuildStartedOf(TheBuildMachine) != BuildStarted() || BuildLogOf(TheBuildMachine, "b1") != BuildLog("b1") {
t.Error("the no-argument forms must name the current role")
}
}
+53 -7
View File
@@ -19,13 +19,57 @@ import (
// act on or a declaration a node reconciles toward; a build is a request that takes minutes and has
// exactly one answer. Too long for request/reply, too particular to be an event.
// TheBuildMachine is the role a build is submitted to.
const TheBuildMachine = "mesh-build-machine"
// TheBuildMachine is the role a build is submitted to: node-scoped, held on every machine that
// builds, and the work shared among them (novox/hq ADR 0190). The name stays for every caller; what
// it names moved from the mesh's one build machine to whichever build agent is idle.
//
// **Switching a live mesh over, in order** — and why no step strands a build. The old seat's
// stream and worker (SEAT_MESH_BUILD_MACHINE, SEAT_MESH_BUILD_MACHINE_worker) stay on the bus until
// removed by hand, and the builder keeps draining them while it is assigned, because a machine
// serves the seat its credential claims (BuildSeatClaimed) and the controller asks the seat that
// has a holder (buildSeatAmong in the command) and hears both seats' outcomes:
//
// 1. Merge the controller and the host's first user list together; the new controller rolls and,
// seeing only the builder assigned, still asks mesh-build-machine — which the builder holds.
// 2. Merge the catalogue's build-agent; the builder builds it and the controller registers it.
// 3. On each machine that builds: `module issue build-agent --node <n>`, then `assign`, then
// `push`. The first holder appears, and from then on asks go to node-build-agent.
// 4. Unassign builder everywhere and `module forget` it.
// 5. By hand: delete SEAT_MESH_BUILD_MACHINE and its worker, drop the retired seat row and
// TheBuildMachineBefore with it, and the second entries in seatsTheControllerAsks and
// ControllerFollows.
const TheBuildMachine = "node-build-agent"
// TheBuildMachineBefore is the role a build was submitted to until ADR 0190: the mesh's one build
// machine, mesh-scoped. Kept named while the handover runs — a machine whose credential claims it
// still serves it, and the controller still hears its outcomes — and dropped with the retired seat
// row once nothing claims it.
const TheBuildMachineBefore = "mesh-build-machine"
// BuildSeatClaimed is the build role a machine serves: the first seat its credential claims, or the
// current role when the credential names none (a credential from before claims travelled in it, or
// one written by hand). **The credential decides, not the binary** (ADR 0190 handover): one build
// machine binary runs as the old `builder` on the old seat and as a `build-agent` on the new one,
// each taking the work the mesh issued it a credential for, so neither drains the other's queue
// and the switch needs no flag day.
func BuildSeatClaimed(claimed []string) string {
for _, seat := range claimed {
if seat != "" {
return seat
}
}
return TheBuildMachine
}
// BuildWork is where a build request lands, and BuildOutcome is where its result does. Derived from
// the seat, so both sides name the role and neither names the other.
func BuildWork() string { return "mesh.seat." + TheBuildMachine + ".accept.build" }
func BuildOutcome() string { return "mesh.seat." + TheBuildMachine + ".event.built" }
// the seat, so both sides name the role and neither names the other. The no-argument forms name the
// current role; the `Of` forms take the seat, for the handover during which two roles exist.
func BuildWork() string { return BuildWorkOf(TheBuildMachine) }
func BuildOutcome() string { return BuildOutcomeOf(TheBuildMachine) }
func BuildWorkOf(seat string) string { return "mesh.seat." + seat + ".accept.build" }
func BuildOutcomeOf(seat string) string {
return "mesh.seat." + seat + ".event.built"
}
// BuildStarted is where a build machine says it has taken a build, and BuildLog is where it says
// what it is doing, one line per message, under the build's own id (novox/hq ADR 0157).
@@ -35,8 +79,10 @@ func BuildOutcome() string { return "mesh.seat." + TheBuildMachine + ".event.bui
// lived in one container's stderr on one machine. Every line is now an event of the role, retained
// with the rest of the mesh's events, so a reader follows a build live by subscribing its subject,
// or reads it back afterwards from the stream, and a viewer is a subscriber and nothing more.
func BuildStarted() string { return "mesh.seat." + TheBuildMachine + ".event.started" }
func BuildLog(id string) string { return "mesh.seat." + TheBuildMachine + ".event.log." + id }
func BuildStarted() string { return BuildStartedOf(TheBuildMachine) }
func BuildLog(id string) string { return BuildLogOf(TheBuildMachine, id) }
func BuildStartedOf(seat string) string { return "mesh.seat." + seat + ".event.started" }
func BuildLogOf(seat, id string) string { return "mesh.seat." + seat + ".event.log." + id }
// BuildStart is what a build machine says the moment it takes a build.
type BuildStart struct {
+1 -1
View File
@@ -26,7 +26,7 @@ func TestTheOldBusAnnouncesABuildUnderBothNames(t *testing.T) {
if KeyRoleBuilt != "built" {
t.Fatalf("the role's event is %q, and a holder emits its verbs bare", KeyRoleBuilt)
}
if TheBuildMachine != "mesh-build-machine" {
if TheBuildMachine != "node-build-agent" {
t.Fatalf("the role is %q", TheBuildMachine)
}
// The two must differ, or one publish would serve both and this doubling would be pointless.
+74 -25
View File
@@ -25,16 +25,34 @@ import (
type natsBuilds struct {
js *broker.JetStream
owned bool
// seat is the build role asked: the one that has a holder (ADR 0190 handover), chosen by the
// controller from what is assigned, so an ask lands where a machine is pulling.
seat string
}
// BuildsOverNATS is the asking side on the bus being built. It dials, because the command that asks
// for a build is a one-shot and holds nothing else.
// BuildsOverNATS is the asking side on the bus being built, asking the current build role. It dials,
// because the command that asks for a build is a one-shot and holds nothing else.
func BuildsOverNATS(address string) (Builders, error) {
return BuildsOverNATSOn(address, TheBuildMachine)
}
// BuildsOverNATSOn is the asking side for one named build role — during the handover from the one
// build machine to build agents, the role that has a holder (ADR 0190).
func BuildsOverNATSOn(address, seat string) (Builders, error) {
js, err := broker.Dial(address)
if err != nil {
return nil, fmt.Errorf("cannot reach the bus at %s to ask for a build: %w", address, err)
}
return &natsBuilds{js: js, owned: true}, nil
return &natsBuilds{js: js, owned: true, seat: seat}, nil
}
// role is the seat asked: what the asker was made for, or the current build role for one made
// without saying (a test building the struct by hand).
func (b *natsBuilds) role() string {
if b.seat == "" {
return TheBuildMachine
}
return b.seat
}
func (b *natsBuilds) Close() {
@@ -51,7 +69,7 @@ func (b *natsBuilds) Ask(ctx context.Context, request BuildRequest) error {
}
publish, cancel := context.WithTimeout(ctx, 30*time.Second)
defer cancel()
if _, err := b.js.Context().Publish(BuildWork(), body, nats.Context(publish)); err != nil {
if _, err := b.js.Context().Publish(BuildWorkOf(b.role()), body, nats.Context(publish)); err != nil {
return fmt.Errorf("cannot submit a build: %w", err)
}
return nil
@@ -63,7 +81,7 @@ func (b *natsBuilds) Submit(ctx context.Context, request BuildRequest,
// Subscribed before the ask, so an outcome cannot arrive before there is anywhere for it to
// land. Core, not the stream: the asker is waiting now, and the durable copy of this outcome is
// the same event on EVENTS, which the controller records.
outcomes, err := b.js.Conn().SubscribeSync(BuildOutcome())
outcomes, err := b.js.Conn().SubscribeSync(BuildOutcomeOf(b.role()))
if err != nil {
return BuildResult{}, fmt.Errorf("cannot listen for a build's outcome: %w", err)
}
@@ -80,7 +98,7 @@ func (b *natsBuilds) Submit(ctx context.Context, request BuildRequest,
// be assumed, because nothing else will ever say so.
publish, cancel := context.WithTimeout(ctx, 30*time.Second)
defer cancel()
if _, err := b.js.Context().Publish(BuildWork(), body, nats.Context(publish)); err != nil {
if _, err := b.js.Context().Publish(BuildWorkOf(b.role()), body, nats.Context(publish)); err != nil {
return BuildResult{}, fmt.Errorf("cannot submit a build: %w", err)
}
@@ -115,9 +133,16 @@ type natsMachine struct {
sub *nats.Subscription
}
// MachineOverNATS takes build work from the role this machine holds.
// MachineOverNATS takes build work from the current build role.
func MachineOverNATS(js *broker.JetStream, on string) BuildMachine {
return &natsMachine{js: js, on: on, seat: TheBuildMachine}
return MachineOverNATSOn(js, on, TheBuildMachine)
}
// MachineOverNATSOn takes build work from the role named — the one this machine's credential claims
// (ADR 0190 handover): its asks come from that seat's worker, and what it says about a build goes
// out as that seat's events, so an outcome is heard where the asker listens.
func MachineOverNATSOn(js *broker.JetStream, on, seat string) BuildMachine {
return &natsMachine{js: js, on: on, seat: seat}
}
func (m *natsMachine) Close() {
@@ -126,29 +151,31 @@ func (m *natsMachine) Close() {
}
}
// Take binds to the role's worker and hands each request over, one at a time.
// Take binds to the role's worker and pulls one request at a time, handing each over.
//
// **Bound, never created.** The work queue and the worker on it are the controller's to define
// (design 25 §3), and a build machine reaches no part of the JetStream API — so a missing one is said
// as the mesh's to answer rather than quietly created with whatever this client defaults to.
//
// **Pulled, one at a time, by whichever holder is free** (novox/hq ADR 0190). Every machine holding
// the role binds this same worker; a machine asks for the next request only when it has finished
// the last, so a slow machine never holds an ask an idle one could take, and a machine that took
// five at once would run five container builds against one runtime and finish all of them slower
// than the first.
func (m *natsMachine) Take(ctx context.Context, do func(context.Context, Build)) error {
worker, found := broker.HolderConsumerFor(m.on, "builder",
worker, found := broker.HolderConsumerFor(m.on, "build-agent",
broker.DeclaredSeat{Name: m.seat, Accepts: []string{"build"}})
if !found {
return fmt.Errorf("%s accepts no work, so there is nothing for this machine to take", m.seat)
}
// One at a time, which the consumer's own ack-pending limit enforces rather than a prefetch
// setting: a machine that took five requests at once would run five container builds against one
// runtime and finish all of them slower than the first.
work := make(chan *nats.Msg, 1)
// **The consumer's own filter, not the one subject this machine cares about.** The client checks
// what is asked for against the consumer's filter and refuses anything that is not the same —
// "subject does not match consumer" — so subscribing `…accept.build` against a consumer filtered
// on `…accept.>` is rejected even though it is narrower. Learned twice now, on two different
// consumers, which is why it is written down here.
filter := worker.Filters[0]
sub, err := m.js.Context().ChanQueueSubscribe(filter, worker.Queue, work,
sub, err := m.js.Context().PullSubscribe(filter, worker.Name,
nats.Bind(worker.Stream, worker.Name), nats.ManualAck())
if err != nil {
return fmt.Errorf(
@@ -159,13 +186,33 @@ func (m *natsMachine) Take(ctx context.Context, do func(context.Context, Build))
m.sub = sub
for {
select {
case <-ctx.Done():
if ctx.Err() != nil {
return nil
case msg, ok := <-work:
if !ok {
return errors.New("the bus stopped delivering build work")
}
// One, and wait a while for it; an empty queue is a timeout, which is the normal state of a
// machine with nothing to build, and is asked again.
fetched, err := sub.Fetch(1, nats.Context(ctx))
switch {
case ctx.Err() != nil:
// Ours ended: the machine is being stopped.
return nil
case errors.Is(err, context.Canceled), errors.Is(err, context.DeadlineExceeded), errors.Is(err, nats.ErrTimeout):
// **An empty queue, not the end.** A fetch on a context without a deadline waits the
// client's own while and then says the deadline passed — the client's, not ours. Read
// as "stop", every idle build machine exited clean every half minute and was started
// again by its supervisor, which looked like a crash loop with nothing in the log to
// say why (2026-10-03, the first build agents). Asked again.
continue
case err != nil:
if sub.IsValid() {
// A transient fault in asking — a reconnect, a slow server — is asked past rather
// than ending the machine; one that outlasts the ack wait redelivers nothing lost.
time.Sleep(time.Second)
continue
}
return fmt.Errorf("the bus stopped delivering build work: %w", err)
}
for _, msg := range fetched {
var request BuildRequest
if err := json.Unmarshal(msg.Data, &request); err != nil {
// Unreadable: terminated rather than retried, because the next attempt reads the same
@@ -178,7 +225,7 @@ func (m *natsMachine) Take(ctx context.Context, do func(context.Context, Build))
// the ask to a second machine nor counts the wait against its deliveries.
working := make(chan struct{})
go stillWorking(msg, working)
do(ctx, &natsBuild{request: request, msg: msg, on: m.on, js: m.js})
do(ctx, &natsBuild{request: request, msg: msg, on: m.on, js: m.js, seat: m.seat})
close(working)
}
}
@@ -189,7 +236,9 @@ type natsBuild struct {
msg *nats.Msg
on string
js *broker.JetStream
seq int
// seat is the role this build was taken from; what the machine says about it is that role's.
seat string
seq int
}
func (b *natsBuild) Request() BuildRequest { return b.request }
@@ -216,7 +265,7 @@ func (b *natsBuild) Announce(ctx context.Context, result BuildResult) error {
}
publish, cancel := context.WithTimeout(ctx, 30*time.Second)
defer cancel()
if _, err := b.js.Context().Publish(BuildOutcome(), body, nats.Context(publish)); err != nil {
if _, err := b.js.Context().Publish(BuildOutcomeOf(b.seat), body, nats.Context(publish)); err != nil {
return fmt.Errorf("cannot announce a build's outcome: %w", err)
}
return nil
@@ -236,7 +285,7 @@ func (b *natsBuild) Began(ctx context.Context) error {
}
publish, cancel := context.WithTimeout(ctx, 30*time.Second)
defer cancel()
if _, err := b.js.Context().Publish(BuildStarted(), body, nats.Context(publish)); err != nil {
if _, err := b.js.Context().Publish(BuildStartedOf(b.seat), body, nats.Context(publish)); err != nil {
return fmt.Errorf("cannot say a build started: %w", err)
}
return nil
@@ -254,7 +303,7 @@ func (b *natsBuild) Say(step, message string) {
if err != nil {
return
}
_ = b.js.Conn().Publish(BuildLog(b.request.ID), body)
_ = b.js.Conn().Publish(BuildLogOf(b.seat, b.request.ID), body)
}
func (b *natsBuild) Hold(after time.Duration) error { return b.msg.NakWithDelay(after) }
+119 -3
View File
@@ -48,7 +48,7 @@ func aBusWithTheBuildRole(t *testing.T) *broker.JetStream {
t.Fatal(err)
}
clean := func() {
_ = js.Context().DeleteStream("SEAT_MESH_BUILD_MACHINE")
_ = js.Context().DeleteStream("SEAT_NODE_BUILD_AGENT")
for _, s := range broker.MeshStreams() {
_ = js.Context().PurgeStream(s.Name)
}
@@ -158,7 +158,7 @@ func TestNatsABuildIsTakenAndItsOutcomeReachesEverybody(t *testing.T) {
// And the work left the queue: a request a machine took and settled must not be given to another.
deadline := time.Now().Add(5 * time.Second)
for time.Now().Before(deadline) {
info, err := js.Context().StreamInfo("SEAT_MESH_BUILD_MACHINE")
info, err := js.Context().StreamInfo("SEAT_NODE_BUILD_AGENT")
if err == nil && info.State.Msgs == 0 {
return
}
@@ -177,7 +177,7 @@ func TestNatsABuildWaitsForAMachineRatherThanFailing(t *testing.T) {
if _, err := js.Context().Publish(BuildWork(), body); err != nil {
t.Fatal(err)
}
info, err := js.Context().StreamInfo("SEAT_MESH_BUILD_MACHINE")
info, err := js.Context().StreamInfo("SEAT_NODE_BUILD_AGENT")
if err != nil || info.State.Msgs != 1 {
t.Fatalf("the work did not queue: %+v %v", info, err)
}
@@ -245,3 +245,119 @@ func TestNatsWorkAMachineDidNotAnswerGoesBackToTheQueue(t *testing.T) {
func quietLog() *log.Logger { return log.New(io.Discard, "", 0) }
var _ = quietLog
// Two machines holding the role share one queue (novox/hq ADR 0190): three asks, each machine takes
// one and the third waits until one of them is done; an ask is never handed to a machine that is
// busy; and a machine that stops mid-ask leaves its ask to the other.
func TestNatsTwoMachinesShareTheWorkAndNeitherIsHandedMoreThanItCanTake(t *testing.T) {
js := aBusWithTheBuildRole(t)
ctx, stop := context.WithCancel(context.Background())
defer stop()
for _, id := range []string{"w-1", "w-2", "w-3"} {
body, _ := json.Marshal(BuildRequest{ID: id, Repository: "/r"})
if _, err := js.Context().Publish(BuildWork(), body); err != nil {
t.Fatal(err)
}
}
type taken struct{ machine, id string }
took := make(chan taken, 8)
release := map[string]chan struct{}{"anchor": make(chan struct{}), "laptop": make(chan struct{})}
machines := map[string]BuildMachine{}
for _, name := range []string{"anchor", "laptop"} {
name := name
m := MachineOverNATS(js, name)
machines[name] = m
defer m.Close()
go func() {
_ = m.Take(ctx, func(ctx context.Context, work Build) {
took <- taken{name, work.Request().ID}
<-release[name]
_ = work.Announce(ctx, BuildResult{ID: work.Request().ID, On: name})
_ = work.Done()
})
}()
}
// Each machine took exactly one, and they are different asks.
first := map[string]string{}
for i := 0; i < 2; i++ {
select {
case got := <-took:
if _, twice := first[got.machine]; twice {
t.Fatalf("%s was handed a second ask while busy with its first", got.machine)
}
first[got.machine] = got.id
case <-time.After(10 * time.Second):
t.Fatalf("only %d machine(s) took work; two idle holders should both have", len(first))
}
}
if first["anchor"] == first["laptop"] {
t.Fatalf("both machines took %q: the queue is not shared, it is copied", first["anchor"])
}
// The third waits: nobody is free.
select {
case got := <-took:
t.Fatalf("%s was handed %s while both machines were busy", got.machine, got.id)
case <-time.After(2 * time.Second):
}
// One finishes, and only then is the third taken — by that machine, the one that is free.
close(release["anchor"])
release["anchor"] = make(chan struct{})
select {
case got := <-took:
if got.machine != "anchor" {
t.Fatalf("the third ask went to %s, which is still busy", got.machine)
}
case <-time.After(10 * time.Second):
t.Fatal("the third ask was never taken after a machine became free")
}
// A machine that stops mid-ask leaves its ask unacknowledged, and the ack wait brings it round
// to whoever is left — the path TestNatsWorkAMachineDidNotAnswerGoesBackToTheQueue proves with
// an explicit hand-back, because the real wait is a minute. Here: the laptop goes, anchor
// finishes, and with nothing queued nothing more is taken by the machine that is left.
machines["laptop"].Close()
close(release["anchor"])
select {
case got := <-took:
t.Fatalf("%s took %s; the queue should be empty", got.machine, got.id)
case <-time.After(2 * time.Second):
}
}
// During the handover (ADR 0190) two build roles exist. A machine whose credential claims the retired
// one takes an ask published to that seat and answers as that seat; the asker of that seat hears it.
func TestNatsAMachineOnTheRetiredBuildRoleTakesThatRolesAsks(t *testing.T) {
js := aBusWithTheBuildRole(t)
seats := []broker.DeclaredSeat{{Name: TheBuildMachineBefore, Accepts: []string{"build"}, Emits: []string{"built"}}}
if err := broker.RaiseSeats(js, seats, map[string]broker.Holder{TheBuildMachineBefore: {Node: "anchor", Module: "builder"}}); err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = js.Context().DeleteStream("SEAT_MESH_BUILD_MACHINE") })
ctx, stop := context.WithCancel(context.Background())
defer stop()
machine := MachineOverNATSOn(js, "anchor", TheBuildMachineBefore)
defer machine.Close()
go func() {
_ = machine.Take(ctx, func(ctx context.Context, work Build) {
_ = work.Began(ctx)
_ = work.Announce(ctx, BuildResult{ID: work.Request().ID, Repository: work.Request().Repository, On: "anchor", Commit: "abc"})
_ = work.Done()
})
}()
asker, err := BuildsOverNATSOn(os.Getenv("MESH_TEST_NATS"), TheBuildMachineBefore)
if err != nil {
t.Fatal(err)
}
defer asker.Close()
result, err := asker.Submit(ctx, BuildRequest{ID: "build-old-seat", Repository: "r"}, 20*time.Second)
if err != nil {
t.Fatal(err)
}
if result.On != "anchor" || result.ID != "build-old-seat" {
t.Errorf("the retired role's holder did not answer: %+v", result)
}
}
+17
View File
@@ -305,6 +305,23 @@ func (e Enrolment) Heard(ctx context.Context, report Report) (news bool, err err
return false, err
}
}
// What filters the machine, and the state of its found firewall (novox/hq ADR 0168), whenever
// it says — every apply of a host that knows how, adopted or converged; never cleared by a
// report that carries none, which is every bare word that the node is there.
if len(report.Filters) > 0 || report.FoundFirewall != nil {
filters := make([]inventory.Filter, 0, len(report.Filters))
for _, f := range report.Filters {
filters = append(filters, inventory.Filter{Where: f.Where, Owner: f.Owner, Refuses: f.Refuses})
}
var found *inventory.FoundFirewall
if report.FoundFirewall != nil {
found = &inventory.FoundFirewall{Kind: report.FoundFirewall.Kind, Active: report.FoundFirewall.Active,
RetiredBy: report.FoundFirewall.RetiredBy}
}
if err := e.Inventory.RecordFiltering(ctx, node.ID, filters, found); err != nil {
return false, err
}
}
// Which of its links face outside (novox/hq ADR 0140), whenever it says so. Recorded on every
// report that carries it, adopted or converged, because the filter the mesh composes is written
// around it — and never cleared by a report that carries none, which is every bare word that the
+42
View File
@@ -218,3 +218,45 @@ func TestWhatAnAdoptedNodeHoldsIsKeptAndAnAliveWordDoesNotWipeIt(t *testing.T) {
t.Fatalf("a report from an adopted node holding nothing did not empty held: %+v", got)
}
}
// What filters a machine, and the state of its found firewall, are kept from every report that
// carries them and never cleared by one that does not (novox/hq ADR 0168).
func TestWhatFiltersAMachineIsKeptFromItsReport(t *testing.T) {
inv, _, _ := heardFrom(t, link.Report{
Node: "home-server", Applied: []string{"a"},
Filters: []link.Filter{
{Where: "table inet mesh, chain forward", Owner: "mesh", Refuses: "policy drop"},
{Where: "chain HAL-MESH-ONLY (iptables-legacy)", Owner: "other", Refuses: "-j DROP"},
},
FoundFirewall: &link.FoundFirewall{Kind: "ufw", Active: false, RetiredBy: "found-inactive"},
})
ctx := context.Background()
f, err := inv.FilteringOf(ctx, "home-server")
if err != nil {
t.Fatal(err)
}
if len(f.Filters) != 2 || f.Filters[1].Owner != inventory.FilterOther || f.Alone() {
t.Fatalf("recorded %+v", f)
}
if f.FoundFirewall == nil || f.FoundFirewall.RetiredBy != "found-inactive" || f.FoundFirewall.Active {
t.Fatalf("the found firewall's state: %+v", f.FoundFirewall)
}
if len(f.Others()) != 1 || f.Others()[0].Where != "chain HAL-MESH-ONLY (iptables-legacy)" {
t.Fatalf("others: %+v", f.Others())
}
// A bare word that the node is there clears nothing.
if _, err := (link.Enrolment{Inventory: inv}).Heard(ctx, link.Report{Node: "home-server"}); err != nil {
t.Fatal(err)
}
if again, _ := inv.FilteringOf(ctx, "home-server"); len(again.Filters) != 2 {
t.Fatalf("a bare report cleared what filters the machine: %+v", again)
}
// The next full report replaces it: the chain removed by hand is gone from the record.
if _, err := (link.Enrolment{Inventory: inv}).Heard(ctx, link.Report{Node: "home-server", Applied: []string{"a"},
Filters: []link.Filter{{Where: "table inet mesh, chain forward", Owner: "mesh", Refuses: "policy drop"}}}); err != nil {
t.Fatal(err)
}
if again, _ := inv.FilteringOf(ctx, "home-server"); len(again.Filters) != 1 || !again.Alone() {
t.Fatalf("the next report did not replace what filters the machine: %+v", again)
}
}
+24
View File
@@ -197,6 +197,15 @@ type Report struct {
// Strays is what runs on the machine that the mesh neither wrote nor holds (ADR 0163).
Strays []Stray `json:"strays,omitempty"`
// Filters is what filters the machine now: every table and chain that refuses traffic, with
// its owner — the mesh's, the found firewall's, the container runtime's own, a ban, or other
// (novox/hq ADR 0168). Every machine reports it, adopted or converged; absent from a host older
// than this.
Filters []Filter `json:"filters,omitempty"`
// FoundFirewall is the state of the firewall a converged machine was found with: in force now
// or not, and how it came to be inactive — the mesh disabled it, or it was found so (ADR 0168).
FoundFirewall *FoundFirewall `json:"found_firewall,omitempty"`
// Profile is what the machine can do, detected again by this apply (novox/hq ADR 0161): the
// same shape enrolment sends, so a machine that gained or lost a capability — switched its
// network manager — is known at its next push and not at its next enrolment. Absent from a host
@@ -278,6 +287,21 @@ type Held struct {
Facts map[string]any `json:"facts,omitempty"`
}
// A Filter is one place on a machine that refuses traffic, with its owner (novox/hq ADR 0168):
// the host's own shape, carried as data.
type Filter struct {
Where string `json:"where"`
Owner string `json:"owner"`
Refuses string `json:"refuses"`
}
// FoundFirewall is the state of a converged machine's found firewall (ADR 0168).
type FoundFirewall struct {
Kind string `json:"kind"`
Active bool `json:"active"`
RetiredBy string `json:"retired_by,omitempty"`
}
// A Stray is a container a machine runs that the mesh neither wrote nor holds (ADR 0163).
type Stray struct {
Kind string `json:"kind"`
+3 -2
View File
@@ -223,10 +223,11 @@ func kindOfSubject(subject string) (string, bool) {
return KindCatchUp, true
case broker.ControllerFollows[3]:
return KindSourceMoved, true
case BuildOutcome():
case BuildOutcome(), BuildOutcomeOf(TheBuildMachineBefore):
// A build's outcome is the role's event now, so it arrives on the events stream rather than
// the control branch — and is acted on by the same handler, because what the controller does
// with it did not change (novox/hq ADR 0121).
// with it did not change (novox/hq ADR 0121). From either build role while the handover
// runs (ADR 0190): the old builder still answers on the retired seat until it is unassigned.
return KindBuilt, true
}
return "", false
+3 -4
View File
@@ -169,10 +169,9 @@ func (g *Generator) Graph() Graph { return g.graph }
//
// - No floor: no header, no localhost, no `127.0.1.1` — those are the machine's, above the region.
// - A machine's own line is marked, and its mesh name resolves to its mesh address, not loopback.
// - `.Names` is every name the mesh serves (issue 111), so anything on the machine reaching a
// routed name through its resolver finds the machine serving it; machines with no address yet
// are already left out of the set. A routed name is one alias, itself — a machine has a bare
// name beside its full one, a routed name has nothing beside it (issue 157).
// - `.Names` is the machines (novox/hq ADR 0191): a route's internal name is under its node's
// internal domain and the resolver answers it by wildcard, and a public name is public DNS's.
// Machines with no address yet are already left out of the set.
const hostsTemplate = "# The mesh's names. This region is replaced whenever a machine joins or leaves.\n" +
"{{range .Names}}{{.Address}}\t{{.FQDN}}{{if ne .Name .FQDN}}\t{{.Name}}{{end}}{{if eq .Name $.Node}}\t# this machine{{end}}\n{{end}}"