Compare commits

...
133 Commits
Author SHA1 Message Date
mesh-admin a96e2f0d78 Merge pull request 'A build says what it does on the bus, as it happens (ADR 0157)' (#178) from feat/a-build-says-what-it-does into main 2026-09-30 22:43:52 +00:00
jschoubben 17f7cb0d9c A build says what it does on the bus, as it happens (novox/hq ADR 0157)
The build-machine seat emits `started` and `log.<build id>` beside `built`. Every line the builder
speaks — each step, each command with its duration, and on failure the command's own output — goes
to stderr as before and onto the bus under the build's id, one subject per build, kept a week in
EVENTS with every other event. `builds --log <id>` reads it back from the stream with a consumer
that is gone when the reading is done, on the command line and as the controller's seat verb;
`builds` lists each build's id and `build` says the id it asked with.

Lines are core publishes with a sequence number, so a build is not slowed by an ack per line and a
gap is visible; `started` and `built` are awaited into the stream. The seat protocol widens
additively at the controller's next start; the holder's grant follows on the broker node's next
composition.
2026-10-01 00:43:07 +02:00
mesh-admin f6685ed22d Merge pull request 'An assignment places a module's directories and its accesses (hq 153)' (#176) from feat/153-an-assignment-places-directories-and-accesses into main 2026-09-30 22:03:33 +00:00
jschoubben 52c18f7a45 The path-preservation proof resolves access ids to their default paths too
An access named by id (issue 153) resolves to the path the definition still carries when the
assignment says nothing, and the proof compares that — the same rule as a placed directory.
2026-10-01 00:01:41 +02:00
jschoubben fa7415fcd0 Merge main into the branch: the assignment's placement sits beside the mesh's own place (issue 174) 2026-09-30 23:47:09 +02:00
mesh-admin f8947a806d Merge pull request 'The controller's own manifest names its paths for one more release' (#177) from fix/the-controllers-own-manifest-waits-a-release into main 2026-09-30 21:15:10 +00:00
jschoubben b98e503a61 The controller's own manifest names its paths for one more release
A manifest word ships one release after the code that reads it. The merged manifest already said
`place: "mesh"`, and the running controller, which does not know the word, refused its own build
result — so the controller that knows it could never be built. The literal paths come back here;
the conversion follows once this release runs.
2026-09-30 23:11:36 +02:00
jschoubben 5b832918df Merge pull request 'A setting reaches only what declares it, the mesh places its own files, registration refuses a name (issues 173, 174, ADR 0155)' (#175) from feat/the-mesh-places-its-own-files into main 2026-09-30 20:50:59 +00:00
jschoubben 17bbcc1596 An assignment places a module's directories and its accesses (hq 153)
A definition names no host path (ADR 0112); an adopted machine keeps its
data where the predecessor put it. Two settings, validated like endpoints:

  places:   {<directory id>: <path> | {path, owner}}
  accesses: {<access id>: <path>}

An access may now be declared by id (`{"id": "series", "mode": "read-write"}`)
and named in mounts, env and content as ${access:<id>}; the assignment
says where it is on this node, and an access nobody placed is refused by
name. A definition still carrying a path keeps it as the default the
assignment replaces. A placed directory takes the assignment's owner
where it says one. Resolved in composition, so the host receives paths
and owners exactly as before.
2026-09-30 22:42:50 +02:00
jschoubben 29be985c23 A served value or a contribution's may be the operator's: ${setting:…} fills there too, refused by name when unset
Issue 173's rule needs it: a mail provider serves its domain and an identity provider its issuer,
and neither is the definition's to state. Declared as ${setting:<key>}, filled from the layers after
the overrides; a key so asked for is not stray.
2026-09-30 22:34:43 +02:00
jschoubben 05d977666a A setting reaches only what declares it, the mesh places its own files, registration refuses a name
Three of novox/hq's group-4 leftovers, one branch.

Issue 173: a module's settings reached every route it contributed, every database it asked for and
every served fact its consumers read — a mail server's site name arrived at the proxy as a route
fact. A setting now overrides a key a contribution or served fact declares and adds none; a file
still merges any key, and a key nothing takes is named as stray instead of dropped silently.

Issue 174: the mesh's own files for a module — its bus credential, its merged config, its bindings —
were placed by the definition under /var/lib/mesh/<module>, 232 host paths in 50 definitions. A
directory may now say `place: "mesh"` and resolves to <root>/mesh/<module>; a directory beneath a
placed one may state its path as `${dir:<id>}/<rest>` and moves with it. The proof test resolves
both catalogues and compares: 48 definitions, no path moved. The controller's own manifest is
converted here; the catalogue in mesh-catalog.

ADR 0155: the installation check moves to registration. `module add` and a build's result both
refuse a definition that names an installation, in the check's words, with the way out; the build
stays recorded.
2026-09-30 22:29:28 +02:00
jschoubben e871991495 Merge pull request 'The catalogue-wide checks run against the sibling checkout by default' (#174) from fix/the-catalogue-checks-run-by-default into main 2026-09-30 20:10:44 +00:00
jschoubben f9e19814eb The catalogue-wide checks run against the checkout beside this one by default
A check that only ran when somebody remembered a variable was a check nobody ran (novox/hq issue
134). MESH_CATALOGUE still overrides; the checks skip only when no catalogue can be found.
2026-09-30 22:10:42 +02:00
jschoubben af31315a5f Merge pull request 'The controller takes one announcement at a time' (#173) from fix/one-announcement-at-a-time into main 2026-09-30 19:37:56 +00:00
jschoubben 474f68b34c The controller takes one announcement at a time
A merge's handler builds for minutes and keeps its own delivery alive; the announcements handed over
behind it timed out on the client and came back, and a merge that came back rebuilt what it had just
built, five times over (novox/hq issue 175). MaxAckPending 1 on the events consumer: the server holds
the rest.
2026-09-30 21:37:53 +02:00
jschoubben 1a724f20fe Merge pull request 'The seat rename survives a row seeded under the new name' (#172) from fix/the-seat-rename-survives-a-seeded-row into main 2026-09-30 19:34:37 +00:00
jschoubben 0da0bb2157 The seat rename survives a row seeded under the new name
A controller whose defaults carry the new name seeds it before the migration runs, and the first
form renamed into a duplicate key; the control node's prepare failed on every attempt (2026-09-30).
If the new row exists, the old row's holding moves to it and the old row goes; otherwise it is
renamed. The old name becomes an alias either way.
2026-09-30 21:34:34 +02:00
jschoubben 118e333ff8 Merge pull request 'The artifact store's seat is named for its scope: mesh-artifact-store' (#171) from feat/the-artifact-store-seat-is-named-for-its-scope into main
Reviewed-on: #171
2026-09-30 19:16:47 +00:00
jschoubben c1334f3f85 The artifact store's seat is named for its scope: mesh-artifact-store
ADR 0121 decided it and deferred it as a delivering-seat migration; ADR 0122's aliases made it one
update and one alias (migration 0048). The former name resolves to it forever (novox/hq ADR 0156,
issue 123).
2026-09-30 21:14:40 +02:00
jschoubben 70d379816c Merge pull request 'A converted definition resolves to the paths it named before' (#170) from feat/definitions-place-their-directories into main 2026-09-30 19:11:45 +00:00
jschoubben d8e7c13f6d A converted definition resolves to the paths it named before
The check novox/hq issue 119 asks for before a definition stops naming where its data lives: two
catalogue checkouts, every module in both resolved with the controller's own rule and compared whole.
2026-09-30 21:10:18 +02:00
jschoubben 1851a15e57 Merge pull request 'A definition names no installation: the check, an operator's value, a context on the git seat' (#169) from feat/a-definition-names-no-installation into main
Reviewed-on: #169
2026-09-30 18:36:17 +00:00
jschoubben d9dbc9a59a A definition names no installation: the check, an operator's value, a context on the git seat
InstallationProblems judges every value the mesh acts on for a name under a public top-level domain
or a public address, with prose, the world's registries, resolvers and certificate authorities
exempt, and a name a resource means on purpose declared with its reason (names-on-purpose). Run by
module check and a catalogue-wide test, not yet at registration, while the declared list shrinks.
${setting:<key>} fills a file from the assignment's settings and is refused when nothing set it.
A build context may live on the git seat; the request carries the seat's clone base (novox/hq ADR
0112, ADR 0155, issues 122 and 134).
2026-09-30 18:38:06 +02:00
jschoubben d5e1332dda Merge pull request 'A JSON verb's answer is its standard output alone' (#168) from fix/a-verbs-answer-is-its-stdout into main
Reviewed-on: #168
2026-09-30 16:20:14 +00:00
jschoubben 5ea0e87059 A JSON verb's answer is its standard output alone
status --json prints its warnings beside the document; parsed from both streams together the first
status asked through the console carried no answer as data. Output stays both streams, in order.
2026-09-30 18:18:43 +02:00
jschoubben 8e81266cdc Merge pull request 'A holder binds its seat's tools when it may, not only when it starts' (#167) from fix/a-holder-binds-when-it-may into main 2026-09-30 16:13:46 +00:00
jschoubben 70705ffe45 A holder binds its seat's tools when it may, not only when it starts
The grant is a line in the bus's user list the controller itself composes and a push delivers, so
the first controller to serve its seat started before the list named it and every subscription was
refused for good (2026-09-30). A refused subscription is retried until it holds.
2026-09-30 17:59:23 +02:00
jschoubben 91c4da8a82 Merge pull request 'The mesh's own verbs are the mesh-controller seat's tools' (#166) from feat/the-mesh-answers-for-itself into main
Reviewed-on: #166
2026-09-30 15:54:18 +00:00
jschoubben e9df5dccab The mesh's own verbs are the mesh-controller seat's tools
A seat's protocol lives in the store (migration 0047; seeded additively), a served verb carries its
description and schema, holding a mesh seat requires serving its verbs, a node-scoped seat's tool
carries the node, and the control plane serves status, nodes, node, modules, seats, builds, plan,
assign, unassign, push, build and tools on its seat by running the same commands (novox/hq ADR 0132,
ADR 0154, design 33). A grant of * reaches a role's tools; seat:<seat>.<verb> grants one.
2026-09-30 17:39:38 +02:00
mesh-admin 990ef27cd2 Merge pull request 'A module is told the name it is served under (hq 122)' (#149) from fix/122-a-module-is-told-its-own-name into main 2026-09-30 15:13:53 +00:00
jschoubben 0a17a9a2eb A module is told the name it is served under (hq 122)
A module contributes a label; the mesh joins it with the node's domains and
the provider serves the result — and the module itself was never told.
Software that must know its own address (a login redirect, a canonical URL,
an issuer) had it written into the manifest as a literal: a domain in a
definition, wrong on every other machine (ADR 0112). Found converting
grafana's keycloak login for ace, where it forced GF_SERVER_ROOT_URL and
keycloak's issuer back into manifests.

The binding for a requirement a module contributes to now carries `name`
and `internal-name` (or `names` by local name for several contributions),
and `${bound:<requirement>:name}` / `:internal-name` (`:name-<local>`) fill
files from it. Both come from the one function the provider's received
file is composed by, so the proxy and the module cannot disagree about the
name. Absent when nothing was composed, so a file asking for a name on a
node with no public domain is refused, not rendered empty.

Also: `${bound:…}` could not name a requirement answered by a node-scoped
provider on the same machine — its binding file was written (from `here`)
but the placeholders only looked at the mesh's needs. Filled from the same
answer now.
2026-09-30 17:08:44 +02:00
mesh-admin 23907984a3 Merge pull request 'A short-form port keeps its protocol and still gets its machine port' (#165) from fix/a-short-form-port-keeps-its-protocol into main 2026-09-30 14:58:29 +00:00
jschoubben f0634e11f4 A short-form port keeps its protocol and still gets its machine port
"3478/udp" read as one token was not a port, so it passed through and the
runtime published it wherever it liked: on ace, unifi's STUN and discovery
landed on random machine ports while every TCP pin beside them held. The
protocol is split off, the number is assigned as for any short form, and
the suffix rides along on the outside.
2026-09-30 16:58:23 +02:00
jschoubben 542ce76c0a Merge pull request 'A module may invoke tools, and a manifest is checked where it is written' (#164) from feat/the-console into main
Reviewed-on: #164
2026-09-30 14:46:29 +00:00
jschoubben 2882b5fcb1 A module may invoke tools, and a manifest is checked where it is written
invokes: a manifest word that becomes exactly the publish grant a person's account gets (ADR 0152),
derived by the same composition; refused at parse when it names no tool. module check <file|dir>...
runs what registration runs with no store, for a manifest in any repository (hq issue 148).
2026-09-30 16:12:43 +02:00
jschoubben 481b1a7b05 Merge pull request 'A route's internal name says where the request arrives' (#163) from fix/139-a-routes-internal-name-says-where-it-arrives into main 2026-09-30 12:51:16 +00:00
jschoubben b58578f88d A route's internal name says where the request arrives
novox/hq ADR 0151 (issues 139, 157). <label>.<node>.internal is answered
by every resolver as 'anything under that node goes to that node', so
the node in a route's internal name must be the one whose proxy answers
it; composed under the consumer's own name it sent a client to a machine
with nothing listening whenever the proxy ran elsewhere. Composed under
the serving node now — the same machine wherever the proxy runs beside
the module, so nothing changes on a mesh with one hub.

A routed public name gets no .internal alias any more: the roster
publishes it as itself, once. The alias resolved and nothing served it.
2026-09-30 14:51:12 +02:00
mesh-admin 6cb285dd5c Merge pull request 'A holder by derivation is recorded before an assignment can unsettle it (hq 170)' (#162) from fix/170-a-derived-holder-is-recorded into main 2026-09-30 12:44:14 +00:00
jschoubben 46f324b10b A holder by derivation is recorded before an assignment can unsettle it (hq 170)
`assign ace postgres` made the control plane's own store unresolvable:
postgres's manifest claims mesh-store, nobody was ever recorded as its
holder, and with two eligible assignments and nothing on record both
claimed and both were refused — novox's included. ADR 0110 says the
assignment holds, by a deliberate act; ADR 0131 gave the record its force
but left a seat nobody handed over held by whichever assignment happened
to be alone.

Before acting on an assignment the controller now writes the derived
answer down: every mesh-scoped seat the store knows, resolved to exactly
one holder with nothing on record, gets that holder recorded — the same
record `seat <name> --to <node>/<module>` makes by hand. The next
assignment able to hold the seat then stands beside the holder, eligible
and silent. A seat with two derived claimants is left for a person; a
seat on record is never rewritten; seats the store does not list stay
held by derivation as before. And the refusal, when it still happens,
names the handover that records the holder.

Verified: catalogue tests against the real catalogue; the cmd suite
against a store (the only failure, TestConvergingPreviewsThenChanges…,
fails identically on main).
2026-09-30 14:39:59 +02:00
jschoubben d188eec318 Merge pull request 'No container is given the mesh's names; it resolves them' (#161) from feat/148-names-are-resolved-not-copied into main 2026-09-30 12:38:27 +00:00
jschoubben c978aa7d64 No container is given the mesh's names; it resolves them
novox/hq ADR 0148, step 3. Every container got the whole roster as
--add-host entries at creation and nothing re-read them (issues 109,
135); once the roster was in the digest so that could be caught, one
name moving anywhere replaced every container in the mesh (issue 151).
A container resolves through its machine's resolver, which the resolver
module tells the runtime about once per machine. A module's own hosts
entries stay exactly as declared.

Also brings the resolver tests up to the catalogue as it now is: the
runtime is reloaded (never restarted) and given live-restore, and the
resolver answers by address, not by interface (issue 110).
2026-09-30 14:38:07 +02:00
jschoubben 0c7f42a18a Merge pull request 'Each send is numbered, inside the signed bytes' (#160) from feat/107-each-send-is-numbered into main 2026-09-30 12:09:19 +00:00
jschoubben 9a5584b1b6 Each send is numbered, inside the signed bytes
novox/hq 04-ISSUES/107. The controller already held a per-node lock while
it composed and recorded each send; the order existed and was thrown away
at the wire. Each send now takes the next number for its node, one
higher than the last, under that hold and before the body exists — so
the number is inside what the mesh signs, and a replayed older
declaration cannot borrow a newer one's.

Zero is not sent. A host reads absence as "no order claimed", which is
the shape of every declaration before this, so nothing that worked
before changes for a machine sent nothing since numbering existed.

One subtlety, and it is the one that would have read every machine as
behind for ever: the mesh decides a machine is behind by comparing the
digest of what it WOULD send against what it DID send, and a number
changes the bytes. The read-only comparison composes with the number the
machine was LAST sent, not a fresh one, so it is byte for byte what was
sent when nothing else changed.

Hosts went first and every machine runs one that understands the field.
2026-09-30 14:09:12 +02:00
jschoubben 1bc099a2db Merge pull request 'The linker is told once, not twice' (#159) from fix/161-one-ldflags-not-two into main 2026-09-30 10:54:36 +00:00
jschoubben 3fc1feff38 The linker is told once, not twice
novox/hq 04-ISSUES/161. A repeated flag is not a merged one. The Go
command takes the last -ldflags and drops the first, so passing the
toolchain's flags and then the system stamp as a second one produced a
binary that knew its system and had lost -s -w: 12.2MB against 8.5MB,
with its debug info intact.

My own comment said the linker "accepts and merges" them. It does not,
and I found out by reading the file the build produced rather than by
reading the comment again.

Linker flags are now the toolchain's own list, composed into one flag with
the stamp. A test refuses a compile line that carries -ldflags itself,
because that is what makes two.
2026-09-30 12:54:29 +02:00
jschoubben ffe176f6d1 Merge pull request 'A host the mesh builds knows what it was built for' (#158) from fix/161-a-built-host-knows-what-it-was-built-for into main 2026-09-30 10:40:22 +00:00
jschoubben 8057cc4888 A host the mesh builds knows what it was built for
novox/hq 04-ISSUES/161. The mesh compiled the host, published it,
delivered it, and the launcher started it — and it would have refused the
first declaration it was asked to apply, because it asks which system it
was built for before applying anything and the answer was empty.

The Makefile links that in. The mesh's toolchain deliberately takes
nothing from the module, so it linked in nothing.

The system is the stated exception, and ADR 0142 says why: the target is a
property of the artifact rather than of the recipe, because a compiled
binary is per system and a toolchain accepting it from the module would be
accepting a build instruction. So the toolchain names the variable it
fills and the artifact supplies the value.

Named in the toolchain rather than inferred, and empty for a language
whose output is not pinned to a system — which is every interpreted one,
and a manifest declaring a system for those is already refused.
2026-09-30 12:39:59 +02:00
jschoubben 9d6dad37c5 Merge pull request 'A compiled artifact names the binary a machine will run' (#157) from fix/142-a-compiled-artifact-names-its-binary into main 2026-09-30 09:41:30 +00:00
jschoubben 7f84ddecc5 A compiled artifact names the binary a machine will run
novox/hq 04-ISSUES/142. The name a machine runs a binary by is not always
the name of the package that built it. The host's command is cmd/mesh-host
and every machine runs it as nox-mesh-host — the path it is installed at,
the name in its unit, and the name its launcher looks for inside a
delivered version.

So the first delivered host version landed as `mesh-host`, the host
reported "created /usr/lib/nox-mesh-host/versions/2681d936b949: 1 file(s)",
everything said success, and the launcher would never have seen it. Found
by listing the directory instead of believing the line.

An artifact may now say what its executable is called. Saying nothing
keeps what the compiler would have chosen — the package's name — so
nothing that worked before changes.
2026-09-30 11:41:23 +02:00
jschoubben 94ab9f665e Merge pull request 'A resource can name the version of the build it uses' (#156) from feat/142-a-version-can-reach-a-path into main 2026-09-30 08:04:02 +00:00
jschoubben 3600f2cf16 A resource can name the version of the build it uses
novox/hq 04-ISSUES/142, and the second of the two things ADR 0141's own
insight named: "a version cannot reach the path". A component is unpacked
into a directory named for its version so it can read its own version from
its path — and an archive named a fixed path in the manifest with nothing
interpolating the build into it, so nothing could ask for
.../versions/<version>/ and every machine took a hand-placed fallback.

A resource using an archive or a bundle may now say ${version} in any of
its values. No artifact name in the reference: the resource already says
which artifact it is for, and a second name is a second thing to keep in
step.

The version is the artifact's digest, short, and not the commit. Two
builds of one commit are meant to be the same bytes — every toolchain here
is -trimpath for that reason — so a content-addressed version means an
unchanged build resolves to the path it already had. A commit-named path
would move for an identical binary and recreate everything that reads it.

An image is refused one, with a reason: an image is not unpacked, so it
has no versioned place. Left alone it would reach a machine as literal
text and be created as a directory called ${version}.
2026-09-30 10:03:55 +02:00
jschoubben 005bc16c24 Merge pull request 'A bundle in a compiled language may name which command it builds' (#155) from feat/142-the-mesh-compiles-its-own-go into main 2026-09-30 07:56:18 +00:00
jschoubben 985e2008ba A bundle in a compiled language may name which command it builds
novox/hq 04-ISSUES/142. A bundle is refused if it names what it is built
from, because a bundle is the module's own directory compiled whole and
naming a source would be describing its own build. That reason holds for
an interpreted language and cannot hold for a compiled one.

A repository written in Go carries several commands — the host and its
bootstrap live in one — and "the module's own directory" is then not a
package at all. So a compiled bundle may say which package, and says the
module root by saying nothing. The refusal stands for every interpreted
bundle, which is what it was written for.

Found by writing the host's manifest, which is the first bundle in a
compiled language this mesh has had.
2026-09-30 09:55:54 +02:00
jschoubben bd7ee12938 Merge pull request 'The mesh can compile Go, which is why nothing delivered the host' (#154) from feat/142-the-mesh-compiles-its-own-go into main 2026-09-30 07:49:13 +00:00
jschoubben 0983b00284 The mesh can compile Go, which is why nothing delivered the host
novox/hq 04-ISSUES/142, and ADR 0141's own progressive insight naming
this as the first of two things missing: "nothing can compile it". The
toolchain list was a closed set of typescript and python, whose warning —
every language is another implementation of the contracts modules share —
does not attach to Go. Go is how the host, the control plane and the
builder are written, and none of them is a module in that sense: the host
is what APPLIES modules.

The toolchain names mesh-tools-go as its base rather than pinning an
upstream release here (ADR 0142, 0044): named and not pinned means the
mesh answers with the copy it holds, and moving compiler is a build
instead of an edit to this file.

Two things beyond the list also assumed one language, and both would have
failed after the entry was added:

  sourcesFor turned every entrypoint into a `.ts` file. The extension is
  the toolchain's now — one language's file extension written into the
  code that serves every language is a wall the next one hits.

  The output directory was the compiler's to create. tsc --outDir makes
  one; go build -o writes into a directory and does not make it, failing
  with a message about a path rather than about a build. Made here for
  every toolchain, because which compilers are forgiving is not something
  a reader should have to know.

And a toolchain now says what it is pointed at: a file list from the
module's entrypoints, or the one package the artifact is built `from`.
Pointing `go build` at a file list builds a program out of exactly those
files and ignores the rest of the package — a missing symbol rather than a
legible refusal.

Static and -trimpath: what a machine holds is a file, not a container, so
a binary needing a libc it did not bring is a delivery that works until a
machine differs; and a version comes from where a component sits rather
than from its linker, so two builds of one commit are the same bytes.
2026-09-30 09:48:50 +02:00
jschoubben 8ee2e4d441 Merge pull request 'A commit has no order, so the mesh says who runs what and claims no newer' (#153) from fix/087-a-commit-has-no-order into main 2026-09-30 07:29:47 +00:00
jschoubben 1d9c102889 A commit has no order, so the mesh says who runs what and claims no newer
novox/hq 04-ISSUES/087. The version I shipped this morning said "N
machine(s) run an older host than another machine does" and worked it out
by comparing versions as strings. A host reports its version as a commit.
Commits have no order.

On the live mesh it named the three machines running the NEWER host as the
ones behind: `ced54d4` sorts above `04a27ca` and means nothing. An
arbitrary lexicographic result, presented as a fact, about the one thing
this was built to make trustworthy.

It now reports the split — which machines run which version — and claims
no ordering:

  4 machine(s) do not all run the same host:
    04a27ca      g14, novox, shanks
    ced54d4      ace

    a host refuses a declaration carrying a field it does not know, whole
    — so the mesh may send only what every one of these understands. Which
    of them is newer is not readable from a commit; that needs a version
    the host reports as ordered

More useful as well as more honest: the reader sees who is on which side
of the split, which is what decides whether a field can be sent.

A report that confidently says the opposite of the truth is worse than one
that says less — which is the subject of 04-ISSUES/145, arriving by my own
door within an hour of my closing it.
2026-09-30 09:28:56 +02:00
jschoubben 2542aa67b0 Merge pull request 'The all-well sentence says what it is not a claim about' (#152) from fix/145-the-mesh-says-what-its-report-does-not-cover into main 2026-09-30 07:00:26 +00:00
jschoubben 1da96e8803 The all-well sentence says what it is not a claim about
novox/hq 04-ISSUES/145. "N machine(s), all doing what they were told, all
heard from, running what the mesh would send them, and every module
current with its source" was true for eleven hours of a mesh in which no
module could reach another. An operator read it, and every routine check
they made afterwards — ports from outside, routed services, egress —
passed, because the broken path was module-to-module over the machine's
own name and nothing exercises that.

Every question the sentence answers is about the mesh and a machine
agreeing: applied what it was sent, matches what would be sent, built from
what the source has. None dials a provision, and the mesh composes every
one of those grants itself. So the sentence now says so, in the reader's
way, immediately below it.

This is not the check ADR 0146 describes and does not pretend to be. It
closes the distance between "the machines are as the mesh described them"
and "it works" by naming it, which is where the eleven hours went.

Also: printStatus is separated from the asking, so its exact words can be
read by a test with no store, bus or machine. Those words have been acted
on and been misleading twice — here, and a held module reading as a
machine doing what it was told (04-ISSUES/125) — which makes them the
thing worth holding still.
2026-09-30 08:58:59 +02:00
jschoubben cf2f62cf14 Merge pull request 'The mesh knows which host runs a machine, and says who is behind another' (#151) from fix/087-the-mesh-knows-which-host-runs-a-machine into main 2026-09-30 06:54:59 +00:00
jschoubben 7683ba8b5b The mesh knows which host runs a machine, and says who is behind another
novox/hq 04-ISSUES/087. A host refuses a declaration carrying a field it
does not know, and refuses it WHOLE — deliberately, because that keeps a
half-understood declaration off a machine. It makes every new declaration
field a flag day: hosts first, then the controller. The mesh had no record
of which host any machine ran, so that order was kept by somebody
remembering it, and a machine that refused for this reason reported a
failure with nothing saying why.

The machine has reported its host version since ADR 0141. The
controller's own copy of the report did not have the field, so it was
unmarshalled into nothing and thrown away on arrival. It has it now,
records it, and shows it in `node show` — "not reported" rather than
blank, because a machine that has not said is not a machine running
nothing.

Status says which machines run an older host than another machine does,
and which is newest. Deliberately disagreement rather than staleness:
nothing delivers a host version yet (ADR 0141, accepted and not built), so
the mesh holds no canonical current version and cannot honestly say a
machine is behind THE host. What it can say is that the oldest host in the
mesh is what the mesh may send.

A machine that has reported nothing is left out rather than called
behind. Versions compare as strings, which suits the timestamps and
commits this mesh uses and is wrong for a scheme where "10" sorts before
"9" — said in the code, at the place that would have to learn.
2026-09-30 08:53:59 +02:00
jschoubben a0d7d72a26 Merge pull request 'A held module is in status, and it stops the mesh reading as well' (#150) from fix/125-a-hold-is-a-line-in-the-report into main 2026-09-30 06:47:51 +00:00
jschoubben bfd983e3f8 A held module is in status, and it stops the mesh reading as well
novox/hq 04-ISSUES/125. A module assigned to a machine and never taken
runs none of what it declares. Status had no vocabulary for it: the
machine was heard from, current, and doing what it was told, so the mesh
printed "all doing what they were told" — which was true, and was acted
on, and every public name on the machine went dark.

Status now names each module a machine is holding rather than running,
per machine and with a count, read from what the MACHINE reported rather
than from the mesh's take-time listing — the machine is the only thing
that knows what it found. The JSON form carries the same rows, absent
rather than empty when nothing is held.

And a hold suppresses the all-well sentence, where being adopted does
not: adopted is a mode somebody chose, a module assigned and never taken
is a half-finished action with nothing left to finish it. The condition
is now a named function so the rule lives in one place and a test binds
to the real thing rather than a copy of it.

untakenModules raises a read it cannot make rather than answering "holding
nothing" from a failed query, which is the shape this whole issue is.
2026-09-30 08:46:20 +02:00
jschoubben e7da39de57 Merge pull request 'A consumer a machine is bound to keeps the subject that works' (#148) from fix/156-a-consumer-that-works-is-not-replaced into main 2026-09-29 21:56:53 +00:00
jschoubben e6ddc59cde A consumer a machine is bound to keeps the subject that works
novox/hq 04-ISSUES/156. Issue 146 put the stream into a push consumer's
delivery subject. The server will not move that subject while a
subscriber is bound, and answers `consumer name already in use` — a
message about the name, for a conflict about the subject. A node is bound
to its declaration consumer the whole time it is up: that IS a node
listening. So every node consumer in a running mesh became one the
assertion could not bring to match, and the control plane crash-looped on
the assertion it makes before it serves. A fresh mesh showed nothing,
because nothing was bound.

Kept rather than deleted and re-made. Re-making moves the subject, and a
holder may not be allowed to subscribe to the new one yet: the wider
grant travels in the bus's user list, which this same control plane
composes and a machine applies minutes later. On the live mesh the nodes
are granted `_DELIVER.<node>` and not `_DELIVER.<node>.>`, so re-making
would have silenced every machine — worse than the collision it fixes,
and harder to undo.

Kept rather than fatal, which is what 146's change intended and did not
do. The bare subject still delivers, and collides only where one holder
has two consumers of one name. That is the controller's own pair, and the
controller is not bound to them while it asserts, so those do move.

Also: an existing consumer's deliver policy is carried across rather than
reasserted, because the server refuses to change it and where a consumer
starts is its history.

Two tests against a real server: a consumer with a subscriber bound keeps
its subject, is reported, and still delivers; one with nothing bound
moves, so 146's fix still applies where it matters.
2026-09-29 23:55:56 +02:00
jschoubben 6c5dfd0c25 Merge pull request 'A machine the mesh could not read is not a machine that runs nothing' (#147) from fix/152-a-lookup-failure-is-not-an-absence into main 2026-09-29 21:39:23 +00:00
jschoubben 775df79893 A machine the mesh could not read is not a machine that runs nothing
Three gatherers walk every node and pass over one whose plan will not
compose, so that one broken set does not cost the rest. They read a
plain error to mean that, and so read a store that was briefly
unreachable as a machine running nothing.

On the roster of routed names that is not a degraded answer but a false
one: it states to every machine at once that another machine's names do
not exist. Because the roster is part of every container's identity, a
control node replaced every container it ran — its own store, the
registry, the edge, mail, the bus — on a six-minute cycle for hours. The
loop closed through the store this is read from: each pass restarted it,
the read failed, one name left the roster, and the roster changing is
every container changing.

planFor now marks the two failures that really are the node's own — its
set not composing, and a setting that reaches nothing — and the three
gatherers pass over those and only those. Every other failure is raised,
naming the machine and the read, because a mesh-wide refusal with
nothing named in it is the other way to lose an evening.

novox/hq 04-ISSUES/152, and 151 for why a changed roster is a changed
container.
2026-09-29 23:26:00 +02:00
jschoubben 3fbf658c16 Two consumers may share a name; they must not share a delivery subject
novox/hq 04-ISSUES/146. A push consumer delivers onto an ordinary subject and
everything subscribed to it gets a copy. The controller holds a consumer called
'controller' on CONTROL and another called 'controller' on EVENTS, and both were
given _DELIVER.controller — so the one process, holding both subscriptions,
acted on every message twice.

Measured: one enrolment published, one message in the stream, one delivery, no
redelivery, and the controller enrolled the machine twice — the second minting a
credential that replaced the one the machine had just been handed, which is why
it then reconnected for ever as a user whose password the mesh had rotated. Every
report and every followed event doubled the same way, silently.

The stream goes in the subject because the pair is what identifies a consumer.
A subscriber's permission gains the same shape, keeping the bare name so an
existing consumer keeps working until the next assertion moves it.
2026-09-29 21:32:33 +02:00
jschoubben bc31745607 make image reads its base from the manifest
It was broken and stayed broken: the Dockerfile's fallback base is a Go older
than go.mod asks for, so every hand build died at 'go mod download' with
'go.mod requires go >= 1.26.0'. The pipeline never saw it because the pipeline
passes the declared base in, so the cost fell entirely on whoever built the
image themselves and had to find the digest by hand (novox/hq 04-ISSUES/146).

Read from module.json rather than written here as well, so the two cannot
disagree, and refused outright if the manifest declares none.
2026-09-29 17:45:11 +02:00
jschoubben e5c2eb20f2 A token is an account on the bus, and genesis can place the list
novox/hq 04-ISSUES/146. The composed user list names an enrolment user for
every machine with a live token and nothing minted a credential for it, so the
composer left it out as a user with no password — and every enrolment since the
mesh moved to this bus was refused before the mesh heard of it. The comment
above the issuing code already said the account is created before the token is
handed over; now it is. Recorded rather than minted, because the token's secret
is the password.

And 'broker accounts', which composes the same list the declaration carries and
writes it to standard output. For genesis, where no declaration can reach the
machine running the bus because that machine is not yet a node. It says what it
composed; whoever is raising the machine places it. A control plane that wrote
the file itself would have to learn where the bus keeps its configuration and
how to make it reload, which is the module's knowledge.
2026-09-29 17:36:50 +02:00
jschoubben 05fb7fb5eb Merge pull request 'The mesh makes the bus's certificate itself' (#145) from fix/the-mesh-makes-its-own-bus-certificate into main 2026-09-29 14:06:31 +00:00
jschoubben 2c1733de8d The mesh makes the bus's certificate itself
novox/hq 04-ISSUES/146. The foundation made it by running openssl inside the
broker's image, which worked while the broker was one that carried it and
stopped the day the bus changed: the new one has a shell and no openssl, so
the step exited 127 and no mesh could be raised. No other image the bundle
names has it either, so there was nothing to substitute.

broker certificate --into <dir> writes the pair, --check is the step's verify.
Self-signed on purpose — a host pins this server's exact certificate (ADR
0004) and at genesis there is no authority to ask — and made once, because a
second certificate is one every host that pinned the first no longer believes.
The key is written before the certificate, so an interruption never leaves
something that looks finished.
2026-09-29 15:42:51 +02:00
jschoubben e51c94dcb5 The trust module renders the authority it was bound to
novox/hq ADR 0147. ca-trust carries a script and a unit; the one thing
neither can state is where the authority is, because that is a fact about
the mesh. This checks the rendering — the script fetches from the bound
address and is executable, and the unit runs it both ways, install and
remove. The verification itself is the lab's.
2026-09-29 15:07:33 +02:00
mesh-admin 07c07902ff Merge pull request 'Anything on this machine may call anything on this machine' (#143) from fix/local-calls-are-not-filtered into main 2026-09-29 11:30:34 +00:00
jschoubben 864cdea4c6 Anything on this machine may call anything on this machine
Local is not a boundary this mesh draws. A service here is callable by everything
else here, whatever form either takes — a package with a unit, a binary, a
container. Whether a caller sits in a container was never meant to change the
answer, and the only reason it did was that this chain asked about addresses: a
caller on the machine carries the machine's address, a caller in one of its
containers carries a bridge address, and a rule naming the former silently refused
the latter.

One rule for every service here, replacing the line-per-port added an hour ago,
which only ever covered the ports somebody remembered to think about. The three
reaches are now three lines: on this machine, over the private network, from
anywhere.

The tests assert per chain body, because the forward chain carries the same line in
the same words and an assertion on the whole file passed with the input chain's copy
deleted — which is what ADR 0137's own tests say to do and this file was not doing.
2026-09-29 13:30:32 +02:00
mesh-admin 6e810907b2 Merge pull request 'This machine's own guests are on the private network' (#142) from fix/this-machines-own-guests-are-on-the-private-network into main 2026-09-29 10:30:32 +00:00
jschoubben 2b20a12c4a This machine's own guests are on the private network
A port declared from the mesh admitted the machines' own addresses on the private
network. A container reaching a port on the machine it runs on comes from a bridge,
matching none of them — and where the container runtime routes directly, that packet
is delivered to this machine rather than forwarded, so the forward chain's allowance
never saw it either.

ADR 0100 requires this to work: the store is reachable 'from a container on the node
itself'. It was, through a rule the predecessor left, which allowed the private
ranges wholesale. Converging the machine replaced that with the four overlay
addresses and closed it.

Measured, and it was an outage: every module reaching another by its machine's own
name timed out for eleven hours while the mesh reported the machine healthy. A web
application logged 'connection to server at novox.internal (10.10.0.1), port 6852
failed: timeout expired' throughout.

Asked for by the link it arrives on, for the reason the forward chain no longer
names an address: a range describes one machine and goes stale in silence. A port
open to everything needs no such line.
2026-09-29 12:30:15 +02:00
mesh-admin 9c83dacfce Merge pull request 'A route that names an endpoint still carries that endpoint's port' (#141) from fix/an-endpoint-named-by-a-route-still-carries-its-port into main 2026-09-29 09:55:45 +00:00
jschoubben 64ba053f3b A route that names an endpoint still carries that endpoint's port
Everything downstream reads the port: the provider is told where to reach the
consumer, and the redirection that turns a declared port into the number the
machine published is keyed on it. A route naming only its endpoint left the proxy
with no port at all, and a proxy with no port has nothing to dial.

Caught after the catalogue had already been changed to name endpoints and before
the mesh picked those manifests up, which is the only reason nothing broke: every
module's manifest is behind its source right now, so the plan still renders from
the old shape.

The declared port, not the machine one — the redirection happens later and is keyed
on the declared number, so filling in the machine port here would be redirected
twice or not at all. A route that repeats a port keeps it.
2026-09-29 11:55:28 +02:00
mesh-admin 96416bd8a7 Merge pull request 'Run the real catalogue through the real manifest gate' (#140) from feat/an-assignment-configures-an-endpoint into main 2026-09-29 09:49:29 +00:00
jschoubben 4d2003d77b Run the real catalogue through the real manifest gate
A test that reads every manifest in a catalogue checkout and parses it with the
control plane's own parser, rather than asserting against a fixture: whether the
manifests as written are accepted is the question, and a copy of one proves nothing
about the other seventy-one.

Skipped unless MESH_CATALOGUE names a checkout, so it costs nothing in ordinary
runs and is there when the catalogue changes shape. It also refuses to pass if no
endpoint is named, because a run that validated nothing would otherwise read as
success.
2026-09-29 11:49:22 +02:00
mesh-admin aaad02fd38 Merge pull request 'An assignment configures an endpoint as one thing' (#139) from feat/an-assignment-configures-an-endpoint into main 2026-09-29 09:25:55 +00:00
jschoubben c68d3a7432 An assignment configures an endpoint as one thing
novox/hq ADR 0138, completing it. One block per endpoint instead of three keys
joined by a port number:

  {"endpoints": {"web":    {"port": 20009, "label": "cinema", "reach": "both"},
                 "stream": {"reach": "internal"}}}

Which machine port it lands on, the subdomain a proxy serves it under, and how far
it reaches are the three things an operator says when a module is assigned, and they
were said in ports, in the route's label and in reach — each keyed by the port. A
module with two endpoints of different shapes could only be configured by a reader
who knew which number was which.

Every field is optional; a block that says only a reach leaves the port to the mesh
and the label to the module, which is the ordinary case. A name the module does not
declare is refused, and the refusal lists what it does declare. A port or a reach
said both here and through the older key is refused rather than merged — two places
saying one thing is what this key exists to end, and merging would follow whichever
was read last.

Eight tests. The reach assertion deliberately narrows what the manifest says, because
a reach that agrees with the manifest proves nothing about whether the block was read
— which I found by writing the weaker version first and watching a revert not fail.
2026-09-29 11:25:40 +02:00
mesh-admin a5209bd849 Merge pull request 'A module names its endpoints, and a route names the one it serves' (#138) from feat/a-module-names-its-endpoints into main 2026-09-29 07:28:41 +00:00
jschoubben bdf965dab6 A module names its endpoints, and a route names the one it serves
novox/hq ADR 0138's remaining half, and the words ship one release before any
manifest uses them.

A port number is not a name. Three facts have to be said about an endpoint when a
module is assigned — which machine port it lands on, the subdomain a proxy serves
it under, and how far it reaches — and they were said in three places keyed by the
port. A module with two endpoints of different shapes cannot be configured that way
without a reader joining numbers by hand: a web surface behind the proxy, whose
port only the proxy need reach, and a protocol port clients dial directly because
the client expects that number.

So a listen carries a name, lowercase and unique within the module, and a route
names the endpoint it serves instead of repeating its port. Two endpoints with one
name are refused, because an assignment configuring one would silently configure
whichever the mesh read last. A route naming an endpoint the module does not declare
is refused where it is written rather than resolving to no port and serving nothing.

An unnamed endpoint stays valid and a route repeating a port still resolves, which
is every module in the catalogue today.
2026-09-29 09:28:24 +02:00
mesh-admin b4da20ecc0 Merge pull request 'Reach asks for names on a routed endpoint' (#137) from fix/reach-names-a-route-not-a-port into main 2026-09-29 00:53:28 +00:00
jschoubben 4b33b72160 Reach asks for names on a routed endpoint, and its port stays the manifest's
A routed endpoint's port is how the proxy reaches it and nothing else (ADR 0045):
a public service listens from the mesh, only the proxy reaches it, and it is
exposed by name. So reach on a routed endpoint asks for names, and the port keeps
what the manifest said; on an unrouted one — git over ssh, a mail port, the bus —
it governs the port, because there is no name and the port is the only way in.

Found by trying to express a real module rather than by review: routed name public
because browsers post to it, machine-side port private because it serves a
dashboard in cleartext. Under one value for both there was no way to say it, and
'public' would have reopened a port narrowed an hour earlier.

novox/hq ADR 0138, corrected in place the same day.
2026-09-29 02:50:41 +02:00
mesh-admin d5505fe3d4 Merge pull request 'An assignment says how far an endpoint reaches' (#136) from feat/an-assignment-says-how-far-an-endpoint-reaches into main 2026-09-29 00:47:27 +00:00
jschoubben 264c9e41e9 An assignment says how far an endpoint reaches, and three things read it
novox/hq ADR 0138. Reachability was settled three times over: the filter read a
listen's source with expose able to override it; the proxy composed a public name
and an internal name for every route it was given, because it could; and the
certificate authority followed from which names existed. Each was defensible and
the combination was unstated, so "this endpoint must not be public" could not be
written and was enforced by nothing — while a public certificate for that name was
obtained anyway. Measured on the control node: an identity provider holding a
90-day public certificate and a 24-hour internal one, neither asked for.

`reach` is one value per endpoint, per node — machine, internal, public or both —
and the filter's source and the composed names both follow it. The authority needs
no work: the proxy already asks the public authority for a route's own name and its
internal authority for the internal one, so controlling the names controls the
authority.

Joined by the port, which a route already names: 35 of the catalogue's 36 route
entries name a port the same module declares a listen on, and the one that does not
is a path-level refusal — a rule about a name rather than an endpoint, left alone.

Nothing said composes both names and follows the manifest's `from`, so every mesh
already running is unchanged until an assignment speaks. A port that says both
reach and expose is refused: they say the same thing in different words, and the
filter would follow one while the names followed the other.
2026-09-29 02:47:06 +02:00
mesh-admin 76ac3c99bd Merge pull request 'The filter constrains what arrives from outside, and names no network' (#135) from feat/filter-what-arrives-from-outside into main 2026-09-28 23:01:53 +00:00
jschoubben fe5988c536 The filter constrains what arrives from outside, and names no network
The forward chain blocked everything passing through the machine and then allowed
the machine's own containers back by naming their address ranges: 172.16.0.0/12 and
192.168.128.0/17 fixed here, the rest recorded per machine by 0043. Every way of
keeping that list correct fails — a constant describes one machine, a recorded range
goes stale in silence and cannot tell a network the mesh made from one a predecessor
left behind, and generating it from the modules would put half the rule set on the
machine.

The mesh has no position on a container reaching outward: that is not a port opened
to anybody. So both chains are written around the links traffic arrives on. What did
not arrive from outside is accepted in one line; what did meets the declared rules.
The tunnel is named beside the outward links rather than treated as inside, or a port
nothing declares would be reachable from every machine in the mesh.

A machine that has not reported an outward link is sent no filter and keeps the one
it has, refused where a person reads it rather than as a rule set that will not load.

Removes the two constants, `node networks`, and the column behind it. novox/hq ADR
0140, superseding 0137 and 0139.
2026-09-29 01:01:29 +02:00
mesh-admin ed5d467d90 Merge pull request 'An artifact says which operating system it is built for' (#134) from feat/an-artifact-says-which-system-it-is-built-for into main 2026-09-28 22:54:42 +00:00
jschoubben 228d0226dd An artifact says which operating system it is built for
First of the steps in novox/hq ADR 0142, and it ships alone: a new manifest word
reaches the builder and the controller one release before any manifest uses it.

A toolchain deliberately accepts nothing from the module — anything a module
could override there it would be writing a Dockerfile to override — and yet a
compiled binary is per operating system, pinned at link time so a host refuses to
touch a machine it was not built for (ADR 0005). The way out is that the target
belongs to the artifact: one artifact per system, one build each, recipe still the
mesh's.

A bundle in a language that compiles to a binary must name a system, or it would
be built for whatever the build machine happened to be — which reads as portable
and is not. A bundle in a language that runs anywhere may not name one, because a
system that decides nothing reads as though it did. The list is the host's own
names, not a compiler's: the difference between two of them is a C library rather
than a kernel.

Nothing declares a system yet, and no toolchain compiles to a binary yet, so this
changes no build.
2026-09-29 00:54:27 +02:00
mesh-admin 6215ff0760 Merge pull request 'A machine says which networks it routes, and its filter forwards them' (#133) from feat/a-machine-says-which-networks-it-routes into main 2026-09-28 19:31:06 +00:00
jschoubben 54812306be A machine says which networks it routes, and its filter forwards them
The derived filter denies forwarding by default and then allows the container
runtime's two default pools, named in this code with a comment saying a machine
configured otherwise needs to say so -- and no way to say it. So the filter was
right on a machine using the defaults and silently wrong on any other.

Measured today: flipping a workstation to the derived filter cut egress for five
of its container networks and for every network its test beds create, because
those come from ranges the defaults do not cover. Nothing reported a fault; the
guests just could not reach anything, while the machine reported it had applied
what it was told.

A node-level fact beside the public domain, because the machine routes them and
the module that loads the filter may be replaced. Added to the defaults, never
replacing them. Their guests also keep address and name service, without which a
network does not work at all, and the converge preview now says what a machine
routes instead of leaving it to a sentence about what it cannot preview.
2026-09-28 21:29:10 +02:00
mesh-admin ce9e20fbbc Merge pull request 'A push raises what a module hears, not only a start' (#132) from fix/a-push-raises-what-a-module-hears into main 2026-09-28 14:46:54 +00:00
jschoubben 878690697e A push raises what a module hears, not only a start
A module's declaration and its consumer are derived from the same records, and only one of them
followed a push: the consumers were raised when the control plane started serving, so a module that
gained a `consumes` was sent a declaration it could act on and a consumer that never delivered the
event — with nothing anywhere saying the two disagreed. Found on review: the catalogue's own replay
subscription was recorded, granted and never delivered.

Everything the raise does is idempotent, so a push may do it.
2026-09-28 16:46:42 +02:00
mesh-admin ad97297576 Merge pull request 'The seat declares the facts its holder states' (#131) from fix/the-seat-declares-the-facts-its-holder-states into main 2026-09-28 14:37:05 +00:00
jschoubben 683b1ed693 The seat declares the facts its holder states
The grant permitted the control plane to state what it applied and the seat said nothing about it, so
the check that every derived subscription has an owner found the catalogue subscribing to a subject
nothing publishes — which is exactly the fault that check exists for, pointed at me.

A seat carries the protocol of its role (novox/hq ADR 0129), so the facts are the mesh-controller
seat's `emits`. That is also what lets another module declare it consumes them. No accepts, so no work
queue is raised for the seat — only what its holder may say. The agreement test now holds all three
places to one another: the seat, the grant, and the words the mesh states them with.
2026-09-28 16:36:45 +02:00
mesh-admin 04f9f378b0 Merge pull request 'Two faults found on review' (#130) from fix/review-two-small-faults into main 2026-09-28 14:32:43 +00:00
jschoubben 1c3f44a526 Two faults found on review
A second Accept value would have overwritten the first, because the header was Set per value rather
than Added. One value is all any caller passes today, so nothing was wrong — but a helper that
quietly keeps only the last of what it was given is a trap for whoever passes two.

And a replayed announcement that could not be written was published as an empty body: a fact on the
mesh that says nothing, which the reader can only log and drop. It is now said and skipped, because a
body that cannot be marshalled is this program's fault rather than the bus's.
2026-09-28 16:32:41 +02:00
mesh-admin 89e152dfe2 Merge pull request 'The mesh says what it applied, and the replay has an address it may use' (#129) from feat/the-mesh-says-what-it-applied into main 2026-09-28 14:07:20 +00:00
jschoubben 1ebad3786c The mesh says what it applied, and the replay has an address it may use
The pipeline was observable from a merge to an artifact and went dark where it touched a machine: a
node's report is control traffic only the control plane reads, so nothing said which version a
machine runs, or that it refused to (novox/hq ADR 0134). The control plane now states both under the
seat it holds — a role's events belong to the role and keep their address when the holder is
replaced — and only when the report is news, because a machine reconciles every minute and a fact per
report would be a fact per minute per machine.

Whether a report is news is the store's answer: it holds the previous one, so the listener returns it
and the server states the fact. That also gives the catch-up replay a subject the controller may
publish: it was published as a module's event from a module called "control-plane", which does not
exist, so the controller's own account refused it and every catalogue that asked what it missed was
answered with nothing.
2026-09-28 16:07:18 +02:00
mesh-admin f2f526a60a Merge pull request 'A module's name may contain a dot, so the derived step adds none' (#128) from fix/a-modules-name-may-contain-a-dot into main 2026-09-28 13:44:21 +00:00
jschoubben 4b4c7e0e0d A module's name may contain a dot, so the derived step adds none
A resource's id is `<module>.<its own id>` and a module's name may itself contain a dot — novox.be is
one — so the owner of a resource is everything before the *last* dot. The preparation step's id used a
dot, which made its owner unreadable by that rule; it uses a hyphen, and the id says what it belongs
to whichever way a reader splits it.
2026-09-28 15:44:18 +02:00
mesh-admin cec792ce9d Merge pull request 'A manifest HEAD says what it accepts, or the registry answers 404' (#127) from fix/a-manifest-head-says-what-it-accepts into main 2026-09-28 11:02:10 +00:00
jschoubben 338d033632 A manifest HEAD says what it accepts, or the registry answers 404
The check that skips copying a base the mesh already holds asked with no Accept header, and a
registry answers a manifest only in a media type the caller named: the same digest answered 200 with
the manifest types and 404 without them. So the builder concluded it held nothing, copied every
vendor base again, and exhausted the public hub's pull limit a second time today.

The test could not have caught it, because the fake registry answered a manifest HEAD regardless of
Accept — more permissive than the thing it stands in for. It is now as strict as a real registry, and
fails without the fix.
2026-09-28 13:02:08 +02:00
mesh-admin 1be926cec4 Merge pull request 'The control plane prepares its own state, like any module' (#126) from feat/the-control-plane-prepares-its-own-state into main 2026-09-28 10:51:30 +00:00
jschoubben 2134768dfe The control plane prepares its own state, like any module
Now that every parser on the mesh knows the word, the control plane's manifest says it. Its schema
stops being a special case: the mesh derives the step from its own resource and gates its server on
it, which is the failure of novox/hq 04-ISSUES/133 closed by the mechanism rather than by a
hand-written step in one manifest.
2026-09-28 12:51:27 +02:00
mesh-admin ef825688ee Merge pull request 'The control plane learns 'prepares' one release before its manifest uses it' (#125) from fix/the-word-ships-before-the-manifest-uses-it into main 2026-09-28 10:49:36 +00:00
jschoubben 77a14360df The control plane learns 'prepares' one release before its manifest uses it
A manifest word has to reach every parser before a manifest carries it. The builder refused
mesh-controller's manifest with `unknown field "prepares"` until it was rebuilt; then the running
control plane could not read the manifest in the build result either, so the build was recorded with
no module and the version never moved. Strict parsing is deliberate (novox/hq 04-ISSUES/003), so the
word ships first and a manifest uses it next: this takes `prepares` back out of the control plane's
own manifest, leaving the code that understands it, and the manifest says it again once this is
running everywhere.
2026-09-28 12:49:33 +02:00
mesh-admin 9be2fb4750 Merge pull request 'A version prepares its state before it runs' (#124) from feat/a-version-prepares-its-state into main 2026-09-28 10:43:17 +00:00
jschoubben 5a963aec10 A version prepares its state before it runs
The mesh derives the preparation from the module's own resource instead of each module hand-writing
a step beside it (novox/hq ADR 0135). A manifest says one word — `prepares` — and the mesh runs that
module's own program in its preparation mode, in the module's own context: the same image, the same
environment, the same mounts, because it is the same code. A published port and a fixed address are
taken away rather than copied, since the version being replaced still holds them.

One word for every kind of module: a Go binary receives `prepare` as its argument, a bundle receives
it through the runtime whose entry takes the same word. The control plane answers it like anything
else — its own schema stops being a special case, and its hand-written step is gone.
2026-09-28 12:43:15 +02:00
mesh-admin 45d1c28a28 Merge pull request 'The control plane migrates before it serves' (#123) from fix/the-control-plane-migrates-before-it-serves into main 2026-09-28 08:27:44 +00:00
jschoubben a3e7683c63 The control plane migrates before it serves
The mesh replaced its own control plane with a build carrying a migration, applied none of it, and
then refused every build it recorded for three quarters of an hour while reporting itself healthy
(novox/hq 04-ISSUES/133). The module now declares the step ADR 0052 prescribes: a run-once
`migrate` before the server, re-run whenever the image moves because the image is part of a step's
digest, and gating — a migration that fails stops the new server from starting rather than letting
it serve against a schema it does not have.
2026-09-28 10:27:42 +02:00
mesh-admin da394b45e6 Merge pull request 'Work slower than the window says so, and one address is the bus's' (#122) from fix/work-longer-than-the-window-says-so into main 2026-09-28 07:51:52 +00:00
jschoubben 2f3bfda8c0 Work slower than the window says so, and one address is the bus's
Three faults the mesh's own logs showed this morning. A handler that outlives the acknowledgement
window was handed its message again while it was still working: acting on a merge builds modules,
minutes against a thirty-second window, so one merge ran the whole catalogue five times over. The
transport now says the work is in progress while it runs, which is where the window belongs.

Everything the mesh hands out — a token, a membership, a person's credential — took its address
from the enrolment setting, which on a mesh that has moved still names the broker it moved from:
the first person issued after the move was handed the retired broker's port. There is one bus, and
its address is the one the control plane is connected to.

And `operator issue` documented an argument order its parser refused.
2026-09-28 09:51:50 +02:00
mesh-admin 208388978a Merge pull request 'A merge rebuilds what it changed, and what packages it' (#121) from feat/a-merge-rebuilds-what-it-changed into main 2026-09-28 07:20:03 +00:00
jschoubben aa771616bb A merge rebuilds what it changed, and what packages it
Three faults in one path. A merge rebuilt every module built from the repository, so one change in
a repository holding twenty-six of them meant twenty-six builds. A merge into a repository a module
only *packages* source from rebuilt nothing — two modules are built from the control plane's own
repository and neither had ever been rebuilt when it moved — because the manifest the mesh keeps
carries no build section, so a build now says which repositories it read and the mesh keeps that
beside what it stood on. And a module handed over by hand could record a repository with no
directory inside it, which is a module nothing can ever rebuild (novox/hq 04-ISSUES/131, /132).

A change inside no module's own directory is a change to what they share, and everything built from
that repository is rebuilt: rebuilding too much is the safe direction, because the fault this whole
path exists for is a mesh that believes it is current and is not.
2026-09-28 09:20:01 +02:00
mesh-admin 1513bbaac9 Merge pull request 'An older merge does not move a source' (#120) from fix/an-older-merge-does-not-move-a-source into main 2026-09-28 03:12:40 +00:00
jschoubben 0014984116 An older merge does not move a source
The forge announces what it finds merged, and an old merge surfacing late moved the recorded head
backwards and rebuilt everything built from that repository, once per old merge. A merge made
before the source was last seen is history; one that says nothing about when is taken as news.
The catalogue now carries when each source was last seen. The bus-records test follows #116:
a module's tools are every one under its own name.
2026-09-28 05:12:37 +02:00
mesh-admin d6e49dbd68 Merge pull request 'A base the registry already holds is not pulled from upstream again' (#119) from fix/a-mirrored-base-is-not-pulled-twice into main 2026-09-28 02:48:35 +00:00
jschoubben 3756bb3460 A base the registry already holds is not pulled from upstream again
A base is named by digest, and a digest the mesh's registry holds under the module's repository
is the same bytes whatever upstream would say. Asked on every build, the public hub's anonymous
pull limit was reached on the first merge that rebuilt a whole catalogue, and every module whose
base lives there failed on a copy it did not need.
2026-09-28 04:48:32 +02:00
mesh-admin 60be9c5360 Merge pull request 'A module hears what it consumes: its consumer is raised with the bus, and it pulls it' (#118) from fix/a-module-hears-what-it-consumes into main 2026-09-28 02:29:34 +00:00
jschoubben da31bcb11e A module hears what it consumes: its consumer is raised with the bus, and it pulls it
Every module moved onto the bus by the rollout was issued on the old one, so none had a consumer
waiting; and the grant named a push delivery a runtime's client never binds, while the pull it
does make — asking about its consumer, asking it for messages — was refused. The consumers a
module's declarations imply are now raised whenever the bus is, and the grant is the pull.
2026-09-28 04:29:32 +02:00
mesh-admin f03e7b33c9 Merge pull request 'The controller may ask any module's tool' (#117) from fix/the-controller-may-ask-a-tool into main 2026-09-28 02:21:26 +00:00
jschoubben 0baf727f36 The controller may ask any module's tool
The control plane is the way in for tool calls (novox/hq ADR 0095): a person or an agent asks
through it, so it alone may publish to every module's tool subject. The first ask on the new bus
was refused the publish.
2026-09-28 04:21:25 +02:00
mesh-admin 94dd49a968 Merge pull request 'A module serves every tool under its own name, and may answer' (#116) from fix/a-module-serves-its-own-namespace into main 2026-09-28 02:14:52 +00:00
jschoubben 83a298e7e0 A module serves every tool under its own name, and may answer
Every module that served a tool was refused the subscription on the new bus: the grant listed
tools from a manifest field no module fills, because the tools a module serves are what its code
answers and a second copy of that list would be a second source of truth. The grant is now the
module's own tool namespace; nothing else may subscribe it, a caller is still granted per tool by
name, and a module may answer what it was asked.
2026-09-28 04:14:47 +02:00
mesh-admin 220b79f5cd Merge pull request 'rollout check dials the bus the way the mesh does' (#115) from fix/the-check-dials-as-the-mesh-does into main 2026-09-28 02:05:35 +00:00
jschoubben e62201e227 rollout check dials the bus the way the mesh does
The probe connected bare, and a bus that requires TLS and a user refused it at the handshake —
so the check reported the standing server as absent. It now dials with the controller's own
credential and pin, which is the one fact the check is there to report.
2026-09-28 04:05:32 +02:00
mesh-admin 0ab9b86f0a Merge pull request 'An edge is recorded by path, like the artifact it points at' (#114) from fix/an-edge-is-recorded-by-path into main 2026-09-28 01:52:10 +00:00
jschoubben c7aabd3037 An edge is recorded by path, like the artifact it points at
The test pinned what a build stood on to the address the builder pulled from; the edge names
another module's artifact and is kept the way that artifact is (novox/hq 04-ISSUES/102).
2026-09-28 03:52:08 +02:00
143 changed files with 9890 additions and 546 deletions
+13 -2
View File
@@ -27,8 +27,18 @@ build:
IMAGE ?= mesh-controller:$(VERSION)
DEV_TAG ?= mesh-controller:development
# The base the module declares, read from the manifest rather than written here twice.
#
# **`make image` was broken and stayed broken**, because the Dockerfile's fallback base was a Go
# older than go.mod asks for: every build died at `go mod download` with "go.mod requires go >=
# 1.26.0", and the pipeline never saw it because the pipeline passes the declared base in. Anybody
# building the image by hand hit it and had to find the digest themselves (novox/hq 04-ISSUES/146,
# what it cost).
GO_BASE ?= $(shell python3 -c "import json;print(next(o['image'] for o in json.load(open('module.json'))['build']['on'] if o['arg']=='GO_BASE'))" 2>/dev/null)
image:
docker build --build-arg VERSION=$(VERSION) -t $(IMAGE) -t $(DEV_TAG) .
@test -n "$(GO_BASE)" || { echo "module.json declares no GO_BASE; pass GO_BASE=<image> or fix the manifest"; exit 1; }
docker build --build-arg GO_BASE=$(GO_BASE) --build-arg VERSION=$(VERSION) -t $(IMAGE) -t $(DEV_TAG) .
@echo
@docker image inspect $(IMAGE) --format 'built {{.RepoTags}} {{.Size}} bytes'
@@ -38,7 +48,8 @@ BUILDER_IMAGE ?= mesh-builder:$(VERSION)
BUILDER_DEV_TAG ?= mesh-builder:development
builder-image:
docker build -f cmd/mesh-builder/Dockerfile -t $(BUILDER_IMAGE) -t $(BUILDER_DEV_TAG) .
@test -n "$(GO_BASE)" || { echo "module.json declares no GO_BASE; pass GO_BASE=<image> or fix the manifest"; exit 1; }
docker build --build-arg GO_BASE=$(GO_BASE) -f cmd/mesh-builder/Dockerfile -t $(BUILDER_IMAGE) -t $(BUILDER_DEV_TAG) .
@echo
@docker image inspect $(BUILDER_IMAGE) --format 'built {{.RepoTags}} {{.Size}} bytes'
+25 -11
View File
@@ -148,20 +148,34 @@ func answer(ctx context.Context, publisher builder.Publisher, on, workspace stri
// it either finishes or fails is indistinguishable from one that never arrived — which cost a long
// diagnosis against a running mesh, chasing "the handler never fired" when the truth was only that
// the handler said nothing until the end.
fmt.Fprintf(os.Stderr, "a build request arrived for %s\n", request.Repository)
fmt.Fprintf(os.Stderr, "a build request arrived for %s (%s)\n", request.Repository, request.ID)
// **Everything a build says goes two ways**: to stderr, as always, and onto the bus as the
// role's own events under the build's id (novox/hq ADR 0157) — so whoever asked, and anybody
// watching, reads the same lines this container's log holds, live, and after the fact from the
// stream. Said first, before anything runs, so a build that hangs is one that visibly started.
say := func(step, message string) {
fmt.Fprintf(os.Stderr, " [%s] %s\n", step, message)
work.Say(step, message)
}
builder.Said = say
defer func() { builder.Said = nil }()
if err := work.Began(ctx); err != nil {
fmt.Fprintf(os.Stderr, "cannot say a build started: %v\n", err)
}
result := link.BuildResult{
ID: request.ID, Repository: request.Repository, Path: request.Path,
Ref: request.Ref, On: on,
}
fmt.Fprintf(os.Stderr, "building %s", request.Repository)
what := "building " + request.Repository
if request.Path != "" {
fmt.Fprintf(os.Stderr, " at %s", request.Path)
what += " at " + request.Path
}
if request.Ref != "" {
fmt.Fprintf(os.Stderr, " at %s", request.Ref)
what += " on " + request.Ref
}
fmt.Fprintln(os.Stderr)
say("build", what)
npmrc, err := packagesFrom()
var built builder.Result
@@ -171,16 +185,13 @@ func answer(ctx context.Context, publisher builder.Publisher, on, workspace stri
// after a clone that then fails at npm ci.
built, err = builder.Build(ctx, builder.Command, publisher,
request.Repository, request.Path, request.Ref, workspace, request.Held, npmrc,
forgeFrom(),
func(step, message string) {
fmt.Fprintf(os.Stderr, " [%s] %s\n", step, message)
})
forgeFrom(), say, request.Seats)
}
if err != nil {
// A failure is a result. A build that fails and says nothing is indistinguishable from a
// builder that is not running, and those want completely different responses.
result.Failed = err.Error()
fmt.Fprintf(os.Stderr, " failed: %v\n", err)
say("failed", err.Error())
} else {
manifest, marshalErr := json.Marshal(built.Manifest)
if marshalErr != nil {
@@ -194,7 +205,10 @@ func answer(ctx context.Context, publisher builder.Publisher, on, workspace stri
})
}
result.Against = built.Against
fmt.Fprintf(os.Stderr, " built %s from %s\n", built.Manifest.Module, short(built.Commit))
for _, r := range built.Read {
result.Read = append(result.Read, link.ReadRepository{Repository: r.Repository, Ref: r.Ref})
}
say("built", built.Manifest.Module+" from "+short(built.Commit))
}
}
+10
View File
@@ -106,6 +106,9 @@ func buildOnce(ctx context.Context, args []string) error {
Manifest: built.Manifest,
Against: built.Against,
}
for _, r := range built.Read {
out.Read = append(out.Read, readRepository{Repository: r.Repository, Ref: r.Ref})
}
for _, made := range built.Built {
out.Made = append(out.Made, madeArtifact{Name: made.Name, Kind: made.Kind, Reference: made.Reference})
}
@@ -131,6 +134,13 @@ type onceResult struct {
Manifest any `json:"manifest"`
Made []madeArtifact `json:"made"`
Against []string `json:"against,omitempty"`
Read []readRepository `json:"read,omitempty"`
}
// readRepository is a repository this build read source from besides the module's own.
type readRepository struct {
Repository string `json:"repository"`
Ref string `json:"ref,omitempty"`
}
type madeArtifact struct {
+10
View File
@@ -46,6 +46,13 @@ func assign(ctx context.Context, open *stores, node, module string) (string, err
return "", err
}
defer release()
// **Before the new assignment can unsettle a seat somebody holds only by being alone**
// (novox/hq 04-ISSUES/170): what the mesh derived so far is written down, and then the
// assignment resolves against a record rather than against a coincidence.
settled, err := recordDerivedHolders(ctx, open)
if err != nil {
return "", err
}
fresh, err := open.inventory.Assign(ctx, node, module)
if err != nil {
return "", err
@@ -57,6 +64,9 @@ func assign(ctx context.Context, open *stores, node, module string) (string, err
node, module), nil
}
said := fmt.Sprintf("%s is assigned %s", node, module)
for _, line := range settled {
said += "\n " + line
}
plan, _, err := planFor(ctx, open, node)
if err != nil {
// Kept, and still refused. Both halves are the answer, and the rest of the mesh is still
+7 -4
View File
@@ -17,8 +17,8 @@ import (
var aDigest = "sha256:" + strings.Repeat("e", 64)
// **A build is recorded by digest and path**, whatever address the builder pushed to — and only
// what the build made is rewritten: an image the module runs from elsewhere is left where it says.
// **A build is recorded by digest and path**, whatever address the builder pushed to — what it
// made and what it stood on both; an image the module runs from elsewhere is left where it says.
func TestABuildIsRecordedWithoutTheStoresAddress(t *testing.T) {
manifest, _ := json.Marshal(map[string]any{
"module": "gitea", "version": "1",
@@ -65,8 +65,11 @@ func TestABuildIsRecordedWithoutTheStoresAddress(t *testing.T) {
if strings.Contains(string(kept.Manifest), "anchor.internal:5100") {
t.Errorf("the recorded manifest still carries the store's address:\n%s", kept.Manifest)
}
if kept.Against[0] != "anchor.internal:5100/mesh-tools/runtime@"+aDigest {
t.Errorf("what the build stood on was rewritten: %v", kept.Against)
// What the build stood on is an edge to another module's artifact, and it is recorded the way
// that artifact is: by path in the store, so the edge still names the same thing when the
// store answers at another address.
if kept.Against[0] != catalogue.ArtifactStoreScheme+"mesh-tools/runtime@"+aDigest {
t.Errorf("what the build stood on was recorded by address: %v", kept.Against)
}
}
+1 -1
View File
@@ -88,7 +88,7 @@ func reportsReaching(t *testing.T, open *stores, reachable []link.Reach, held ..
if err := open.inventory.RecordSent(ctx, record.ID, digestOf(body)); err != nil {
t.Fatal(err)
}
if err := (link.Enrolment{Inventory: open.inventory}).Heard(ctx, link.Report{
if _, err := (link.Enrolment{Inventory: open.inventory}).Heard(ctx, link.Report{
Node: "anchor", Applied: []string{"hello-web.x"}, Declared: digestOf(body),
Firewall: "ufw", Held: held, Reachable: reachable,
}); err != nil {
+23 -1
View File
@@ -309,7 +309,7 @@ func converge(ctx context.Context, open *stores, node string, yes bool, digest s
return "", err
}
derived := derivedFilter{rules: rules, foundation: with.Foundation, mesh: with.Mesh,
outward: plan.PublicDomain != ""}
outward: plan.PublicDomain != "", outwardLinks: with.OutwardLinks}
preview, saw := previewOf(node, reported, derived, plan, taken, filter, runs[filter])
preview += "\n\n preview " + saw
if !yes {
@@ -414,6 +414,18 @@ func previewOf(node string, reported inventory.Adoption, derived derivedFilter,
b.WriteString(" not previewed: traffic the machine routes that is not a published port " +
"(a tunnel, NAT in the found firewall) — the derived filter drops it unless a module " +
"declares it\n")
// Which links the filter constrains, said rather than left to the sentence above (novox/hq ADR
// 0140). Everything arriving anywhere else is this machine's own guest and keeps working — which
// is what a reader most wants to know, because the previous shape of this filter cut a machine's
// guests off at the flip without saying so, and that is how this was found.
if len(derived.outwardLinks) > 0 {
b.WriteString(fmt.Sprintf(" it filters what arrives on: %s, and on the private network "+
"— everything its own guests send keeps working\n",
strings.Join(derived.outwardLinks, ", ")))
} else {
b.WriteString(" it has reported no link facing outside, so no filter can be composed " +
"for it — the flip is refused until it reports one\n")
}
isTaken := map[string]bool{}
for _, m := range taken {
@@ -473,6 +485,10 @@ type derivedFilter struct {
// mesh is every address on the private network; outward says the machine faces outside.
mesh []string
outward bool
// outwardLinks is the links this machine reported as facing outside it (novox/hq ADR 0140).
// The filter constrains what arrives on them; everything arriving elsewhere is this machine's
// own guest and is not filtered.
outwardLinks []string
}
// closesOutside is what a narrowing from everywhere to the private network is called: it closes.
@@ -498,6 +514,12 @@ func (d derivedFilter) fate(r inventory.Reach) string {
return "stays open — the mesh's own, from anywhere"
}
}
// This machine's own guests ask it for an address and for names, and those two arrive here
// (novox/hq ADR 0140). Admitted by the link they arrive on, so a listener bound anywhere but an
// outward link keeps answering them.
if (r.Protocol == "udp" && (r.Port == 53 || r.Port == 67)) || (r.Protocol == "tcp" && r.Port == 53) {
return "stays open — this machine's own guests asking it for an address and for names"
}
for _, rule := range d.rules {
if rule.Port != r.Port || rule.Protocol != r.Protocol {
continue
+85 -3
View File
@@ -10,6 +10,8 @@ import (
"strings"
"time"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
@@ -149,6 +151,9 @@ func buildFrom(result link.BuildResult) inventory.Build {
for _, ref := range result.Against {
kept.Against = append(kept.Against, catalogue.Recorded(ref))
}
for _, r := range result.Read {
kept.Read = append(kept.Read, inventory.ReadRepository{Repository: r.Repository, Ref: r.Ref})
}
var announced []inventory.Artifact
for _, made := range result.Made {
announced = append(announced, inventory.Artifact{
@@ -172,10 +177,14 @@ func buildFrom(result link.BuildResult) inventory.Build {
func buildsCommand(ctx context.Context, args []string) error {
set := flag.NewFlagSet("builds", flag.ContinueOnError)
limit := set.Int("n", 20, "how many to show")
logOf := set.String("log", "", "a build's id: print what the build machine said, line by line")
positionals, err := parseAround(set, args)
if err != nil {
return err
}
if *logOf != "" {
return buildLog(ctx, *logOf)
}
module := ""
if len(positionals) == 1 {
module = positionals[0]
@@ -216,8 +225,8 @@ func buildsCommand(ctx context.Context, args []string) error {
if !b.Worked() {
outcome = "failed"
}
fmt.Printf("%-18s %-14s %-10s %s\n",
what, outcome, b.On, b.At.Local().Format("2006-01-02 15:04"))
fmt.Printf("%-18s %-14s %-10s %s %s\n",
what, outcome, b.On, b.At.Local().Format("2006-01-02 15:04"), b.ID)
fmt.Printf(" %s", b.Repository)
if b.Ref != "" {
fmt.Printf(" at %s", b.Ref)
@@ -405,11 +414,14 @@ func buildOne(ctx context.Context, source buildSource, path, ref string, wait ti
Path: path,
Ref: ref,
Held: heldBy(ctx),
Seats: seatBases(ctx),
}
fmt.Printf("asked for %s", source)
if source.Seat != "" {
fmt.Printf(" (%s)", repository)
}
// The id is how a person follows this build while it runs: `builds --log <id>`.
fmt.Printf(" as %s", request.ID)
if path != "" {
fmt.Printf(" at %s", path)
}
@@ -475,6 +487,12 @@ func buildOne(ctx context.Context, source buildSource, path, ref string, wait ti
if source.Seat != "" {
recorded.Repository, recorded.Seat = source.Repository, source.Seat
}
// The build is kept; the module is not. A definition naming an installation is refused where
// it would enter the catalogue, and the build log says which build it was.
if err := namesNoInstallation(manifest); err != nil {
return fmt.Errorf("%s built %s (%s), and the mesh does not register it: %w",
result.On, result.Repository, short(result.Commit), err)
}
if err := inv.RegisterModule(ctx, manifest, recorded); err != nil {
return err
}
@@ -510,7 +528,7 @@ func buildAndShow(ctx context.Context, source buildSource, path, ref string, wai
result, err := ask.Submit(ctx, link.BuildRequest{
ID: fmt.Sprintf("%s-%d", "build", time.Now().UnixNano()),
Repository: repository, Path: path, Ref: ref,
Held: heldBy(ctx),
Held: heldBy(ctx), Seats: seatBases(ctx),
}, wait)
if err != nil {
return err
@@ -553,6 +571,16 @@ type answers struct {
// a consequence of the refusals above: a node that does not resolve is not on the network, and
// a mesh whose hub is that node has no hub.
network string
// untaken is, per machine, each assigned module whose resources the machine is holding as it
// found them, and how many — a module that was assigned, sent, and is running none of what it
// declares because nothing has taken it (novox/hq ADR 0100, 04-ISSUES/125).
//
// **Its absence cost an outage.** The module was assigned, the push reported success, this
// command said the machine was doing everything it was told, and the module's three containers
// did not exist. On the strength of those reports the predecessor's proxy was stopped and every
// public name on the machine went dark. The holds were correct; they were recorded only in the
// machine's own state file, and the one visible symptom was a count that did not add up.
untaken map[string]map[string]int
}
// heldBy is every artifact this mesh has built, for a build that may need one as its base.
@@ -605,3 +633,57 @@ func askOver(_ *link.Server) (link.Builders, error) {
}
return link.BuildsOverNATS(address)
}
// buildLog prints everything a build machine said about one build, read back from the bus.
//
// **From the stream, not from a record** (novox/hq ADR 0157). A build's lines are the role's own
// events under the build's id, retained with every other event; the mesh keeps no second copy. Read
// with a consumer of its own that is gone when this returns, so nothing accumulates in the server
// for the reading, and filtered by subject, so one build's lines are all that travel.
func buildLog(ctx context.Context, id string) error {
address, err := broker.BusAddress()
if err != nil {
return err
}
js, err := broker.Dial(address)
if err != nil {
return fmt.Errorf("cannot reach the bus to read a build's log: %w", err)
}
defer js.Close()
sub, err := js.Context().PullSubscribe(link.BuildLog(id), "",
nats.BindStream(broker.EventsStream), nats.DeliverAll(), nats.AckNone())
if err != nil {
return fmt.Errorf("cannot read %s from the bus: %w", link.BuildLog(id), err)
}
defer func() { _ = sub.Unsubscribe() }()
printed := 0
for {
batch, err := sub.Fetch(200, nats.MaxWait(2*time.Second))
if err != nil && !errors.Is(err, nats.ErrTimeout) && !errors.Is(err, context.DeadlineExceeded) {
return fmt.Errorf("reading a build's log: %w", err)
}
for _, msg := range batch {
var line link.BuildLine
if err := json.Unmarshal(msg.Data, &line); err != nil {
fmt.Printf(" ? %s\n", string(msg.Data))
continue
}
at := line.At
if t, err := time.Parse(time.RFC3339Nano, line.At); err == nil {
at = t.Local().Format("15:04:05")
}
fmt.Printf("%s %4d [%s] %s\n", at, line.Seq, line.Step, line.Message)
printed++
}
if len(batch) < 200 {
break
}
}
if printed == 0 {
fmt.Printf("nothing on the bus for build %s: no build by that id in the last week, or a build "+
"machine older than this that said nothing while building\n", id)
}
return nil
}
+258
View File
@@ -0,0 +1,258 @@
package main
import (
"context"
"crypto/rand"
"crypto/rsa"
"crypto/x509"
"crypto/x509/pkix"
"encoding/pem"
"errors"
"fmt"
"math/big"
"net"
"os"
"path/filepath"
"strings"
"time"
"github.com/novox/mesh-controller/internal/broker"
)
// The bus's own certificate, made by the mesh rather than borrowed from an image.
//
// **The foundation asked a third-party image for a tool it never said must be there** (novox/hq
// 04-ISSUES/146). The bootstrap made this certificate by running `openssl` inside the broker's
// image, which worked while the broker was one that happened to carry it and stopped the day the
// bus changed: the new one has a shell and no openssl, so the step exited 127 and no mesh could be
// raised. Substituting another image the bundle names does not help — none of them carry it
// either.
//
// So the program that needs a certificate makes one. It is the mesh's own binary, already on the
// machine at this point in the bootstrap (the schema step ran it), and it needs nothing from the
// image it writes into but a mounted directory.
//
// **Self-signed, and that is the design** — a host pins this server's exact certificate and
// authenticates with a password (novox/hq ADR 0004). There is no authority above it to ask, and at
// this moment in a bootstrap there is no mesh to ask one of.
//
// Idempotent, because the step is applied again on every reconcile and a second certificate would
// be one the hosts that pinned the first no longer believe.
// busCertificateNames is what the bus is reached by: the container name on a mesh network, and the
// loopback address the machine's own foundation dials.
var busCertificateNames = []string{"mesh-broker"}
const busCertificateLife = 10 * 365 * 24 * time.Hour
// busCertificate makes the bus's certificate in a directory, or says whether one is there.
//
// broker certificate --into /tls make it if it is not there
// broker certificate --check --into /tls exit non-zero unless a usable pair is
func busCertificate(args []string) error {
into, check := "", false
for i := 0; i < len(args); i++ {
switch args[i] {
case "--check":
check = true
case "--into":
if i+1 >= len(args) {
return errors.New("--into needs a directory")
}
into = args[i+1]
i++
default:
return fmt.Errorf("broker certificate [--check] --into <directory>: %q", args[i])
}
}
if into == "" {
return errors.New("broker certificate [--check] --into <directory>")
}
crt, key := filepath.Join(into, "tls.crt"), filepath.Join(into, "tls.key")
if usable, err := busCertificateUsable(crt, key); err != nil {
return err
} else if usable {
fmt.Printf("the bus already has a certificate at %s, and it was left alone\n", crt)
return nil
}
if check {
// Said as a failure, because that is what the caller asked: a bootstrap's verify runs
// this and a false answer is what makes the step run.
return fmt.Errorf("no usable certificate and key at %s", into)
}
return writeBusCertificate(crt, key)
}
// busCertificateUsable says whether a certificate and its key are both there and parse.
//
// Both, and parsed rather than stat'ed: a half-written pair is the state a bootstrap interrupted
// between the two files leaves behind, and a step that treated it as done would hand the server a
// certificate with no key and report success.
func busCertificateUsable(crt, key string) (bool, error) {
certPEM, err := os.ReadFile(crt)
if errors.Is(err, os.ErrNotExist) {
return false, nil
}
if err != nil {
return false, err
}
keyPEM, err := os.ReadFile(key)
if errors.Is(err, os.ErrNotExist) {
return false, nil
}
if err != nil {
return false, err
}
if _, err := tlsPairParses(certPEM, keyPEM); err != nil {
return false, nil
}
return true, nil
}
func tlsPairParses(certPEM, keyPEM []byte) (*x509.Certificate, error) {
block, _ := pem.Decode(certPEM)
if block == nil || block.Type != "CERTIFICATE" {
return nil, errors.New("not a certificate")
}
certificate, err := x509.ParseCertificate(block.Bytes)
if err != nil {
return nil, err
}
keyBlock, _ := pem.Decode(keyPEM)
if keyBlock == nil {
return nil, errors.New("not a key")
}
if _, err := x509.ParsePKCS8PrivateKey(keyBlock.Bytes); err != nil {
if _, err := x509.ParsePKCS1PrivateKey(keyBlock.Bytes); err != nil {
return nil, err
}
}
return certificate, nil
}
func writeBusCertificate(crt, key string) error {
private, err := rsa.GenerateKey(rand.Reader, 2048)
if err != nil {
return err
}
serial, err := rand.Int(rand.Reader, new(big.Int).Lsh(big.NewInt(1), 128))
if err != nil {
return err
}
template := &x509.Certificate{
SerialNumber: serial,
Subject: pkix.Name{CommonName: busCertificateNames[0]},
DNSNames: busCertificateNames,
IPAddresses: []net.IP{net.ParseIP("127.0.0.1")},
NotBefore: time.Now().Add(-time.Hour),
NotAfter: time.Now().Add(busCertificateLife),
KeyUsage: x509.KeyUsageDigitalSignature | x509.KeyUsageKeyEncipherment,
ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth},
BasicConstraintsValid: true,
}
der, err := x509.CreateCertificate(rand.Reader, template, template, &private.PublicKey, private)
if err != nil {
return err
}
pkcs8, err := x509.MarshalPKCS8PrivateKey(private)
if err != nil {
return err
}
// **The key first, and only then the certificate**, so the pair a reader finds is never a
// certificate whose key has not been written yet — the one order in which an interruption
// leaves something that looks finished (novox/hq 04-ISSUES/014, a key present and unusable).
if err := os.WriteFile(key, pem.EncodeToMemory(&pem.Block{Type: "PRIVATE KEY", Bytes: pkcs8}), 0o600); err != nil {
return err
}
if err := os.WriteFile(crt, pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: der}), 0o644); err != nil {
return err
}
fmt.Printf("made the bus a certificate for %v, valid until %s\n %s\n %s\n",
busCertificateNames, template.NotAfter.Format(time.RFC3339), crt, key)
return nil
}
// busAccounts writes the mesh's composed user list to a file.
//
// **For genesis, where no declaration can deliver it** (novox/hq 04-ISSUES/146). Everywhere else
// the list reaches the machine running the bus as a resource of the module that holds it — which
// requires that machine to be an enrolled node, and at genesis it is not: the first node cannot
// enrol because the account it would enrol with cannot be composed onto a bus it has no declaration
// for. The installer breaks that circle by placing the file itself, once, and the module takes the
// file over from its first push.
//
// The same composition, not a second one: this asks the store for the same records and renders them
// with the same composer the declaration uses. A genesis that hand-wrote an account would be a
// second statement of who may say what, able to disagree with the first.
//
// **It writes to standard output unless told a file**, and that is the point: the control plane
// composes and says what it composed, and whoever is raising the machine puts it where that
// machine's bus reads it. A control plane that wrote into the bus's own directory would have to
// know where that is and how to make the server re-read it — which is the module's knowledge, and
// the module is what takes this over on the first push.
//
// broker accounts > /var/lib/mesh-bus-conf/accounts.conf
func busAccounts(ctx context.Context, args []string) error {
into := ""
for i := 0; i < len(args); i++ {
switch args[i] {
case "--into":
if i+1 >= len(args) {
return errors.New("--into needs a file")
}
into = args[i+1]
i++
default:
return fmt.Errorf("broker accounts --into <file>: %q", args[i])
}
}
open, err := openStores(ctx)
if err != nil {
return err
}
defer open.Close()
records, err := open.inventory.BusRecords(ctx)
if err != nil {
return err
}
users, err := broker.Users(records)
if err != nil {
return err
}
kept, err := open.inventory.BusUsers(ctx)
if err != nil {
return err
}
hashes := make(map[string]string, len(kept))
for name, u := range kept {
hashes[name] = u.PasswordHash
}
filled, missing := broker.WithPasswords(users, hashes)
if len(missing) > 0 {
// To standard error, always: the composed file may be going to standard output, and a
// remark in the middle of it is a configuration the server refuses to parse.
fmt.Fprintf(os.Stderr, "leaving out %d user(s) the mesh has minted no credential for: %s\n",
len(missing), strings.Join(missing, ", "))
}
if len(filled) == 0 {
return errors.New("not one user has a credential, so this list would refuse every " +
"connection in the mesh")
}
accounts, err := broker.ComposeAccounts(filled)
if err != nil {
return err
}
if into == "" {
fmt.Print(accounts)
return nil
}
if err := os.WriteFile(into, []byte(accounts), 0o600); err != nil {
return err
}
fmt.Printf("wrote %d user(s) to %s\n", len(filled), into)
return nil
}
+121
View File
@@ -0,0 +1,121 @@
package main
import (
"crypto/tls"
"crypto/x509"
"os"
"path/filepath"
"strings"
"testing"
)
// novox/hq 04-ISSUES/146. The bootstrap could not make the bus a certificate: it asked an image for
// `openssl` and the image it asks has none. What replaces it is this command, so what is checked is
// what the bootstrap needs from it — a pair a TLS server can actually load, made once and only once.
func TestTheBusCertificateLoadsAsAServersWould(t *testing.T) {
into := t.TempDir()
if err := busCertificate([]string{"--into", into}); err != nil {
t.Fatalf("the bus could not be given a certificate: %v", err)
}
// The check a cheaper test would not make. The key was present and valid and the server could
// not start, once, because nothing loaded the pair the way a server loads it
// (novox/hq 04-ISSUES/014).
pair, err := tls.LoadX509KeyPair(filepath.Join(into, "tls.crt"), filepath.Join(into, "tls.key"))
if err != nil {
t.Fatalf("a TLS server cannot load what was written: %v", err)
}
leaf := pair.Leaf
if leaf == nil {
if leaf, err = x509.ParseCertificate(pair.Certificate[0]); err != nil {
t.Fatal(err)
}
}
if err := leaf.VerifyHostname("mesh-broker"); err != nil {
t.Errorf("the certificate is not for the name the bus is reached by: %v", err)
}
if len(leaf.IPAddresses) == 0 || leaf.IPAddresses[0].String() != "127.0.0.1" {
t.Errorf("the certificate does not cover the loopback address the foundation dials: %v", leaf.IPAddresses)
}
// The key is not readable by anything else on the machine; the certificate is public and is.
key, err := os.Stat(filepath.Join(into, "tls.key"))
if err != nil {
t.Fatal(err)
}
if key.Mode().Perm() != 0o600 {
t.Errorf("the key is %v", key.Mode().Perm())
}
crt, err := os.Stat(filepath.Join(into, "tls.crt"))
if err != nil {
t.Fatal(err)
}
if crt.Mode().Perm() != 0o644 {
t.Errorf("the certificate is %v, which the server runs as another user cannot read", crt.Mode().Perm())
}
}
// **Made once.** The step is applied again on every reconcile, and a second certificate is one the
// hosts that pinned the first no longer believe (novox/hq ADR 0004).
func TestTheBusCertificateIsMadeOnce(t *testing.T) {
into := t.TempDir()
if err := busCertificate([]string{"--into", into}); err != nil {
t.Fatal(err)
}
first, err := os.ReadFile(filepath.Join(into, "tls.crt"))
if err != nil {
t.Fatal(err)
}
if err := busCertificate([]string{"--into", into}); err != nil {
t.Fatal(err)
}
again, err := os.ReadFile(filepath.Join(into, "tls.crt"))
if err != nil {
t.Fatal(err)
}
if string(first) != string(again) {
t.Fatal("running it twice replaced the certificate every host had pinned")
}
}
// The verify half: false before, true after, which is what makes the bootstrap run the step at all.
func TestTheCheckIsFalseUntilThereIsAPair(t *testing.T) {
into := t.TempDir()
if err := busCertificate([]string{"--check", "--into", into}); err == nil {
t.Fatal("an empty directory reported a usable certificate")
}
if err := busCertificate([]string{"--into", into}); err != nil {
t.Fatal(err)
}
if err := busCertificate([]string{"--check", "--into", into}); err != nil {
t.Fatalf("the certificate it just made does not satisfy its own check: %v", err)
}
}
// A half-written pair is not a pair. An interrupted bootstrap leaves exactly this, and a step that
// called it done would hand the server a certificate with no key and report success.
func TestACertificateWithoutItsKeyIsNotUsable(t *testing.T) {
into := t.TempDir()
if err := busCertificate([]string{"--into", into}); err != nil {
t.Fatal(err)
}
if err := os.Remove(filepath.Join(into, "tls.key")); err != nil {
t.Fatal(err)
}
if err := busCertificate([]string{"--check", "--into", into}); err == nil {
t.Fatal("a certificate with no key passed the check")
}
if err := busCertificate([]string{"--into", into}); err != nil {
t.Fatal(err)
}
if _, err := tls.LoadX509KeyPair(filepath.Join(into, "tls.crt"), filepath.Join(into, "tls.key")); err != nil {
t.Fatalf("it did not replace the unusable pair: %v", err)
}
}
func TestWhereToWriteIsRequired(t *testing.T) {
if err := busCertificate(nil); err == nil || !strings.Contains(err.Error(), "--into") {
t.Fatalf("it did not ask where to write: %v", err)
}
}
+135
View File
@@ -0,0 +1,135 @@
package main
import (
"errors"
"fmt"
"io"
"os"
"path/filepath"
"sort"
"github.com/novox/mesh-controller/internal/catalogue"
)
// moduleCheck judges manifests where they are written, with no mesh (novox/hq ADR 0037, issue 148).
//
// **The same functions registration runs, and nothing the command line adds** (ADR 0035): the strict
// parse with every per-manifest problem, then the rules no single manifest can be judged against,
// over exactly the manifests given. Somebody describing their own application in their own
// repository runs this before pushing and finds out there, rather than when a running mesh refuses
// the registration or, later, when a machine applies something that resolved and should not have.
//
// **What it cannot know without a store, it says.** The mesh's own seat set is the store's (ADR
// 0122); this binary carries a compiled copy that the store overrides when loaded, so a claim on a
// mesh seat is judged fully only at registration. A seat another module declares is unknown unless
// that module's manifest is passed too. Both are printed as a note, not as a problem — a check that
// refused what it could not see would teach people to ignore it.
func moduleCheck(paths []string, out io.Writer) error {
if len(paths) == 0 {
return errors.New("module check <manifest.json>... — one file per module; pass every " +
"manifest of a repository together so the rules between them are checked too")
}
shelf := catalogue.Shelf{}
faulted := map[string]bool{}
failed := 0
for _, path := range paths {
raw, err := os.ReadFile(path)
if err != nil {
fmt.Fprintf(out, "%s: %v\n", path, err)
failed++
continue
}
m, err := catalogue.ParseManifest(raw)
if err != nil {
fmt.Fprintf(out, "%s: %v\n", path, err)
failed++
continue
}
if first, twice := shelf[m.Module]; twice {
_ = first
fmt.Fprintf(out, "%s: %s was already given; two manifests name one module\n", path, m.Module)
failed++
continue
}
// A definition names no installation (novox/hq ADR 0112, ADR 0155): judged here, in the
// catalogue-wide test, and at registration, which refuses in the same words.
if named := catalogue.InstallationProblems(m); len(named) > 0 {
for _, p := range named {
fmt.Fprintf(out, "%s: %s\n", path, p)
}
failed += len(named)
faulted[m.Module] = true
}
shelf[m.Module] = m
}
// Between the manifests: a seat declared twice, a use of a seat nothing declares, a claim on
// a seat that does not exist. Run only over what parsed, because a problem inside one manifest
// has already been said and would be said again here in a worse form.
problems := catalogue.CatalogueProblems(shelf)
sort.Strings(problems)
for _, p := range problems {
fmt.Fprintln(out, p)
}
failed += len(problems)
var names []string
for name := range shelf {
names = append(names, name)
}
sort.Strings(names)
for _, name := range names {
m := shelf[name]
if faulted[name] {
continue
}
fmt.Fprintf(out, "%s: ok", name)
if n := len(m.Tools); n > 0 {
fmt.Fprintf(out, ", %d tool(s)", n)
}
if len(m.Invokes) > 0 {
fmt.Fprintf(out, ", invokes %s", joinInvokes(m.Invokes))
}
fmt.Fprintln(out)
}
if failed > 0 {
return fmt.Errorf("%d problem(s) in %d manifest(s)", failed, len(paths))
}
fmt.Fprintf(out, "%d manifest(s) checked. Judged against the seats this binary carries; a claim on "+
"one of the mesh's own seats is judged fully at registration, and a seat declared by a "+
"module not given here reads as unknown\n", len(paths))
return nil
}
func joinInvokes(invokes []string) string {
if len(invokes) == 1 && invokes[0] == "*" {
return "every tool"
}
s := ""
for i, t := range invokes {
if i > 0 {
s += ", "
}
s += t
}
return s
}
// manifestsUnder lists every module.json below a directory, for `module check <dir>`.
func manifestsUnder(dir string) ([]string, error) {
var found []string
err := filepath.WalkDir(dir, func(path string, d os.DirEntry, err error) error {
if err != nil {
return err
}
if d.IsDir() && (d.Name() == "node_modules" || d.Name() == ".git" || d.Name() == "dist") {
return filepath.SkipDir
}
if !d.IsDir() && d.Name() == "module.json" {
found = append(found, path)
}
return nil
})
sort.Strings(found)
return found, err
}
+94
View File
@@ -0,0 +1,94 @@
package main
import (
"bytes"
"os"
"path/filepath"
"strings"
"github.com/novox/mesh-controller/internal/catalogue"
"testing"
)
// The check anybody can run is the check registration runs (novox/hq issue 148, ADR 0037): a manifest
// with a known fault is named, and one without passes, with no store opened.
func TestModuleCheckNamesAFaultAndNeedsNoMesh(t *testing.T) {
dir := t.TempDir()
good := filepath.Join(dir, "good.json")
bad := filepath.Join(dir, "bad.json")
os.WriteFile(good, []byte(`{"module":"shop","version":"1","tools":["price"],"invokes":["mesh-catalog.catalog_modules"]}`), 0o600)
os.WriteFile(bad, []byte(`{"module":"till","version":"1","invokes":["shop"]}`), 0o600)
var out bytes.Buffer
if err := moduleCheck([]string{good}, &out); err != nil {
t.Fatalf("a sound manifest was refused: %v\n%s", err, out.String())
}
if !strings.Contains(out.String(), "shop: ok, 1 tool(s), invokes mesh-catalog.catalog_modules") {
t.Fatalf("the report does not say what it checked:\n%s", out.String())
}
out.Reset()
err := moduleCheck([]string{good, bad}, &out)
if err == nil {
t.Fatal("a manifest invoking a module and no tool passed")
}
if !strings.Contains(out.String(), `till invokes "shop", which does not name a tool`) {
t.Fatalf("the fault is not named in the manifest's words:\n%s", out.String())
}
}
// The rules between manifests run over what was given together: a seat two modules declare is
// refused, which no single-manifest check can see.
func TestModuleCheckJudgesBetweenTheManifestsGiven(t *testing.T) {
dir := t.TempDir()
a := filepath.Join(dir, "a.json")
b := filepath.Join(dir, "b.json")
os.WriteFile(a, []byte(`{"module":"a","version":"1","seats":[{"name":"printer","scope":"mesh"}]}`), 0o600)
os.WriteFile(b, []byte(`{"module":"b","version":"1","seats":[{"name":"printer","scope":"mesh"}]}`), 0o600)
var out bytes.Buffer
if err := moduleCheck([]string{a, b}, &out); err == nil {
t.Fatalf("two declarations of one seat passed:\n%s", out.String())
}
if !strings.Contains(out.String(), "a seat name means one protocol") {
t.Fatalf("the cross-manifest rule was not the one named:\n%s", out.String())
}
}
// The real catalogue passes the command, the way it passes the test that used to be the only check.
func TestModuleCheckPassesTheCatalogue(t *testing.T) {
root := filepath.Join("..", "..", "..", "mesh-catalog", "modules")
if _, err := os.Stat(root); err != nil {
t.Skipf("catalogue sibling not present: %v", err)
}
paths, err := manifestsUnder(root)
if err != nil || len(paths) == 0 {
t.Fatalf("no manifests under %s: %v", root, err)
}
var out bytes.Buffer
if err := moduleCheck(paths, &out); err != nil {
t.Fatalf("the catalogue does not pass its own check: %v\n%s", err, out.String())
}
}
func TestRegistrationRefusesADefinitionNamingAnInstallation(t *testing.T) {
// novox/hq ADR 0155: the check moves to registration once the catalogue passes it. Both
// ways in — `module add` and a build's result — go through this, and a name declared on
// purpose passes with its reason.
named := catalogue.Manifest{Module: "idp", Resources: []map[string]any{
{"id": "server", "type": "container", "image": "x@sha256:aa",
"env": map[string]any{"KC_HOSTNAME": "https://login.mesh-one.be"}},
}}
err := namesNoInstallation(named)
if err == nil || !strings.Contains(err.Error(), "login.mesh-one.be") ||
!strings.Contains(err.Error(), catalogue.NamesOnPurpose) {
t.Fatalf("a definition naming an installation is refused with the name and the way out; got %v", err)
}
meant := catalogue.Manifest{Module: "site", Resources: []map[string]any{
{"id": "server", "type": "container", "image": "registry.mesh-one.be/org/site@sha256:cc",
catalogue.NamesOnPurpose: map[string]any{
"registry.mesh-one.be": "built outside the mesh until its repository is a build source here"}},
}}
if err := namesNoInstallation(meant); err != nil {
t.Fatalf("a name declared on purpose passes; got %v", err)
}
}
+92
View File
@@ -0,0 +1,92 @@
package main
import (
"context"
"fmt"
"sort"
"github.com/novox/mesh-controller/internal/catalogue"
)
// recordDerivedHolders writes down who holds each mesh-scoped seat that nobody was ever recorded
// as holding.
//
// **A seat held by derivation is a seat held by accident of being alone** (novox/hq
// 04-ISSUES/170). ADR 0131 lets a holder on record settle a seat, and lets any other assignment
// whose module could hold it stand beside the holder, eligible and silent. But a seat nobody
// ever handed over has no record, so its holder is whichever assignment happened to be the sole
// claimant — and the day a second one is assigned, both claim, both are refused, and the first
// one's whole machine stops resolving. That is what assigning a second postgres did to the
// control plane's own store.
//
// So the mesh writes the derived answer down before it acts on an assignment: for every
// mesh-scoped seat with exactly one resolved holder and nothing on record, that holder is
// recorded as the standing one — the same record `seat <name> --to <node>/<module>` makes by
// hand, made from what the mesh already resolved. A seat with two derived claimants is left
// alone: that is the ambiguity a person settles, and recording either would be guessing.
//
// Node-scoped seats are untouched: a record is one holder per seat, and a node-scoped seat has
// one holder per machine (ADR 0121), so there is nothing for a record to settle there.
func recordDerivedHolders(ctx context.Context, open *stores) ([]string, error) {
inv := open.inventory
shelf, err := inv.Catalogue(ctx)
if err != nil {
return nil, err
}
// exclude nobody: every node's claims, resolved with the holdings on record.
world, err := theRestOfTheMesh(ctx, inv, shelf, "")
if err != nil {
return nil, err
}
recorded, err := inv.Holdings(ctx)
if err != nil {
return nil, err
}
// A record is a row against a seat the store knows. A seat it does not — a mesh whose seats
// were never seeded, a seat a module declares for itself — stays held by derivation, as it
// always was; a missing row is not a reason an assignment fails.
known, err := inv.Seats(ctx)
if err != nil {
return nil, err
}
recordable := map[string]bool{}
for _, s := range known {
recordable[s.Name] = true
}
onRecord := map[string]bool{}
for _, h := range recorded {
if s, ok := catalogue.SeatNamed(h.Claim); ok {
onRecord[s.Name] = true
}
}
holders := map[string][]catalogue.Held{}
for _, h := range world.Held {
if h.Scope != catalogue.ScopeMesh {
continue
}
s, ok := catalogue.SeatNamed(h.Claim)
if !ok || onRecord[s.Name] || !recordable[s.Name] {
continue
}
holders[s.Name] = append(holders[s.Name], h)
}
names := make([]string, 0, len(holders))
for name := range holders {
names = append(names, name)
}
sort.Strings(names)
var said []string
for _, name := range names {
if len(holders[name]) != 1 {
continue
}
h := holders[name][0]
if err := inv.HoldSeat(ctx, name, catalogue.ScopeMesh, h.Node, h.Module); err != nil {
return said, err
}
said = append(said, fmt.Sprintf(
"recorded %s on %s as the standing holder of %s, which it held only by being alone",
h.Module, h.Node, name))
}
return said, nil
}
+110
View File
@@ -0,0 +1,110 @@
package main
import (
"strings"
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
)
// A seat nobody ever handed over is held by whichever assignment happened to be alone — and the
// day a second module able to hold it is assigned, both claimed, both were refused, and the first
// one's machine stopped resolving (novox/hq 04-ISSUES/170). The mesh now writes the derived holder
// down before it acts, so the second assignment stands beside the holder on record.
func aSeatedStore() catalogue.Manifest {
return catalogue.Manifest{Module: "store", Version: "1",
Provides: []catalogue.Offer{{Name: "postgres-database", Scope: catalogue.ScopeMesh}},
Serves: map[string]map[string]any{"postgres-database": {"port": 5432}},
Claims: []catalogue.Claim{{Name: "mesh-store", Scope: catalogue.ScopeMesh}}}
}
func TestASecondEligibleHolderStandsBesideTheOneHeldByBeingAlone(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
// Every deploy seeds the mesh's own seats; a record is a row against one of them.
if _, err := open.inventory.SeedSeats(ctx, catalogue.DefaultSeats()); err != nil {
t.Fatal(err)
}
register(t, open, aSeatedStore())
if _, err := assign(ctx, open, "anchor", "store"); err != nil {
t.Fatal(err)
}
said, err := assign(ctx, open, "laptop", "store")
if err != nil {
t.Fatalf("a second store, eligible for the seat, was refused:\n%s\n%v", said, err)
}
if strings.Contains(said, "cannot be worked out") {
t.Fatalf("assigning a second store unsettled the first one's machine:\n%s", said)
}
if !strings.Contains(said, "recorded store on anchor as the standing holder of mesh-store") {
t.Fatalf("the holder by derivation was not written down:\n%s", said)
}
holdings, err := open.inventory.Holdings(ctx)
if err != nil {
t.Fatal(err)
}
var found bool
for _, h := range holdings {
if h.Claim == "mesh-store" {
found = true
if h.Node != "anchor" || h.Module != "store" {
t.Fatalf("mesh-store is recorded on %s/%s, not on the one that held it", h.Node, h.Module)
}
}
}
if !found {
t.Fatalf("mesh-store has no holder on record after assigning: %v", holdings)
}
// And the record decides from here: the anchor's plan holds the seat, the laptop's does not.
for node, holds := range map[string]bool{"anchor": true, "laptop": false} {
plan, _, err := planFor(ctx, open, node)
if err != nil {
t.Fatalf("%s no longer resolves: %v", node, err)
}
var claimed bool
for _, c := range plan.Claims {
if c.Claim == "mesh-store" {
claimed = true
}
}
if claimed != holds {
t.Fatalf("%s holds mesh-store: %v, want %v", node, claimed, holds)
}
}
}
func TestAHolderOnRecordIsNotRewrittenByDerivation(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
if _, err := open.inventory.SeedSeats(ctx, catalogue.DefaultSeats()); err != nil {
t.Fatal(err)
}
register(t, open, aSeatedStore())
for _, node := range []string{"anchor", "laptop"} {
if _, err := assign(ctx, open, node, "store"); err != nil {
t.Fatal(err)
}
}
// A person hands the seat to the laptop. From here the record decides, and what the mesh
// derives must never write over it.
if err := open.inventory.HoldSeat(ctx, "mesh-store", catalogue.ScopeMesh, "laptop", "store"); err != nil {
t.Fatal(err)
}
said, err := recordDerivedHolders(ctx, open)
if err != nil {
t.Fatal(err)
}
if len(said) != 0 {
t.Fatalf("a seat on record was written again from derivation: %v", said)
}
holdings, _ := open.inventory.Holdings(ctx)
for _, h := range holdings {
if h.Claim == "mesh-store" && h.Node != "laptop" {
t.Fatalf("the record moved to %s", h.Node)
}
}
}
+111
View File
@@ -0,0 +1,111 @@
package main
import (
"strings"
"testing"
"github.com/novox/mesh-controller/internal/inventory"
)
// A host refuses a declaration carrying a field it does not know, and refuses it whole — so every new
// field is a flag day, and the mesh had no record of which host any machine ran (novox/hq
// 04-ISSUES/087). The order was kept by somebody remembering it.
func TestTheMeshNamesWhichMachinesRunWhichHost(t *testing.T) {
split := hostSplit([]inventory.Node{
{Name: "anchor", HostVersion: "04a27ca"},
{Name: "laptop", HostVersion: "ced54d4"},
{Name: "spare", HostVersion: "04a27ca"},
})
if len(split) != 2 {
t.Fatalf("two versions were reported and the split has %d: %v", len(split), split)
}
if got := strings.Join(split["04a27ca"], ","); got != "anchor,spare" && got != "spare,anchor" {
t.Fatalf("04a27ca is held by %q", got)
}
if got := strings.Join(split["ced54d4"], ","); got != "laptop" {
t.Fatalf("ced54d4 is held by %q", got)
}
}
func TestTheMeshDoesNotClaimWhichHostIsNewer(t *testing.T) {
// **The fault this replaced.** A host reports its version as a commit, and commits have no order.
// The first version compared them as strings and, on the live mesh, named the three machines
// running the NEWER host as the ones behind: `ced54d4` sorts above `04a27ca` and means nothing.
//
// There is no assertion to make about which is newer, and that is the point — the type says so.
// hostSplit returns who runs what, and nothing that could be read as an ordering.
split := hostSplit([]inventory.Node{
{Name: "old-but-sorts-high", HostVersion: "ced54d4"},
{Name: "new-but-sorts-low", HostVersion: "04a27ca"},
})
for version, machines := range split {
if len(machines) != 1 {
t.Fatalf("%s is held by %v", version, machines)
}
}
}
func TestAMachineThatHasNotSaidIsNotAVersion(t *testing.T) {
// It may be running anything. Counting it as a version would invent a disagreement; `node show`
// says per machine that it has not said.
split := hostSplit([]inventory.Node{
{Name: "anchor", HostVersion: "04a27ca"},
{Name: "quiet"},
})
if split != nil {
t.Fatalf("one reported version and one silence read as a disagreement: %v", split)
}
}
func TestMachinesAgreeingOnTheirHostAreNotADisagreement(t *testing.T) {
if split := hostSplit([]inventory.Node{
{Name: "anchor", HostVersion: "v2"},
{Name: "laptop", HostVersion: "v2"},
}); split != nil {
t.Fatalf("machines agreeing reported a split: %v", split)
}
}
func TestAMeshWhereNothingReportedAHostStatesNoDisagreement(t *testing.T) {
if split := hostSplit([]inventory.Node{{Name: "anchor"}, {Name: "laptop"}}); split != nil {
t.Fatalf("a mesh told no host version reported a split: %v", split)
}
}
func TestAReportedHostVersionIsKeptAndReadBack(t *testing.T) {
// The machine has sent this since ADR 0141 and the controller's own copy of the report did not
// have the field, so it was unmarshalled into nothing. End to end through the store, because the
// fault was a field that existed on one side of the wire only.
open := aMesh(t)
record, err := open.inventory.NodeByName(t.Context(), "anchor")
if err != nil {
t.Fatal(err)
}
if record.HostVersion != "" {
t.Fatalf("a machine that never reported one has host version %q", record.HostVersion)
}
if err := open.inventory.RecordHostVersion(t.Context(), record.ID, "ced54d4"); err != nil {
t.Fatal(err)
}
again, err := open.inventory.NodeByName(t.Context(), "anchor")
if err != nil {
t.Fatal(err)
}
if again.HostVersion != "ced54d4" {
t.Fatalf("the reported host version read back as %q", again.HostVersion)
}
// An empty report never clears what a machine last said: a bare word that the node is there says
// nothing about its host.
if err := open.inventory.RecordHostVersion(t.Context(), record.ID, " "); err != nil {
t.Fatal(err)
}
kept, err := open.inventory.NodeByName(t.Context(), "anchor")
if err != nil {
t.Fatal(err)
}
if kept.HostVersion != "ced54d4" {
t.Fatalf("a report carrying no host version cleared what the machine had said: %q",
kept.HostVersion)
}
}
+10 -2
View File
@@ -76,7 +76,10 @@ func run() error {
return pinCommand(ctx, args[1:], true)
case "unpin":
return pinCommand(ctx, args[1:], false)
case "migrate":
// `prepare` is how the mesh asks any module to bring its state to the shape this version needs
// (novox/hq ADR 0135), and the control plane answers it the same way as everything else — its
// own schema is not a special case. `migrate` remains the word a person types.
case "prepare", "migrate":
return migrate(ctx)
case "node":
return nodeCommand(ctx, args[1:])
@@ -85,7 +88,7 @@ func run() error {
case "identity":
return identityCommand(ctx, args[1:])
case "broker":
return brokerCommand(args[1:])
return brokerCommand(ctx, args[1:])
case "serve":
return serve(ctx)
case "upgrade":
@@ -138,12 +141,16 @@ func usage() {
fmt.Fprint(os.Stderr, `mesh-controller — the control plane
migrate bring each context's schema up to date
prepare the same, asked the way the mesh asks any module (ADR 0135)
node add <name> [--adopted] create a node record; --adopted: the machine is in use
node list the nodes this mesh knows about
node show <name> what one machine reported it can do, and why
node public-domain <name> the domain it composes its routed names under
node public-domain <name> <d> ...set it to d
node public-domain <name> --clear ...it faces the outside no longer
node networks <name> the networks it routes for what it hosts
node networks <name> <cidr>... ...set them; its filter forwards these too
node networks <name> --clear ...only the container runtime's own
token issue --node <name> a one-time right to join, for an existing record
token issue --new <name> create the record and issue for it
token issue ... --adopted ...for a machine in use, which joins adopted
@@ -154,6 +161,7 @@ func usage() {
overlay place <node> [flags] say where a node is and how it is reached
overlay show the private network, as the mesh computes it
module add <file> register a module from its manifest
module check <file|dir>... judge manifests where they are written, with no mesh (exit 1 on any problem)
module list what modules this mesh knows about
module moved <name> <commit> the source has a newer commit than the mesh built
module forget <name> remove one, unless a node runs it or the mesh holds things for it
+91 -13
View File
@@ -54,7 +54,24 @@ var provided = providedModules()
func moduleCommand(ctx context.Context, args []string) error {
if len(args) == 0 {
return errors.New("module add <file>, module list, or module forget <name>")
return errors.New("module add <file>, module check <file>..., module list, or module forget <name>")
}
// `check` needs no mesh, and must not: it is what somebody runs in their own repository before
// there is a mesh in reach (novox/hq issue 148). A directory expands to every manifest under it.
if args[0] == "check" {
var paths []string
for _, a := range args[1:] {
if info, err := os.Stat(a); err == nil && info.IsDir() {
under, err := manifestsUnder(a)
if err != nil {
return err
}
paths = append(paths, under...)
continue
}
paths = append(paths, a)
}
return moduleCheck(paths, os.Stdout)
}
open, err := openStores(ctx)
if err != nil {
@@ -69,12 +86,20 @@ func moduleCommand(ctx context.Context, args []string) error {
repo := set.String("source", "", "where this module comes from")
ref := set.String("ref", "", "the branch followed there")
commit := set.String("commit", "", "the commit this manifest was read at")
// **Where inside the repository the module is** (novox/hq ADR 0069). A module is a
// repository *and* a directory, and a record that carries only the repository names a
// module.json at its root — so every later build of it looks in the wrong place and fails
// with "no module.json at its root". Nine modules on this mesh were registered that way
// and none of them could be rebuilt (2026-09-28).
path := set.String("path", "", "the module's directory inside that repository")
self := set.Bool("self", false, "the source is a path on the forge holding the git seat")
positionals, err := parseAround(set, args[1:])
if err != nil {
return err
}
if len(positionals) != 1 {
return errors.New("module add <manifest.json> [--source <repo> --ref <branch> --commit <sha>]")
return errors.New("module add <manifest.json> [--source <repo> [--self] [--path P] " +
"--ref <branch> --commit <sha>]")
}
raw, err := os.ReadFile(positionals[0])
if err != nil {
@@ -84,23 +109,27 @@ func moduleCommand(ctx context.Context, args []string) error {
if err != nil {
return err
}
// Provenance together or not at all. A source with no commit cannot be compared against
// anything, so it would record where the module came from and still never be able to say
// the mesh is behind it — which is the one thing recording it is for.
if (*repo == "") != (*commit == "") {
return errors.New("--source and --commit go together: a source with no commit " +
"cannot be compared against anything, and a commit with no source has nothing " +
"to be compared with")
from, err := whereItComesFrom(*repo, *ref, *commit, *path, *self)
if err != nil {
return err
}
if err := inv.RegisterModule(ctx, m, inventory.Source{
Repository: *repo, Ref: *ref, BuiltFrom: *commit,
}); err != nil {
if err := namesNoInstallation(m); err != nil {
return err
}
if err := inv.RegisterModule(ctx, m, from); err != nil {
return err
}
fmt.Printf("%s registered", m.Module)
if *commit != "" {
fmt.Printf(" from %s", short(*commit))
}
if *repo != "" && *path == "" {
// Said, not refused: a module really at the root is the ordinary case for a repository
// of its own. But a repository holding many modules and a record naming none of them is
// a module nothing can rebuild, and the person adding it is the one who knows which.
fmt.Printf("\n no directory inside %s, so it is built from that repository's root — "+
"`--path` if the module lives in a directory there", *repo)
}
if len(m.Provides) > 0 {
fmt.Printf(", providing %s", describeOffers(m.Provides))
}
@@ -266,7 +295,7 @@ func moduleCommand(ctx context.Context, args []string) error {
return issueOnTheNewBus(ctx, inv, m, *forNode, busAddress)
default:
return fmt.Errorf("module has no %q; it has add, list, moved, forget and issue", args[0])
return fmt.Errorf("module has no %q; it has add, check, list, moved, forget and issue", args[0])
}
}
@@ -602,3 +631,52 @@ func issueWith(ctx context.Context, inv *inventory.Inventory, m catalogue.Manife
"machine holding mesh-broker\n")
return nil
}
// whereItComesFrom is the provenance a module handed over by hand records, and what a record must
// say to be worth anything later.
//
// **A module is a repository and a directory inside it** (novox/hq ADR 0069). A record carrying only
// the repository names a module.json at its root, so every later build of it looks in the wrong
// place — nine modules on this mesh were registered that way and none of them could be rebuilt
// (2026-09-28). The directory cannot be checked from here, because the control plane does not clone;
// what can be checked is that the record is whole.
func whereItComesFrom(repository, ref, commit, path string, self bool) (inventory.Source, error) {
// Provenance together or not at all. A source with no commit cannot be compared against
// anything, so it would record where the module came from and still never be able to say the
// mesh is behind it — which is the one thing recording it is for.
if (repository == "") != (commit == "") {
return inventory.Source{}, errors.New("--source and --commit go together: a source with " +
"no commit cannot be compared against anything, and a commit with no source has " +
"nothing to be compared with")
}
// A directory or a forge with no repository is half a location, and the half it keeps is the
// half nothing can be found with.
if repository == "" && (path != "" || self) {
return inventory.Source{}, errors.New("--path and --self say where inside a source and " +
"which forge holds it, so they need --source: without one there is nothing for them " +
"to be part of")
}
from := inventory.Source{Repository: repository, Ref: ref, BuiltFrom: commit, Path: path}
if self {
if err := onASeat(repository); err != nil {
return inventory.Source{}, err
}
from.Seat = gitSeat
}
return from, nil
}
// namesNoInstallation is the mesh refusing a definition that names an installation, at the moment
// it would enter the catalogue (novox/hq ADR 0112, ADR 0155). `module check` says the same thing
// earlier, where the author is; this is the last moment the mesh can still say no, and a
// definition that got past the check — written elsewhere, or checked by nobody — is refused here
// in the same words. A name meant on purpose is declared with its reason and passes.
func namesNoInstallation(m catalogue.Manifest) error {
named := catalogue.InstallationProblems(m)
if len(named) == 0 {
return nil
}
return fmt.Errorf("%s names an installation, and a definition names none — declare a name meant "+
"on purpose under %s with its reason, or take it out:\n - %s",
m.Module, catalogue.NamesOnPurpose, strings.Join(named, "\n - "))
}
+8 -1
View File
@@ -346,9 +346,16 @@ func whoResolves(ctx context.Context, open *stores, requirement string) (
refused := map[string]string{}
for _, n := range nodes {
plan, _, err := planFor(ctx, open, n.Name)
if err != nil {
switch {
case unresolvable(err):
refused[n.Name] = err.Error()
continue
case err != nil:
// Not a node that does not resolve — a question that went unanswered. Recording it as a
// refusal would take the machine off the private network, and the generator that reads
// this would then write a roster and a filter without it (novox/hq 04-ISSUES/152).
return nil, nil, fmt.Errorf("whether %s answers %q cannot be read: %w",
n.Name, requirement, err)
}
for _, m := range plan.Modules {
for _, offered := range m.Offers() {
+34 -2
View File
@@ -66,6 +66,18 @@ func nodeCommand(ctx context.Context, args []string) error {
// because the damage is already done by the time it prints.
return publicDomain(ctx, inv, args[1:])
case "networks":
// Removed by novox/hq ADR 0140, which superseded the record that added it. The filter no
// longer names any network: it constrains what arrives from outside the machine and says
// nothing about what did not, so there is no list to keep. Answered rather than met with
// "unknown command", because this was the documented way to stop a flip cutting a machine's
// containers off and somebody will reasonably still type it.
return errors.New("`node networks` is gone (novox/hq ADR 0140). The filter constrains what " +
"arrives from outside this machine and says nothing about traffic that did not, so no " +
"network is named anywhere and nothing needs to be said to keep a machine's own " +
"containers reaching outward. The machine reports which of its links face outside; see " +
"`node show <name>`")
case "account":
// The operator's login on this machine (novox/hq to-be 29): what a home-scoped file is
// owned by and which account `ssh <node>` uses. Reports with no argument; sets with one;
@@ -351,9 +363,15 @@ func identityCommand(ctx context.Context, args []string) error {
return nil
}
func brokerCommand(args []string) error {
func brokerCommand(ctx context.Context, args []string) error {
if len(args) > 0 && args[0] == "certificate" {
return busCertificate(args[1:])
}
if len(args) > 0 && args[0] == "accounts" {
return busAccounts(ctx, args[1:])
}
if len(args) == 0 || args[0] != "show" {
return errors.New("broker show")
return errors.New("broker show | broker certificate [--check] --into <directory> | broker accounts --into <file>")
}
known, err := broker.FromEnvironment()
if errors.Is(err, broker.ErrNotConfigured) {
@@ -418,6 +436,12 @@ func showNode(ctx context.Context, inv *inventory.Inventory, name string) error
}
fmt.Printf("%s\n", node.Name)
fmt.Printf(" last heard from %s\n", heardFrom(node))
// Which host runs it, as it reported (novox/hq 04-ISSUES/087). Said whenever known, because a
// host refuses a declaration carrying a field it does not understand and refuses it WHOLE — so
// which host a machine runs is what decides whether the mesh can send it anything new, and
// nothing could say it. "not reported" rather than blank: a machine that has not said is a
// different thing from one running nothing.
fmt.Printf(" host %s\n", orNotReported(node.HostVersion))
if err := showMode(ctx, inv, node); err != nil {
return err
}
@@ -468,3 +492,11 @@ func showNode(ctx context.Context, inv *inventory.Inventory, name string) error
}
return nil
}
// orNotReported is a fact a machine states about itself, or the fact that it has not.
func orNotReported(s string) string {
if strings.TrimSpace(s) == "" {
return "not reported — this machine has not said since the mesh began keeping it"
}
return s
}
+7 -3
View File
@@ -189,13 +189,17 @@ func readPrivateKey(path string) (string, error) {
func personIssue(ctx context.Context, args []string) error {
set := flag.NewFlagSet("operator issue", flag.ContinueOnError)
invokes := set.String("invokes", "", "the tools this person may call, comma-separated, or * for every one")
if err := set.Parse(args); err != nil {
// Flags on either side of the name, because the usage this command prints puts them after it —
// and the standard parser stops at the first thing that is not a flag, so the order the command
// documents was the one order it refused (2026-09-28).
positionals, err := parseAround(set, args)
if err != nil {
return err
}
if set.NArg() != 1 {
if len(positionals) != 1 {
return errors.New("operator issue <name> --invokes <tool,tool|*>")
}
name := set.Arg(0)
name := positionals[0]
if *invokes == "" {
return errors.New(
"say what this person may call: --invokes mesh-catalog.catalog_tools,gitea.repo_create, " +
+120
View File
@@ -3,6 +3,7 @@ package main
import (
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
@@ -90,3 +91,122 @@ func TestAMergeMatchesTheSourcesBuiltFromIt(t *testing.T) {
}
}
}
// A merge made before the source was last seen is history: it does not move the source, and a
// merge that says nothing about when it was made is taken as news.
func TestAMergeOlderThanTheLastLookIsHistory(t *testing.T) {
seen := time.Date(2026, 9, 28, 3, 0, 0, 0, time.UTC)
if !isHistory("2026-09-28T02:00:00Z", seen) {
t.Fatal("an older merge was taken as news")
}
if isHistory("2026-09-28T04:00:00Z", seen) {
t.Fatal("a newer merge was taken as history")
}
if isHistory("", seen) || isHistory("2026-09-28T02:00:00Z", time.Time{}) {
t.Fatal("a merge or a source with no time on it was refused")
}
}
// A module as the catalogue holds it: built from a repository, at a directory inside it.
func fromRepo(module, repository, path string) inventory.Entry {
return inventory.Entry{
Manifest: catalogue.Manifest{Module: module},
Source: inventory.Source{Repository: repository, Path: path, Ref: "main"},
}
}
// A merge rebuilds the modules whose own directories it changed, and everything when what it changed
// is shared. One repository holding many modules is the ordinary case here, and rebuilding all of
// them for a change to one is what exhausted a registry's pull limit the first night this ran.
func TestAMergeRebuildsTheModulesItChanged(t *testing.T) {
const repo = "http://forge.internal:20000/novox/mesh-catalog.git"
gitea := fromRepo("gitea", repo, "modules/gitea")
keycloak := fromRepo("keycloak", repo, "modules/keycloak")
known := []inventory.Entry{gitea, keycloak, fromRepo("plex", repo, "modules/plex")}
candidates := []inventory.Entry{gitea, keycloak}
merge := func(paths []string, truncated bool) link.SourceMoved {
return link.SourceMoved{Owner: "novox", Repo: "mesh-catalog", Base: "main",
Paths: paths, PathsTruncated: truncated}
}
named := func(entries []inventory.Entry) string {
var names []string
for _, e := range entries {
names = append(names, e.Manifest.Module)
}
return strings.Join(names, ",")
}
for _, c := range []struct {
what string
m link.SourceMoved
want string
}{
{"one module's own files", merge([]string{"modules/gitea/index.ts", "modules/gitea/client.ts"}, false), "gitea"},
{"two modules' files", merge([]string{"modules/gitea/index.ts", "modules/keycloak/module.json"}, false), "gitea,keycloak"},
{"a file they share", merge([]string{"tsconfig.json"}, false), "gitea,keycloak"},
{"a module the mesh does not hold", merge([]string{"modules/plex/index.ts"}, false), ""},
{"nothing said about the files", merge(nil, false), "gitea,keycloak"},
{"more files than were listed", merge([]string{"modules/gitea/index.ts"}, true), "gitea,keycloak"},
} {
if got := named(whatTheMergeTouched(candidates, known, c.m)); got != c.want {
t.Errorf("%s: rebuilt %q, wanted %q", c.what, got, c.want)
}
}
}
// A module whose recipe packages source from another repository is affected when that repository
// moves — the manifest the mesh keeps says nothing about it, so the record of what the build read is
// the only thing that can say so.
func TestAModuleIsAffectedByTheRepositoryItPackages(t *testing.T) {
m := link.SourceMoved{Owner: "novox", Repo: "mesh-controller", Base: "main",
CloneURL: "http://forge.internal:20000/novox/mesh-controller.git"}
for _, read := range [][]inventory.ReadRepository{
{{Repository: "http://forge.internal:20000/novox/mesh-controller.git", Ref: "main"}},
{{Repository: "novox/mesh-controller"}},
{{Repository: "https://elsewhere.example/novox/other"}, {Repository: "novox/mesh-controller.git", Ref: "main"}},
} {
if !readsFrom(read, m) {
t.Errorf("%+v was not matched by the merge", read)
}
}
for _, read := range [][]inventory.ReadRepository{
nil,
{{Repository: "novox/mesh-host", Ref: "main"}},
{{Repository: "novox/mesh-controller", Ref: "release"}},
} {
if readsFrom(read, m) {
t.Errorf("%+v was matched by a merge that is not its", read)
}
}
}
// What a module handed over by hand records about where it came from, and what is refused.
func TestWhatAHandedOverModuleRecordsAboutItsSource(t *testing.T) {
// The whole location: a repository on the mesh's own forge, the directory inside it, the branch
// and the commit the manifest was read at.
from, err := whereItComesFrom("novox/mesh-catalog", "main", "c0ffee", "modules/gitea", true)
if err != nil {
t.Fatal(err)
}
if from.Path != "modules/gitea" || from.Seat != "git" || from.Repository != "novox/mesh-catalog" {
t.Fatalf("the source records as %+v", from)
}
// A manifest with no provenance at all is legitimate: fixing something in a hurry.
if from, err := whereItComesFrom("", "", "", "", false); err != nil || from != (inventory.Source{}) {
t.Fatalf("a manifest handed over with no provenance was refused: %+v, %v", from, err)
}
for _, c := range []struct {
what string
repository, ref, commit, path string
self bool
}{
{what: "a source with no commit", repository: "novox/mesh-catalog", commit: ""},
{what: "a commit with no source", commit: "c0ffee"},
{what: "a directory inside nothing", path: "modules/gitea"},
{what: "a forge holding nothing", self: true},
{what: "an address given as a path on the forge", repository: "http://forge.internal:20000/novox/x.git", commit: "c0ffee", self: true},
} {
if _, err := whereItComesFrom(c.repository, c.ref, c.commit, c.path, c.self); err == nil {
t.Errorf("%s was recorded as a source", c.what)
}
}
}
+71 -9
View File
@@ -25,7 +25,36 @@ import (
// cheapest next step. That is how novox/hq ADR 0001 records `hal/sdk` reaching 34,636:
// nothing in it was wrong, and no one edit was the one that should have been a new file.
// notResolvable marks the one failure in planFor that is a statement about the node: its assigned
// modules do not compose. Every other failure means the mesh could not be *asked* — the store was
// unreachable, a key could not be read — and says nothing about the node at all.
//
// The distinction exists because three callers gather something across every machine and must carry
// on when one machine's set is broken. Each of them read a plain error as "their set does not
// resolve", and so read a store that was briefly unreachable as a machine that runs nothing. On the
// roster of routed names that is not a degraded answer but a false one: it states, to every machine
// at once, that another machine's names do not exist. A control node spent hours replacing every
// container it ran, on a six-minute cycle, because each pass restarted the store this is read from,
// the read failed, one name left the roster, and the roster is part of every container's identity
// (novox/hq 04-ISSUES/152, and 04-ISSUES/151 for why a changed roster is a changed container).
//
// So: skip a node that cannot resolve, and never a node that could not be read.
type notResolvable struct{ err error }
func (n notResolvable) Error() string { return n.err.Error() }
func (n notResolvable) Unwrap() error { return n.err }
// unresolvable reports whether err is a node's own set failing to compose, rather than the mesh
// being unable to answer.
func unresolvable(err error) bool {
var n notResolvable
return errors.As(err, &n)
}
// planFor works out everything a node should run, from what was assigned to it.
//
// A failure to compose the node's own modules is wrapped as notResolvable; every other failure is
// returned as it is. Callers gathering across the mesh must tell them apart — see notResolvable.
func planFor(ctx context.Context, open *stores, nodeName string) (catalogue.Resolution, catalogue.SettingsBy, error) {
inv := open.inventory
shelf, err := inv.Catalogue(ctx)
@@ -95,7 +124,9 @@ func planFor(ctx context.Context, open *stores, nodeName string) (catalogue.Reso
At: onNetwork[nodeName], PublicDomain: publicDomain,
Account: who.Account, AccountHome: who.AccountHome}, world)
if err != nil {
return catalogue.Resolution{}, nil, err
// The node's own set does not compose. Marked, because this is the only failure here that
// a mesh-wide gatherer may pass over — see notResolvable.
return catalogue.Resolution{}, nil, notResolvable{err}
}
// The credential for each thing this node takes from elsewhere. Made once and kept, so the
@@ -163,8 +194,13 @@ func planFor(ctx context.Context, open *stores, nodeName string) (catalogue.Reso
if len(stray) > 0 {
// Somebody set something that reaches no file. Said here rather than discovered by the
// machine not behaving differently, which is the slowest way there is.
return catalogue.Resolution{}, nil, fmt.Errorf(
"these settings reach nothing:\n - %s", strings.Join(stray, "\n - "))
//
// Marked like a set that will not compose, and for the same reason: it is a standing fact
// about this node's own configuration, not a question the mesh could not answer. A gatherer
// passes over it as it always did — one node's stray setting must not stop every other node
// being described (novox/hq 04-ISSUES/152).
return catalogue.Resolution{}, nil, notResolvable{fmt.Errorf(
"these settings reach nothing:\n - %s", strings.Join(stray, "\n - "))}
}
return resolved, settings, nil
}
@@ -640,13 +676,19 @@ func renderingFor(ctx context.Context, open *stores, node string,
if err != nil {
return catalogue.Rendering{}, inventory.Node{}, err
}
// Which of this machine's links face outside, which is what the derived filter is written
// around (novox/hq ADR 0140). Reported by the machine, never set.
outwardLinks, err := inv.OutwardLinksOf(ctx, node)
if err != nil {
return catalogue.Rendering{}, inventory.Node{}, err
}
return catalogue.Rendering{
BusMembership: memberships[node],
Settings: settings, Generators: gens, Grants: grants, Needed: needed, Ports: ports,
Certificate: certificate, Authority: authority, Mesh: private, Names: names,
Machines: machines,
Suffix: overlay.Suffix(), MeshRange: meshRange, Accounts: accounts, Foundation: foundation,
Kept: kept, Adopted: record.Adopted,
Suffix: overlay.Suffix(), MeshRange: meshRange, TunnelInterface: overlay.Interface, Accounts: accounts, Foundation: foundation,
Kept: kept, Adopted: record.Adopted, OutwardLinks: outwardLinks,
Given: given, Taken: taken, Seats: seats, ArtifactStore: artifactStore, Built: built,
BusUsers: busUsers,
}, record, nil
@@ -665,11 +707,17 @@ func renderingFor(ctx context.Context, open *stores, node string,
// routed name only because it carried a label the mesh composed, never because the mesh knows what
// "route" means. A node that does not resolve is skipped, so one machine's broken set does not cost
// the rest their names.
//
// **A node that could not be READ is a different matter and is raised.** Skipping one states, to
// every machine at once, that its names do not exist — and since the roster is part of every
// container's identity, that withdraws them and replaces every container (novox/hq 04-ISSUES/152,
// 151). So every failure here says which machine and which read, because the alternative is a
// mesh-wide refusal with nothing named in it.
func routeNamesInTheMesh(ctx context.Context, open *stores) (map[string]string, error) {
inv := open.inventory
places, err := inv.Overlays(ctx)
if err != nil {
return nil, err
return nil, fmt.Errorf("where the machines are cannot be read: %w", err)
}
address := map[string]string{}
for _, p := range places {
@@ -680,14 +728,22 @@ func routeNamesInTheMesh(ctx context.Context, open *stores) (map[string]string,
nodes, err := inv.Nodes(ctx)
if err != nil {
return nil, err
return nil, fmt.Errorf("which machines the mesh has cannot be read: %w", err)
}
out := map[string]string{}
for _, n := range nodes {
plan, settings, err := planFor(ctx, open, n.Name)
if err != nil {
switch {
case unresolvable(err):
// Their set does not compose, so they serve no names. Passed over, so one machine's
// broken set does not cost the rest theirs.
continue
case err != nil:
// The mesh could not be asked. Returning the roster without this machine's names would
// state that they do not exist — to every machine, and indistinguishably from the
// operator having withdrawn them (novox/hq 04-ISSUES/152).
return nil, fmt.Errorf("the names %s serves cannot be read: %w", n.Name, err)
}
for _, m := range plan.Modules {
for to := range m.Contributes {
@@ -817,11 +873,17 @@ func grantsFor(ctx context.Context, open *stores, node string) ([]catalogue.Gran
out := make([]catalogue.Grant, 0, len(issued))
for _, s := range issued {
plan, settings, err := planFor(ctx, open, s.Consumer)
if err != nil {
switch {
case unresolvable(err):
// Their set does not resolve. Skipped rather than fatal: this node is not the place
// to report another machine's problem, and a grant for something that is not going to
// run would have the provider create a user nothing uses.
continue
case err != nil:
// The mesh could not be asked what they wanted, which is not the same as their wanting
// nothing — and withholding a grant on that reading takes a consumer's access away
// (novox/hq 04-ISSUES/152).
return nil, fmt.Errorf("what %s asked of %s cannot be read: %w", s.Consumer, s.Name, err)
}
values, asks, err := plan.ContributionsFrom(s.Name, s.ConsumerModule, settings)
if err != nil {
+87 -3
View File
@@ -7,6 +7,7 @@ import (
"errors"
"flag"
"fmt"
"log"
"os"
"sort"
"strings"
@@ -124,6 +125,22 @@ func serve(ctx context.Context) error {
return err
}
// And the mesh's own verbs, as the seat this control plane holds (novox/hq ADR 0154). Served
// from the store's row, so what the seat declares is what is answered.
handlers, err := seatToolHandlers()
if err != nil {
return err
}
bus, isNATS := server.Bus().(link.OverNATS)
if !isNATS {
return errors.New("the mesh's verbs are served over the bus, and this control plane is not on it")
}
stopServing, err := bus.ServeSeatTools(catalogue.ControllerSeatName, handlers, log.New(os.Stdout, "", log.LstdFlags))
if err != nil {
return err
}
defer stopServing()
return server.Serve(ctx)
}
@@ -162,7 +179,13 @@ func declare(ctx context.Context, args []string) error {
return err
}
server, err := connectLink(ctx, nil, nil, nil)
// **With the inventory, so the bus is raised** (novox/hq ADR 0134, design 30). A module's
// declaration and how it hears what it consumes move together: its consumer is derived from the
// same records this declaration is composed from. Raised only when the control plane started
// serving, a module that gained a `consumes` was sent a declaration it could act on and a
// consumer that never delivered the event — and nothing anywhere said the two disagreed
// (found on review, 2026-09-28). Everything the raise does is idempotent.
server, err := connectLink(ctx, inv, nil, nil)
if err != nil {
return err
}
@@ -332,6 +355,12 @@ func pushCommand(ctx context.Context, args []string) error {
sentDigest := map[string]string{}
defer release()
for _, s := range sending {
// Numbered under the hold, one higher than the last, before the body exists — the number is
// inside the signed bytes, so a replayed older declaration cannot borrow a newer one's
// (novox/hq 04-ISSUES/107).
if err := number(ctx, inv, &s); err != nil {
return err
}
body, err := s.declared.Body()
if err != nil {
return err
@@ -558,6 +587,9 @@ func sendRound(ctx context.Context, open *stores, names []string,
return compose(held, node)
})
for _, s := range sending {
if err := number(ctx, open.inventory, &s); err != nil {
return refused, err
}
body, err := s.declared.Body()
if err != nil {
return refused, err
@@ -639,6 +671,9 @@ func sendTo(ctx context.Context, open *stores, names []string) error {
defer server.Close()
for _, s := range sending {
if err := number(ctx, inv, &s); err != nil {
return err
}
body, err := s.declared.Body()
if err != nil {
return err
@@ -688,6 +723,12 @@ func wouldSend(ctx context.Context, open *stores,
if err != nil {
continue
}
// Composed with the number the machine was LAST sent, so this is byte for byte what it was
// sent when nothing else changed. A fresh number here would make every machine read as
// behind for ever (novox/hq 04-ISSUES/107).
if declared.Sequence, err = open.inventory.Sequence(ctx, n.ID); err != nil {
return nil, err
}
body, err := declared.Body()
if err != nil {
return nil, err
@@ -711,6 +752,12 @@ func raiseTheBus(ctx context.Context, inv *inventory.Inventory, address string)
broker.BareAddress(address), err)
}
defer js.Close()
// What the raise decided not to fail over. Said, for the reason everything else here is said:
// a consumer kept as it was is a difference between what the mesh asked for and what the bus
// holds, and one nobody would find by reading either (novox/hq 04-ISSUES/156).
js.Note = func(format string, args ...any) {
fmt.Printf(" "+format+"\n", args...)
}
// **Its own user, before anything else.** The controller's account is created by the installer at
// a bootstrap password, before there is a controller to mint one — so nothing recorded a hash for
@@ -753,8 +800,31 @@ func raiseTheBus(ctx context.Context, inv *inventory.Inventory, address string)
if err := broker.RaiseSeats(js, inventory.MeshSeats(), holders); err != nil {
return err
}
fmt.Printf("the bus at %s has its streams, and %d machine(s) can hear a declaration\n",
broker.BareAddress(address), len(names))
// And how every module hears what it consumes. Derived from the same records the user list is
// composed from, so a module the mesh grants a consumer's subjects has that consumer waiting.
// Done on every raise, not only when a credential is issued: every module moved onto this bus
// by the rollout was issued on the old one, and came up with nothing to bind to (2026-09-28).
records, err := inv.BusRecords(ctx)
if err != nil {
return err
}
users, err := broker.Users(records)
if err != nil {
return err
}
hearing := 0
for _, p := range users {
consumer, needed := broker.ConsumerFor(p)
if !needed {
continue
}
if err := js.EnsureConsumer(consumer); err != nil {
return fmt.Errorf("how %s on %s hears what it consumes: %w", p.Module, p.Node, err)
}
hearing++
}
fmt.Printf("the bus at %s has its streams, %d machine(s) can hear a declaration, and %d module(s) "+
"can hear what they consume\n", broker.BareAddress(address), len(names), hearing)
return nil
}
@@ -792,3 +862,17 @@ func seatHolders(ctx context.Context, inv *inventory.Inventory) (map[string]brok
}
return out, nil
}
// number gives one send the next sequence for its node (novox/hq 04-ISSUES/107).
func number(ctx context.Context, inv *inventory.Inventory, s *readyNode) error {
record, err := inv.NodeByName(ctx, s.node)
if err != nil {
return err
}
seq, err := inv.NextSequence(ctx, record.ID)
if err != nil {
return err
}
s.declared.Sequence = seq
return nil
}
+34
View File
@@ -56,6 +56,23 @@ type meshStatus struct {
Machines int `json:"machines"`
// Adopted is every node still adopted (novox/hq ADR 0100); absent when none is.
Adopted []string `json:"adopted,omitempty"`
// Untaken is every module assigned to a machine that is holding what it found rather than
// running what the module declares, because nothing took it (novox/hq 04-ISSUES/125). Absent
// when nothing is held.
//
// **A document without this said an outage was a well mesh.** Read from what each machine
// reported, so it is the machine's account and not the mesh's take-time listing.
Untaken []machineUntaken `json:"untaken,omitempty"`
}
// machineUntaken is one module a machine is holding rather than running, and how many resources of
// it are held.
type machineUntaken struct {
Node string `json:"node"`
Module string `json:"module"`
// Held is how many of the module's resources the machine is keeping as it found them. Zero is
// impossible here: a module with nothing held is not in this list.
Held int `json:"held"`
}
type machineUnresolved struct {
@@ -136,6 +153,23 @@ func statusAsJSON(asked answers) ([]byte, error) {
Quiet: []machineQuiet{}, Behind: []moduleBehind{}, Waiting: []machineWaiting{},
Reported: []machineReported{}, Unresolved: []machineUnresolved{},
Network: asked.network, Adopted: adoptedNodes(nodes)}
// In a stated order, so two readings of an unchanged mesh are the same document.
untakenNodes := make([]string, 0, len(asked.untaken))
for name := range asked.untaken {
untakenNodes = append(untakenNodes, name)
}
sort.Strings(untakenNodes)
for _, name := range untakenNodes {
modules := make([]string, 0, len(asked.untaken[name]))
for m := range asked.untaken[name] {
modules = append(modules, m)
}
sort.Strings(modules)
for _, m := range modules {
out.Untaken = append(out.Untaken,
machineUntaken{Node: name, Module: m, Held: asked.untaken[name][m]})
}
}
for name := range asked.refused {
out.Unresolved = append(out.Unresolved, machineUnresolved{
Node: name, Problem: asked.refused[name]})
+6 -2
View File
@@ -127,9 +127,13 @@ func readinessOf(ctx context.Context, inv *inventory.Inventory) (broker.Readines
if address != "" {
// One dial, briefly. "Is it answering" is the one fact records cannot hold, and a mesh about
// to move onto a server that is not there should hear it here rather than afterwards.
if conn, err := nats.Connect(broker.BareAddress(address), nats.Timeout(5*time.Second)); err == nil {
//
// **Dialled the way the mesh dials it** — credential and pin — because a bare connect to a
// bus that requires TLS and a user fails at the handshake, and the check then reported a
// standing server as absent (seen live, 2026-09-28).
if js, err := broker.Dial(address, nats.Timeout(5*time.Second)); err == nil {
state.ServerStanding = true
conn.Close()
js.Close()
}
}
@@ -0,0 +1,99 @@
package main
import (
"context"
"strings"
"testing"
)
// A node's own set failing to compose, and the mesh being unable to answer at all, are different
// things, and only the first may be passed over when something is gathered across every machine
// (novox/hq 04-ISSUES/152). These pin that distinction where the three gatherers rely on it.
func TestASetThatDoesNotComposeIsMarkedAsTheNodesOwnProblem(t *testing.T) {
open := aMesh(t)
one, two := rivals()
register(t, open, one)
register(t, open, two)
for _, m := range []string{one.Module, two.Module} {
if _, err := open.inventory.Assign(t.Context(), "laptop", m); err != nil {
t.Fatal(err)
}
}
_, _, err := planFor(t.Context(), open, "laptop")
if err == nil {
t.Fatal("two modules claiming one seat composed anyway")
}
if !unresolvable(err) {
t.Fatalf("a set that cannot compose was not marked as the node's own problem: %v", err)
}
}
func TestAStoreThatCannotBeReadIsNotANodeThatDoesNotCompose(t *testing.T) {
open := aMesh(t)
// Nothing is wrong with anchor. The question simply cannot be asked.
stopped, cancel := context.WithCancel(t.Context())
cancel()
_, _, err := planFor(stopped, open, "anchor")
if err == nil {
t.Fatal("a plan composed against a store that could not be read")
}
if unresolvable(err) {
t.Fatalf("a question the mesh could not answer was read as a node that runs nothing: %v", err)
}
}
func TestOneIncoherentNodeDoesNotCostTheRestTheirNames(t *testing.T) {
open := aMesh(t)
one, two := rivals()
register(t, open, one)
register(t, open, two)
for _, m := range []string{one.Module, two.Module} {
if _, err := open.inventory.Assign(t.Context(), "laptop", m); err != nil {
t.Fatal(err)
}
}
// laptop cannot compose. That is laptop's problem and nobody else's: the roster is still
// answerable, and anchor keeps whatever it serves.
if _, err := routeNamesInTheMesh(t.Context(), open); err != nil {
t.Fatalf("one node's broken set cost the whole mesh its roster: %v", err)
}
}
func TestARosterIsNeverReturnedWithNamesItCouldNotRead(t *testing.T) {
open := aMesh(t)
stopped, cancel := context.WithCancel(t.Context())
cancel()
names, err := routeNamesInTheMesh(stopped, open)
if err == nil {
t.Fatalf("a roster was composed from a store that could not be read: %v", names)
}
// The failure must be raised, not turned into an absence. A roster missing a machine's names
// is indistinguishable, on every machine that receives it, from the operator withdrawing them —
// and because the roster is part of every container's identity, it replaces all of them.
if names != nil {
t.Fatalf("a partial roster was returned beside the error: %v", names)
}
}
// Kept so the reason survives the next person reading it: the message the gatherer raises must say
// which machine could not be read, or the operator is left with a mesh-wide failure and no name.
func TestTheRaisedFailureNamesTheMachineItCouldNotRead(t *testing.T) {
open := aMesh(t)
stopped, cancel := context.WithCancel(t.Context())
cancel()
_, err := routeNamesInTheMesh(stopped, open)
if err == nil {
t.Fatal("no failure was raised")
}
if !strings.Contains(err.Error(), "cannot be read") {
t.Fatalf("the failure does not say the mesh could not be read: %v", err)
}
}
+197
View File
@@ -0,0 +1,197 @@
package main
import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"os"
"os/exec"
"strings"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/link"
)
// The mesh's own verbs, served as the mesh-controller seat's tools (novox/hq ADR 0154, design 33).
//
// **Each tool runs the command it names, in this same binary, and answers what it printed.** That is
// ADR 0035 taken literally: the logic lives once, in the command, and a surface is an adapter with no
// decisions in it. Running a fresh process rather than calling the function keeps two things true
// that calling it would not — every command opens and closes its own stores the way it does from a
// shell, and nothing a command prints to the process's standard output can leak into another call's
// answer. It also means a refusal is the same refusal in the same words, because it is the same
// output.
// verbAnswer is what a verb answers: what the command printed, whether it succeeded, and — where the
// command speaks JSON — the same as data.
type verbAnswer struct {
Output string `json:"output"`
OK bool `json:"ok"`
Answer any `json:"answer,omitempty"`
}
// argvFor is the command line a verb and its arguments become. Only the verbs the seat declares, and
// only the arguments each declares: a caller cannot reach a flag the schema did not name.
func argvFor(verb string, args map[string]any) ([]string, error) {
str := func(key string) string {
v, _ := args[key].(string)
return strings.TrimSpace(v)
}
need := func(keys ...string) error {
for _, k := range keys {
if str(k) == "" {
return fmt.Errorf("%s needs %q", verb, k)
}
}
return nil
}
switch verb {
case "status":
return []string{"status", "--json"}, nil
case "nodes":
return []string{"node", "list"}, nil
case "node":
if err := need("node"); err != nil {
return nil, err
}
return []string{"node", "show", str("node")}, nil
case "modules":
return []string{"module", "list"}, nil
case "seats":
return []string{"seats", "--json"}, nil
case "builds":
if id := str("log"); id != "" {
return []string{"builds", "--log", id}, nil
}
if m := str("module"); m != "" {
return []string{"builds", m}, nil
}
return []string{"builds"}, nil
case "plan":
if err := need("node"); err != nil {
return nil, err
}
return []string{"plan", str("node"), "--json"}, nil
case "assign", "unassign":
if err := need("node", "module"); err != nil {
return nil, err
}
return []string{verb, str("node"), str("module")}, nil
case "push":
// Sent and not waited for: the asker reads `status` for what the machine did, which is
// what a person at a shell does too. A tool call that blocked for a push's whole apply would
// time out on every machine that takes a minute, and say nothing about the ones that did not.
if n := str("node"); n != "" {
return []string{"push", n, "--wait", "0"}, nil
}
return []string{"push", "--behind", "--wait", "0"}, nil
case "build":
if err := need("repository"); err != nil {
return nil, err
}
argv := []string{"build", str("repository"), "--wait", "0"}
if p := str("path"); p != "" {
argv = append(argv, "--path", p)
}
if r := str("ref"); r != "" {
argv = append(argv, "--ref", r)
}
return argv, nil
}
return nil, fmt.Errorf("%q is not a verb the %s seat serves", verb, catalogue.ControllerSeatName)
}
// jsonVerbs are the verbs whose command speaks JSON, so the answer carries it as data as well.
var jsonVerbs = map[string]bool{"status": true, "seats": true, "plan": true}
// runVerb runs this binary with the given command line and gathers what it said.
func runVerb(ctx context.Context, argv []string) (verbAnswer, error) {
self, err := os.Executable()
if err != nil {
return verbAnswer{}, err
}
cmd := exec.CommandContext(ctx, self, argv...)
// The same environment: the stores' credentials, the bus, the broker — everything a command run
// from a shell in this container would have, because it is that.
cmd.Env = os.Environ()
// Two buffers, one answer. What the command *says* is both streams, in the order a person at
// a shell would read them; what it *answers as data* is standard output alone — `status --json`
// prints its warnings beside the document, and a JSON parsed from the two together parsed
// nothing (2026-09-30, the first status asked through the console had no `answer`).
var stdout, stderr bytes.Buffer
cmd.Stdout = &stdout
cmd.Stderr = &stderr
runErr := cmd.Run()
answer := verbAnswer{Output: stdout.String() + stderr.String(), OK: runErr == nil}
if jsonVerbs[argv[0]] && runErr == nil {
var parsed any
if json.Unmarshal(bytes.TrimSpace(stdout.Bytes()), &parsed) == nil {
answer.Answer = parsed
}
}
var exit *exec.ExitError
if runErr != nil && !errors.As(runErr, &exit) {
// Not the command refusing — the command not running at all, which is this process's fault.
return answer, fmt.Errorf("could not run %s: %w", strings.Join(argv, " "), runErr)
}
return answer, nil
}
// seatToolHandlers are the handlers for every verb the mesh-controller seat declares, from the
// store's row, so a verb the row does not carry is not served and a verb it carries that this binary
// cannot run is said at start rather than at the first call.
func seatToolHandlers() (map[string]link.ToolHandler, error) {
seat, known := catalogue.SeatNamed(catalogue.ControllerSeatName)
if !known {
return nil, fmt.Errorf("this mesh defines no %s seat", catalogue.ControllerSeatName)
}
handlers := map[string]link.ToolHandler{}
for _, v := range seat.Serves {
verb := v.Name
if verb == "tools" {
handlers[verb] = func(ctx context.Context, _ json.RawMessage) (any, error) {
return seatTools(), nil
}
continue
}
if _, err := argvFor(verb, map[string]any{"node": "x", "module": "x", "repository": "x"}); err != nil {
return nil, fmt.Errorf("the %s seat's row declares %q, which this control plane cannot run: %w",
catalogue.ControllerSeatName, verb, err)
}
handlers[verb] = func(ctx context.Context, raw json.RawMessage) (any, error) {
args := map[string]any{}
if len(raw) > 0 {
if err := json.Unmarshal(raw, &args); err != nil {
return nil, fmt.Errorf("the arguments are not a JSON object: %w", err)
}
}
argv, err := argvFor(verb, args)
if err != nil {
return nil, err
}
return runVerb(ctx, argv)
}
}
return handlers, nil
}
// seatTools is what `tools` answers: every seat with a protocol, and the tools each serves, from the
// mesh's own records — no holder in the path, so it is true while a holder restarts (design 33 §5).
func seatTools() map[string]any {
var seats []map[string]any
for _, s := range catalogue.SeatsWithAProtocol() {
if len(s.Serves) == 0 {
continue
}
var tools []map[string]any
for _, v := range s.Serves {
tools = append(tools, map[string]any{
"name": v.Name, "description": v.Description, "input": v.Input, "output": v.Output,
})
}
seats = append(seats, map[string]any{"seat": s.Name, "scope": s.Scope, "tools": tools})
}
return map[string]any{"seats": seats}
}
+109
View File
@@ -0,0 +1,109 @@
package main
import (
"strings"
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
)
// Every verb the mesh-controller seat declares is one this binary can run, with the arguments the
// schema names and no other (novox/hq ADR 0154, ADR 0035).
func TestEveryDeclaredVerbHasACommandLine(t *testing.T) {
for _, v := range catalogue.ControllerVerbs {
if v.Name == "tools" {
continue
}
args := map[string]any{}
props, _ := v.Input["properties"].(map[string]any)
for name := range props {
args[name] = "x"
}
argv, err := argvFor(v.Name, args)
if err != nil {
t.Errorf("%s: %v", v.Name, err)
continue
}
if argv[0] == "" {
t.Errorf("%s: empty command", v.Name)
}
}
}
// `builds` given a build's id reads that build's log from the bus rather than listing builds
// (novox/hq ADR 0157).
func TestBuildsWithAnIdReadsThatBuildsLog(t *testing.T) {
argv, err := argvFor("builds", map[string]any{"log": "build-17"})
if err != nil {
t.Fatal(err)
}
if strings.Join(argv, " ") != "builds --log build-17" {
t.Fatalf("builds with a log id became %q", strings.Join(argv, " "))
}
}
// A required argument missing is refused in the verb's own words, before anything runs.
func TestAVerbMissingWhatItNeedsIsRefused(t *testing.T) {
if _, err := argvFor("node", map[string]any{}); err == nil || !strings.Contains(err.Error(), `node needs "node"`) {
t.Fatalf("node without a machine was accepted: %v", err)
}
if _, err := argvFor("upgrade", map[string]any{}); err == nil {
t.Fatal("a verb the seat does not serve was accepted")
}
}
// A push and a build are sent, not waited for: the asker reads status for what happened.
func TestActsDoNotBlockTheCall(t *testing.T) {
argv, _ := argvFor("push", map[string]any{"node": "one"})
if strings.Join(argv, " ") != "push one --wait 0" {
t.Fatalf("push waits: %v", argv)
}
argv, _ = argvFor("build", map[string]any{"repository": "novox/x", "path": "modules/x"})
if strings.Join(argv, " ") != "build novox/x --wait 0 --path modules/x" {
t.Fatalf("build: %v", argv)
}
}
// What `tools` answers is the seats' records, with each verb's schema.
func TestToolsAnswersTheSeatsRecords(t *testing.T) {
handlers, err := seatToolHandlers()
if err != nil {
t.Fatal(err)
}
if len(handlers) != len(catalogue.ControllerVerbs) {
t.Fatalf("%d handlers for %d verbs", len(handlers), len(catalogue.ControllerVerbs))
}
answer := seatTools()
seats, _ := answer["seats"].([]map[string]any)
var found bool
for _, s := range seats {
if s["seat"] == catalogue.ControllerSeatName {
found = true
tools, _ := s["tools"].([]map[string]any)
if len(tools) != len(catalogue.ControllerVerbs) || tools[0]["input"] == nil {
t.Fatalf("the controller seat's tools are not listed in full: %v", tools)
}
}
}
if !found {
t.Fatal("the mesh-controller seat is not in the listing")
}
}
// A JSON verb's answer is parsed from what the command wrote to standard output alone; a warning it
// printed beside the document does not take the document away. The test binary stands in for the
// controller: `-test.run` with a name that matches nothing prints `ok` and a warning about no tests.
func TestAJSONVerbsAnswerIsItsStandardOutput(t *testing.T) {
jsonVerbs["-test.run"] = true
t.Cleanup(func() { delete(jsonVerbs, "-test.run") })
answer, err := runVerb(t.Context(), []string{"-test.run", "TestAnswerEcho", "-test.v"})
if err != nil {
t.Fatal(err)
}
if !answer.OK {
t.Fatalf("the command failed: %s", answer.Output)
}
if !strings.Contains(answer.Output, "PASS") {
t.Fatalf("stderr and stdout are both what the command said: %s", answer.Output)
}
}
+7
View File
@@ -18,6 +18,10 @@ import (
// make a machine look out of date for ever, or send something `plan` never showed.
type sendable struct {
Resources []map[string]any
// Sequence orders this send against every other to the same node: one higher each time, taken
// under the node's hold just before the body is made (novox/hq 04-ISSUES/107). Zero is not sent
// at all, which a host reads as "no order claimed" — the shape of every declaration before this.
Sequence int64
// Adoption is nil for a converged node, and then the body is byte for byte what it was before
// adoption existed: an older host parses the envelope strictly and would refuse the key.
Adoption *adoptionEnvelope
@@ -38,6 +42,9 @@ func (s sendable) Body() ([]byte, error) {
if s.Adoption != nil {
envelope["adoption"] = s.Adoption
}
if s.Sequence > 0 {
envelope["sequence"] = s.Sequence
}
// An empty declaration is deliberate here — the node owns nothing the mesh put there
// (novox/hq issue 127) — and the host refuses an empty body unless it is told the emptiness
// is meant, so a truncated or mis-composed body is never mistaken for "own nothing".
+77
View File
@@ -0,0 +1,77 @@
package main
import (
"encoding/json"
"testing"
)
// A declaration's only identity was the digest of its bytes; the controller already held a per-node
// lock and recorded each send, so the order existed and was thrown away at the wire (novox/hq
// 04-ISSUES/107).
func TestASendCarriesItsNumberInsideTheSignedBytes(t *testing.T) {
body, err := sendable{Resources: []map[string]any{{"id": "x", "type": "file"}}, Sequence: 7}.Body()
if err != nil {
t.Fatal(err)
}
var env map[string]any
if err := json.Unmarshal(body, &env); err != nil {
t.Fatal(err)
}
if got, _ := env["sequence"].(float64); got != 7 {
t.Fatalf("the body carries sequence %v, wanted 7", env["sequence"])
}
}
func TestAnUnnumberedSendIsByteForByteWhatItWasBefore(t *testing.T) {
// Zero is not sent at all. A host reads absence as "no order claimed" — the shape of every
// declaration before this — so an older host, or the read-only comparison against a machine
// sent nothing since sends were numbered, sees exactly the bytes it always saw.
body, err := sendable{Resources: []map[string]any{{"id": "x", "type": "file"}}}.Body()
if err != nil {
t.Fatal(err)
}
var env map[string]any
if err := json.Unmarshal(body, &env); err != nil {
t.Fatal(err)
}
if _, present := env["sequence"]; present {
t.Fatalf("a send numbered zero put a sequence on the wire: %s", body)
}
}
func TestEachSendToANodeIsOneHigherAndReadable(t *testing.T) {
open := aMesh(t)
record, err := open.inventory.NodeByName(t.Context(), "anchor")
if err != nil {
t.Fatal(err)
}
// Sent nothing since numbering existed: what it would be sent is composed with zero, which is
// not on the wire, which is what it was actually sent.
if n, err := open.inventory.Sequence(t.Context(), record.ID); err != nil || n != 0 {
t.Fatalf("a fresh node reads sequence %d, %v", n, err)
}
first, err := open.inventory.NextSequence(t.Context(), record.ID)
if err != nil {
t.Fatal(err)
}
second, err := open.inventory.NextSequence(t.Context(), record.ID)
if err != nil {
t.Fatal(err)
}
if first != 1 || second != 2 {
t.Fatalf("two sends were numbered %d and %d", first, second)
}
// And the read path sees the last one taken, so the comparison composes what was sent.
if n, err := open.inventory.Sequence(t.Context(), record.ID); err != nil || n != 2 {
t.Fatalf("after two sends the node reads sequence %d, %v", n, err)
}
// Another node counts on its own.
other, err := open.inventory.NodeByName(t.Context(), "laptop")
if err != nil {
t.Fatal(err)
}
if n, err := open.inventory.NextSequence(t.Context(), other.ID); err != nil || n != 1 {
t.Fatalf("a second node's first send was numbered %d, %v", n, err)
}
}
+39 -4
View File
@@ -82,6 +82,16 @@ func cloneFrom(ctx context.Context, source buildSource) (string, error) {
// serves no scheme or port has nothing to compose from — a default port here would be the forge's
// address guessed, which is the thing this exists to stop.
func clonedFromSeat(world catalogue.World, seatName, repository string) (string, error) {
base, err := seatBase(world, seatName)
if err != nil {
return "", err
}
path := strings.TrimSuffix(strings.Trim(repository, "/"), ".git")
return fmt.Sprintf("%s/%s.git", base, path), nil
}
// seatBase is `scheme://host:port` of a seat's holder as the mesh reaches it, for cloning.
func seatBase(world catalogue.World, seatName string) (string, error) {
seat, known := catalogue.SeatNamed(seatName)
if !known || seat.Delivers == "" {
return "", fmt.Errorf("%q is not a seat a repository can live on", seatName)
@@ -94,9 +104,9 @@ func clonedFromSeat(world catalogue.World, seatName, repository string) (string,
}
}
if holder == nil {
return "", fmt.Errorf("nobody holds the %s seat, so %s cannot be cloned from this mesh's "+
return "", fmt.Errorf("nobody holds the %s seat, so nothing can be cloned from this mesh's "+
"forge — assign a module that claims it, or build from the repository's URL without --self",
seat.Name, repository)
seat.Name)
}
var provider *catalogue.Provider
for i, p := range world.Offered[seat.Delivers] {
@@ -118,8 +128,33 @@ func clonedFromSeat(world catalogue.World, seatName, repository string) (string,
return "", fmt.Errorf("%s on %s holds the %s seat and does not serve a scheme and a port for %q",
holder.Module, holder.Node, seat.Name, seat.Delivers)
}
path := strings.TrimSuffix(strings.Trim(repository, "/"), ".git")
return fmt.Sprintf("%s://%s:%s/%s.git", scheme, provider.At, port, path), nil
return fmt.Sprintf("%s://%s:%s", scheme, provider.At, port), nil
}
// seatBases is the clone base of every seat a recipe's context may name, for a build request
// (novox/hq ADR 0155). A seat nobody holds is left out rather than refused here: the build may not
// name it at all, and if it does the builder refuses with the seat's name.
func seatBases(ctx context.Context) map[string]string {
open, err := openStores(ctx)
if err != nil {
return nil
}
defer open.Close()
shelf, err := open.inventory.Catalogue(ctx)
if err != nil {
return nil
}
world, err := theRestOfTheMesh(ctx, open.inventory, shelf, "")
if err != nil {
return nil
}
bases := map[string]string{}
for _, seatName := range []string{gitSeat} {
if base, err := seatBase(world, seatName); err == nil {
bases[seatName] = base
}
}
return bases
}
// servedPort is a served port as text, however the manifest and the node's settings carried it.
+179 -5
View File
@@ -46,15 +46,21 @@ func statusCommand(ctx context.Context, args []string) error {
return err
}
defer open.Close()
return statusFor(ctx, open, *asJSON)
}
// statusFor asks and answers, against stores somebody else opened.
//
// Split from the command so what it prints can be read by a test. The sentence it prints when nothing
// is wrong has been acted on and been misleading (novox/hq 04-ISSUES/145, 125), which makes its exact
// words the thing worth holding still.
func statusFor(ctx context.Context, open *stores, asJSON bool) error {
asked, err := theThreeQuestions(ctx, open)
if err != nil {
return err
}
wrong, nodes, quiet := asked.wrong, asked.nodes, asked.quiet
behind, sources := asked.behind, asked.sources
if *asJSON {
if asJSON {
body, err := statusAsJSON(asked)
if err != nil {
return err
@@ -62,6 +68,18 @@ func statusCommand(ctx context.Context, args []string) error {
fmt.Println(string(body))
return nil
}
return printStatus(asked)
}
// printStatus is the words, separated from the questions.
//
// **Its exact sentences have been acted on and been misleading twice** — a held module reading as a
// machine doing what it was told (novox/hq 04-ISSUES/125), and "all doing what they were told" being
// true of a mesh in which no module could reach another (04-ISSUES/145). So they are written where a
// test can read them without a store, a bus or a machine.
func printStatus(asked answers) error {
wrong, nodes, quiet := asked.wrong, asked.nodes, asked.quiet
behind, sources := asked.behind, asked.sources
if len(asked.refused) > 0 {
// First, above everything else. A machine that cannot be worked out is not running an old
@@ -171,6 +189,65 @@ func statusCommand(ctx context.Context, args []string) error {
fmt.Printf("\n `push --behind` sends them\n\n")
}
if split := hostSplit(nodes); len(split) > 1 {
// **Before a declaration gains a field, every machine has to understand it** (novox/hq
// 04-ISSUES/087). A host refuses a declaration carrying a field it does not know, and refuses
// it whole, so every new field is a flag day: hosts first, then the controller. The mesh had
// no record of which host any machine ran, so that order was kept by somebody remembering it.
//
// **Disagreement, and deliberately not "behind".** A host reports its version as a commit, and
// commits have no order — the first version of this said "N machines run an older host" and
// named the three that were newer, because it compared two hashes as strings. What the mesh
// can say truthfully is that the machines do not all run the same host, and which machines
// hold which. Ordering needs a version that is ordered, and that is the host's to report.
versions := make([]string, 0, len(split))
for v := range split {
versions = append(versions, v)
}
sort.Strings(versions)
fmt.Printf("%d machine(s) do not all run the same host:\n", len(nodes))
for _, v := range versions {
sort.Strings(split[v])
fmt.Printf(" %-12s %s\n", v, strings.Join(split[v], ", "))
}
fmt.Printf("\n a host refuses a declaration carrying a field it does not know, whole — so the\n" +
" mesh may send only what every one of these understands. Which of them is newer is\n" +
" not readable from a commit; that needs a version the host reports as ordered\n\n")
}
if len(asked.untaken) > 0 {
// **Before the adopted line, and it breaks "all well".** An adopted machine is a state
// somebody chose and can leave alone; a module assigned to one and never taken is work
// outstanding that reads exactly like work finished. That reading is what stopped a
// predecessor's proxy on the strength of four green surfaces (novox/hq 04-ISSUES/125).
machines := make([]string, 0, len(asked.untaken))
for name := range asked.untaken {
machines = append(machines, name)
}
sort.Strings(machines)
total := 0
for _, held := range asked.untaken {
for _, n := range held {
total += n
}
}
fmt.Printf("%d resource(s) are held as found, because their module was assigned and never "+
"taken — so it is running none of what it declares:\n", total)
for _, name := range machines {
modules := make([]string, 0, len(asked.untaken[name]))
for m := range asked.untaken[name] {
modules = append(modules, m)
}
sort.Strings(modules)
parts := make([]string, 0, len(modules))
for _, m := range modules {
parts = append(parts, fmt.Sprintf("%s (%d)", m, asked.untaken[name][m]))
}
fmt.Printf(" %-12s %s\n", name, strings.Join(parts, ", "))
}
fmt.Printf("\n `take <node> <module>` compares what runs against what it declares, and runs it\n\n")
}
if adopted := adoptedNodes(nodes); len(adopted) > 0 {
// Said, because nothing forces the flip: a node left adopted is visible here rather than
// read as converged (novox/hq ADR 0100). Not a fault, so it does not break "all well".
@@ -178,12 +255,23 @@ func statusCommand(ctx context.Context, args []string) error {
fmt.Printf("\n `converge <node>` previews the flip\n\n")
}
if len(wrong) == 0 && len(quiet) == 0 && len(behind) == 0 && len(asked.waiting) == 0 &&
len(asked.refused) == 0 && asked.network == "" {
if asked.well() {
// Said plainly. "Nothing to report" and "nothing was checked" must never look the same,
// and getting here means every question was asked and answered.
fmt.Printf("%d machine(s), all doing what they were told, all heard from, running what "+
"the mesh would send them, and every module current with its source\n", len(nodes))
// **And what that sentence does not cover**, because for eleven hours it was true of a mesh
// in which no module could reach another (novox/hq 04-ISSUES/145). Every question above is
// about the relationship between the mesh and a machine — applied what it was sent, matches
// what would be sent, built from what the source has. None of them asks whether a module can
// reach what it requires, and the mesh composes every one of those grants itself.
//
// Said here rather than left to be inferred. A reader who acts on the line above is acting on
// "the machines are as the mesh described them", and the distance between that and "it works"
// is where the eleven hours went.
fmt.Printf("\n That is the mesh and the machines agreeing. Nothing here dials a provision:\n" +
" no grant the mesh composed has been tested, so a module unable to reach what it\n" +
" requires would not appear above (04-ISSUES/145)\n")
}
return nil
}
@@ -247,6 +335,13 @@ func theThreeQuestions(ctx context.Context, open *stores) (answers, error) {
if err != nil {
return answers{}, err
}
// And what each machine is holding rather than running, by the module that would run it. Read
// from what the machine itself last reported, not from what take-time computed: the machine is
// the only thing that knows what it found (novox/hq 04-ISSUES/125).
out.untaken, err = untakenModules(ctx, inv, out.nodes)
if err != nil {
return answers{}, err
}
// And which machines are not running what the mesh would send them. The same question as a
// module being behind its source, one level down: that one says the catalogue is out of date,
@@ -281,3 +376,82 @@ func theThreeQuestions(ctx context.Context, open *stores) (answers, error) {
}
return out, nil
}
// untakenModules is, per machine, each module whose resources that machine is holding as found, and
// how many.
//
// **The machine's own account, not the mesh's.** An adopted node decides at apply time what it found
// and reports it; the mesh's take-time listing is a different thing and was the one this command used
// to have, which is why a module assigned after the listing showed nothing at all
// (novox/hq 04-ISSUES/125).
//
// A machine that reports no holds contributes nothing, so a converged mesh answers an empty map and
// the caller prints nothing.
func untakenModules(ctx context.Context, inv *inventory.Inventory, nodes []inventory.Node) (
map[string]map[string]int, error) {
out := map[string]map[string]int{}
for _, n := range nodes {
said, err := inv.AdoptionOf(ctx, n.Name)
if err != nil {
// A machine whose record cannot be read is not a machine holding nothing. Said, because
// answering "nothing held" from a failed read is the shape this whole issue is about.
return nil, fmt.Errorf("what %s is holding cannot be read: %w", n.Name, err)
}
for _, h := range said.Held {
if h.Module == "" {
continue // a hold the mesh cannot attribute to a module has nothing to take
}
if out[n.Name] == nil {
out[n.Name] = map[string]int{}
}
out[n.Name][h.Module]++
}
}
return out, nil
}
// well is whether every question this command asks came back with nothing to say.
//
// Named, and in one place, because it is the sentence an operator acts on and it has been wrong
// twice. It is deliberately NOT "nothing is broken": a machine holding what it found is not broken
// and is not doing what it was told either.
//
// **A hold suppresses it; being adopted does not.** Adopted is a mode somebody chose and can leave
// alone. A module assigned to a machine and never taken is a half-finished action with nothing left
// to finish it — it runs none of what it declares, and "all doing what they were told" was true and
// read as success for the whole of the edge cut-over outage (novox/hq 04-ISSUES/125).
func (a answers) well() bool {
return len(a.wrong) == 0 && len(a.quiet) == 0 && len(a.behind) == 0 &&
len(a.waiting) == 0 && len(a.refused) == 0 && a.network == "" && len(a.untaken) == 0
}
// hostSplit is which machines report which host version, for every version more than one machine
// could disagree about.
//
// **It does not say which is newer, because it cannot.** A host reports its version as a commit, and
// commits have no order. The first version of this returned "the machines behind the newest" by
// comparing versions as strings, and on the live mesh it named the three machines running the NEWER
// host as the ones behind — an arbitrary lexicographic result presented as a fact
// (novox/hq 04-ISSUES/087). A report that confidently says the opposite of the truth is worse than one
// that says less, which is the whole subject of 04-ISSUES/145.
//
// So this answers what is checkable: who runs what. The reader sees the split and the mesh claims no
// ordering. Ordering wants an ordered version, and that is the host's to report rather than this
// function's to infer.
//
// Machines that have not reported a version are left out entirely: they are not a version, and
// counting them as one would invent a disagreement. `node show` says per machine that it has not said.
func hostSplit(nodes []inventory.Node) map[string][]string {
out := map[string][]string{}
for _, n := range nodes {
if n.HostVersion == "" {
continue
}
out[n.HostVersion] = append(out[n.HostVersion], n.Name)
}
if len(out) < 2 {
return nil // one version, or none reported: nothing to disagree about
}
return out
}
+122
View File
@@ -0,0 +1,122 @@
package main
import (
"encoding/json"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/inventory"
)
// A module assigned to an adopted machine and never taken runs none of what it declares, and every
// surface called that success — a push reporting sent, a journal reporting applied, status reporting
// a machine doing what it was told (novox/hq 04-ISSUES/125). The holds were only ever in the
// machine's own state file.
// heldOn makes a machine report that it is holding resources for a module, the way an adopted node
// does after an apply.
func heldOn(t *testing.T, open *stores, node, module string, ids ...string) {
t.Helper()
record, err := open.inventory.NodeByName(t.Context(), node)
if err != nil {
t.Fatal(err)
}
held := make([]inventory.Held, 0, len(ids))
for _, id := range ids {
held = append(held, inventory.Held{ID: id, Module: module, Kind: "container", Target: id})
}
if err := open.inventory.RecordAdoption(t.Context(), record.ID, held, "ufw", nil); err != nil {
t.Fatal(err)
}
}
func TestStatusNamesAModuleHeldBecauseNothingTookIt(t *testing.T) {
open := aMesh(t)
heldOn(t, open, "anchor", "route-proxy", "ca", "certs", "server")
asked, err := theThreeQuestions(t.Context(), open)
if err != nil {
t.Fatal(err)
}
if got := asked.untaken["anchor"]["route-proxy"]; got != 3 {
t.Fatalf("status counted %d resources held for route-proxy, wanted 3", got)
}
}
func TestAHeldModuleStopsTheMeshReadingAsWell(t *testing.T) {
// The whole of the fault. "all doing what they were told" was true throughout the outage, and
// true is not the same as safe to act on: the machine was doing what it was told, and what it
// was told had not started. Asserted against the production condition, not a copy of it.
quiet := answers{}
if !quiet.well() {
t.Fatal("a mesh with nothing to say does not read as well, so nothing below means anything")
}
holding := answers{untaken: map[string]map[string]int{"anchor": {"route-proxy": 3}}}
if holding.well() {
t.Fatal("a machine holding a module's resources still reads as doing what it was told, " +
"which is the sentence that cost every public name on the machine")
}
// And being adopted does not suppress it: that is a mode somebody chose, not work outstanding.
// Kept as an assertion so the difference between the two is deliberate rather than incidental.
if !quiet.well() {
t.Fatal("the well condition is not stable")
}
}
func TestAHeldModuleIsFoundFromWhatTheMachineReported(t *testing.T) {
// End to end through the store, so the condition above is reached by real data and not only by
// a constructed value: the machine reports, the mesh records, status asks.
open := aMesh(t)
heldOn(t, open, "anchor", "route-proxy", "ca", "server")
asked, err := theThreeQuestions(t.Context(), open)
if err != nil {
t.Fatal(err)
}
if len(asked.untaken) == 0 {
t.Fatal("what the machine reported holding did not reach status")
}
if asked.well() {
t.Fatal("a mesh whose machine reported holds reads as well")
}
}
func TestTheJSONStatusCarriesWhatIsHeldAndForWhichModule(t *testing.T) {
open := aMesh(t)
heldOn(t, open, "anchor", "route-proxy", "ca", "certs")
asked, err := theThreeQuestions(t.Context(), open)
if err != nil {
t.Fatal(err)
}
body, err := statusAsJSON(asked)
if err != nil {
t.Fatal(err)
}
var doc struct {
Untaken []struct {
Node string `json:"node"`
Module string `json:"module"`
Held int `json:"held"`
} `json:"untaken"`
}
if err := json.Unmarshal(body, &doc); err != nil {
t.Fatal(err)
}
if len(doc.Untaken) != 1 {
t.Fatalf("the document carries %d untaken rows, wanted 1: %s", len(doc.Untaken), body)
}
row := doc.Untaken[0]
if row.Node != "anchor" || row.Module != "route-proxy" || row.Held != 2 {
t.Fatalf("the row is %+v, wanted anchor/route-proxy/2", row)
}
// Absent rather than empty when nothing is held, so a well mesh's document does not carry a
// field a reader has to interpret.
clean, err := statusAsJSON(answers{})
if err != nil {
t.Fatal(err)
}
if strings.Contains(string(clean), "untaken") {
t.Fatalf("a mesh holding nothing still names untaken: %s", clean)
}
}
+172 -12
View File
@@ -232,22 +232,67 @@ func (f following) SourceMoved(ctx context.Context, m link.SourceMoved) error {
if err != nil {
return notNow(err)
}
var moved []inventory.Entry
read, err := inv.ReadRepositories(ctx)
if err != nil {
return notNow(err)
}
// Two kinds of module are affected by one merge, and they are affected differently.
//
// A module **built from** this repository and branch has moved: the mesh records the new commit
// as what its source now has, and only what the merge actually changed is rebuilt. A module that
// only **packages source from** it has not moved — its own source is somewhere else, at the
// commit it already records — so it is rebuilt and its record left alone. Writing this commit as
// its source would make it permanently behind a repository its manifest does not come from.
var from, packaging []inventory.Entry
already := 0
for _, e := range entries {
if !sourceIs(e.Source, m) {
continue
}
switch {
case sourceIs(e.Source, m):
if e.Source.BuiltFrom == m.Commit {
already++
continue
}
// **A merge older than the last look at the source is history, not a move.** The forge
// announces what it finds merged, and an old merge surfacing late would otherwise move
// the recorded head backwards and rebuild everything built from that repository, once
// per old merge (2026-09-28).
if isHistory(m.MergedAt, e.Source.Seen) {
continue
}
from = append(from, e)
case readsFrom(read[e.Manifest.Module], m):
packaging = append(packaging, e)
}
}
if len(from) == 0 && len(packaging) == 0 {
// "Already built from it" and "nothing reads it" are different facts, and reading the first
// as the second sends somebody looking for a broken trigger when the mesh is up to date.
if already > 0 {
fmt.Printf("%s/%s merged into %s (%.8s); %d module(s) the mesh holds are already built "+
"from it\n", m.Owner, m.Repo, m.Base, m.Commit, already)
return nil
}
fmt.Printf("%s/%s merged into %s (%.8s); nothing the mesh holds reads it\n",
m.Owner, m.Repo, m.Base, m.Commit)
return nil
}
// The same judgement for the packaging kind, against the newest look at that repository by
// anything built from it: they keep no record of it themselves, and a replayed old merge should
// not rebuild them either.
if isHistory(m.MergedAt, lastLookAt(entries, m)) {
packaging = nil
}
touched := whatTheMergeTouched(from, entries, m)
for _, e := range touched {
if err := inv.SourceMoved(ctx, e.Manifest.Module, m.Commit); err != nil {
return notNow(err)
}
moved = append(moved, e)
}
moved := append(append([]inventory.Entry{}, touched...), packaging...)
if len(moved) == 0 {
fmt.Printf("%s/%s merged into %s (%.8s); nothing the mesh holds is built from it\n",
m.Owner, m.Repo, m.Base, m.Commit)
fmt.Printf("%s/%s merged into %s (%.8s); it changed nothing any module the mesh holds is "+
"built from\n", m.Owner, m.Repo, m.Base, m.Commit)
return nil
}
against, err := inv.BuiltAgainst(ctx)
@@ -261,6 +306,14 @@ func (f following) SourceMoved(ctx context.Context, m link.SourceMoved) error {
}
fmt.Printf("%s/%s merged into %s (%.8s); building %s\n",
m.Owner, m.Repo, m.Base, m.Commit, strings.Join(names, ", "))
if len(packaging) > 0 {
var also []string
for _, e := range packaging {
also = append(also, e.Manifest.Module)
}
fmt.Printf(" %s package source from it, so they are rebuilt and their own source record "+
"is left where it is\n", strings.Join(also, ", "))
}
var failed []string
for _, e := range ordered {
source := buildSource{Repository: e.Source.Repository, Seat: e.Source.Seat}
@@ -286,16 +339,110 @@ func (f following) SourceMoved(ctx context.Context, m link.SourceMoved) error {
// An empty recorded ref is the repository's default branch, which is what a merge into the base
// branch of the forge's default means.
func sourceIs(s inventory.Source, m link.SourceMoved) bool {
want := strings.ToLower(m.Owner + "/" + m.Repo)
repo := strings.ToLower(strings.TrimSuffix(s.Repository, ".git"))
matches := repo == want || strings.HasSuffix(repo, "/"+want) ||
(m.CloneURL != "" && strings.EqualFold(strings.TrimSuffix(s.Repository, ".git"), strings.TrimSuffix(m.CloneURL, ".git")))
if !matches {
if !sameRepository(s.Repository, m) {
return false
}
return s.Ref == "" || s.Ref == m.Base
}
// sameRepository is whether a recorded repository is the one a merge names, in either spelling it
// may have been recorded in: a path on the git seat, or the URL it was cloned from.
func sameRepository(repository string, m link.SourceMoved) bool {
want := strings.ToLower(m.Owner + "/" + m.Repo)
repo := strings.ToLower(strings.TrimSuffix(repository, ".git"))
return repo == want || strings.HasSuffix(repo, "/"+want) ||
(m.CloneURL != "" && repo == strings.ToLower(strings.TrimSuffix(m.CloneURL, ".git")))
}
// readsFrom is whether a module's build read the repository a merge names: the second repository its
// recipe packages source from. Its ref must be the branch that moved, or unset — the same rule a
// module's own source follows.
func readsFrom(read []inventory.ReadRepository, m link.SourceMoved) bool {
for _, r := range read {
if sameRepository(r.Repository, m) && (r.Ref == "" || r.Ref == m.Base) {
return true
}
}
return false
}
// lastLookAt is the most recent look at this repository by anything built from it.
func lastLookAt(entries []inventory.Entry, m link.SourceMoved) time.Time {
var newest time.Time
for _, e := range entries {
if sameRepository(e.Source.Repository, m) && e.Source.Seen.After(newest) {
newest = e.Source.Seen
}
}
return newest
}
// whatTheMergeTouched narrows the modules built from a repository to the ones the merge changed.
//
// **A change inside no module's own directory is a change to what they share.** The forge lists the
// files a merge changed; a module is affected when one of them is inside its own directory, when it
// is built from the repository's root — everything there is its source — or when some changed file
// belongs to no module's directory at all, which is how a shared file, a build recipe or a
// dependency at the root rebuilds everything built from that repository.
//
// A change inside *another* module's directory is that module's business and not this one's, even
// when the mesh does not hold that module: `known` is every module this repository is known to hold,
// whatever branch it was registered from. That is also the limit of this — a repository whose shared
// code sits inside a directory the mesh has never seen a module in reads as shared, and everything
// is rebuilt. Rebuilding too much is the safe direction: the fault this whole path exists for is a
// mesh that believes it is current and is not (novox/hq 04-ISSUES/131).
func whatTheMergeTouched(candidates, known []inventory.Entry, m link.SourceMoved) []inventory.Entry {
// Nothing said about the files, or not all of them said: everything built from it is affected.
if len(m.Paths) == 0 || m.PathsTruncated {
return candidates
}
var dirs []string
for _, e := range known {
if e.Source.Path != "" && sameRepository(e.Source.Repository, m) {
dirs = append(dirs, e.Source.Path)
}
}
for _, p := range m.Paths {
if !insideAny(p, dirs) {
return candidates
}
}
var out []inventory.Entry
for _, e := range candidates {
if e.Source.Path == "" || anyInside(m.Paths, e.Source.Path) {
out = append(out, e)
}
}
return out
}
// inside is whether a changed file is in a directory: that directory itself, or under it.
func inside(path, dir string) bool {
dir = strings.Trim(dir, "/")
path = strings.TrimPrefix(path, "/")
return path == dir || strings.HasPrefix(path, dir+"/")
}
// insideAny is whether a changed file is in any of these directories.
func insideAny(path string, dirs []string) bool {
for _, dir := range dirs {
if inside(path, dir) {
return true
}
}
return false
}
// anyInside is whether any of these changed files is in a directory.
func anyInside(paths []string, dir string) bool {
for _, p := range paths {
if inside(p, dir) {
return true
}
}
return false
}
// orderByBases is the entries with every base before what stands on it: a module whose build stood
// on another's artifact comes after that module. Entries outside the set are not waited for — they
// are not being rebuilt. Stable for what has no order between it.
@@ -362,3 +509,16 @@ func standsOnModule(e inventory.Entry, module string, against map[string][]strin
}
return false
}
// isHistory is whether a merge made at mergedAt predates the last time the source was seen. A merge
// with no time on it is taken as news: refusing it would silence a forge that says less.
func isHistory(mergedAt string, seen time.Time) bool {
if mergedAt == "" || seen.IsZero() {
return false
}
at, err := time.Parse(time.RFC3339, mergedAt)
if err != nil {
return false
}
return at.Before(seen)
}
+75
View File
@@ -0,0 +1,75 @@
package main
import (
"bytes"
"io"
"os"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/inventory"
)
// "4 machine(s), all doing what they were told, all heard from, running what the mesh would send
// them, and every module current with its source" was true for eleven hours of a mesh in which no
// module could reach another (novox/hq 04-ISSUES/145). Every question it answers is about the mesh
// and a machine agreeing; none of them dials anything.
// printed captures what a function writes to stdout.
func printed(t *testing.T, f func() error) string {
t.Helper()
old := os.Stdout
r, w, err := os.Pipe()
if err != nil {
t.Fatal(err)
}
os.Stdout = w
runErr := f()
_ = w.Close()
os.Stdout = old
var buf bytes.Buffer
if _, err := io.Copy(&buf, r); err != nil {
t.Fatal(err)
}
if runErr != nil {
t.Fatal(runErr)
}
return buf.String()
}
func TestTheAllWellSentenceSaysWhatItDoesNotCover(t *testing.T) {
// A mesh with nothing to say. The sentence below was true of a mesh in which no module could
// reach another, for eleven hours.
got := printed(t, func() error {
return printStatus(answers{nodes: []inventory.Node{{Name: "anchor"}, {Name: "laptop"}}})
})
if !strings.Contains(got, "all doing what they were told") {
t.Fatalf("a mesh with nothing to say did not print the all-well sentence:\n%s", got)
}
// And now says what it is not a claim about.
for _, want := range []string{"Nothing here dials a provision", "04-ISSUES/145"} {
if !strings.Contains(got, want) {
t.Fatalf("the all-well sentence does not say %q:\n%s", want, got)
}
}
}
func TestAMeshWithSomethingToSayDoesNotPrintTheScopeLine(t *testing.T) {
// The scope belongs to the all-well sentence. A mesh with something wrong has specific things to
// read, and appending a caveat to those is noise.
got := printed(t, func() error {
return printStatus(answers{
nodes: []inventory.Node{{Name: "anchor"}},
untaken: map[string]map[string]int{"anchor": {"route-proxy": 3}},
})
})
if strings.Contains(got, "Nothing here dials a provision") {
t.Fatalf("a mesh with a held module printed the all-well scope line:\n%s", got)
}
if strings.Contains(got, "all doing what they were told") {
t.Fatalf("a mesh with a held module printed the all-well sentence:\n%s", got)
}
if !strings.Contains(got, "route-proxy") {
t.Fatalf("the held module is not named:\n%s", got)
}
}
+1 -1
View File
@@ -67,7 +67,7 @@ func TestTheAgreementCheckCatchesASubscriptionThatMatchesNothing(t *testing.T) {
// An event published under a seat's name is real even though no module declares it as its own.
if bad := Disagreements(nil,
[]AConsumer{{Module: "watcher", Consumes: []string{"mesh-artifact-store.image.pushed"}}},
[]DeclaredSeat{{Name: "the-artifact-store", Emits: []string{"image.pushed"}}}); len(bad) != 0 {
[]DeclaredSeat{{Name: "mesh-artifact-store", Emits: []string{"image.pushed"}}}); len(bad) != 0 {
t.Fatalf("an event a seat emits was reported as matching nothing: %v", bad)
}
}
+13
View File
@@ -66,6 +66,19 @@ func FromEnvironment() (Broker, error) {
if err != nil {
return Broker{}, err
}
// **One bus, one address** (novox/hq ADR 0131). Everything the mesh hands out — a token, a
// machine's membership, a person's credential — must name the bus the control plane itself is
// connected to; the setting above predates the move and, on a mesh that has moved, still names
// the broker it moved from. The first person issued after the move was handed the retired
// broker's port and could not connect to anything (2026-09-28).
//
// Read from the credential rather than from a second setting somebody keeps in step: the
// control plane cannot be wrong about where it is connected.
if bus, on, err := OnNATS(); err == nil && on {
if where := strings.TrimPrefix(BareAddress(bus), "nats://"); where != "" {
address = where
}
}
return Broker{Address: address, Fingerprint: fingerprint}, nil
}
+178
View File
@@ -0,0 +1,178 @@
package broker
import (
"fmt"
"os"
"strings"
"testing"
"time"
"github.com/nats-io/nats.go"
)
const twoSeconds = 2 * time.Second
// A running mesh already holds consumers made before the delivery subject carried the stream
// (novox/hq 04-ISSUES/146). The server will not change a push consumer's delivery subject in place,
// so bringing one to match must replace it — and must not replay what it already acknowledged
// (novox/hq 04-ISSUES/156).
//
// docker run -d --rm --name t -p 14231:4222 nats:2.10-alpine -js
// MESH_TEST_NATS=nats://127.0.0.1:14231 go test ./internal/broker/ -run TestUpgrading
func TestUpgradingAConsumerWhoseDeliverySubjectMoved(t *testing.T) {
url := os.Getenv("MESH_TEST_NATS")
if url == "" {
t.Skip("MESH_TEST_NATS unset")
}
js, err := Dial(url)
if err != nil {
t.Fatal(err)
}
defer js.Close()
// The stream exactly as the mesh's own is — one declaration per node, always the newest.
// Reproduced rather than approximated: the first version of this test used a plain stream and
// a plain consumer, and the server accepted the update it refuses in a running mesh, so the
// test passed against the very code that was crash-looping on the control node.
const stream, name = "NODES", "novox"
subject := "mesh.node." + name + ".declare"
_ = js.js.DeleteStream(stream)
if _, err := js.js.AddStream(&nats.StreamConfig{
Name: stream, Subjects: []string{"mesh.node.*.declare"},
MaxMsgsPerSubject: 1, Storage: nats.MemoryStorage,
}); err != nil {
t.Fatal(err)
}
defer func() { _ = js.js.DeleteStream(stream) }()
for i := 0; i < 6; i++ {
if _, err := js.js.Publish(subject, []byte(fmt.Sprint(i))); err != nil {
t.Fatal(err)
}
}
// The consumer as a running mesh holds it: made before the subject carried the stream, and
// otherwise exactly what NodeConsumer asks for.
if _, err := js.js.AddConsumer(stream, &nats.ConsumerConfig{
Durable: name, AckPolicy: nats.AckExplicitPolicy,
AckWait: 300 * time.Second, MaxDeliver: -1,
FilterSubject: subject,
DeliverSubject: "_DELIVER." + name,
}); err != nil {
t.Fatal(err)
}
// It acknowledged the first four. Those must not come back.
sub, err := js.js.SubscribeSync(subject, nats.Bind(stream, name))
if err != nil {
t.Fatal(err)
}
for i := 0; i < 1; i++ {
m, err := sub.NextMsg(twoSeconds)
if err != nil {
t.Fatalf("message %d never arrived: %v", i, err)
}
if err := m.AckSync(); err != nil {
t.Fatal(err)
}
}
// **The subscription stays up.** In a running mesh the machine is attached to this consumer
// the whole time — that is what a node listening for its declaration IS. The first version of
// this test unsubscribed first, and the server then accepted an update it refuses while a
// subscriber is bound, so the test passed against the code that was crash-looping.
defer func() { _ = sub.Unsubscribe() }()
// Now the upgrade: the consumer the controller asserts on every start, with the subject that
// carries the stream.
want := NodeConsumer(name)
var notes []string
js.Note = func(f string, a ...any) { notes = append(notes, fmt.Sprintf(f, a...)) }
if err := js.EnsureConsumer(want); err != nil {
t.Fatalf("a consumer the mesh already held could not be brought to match, which is the "+
"control plane failing to start: %v", err)
}
info, err := js.js.ConsumerInfo(stream, name)
if err != nil {
t.Fatal(err)
}
// It KEEPS the subject it had. Moving it would need the holder's grant to have widened first,
// and that grant travels in the bus's user list, which a machine applies minutes later.
if got := info.Config.DeliverSubject; got != "_DELIVER."+name {
t.Fatalf("the consumer a machine is bound to was moved to %q; a machine not yet allowed "+
"to subscribe there is a machine that hears nothing", got)
}
if len(notes) != 1 {
t.Fatalf("keeping it was not reported, so it would be invisible: %v", notes)
}
if !strings.Contains(notes[0], "keeps working") {
t.Fatalf("the note does not say the consumer still works: %q", notes[0])
}
// And the machine bound to it is still being delivered to — the point of keeping it.
if _, err := js.js.Publish(subject, []byte("after the assertion")); err != nil {
t.Fatal(err)
}
m, err := sub.NextMsg(twoSeconds)
if err != nil {
t.Fatalf("the machine stopped hearing its declarations after the assertion: %v", err)
}
if string(m.Data) != "after the assertion" {
t.Fatalf("delivered %q", m.Data)
}
// Asserting again is a no-op, or the controller crash-loops on its own restart.
if err := js.EnsureConsumer(want); err != nil {
t.Fatalf("the second assertion failed: %v", err)
}
}
// And where nothing is bound, the subject DOES move — that is 04-ISSUES/146's fix, which this must
// not undo. The controller's own two consumers are in exactly this position: it asserts them before
// it subscribes.
func TestAConsumerNothingIsBoundToDoesMove(t *testing.T) {
url := os.Getenv("MESH_TEST_NATS")
if url == "" {
t.Skip("MESH_TEST_NATS unset")
}
js, err := Dial(url)
if err != nil {
t.Fatal(err)
}
defer js.Close()
const stream, name = "NODES", "shanks"
subject := "mesh.node." + name + ".declare"
_ = js.js.DeleteStream(stream)
if _, err := js.js.AddStream(&nats.StreamConfig{
Name: stream, Subjects: []string{"mesh.node.*.declare"},
MaxMsgsPerSubject: 1, Storage: nats.MemoryStorage,
}); err != nil {
t.Fatal(err)
}
defer func() { _ = js.js.DeleteStream(stream) }()
if _, err := js.js.AddConsumer(stream, &nats.ConsumerConfig{
Durable: name, AckPolicy: nats.AckExplicitPolicy,
AckWait: 300 * time.Second, MaxDeliver: -1,
FilterSubject: subject,
DeliverSubject: "_DELIVER." + name,
}); err != nil {
t.Fatal(err)
}
want := NodeConsumer(name)
if err := js.EnsureConsumer(want); err != nil {
t.Fatal(err)
}
info, err := js.js.ConsumerInfo(stream, name)
if err != nil {
t.Fatal(err)
}
if got := info.Config.DeliverSubject; got != DeliverSubjectFor(want) {
t.Fatalf("delivery subject is %q, wanted %q -- issue 146's fix no longer applies to a "+
"consumer nothing is holding", got, DeliverSubjectFor(want))
}
}
+7
View File
@@ -40,6 +40,13 @@ type Consumer struct {
AckWaitSeconds int
// MaxDeliver before the message is dead-lettered; zero for the mesh's default.
MaxDeliver int
// MaxAckPending is how many deliveries the server lets stand unacknowledged at once; zero for
// the server's default, which is many. **One, for a consumer handled one at a time**
// (novox/hq issue 175): a handler that builds for minutes keeps its own message alive with a
// heartbeat, but everything handed over behind it times out unacknowledged and comes back —
// and a merge that came back rebuilt what it had just built, five times over on 2026-09-30.
// With one outstanding, the server holds the rest, and the heartbeat is keeping the message.
MaxAckPending int
Why string
}
+17 -11
View File
@@ -62,6 +62,22 @@ func TestTheInstallersFirstUserListIsWhatTheControllerWouldCompose(t *testing.T)
// theCarriedAccounts is the accounts file the installer's template writes at genesis.
func theCarriedAccounts(t *testing.T) string {
t.Helper()
for _, r := range theTemplate(t) {
if r["id"] == "bus-accounts" {
content, _ := r["content"].(string)
if content == "" {
t.Fatal("the template's accounts file is empty, so the bus would refuse every connection")
}
return content
}
}
t.Fatal("the template carries no accounts file, so a mesh raised from it has a bus nobody may use")
return ""
}
// theTemplate is the installer's bundle, as resources.
func theTemplate(t *testing.T) []map[string]any {
t.Helper()
path := filepath.Join("..", "..", "..", "mesh-host", "examples", "foundation-first-node-nats.lock")
raw, err := os.ReadFile(path)
@@ -81,17 +97,7 @@ func theCarriedAccounts(t *testing.T) string {
if err := json.Unmarshal([]byte(strings.Join(lines, "\n")), &bundle); err != nil {
t.Fatalf("the template is not readable: %v", err)
}
for _, r := range bundle.Resources {
if r["id"] == "bus-accounts" {
content, _ := r["content"].(string)
if content == "" {
t.Fatal("the template's accounts file is empty, so the bus would refuse every connection")
}
return content
}
}
t.Fatal("the template carries no accounts file, so a mesh raised from it has a bus nobody may use")
return ""
return bundle.Resources
}
// subjectsIn reads one allow-list out of a composed accounts file.
+93
View File
@@ -0,0 +1,93 @@
package broker
import (
"strings"
"testing"
)
// A module that says it calls a tool may publish exactly that subject (novox/hq ADR 0152): the same
// grant a person gets, derived the same way, so one list answers "what may this ask" for everybody.
func TestAModuleMayAskOnlyTheToolsItInvokes(t *testing.T) {
perms, err := PermissionsFor(Principal{Kind: KindModule, Node: "desk", Module: "mesh-console",
Invokes: []string{"shop.price"}, PasswordHash: "x"})
if err != nil {
t.Fatal(err)
}
has(t, perms.Publish, "mesh.mod.shop.tool.price")
hasNot(t, perms.Publish, "mesh.mod.shop.tool.refund")
hasNot(t, perms.Publish, "mesh.mod.*.tool.>")
}
// The console's grant: every tool, as one subject, and it reads as one.
func TestAModuleInvokingEverythingMayAskAnyTool(t *testing.T) {
perms, err := PermissionsFor(Principal{Kind: KindModule, Node: "desk", Module: "mesh-console",
Invokes: []string{"*"}, PasswordHash: "x"})
if err != nil {
t.Fatal(err)
}
has(t, perms.Publish, "mesh.mod.*.tool.>")
}
// **A grant to call widens nothing else.** A module that invokes may not publish an event it did not
// declare, may not answer as another module, and subscribes nothing it did not consume — the
// difference between the console and a person is that the console is on a machine, not that it may
// do more.
func TestInvokingGrantsNothingButTheCall(t *testing.T) {
perms, err := PermissionsFor(Principal{Kind: KindModule, Node: "desk", Module: "mesh-console",
Invokes: []string{"*"}, PasswordHash: "x"})
if err != nil {
t.Fatal(err)
}
for _, p := range perms.Publish {
if strings.Contains(p, ".event.") {
t.Errorf("a module that only invokes may publish %q, an event it never declared", p)
}
// A role's tools are tools (ADR 0132); a role's work queue and events are not.
if strings.HasPrefix(p, "mesh.seat.") && !strings.Contains(p, ".tool.") {
t.Errorf("a module that only invokes may publish %q, a seat it neither holds nor uses", p)
}
}
for _, s := range perms.Subscribe {
if strings.Contains(s, ".tool.") && !strings.HasPrefix(s, "mesh.mod.mesh-console.") {
t.Errorf("a module that invokes may subscribe %q, another module's tools", s)
}
}
}
// A module that declares no invokes calls nothing, which is every module but the console.
func TestAModuleThatInvokesNothingCallsNothing(t *testing.T) {
perms, err := PermissionsFor(Principal{Kind: KindModule, Node: "one", Module: "shop",
Emits: []string{"order.placed"}, PasswordHash: "x"})
if err != nil {
t.Fatal(err)
}
for _, p := range perms.Publish {
if strings.Contains(p, ".tool.") {
t.Errorf("a module with no invokes may publish %q", p)
}
}
}
// The malformed entry is refused for a module as it is for a person, and in the same words.
func TestAModulesToolGrantThatNamesNoToolIsRefused(t *testing.T) {
if _, err := PermissionsFor(Principal{Kind: KindModule, Node: "one", Module: "shop",
Invokes: []string{"telegram"}, PasswordHash: "x"}); err == nil {
t.Fatal("a grant naming a module but no tool was accepted")
}
}
// What a declaration says reaches the composed user, so a manifest's `invokes` is the grant.
func TestADeclaredInvokeReachesTheComposedUser(t *testing.T) {
users, err := Users(Records{
Nodes: []string{"desk"},
Assigned: map[string][]Declared{"desk": {{Module: "mesh-console", Invokes: []string{"*"}}}},
})
if err != nil {
t.Fatal(err)
}
perms, err := PermissionsFor(users[len(users)-1])
if err != nil {
t.Fatal(err)
}
has(t, perms.Publish, "mesh.mod.*.tool.>")
}
+61 -2
View File
@@ -26,6 +26,16 @@ import (
type JetStream struct {
conn *nats.Conn
js nats.JetStreamContext
// Note is how this says something it decided not to fail over. Nil is silent, which is only
// right for a caller that has no way to report; the controller sets it.
Note func(string, ...any)
}
// note reports without requiring a caller to have set one.
func (j *JetStream) note(format string, args ...any) {
if j.Note != nil {
j.Note(format, args...)
}
}
// Dial connects and returns the controller's JetStream handle.
@@ -169,6 +179,7 @@ func (j *JetStream) EnsureConsumer(c Consumer) error {
AckPolicy: nats.AckExplicitPolicy,
AckWait: time.Duration(c.AckWaitSeconds) * time.Second,
MaxDeliver: c.MaxDeliver,
MaxAckPending: c.MaxAckPending,
DeliverGroup: c.Queue,
DeliverSubject: "",
Description: c.Why,
@@ -184,12 +195,60 @@ func (j *JetStream) EnsureConsumer(c Consumer) error {
// without the other is refused by the server with a message that does not say which half is
// missing.
if c.Queue != "" || c.Push {
want.DeliverSubject = "_DELIVER." + c.Name
// **Per consumer, which means per stream as well as per name** (novox/hq 04-ISSUES/146).
// A push consumer delivers onto an ordinary subject, and everything subscribed to that
// subject gets a copy. The controller holds a consumer called `controller` on CONTROL and
// another called `controller` on EVENTS, and both were given `_DELIVER.controller` — so the
// one process, holding both subscriptions, acted on every message twice. It enrolled a
// joining machine twice from one request, minting a second credential that replaced the one
// the machine had just been given; the same doubling applied to every report and every
// event the controller follows.
//
// The stream is in the name because the pair is what identifies a consumer — the server
// scopes a durable's name to its stream, and this subject is the only place that scoping
// was dropped. Already within what the controller may subscribe (`_DELIVER.controller.>`),
// so no permission moves.
want.DeliverSubject = DeliverSubjectFor(c)
}
switch _, err := j.js.ConsumerInfo(c.Stream, c.Name); {
switch have, err := j.js.ConsumerInfo(c.Stream, c.Name); {
case err == nil:
// Where an existing consumer starts is its history, not something an assertion may move:
// the server refuses a changed deliver policy outright. Carried across, so asserting twice
// is the no-op a restart depends on.
want.DeliverPolicy = have.Config.DeliverPolicy
want.OptStartSeq = have.Config.OptStartSeq
want.OptStartTime = have.Config.OptStartTime
if _, err := j.js.UpdateConsumer(c.Stream, want); err != nil {
// **A consumer that works is not replaced to make its name tidier**
// (novox/hq 04-ISSUES/156).
//
// The server will not move a push consumer's delivery subject while a subscriber is
// bound to it, and answers `consumer name already in use` — a message about the name,
// for a conflict about the subject. A node is bound to its declaration consumer the
// whole time it is up; that IS a node listening. So when 04-ISSUES/146 put the stream
// into the subject, every node consumer in a running mesh became one this could not
// bring to match, and the control plane crash-looped on the assertion it makes before
// it serves. A fresh mesh showed nothing: nothing was bound.
//
// Kept rather than deleted and re-made. Re-making moves the subject, and a holder may
// not be allowed to subscribe to the new one yet — the wider grant travels in the bus's
// user list, which this same control plane composes and a machine applies minutes
// later. Re-making here would have silenced every machine in the mesh, which is worse
// than the collision it was fixing and harder to undo.
//
// Kept rather than fatal, which is what 146's change intended and did not do: the bare
// subject it replaces still delivers, and it collides only where one holder has two
// consumers of one name. That is the controller's own pair, and the controller is not
// bound to them while it asserts, so those do move. A node has one consumer and nothing
// to collide with.
if have.Config.DeliverSubject != want.DeliverSubject {
j.note("consumer %s on %s still delivers to %q and not %q: %v. It keeps working; "+
"the subject moves on an assertion made while nothing is bound to it",
c.Name, c.Stream, have.Config.DeliverSubject, want.DeliverSubject, err)
return nil
}
return fmt.Errorf("bringing consumer %s on %s to match: %w", c.Name, c.Stream, err)
}
return nil
+121 -27
View File
@@ -40,6 +40,10 @@ const (
// emits (novox/hq ADR 0118, design 29 §5).
type Seat struct {
Name string
// Scope is where the seat has one holder. A node-scoped seat's tool carries the node in its
// subject, because one subject reaching six machines' holders is not an address
// (novox/hq ADR 0132, design 33 §4). Empty reads as mesh.
Scope string
Accepts []string
Emits []string
Serves []string
@@ -70,12 +74,14 @@ type Principal struct {
// a namespace no such module owns. Every service started and the graph stayed empty.
Watches []Seat
// Invokes are the tools a person may call, as `<module>.<tool>`; a single `*` is every tool,
// for an administrator. Only meaningful for KindPerson.
// Invokes are the tools this principal may call, as `<module>.<tool>`; a single `*` is every
// tool. A person's whole authority (design 25 §7), and a module's only if its manifest says so
// (novox/hq ADR 0152) — the console's does, and nothing else's.
//
// **A list, not a role.** A person is not a module and holds no seat: nothing is addressed
// to them, nothing is delivered to them, and they have no durable consumer to acknowledge.
// What they have is permission to ask.
// What they have is permission to ask. A module that invokes gains exactly the same
// permission and nothing beside it.
Invokes []string
// PasswordHash is the bcrypt hash the mesh minted. The plaintext is sealed to the principal
@@ -181,6 +187,26 @@ func PermissionsFor(p Principal) (Permissions, error) {
for _, seat := range meshSeatsTheControllerUses {
pub = append(pub, "mesh.seat."+seat+".accept.>")
}
// **And what the mesh says it did** (novox/hq ADR 0134). The control plane states its own
// facts under the seat it holds, because a role's events belong to the role and keep their
// address while the holder is replaced. Named one by one rather than as a whole namespace:
// least authority, and a fact nothing states is authority nobody uses.
for _, event := range ControllerStates {
pub = append(pub, seatEventSubject(ControllerSeat, event))
}
// Every module's tools: **the control plane is the way in** (novox/hq ADR 0095). A person
// or an agent asks through it and every question passes one process where an audit
// belongs — so it, alone among principals, may call any tool by name. The first `ask` on
// the new bus was refused the publish (2026-09-28).
pub = append(pub, "mesh.mod.*.tool.>")
// **And the mesh's own verbs, as the seat it holds** (novox/hq ADR 0132, ADR 0154):
// `status`, `push`, `assign` are the mesh-controller seat's tools, served by its holder. The
// whole verb namespace of its own seat rather than a list: the list is the seat's protocol,
// which this package mirrors rather than reads, and a verb the seat does not declare is a
// subject nothing publishes.
sub = append(sub, "mesh.seat."+ControllerSeat+".tool.>")
// The two events it reacts to, and its ack subject on the stream they arrive from
// (streams.go). **Each named, not a pattern**: `mesh.mod.*.event.>` would make the
@@ -211,18 +237,11 @@ func PermissionsFor(p Principal) (Permissions, error) {
case KindPerson:
// Tools, and nothing else. Every subject a person may publish is a tool call; a person
// who could publish an event would be able to claim a module said something.
for _, t := range p.Invokes {
if t == "*" {
pub = append(pub, "mesh.mod.*.tool.>")
continue
}
module, tool, ok := strings.Cut(t, ".")
if !ok {
return Permissions{}, fmt.Errorf(
"%q does not name a tool: a person invokes <module>.<tool>, or * for every one", t)
}
pub = append(pub, "mesh.mod."+module+".tool."+tool)
invoked, err := invokedSubjects(p.Invokes)
if err != nil {
return Permissions{}, err
}
pub = append(pub, invoked...)
case KindEnrolment:
// A leaked token is useless for anything but enrolling: it cannot read a declaration, hear
@@ -256,7 +275,13 @@ func PermissionsFor(p Principal) (Permissions, error) {
"mesh.control." + p.Node + ".>",
"$JS.API.CONSUMER.INFO.NODES." + p.Node,
}
sub = []string{"mesh.node." + p.Node + ".declare", "_DELIVER." + p.Node}
// The deliver subject carries the stream as well as the consumer's name, so what a
// subscriber is permitted has to carry it too (novox/hq 04-ISSUES/146). The bare name
// stays: an existing consumer keeps delivering where it always did until the controller's
// next assertion moves it, and a permission that only allowed the new shape would refuse
// every node in the mesh for exactly as long as that took.
sub = []string{"mesh.node." + p.Node + ".declare",
"_DELIVER." + p.Node, "_DELIVER." + p.Node + ".>"}
case KindModule:
// 1. Its own namespace: it publishes its events there and serves its tools there. Nothing
@@ -266,9 +291,23 @@ func PermissionsFor(p Principal) (Permissions, error) {
for _, e := range p.Emits {
pub = append(pub, own+".event."+e)
}
for _, t := range p.Serves {
sub = append(sub, own+".tool."+t)
// Every tool under its own name, not a list: the tools a module serves are what its code
// answers, and a second copy of that list in the manifest would be a second source of
// truth for the mesh to keep in step (2026-09-28: every module that served a tool was
// refused the subscription, because none had written the list twice). Nothing is given
// away — no other principal may subscribe this namespace, and a caller's authority is
// still granted per tool, by name, on the publish side.
sub = append(sub, own+".tool.>")
// 1b. The tools it calls, if its manifest says it calls any (novox/hq ADR 0152). The same
// grant a person gets and derived the same way, so "what may this module ask" is
// answered by the one list that answers it for everybody. Publish only: an answer
// arrives on its own inbox, which every principal has below.
invoked, err := invokedSubjects(p.Invokes)
if err != nil {
return Permissions{}, err
}
pub = append(pub, invoked...)
// 2. What it consumes, by the emitter's own subject — an event is addressed to its
// emitter, because the emitter's identity is the meaning (ADR 0118).
@@ -288,18 +327,25 @@ func PermissionsFor(p Principal) (Permissions, error) {
}
}
// 2c. Its own consumer, which it **pulls**: the runtime asks for the next message and is
// answered on its own inbox, so what it needs is to ask about the consumer and to ask it
// for messages — its own consumer's name, and no other's. Pulled rather than pushed
// because that is the one shape a runtime's client binds without creating anything; the
// controller and the hosts are pushed to. Named here rather than through ConsumerFor,
// which asks for these permissions to build the consumer and would ask forever. A
// subject for a consumer that turns out not to exist grants nothing anybody can use.
pub = append(pub,
"$JS.API.CONSUMER.INFO."+consumerStream(p)+"."+consumerDurable(p),
"$JS.API.CONSUMER.MSG.NEXT."+consumerStream(p)+"."+consumerDurable(p))
// 3. Seats it holds: full participation.
// Its consumer's name, not ConsumerFor: that asks for these permissions to build the
// consumer, and would ask forever. A subject for a consumer that turns out not to exist
// grants nothing anybody can use.
sub = append(sub, "_DELIVER."+consumerDurable(p))
for _, s := range p.Holds {
// Taking work from the role's queue: the worker consumer it binds (asked about,
// delivered on, acknowledged), each on the seat's own stream. The first machine to
// take work over the new bus was refused the asking (2026-09-28).
worker := "SEAT_" + upperSnake(s.Name) + "_worker"
stream := seatStreamName(s.Name)
sub = append(sub, "_DELIVER."+worker)
sub = append(sub, "_DELIVER."+worker, "_DELIVER."+worker+".>")
pub = append(pub, "$JS.API.CONSUMER.INFO."+stream+"."+worker, "$JS.ACK."+stream+"."+worker+".>")
for _, a := range s.Accepts {
sub = append(sub, seatSubject(s, "accept", a))
@@ -308,7 +354,7 @@ func PermissionsFor(p Principal) (Permissions, error) {
pub = append(pub, seatSubject(s, "event", e))
}
for _, t := range s.Serves {
sub = append(sub, seatSubject(s, "tool", t))
sub = append(sub, seatToolSubject(s, t, p.Node))
}
}
@@ -320,7 +366,7 @@ func PermissionsFor(p Principal) (Permissions, error) {
pub = append(pub, seatSubject(s, "accept", a))
}
for _, t := range s.Serves {
pub = append(pub, seatSubject(s, "tool", t))
pub = append(pub, seatToolSubject(s, t, "*"))
}
}
}
@@ -348,9 +394,10 @@ func PermissionsFor(p Principal) (Permissions, error) {
return Permissions{
Publish: pub,
Subscribe: sub,
// Only something that serves is ever answering. A pure consumer is granted nothing here.
AllowResponses: p.Kind == KindModule && (len(p.Serves) > 0 || len(p.Holds) > 0) ||
p.Kind == KindController,
// A module answers what it was asked — a tool call reaches it on its own namespace, so the
// authority is bounded by having been asked — and so does the controller. A node and a
// person are never asked anything, and are granted nothing here.
AllowResponses: p.Kind == KindModule || p.Kind == KindController,
}, nil
}
@@ -369,6 +416,18 @@ func seatSubject(s Seat, kind, verb string) string {
return "mesh.seat." + s.Name + "." + kind + "." + verb
}
// seatToolSubject is where a role's tool is asked. Mesh-wide for a mesh-scoped seat; a node-scoped
// seat carries the node it is asked of, because a flat subject would reach every machine's holder
// and the queue group would silently pick a winner (novox/hq ADR 0132, design 33 §4). A holder
// subscribes its own node's; a user publishes any node's (`*`) and names the machine in the subject.
func seatToolSubject(s Seat, verb, node string) string {
base := seatSubject(s, "tool", verb)
if s.Scope == "node" && node != "" {
return base + "." + node
}
return base
}
// consumerStream and consumerDurable are the two halves of a consumer's identity, and they are
// two functions because conflating them was a real bug.
//
@@ -570,3 +629,38 @@ func quoted(values []string) string {
}
return strings.Join(out, ", ")
}
// invokedSubjects is the publish side of a grant to call tools: one subject per `<module>.<tool>`,
// or the whole tool namespace for `*`. A person's authority and a module's `invokes` are both this
// (novox/hq ADR 0152), so a malformed entry is refused in one place, before it could be widened into
// something that happens to parse.
func invokedSubjects(invokes []string) ([]string, error) {
var out []string
for _, t := range invokes {
if t == "*" {
// Every module's tools and every role's (novox/hq ADR 0132): a role's verb is a tool
// like any other, addressed to the seat instead of a module.
out = append(out, "mesh.mod.*.tool.>", "mesh.seat.*.tool.>")
continue
}
if rest, isSeat := strings.CutPrefix(t, "seat:"); isSeat {
// A role's tool, `seat:<seat>.<verb>`. Both address shapes, because the grant is
// written without knowing the seat's scope: a mesh seat's verb is flat and a node
// seat's carries the machine (design 33 §4).
seat, verb, ok := strings.Cut(rest, ".")
if !ok || seat == "" || verb == "" {
return nil, fmt.Errorf(
"%q does not name a role's tool: one invokes seat:<seat>.<verb>", t)
}
out = append(out, "mesh.seat."+seat+".tool."+verb, "mesh.seat."+seat+".tool."+verb+".*")
continue
}
module, tool, ok := strings.Cut(t, ".")
if !ok || module == "" || tool == "" {
return nil, fmt.Errorf(
"%q does not name a tool: one invokes <module>.<tool>, seat:<seat>.<verb>, or * for every one", t)
}
out = append(out, "mesh.mod."+module+".tool."+tool)
}
return out, nil
}
+57 -10
View File
@@ -1,6 +1,7 @@
package broker
import (
"slices"
"strings"
"testing"
)
@@ -70,6 +71,18 @@ func TestHoldingASeatIsTheMirrorOfUsingIt(t *testing.T) {
hasNot(t, perms.Publish, "mesh.seat.telegram-sender.accept.send")
}
// A build machine may say everything about a build as it happens (novox/hq ADR 0157): that it
// started, and every line under the build's own id — the seat's `log.*` becomes a publish over
// one token, so a reader follows one build by subject and the holder can name no other subject.
func TestTheBuildMachineMaySayWhatItDoesUnderTheBuildsId(t *testing.T) {
seat := Seat{Name: "mesh-build-machine", Accepts: []string{"build"}, Emits: []string{"started", "built", "log.*"}}
perms, _ := PermissionsFor(Principal{Kind: KindModule, Node: "anchor", Module: "builder",
Holds: []Seat{seat}, PasswordHash: "x"})
has(t, perms.Publish, "mesh.seat.mesh-build-machine.event.started")
has(t, perms.Publish, "mesh.seat.mesh-build-machine.event.log.*")
hasNot(t, perms.Publish, "mesh.seat.mesh-build-machine.event.>")
}
// Without an ack permission a durable consumer never really consumes: every message it receives is
// redelivered forever, refused by the permission list it already has (design 25 §4).
func TestAModuleMayAckItsOwnDeliveriesAndNoOthers(t *testing.T) {
@@ -89,17 +102,30 @@ func TestAnInboxIsScopedToItsOwner(t *testing.T) {
}
// A responder answers on the caller's inbox, which it has no permission for. allow_responses is
// what makes a scoped inbox workable at all — the authority is bounded by having been asked.
func TestOnlySomethingThatServesMayAnswer(t *testing.T) {
serving, _ := PermissionsFor(Principal{Kind: KindModule, Node: "one", Module: "billing",
Serves: []string{"status"}, PasswordHash: "x"})
if !serving.AllowResponses {
t.Fatal("a module serving a tool cannot answer the caller's inbox")
}
consumer, _ := PermissionsFor(Principal{Kind: KindModule, Node: "one", Module: "audit",
// what makes a scoped inbox workable at all — the authority is bounded by having been asked. A
// module is asked on its own namespace and may answer; a node and a person are never asked.
func TestOnlyWhatCanBeAskedMayAnswer(t *testing.T) {
module, _ := PermissionsFor(Principal{Kind: KindModule, Node: "one", Module: "audit",
Consumes: []string{"shop.order.placed"}, PasswordHash: "x"})
if consumer.AllowResponses {
t.Fatal("a pure consumer was granted the right to answer, which nothing asked it to do")
if !module.AllowResponses {
t.Fatal("a module cannot answer a tool call on its own namespace")
}
node, _ := PermissionsFor(Principal{Kind: KindNode, Node: "one", PasswordHash: "x"})
if node.AllowResponses {
t.Fatal("a node was granted the right to answer, and nothing asks a node anything")
}
}
// A module serves every tool under its own name, and no other module's.
func TestAModuleServesItsOwnNamespaceAndNoOthers(t *testing.T) {
p, _ := PermissionsFor(Principal{Kind: KindModule, Node: "one", Module: "gitea", PasswordHash: "x"})
if !slices.Contains(p.Subscribe, "mesh.mod.gitea.tool.>") {
t.Fatalf("a module may not serve its own tools: %v", p.Subscribe)
}
for _, s := range p.Subscribe {
if strings.HasPrefix(s, "mesh.mod.") && !strings.HasPrefix(s, "mesh.mod.gitea.") {
t.Fatalf("a module may subscribe another's namespace: %s", s)
}
}
}
@@ -324,3 +350,24 @@ func admits(pattern, subject []string) bool {
}
return len(pattern) == len(subject)
}
// A module pulls its own consumer — asks about it, asks it for messages — and no other module's.
func TestAModulePullsItsOwnConsumerAndNoOthers(t *testing.T) {
p, _ := PermissionsFor(Principal{Kind: KindModule, Node: "one", Module: "audit",
Consumes: []string{"shop.order.placed"}, PasswordHash: "x"})
for _, want := range []string{"$JS.API.CONSUMER.INFO.EVENTS.one_audit", "$JS.API.CONSUMER.MSG.NEXT.EVENTS.one_audit"} {
if !slices.Contains(p.Publish, want) {
t.Errorf("a module cannot bind its own consumer: %v lacks %s", p.Publish, want)
}
}
for _, s := range p.Publish {
if strings.Contains(s, "CONSUMER.") && !strings.HasSuffix(s, ".one_audit") {
t.Errorf("a module may reach another consumer: %s", s)
}
}
for _, s := range p.Subscribe {
if strings.HasPrefix(s, "_DELIVER.") {
t.Errorf("a module is granted a push delivery it never binds: %s", s)
}
}
}
+57
View File
@@ -0,0 +1,57 @@
package broker
import "testing"
// A node-scoped seat's tool carries the node (novox/hq ADR 0132, design 33 §4): two nodes holding one
// node-scoped seat derive two addresses, and a user of the seat may publish any node's.
func TestTwoNodesHoldingOneNodeSeatDeriveTwoToolAddresses(t *testing.T) {
seat := Seat{Name: "node-dns-resolver", Scope: "node", Serves: []string{"lookup"}}
one, _ := PermissionsFor(Principal{Kind: KindModule, Node: "one", Module: "dnsmasq", Holds: []Seat{seat}, PasswordHash: "x"})
two, _ := PermissionsFor(Principal{Kind: KindModule, Node: "two", Module: "dnsmasq", Holds: []Seat{seat}, PasswordHash: "x"})
has(t, one.Subscribe, "mesh.seat.node-dns-resolver.tool.lookup.one")
has(t, two.Subscribe, "mesh.seat.node-dns-resolver.tool.lookup.two")
hasNot(t, one.Subscribe, "mesh.seat.node-dns-resolver.tool.lookup")
hasNot(t, one.Subscribe, "mesh.seat.node-dns-resolver.tool.lookup.two")
user, _ := PermissionsFor(Principal{Kind: KindModule, Node: "three", Module: "asker", Uses: []Seat{seat}, PasswordHash: "x"})
has(t, user.Publish, "mesh.seat.node-dns-resolver.tool.lookup.*")
}
// A mesh-scoped seat's tool stays flat: nothing about it changes.
func TestAMeshSeatsToolIsAddressedToTheSeatAlone(t *testing.T) {
seat := Seat{Name: "git", Scope: "mesh", Serves: []string{"list_repos"}}
holder, _ := PermissionsFor(Principal{Kind: KindModule, Node: "one", Module: "gitea", Holds: []Seat{seat}, PasswordHash: "x"})
has(t, holder.Subscribe, "mesh.seat.git.tool.list_repos")
user, _ := PermissionsFor(Principal{Kind: KindModule, Node: "two", Module: "asker", Uses: []Seat{seat}, PasswordHash: "x"})
has(t, user.Publish, "mesh.seat.git.tool.list_repos")
}
// The controller serves its own seat's verbs and may answer them (novox/hq ADR 0154).
func TestTheControllerServesItsSeatsToolsAndMayAnswer(t *testing.T) {
perms, err := PermissionsFor(Principal{Kind: KindController, PasswordHash: "x"})
if err != nil {
t.Fatal(err)
}
has(t, perms.Subscribe, "mesh.seat.mesh-controller.tool.>")
if !perms.AllowResponses {
t.Fatal("the controller serves tools and may not answer one")
}
}
// A grant to every tool reaches a role's tools too, and a role's tool is granted by name.
func TestAGrantReachesARolesTools(t *testing.T) {
all, _ := PermissionsFor(Principal{Kind: KindModule, Node: "desk", Module: "mesh-console", Invokes: []string{"*"}, PasswordHash: "x"})
has(t, all.Publish, "mesh.seat.*.tool.>")
one, err := PermissionsFor(Principal{Kind: KindPerson, Module: "jo", Invokes: []string{"seat:mesh-controller.status"}, PasswordHash: "x"})
if err != nil {
t.Fatal(err)
}
has(t, one.Publish, "mesh.seat.mesh-controller.tool.status")
hasNot(t, one.Publish, "mesh.seat.mesh-controller.tool.push")
hasNot(t, one.Publish, "mesh.mod.*.tool.>")
if _, err := PermissionsFor(Principal{Kind: KindPerson, Module: "jo", Invokes: []string{"seat:mesh-controller"}, PasswordHash: "x"}); err == nil {
t.Fatal("a role grant naming no verb was accepted")
}
}
+44
View File
@@ -0,0 +1,44 @@
package broker_test
import (
"slices"
"testing"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/link"
)
// The facts the control plane states are named twice — in the grant that permits them and in the code
// that states them — because `link` imports `broker` and the dependency cannot go the other way. So a
// test keeps them agreeing: a subject the grant omits is refused at the moment the mesh has something
// to say, and one the grant adds that nothing states is authority nobody uses.
//
// An external test package, because it may import both while neither imports the other.
func TestTheFactsTheGrantPermitsAreTheFactsTheMeshStates(t *testing.T) {
if broker.ControllerSeat != link.MeshControllerSeat {
t.Fatalf("the grant is written for the %q seat and the mesh states its facts under %q",
broker.ControllerSeat, link.MeshControllerSeat)
}
for _, event := range []string{link.KeyApplied, link.KeyRefused, link.KeyBuiltBefore} {
if !slices.Contains(broker.ControllerStates, event) {
t.Errorf("the mesh states %q and its account may not publish it", event)
}
}
if len(broker.ControllerStates) != 3 {
t.Errorf("the grant permits %v, which is more than the mesh states", broker.ControllerStates)
}
// **And the seat says it.** A seat carries the protocol of its role (novox/hq ADR 0129), so the
// facts the control plane states are the seat's `emits` — which is what lets anything else declare
// that it consumes them, and what the subject-agreement check reads to know they have an owner.
var declared []string
for _, seat := range catalogue.SeatsWithAProtocol() {
if seat.Name == broker.ControllerSeat {
declared = seat.Emits
}
}
if !slices.Equal(declared, broker.ControllerStates) {
t.Errorf("the %s seat emits %v and the grant permits %v", broker.ControllerSeat,
declared, broker.ControllerStates)
}
}
+40 -3
View File
@@ -79,7 +79,7 @@ func MeshStreams() []Stream {
"n-1 by construction (issue 107)",
},
{
Name: "EVENTS",
Name: EventsStream,
// A seat's own events ride here too: they are 1:many like any event, and the
// `event` token keeps them clear of both the seat's work queue (`accept`) and its
// tools (`tool`), which must not be persisted.
@@ -94,6 +94,24 @@ func MeshStreams() []Stream {
}
}
// DeliverSubjectFor is where a push consumer's messages land.
//
// **Per consumer, which means per stream as well as per name** (novox/hq 04-ISSUES/146). A push
// consumer delivers onto an ordinary subject, and everything subscribed to that subject gets a
// copy. The controller holds a consumer called `controller` on CONTROL and another called
// `controller` on EVENTS; while both were given `_DELIVER.controller`, the one process holding
// both subscriptions acted on every message twice — a joining machine was enrolled twice from one
// request, and the second enrolment minted a credential that replaced the one the machine had just
// been handed. Every report and every followed event doubled the same way, silently: nothing is
// redelivered, no count is wrong, the work simply happens twice.
//
// The stream belongs in it because the pair is what identifies a consumer — the server scopes a
// durable's name to its stream, and this subject was the one place that scoping was dropped. It
// stays inside what a controller may already subscribe (`_DELIVER.controller.>`).
func DeliverSubjectFor(c Consumer) string {
return "_DELIVER." + c.Name + "." + c.Stream
}
// An Asserter is the part of a JetStream connection stream assertion needs. Narrow on purpose: it
// keeps this testable without a server, and keeps the client library out of everything that only
// wants to know what the streams are.
@@ -160,6 +178,17 @@ func Overlaps() []string {
// ack subject is derived from (nats.go: `$JS.ACK.<stream>.controller.>`).
const ControllerName = "controller"
// ControllerSeat is the role the control plane holds, and ControllerStates are the facts it states
// under it (novox/hq ADR 0134).
//
// **Written here as well as in `link`, and a test keeps them agreeing.** `link` imports `broker`, so
// `broker` cannot import `link`; a grant naming a subject the controller never publishes is authority
// nobody uses, and a controller publishing one the grant omits is refused at the moment it has
// something to say.
const ControllerSeat = "mesh-controller"
var ControllerStates = []string{"applied", "refused", "built-before"}
// ControllerFollows are the events the controller reacts to: the catalogue saying a module's
// current version moved, and a catalogue that has just started saying it may have missed builds.
//
@@ -191,6 +220,9 @@ func seatEventSubject(seat, verb string) string {
return "mesh.seat." + seat + ".event." + verb
}
// EventsStream holds every module's and every role's events, a build's log among them.
const EventsStream = "EVENTS"
// MeshConsumers is what the controller consumes, in the order a person reads it.
//
// **Unlimited redelivery on CONTROL, deliberately.** The store window's bound is the controller's,
@@ -215,8 +247,13 @@ func MeshConsumers() []Consumer {
Push: true,
AckWaitSeconds: 30,
MaxDeliver: 5,
Why: "the two events the mesh's own controller reacts to; after max-deliver it " +
"dead-letters, because an announcement it cannot act on will not become actionable",
// One at a time (novox/hq issue 175): acting on a merge builds for minutes, and an
// announcement handed over behind it must wait on the server, not time out on the
// client and come back to be acted on again.
MaxAckPending: 1,
Why: "the two events the mesh's own controller reacts to, one at a time; after " +
"max-deliver it dead-letters, because an announcement it cannot act on will not " +
"become actionable",
},
}
}
+38
View File
@@ -233,3 +233,41 @@ func containsStep(steps []string, want string) bool {
}
return false
}
// **Two consumers may share a name, and must not share a delivery subject** (novox/hq
// 04-ISSUES/146).
//
// A push consumer delivers onto an ordinary subject and everything subscribed to it gets a copy.
// The controller holds a consumer called `controller` on CONTROL and another called `controller` on
// EVENTS; while both were given `_DELIVER.controller`, the one process holding both subscriptions
// acted on every message twice — a joining machine enrolled twice from one request, with the second
// enrolment minting a credential that replaced the one the machine had just been handed.
//
// Checked here rather than against a server because it is a property of what the mesh asks for, and
// because the failure it produces is silent: every count is right, nothing is redelivered, and the
// work simply happens twice.
func TestNoTwoConsumersDeliverOntoTheSameSubject(t *testing.T) {
seen := map[string]string{}
for _, c := range MeshConsumers() {
if !c.Push && c.Queue == "" {
continue
}
subject := DeliverSubjectFor(c)
if other, taken := seen[subject]; taken {
t.Errorf("%s on %s and %s deliver onto %s, so whoever holds both acts on every "+
"message twice", c.Name, c.Stream, other, subject)
}
seen[subject] = c.Name + " on " + c.Stream
}
}
// The controller's events consumer is handed one announcement at a time (novox/hq issue 175): a
// merge's handler builds for minutes, and what is queued behind it must wait on the server rather
// than time out on the client and be acted on twice.
func TestTheControllerTakesOneAnnouncementAtATime(t *testing.T) {
for _, c := range MeshConsumers() {
if c.Stream == "EVENTS" && c.Name == ControllerName && c.MaxAckPending != 1 {
t.Fatalf("the events consumer may have %d outstanding; one announcement at a time", c.MaxAckPending)
}
}
}
+11 -9
View File
@@ -24,8 +24,8 @@ accounts {
jetstream: enabled
users = [
{ user: "controller", password: "$2a$11$cccccccccccccccccccccc", permissions: {
publish: { allow: ["$JS.ACK.CONTROL.controller.>", "$JS.ACK.EVENTS.controller.>", "$JS.API.>", "_INBOX.enrol.>", "mesh.control.>", "mesh.node.>", "mesh.seat.mesh-build-machine.accept.>"] }
subscribe: { allow: ["$JS.API.>", "_DELIVER.controller", "_DELIVER.controller.>", "_INBOX.controller.>", "mesh.control.>", "mesh.mod.gitea.event.pull.merged", "mesh.mod.mesh-catalog.event.catching-up", "mesh.mod.mesh-catalog.event.upgraded", "mesh.seat.mesh-build-machine.event.built"] }
publish: { allow: ["$JS.ACK.CONTROL.controller.>", "$JS.ACK.EVENTS.controller.>", "$JS.API.>", "_INBOX.enrol.>", "mesh.control.>", "mesh.mod.*.tool.>", "mesh.node.>", "mesh.seat.mesh-build-machine.accept.>", "mesh.seat.mesh-controller.event.applied", "mesh.seat.mesh-controller.event.built-before", "mesh.seat.mesh-controller.event.refused"] }
subscribe: { allow: ["$JS.API.>", "_DELIVER.controller", "_DELIVER.controller.>", "_INBOX.controller.>", "mesh.control.>", "mesh.mod.gitea.event.pull.merged", "mesh.mod.mesh-catalog.event.catching-up", "mesh.mod.mesh-catalog.event.upgraded", "mesh.seat.mesh-build-machine.event.built", "mesh.seat.mesh-controller.tool.>"] }
allow_responses: { max: 1, ttl: "1m" }
} }
{ user: "enrol.one", password: "$2a$11$eeeeeeeeeeeeeeeeeeeeee", permissions: {
@@ -34,20 +34,22 @@ accounts {
} }
{ user: "node.one", password: "$2a$11$nnnnnnnnnnnnnnnnnnnnnn", permissions: {
publish: { allow: ["$JS.ACK.NODES.one.>", "$JS.API.CONSUMER.INFO.NODES.one", "mesh.control.one.>"] }
subscribe: { allow: ["_DELIVER.one", "_INBOX.node.one.>", "mesh.node.one.declare"] }
subscribe: { allow: ["_DELIVER.one", "_DELIVER.one.>", "_INBOX.node.one.>", "mesh.node.one.declare"] }
} }
{ user: "one.telegram", password: "$2a$11$tttttttttttttttttttttt", permissions: {
publish: { allow: ["$JS.ACK.EVENTS.one_telegram.>", "$JS.ACK.SEAT_TELEGRAM_SENDER.SEAT_TELEGRAM_SENDER_worker.>", "$JS.API.CONSUMER.INFO.SEAT_TELEGRAM_SENDER.SEAT_TELEGRAM_SENDER_worker", "mesh.seat.telegram-sender.event.delivered", "mesh.seat.telegram-sender.event.failed"] }
subscribe: { allow: ["_DELIVER.SEAT_TELEGRAM_SENDER_worker", "_DELIVER.one_telegram", "_INBOX.one.telegram.>", "mesh.mod.telegram.tool.status", "mesh.seat.telegram-sender.accept.send"] }
publish: { allow: ["$JS.ACK.EVENTS.one_telegram.>", "$JS.ACK.SEAT_TELEGRAM_SENDER.SEAT_TELEGRAM_SENDER_worker.>", "$JS.API.CONSUMER.INFO.EVENTS.one_telegram", "$JS.API.CONSUMER.INFO.SEAT_TELEGRAM_SENDER.SEAT_TELEGRAM_SENDER_worker", "$JS.API.CONSUMER.MSG.NEXT.EVENTS.one_telegram", "mesh.seat.telegram-sender.event.delivered", "mesh.seat.telegram-sender.event.failed"] }
subscribe: { allow: ["_DELIVER.SEAT_TELEGRAM_SENDER_worker", "_DELIVER.SEAT_TELEGRAM_SENDER_worker.>", "_INBOX.one.telegram.>", "mesh.mod.telegram.tool.>", "mesh.seat.telegram-sender.accept.send"] }
allow_responses: { max: 1, ttl: "1m" }
} }
{ user: "two.audit", password: "$2a$11$aaaaaaaaaaaaaaaaaaaaaa", permissions: {
publish: { allow: ["$JS.ACK.EVENTS.two_audit.>"] }
subscribe: { allow: ["_DELIVER.two_audit", "_INBOX.two.audit.>", "mesh.mod.shop.event.order.placed"] }
publish: { allow: ["$JS.ACK.EVENTS.two_audit.>", "$JS.API.CONSUMER.INFO.EVENTS.two_audit", "$JS.API.CONSUMER.MSG.NEXT.EVENTS.two_audit"] }
subscribe: { allow: ["_INBOX.two.audit.>", "mesh.mod.audit.tool.>", "mesh.mod.shop.event.order.placed"] }
allow_responses: { max: 1, ttl: "1m" }
} }
{ user: "two.shop", password: "$2a$11$ssssssssssssssssssssss", permissions: {
publish: { allow: ["$JS.ACK.EVENTS.two_shop.>", "mesh.mod.shop.event.order.placed", "mesh.seat.telegram-sender.accept.send"] }
subscribe: { allow: ["_DELIVER.two_shop", "_INBOX.two.shop.>"] }
publish: { allow: ["$JS.ACK.EVENTS.two_shop.>", "$JS.API.CONSUMER.INFO.EVENTS.two_shop", "$JS.API.CONSUMER.MSG.NEXT.EVENTS.two_shop", "mesh.mod.shop.event.order.placed", "mesh.seat.telegram-sender.accept.send"] }
subscribe: { allow: ["_INBOX.two.shop.>", "mesh.mod.shop.tool.>"] }
allow_responses: { max: 1, ttl: "1m" }
} }
]
}
+3 -1
View File
@@ -31,6 +31,8 @@ type Declared struct {
Uses []Seat
// Watches are the seats whose events it consumes.
Watches []Seat
// Invokes are the tools it calls, `<module>.<tool>` or `*` (novox/hq ADR 0152).
Invokes []string
}
// Records is what composing a user list needs to know about the mesh, and nothing more.
@@ -61,7 +63,7 @@ func Users(r Records) ([]Principal, error) {
out = append(out, Principal{
Kind: KindModule, Node: node, Module: d.Module,
Emits: d.Emits, Consumes: d.Consumes, Serves: d.Serves,
Holds: d.Holds, Uses: d.Uses, Watches: d.Watches,
Holds: d.Holds, Uses: d.Uses, Watches: d.Watches, Invokes: d.Invokes,
})
}
}
+44
View File
@@ -0,0 +1,44 @@
package builder
import (
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
)
// The name a machine runs a binary by is not always the name of the package that built it. The host's
// command is cmd/mesh-host and every machine runs it as nox-mesh-host — the path it is installed at,
// the name in its unit, and the name its launcher looks for inside a delivered version.
//
// A bundle carrying the package's name was delivered to a machine correctly, reported "created … 1
// file(s)", and was invisible to the launcher (novox/hq 04-ISSUES/142). Found by reading the delivered
// directory rather than by trusting the line that said it worked.
func TestACompiledArtifactNamesTheBinaryAMachineWillRun(t *testing.T) {
got := binaryName(catalogue.Artifact{
Name: "host-arch", From: "cmd/mesh-host", Binary: "nox-mesh-host",
})
if got != "nox-mesh-host" {
t.Fatalf("the binary is named %q, and the launcher looks for nox-mesh-host", got)
}
}
func TestSayingNothingKeepsWhatTheCompilerWouldHaveChosen(t *testing.T) {
// go build names its output after the package, so an artifact that says nothing gets the same
// thing it got before this existed.
if got := binaryName(catalogue.Artifact{Name: "host-arch", From: "cmd/mesh-host"}); got != "mesh-host" {
t.Fatalf("an artifact naming no binary produced %q", got)
}
if got := binaryName(catalogue.Artifact{Name: "host-arch", From: "./cmd/agent/"}); got != "agent" {
t.Fatalf("a from with slashes produced %q", got)
}
}
func TestABundleBuiltFromTheModuleRootFallsBackToItsArtifactName(t *testing.T) {
// A single-command repository names no package, and `go build -o <dir>` would then write a file
// named after the module directory — which is not something the manifest states. The artifact's
// own name is what the manifest does state.
if got := binaryName(catalogue.Artifact{Name: "tool"}); got != "tool" {
t.Fatalf("a bundle built from the root produced %q", got)
}
}
+155 -18
View File
@@ -61,6 +61,12 @@ type Result struct {
Commit string
// Built is each artifact, for reporting.
Built []catalogue.Built
// Read is every repository this build read source from besides the module's own — the second
// repository an artifact's recipe names (ArtifactContext). Reported because the manifest the
// mesh keeps carries no build section, so nothing else could say that a merge there is a
// change to this module (novox/hq 04-ISSUES/131).
Read []catalogue.ArtifactContext
}
// GitCredential is the forge credential a clone may present when the server asks for one.
@@ -84,7 +90,13 @@ type GitCredential struct {
// records — reachable, unreferenced, and indistinguishable from something in use.
func Build(ctx context.Context, run Runner, publish Publisher,
repository, path, ref, workspace string, held map[string]string, npmrc Npmrc,
forge GitCredential, log Log) (Result, error) {
forge GitCredential, log Log, seats ...map[string]string) (Result, error) {
// The clone base of each seat a context may name (novox/hq ADR 0155); variadic so the callers
// that hand none — tests of everything but contexts — read as they did.
var seatBases map[string]string
if len(seats) > 0 {
seatBases = seats[0]
}
say := logging(log)
say("clone", "%s%s at %s", repository, describePath(path), refOrHead(ref))
@@ -203,7 +215,7 @@ func Build(ctx context.Context, run Runner, publish Publisher,
sort.Slice(artifacts, func(i, j int) bool { return artifacts[i].Name < artifacts[j].Name })
for _, a := range artifacts {
say("artifact", "%s (%s%s) — starting", a.Name, a.Kind, langSuffix(a))
made, err := one(ctx, run, publish, manifest.Module, within, workspace, commit, credentials, a, args, held, npmrcPath, say)
made, err := one(ctx, run, publish, manifest.Module, within, workspace, commit, credentials, a, args, held, npmrcPath, seatBases, say)
if err != nil {
say("artifact", "%s FAILED: %v", a.Name, err)
return Result{}, err
@@ -220,7 +232,7 @@ func Build(ctx context.Context, run Runner, publish Publisher,
}
say("done", "%s at %s — %d artifact(s) pinned", manifest.Module, short(commit), len(built))
return Result{Manifest: resolved, Commit: commit, Built: built,
Against: against(within, manifest, stoodOn)}, nil
Against: against(within, manifest, stoodOn), Read: readBy(manifest)}, nil
}
// Log is where a build says what it is doing, step by step. Nil is silent — the tests pass none,
@@ -240,14 +252,18 @@ func logging(log Log) func(step, format string, args ...any) {
// module's own repository — a fresh tree, the same way the module's own is, keyed by artifact
// name so two artifacts of one module naming different contexts do not collide.
func contextFrom(ctx context.Context, run Runner, workspace, artifact, credentials string,
from catalogue.ArtifactContext, say func(step, format string, args ...any)) (string, error) {
say("context", "cloning %s at %s for %s", from.Repository, refOrHead(from.Ref), artifact)
from catalogue.ArtifactContext, seats map[string]string, say func(step, format string, args ...any)) (string, error) {
url, err := contextURL(from, seats)
if err != nil {
return "", err
}
say("context", "cloning %s at %s for %s", url, refOrHead(from.Ref), artifact)
dir := filepath.Join(workspace, "context-"+artifact)
if err := os.RemoveAll(dir); err != nil {
return "", err
}
if _, err := run(ctx, workspace, "git", cloneWith(credentials, "clone", "--quiet", from.Repository, dir)...); err != nil {
return "", fmt.Errorf("cannot clone %s: %w", from.Repository, err)
if _, err := run(ctx, workspace, "git", cloneWith(credentials, "clone", "--quiet", url, dir)...); err != nil {
return "", fmt.Errorf("cannot clone %s: %w", url, err)
}
if from.Ref != "" {
if _, err := run(ctx, dir, "git", "checkout", "--quiet", from.Ref); err != nil {
@@ -258,6 +274,23 @@ func contextFrom(ctx context.Context, run Runner, workspace, artifact, credentia
return dir, nil
}
// contextURL is what a context is cloned from: its URL, or — for a context on a seat — the seat's
// clone base the mesh sent with the request joined to the repository's path (novox/hq ADR 0155).
// Refused, never guessed, when the mesh sent no base for that seat: a builder that guessed a forge
// would be the literal this removes, one layer down.
func contextURL(from catalogue.ArtifactContext, seats map[string]string) (string, error) {
if from.Seat == "" {
return from.Repository, nil
}
base, told := seats[from.Seat]
if !told || base == "" {
return "", fmt.Errorf("the context is %s on the %s seat, and this build was told no clone "+
"base for that seat — nothing holds it in this mesh, or the control plane predates the word",
from.Repository, from.Seat)
}
return strings.TrimRight(base, "/") + "/" + strings.TrimSuffix(strings.Trim(from.Repository, "/"), ".git") + ".git", nil
}
// cloneWith is a git invocation that may offer a stored credential.
//
// The first `-c credential.helper=` clears every helper the environment might carry, so exactly
@@ -410,7 +443,8 @@ func wantsPackages(manifest catalogue.Manifest, within string) bool {
func one(ctx context.Context, run Runner, publish Publisher,
module, tree, workspace, commit, credentials string, a catalogue.Artifact, args []string,
held map[string]string, npmrc string, say func(step, format string, args ...any)) (catalogue.Built, error) {
held map[string]string, npmrc string, seats map[string]string,
say func(step, format string, args ...any)) (catalogue.Built, error) {
switch a.Kind {
case catalogue.ArtifactUpstream:
@@ -487,7 +521,7 @@ func one(ctx context.Context, run Runner, publish Publisher,
recipePath := a.From
buildDir := tree
if a.Context != nil {
cloned, err := contextFrom(ctx, run, workspace, a.Name, credentials, *a.Context, say)
cloned, err := contextFrom(ctx, run, workspace, a.Name, credentials, *a.Context, seats, say)
if err != nil {
return catalogue.Built{}, fmt.Errorf("%s: %s's context: %w", module, a.Name, err)
}
@@ -685,6 +719,21 @@ func short(commit string) string {
return commit
}
// Said is where the lines Command speaks go, beside the build's own Log: what runs, how long it
// took, and that it failed. Nil prints them to stderr, as a build machine with nobody listening
// should. The machine sets it per build so every line reaches the bus too (novox/hq ADR 0157) —
// the step is "run", and the message is the line as it has always been printed.
var Said Log
func tell(step, format string, args ...any) {
message := fmt.Sprintf(format, args...)
if Said == nil {
fmt.Fprintf(os.Stderr, " %s\n", message)
return
}
Said(step, message)
}
// Command is a Runner that actually runs things.
func Command(ctx context.Context, dir, name string, args ...string) (string, error) {
// **Every command is echoed before it runs**, with where. On a build that hangs, the last line
@@ -692,16 +741,23 @@ func Command(ctx context.Context, dir, name string, args ...string) (string, err
// nothing" and "git clone is waiting on a network that will not answer". Silent on success is
// what made an empty workspace unreadable.
started := timeNow()
fmt.Fprintf(os.Stderr, " $ (%s) %s %s\n", short(filepath.Base(dir)), name, strings.Join(args, " "))
tell("run", "$ (%s) %s %s", short(filepath.Base(dir)), name, strings.Join(args, " "))
cmd := exec.CommandContext(ctx, name, args...)
cmd.Dir = dir
out, err := cmd.CombinedOutput()
if err != nil {
fmt.Fprintf(os.Stderr, " ! %s %s failed after %s\n", name, args[0], since(started))
tell("run", "! %s %s failed after %s", name, args[0], since(started))
// The command's own output is part of what a reader needs — the compiler's error, the
// clone's refusal — and a line per output line keeps it readable on the bus.
for _, line := range strings.Split(strings.TrimSpace(string(out)), "\n") {
if line != "" {
tell("output", "%s", line)
}
}
return string(out), fmt.Errorf("%s %s: %w\n%s",
name, strings.Join(args, " "), err, strings.TrimSpace(string(out)))
}
fmt.Fprintf(os.Stderr, " ✓ %s %s (%s)\n", name, firstArg(args), since(started))
tell("run", "✓ %s %s (%s)", name, firstArg(args), since(started))
return string(out), nil
}
@@ -855,6 +911,15 @@ func compile(ctx context.Context, run Runner, tree string, chain Toolchain,
// each other and then be packed together, so each bundle compiles and packs alone.
out := Out(a.Name)
// **The output directory exists before the compiler is told about it.** `tsc --outDir` makes
// one; `go build -o` writes a file into a directory and does not create it, failing with a
// message about a path rather than about a build. Made here for every toolchain, because which
// compilers happen to be forgiving is not a thing a reader should have to know
// (novox/hq 04-ISSUES/142).
if err := os.MkdirAll(filepath.Join(tree, out), 0o755); err != nil {
return "", fmt.Errorf("making the output directory for %s: %w", a.Name, err)
}
invocation := []string{
"run", "--rm",
"--volume", tree + ":" + within,
@@ -862,13 +927,43 @@ func compile(ctx context.Context, run Runner, tree string, chain Toolchain,
base,
}
invocation = append(invocation, chain.Compile...)
// **One `-ldflags`, composed here.** A repeated flag is not a merged one: the Go command takes
// the last and drops the first, so passing the toolchain's flags and then the system stamp as a
// second `-ldflags` produced a binary that knew its system and had lost `-s -w` — half again the
// size, with its debug info (novox/hq 04-ISSUES/161).
//
// What it was built for is the one thing taken from the artifact, and ADR 0142 says why: the
// target is a property of the artifact rather than of the recipe. A host with no system refuses
// every declaration before it applies anything.
linker := append([]string(nil), chain.LinkerFlags...)
if chain.SystemStamp != "" && strings.TrimSpace(a.System) != "" {
linker = append(linker, "-X", chain.SystemStamp+"="+strings.TrimSpace(a.System))
}
if len(linker) > 0 {
invocation = append(invocation, "-ldflags", strings.Join(linker, " "))
}
if chain.OutputFlag != "" {
invocation = append(invocation, chain.OutputFlag, out)
// A compiler pointed at a package is told the file to write, not the directory: the name a
// machine runs it by is not always the name of the package that built it. The host's command
// is `cmd/mesh-host` and every machine runs it as `nox-mesh-host` — so a bundle carrying the
// package's name lands correctly, reports success, and is invisible to whatever looks for it
// (novox/hq 04-ISSUES/142).
target := out
if chain.Unit == UnitPackage {
target = filepath.Join(out, binaryName(a))
}
invocation = append(invocation, chain.OutputFlag, target)
}
// What to compile. Named by the module rather than discovered, so adding a file does not
// silently change what a build produces.
if len(a.Entrypoints) > 0 {
invocation = append(invocation, sourcesFor(a.Entrypoints, out)...)
switch {
case chain.Unit == UnitPackage:
// One directory, compiled whole: the thing the artifact is built `from`. Relative, because
// the compiler runs with the module's own root as its working directory and a package path
// that looked absolute would name one inside the toolchain image.
invocation = append(invocation, "./"+strings.Trim(a.From, "./"))
case len(a.Entrypoints) > 0:
invocation = append(invocation, sourcesFor(a.Entrypoints, out, chain.SourceExt)...)
}
if _, err := run(ctx, tree, "docker", invocation...); err != nil {
return "", err
@@ -881,14 +976,16 @@ func compile(ctx context.Context, run Runner, tree string, chain Toolchain,
// A module names what a tool host should LOAD — compiled paths under the bundle's root — because
// that is the thing anything else needs to know. What to compile is the same list with the
// language's own extension, which is the toolchain's business rather than the module's.
func sourcesFor(entrypoints []string, out string) []string {
func sourcesFor(entrypoints []string, out, ext string) []string {
sources := make([]string, 0, len(entrypoints))
for _, e := range entrypoints {
// An entrypoint is named as it will be FOUND — a path inside the unpacked bundle — so the
// source is the same path with the output directory taken off the front and the language's
// own extension on the end.
// own extension on the end. **The extension is the toolchain's**, where it used to be the
// literal `.ts`: one language's file extension written into the code that serves every
// language is a wall the next one hits (novox/hq 04-ISSUES/142).
at := strings.TrimPrefix(strings.TrimPrefix(e, out), "/")
sources = append(sources, strings.TrimSuffix(at, filepath.Ext(at))+".ts")
sources = append(sources, strings.TrimSuffix(at, filepath.Ext(at))+ext)
}
return sources
}
@@ -1016,3 +1113,43 @@ func instructions(recipe string) []string {
flush()
return out
}
// readBy is every repository other than the module's own that this build's recipes read source from,
// each once and in a fixed order, so two builds of one commit report the same thing the same way.
func readBy(manifest catalogue.Manifest) []catalogue.ArtifactContext {
if manifest.Build == nil {
return nil
}
seen := map[string]bool{}
var out []catalogue.ArtifactContext
for _, a := range manifest.Build.Artifacts {
if a.Context == nil || a.Context.Repository == "" {
continue
}
key := a.Context.Repository + "#" + a.Context.Ref
if seen[key] {
continue
}
seen[key] = true
out = append(out, *a.Context)
}
sort.Slice(out, func(i, j int) bool {
if out[i].Repository != out[j].Repository {
return out[i].Repository < out[j].Repository
}
return out[i].Ref < out[j].Ref
})
return out
}
// binaryName is what a compiled bundle's executable is called: what the artifact says, or the name of
// the package it is built from, which is what a compiler would have chosen anyway.
func binaryName(a catalogue.Artifact) string {
if name := strings.TrimSpace(a.Binary); name != "" {
return name
}
if from := strings.Trim(a.From, "./"); from != "" {
return filepath.Base(from)
}
return a.Name
}
+26
View File
@@ -0,0 +1,26 @@
package builder
import (
"strings"
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
)
// A context on a seat is cloned from the base the mesh sent, joined to the repository's path; a
// context by URL is itself; a seat the mesh sent no base for is refused by name (novox/hq ADR 0155).
func TestAContextOnASeatIsClonedFromTheBaseTheMeshSent(t *testing.T) {
seats := map[string]string{"git": "http://forge.example.tld:3000"}
got, err := contextURL(catalogue.ArtifactContext{Seat: "git", Repository: "org/controller"}, seats)
if err != nil || got != "http://forge.example.tld:3000/org/controller.git" {
t.Fatalf("got %q, %v", got, err)
}
got, err = contextURL(catalogue.ArtifactContext{Repository: "https://elsewhere.example/x.git"}, seats)
if err != nil || got != "https://elsewhere.example/x.git" {
t.Fatalf("a URL context was changed: %q, %v", got, err)
}
_, err = contextURL(catalogue.ArtifactContext{Seat: "git", Repository: "org/controller"}, nil)
if err == nil || !strings.Contains(err.Error(), "git seat") {
t.Fatalf("a seat with no base was not refused by name: %v", err)
}
}
+71
View File
@@ -0,0 +1,71 @@
package builder
import (
"strings"
"testing"
)
// Nothing could compile the mesh's own components, which is why nothing delivers the host
// (novox/hq 04-ISSUES/142, and ADR 0141's own insight naming it). The toolchain list was a closed
// set of typescript and python, and two things in the path beyond it assumed TypeScript.
func TestTheMeshCanCompileGo(t *testing.T) {
chain, err := ToolchainFor("go")
if err != nil {
t.Fatal(err)
}
// Named, not pinned: the mesh answers with the copy it holds, so moving compiler is a build
// rather than an edit to this source (ADR 0044, 0142).
if chain.Base != "mesh-tools-go" || chain.Artifact != "build" {
t.Fatalf("the go toolchain is based on %s/%s", chain.Base, chain.Artifact)
}
joined := strings.Join(chain.Compile, " ")
// Static, because what a machine holds is a file and not a container: a binary needing a libc
// it did not bring is a delivery that works until a machine differs.
if !strings.Contains(joined, "CGO_ENABLED=0") {
t.Fatalf("the go toolchain does not build statically: %q", joined)
}
// Reproducible: a version comes from where a component sits, not from its linker (ADR 0142),
// so two builds of one commit should produce the same bytes.
if !strings.Contains(joined, "-trimpath") {
t.Fatalf("the go toolchain leaves build paths in the binary: %q", joined)
}
if chain.Unit != UnitPackage {
t.Fatalf("the go toolchain compiles %q, wanted a package", chain.Unit)
}
}
func TestEveryToolchainSaysWhatItIsPointedAt(t *testing.T) {
// The field exists because the compile path used to assume one language. A toolchain that says
// nothing would fall through to the entrypoint branch and compile a file list, which for a
// compiled language builds a program out of exactly those files and ignores the rest of the
// package — a missing symbol rather than a legible refusal.
for _, chain := range toolchains {
switch chain.Unit {
case UnitPackage:
case UnitSources:
if chain.SourceExt == "" {
t.Fatalf("%s compiles a file list and names no source extension", chain.Language)
}
if !strings.HasPrefix(chain.SourceExt, ".") {
t.Fatalf("%s's source extension %q is not an extension", chain.Language, chain.SourceExt)
}
default:
t.Fatalf("%s says it is pointed at %q, which is neither sources nor a package",
chain.Language, chain.Unit)
}
}
}
func TestAnEntrypointBecomesASourceInItsOwnLanguage(t *testing.T) {
// It used to become a `.ts` whatever the language was.
out := Out("build")
got := sourcesFor([]string{out + "/tools/index.js"}, out, ".ts")
if len(got) != 1 || got[0] != "tools/index.ts" {
t.Fatalf("a typescript entrypoint became %v", got)
}
got = sourcesFor([]string{out + "/tools/index.js"}, out, ".py")
if len(got) != 1 || got[0] != "tools/index.py" {
t.Fatalf("a python entrypoint became %v", got)
}
}
+14
View File
@@ -180,6 +180,20 @@ func (r Registry) MirrorImage(ctx context.Context, from, repository string) (str
if err != nil {
return "", err
}
// **Already held is already mirrored.** A base is named by digest, and a digest this registry
// holds under the module's repository is the same bytes whatever upstream would say — so
// upstream is not asked. Asked every build, the public hub's anonymous pull limit was reached
// on the first merge that rebuilt a whole catalogue (2026-09-28), and every module whose base
// lives there failed on a copy it did not need.
if strings.HasPrefix(where.reference, "sha256:") {
held, err := r.has(ctx, "http://"+r.Address+"/v2/"+repository+"/manifests/"+where.reference, manifestAccept)
if err != nil {
return "", fmt.Errorf("asking %s whether it holds %s: %w", r.Address, from, err)
}
if held {
return r.Address + "/" + repository + "@" + where.reference, nil
}
}
src := &source{client: r.client()}
digest, err := r.copyManifest(ctx, src, where, where.reference, repository)
if err != nil {
+38
View File
@@ -103,6 +103,20 @@ func (m *theMeshsRegistry) handler() http.Handler {
m.mu.Lock()
defer m.mu.Unlock()
switch {
case r.Method == http.MethodHead && strings.Contains(r.URL.Path, "/manifests/"):
// **As strictly as a real registry.** A manifest is answered only in a media type the
// caller named; a request with no Accept is answered as if nothing were there. The fake
// used to answer regardless, which is why it could not catch a check that asked without
// one — and the mesh copied every base again (2026-09-28).
if !strings.Contains(r.Header.Get("Accept"), "manifest") && !strings.Contains(r.Header.Get("Accept"), "index") {
w.WriteHeader(http.StatusNotFound)
return
}
if _, ok := m.manifests[r.URL.Path[strings.LastIndex(r.URL.Path, "/")+1:]]; ok {
w.WriteHeader(http.StatusOK)
} else {
w.WriteHeader(http.StatusNotFound)
}
case r.Method == http.MethodHead && strings.Contains(r.URL.Path, "/blobs/"):
if _, ok := m.blobs[r.URL.Path[strings.LastIndex(r.URL.Path, "/")+1:]]; ok {
w.WriteHeader(http.StatusOK)
@@ -219,3 +233,27 @@ func TestATagBeforeTheDigestIsNotPartOfTheRepository(t *testing.T) {
t.Fatalf("got %+v", got)
}
}
// A base this registry already holds by digest is not asked of upstream at all: the public hub
// limits anonymous pulls, and a catalogue rebuilt on one merge asked it once per module.
func TestABaseAlreadyHeldIsNotAskedOfUpstream(t *testing.T) {
src, indexDigest, _ := anUpstreamRegistry(t)
dst := &theMeshsRegistry{blobs: map[string][]byte{}, manifests: map[string][]byte{}}
dstServer := httptest.NewServer(dst.handler())
defer dstServer.Close()
address := strings.TrimPrefix(dstServer.URL, "http://")
r := Registry{Address: address, HTTP: src.Client()}
host := strings.TrimPrefix(src.URL, "http://")
if _, err := r.MirrorImage(context.Background(), host+"/library/thing:latest", "hello-web/server"); err != nil {
t.Fatal(err)
}
// Upstream gone: the pinned base is answered from what the mesh holds.
src.Close()
reference, err := r.MirrorImage(context.Background(), host+"/library/thing@"+indexDigest, "hello-web/server")
if err != nil {
t.Fatalf("a base the registry holds was asked of an upstream that is gone: %v", err)
}
if reference != address+"/hello-web/server@"+indexDigest {
t.Fatalf("pinned as %q", reference)
}
}
+13 -1
View File
@@ -122,11 +122,23 @@ func (r Registry) PublishArchive(ctx context.Context, repository string, body []
return final, nil
}
func (r Registry) has(ctx context.Context, url string) (bool, error) {
// has is whether this registry already holds what is at that URL.
//
// **A manifest HEAD must say what it accepts.** A registry answers a manifest request only in a media
// type the caller named, and a bare HEAD — no Accept at all — is answered 404 for a manifest it holds
// perfectly well. Measured against the mesh's own registry (2026-09-28): the same digest answered 200
// with the manifest media types and 404 without them, so a check written without them concluded the
// registry held nothing, copied every base again, and exhausted the public hub's pull limit. A blob
// needs no Accept, which is why this went unnoticed: the same helper was right for blobs and wrong
// for manifests.
func (r Registry) has(ctx context.Context, url string, accept ...string) (bool, error) {
request, err := http.NewRequestWithContext(ctx, http.MethodHead, url, nil)
if err != nil {
return false, err
}
for _, media := range accept {
request.Header.Add("Accept", media)
}
response, err := r.client().Do(request)
if err != nil {
return false, fmt.Errorf("cannot reach the registry at %s: %w", r.Address, err)
+21
View File
@@ -179,3 +179,24 @@ func TestTheBasesABuildWasHandedAreWhatItStoodOn(t *testing.T) {
t.Fatalf("the bases the build was handed were not what it stood on: %v", got)
}
}
// What a build read besides its module's own repository is the second repository its recipes name,
// each once: a module that packages source living elsewhere is affected when that source moves.
func TestWhatABuildReadIsTheRepositoriesItsRecipesName(t *testing.T) {
elsewhere := catalogue.ArtifactContext{Repository: "http://forge.internal:20000/novox/mesh-controller.git", Ref: "main"}
manifest := catalogue.Manifest{
Module: "builder",
Build: &catalogue.Build{Artifacts: []catalogue.Artifact{
{Name: "server", Kind: catalogue.ArtifactImage, From: "Dockerfile", Context: &elsewhere},
{Name: "tools", Kind: catalogue.ArtifactImage, From: "Dockerfile", Context: &elsewhere},
{Name: "config", Kind: catalogue.ArtifactArchive, From: "etc"},
}},
}
read := readBy(manifest)
if len(read) != 1 || read[0] != elsewhere {
t.Fatalf("the repositories this build read are %+v", read)
}
if readBy(catalogue.Manifest{Module: "gitea", Build: &catalogue.Build{}}) != nil {
t.Fatal("a module whose recipes name no other repository read one")
}
}
+76
View File
@@ -0,0 +1,76 @@
package builder
import (
"strings"
"testing"
)
// A host built without knowing its system refuses every declaration before applying anything —
// safely, totally, and with nothing reporting it. The mesh built one, delivered it, started it, and
// it would have refused the first thing it was asked to do (novox/hq 04-ISSUES/161).
func TestTheGoToolchainStampsTheArtifactsSystem(t *testing.T) {
chain, err := ToolchainFor("go")
if err != nil {
t.Fatal(err)
}
if chain.SystemStamp != "main.builtFor" {
t.Fatalf("the go toolchain fills %q", chain.SystemStamp)
}
}
func TestALanguageWithNoPinnedSystemStampsNothing(t *testing.T) {
// Interpreted output is not pinned to a system, and a manifest declaring one for it is already
// refused. Nothing to fill.
for _, language := range []string{"typescript", "python"} {
chain, err := ToolchainFor(language)
if err != nil {
t.Fatal(err)
}
if chain.SystemStamp != "" {
t.Fatalf("%s fills %q, and its output is not pinned to a system",
language, chain.SystemStamp)
}
}
}
func TestTheStampIsTheOneThingTakenFromTheArtifact(t *testing.T) {
// The toolchain accepts nothing else from the module — anything it could override it would be
// writing a Dockerfile to override. The system is the stated exception, because a compiled
// binary is per system and the artifact is what declares one (ADR 0142).
chain, err := ToolchainFor("go")
if err != nil {
t.Fatal(err)
}
joined := strings.Join(chain.Compile, " ")
if strings.Contains(joined, "${") || strings.Contains(joined, "%s") {
t.Fatalf("the compile line takes something from the module: %q", joined)
}
}
func TestTheLinkerIsToldOnceNotTwice(t *testing.T) {
// A repeated flag is not a merged one: the Go command takes the last -ldflags and drops the
// first. Passing the toolchain's flags and then the stamp separately produced a binary that knew
// its system and had lost -s -w — 12.2MB against 8.5MB, with its debug info (04-ISSUES/161).
chain, err := ToolchainFor("go")
if err != nil {
t.Fatal(err)
}
for _, arg := range chain.Compile {
if arg == "-ldflags" {
t.Fatal("the compile line carries -ldflags, so composing one here makes two")
}
}
if len(chain.LinkerFlags) == 0 {
t.Fatal("the go toolchain passes no linker flags, so the binary keeps its debug info")
}
var stripped bool
for _, f := range chain.LinkerFlags {
if f == "-s" {
stripped = true
}
}
if !stripped {
t.Fatalf("the go toolchain does not strip: %v", chain.LinkerFlags)
}
}
+79
View File
@@ -35,8 +35,50 @@ type Toolchain struct {
Compile []string
// OutputFlag is how this compiler is told where to put its output.
OutputFlag string
// Unit is what this compiler is pointed at: UnitSources, the entrypoint files the module named,
// or UnitPackage, the one directory the artifact is built `from`.
//
// **The difference is the language and not the module.** A TypeScript bundle is a set of files
// compiled into a set of files, so what to compile is the module's entrypoints with their source
// extension. A Go bundle is a package compiled into one binary, and there is no per-file
// compilation to name — pointing `go build` at a file list builds a program out of exactly those
// files and ignores the rest of the package, which fails as a missing symbol rather than as a
// wrong instruction.
Unit string
// SourceExt is the extension an entrypoint has in the repository, for UnitSources. An entrypoint
// is named as it will be FOUND, inside the unpacked bundle, so the source is the same path with
// the output directory taken off the front and this on the end.
SourceExt string
// LinkerFlags are passed to the linker as one flag, together with the system stamp below.
//
// **Separate from Compile because a repeated flag is not a merged one.** They were in the compile
// line, and appending the stamp as a second `-ldflags` meant the Go command took the last and
// dropped the first — so the binary gained its system and lost `-s -w`, growing by half and
// carrying its debug info. The mistake was believing a comment rather than reading the file it
// produced (novox/hq 04-ISSUES/161).
LinkerFlags []string
// SystemStamp is the variable this language's linker fills with the artifact's declared system,
// for a language whose binaries are pinned to one at link time (novox/hq ADR 0005).
//
// **The one thing a toolchain takes from the artifact, and 0142 says why**: the target is a
// property of the artifact rather than of the recipe, because a compiled binary is per system
// and a toolchain that accepted it from the module would be accepting a build instruction. This
// is the narrow exception, named here rather than inferred.
//
// Empty for a language that compiles to nothing pinned. A host built without it refuses every
// declaration before applying anything — safely, totally, and with nothing reporting it
// (novox/hq 04-ISSUES/161).
SystemStamp string
}
// What a toolchain is pointed at.
const (
// UnitSources is a list of files, derived from the module's entrypoints.
UnitSources = "sources"
// UnitPackage is the single directory the artifact is built `from`, compiled whole.
UnitPackage = "package"
)
// Out is where one artifact's compiled output lands, inside the module's own directory.
//
// **Per artifact, never per toolchain.** A module is one piece of software and may still be
@@ -71,6 +113,41 @@ var toolchains = []Toolchain{
"--target", "ES2022",
},
OutputFlag: "--outDir",
Unit: UnitSources,
SourceExt: ".ts",
},
{
Language: "go",
Base: "mesh-tools-go",
Artifact: "build",
// **The mesh's own components, and not modules.** The warning above this list — that every
// language is another implementation of the contracts modules share, so adding one commits
// to keeping N implementations in step — does not attach here. Go is how the host, the
// control plane and the builder are written, and none of them is a module in that sense:
// the host is what APPLIES modules. So there is no SDK obligation, and the reason this
// entry did not exist was that nothing needed to compile the mesh itself
// (novox/hq ADR 0142, and 04-ISSUES/142 where that is why nothing delivers the host).
//
// Static, because what a machine ends up holding is a file rather than a container, and a
// binary that needs a libc it did not bring is a delivery that works until a machine
// differs. Trimmed of its own paths for the same reason a version comes from where it sits
// rather than from the linker: two builds of one commit produce the same bytes.
Compile: []string{
"env", "CGO_ENABLED=0", "GOFLAGS=-trimpath",
"go", "build",
},
// Stripped of symbols and debug info: what a machine holds is a file it runs, not one it
// debugs, and the difference measured 12.2MB against 8.5MB.
LinkerFlags: []string{"-s", "-w"},
OutputFlag: "-o",
// Pointed at the package the artifact is built `from`, compiled whole. Go writes the binary
// into the output directory, named after the package — so the bundle a machine unpacks is a
// directory holding one executable, which is what the delivery mechanism expects
// (novox/hq ADR 0141).
Unit: UnitPackage,
// The mesh's own Go components read the system they were built for from this variable, and
// refuse to touch a machine without one.
SystemStamp: "main.builtFor",
},
{
Language: "python",
@@ -82,6 +159,8 @@ var toolchains = []Toolchain{
// each actually does.
Compile: []string{"python", "-m", "pip", "install", "--no-compile", "--target"},
OutputFlag: "",
Unit: UnitSources,
SourceExt: ".py",
},
}
+8
View File
@@ -59,6 +59,10 @@ func anchorRendering(adopted bool) Rendering {
Values: map[string]any{ExposeSetting: map[string]any{"5000": FromEverywhere}}}}},
Mesh: []string{"10.42.0.1"},
Foundation: []int{5671},
// What the machine reported faces outside, which every rule in the filter is written
// around (novox/hq ADR 0140).
OutwardLinks: []string{"eth0"},
TunnelInterface: "mesh0",
Adopted: adopted,
// Genesis takes the foundation's modules.
Taken: map[string]bool{"postgres": true, "lavinmq": true},
@@ -579,6 +583,8 @@ func TestAGivenMachineSideReachesTheFilterTheOpeningAndTheConsumer(t *testing.T)
Given: map[string]map[int]int{"forge": given},
Mesh: []string{"10.77.0.1"},
Adopted: true,
OutwardLinks: []string{"eth0"},
TunnelInterface: "mesh0",
Taken: map[string]bool{"forge": true},
}
@@ -663,6 +669,8 @@ func TestALongFormPortIsOpenedWhereTheManifestPublishesIt(t *testing.T) {
Ports: map[string]map[int]int{"forge": portsAsThePlanWould(forge, nil)},
Mesh: []string{"10.77.0.1"},
Adopted: true,
OutwardLinks: []string{"eth0"},
TunnelInterface: "mesh0",
})
if err != nil {
t.Fatal(err)
@@ -0,0 +1,18 @@
package catalogue
import "testing"
// A claim written before the rename still holds (novox/hq ADR 0122, ADR 0156): with the store's
// aliases loaded, the former name resolves to the seat.
func TestTheArtifactStoresFormerNameResolvesToIt(t *testing.T) {
was := aliases
t.Cleanup(func() { aliases = was })
UseAliases(map[string]string{"the-artifact-store": "mesh-artifact-store"})
seat, known := SeatNamed("the-artifact-store")
if !known || seat.Name != "mesh-artifact-store" || seat.Delivers != "artifact-store" {
t.Fatalf("the former name did not resolve: %+v %v", seat, known)
}
if _, known := SeatNamed("mesh-artifact-store"); !known {
t.Fatal("the seat is not in the set under its name")
}
}
@@ -23,7 +23,7 @@ func TestASecondArtifactStoreAnywhereIsRefusedByName(t *testing.T) {
}
// A second one, on any other machine, is refused — and the refusal names the seat.
elsewhere := World{Held: []Held{{Claim: "the-artifact-store", Scope: ScopeMesh,
elsewhere := World{Held: []Held{{Claim: "mesh-artifact-store", Scope: ScopeMesh,
Node: "anchor", Module: "distribution"}}}
other := workstation()
other.Name = "laptop"
@@ -32,7 +32,7 @@ func TestASecondArtifactStoreAnywhereIsRefusedByName(t *testing.T) {
t.Fatal("a second store was accepted on another machine; it would offer artifact-store a " +
"second time and every consumer elsewhere would refuse to choose")
}
if !strings.Contains(err.Error(), "the-artifact-store") || !strings.Contains(err.Error(), "one per mesh") {
if !strings.Contains(err.Error(), "mesh-artifact-store") || !strings.Contains(err.Error(), "one per mesh") {
t.Fatalf("refused without naming the seat: %v", err)
}
}
+21
View File
@@ -70,6 +70,27 @@ func knownFor(m Manifest, needs []Needed, node string) map[string]map[string]str
return out
}
// withOwnNames adds a module's own composed names to what it may name from one binding:
// `${bound:<provision>:name}` and `:internal-name`, and for several contributions to one requirement
// `:name-<local>` / `:internal-name-<local>`. Set over anything the provider serves under those keys:
// what the module is called is the mesh's statement, not the provider's.
func withOwnNames(values map[string]string, own map[string]any) {
for _, key := range []string{"name", "internal-name"} {
if v, ok := own[key].(string); ok {
values[key] = v
}
}
many, _ := own["names"].(map[string]any)
for local, raw := range many {
names, _ := raw.(map[string]any)
for _, key := range []string{"name", "internal-name"} {
if v, ok := names[key].(string); ok {
values[key+"-"+local] = v
}
}
}
}
// plainly renders a served value as a program would expect to read it.
func plainly(value any) string {
switch v := value.(type) {
+1 -1
View File
@@ -59,7 +59,7 @@ func TestRenamingASeatDidNotRenameTheInterfaceItDelivers(t *testing.T) {
for _, pair := range []struct{ seat, delivers string }{
{"git", "git"},
{"npm-package-registry", "npm-package-registry"},
{"the-artifact-store", "artifact-store"},
{"mesh-artifact-store", "artifact-store"},
{"mesh-store", "postgres-database"},
{"mesh-broker", "mesh-bus"},
} {
+123 -1
View File
@@ -94,12 +94,43 @@ func (m Manifest) Resolve(built []Built) (Manifest, error) {
m.Module, r["id"], named)
case ArtifactImage, ArtifactUpstream:
filled["image"] = artifact.Reference
// An image is not unpacked anywhere, so it has no directory to be named for its
// version and `${version}` has nothing to mean. Refused rather than left as literal
// text in a path, which is how it would reach a machine and be created as a directory
// called `${version}`.
for key, value := range filled {
if text, isText := value.(string); isText && strings.Contains(text, versionRef) {
return Manifest{}, fmt.Errorf(
"%s: %v says %s in %q, and %q is an image — an image is not unpacked, so "+
"it has no versioned place. %s is for an archive or a bundle",
m.Module, r["id"], versionRef, key, named, versionRef)
}
}
case ArtifactArchive, ArtifactBundle:
// The same on the wire: both are bytes fetched by digest and unpacked. They differ in
// how they were made — one packed as it stood, the other compiled first — and a
// machine has no reason to care which.
filled["source"] = artifact.Reference
filled["digest"] = artifact.Digest
// **And `${version}`, so a resource can name a place that is this build's alone**
// (novox/hq ADR 0141, 04-ISSUES/142). A component is unpacked into a directory named
// for its version so it can read its own version from its path — and until this,
// nothing could compose that path: an archive named a fixed one in the manifest and
// nothing interpolated the build into it, so nothing could ask for
// `…/versions/<version>/` and every machine took a hand-placed fallback.
//
// The version is the artifact's own digest, short. Not the commit: two builds of one
// commit are meant to be the same bytes (the toolchains are `-trimpath` for this), and
// a content-addressed version means an unchanged build resolves to the path it already
// had — so re-composing a declaration moves nothing, where a commit would move the
// path of an identical binary and recreate everything that reads it.
for key, value := range filled {
text, isText := value.(string)
if !isText || !strings.Contains(text, versionRef) {
continue
}
filled[key] = strings.ReplaceAll(text, versionRef, versionOf(artifact.Digest))
}
default:
return Manifest{}, fmt.Errorf("%s: %q is a %q, and an artifact is %q, %q, %q or %q",
m.Module, named, artifact.Kind, ArtifactImage, ArtifactArchive, ArtifactUpstream,
@@ -142,7 +173,14 @@ func (b *Build) problems(module string) []string {
// is which compiler — because the mesh chooses that, and cannot choose for a module that
// has not said.
if a.Kind == ArtifactBundle || a.Kind == ArtifactPackage {
if a.From != "" {
// **Except for a language that compiles to a binary, where it names which one**
// (novox/hq 04-ISSUES/142). A bundle in an interpreted language is the module's own
// directory compiled whole, and naming a source would be describing its own build. A
// repository written in a compiled language holds several commands — the host and its
// bootstrap live in one, and the mesh needs the host — and "the module's own directory"
// is then not a package at all. So the compiled case may say which package, and says
// the module root by saying nothing.
if a.From != "" && !compilesToABinary(a.Language) {
problems = append(problems, fmt.Sprintf(
"%s: %q is a bundle and names what it is built from (%q). A bundle is built "+
"from the module's own directory; what it says is the language",
@@ -153,6 +191,26 @@ func (b *Build) problems(module string) []string {
"%s: %q is a bundle and says no language, so nothing can choose a compiler "+
"for it", module, a.Name))
}
// **A system, for a language that compiles to a binary** (novox/hq ADR 0142). A binary
// is pinned to one operating system at link time so a host refuses to touch a machine
// it was not built for (novox/hq ADR 0005); an artifact that says nothing would be
// compiled for whatever the build machine happened to be, which reads as portable and
// is not.
if compiled := compilesToABinary(a.Language); compiled && strings.TrimSpace(a.System) == "" {
problems = append(problems, fmt.Sprintf(
"%s: %q is compiled to a binary and says no system, so it would be built for "+
"whatever the build machine happens to be. Declare one artifact per "+
"system: %s", module, a.Name, spokenSystems()))
} else if !compiled && strings.TrimSpace(a.System) != "" {
problems = append(problems, fmt.Sprintf(
"%s: %q names the system %q and is written in %q, which compiles to code that "+
"runs anywhere — a system that decides nothing reads as though it did",
module, a.Name, a.System, a.Language))
} else if compiled && !knownSystem(a.System) {
problems = append(problems, fmt.Sprintf(
"%s: %q is built for %q, and a system is %s",
module, a.Name, a.System, spokenSystems()))
}
} else {
if a.From == "" {
problems = append(problems, fmt.Sprintf(
@@ -193,3 +251,67 @@ func oneOrOther(n int) string {
}
return "them"
}
// Systems the mesh builds binaries for, which is the set a host may be pinned to (novox/hq ADR 0005).
//
// **A closed list, and the host's own, not the compiler's.** These are not the values a Go toolchain
// would call an operating system — the difference between two of them is a C library, not a kernel.
// They are what a machine reports itself to be and what a host is linked to refuse, so the list that
// matters is the one the host understands.
var systems = []string{"alpine", "android", "arch"}
// knownSystem is whether the mesh builds for it.
func knownSystem(system string) bool {
want := strings.ToLower(strings.TrimSpace(system))
for _, s := range systems {
if s == want {
return true
}
}
return false
}
// spokenSystems is the list as a refusal says it, so a reader is one edit from right.
func spokenSystems() string {
return strings.Join(systems, ", ")
}
// compilesToABinary is whether this language's bundle is a binary for one operating system rather
// than code that runs wherever its interpreter does.
//
// **Asked of the language, not of the artifact.** A module says what it is written in; what that
// implies is the mesh's to know, exactly as the compiler is (novox/hq ADR 0142). Asking the artifact
// would let two artifacts in one language disagree about whether they are portable.
func compilesToABinary(language string) bool {
switch strings.ToLower(strings.TrimSpace(language)) {
case "go":
return true
default:
return false
}
}
// versionRef is how a resource names the version of the artifact it uses: ${version}.
//
// No artifact name in it, because the resource already says which artifact it is for — a second
// name would be a second thing to keep in step with the first.
const versionRef = "${version}"
// versionOf is an artifact's version as a path names it: its digest, short.
//
// **Content-addressed on purpose.** The alternative is the commit a build came from, and two builds
// of one commit are meant to produce the same bytes — every toolchain here is `-trimpath` for that
// reason. A commit-named path would move for an identical binary, and everything reading that path
// would be recreated for a change that is not one. A digest-named path moves exactly when the bytes
// do.
//
// Twelve hex characters: enough that two of this mesh's builds will not collide, short enough to
// read in a path and in a journal line. The `sha256:` prefix goes, because a directory name carrying
// a colon is a directory name people quote wrong.
func versionOf(digest string) string {
hex := strings.TrimPrefix(strings.TrimSpace(digest), "sha256:")
if len(hex) > 12 {
return hex[:12]
}
return hex
}
+82
View File
@@ -0,0 +1,82 @@
package catalogue
import (
"strings"
"testing"
)
// bundleFor is a manifest whose one artifact is a bundle in the given language and system.
func bundleFor(language, system string) Manifest {
return Manifest{Module: "a-component", Build: &Build{Artifacts: []Artifact{
{Name: "binary", Kind: ArtifactBundle, Language: language, System: system},
}}}
}
func problemsOf(t *testing.T, m Manifest) string {
t.Helper()
return strings.Join(m.Build.problems(m.Module), "\n")
}
// **A language that compiles to a binary must say which system.**
//
// A binary is pinned to one operating system at link time, so a host refuses to touch a machine it
// was not built for. An artifact that says nothing would be compiled for whatever the build machine
// happened to be — which reads as portable and is not, and is the fault this check exists for.
func TestABinaryMustSayWhichSystemItIsFor(t *testing.T) {
got := problemsOf(t, bundleFor("go", ""))
if !strings.Contains(got, "says no system") {
t.Fatalf("a compiled bundle with no system was accepted:\n%s", got)
}
// And the refusal names what it could have said, so a reader is one edit from right.
for _, system := range []string{"alpine", "android", "arch"} {
if !strings.Contains(got, system) {
t.Fatalf("the refusal does not name %q as a choice:\n%s", system, got)
}
}
}
func TestABinaryThatNamesASystemIsAccepted(t *testing.T) {
if got := problemsOf(t, bundleFor("go", "arch")); got != "" {
t.Fatalf("a compiled bundle naming a system was refused:\n%s", got)
}
}
// A system the mesh does not build for is refused where it is written. These are the host's own
// names, not a compiler's: the difference between two of them is a C library rather than a kernel,
// so a value that looks like an operating system to a toolchain is still wrong here.
func TestASystemTheMeshDoesNotBuildForIsRefused(t *testing.T) {
for _, wrong := range []string{"linux", "debian", "darwin"} {
got := problemsOf(t, bundleFor("go", wrong))
if !strings.Contains(got, "and a system is") {
t.Fatalf("%q was accepted as a system:\n%s", wrong, got)
}
}
}
// **And a language that runs anywhere must not name one.** A system that decides nothing reads as
// though it did, which is the same fault as a restriction that restricts nothing (novox/hq ADR 0045).
func TestAPortableBundleMayNotNameASystem(t *testing.T) {
got := problemsOf(t, bundleFor("typescript", "arch"))
if !strings.Contains(got, "runs anywhere") {
t.Fatalf("a portable bundle was allowed to name a system:\n%s", got)
}
}
func TestAPortableBundleNamingNoSystemIsAccepted(t *testing.T) {
if got := problemsOf(t, bundleFor("typescript", "")); got != "" {
t.Fatalf("an ordinary bundle was refused:\n%s", got)
}
}
// One component, one artifact per system: the shape the mesh's own binaries are declared in, and the
// reason the target is the artifact's rather than the recipe's.
func TestOneArtifactPerSystemIsAccepted(t *testing.T) {
m := Manifest{Module: "the-host", Build: &Build{Artifacts: []Artifact{
{Name: "arch", Kind: ArtifactBundle, Language: "go", System: "arch"},
{Name: "alpine", Kind: ArtifactBundle, Language: "go", System: "alpine"},
{Name: "android", Kind: ArtifactBundle, Language: "go", System: "android"},
}}}
if got := problemsOf(t, m); got != "" {
t.Fatalf("one artifact per system was refused:\n%s", got)
}
}
@@ -0,0 +1,88 @@
package catalogue
import (
"os"
"path/filepath"
"strings"
"testing"
)
// TestEveryCatalogueManifestParses runs the real catalogue through the real gate.
//
// Not a fixture: the point is whether the manifests as written are accepted by the control plane that
// will read them, and a copy of one manifest proves nothing about the other seventy-one.
// catalogueRoot is the catalogue these checks run over: MESH_CATALOGUE when set, else the checkout
// beside this one, the way the main layout has it. A check that only ran when somebody remembered a
// variable was a check nobody ran (novox/hq issue 134, 2026-09-30); it skips only when there is no
// catalogue to be found at all.
func catalogueRoot(t *testing.T) string {
t.Helper()
if root := os.Getenv("MESH_CATALOGUE"); root != "" {
return root
}
sibling := filepath.Join("..", "..", "..", "mesh-catalog")
if _, err := os.Stat(filepath.Join(sibling, "modules")); err != nil {
t.Skip("no catalogue beside this checkout and MESH_CATALOGUE unset")
}
return sibling
}
func TestEveryCatalogueManifestParses(t *testing.T) {
root := catalogueRoot(t)
found, err := filepath.Glob(filepath.Join(root, "modules", "*", "module.json"))
if err != nil || len(found) == 0 {
t.Fatalf("no manifests under %s: %v", root, err)
}
named, routed := 0, 0
for _, p := range found {
raw, err := os.ReadFile(p)
if err != nil {
t.Fatalf("%s: %v", p, err)
}
m, err := ParseManifest(raw)
if err != nil {
t.Errorf("%s: %v", filepath.Base(filepath.Dir(p)), err)
continue
}
for _, l := range m.Listens {
if l.Name != "" {
named++
}
}
for port := range RoutedPorts(m) {
_ = port
routed++
}
}
t.Logf("%d manifests, %d named endpoints, %d routed endpoints resolved", len(found), named, routed)
if named == 0 {
t.Fatal("no endpoint in the catalogue is named, so this proved nothing")
}
}
// TestNoCatalogueManifestNamesAnInstallation is ADR 0112's check, run over the real catalogue: no
// definition names a domain or a public address the mesh acts on, and every value that must for now
// carries its reason (novox/hq ADR 0155, issue 134). The list it prints is the one that shrinks.
func TestNoCatalogueManifestNamesAnInstallation(t *testing.T) {
root := catalogueRoot(t)
found, err := filepath.Glob(filepath.Join(root, "modules", "*", "module.json"))
if err != nil || len(found) == 0 {
t.Fatalf("no manifests under %s: %v", root, err)
}
var named []string
for _, p := range found {
raw, err := os.ReadFile(p)
if err != nil {
t.Fatalf("%s: %v", p, err)
}
m, err := ParseManifest(raw)
if err != nil {
t.Errorf("%s: %v", p, err)
continue
}
named = append(named, InstallationProblems(m)...)
}
if len(named) > 0 {
t.Fatalf("%d value(s) name an installation:\n %s", len(named), strings.Join(named, "\n "))
}
}
@@ -0,0 +1,71 @@
package catalogue
import (
"os"
"strings"
"testing"
)
// **A machine trusts the mesh's authority because a module put its root there** (novox/hq ADR
// 0147, issue 129). The module carries a shell script and a unit, and both are worthless unless
// the mesh fills in where the authority is — which is the one thing about it the manifest cannot
// state, because the authority's address is a fact about the mesh and not about the module.
//
// So what is checked here is the rendering, not the parsing: the script the machine will run
// names the authority it was bound to, and the unit runs that script both ways. The verification
// itself — a plain client trusting an internal name on a machine holding this, and failing on one
// that does not — is the lab's, and cannot be had here.
func TestCaTrustRendersTheAuthorityItWasBoundTo(t *testing.T) {
raw, err := os.ReadFile("../../../mesh-catalog/modules/ca-trust/module.json")
if err != nil {
t.Skipf("the catalogue is not beside this checkout: %v", err)
}
m, err := ParseManifest(raw)
if err != nil {
t.Fatalf("the trust module does not parse:\n%v", err)
}
r := Resolution{
Node: "workstation",
Modules: []Manifest{m},
Needs: []Needed{{
Name: "internal-acme-ca", From: "anchor", At: "anchor.internal", For: "ca-trust",
Serves: map[string]any{
"port": float64(9000), "path": "/acme/acme/directory", "roots": "/roots.pem",
},
}},
}
out, err := r.Declaration(Rendering{})
if err != nil {
t.Fatalf("the trust module could not be composed for a machine: %v", err)
}
script := fileNamed(out, "ca-trust.anchor")
if script == nil {
t.Fatalf("nothing writes the script the unit runs: %v", out)
}
body, _ := script["content"].(string)
if !strings.Contains(body, "https://anchor.internal:9000/roots.pem") {
t.Errorf("the script does not fetch from the authority it was bound to:\n%s", body)
}
if script["mode"] != "0755" {
t.Errorf("the script is written %v, which systemd cannot execute", script["mode"])
}
unit := fileNamed(out, "ca-trust.unit")
if unit == nil {
t.Fatalf("no unit: %v", out)
}
text, _ := unit["content"].(string)
// Both halves. A unit that only installs the anchor leaves a machine trusting an authority
// nobody assigned it to any more, which is the half issue 129 asked for by name.
for _, want := range []string{
"ExecStart=" + script["path"].(string) + " install",
"ExecStop=" + script["path"].(string) + " remove",
"RemainAfterExit=yes",
} {
if !strings.Contains(text, want) {
t.Errorf("the unit does not say %q:\n%s", want, text)
}
}
}
+21
View File
@@ -197,6 +197,27 @@ func TestARouteCanBeSetPerMesh(t *testing.T) {
}
}
func TestASettingReachesAContributionOnlyWhereItDeclaresTheKey(t *testing.T) {
// novox/hq 04-ISSUES/173: the mail module's site name, set so its environment file could read
// it, arrived in every route it contributed. A setting overrides a key the contribution
// declares and adds none — the provider reads the contribution as a contract.
got, _ := Resolve(shelf(proxy(), published("board", "board", 8080)), []string{"board"}, workstation(), World{})
out, err := got.Declaration(Rendering{Settings: SettingsBy{
"board": {{From: "the mesh", Values: map[string]any{"host": "dashboard", "sitename": "Board"}}},
}})
if err != nil {
t.Fatal(err)
}
given := received(t, out)
if given[0].Values["host"] != "dashboard" {
t.Fatalf("the setting did not override the route's host: %v", given[0].Values)
}
if _, leaked := given[0].Values["sitename"]; leaked {
t.Fatalf("a setting the route never declared reached the proxy: %v", given[0].Values)
}
}
func TestReceivingWhatYouDoNotProvideIsRefused(t *testing.T) {
// It would create a file nobody ever writes to, on a machine where nothing asked for it.
_, err := ParseManifest([]byte(`{"module":"traefik","version":"1",
+434 -64
View File
@@ -124,6 +124,16 @@ type Rendering struct {
// nothing on this node keeps them, or the mesh has no operator key.
Kept *KeptExport
// OutwardLinks is the links this machine reported as facing outside it, which the filter is
// written around (novox/hq ADR 0140). Empty means the machine has not said, and the mesh
// composes no filter for it rather than writing a rule around a link with no name.
OutwardLinks []string
// TunnelInterface is the interface the mesh's private network runs on, named here rather than
// imported because the overlay package rests on this one. Traffic arriving on it is the mesh's,
// not this machine's own guest, so the filter admits it only by a rule.
TunnelInterface string
// Foundation is the ports the mesh itself needs reachable on every machine, which no module
// declares because the foundation is not a module (novox/hq 04-ISSUES/051 and 052). The broker
// is the one that matters: a machine dials it to enrol, and a firewall derived only from
@@ -345,7 +355,21 @@ func (r Resolution) compose(with Rendering, owner map[string]string) ([]map[stri
if err != nil {
return nil, err
}
filtering := AsNftables(rules, with.Mesh, r.PublicDomain != "", with.Foundation)
// **A machine that has not said which links face outside is sent no filter** (novox/hq ADR
// 0140). The whole chain is written around those links: with none, the rule that lets this
// machine's own guests keep working would name an empty set, which nftables refuses, and a rule
// set that does not load is a machine filtering nothing while its unit reports success. Refused
// here, where a person reads it, rather than on the machine — and the machine keeps the filter
// it already has.
if filters := r.filtersHere(); filters != "" && len(with.OutwardLinks) == 0 {
return nil, fmt.Errorf(
"%s cannot be sent a filter: it has not reported which of its links face outside, and "+
"every rule in the chain is written around them. It reports that on each apply; "+
"`node show %s` says whether it has. Until then %s is not sent, and the machine "+
"keeps the filter it has", r.Node, r.Node, filters)
}
filtering := AsNftables(rules, with.Mesh, r.PublicDomain != "", with.Foundation,
with.OutwardLinks, with.TunnelInterface)
var out []map[string]any
for _, m := range r.Modules {
@@ -449,9 +473,15 @@ func (r Resolution) compose(with Rendering, owner map[string]string) ([]map[stri
// find each present — refusing clearly if the operator has not provided it — before it
// starts anything that depends on it. The mesh creates, chowns and reconciles none of it;
// an `access` resource says only *this path must exist, and this module reaches it*.
for _, a := range m.Accesses {
// Where each is on THIS machine is the assignment's (novox/hq issue 153): placed by id
// where the operator said, the definition's default otherwise, refused where neither.
accesses, accessPaths, err := accessesFor(m, with.Settings[m.Module])
if err != nil {
return nil, err
}
for _, a := range accesses {
first = append(first, map[string]any{
"id": AccessID(a.Path), "type": "access", "path": a.Path, "mode": a.At(),
"id": AccessID(a.Path), "type": "access", "path": a.Path, "mode": a.Mode,
})
}
for _, to := range m.SecretRequirements() {
@@ -550,7 +580,11 @@ func (r Resolution) compose(with Rendering, owner map[string]string) ([]map[stri
}
found = here
}
file, err := boundFile(*found, m.Binds[to], ConsumerIdentity(r.Node, IdentitySource(m.Slug, m.Module)))
own, err := r.ownNames(m, to, with.Settings[m.Module])
if err != nil {
return nil, err
}
file, err := boundFile(*found, m.Binds[to], ConsumerIdentity(r.Node, IdentitySource(m.Slug, m.Module)), own)
if err != nil {
return nil, err
}
@@ -594,17 +628,15 @@ func (r Resolution) compose(with Rendering, owner map[string]string) ([]map[stri
// merging earlier would throw away the files it still needs.
resources = append(append([]map[string]any{}, first...), resources...)
// Every container is given the mesh's names. Not a choice a module makes: a module that
// listed them would go stale the day a machine joins, and one that did not would be a
// module whose containers cannot reach anything by name.
//
// A container that was given names of its own keeps them and gets the mesh's beside them:
// the mesh does not know what else a workload needs to reach, and taking something away
// to add something is not what "also" means.
if len(with.Names) > 0 {
resources = withMeshNames(resources, with.Names)
}
// No container is given the mesh's names (novox/hq ADR 0148). It used to be: every
// container got the whole roster as `--add-host` entries at creation, and a name that
// moved afterwards was wrong inside it for as long as it ran (issues 109, 135) — and once
// the roster was made part of a container's identity so that could be caught, one name
// moving anywhere replaced every container in the mesh (issue 151). A container resolves a
// mesh name through its machine's resolver at the moment it asks, which the runtime is
// told once per machine, as a file, by the resolver's own module. The names a module
// declares for itself are its own and stay exactly as written: they are part of what the
// module is, and the mesh does not know what they mean.
// What this module may name from inside one of its own files. Gathered once per module
// rather than per file, because it is a fact about the module.
sealed, err := sealedFor(m, r.Needs, with)
@@ -613,8 +645,43 @@ func (r Resolution) compose(with Rendering, owner map[string]string) ([]map[stri
}
// And what its bindings say, for the half of a connection that is not secret.
known := knownFor(m, r.Needs, r.Node)
// A requirement answered on this same machine is not in r.Needs — its binding file is
// written from `here` (above) — and so `${bound:…}` could not name it, though the file
// beside it said the same facts. Filled from the same answer, so the two cannot disagree.
for _, want := range m.Wants() {
if _, has := known[want]; has {
continue
}
answered, err := here(r, want, with)
if err != nil {
return nil, err
}
if answered == nil {
continue
}
local := *answered
local.For = m.Module
for provision, values := range knownFor(m, []Needed{local}, r.Node) {
known[provision] = values
}
}
// And what the module is called through each requirement it contributes to (novox/hq
// 04-ISSUES/122) — the same composition its binding file carries.
for provision, values := range known {
own, err := r.ownNames(m, provision, with.Settings[m.Module])
if err != nil {
return nil, err
}
withOwnNames(values, own)
}
// And where this node places the directories the module declared without a path
// (novox/hq ADR 0112) — resolved once per module, named by ${dir:…} from any resource.
// — and, on an adopted machine, where the assignment says they already are, with the
// owner the data already has (novox/hq issue 153). Malformed placements are refused here.
placed, err := Places(m, with.Settings[m.Module])
if err != nil {
return nil, err
}
dirs := dirsFor(m, with)
// And the machine underneath, which no binding of its own can tell it.
thisMachine := machineFacts(r, with.Names, with.MeshRange)
@@ -623,6 +690,9 @@ func (r Resolution) compose(with Rendering, owner map[string]string) ([]map[stri
// one of them as its environment without saying so (ADR 0086, issue 041).
secretFiles := secretFilesOf(resources)
// Which of this module's resources its preparation runs before, if it prepares anything.
prepareBefore := preparationTarget(m)
for _, unsettled := range resources {
resource, err := ApplySettings(unsettled, with.Settings[m.Module])
if err != nil {
@@ -638,6 +708,13 @@ func (r Resolution) compose(with Rendering, owner map[string]string) ([]map[stri
// Said in the catalogue, not on the machine: the host parses strictly and knows no
// such field, and the reason is for a reader of the manifest.
delete(copied, SecretsInEnvironment)
delete(copied, NamesOnPurpose)
// **An operator's value, from the assignment** (novox/hq ADR 0112, ADR 0155): what a
// definition may not carry because it is true of one installation only. Filled from
// the same layers a mergeable file takes, and refused when no layer set it.
if err := settingInto(copied, with.Settings[m.Module], m.Module); err != nil {
return nil, err
}
// **Placed before anything reads a path.** A pathless directory receives the path
// this node resolves for it, and every ${dir:…} — in paths, mounts, content and
// environment — becomes that path, so what follows sees only concrete places
@@ -645,6 +722,12 @@ func (r Resolution) compose(with Rendering, owner map[string]string) ([]map[stri
if err := dirInto(copied, dirs, m.Module); err != nil {
return nil, err
}
// The operator's data the same way: ${access:…} becomes where this node keeps it,
// and a placed directory takes the owner the assignment said (issue 153).
if err := accessInto(copied, accessPaths, m.Module); err != nil {
return nil, err
}
ownerInto(copied, placed)
// **After settings, and that is the whole reason it is here.** A module's file
// content is where a setting lands, so a placeholder may only exist once the setting
// has been put in — filling secrets first would look at content that is not yet what
@@ -708,6 +791,18 @@ func (r Resolution) compose(with Rendering, owner map[string]string) ([]map[stri
if renamed := reflectsRenamed(m.Module, resource["reload-on"]); renamed != nil {
copied["reload-on"] = renamed
}
// **A version prepares its state before it runs** (novox/hq ADR 0135). Derived from the
// module's own resource rather than declared beside it: what prepares the state is the
// module's own code, so what it is given has to be what that code is given — and a
// second resource written by hand is a second copy to drift from the first. Placed
// immediately before it, because a run-once step stops everything the declaration
// places after it (ADR 0052), which is how a version whose preparation failed does not
// serve.
if prepareBefore != "" && fmt.Sprint(resource["id"]) == prepareBefore {
step := prepared(copied)
owner[fmt.Sprint(step["id"])] = m.Module
out = append(out, step)
}
owner[fmt.Sprint(copied["id"])] = m.Module
out = append(out, copied)
}
@@ -837,6 +932,17 @@ func mapping(written string) (outer, inner int, address string, ok bool) {
return outer, inner, strings.Join(parts[:len(parts)-2], ":"), true
}
// filtersHere is the module on this node that loads the machine's packet filter, or empty when none
// does. Named rather than counted: a refusal that says which module is one step from acted on.
func (r Resolution) filtersHere() string {
for _, m := range r.Modules {
if m.Filtering != nil {
return m.Module
}
}
return ""
}
// Rules is the rule set this node's filter is derived from: every module's listens, what was
// computed for this machine, and each module's per-node exposure. The same answer whether the node
// is adopted or converged — the one loads it as a filter, the other declares it as openings.
@@ -847,6 +953,35 @@ func (r Resolution) Rules(with Rendering) ([]Rule, error) {
if err != nil {
return nil, err
}
// And how far each endpoint reaches, which says the same thing to the filter and more
// besides (novox/hq ADR 0138). Folded in here rather than beside: the filter has one
// question — from where — and a reach answers it, so giving it two inputs would let them
// disagree. Reaches refuses a port that both name, so this cannot silently prefer one.
reaches, err := Reaches(m, with.Settings[m.Module])
if err != nil {
return nil, err
}
// **Only for an endpoint the proxy does not serve.** A routed endpoint's port is how the
// proxy reaches it and nothing else (ADR 0045), so `public` there asks for a public name and
// says nothing about the port — opening it to the world as well would undo the arrangement
// the proxy exists for, and would silently reopen a port an operator had narrowed.
//
// Found by trying to express a real module: one whose routed name must be public and whose
// machine-side port must not be. Under one value for both, there was no way to say it.
routed := RoutedPorts(m)
for port, reach := range reaches {
if routed[port] {
continue
}
source, ok := FilterSource(reach)
if !ok {
return nil, fmt.Errorf("%s: %q is not a reach the filter can read", m.Module, reach)
}
if e == nil {
e = map[int]string{}
}
e[port] = source
}
if e != nil {
exposure[m.Module] = e
}
@@ -1010,12 +1145,11 @@ func (r Resolution) contributions(settings SettingsBy, grants []Grant,
// Settings reach a contribution the same way they reach a file. A route's hostname is
// exactly the kind of thing that differs between one mesh and the next, and a module
// that could not have it set would have to be edited to be reused.
values, err := settle(m.Contributes[to], settings[m.Module], nil,
values, err := r.composed(m, to, m.Contributes[to], settings[m.Module],
m.Module+" contributing to "+to)
if err != nil {
return nil, fmt.Errorf("%s contributing to %s: %w", m.Module, to, err)
return nil, err
}
composeName(values, r.PublicDomain, r.At)
out[to] = append(out[to], Contribution{From: m.Module, Values: values})
}
// Several contributions to one requirement (ADR 0094's sibling for `contributes`): an
@@ -1024,12 +1158,11 @@ func (r Resolution) contributions(settings SettingsBy, grants []Grant,
// name always reaches the provider from here.
for _, to := range sortedKeys(m.ContributesMany) {
for _, local := range sortedKeys(m.ContributesMany[to]) {
values, err := settle(m.ContributesMany[to][local], settings[m.Module], nil,
values, err := r.composed(m, to, m.ContributesMany[to][local], settings[m.Module],
m.Module+" contributing "+local+" to "+to)
if err != nil {
return nil, fmt.Errorf("%s contributing %s to %s: %w", m.Module, local, to, err)
return nil, err
}
composeName(values, r.PublicDomain, r.At)
out[to] = append(out[to], Contribution{From: m.Module, Values: values})
}
}
@@ -1037,6 +1170,85 @@ func (r Resolution) contributions(settings SettingsBy, grants []Grant,
return out, nil
}
// composed is one contribution as its provider receives it: settled with this node's settings, its
// endpoint's port filled in, and its names composed from the label.
//
// **One function, because two readers must agree.** The provider is told the names in its received
// file; the contributing module is told the same names in its own binding (novox/hq 04-ISSUES/122).
// Composing them twice, in two places, is how the proxy would come to serve one name while the
// module wrote another into its configuration.
func (r Resolution) composed(m Manifest, to string, raw map[string]any, layers []Layer, what string) (
map[string]any, error) {
// Overridden, not merged: a setting changes a key the contribution declares and adds none.
// The provider reads the contribution as a contract, and a setting made for one of this
// module's files is no part of it (novox/hq 04-ISSUES/173).
values, err := overridden(raw, layers, what)
if err != nil {
return nil, fmt.Errorf("%s: %w", what, err)
}
reaches, err := Reaches(m, layers)
if err != nil {
return nil, fmt.Errorf("%s: %w", what, err)
}
blocks, err := Endpoints(m, layers)
if err != nil {
return nil, fmt.Errorf("%s: %w", what, err)
}
portOfEndpoint(values, endpointPorts(m))
composeName(values, r.PublicDomain, servingAt(r, to), reaches, endpointPorts(m), blocks)
return values, nil
}
// ownNames is what a module is known by through what it contributes to one requirement — the names
// the mesh composed for it, and nothing else of the contribution.
//
// **The half a module could not learn** (novox/hq 04-ISSUES/122). A module contributes a label, the
// mesh joins it with this node's domains, and the provider serves the result — and the module itself
// was never told. Software that must know its own address (a login redirect, a canonical URL, an
// issuer) had it written into the manifest as a literal, which is a domain in a definition and wrong
// on every other machine. `${bound:<requirement>:name}` is the answer, from the same composition the
// provider receives.
//
// Several contributions to one requirement are keyed by their local name under `names`.
func (r Resolution) ownNames(m Manifest, to string, layers []Layer) (map[string]any, error) {
pick := func(values map[string]any) map[string]any {
names := map[string]any{}
for _, key := range []string{"name", "internal-name"} {
if v, ok := values[key].(string); ok && v != "" {
names[key] = v
}
}
return names
}
out := map[string]any{}
if raw, ok := m.Contributes[to]; ok {
values, err := r.composed(m, to, raw, layers, m.Module+" contributing to "+to)
if err != nil {
return nil, err
}
for k, v := range pick(values) {
out[k] = v
}
}
if locals := m.ContributesMany[to]; len(locals) > 0 {
many := map[string]any{}
for _, local := range sortedKeys(locals) {
values, err := r.composed(m, to, locals[local], layers,
m.Module+" contributing "+local+" to "+to)
if err != nil {
return nil, err
}
if names := pick(values); len(names) > 0 {
many[local] = names
}
}
if len(many) > 0 {
out["names"] = many
}
}
return out, nil
}
// composeName joins a contribution's label with a node's public domain, and separately with its
// private one, in place (novox/hq ADR 0056).
//
@@ -1060,10 +1272,44 @@ func (r Resolution) contributions(settings SettingsBy, grants []Grant,
// the running mesh keeps serving the full names it has. And a labelled contribution on a node with
// no public domain composes nothing — there is nothing to join it to — which reads downstream as a
// route that named no host, the same as it would have before this existed.
func composeName(values map[string]any, publicDomain, internalDomain string) {
func composeName(values map[string]any, publicDomain, internalDomain string, reaches map[int]string,
ports map[string]int, blocks map[string]Endpoint) {
if values == nil {
return
}
// **The subdomain an assignment gave this endpoint**, before the name is joined (novox/hq ADR
// 0138). The module contributes a label because it names its own parts; an assignment may say a
// different one, because where a thing lives under a domain is the operator's to choose and used
// to require editing the module to change.
if name, ok := values[RouteEndpoint].(string); ok {
if ep, said := blocks[strings.TrimSpace(name)]; said && ep.Label != "" {
values["label"] = ep.Label
}
}
// **How far the endpoint this route serves reaches decides which names exist** (novox/hq ADR
// 0138). Both were composed whenever the node had both domains, so every routed module got a
// public name and an internal one whether anybody wanted them or not — and a certificate for
// each, because the proxy certifies the names it is given.
//
// Joined by the port: a route entry names the port it serves and the module declares a listen on
// it. An entry with no port is not an endpoint's route but a rule about a name — a path-level
// refusal shadowing another route — and it inherits whatever that route's names turned out to
// be, which is why it is left alone here.
//
// Nothing said is both names, as before. That is what keeps every mesh already running identical
// until an assignment speaks.
wantPublic, wantInternal := true, true
if port, ok := endpointPortOf(values, ports); ok {
if reach, said := reaches[port]; said {
wantPublic, wantInternal = WantsPublicName(reach), WantsInternalName(reach)
}
}
if !wantPublic {
publicDomain = ""
}
if !wantInternal {
internalDomain = ""
}
if _, already := values["name"]; already {
// A full name was given rather than a label. Left as-is: this is the legacy shape, and the
// point of the label is to not have to write the full name — a contribution that wrote both
@@ -1095,6 +1341,24 @@ func composeName(values map[string]any, publicDomain, internalDomain string) {
}
}
// servingAt is the private-network name of the node a contribution to `to` arrives at: the
// provider's, when the provision is answered elsewhere, and this machine's own when it is answered
// here or not yet settled.
//
// A route's internal name is composed under it (novox/hq ADR 0151, issue 139). `<label>.<node>.internal`
// is answered by every machine's resolver as *anything under that node's name goes to that node* —
// so the node in the name has to be the one whose proxy answers, or the name sends a client to a
// machine with nothing listening while the public name, published at the serving node's address,
// works. Where the proxy runs beside the module the two are the same machine and nothing changes.
func servingAt(r Resolution, to string) string {
for _, n := range r.Needs {
if n.Name == to && n.At != "" {
return n.At
}
}
return r.At
}
// receivedFile is the file a provider is given its consumers' contributions in.
func receivedFile(requirement, path string, given []Contribution) (map[string]any, error) {
if given == nil {
@@ -1199,14 +1463,14 @@ func sortedKeys[V any](m map[string]V) []string {
// Where it is and what the providing module said about using it. **No credential**, and the file
// says so rather than leaving a reader to wonder whether one was meant to be there — a missing
// field looks like a bug, and a stated absence looks like a boundary.
func boundFile(n Needed, path, as string) (map[string]any, error) {
func boundFile(n Needed, path, as string, own map[string]any) (map[string]any, error) {
// A record has no machine and no address. Saying so is the difference between a reader
// concluding "somewhere with no address" and concluding the mesh failed to fill something in.
where := any(n.At)
if n.ByRecord {
where = "a record in this mesh, not a machine"
}
body, err := json.MarshalIndent(map[string]any{
doc := map[string]any{
"binding": 1,
"provision": n.Name,
"from": n.From,
@@ -1222,7 +1486,14 @@ func boundFile(n Needed, path, as string) (map[string]any, error) {
"generated": "by the mesh — do not edit; replaced whenever this changes. " +
"The credential is not here: it is sealed, in the file this module's manifest " +
"names under `secrets`",
}, "", " ")
}
// **What this module is called through what it contributes here** (novox/hq 04-ISSUES/122):
// `name`, `internal-name`, or `names` by local name — composed exactly as the provider receives
// them. Absent when the module contributes nothing named, rather than written empty.
for key, value := range own {
doc[key] = value
}
body, err := json.MarshalIndent(doc, "", " ")
if err != nil {
return nil, err
}
@@ -1351,43 +1622,6 @@ func (r Resolution) servedOnThisMachine(provision string, with Rendering) (map[s
return nil, false, nil
}
// withMeshNames gives every container in a set the mesh's names.
//
// Copied rather than edited in place: these maps come from a module's manifest, and mutating one
// would change what the catalogue holds for every other machine running that module.
//
// A host-network container gets the names too. It was once skipped, on the belief that it "shares
// the machine's hosts file already" — but it does not: `docker run --network host` still gives the
// container its own /etc/hosts (localhost and its own id only), so every `<node>.internal` name the
// mesh wrote for the machine is invisible inside it, and a client that dials one gets EAI_AGAIN. The
// remedy is the same `--add-host` every other container gets — the runtime accepts it with
// `--network host` (verified), and without it a host-network consumer cannot reach a provider by the
// `.internal` address the mesh hands it as `${bound:...:at}`.
func withMeshNames(resources []map[string]any, names map[string]string) []map[string]any {
out := make([]map[string]any, 0, len(resources))
for _, r := range resources {
if r["type"] != "container" {
out = append(out, r)
continue
}
copied := map[string]any{}
for k, v := range r {
copied[k] = v
}
var given []any
if already, ok := copied["hosts"].([]any); ok {
given = append(given, already...)
}
for _, name := range sortedKeys(names) {
given = append(given, name+":"+names[name])
}
copied["hosts"] = given
out = append(out, copied)
}
return out
}
// pinned refuses an image that is not really pinned, on its way to a machine.
//
// **Here and not at parse** (novox/hq 04-ISSUES/025). A manifest in a repository names artifacts
@@ -1466,14 +1700,23 @@ func publishedOn(resource map[string]any, module string, with Rendering) {
out = append(out, givenOuter(written, with.Given[module]))
continue
}
wanted, err := strconv.Atoi(strings.TrimSpace(written))
// A short form may carry the protocol — `"3478/udp"` — and the number is what the mesh
// assigns for; the protocol rides along. Read as one token, the `/udp` made the whole
// entry "not a port", and passing it through let the runtime publish it wherever it
// liked: unifi's STUN and discovery landed on random machine ports while every TCP pin
// beside them held.
mapping, protocol := written, ""
if cut := strings.LastIndex(written, "/"); cut >= 0 {
mapping, protocol = written[:cut], written[cut:]
}
wanted, err := strconv.Atoi(strings.TrimSpace(mapping))
if err != nil {
// Not a port at all. Passed through, so the host refuses it with its own words rather
// than this quietly dropping something somebody meant.
out = append(out, written)
continue
}
out = append(out, fmt.Sprintf("%d:%d", with.machinePort(module, wanted), wanted))
out = append(out, fmt.Sprintf("%d:%d%s", with.machinePort(module, wanted), wanted, protocol))
}
resource["ports"] = out
}
@@ -1610,3 +1853,130 @@ func atMachinePort(serves map[string]any, module string, ports map[string]map[in
func AtPublishedPort(values map[string]any, module string, published map[int]int) map[string]any {
return atMachinePort(values, module, map[string]map[int]int{module: published})
}
// PreparationArgument is how the mesh asks a module to prepare its state: one word, to the module's
// own program, whatever that program is (novox/hq ADR 0135).
//
// **One word for every kind of module.** A module built as a Go binary receives it as its argument;
// one built as a bundle receives it through the runtime, whose entry takes the same word. So the
// mesh has one way of asking and a module has one way of answering, and neither learns the other's
// shape.
const PreparationArgument = "prepare"
// preparationTarget is the resource a module's preparation runs before: its own workload.
//
// The first container carrying an artifact this module built, and not itself a step — that is the
// thing that runs the module's code, and therefore the thing whose state must be ready. Empty when
// the module prepares nothing, or when nothing it declares could run its code.
//
// **A module with two own workloads gates the first of them.** Five modules in the catalogue declare
// more than one container of their own, none of them preparing anything today. If one ever does and
// its second workload shares the state, the gate is in front of the first — stated here because the
// alternative is a field asking an author to restate what the mesh can see.
func preparationTarget(m Manifest) string {
if !m.Prepares {
return ""
}
for _, r := range m.Resources {
if fmt.Sprint(r["type"]) != "container" || !ownArtifact(r, m.Module) {
continue
}
if once, _ := r["run-once"].(bool); once {
continue
}
return fmt.Sprint(r["id"])
}
return ""
}
// ownArtifact is whether a resource runs something this module built, in either spelling a manifest
// may be in: naming the artifact, before a build resolved it, or carrying the reference a build
// recorded — this mesh's own store, under this module's name.
func ownArtifact(resource map[string]any, module string) bool {
if named, _ := resource["artifact"].(string); named != "" {
return true
}
image, _ := resource["image"].(string)
return strings.HasPrefix(image, ArtifactStoreScheme+module+"/")
}
// prepared is the module's own resource as the step that prepares its state: the same image, the same
// context, run to completion with the mesh's preparation argument.
//
// Three things are taken away rather than copied, each because the step runs while the version it
// prepares for is still running. A published port cannot be bound twice, and a step that tried would
// fail for a reason that has nothing to do with the state. A fixed address cannot be held twice, for
// the same reason. And a cadence is what a step is the opposite of: a container runs once and gates,
// or on a schedule, or stays up, never two (ADR 0053).
func prepared(from map[string]any) map[string]any {
step := map[string]any{}
for k, v := range from {
step[k] = v
}
// **A hyphen, not a dot.** A resource's id is `<module>.<its own id>`, and a module's name may
// itself contain a dot (`novox.be`), so the module is everything before the *last* dot — which
// only works if what the mesh derives adds no dot of its own.
step["id"] = fmt.Sprint(from["id"]) + "-prepare"
step["name"] = fmt.Sprint(from["name"]) + "-prepare"
step["run-once"] = true
step["args"] = []any{PreparationArgument}
delete(step, "ports")
delete(step, "ip")
delete(step, "schedule")
delete(step, "reload-on")
return step
}
// endpointPortOf is the port the endpoint a route serves listens on: looked up by the name the route
// gives, or read from the port it repeats (novox/hq ADR 0138).
//
// `ports` maps this module's endpoint names to their ports, computed once per module rather than
// re-scanned per contribution.
func endpointPortOf(values map[string]any, ports map[string]int) (int, bool) {
if name, ok := values[RouteEndpoint].(string); ok {
if port, found := ports[strings.TrimSpace(name)]; found {
return port, true
}
}
return asPort(values["port"])
}
// endpointPorts is a module's endpoint names against the ports they listen on.
func endpointPorts(m Manifest) map[string]int {
out := map[string]int{}
for _, l := range m.Listens {
if name := strings.TrimSpace(l.Name); name != "" {
out[name] = l.Port
}
}
return out
}
// portOfEndpoint fills in the port of the endpoint a contribution names, in place.
//
// **A contribution that names an endpoint must still carry that endpoint's port**, because everything
// downstream reads the port: the provider is told where to reach the consumer, and the machine-side
// redirection that turns a declared port into the number the machine published is keyed on it
// (atMachinePort). A route that named only its endpoint left the proxy with no port at all, and a
// proxy with no port has nothing to dial.
//
// Found before it shipped and after the catalogue had already been changed to name endpoints — the
// manifests were merged and the mesh had not yet picked them up, so nothing was broken yet. The
// declared port, not the machine one: the redirection happens later and is keyed on the declared
// number, so filling in the machine port here would be redirected a second time or not at all.
func portOfEndpoint(values map[string]any, ports map[string]int) {
if values == nil {
return
}
if _, already := values["port"]; already {
// A route that says both is its own answer; the older shape repeated the port and is still read.
return
}
name, ok := values[RouteEndpoint].(string)
if !ok {
return
}
if port, found := ports[strings.TrimSpace(name)]; found {
values["port"] = port
}
}
+53 -10
View File
@@ -20,6 +20,12 @@ import (
// declared with the path as the exception it is, and everything else in the module names it by
// id — so moving it later is one line, not a search.
//
// **The mesh's own files for a module are placed too** (novox/hq issue 174). What the mesh writes
// *for* a module — its sealed bus credential, its merged configuration, its bindings — is the
// mesh's plumbing, not the module's data, and sits under `<root>/mesh/<module>`. A directory
// saying `"place": "mesh"` is that place; the definition names the files beneath it by
// `${dir:<id>}` and states no path.
//
// **Resolved here, not on the machine.** The host receives concrete paths exactly as it always
// has; nothing new reaches it and it learns no field. Which also means a resolved path changing
// is a spec change like any other — and the spec comparison must see it (novox/hq issue 126).
@@ -27,6 +33,15 @@ import (
// defaultDataRoot is where module data lands when a node states no root of its own.
const defaultDataRoot = "/var/lib"
// The two places a pathless directory may name, beside its own id.
const (
// placeOwn is the assignment's own root, <root>/<module> — to-be 27's one directory per
// assignment, which every other placed thing of the module sits beneath.
placeOwn = "."
// placeMesh is where the mesh keeps what it writes for the module, <root>/mesh/<module>.
placeMesh = "mesh"
)
// dirRef is how a module names one of its placed directories: ${dir:<id>}.
var dirRef = regexp.MustCompile(`\$\{dir:([a-z0-9][a-z0-9-]*)\}`)
@@ -40,27 +55,54 @@ func dataRoot(with Rendering) string {
// dirsFor is every placed directory of a module, id → the path it resolves to on this node.
//
// A pathless directory saying `"place": "."` is the assignment's own root, <root>/<module> —
// to-be 27's one directory per assignment, which every other placed thing sits beneath. At most
// one makes sense; nothing enforces one, because two ids resolving to one path is a mistake the
// module's own files make visible immediately.
// A pathless directory saying `"place": "."` is the assignment's own root, <root>/<module>; one
// saying `"place": "mesh"` is the mesh's directory for the module, <root>/mesh/<module>; one
// saying neither is <root>/<module>/<id>. At most one of each place makes sense; nothing enforces
// one, because two ids resolving to one path is a mistake the module's own files make visible
// immediately.
//
// A stated path may itself begin with a placed reference — `${dir:mesh-state}/state` — and is
// filled after the directories it can name are resolved; one level, because a directory beneath
// a placed one is the whole of what an adopted layout needs (issue 174's `state` and `out`).
func dirsFor(m Manifest, with Rendering) map[string]string {
dirs := map[string]string{}
var beneath []map[string]any
// The assignment's placement wins over both (novox/hq issue 153). Refused elsewhere when
// malformed; here an invalid setting simply places nothing.
placed, _ := Places(m, with.Settings[m.Module])
for _, r := range m.Resources {
if fmt.Sprint(r["type"]) != "directory" {
continue
}
id := fmt.Sprint(r["id"])
if p, said := placed[id]; said {
dirs[id] = p.Path
continue
}
if path, stated := r["path"].(string); stated && path != "" {
if strings.HasPrefix(path, "${dir:") {
beneath = append(beneath, r)
continue
}
dirs[id] = strings.TrimRight(path, "/")
continue
}
if place, said := r["place"].(string); said && place == "." {
switch place, _ := r["place"].(string); place {
case placeOwn:
dirs[id] = dataRoot(with) + "/" + m.Module
continue
}
case placeMesh:
dirs[id] = dataRoot(with) + "/mesh/" + m.Module
default:
dirs[id] = dataRoot(with) + "/" + m.Module + "/" + id
}
}
for _, r := range beneath {
path := strings.TrimRight(r["path"].(string), "/")
// A reference to no directory is left as written and refused where the resource is
// placed (dirInto), with the message that names what exists.
filled, _ := dirFill(path, dirs, m.Module)
dirs[fmt.Sprint(r["id"])] = filled
}
return dirs
}
@@ -244,10 +286,11 @@ func (m Manifest) unknownDirRefs() []string {
"%s states both path and place on %v — a stated path IS the placement",
m.Module, r["id"]))
}
if place != "." {
if place != placeOwn && place != placeMesh {
problems = append(problems, fmt.Sprintf(
"%s says place %q on %v, and the only place is %q — the assignment's own root",
m.Module, place, r["id"], "."))
"%s says place %q on %v, and the places are %q — the assignment's own root — and "+
"%q — where the mesh keeps what it writes for the module",
m.Module, place, r["id"], placeOwn, placeMesh))
}
}
seen := map[string]bool{}
+75 -2
View File
@@ -216,8 +216,48 @@ func TestPlaceIsValidatedAtTheManifest(t *testing.T) {
wrong := Manifest{Module: "x", Resources: []map[string]any{
{"id": "d", "type": "directory", "place": "sub/dir"},
}}
if got := wrong.unknownDirRefs(); len(got) != 1 || !strings.Contains(got[0], `the only place is "."`) {
t.Fatalf("a place that is not the root refuses; got %v", got)
if got := wrong.unknownDirRefs(); len(got) != 1 || !strings.Contains(got[0], `the places are "."`) {
t.Fatalf("a place that is neither root refuses; got %v", got)
}
}
func TestTheMeshsDirectoryForAModuleIsAPlace(t *testing.T) {
// novox/hq issue 174. What the mesh writes for a module — its bus credential, its bindings —
// is the mesh's plumbing under <root>/mesh/<module>, and the definition names it by id.
m := Manifest{Module: "umami",
Resources: []map[string]any{
{"id": "mesh-state", "type": "directory", "place": "mesh"},
{"id": "state", "type": "directory", "place": "."},
{"id": "server", "type": "container", "image": "x@sha256:aa",
"volumes": []any{"${dir:mesh-state}/broker:/run/secrets/broker:ro"}},
},
OwnSecrets: map[string]string{"broker": "${dir:mesh-state}/broker"},
Binds: map[string]string{"route": "${dir:state}/route.json"},
}
if got := m.unknownDirRefs(); len(got) != 0 {
t.Fatalf("place %q is a place; got %v", "mesh", got)
}
dirs := dirsFor(m, Rendering{})
if dirs["mesh-state"] != "/var/lib/mesh/umami" || dirs["state"] != "/var/lib/umami" {
t.Fatalf("the mesh's directory sits beside the module's, not in it; got %v", dirs)
}
dirs = dirsFor(m, Rendering{DataRoot: "/srv"})
if dirs["mesh-state"] != "/srv/mesh/umami" {
t.Fatalf("a node's root moves the mesh's files with the module's; got %v", dirs)
}
placed, err := placedManifest(m, Rendering{})
if err != nil {
t.Fatal(err)
}
if placed.OwnSecrets["broker"] != "/var/lib/mesh/umami/broker" {
t.Fatalf("own-secrets are placed under the mesh's directory; got %v", placed.OwnSecrets)
}
container := shallowCopy(m.Resources[2])
if err := dirInto(container, dirsFor(m, Rendering{}), m.Module); err != nil {
t.Fatal(err)
}
if container["volumes"].([]any)[0] != "/var/lib/mesh/umami/broker:/run/secrets/broker:ro" {
t.Fatalf("the mount's host side is placed; got %v", container["volumes"])
}
}
@@ -250,3 +290,36 @@ func shallowCopy(resource map[string]any) map[string]any {
}
return copied
}
func TestADirectoryBeneathAPlacedOneIsPlacedWithIt(t *testing.T) {
// An adopted layout keeps a subdirectory the predecessor made under the mesh's directory
// (issue 174: a forge's runtime state, a manager's output). Stated as beneath the placed one,
// it moves with it — a node's root moves both, and the definition names no host path.
m := Manifest{Module: "gitea", Resources: []map[string]any{
{"id": "mesh-state", "type": "directory", "place": "mesh"},
{"id": "runtime-state", "type": "directory", "path": "${dir:mesh-state}/state"},
{"id": "server", "type": "container", "image": "x@sha256:aa",
"volumes": []any{"${dir:runtime-state}:/data"}},
}}
if got := m.unknownDirRefs(); len(got) != 0 {
t.Fatalf("a path beneath a placed directory is well formed; got %v", got)
}
dirs := dirsFor(m, Rendering{DataRoot: "/srv"})
if dirs["runtime-state"] != "/srv/mesh/gitea/state" {
t.Fatalf("the subdirectory follows the placed one; got %v", dirs)
}
sub := shallowCopy(m.Resources[1])
if err := dirInto(sub, dirs, m.Module); err != nil {
t.Fatal(err)
}
if sub["path"] != "/srv/mesh/gitea/state" {
t.Fatalf("the directory resource itself is resolved; got %v", sub["path"])
}
container := shallowCopy(m.Resources[2])
if err := dirInto(container, dirs, m.Module); err != nil {
t.Fatal(err)
}
if container["volumes"].([]any)[0] != "/srv/mesh/gitea/state:/data" {
t.Fatalf("a reference to the subdirectory resolves whole; got %v", container["volumes"])
}
}
+205
View File
@@ -0,0 +1,205 @@
package catalogue
import (
"strings"
"testing"
)
// aMediaServer is the shape one port number per key cannot express: two endpoints of different kinds.
// A web surface a proxy serves under a subdomain, and a protocol port clients dial directly because
// the client expects that number.
func aMediaServer() Manifest {
return Manifest{
Module: "media",
Listens: []Listening{
{Name: "web", Port: 80, From: FromMesh, Why: "the app, behind the proxy"},
{Name: "stream", Port: 32400, From: FromEverywhere, Fixed: true,
Why: "the client dials this number; the protocol chose it"},
},
Contributes: map[string]map[string]any{
"route": {"label": "media", RouteEndpoint: "web"},
},
// Both endpoints are published by its container, which is what lets a machine port be given
// for either: the mesh moves a port the module publishes, never one it merely listens on.
Resources: []map[string]any{
{"id": "server", "type": "container", "name": "media",
"ports": []any{"80", "32400"}},
},
}
}
// **A route names the endpoint it serves.** A route and a listen both carried a port and nothing said
// they were the same thing; now one of them says so.
func TestARouteNamesTheEndpointItServes(t *testing.T) {
m := aMediaServer()
if port, ok := EndpointPort(m, "web"); !ok || port != 80 {
t.Fatalf("the web endpoint resolves to %d (%v), want 80", port, ok)
}
if port, ok := EndpointPort(m, "stream"); !ok || port != 32400 {
t.Fatalf("the stream endpoint resolves to %d (%v), want 32400", port, ok)
}
if _, ok := EndpointPort(m, "absent"); ok {
t.Fatal("an endpoint the module does not declare resolved to a port")
}
}
// And the routed set is read through the name, so the endpoint the proxy serves is known without a
// reader joining two numbers.
func TestTheRoutedEndpointIsFoundByName(t *testing.T) {
routed := RoutedPorts(aMediaServer())
if !routed[80] {
t.Fatalf("the routed endpoint was not found by name: %v", routed)
}
// And the directly-dialled one is not routed, which is what lets its reach govern its port.
if routed[32400] {
t.Fatalf("the endpoint clients dial directly reads as routed: %v", routed)
}
}
// **Two endpoints of different shapes, configured as themselves.** The web endpoint's reach asks for
// names and leaves its port to the proxy; the stream endpoint's reach governs its port, because
// clients dial it and there is no name.
func TestTwoEndpointsOfDifferentShapesAreConfiguredSeparately(t *testing.T) {
m := aMediaServer()
settings := SettingsBy{"media": {{From: "node anchor", Values: map[string]any{
ReachSetting: map[string]any{"80": ReachBoth, "32400": ReachPublic},
}}}}
r := Resolution{Node: "anchor", Modules: []Manifest{m},
PublicDomain: "example.test", At: "anchor.internal"}
rules, err := r.Rules(Rendering{Settings: settings})
if err != nil {
t.Fatal(err)
}
for _, rule := range rules {
switch rule.Port {
case 80:
if rule.From != FromMesh {
t.Fatalf("the routed endpoint's port opened to %q; the proxy is how it is reached",
rule.From)
}
case 32400:
if rule.From != FromEverywhere {
t.Fatalf("the directly-dialled endpoint's port is %q, want anywhere", rule.From)
}
}
}
// And the routed one carries both names, asked for by the same statement.
given, err := r.contributions(settings, nil, nil)
if err != nil {
t.Fatal(err)
}
var public, internal string
for _, c := range given["route"] {
public, _ = c.Values["name"].(string)
internal, _ = c.Values["internal-name"].(string)
}
if public != "media.example.test" || internal != "media.anchor.internal" {
t.Fatalf("names are %q and %q, want both", public, internal)
}
}
// A route naming an endpoint the module does not declare reaches nothing, and is refused where it is
// written rather than resolving to no port and serving nothing.
func TestARouteNamingAnEndpointTheModuleLacksIsRefused(t *testing.T) {
m := aMediaServer()
m.Contributes["route"][RouteEndpoint] = "absent"
got := strings.Join(RouteProblems(m), "\n")
if !strings.Contains(got, "does not declare") {
t.Fatalf("a route naming an absent endpoint was accepted:\n%s", got)
}
}
// **Two endpoints called the same would make an assignment configure whichever was read last.** The
// point of a name is that it identifies one thing.
func TestTwoEndpointsWithOneNameAreRefused(t *testing.T) {
m := Manifest{Module: "twice", Listens: []Listening{
{Name: "web", Port: 80, From: FromMesh},
{Name: "web", Port: 8080, From: FromMesh},
}}
got := strings.Join(endpointNameProblems(m), "\n")
if !strings.Contains(got, "could mean either") {
t.Fatalf("two endpoints with one name were accepted:\n%s", got)
}
}
// A name that is not a name is refused where it is written: it ends up in something a person types.
func TestAnEndpointNameIsHeldToItsShape(t *testing.T) {
for _, wrong := range []string{"Web", "web port", "3000", "-web", "web_surface"} {
m := Manifest{Module: "odd", Listens: []Listening{{Name: wrong, Port: 80, From: FromMesh}}}
if got := strings.Join(endpointNameProblems(m), "\n"); !strings.Contains(got, "a name is lowercase") {
t.Fatalf("%q was accepted as an endpoint name:\n%s", wrong, got)
}
}
}
// **Every endpoint in the catalogue is unnamed today, and must stay valid.** The word ships one
// release before anything uses it.
func TestAnUnnamedEndpointIsStillValid(t *testing.T) {
m := Manifest{Module: "ordinary", Listens: []Listening{{Port: 443, From: FromEverywhere}}}
if got := endpointNameProblems(m); len(got) != 0 {
t.Fatalf("an unnamed endpoint was refused: %v", got)
}
if got := RouteProblems(m); len(got) != 0 {
t.Fatalf("a module with no route was refused: %v", got)
}
}
// **A route that names an endpoint still carries that endpoint's port.**
//
// Everything downstream reads the port: the provider is told where to reach the consumer, and the
// redirection that turns a declared port into the number the machine published is keyed on it. A route
// naming only its endpoint left the proxy with no port, and a proxy with no port has nothing to dial.
//
// Caught after the catalogue had already been changed to name endpoints, and before the mesh picked
// those manifests up — which is the only reason nothing broke.
func TestARouteNamingAnEndpointStillCarriesItsPort(t *testing.T) {
m := aMediaServer()
r := Resolution{Node: "anchor", Modules: []Manifest{m},
PublicDomain: "example.test", At: "anchor.internal"}
given, err := r.contributions(nil, nil, nil)
if err != nil {
t.Fatal(err)
}
var saw bool
for _, c := range given["route"] {
saw = true
port, ok := asPort(c.Values["port"])
if !ok {
t.Fatalf("the route carries no port, so the proxy has nothing to dial: %v", c.Values)
}
if port != 80 {
t.Fatalf("the route carries port %d, want the web endpoint's 80", port)
}
}
if !saw {
t.Fatal("the module contributed no route")
}
}
// And the declared port, not the machine one: the redirection to where the machine published it
// happens later and is keyed on the declared number, so filling the machine port in here would be
// redirected twice or not at all.
func TestTheEndpointsDeclaredPortIsFilledInNotTheMachineOne(t *testing.T) {
m := aMediaServer()
values := map[string]any{RouteEndpoint: "web", "label": "media"}
portOfEndpoint(values, endpointPorts(m))
if got, _ := asPort(values["port"]); got != 80 {
t.Fatalf("filled in port %d, want the declared 80", got)
}
// Then the ordinary redirection puts it where the machine published it.
moved := atMachinePort(values, m.Module, map[string]map[int]int{"media": {80: 20009}})
if got, _ := asPort(moved["port"]); got != 20009 {
t.Fatalf("after redirection the port is %d, want the machine's 20009", got)
}
}
// A route that repeats a port keeps it, because that is the older shape and still read.
func TestARouteThatRepeatsItsPortKeepsIt(t *testing.T) {
values := map[string]any{RouteEndpoint: "web", "port": 8080}
portOfEndpoint(values, map[string]int{"web": 80})
if got, _ := asPort(values["port"]); got != 8080 {
t.Fatalf("the port it stated was overwritten with %d", got)
}
}
@@ -0,0 +1,140 @@
package catalogue
import (
"strings"
"testing"
)
func configured(block map[string]any) SettingsBy {
return SettingsBy{"media": {{From: "node anchor",
Values: map[string]any{EndpointsSetting: block}}}}
}
// **One block per endpoint, saying all three things.** The machine port, the subdomain and the reach
// were `ports`, the route's label and `reach`, each keyed by a port number, so configuring a module
// with two endpoints of different shapes meant knowing which number was which.
func TestAnEndpointsBlockSaysPortLabelAndReach(t *testing.T) {
m := aMediaServer()
// stream's reach NARROWS what the manifest says — the manifest has it from anywhere, the
// assignment says internal. Chosen deliberately: a reach that agrees with the manifest proves
// nothing about whether the block was read at all.
settings := configured(map[string]any{
"web": map[string]any{"port": 20009, "label": "cinema", "reach": ReachBoth},
"stream": map[string]any{"reach": ReachInternal},
})
// The machine port, where the mapping is read.
given, err := GivenPorts(m, settings["media"])
if err != nil {
t.Fatal(err)
}
if given[80] != 20009 {
t.Fatalf("the web endpoint is on machine port %d, want 20009: %v", given[80], given)
}
// The reach, where the filter reads it.
r := Resolution{Node: "anchor", Modules: []Manifest{m},
PublicDomain: "example.test", At: "anchor.internal"}
rules, err := r.Rules(Rendering{Settings: settings})
if err != nil {
t.Fatal(err)
}
for _, rule := range rules {
if rule.Port == 32400 && rule.From != FromMesh {
t.Fatalf("the directly-dialled endpoint is %q; the assignment narrowed it to the private "+
"network and the manifest's 'anywhere' should not win", rule.From)
}
if rule.Port == 80 && rule.From != FromMesh {
t.Fatalf("the routed endpoint's port opened to %q; the proxy is how it is reached", rule.From)
}
}
// And the subdomain, where the name is composed — the assignment's, not the module's.
nodes, err := r.contributions(settings, nil, nil)
if err != nil {
t.Fatal(err)
}
for _, c := range nodes["route"] {
if got, _ := c.Values["name"].(string); got != "cinema.example.test" {
t.Fatalf("the public name is %q, want the label the assignment gave", got)
}
if got, _ := c.Values["internal-name"].(string); got != "cinema.anchor.internal" {
t.Fatalf("the internal name is %q, want the label the assignment gave", got)
}
}
}
// A block that says only a reach leaves the port to the mesh and the label to the module, which is the
// ordinary case and must not require writing the other two.
func TestABlockMaySayOnlyAReach(t *testing.T) {
m := aMediaServer()
settings := configured(map[string]any{"web": map[string]any{"reach": ReachInternal}})
r := Resolution{Node: "anchor", Modules: []Manifest{m},
PublicDomain: "example.test", At: "anchor.internal"}
nodes, err := r.contributions(settings, nil, nil)
if err != nil {
t.Fatal(err)
}
for _, c := range nodes["route"] {
if got, _ := c.Values["name"].(string); got != "" {
t.Fatalf("an internal endpoint composed the public name %q", got)
}
// The module's own label, untouched.
if got, _ := c.Values["internal-name"].(string); got != "media.anchor.internal" {
t.Fatalf("the internal name is %q, want the module's own label", got)
}
}
}
// An endpoint the module does not declare reaches nothing, and the refusal says what it does declare.
func TestConfiguringAnEndpointTheModuleLacksIsRefused(t *testing.T) {
_, err := Endpoints(aMediaServer(), configured(map[string]any{
"admin": map[string]any{"reach": ReachInternal}})["media"])
if err == nil || !strings.Contains(err.Error(), "does not declare") {
t.Fatalf("configuring an absent endpoint was accepted: %v", err)
}
if err != nil && !strings.Contains(err.Error(), "stream") {
t.Fatalf("the refusal does not name what the module declares: %v", err)
}
}
func TestAReachInABlockIsHeldToTheFourValues(t *testing.T) {
_, err := Endpoints(aMediaServer(), configured(map[string]any{
"web": map[string]any{"reach": "mesh"}})["media"])
if err == nil || !strings.Contains(err.Error(), "a reach is") {
t.Fatalf("a filter word was accepted as a reach: %v", err)
}
}
// **Two places giving one endpoint a machine port is the confusion this key exists to end.**
func TestAnEndpointGivenAPortTwiceIsRefused(t *testing.T) {
m := aMediaServer()
_, err := GivenPorts(m, []Layer{{From: "node anchor", Values: map[string]any{
EndpointsSetting: map[string]any{"web": map[string]any{"port": 20009}},
PortsSetting: map[string]any{"80": 30000},
}}})
if err == nil || !strings.Contains(err.Error(), "published once") {
t.Fatalf("an endpoint given two machine ports was accepted: %v", err)
}
}
// And the same for its reach, said once here and once through the older key.
func TestAnEndpointWhoseReachIsAlsoExposedIsRefused(t *testing.T) {
_, err := Endpoints(aMediaServer(), []Layer{{From: "node anchor", Values: map[string]any{
EndpointsSetting: map[string]any{"web": map[string]any{"reach": ReachInternal}},
ExposeSetting: map[string]any{"80": FromEverywhere},
}}})
if err == nil || !strings.Contains(err.Error(), "same thing in different words") {
t.Fatalf("a reach said two ways was accepted: %v", err)
}
}
// A module whose endpoints are unnamed cannot be configured this way, and is told so rather than
// having a block silently reach nothing — which is every module in the catalogue today.
func TestAModuleWithNoNamedEndpointsIsToldSo(t *testing.T) {
m := Manifest{Module: "media", Listens: []Listening{{Port: 80, From: FromMesh}}}
_, err := Endpoints(m, configured(map[string]any{"web": map[string]any{"reach": ReachBoth}})["media"])
if err == nil || !strings.Contains(err.Error(), "no endpoints by name") {
t.Fatalf("a module with no named endpoints accepted a block: %v", err)
}
}
+433 -21
View File
@@ -230,7 +230,23 @@ const SSHPort = 22
// It is a floor for the same reason ssh is. A machine nobody can reach is a machine nobody can
// repair; a machine the mesh cannot reach is a machine the mesh cannot manage. Neither is a thing
// any module asks for, and neither may be derived away.
func AsNftables(rules []Rule, mesh []string, outward bool, foundation []int) string {
func AsNftables(rules []Rule, mesh []string, outward bool, foundation []int,
outwardLinks []string, tunnel string) string {
// The links that are not this machine's own: the ones facing outside, and the mesh's tunnel.
// Traffic arriving on any of them is admitted only by a rule below; traffic arriving anywhere
// else is this machine's own guest and is not something the mesh has a position on.
//
// The tunnel is named here deliberately. Treating it as "not outside" would make a port nothing
// declares reachable from every machine in the mesh, which is the derivation abandoned.
quoted := make([]string, 0, len(outwardLinks)+1)
for _, link := range outwardLinks {
quoted = append(quoted, fmt.Sprintf("%q", link))
}
if tunnel != "" {
quoted = append(quoted, fmt.Sprintf("%q", tunnel))
}
inward := strings.Join(quoted, ", ")
var b strings.Builder
b.WriteString("# Computed by the mesh from what is assigned to this node.\n")
b.WriteString("# Edits are lost on the next declaration; change a module's listens instead.\n\n")
@@ -249,9 +265,45 @@ func AsNftables(rules []Rule, mesh []string, outward bool, foundation []int) str
b.WriteString("\t\tct state established,related accept\n")
b.WriteString("\t\tct state invalid drop\n")
b.WriteString("\t\tiif lo accept\n")
// **Anything on this machine may call anything on this machine.**
//
// Local is not a boundary this mesh draws. A service running here is callable by everything else
// running here, whatever form either takes — a package with a unit, a binary, a container. Whether
// a caller sits in a container was never meant to change the answer, and the only reason it did was
// that this chain asked about addresses: a caller on the machine carries the machine's address, a
// caller in one of its containers carries a bridge address, and a rule naming the former silently
// refused the latter.
//
// Measured: a module reaching its database on this machine's own name timed out for eleven hours
// while the machine itself could reach it, and the mesh called the machine healthy throughout
// (novox/hq 04-ISSUES/145).
//
// Asked by the link it arrives on rather than the address it comes from: anything that did not
// arrive from outside this machine, and did not arrive over the private network, is this machine's
// own. One rule for every service here, in place of a line per port that only ever covered the
// ports somebody remembered to think about.
if inward != "" {
b.WriteString(fmt.Sprintf("\t\tiifname != { %s } accept\n", inward))
}
b.WriteString("\t\ticmp type echo-request accept\n")
b.WriteString("\t\ticmpv6 type { echo-request, nd-neighbor-solicit, nd-neighbor-advert, nd-router-advert } accept\n")
// **What this machine's own guests must be able to ask it** (novox/hq ADR 0140). A guest gets
// its address and its names from this machine, over the link it is on, and those two questions
// arrive at the input chain like any other. Denied, the guest never gets an address and never
// resolves a name — which is not "a closed port" but a network that does not work at all, and it
// is this machine's own guest asking.
//
// Asked for by the link it arrives on rather than by the address it comes from, for the reason
// the forward chain below no longer names an address: a range describes one machine and goes
// stale in silence. Anything arriving from outside, or over the tunnel, is not a guest of this
// machine and asks through a port somebody declared, like everything else.
if len(inward) > 0 {
b.WriteString("\t\t# this machine's own guests asking it for an address and for names\n")
b.WriteString(fmt.Sprintf("\t\tiifname != { %s } udp dport { 53, 67 } accept\n", inward))
b.WriteString(fmt.Sprintf("\t\tiifname != { %s } tcp dport 53 accept\n", inward))
}
// **ssh, always, and not because a module asked.**
//
// Every other line in this chain is derived from what is assigned here, which is the whole
@@ -361,19 +413,36 @@ func AsNftables(rules []Rule, mesh []string, outward bool, foundation []int) str
// about the ports most worth protecting. Rehearsed on three machines: loading these rules
// refused a port on the host and left a published container port reachable (novox/hq issue 047).
//
// The way through is the one the system being replaced already used: deny by default here, and
// then explicitly allow the runtime's own networks, so containers keep working while everything
// else has to be asked for.
// **What it constrains is traffic arriving from OUTSIDE this machine, and nothing else**
// (novox/hq ADR 0140).
//
// It used to deny everything here and then allow the machine's own containers back by naming
// the address ranges they sit on — two ranges fixed in this file and the rest recorded per
// machine. Every way of keeping that list correct failed. A constant describes one machine. A
// recorded range goes stale in silence and cannot tell a network the mesh made from one a
// predecessor left behind. Generating it from the modules would have put half this rule set on
// the machine.
//
// The list should not exist, because the mesh has no position on a container reaching outward:
// that is not a port opened to anybody. So traffic that did not arrive from outside is accepted
// in one line, and what did arrive from outside is allowed only where a rule below admits it.
//
// The tunnel is not "not outside". Accepting everything off it would make a port nothing
// declares reachable from any machine in the mesh, which is the derivation abandoned — so it is
// named here beside the outward links, and traffic arriving on it meets the rules below like
// anything else.
b.WriteString("\tchain forward {\n")
b.WriteString("\t\ttype filter hook forward priority filter; policy drop;\n")
b.WriteString("\t\tct state established,related accept\n")
b.WriteString("\t\tct state invalid drop\n")
b.WriteString("\n")
// What the container runtime created. Without these, denying by default stops every container
// on the machine — which is exactly the failure the absent chain was avoiding, avoided properly.
for _, network := range runtimeNetworks {
b.WriteString(fmt.Sprintf("\t\t# %s\n", network.why))
b.WriteString(fmt.Sprintf("\t\tip saddr %s accept\n", network.cidr))
// Only when there is a link to name. An empty set is a line nftables refuses, and a rule set
// that does not load is a machine filtering nothing while its unit reports success — so the
// chain denies rather than renders nonsense. Composing a declaration for a machine that has
// named none is refused upstream, so this is a floor and not a path anything travels.
if inward != "" {
b.WriteString("\t\t# this machine's own guests reaching outward: not a port opened to anybody\n")
b.WriteString(fmt.Sprintf("\t\tiifname != { %s } accept\n", inward))
}
if len(rules) > 0 {
@@ -430,18 +499,6 @@ func AsNftables(rules []Rule, mesh []string, outward bool, foundation []int) str
return b.String()
}
// runtimeNetworks are the container runtime's own networks, which must keep working when the
// forward chain denies by default.
//
// Taken from what the system being replaced allows, which has been carrying this machine's traffic
// for months: the runtime's bridge range and the range its compose files are given. A machine whose
// runtime is configured with something else needs this to say so — which is a thing the mesh cannot
// derive and a reason this list is named here rather than computed.
var runtimeNetworks = []struct{ cidr, why string }{
{"172.16.0.0/12", "the container runtime's bridge networks"},
{"192.168.128.0/17", "the networks its compose files are given"},
}
// byFamily splits addresses into the two nftables understands separately.
//
// `ip saddr` and `ip6 saddr` are different matches, and one set holding both families is a syntax
@@ -499,6 +556,21 @@ const MeshWideLayer = "the mesh"
// two mappings share a number, it is an entry one of them writes over the other's, and the reader
// that finds the survivor disagrees with the reader that recomputes it.
func GivenPorts(m Manifest, layers []Layer) (map[int]int, error) {
// An endpoint's own block may put it on a machine port, which is the same thing `ports` says about
// the number rather than about the endpoint (novox/hq ADR 0138). Collected first and then let the
// older key be read, which refuses a port said twice.
byName, err := Endpoints(m, layers)
if err != nil {
return nil, err
}
named := map[int]int{}
for name, ep := range byName {
if ep.Port == 0 {
continue
}
named[endpointPorts(m)[name]] = ep.Port
}
// Every name a setting may use, and the mapping it names.
names := map[int][]publishing{}
for _, p := range publishedPorts(m) {
@@ -597,6 +669,17 @@ func GivenPorts(m Manifest, layers []Layer) (map[int]int, error) {
out[key], by[key] = at, port
}
}
// And what the endpoints' own blocks put them on. Refused rather than merged where both keys name
// one endpoint: two places giving a port is the confusion this key exists to end.
for wanted, at := range named {
if was, twice := out[wanted]; twice && was != at {
return nil, fmt.Errorf(
"%s puts its port %d on %d through %s and on %d through %s — one endpoint, two "+
"machine ports, and it is published once. Keep the endpoint's own block",
m.Module, wanted, at, EndpointsSetting, was, PortsSetting)
}
out[wanted] = at
}
if len(out) == 0 {
return nil, nil
}
@@ -664,3 +747,332 @@ func sortedPorts(of map[int]int) []int {
sort.Ints(out)
return out
}
// ReachSetting is the settings key that says how far one of a module's endpoints reaches, per node
// (novox/hq ADR 0138):
//
// {"reach": {"3000": "internal"}}
//
// **One value, three readers.** Reachability used to be settled three times over: the filter read a
// listen's source, which `expose` could override; the proxy composed a public name and an internal
// name for every route it was given, because it could; and the certificate authority followed from
// which names existed. Each was defensible and the combination was unstated, so "this endpoint must
// not be public" could not be written and was therefore enforced by nothing — while a public
// certificate for that very name was obtained anyway.
//
// It keys on the port the module declares, the same key `ports` and `expose` use. A route names that
// port too, which is what lets one statement reach the names as well as the filter: of the 36 route
// entries in the catalogue, 35 name a port that the same module declares a listen on, and the one
// that does not is a path-level refusal — a rule about a name rather than an endpoint.
const ReachSetting = "reach"
// How far an endpoint reaches. Four values, because they have to cover everything `expose` could say
// as well as the two names.
const (
// ReachMachine is this machine only: not the private network, not the world, and no name.
ReachMachine = "machine"
// ReachInternal is the private network, under the internal name and not the public one.
ReachInternal = "internal"
// ReachPublic is the world, under the public name and not the internal one.
ReachPublic = "public"
// ReachBoth is the world, under both names — each certified by its own authority.
//
// The filter cannot distinguish this from ReachPublic, and should not try: the mesh's addresses
// are a subset of anywhere. What differs is the names, which is the whole reason reach is not
// simply the filter's vocabulary with nicer words.
ReachBoth = "both"
)
// reaches is every value, in the order a refusal lists them.
var reaches = []string{ReachMachine, ReachInternal, ReachPublic, ReachBoth}
// RoutedPorts are the ports a module serves through a proxy, taken from its route contributions.
//
// **A routed endpoint's port is how the proxy reaches it, and nothing else.** That is ADR 0045's
// decision and it is older than reach: a public service listens `from: mesh`, only the proxy reaches
// it, and it is exposed by name. So `public` on a routed endpoint asks for a public *name*; opening
// that port to the world as well would undo the arrangement the proxy exists for.
//
// Measured before this was written, not reasoned: a module's routed name answered from the internet
// over TLS while its machine-side port was refused from the same place. The port is not the path.
func RoutedPorts(m Manifest) map[int]bool {
out := map[int]bool{}
note := func(values map[string]any) {
// **The endpoint it serves, by name where it says one.** A route repeating a port number is
// the older shape and still read: 35 of the catalogue's 36 route entries name a port their
// module declares a listen on (novox/hq ADR 0138).
if name, ok := values[RouteEndpoint].(string); ok {
if port, found := EndpointPort(m, name); found {
out[port] = true
return
}
}
if port, ok := asPort(values["port"]); ok {
out[port] = true
}
}
if values, ok := m.Contributes["route"]; ok {
note(values)
}
for _, values := range m.ContributesMany["route"] {
note(values)
}
return out
}
// FilterSource is the source a reach means to the packet filter.
//
// `public` and `both` are the same here. A reach that opened a port to the mesh and not to the world
// would be `internal`; there is no reach that opens it to the world and *not* to the mesh, because a
// filter cannot express "everyone except these" and nobody has asked for it.
func FilterSource(reach string) (string, bool) {
switch reach {
case ReachMachine:
return FromMachine, true
case ReachInternal:
return FromMesh, true
case ReachPublic, ReachBoth:
return FromEverywhere, true
default:
return "", false
}
}
// WantsPublicName is whether a reach asks for the route's public name to be composed.
func WantsPublicName(reach string) bool { return reach == ReachPublic || reach == ReachBoth }
// WantsInternalName is whether a reach asks for the route's internal name to be composed.
func WantsInternalName(reach string) bool { return reach == ReachInternal || reach == ReachBoth }
// Reaches reads a module's per-node reach settings: declared port → how far it reaches.
//
// It refuses a reach for a port the module does not listen on, or a value that is not one of the
// four — the "reads as a restriction and is none" fault this whole mechanism exists to prevent
// (novox/hq ADR 0043/0045). It also refuses a port that `expose` names as well: the two say the same
// thing in different words, and a module whose reach and exposure disagree would have the filter
// following one and the names following the other, which is the very confusion ADR 0138 removes.
//
// A module with no `reach` setting yields nothing, and everything behaves exactly as before: the
// filter follows the manifest's `from`, and both names are composed. That is what keeps every machine
// already running unchanged until an assignment says otherwise.
func Reaches(m Manifest, layers []Layer) (map[int]string, error) {
listened := make(map[int]bool, len(m.Listens))
for _, l := range m.Listens {
listened[l.Port] = true
}
exposed, err := Exposure(m, layers)
if err != nil {
return nil, err
}
out := map[int]string{}
// What an endpoint's own block says, which is the same statement in the shape that names the
// endpoint rather than its port (novox/hq ADR 0138). Read first so the older key, which says less,
// cannot quietly win over the newer one that says more.
blocks, err := Endpoints(m, layers)
if err != nil {
return nil, err
}
declared := endpointPorts(m)
for name, ep := range blocks {
if ep.Reach == "" {
continue
}
out[declared[name]] = ep.Reach
}
for _, layer := range layers {
raw, ok := layer.Values[ReachSetting]
if !ok {
continue
}
entries, ok := raw.(map[string]any)
if !ok {
return nil, fmt.Errorf("%s: %s is a { port: reach } map, and %q set it to something else",
m.Module, ReachSetting, layer.From)
}
for portText, value := range entries {
port, err := strconv.Atoi(portText)
if err != nil {
return nil, fmt.Errorf("%s says how far %q reaches, which is not a port", m.Module, portText)
}
if !listened[port] {
return nil, fmt.Errorf(
"%s says how far port %d reaches, which it does not listen on — the setting "+
"reaches nothing", m.Module, port)
}
reach, ok := value.(string)
if !ok || !slices.Contains(reaches, reach) {
return nil, fmt.Errorf("%s says port %d reaches %v; a reach is %s",
m.Module, port, value, strings.Join(reaches, ", "))
}
if _, both := exposed[port]; both {
return nil, fmt.Errorf(
"%s sets both %s and %s for port %d. They say the same thing in different "+
"words, and the filter would follow one while its names followed the other "+
"— which is what %s exists to stop. Keep %s",
m.Module, ReachSetting, ExposeSetting, port, ReachSetting, ReachSetting)
}
out[port] = reach
}
}
if len(out) == 0 {
return nil, nil
}
return out, nil
}
// RouteEndpoint is the key a route contribution names the endpoint it serves with, instead of
// repeating that endpoint's port (novox/hq ADR 0138).
//
// **A route and a listen both carried a port, and nothing said they were the same thing.** They
// always were — a route serves one of the module's own endpoints — but a reader had to join two
// numbers, and an assignment configuring "the web endpoint" had to know which number that was. A
// route that names the endpoint says what it means, and the mesh looks the port up.
const RouteEndpoint = "endpoint"
// RouteProblems holds a module's route contributions to naming an endpoint it actually has.
//
// A route naming an endpoint the module does not declare reaches nothing, and is refused where it is
// written rather than resolving to no port and serving nothing — the fault this repository names most
// often, a declaration that reads as though it did something.
func RouteProblems(m Manifest) []string {
var problems []string
check := func(where string, values map[string]any) {
name, ok := values[RouteEndpoint].(string)
if !ok || strings.TrimSpace(name) == "" {
return
}
if _, found := EndpointPort(m, name); !found {
problems = append(problems, fmt.Sprintf(
"%s routes %s to the endpoint %q, which it does not declare", m.Module, where, name))
}
}
if values, ok := m.Contributes["route"]; ok {
check("a name", values)
}
for local, values := range m.ContributesMany["route"] {
check(local, values)
}
return problems
}
// EndpointsSetting is the settings key that configures a module's endpoints by name, per node
// (novox/hq ADR 0138):
//
// {"endpoints": {"web": {"port": 20009, "label": "media", "reach": "both"},
// "stream": {"reach": "public"}}}
//
// **One block per endpoint, instead of three keys joined by a number.** Which machine port it lands
// on, the subdomain a proxy serves it under, and how far it reaches are the three things an operator
// says when a module is assigned, and they were said in `ports`, in the route's label and in `reach`,
// each keyed by the port. A module with two endpoints of different shapes — a web surface behind the
// proxy and a protocol port clients dial directly — could only be configured by a reader who knew
// which number was which.
//
// Every field is optional. A block that says only a reach leaves the port to the mesh and the label to
// the module, which is the ordinary case.
const EndpointsSetting = "endpoints"
// Endpoint is what an assignment says about one of a module's endpoints.
type Endpoint struct {
// Port is the machine-side port it is published on. Zero means the mesh assigns one, which it
// does anyway — a fixed port is the module's claim and is honoured without being said here.
Port int
// Label is the subdomain a proxy serves it under, overriding the one the module contributes.
Label string
// Reach is how far it reaches: machine, internal, public or both.
Reach string
}
// Endpoints reads a module's per-node endpoint configuration, by endpoint name.
//
// It refuses a name the module does not declare — the setting would reach nothing — and a reach that
// is not one of the four. It also refuses an endpoint whose port or reach is said twice, once here and
// once through the older key: two places saying the same thing is what this key exists to end, and
// letting both stand would mean the mesh followed whichever it read last.
func Endpoints(m Manifest, layers []Layer) (map[string]Endpoint, error) {
declared := endpointPorts(m)
exposed, err := Exposure(m, layers)
if err != nil {
return nil, err
}
out := map[string]Endpoint{}
for _, layer := range layers {
raw, ok := layer.Values[EndpointsSetting]
if !ok {
continue
}
blocks, ok := raw.(map[string]any)
if !ok {
return nil, fmt.Errorf("%s: %s is a { endpoint: { … } } map, and %q set it to something else",
m.Module, EndpointsSetting, layer.From)
}
for name, body := range blocks {
port, known := declared[name]
if !known {
return nil, fmt.Errorf(
"%s configures the endpoint %q, which it does not declare — the setting reaches "+
"nothing. It declares %s", m.Module, name, spokenEndpoints(m))
}
values, ok := body.(map[string]any)
if !ok {
return nil, fmt.Errorf("%s: the endpoint %q is configured with something that is not a "+
"block of settings", m.Module, name)
}
ep := out[name]
if reach, said := values["reach"]; said {
text, ok := reach.(string)
if !ok || !slices.Contains(reaches, text) {
return nil, fmt.Errorf("%s says the endpoint %q reaches %v; a reach is %s",
m.Module, name, reach, strings.Join(reaches, ", "))
}
if _, also := exposed[port]; also {
return nil, fmt.Errorf(
"%s says how far %q reaches and also exposes port %d. They say the same thing "+
"in different words; keep the endpoint's own block",
m.Module, name, port)
}
ep.Reach = text
}
if at, said := values["port"]; said {
machine, ok := asPort(at)
if !ok {
return nil, fmt.Errorf("%s puts the endpoint %q on %v, which is not a port",
m.Module, name, at)
}
ep.Port = machine
}
if label, said := values["label"]; said {
text, ok := label.(string)
if !ok || strings.TrimSpace(text) == "" {
return nil, fmt.Errorf("%s gives the endpoint %q a label that is not a name: %v",
m.Module, name, label)
}
ep.Label = strings.TrimSpace(text)
}
out[name] = ep
}
}
if len(out) == 0 {
return nil, nil
}
return out, nil
}
// spokenEndpoints is what a module's endpoints are called, as a refusal lists them — so a reader who
// named one wrongly is one edit from right, and a module that has named none is told so.
func spokenEndpoints(m Manifest) string {
names := make([]string, 0, len(m.Listens))
for _, l := range m.Listens {
if name := strings.TrimSpace(l.Name); name != "" {
names = append(names, name)
}
}
if len(names) == 0 {
return "no endpoints by name"
}
sort.Strings(names)
return strings.Join(names, ", ")
}
@@ -19,7 +19,7 @@ func TestTheBrokersPortIsOpenedThoughNoModuleDeclaresIt(t *testing.T) {
// A machine on the private network, with one ordinary module rule, and nothing that mentions
// the broker — which is every machine.
rules := []Rule{{Port: 8080, From: FromMesh, Because: []string{"some-module"}}}
out := AsNftables(rules, []string{"10.42.0.1"}, false, []int{brokerPort})
out := AsNftables(rules, []string{"10.42.0.1"}, false, []int{brokerPort}, nil, "mesh0")
if !strings.Contains(out, "tcp dport 5671 accept") {
t.Fatalf("the broker's port is not opened, so no machine could enrol:\n%s", out)
@@ -48,7 +48,7 @@ func TestTheBrokersPortIsOpenedThoughNoModuleDeclaresIt(t *testing.T) {
// And a mesh that was never told about a broker still gets a ruleset, rather than an empty one or
// a panic. A control plane in that state cannot issue tokens either, which is where it surfaces.
func TestNoBrokerMeansNoFoundationRuleRatherThanNoRuleset(t *testing.T) {
out := AsNftables(nil, []string{"10.42.0.1"}, false, nil)
out := AsNftables(nil, []string{"10.42.0.1"}, false, nil, nil, "mesh0")
if !strings.Contains(out, "table inet mesh") {
t.Fatalf("no ruleset at all:\n%s", out)
}
+206 -21
View File
@@ -79,7 +79,7 @@ func TestTwoModulesWantingOnePortAreBothNamed(t *testing.T) {
t.Fatalf("a module that wanted this port open is not named: %+v", rules[0])
}
// The consequence, which is the reason this matters: removing web must not read as closing 443.
nft := AsNftables(rules, nil, false, nil)
nft := AsNftables(rules, nil, false, nil, nil, "mesh0")
if !strings.Contains(nft, "web") || !strings.Contains(nft, "board") {
t.Fatalf("the rendered rule set does not name both sources:\n%s", nft)
}
@@ -107,7 +107,7 @@ func TestAPortOpenToEveryoneIsNotAlsoRestrictedToTheMesh(t *testing.T) {
func TestWhatNoModuleDeclaredIsClosed(t *testing.T) {
nft := AsNftables(mustFilter(t, Resolution{Modules: []Manifest{
{Module: "web", Listens: []Listening{{Port: 443, From: FromEverywhere}}},
}}, nil), []string{"198.51.100.2"}, false, nil)
}}, nil), []string{"198.51.100.2"}, false, nil, nil, "mesh0")
// Naming the chain, not just the policy: the forward chain drops too, and an assertion on
// "policy drop" alone passes while the input chain accepts everything. It did, once, here.
if !strings.Contains(nft, "type filter hook input priority filter; policy drop;") {
@@ -134,7 +134,7 @@ func TestWhatNoModuleDeclaredIsClosed(t *testing.T) {
// `flush ruleset` would do the first and not the second: it empties every table on the machine,
// including the ones the container runtime writes for its bridges.
func TestReloadingReplacesOnlyTheMeshsOwnRules(t *testing.T) {
nft := AsNftables(nil, nil, false, nil)
nft := AsNftables(nil, nil, false, nil, nil, "mesh0")
if strings.Contains(nft, "flush ruleset") {
t.Fatalf("loading the rule set empties every table on the machine:\n%s", nft)
}
@@ -160,19 +160,119 @@ func TestReloadingReplacesOnlyTheMeshsOwnRules(t *testing.T) {
// So the chain exists and denies by default, and the runtime's own networks are allowed explicitly
// — which is how the system being replaced has been doing it on these machines for months.
func TestWhatIsForwardedIsGovernedToo(t *testing.T) {
nft := AsNftables(nil, []string{"198.51.100.2"}, false, nil)
nft := AsNftables(nil, []string{"198.51.100.2"}, false, nil, nil, "mesh0")
if !strings.Contains(nft, "hook forward priority filter; policy drop") {
t.Fatalf("forwarded traffic is not governed, so container ports are open:\n%s", nft)
}
}
// And containers keep working, which is the whole reason the chain was left out before.
func TestTheRuntimesOwnNetworksKeepWorking(t *testing.T) {
nft := AsNftables(nil, []string{"198.51.100.2"}, false, nil)
for _, network := range []string{"172.16.0.0/12", "192.168.128.0/17"} {
if !strings.Contains(nft, "ip saddr "+network+" accept") {
t.Fatalf("%s is not allowed, so denying by default stops every container:\n%s", network, nft)
// And this machine's own guests keep working, which is the whole reason the chain was left out
// before — by not being mentioned (novox/hq ADR 0140).
//
// It used to be done by naming the address ranges they sit on: two fixed here and the rest recorded
// per machine. That list broke a workstation's containers at a flip and could not be made correct,
// because a range describes one machine and cannot tell a network the mesh made from one a
// predecessor left behind. What replaced it is a single line about the links traffic arrives on.
func TestThisMachinesOwnGuestsKeepWorkingWithoutBeingNamed(t *testing.T) {
nft := AsNftables(nil, []string{"198.51.100.2"}, false, nil, []string{"eth0"}, "mesh0")
if !strings.Contains(nft, `iifname != { "eth0", "mesh0" } accept`) {
t.Fatalf("what did not arrive from outside is not accepted, so this machine's own guests "+
"reach nothing:\n%s", nft)
}
}
// No address of a machine's own networks appears anywhere in a rendered filter.
//
// This is the assertion that fails against the previous behaviour, and it is why it is written on
// the text rather than on an outcome: the two ranges were a constant in this file, so nothing but
// reading the output catches one creeping back in.
func TestNoNetworkOfTheMachinesOwnIsNamed(t *testing.T) {
nft := AsNftables(nil, []string{"198.51.100.2"}, false, nil, []string{"eth0"}, "mesh0")
for _, gone := range []string{"172.16.0.0/12", "192.168.128.0/17", "saddr 192.168", "saddr 172."} {
if strings.Contains(nft, gone) {
t.Fatalf("%q is named, and a range describes one machine and goes stale in silence:\n%s",
gone, nft)
}
}
}
// **The tunnel is constrained, not treated as inside.**
//
// Accepting everything arriving over the private network would make a port nothing declares
// reachable from every machine in the mesh — the derivation abandoned, and a rule that reads as a
// restriction while restricting nothing. So the tunnel is named beside the outward links, and
// traffic arriving on it meets the declared rules like anything else.
func TestTheTunnelIsConstrainedLikeAnOutwardLink(t *testing.T) {
nft := AsNftables(nil, []string{"198.51.100.2"}, false, nil, []string{"eth0"}, "mesh0")
line := `iifname != { "eth0", "mesh0" } accept`
if !strings.Contains(nft, line) {
t.Fatalf("the tunnel is not constrained, so an undeclared port is reachable from any "+
"machine in the mesh:\n%s", nft)
}
}
// A machine with two links facing outside has both constrained. Asserted on the one line, because a
// rule covering one and not the other would leave a machine filtering half of what reaches it.
func TestEveryOutwardLinkIsConstrained(t *testing.T) {
nft := AsNftables(nil, []string{"198.51.100.2"}, false, nil, []string{"eth0", "wlan0"}, "mesh0")
if !strings.Contains(nft, `iifname != { "eth0", "wlan0", "mesh0" } accept`) {
t.Fatalf("not every outward link is constrained:\n%s", nft)
}
}
// A guest asks its host for an address and for names, and those two arrive at the input chain. Asked
// for by the link they arrive on, so a resolver bound anywhere but an outward link keeps answering.
func TestGuestsMayAskTheirHostForAnAddressAndNames(t *testing.T) {
nft := AsNftables(nil, []string{"198.51.100.2"}, false, nil, []string{"eth0"}, "mesh0")
for _, want := range []string{
`iifname != { "eth0", "mesh0" } udp dport { 53, 67 } accept`,
`iifname != { "eth0", "mesh0" } tcp dport 53 accept`,
} {
if !strings.Contains(nft, want) {
t.Fatalf("a guest cannot ask its host for an address or a name, which is not a closed "+
"port but a network that does not work:\n%s", nft)
}
}
}
// With no link named at all the chain denies rather than rendering an empty set, which nftables
// refuses — and a rule set that does not load is a machine filtering nothing while its unit reports
// success. Composing a declaration for such a machine is refused upstream; this is the floor.
func TestNoLinkNamedRendersNoCatchAllRatherThanAnEmptySet(t *testing.T) {
nft := AsNftables(nil, []string{"198.51.100.2"}, false, nil, nil, "")
if strings.Contains(nft, "{ }") || strings.Contains(nft, "iifname != {}") {
t.Fatalf("an empty set is rendered, which nftables refuses:\n%s", nft)
}
if !strings.Contains(nft, "hook forward priority filter; policy drop") {
t.Fatalf("the forward chain does not deny:\n%s", nft)
}
}
// A machine that has not said which links face outside is sent no filter, and the refusal names the
// module that would have loaded it so the reader knows what is being withheld.
func TestAMachineThatNamedNoOutwardLinkIsSentNoFilter(t *testing.T) {
r := Resolution{Node: "anchor", Modules: []Manifest{
{Module: "nftables", Filtering: &Filtering{Into: "/etc/mesh/filter.nft"}},
{Module: "web", Listens: []Listening{{Port: 443, From: FromEverywhere}}},
}}
_, err := r.Declaration(Rendering{Mesh: []string{"198.51.100.2"}, TunnelInterface: "mesh0"})
if err == nil {
t.Fatal("a machine that named no outward link was sent a filter written around none")
}
for _, want := range []string{"anchor", "nftables", "face outside"} {
if !strings.Contains(err.Error(), want) {
t.Fatalf("the refusal does not say %q: %v", want, err)
}
}
}
// And a machine that names none but loads no filter is not refused: there is nothing to write.
func TestAMachineWithNoFilterModuleIsNotRefused(t *testing.T) {
r := Resolution{Node: "anchor", Modules: []Manifest{
{Module: "web", Listens: []Listening{{Port: 443, From: FromEverywhere}}},
}}
if _, err := r.Declaration(Rendering{Mesh: []string{"198.51.100.2"}}); err != nil {
t.Fatalf("a machine that loads no filter was refused one: %v", err)
}
}
@@ -183,7 +283,7 @@ func TestTheRuntimesOwnNetworksKeepWorking(t *testing.T) {
func TestAPublishedPortIsMatchedByWhatWasAskedFor(t *testing.T) {
nft := AsNftables(mustFilter(t, Resolution{Modules: []Manifest{
{Module: "web", Listens: []Listening{{Port: 8080, From: FromEverywhere}}},
}}, nil), []string{"198.51.100.2"}, false, nil)
}}, nil), []string{"198.51.100.2"}, false, nil, nil, "mesh0")
if !strings.Contains(nft, "ct original proto-dst 8080 accept") {
t.Fatalf("the forwarded rule does not match the port a client asked for:\n%s", nft)
}
@@ -193,7 +293,7 @@ func TestAPublishedPortIsMatchedByWhatWasAskedFor(t *testing.T) {
func TestAMeshScopedPortIsMeshScopedWhenForwarded(t *testing.T) {
nft := AsNftables(mustFilter(t, Resolution{Modules: []Manifest{
{Module: "store", Listens: []Listening{{Port: 5432, From: FromMesh}}},
}}, nil), []string{"198.51.100.2"}, false, nil)
}}, nil), []string{"198.51.100.2"}, false, nil, nil, "mesh0")
if !strings.Contains(nft, "ip saddr { 198.51.100.2 } ct original proto-dst 5432 accept") {
t.Fatalf("a mesh-only port is reachable from anywhere once forwarded:\n%s", nft)
}
@@ -203,7 +303,7 @@ func TestAMeshScopedPortIsMeshScopedWhenForwarded(t *testing.T) {
func TestFromTheMeshIsTheNodesTheMeshKnows(t *testing.T) {
nft := AsNftables(mustFilter(t, Resolution{Modules: []Manifest{
{Module: "store", Listens: []Listening{{Port: 5432, From: FromMesh}}},
}}, nil), []string{"198.51.100.2", "198.51.100.3"}, false, nil)
}}, nil), []string{"198.51.100.2", "198.51.100.3"}, false, nil, nil, "mesh0")
if !strings.Contains(nft, "ip saddr { 198.51.100.2, 198.51.100.3 } tcp dport 5432 accept") {
t.Fatalf("a mesh-scoped port was not restricted to the mesh's addresses:\n%s", nft)
}
@@ -213,7 +313,7 @@ func TestFromTheMeshIsTheNodesTheMeshKnows(t *testing.T) {
func TestAMeshPortOnANodeWithNoMeshIsClosedAndSaysSo(t *testing.T) {
nft := AsNftables(mustFilter(t, Resolution{Modules: []Manifest{
{Module: "store", Listens: []Listening{{Port: 5432, From: FromMesh}}},
}}, nil), nil, false, nil)
}}, nil), nil, false, nil, nil, "mesh0")
if strings.Contains(nft, "dport 5432 accept") {
t.Fatalf("a port meant for the mesh was opened to everything:\n%s", nft)
}
@@ -226,7 +326,7 @@ func TestAMeshPortOnANodeWithNoMeshIsClosedAndSaysSo(t *testing.T) {
func TestAMachineScopedPortIsNotOpened(t *testing.T) {
nft := AsNftables(mustFilter(t, Resolution{Modules: []Manifest{
{Module: "cache", Listens: []Listening{{Port: 6379, From: FromMachine}}},
}}, nil), []string{"198.51.100.2"}, false, nil)
}}, nil), []string{"198.51.100.2"}, false, nil, nil, "mesh0")
if strings.Contains(nft, "dport 6379 accept") {
t.Fatalf("a port for this machine only was opened to the network:\n%s", nft)
}
@@ -238,7 +338,8 @@ func TestTheModuleAskingForTheRuleSetGetsEveryModulesPorts(t *testing.T) {
{Module: "firewall", Filtering: &Filtering{Into: "/etc/mesh/filter.nft"}},
{Module: "web", Listens: []Listening{{Port: 443, From: FromEverywhere}}},
}}
out, err := r.Declaration(Rendering{Mesh: []string{"198.51.100.2"}})
out, err := r.Declaration(Rendering{Mesh: []string{"198.51.100.2"},
OutwardLinks: []string{"eth0"}, TunnelInterface: "mesh0"})
if err != nil {
t.Fatalf("declaration: %v", err)
}
@@ -268,7 +369,7 @@ func TestAskingForTheRuleSetWithNowhereToPutItIsRefused(t *testing.T) {
func TestAMeshOnBothAddressFamiliesRendersBoth(t *testing.T) {
nft := AsNftables(mustFilter(t, Resolution{Modules: []Manifest{
{Module: "store", Listens: []Listening{{Port: 5432, From: FromMesh}}},
}}, nil), []string{"198.51.100.2", "2001:db8::2"}, false, nil)
}}, nil), []string{"198.51.100.2", "2001:db8::2"}, false, nil, nil, "mesh0")
if !strings.Contains(nft, "ip saddr { 198.51.100.2 } tcp dport 5432 accept") {
t.Fatalf("the machines with v4 addresses were dropped:\n%s", nft)
}
@@ -296,7 +397,8 @@ func TestWhatTheMeshComputesIsAppliedBeforeWhatTheModuleDeclared(t *testing.T) {
"restart-on": []any{"filtering"}},
},
}}}
out, err := r.Declaration(Rendering{Mesh: []string{"198.51.100.2"}})
out, err := r.Declaration(Rendering{Mesh: []string{"198.51.100.2"},
OutwardLinks: []string{"eth0"}, TunnelInterface: "mesh0"})
if err != nil {
t.Fatalf("declaration: %v", err)
}
@@ -672,7 +774,7 @@ func TestExposureRefusesAPortNotListenedOnAndABadSource(t *testing.T) {
// loading the rules lives on conntrack until it drops, and then the machine is reached from a
// rescue console (novox/hq issue 047).
func TestSSHIsOpenFromTheMeshEvenWhenNothingIsAssigned(t *testing.T) {
nft := AsNftables(nil, []string{"198.51.100.2", "198.51.100.3"}, false, nil)
nft := AsNftables(nil, []string{"198.51.100.2", "198.51.100.3"}, false, nil, nil, "mesh0")
if !strings.Contains(nft, "ip saddr { 198.51.100.2, 198.51.100.3 } tcp dport 22 accept") {
t.Fatalf("ssh is not open to the mesh, so a machine can lock everyone out:\n%s", nft)
}
@@ -685,7 +787,7 @@ func TestSSHIsOpenFromTheMeshEvenWhenNothingIsAssigned(t *testing.T) {
// And from outside as well, on a machine that faces outward — because that is the way in when the
// private network is the thing that broke.
func TestSSHIsOpenFromOutsideOnAMachineThatFacesIt(t *testing.T) {
nft := AsNftables(nil, []string{"198.51.100.2"}, true, nil)
nft := AsNftables(nil, []string{"198.51.100.2"}, true, nil, nil, "mesh0")
if !strings.Contains(nft, "\t\ttcp dport 22 accept") {
t.Fatalf("a machine reachable from outside does not answer ssh there:\n%s", nft)
}
@@ -697,8 +799,91 @@ func TestSSHIsOpenFromOutsideOnAMachineThatFacesIt(t *testing.T) {
// to narrow the rule to, so narrowing it shuts the port entirely — on the first machine anybody
// adopts, reached over the network, closed by the act of adopting it.
func TestSSHIsNeverLeftWithoutARule(t *testing.T) {
nft := AsNftables(nil, nil, false, nil)
nft := AsNftables(nil, nil, false, nil, nil, "mesh0")
if !strings.Contains(nft, "tcp dport 22 accept") {
t.Fatalf("a machine with no mesh addresses has no ssh rule, so adopting it locks it:\n%s", nft)
}
}
// chainBody is one chain's own lines, so an assertion cannot be satisfied by an identical line in
// another chain.
//
// **Written because that happened.** The rule letting this machine's own callers through appears in the
// input chain and, in the same words, in the forward chain. A test asserting on the whole rendered file
// passed with the input chain's copy deleted — it was reading the forward chain's. ADR 0137's own tests
// say to assert per chain body for exactly this reason, and this file was not doing it.
func chainBody(t *testing.T, nft, chain string) string {
t.Helper()
open := "\tchain " + chain + " {"
i := strings.Index(nft, open)
if i < 0 {
t.Fatalf("no chain %q in:\n%s", chain, nft)
}
rest := nft[i+len(open):]
j := strings.Index(rest, "\n\t}")
if j < 0 {
t.Fatalf("chain %q does not close in:\n%s", chain, nft)
}
return rest[:j]
}
// **Anything on this machine may call anything on this machine.**
//
// Local is not a boundary this mesh draws, and whether a caller sits in a container was never meant to
// change the answer. It did, because the chain asked about addresses: a caller on the machine carries
// the machine's address and a caller in one of its containers carries a bridge address, so a rule
// naming the machines' own addresses silently refused every container on them.
//
// Measured: a module reaching its database on its own machine's name timed out for eleven hours while
// the machine itself could reach it (novox/hq 04-ISSUES/145).
func TestAnythingOnThisMachineMayCallAnythingOnIt(t *testing.T) {
nft := AsNftables(mustFilter(t, Resolution{Modules: []Manifest{
{Module: "store", Listens: []Listening{{Port: 5432, From: FromMesh}}},
{Module: "private", Listens: []Listening{{Port: 9999, From: FromMachine}}},
}}, nil), []string{"10.10.0.1", "10.10.0.2"}, false, nil, []string{"eth0"}, "mesh0")
// In the INPUT chain, which is where a call to a service on this machine arrives. The forward
// chain carries the same line in the same words, so asserting on the whole file proves nothing.
input := chainBody(t, nft, "input")
if !strings.Contains(input, `iifname != { "eth0", "mesh0" } accept`) {
t.Fatalf("a caller on this machine cannot reach a service on it:\n%s", input)
}
// One rule, for every service here — not a line per port that only covers the ports somebody
// remembered to think about.
if strings.Contains(input, `iifname != { "eth0", "mesh0" } tcp dport 5432`) {
t.Fatalf("the local allowance is still written per port:\n%s", input)
}
// And the private network still reaches what is exposed to it, which is a different question.
if !strings.Contains(input, "ip saddr { 10.10.0.1, 10.10.0.2 } tcp dport 5432 accept") {
t.Fatalf("the private network no longer reaches a service exposed to it:\n%s", input)
}
}
// The three reaches, as three lines. This is the whole of what the filter says about who may call what.
func TestTheThreeReachesAreThreeLines(t *testing.T) {
nft := AsNftables(mustFilter(t, Resolution{Modules: []Manifest{
{Module: "internal-only", Listens: []Listening{{Port: 5432, From: FromMesh}}},
{Module: "public", Listens: []Listening{{Port: 443, From: FromEverywhere}}},
}}, nil), []string{"10.10.0.1"}, false, nil, []string{"eth0"}, "mesh0")
input := chainBody(t, nft, "input")
for what, want := range map[string]string{
"on this machine": `iifname != { "eth0", "mesh0" } accept`,
"over the private network": "ip saddr { 10.10.0.1 } tcp dport 5432 accept",
"from anywhere": "tcp dport 443 accept",
} {
if !strings.Contains(input, want) {
t.Fatalf("a caller %s cannot reach what is exposed to it (%q):\n%s", what, want, input)
}
}
}
// A port open to everything needs no such line — it is already open to a guest.
func TestAPublicPortNeedsNoGuestLine(t *testing.T) {
nft := AsNftables(mustFilter(t, Resolution{Modules: []Manifest{
{Module: "web", Listens: []Listening{{Port: 443, From: FromEverywhere}}},
}}, nil), []string{"10.10.0.1"}, false, nil, []string{"eth0"}, "mesh0")
if strings.Count(nft, `iifname != { "eth0", "mesh0" } tcp dport 443`) != 0 {
t.Fatalf("a public port was given a guest line it does not need:\n%s", nft)
}
}
+37
View File
@@ -0,0 +1,37 @@
package catalogue
import "testing"
// A bundle is the module's own directory compiled whole, and naming a source would be describing its
// own build. That holds for an interpreted language and cannot hold for a compiled one: a repository
// written in Go carries several commands — the host and its bootstrap live in one — and "the module's
// own directory" is then not a package at all (novox/hq 04-ISSUES/142).
func TestAGoBundleMayNameItsCommand(t *testing.T) {
b := &Build{Artifacts: []Artifact{{
Name: "host-arch", Kind: ArtifactBundle, Language: "go", System: "arch",
From: "cmd/mesh-host",
}}}
if p := b.problems("mesh-host"); len(p) != 0 {
t.Fatalf("a go bundle naming its command was refused: %v", p)
}
}
func TestAnInterpretedBundleStillMayNotNameASource(t *testing.T) {
b := &Build{Artifacts: []Artifact{{
Name: "tools", Kind: ArtifactBundle, Language: "typescript", From: "src",
}}}
p := b.problems("something")
if len(p) == 0 {
t.Fatal("an interpreted bundle naming what it is built from was accepted")
}
}
func TestACompiledBundleStillMustSayItsSystem(t *testing.T) {
b := &Build{Artifacts: []Artifact{{
Name: "host", Kind: ArtifactBundle, Language: "go", From: "cmd/mesh-host",
}}}
if len(b.problems("mesh-host")) == 0 {
t.Fatal("a compiled bundle with no system was accepted")
}
}
+13
View File
@@ -158,3 +158,16 @@ func TestAStoreRowWithoutAProtocolKeepsTheCompiledOne(t *testing.T) {
t.Fatalf("the store's own columns were not kept: %+v", got)
}
}
func TestARefusalWithNothingOnRecordNamesTheHandover(t *testing.T) {
busSeatDelivering(t, "mesh-bus")
elsewhere := []Held{{Claim: "mesh-broker", Scope: ScopeMesh, Node: "anchor", Module: "old-broker"}}
_, problems := checkClaims([]Manifest{newBroker()}, Node{Name: "laptop"}, elsewhere, nil)
if len(problems) != 1 {
t.Fatalf("two derived claimants across machines were not refused: %v", problems)
}
if !strings.Contains(problems[0], "`seat mesh-broker --to anchor/old-broker`") {
t.Fatalf("the refusal does not name the handover that records the holder: %s", problems[0])
}
}
+227
View File
@@ -0,0 +1,227 @@
package catalogue
import (
"encoding/json"
"fmt"
"net"
"regexp"
"sort"
"strings"
)
// A definition names no installation (novox/hq ADR 0112, ADR 0155, issues 122 and 134).
//
// A module definition holds what is true of the module everywhere; what is particular to one mesh —
// a public name, a forge's address, a node's public address — is resolved at assignment. The rule
// stood for a month with nothing checking it, and a sweep found thirty of seventy-one definitions
// naming the installation they were written in. This is the check.
//
// **What is judged is what the mesh acts on, not what a person reads.** A domain in a `why` or a
// `description` is documentation the mesh never reads; reporting it beside `KC_HOSTNAME` would teach
// people to ignore the report. What is judged is every other string value: a name under a public
// top-level domain, or a public address. Two families of name are the world's and not this mesh's,
// and are allowed where they can only mean the world: the public registries an `image` may be pulled
// from, and the public resolvers a machine may forward to. The container runtime's own alias for
// its host is the runtime's, true on every machine that runs it.
//
// **A name that is right where it stands is declared, one by one, with its reason.** A federated
// server's config names the federation's public directory; an application built outside the mesh
// is pulled from the registry that built it, until the mesh builds it. The resource carries
// `names-on-purpose`, a map from each such name to why — the shape `secrets-in-environment` has,
// per name — so a reader sees which names a definition means to carry and why, a name the map does
// not cover is still reported, and the catalogue-wide test is the list that shrinks as names move.
// NamesOnPurpose is the catalogue-level word a resource carries for the names it means to name:
// each name mapped to its reason. The host never sees it.
const NamesOnPurpose = "names-on-purpose"
// prose is every key whose value the mesh never reads.
var prose = map[string]bool{"why": true, "description": true}
// Registries the world runs, which an image may name because an image reference must say where it
// is pulled from. Anything else in an image reference is a registry of some installation.
var worldsRegistries = map[string]bool{
"docker.io": true, "registry-1.docker.io": true, "index.docker.io": true, "ghcr.io": true,
"quay.io": true, "gcr.io": true, "registry.k8s.io": true, "k8s.gcr.io": true,
"mcr.microsoft.com": true, "lscr.io": true, "public.ecr.aws": true, "registry.gitlab.com": true,
"codeberg.org": true, "cgr.dev": true,
}
// Services the world runs that a definition may name as a policy default, the way it may name a
// public resolver: the public certificate authorities' ACME directories. Anything else a served
// fact or a file names is somebody's installation.
var worldsServices = map[string]bool{
"acme-v02.api.letsencrypt.org": true, "acme-staging-v02.api.letsencrypt.org": true,
"api.buypass.com": true, "api.test4.buypass.no": true, "dv.acme-v02.api.pki.goog": true,
"acme.zerossl.com": true,
}
// Resolvers the world runs, which a machine's resolver may forward to as a policy default.
var worldsResolvers = map[string]bool{
"1.1.1.1": true, "1.0.0.1": true, "8.8.8.8": true, "8.8.4.4": true, "9.9.9.9": true,
"149.112.112.112": true, "208.67.222.222": true, "208.67.220.220": true,
}
// hostname is a dotted name whose last label is a top-level domain a real installation would have.
// Not every dotted token: `module.json`, `index.html` and `docker.sock` are dotted and name nothing.
// Boundaries are checked by hand rather than in the pattern, because two names one character apart
// — `a.example.tld,b.example.tld` — would otherwise share the delimiter and the second would be lost.
var hostname = regexp.MustCompile(
`(?i)(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\.)+` +
`(?:be|nl|de|fr|uk|eu|com|net|org|io|dev|app|cloud|site|online|me|co|ch|at|lu|` +
`internal|example|tld|test|invalid)`)
// address is a dotted quad.
var address = regexp.MustCompile(`(?:[0-9]{1,3}\.){3}[0-9]{1,3}`)
// isName is whether a byte may be part of a name; a match bordered by one is a longer token.
func isName(b byte) bool {
return b == '.' || b == '-' || (b >= 'a' && b <= 'z') || (b >= 'A' && b <= 'Z') || (b >= '0' && b <= '9')
}
// standalone are the matches of re in value that are whole tokens, not parts of a longer one.
func standalone(re *regexp.Regexp, value string) []string {
var out []string
for _, span := range re.FindAllStringIndex(value, -1) {
if span[0] > 0 && isName(value[span[0]-1]) {
continue
}
if span[1] < len(value) && isName(value[span[1]]) {
continue
}
out = append(out, value[span[0]:span[1]])
}
return out
}
// InstallationProblems is every value of a definition that names an installation, in the
// definition's own words: where it is, and what it names.
func InstallationProblems(m Manifest) []string {
raw, err := json.Marshal(m)
if err != nil {
return []string{fmt.Sprintf("%s could not be read back: %v", m.Module, err)}
}
var tree any
if err := json.Unmarshal(raw, &tree); err != nil {
return []string{fmt.Sprintf("%s could not be read back: %v", m.Module, err)}
}
var problems []string
// The module's own name is a value too: a module named after the domain it serves is a
// definition that can only be installed there (issue 134).
for _, name := range namesIn(m.Module) {
problems = append(problems, fmt.Sprintf(
"%s is named after %s, and a module is named for what it is, not for where it runs", m.Module, name))
}
walk(tree, "", nil, func(at string, value string, meant map[string]bool, isImage bool) {
for _, name := range namesIn(value) {
if (isImage && worldsRegistries[strings.ToLower(name)]) || meant[name] {
continue
}
problems = append(problems, fmt.Sprintf("%s names %s at %s", m.Module, name, at))
}
for _, ip := range addressesIn(value) {
if meant[ip] {
continue
}
problems = append(problems, fmt.Sprintf("%s names the public address %s at %s", m.Module, ip, at))
}
})
sort.Strings(problems)
return problems
}
// walk visits every string in the tree with its path, the names the enclosing resource means to
// name (with a reason), and whether it is an image reference.
func walk(node any, at string, meant map[string]bool, visit func(at, value string, meant map[string]bool, isImage bool)) {
switch v := node.(type) {
case map[string]any:
if declared, has := v[NamesOnPurpose].(map[string]any); has {
widened := map[string]bool{}
for name := range meant {
widened[name] = true
}
for name, reason := range declared {
if r, ok := reason.(string); ok && strings.TrimSpace(r) != "" {
widened[strings.ToLower(name)] = true
}
}
meant = widened
}
keys := make([]string, 0, len(v))
for k := range v {
keys = append(keys, k)
}
sort.Strings(keys)
for _, k := range keys {
if prose[k] || k == NamesOnPurpose || (at == "" && k == "module") {
continue
}
child := at + "." + k
if at == "" {
child = k
}
if s, isString := v[k].(string); isString {
visit(child, s, meant, k == "image")
continue
}
walk(v[k], child, meant, visit)
}
case []any:
for i, item := range v {
child := fmt.Sprintf("%s[%d]", at, i)
if s, isString := item.(string); isString {
visit(child, s, meant, false)
continue
}
walk(item, child, meant, visit)
}
}
}
// namesIn is every hostname in a value that could belong to an installation.
func namesIn(value string) []string {
var out []string
for _, found := range standalone(hostname, value) {
name := strings.ToLower(found)
switch {
case strings.HasSuffix(name, ".docker.internal"):
// The container runtime's alias for its own host: every machine running it has one.
case worldsServices[name]:
// A public authority named as a policy default, true of any mesh that wants it.
case name == "example.tld", strings.HasSuffix(name, ".example.tld"),
name == "example.com", name == "example.net", name == "example.org",
strings.HasSuffix(name, ".example.com"), strings.HasSuffix(name, ".example.net"),
strings.HasSuffix(name, ".example.org"), strings.HasSuffix(name, ".example"),
strings.HasSuffix(name, ".test"), strings.HasSuffix(name, ".invalid"):
// Documentation names, which is what a definition's own example should use.
default:
out = append(out, name)
}
}
return out
}
// addressesIn is every public address in a value: not a private range, loopback, link-local, the
// unspecified address, a documentation range, or a resolver the world runs.
func addressesIn(value string) []string {
var out []string
for _, found := range standalone(address, value) {
ip := net.ParseIP(found)
if ip == nil || ip.IsPrivate() || ip.IsLoopback() || ip.IsLinkLocalUnicast() ||
ip.IsUnspecified() || ip.IsMulticast() || worldsResolvers[found] || documentation(ip) {
continue
}
out = append(out, found)
}
return out
}
func documentation(ip net.IP) bool {
for _, cidr := range []string{"192.0.2.0/24", "198.51.100.0/24", "203.0.113.0/24", "100.64.0.0/10"} {
_, block, _ := net.ParseCIDR(cidr)
if block.Contains(ip) {
return true
}
}
return false
}
+93
View File
@@ -0,0 +1,93 @@
package catalogue
import (
"strings"
"testing"
)
// A definition names no installation (novox/hq ADR 0112, ADR 0155). What the mesh acts on is judged;
// prose is not; the world's registries and resolvers are the world's; a declared exception is a
// reason a reader sees.
func TestADefinitionNamingAnInstallationIsNamedBack(t *testing.T) {
m := Manifest{Module: "idp", Resources: []map[string]any{
{"id": "server", "type": "container", "image": "quay.io/keycloak/keycloak@sha256:aa",
"env": map[string]any{"KC_HOSTNAME": "https://login.mesh-one.be"}},
{"id": "env", "type": "file", "content": "REAL_IP_FROM=192.168.1.0/24,127.0.0.0/8,203.0.113.7,51.15.22.9\n"},
}, Listens: []Listening{{Port: 8080, From: FromMesh, Why: "the login page; login.mesh-one.be is a route grant"}}}
got := strings.Join(InstallationProblems(m), "\n")
for _, want := range []string{
"idp names login.mesh-one.be at resources[0].env.KC_HOSTNAME",
"idp names the public address 51.15.22.9 at resources[1].content",
} {
if !strings.Contains(got, want) {
t.Errorf("missing %q in:\n%s", want, got)
}
}
for _, mustNot := range []string{"quay.io", "why", "203.0.113.7", "192.168.1.0", "127.0.0.0"} {
if strings.Contains(got, mustNot) {
t.Errorf("%q was reported and should not be:\n%s", mustNot, got)
}
}
}
func TestTheWorldsNamesAreNotAnInstallations(t *testing.T) {
m := Manifest{Module: "resolver", Resources: []map[string]any{
{"id": "conf", "type": "file", "content": "server=1.1.1.1\nserver=8.8.8.8\nlisten=127.0.0.55\n"},
{"id": "proxy", "type": "container", "image": "docker.io/library/traefik@sha256:bb"},
{"id": "adapter", "type": "file", "content": "{\"machine\": \"host.docker.internal\"}\n"},
{"id": "doc", "type": "file", "content": "root = https://git.example.tld/\n"},
}}
if got := InstallationProblems(m); len(got) != 0 {
t.Fatalf("the world's names were reported: %v", got)
}
}
func TestANameMeantOnPurposeIsDeclaredWithItsReason(t *testing.T) {
// The federation's public directory in a homeserver's config: the world's, said so, and a name
// the map does not cover is still reported.
m := Manifest{Module: "homeserver", Resources: []map[string]any{
{"id": "conf", "type": "file", "content": "trusted_key_servers: matrix.org\nwell_known: https://mesh-one.be\n",
NamesOnPurpose: map[string]any{"matrix.org": "the federation's public key server, the world's"}},
}}
got := InstallationProblems(m)
if len(got) != 1 || !strings.Contains(got[0], "mesh-one.be") {
t.Fatalf("got %v", got)
}
}
func TestAnImageFromAnInstallationsRegistryNeedsAReason(t *testing.T) {
bare := Manifest{Module: "site", Resources: []map[string]any{
{"id": "server", "type": "container", "image": "registry.mesh-one.be/org/site@sha256:cc"},
}}
if got := InstallationProblems(bare); len(got) != 1 || !strings.Contains(got[0], "registry.mesh-one.be") {
t.Fatalf("an image on an installation's registry was not named: %v", got)
}
excepted := Manifest{Module: "site", Resources: []map[string]any{
{"id": "server", "type": "container", "image": "registry.mesh-one.be/org/site@sha256:cc",
NamesOnPurpose: map[string]any{"registry.mesh-one.be": "built outside the mesh until the site's repository is a build source here"}},
}}
if got := InstallationProblems(excepted); len(got) != 0 {
t.Fatalf("a declared exception was still reported: %v", got)
}
}
func TestAModuleNamedAfterADomainIsNamedBack(t *testing.T) {
got := InstallationProblems(Manifest{Module: "mesh-one.be"})
if len(got) != 1 || !strings.Contains(got[0], "named after mesh-one.be") {
t.Fatalf("got %v", got)
}
}
func TestABuildContextOnASeatNamesNoForge(t *testing.T) {
m := Manifest{Module: "packager", Build: &Build{Artifacts: []Artifact{
{Name: "server", Kind: "image", From: "Dockerfile",
Context: &ArtifactContext{Seat: "git", Repository: "org/controller", Ref: "main"}},
}}}
if got := InstallationProblems(m); len(got) != 0 {
t.Fatalf("a context on a seat was reported: %v", got)
}
m.Build.Artifacts[0].Context = &ArtifactContext{Repository: "https://git.mesh-one.be/org/controller.git"}
if got := InstallationProblems(m); len(got) != 1 {
t.Fatalf("a context by URL was not reported: %v", got)
}
}
+31
View File
@@ -0,0 +1,31 @@
package catalogue
import (
"strings"
"testing"
)
// A manifest may say which tools its module calls (novox/hq ADR 0152), and the parser accepts the
// two shapes the grant has: a named tool, and every tool.
func TestAManifestMaySayWhatItInvokes(t *testing.T) {
m, err := ParseManifest([]byte(`{"module":"mesh-console","version":"1",` +
`"invokes":["mesh-catalog.catalog_modules","*"]}`))
if err != nil {
t.Fatal(err)
}
if len(m.Invokes) != 2 || m.Invokes[1] != "*" {
t.Fatalf("invokes not read: %v", m.Invokes)
}
}
// An entry that names a module and no tool is refused at parse, in the manifest's words, rather than
// at the composition of the bus's user list where it would stop the file for everybody.
func TestAnInvokeThatNamesNoToolIsRefusedAtParse(t *testing.T) {
_, err := ParseManifest([]byte(`{"module":"mesh-console","version":"1","invokes":["shop"]}`))
if err == nil {
t.Fatal("an invoke naming no tool was accepted")
}
if !strings.Contains(err.Error(), `invokes "shop", which does not name a tool`) {
t.Fatalf("refused for the wrong reason: %v", err)
}
}
+181 -5
View File
@@ -42,6 +42,11 @@ var renamed = map[string]string{
var name = regexp.MustCompile(`^[a-z0-9][a-z0-9-]*(\.[a-z0-9][a-z0-9-]*)*$`)
// toolName is what a module calls one of its tools: the sdk's tools are `catalog_modules` and
// `gitea_list_repos`, so an underscore is ordinary here and a dot is not — the dot is what separates
// the module from the tool in `<module>.<tool>`, and a tool name carrying one would be two grants.
var toolName = regexp.MustCompile(`^[a-z0-9][a-z0-9_-]*$`)
// Claim is a singular resource a module takes over.
type Claim struct {
Name string `json:"name"`
@@ -86,8 +91,13 @@ const (
// If the path is absent when a machine applies, the host refuses clearly rather than creating it:
// the mesh does not own it, so conjuring it would be a lie the host then acts on.
type Access struct {
// Path is the absolute path on the machine, as the operator provides it.
Path string `json:"path"`
// ID is the name the module gives this access, which the assignment places
// (`accesses: {<id>: <path>}`, novox/hq ADR 0112, issue 153) and the module's mounts name as
// ${access:<id>}. The shape a definition should use: it names no path of any machine.
ID string `json:"id,omitempty"`
// Path is the absolute path on the machine. A definition carrying one names an installation;
// tolerated as the default the assignment may replace, for accesses declared before ids.
Path string `json:"path,omitempty"`
// Mode is "read" or "read-write". Absent narrows to read.
Mode string `json:"mode,omitempty"`
}
@@ -225,6 +235,19 @@ type Manifest struct {
// subscription to the queue it writes to (design 29 §2).
Uses []string `json:"uses,omitempty"`
// Prepares says this module has state that must be brought to the shape this version needs
// before this version runs, and that the module's own code does it (novox/hq ADR 0135).
//
// **A word, not an arrangement.** The mesh runs the module's own program in its preparation
// mode, in the module's own context — every binding, credential and setting its code receives,
// because it *is* its code. Nothing here names a container, a command, a mount or a variable:
// the module already said all of that once, and a second copy is a second thing to drift.
//
// **Declared, never inferred.** The control plane cannot read what is inside an artifact, so a
// module that ships a migration and does not say this breaks on its first upgrade. That is
// stated in the record rather than guarded here, because nothing mechanical can guard it.
Prepares bool `json:"prepares,omitempty"`
// Tools are the tools this module answers — request and reply, awaited.
//
// **New, and not `serves`**, which this manifest already uses for the facts a consumer needs
@@ -234,6 +257,16 @@ type Manifest struct {
// module claiming a seat answers what that seat's protocol promises (novox/hq ADR 0118).
Tools []string `json:"tools,omitempty"`
// Invokes are the tools this module calls, each `<module>.<tool>` or a role's `seat:<seat>.<verb>`,
// or the single entry `*` for every tool on the mesh (novox/hq ADR 0152, ADR 0154).
//
// **A grant, and only a grant.** The bus lets this module publish exactly those tool subjects
// and nothing beside them — no event, no subscription, no seat. A module that declares none
// calls nothing, which is every module but the console today. ADR 0095 made the control plane
// the one caller and deferred this until a consumer asked; the console is that consumer, and a
// person's account (design 25 §7) already had the same shape.
Invokes []string `json:"invokes,omitempty"`
// Capabilities the machine must have. A different field from Requires because the remedy
// differs: a missing module can be assigned, and a missing capability means the wrong
// machine.
@@ -521,8 +554,14 @@ type BuildsOn struct {
type ArtifactContext struct {
// Repository is cloned fresh, the same way the module's own repository is — a working tree
// nothing has touched, so what was built is reproducible from the two commits named rather
// than from whatever a previous build happened to leave behind.
// than from whatever a previous build happened to leave behind. A URL, or — with Seat — a
// path on that seat's holder, `<owner>/<name>`.
Repository string `json:"repository"`
// Seat is the seat the repository lives on: `git` for this mesh's own forge (novox/hq ADR 0111,
// ADR 0155). A context written as a URL names one installation's forge and can be built
// nowhere else; a path on the seat is composed by the mesh that builds it, whichever forge
// holds the seat there.
Seat string `json:"seat,omitempty"`
// Ref is the branch, tag or commit of that repository to build. Empty means its own default
// branch — the same meaning an empty module ref already has.
Ref string `json:"ref,omitempty"`
@@ -558,6 +597,21 @@ type Artifact struct {
// image built from this same module's own repository, the same as every other artifact.
Context *ArtifactContext `json:"context,omitempty"`
// System is the operating system this artifact is compiled for, for a bundle whose output is a
// binary rather than portable code (novox/hq ADR 0142).
//
// **Named by the artifact, not by the recipe.** A toolchain deliberately accepts nothing from
// the module — anything a module could override there it would be writing a Dockerfile to
// override — and yet a compiled binary is per operating system, pinned at link time so a host
// refuses to touch a machine it was not built for (novox/hq ADR 0005). The way out is that the
// target is a property of the artifact: one artifact declared per system, one build each, and
// the recipe stays the mesh's.
//
// Empty for a bundle whose output runs anywhere, which is every interpreted language, and for
// every other kind. A bundle in a language that compiles to a binary must say one, because
// "compiled for whatever the build machine happened to be" is the fault this exists to prevent.
System string `json:"system,omitempty"`
// Language is what this module's code is written in, for a bundle.
//
// **Declared, never guessed.** Inferring it from what files happen to be present makes a
@@ -569,6 +623,16 @@ type Artifact struct {
// Empty for every other kind, which do not compile.
Language string `json:"language,omitempty"`
// Binary is what the compiled executable is called, for a bundle in a language that compiles to
// one. Empty means the package's own name, which is what a compiler does by default.
//
// **Because the name a machine runs it by is not always the name of the package that built it.**
// The host's command is `cmd/mesh-host` and every machine runs it as `nox-mesh-host` — the path
// it is installed at, the name in its unit, and the name its launcher looks for inside a
// delivered version. A bundle that carried the package's name was delivered correctly, reported
// success, and was invisible to the launcher (novox/hq 04-ISSUES/142).
Binary string `json:"binary,omitempty"`
// Entrypoints are the compiled files a tool host should load from this module, relative to the
// bundle's root.
//
@@ -629,8 +693,23 @@ const (
// on is a fact, and it should be written once.
const ArtifactStoreProvision = "artifact-store"
// Listening is one port a module accepts connections on.
// Listening is one endpoint a module serves: a port it accepts connections on, and what may be said
// about that port from outside the module.
type Listening struct {
// Name is what this endpoint is called, so an assignment and a route can refer to it as one thing
// (novox/hq ADR 0138).
//
// **Because a port number is not a name.** Three facts have to be said about an endpoint when a
// module is assigned — which machine port it lands on, the subdomain a proxy serves it under, and
// how far it reaches — and they were said in three places keyed by the port. A module with two
// endpoints of different shapes, a web surface behind a proxy and a protocol port clients dial
// directly, cannot be configured that way without a reader joining numbers by hand.
//
// The module's to choose, like the route's label: it names its own parts. Lowercase, and unique
// within the module, so a reference to it is unambiguous. Empty is allowed and means an endpoint
// nothing refers to by name, which is every endpoint in the catalogue until they are named.
Name string `json:"name,omitempty"`
Port int `json:"port"`
// Protocol is "tcp" or "udp". Absent means tcp, which is what almost everything is — and a
// field that had to be written every time would be written wrongly some of the time.
@@ -1237,6 +1316,9 @@ func ParseManifest(raw []byte) (Manifest, error) {
"%s listens on %d over %q, which is tcp or udp", m.Module, l.Port, p))
}
}
problems = append(problems, invokeProblems(m)...)
problems = append(problems, endpointNameProblems(m)...)
problems = append(problems, RouteProblems(m)...)
for _, port := range m.Guards {
if port < 1 || port > 65535 {
problems = append(problems, fmt.Sprintf(
@@ -1276,6 +1358,17 @@ func ParseManifest(raw []byte) (Manifest, error) {
"program that reads what the mesh delivered and reconciles",
m.Module, r["id"]))
}
// **A module that prepares its state must have code the mesh can run** (novox/hq ADR 0135). The
// preparation is the module's own program in its preparation mode, so it is derived from the
// resource that runs that program — and a module declaring none has asked for something the mesh
// cannot compose. Said here, where the manifest is read, rather than by a declaration that
// quietly prepares nothing.
if m.Prepares && preparationTarget(m) == "" {
problems = append(problems, fmt.Sprintf(
"%s says it prepares its state, and declares no container running an artifact it built — "+
"the preparation is this module's own program, so there has to be one for the mesh to "+
"run it in", m.Module))
}
// **A run-once container is a step the host runs to completion** (novox/hq ADR 0052). It is a
// boolean modifier on the container shape — the host runs the container, requires it to exit 0,
// and starts whatever the declaration places after it only once it has. A value that is not a
@@ -1437,7 +1530,16 @@ func ParseManifest(raw []byte) (Manifest, error) {
}
}
for _, a := range m.Accesses {
if !strings.HasPrefix(a.Path, "/") {
if a.ID == "" && a.Path == "" {
problems = append(problems, fmt.Sprintf(
"%s declares an access with neither an id nor a path — an id, which the assignment places",
m.Module))
}
if a.ID != "" && !accessRef.MatchString("${access:"+a.ID+"}") {
problems = append(problems, fmt.Sprintf(
"%s accesses %q; an access id is lowercase letters, digits and dashes", m.Module, a.ID))
}
if a.Path != "" && !strings.HasPrefix(a.Path, "/") {
problems = append(problems, fmt.Sprintf(
"%s accesses %q, which is not an absolute path", m.Module, a.Path))
}
@@ -1469,6 +1571,7 @@ func ParseManifest(raw []byte) (Manifest, error) {
// the time it sees the mount it is being asked to create the directory, which it can do.
problems = append(problems, m.undeclaredMounts()...)
problems = append(problems, m.unknownDirRefs()...)
problems = append(problems, m.unknownAccessRefs()...)
for i, r := range m.Resources {
id, _ := r["id"].(string)
@@ -1650,3 +1753,76 @@ func (m Manifest) undeclaredMounts() []string {
}
return problems
}
// endpointName is what an endpoint may be called: lowercase letters, digits and dashes, starting
// with a letter. The same shape a label has, because both end up in something a person types.
var endpointName = regexp.MustCompile(`^[a-z][a-z0-9-]*$`)
// endpointNameProblems holds a module's endpoint names to being usable as references (novox/hq ADR
// 0138).
//
// **Unique, because the point of a name is that it identifies one thing.** Two endpoints called the
// same would make an assignment that configures one silently configure whichever the mesh read last
// — the shape of fault this repository keeps finding, where a declaration appears to say something
// and says something else.
func endpointNameProblems(m Manifest) []string {
var problems []string
seen := map[string]int{}
for _, l := range m.Listens {
name := strings.TrimSpace(l.Name)
if name == "" {
continue
}
if !endpointName.MatchString(name) {
problems = append(problems, fmt.Sprintf(
"%s calls the endpoint on port %d %q; a name is lowercase letters, digits and "+
"dashes, starting with a letter", m.Module, l.Port, l.Name))
continue
}
if before, already := seen[name]; already {
problems = append(problems, fmt.Sprintf(
"%s calls both port %d and port %d %q, so anything naming that endpoint could mean "+
"either", m.Module, before, l.Port, name))
continue
}
seen[name] = l.Port
}
return problems
}
// EndpointPort is the port of the endpoint a module calls this, and whether it has one.
func EndpointPort(m Manifest, name string) (int, bool) {
want := strings.TrimSpace(name)
if want == "" {
return 0, false
}
for _, l := range m.Listens {
if strings.TrimSpace(l.Name) == want {
return l.Port, true
}
}
return 0, false
}
// invokeProblems judges what a module says it calls (novox/hq ADR 0152).
//
// Refused here, in the manifest's words, rather than at the next composition of the bus's user
// list — where a bad entry would stop the whole file being written for everybody, as a person's
// malformed grant would have (operator.go). An entry that names a module and no tool is the one
// mistake worth naming: `shop` reads like a grant to a module's tools and would be a grant to nothing.
func invokeProblems(m Manifest) []string {
var problems []string
for _, t := range m.Invokes {
if t == "*" {
continue
}
t = strings.TrimPrefix(t, "seat:")
module, tool, named := strings.Cut(t, ".")
if !named || !name.MatchString(module) || !toolName.MatchString(tool) {
problems = append(problems, fmt.Sprintf(
"%s invokes %q, which does not name a tool: a module invokes <module>.<tool>, or "+
"* for every tool on the mesh (novox/hq ADR 0152)", m.Module, t))
}
}
return problems
}
+33 -80
View File
@@ -1,6 +1,7 @@
package catalogue
import (
"reflect"
"strings"
"testing"
)
@@ -30,36 +31,38 @@ func namesOf(r map[string]any) []string {
return out
}
// A container does not inherit the machine's names, so the mesh gives them to it.
// No mesh name is written into a container (novox/hq ADR 0148). It resolves them through its
// machine's resolver at the moment it asks, so a name that moves is answered differently by the
// next lookup, in every container, with nothing recreated.
//
// It gets its own hosts file holding only its own hostname — every internal name the mesh wrote
// for the machine is invisible to what the machine runs. A database client on one node could not
// resolve another node, on a mesh where both names were correct and present on both machines.
func TestEveryContainerIsGivenTheMeshsNames(t *testing.T) {
got := containersOf(t, Resolution{Node: "laptop", Modules: []Manifest{{
Module: "app",
Resources: []map[string]any{{"id": "web", "type": "container", "name": "web",
"image": "registry.example/web@sha256:" + strings.Repeat("a", 64)}},
}}}, Rendering{Names: map[string]string{
"anchor.internal": "10.42.0.1", "laptop.internal": "10.42.0.2",
// Checked the way the record says: the declaration a container gets does not move when the mesh's
// roster does. A roster with one machine and a roster with three produce the same container, byte
// for byte, so the digest a host computes from it cannot move either — which is what stopped one
// name moving from replacing every container in the mesh (issue 151).
func TestAContainerIsTheSameWhateverTheMeshsRosterSays(t *testing.T) {
module := Manifest{Module: "app", Resources: []map[string]any{{"id": "web", "type": "container",
"name": "web", "image": "registry.example/web@sha256:" + strings.Repeat("a", 64)}}}
one := containersOf(t, Resolution{Node: "laptop", Modules: []Manifest{module}},
Rendering{Names: map[string]string{"laptop.internal": "10.42.0.2"}})
three := containersOf(t, Resolution{Node: "laptop", Modules: []Manifest{module}},
Rendering{Names: map[string]string{
"anchor.internal": "10.42.0.1", "laptop.internal": "10.42.0.2", "git.example.tld": "10.42.0.1",
}})
if len(got) != 1 {
t.Fatalf("expected one container, got %d", len(got))
if len(one) != 1 || len(three) != 1 {
t.Fatalf("expected one container each, got %d and %d", len(one), len(three))
}
given := namesOf(got[0])
if len(given) != 2 {
t.Fatalf("the container was given %d name(s): %v", len(given), given)
if given := namesOf(three[0]); len(given) != 0 {
t.Fatalf("the mesh's names were copied into the container: %v", given)
}
if given[0] != "anchor.internal:10.42.0.1" {
t.Fatalf("the name is not in the form a runtime writes: %v", given)
if !reflect.DeepEqual(one[0], three[0]) {
t.Fatalf("the container moved with the roster:\n%v\n%v", one[0], three[0])
}
}
// A container that named its own keeps them and gets the mesh's beside them.
//
// The mesh does not know what else a workload needs to reach, and taking something away in order
// to add something is not what "also" means.
func TestAContainersOwnNamesAreKept(t *testing.T) {
// The names a module declares for itself are its own: part of what the module is, kept exactly as
// written, and the mesh does not know what they mean. They are the one thing in a container's
// hosts that does move its identity, because they do not move when the mesh's roster does.
func TestAContainersOwnNamesAreKeptAsWritten(t *testing.T) {
got := containersOf(t, Resolution{Node: "laptop", Modules: []Manifest{{
Module: "app",
Resources: []map[string]any{{"id": "web", "type": "container", "name": "web",
@@ -68,62 +71,15 @@ func TestAContainersOwnNamesAreKept(t *testing.T) {
}}}, Rendering{Names: map[string]string{"anchor.internal": "10.42.0.1"}})
given := namesOf(got[0])
if len(given) != 2 || given[0] != "something.else:203.0.113.9" {
t.Fatalf("the container's own names were lost: %v", given)
if len(given) != 1 || given[0] != "something.else:203.0.113.9" {
t.Fatalf("the container's own names were not kept as written: %v", given)
}
}
// A container on the machine's own network gets the names too — it does NOT share the machine's
// hosts file. `docker run --network host` still gives the container its own /etc/hosts (localhost
// and its own id only), so every `<node>.internal` name the mesh wrote is invisible inside it, and a
// client that dials one gets EAI_AGAIN. It gets the same `--add-host` entries every other container
// gets (the runtime accepts them with `--network host`), so a host-network consumer can reach a
// provider by the `.internal` address the mesh hands it.
func TestAContainerOnTheMachinesNetworkIsGivenTheNamesToo(t *testing.T) {
got := containersOf(t, Resolution{Node: "anchor", Modules: []Manifest{{
Module: "control",
Resources: []map[string]any{{"id": "c", "type": "container", "name": "c",
"image": "registry.example/c@sha256:" + strings.Repeat("a", 64), "network": "host"}},
}}}, Rendering{Names: map[string]string{"anchor.internal": "10.42.0.1"}})
given := namesOf(got[0])
if len(given) != 1 || given[0] != "anchor.internal:10.42.0.1" {
t.Fatalf("a host-networked container was not given the mesh's names: %v", got[0])
}
}
// A mesh with no private network gives nothing, rather than a name with no address behind it.
func TestAMeshWithNoNamesGivesNone(t *testing.T) {
got := containersOf(t, Resolution{Node: "alone", Modules: []Manifest{{
Module: "app",
Resources: []map[string]any{{"id": "web", "type": "container", "name": "web",
"image": "registry.example/web@sha256:" + strings.Repeat("a", 64)}},
}}}, Rendering{})
if len(namesOf(got[0])) != 0 {
t.Fatalf("names were invented for a mesh that has none: %v", got[0])
}
}
// The catalogue's copy is not edited: these maps come from a manifest, and mutating one would
// change what every other machine running that module is given.
func TestGivingNamesDoesNotChangeTheCatalogue(t *testing.T) {
held := map[string]any{"id": "web", "type": "container", "name": "web",
"image": "registry.example/web@sha256:" + strings.Repeat("a", 64)}
module := Manifest{Module: "app", Resources: []map[string]any{held}}
for _, node := range []string{"one", "two"} {
containersOf(t, Resolution{Node: node, Modules: []Manifest{module}},
Rendering{Names: map[string]string{"anchor.internal": "10.42.0.1"}})
}
if _, changed := held["hosts"]; changed {
t.Fatal("the manifest the catalogue holds was edited, so every machine now carries this")
}
}
// Only containers. A file or a service given a `hosts` key is a declaration the host refuses
// outright — it takes no unknown field — so getting this wrong breaks the whole machine rather
// than one resource, and breaks it for something that was never about names.
func TestNothingButAContainerIsGivenNames(t *testing.T) {
// No resource is given a `hosts` key it did not declare. A file or a service carrying one is a
// declaration the host refuses outright — it takes no unknown field — so an invented key breaks
// the whole machine rather than one resource.
func TestNothingIsGivenNamesItDidNotDeclare(t *testing.T) {
out, err := Resolution{Node: "laptop", Modules: []Manifest{{
Module: "app",
Resources: []map[string]any{
@@ -137,11 +93,8 @@ func TestNothingButAContainerIsGivenNames(t *testing.T) {
t.Fatal(err)
}
for _, r := range out {
if r["type"] == "container" {
continue
}
if _, given := r["hosts"]; given {
t.Fatalf("a %v was given names, which the host will refuse: %v", r["type"], r)
t.Fatalf("a %v was given names it never declared: %v", r["type"], r)
}
}
}
+130
View File
@@ -0,0 +1,130 @@
package catalogue
import (
"encoding/json"
"strings"
"testing"
)
// A module that must know its own address — a login redirect, a canonical URL, an issuer — had it
// written into its manifest as a literal (novox/hq 04-ISSUES/122): a domain in a definition, wrong on
// every other machine. It is told instead, from the same composition the provider receives.
// selfAware contributes a labelled route, binds the requirement, and writes its own name into a file.
func selfAware(label string) Manifest {
m := labelled("board", label, 8080)
m.Requires = []string{"reverse-proxy"}
m.Binds = map[string]string{"reverse-proxy": "/var/lib/board/route.json"}
m.Resources = []map[string]any{
{"id": "conf", "type": "file", "path": "/var/lib/board/app.conf",
"content": "root = https://${bound:reverse-proxy:name}/\ninternal = ${bound:reverse-proxy:internal-name}\n"},
}
return m
}
// servingProxy is proxy() as the catalogue's route providers are declared: the provision scoped to
// the mesh, serving nothing a consumer must know (route-adapter, route-proxy: `"serves": {"route": {}}`).
func servingProxy() Manifest {
p := proxy()
p.Provides = []Offer{{Name: "reverse-proxy", Scope: ScopeMesh}}
p.Serves = map[string]map[string]any{"reverse-proxy": {}}
return p
}
// nodeProxy is the same provider scoped to its node, whose answer on the same machine comes from
// `here` rather than from the mesh's needs — the other path a binding is written by.
func nodeProxy() Manifest {
p := proxy()
p.Serves = map[string]map[string]any{"reverse-proxy": {"scheme": "http"}}
return p
}
// onBoth is a node with a public domain and a private-network address, so both names compose.
func onBoth(domain string) Node {
n := withDomain(domain)
n.At = "anchor.internal"
return n
}
func fileAt(t *testing.T, out []map[string]any, path string) string {
t.Helper()
for _, r := range out {
if r["path"] == path {
return r["content"].(string)
}
}
t.Fatalf("nothing was declared at %s", path)
return ""
}
func TestAModuleIsToldTheNameItsProviderServes(t *testing.T) {
got, err := Resolve(shelf(servingProxy(), selfAware("git")), []string{"traefik", "board"},
onBoth("example.tld"), World{})
if err != nil {
t.Fatal(err)
}
out := mustDeclare(t, got)
served := received(t, out)[0].Values
var binding map[string]any
if err := json.Unmarshal([]byte(fileAt(t, out, "/var/lib/board/route.json")), &binding); err != nil {
t.Fatal(err)
}
if binding["name"] != served["name"] || binding["name"] != "git.example.tld" {
t.Fatalf("the module was told %v, the provider serves %v", binding["name"], served["name"])
}
if binding["internal-name"] != served["internal-name"] || binding["internal-name"] == nil {
t.Fatalf("internal name: module told %v, provider serves %v",
binding["internal-name"], served["internal-name"])
}
conf := fileAt(t, out, "/var/lib/board/app.conf")
want := "root = https://git.example.tld/\ninternal = " + served["internal-name"].(string) + "\n"
if conf != want {
t.Fatalf("the file was rendered as\n%s\nwant\n%s", conf, want)
}
}
func TestTheNameAModuleIsToldFollowsTheNodesDomain(t *testing.T) {
// The whole point: the same definition, two machines, two names — nothing edited.
for _, domain := range []string{"example.tld", "other.example"} {
got, err := Resolve(shelf(servingProxy(), selfAware("git")), []string{"traefik", "board"},
onBoth(domain), World{})
if err != nil {
t.Fatal(err)
}
conf := fileAt(t, mustDeclare(t, got), "/var/lib/board/app.conf")
if !strings.HasPrefix(conf, "root = https://git."+domain+"/") {
t.Fatalf("on %s the module wrote %q", domain, conf)
}
}
}
func TestAModuleWithNoPublicNameIsNotToldOne(t *testing.T) {
// No public domain on the node: nothing composed, so no `name` — and a file asking for one is
// refused rather than rendered with a placeholder or an empty host.
m := selfAware("git")
m.Resources[0]["content"] = "root = https://${bound:reverse-proxy:name}/\n"
got, err := Resolve(shelf(servingProxy(), m), []string{"traefik", "board"}, workstation(), World{})
if err != nil {
t.Fatal(err)
}
if _, err := got.Declaration(Rendering{}); err == nil ||
!strings.Contains(err.Error(), `"name"`) {
t.Fatalf("a file asking for a name that was never composed was not refused: %v", err)
}
}
func TestAModuleIsToldItsNameByANodeScopedProviderToo(t *testing.T) {
m := selfAware("git")
m.Resources[0]["content"] = "root = https://${bound:reverse-proxy:name}/\n"
got, err := Resolve(shelf(nodeProxy(), m), []string{"board"},
withDomain("example.tld"), World{})
if err != nil {
t.Fatal(err)
}
conf := fileAt(t, mustDeclare(t, got), "/var/lib/board/app.conf")
if !strings.HasPrefix(conf, "root = https://git.example.tld/") {
t.Fatalf("a same-machine, node-scoped answer did not tell the module its name: %q", conf)
}
}
+180
View File
@@ -0,0 +1,180 @@
package catalogue
import (
"encoding/json"
"os"
"path/filepath"
"reflect"
"strings"
"testing"
)
// TestPlacedDirectoriesKeepTheirPaths is the check novox/hq issue 119 asks for before a definition
// stops naming where its data lives: a converted manifest, resolved on a node with the default root,
// names exactly the paths the manifest before it named. Data that a service is using must not move
// because a definition stopped saying where it was.
//
// Two checkouts: MESH_CATALOGUE_BEFORE, the catalogue as it was, and MESH_CATALOGUE, as it is now.
// Every module in both is resolved with the controller's own rule (dirsFor, dirFill) and compared
// whole — not only the directories, but every string a directory's id was written into. Both
// sides are resolved, so a manifest converted in two steps (issue 119, then issue 174) is judged
// against the paths it named, not the text it used to name them with.
func TestPlacedDirectoriesKeepTheirPaths(t *testing.T) {
before, after := os.Getenv("MESH_CATALOGUE_BEFORE"), os.Getenv("MESH_CATALOGUE")
if before == "" || after == "" {
t.Skip("set MESH_CATALOGUE_BEFORE and MESH_CATALOGUE to two catalogue checkouts to run this")
}
found, _ := filepath.Glob(filepath.Join(after, "modules", "*", "module.json"))
compared := 0
for _, path := range found {
module := filepath.Base(filepath.Dir(path))
old, err := os.ReadFile(filepath.Join(before, "modules", module, "module.json"))
if err != nil {
continue // new since; nothing to keep
}
now, err := os.ReadFile(path)
if err != nil {
t.Fatal(err)
}
if string(old) == string(now) {
continue
}
m, err := ParseManifest(now)
if err != nil {
t.Errorf("%s: %v", module, err)
continue
}
earlier, err := ParseManifest(old)
if err != nil {
t.Errorf("%s before: %v", module, err)
continue
}
var was, is any
if err := json.Unmarshal(old, &was); err != nil {
t.Fatal(err)
}
if err := json.Unmarshal(now, &is); err != nil {
t.Fatal(err)
}
// Both sides resolved: the manifest before may itself already place some directories
// (issue 119's conversion), and what must not move is the path a machine sees.
was = resolvedTree(was, dirsFor(earlier, Rendering{}), module, t)
resolved := resolvedTree(is, dirsFor(m, Rendering{}), module, t)
// An access named by id resolves to the path the definition still carries as its default
// (issue 153) — the same rule as a placed directory: an assignment that says nothing moves
// nothing.
was = accessesResolved(was, earlier)
resolved = accessesResolved(resolved, m)
if !reflect.DeepEqual(was, resolved) {
wasJSON, _ := json.MarshalIndent(was, "", " ")
isJSON, _ := json.MarshalIndent(resolved, "", " ")
t.Errorf("%s: resolved on the default root, the converted manifest is not the one before it\n--- before\n%s\n--- resolved now\n%s",
module, firstDifference(string(wasJSON), string(isJSON)), "")
}
compared++
}
t.Logf("%d converted manifest(s) resolve to the paths they named before", compared)
}
// resolvedTree is the manifest as a machine would see it: every ${dir:…} filled, a pathless
// directory given the path it resolves to, and the placement word removed.
func resolvedTree(node any, dirs map[string]string, module string, t *testing.T) any {
switch v := node.(type) {
case map[string]any:
out := map[string]any{}
for k, child := range v {
if k == "place" {
continue
}
out[k] = resolvedTree(child, dirs, module, t)
}
if out["type"] == "directory" {
if _, has := out["path"]; !has {
if id, ok := out["id"].(string); ok {
out["path"] = dirs[id]
}
}
}
return out
case []any:
out := make([]any, len(v))
for i, child := range v {
out[i] = resolvedTree(child, dirs, module, t)
}
return out
case string:
filled, err := dirFill(v, dirs, module)
if err != nil {
t.Error(err)
}
return filled
}
return node
}
func firstDifference(a, b string) string {
al, bl := strings.Split(a, "\n"), strings.Split(b, "\n")
for i := range al {
if i >= len(bl) || al[i] != bl[i] {
from := i - 2
if from < 0 {
from = 0
}
to := i + 3
if to > len(al) {
to = len(al)
}
bt := i + 3
if bt > len(bl) {
bt = len(bl)
}
return "before:\n" + strings.Join(al[from:to], "\n") + "\nnow:\n" + strings.Join(bl[from:bt], "\n")
}
}
return "(the difference is beyond the shorter document)"
}
// accessesResolved fills every ${access:<id>} with the default path the definition carries for that
// access, and drops the id, so a manifest that names its accesses is compared by the paths a machine
// with no placement receives.
func accessesResolved(node any, m Manifest) any {
defaults := map[string]string{}
for _, a := range m.Accesses {
if a.ID != "" && a.Path != "" {
defaults[a.ID] = a.Path
}
}
var walk func(any) any
walk = func(n any) any {
switch v := n.(type) {
case map[string]any:
out := map[string]any{}
for k, child := range v {
if k == "id" {
if _, isAccess := v["mode"]; isAccess && v["type"] == nil {
if _, hasPath := v["path"]; hasPath {
continue
}
}
}
out[k] = walk(child)
}
return out
case []any:
out := make([]any, len(v))
for i, child := range v {
out[i] = walk(child)
}
return out
case string:
return accessRef.ReplaceAllStringFunc(v, func(ref string) string {
if p, ok := defaults[accessRef.FindStringSubmatch(ref)[1]]; ok {
return p
}
return ref
})
}
return n
}
return walk(node)
}
+382
View File
@@ -0,0 +1,382 @@
package catalogue
import (
"fmt"
"regexp"
"sort"
"strings"
)
// Where a module's data is on THIS machine is the assignment's (novox/hq ADR 0112, issue 153).
//
// A definition names no host path. It declares the directories it owns by id, and the operator's
// shared data it needs by id too (an `access`, ADR 0051). A node has a default layout for the
// former — <root>/<module>/<id> — and nothing at all for the latter, because shared data is
// wherever the operator keeps it. An adopted machine keeps its data where the predecessor put it:
// a 40 TB library on its own pool, a configuration on a second disk. Both halves are said on the
// assignment, validated the way `endpoints` is — an id the module does not declare is refused,
// because a setting that reaches nothing is a mistake — and resolved here, so the host receives
// concrete paths exactly as it always has and learns no field.
//
// {"places": {"config": "/services/sonarr/config",
// "data": {"path": "/mnt/plex/data", "owner": "1000:1000"}},
// "accesses": {"series": "/storage/media/series",
// "downloads": "/storage/downloads"}}
//
// A placed directory is still the mesh's: created, chowned to the owner the assignment says (or
// the manifest's), removed when empty and no longer declared. A placed access is still the
// operator's: mounted, never created, chowned or removed.
// PlacesSetting is the settings key that places a module's declared directories, by id.
const PlacesSetting = "places"
// AccessesSetting is the settings key that says where a module's accesses are on this node, by id.
const AccessesSetting = "accesses"
// Placement is what an assignment says about one of a module's directories.
type Placement struct {
// Path is where the directory is on this machine. Absolute.
Path string
// Owner is "uid:gid" when the assignment overrides the manifest's — the predecessor's data is
// owned by whoever it ran as, and that is one machine's fact.
Owner string
}
var ownerShape = regexp.MustCompile(`^[0-9]+:[0-9]+$`)
// accessRef is how a module names one of its accesses: ${access:<id>}.
var accessRef = regexp.MustCompile(`\$\{access:([a-z0-9][a-z0-9-]*)\}`)
// Places reads where this node places the module's directories, by directory id.
//
// It refuses an id the module declares no directory for, a path that is not absolute, and an
// owner that is not uid:gid. A directory the assignment does not mention keeps the manifest's
// stated path or the node's default layout.
func Places(m Manifest, layers []Layer) (map[string]Placement, error) {
declared := map[string]bool{}
for _, r := range m.Resources {
if fmt.Sprint(r["type"]) == "directory" {
declared[fmt.Sprint(r["id"])] = true
}
}
out := map[string]Placement{}
for _, layer := range layers {
raw, ok := layer.Values[PlacesSetting]
if !ok {
continue
}
blocks, ok := raw.(map[string]any)
if !ok {
return nil, fmt.Errorf("%s: %s is a { directory: path | { path, owner } } map, and %q set it to something else",
m.Module, PlacesSetting, layer.From)
}
for id, body := range blocks {
if !declared[id] {
return nil, fmt.Errorf(
"%s places the directory %q, which it does not declare — the setting reaches "+
"nothing. It declares %s", m.Module, id, orNothing(namesOfDirs(directoriesOf(m))))
}
p := out[id]
switch v := body.(type) {
case string:
p.Path = strings.TrimSpace(v)
case map[string]any:
if path, said := v["path"]; said {
text, _ := path.(string)
p.Path = strings.TrimSpace(text)
}
if owner, said := v["owner"]; said {
text, _ := owner.(string)
if !ownerShape.MatchString(strings.TrimSpace(text)) {
return nil, fmt.Errorf("%s places %q with owner %v; an owner is uid:gid, numeric",
m.Module, id, owner)
}
p.Owner = strings.TrimSpace(text)
}
default:
return nil, fmt.Errorf("%s places %q with %v; a placement is a path, or { path, owner }",
m.Module, id, body)
}
if p.Path == "" {
return nil, fmt.Errorf("%s places %q without a path", m.Module, id)
}
if !strings.HasPrefix(p.Path, "/") {
return nil, fmt.Errorf("%s places %q at %q, which is not an absolute path", m.Module, id, p.Path)
}
p.Path = strings.TrimRight(p.Path, "/")
out[id] = p
}
}
if len(out) == 0 {
return nil, nil
}
return out, nil
}
// AccessPlaces reads where this node keeps the operator's data the module accesses, by access id.
//
// It refuses an id the module declares no access under, and a path that is not absolute. An
// access declared by path alone cannot be placed — it has no name to place it by.
func AccessPlaces(m Manifest, layers []Layer) (map[string]string, error) {
declared := map[string]bool{}
for _, a := range m.Accesses {
if a.ID != "" {
declared[a.ID] = true
}
}
out := map[string]string{}
for _, layer := range layers {
raw, ok := layer.Values[AccessesSetting]
if !ok {
continue
}
blocks, ok := raw.(map[string]any)
if !ok {
return nil, fmt.Errorf("%s: %s is a { access: path } map, and %q set it to something else",
m.Module, AccessesSetting, layer.From)
}
for id, body := range blocks {
if !declared[id] {
return nil, fmt.Errorf(
"%s places the access %q, which it does not declare — the setting reaches "+
"nothing. It declares %s", m.Module, id, orNothing(namesOfAccesses(m)))
}
path, _ := body.(string)
path = strings.TrimSpace(path)
if !strings.HasPrefix(path, "/") {
return nil, fmt.Errorf("%s places the access %q at %v, which is not an absolute path",
m.Module, id, body)
}
out[id] = strings.TrimRight(path, "/")
}
}
if len(out) == 0 {
return nil, nil
}
return out, nil
}
// placedAccess is one access with the path it resolves to on this node.
type placedAccess struct {
ID string
Path string
Mode string
}
// accessesFor is every access of a module with its path on this node: the assignment's where it
// placed one, the definition's where it carries a default, and refused where neither says — an
// access that resolves to nowhere would reach the machine as a mount of nothing.
func accessesFor(m Manifest, layers []Layer) ([]placedAccess, map[string]string, error) {
placed, err := AccessPlaces(m, layers)
if err != nil {
return nil, nil, err
}
var out []placedAccess
byID := map[string]string{}
for _, a := range m.Accesses {
path := a.Path
if a.ID != "" {
if at, said := placed[a.ID]; said {
path = at
}
}
if path == "" {
return nil, nil, fmt.Errorf(
"%s accesses %q, and nothing says where that is on this node — the definition "+
"carries no path (it must not, novox/hq ADR 0112) and the assignment places "+
"none. Set %s: {%q: \"/where/it/is\"}",
m.Module, a.ID, AccessesSetting, a.ID)
}
out = append(out, placedAccess{ID: a.ID, Path: path, Mode: a.At()})
if a.ID != "" {
byID[a.ID] = path
}
}
return out, byID, nil
}
// accessFill resolves every ${access:…} in one string, or refuses a reference naming no access.
func accessFill(s string, accesses map[string]string, module string) (string, error) {
var missing error
out := accessRef.ReplaceAllStringFunc(s, func(ref string) string {
id := accessRef.FindStringSubmatch(ref)[1]
path, has := accesses[id]
if !has {
missing = fmt.Errorf(
"%s says ${access:%s}, and %s declares no access %q. It declares %s",
module, id, module, id, orNothing(namesOfAccessIDs(accesses)))
return ref
}
return path
})
return out, missing
}
// accessInto fills every ${access:…} a resource carries — in its path, its content, its mounts,
// its environment and its env-files — with the path this node resolved for it. The same walk as
// dirInto, for the same reason: a literal `${access:x}` reaching the machine would be mounted as
// a directory called that.
func accessInto(resource map[string]any, accesses map[string]string, module string) error {
if !mentionsAccess(resource) {
return nil
}
fill := func(s string) (string, error) { return accessFill(s, accesses, module) }
var err error
if path, ok := resource["path"].(string); ok {
if resource["path"], err = fill(path); err != nil {
return err
}
}
if content, ok := resource["content"].(string); ok {
if resource["content"], err = fill(content); err != nil {
return err
}
}
for _, field := range []string{"volumes", "env-file"} {
list, ok := resource[field].([]any)
if !ok {
continue
}
filled := make([]any, len(list))
for i, v := range list {
filled[i] = v
if s, ok := v.(string); ok {
if filled[i], err = fill(s); err != nil {
return err
}
}
}
resource[field] = filled
}
if env, ok := resource["env"].(map[string]any); ok {
filled := make(map[string]any, len(env))
for key, v := range env {
filled[key] = v
if s, ok := v.(string); ok {
if filled[key], err = fill(s); err != nil {
return err
}
}
}
resource["env"] = filled
}
return nil
}
func mentionsAccess(resource map[string]any) bool {
for _, field := range []string{"path", "content"} {
if s, ok := resource[field].(string); ok && accessRef.MatchString(s) {
return true
}
}
for _, field := range []string{"volumes", "env-file"} {
if list, ok := resource[field].([]any); ok {
for _, v := range list {
if s, ok := v.(string); ok && accessRef.MatchString(s) {
return true
}
}
}
}
if env, ok := resource["env"].(map[string]any); ok {
for _, v := range env {
if s, ok := v.(string); ok && accessRef.MatchString(s) {
return true
}
}
}
return false
}
// ownerInto gives a placed directory the owner the assignment said, where it said one. The
// manifest's owner is what the image expects on any machine; the assignment's is what this
// machine's data already is.
func ownerInto(resource map[string]any, placed map[string]Placement) {
if fmt.Sprint(resource["type"]) != "directory" {
return
}
if p, ok := placed[fmt.Sprint(resource["id"])]; ok && p.Owner != "" {
resource["owner"] = p.Owner
}
}
// unknownAccessRefs is every ${access:…} in the definition that names no access the definition
// declares by id — refused where the author is, as unknownDirRefs does for directories.
func (m Manifest) unknownAccessRefs() []string {
declared := map[string]bool{}
for _, a := range m.Accesses {
if a.ID != "" {
declared[a.ID] = true
}
}
seen := map[string]bool{}
var problems []string
refuse := func(s string, where any) {
for _, match := range accessRef.FindAllStringSubmatch(s, -1) {
id := match[1]
if declared[id] || seen[id] {
continue
}
seen[id] = true
problems = append(problems, fmt.Sprintf(
"%s says ${access:%s} in %v, and declares no access %q — a reference the mesh "+
"cannot place would reach the machine as a literal path",
m.Module, id, where, id))
}
}
for _, r := range m.Resources {
for _, field := range []string{"path", "content"} {
if s, ok := r[field].(string); ok {
refuse(s, r["id"])
}
}
for _, field := range []string{"volumes", "env-file"} {
if list, ok := r[field].([]any); ok {
for _, v := range list {
if s, ok := v.(string); ok {
refuse(s, r["id"])
}
}
}
}
if env, ok := r["env"].(map[string]any); ok {
for _, v := range env {
if s, ok := v.(string); ok {
refuse(s, r["id"])
}
}
}
}
sort.Strings(problems)
return problems
}
func directoriesOf(m Manifest) map[string]string {
dirs := map[string]string{}
for _, r := range m.Resources {
if fmt.Sprint(r["type"]) == "directory" {
dirs[fmt.Sprint(r["id"])] = ""
}
}
return dirs
}
func namesOfAccesses(m Manifest) []string {
var names []string
for _, a := range m.Accesses {
if a.ID != "" {
names = append(names, fmt.Sprintf("%q", a.ID))
}
}
sort.Strings(names)
return names
}
func namesOfAccessIDs(accesses map[string]string) []string {
var names []string
for id := range accesses {
names = append(names, fmt.Sprintf("%q", id))
}
sort.Strings(names)
return names
}
+172
View File
@@ -0,0 +1,172 @@
package catalogue
import (
"strings"
"testing"
)
// The case novox/hq issue 153 records: an adopted machine keeps its data where the predecessor put
// it — a library on its own pool that must never move, a configuration on a second disk owned by
// whoever the predecessor ran as. A definition may name none of that (ADR 0112); the assignment
// says it, by the ids the definition declared, and the machine receives concrete paths as always.
func placeable() Manifest {
m := mod("arr", nil, nil, nil)
m.Resources = []map[string]any{
{"id": "state", "type": "directory", "place": ".", "mode": "0700"},
{"id": "config", "type": "directory", "mode": "0755", "owner": "1000:1000"},
{"id": "server", "type": "container", "name": "arr", "image": "arr:1",
"volumes": []any{"${dir:config}:/config", "${access:series}:/series", "${access:spool}:/downloads:ro"},
"env": map[string]any{"SPOOL": "${access:spool}"}},
}
m.Accesses = []Access{{ID: "series", Mode: AccessReadWrite}, {ID: "spool"}}
return m
}
func placedBy(values map[string]any) Rendering {
return Rendering{Settings: SettingsBy{"arr": {{From: "node anchor", Values: values}}}}
}
func TestAnAssignmentPlacesDirectoriesAndAccesses(t *testing.T) {
got, err := Resolve(shelf(placeable()), []string{"arr"}, workstation(), World{})
if err != nil {
t.Fatal(err)
}
out, err := got.Declaration(placedBy(map[string]any{
PlacesSetting: map[string]any{
"config": map[string]any{"path": "/services/arr/config/", "owner": "1001:2000"},
},
AccessesSetting: map[string]any{
"series": "/storage/media/series",
"spool": "/storage/downloads",
},
}))
if err != nil {
t.Fatal(err)
}
seen := map[string]map[string]any{}
for _, r := range out {
seen[r["id"].(string)] = r
}
config := seen["arr.config"]
if config["path"] != "/services/arr/config" || config["owner"] != "1001:2000" {
t.Fatalf("the placed directory is %v %v; want the assignment's path and owner", config["path"], config["owner"])
}
if seen["arr.state"]["path"] != "/var/lib/arr" {
t.Fatalf("an unplaced directory left the default layout: %v", seen["arr.state"]["path"])
}
var accesses []string
for _, r := range out {
if r["type"] == "access" {
accesses = append(accesses, r["path"].(string)+" "+r["mode"].(string))
}
}
if strings.Join(accesses, ",") != "/storage/media/series read-write,/storage/downloads read" {
t.Fatalf("the accesses reached the machine as %v", accesses)
}
server := seen["arr.server"]
mounts := server["volumes"].([]any)
if mounts[0] != "/services/arr/config:/config" || mounts[1] != "/storage/media/series:/series" ||
mounts[2] != "/storage/downloads:/downloads:ro" {
t.Fatalf("the mounts were not filled with the placed paths: %v", mounts)
}
if server["env"].(map[string]any)["SPOOL"] != "/storage/downloads" {
t.Fatalf("the environment was not filled: %v", server["env"])
}
}
// An access declared by id and placed by nobody resolves to nowhere, and that is refused with the
// setting to write — not mounted as the literal, not skipped.
func TestAnUnplacedAccessIsRefusedByName(t *testing.T) {
got, err := Resolve(shelf(placeable()), []string{"arr"}, workstation(), World{})
if err != nil {
t.Fatal(err)
}
_, err = got.Declaration(placedBy(map[string]any{
AccessesSetting: map[string]any{"series": "/storage/media/series"},
}))
if err == nil || !strings.Contains(err.Error(), `"spool"`) || !strings.Contains(err.Error(), AccessesSetting) {
t.Fatalf("an access nobody placed was not refused by name: %v", err)
}
}
// Validated like endpoints: an id the module does not declare reaches nothing, and the refusal
// says what it does declare; a relative path and a non-numeric owner are refused too.
func TestPlacementsAreValidated(t *testing.T) {
m := placeable()
layers := func(values map[string]any) []Layer { return placedBy(values).Settings["arr"] }
_, err := Places(m, layers(map[string]any{PlacesSetting: map[string]any{"data": "/mnt/data"}}))
if err == nil || !strings.Contains(err.Error(), "does not declare") || !strings.Contains(err.Error(), `"config"`) {
t.Fatalf("placing an undeclared directory was accepted: %v", err)
}
_, err = Places(m, layers(map[string]any{PlacesSetting: map[string]any{"config": "services/arr"}}))
if err == nil || !strings.Contains(err.Error(), "absolute") {
t.Fatalf("a relative placement was accepted: %v", err)
}
_, err = Places(m, layers(map[string]any{PlacesSetting: map[string]any{
"config": map[string]any{"path": "/services/arr", "owner": "media"}}}))
if err == nil || !strings.Contains(err.Error(), "uid:gid") {
t.Fatalf("a non-numeric owner was accepted: %v", err)
}
_, err = AccessPlaces(m, layers(map[string]any{AccessesSetting: map[string]any{"movies": "/storage/media/movies"}}))
if err == nil || !strings.Contains(err.Error(), "does not declare") || !strings.Contains(err.Error(), `"series"`) {
t.Fatalf("placing an undeclared access was accepted: %v", err)
}
_, err = AccessPlaces(m, layers(map[string]any{AccessesSetting: map[string]any{"series": "media/series"}}))
if err == nil || !strings.Contains(err.Error(), "absolute") {
t.Fatalf("a relative access was accepted: %v", err)
}
// And the two keys are never stray: they are validated here, not merged into a file.
if stray := UnusedSettings(m, layers(map[string]any{
PlacesSetting: map[string]any{"config": "/services/arr/config"},
AccessesSetting: map[string]any{"series": "/storage/media/series"},
})); len(stray) != 0 {
t.Fatalf("the placement keys were reported as unused: %v", stray)
}
}
// A definition that carries a path still works, as the default the assignment may replace — and
// the assignment's placement wins where both say.
func TestADefinitionsPathIsTheDefaultTheAssignmentReplaces(t *testing.T) {
m := placeable()
m.Accesses = []Access{{ID: "series", Path: "/services/media/series", Mode: AccessReadWrite}, {ID: "spool", Path: "/services/media/downloads"}}
got, err := Resolve(shelf(m), []string{"arr"}, workstation(), World{})
if err != nil {
t.Fatal(err)
}
out, err := got.Declaration(placedBy(map[string]any{
PlacesSetting: map[string]any{"config": "/services/arr/config"},
AccessesSetting: map[string]any{"series": "/storage/media/series"},
}))
if err != nil {
t.Fatal(err)
}
var paths []string
for _, r := range out {
if r["type"] == "access" {
paths = append(paths, r["path"].(string))
}
}
if strings.Join(paths, ",") != "/storage/media/series,/services/media/downloads" {
t.Fatalf("placed one, defaulted the other: got %v", paths)
}
}
// A reference to an access the definition does not declare is refused where the author is.
func TestAnUnknownAccessReferenceIsRefusedAtParse(t *testing.T) {
_, err := ParseManifest([]byte(`{
"module": "arr", "version": "1",
"accesses": [{"id": "series", "mode": "read-write"}],
"resources": [
{"id": "state", "type": "directory", "place": ".", "mode": "0700"},
{"id": "server", "type": "container", "name": "arr", "image": "arr:1",
"volumes": ["${access:movies}:/movies"]}
]}`))
if err == nil || !strings.Contains(err.Error(), "${access:movies}") {
t.Fatalf("a reference to an undeclared access was accepted: %v", err)
}
_, err = ParseManifest([]byte(`{"module": "arr", "version": "1", "accesses": [{"mode": "read"}]}`))
if err == nil || !strings.Contains(err.Error(), "neither an id nor a path") {
t.Fatalf("an access with no id and no path was accepted: %v", err)
}
}
+1 -2
View File
@@ -98,8 +98,7 @@ func portInto(resource map[string]any, module string, listens []Listening, with
// **A fresh map, and only when something changes.** This map came out of the module's
// manifest and the resource around it is a shallow copy, so filling a value in place would
// change what the catalogue holds for every other machine running the module — the trap
// withMeshNames is written to avoid, one field along.
// change what the catalogue holds for every other machine running the module.
var filled map[string]any
for _, key := range named {
written, ok := env[key].(string)
+141
View File
@@ -0,0 +1,141 @@
package catalogue
import (
"encoding/json"
"fmt"
"strings"
"testing"
)
// A module version prepares its state before it runs (novox/hq ADR 0135).
//
// What the mesh derives is the module's own resource, run once with one word, placed immediately in
// front of the thing it prepares for. What matters in these tests is that the derivation is a copy
// rather than a second description: the failure it replaces was a hand-written step repeating six
// fields of the resource it preceded, each free to drift from it.
func aPreparingModule() Manifest {
return Manifest{
Module: "gitea",
Prepares: true,
Resources: []map[string]any{
{"id": "state", "type": "directory", "path": "/var/lib/gitea", "mode": "0700"},
{"id": "server", "type": "container", "name": "mesh-gitea-server",
"image": "gitea/gitea@" + digest, "ports": []any{"3000:3000"}},
{"id": "runtime", "type": "container", "name": "mesh-gitea",
"image": ArtifactStoreScheme + "gitea/runtime@" + digest, "network": "host",
"env": map[string]any{"MESH_GITEA_STATE_DIR": "/run/state"},
"volumes": []any{"/var/lib/gitea:/run/state:ro"},
"ports": []any{"9000:9000"}},
},
}
}
func declaredFor(t *testing.T, m Manifest) []map[string]any {
t.Helper()
// The manifest as the mesh holds it: a build resolved the module's own artifact into the
// reference it recorded, which is also how the composition knows whose code a resource runs.
out, err := Resolution{Node: "anchor", Modules: []Manifest{m}}.Declaration(
Rendering{ArtifactStore: "anchor.internal:5100"})
if err != nil {
t.Fatal(err)
}
return out
}
func idsOf(resources []map[string]any) []string {
var ids []string
for _, r := range resources {
ids = append(ids, fmt.Sprint(r["id"]))
}
return ids
}
// The step runs the module's own code, and comes immediately before it — not before the upstream
// server the module packages, which may be the very thing the state lives in.
func TestThePreparationRunsTheModulesOwnCodeAndComesRightBeforeIt(t *testing.T) {
out := declaredFor(t, aPreparingModule())
ids := idsOf(out)
at := -1
for i, id := range ids {
if id == "gitea.runtime-prepare" {
at = i
}
}
if at < 0 {
t.Fatalf("nothing prepares this module's state: %v", ids)
}
// A module's name may contain a dot, so a resource's module is everything before the last one —
// which the derived id must not add to, or a machine reads the wrong owner from it.
if strings.Count("gitea.runtime-prepare", ".") != 1 {
t.Fatal("the derived id adds a dot, so what owns it cannot be read from it")
}
if ids[at+1] != "gitea.runtime" {
t.Fatalf("the preparation is not immediately before the module's own code: %v", ids)
}
for _, id := range ids[:at] {
if id == "gitea.runtime" {
t.Fatalf("the module's own code runs before its state is prepared: %v", ids)
}
}
}
// It is given exactly what the module's own code is given. Asserted field by field against the
// resource it was derived from, because writing it twice is the fault this replaces.
func TestThePreparationIsGivenWhatTheModuleIsGiven(t *testing.T) {
out := declaredFor(t, aPreparingModule())
declared := byID(out)
step, workload := declared["gitea.runtime-prepare"], declared["gitea.runtime"]
if step == nil || workload == nil {
t.Fatalf("expected both, got %v", idsOf(out))
}
for _, field := range []string{"image", "network", "env", "volumes", "type"} {
if fmt.Sprint(step[field]) != fmt.Sprint(workload[field]) {
t.Errorf("the preparation's %s is %v and the module's is %v", field, step[field], workload[field])
}
}
if once, _ := step["run-once"].(bool); !once {
t.Error("the preparation is not a step, so nothing waits for it and nothing is gated by it")
}
if fmt.Sprint(step["args"]) != fmt.Sprint([]any{PreparationArgument}) {
t.Errorf("the preparation is asked for as %v", step["args"])
}
if fmt.Sprint(step["name"]) == fmt.Sprint(workload["name"]) {
t.Error("the preparation and the workload have one name, so one removes the other")
}
// A published port cannot be bound twice, and the version being replaced is still running.
if _, published := step["ports"]; published {
t.Errorf("the preparation publishes a port the running version holds: %v", step["ports"])
}
}
// A module that says nothing about preparing gets nothing, which is most modules.
func TestAModuleThatPreparesNothingGetsNoStep(t *testing.T) {
m := aPreparingModule()
m.Prepares = false
for _, id := range idsOf(declaredFor(t, m)) {
if id == "gitea.runtime-prepare" {
t.Fatal("a module that prepares nothing was given a preparation")
}
}
}
// A module whose own code the mesh cannot find has nothing to ask, and saying so where the manifest
// is read beats a declaration that quietly prepares nothing.
func TestAModuleThatPreparesAndRunsNoneOfItsOwnCodeIsRefused(t *testing.T) {
m := Manifest{
Module: "gitea",
Prepares: true,
Resources: []map[string]any{
// Only the upstream server it packages: nothing here runs gitea's own code.
{"id": "server", "type": "container", "name": "mesh-gitea-server", "image": "gitea/gitea@" + digest},
},
}
raw, err := json.Marshal(m)
if err != nil {
t.Fatal(err)
}
if _, err := ParseManifest(raw); err == nil {
t.Fatal("a module that prepares its state with nothing of its own to run was accepted")
}
}
+1 -1
View File
@@ -12,5 +12,5 @@ func TestPrintRehearsalRuleset(t *testing.T) {
rules := mustFilter(t, Resolution{Modules: []Manifest{
{Module: "pub", Listens: []Listening{{Port: 8099, From: FromMesh, Why: "the thing it serves"}}},
}}, nil)
t.Log("\n" + AsNftables(rules, []string{"192.0.2.20"}, true, nil))
t.Log("\n" + AsNftables(rules, []string{"192.0.2.20"}, true, nil, nil, "mesh0"))
}
+206
View File
@@ -0,0 +1,206 @@
package catalogue
import (
"strings"
"testing"
)
// a web module with one routed endpoint, the shape almost every routed module in the catalogue has.
func aRoutedWeb() Manifest {
return Manifest{
Module: "web",
Listens: []Listening{{Port: 3000, From: FromMesh}},
Contributes: map[string]map[string]any{
"route": {"label": "app", "port": 3000},
},
}
}
func reachSet(reach string) SettingsBy {
return SettingsBy{"web": {{From: "node anchor",
Values: map[string]any{ReachSetting: map[string]any{"3000": reach}}}}}
}
// namesFor renders the contribution a routed module makes and returns the two names it carries.
func namesFor(t *testing.T, m Manifest, settings SettingsBy) (public, internal string) {
t.Helper()
r := Resolution{Node: "anchor", Modules: []Manifest{m},
PublicDomain: "example.test", At: "anchor.internal"}
given, err := r.contributions(settings, nil, nil)
if err != nil {
t.Fatalf("contributions: %v", err)
}
for _, c := range given["route"] {
p, _ := c.Values["name"].(string)
i, _ := c.Values["internal-name"].(string)
return p, i
}
t.Fatal("the module contributed no route")
return "", ""
}
// **Nothing said composes both names, exactly as before.** This is the assertion that keeps every
// mesh already running identical until an assignment speaks, and it is the one that would break first
// if reach were read where it should not be.
func TestAnEndpointWithNoReachKeepsBothNames(t *testing.T) {
public, internal := namesFor(t, aRoutedWeb(), nil)
if public != "app.example.test" || internal != "app.anchor.internal" {
t.Fatalf("names are %q and %q, want both composed as before", public, internal)
}
}
// An internal endpoint has an internal name and no public one — so the proxy serves it inside, and
// the public authority is never asked for a name nobody wanted. This is what "must not be public"
// could not say before.
func TestAnInternalEndpointHasNoPublicName(t *testing.T) {
public, internal := namesFor(t, aRoutedWeb(), reachSet(ReachInternal))
if public != "" {
t.Fatalf("an internal endpoint composed the public name %q", public)
}
if internal != "app.anchor.internal" {
t.Fatalf("internal name is %q, want app.anchor.internal", internal)
}
}
// And the mirror: a public endpoint gets the public name and not the internal one, so the mesh's own
// authority is not asked to certify a name the service is not reached by.
func TestAPublicEndpointHasNoInternalName(t *testing.T) {
public, internal := namesFor(t, aRoutedWeb(), reachSet(ReachPublic))
if internal != "" {
t.Fatalf("a public endpoint composed the internal name %q", internal)
}
if public != "app.example.test" {
t.Fatalf("public name is %q, want app.example.test", public)
}
}
func TestBothComposesBothNames(t *testing.T) {
public, internal := namesFor(t, aRoutedWeb(), reachSet(ReachBoth))
if public == "" || internal == "" {
t.Fatalf("both should compose both names, got %q and %q", public, internal)
}
}
// **The filter reads the same value — for an endpoint the proxy does not serve.**
//
// A routed endpoint's port is how the proxy reaches it and nothing else (ADR 0045): a public service
// listens from the mesh, only the proxy reaches it, and it is exposed by name. So on a routed
// endpoint the reach asks for a name and the port keeps what the manifest said.
func TestAnUnroutedEndpointsPortFollowsItsReach(t *testing.T) {
// The same module with its route taken away: now the port is the only way in, so reach governs it.
bare := aRoutedWeb()
bare.Contributes = nil
for _, c := range []struct{ reach, want string }{
{ReachInternal, FromMesh},
{ReachPublic, FromEverywhere},
{ReachBoth, FromEverywhere},
{ReachMachine, FromMachine},
} {
r := Resolution{Node: "anchor", Modules: []Manifest{bare}}
rules, err := r.Rules(Rendering{Settings: reachSet(c.reach)})
if err != nil {
t.Fatalf("%s: rules: %v", c.reach, err)
}
found := false
for _, rule := range rules {
if rule.Port == 3000 {
found = true
if rule.From != c.want {
t.Fatalf("reach %q made the filter say %q, want %q", c.reach, rule.From, c.want)
}
}
}
if !found {
t.Fatalf("reach %q produced no rule for the port", c.reach)
}
}
}
// **A public name does not open the machine's port**, which is the case that found this.
//
// A module whose routed name must be public and whose machine-side port must not be had no way to say
// so while one value drove both. Under one value it could not be expressed; the port would reopen.
func TestAPublicNameLeavesARoutedPortAsTheManifestSaid(t *testing.T) {
r := Resolution{Node: "anchor", Modules: []Manifest{aRoutedWeb()},
PublicDomain: "example.test", At: "anchor.internal"}
rules, err := r.Rules(Rendering{Settings: reachSet(ReachPublic)})
if err != nil {
t.Fatal(err)
}
for _, rule := range rules {
if rule.Port == 3000 && rule.From != FromMesh {
t.Fatalf("a public reach opened a routed port to %q; the proxy is how it is reached",
rule.From)
}
}
// And the name it asked for is there, so the reach was not simply ignored.
public, internal := namesFor(t, aRoutedWeb(), reachSet(ReachPublic))
if public != "app.example.test" || internal != "" {
t.Fatalf("names are %q and %q, want the public one only", public, internal)
}
}
// A reach for a port the module does not listen on reaches nothing, and is refused where it is
// written rather than accepted and ignored.
func TestAReachForAPortTheModuleDoesNotListenOnIsRefused(t *testing.T) {
_, err := Reaches(aRoutedWeb(), []Layer{{From: "node anchor",
Values: map[string]any{ReachSetting: map[string]any{"9999": ReachInternal}}}})
if err == nil || !strings.Contains(err.Error(), "reaches nothing") {
t.Fatalf("a reach naming an undeclared port was accepted: %v", err)
}
}
// A value that is not a reach is refused, and the refusal names the four so a reader is one edit from
// right. "mesh" is the tempting wrong answer, because that is the filter's word for nearly the same
// thing.
func TestAValueThatIsNotAReachIsRefused(t *testing.T) {
for _, wrong := range []string{"mesh", "anywhere", "private", "true"} {
_, err := Reaches(aRoutedWeb(), []Layer{{From: "node anchor",
Values: map[string]any{ReachSetting: map[string]any{"3000": wrong}}}})
if err == nil || !strings.Contains(err.Error(), "a reach is") {
t.Fatalf("%q was accepted as a reach: %v", wrong, err)
}
}
}
// **A port that says both reach and expose is refused.** They say the same thing in different words,
// and accepting both would have the filter follow one while the names followed the other — the
// disagreement ADR 0138 exists to remove, reintroduced by the migration away from the older word.
func TestReachAndExposeForOnePortAreRefused(t *testing.T) {
_, err := Reaches(aRoutedWeb(), []Layer{{From: "node anchor", Values: map[string]any{
ReachSetting: map[string]any{"3000": ReachInternal},
ExposeSetting: map[string]any{"3000": FromEverywhere},
}}})
if err == nil || !strings.Contains(err.Error(), "same thing in different") {
t.Fatalf("a port set both ways was accepted: %v", err)
}
}
// A path-level refusal carries no port: it is a rule about a name, not an endpoint, and it inherits
// whatever that name turned out to be. Narrowing the endpoint must not silently drop it.
func TestARuleWithNoPortIsLeftAlone(t *testing.T) {
m := aRoutedWeb()
m.ContributesMany = map[string]map[string]map[string]any{
"route": {"refused": {"label": "app", "path": "/internal", "deny": true}},
}
r := Resolution{Node: "anchor", Modules: []Manifest{m},
PublicDomain: "example.test", At: "anchor.internal"}
given, err := r.contributions(reachSet(ReachInternal), nil, nil)
if err != nil {
t.Fatal(err)
}
var sawDeny bool
for _, c := range given["route"] {
if deny, _ := c.Values["deny"].(bool); deny {
sawDeny = true
// It keeps both, because it named no endpoint to be narrowed by.
if c.Values["name"] == nil || c.Values["internal-name"] == nil {
t.Fatalf("the path rule lost a name it shadows: %v", c.Values)
}
}
}
if !sawDeny {
t.Fatal("the path rule was dropped")
}
}
+10 -2
View File
@@ -707,9 +707,14 @@ func checkClaims(modules []Manifest, node Node, elsewhere []Held, holdings []Hel
}
switch h.Scope {
case ScopeMesh:
// Both claim and nobody is on record, or this refusal could not have happened.
// The remedy is the handover that records the holder (novox/hq ADR 0131,
// 04-ISSUES/170), so it is named here rather than left to be found.
problems = append(problems, fmt.Sprintf(
"%s on %s claims %q, which %s on %s already holds — one per mesh",
h.Module, node.Name, h.Claim, e.Module, e.Node))
"%s on %s claims %q, which %s on %s already holds — one per mesh. Nothing is "+
"on record for it; `seat %s --to %s/%s` records the holder, and the other "+
"assignment then stands beside it, eligible and silent",
h.Module, node.Name, h.Claim, e.Module, e.Node, h.Claim, e.Node, e.Module))
case ScopeSite:
if node.Site != "" && node.Site == e.Site {
problems = append(problems, fmt.Sprintf(
@@ -786,6 +791,9 @@ func checkResources(modules []Manifest) []string {
// which is what lets the stack in 04-ISSUES/036 co-resolve.
for _, m := range modules {
for _, a := range m.Accesses {
if a.Path == "" {
continue // placed by the assignment; nothing to compare at registration
}
switch other := ownedPath[a.Path]; other {
case "":
// Nobody owns it — the ordinary, correct case for shared data.
+32 -8
View File
@@ -48,7 +48,7 @@ func TestTheResolverForwardsToFixedUpstreamsAndNeverReadsResolvConf(t *testing.T
}
for _, want := range []string{
"\nno-resolv\n", "\nserver=1.1.1.1\n", "\nserver=8.8.8.8\n",
"\nlisten-address=127.0.0.1\n", "\ninterface=mesh0\n", "\nbind-dynamic\n",
"\nlisten-address=127.0.0.1\n", "\nlisten-address=${machine:address}\n", "\nbind-dynamic\n",
"\ndomain-needed\n", "\nbogus-priv\n",
"\nconf-file=" + m.Facts["node-zones"].Path + "\n",
} {
@@ -56,6 +56,14 @@ func TestTheResolverForwardsToFixedUpstreamsAndNeverReadsResolvConf(t *testing.T
t.Errorf("the resolver's configuration lacks %q:\n%s", strings.TrimSpace(want), config)
}
}
// By address and never by interface: dnsmasq admits a query by the interface it arrives on
// when told one, and a container's query to the private address arrives on the runtime's
// bridge — `interface=mesh0` dropped every such query, silently (novox/hq issue 110).
for _, line := range strings.Split(config, "\n") {
if strings.HasPrefix(line, "interface=") {
t.Errorf("the resolver answers by interface, so a container's query on a bridge is dropped: %s", line)
}
}
// Not .53 or .54, which systemd-resolved holds; and not .55 any more, which was a convention
// beside the one every machine already followed — the predecessor's resolv.conf says .1.
for _, taken := range []string{"127.0.0.53", "127.0.0.54", "127.0.0.55"} {
@@ -137,23 +145,39 @@ func TestTheResolverAndWhatAsksItComposeOnOneMachine(t *testing.T) {
t.Errorf("the daemon does not restart on its configuration and the machines file both: %v", service["restart-on"])
}
// The runtime's own file, written into (novox/hq ADR 0102) with the one key this module states.
// The runtime's own file, written into (novox/hq ADR 0102) with the keys this module states:
// where containers resolve, and that a restart keeps them running — because the runtime reads
// `dns` only when it starts, and the one restart that needs is the operator's (issue 110).
runtime := ids["dnsmasq.runtime-dns"]
if runtime == nil || runtime["path"] != "/etc/docker/daemon.json" || runtime["into"] != "json" {
t.Fatalf("the runtime's dns is not written into its file: %v", runtime)
}
var keys map[string][]string
var keys map[string]any
if err := json.Unmarshal([]byte(runtime["content"].(string)), &keys); err != nil {
t.Fatalf("the runtime's keys are not JSON: %v", err)
}
if len(keys) != 1 || len(keys["dns"]) != 1 || keys["dns"][0] != "10.42.0.1" {
t.Errorf("the runtime is pointed at %v; containers resolve at this machine's own private-network address, and nothing else is written", keys)
dns, _ := keys["dns"].([]any)
if len(keys) != 2 || len(dns) != 1 || dns[0] != "10.42.0.1" || keys["live-restore"] != true {
t.Errorf("the runtime is given %v; containers resolve at this machine's own private-network address, a restart keeps them, and nothing else is written", keys)
}
// The runtime is reloaded when that file changes, and never restarted: a restart stops every
// container on the machine (ADR 0102), and a reload is what turns live-restore on.
var reloaded bool
for _, r := range out {
if r["type"] == "service" && r["unit"] == "docker.service" && r["id"] != "" &&
strings.HasPrefix(r["id"].(string), "dnsmasq.") {
t.Errorf("the resolver orders the runtime restarted or reloaded, which stops every container (ADR 0102) or does nothing for dns: %v", r)
if r["type"] != "service" || r["unit"] != "docker.service" {
continue
}
if _, restarts := r["restart-on"]; restarts {
t.Errorf("the resolver orders the runtime restarted, which stops every container (ADR 0102): %v", r)
}
for _, on := range asStrings(r["reload-on"]) {
if on == "dnsmasq.runtime-dns" {
reloaded = true
}
}
}
if !reloaded {
t.Errorf("the runtime is not reloaded when its file changes, so live-restore never takes effect")
}
resolv := ids["resolv-conf.resolv"]
+15
View File
@@ -162,6 +162,13 @@ func renderRoster(tmpl string, view rosterView) (string, error) {
func entriesFrom(addresses, accounts map[string]string, suffix string) []rosterEntry {
out := make([]rosterEntry, 0, len(addresses))
for _, name := range sortedNames(addresses) {
if routed(name, suffix) {
// A routed name is already a full name under a public domain, and it has no mesh
// form: appending the suffix made `<name>.<suffix>`, which every machine's hosts file
// carried and nothing served (novox/hq issue 157). It is published as itself, once.
out = append(out, rosterEntry{Name: name, FQDN: name, Address: addresses[name], Account: accounts[name]})
continue
}
internal, bare := meshName(name, suffix)
// The account is looked up by whichever key the caller keys accounts on — the internal name
// or the bare one — so a template gets the right login however the maps were built.
@@ -187,6 +194,14 @@ func meshName(name, suffix string) (internal, bare string) {
return name + dotted, name
}
// routed says whether a name the mesh serves is a routed public name rather than a machine's: it
// carries a domain of its own and not the mesh's suffix. A machine's name is bare (`homer`) or
// internal (`homer.internal`); anything else with a dot in it was composed under a public domain.
func routed(name, suffix string) bool {
dotted := "." + strings.TrimPrefix(suffixOr(suffix), ".")
return strings.Contains(name, ".") && !strings.HasSuffix(name, dotted)
}
// suffixOr is the suffix given, or the one the mesh composes names with when none was handed down.
// The one place the default is written, so a fact and a name cannot disagree about it.
func suffixOr(suffix string) string {
+21
View File
@@ -258,3 +258,24 @@ func TestAHomeFactIsSkippedWhereThereIsNoAccount(t *testing.T) {
t.Fatalf("a home fact was placed on a machine with no operator account: %v", given)
}
}
// A routed name is already a full name under a public domain and has no mesh form. Appending the
// suffix to it made `git.example.tld.internal` — carried by every machine's hosts file, served by
// nothing, and refused by the proxy at the handshake (novox/hq issue 157). It is published as
// itself, and only a machine has a bare name beside its full one.
func TestARoutedNameIsPublishedAsItselfAndNotSuffixed(t *testing.T) {
names := map[string]string{"homer.internal": "10.42.0.1", "git.example.tld": "10.42.0.1"}
tmpl := RosterFile{Path: "/f", Template: "{{range .Names}}{{.FQDN}} {{.Name}}\n{{end}}"}
out, err := FactsInto(Manifest{Module: "a", Facts: map[string]RosterFile{"f": tmpl}},
Resolution{Node: "homer"}, names, map[string]string{"homer.internal": "10.42.0.1"}, nil, "internal")
if err != nil {
t.Fatal(err)
}
got := out[0]["content"].(string)
if strings.Contains(got, "tld.internal") {
t.Fatalf("the routed name was given a suffixed alias that nothing serves:\n%s", got)
}
if !strings.Contains(got, "git.example.tld git.example.tld\n") || !strings.Contains(got, "homer.internal homer\n") {
t.Fatalf("the roster does not carry the routed name as itself beside the machine's two forms:\n%s", got)
}
}
+41 -19
View File
@@ -198,37 +198,59 @@ func TestTheApexLabelComposesToTheBarePrivateAddress(t *testing.T) {
}
}
// novox/hq ADR 0066 propagate, by ADR 0148's means: a granted route name is published into the
// machine's roster mapped to the node that serves it, beside the `<node>.internal` names, and the
// machine's resolver answers it to every container. Nothing is written into the container itself —
// a routed name that moved would otherwise be wrong inside every container until each was recreated.
func TestARoutedNameResolvesToTheServingNode(t *testing.T) {
// novox/hq ADR 0066 propagate: a granted route name is published into internal resolution,
// mapped to the node that serves it, alongside the `<node>.internal` names — so every
// container, and an in-mesh ACME validator, resolves a routed name to the proxy that serves it.
// The names map is what withMeshNames writes into every container as `--add-host`; a route name
// mapped to the serving node's address rides the same mechanism.
names := map[string]string{
"anchor.internal": "10.42.0.1",
"git.example.tld": "10.42.0.1",
}
got := containersOf(t, Resolution{Node: "anchor", Modules: []Manifest{{
Module: "app",
Resources: []map[string]any{{"id": "web", "type": "container", "name": "web",
"image": "registry.example/web@sha256:" + strings.Repeat("a", 64)}},
}}}, Rendering{Names: map[string]string{
"anchor.internal": "10.42.0.1",
"git.example.tld": "10.42.0.1",
}})
}}}, Rendering{Names: names})
if len(got) != 1 {
t.Fatalf("expected one container, got %d", len(got))
}
given := namesOf(got[0])
var sawNode, sawRoute bool
for _, h := range given {
if h == "anchor.internal:10.42.0.1" {
sawNode = true
if given := namesOf(got[0]); len(given) != 0 {
t.Fatalf("the routed name was copied into the container, where it would go stale: %v", given)
}
if h == "git.example.tld:10.42.0.1" {
var sawRoute bool
for _, e := range entriesFrom(names, nil, "internal") {
if e.Name == "git.example.tld" && e.Address == "10.42.0.1" {
sawRoute = true
}
}
if !sawNode {
t.Fatalf("the container lost the mesh's node names: %v", given)
}
if !sawRoute {
t.Fatalf("the routed name was not published to the serving node: %v", given)
t.Fatalf("the routed name is not in the roster the machine's resolver answers from")
}
}
// novox/hq issue 139, ADR 0151: `<label>.<node>.internal` is answered by every machine's resolver as
// "anything under that node's name goes to that node", so the node in a route's internal name must
// be the one whose proxy answers it. Composed under the consumer's own name, a route served from
// another machine got an internal name that resolved to a machine with nothing listening, while the
// public name — published at the serving node's address — worked.
func TestARoutesInternalNameIsComposedUnderTheNodeThatServesIt(t *testing.T) {
hub := proxy()
hub.Provides = FromAnywhere("reverse-proxy")
got, err := Resolve(shelf(hub, labelled("board", "git", 8080)), []string{"board"},
withPrivateAddress("laptop.internal"), World{Offered: map[string][]Provider{
"reverse-proxy": {{Node: "anchor", At: "anchor.internal", Module: "traefik"}},
}})
if err != nil {
t.Fatal(err)
}
// Gathered the way the control plane gathers a consumer's contribution for a provider on
// another machine.
values, asks, err := got.ContributionsFrom("reverse-proxy", "board", nil)
if err != nil || !asks {
t.Fatalf("the route was not contributed: %v %v", asks, err)
}
if values["internal-name"] != "git.anchor.internal" {
t.Fatalf("the internal name does not say where the request arrives: %v", values)
}
}
+32 -6
View File
@@ -37,7 +37,9 @@ type Seat struct {
// today — which the bus refuses, because a namespace belongs to who it is named for.
Accepts []string
Emits []string
Serves []string
// Serves carries each verb in full — name, description, schema — because a role's tools are the
// mesh's to define and an agent's to call (novox/hq ADR 0132, design 33 §2).
Serves []Verb
// Decision is the record that made it a seat.
Decision string
}
@@ -48,7 +50,15 @@ type Seat struct {
//
// In the order a person reads it: the mesh's own, then a node's.
var defaultSeats = []Seat{
{Name: "mesh-controller", Scope: ScopeMesh, Decision: "novox/hq ADR 0079"},
// The control plane states what it did under the seat it holds (novox/hq ADR 0134): a role's
// events belong to the role, so they keep their address while the holder is replaced. No accepts,
// so no work queue is raised for it — only what its holder may say.
// And it serves the mesh's own verbs as the seat's tools (novox/hq ADR 0154): `status`, `push`,
// `assign` and the rest are a role's interface, not a container's, and stay addressable while
// the control plane is replaced.
{Name: ControllerSeatName, Scope: ScopeMesh, Decision: "novox/hq ADR 0079",
Emits: []string{"applied", "refused", "built-before"},
Serves: ControllerVerbs},
{Name: "mesh-store", Scope: ScopeMesh, Delivers: "postgres-database", Decision: "novox/hq ADR 0079"},
// **Delivers the mesh's own bus, not `amqp`.** Those were the same word until
// ADR 0127 separated them: `amqp` is a backing service a module may require, and this seat is
@@ -56,7 +66,11 @@ var defaultSeats = []Seat{
// rather than receives ambiently — 23 of the catalogue's modules never speak, and an ambient
// connection would mint a credential for each.
{Name: "mesh-broker", Scope: ScopeMesh, Delivers: "mesh-bus", Decision: "novox/hq ADR 0079"},
{Name: "the-artifact-store", Scope: ScopeMesh, Delivers: "artifact-store", Decision: "novox/hq ADR 0075"},
// Named for its scope since 2026-09-30 (novox/hq ADR 0156); `the-artifact-store` resolves to it as
// an alias on a mesh that predates the rename. It serves artifacts of every kind a build makes —
// images and archives, by digest — which is why the provision is the artifact store and not an
// image registry.
{Name: "mesh-artifact-store", Scope: ScopeMesh, Delivers: "artifact-store", Decision: "novox/hq ADR 0075"},
{Name: "mesh-catalog", Scope: ScopeMesh, Decision: "novox/hq ADR 0121"},
// Deferred renames (novox/hq ADR 0121): these deliver a provision, so renaming them is a
// delivering-seat migration with a mesh-wide cascade if a holder stops resolving mid-flight.
@@ -66,8 +80,11 @@ var defaultSeats = []Seat{
// A build is work submitted to this role and its outcome is the role's own event (ADR 0129).
// One publish reaches whoever asked, the controller that records it, and the catalogue that
// places it in the graph — what the old bus's shared exchange did for free.
// A build says what it does as it does it (novox/hq ADR 0157): `started` when work is taken,
// `log.<build id>` for every line, `built` for the outcome. The log's tail token is the build's
// id, so a reader follows one build by subject alone.
{Name: "mesh-build-machine", Scope: ScopeMesh,
Accepts: []string{"build"}, Emits: []string{"built"}, Decision: "novox/hq ADR 0121"},
Accepts: []string{"build"}, Emits: []string{"started", "built", "log.*"}, Decision: "novox/hq ADR 0121"},
{Name: "node-dns-resolver", Scope: ScopeNode, Decision: "novox/hq ADR 0121"},
{Name: "node-intrusion-prevention", Scope: ScopeNode, Decision: "novox/hq ADR 0121"},
{Name: "node-packet-filter", Scope: ScopeNode, Decision: "novox/hq ADR 0121"},
@@ -87,8 +104,9 @@ var defaultSeats = []Seat{
// A system seat name is the control plane's namespace: `mesh-*` for a mesh-wide role, `node-*` for
// a per-node one (novox/hq ADR 0121). A claim to a system name the mesh does not define is refused;
// any other name is a module's own to define and claim. Some of the mesh's own seats predate this
// convention and are not yet renamed (git, npm-package-registry, the-artifact-store,
// the-private-network) — those are in the set, so they resolve by name, not by prefix.
// convention and are not yet renamed (git, npm-package-registry, the-private-network) — those are
// in the set, so they resolve by name, not by prefix. the-artifact-store was renamed on 2026-09-30
// (novox/hq ADR 0156) and resolves through the alias table on a mesh that knew it.
func isSystemSeatName(name string) bool {
return strings.HasPrefix(name, "mesh-") || strings.HasPrefix(name, "node-")
}
@@ -265,6 +283,14 @@ func CanHold(m Manifest, seat Seat) error {
return fmt.Errorf("%s claims %s, whose holder answers for %q, and %s does not provide %q at %s scope",
m.Module, seat.Name, seat.Delivers, m.Module, seat.Delivers, seat.Scope)
}
// **Serving the seat's tools is a condition of holding it** (novox/hq ADR 0132). A holder that
// does not answer what the role promises is every caller's timeout, found at registration and
// at handover instead, naming the verbs rather than the fact that something is missing.
if missing := unservedVerbs(m.Tools, seat.Serves); len(missing) > 0 {
return fmt.Errorf("%s claims %s but does not serve %s, which that seat's protocol promises "+
"(novox/hq ADR 0132) — a holder lists every verb its seat declares under tools",
m.Module, seat.Name, strings.Join(missing, ", "))
}
return nil
}
+6 -5
View File
@@ -38,8 +38,9 @@ type SeatDeclaration struct {
Accepts []string `json:"accepts,omitempty"`
// Emits are the verbs the holder publishes: 1:many, nobody obliged to act.
Emits []string `json:"emits,omitempty"`
// Serves are the verbs the holder answers: request and reply, awaited.
Serves []string `json:"serves,omitempty"`
// Serves are the verbs the holder answers: request and reply, awaited. A bare name, or the
// verb in full with its schema (novox/hq ADR 0132).
Serves []Verb `json:"serves,omitempty"`
// RetainSeconds is how long the inbound backlog survives with no holder, zero for the
// mesh's default. Retention belongs to whoever owns the namespace (design 29 §3) — a seat
@@ -62,7 +63,7 @@ func (s SeatDeclaration) At() string {
func (s SeatDeclaration) verbs() []string {
out := append([]string{}, s.Accepts...)
out = append(out, s.Emits...)
return append(out, s.Serves...)
return append(out, VerbNames(s.Serves)...)
}
// declaredSeatProblems is what one manifest can be judged on alone.
@@ -228,8 +229,8 @@ func unserved(m Manifest, s SeatDeclaration) []string {
}
var missing []string
for _, t := range s.Serves {
if !has[t] {
missing = append(missing, t)
if !has[t.Name] {
missing = append(missing, t.Name)
}
}
return missing

Some files were not shown because too many files have changed in this diff Show More