Compare commits

..
Author SHA1 Message Date
jschoubben c21b3aa207 The mesh's interface takes over the found tunnel's MTU
Carries MTU from the reported tunnel (mesh-host#28) through inventory,
the overlay graph's TakeOver, into the generated config's [Interface].
A tuned path keeps its MTU across the takeover instead of regressing to
1420 and hanging transfers no ping would reveal. Two emit tests; a
tunnel with no MTU writes no line.
2026-09-26 22:40:24 +02:00
jschoubben cc252472e2 A taken tunnel brings its ListenPort, even on a node the hub cannot dial
A home node behind NAT (no Endpoint → not Reachable) that took over a
tunnel must still listen on that tunnel's port: its LAN peers dial it
there. ListenPort was gated on Reachable, which conflated 'a peer dials
me here' with 'the hub can dial me' — so the takeover guard refused
overlay-up, and the guard's suggested remedy (re-place with an
endpoint) breaks a NAT'd node's path: it stops keepalive and hands the
hub a private LAN address to dial. TakeOver now carries the found
tunnel's port (already known to the controller), and the interface
listens on it when the node is not otherwise reachable. Two tests;
Endpoint-reachable nodes keep the old path unchanged.
2026-09-26 22:33:27 +02:00
jschoubben d2ab0b2b82 Merge pull request 'The broker opening is only on the broker's host, not every node' (#74) from fix/foundation-opening-only-on-the-broker-host into main 2026-09-26 20:20:50 +00:00
jschoubben 48d8c89749 The broker opening is only on the broker's host, not every node
Enrolling ace applied adoption.opening-tcp-5671-incoming to it, opening
5671 from anywhere (v4+v6) where nothing listens — the ace session
caught it. foundation ports widen the broker's from:mesh port to
from-anywhere so a machine that is not yet on the mesh can make its
first dial; that belongs on the broker's host alone. foundationPortsFor
keeps the port only when a module resolved onto this node listens on
it, so novox opens 5671 and a node that merely dials out opens nothing.
Two tests, both directions.
2026-09-26 22:20:10 +02:00
jschoubben 2f7b407000 Merge pull request 'A carried peer is nameable, and the mesh answers for it (hq 112)' (#73) from feat/112-a-carried-peer-is-nameable into main 2026-09-26 18:10:00 +00:00
jschoubben 952092ccb3 A carried peer is nameable, and the mesh answers for it (hq 112)
The tunnel the hub took over routes to machines the predecessor knows
by name and the mesh knew only by address — taking the resolver in that
state silences three machines at once. Now the operator states which
machine a carried address is (overlay name <address> <name>), the
statement rides tunnel_peer.named, and namesInTheMesh answers for named
not-yet-enrolled peers — one reading, so the hosts fact, a container's
hosts and the resolver cannot disagree. Enrolment verifies the word:
a machine enrolling under a named peer's key with a different name is
refused where the operator can read it, the stated name keeps the
carried address, and an enrolled peer's name is the node's — naming it
again refuses. The issue's rule holds: a name the predecessor answers
for keeps resolving until the machine behind it is a node.
2026-09-26 20:09:42 +02:00
jschoubben ed08cc1adc Merge pull request 'A repeat assignment says nothing changed (ADR 0115)' (#72) from feat/a-second-assignment-says-so into main 2026-09-26 17:02:49 +00:00
jschoubben 50734095b8 A repeat assignment says nothing changed (ADR 0115)
One assignment of a module per node is now the rule, not a limitation —
the operator dropped the multi-assignment requirement, and the schema's
(node, module) key has been the decision since migration 0005. What
changed: Assign reports whether the assignment was new, and the command
says 'already runs — one node runs one of each (ADR 0115); nothing
changed' instead of printing 'is assigned' for a no-op, which read as
an action that happened. Idempotence stays: a repeat is exit 0, because
a script stating what is already true is not wrong.
2026-09-26 19:02:35 +02:00
jschoubben 95426e25cf Merge pull request 'A collision is two places, not two spellings' (#71) from fix/collisions-compare-placed-paths into main 2026-09-26 16:25:40 +00:00
jschoubben fda47558d5 A collision is two places, not two spellings
checkResources compared paths as written, so ${dir:state}/server.env —
the same characters in every module, a different directory in each —
refused the first two placed modules that met. Paths are placed before
they are compared, under the default root, which keeps every real
collision: distinct modules' places are distinct under any one root,
and a module stating another's placed root is caught because a pathless
directory now owns its placed path in the comparison too.
2026-09-26 18:25:28 +02:00
jschoubben 8ea80f9584 Merge pull request 'The assignment's own root is a place, and the manifest's maps are placed' (#70) from feat/the-assignment-root-and-the-manifests-maps into main 2026-09-26 16:18:27 +00:00
jschoubben 2e3b13c0f8 The assignment's own root is a place, and the manifest's maps are placed
Slice two of ADR 0112. A pathless directory saying place "." is the
assignment's one directory, <root>/<module> — to-be 27's shape — and
place never reaches the host, which parses strictly. The maps naming
where bindings, credentials and contributions land (binds, secrets,
own-secrets, receives, grants) fill against the placed directories at
composition, into fresh maps and a fresh module slice, because one
resolution composes for many nodes. The five absolute-path checks on
those maps accept a placed reference — resolution makes it absolute
before anything reads it — while certificate, operator-keeps and
accesses paths stay absolute-only: those are the operator's or another
vocabulary's. unknownDirRefs scans the maps too, and validates place
itself: only on a directory, only ".", never beside a stated path.

Found by the foundation tests validating the sibling catalogue: the
first conversion's blanket replace turned /var/lib/gitea/database.json
into ${dir:data}base.json — which resolves to the right path by pure
string concatenation. Production was saved by a coincidence; the
catalogue cleanup that follows spells it ${dir:state}/database.json.
2026-09-26 18:18:13 +02:00
jschoubben cd2481dcd8 Merge pull request 'A directory the mesh places: ${dir:<id>} and the pathless directory resource' (#69) from feat/a-directory-the-mesh-places into main 2026-09-26 15:52:25 +00:00
jschoubben d2cbc9dbdc A directory the mesh places: ${dir:<id>} and the pathless directory resource
The first executable slice of ADR 0112 / to-be 27, sized to what the
operator settled tonight: a module definition names no host path for
its own data. A directory resource may omit path; composition resolves
it to <root>/<module>/<id>, the root a node's setting on Rendering with
/var/lib as the default — which reproduces exactly the layout novox
converged to by hand. ${dir:<id>} names the place from a resource's
path, content, mounts, environment and env-files, the same shape as
${bound:…}. A directory that states a path keeps it and still answers
by name — that is the adopted-data placement, mssql its live case.

Resolved in the controller at composition, so the wire format and the
host change not at all; a reference naming no directory refuses at the
manifest and again at composition; nested fills are rebuilt, never
written into the manifest's own maps, because one manifest composes
for many nodes.
2026-09-26 17:51:54 +02:00
jschoubben e88d3bd485 Merge pull request 'A route may say the largest body it carries' (#68) from feat/a-route-may-limit-the-body-it-carries into main 2026-09-26 14:01:56 +00:00
jochen a287812e14 A route may say the largest body it carries
Proxy configuration beside insecure, not a fifth policy — ADR 0108 closed that set at four, and both
of these tune how a request is carried rather than deciding what a name admits. A registry is the
case that needs it: image layers arrive as single requests of gigabytes and a proxy's own default
refuses them long before the workload is reached.

Absent is no limit, which is what every route already got. A limit that is not a whole positive
number of bytes takes the route with it, named in the log like a port that is not one — serving it
without the limit would carry exactly what the module said not to carry. Enforced on the declared
length where there is one, and while reading for a chunked body, which declares none: without the
second, a limit is advice.
2026-09-26 16:01:21 +02:00
jschoubben 557f419e71 Merge pull request 'mount the broker TLS directory bind, not the old named volume' (#54) from fix/own-broker-tls-mount-is-the-directory-bind into main 2026-09-26 12:55:32 +00:00
jochen 71c8080359 Declare the broker's TLS directory as an access, not an undeclared bind
The swap from a named volume to the host directory left the mount undeclared, which main's own
manifest check now refuses: a bind the module did not declare is created by the runtime as root, so
the module's owner and mode never reach it and ADR 0030's data rule does not cover it.

The directory is the broker's — lavinmq declares it as its own, mode 0700 — so from here it is an
access, read-only: a pre-existing path this module is granted use of and does not own.
2026-09-26 14:55:04 +02:00
jschoubben cc86f8b433 Merge main 2026-09-26 14:53:55 +02:00
jschoubben 0944311f86 Merge pull request 'Seats are a closed set, a seat's holder answers for what it delivers, and a build source may live on the git seat' (#63) from feat/seats-are-a-closed-set into main 2026-09-26 12:31:16 +00:00
jschoubben 7e42380dcd Merge main 2026-09-26 14:29:00 +02:00
jschoubben 41de152739 Merge pull request 'route-proxy: a policy refusal is also not-my-token' (#67) from fix/a-policy-refusal-is-also-not-my-token into main 2026-09-26 12:26:19 +00:00
jschoubben dad0a153ff route-proxy: a policy refusal is also not-my-token
autocert checks the host policy before the token and answers 403 — the
internal authority does this for every public name, so mail.novox.be's
challenge died on the internal manager's probe one commit after it
stopped dying on the public one's 404. Both shapes of refusal now fall
through to routing; a fifth test pins the 403 case with a refusing
policy.
2026-09-26 14:26:01 +02:00
jschoubben 345722a4fd Merge pull request 'route-proxy: the challenge path falls through for real' (#66) from fix/the-challenge-path-falls-through-for-real into main 2026-09-26 12:22:21 +00:00
jschoubben f145d17fc8 route-proxy: the challenge path falls through for real
autocert's HTTPHandler answers 404 itself for a token it does not hold
and never consults its fallback on the challenge path — the
predecessor's exact fault, rediscovered live when Mailu's renewal died
behind this proxy on cutover day. tokenOrRoute probes each authority
against a buffered writer and hands a token none of them holds to plain
routing, so a consumer's own ACME client answers its own challenge
through an ordinary path-scoped route. Four tests pin it, including the
cache-key shape a restart-surviving token actually has.
2026-09-26 14:21:52 +02:00
jschoubben c3458a2546 Merge pull request 'route-proxy: a second authority for internal names, and https targets' (#64) from feat/route-proxy-internal-acme into main 2026-09-25 19:54:51 +00:00
jschoubben f8919e2079 Merge pull request 'builder: a clone may offer the forge's credential, through git's own store' (#65) from feat/builder-clones-with-the-forges-credential into main 2026-09-25 19:54:39 +00:00
jschoubben 6ac9013d6e builder: a clone may offer the forge's credential, through git's own store
A private repository could not be built: the builder clones anonymously,
and had no way to say who it is. It already holds exactly one credential
to exactly the right place — the package-registry binding and its sealed
secret, one gitea user whose password answers npm and git alike — so a
clone now offers that, and nothing new is minted or carried.

Offered, never pushed: the credential is written as a git
credential-store file (0600, in the workspace, never argv) and named
with -c credential.helper, so git itself decides when it applies — only
on an authentication challenge, and only for the URL it was written
for, scheme, host and port included. A public repository clones exactly
as before; a repository on any other host is never shown it. The same
store rides along on an artifact's own context clone, so a private
module with a private context builds too.
2026-09-25 21:47:32 +02:00
jochen 97448194ac Seats are a closed set, a seat's holder answers for what it delivers, and a build source may live on the git seat
Implements novox/hq ADR 0110 and 0111.

The seat set lives in internal/catalogue/seats.go: fourteen seats, each with a scope, what occupying
it delivers, and the record that made it one. A test asserts the count and a decision per entry, so
changing the set means finding the argument, as the host's vocabulary test does. The first set is
every seat already claimed — including the-private-network, which the network module claims from a
manifest composed in this repository's code, not from any module.json — plus npm-package-registry
(ADR 0109) and git (ADR 0111). A test parses every catalogue manifest and this repository's own and
fails on any refused claim, so closing the set refuses nothing in use.

ParseManifest now refuses a claim on a seat the mesh does not define, a seat claimed at another
scope, and a delivering seat claimed by a module that does not provide what it delivers. A
malformed claim is refused once, for being malformed.

Resolution: among several providers of a mesh provision, a pin still wins; then the holder of the
seat that delivers it; then the only provider; otherwise refused as before. ADR 0009's "never
guessed" holds — the seat is the choice made once, mesh-wide, rather than a pin per consumer node.
A provider now carries the module it came from, because a provider is a (node, module) pair and the
pair is what tells a holder from a neighbour on the same machine.

The planner's second pass is now given the first pass's holdings. Without them, a node consuming a
seat-delivered provision was refused there, and a refused node's own claims dropped out of what the
mesh holds — letting a second holder of one of its seats pass unrefused.

`seats [--json]` lists every seat, what it delivers, and each holder, derived from assignments
every time and never stored. Unheld seats are listed. A stored claim outside the set — possible
for a manifest registered before the set closed, since stored manifests are not re-validated — is
shown rather than hidden.

`build --self <owner>/<repo>` builds from a repository on the git seat's holder. The clone URL is
composed at build time from the holder's node and what it serves for git; the recorded source is the
path and the seat (migration 0032), never an address, so a moved forge changes nothing recorded.
Nobody holding the seat refuses self-hosted builds and says so; external URLs are unchanged. An
address passed with --self is refused rather than recorded as a path.

Replaces three foundation tests that defended the builder's carried package binding. The catalogue
removed that binding when the builder began requiring the registry through a real grant, so the
tests were already failing on main; they now assert the builder requires what the npm seat delivers
and carries no copy of its own, and that the forge holds the npm and git seats.

Verified: go vet clean; the whole suite passes against a throwaway Postgres (make postgres), the new
inventory tests included; gofmt clean apart from cmd/mesh-builder/stdout_test.go, which fails on
main too.
2026-09-25 20:48:10 +02:00
jschoubben 5fad1f89cf route-proxy: a second authority for internal names, and a target a route names the scheme of
Internal aliases were served over plain HTTP only — correctly refused a
public certificate (no public CA can validate a private name), and then
left with nothing. The mesh has two authorities for its two name spaces
(08-connectivity §2), so the proxy now takes an optional internal ACME
directory and dispatches at the handshake by the same question HostPolicy
already answers: which authority may certify this name at all.

A route may also say its target speaks https, with insecure for a backend
whose own certificate nothing would trust — the shape Mailu's webmail
front needs, and the exception: everything else the mesh hands this proxy
stays plain http on the private network.
2026-09-25 20:37:03 +02:00
jschoubben 7ffe6ce21b Merge pull request 'An image artifact may name its own build context, apart from the module's repository' (#62) from feat/an-artifact-may-name-its-own-build-context into main 2026-09-25 15:39:45 +00:00
jschoubben 20e57c3f51 an image artifact may name its own build context, apart from the module's repository
route-proxy's own Dockerfile documents the shape it has always needed and
never had: 'the proxy source is not vendored here... the build context is
the mesh-controller repository root, and this Dockerfile compiles
./examples/route-proxy from it.' Nothing in the mesh could do that — the
build command clones one repository and builds every artifact from
within it, so route-proxy has never once been built through the pipeline,
consistent with it never having been assigned anywhere. Found attempting
exactly that build tonight: 'stat go.mod: file does not exist', because
the context was mesh-catalog, which does not have one.

An image artifact may now carry a context: {repository, ref}, cloned
fresh alongside the module's own tree. The recipe (Dockerfile) is still
read from the module's own directory, at the module's own commit — only
docker build's own context argument moves. Packaging and source stay
exactly as separate as route-proxy's own comment already said they were,
now for real.
2026-09-25 17:39:27 +02:00
jschoubben 7bf23ea052 Merge pull request 'route-proxy serves a route's internal-name alias, never certifies it' (#61) from feat/route-proxy-serves-internal-alias into main 2026-09-25 15:24:36 +00:00
jschoubben 6da55bdfea route-proxy serves a route's internal-name alias, never certifies it
A route now consumed with two hosts when the mesh composed both — the
same host under internal-name reaches the same rule as its public name,
restoring the convenience a predecessor proxy gave for reaching a service
over the VPN without a public TLS round trip (the field composeName now
writes, feat/route-carries-internal-alias — this branch depends on that
one landing for internal-name to ever be populated; builds and tests
clean without it, just serves nothing extra).

Never certified: onlyWhatTheMeshSaid used routed(), which answered yes
for any host in the table regardless of how it got there. A new
eligibleForACME() checks a parallel 'public' set instead — every host
reached through a route's own name, never one reached only through its
internal-name — so an internal alias is proxied but never given its own
failing ACME order. routed() is unchanged and still used for the 404
message, which legitimately wants 'is this host served at all.'
2026-09-25 17:24:13 +02:00
jschoubben 752abaa81d Merge pull request 'A route composes its internal-network alias too, not only its public name' (#60) from feat/route-carries-internal-alias into main 2026-09-25 15:24:01 +00:00
jschoubben 2652287fe1 Merge pull request 'gitea's ssh port test matched a manifest mistake; resolver test used Names, not Machines' (#59) from fix/gitea-ssh-port-and-resolver-machines-test into main 2026-09-25 15:23:48 +00:00
jschoubben af26ed2e07 gitea's ssh port test matched a manifest mistake; resolver test used Names, not Machines
TestTheForgesSshPortIsGivenByTheNumberTheForgeCallsIt exercised a settings
override from '2222' to 222 — but 2222 was never a real port anywhere,
just a mistake in gitea's own manifest (fixed alongside this: listens.port
is now 22, the container's real internal sshd port, matching every other
module's convention, and ports declares 222:22 directly — 222 has always
been the real, fixed public git-ssh port, needing no per-node override).
Split into two tests: the fixed default with no override, and a genuine
override case for a hypothetical node whose predecessor used a different
number, keyed correctly by 22.

TestTheResolverAndWhatAsksItComposeOnOneMachine set Rendering.Names but
FactNodeZones reads Rendering.Machines (novox/hq issue 111 split the two
apart: every name the mesh serves vs. the machines subset) — a loose end
from that merge, not exercised until now. Both are the same map in this
test's scenario, so both fields are set.
2026-09-25 17:07:24 +02:00
jschoubben f996a6707e a route composes its internal-network alias too, not only its public name
Every cutover done on novox tonight (drive, files, files-api, git,
keycloak, umami) dropped the <label>.<node>.internal alias HAL always
paired with the public hostname — found only when the operator tested it
by hand. Not a security boundary (a predecessor proxy served both as a
convenience, reaching a service over the VPN without a public TLS round
trip, not as access control), so restoring it is composing the same
convenience the same way the public name already is: <label> joined to
the node's own private address (r.At), independently of whether a public
domain exists to join the other half to.

composeName's signature changes (publicDomain, internalDomain) but its
shape does not — additive, label-gated, apex-aware, exactly mirroring the
public half it already did. A contribution the mesh writes both names
into is the entire fix; route-adapter and route-proxy pick up internal-
name whenever they're updated to serve it, not before, so this alone
changes nothing about what is live on any node yet.
2026-09-25 16:51:38 +02:00
jschoubben 506426cf94 Merge pull request 'route-proxy: a route carries the policy applied to a request' (#58) from issue/116-route-proxy-has-no-auth-or-ip-restriction into main 2026-09-25 12:21:43 +00:00
jochen e11e1374bd route-proxy: a priority is an ordering, not a port
Found reviewing my own change before merging it, and it was load-bearing rather than cosmetic.

Priority was read with asPort, which caps at 65535. A rule declared above that silently became
priority 0 and stopped shadowing the route it exists to shadow. The one real rule this has to
reproduce is declared at 100000 — so path scoping and refusal would both have shipped looking
complete, passing their tests, and doing nothing on the only case that motivated them.

A priority is an ordering and has no range. asWhole takes any whole number the mesh wrote and
rejects a non-integral one, which was not meant as a priority.

Also: a host may now be routed on some paths and not others, which made the 404 dishonest — it
said "no route for this name" while listing that very name as served, a contradiction an
operator has to disbelieve the proxy to get past. An uncovered path now says so, and a name
that is genuinely not served still lists what is.

Two regression tests, both through the proxy rather than against the parser, because the parser
was where the bug looked fine.
2026-09-25 14:20:06 +02:00
jochen 008ce39ec0 route-proxy: a route carries the policy applied to a request
Implements novox/hq ADR 0108, closing issue 116. The proxy's request path was a host lookup
and a forward, so it applied nothing — while the ingress it replaces relies on four things it
had none of.

Path scoping came first because it is a prerequisite, not a sibling. The table mapped a host
to one target, so a host could not be routed two ways, and the refusal this issue turns on
matches a path on a host already routed to a workload. No amount of authentication or source
filtering would have made it expressible. The table is now host to an ordered list of rules,
matched on path prefix.

The order is total, not just by priority. Sorting on priority alone leaves rules that share
one in whatever order the map produced, so the same declaration would serve differently
between restarts — a fault that works, and works differently each time, which is the hardest
kind to believe when reported. Within a priority the longer path wins, which is also the
intuitive reading.

auth names a secret and never holds one. A declaration carrying a credential is refused
whole rather than served unprotected, so the option ADR 0108 rejected cannot return by
accident. A secret that cannot be read makes the route refuse and say so, rather than serve
the workload unprotected — a gate that cannot check is not a gate that opens, and the
alternative turns a missing file into a silently public admin surface.

Authentication costs one bcrypt comparison on every path including an unknown user, so an
unknown user is not measurably faster than a known one with a wrong password. That difference
is a way to enumerate a route's users from outside it.

Redirects keep the request's own path and query, or canonicalising one name onto another
would land every deep link on the front page and raise no error doing it.

Eleven tests, four of them for the capabilities and two for the failure modes that rot
quietly: the credential-in-a-declaration refusal, and the unreadable secret failing closed.
Nothing else breaks if those stop working, so nothing else would report it.

No new dependency: bcrypt comes from the x/crypto module already required.
2026-09-25 14:00:57 +02:00
jschoubben 856fabda04 Merge pull request 'contributes: a module's grant carries no value where it contributed several times' (#57) from fix/several-contributions-collide-in-grants-v2 into main 2026-09-24 17:39:55 +00:00
jschoubben 8fa5443862 contributes: a module's grant carries no value where it contributed several times
ContributionsFrom settled to whichever of a module's several contributions to
one requirement sorted first, arbitrarily — the grant minted for it then
carried that contribution's label and port under a credential the OTHER
contribution's consumer never sees, and collided with that same
contribution's own entry from contributions() besides.

Confirmed live: minio's two route contributions (files-api, files) produced
three entries in route-adapter's received file — files-api twice, once
credentialed and once not, files not credentialed at all. Every
single-contribution module (gitea, keycloak, umami) already mints an unused
credential for `route` too — route never needs one, by its own
documentation — but with exactly one contribution to match there was nothing
to collide with, so it never surfaced.

Where a module contributes more than once, there is no single value to
settle on. The module still asks, still gets its one credential — a pair
credential is not a place for a label or a port anyway — and each named
contribution reaches the provider on its own, unchanged.

No cleanup needed for the secret already minted live for minio+route: the
sealed blob is a random pair credential unrelated to Values, which is
recomputed fresh on every plan/push regardless.
2026-09-24 18:54:35 +02:00
jschoubben 3ece1a86d7 Merge pull request 'plan: show what a module would open and why' (#56) from feat/plan-shows-what-a-module-would-open into main 2026-09-24 16:42:24 +00:00
jschoubben dc8839246b Merge pull request 'contributes: a module may answer one requirement several times' (#55) from feat/several-route-contributions-per-module into main 2026-09-24 16:41:58 +00:00
jschoubben 524cc2a3ec plan: show what a module would open and why
Every module.json already declares a why for each port under listens,
but plan only ever used it to build the firewall's rule set — nothing
printed it. An operator deciding whether to assign a module had no way
to see what it would open without reading the manifest by hand.

plan <node> now prints each assigned module's listens entries — port,
protocol, source, and its why — right under the module line, so the
same text that feeds the firewall is visible at the point someone is
actually deciding whether to open it.
2026-09-24 18:40:30 +02:00
jschoubben f4bcb320fe contributes: a module may answer one requirement several times
A module's contributes was map[string]map[string]any — one JSON object key
per requirement, structurally exactly one contribution to "route" ever.
minio needs two public hostnames (the S3 API and the console), which is
two different contributions to route from one module, and nothing let it
say so.

This is the same shape of problem ADR 0094 solved for secrets (a module
needing several values from one provider that gives one per pair):
contributes now accepts either the ordinary {label, port} object, or an
object of local names to several such objects. Detected per requirement
key by what's inside, since (unlike secrets' string-vs-object split) both
shapes are JSON objects: an ordinary contribution's fields are scalars, the
several-instance shape is local-name -> object. Confirmed against every
module.json in mesh-catalog before relying on that split.

Both route-proxy and the migration-era route-adapter already key generated
routers off the composed hostname (Values["name"]), not the module name,
so two contributions with the same From reach them as two independent
routes with no changes needed on the receiving side.
2026-09-24 18:36:08 +02:00
jschoubben caf9746759 mesh-controller: mount the broker TLS directory bind, not the old named volume
Found checking whether the named volumes mesh-catalog PR #54/#55 replaced
are actually unused before considering them safe to remove -- this repo
has its own independent volumes declaration for the same TLS material
(mesh-controller reads it directly, not through lavinmq's own resource),
and it still named the old mesh-broker-tls volume.

Right now the content is identical -- copied once during the conversion.
If the cert ever rotates, lavinmq writes the new directory and this would
keep reading stale content from the volume nothing else updates.

Checked both repos for any other reference to the four converted volume
names (mesh-store-data, mesh-broker-data, mesh-broker-tls,
mesh-registry-data): this was the only one.
2026-09-24 17:19:48 +02:00
jschoubben 6090953843 Merge pull request 'Tell the resolver the machines, not the names the mesh merely serves' (#53) from fix/the-resolver-is-told-machines-not-routes into main 2026-09-23 23:32:22 +00:00
jschoubben 2277583e99 Tell the resolver the machines, not the names the mesh merely serves
The map the control plane hands a resolution holds both: the machines, and every name
the mesh was told to route to whichever machine serves it. A container's hosts wants all
of it, so a routed name resolves to the proxy. A resolver's zones want only the machines:
told the mesh's suffix is its own it answers authoritatively for everything under it and
forwards none of it, so a routed name with the suffix appended — drive.example.test.internal
— is a name nobody will ever ask for, standing beside the machines and looking as real.

Found composing the resolver's first assignment on a live machine, before pushing it.
hq issue 111.
2026-09-24 01:31:11 +02:00
jschoubben 6bf42025e1 Merge pull request 'Give the resolver the mesh's suffix as a local domain and a module its machine's address' (#52) from convert/dnsmasq-from-hal into main 2026-09-23 23:13:03 +00:00
jschoubben 0d8264ff55 Give the resolver the mesh's suffix as a local domain and a module its machine's address
hal dnsmasq-app conversion, hq 08-connectivity. Converting the resolver from the module it
replaces made it forward what it cannot answer, which is what the predecessor's does, and
that found two things the controller did not say.

A resolver that forwards must not send a mesh name it does not know upstream: the
`node-zones` fact now carries `local=/<suffix>/` beside the wildcards, written here rather
than in the daemon's configuration because the suffix is the mesh's choice and this file is
the one place the mesh writes what it chose. The default lives in one helper now instead of
being spelled in two functions.

The predecessor points the container runtime's `dns` at the machine's own tunnel address —
a container cannot reach the machine's loopback. A module writing that key needs the
address, and `${machine:at}` is the machine's name; a runtime's resolver list cannot be a
name it would need that resolver to look up. So a module may say `${machine:address}`: what
`at` resolves to, read from the same names the hosts file and the wildcards are written
from, absent — and refused — off the network like `at` is.

The `mesh-resolver` and `resolver-data` constants go: nothing provided or consumed either,
the fact and `mesh-addressing` are the mechanism, and a requirement nothing provides is
refused at resolution.

Tests: the catalogue's dnsmasq, resolv-conf and resolved-split-dns manifests are parsed
and composed as a machine would receive them — fixed upstreams, no-resolv, 127.0.0.1, the
machines file, the runtime's key, the pair that decides what a machine asks refused on one
node; and on a real mesh the resolver's machines file is composed with a wildcard per
machine on the network and composed again without one that left, mirroring the hosts fact.
2026-09-24 01:10:15 +02:00
jschoubben 6073e94a4f Merge pull request 'Adopt the predecessor's tunnel in place: its range, its address, its peers (hq ADR 0105)' (#49) from feat/adopt-the-tunnel into main 2026-09-23 22:38:31 +00:00
jschoubben 4566c5c9aa Adopt the tunnel as a mesh fact, refuse a mismatched takeover, and rekey after enrolment
Review of the ADR 0105 build (hq ADR 0105). Four things it got wrong and one
path it lacked:

- A predecessor spoke's tunnel names one peer, the hub, routed the whole
  range; recording refused it and the whole enrolment failed. Range-routed
  peers are skipped now — only the hub's peers are ever carried.
- The range and the carried peers were conditions on the node being adopted,
  so converging the hub would have renumbered the mesh and dropped the peers
  still reaching it. They are facts of the tunnel record now, mode aside; the
  takeover alone is declared to an adopted node. Converging the hub is refused
  while a carried peer has not enrolled, naming it.
- A push composed a takeover for a hub whose address or endpoint disagreed
  with the tunnel, which would have the host stop the found interface and
  raise the mesh's where no peer listens. The graph refuses to compose it,
  naming both and the placement that fixes it.
- The host's account said taken or not; "found down and the mesh's not up"
  read as not taken. Three states now, and an account on every takeover.
- A hub that enrolled before this feature holds a key of its own, and
  re-enrolling would rotate every key the mesh sealed credentials to. A node
  now rekeys in a report, signed with its identity key over the key it
  leaves, the key it takes and the tunnel; the mesh verifies against the live
  key, refuses a stale or foreign proof, records key and tunnel, and moves a
  hub to the tunnel's address. `overlay show` names the path for a hub that
  found no tunnel.

Also: a carried IPv6 peer is routed /128, and identity.ForTest exists so the
link can be tested against a real identity store.
2026-09-24 00:02:07 +02:00
jschoubben 7ef7669c0c Merge pull request 'An address is read from the node's settings where it is used, never recorded with a port (hq issue 102)' (#50) from fix/addresses-follow-the-node into main 2026-09-23 21:55:03 +00:00
jschoubben cdd3638312 The control plane's own manifest says where the node put the store and the broker
Beside each sealed connection genesis wrote, the port this machine put the
seat's holder at: `${seat:mesh-store:5432}` for the three stores,
`${seat:mesh-broker:…}` for the bus, the plain AMQP port and the management API.
Filled from the node's settings when the control plane composes its own
declaration; empty — the sealed value stands — when the mesh has nothing to add.

On its own, after the commit before it is built and running: a control plane
that does not know the placeholder passes it through as the value, and this
manifest is composed by whatever control plane is running when it is pushed.
The reader ignores an unfilled placeholder either way, and a test holds it to
ignoring exactly what this manifest says.

novox/hq 04-ISSUES/102
2026-09-23 23:49:55 +02:00
jschoubben e07b56ce43 An address is read from the node's settings where it is used, never recorded with a port
Three readers did not follow a moved foundation port (novox/hq 04-ISSUES/102),
and each took the control-node down in its own way: the control plane's own
store and broker connections, sealed at genesis with the port inside; and every
build the mesh ever recorded, kept as `<registry>:<port>/<module>/<artifact>@…`.

The control plane cannot open its own sealed connections to move a port, and it
cannot bind the store as a consumer would — a binding mints a credential. So its
settings get a third twin, `NAME_PORT`, read on top of the sealed value by the
store, the broker, the management API and the bus connection, and filled into
its container by a placeholder that names a seat, `${seat:mesh-store:5432}`,
from the node's given or mesh-assigned ports — never the manifest's number, and
empty when the mesh has nothing to add, so what genesis wrote stands. A value
that is still a placeholder is nothing said, aloud: the manifest naming it lands
in the next commit, once every control plane that composes it knows it.

A build is now recorded by digest and path — `artifact-store://<module>/<artifact>@…`
— and the store's address is composed in where a reference is used: the
declaration, the trust file, the bases a build is handed, a replay to the
catalogue. Over the network as `<node>.internal:<port>`; on the store's own node
before any network exists — every genesis push before its "network" step — by
loopback. A reference recorded before this, with an address, is re-routed the
same way when the mesh built it. The trust file and every provider's address
come from one derivation: the node's given port, over the mesh's assignment,
over the manifest's number.

novox/hq 04-ISSUES/102
2026-09-23 23:49:31 +02:00
jschoubben 1b5ccf4165 Merge pull request 'The artifact store's seat is one per mesh, and the test says so from the catalogue' (#51) from fix/the-artifact-store-is-one-per-mesh into main 2026-09-23 21:42:33 +00:00
jschoubben 26690d89f1 The artifact store's seat is one per mesh, and the test says so from the catalogue
Review of the registry work found the seat node-scoped: a second `distribution` on another
machine resolved cleanly there, and only afterwards did the mesh notice `artifact-store`
offered by two nodes, with every consumer elsewhere refusing to choose. A node-scoped
requirement with one candidate installs that candidate, so anything that wanted the store
beside it would have raised a fresh, empty store on the wrong machine first.

The claim is mesh-scoped in mesh-catalog now; this holds the catalogue's manifest to it —
a second store anywhere is refused by name, where it is assigned.
2026-09-23 23:40:53 +02:00
jschoubben 3c836f0abb Adopt the predecessor's tunnel in place: its range, its address, its peers
On an adopted hub the private network takes over the tunnel it finds rather
than running beside it (hq ADR 0105): two tunnels leave the mesh's unreachable
through the provider's filter, so no machine can ever join.

The node presents the found tunnel when it enrols, under the key it took as
its own; the inventory records it (node.tunnel, tunnel_peer — migration 0031)
and the mesh composes from it: the overlay's range is the adopted tunnel's,
the hub is placed at the tunnel's address on the tunnel's port, and every
peer the tunnel had is carried in the hub's peer list as a peer of the
tunnel, not a node of the mesh, until a node enrols with that key — which
then keeps the address the tunnel had for it. A fresh node never gets an
address the tunnel holds. The hub's declaration tells the host which unit to
take over; the host's account of carrying it is recorded and shown.

Every reader of the range follows the setting; nothing stores it. A found
tunnel under another key is recorded and not adopted, so ADR 0100's
non-overlap rule keeps applying where a tunnel is left running beside the
mesh's. A lab bed and test skeleton for "How it is checked" are under lab/.
2026-09-23 23:26:34 +02:00
75 changed files with 7089 additions and 434 deletions
+49
View File
@@ -24,6 +24,7 @@ import (
"encoding/json"
"errors"
"fmt"
"net/url"
"os"
"os/signal"
"strings"
@@ -202,6 +203,7 @@ func answer(ctx context.Context, channel *amqp.Channel, publisher builder.Publis
// not after a clone that then fails at npm ci.
built, err = builder.Build(ctx, builder.Command, publisher,
request.Repository, request.Path, request.Ref, workspace, request.Held, npmrc,
forgeFrom(),
func(step, message string) {
fmt.Fprintf(os.Stderr, " [%s] %s\n", step, message)
})
@@ -367,6 +369,53 @@ func packagesFrom() (builder.Npmrc, error) {
return builder.Npmrc{Scope: scope, Registry: registry, Token: secret}, nil
}
// forgeFrom is the git credential this builder may offer a clone, composed from the same binding
// and sealed secret its package-registry half already reads: the forge that answers npm is the
// forge that hosts the repositories, and its provisioner applies one password to one user for
// both. Anything missing means no credential, and every clone stays anonymous — which is all a
// mesh of public repositories ever needs.
//
// The URL names the binding's own address — the machine the mesh says the forge is on — so a
// private repository is registered and built by that address, and a clone of anything else is
// never shown this credential (git's credential store matches the whole origin).
func forgeFrom() builder.GitCredential {
path := strings.TrimSpace(os.Getenv("MESH_PACKAGE_BINDING"))
if path == "" {
return builder.GitCredential{}
}
raw, err := os.ReadFile(path)
if err != nil {
return builder.GitCredential{}
}
var told struct {
At string `json:"at"`
As string `json:"as"`
Serves map[string]any `json:"serves"`
}
if err := json.Unmarshal(raw, &told); err != nil || told.At == "" || told.As == "" {
return builder.GitCredential{}
}
secret := strings.TrimSpace(os.Getenv("MESH_NPM_TOKEN"))
if file := strings.TrimSpace(os.Getenv("MESH_NPM_TOKEN_FILE")); file != "" {
if raw, err := os.ReadFile(file); err == nil {
secret = strings.TrimSpace(string(raw))
}
}
if secret == "" {
return builder.GitCredential{}
}
scheme := "https"
if s, ok := told.Serves["scheme"]; ok {
scheme = fmt.Sprintf("%v", s)
}
host := told.At
if port, ok := told.Serves["port"]; ok {
host = fmt.Sprintf("%s:%v", told.At, port)
}
made := url.URL{Scheme: scheme, User: url.UserPassword(told.As, secret), Host: host}
return builder.GitCredential{URL: made.String()}
}
func short(commit string) string {
if len(commit) > 8 {
return commit[:8]
+1
View File
@@ -89,6 +89,7 @@ func buildOnce(ctx context.Context, args []string) error {
return err
}
built, buildErr := builder.Build(ctx, builder.Command, publisher, repository, *path, *ref, where, bases, npmrc,
forgeFrom(),
func(step, message string) { fmt.Fprintf(os.Stderr, " [%s] %s\n", step, message) })
if buildErr != nil {
return buildErr
+8 -1
View File
@@ -46,9 +46,16 @@ func assign(ctx context.Context, open *stores, node, module string) (string, err
return "", err
}
defer release()
if err := open.inventory.Assign(ctx, node, module); err != nil {
fresh, err := open.inventory.Assign(ctx, node, module)
if err != nil {
return "", err
}
if !fresh {
// Nothing changed, and saying "is assigned" would read as an action. One node runs one
// of each — the module's name is the assignment's identity (novox/hq ADR 0115).
return fmt.Sprintf("%s already runs %s — one node runs one of each (ADR 0115); nothing changed",
node, module), nil
}
said := fmt.Sprintf("%s is assigned %s", node, module)
plan, _, err := planFor(ctx, open, node)
if err != nil {
+130 -7
View File
@@ -8,6 +8,7 @@ import (
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/licences"
"github.com/novox/mesh-controller/internal/link"
)
@@ -153,7 +154,7 @@ func TestTheControlPlaneIsToldWhereTheNodePutTheStoreAndTheBroker(t *testing.T)
if err != nil {
t.Fatal(err)
}
control, err := m.Resolve([]catalogue.Built{{Name: "server", Kind: catalogue.ArtifactImage,
control, err := withSeatPorts(m).Resolve([]catalogue.Built{{Name: "server", Kind: catalogue.ArtifactImage,
Reference: "registry.example/control@" + aDigest}})
if err != nil {
t.Fatal(err)
@@ -198,15 +199,137 @@ func TestTheControlPlaneIsToldWhereTheNodePutTheStoreAndTheBroker(t *testing.T)
t.Fatal("the control plane's container is not in its own node's declaration")
}
for key, want := range map[string]string{
"MESH_STORE_INVENTORY_PORT": "6852",
"MESH_STORE_IDENTITY_PORT": "6852",
"MESH_STORE_LICENCES_PORT": "6852",
"MESH_BROKER_AMQP_PORT": "5679",
"MESH_BROKER_ADDRESS_PORT": "5671",
"MESH_BROKER_MANAGEMENT_PORT": "15672",
"MESH_STORE_INVENTORY_PORT": "6852",
"MESH_STORE_IDENTITY_PORT": "6852",
"MESH_STORE_LICENCES_PORT": "6852",
"MESH_BROKER_AMQP_PORT": "5679",
// Neither given nor assigned by the mesh: the manifest's own number is NOT the answer,
// because the sealed value beside it carries the port genesis wrote (finding F3).
"MESH_BROKER_ADDRESS_PORT": "",
"MESH_BROKER_MANAGEMENT_PORT": "",
} {
if env[key] != want {
t.Errorf("the control plane is told %s=%v; the node says %q", key, env[key], want)
}
}
}
// withSeatPorts is the control plane's manifest with the seat placeholders in its environment —
// added here until module.json carries them (the manifest lands one commit after the code that
// fills it, so a control plane one build behind never sees a placeholder it cannot fill).
func withSeatPorts(m catalogue.Manifest) catalogue.Manifest {
seatPorts := map[string]string{
"MESH_STORE_INVENTORY_PORT": "${seat:mesh-store:5432}",
"MESH_STORE_IDENTITY_PORT": "${seat:mesh-store:5432}",
"MESH_STORE_LICENCES_PORT": "${seat:mesh-store:5432}",
"MESH_BROKER_AMQP_PORT": "${seat:mesh-broker:5672}",
"MESH_BROKER_MANAGEMENT_PORT": "${seat:mesh-broker:15672}",
"MESH_BROKER_ADDRESS_PORT": "${seat:mesh-broker:5671}",
}
out := m
out.Resources = nil
for _, r := range m.Resources {
if r["type"] != "container" {
out.Resources = append(out.Resources, r)
continue
}
copied := map[string]any{}
for k, v := range r {
copied[k] = v
}
env := map[string]any{}
if had, ok := r["env"].(map[string]any); ok {
for k, v := range had {
env[k] = v
}
}
for k, v := range seatPorts {
if _, said := env[k]; !said {
env[k] = v
}
}
copied["env"] = env
out.Resources = append(out.Resources, copied)
}
return out
}
// aLoneNode is one capable machine with nothing placed on any network — the control-node during
// genesis, before the "network" step, which is after the store, the broker, the vault and the
// catalogue have each been built and pushed (finding F2).
func aLoneNode(t *testing.T) *stores {
t.Helper()
inventory.ForTest(t)
licences.ForTest(t)
open, err := openStores(t.Context())
if err != nil {
t.Fatal(err)
}
t.Cleanup(open.Close)
for _, m := range provided {
if err := open.inventory.Provide(t.Context(), m); err != nil {
t.Fatal(err)
}
}
record, err := open.inventory.AddNode(t.Context(), "anchor")
if err != nil {
t.Fatal(err)
}
reported, _ := json.Marshal(map[string]any{"capabilities": []map[string]any{
{"name": "container-runtime", "present": true}}})
var profile map[string]any
_ = json.Unmarshal(reported, &profile)
if err := open.inventory.RecordProfile(t.Context(), record.ID, profile); err != nil {
t.Fatal(err)
}
if err := open.inventory.RecordSealingKey(t.Context(), record.ID, aPublicKey(t)); err != nil {
t.Fatal(err)
}
return open
}
// **Before the network exists, the store's own node reaches it by loopback** — never refused,
// never handed the scheme: a genesis pushes the store, the broker, the vault and the catalogue to
// a node on no network, and builds the catalogue on a base it must be able to pull.
func TestOnANodeWithNoNetworkTheStoreIsReachedByLoopback(t *testing.T) {
open := aLoneNode(t)
ctx := t.Context()
register(t, open, aStore())
register(t, open, catalogue.Manifest{Module: "builder", Version: "1",
Requires: []string{catalogue.ArtifactStoreProvision},
Resources: []map[string]any{{"id": "server", "type": "container", "name": "mesh-builder",
"image": "registry.example/mesh-builder@" + aDigest}}})
if err := open.inventory.RecordBuild(ctx, inventory.Build{
ID: "b1", Repository: "r", Module: "postgres", Commit: "abc",
Made: []inventory.Artifact{{Name: "runtime", Kind: "image",
Reference: catalogue.ArtifactStoreScheme + "postgres/runtime@" + aDigest}},
}); err != nil {
t.Fatal(err)
}
register(t, open, catalogue.Manifest{Module: "postgres", Version: "1",
Resources: []map[string]any{{"id": "runtime", "type": "container", "name": "mesh-postgres",
"image": catalogue.ArtifactStoreScheme + "postgres/runtime@" + aDigest}}})
for _, module := range []string{"distribution", "builder", "postgres"} {
if _, err := assign(ctx, open, "anchor", module); err != nil {
t.Fatal(err)
}
}
if err := open.inventory.SetSettings(ctx, "anchor", "distribution",
map[string]any{catalogue.PortsSetting: map[string]any{"5000": 5100}}); err != nil {
t.Fatal(err)
}
var image any
for _, r := range composed(t, open, "anchor").Resources {
if r["id"] == "postgres.runtime" {
image = r["image"]
}
}
if image != "127.0.0.1:5100/postgres/runtime@"+aDigest {
t.Fatalf("on the store's own node, off any network, the image is fetched as %v", image)
}
held := heldBy(ctx)
if got := held["postgres/runtime"]; got != "127.0.0.1:5100/postgres/runtime@"+aDigest {
t.Fatalf("a builder beside the store is handed the base %q", got)
}
}
+65 -1
View File
@@ -14,6 +14,7 @@ import (
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/overlay"
)
// A node is adopted or converged (novox/hq ADR 0100), and it is said to be adopted wherever the
@@ -45,6 +46,9 @@ func showMode(ctx context.Context, inv *inventory.Inventory, node inventory.Node
return nil
}
fmt.Printf(" firewall found %s\n", orNone(said.Firewall))
if err := showTunnel(ctx, inv, node.Name); err != nil {
return err
}
if len(said.Held) == 0 {
fmt.Printf(" holding nothing found\n")
}
@@ -63,6 +67,44 @@ func showMode(ctx context.Context, inv *inventory.Inventory, node inventory.Node
return nil
}
// showTunnel is the node show lines about the tunnel an adopted node found and carried (novox/hq
// ADR 0105): what it presented at enrolment, and what it last said about taking it over.
func showTunnel(ctx context.Context, inv *inventory.Inventory, name string) error {
tunnel, err := inv.TunnelOf(ctx, name)
if errors.Is(err, inventory.ErrNoTunnel) {
return nil
}
if err != nil {
return err
}
fmt.Printf(" tunnel found %s on port %d, %s in %s, %d peer(s)\n",
tunnel.Interface, tunnel.Port, tunnel.Address, tunnel.Range, len(tunnel.Peers))
carried, said, err := inv.CarriedTunnelOf(ctx, name)
if err != nil {
return err
}
switch {
case !said:
fmt.Printf(" %-17s not yet taken over — the node has not said so\n", "")
case carried.State == inventory.CarriedTaken:
fmt.Printf(" %-17s taken over: %s is down and disabled, never flushed; the mesh's interface "+
"runs with its key, port and %d peer(s)\n", "", carried.Interface, carried.Peers)
case carried.State == inventory.CarriedDown:
fmt.Printf(" %-17s TUNNEL DOWN: %s is stopped and the mesh's interface is not up — the peers "+
"reach nothing. On the machine: systemctl start %s\n", "", carried.Interface,
"wg-quick@"+carried.Interface)
default:
fmt.Printf(" %-17s NOT taken over: %s is still the interface the peers reach\n", "", carried.Interface)
}
if said && carried.Note != "" {
fmt.Printf(" %-17s %s\n", "", carried.Note)
}
if said && carried.Kept != "" {
fmt.Printf(" %-17s its configuration's original kept at %s\n", "", carried.Kept)
}
return nil
}
func orNone(s string) string {
if s == "" {
return "none reported"
@@ -213,6 +255,28 @@ func converge(ctx context.Context, open *stores, node string, yes bool, digest s
return "", fmt.Errorf("%s still holds what it found, and a service is taken on its own, "+
"never by the flip:\n%s", node, strings.Join(holding, "\n"))
}
// And refused while a peer of the tunnel this hub took over has not enrolled (novox/hq ADR
// 0105): the flip loads the derived filter and retires the found firewall, and a machine the
// mesh has no record of is not one the filter admits — it would go dark.
if _, hubName, adopted, err := inv.AdoptedTunnel(ctx); err != nil {
return "", err
} else if adopted && hubName == node {
carried, err := inv.CarriedPeers(ctx)
if err != nil {
return "", err
}
var waiting []string
for _, c := range carried {
if c.EnrolledAs == "" {
waiting = append(waiting, fmt.Sprintf(" %s at %s", overlay.CarriedName(c.PublicKey), c.Address))
}
}
if len(waiting) > 0 {
return "", fmt.Errorf("%s carries peers of the tunnel it took over that have not enrolled, and "+
"converging would cut them off — enrol each first (`overlay show` says which are enrolled):\n%s",
node, strings.Join(waiting, "\n"))
}
}
shelf, err := inv.Catalogue(ctx)
if err != nil {
@@ -291,7 +355,7 @@ func converge(ctx context.Context, open *stores, node string, yes bool, digest s
strings.Join(assigned, ", "))
}
if !slices.Contains(assigned, filter) {
if err := inv.Assign(ctx, node, filter); err != nil {
if _, err := inv.Assign(ctx, node, filter); err != nil {
return "", err
}
if _, _, err := planFor(ctx, open, node); err != nil {
+44 -12
View File
@@ -46,25 +46,35 @@ func buildCommand(ctx context.Context, args []string) error {
// retype each repository is asking them to be the loop. Naming a repository and asking which
// ones need building are different requests, so they are not combined.
behind := set.Bool("behind", false, "every module the mesh holds older than its source has")
// A repository on the mesh's own forge, named by its path there (novox/hq ADR 0111). Without it
// the repository is external, cloned exactly as given — see source.go.
self := set.Bool("self", false, "the repository is a path on the forge holding the git seat")
positionals, err := parseAround(set, args)
if err != nil {
return err
}
if *behind {
if len(positionals) != 0 {
if len(positionals) != 0 || *self {
return errors.New("build <repository> or build --behind, not both: one names a " +
"repository and the other asks which need building")
}
return buildBehind(ctx, *wait)
}
if len(positionals) != 1 {
return errors.New("build <repository> [--ref R] [--wait D] [--dry-run]")
return errors.New("build <repository> [--self] [--path P] [--ref R] [--wait D] [--dry-run]")
}
source := buildSource{Repository: positionals[0]}
if *self {
if err := onASeat(source.Repository); err != nil {
return err
}
source.Seat = gitSeat
}
if *dryRun {
return buildAndShow(ctx, positionals[0], *path, *ref, *wait)
return buildAndShow(ctx, source, *path, *ref, *wait)
}
return buildOne(ctx, positionals[0], *path, *ref, *wait)
return buildOne(ctx, source, *path, *ref, *wait)
}
// buildFrom turns what a builder said into what the mesh keeps.
@@ -325,7 +335,8 @@ func buildBehind(ctx context.Context, wait time.Duration) error {
// Its own recorded ref, not its head commit: a module tracking a branch should be built
// from that branch, and pinning to the commit the mesh happened to notice would quietly
// turn a tracked branch into a pin.
if err := buildOne(ctx, e.Source.Repository, e.Source.Path, e.Source.Ref, wait); err != nil {
source := buildSource{Repository: e.Source.Repository, Seat: e.Source.Seat}
if err := buildOne(ctx, source, e.Source.Path, e.Source.Ref, wait); err != nil {
fmt.Printf(" %v\n", err)
failed = append(failed, e.Manifest.Module)
}
@@ -343,7 +354,14 @@ func buildBehind(ctx context.Context, wait time.Duration) error {
// buildOne asks a build machine for one repository and records everything that came back.
//
// Separated from the command so `--behind` can walk a list without a second path to the same act.
func buildOne(ctx context.Context, repository, path, ref string, wait time.Duration) error {
func buildOne(ctx context.Context, source buildSource, path, ref string, wait time.Duration) error {
// Before anything is asked of a builder: a source on a seat nobody holds is refused here, with
// the reason, rather than sent to a machine to fail at `git clone`.
repository, err := cloneFrom(ctx, source)
if err != nil {
return err
}
ident, err := openIdentity(ctx)
if err != nil {
return err
@@ -365,7 +383,10 @@ func buildOne(ctx context.Context, repository, path, ref string, wait time.Durat
Ref: ref,
Held: heldBy(ctx),
}
fmt.Printf("asked for %s", request.Repository)
fmt.Printf("asked for %s", source)
if source.Seat != "" {
fmt.Printf(" (%s)", repository)
}
if path != "" {
fmt.Printf(" at %s", path)
}
@@ -414,11 +435,18 @@ func buildOne(ctx context.Context, repository, path, ref string, wait time.Durat
}
// Recorded with where it came from, so "is this current?" is answerable without building it
// again (novox/hq ADR 0009).
if err := inv.RegisterModule(ctx, manifest, inventory.Source{
// again (novox/hq ADR 0009). **For a source on a seat, as the path and the seat, never the URL
// just cloned** (ADR 0111): the URL is where the forge runs today, and recording it would put
// the forge's address back into every module built from it. The build log above keeps the URL,
// because that is what was cloned.
recorded := inventory.Source{
Repository: result.Repository, Path: result.Path, Ref: result.Ref,
BuiltFrom: result.Commit, Head: result.Commit,
}); err != nil {
}
if source.Seat != "" {
recorded.Repository, recorded.Seat = source.Repository, source.Seat
}
if err := inv.RegisterModule(ctx, manifest, recorded); err != nil {
return err
}
fmt.Printf("\n%s %s, built on %s from %s\n",
@@ -428,7 +456,11 @@ func buildOne(ctx context.Context, repository, path, ref string, wait time.Durat
}
// buildAndShow builds and prints the manifest without recording anything.
func buildAndShow(ctx context.Context, repository, path, ref string, wait time.Duration) error {
func buildAndShow(ctx context.Context, source buildSource, path, ref string, wait time.Duration) error {
repository, err := cloneFrom(ctx, source)
if err != nil {
return err
}
ident, err := openIdentity(ctx)
if err != nil {
return err
@@ -510,7 +542,7 @@ func heldBy(ctx context.Context) map[string]string {
fmt.Fprintf(os.Stderr, "could not read what this mesh has built: %v\n", err)
return nil
}
address, err := whereTheStoreIs(ctx, open.inventory)
address, err := whereABuilderReachesTheStore(ctx, open.inventory)
if err != nil {
fmt.Fprintf(os.Stderr, "could not find the artifact store on this mesh's network, so a "+
"module naming a base will be handed a reference nothing can fetch: %v\n", err)
@@ -0,0 +1,33 @@
package main
// The broker opening belongs only on the node that listens on it (novox/hq: it leaked onto
// every enrolled node's declaration, opening a from-anywhere hole for a port nothing there
// serves). foundationPortsFor is the scope.
import (
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
)
func TestTheBrokerHostGetsTheFoundationOpening(t *testing.T) {
broker := catalogue.Manifest{Module: "lavinmq", Listens: []catalogue.Listening{
{Port: 5671, Protocol: "tcp", From: "mesh"},
{Port: 5672, Protocol: "tcp", From: "mesh"},
}}
got := foundationPortsFor(5671, []catalogue.Manifest{broker})
if len(got) != 1 || got[0] != 5671 {
t.Fatalf("the node that listens on the broker port keeps it; got %v", got)
}
}
func TestANodeThatOnlyDialsTheBrokerGetsNoOpening(t *testing.T) {
// ace's set: things that reach the broker as a client, none listening on 5671.
ace := []catalogue.Manifest{
{Module: "plex", Listens: []catalogue.Listening{{Port: 32400, Protocol: "tcp", From: "anywhere"}}},
{Module: "postgres", Listens: []catalogue.Listening{{Port: 5432, Protocol: "tcp", From: "mesh"}}},
}
if got := foundationPortsFor(5671, ace); got != nil {
t.Fatalf("a node that only dials out opens nothing for the broker; got %v", got)
}
}
+3
View File
@@ -114,6 +114,8 @@ func run() error {
return planCommand(ctx, args[1:])
case "push":
return pushCommand(ctx, args[1:])
case "seats":
return seatsCommand(ctx, args[1:])
case "status":
return statusCommand(ctx, args[1:])
case "version":
@@ -157,6 +159,7 @@ func usage() {
upgrade <name> roll-out [--together] ...send it to the machines running it
upgrade <name> record ...record that they are behind, and send nothing
status [--json] what is wrong, what is quiet, and what is out of date
seats [--json] every seat this mesh defines, what it delivers, and who holds it
board [--listen ADDR] the same three questions, as a page that holds nothing
api --issuer URL [--listen A] assign and unassign over http, for a surface that is not here
assign <node> <module> put a module on a node
+1 -1
View File
@@ -73,7 +73,7 @@ func aMesh(t *testing.T) *stores {
if err := open.inventory.RecordOverlayKey(t.Context(), record.ID, aPublicKey(t)); err != nil {
t.Fatal(err)
}
if err := open.inventory.Assign(t.Context(), name, overlay.Name); err != nil {
if _, err := open.inventory.Assign(t.Context(), name, overlay.Name); err != nil {
t.Fatal(err)
}
}
+232 -18
View File
@@ -7,6 +7,7 @@ import (
"fmt"
"os"
"sort"
"strconv"
"strings"
"time"
@@ -21,16 +22,33 @@ import (
// cheapest next step. That is how novox/hq ADR 0001 records `hal/sdk` reaching 34,636:
// nothing in it was wrong, and no one edit was the one that should have been a new file.
func overlayCIDR() string {
if v := strings.TrimSpace(os.Getenv(OverlayCIDRVar)); v != "" {
return v
// DefaultOverlayCIDR is the range the mesh allocates from when nothing says another.
const DefaultOverlayCIDR = "10.42.0.0/16"
// overlayRange is the range the mesh allocates node addresses from.
//
// **The adopted tunnel's range first** (novox/hq ADR 0105): a hub that took over the tunnel it
// found is at that tunnel's address, its peers are at theirs, and every node's address is
// composed from the same range — the hub's, and every binding, hosts entry and endpoint derived
// from it. Those are readers of this; none of them stores the range. Without an adopted tunnel,
// the range genesis was told, or the default.
func overlayRange(ctx context.Context, inv *inventory.Inventory) (string, error) {
tunnel, _, adopted, err := inv.AdoptedTunnel(ctx)
if err != nil {
return "", err
}
return "10.42.0.0/16"
if adopted {
return tunnel.Range, nil
}
if v := strings.TrimSpace(os.Getenv(OverlayCIDRVar)); v != "" {
return v, nil
}
return DefaultOverlayCIDR, nil
}
func overlayCommand(ctx context.Context, args []string) error {
if len(args) == 0 {
return errors.New("overlay place <node> [flags], or overlay show")
return errors.New("overlay place <node> [flags], overlay name <address> <name>, or overlay show")
}
// Answered before anything is opened. A message about which command to use should not need a
// database to say so, and needing one turns a redirect into a connection error.
@@ -51,12 +69,29 @@ func overlayCommand(ctx context.Context, args []string) error {
return overlayPlace(ctx, inv, args[1:])
case "show":
return overlayShow(ctx, open)
case "name":
return overlayName(ctx, inv, args[1:])
default:
return fmt.Errorf("overlay has no %q; it has place and show", args[0])
return fmt.Errorf("overlay has no %q; it has place, name and show", args[0])
}
}
// overlayName is the operator saying which machine a carried address is (novox/hq issue 112),
// so the mesh answers for its name until the machine enrols and verifies it.
func overlayName(ctx context.Context, inv *inventory.Inventory, args []string) error {
if len(args) != 2 {
return errors.New("overlay name <carried-address> <node-name>")
}
address, name := args[0], args[1]
if err := inv.NamePeer(ctx, address, name); err != nil {
return err
}
fmt.Printf("the peer at %s is %s until it enrols — the mesh answers for %s.<suffix> from the "+
"operator's word, and enrolment under this key must use this name\n", address, name, name)
return nil
}
func overlayPlace(ctx context.Context, inv *inventory.Inventory, args []string) error {
if len(args) == 0 {
return errors.New(
@@ -110,16 +145,46 @@ func overlayPlace(ctx context.Context, inv *inventory.Inventory, args []string)
}
}
// A hub that took over a tunnel listens on that tunnel's port — it is what the peers dial, and
// the reason the port is worth having (novox/hq ADR 0105). An endpoint on another port would
// have the mesh's interface up where no peer is listening for it.
found, err := inv.NodeByName(ctx, node)
if err != nil {
return err
}
var tunnel inventory.Tunnel
adoptsTunnel := false
if *hub && found.Adopted {
if t, err := inv.TunnelOf(ctx, node); err == nil {
tunnel = t
placed, _ := inv.Overlays(ctx)
for _, o := range placed {
if o.Name == node && o.Key == t.PublicKey {
adoptsTunnel = true
}
}
} else if !errors.Is(err, inventory.ErrNoTunnel) {
return err
}
}
if adoptsTunnel {
if port := portOfEndpoint(*endpoint); port != strconv.Itoa(tunnel.Port) {
return fmt.Errorf("%s takes over the tunnel it found on %s, which listens on port %d, and "+
"its endpoint %q names another port: the peers dial the tunnel's port, so the hub's "+
"endpoint must be on it", node, tunnel.Interface, tunnel.Port, *endpoint)
}
}
// Declared, all three. The address is evidence of reachability and is not the fact, and hub
// election by address prefix fails silently (novox/hq ADR 0007).
if err := inv.SetPlace(ctx, node, *endpoint, *site, *hub, ""); err != nil {
return err
}
found, err := inv.NodeByName(ctx, node)
cidr, err := overlayRange(ctx, inv)
if err != nil {
return err
}
address, err := inv.AssignAddress(ctx, found.ID, overlayCIDR())
address, err := inv.AssignAddress(ctx, found.ID, cidr)
if err != nil {
return err
}
@@ -128,6 +193,10 @@ func overlayPlace(ctx context.Context, inv *inventory.Inventory, args []string)
fmt.Println(" credentials issued for it before this placement keep their old broker address —" +
" `module issue` them again and push (novox/hq issue 059)")
switch {
case adoptsTunnel:
fmt.Printf(" the hub — it takes over the tunnel it found on %s: range %s, port %d, "+
"%d peer(s) carried until they enrol\n", tunnel.Interface, tunnel.Range, tunnel.Port,
len(tunnel.Peers))
case *hub:
fmt.Println(" the hub — every node not sharing a site routes through it")
case *endpoint == "":
@@ -154,15 +223,47 @@ func network(ctx context.Context, inv *inventory.Inventory, on map[string]bool,
if err != nil {
return nil, err
}
// The tunnels adopted nodes take over, and the peers the hub's carries (novox/hq ADR 0105).
tunnels, err := inv.Tunnels(ctx)
if err != nil {
return nil, err
}
carried, err := inv.CarriedPeers(ctx)
if err != nil {
return nil, err
}
nodes := make([]overlay.Node, 0, len(places))
for _, p := range places {
if !on[p.Name] {
continue
}
nodes = append(nodes, overlay.Node{
n := overlay.Node{
Name: p.Name, Key: p.Key, Endpoint: p.Endpoint,
Site: p.Site, Hub: p.Hub, Address: p.Address,
})
}
if t, takes := tunnels[p.Name]; takes && t.NodeAdopted {
// Only an adopted node is told to take the found unit over: on a converged one there
// is nothing found to keep, and the host refuses the field. The range and the carried
// peers do not depend on the mode; the takeover does.
//
// **Refused, not composed, when the hub's record disagrees with the tunnel.** A
// declaration that stopped the found unit and raised the mesh's interface on another
// port or address would leave every peer dark while reporting the tunnel taken — so a
// hub placed before it took the tunnel over (or at the wrong port) is named here, and
// nothing is sent until it is re-placed.
if wrong := disagrees(p, t.Tunnel); wrong != "" {
return nil, fmt.Errorf("%s takes over the tunnel on %s and its placement disagrees with it: %s. "+
"Re-place it — `overlay place %s --hub --endpoint <host>:%d …` — and push again; "+
"nothing was composed", p.Name, t.Interface, wrong, p.Name, t.Port)
}
n.TakesOver = &overlay.TakeOver{Interface: t.Interface, Unit: t.Unit, Config: t.Config, Port: t.Port, MTU: t.MTU}
}
if p.Hub {
for _, c := range carried {
n.Carried = append(n.Carried, overlay.Carried{Key: c.PublicKey, Address: c.Address})
}
}
nodes = append(nodes, n)
}
if len(nodes) == 0 {
// Nobody was given it. An empty network is a legitimate mesh, not a broken one, so this
@@ -170,7 +271,11 @@ func network(ctx context.Context, inv *inventory.Inventory, on map[string]bool,
// "no hub" to somebody who never asked for a network would be a lie about the cause.
return overlay.Empty(), nil
}
g, err := overlay.From(nodes, overlayCIDR(), "")
cidr, err := overlayRange(ctx, inv)
if err != nil {
return nil, err
}
g, err := overlay.From(nodes, cidr, "")
if g != nil {
// The artifact store, as this network reaches it. Found rather than configured: the
// provider is whichever module offers it, on whichever machine holds that module — and if
@@ -292,6 +397,17 @@ func overlayShow(ctx context.Context, open *stores) error {
return nil
}
// The tunnel the hub took over, if any, and the peers carried from it (novox/hq ADR 0105):
// listed apart from the nodes, because they are peers of the tunnel and not nodes of the
// mesh until they enrol — and once one has, it is listed as the node it became.
tunnel, hubName, adopted, err := open.inventory.AdoptedTunnel(ctx)
if err != nil {
return err
}
carried, err := open.inventory.CarriedPeers(ctx)
if err != nil {
return err
}
for _, n := range nodes {
place := n.Address
if place == "" {
@@ -301,22 +417,74 @@ func overlayShow(ctx context.Context, open *stores) error {
}
fmt.Printf("%-16s %-14s", n.Name, place)
switch {
case n.Hub && adopted:
fmt.Printf(" hub — over the tunnel it took over on %s (range %s, port %d)",
tunnel.Interface, tunnel.Range, tunnel.Port)
case n.Hub && hubName == n.Name && tunnel.Interface != "":
fmt.Printf(" hub — found a tunnel on %s and did NOT take it over: its key is not the tunnel's; "+
"`mesh-host overlay take --tunnel %s` on the machine takes it", tunnel.Interface, tunnel.Interface)
case n.Hub:
fmt.Print(" hub")
fmt.Print(" hub — found no tunnel; if the machine runs the predecessor's, " +
"`mesh-host overlay take --tunnel <iface>` there adopts it (novox/hq ADR 0105)")
case !n.Reachable():
fmt.Print(" not dialable")
}
if n.Site != "" {
fmt.Printf(" at %s", n.Site)
}
if n.TakesOver != nil && !n.Hub {
fmt.Printf(" takes over %s", n.TakesOver.Interface)
}
fmt.Println()
for _, p := range computed[n.Name] {
fmt.Printf(" → %-14s %-18s %s\n", p.Name, p.Allowed, p.Why)
}
}
if len(carried) > 0 {
fmt.Printf("\npeers of the tunnel %s took over — not nodes of the mesh until they enrol:\n", hubName)
for _, c := range carried {
state := "not yet enrolled"
if c.EnrolledAs != "" {
state = "enrolled as " + c.EnrolledAs + ", which keeps this address"
}
fmt.Printf(" %-16s %-14s %s\n", overlay.CarriedName(c.PublicKey), c.Address, state)
}
}
return nil
}
// disagrees says how a node's placement differs from the tunnel it takes over — its address not
// the tunnel's, its endpoint not on the tunnel's port — or nothing when both agree.
func disagrees(p inventory.Overlay, t inventory.Tunnel) string {
var wrong []string
want := t.Address
if i := strings.Index(want, "/"); i >= 0 {
want = want[:i]
}
if p.Address != want {
wrong = append(wrong, fmt.Sprintf("its address is %s and the tunnel's is %s", orNothing(p.Address), want))
}
if p.Reachable() && portOfEndpoint(p.Endpoint) != strconv.Itoa(t.Port) {
wrong = append(wrong, fmt.Sprintf("its endpoint %s is not on the tunnel's port %d", p.Endpoint, t.Port))
}
return strings.Join(wrong, "; ")
}
func orNothing(s string) string {
if s == "" {
return "unset"
}
return s
}
// portOfEndpoint is the port in host:port, or empty.
func portOfEndpoint(endpoint string) string {
if i := strings.LastIndex(endpoint, ":"); i >= 0 {
return endpoint[i+1:]
}
return ""
}
// SilentFor is how long a node may be quiet before the mesh says so.
//
// A node speaks every minute, so three of them missed is a gap rather than a slow one. The number
@@ -437,6 +605,24 @@ func namesInTheMesh(ctx context.Context, inv *inventory.Inventory,
for _, p := range places {
out[overlay.InternalName(p.Name)] = p.Address
}
// And the carried peers the operator has named (novox/hq issue 112): machines the
// predecessor's resolver answers for and the mesh routes to, known by name on the operator's
// word until they enrol — at which point enrolment verifies the name and the node's own
// entry takes over above. A name the predecessor answers for must keep resolving until the
// machine behind it is a node; without these, taking the resolver silences three machines.
carried, err := inv.CarriedPeers(ctx)
if err != nil {
return nil, err
}
for _, p := range carried {
if p.Named == "" || p.EnrolledAs != "" {
continue
}
if _, taken := out[overlay.InternalName(p.Named)]; taken {
continue // a node of the mesh owns the name; the stale statement loses
}
out[overlay.InternalName(p.Named)] = p.Address
}
return out, nil
}
@@ -494,11 +680,18 @@ func artifactStoreOnNetwork(ctx context.Context, inv *inventory.Inventory,
return "", "", false, nil
}
// artifactStoreAddress is the artifact store as this network reaches it, `<node>.internal:<port>`,
// or "" when the mesh has none on its network yet — the address composed into every reference
// the mesh built, at the moment it is used and never before (novox/hq 04-ISSUES/102).
// artifactStoreAddress is the artifact store as `forNode` reaches it: `<node>.internal:<port>`
// over the private network, or — when nothing is on the network yet — `127.0.0.1:<port>` for the
// node that holds the store itself, and "" for any other. The address composed into every
// reference the mesh built, at the moment it is used and never before (novox/hq 04-ISSUES/102).
//
// **Genesis places the network after the store, the broker, the vault and the catalogue.** Each
// of those is built and pushed to a node that is on no network, and the store is on that same
// node; an answer of "no store" there would refuse every one of those pushes and hand every one
// of those builds a base nothing can pull. Loopback is the truth on that machine, and it is the
// address genesis itself reaches the store by.
func artifactStoreAddress(ctx context.Context, inv *inventory.Inventory,
shelf map[string]catalogue.Manifest) (string, error) {
shelf map[string]catalogue.Manifest, forNode string) (string, error) {
onNetwork, err := whereEveryoneIs(ctx, inv, shelf)
if err != nil {
return "", err
@@ -508,8 +701,29 @@ func artifactStoreAddress(ctx context.Context, inv *inventory.Inventory,
on[name] = true
}
node, port, found, err := artifactStoreOnNetwork(ctx, inv, on)
if err != nil || !found {
if err != nil {
return "", err
}
return overlay.InternalName(node) + ":" + port, nil
if found {
return overlay.InternalName(node) + ":" + port, nil
}
holder, port, found, err := artifactStoreHolder(ctx, inv)
if err != nil || !found || holder != forNode {
return "", err
}
return "127.0.0.1:" + port, nil
}
// artifactStoreHolder is whichever node is assigned a module offering the artifact store, on or
// off the network, and the port that node put it on.
func artifactStoreHolder(ctx context.Context, inv *inventory.Inventory) (node, port string, found bool, err error) {
nodes, err := inv.Nodes(ctx)
if err != nil {
return "", "", false, err
}
all := map[string]bool{}
for _, n := range nodes {
all[n.Name] = true
}
return artifactStoreOnNetwork(ctx, inv, all)
}
+195
View File
@@ -2,9 +2,11 @@ package main
import (
"context"
"os"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/overlay"
)
@@ -108,3 +110,196 @@ func TestOnlyAMachineOnThePrivateNetworkIsNamed(t *testing.T) {
t.Fatalf("a machine that left the network is still named, or the one that stayed is not: %v", names)
}
}
// novox/hq ADR 0105: the range every address is composed from is the adopted tunnel's, read from
// the tunnel the hub holds — never stored anywhere else.
func TestTheOverlaysRangeIsTheAdoptedTunnels(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
t.Setenv(OverlayCIDRVar, "10.99.0.0/16")
before, err := overlayRange(ctx, inv)
if err != nil || before != "10.99.0.0/16" {
t.Fatalf("without an adopted tunnel the range is not what genesis said: %q %v", before, err)
}
// The hub becomes what genesis makes of a machine in use: adopted, enrolled with the found
// tunnel's key, and presenting the tunnel.
if err := inv.SetAdopted(ctx, "anchor", true); err != nil {
t.Fatal(err)
}
hub, err := inv.NodeByName(ctx, "anchor")
if err != nil {
t.Fatal(err)
}
const key = "THE-TUNNELS-KEY========================="
if err := inv.RecordOverlayKey(ctx, hub.ID, key); err != nil {
t.Fatal(err)
}
if err := inv.RecordTunnel(ctx, hub.ID, inventory.Tunnel{
Interface: "wg0", Unit: "wg-quick@wg0", Config: "/etc/wireguard/wg0.conf", Port: 51900,
Address: "192.0.2.1/24", Range: "192.0.2.0/24", PublicKey: key,
Peers: []inventory.TunnelPeer{{PublicKey: "PEER-TWO", Address: "192.0.2.2"}},
}); err != nil {
t.Fatal(err)
}
after, err := overlayRange(ctx, inv)
if err != nil || after != "192.0.2.0/24" {
t.Fatalf("with an adopted tunnel the range is %q (%v), not the tunnel's", after, err)
}
// A placement whose endpoint is on another port than the tunnel's is refused: the peers dial
// the tunnel's port.
err = overlayPlace(ctx, inv, []string{"anchor", "--endpoint", "198.51.100.10:51820", "--site", "hosting", "--hub"})
if err == nil || !strings.Contains(err.Error(), "51900") {
t.Fatalf("an endpoint off the tunnel's port was accepted: %v", err)
}
// On the tunnel's port, the hub is placed at the tunnel's address — whatever it had before.
if err := inv.SetPlace(ctx, "anchor", "", "", false, ""); err != nil {
t.Fatal(err)
}
if err := overlayPlace(ctx, inv, []string{"anchor", "--endpoint", "198.51.100.10:51900", "--site", "hosting", "--hub"}); err != nil {
t.Fatal(err)
}
if placed := placementOf(t, ctx, inv, "anchor"); placed.Address != "192.0.2.1" {
t.Fatalf("the hub was placed at %s, not the tunnel's own address", placed.Address)
}
// And the hub's declaration carries the peer and the takeover.
nodes, computed, err := graph(ctx, open)
if err != nil {
t.Fatal(err)
}
var hubNode overlay.Node
for _, n := range nodes {
if n.Name == "anchor" {
hubNode = n
}
}
if hubNode.TakesOver == nil || hubNode.TakesOver.Unit != "wg-quick@wg0" {
t.Errorf("the hub is not told to take over the found tunnel: %+v", hubNode)
}
carried := false
for _, p := range computed["anchor"] {
if p.Key == "PEER-TWO" && p.Allowed == "192.0.2.2/32" {
carried = true
}
}
if !carried {
t.Errorf("the hub's peer list does not carry the tunnel's peer: %+v", computed["anchor"])
}
}
// A takeover is composed only for a hub whose placement agrees with the tunnel: an address or an
// endpoint port that differs would have the host stop the found interface and raise the mesh's
// where no peer is listening.
func TestATakeoverIsNotComposedForAHubPlacedOffItsTunnel(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
if err := inv.SetAdopted(ctx, "anchor", true); err != nil {
t.Fatal(err)
}
hub, err := inv.NodeByName(ctx, "anchor")
if err != nil {
t.Fatal(err)
}
const key = "THE-TUNNELS-KEY========================="
if err := inv.RecordOverlayKey(ctx, hub.ID, key); err != nil {
t.Fatal(err)
}
if err := inv.RecordTunnel(ctx, hub.ID, inventory.Tunnel{
Interface: "wg0", Unit: "wg-quick@wg0", Config: "/etc/wireguard/wg0.conf", Port: 51900,
Address: "192.0.2.1/24", Range: "192.0.2.0/24", PublicKey: key}); err != nil {
t.Fatal(err)
}
// aMesh placed anchor at 10.77.0.1 on :51820 — the record of a hub placed before it took the
// tunnel over.
_, _, err = graph(ctx, open)
if err == nil {
t.Fatal("a takeover was composed for a hub whose address and port are not the tunnel's")
}
for _, want := range []string{"10.77.0.1", "192.0.2.1", "51820", "51900", "overlay place anchor"} {
if !strings.Contains(err.Error(), want) {
t.Errorf("the refusal does not say %q: %v", want, err)
}
}
// Re-placed on the tunnel, it composes.
if err := inv.SetPlace(ctx, "anchor", "", "", false, ""); err != nil {
t.Fatal(err)
}
if err := overlayPlace(ctx, inv, []string{"anchor", "--endpoint", "198.51.100.10:51900", "--site", "here", "--hub"}); err != nil {
t.Fatal(err)
}
if _, _, err := graph(ctx, open); err != nil {
t.Fatalf("re-placed on the tunnel, the graph still refuses: %v", err)
}
}
// theResolver is the catalogue's dnsmasq module as it is, or the test is skipped where the
// catalogue is not beside this checkout.
func theResolver(t *testing.T) catalogue.Manifest {
t.Helper()
raw, err := os.ReadFile("../../../mesh-catalog/modules/dnsmasq/module.json")
if err != nil {
t.Skipf("the catalogue is not beside this checkout: %v", err)
}
m, err := catalogue.ParseManifest(raw)
if err != nil {
t.Fatalf("dnsmasq does not parse:\n%v", err)
}
return m
}
// The resolver is handed every machine on the private network as a wildcard, the same set and the
// same source as the hosts file, and is handed it again when a machine leaves — through the
// module's own manifest asking for the fact, with no module of the mesh's own in between (hal
// dnsmasq-app conversion, novox/hq 08-connectivity). The runtime on that machine is pointed at the
// machine's own address, where the resolver answers for its containers.
func TestTheResolverIsToldEveryMachineOnTheNetworkAndToldAgainWhenOneLeaves(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
register(t, open, theResolver(t))
if _, err := assign(ctx, open, "anchor", "dnsmasq"); err != nil {
t.Fatal(err)
}
zones := func() string {
t.Helper()
for _, r := range composed(t, open, "anchor").Resources {
if r["id"] == "dnsmasq.fact-node-zones" {
if r["path"] != "/etc/mesh-resolver/nodes.conf" {
t.Fatalf("the machines were written somewhere the resolver does not read: %v", r["path"])
}
return r["content"].(string)
}
}
t.Fatal("the resolver was not handed the machines")
return ""
}
first := zones()
for _, want := range []string{
"local=/internal/", "address=/anchor.internal/10.77.0.1\n", "address=/laptop.internal/10.77.0.2\n",
} {
if !strings.Contains(first, want) {
t.Errorf("the resolver's machines lack %q:\n%s", want, first)
}
}
for _, r := range composed(t, open, "anchor").Resources {
if r["id"] == "dnsmasq.runtime-dns" {
if !strings.Contains(r["content"].(string), `"10.77.0.1"`) || r["into"] != "json" {
t.Errorf("the runtime is not pointed at this machine's own address, written into its file: %v", r)
}
}
}
// The laptop keeps its place and its address, and stops running the network.
if err := open.inventory.Unassign(ctx, "laptop", overlay.Name); err != nil {
t.Fatal(err)
}
after := zones()
if strings.Contains(after, "laptop") || !strings.Contains(after, "address=/anchor.internal/10.77.0.1\n") {
t.Fatalf("a machine that left the network is still a wildcard, or the one that stayed is not:\n%s", after)
}
}
+91 -32
View File
@@ -217,6 +217,7 @@ func theRestOfTheMesh(ctx context.Context, inv *inventory.Inventory,
}
offered := map[string][]catalogue.Provider{}
var firstHeld []catalogue.Held
for _, o := range others {
got, err := catalogue.Resolve(shelf, o.assigned, o.node, catalogue.World{Unchecked: true})
if err != nil {
@@ -224,6 +225,7 @@ func theRestOfTheMesh(ctx context.Context, inv *inventory.Inventory,
// report, and nothing of theirs is running, so it offers nothing.
continue
}
firstHeld = append(firstHeld, got.Claims...)
for _, m := range got.Modules {
for _, name := range m.OffersAt(catalogue.ScopeMesh) {
// What that module says a consumer needs to know, with that node's settings on
@@ -234,7 +236,7 @@ func theRestOfTheMesh(ctx context.Context, inv *inventory.Inventory,
return catalogue.World{}, err
}
offered[name] = append(offered[name], catalogue.Provider{
Node: o.node.Name, At: o.node.At, Serves: serves})
Node: o.node.Name, At: o.node.At, Serves: serves, Module: m.Module})
}
}
}
@@ -244,14 +246,20 @@ func theRestOfTheMesh(ctx context.Context, inv *inventory.Inventory,
})
}
world := catalogue.World{Offered: offered}
// **The second pass is given the first pass's holdings.** A seat's holder answers a requirement
// with several providers (novox/hq ADR 0110), so a node consuming one resolves only once the
// holder is known. Without them its set is refused here, and a refused node's own claims drop
// out of what the mesh holds — so a second holder of one of its seats would pass unrefused.
world := catalogue.World{Offered: offered, Held: firstHeld}
var held []catalogue.Held
for _, o := range others {
got, err := catalogue.Resolve(shelf, o.assigned, o.node, world)
if err != nil {
continue
}
world.Held = append(world.Held, got.Claims...)
held = append(held, got.Claims...)
}
world.Held = held
return world, nil
}
@@ -481,17 +489,10 @@ func renderingFor(ctx context.Context, open *stores, node string,
return catalogue.Rendering{}, inventory.Node{}, err
}
// Where this node put the foundation's servers, for the control plane's own connections
// (novox/hq 04-ISSUES/102): read from the node's settings for whatever claims each seat,
// exactly as a consumer's binding is, never from what genesis wrote into a secret.
seats, err := seatsOn(ctx, inv, shelf, node, plan.Modules)
if err != nil {
return catalogue.Rendering{}, inventory.Node{}, err
}
// And the artifact store as this network reaches it now — the address every image and
// archive the mesh built is fetched through, composed here and recorded nowhere — with what
// the mesh has built, so a reference recorded with an address before that is re-routed too.
artifactStore, err := artifactStoreAddress(ctx, inv, shelf)
// The artifact store as this node reaches it now — the address every image and archive the
// mesh built is fetched through, composed here and recorded nowhere — with what the mesh has
// built, so a reference recorded with an address before that is re-routed too.
artifactStore, err := artifactStoreAddress(ctx, inv, shelf, node)
if err != nil {
return catalogue.Rendering{}, inventory.Node{}, err
}
@@ -516,6 +517,12 @@ func renderingFor(ctx context.Context, open *stores, node string,
// `<node>.internal` names above, so a container — or an internal ACME validator — resolves a
// routed name to the proxy that serves it, mesh-wide. The mesh publishes the names it was told
// to serve and knows nothing about what they mean.
// Kept apart from the machines, because a fact about the machines must not be handed the names
// the mesh merely serves (novox/hq 04-ISSUES/111).
machines := make(map[string]string, len(names))
for name, at := range names {
machines[name] = at
}
routes, err := routeNamesInTheMesh(ctx, open)
if err != nil {
return catalogue.Rendering{}, inventory.Node{}, err
@@ -527,11 +534,19 @@ func renderingFor(ctx context.Context, open *stores, node string,
// The ports the mesh itself needs open, which no module declares. Read from the broker this
// control plane was told about rather than written down twice: the address a node is handed in
// its token and the port its machine must accept on are the same fact.
//
// **Only on the node that listens on it** (novox/hq issue: the broker opening leaked onto
// every node). The opening exists to WIDEN the broker's port to from-anywhere — a machine
// enrolling is not on the mesh yet, so the broker's own `from: mesh` listen would refuse its
// first dial. That widening belongs on the broker's host and nowhere else: a node that only
// dials out needs no incoming rule, and an opening for a port nothing here listens on is a
// from-anywhere hole for a dead port. So the foundation port is kept only when a module
// resolved onto THIS node actually listens on it.
var foundation []int
if b, err := broker.FromEnvironment(); err == nil {
if _, port, err := net.SplitHostPort(b.Address); err == nil {
if n, err := strconv.Atoi(port); err == nil {
foundation = append(foundation, n)
foundation = foundationPortsFor(n, plan.Modules)
}
}
}
@@ -571,10 +586,18 @@ func renderingFor(ctx context.Context, open *stores, node string,
taken[m] = true
}
}
// Where this node put the foundation's servers, for the control plane's own connections
// (novox/hq 04-ISSUES/102): read from the node's settings for whatever claims each seat,
// exactly as a consumer's binding is, never from what genesis wrote into a secret.
seats, err := seatsOn(ctx, inv, shelf, node, plan.Modules, record.Adopted, taken)
if err != nil {
return catalogue.Rendering{}, inventory.Node{}, err
}
return catalogue.Rendering{
Settings: settings, Generators: gens, Grants: grants, Needed: needed, Ports: ports,
Certificate: certificate, Authority: authority, Mesh: private, Names: names,
Suffix: overlay.Suffix(), Foundation: foundation, Kept: kept, Adopted: record.Adopted,
Machines: machines,
Suffix: overlay.Suffix(), Foundation: foundation, Kept: kept, Adopted: record.Adopted,
Given: given, Taken: taken, Seats: seats, ArtifactStore: artifactStore, Built: built,
}, record, nil
}
@@ -793,6 +816,22 @@ func grantsFor(ctx context.Context, open *stores, node string) ([]catalogue.Gran
return out, nil
}
// listensLines is what a person is told about what this module would open, and why — the same
// `why` every listens entry already carries for the firewall it also feeds (novox/hq ADR 0007), so
// deciding whether to assign a module can see what it would open before it opens it, not only
// after. A module with nothing to listen on prints nothing extra, same as today.
func listensLines(m catalogue.Manifest) []string {
var out []string
for _, l := range m.Listens {
if l.Why == "" {
out = append(out, fmt.Sprintf(" listens %d/%s from %s", l.Port, l.At(), l.From))
continue
}
out = append(out, fmt.Sprintf(" listens %d/%s from %s — %s", l.Port, l.At(), l.From, l.Why))
}
return out
}
func planCommand(ctx context.Context, args []string) error {
set := flag.NewFlagSet("plan", flag.ContinueOnError)
// Because "one resource" does not tell you whether the settings landed. Being able to read
@@ -846,6 +885,9 @@ func planCommand(ctx context.Context, args []string) error {
fmt.Printf("%s would run:\n", args[0])
for _, m := range plan.Modules {
fmt.Printf(" %-20s %s\n", m.Module, plan.Because[m.Module])
for _, line := range listensLines(m) {
fmt.Println(line)
}
}
// What was assigned here and cannot run here. Said with the rest rather than as a refusal: it is
// one module on the wrong machine, the others still run, and the remedy is to move this one.
@@ -1023,7 +1065,7 @@ func portsGivenOn(ctx context.Context, inv *inventory.Inventory, node string,
// and not yet assigned (04-ISSUES/085), and the control plane must follow that setting from the
// first declaration it composes for itself. A holder in this node's set wins over one that is not.
func seatsOn(ctx context.Context, inv *inventory.Inventory, shelf map[string]catalogue.Manifest,
node string, inSet []catalogue.Manifest) (map[string]map[int]int, error) {
node string, inSet []catalogue.Manifest, adopted bool, taken map[string]bool) (map[string]map[int]int, error) {
assigned := map[string]bool{}
for _, m := range inSet {
assigned[m.Module] = true
@@ -1045,26 +1087,26 @@ func seatsOn(ctx context.Context, inv *inventory.Inventory, shelf map[string]cat
if len(claims) == 0 {
continue
}
// **Only a port the node was given or the mesh assigned — never the manifest's own
// number.** The sealed value the answer sits beside carries the port genesis wrote, which
// on a given-port node is the predecessor's; a manifest's long-form mapping is the
// catalogue's default, and answering with it would override the right number with one
// the mesh never checked (the contract in seat_into.go). And on an adopted node a holder
// assigned but not yet taken is the found container, on the ports it was found with, not
// the declaration's — so its mesh-assigned ports do not count there either; a given port
// does, because a given port is the found one by construction (ADR 0100).
ports, _, err := portsGivenOn(ctx, inv, node, m)
if err != nil {
return nil, err
}
if assigned[name] {
// Running here, so what its manifest publishes the long way is where this machine
// has it — the same reading the declaration itself makes (ADR 0038). Only for a
// holder in this node's set: a manifest's number says nothing about a machine the
// module does not run on.
var declared []int
for _, l := range m.Listens {
declared = append(declared, l.Port)
if adopted && !taken[name] {
layers, err := inv.SettingsFor(ctx, node, m.Module)
if err != nil {
return nil, err
}
for _, wanted := range append(declared, m.Guards...) {
if _, said := ports[wanted]; said {
continue
}
if at, mayAssign := m.MachineSide(wanted); !mayAssign && at != 0 {
ports[wanted] = at
}
ports = map[int]int{}
if given, err := catalogue.GivenPorts(m, layers); err == nil {
ports = given
}
}
if len(ports) == 0 {
@@ -1101,3 +1143,20 @@ func portsOn(
}
return out, nil
}
// foundationPortsFor is the broker port, kept only when a module resolved onto this node listens
// on it (novox/hq issue: the broker opening leaked onto every node). The foundation opening
// exists to WIDEN the broker's `from: mesh` port to from-anywhere, because a machine enrolling is
// not on the mesh yet and its first dial would be refused. That widening belongs on the broker's
// host alone: a node that only dials out needs no incoming rule, and an opening for a port
// nothing here listens on is a from-anywhere hole for a dead port.
func foundationPortsFor(brokerPort int, modules []catalogue.Manifest) []int {
for _, m := range modules {
for _, l := range m.Listens {
if l.Port == brokerPort {
return []int{brokerPort}
}
}
}
return nil
}
+37
View File
@@ -0,0 +1,37 @@
package main
import (
"strings"
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
)
// `plan` tells a person what a module would open and why, from the same `why` every listens
// entry already carries for the firewall (novox/hq ADR 0007) — so deciding whether to assign a
// module does not need reading its manifest first.
func TestListensLinesShowWhatAModuleWouldOpenAndWhy(t *testing.T) {
m := catalogue.Manifest{Module: "minio", Listens: []catalogue.Listening{
{Port: 9000, From: catalogue.FromMesh, Why: "the S3 endpoint"},
{Port: 9001, From: catalogue.FromMesh},
}}
got := listensLines(m)
if len(got) != 2 {
t.Fatalf("two listens entries, got %d: %v", len(got), got)
}
if !strings.Contains(got[0], "9000/tcp") || !strings.Contains(got[0], "the S3 endpoint") {
t.Errorf("the port and its why did not both appear: %q", got[0])
}
if strings.Contains(got[1], "—") {
t.Errorf("a listens entry with no why should not print a dash: %q", got[1])
}
if !strings.Contains(got[1], "9001/tcp") {
t.Errorf("the port still appears without a why: %q", got[1])
}
}
func TestListensLinesAreEmptyForAModuleWithNothingToListenOn(t *testing.T) {
if got := listensLines(catalogue.Manifest{Module: "board"}); len(got) != 0 {
t.Errorf("a module with no listens should print nothing, got %v", got)
}
}
+42 -4
View File
@@ -101,12 +101,50 @@ func routedArtifacts(made []inventory.Artifact, address string) []inventory.Arti
return out
}
// whereTheStoreIs is the artifact store's address as this network reaches it, or "" — read for a
// caller that has the inventory open and nothing else in hand.
func whereTheStoreIs(ctx context.Context, inv *inventory.Inventory) (string, error) {
// whereTheStoreIs is the artifact store's address as something on `forNode` reaches it, or "" —
// read for a caller that has the inventory open and nothing else in hand. With no node named, the
// store's own node: loopback when nothing is on the network yet.
func whereTheStoreIs(ctx context.Context, inv *inventory.Inventory, forNode string) (string, error) {
shelf, err := inv.Catalogue(ctx)
if err != nil {
return "", err
}
return artifactStoreAddress(ctx, inv, shelf)
if forNode == "" {
if holder, _, found, err := artifactStoreHolder(ctx, inv); err != nil {
return "", err
} else if found {
forNode = holder
}
}
return artifactStoreAddress(ctx, inv, shelf, forNode)
}
// whereABuilderReachesTheStore is the store's address for the machine that builds: the network's
// when there is one, else loopback on the store's own node — when that node also holds a module
// requiring the store, which is what a builder is (genesis: one node holds both).
func whereABuilderReachesTheStore(ctx context.Context, inv *inventory.Inventory) (string, error) {
shelf, err := inv.Catalogue(ctx)
if err != nil {
return "", err
}
holder, _, found, err := artifactStoreHolder(ctx, inv)
if err != nil || !found {
return "", err
}
assigned, err := inv.Assigned(ctx, holder)
if err != nil {
return "", err
}
besideIt := false
for _, a := range assigned {
for _, r := range shelf[a].Requires {
if r == catalogue.ArtifactStoreProvision {
besideIt = true
}
}
}
if !besideIt {
holder = ""
}
return artifactStoreAddress(ctx, inv, shelf, holder)
}
+150
View File
@@ -0,0 +1,150 @@
package main
import (
"context"
"encoding/json"
"flag"
"fmt"
"os"
"sort"
"strings"
"text/tabwriter"
"github.com/novox/mesh-controller/internal/catalogue"
)
// What this mesh can have one of, and who fills each (novox/hq ADR 0110).
//
// **Derived every time, never stored.** A seat is held by a module assignment, so the answer is
// computed from assignments by the same resolution that decides what every machine runs. A table
// of holders kept beside the assignments would be a second copy of one fact, and the first thing
// to be wrong about it.
// seatHolder is one assignment holding a seat.
type seatHolder struct {
Node string `json:"node"`
Module string `json:"module"`
}
// seatRow is one seat and who holds it. Unheld is an answer — "this mesh has no X" — not a fault.
type seatRow struct {
Seat string `json:"seat"`
Scope string `json:"scope"`
Delivers string `json:"delivers,omitempty"`
Decision string `json:"decision"`
Holders []seatHolder `json:"holders"`
}
// seatsHeld is every seat the mesh defines with its holders, and every claim held that names no
// seat in the set.
//
// **The second list is not empty by construction.** Manifests are held to the set when they are
// registered, and a mesh can hold one registered before the set closed. Leaving its claim out of the
// overview would make the one thing the overview is for — what does this mesh have — quietly
// incomplete.
func seatsHeld(seats []catalogue.Seat, held []catalogue.Held) ([]seatRow, []catalogue.Held) {
defined := map[string]bool{}
rows := make([]seatRow, 0, len(seats))
for _, s := range seats {
defined[s.Name] = true
row := seatRow{Seat: s.Name, Scope: s.Scope, Delivers: s.Delivers, Decision: s.Decision,
Holders: []seatHolder{}}
seen := map[seatHolder]bool{}
for _, h := range held {
if h.Claim != s.Name || h.Scope != s.Scope {
continue
}
holder := seatHolder{Node: h.Node, Module: h.Module}
if !seen[holder] {
seen[holder] = true
row.Holders = append(row.Holders, holder)
}
}
sort.Slice(row.Holders, func(i, j int) bool {
if row.Holders[i].Node != row.Holders[j].Node {
return row.Holders[i].Node < row.Holders[j].Node
}
return row.Holders[i].Module < row.Holders[j].Module
})
rows = append(rows, row)
}
var outside []catalogue.Held
for _, h := range held {
if !defined[h.Claim] {
outside = append(outside, h)
}
}
sort.Slice(outside, func(i, j int) bool {
if outside[i].Claim != outside[j].Claim {
return outside[i].Claim < outside[j].Claim
}
return outside[i].Node < outside[j].Node
})
return rows, outside
}
func seatsCommand(ctx context.Context, args []string) error {
set := flag.NewFlagSet("seats", flag.ContinueOnError)
asJSON := set.Bool("json", false, "the same, as JSON")
if err := set.Parse(args); err != nil {
return err
}
open, err := openStores(ctx)
if err != nil {
return err
}
defer open.Close()
inv := open.inventory
shelf, err := inv.Catalogue(ctx)
if err != nil {
return err
}
// Every node, none excluded: the same view of what each machine holds that planning uses.
world, err := theRestOfTheMesh(ctx, inv, shelf, "")
if err != nil {
return err
}
rows, outside := seatsHeld(catalogue.Seats(), world.Held)
if *asJSON {
out := struct {
Seats []seatRow `json:"seats"`
Outside []catalogue.Held `json:"outside,omitempty"`
}{rows, outside}
body, err := json.MarshalIndent(out, "", " ")
if err != nil {
return err
}
fmt.Println(string(body))
return nil
}
w := tabwriter.NewWriter(os.Stdout, 0, 0, 2, ' ', 0)
fmt.Fprintln(w, "SEAT\tSCOPE\tDELIVERS\tHELD BY")
for _, r := range rows {
delivers := r.Delivers
if delivers == "" {
delivers = "—"
}
holders := "unheld"
if len(r.Holders) > 0 {
parts := make([]string, 0, len(r.Holders))
for _, h := range r.Holders {
parts = append(parts, h.Module+" on "+h.Node)
}
holders = strings.Join(parts, ", ")
}
fmt.Fprintf(w, "%s\t%s\t%s\t%s\n", r.Seat, r.Scope, delivers, holders)
}
if err := w.Flush(); err != nil {
return err
}
if len(outside) > 0 {
fmt.Println("\nheld, and not a seat this mesh defines (registered before the set closed — novox/hq ADR 0110):")
for _, h := range outside {
fmt.Printf(" %s %s on %s\n", h.Claim, h.Module, h.Node)
}
}
return nil
}
+64
View File
@@ -0,0 +1,64 @@
package main
import (
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
)
// The overview of what a mesh has (novox/hq ADR 0110).
func TestEverySeatIsListedIncludingTheOnesNobodyHolds(t *testing.T) {
// An unheld seat is an answer — "this mesh has no forge" — so it is listed rather than omitted.
rows, _ := seatsHeld(catalogue.Seats(), []catalogue.Held{
{Claim: "mesh-store", Scope: catalogue.ScopeMesh, Node: "anchor", Module: "postgres"},
})
if len(rows) != len(catalogue.Seats()) {
t.Fatalf("%d seats listed of %d", len(rows), len(catalogue.Seats()))
}
for _, r := range rows {
switch r.Seat {
case "mesh-store":
if len(r.Holders) != 1 || r.Holders[0].Module != "postgres" || r.Holders[0].Node != "anchor" {
t.Errorf("mesh-store is held by %+v", r.Holders)
}
if r.Delivers != "postgres-database" {
t.Errorf("mesh-store does not say what it delivers: %q", r.Delivers)
}
case "git":
if len(r.Holders) != 0 {
t.Errorf("git is held by %+v in a mesh with no forge", r.Holders)
}
}
}
}
func TestANodeSeatListsEveryMachineHoldingIt(t *testing.T) {
rows, _ := seatsHeld(catalogue.Seats(), []catalogue.Held{
{Claim: "the-packet-filter", Scope: catalogue.ScopeNode, Node: "node2", Module: "nftables"},
{Claim: "the-packet-filter", Scope: catalogue.ScopeNode, Node: "anchor", Module: "nftables"},
// Resolved twice, reported once: a machine is one holder however many passes saw it.
{Claim: "the-packet-filter", Scope: catalogue.ScopeNode, Node: "anchor", Module: "nftables"},
})
for _, r := range rows {
if r.Seat != "the-packet-filter" {
continue
}
if len(r.Holders) != 2 || r.Holders[0].Node != "anchor" || r.Holders[1].Node != "node2" {
t.Fatalf("the packet filter is held by %+v", r.Holders)
}
return
}
t.Fatal("the packet filter is not listed")
}
func TestAClaimOutsideTheSetIsShownNotHidden(t *testing.T) {
// A manifest registered before the set closed can still hold one. Leaving it out would make
// the overview quietly incomplete, which is the one thing it may not be.
_, outside := seatsHeld(catalogue.Seats(), []catalogue.Held{
{Claim: "the-controller", Scope: catalogue.ScopeMesh, Node: "anchor", Module: "mesh-controller"},
})
if len(outside) != 1 || outside[0].Claim != "the-controller" {
t.Fatalf("a claim outside the set was not shown: %+v", outside)
}
}
+136
View File
@@ -0,0 +1,136 @@
package main
import (
"context"
"fmt"
"strconv"
"strings"
"github.com/novox/mesh-controller/internal/catalogue"
)
// where a build's repository is (novox/hq ADR 0111).
//
// A repository is on the mesh's own forge, or it is anywhere else. The first is recorded as its path
// on the forge holding the git seat, and cloned from wherever that forge runs at the moment of
// building; the second is a URL, recorded and cloned exactly as given. The build machine is not told
// the difference — it is handed a URL either way — because only the control plane knows where the
// seat's holder runs.
// gitSeat is the seat a self-hosted repository lives on.
const gitSeat = "git"
// buildSource is where a build's repository is: a URL, or a path on a seat's holder.
type buildSource struct {
Repository string
Seat string
}
// String is the source as a person reads it, which for one on a seat is not the URL: the URL is a
// fact about where the forge happens to run today.
func (s buildSource) String() string {
if s.Seat == "" {
return s.Repository
}
return fmt.Sprintf("%s on the %s seat", s.Repository, s.Seat)
}
// onASeat refuses an address given as a path on the forge.
//
// **A URL here would be recorded as a path**, and then composed onto the forge's address as one —
// cloning `http://forge:3000/https://github.com/…`. Refused by what an address plainly looks like,
// not repaired: `--self` promises a path, and something that is not one is a mistake to name.
func onASeat(repository string) error {
if strings.Contains(repository, ":") || strings.HasPrefix(repository, "/") ||
strings.Trim(repository, "/") == "" {
return fmt.Errorf("--self takes the repository's path on the forge, such as novox/mesh-catalog, "+
"and %q is not one — without --self it is built from exactly what is given", repository)
}
return nil
}
// cloneFrom is the URL a build machine clones for a source.
//
// A URL is itself. A path on a seat is composed from the seat's holder as the mesh sees it now —
// the same view planning takes of every machine, so the forge a build clones from is the forge the
// mesh says holds the seat.
func cloneFrom(ctx context.Context, source buildSource) (string, error) {
if source.Seat == "" {
return source.Repository, nil
}
open, err := openStores(ctx)
if err != nil {
return "", err
}
defer open.Close()
shelf, err := open.inventory.Catalogue(ctx)
if err != nil {
return "", err
}
world, err := theRestOfTheMesh(ctx, open.inventory, shelf, "")
if err != nil {
return "", err
}
return clonedFromSeat(world, source.Seat, source.Repository)
}
// clonedFromSeat composes the clone URL for a repository on a seat's holder.
//
// **Refused, never defaulted, at every step that has no answer.** Nobody holding the seat is a mesh
// without a forge of its own: it builds from external repositories and must say so rather than fail
// to clone. A holder off the private network cannot be reached by any build machine. A holder that
// serves no scheme or port has nothing to compose from — a default port here would be the forge's
// address guessed, which is the thing this exists to stop.
func clonedFromSeat(world catalogue.World, seatName, repository string) (string, error) {
seat, known := catalogue.SeatNamed(seatName)
if !known || seat.Delivers == "" {
return "", fmt.Errorf("%q is not a seat a repository can live on", seatName)
}
var holder *catalogue.Held
for i, h := range world.Held {
if h.Claim == seat.Name && h.Scope == seat.Scope {
holder = &world.Held[i]
break
}
}
if holder == nil {
return "", fmt.Errorf("nobody holds the %s seat, so %s cannot be cloned from this mesh's "+
"forge — assign a module that claims it, or build from the repository's URL without --self",
seat.Name, repository)
}
var provider *catalogue.Provider
for i, p := range world.Offered[seat.Delivers] {
if p.Node == holder.Node && p.Module == holder.Module {
provider = &world.Offered[seat.Delivers][i]
}
}
if provider == nil {
return "", fmt.Errorf("%s on %s holds the %s seat and offers no %q to clone from",
holder.Module, holder.Node, seat.Name, seat.Delivers)
}
if provider.At == "" {
return "", fmt.Errorf("%s on %s holds the %s seat and is not on the private network, so no "+
"build machine can reach it", holder.Module, holder.Node, seat.Name)
}
scheme, _ := provider.Serves["scheme"].(string)
port := servedPort(provider.Serves["port"])
if scheme == "" || port == "" {
return "", fmt.Errorf("%s on %s holds the %s seat and does not serve a scheme and a port for %q",
holder.Module, holder.Node, seat.Name, seat.Delivers)
}
path := strings.TrimSuffix(strings.Trim(repository, "/"), ".git")
return fmt.Sprintf("%s://%s:%s/%s.git", scheme, provider.At, port, path), nil
}
// servedPort is a served port as text, however the manifest and the node's settings carried it.
func servedPort(v any) string {
switch p := v.(type) {
case float64:
return strconv.Itoa(int(p))
case int:
return strconv.Itoa(p)
case string:
return p
}
return ""
}
+108
View File
@@ -0,0 +1,108 @@
package main
import (
"strings"
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
)
// Defends novox/hq ADR 0111: a build source is on the git seat, or it is external.
func forgeHolding(port any) catalogue.World {
return catalogue.World{
Held: []catalogue.Held{{Claim: "git", Scope: catalogue.ScopeMesh, Node: "anchor", Module: "gitea"}},
Offered: map[string][]catalogue.Provider{"git": {
// A second forge that does not hold the seat, so taking the first one found would be wrong.
{Node: "archive", At: "archive.internal", Module: "gitea-mirror",
Serves: map[string]any{"scheme": "http", "port": float64(3000)}},
{Node: "anchor", At: "anchor.internal", Module: "gitea",
Serves: map[string]any{"scheme": "http", "port": port}},
}},
}
}
func TestARepositoryOnTheSeatIsClonedFromItsHolder(t *testing.T) {
got, err := clonedFromSeat(forgeHolding(float64(3000)), "git", "novox/mesh-catalog")
if err != nil {
t.Fatal(err)
}
if got != "http://anchor.internal:3000/novox/mesh-catalog.git" {
t.Fatalf("cloned from %s", got)
}
}
func TestAMovedForgeIsFollowedWithoutRewritingAnything(t *testing.T) {
// The whole point: the node gave the forge another port, and the same recorded path clones
// from the new one. Nothing recorded contained the old one to be wrong.
got, err := clonedFromSeat(forgeHolding(float64(3100)), "git", "novox/mesh-catalog")
if err != nil {
t.Fatal(err)
}
if !strings.Contains(got, ":3100/") {
t.Fatalf("the moved port was not followed: %s", got)
}
}
func TestWithNobodyHoldingTheSeatASelfHostedBuildIsRefusedAndSaysWhy(t *testing.T) {
_, err := clonedFromSeat(catalogue.World{}, "git", "novox/mesh-catalog")
if err == nil {
t.Fatal("a repository was cloned from a forge the mesh does not have")
}
for _, want := range []string{"nobody holds the git seat", "without --self"} {
if !strings.Contains(err.Error(), want) {
t.Fatalf("the refusal does not say %q: %v", want, err)
}
}
}
func TestAnExternalRepositoryIsClonedExactlyAsGiven(t *testing.T) {
// Unaffected by the seat, held or not: GitHub and GitLab are the ordinary cases.
given := "https://github.com/someone/something.git"
got, err := cloneFrom(t.Context(), buildSource{Repository: given})
if err != nil {
t.Fatal(err)
}
if got != given {
t.Fatalf("an external repository became %s", got)
}
}
func TestAHolderOffThePrivateNetworkIsRefused(t *testing.T) {
world := forgeHolding(float64(3000))
world.Offered["git"][1].At = ""
if _, err := clonedFromSeat(world, "git", "novox/mesh-catalog"); err == nil ||
!strings.Contains(err.Error(), "private network") {
t.Fatalf("a forge nothing can reach was cloned from: %v", err)
}
}
func TestAHolderServingNoPortIsRefusedRatherThanGuessed(t *testing.T) {
// A default port would be the forge's address guessed, which is what this exists to stop.
if _, err := clonedFromSeat(forgeHolding(nil), "git", "novox/mesh-catalog"); err == nil {
t.Fatal("a port was guessed for a forge that serves none")
}
}
func TestAnAddressGivenAsAPathOnTheForgeIsRefused(t *testing.T) {
for _, bad := range []string{
"https://github.com/someone/something.git",
"git@anchor:novox/mesh-catalog.git",
"/srv/git/mesh-catalog",
"",
} {
if err := onASeat(bad); err == nil {
t.Errorf("--self accepted %q as a path on the forge", bad)
}
}
if err := onASeat("novox/mesh-catalog"); err != nil {
t.Errorf("a path on the forge was refused: %v", err)
}
}
func TestASourceOnTheSeatReadsAsAPathNotAnAddress(t *testing.T) {
s := buildSource{Repository: "novox/mesh-catalog", Seat: "git"}
if got := s.String(); got != "novox/mesh-catalog on the git seat" {
t.Fatalf("read as %q", got)
}
}
+1 -1
View File
@@ -184,7 +184,7 @@ func (f following) Announceable(ctx context.Context) ([]link.Announcement, error
if err != nil {
return nil, err
}
address, err := whereTheStoreIs(ctx, f.open.inventory)
address, err := whereTheStoreIs(ctx, f.open.inventory, "")
if err != nil {
return nil, err
}
+109
View File
@@ -0,0 +1,109 @@
package main
// The challenge path falls through for real. autocert's own HTTPHandler answers 404 itself for a
// token it does not hold and never consults its fallback on the challenge path — the
// predecessor's fault, the edge owning /.well-known/acme-challenge outright, rediscovered live
// when Mailu's renewal died behind this proxy on cutover day (2026-09-26). These tests pin the
// three behaviours tokenOrRoute exists for.
import (
"context"
"fmt"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"testing"
"golang.org/x/crypto/acme/autocert"
)
func routedTo(t *testing.T, marker string) http.Handler {
t.Helper()
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusOK)
if _, err := w.Write([]byte(marker)); err != nil {
t.Fatal(err)
}
})
}
func TestATokenNoAuthorityHoldsIsRoutedNot404d(t *testing.T) {
m := &autocert.Manager{Prompt: autocert.AcceptTOS, Cache: autocert.DirCache(t.TempDir())}
h := tokenOrRoute(routedTo(t, "the workload answered"), m)
rec := httptest.NewRecorder()
h.ServeHTTP(rec, httptest.NewRequest("GET", "http://mail.example/.well-known/acme-challenge/somebody-elses-token", nil))
if rec.Code != http.StatusOK || rec.Body.String() != "the workload answered" {
t.Fatalf("a token no authority holds must reach plain routing; got %d %q", rec.Code, rec.Body.String())
}
}
func TestATokenAManagerHoldsIsAnsweredByIt(t *testing.T) {
// autocert reads a token it does not have in memory from its cache, under "<token>+http-01" —
// which is also how a token would survive the manager restarting mid-issuance.
dir := t.TempDir()
if err := os.WriteFile(filepath.Join(dir, "held-token+http-01"), []byte("the-key-authorization"), 0o600); err != nil {
t.Fatal(err)
}
m := &autocert.Manager{Prompt: autocert.AcceptTOS, Cache: autocert.DirCache(dir)}
h := tokenOrRoute(routedTo(t, "must not be reached"), m)
rec := httptest.NewRecorder()
h.ServeHTTP(rec, httptest.NewRequest("GET", "http://mail.example/.well-known/acme-challenge/held-token", nil))
if rec.Code != http.StatusOK || rec.Body.String() != "the-key-authorization" {
t.Fatalf("the manager holding a token answers it; got %d %q", rec.Code, rec.Body.String())
}
}
func TestASecondAuthorityIsProbedBeforeRouting(t *testing.T) {
first := &autocert.Manager{Prompt: autocert.AcceptTOS, Cache: autocert.DirCache(t.TempDir())}
dir := t.TempDir()
if err := os.WriteFile(filepath.Join(dir, "internal-token+http-01"), []byte("internal-key"), 0o600); err != nil {
t.Fatal(err)
}
second := &autocert.Manager{Prompt: autocert.AcceptTOS, Cache: autocert.DirCache(dir)}
h := tokenOrRoute(routedTo(t, "must not be reached"), first, second)
rec := httptest.NewRecorder()
h.ServeHTTP(rec, httptest.NewRequest("GET", "http://git.internal/.well-known/acme-challenge/internal-token", nil))
if rec.Code != http.StatusOK || rec.Body.String() != "internal-key" {
t.Fatalf("the second authority's token is found by probing past the first; got %d %q", rec.Code, rec.Body.String())
}
}
func TestAnAuthorityWhosePolicyRefusesTheNameIsProbedPast(t *testing.T) {
// autocert checks the host policy before the token and answers 403 — the internal authority
// does this for every public name. A policy refusal is as much "not mine" as a missing token:
// the request must still reach plain routing, where the workload's own ACME client answers.
refusing := &autocert.Manager{
Prompt: autocert.AcceptTOS,
Cache: autocert.DirCache(t.TempDir()),
HostPolicy: func(ctx context.Context, host string) error {
return fmt.Errorf("no internal-only route for %q in this mesh", host)
},
}
h := tokenOrRoute(routedTo(t, "the workload answered"), refusing)
rec := httptest.NewRecorder()
h.ServeHTTP(rec, httptest.NewRequest("GET", "http://mail.example/.well-known/acme-challenge/mailus-token", nil))
if rec.Code != http.StatusOK || rec.Body.String() != "the workload answered" {
t.Fatalf("a policy refusal must fall through to routing; got %d %q", rec.Code, rec.Body.String())
}
}
func TestAnOrdinaryPathNeverTouchesTheChallengeMachinery(t *testing.T) {
m := &autocert.Manager{Prompt: autocert.AcceptTOS, Cache: autocert.DirCache(t.TempDir())}
h := tokenOrRoute(routedTo(t, "routed"), m)
rec := httptest.NewRecorder()
h.ServeHTTP(rec, httptest.NewRequest("GET", "http://site.example/index.html", nil))
if rec.Code != http.StatusOK || rec.Body.String() != "routed" {
t.Fatalf("an ordinary path goes straight to routing; got %d %q", rec.Code, rec.Body.String())
}
}
+624 -73
View File
@@ -10,10 +10,31 @@
// program. What lives here is that contract, written as something that runs so it can be read
// rather than described.
//
// **A route also carries what a request arriving at it may do** (novox/hq ADR 0108). The grant used
// to say only where to send traffic, so this proxy applied nothing; the four things the ingress it
// replaces actually relies on are now part of the contribution. The set is closed at four, because
// an open middleware surface recreates the thing being replaced and is far harder to narrow later
// than a closed one is to widen.
//
// What it is given, written by the host from an ordinary declaration:
//
// $ROUTES every consumer, the name it asked for, and where the mesh says that machine is
//
// Each contribution's values carry the name and port as before, and optionally:
//
// path the path prefix this rule is scoped to; absent means every path
// priority which rule wins where two match; higher first, and the order is total
// deny refuse the request outright — the shape an incident mitigation needs
// redirect answer with a permanent redirect to this name, keeping the path and query
// auth the *path of a secret* holding `user:hash` lines, never the credential itself
//
// A host may appear more than once, which is what path scoping means: one rule refusing a path
// while another serves everything else on the same name.
//
// **`auth` names a secret and never holds one.** A declaration carrying a credential is refused
// outright rather than served unprotected, and a secret that cannot be read makes the route refuse
// rather than open — a gate that cannot check is not a gate that opens.
//
// It re-reads on change rather than being restarted, for the same reason the provisioner does:
// a route arriving or leaving is an ordinary event and must not drop the connections of every
// other workload.
@@ -23,6 +44,7 @@ import (
"bytes"
"context"
"crypto/sha256"
"crypto/subtle"
"crypto/tls"
"crypto/x509"
"encoding/hex"
@@ -42,6 +64,7 @@ import (
"golang.org/x/crypto/acme"
"golang.org/x/crypto/acme/autocert"
"golang.org/x/crypto/bcrypt"
)
// Where public certificates come from when nothing says otherwise.
@@ -74,10 +97,23 @@ func issuer() string {
// to what it may serve.
func onlyWhatTheMeshSaid(held *table) autocert.HostPolicy {
return func(_ context.Context, host string) error {
if _, known := held.find(host); known {
if held.eligibleForACME(host) {
return nil
}
return fmt.Errorf("no route for %q in this mesh, so no certificate is asked for", host)
return fmt.Errorf("no public route for %q in this mesh, so no certificate is asked for", host)
}
}
// onlyInternalNamesTheMeshSaid is onlyWhatTheMeshSaid's mirror for the internal authority — the
// same quota-spending concern applies even to an authority with no rate limit of its own, because
// an order for a name this proxy does not actually route is a bug worth refusing rather than
// serving.
func onlyInternalNamesTheMeshSaid(held *table) autocert.HostPolicy {
return func(_ context.Context, host string) error {
if held.eligibleForInternalACME(host) {
return nil
}
return fmt.Errorf("no internal-only route for %q in this mesh, so no certificate is asked for", host)
}
}
@@ -95,48 +131,194 @@ type contribution struct {
Values map[string]any `json:"values"`
}
// policy is what a rule does with a request that matched it.
//
// **Decided by the mesh, not here** (novox/hq ADR 0108). A route grant used to hand back a name and
// say nothing about what the name admitted, so this proxy admitted everything. The set is closed at
// four — authentication, refusal, path scoping, redirect — because an open middleware surface
// recreates the thing being replaced and is far harder to narrow later than a closed one is to widen.
type policy struct {
// deny refuses the request outright, whatever it is.
deny bool
// redirectTo answers with a permanent redirect instead of proxying. The request's own path and
// query are carried across, which is what canonicalising one public name onto another means.
redirectTo string
// users is what a request must present, read at load time from the secret the declaration
// *named*. A declaration never carries the credential itself.
users map[string]string
// sealed is set when authentication was declared and the secret could not be read. The rule then
// refuses everything and says why.
//
// **Fail closed.** The alternative — serve the route unauthenticated because the gate is
// missing — turns an unreadable file into a silently public admin surface, which is the exact
// outcome ADR 0108 exists to prevent. A gate that cannot check is not a gate that opens.
sealed string
}
// rule is one way a host may be routed. A host may have several, which is what path scoping means.
type rule struct {
path string // "" matches every path
priority int
policy policy
to *httputil.ReverseProxy
target string
// insecure skips certificate verification when target is reached over https. For a backend
// that terminates TLS with its own certificate this proxy has no reason to trust — Mailu's
// webmail front is the first of these — never for anything reached over plain http, where
// there is nothing to verify in the first place.
insecure bool
// maxRequestBody is the largest body, in bytes, this route carries. Zero is no limit, which is
// what every route gets by saying nothing: this proxy has never limited a body, and a default
// arriving with the field would change every route that never asked for one.
//
// **Configuration, not a policy** (novox/hq ADR 0108 closed that set at four). It belongs beside
// `insecure` for the same reason `insecure` is not a policy: both tune how this proxy carries a
// request to a backend, rather than deciding what the name admits or who may reach it. A
// registry is the case that needs it — image layers arrive as single requests of gigabytes, and
// a proxy's own default refuses them long before the workload is reached.
maxRequestBody int64
}
// table is what the proxy is currently serving, replaced whole whenever the file changes.
//
// Replaced rather than merged: the file is the whole truth about who has a route, so merging
// would keep serving a name whose module was unassigned — which is the stale-route fault
// 08-connectivity lists as open, reintroduced one level down.
//
// Keyed by host to an *ordered* list rather than to one target, because two of the four policies
// need a single host routed more than one way: a refusal on a path the ordinary route also matches,
// and a certificate-challenge path on a host that otherwise serves a workload.
type table struct {
mu sync.RWMutex
to map[string]*httputil.ReverseProxy
targets map[string]string
mu sync.RWMutex
to map[string][]rule
// public is which routed hosts are eligible for a real certificate — every host reached as a
// route's own `name`, never one reached only as its `internal-name`. A private alias can never
// pass ACME's own validation (it has no public DNS to prove it against), so asking for it is
// not merely pointless but the failing order onlyWhatTheMeshSaid exists to prevent.
public map[string]bool
}
func (t *table) set(routes map[string]string) {
made := map[string]*httputil.ReverseProxy{}
for name, target := range routes {
where, err := url.Parse(target)
if err != nil {
log.Printf("route %s points at %q, which is not a URL: %v", name, target, err)
func (t *table) set(routes map[string][]rule, public map[string]bool) {
made := map[string][]rule{}
for host, rules := range routes {
kept := make([]rule, 0, len(rules))
for _, r := range rules {
// A rule that only refuses or only redirects has nowhere to send anything, and needs
// nowhere: it answers by itself.
if r.policy.deny || r.policy.redirectTo != "" {
kept = append(kept, r)
continue
}
where, err := url.Parse(r.target)
if err != nil {
log.Printf("route %s points at %q, which is not a URL: %v", host, r.target, err)
continue
}
r.to = httputil.NewSingleHostReverseProxy(where)
if r.insecure {
r.to.Transport = &http.Transport{TLSClientConfig: &tls.Config{InsecureSkipVerify: true}}
}
kept = append(kept, r)
}
if len(kept) == 0 {
continue
}
made[name] = httputil.NewSingleHostReverseProxy(where)
inOrder(kept)
made[host] = kept
}
t.mu.Lock()
t.to, t.targets = made, routes
t.to = made
t.public = public
t.mu.Unlock()
}
func (t *table) find(host string) (*httputil.ReverseProxy, bool) {
// The port is not part of the name. A request to app.example:8080 is for app.example.
// inOrder puts the rules for one host into the order they are matched in, and does so totally.
//
// **Equal priorities must resolve identically every time** (ADR 0108). Sorting only by priority
// leaves rules that share one in whatever order the map produced, so the same declaration would
// serve differently between restarts — a proxy that is not reproducible. Longest path first within a
// priority is also the intuitive reading: the more specific rule wins. The last two keys exist only
// to make the order total.
func inOrder(rules []rule) {
sort.SliceStable(rules, func(i, j int) bool {
a, b := rules[i], rules[j]
if a.priority != b.priority {
return a.priority > b.priority
}
if len(a.path) != len(b.path) {
return len(a.path) > len(b.path)
}
if a.path != b.path {
return a.path < b.path
}
return a.target < b.target
})
}
// find is the rule that answers this request, or nothing if the host is not routed here at all.
func (t *table) find(host, path string) (rule, bool) {
t.mu.RLock()
defer t.mu.RUnlock()
for _, r := range t.to[bareHost(host)] {
if r.path == "" || strings.HasPrefix(path, r.path) {
return r, true
}
}
return rule{}, false
}
// routed says whether this proxy serves the name at all, whatever the path.
//
// Separate from find because certificate issuance is a question about the *name*: a host whose only
// rules are path-scoped is still a name this proxy answers to, and still needs a certificate.
// eligibleForACME says whether this proxy may ask a certificate authority for this name — every
// host reached as a route's own public `name`, never one reached only as its `internal-name`
// alias, which no public CA can ever validate.
func (t *table) eligibleForACME(host string) bool {
t.mu.RLock()
defer t.mu.RUnlock()
bare := bareHost(host)
return len(t.to[bare]) > 0 && t.public[bare]
}
func (t *table) routed(host string) bool {
t.mu.RLock()
defer t.mu.RUnlock()
return len(t.to[bareHost(host)]) > 0
}
// eligibleForInternalACME says whether this proxy may ask its *internal* authority for a
// certificate for this name — every host it routes that is not also a route's public `name`.
//
// **The mesh has two name spaces and two authorities** (novox/hq 03-DESIGN/01-to-be/08-connectivity
// §2): a public name is certified by a public CA, an internal one by the mesh's own. This is
// composed only from `to` and `public`, which routesFrom already builds correctly — a host never
// lands in both a route's own `name` and only its `internal-name`, so nothing new has to be
// tracked to tell the two apart.
func (t *table) eligibleForInternalACME(host string) bool {
t.mu.RLock()
defer t.mu.RUnlock()
bare := bareHost(host)
return len(t.to[bare]) > 0 && !t.public[bare]
}
// bareHost is the name without the port, lower-cased.
//
// The port is not part of the name: a request to app.example:8080 is for app.example. Lower-cased
// because a Host header is not case-sensitive, and a route that only answers the spelling in the
// manifest answers half the requests made to it.
func bareHost(host string) string {
if h, _, err := net.SplitHostPort(host); err == nil {
host = h
}
t.mu.RLock()
defer t.mu.RUnlock()
p, ok := t.to[strings.ToLower(host)]
return p, ok
return strings.ToLower(host)
}
func (t *table) names() []string {
t.mu.RLock()
defer t.mu.RUnlock()
out := make([]string, 0, len(t.targets))
for name := range t.targets {
out := make([]string, 0, len(t.to))
for name := range t.to {
out = append(out, name)
}
sort.Strings(out)
@@ -162,7 +344,7 @@ func run() error {
held := newTable()
read := func() {
routes, err := routesFrom(path)
routes, public, err := routesFrom(path)
if err != nil {
// Kept serving what it had. A file being rewritten is momentarily unreadable, and
// dropping every route because one read landed mid-write would turn an ordinary
@@ -170,7 +352,7 @@ func run() error {
log.Printf("cannot read %s, keeping what is already served: %v", path, err)
return
}
held.set(routes)
held.set(routes, public)
log.Printf("serving %d route(s): %s", len(routes), strings.Join(held.names(), ", "))
}
read()
@@ -197,16 +379,158 @@ func run() error {
return fmt.Errorf("TLS_LISTEN is set and ACME_CACHE is not: certificates need somewhere " +
"to persist, or every restart orders them again")
}
client := &acme.Client{DirectoryURL: issuer()}
// An issuer that is not one of the public ones serves its own API over TLS with a certificate
// nothing trusts yet — the lab's, or an internal step-ca. Trusting it is a deliberate act and
// names a file, rather than the client being told to skip verification: *skip* would also
// apply on the day this points at a public issuer, and nothing would say so.
publicManager, err := newManager(cache, issuer(), strings.TrimSpace(os.Getenv("ACME_CA_BUNDLE")),
onlyWhatTheMeshSaid(held))
if err != nil {
return err
}
log.Printf("issuing public certificates from %s, for whatever the mesh routes here", issuer())
// The internal authority is optional: unset means this proxy serves internal-only aliases over
// plain HTTP exactly as it always has, which is the standalone-binary default and a safe one —
// it asks nothing of an authority it was not told about.
var internalManager *autocert.Manager
if directory := strings.TrimSpace(os.Getenv("INTERNAL_ACME_DIRECTORY")); directory != "" {
internalManager, err = newManager(cache, directory, strings.TrimSpace(os.Getenv("INTERNAL_ACME_CA_BUNDLE")),
onlyInternalNamesTheMeshSaid(held))
if err != nil {
return fmt.Errorf("internal certificate authority: %w", err)
}
log.Printf("issuing internal certificates from %s, for every internal-only alias this routes",
directory)
}
// Port 80 answers the HTTP-01 challenge and goes on proxying everything else. The challenge
// must be answered *at the name being certified*, which is why issuance happens on the node
// that is publicly reachable rather than wherever the workload runs.
//
// **autocert's own HTTPHandler does not fall through on the challenge path.** For a token it
// does not hold it answers 404 itself; its fallback only ever sees non-challenge paths — which
// is exactly the predecessor's fault, the edge owning `/.well-known/acme-challenge` outright,
// rediscovered live when Mailu's renewal died behind this proxy on cutover day. tokenOrRoute
// probes each manager and hands a token neither authority recognises to plain routing, which
// is what lets a consumer's own ACME client — Mailu's, certifying its own name for a protocol
// this proxy never proxies — answer its own challenge through an ordinary path-scoped route.
port80 := tokenOrRoute(handler(held), publicManager)
if internalManager != nil {
port80 = tokenOrRoute(handler(held), publicManager, internalManager)
}
go func() {
if err := http.ListenAndServe(listen, port80); err != nil {
log.Printf("plain HTTP stopped: %v", err)
}
}()
tlsConfig := publicManager.TLSConfig()
if internalManager != nil {
// Dispatched by which authority may certify this name at all — the same question
// eligibleForInternalACME already answers, asked once more at handshake time rather than
// only when an order is placed, since a cached certificate is served here on every request
// and never goes through HostPolicy again.
fromPublic, fromInternal := tlsConfig.GetCertificate, internalManager.TLSConfig().GetCertificate
tlsConfig.GetCertificate = func(hello *tls.ClientHelloInfo) (*tls.Certificate, error) {
if held.eligibleForInternalACME(hello.ServerName) {
return fromInternal(hello)
}
return fromPublic(hello)
}
}
server := &http.Server{
Addr: secure,
Handler: handler(held),
TLSConfig: tlsConfig,
}
return server.ListenAndServeTLS("", "")
}
// tokenOrRoute serves port 80: each manager answers the challenge tokens it is itself holding,
// and a token none of them holds is routed like any other request instead of being 404'd at the
// edge.
//
// autocert gives no way to ask "is this your token?" — its HTTPHandler both answers and refuses —
// so each manager is probed against a buffered writer and its refusal (404 on the challenge path)
// is discarded in favour of the next candidate. The probe is cheap: the handler answers from
// memory, and the path only carries traffic while an issuance is actually running.
func tokenOrRoute(routes http.Handler, managers ...*autocert.Manager) http.Handler {
const challengePrefix = "/.well-known/acme-challenge/"
// Non-challenge paths never reach a manager at all; autocert's tryHTTP01 switch still has to
// be armed, which HTTPHandler is the only exported way to do.
probes := make([]http.Handler, len(managers))
for i, m := range managers {
probes[i] = m.HTTPHandler(routes)
}
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if !strings.HasPrefix(r.URL.Path, challengePrefix) {
routes.ServeHTTP(w, r)
return
}
for _, probe := range probes {
buffered := &probedResponse{header: make(http.Header)}
probe.ServeHTTP(buffered, r)
// Two shapes of "not mine": 404, a token this manager is not holding — and 403, a
// name its host policy would never certify at all (autocert checks the policy before
// the token, so the internal authority answers 403 for every public name).
if buffered.status == http.StatusNotFound || buffered.status == http.StatusForbidden {
continue
}
buffered.replayTo(w)
return
}
routes.ServeHTTP(w, r) // no authority holds it: the workload behind a routed path may
})
}
// probedResponse buffers one handler's answer so a refusal can be discarded unseen.
type probedResponse struct {
header http.Header
status int
body bytes.Buffer
}
func (p *probedResponse) Header() http.Header { return p.header }
func (p *probedResponse) WriteHeader(status int) {
if p.status == 0 {
p.status = status
}
}
func (p *probedResponse) Write(b []byte) (int, error) {
if p.status == 0 {
p.status = http.StatusOK
}
return p.body.Write(b)
}
func (p *probedResponse) replayTo(w http.ResponseWriter) {
for k, vs := range p.header {
for _, v := range vs {
w.Header().Add(k, v)
}
}
status := p.status
if status == 0 {
status = http.StatusOK
}
w.WriteHeader(status)
_, _ = w.Write(p.body.Bytes())
}
// newManager is one ACME authority's autocert manager: where to ask, what to trust it with, and
// which names it may be asked to certify.
//
// **Trusting an authority names a file rather than skipping verification.** An issuer that is not
// one of the public ones — the lab's, or the mesh's own step-ca — serves its own ACME API over TLS
// with a certificate nothing trusts yet. *Skip* would also apply the day this points at a public
// issuer, and nothing would say so; naming a bundle is a deliberate, visible act instead.
func newManager(cache, directory, bundle string, policy autocert.HostPolicy) (*autocert.Manager, error) {
client := &acme.Client{DirectoryURL: directory}
var root []byte
if bundle := strings.TrimSpace(os.Getenv("ACME_CA_BUNDLE")); bundle != "" {
if bundle != "" {
read, err := os.ReadFile(bundle)
if err != nil {
return fmt.Errorf("ACME_CA_BUNDLE names %s and it cannot be read: %w", bundle, err)
return nil, fmt.Errorf("the CA bundle names %s and it cannot be read: %w", bundle, err)
}
root = read
// An empty bundle means the issuer's root is already in the system trust store — a public
@@ -218,7 +542,7 @@ func run() error {
if strings.TrimSpace(string(root)) != "" {
pool := x509.NewCertPool()
if !pool.AppendCertsFromPEM(root) {
return fmt.Errorf("%s holds no certificate this can trust", bundle)
return nil, fmt.Errorf("%s holds no certificate this can trust", bundle)
}
client.HTTPClient = &http.Client{
Timeout: 30 * time.Second,
@@ -227,31 +551,16 @@ func run() error {
}
}
// Where this authority's account and certificates are kept. Per authority, not per proxy — see
// forThisAuthority, which is what makes a re-initialised CA heal itself.
mine := forThisAuthority(cache, issuer(), root)
manager := &autocert.Manager{
// forThisAuthority, which is what makes a re-initialised CA heal itself, and what lets the
// public and internal authorities share one ACME_CACHE without colliding: they hash to
// different names because their directories differ.
mine := forThisAuthority(cache, directory, root)
return &autocert.Manager{
Cache: autocert.DirCache(mine),
Prompt: autocert.AcceptTOS,
HostPolicy: onlyWhatTheMeshSaid(held),
HostPolicy: policy,
Client: client,
}
log.Printf("issuing from %s into %s, for whatever the mesh routes here", issuer(), mine)
// Port 80 answers the HTTP-01 challenge and goes on proxying everything else. The challenge
// must be answered *at the name being certified*, which is why issuance happens on the node
// that is publicly reachable rather than wherever the workload runs.
go func() {
if err := http.ListenAndServe(listen, manager.HTTPHandler(handler(held))); err != nil {
log.Printf("plain HTTP stopped: %v", err)
}
}()
server := &http.Server{
Addr: secure,
Handler: handler(held),
TLSConfig: manager.TLSConfig(),
}
return server.ListenAndServeTLS("", "")
}, nil
}
// forThisAuthority is where one ACME authority's account and certificates are kept.
@@ -285,61 +594,303 @@ func forThisAuthority(cache, directory string, root []byte) string {
// newTable is an empty routing table.
func newTable() *table {
return &table{to: map[string]*httputil.ReverseProxy{}, targets: map[string]string{}}
return &table{to: map[string][]rule{}}
}
// handler is the proxy itself, separated so it can be driven by a test without a listener.
func handler(held *table) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
proxy, known := held.find(r.Host)
matched, known := held.find(r.Host, r.URL.Path)
if !known {
// **Named, not a bare 404.** A route that was withdrawn and a name that never existed
// are different things, and a proxy that says only "not found" makes an operator go
// and read the mesh to tell them apart. What it is serving is the answer to both.
//
// And since a host may now be routed only on some paths, those are a third thing:
// saying "no route for this name" while listing that very name as served is a
// contradiction an operator would have to disbelieve the proxy to get past.
w.Header().Set("Content-Type", "text/plain; charset=utf-8")
w.WriteHeader(http.StatusNotFound)
if held.routed(r.Host) {
fmt.Fprintf(w, "%s is served here, but no route covers %q.\n",
bareHost(r.Host), r.URL.Path)
return
}
fmt.Fprintf(w, "no route for %q in this mesh.\nserving: %s\n",
r.Host, strings.Join(held.names(), ", "))
return
}
proxy.ServeHTTP(w, r)
switch {
case matched.policy.sealed != "":
// Declared a gate, cannot check it. Refused, and says why — an operator reading this
// learns the secret is missing, rather than wondering why a protected name is 503.
w.Header().Set("Content-Type", "text/plain; charset=utf-8")
w.WriteHeader(http.StatusServiceUnavailable)
fmt.Fprintf(w, "this route requires authentication and its credentials cannot be read: %s\n",
matched.policy.sealed)
return
case matched.policy.deny:
http.Error(w, "this path is not served to you", http.StatusForbidden)
return
case matched.policy.redirectTo != "":
http.Redirect(w, r, canonical(matched.policy.redirectTo, r.URL), http.StatusMovedPermanently)
return
case len(matched.policy.users) > 0 && !allowed(matched.policy.users, r):
// The realm is the name asked for, so a browser's prompt says which route it is for.
w.Header().Set("WWW-Authenticate", fmt.Sprintf("Basic realm=%q, charset=\"UTF-8\"", bareHost(r.Host)))
http.Error(w, "unauthorized", http.StatusUnauthorized)
return
}
if matched.maxRequestBody > 0 {
// Refused on the declared length where there is one, so an upload that cannot succeed
// is answered before it is carried; and capped while reading for a chunked body, which
// declares no length at all. Without the second, a limit is advice.
if r.ContentLength > matched.maxRequestBody {
http.Error(w, fmt.Sprintf("request body too large for this route: %d bytes is the most it carries",
matched.maxRequestBody), http.StatusRequestEntityTooLarge)
return
}
r.Body = http.MaxBytesReader(w, r.Body, matched.maxRequestBody)
}
matched.to.ServeHTTP(w, r)
})
}
// routesFrom reads what the mesh wrote and turns it into name → target.
func routesFrom(path string) (map[string]string, error) {
// canonical is where a redirect sends this request.
//
// The declaration names the destination *name*; the request keeps its own path and query. That is
// what canonicalising one public name onto another means — a link to a page under the old name has
// to arrive at the same page under the new one, or the redirect silently loses every deep link.
func canonical(to string, from *url.URL) string {
where, err := url.Parse(to)
if err != nil {
return to
}
if where.Path == "" || where.Path == "/" {
where.Path = from.Path
}
if where.RawQuery == "" {
where.RawQuery = from.RawQuery
}
return where.String()
}
// allowed says whether the request presented credentials this route accepts.
//
// **Every path costs one bcrypt comparison**, including an unknown user, which is why the miss
// compares against a fixed hash rather than returning early. Returning early would make an unknown
// user measurably faster than a known one with a wrong password, and that difference is a way to
// enumerate the users of a route from outside it.
func allowed(users map[string]string, r *http.Request) bool {
// A hash of nothing anybody knows. Its only job is to cost what a real comparison costs.
const absent = "$2a$10$N9qo8uLOickgx2ZMRZoMyeIjZAgcfl7p92ldGxad68LJZdL17lhWy"
user, password, ok := r.BasicAuth()
if !ok {
return false
}
want, known := users[user]
if !known {
want = absent
}
if err := bcrypt.CompareHashAndPassword([]byte(want), []byte(password)); err != nil {
return false
}
// `known` is checked after the comparison, not instead of it, so the timing is the same either
// way. subtle.ConstantTimeByteEq keeps the branch from being the thing that differs.
return subtle.ConstantTimeByteEq(boolByte(known), 1) == 1
}
func boolByte(b bool) byte {
if b {
return 1
}
return 0
}
// routesFrom reads what the mesh wrote and turns it into host → the rules for that host, and
// which of those hosts is a public name — the second is `name`, ACME-eligible; a host reached
// only through `internal-name` never appears there.
func routesFrom(path string) (map[string][]rule, map[string]bool, error) {
raw, err := os.ReadFile(path)
if err != nil {
return nil, err
return nil, nil, err
}
var said given
if err := json.Unmarshal(raw, &said); err != nil {
return nil, err
return nil, nil, err
}
out := map[string]string{}
out := map[string][]rule{}
public := map[string]bool{}
for _, c := range said.Given {
name, _ := c.Values["name"].(string)
if name == "" {
log.Printf("%s on %s asked for a route and named nothing; skipped", c.From, c.Node)
continue
}
port, ok := asPort(c.Values["port"])
if !ok {
log.Printf("%s on %s asked for route %q and gave no usable port; skipped",
c.From, c.Node, name)
host := strings.ToLower(name)
public[host] = true
made := rule{path: asPath(c.Values["path"])}
if p, ok := asWhole(c.Values["priority"]); ok {
made.priority = p
}
made.policy.deny, _ = c.Values["deny"].(bool)
made.policy.redirectTo, _ = c.Values["redirect"].(string)
if named, carried := c.Values["auth"].(string); carried && strings.TrimSpace(named) != "" {
// **A declaration names a secret; it never holds one** (ADR 0108). Refused rather than
// tolerated, and the whole rule is dropped rather than served unprotected — the
// rejected option cannot come back by accident, which is the failure this check exists
// to make impossible.
if looksLikeACredential(named) {
log.Printf("%s on %s declared route %q with a credential in the declaration rather "+
"than the name of a secret; the whole route is refused (novox/hq ADR 0108)",
c.From, c.Node, name)
continue
}
users, err := usersFrom(named)
if err != nil {
// Fail closed: the rule is kept so the name stays routed and answers, and it
// answers by refusing. Dropping it instead would make the name 404 and read as a
// withdrawn route rather than an unreadable secret.
made.policy.sealed = err.Error()
}
made.policy.users = users
}
// Only a rule that actually proxies needs somewhere to send the request.
if !made.policy.deny && made.policy.redirectTo == "" {
port, ok := asPort(c.Values["port"])
if !ok {
log.Printf("%s on %s asked for route %q and gave no usable port; skipped",
c.From, c.Node, name)
continue
}
// Where the mesh says that machine is. Empty means it is this one — a workload beside
// the proxy is ordinary, and reaching it over loopback is both correct and the only
// thing that works when there is no private network.
at := c.At
if at == "" {
at = "127.0.0.1"
}
// http unless the contribution says otherwise. A backend that terminates its own TLS
// with a certificate this proxy has no reason to trust — Mailu's webmail front is the
// first of these — is the reason `insecure` exists, and it stays the exception: every
// other target the mesh hands this proxy is a plain workload on the private network.
scheme, _ := c.Values["scheme"].(string)
scheme = strings.ToLower(strings.TrimSpace(scheme))
if scheme == "" {
scheme = "http"
}
if scheme != "http" && scheme != "https" {
log.Printf("%s on %s asked for route %q with scheme %q, which is neither http "+
"nor https; skipped", c.From, c.Node, name, scheme)
continue
}
made.insecure, _ = c.Values["insecure"].(bool)
// A limit this proxy cannot read is a route it does not serve, named like a port that
// is not a port. Serving it without the limit would carry exactly what the module said
// not to carry, and report success doing it.
if asked, said := c.Values["max-request-body"]; said {
bytes, whole := asWhole(asked)
if !whole || bytes <= 0 {
log.Printf("%s on %s asked for route %q with a max-request-body of %v, which is "+
"not a whole positive number of bytes; skipped", c.From, c.Node, name, asked)
continue
}
made.maxRequestBody = int64(bytes)
}
made.target = fmt.Sprintf("%s://%s:%d", scheme, at, port)
}
out[host] = append(out[host], made)
// The internal-network alias, the same rule under a second host — a predecessor proxy
// answered both for one route, as a convenience (reaching a service over the VPN without a
// public TLS round trip), not as an access boundary; composing it here restores exactly
// that, nothing more. Absent whenever the node composed no internal name (novox/hq ADR
// 0056's internalDomain half) — the same "nothing to join a label to" case the public name
// already has.
if internal, _ := c.Values["internal-name"].(string); strings.TrimSpace(internal) != "" {
out[strings.ToLower(internal)] = append(out[strings.ToLower(internal)], made)
}
}
return out, public, nil
}
// asWhole is any whole number the mesh wrote, whatever its magnitude.
//
// **Not asPort.** Priority was read with the port reader first, which caps at 65535 — so a rule
// declared at a priority above that silently became priority 0 and stopped shadowing the route it
// exists to shadow. The one real rule this has to reproduce is declared at 100000, so the bug was
// exactly load-bearing. A priority is an ordering, not a port: it has no range.
func asWhole(v any) (int, bool) {
switch n := v.(type) {
case float64:
// JSON makes a float of every number, so a non-integral one was not meant as a priority.
if n != float64(int(n)) {
return 0, false
}
return int(n), true
case int:
return n, true
}
return 0, false
}
// asPath is the path prefix a rule is scoped to, or "" for every path.
func asPath(v any) string {
p, _ := v.(string)
p = strings.TrimSpace(p)
if p == "" {
return ""
}
if !strings.HasPrefix(p, "/") {
p = "/" + p
}
return p
}
// looksLikeACredential is the check that keeps a secret out of a declaration.
//
// It errs towards refusing: a value holding a `:` (the htpasswd separator) or opening with a bcrypt
// identifier is a credential, not a path, and no filesystem path the mesh writes needs either. A
// false refusal is a loud log and a route that does not serve; a false accept is a credential
// committed to a declaration, which is the thing being prevented.
func looksLikeACredential(v string) bool {
v = strings.TrimSpace(v)
return strings.Contains(v, ":") || strings.HasPrefix(v, "$2")
}
// usersFrom reads the credentials the mesh mounted, in the one format every htpasswd already is.
func usersFrom(path string) (map[string]string, error) {
raw, err := os.ReadFile(path)
if err != nil {
return nil, fmt.Errorf("cannot read the secret named for this route: %w", err)
}
users := map[string]string{}
for _, line := range strings.Split(string(raw), "\n") {
line = strings.TrimSpace(line)
if line == "" || strings.HasPrefix(line, "#") {
continue
}
// Where the mesh says that machine is. Empty means it is this one — a workload beside the
// proxy is ordinary, and reaching it over loopback is both correct and the only thing
// that works when there is no private network.
at := c.At
if at == "" {
at = "127.0.0.1"
user, hash, ok := strings.Cut(line, ":")
if !ok || user == "" || hash == "" {
continue
}
out[strings.ToLower(name)] = fmt.Sprintf("http://%s:%d", at, port)
users[user] = hash
}
return out, nil
if len(users) == 0 {
return nil, fmt.Errorf("the secret named for this route holds no usable credentials")
}
return users, nil
}
// asPort accepts what JSON makes of a number, which is a float even when it was written 8080.
+273
View File
@@ -0,0 +1,273 @@
package main
import (
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"strconv"
"strings"
"testing"
"golang.org/x/crypto/bcrypt"
)
// What a route carries about the requests arriving at it — novox/hq ADR 0108.
//
// Each test here is one of the four capabilities that record closed the set at, plus the negative
// case it promised would be refused. The negative case is the one that rots quietly: nothing fails
// if it stops working, so nothing tells you it has.
// served starts a workload and gives back the host and port the mesh would have recorded for it.
func served(t *testing.T, body string) (string, int) {
t.Helper()
workload := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
_, _ = w.Write([]byte(body))
}))
t.Cleanup(workload.Close)
host, port, _ := strings.Cut(strings.TrimPrefix(workload.URL, "http://"), ":")
n, err := strconv.Atoi(port)
if err != nil {
t.Fatal(err)
}
return host, n
}
// ask makes one request through the proxy for a given name and path, without following redirects.
func ask(t *testing.T, proxy, name, path string, auth [2]string) *http.Response {
t.Helper()
req, err := http.NewRequest(http.MethodGet, proxy+path, nil)
if err != nil {
t.Fatal(err)
}
req.Host = name
if auth[0] != "" {
req.SetBasicAuth(auth[0], auth[1])
}
client := &http.Client{CheckRedirect: func(*http.Request, []*http.Request) error {
return http.ErrUseLastResponse
}}
answer, err := client.Do(req)
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = answer.Body.Close() })
return answer
}
func proxyFor(t *testing.T, routesJSON string) string {
t.Helper()
path := filepath.Join(t.TempDir(), "routes.json")
if err := os.WriteFile(path, []byte(routesJSON), 0o644); err != nil {
t.Fatal(err)
}
routes, public, err := routesFrom(path)
if err != nil {
t.Fatal(err)
}
held := newTable()
held.set(routes, public)
server := httptest.NewServer(handler(held))
t.Cleanup(server.Close)
return server.URL
}
// A refusal on a path shadows the ordinary route for that path and leaves every other path alone.
//
// **This is why path scoping is a prerequisite and not a sibling capability.** The rule being
// reproduced matches a path on a host that is already routed to a workload, so a table mapping a
// host to one target cannot express it at all — no amount of authentication or source filtering
// would have helped.
func TestARefusedPathShadowsTheRouteAndLeavesTheRestServed(t *testing.T) {
at, port := served(t, "the workload")
proxy := proxyFor(t, `{"given":[
{"from":"forge","node":"anchor","at":"`+at+`","values":{"name":"forge.example","port":`+strconv.Itoa(port)+`}},
{"from":"forge","node":"anchor","values":{"name":"forge.example","path":"/api/internal","priority":100,"deny":true}}
]}`)
if got := ask(t, proxy, "forge.example", "/api/internal/hook", [2]string{}).StatusCode; got != http.StatusForbidden {
t.Fatalf("the refused path answered %d, so the block that was put in front of it during an "+
"incident is not in front of it any more", got)
}
if got := ask(t, proxy, "forge.example", "/", [2]string{}).StatusCode; got != http.StatusOK {
t.Fatalf("refusing one path took the whole route with it: %d", got)
}
}
// A redirect answers with the redirect, and the request keeps its own path and query.
//
// Losing the path would turn canonicalising one name onto another into "every deep link now lands
// on the front page", which is the kind of breakage that produces no error anywhere.
func TestARedirectKeepsThePathAndQuery(t *testing.T) {
proxy := proxyFor(t, `{"given":[
{"from":"site","node":"anchor","values":{"name":"www.example","redirect":"https://example/"}}
]}`)
answer := ask(t, proxy, "www.example", "/deep/page?ref=1", [2]string{})
if answer.StatusCode != http.StatusMovedPermanently {
t.Fatalf("a declared redirect answered %d", answer.StatusCode)
}
where := answer.Header.Get("Location")
if !strings.Contains(where, "/deep/page") || !strings.Contains(where, "ref=1") {
t.Fatalf("the redirect dropped the path or the query: %q", where)
}
}
// Authentication refuses a request with no credentials, admits one with the right ones, and refuses
// the wrong ones — with the credentials read from the secret the declaration *named*.
func TestAuthenticationAdmitsOnlyWhatTheSecretSays(t *testing.T) {
at, port := served(t, "the console")
hash, err := bcrypt.GenerateFromPassword([]byte("correct horse"), bcrypt.MinCost)
if err != nil {
t.Fatal(err)
}
secret := filepath.Join(t.TempDir(), "console-auth")
if err := os.WriteFile(secret, []byte("# a comment\nadmin:"+string(hash)+"\n"), 0o600); err != nil {
t.Fatal(err)
}
proxy := proxyFor(t, `{"given":[
{"from":"console","node":"anchor","at":"`+at+`","values":{"name":"console.example","port":`+strconv.Itoa(port)+`,"auth":"`+secret+`"}}
]}`)
if got := ask(t, proxy, "console.example", "/", [2]string{}).StatusCode; got != http.StatusUnauthorized {
t.Fatalf("an admin surface with no login of its own answered %d without credentials", got)
}
if got := ask(t, proxy, "console.example", "/", [2]string{"admin", "wrong"}).StatusCode; got != http.StatusUnauthorized {
t.Fatalf("the wrong password answered %d", got)
}
if got := ask(t, proxy, "console.example", "/", [2]string{"admin", "correct horse"}).StatusCode; got != http.StatusOK {
t.Fatalf("the right password answered %d", got)
}
}
// The negative case ADR 0108 promised would be refused: a credential in the declaration.
//
// **Refused whole, not tolerated and not served unprotected.** A hash carried in a declaration was
// the rejected option; nothing in the running system should quietly accept it later, because the
// precedent is far easier to set than to withdraw. If this test is deleted the option returns and
// nothing else notices.
func TestACredentialInTheDeclarationIsRefusedRatherThanServed(t *testing.T) {
inline := []string{
`{"given":[{"from":"c","node":"n","at":"127.0.0.1","values":{"name":"c.example","port":8080,"auth":"admin:$2a$10$abcdefghijklmnopqrstuv"}}]}`,
`{"given":[{"from":"c","node":"n","at":"127.0.0.1","values":{"name":"c.example","port":8080,"auth":"$2a$10$abcdefghijklmnopqrstuv"}}]}`,
}
for _, body := range inline {
path := filepath.Join(t.TempDir(), "routes.json")
if err := os.WriteFile(path, []byte(body), 0o644); err != nil {
t.Fatal(err)
}
routes, _, err := routesFrom(path)
if err != nil {
t.Fatal(err)
}
if len(routes) != 0 {
t.Fatalf("a declaration carrying a credential was served anyway: %v", routes)
}
}
}
// Authentication declared, secret unreadable: the route refuses. It does not serve unprotected.
//
// **Fail closed.** The alternative turns a missing file into a silently public admin surface, which
// is the outcome the whole record exists to prevent. It answers rather than 404s, so an operator
// sees "cannot read the credentials" instead of concluding the route was withdrawn.
func TestAnUnreadableSecretFailsClosed(t *testing.T) {
at, port := served(t, "the console")
missing := filepath.Join(t.TempDir(), "not-mounted")
proxy := proxyFor(t, `{"given":[
{"from":"console","node":"anchor","at":"`+at+`","values":{"name":"console.example","port":`+strconv.Itoa(port)+`,"auth":"`+missing+`"}}
]}`)
answer := ask(t, proxy, "console.example", "/", [2]string{})
if answer.StatusCode == http.StatusOK {
t.Fatal("a route whose credentials could not be read served the workload unprotected")
}
if answer.StatusCode != http.StatusServiceUnavailable {
t.Fatalf("expected the route to say it cannot check, got %d", answer.StatusCode)
}
}
// Equal priorities resolve the same way every time, so the same declaration serves the same way
// after a restart.
//
// Sorting only by priority leaves rules that share one in whatever order the map produced. The
// proxy would still work, and would work differently between restarts — which is the hardest kind
// of fault to believe when it is reported.
func TestRulesThatShareAPriorityAreStillTotallyOrdered(t *testing.T) {
first := []rule{
{path: "/a", priority: 10, target: "http://x:1"},
{path: "/bb", priority: 10, target: "http://y:2"},
{path: "", priority: 10, target: "http://z:3"},
}
second := []rule{
{path: "", priority: 10, target: "http://z:3"},
{path: "/bb", priority: 10, target: "http://y:2"},
{path: "/a", priority: 10, target: "http://x:1"},
}
inOrder(first)
inOrder(second)
for i := range first {
if first[i].path != second[i].path || first[i].target != second[i].target {
t.Fatalf("two orderings of the same rules disagree at %d: %q vs %q",
i, first[i].path, second[i].path)
}
}
// And the more specific rule is matched first, which is the intuitive reading.
if first[0].path != "/bb" {
t.Fatalf("the longest path is not matched first: %q", first[0].path)
}
}
// Priority decides before path length does, so a rule can be made to win regardless of specificity.
func TestPriorityOutranksPathLength(t *testing.T) {
rules := []rule{
{path: "/very/long/path", priority: 1, target: "http://x:1"},
{path: "", priority: 100, target: "http://y:2"},
}
inOrder(rules)
if rules[0].priority != 100 {
t.Fatalf("a higher priority did not win: %+v", rules[0])
}
}
// A priority above a port number survives, because a priority is an ordering and not a port.
//
// **Found by review, and it was load-bearing.** Priority was first read with the port reader, which
// caps at 65535 — so a rule declared above that silently became priority 0 and stopped shadowing the
// route it exists to shadow. The one real rule this has to reproduce is declared at 100000, so the
// capability would have shipped looking complete and doing nothing.
func TestAPriorityAboveAPortNumberSurvives(t *testing.T) {
at, port := served(t, "the workload")
proxy := proxyFor(t, `{"given":[
{"from":"forge","node":"anchor","at":"`+at+`","values":{"name":"forge.example","port":`+strconv.Itoa(port)+`}},
{"from":"forge","node":"anchor","values":{"name":"forge.example","path":"/api/internal","priority":100000,"deny":true}}
]}`)
if got := ask(t, proxy, "forge.example", "/api/internal/hook", [2]string{}).StatusCode; got != http.StatusForbidden {
t.Fatalf("a rule declared at priority 100000 answered %d instead of refusing", got)
}
}
// A host routed only on some paths says so, rather than claiming the name is not served here.
//
// Saying "no route for this name" while listing that very name as served is a contradiction an
// operator has to disbelieve the proxy to get past — and path scoping makes it reachable, because a
// host can now have rules that none of this request's paths match.
func TestAHostRoutedOnlyOnSomePathsSaysSo(t *testing.T) {
proxy := proxyFor(t, `{"given":[
{"from":"forge","node":"anchor","values":{"name":"forge.example","path":"/api/internal","deny":true}}
]}`)
answer := ask(t, proxy, "forge.example", "/elsewhere", [2]string{})
if answer.StatusCode != http.StatusNotFound {
t.Fatalf("an uncovered path answered %d", answer.StatusCode)
}
body := make([]byte, 256)
n, _ := answer.Body.Read(body)
said := string(body[:n])
if !strings.Contains(said, "is served here") || !strings.Contains(said, "/elsewhere") {
t.Fatalf("the refusal does not distinguish an uncovered path from an unserved name: %q", said)
}
}
+308 -16
View File
@@ -2,6 +2,8 @@ package main
import (
"context"
"fmt"
"io"
"net/http"
"net/http/httptest"
"os"
@@ -19,10 +21,37 @@ func write(t *testing.T, body string) string {
return path
}
// plain is the table an ordinary set of routes makes: one host, one target, no policy.
func plain(routes map[string]string) map[string][]rule {
out := map[string][]rule{}
for host, target := range routes {
out[host] = []rule{{target: target}}
}
return out
}
// allPublic is every host in a routes map, ACME-eligible — the ordinary case for a test with no
// internal-name alias of its own to distinguish.
func allPublic(routes map[string][]rule) map[string]bool {
out := map[string]bool{}
for host := range routes {
out[host] = true
}
return out
}
// targetOf is where a host's first matching rule sends a request.
func targetOf(routes map[string][]rule, host string) string {
if rules := routes[host]; len(rules) > 0 {
return rules[0].target
}
return ""
}
// A route is a grant: the consumer supplies a target, and where that machine is comes from the
// mesh rather than from a naming convention the proxy has to know.
func TestARouteGoesToWhereTheMeshSaysTheConsumerIs(t *testing.T) {
routes, err := routesFrom(write(t, `{"contributions":1,"requirement":"route","given":[
routes, _, err := routesFrom(write(t, `{"contributions":1,"requirement":"route","given":[
{"from":"app","node":"laptop","at":"laptop.internal","values":{"name":"App.Example","port":8080}}
]}`))
if err != nil {
@@ -30,28 +59,67 @@ func TestARouteGoesToWhereTheMeshSaysTheConsumerIs(t *testing.T) {
}
// Lower-cased, because a Host header is not case-sensitive and a route that only answers the
// spelling in the manifest answers half the requests made to it.
if routes["app.example"] != "http://laptop.internal:8080" {
if targetOf(routes, "app.example") != "http://laptop.internal:8080" {
t.Fatalf("the route does not point at the consumer: %v", routes)
}
}
// A route with an internal-name alias is reachable under both hostnames, pointed at the same
// target — the same convenience a predecessor proxy gave for reaching a service over the VPN
// without a public TLS round trip.
func TestARouteWithAnInternalNameIsReachableUnderBoth(t *testing.T) {
routes, public, err := routesFrom(write(t, `{"given":[
{"from":"app","node":"anchor","at":"anchor.internal",
"values":{"name":"app.example","internal-name":"app.anchor.internal","port":8080}}
]}`))
if err != nil {
t.Fatal(err)
}
if targetOf(routes, "app.example") != "http://anchor.internal:8080" {
t.Fatalf("the public name does not point at the consumer: %v", routes)
}
if targetOf(routes, "app.anchor.internal") != "http://anchor.internal:8080" {
t.Fatalf("the internal alias does not point at the same consumer: %v", routes)
}
if !public["app.example"] {
t.Errorf("the public name is not eligible for a certificate: %v", public)
}
if public["app.anchor.internal"] {
t.Errorf("the internal alias is eligible for a certificate no public CA could ever issue: %v",
public)
}
}
// A route with no internal-name composed gets no second host — the ordinary case, unchanged.
func TestARouteWithNoInternalNameGetsNoAlias(t *testing.T) {
routes, _, err := routesFrom(write(t, `{"given":[
{"from":"app","node":"anchor","values":{"name":"app.example","port":8080}}
]}`))
if err != nil {
t.Fatal(err)
}
if len(routes) != 1 {
t.Fatalf("a route with no internal-name grew a second host: %v", routes)
}
}
// A workload beside the proxy is ordinary, and reaching it over loopback is both correct and the
// only thing that works when there is no private network.
func TestAConsumerOnTheProxysOwnMachineIsReachedOverLoopback(t *testing.T) {
routes, err := routesFrom(write(t, `{"given":[
routes, _, err := routesFrom(write(t, `{"given":[
{"from":"app","node":"anchor","values":{"name":"app.example","port":9000}}
]}`))
if err != nil {
t.Fatal(err)
}
if routes["app.example"] != "http://127.0.0.1:9000" {
if targetOf(routes, "app.example") != "http://127.0.0.1:9000" {
t.Fatalf("a workload on this machine was not reachable: %v", routes)
}
}
// Skipped rather than served wrongly. A route with no port would proxy to :0.
func TestAContributionMissingWhatARouteNeedsIsSkipped(t *testing.T) {
routes, err := routesFrom(write(t, `{"given":[
routes, _, err := routesFrom(write(t, `{"given":[
{"from":"a","node":"n","at":"n.internal","values":{"name":"no-port.example"}},
{"from":"b","node":"n","at":"n.internal","values":{"port":8080}},
{"from":"c","node":"n","at":"n.internal","values":{"name":"fine.example","port":8080}}
@@ -59,11 +127,73 @@ func TestAContributionMissingWhatARouteNeedsIsSkipped(t *testing.T) {
if err != nil {
t.Fatal(err)
}
if len(routes) != 1 || routes["fine.example"] == "" {
if len(routes) != 1 || targetOf(routes, "fine.example") == "" {
t.Fatalf("an unusable contribution was served: %v", routes)
}
}
// A route may name a target reached over https, for a backend that terminates its own TLS — the
// shape Mailu's webmail front needs, which this proxy reaches as a plain workload otherwise.
func TestARouteMayTargetHttps(t *testing.T) {
routes, _, err := routesFrom(write(t, `{"given":[
{"from":"mail","node":"anchor","at":"anchor.internal",
"values":{"name":"mail.example","port":7443,"scheme":"https","insecure":true}}
]}`))
if err != nil {
t.Fatal(err)
}
if targetOf(routes, "mail.example") != "https://anchor.internal:7443" {
t.Fatalf("an https target was not built as one: %v", routes)
}
if !routes["mail.example"][0].insecure {
t.Fatal("insecure was declared and not carried onto the rule")
}
}
// A scheme that is neither http nor https is refused rather than guessed at.
func TestARouteWithAnUnknownSchemeIsSkipped(t *testing.T) {
routes, _, err := routesFrom(write(t, `{"given":[
{"from":"a","node":"n","at":"n.internal","values":{"name":"bad.example","port":80,"scheme":"ftp"}}
]}`))
if err != nil {
t.Fatal(err)
}
if len(routes) != 0 {
t.Fatalf("a route with an unusable scheme was served: %v", routes)
}
}
// End to end: a backend terminating TLS with a certificate nothing would ordinarily trust is still
// reached when the route declared `insecure`, and the response comes back through unmodified.
func TestTheProxyReachesAnInsecureHttpsBackend(t *testing.T) {
workload := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
_, _ = w.Write([]byte("the workload, over its own TLS"))
}))
defer workload.Close()
target := strings.TrimPrefix(workload.URL, "https://")
held := newTable()
routes := map[string][]rule{"mail.example": {{target: "https://" + target, insecure: true}}}
held.set(routes, allPublic(routes))
proxy := httptest.NewServer(handler(held))
defer proxy.Close()
asked, err := http.NewRequest(http.MethodGet, proxy.URL, nil)
if err != nil {
t.Fatal(err)
}
asked.Host = "mail.example"
answer, err := http.DefaultClient.Do(asked)
if err != nil {
t.Fatal(err)
}
defer answer.Body.Close()
if answer.StatusCode != http.StatusOK {
t.Fatalf("an insecure https backend was not reached: %d", answer.StatusCode)
}
}
// End to end through the proxy itself: a request for the name reaches the workload, and a name
// nobody asked for is refused in a way that says what IS served.
func TestTheProxyReachesTheWorkloadAndNamesWhatItServes(t *testing.T) {
@@ -75,7 +205,7 @@ func TestTheProxyReachesTheWorkloadAndNamesWhatItServes(t *testing.T) {
host, port, _ := strings.Cut(target, ":")
held := newTable()
held.set(map[string]string{"app.example": "http://" + host + ":" + port})
held.set(plain(map[string]string{"app.example": "http://" + host + ":" + port}), allPublic(plain(map[string]string{"app.example": "http://" + host + ":" + port})))
proxy := httptest.NewServer(handler(held))
defer proxy.Close()
@@ -120,16 +250,17 @@ func TestTheProxyReachesTheWorkloadAndNamesWhatItServes(t *testing.T) {
// nothing fails more visibly than a stale grant, which is exactly why it must not survive.
func TestWithdrawingARouteStopsServingIt(t *testing.T) {
held := newTable()
held.set(map[string]string{
initial := plain(map[string]string{
"going.example": "http://a.internal:80",
"staying.example": "http://b.internal:80",
})
held.set(map[string]string{"staying.example": "http://b.internal:80"})
held.set(initial, allPublic(initial))
held.set(plain(map[string]string{"staying.example": "http://b.internal:80"}), allPublic(plain(map[string]string{"staying.example": "http://b.internal:80"})))
if _, still := held.find("going.example"); still {
if _, still := held.find("going.example", "/"); still {
t.Fatal("a route whose module was unassigned is still served")
}
if _, kept := held.find("staying.example"); !kept {
if _, kept := held.find("staying.example", "/"); !kept {
t.Fatal("withdrawing one route took another with it")
}
}
@@ -137,8 +268,8 @@ func TestWithdrawingARouteStopsServingIt(t *testing.T) {
// A Host header carries a port and the name does not.
func TestARequestNamingAPortStillFindsItsRoute(t *testing.T) {
held := newTable()
held.set(map[string]string{"app.example": "http://a.internal:8080"})
if _, found := held.find("app.example:8080"); !found {
held.set(plain(map[string]string{"app.example": "http://a.internal:8080"}), allPublic(plain(map[string]string{"app.example": "http://a.internal:8080"})))
if _, found := held.find("app.example:8080", "/"); !found {
t.Fatal("a request to app.example:8080 did not find the route for app.example")
}
}
@@ -168,7 +299,7 @@ func TestTheIssuerIsStagingUnlessNamed(t *testing.T) {
// rate limit — and the proxy would look healthy throughout.
func TestNoCertificateIsAskedForOnAnUnroutedName(t *testing.T) {
held := newTable()
held.set(map[string]string{"photos.example": "http://127.0.0.1:8080"})
held.set(plain(map[string]string{"photos.example": "http://127.0.0.1:8080"}), allPublic(plain(map[string]string{"photos.example": "http://127.0.0.1:8080"})))
policy := onlyWhatTheMeshSaid(held)
if err := policy(context.Background(), "photos.example"); err != nil {
@@ -181,18 +312,179 @@ func TestNoCertificateIsAskedForOnAnUnroutedName(t *testing.T) {
}
}
// A certificate is asked for on a route's public name, never on its internal-network alias — no
// public CA can validate a private name, and asking anyway would only spend the account's rate
// limit on an order that can never succeed.
func TestNoCertificateIsAskedForOnAnInternalAlias(t *testing.T) {
routes, public, err := routesFrom(write(t, `{"given":[
{"from":"app","node":"anchor","at":"anchor.internal",
"values":{"name":"app.example","internal-name":"app.anchor.internal","port":8080}}
]}`))
if err != nil {
t.Fatal(err)
}
held := newTable()
held.set(routes, public)
policy := onlyWhatTheMeshSaid(held)
if err := policy(context.Background(), "app.example"); err != nil {
t.Errorf("the route's public name was refused a certificate: %v", err)
}
if err := policy(context.Background(), "app.anchor.internal"); err == nil {
t.Error("a certificate was ordered for the internal alias, which no public CA can validate")
}
}
// A certificate is asked of the *internal* authority only for a name that is routed here and is
// not a route's own public name — the internal-network alias, never the route it accompanies.
func TestTheInternalAuthorityOnlyCertifiesInternalOnlyAliases(t *testing.T) {
routes, public, err := routesFrom(write(t, `{"given":[
{"from":"app","node":"anchor","at":"anchor.internal",
"values":{"name":"app.example","internal-name":"app.anchor.internal","port":8080}}
]}`))
if err != nil {
t.Fatal(err)
}
held := newTable()
held.set(routes, public)
policy := onlyInternalNamesTheMeshSaid(held)
if err := policy(context.Background(), "app.anchor.internal"); err != nil {
t.Errorf("the internal alias was refused by its own authority: %v", err)
}
if err := policy(context.Background(), "app.example"); err == nil {
t.Error("the internal authority certified a route's public name, which the public authority already covers")
}
if err := policy(context.Background(), "unrouted.internal"); err == nil {
t.Error("the internal authority certified a name nobody routed here")
}
}
// A route withdrawn stops being certifiable, without the proxy restarting.
func TestWithdrawingARouteWithdrawsItsCertificate(t *testing.T) {
held := newTable()
held.set(map[string]string{"photos.example": "http://127.0.0.1:8080"})
held.set(plain(map[string]string{"photos.example": "http://127.0.0.1:8080"}), allPublic(plain(map[string]string{"photos.example": "http://127.0.0.1:8080"})))
policy := onlyWhatTheMeshSaid(held)
if err := policy(context.Background(), "photos.example"); err != nil {
t.Fatal(err)
}
held.set(nil)
held.set(nil, nil)
if err := policy(context.Background(), "photos.example"); err == nil {
t.Fatal("a withdrawn route can still order certificates, so the policy read a copy taken " +
"once rather than what is served now")
}
}
// A route may say the largest body it carries, and the proxy holds requests to it.
//
// The registry is why: image layers arrive as single requests of gigabytes, and a proxy's own
// default refuses them long before the workload is reached. It is configuration beside `insecure`,
// not a fifth policy — novox/hq ADR 0108 closed that set at four.
func TestARouteMayLimitTheBodyItCarries(t *testing.T) {
routes, _, err := routesFrom(write(t, `{"given":[
{"from":"registry","node":"anchor","at":"anchor.internal",
"values":{"name":"images.example","port":5000,"max-request-body":21474836480}}
]}`))
if err != nil {
t.Fatal(err)
}
rules := routes["images.example"]
if len(rules) != 1 {
t.Fatalf("the route is not served once: %v", routes)
}
if rules[0].maxRequestBody != 21474836480 {
t.Fatalf("the limit did not survive the contribution: %d", rules[0].maxRequestBody)
}
}
// Saying nothing leaves the route unlimited, which is what every route already got.
func TestARouteThatSaysNothingCarriesAnySize(t *testing.T) {
routes, _, err := routesFrom(write(t, `{"given":[
{"from":"app","node":"anchor","at":"anchor.internal","values":{"name":"app.example","port":8080}}
]}`))
if err != nil {
t.Fatal(err)
}
if got := routes["app.example"][0].maxRequestBody; got != 0 {
t.Fatalf("a route that asked for no limit got one: %d", got)
}
}
// A limit that is not a whole positive number of bytes takes the route with it. Serving it without
// the limit would carry exactly what the module said not to carry, and report success doing it.
func TestARouteWithAnUnusableLimitIsSkipped(t *testing.T) {
for _, asked := range []string{`"lots"`, `-1`, `0`, `1.5`} {
routes, _, err := routesFrom(write(t, `{"given":[
{"from":"registry","node":"anchor","at":"anchor.internal",
"values":{"name":"images.example","port":5000,"max-request-body":`+asked+`}}
]}`))
if err != nil {
t.Fatal(err)
}
if len(routes["images.example"]) != 0 {
t.Errorf("a route asking for a max-request-body of %s was served anyway: %v", asked, routes)
}
}
}
// A request larger than the route carries is refused by the proxy, with the limit named, and the
// workload never sees it. A request within it is proxied normally.
func TestABodyOverTheLimitIsRefusedAndOneUnderItIsCarried(t *testing.T) {
var reached int
workload := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
body, _ := io.ReadAll(r.Body)
reached++
fmt.Fprintf(w, "carried %d bytes", len(body))
}))
defer workload.Close()
at := strings.TrimPrefix(workload.URL, "http://")
host, port, _ := strings.Cut(at, ":")
routes, _, err := routesFrom(write(t, `{"given":[
{"from":"registry","node":"anchor","at":"`+host+`",
"values":{"name":"images.example","port":`+port+`,"max-request-body":8}}
]}`))
if err != nil {
t.Fatal(err)
}
held := newTable()
held.set(routes, map[string]bool{})
proxy := httptest.NewServer(handler(held))
defer proxy.Close()
over, err := post(proxy.URL, "images.example", "123456789")
if err != nil {
t.Fatal(err)
}
defer over.Body.Close()
if over.StatusCode != http.StatusRequestEntityTooLarge {
t.Fatalf("a body over the limit answered %d, not 413", over.StatusCode)
}
if reached != 0 {
t.Fatalf("the workload was reached by a request the route said it would not carry")
}
under, err := post(proxy.URL, "images.example", "1234")
if err != nil {
t.Fatal(err)
}
defer under.Body.Close()
if under.StatusCode != http.StatusOK {
t.Fatalf("a body within the limit answered %d, not 200", under.StatusCode)
}
if reached != 1 {
t.Fatalf("the workload was not reached by a request within the limit")
}
}
// post sends a body to the proxy as the named route, since a route is found by the Host header.
func post(url, host, body string) (*http.Response, error) {
asked, err := http.NewRequest(http.MethodPost, url, strings.NewReader(body))
if err != nil {
return nil, err
}
asked.Host = host
asked.Header.Set("Content-Type", "application/octet-stream")
return http.DefaultClient.Do(asked)
}
+87 -7
View File
@@ -63,6 +63,19 @@ type Result struct {
Built []catalogue.Built
}
// GitCredential is the forge credential a clone may present when the server asks for one.
//
// **Offered, never pushed.** It is written as a git credential-store file and named to git with
// `-c credential.helper=store`, so git itself decides when it applies: only on an authentication
// challenge, and only for the URL it was written for — scheme, host and port included. A public
// repository clones exactly as before, and a repository on any other host is never shown it.
type GitCredential struct {
// URL is the credential-store line — scheme://user:password@host[:port] — naming the one
// server this credential belongs to. Empty means the builder holds none and every clone is
// anonymous, as it always was.
URL string
}
// Build clones a repository at a ref, reads its manifest, produces what it declares, publishes
// each, and returns the manifest the mesh should hold.
//
@@ -70,7 +83,8 @@ type Result struct {
// archive failed would otherwise leave half of itself in the store under a digest the mesh never
// records — reachable, unreferenced, and indistinguishable from something in use.
func Build(ctx context.Context, run Runner, publish Publisher,
repository, path, ref, workspace string, held map[string]string, npmrc Npmrc, log Log) (Result, error) {
repository, path, ref, workspace string, held map[string]string, npmrc Npmrc,
forge GitCredential, log Log) (Result, error) {
say := logging(log)
say("clone", "%s%s at %s", repository, describePath(path), refOrHead(ref))
@@ -80,6 +94,15 @@ func Build(ctx context.Context, run Runner, publish Publisher,
if err := os.MkdirAll(workspace, 0o755); err != nil {
return Result{}, err
}
// The credential is a file git reads, never an argument: a URL carrying a password in argv
// would be readable by anything that can list processes for as long as a clone runs.
credentials := ""
if forge.URL != "" {
credentials = filepath.Join(workspace, "git-credentials")
if err := os.WriteFile(credentials, []byte(forge.URL+"\n"), 0o600); err != nil {
return Result{}, err
}
}
tree := filepath.Join(workspace, "source")
if err := os.RemoveAll(tree); err != nil {
return Result{}, err
@@ -87,7 +110,7 @@ func Build(ctx context.Context, run Runner, publish Publisher,
// A fresh clone every time rather than a fetch into a tree that is already there. A build
// that reuses a working tree can succeed because of something a previous build left behind,
// and that is a build nobody can reproduce.
if _, err := run(ctx, workspace, "git", "clone", "--quiet", repository, tree); err != nil {
if _, err := run(ctx, workspace, "git", cloneWith(credentials, "clone", "--quiet", repository, tree)...); err != nil {
say("clone", "FAILED: %v", err)
return Result{}, fmt.Errorf("cannot clone %s: %w", repository, err)
}
@@ -177,7 +200,7 @@ func Build(ctx context.Context, run Runner, publish Publisher,
sort.Slice(artifacts, func(i, j int) bool { return artifacts[i].Name < artifacts[j].Name })
for _, a := range artifacts {
say("artifact", "%s (%s%s) — starting", a.Name, a.Kind, langSuffix(a))
made, err := one(ctx, run, publish, manifest.Module, within, commit, a, args, held, npmrcPath, say)
made, err := one(ctx, run, publish, manifest.Module, within, workspace, commit, credentials, a, args, held, npmrcPath, say)
if err != nil {
say("artifact", "%s FAILED: %v", a.Name, err)
return Result{}, err
@@ -210,6 +233,43 @@ func logging(log Log) func(step, format string, args ...any) {
}
}
// contextFrom clones an image artifact's own build context, when it names one apart from this
// module's own repository — a fresh tree, the same way the module's own is, keyed by artifact
// name so two artifacts of one module naming different contexts do not collide.
func contextFrom(ctx context.Context, run Runner, workspace, artifact, credentials string,
from catalogue.ArtifactContext, say func(step, format string, args ...any)) (string, error) {
say("context", "cloning %s at %s for %s", from.Repository, refOrHead(from.Ref), artifact)
dir := filepath.Join(workspace, "context-"+artifact)
if err := os.RemoveAll(dir); err != nil {
return "", err
}
if _, err := run(ctx, workspace, "git", cloneWith(credentials, "clone", "--quiet", from.Repository, dir)...); err != nil {
return "", fmt.Errorf("cannot clone %s: %w", from.Repository, err)
}
if from.Ref != "" {
if _, err := run(ctx, dir, "git", "checkout", "--quiet", from.Ref); err != nil {
return "", fmt.Errorf("%s has no %s: %w", from.Repository, from.Ref, err)
}
}
say("context", "done")
return dir, nil
}
// cloneWith is a git invocation that may offer a stored credential.
//
// The first `-c credential.helper=` clears every helper the environment might carry, so exactly
// one place answers an authentication challenge: the file the builder wrote. Without a file, the
// invocation is exactly what it always was.
func cloneWith(credentials string, rest ...string) []string {
if credentials == "" {
return rest
}
return append([]string{
"-c", "credential.helper=",
"-c", "credential.helper=store --file=" + credentials,
}, rest...)
}
func describePath(path string) string {
if path == "" {
return ""
@@ -333,7 +393,7 @@ func wantsPackages(manifest catalogue.Manifest, within string) bool {
}
func one(ctx context.Context, run Runner, publish Publisher,
module, tree, commit string, a catalogue.Artifact, args []string,
module, tree, workspace, commit, credentials string, a catalogue.Artifact, args []string,
held map[string]string, npmrc string, say func(step, format string, args ...any)) (catalogue.Built, error) {
switch a.Kind {
@@ -405,7 +465,27 @@ func one(ctx context.Context, run Runner, publish Publisher,
"and start FROM ${<NAME>} (novox/hq ADR 0097)",
module, a.From, strings.Join(bases, ", "))
}
invocation := append([]string{"build", "-f", a.From, "-t", local}, args...)
// The recipe is always read from this module's own tree, at this module's own commit — only
// the context docker build's final argument names can come from somewhere else, when the
// artifact says so.
recipePath := a.From
buildDir := tree
if a.Context != nil {
cloned, err := contextFrom(ctx, run, workspace, a.Name, credentials, *a.Context, say)
if err != nil {
return catalogue.Built{}, fmt.Errorf("%s: %s's context: %w", module, a.Name, err)
}
// docker build accepts -f outside the context it is given; the recipe stays exactly
// where it was read from and validated against, absolute so the working directory
// switching to the cloned context does not change which file that is.
absRecipe, err := filepath.Abs(filepath.Join(tree, a.From))
if err != nil {
return catalogue.Built{}, fmt.Errorf("%s: %s's recipe: %w", module, a.Name, err)
}
recipePath = absRecipe
buildDir = cloned
}
invocation := append([]string{"build", "-f", recipePath, "-t", local}, args...)
if a.Target != "" {
invocation = append(invocation, "--target", a.Target)
}
@@ -417,8 +497,8 @@ func one(ctx context.Context, run Runner, publish Publisher,
invocation = append(invocation, "--network", "host")
}
invocation = append(invocation, ".")
say("image", "docker build -f %s", a.From)
if _, err := run(ctx, tree, "docker", invocation...); err != nil {
say("image", "docker build -f %s", recipePath)
if _, err := run(ctx, buildDir, "docker", invocation...); err != nil {
return catalogue.Built{}, fmt.Errorf("%s: building %s failed: %w", module, a.Name, err)
}
say("image", "built, publishing")
+199 -14
View File
@@ -18,13 +18,19 @@ import (
// tree, that two builds of one commit produce one digest. Running docker here would test docker.
type recorded struct {
ran []string
ran []string
// dirs is the directory each entry in ran was run from, same index — so a test can ask not
// only what ran but where.
dirs []string
images map[string]string
archives map[string]string
failPush bool
// contents is what a clone of this repository lands, so the fake clone can restore the tree
// Build deliberately removes first.
contents map[string]string
// secondary is what a clone of a repository OTHER than the one under test lands, keyed by
// that repository's URL — an artifact's own build context, cloned apart from the module.
secondary map[string]map[string]string
// stamped is the modification time the clone gives every file. Set differently between two
// builds of one commit, because otherwise both land in the same second and a packer that
// carried timestamps would still produce one digest — which is a test that passes for a
@@ -35,13 +41,28 @@ type recorded struct {
func (r *recorded) run(_ context.Context, dir, name string, args ...string) (string, error) {
line := name + " " + strings.Join(args, " ")
r.ran = append(r.ran, line)
r.dirs = append(r.dirs, dir)
// A clone may carry `-c` configuration in front of the verb — the credential store — so the
// verb is found rather than assumed first.
isClone := false
for _, a := range args {
if a == "clone" {
isClone = true
break
}
}
switch {
case name == "git" && len(args) > 0 && args[0] == "clone":
case name == "git" && isClone:
repository := args[len(args)-2]
tree := args[len(args)-1]
if err := os.MkdirAll(tree, 0o755); err != nil {
return "", err
}
for path, body := range r.contents {
lands := r.contents
if by, is := r.secondary[repository]; is {
lands = by
}
for path, body := range lands {
full := filepath.Join(tree, path)
if err := os.MkdirAll(filepath.Dir(full), 0o755); err != nil {
return "", err
@@ -59,7 +80,6 @@ func (r *recorded) run(_ context.Context, dir, name string, args ...string) (str
case name == "git" && len(args) > 0 && args[0] == "rev-parse":
return "c0ffeec0ffeec0ffeec0ffeec0ffeec0ffeec0ff\n", nil
}
_ = dir
return "", nil
}
@@ -108,7 +128,7 @@ func TestABuildProducesAManifestThePinsAreIn(t *testing.T) {
r, workspace := aRepository(t, withBoth, map[string]string{
"Dockerfile": "FROM scratch", "files/theme.conf": "dark",
})
got, err := Build(context.Background(), r.run, r, "https://forge.invalid/meshboard.git", "", "", workspace, nil, Npmrc{}, nil)
got, err := Build(context.Background(), r.run, r, "https://forge.invalid/meshboard.git", "", "", workspace, nil, Npmrc{}, GitCredential{}, nil)
if err != nil {
t.Fatal(err)
}
@@ -134,7 +154,7 @@ func TestTwoBuildsOfOneCommitProduceOneDigest(t *testing.T) {
})
// A year apart, so a packer carrying timestamps cannot accidentally agree.
r.stamped = time.Date(2020+i, time.March, 3, 4, 5, 6, 0, time.UTC)
got, err := Build(context.Background(), r.run, r, "https://forge.invalid/x.git", "", "", workspace, nil, Npmrc{}, nil)
got, err := Build(context.Background(), r.run, r, "https://forge.invalid/x.git", "", "", workspace, nil, Npmrc{}, GitCredential{}, nil)
if err != nil {
t.Fatal(err)
}
@@ -154,7 +174,7 @@ func TestNothingIsPublishedUntilEverythingIsBuilt(t *testing.T) {
// unreferenced, and indistinguishable from something in use.
r, workspace := aRepository(t, withBoth, map[string]string{"Dockerfile": "FROM scratch"})
// `files` is missing, so packing the archive fails — after the image would have been pushed.
_, err := Build(context.Background(), r.run, r, "https://forge.invalid/x.git", "", "", workspace, nil, Npmrc{}, nil)
_, err := Build(context.Background(), r.run, r, "https://forge.invalid/x.git", "", "", workspace, nil, Npmrc{}, GitCredential{}, nil)
if err == nil {
t.Fatal("a build with a missing input succeeded")
}
@@ -166,7 +186,7 @@ func TestNothingIsPublishedUntilEverythingIsBuilt(t *testing.T) {
func TestARepositoryWithNoManifestSaysSo(t *testing.T) {
workspace := t.TempDir()
r := &recorded{contents: map[string]string{"README.md": "nothing to see"}}
_, err := Build(context.Background(), r.run, r, "https://forge.invalid/x.git", "", "", workspace, nil, Npmrc{}, nil)
_, err := Build(context.Background(), r.run, r, "https://forge.invalid/x.git", "", "", workspace, nil, Npmrc{}, GitCredential{}, nil)
if err == nil {
t.Fatal("a repository with nothing saying what it is was built")
}
@@ -179,7 +199,7 @@ func TestAModuleThatBuildsNothingStillProducesAManifest(t *testing.T) {
// Most of what a person installs is configuration.
r, workspace := aRepository(t, `{"module":"shell","version":"1","resources":[
{"id":"rc","type":"file","path":"/etc/zsh/zshrc","content":"setopt"}]}`, nil)
got, err := Build(context.Background(), r.run, r, "https://forge.invalid/shell.git", "", "", workspace, nil, Npmrc{}, nil)
got, err := Build(context.Background(), r.run, r, "https://forge.invalid/shell.git", "", "", workspace, nil, Npmrc{}, GitCredential{}, nil)
if err != nil {
t.Fatal(err)
}
@@ -209,7 +229,7 @@ func TestTheTreeIsFreshEveryTime(t *testing.T) {
if err := os.WriteFile(leftover, []byte("stale"), 0o644); err != nil {
t.Fatal(err)
}
if _, err := Build(context.Background(), r.run, r, "https://forge.invalid/x.git", "", "", workspace, nil, Npmrc{}, nil); err != nil {
if _, err := Build(context.Background(), r.run, r, "https://forge.invalid/x.git", "", "", workspace, nil, Npmrc{}, GitCredential{}, nil); err != nil {
t.Fatal(err)
}
if _, err := os.Stat(leftover); err == nil {
@@ -222,7 +242,7 @@ func TestABuildThatCannotPushFails(t *testing.T) {
"Dockerfile": "FROM scratch", "files/a": "b",
})
r.failPush = true
if _, err := Build(context.Background(), r.run, r, "https://forge.invalid/x.git", "", "", workspace, nil, Npmrc{}, nil); err == nil {
if _, err := Build(context.Background(), r.run, r, "https://forge.invalid/x.git", "", "", workspace, nil, Npmrc{}, GitCredential{}, nil); err == nil {
t.Fatal("a build that could publish nothing reported success")
}
}
@@ -236,7 +256,7 @@ func TestAnUpstreamImageIsMirroredRatherThanBuilt(t *testing.T) {
"resources":[{"id":"db","type":"container","name":"mesh-postgres","artifact":"store"}]}`
r, workspace := aRepository(t, mirrors, nil)
got, err := Build(context.Background(), r.run, r, "https://forge.invalid/postgres.git", "", "", workspace, nil, Npmrc{}, nil)
got, err := Build(context.Background(), r.run, r, "https://forge.invalid/postgres.git", "", "", workspace, nil, Npmrc{}, GitCredential{}, nil)
if err != nil {
t.Fatal(err)
}
@@ -302,7 +322,7 @@ func TestAModuleIsBuiltFromItsPathWithinTheRepository(t *testing.T) {
"modules/other/" + ManifestName: `{"module":"other","version":"1"}`,
}}
got, err := Build(context.Background(), r.run, r,
"https://forge.invalid/catalogue.git", "modules/shell", "", t.TempDir(), nil, Npmrc{}, nil)
"https://forge.invalid/catalogue.git", "modules/shell", "", t.TempDir(), nil, Npmrc{}, GitCredential{}, nil)
if err != nil {
t.Fatal(err)
}
@@ -321,7 +341,7 @@ func TestAPathThatLeavesTheRepositoryIsRefused(t *testing.T) {
for _, escaping := range []string{"../../etc", "/etc"} {
r := &recorded{contents: map[string]string{ManifestName: withBoth}}
_, err := Build(context.Background(), r.run, r,
"https://forge.invalid/x.git", escaping, "", t.TempDir(), nil, Npmrc{}, nil)
"https://forge.invalid/x.git", escaping, "", t.TempDir(), nil, Npmrc{}, GitCredential{}, nil)
if err == nil {
t.Fatalf("%q was accepted as a module's path", escaping)
}
@@ -331,3 +351,168 @@ func TestAPathThatLeavesTheRepositoryIsRefused(t *testing.T) {
}
}
}
// **Packaging and source are allowed to live apart** — a module that ships only the recipe for
// source that lives in a second repository (the reference route-proxy, packaged in the catalogue
// but built from mesh-controller's own repository) names where that source actually is, rather
// than vendoring a second copy the two could drift from.
func TestAnArtifactWithItsOwnContextIsBuiltFromThere(t *testing.T) {
const withContext = `{"module":"route-proxy","version":"1",
"build":{"artifacts":[
{"name":"server","kind":"image","from":"Dockerfile",
"context":{"repository":"https://forge.invalid/source.git","ref":"main"}}]}}`
r := &recorded{
contents: map[string]string{
ManifestName: withContext,
// The recipe lives with the packaging, not the source — read from here regardless of
// where the build context comes from. FROM scratch declares no base, so what is under
// test — where the context comes from — is not entangled with ADR 0097's own checks.
"Dockerfile": "FROM scratch\nCOPY go.mod ./\n",
},
secondary: map[string]map[string]string{
// go.mod exists only in the second repository. A build context taken from the wrong
// place would never find it, which a real docker build would refuse on — the fake
// does not read files, so what is checked below is that the build was even pointed
// at the right place, not that COPY would have succeeded.
"https://forge.invalid/source.git": {"go.mod": "module route-proxy\n"},
},
}
_, err := Build(context.Background(), r.run, r,
"https://forge.invalid/catalogue.git", "", "", t.TempDir(), nil, Npmrc{}, GitCredential{}, nil)
if err != nil {
t.Fatal(err)
}
var clonedSource bool
for _, line := range r.ran {
if strings.HasPrefix(line, "git clone") && strings.Contains(line, "https://forge.invalid/source.git") {
clonedSource = true
}
}
if !clonedSource {
t.Fatalf("the artifact's own context was never cloned: %v", r.ran)
}
buildIndex := -1
for i, line := range r.ran {
if strings.HasPrefix(line, "docker build ") {
buildIndex = i
}
}
if buildIndex == -1 {
t.Fatal("no docker build was run")
}
build := r.ran[buildIndex]
buildDir := r.dirs[buildIndex]
if !strings.Contains(buildDir, "context-server") {
t.Errorf("docker build ran from %q, not the artifact's own cloned context", buildDir)
}
recipe := strings.SplitN(strings.SplitN(build, "-f ", 2)[1], " ", 2)[0]
if !filepath.IsAbs(recipe) {
t.Errorf("the recipe %q is not an absolute path, so it is read relative to whatever "+
"directory the build context moved to rather than where it actually is", recipe)
}
if !strings.HasSuffix(recipe, string(filepath.Separator)+"Dockerfile") {
t.Errorf("the recipe is not the module's own Dockerfile: %q", recipe)
}
if !strings.HasSuffix(build, " .") {
t.Errorf("the build was not given a context: %s", build)
}
}
// The forge credential is offered through git's own credential store — a file, never argv — and
// git decides when it applies. What is checked: the clone names the store, the secret never
// appears in a command line, and the file holds exactly the URL at 0600.
func TestABuildOffersTheForgesCredentialThroughGitsOwnStore(t *testing.T) {
r, workspace := aRepository(t, withBoth, map[string]string{
"Dockerfile": "FROM scratch", "files/theme.conf": "dark",
})
_, err := Build(context.Background(), r.run, r, "https://forge.invalid/meshboard.git", "", "",
workspace, nil, Npmrc{},
GitCredential{URL: "http://mesh_novox_builder:sw0rdfi5h@forge.invalid:20000"}, nil)
if err != nil {
t.Fatal(err)
}
stored := filepath.Join(workspace, "git-credentials")
clone := r.ran[0]
if !strings.Contains(clone, "credential.helper=store --file="+stored) {
t.Fatalf("the clone does not name the credential store: %s", clone)
}
for _, line := range r.ran {
if strings.Contains(line, "sw0rdfi5h") {
t.Fatalf("the secret is in a command line, readable by anything that can list processes: %s", line)
}
}
raw, err := os.ReadFile(stored)
if err != nil {
t.Fatal(err)
}
if strings.TrimSpace(string(raw)) != "http://mesh_novox_builder:sw0rdfi5h@forge.invalid:20000" {
t.Fatalf("the store does not hold the credential as given: %q", raw)
}
info, err := os.Stat(stored)
if err != nil {
t.Fatal(err)
}
if info.Mode().Perm() != 0o600 {
t.Fatalf("the credential file is readable beyond its owner: %v", info.Mode())
}
}
// Without a credential, a clone is exactly the invocation it always was, and no credential file
// appears — the builder a mesh of public repositories runs is unchanged.
func TestABuildWithNoCredentialClonesExactlyAsBefore(t *testing.T) {
r, workspace := aRepository(t, withBoth, map[string]string{
"Dockerfile": "FROM scratch", "files/theme.conf": "dark",
})
_, err := Build(context.Background(), r.run, r, "https://forge.invalid/meshboard.git", "", "",
workspace, nil, Npmrc{}, GitCredential{}, nil)
if err != nil {
t.Fatal(err)
}
if !strings.HasPrefix(r.ran[0], "git clone --quiet ") {
t.Fatalf("a credential-less clone grew flags: %s", r.ran[0])
}
if _, err := os.Stat(filepath.Join(workspace, "git-credentials")); !os.IsNotExist(err) {
t.Fatal("a credential file was written with no credential to put in it")
}
}
// An artifact's own context is cloned with the same offer: a private module whose context is a
// second private repository on the same forge builds, and the secret still never reaches argv.
func TestAContextCloneCarriesTheSameCredentialStore(t *testing.T) {
const withContext = `{"module":"route-proxy","version":"1",
"build":{"artifacts":[
{"name":"server","kind":"image","from":"Dockerfile",
"context":{"repository":"https://forge.invalid/source.git","ref":"main"}}]}}`
r := &recorded{
contents: map[string]string{
ManifestName: withContext,
"Dockerfile": "FROM scratch\nCOPY go.mod ./\n",
},
secondary: map[string]map[string]string{
"https://forge.invalid/source.git": {"go.mod": "module route-proxy\n"},
},
}
workspace := t.TempDir()
_, err := Build(context.Background(), r.run, r,
"https://forge.invalid/catalogue.git", "", "", workspace, nil, Npmrc{},
GitCredential{URL: "https://builder:s3cret@forge.invalid"}, nil)
if err != nil {
t.Fatal(err)
}
stored := filepath.Join(workspace, "git-credentials")
var contextClone string
for _, line := range r.ran {
if strings.Contains(line, "clone") && strings.Contains(line, "source.git") {
contextClone = line
}
}
if contextClone == "" {
t.Fatalf("the context was never cloned: %v", r.ran)
}
if !strings.Contains(contextClone, "credential.helper=store --file="+stored) {
t.Fatalf("the context clone does not name the credential store: %s", contextClone)
}
}
+4 -4
View File
@@ -54,7 +54,7 @@ func TestABundleIsCompiledAndPackedWithNoDockerfile(t *testing.T) {
held := map[string]string{"mesh-tools/build": "registry.invalid/mesh-tools/build@sha256:" + strings.Repeat("b", 64)}
got, err := Build(context.Background(), compiling{r}.run, r,
"https://forge.invalid/greeter.git", "", "", workspace, held, Npmrc{}, nil)
"https://forge.invalid/greeter.git", "", "", workspace, held, Npmrc{}, GitCredential{}, nil)
if err != nil {
t.Fatalf("a module with a language and no Dockerfile did not build: %v", err)
}
@@ -91,7 +91,7 @@ func TestABundleWhoseToolchainIsNotHeldIsRefusedFirst(t *testing.T) {
r, workspace := aRepository(t, aBundle, map[string]string{"index.ts": "console.log(1)"})
_, err := Build(context.Background(), compiling{r}.run, r,
"https://forge.invalid/greeter.git", "", "", workspace, nil, Npmrc{}, nil)
"https://forge.invalid/greeter.git", "", "", workspace, nil, Npmrc{}, GitCredential{}, nil)
if err == nil {
t.Fatal("a bundle was built with no toolchain to compile it in")
}
@@ -112,7 +112,7 @@ func TestABundleInAnUnknownLanguageIsRefused(t *testing.T) {
_, err := Build(context.Background(), compiling{r}.run, r,
"https://forge.invalid/greeter.git", "", "", workspace,
map[string]string{"mesh-tools/build": "registry.invalid/x@sha256:" + strings.Repeat("c", 64)}, Npmrc{}, nil)
map[string]string{"mesh-tools/build": "registry.invalid/x@sha256:" + strings.Repeat("c", 64)}, Npmrc{}, GitCredential{}, nil)
if err == nil {
t.Fatal("a language nothing can compile was accepted")
}
@@ -140,7 +140,7 @@ func TestTwoBundlesInOneModuleArePackedSeparately(t *testing.T) {
held := map[string]string{"mesh-tools/build": "registry.invalid/mesh-tools/build@sha256:" + strings.Repeat("b", 64)}
got, err := Build(context.Background(), compiling{r}.run, r,
"https://forge.invalid/greeter.git", "", "", workspace, held, Npmrc{}, nil)
"https://forge.invalid/greeter.git", "", "", workspace, held, Npmrc{}, GitCredential{}, nil)
if err != nil {
t.Fatalf("a module with two bundles did not build: %v", err)
}
+5 -5
View File
@@ -71,7 +71,7 @@ func TestAnImageBuildGetsTheCredentialInTheContextAndHostNetwork(t *testing.T) {
r, workspace := aRepository(t, withBoth, map[string]string{"Dockerfile": "FROM scratch\nCOPY .npmrc ./", "files/x": "y"})
n := Npmrc{Scope: "@novox", Registry: "https://forge.invalid/api/packages/novox/npm/", Token: "t"}
if _, err := Build(context.Background(), r.run, r,
"https://forge.invalid/meshboard.git", "", "", workspace, nil, n, nil); err != nil {
"https://forge.invalid/meshboard.git", "", "", workspace, nil, n, GitCredential{}, nil); err != nil {
t.Fatalf("the build failed: %v", err)
}
@@ -101,7 +101,7 @@ func TestAnImageBuildGetsTheCredentialInTheContextAndHostNetwork(t *testing.T) {
func TestAnImageBuildWithoutACredentialGetsNoHostNetwork(t *testing.T) {
r, workspace := aRepository(t, withBoth, map[string]string{"Dockerfile": "FROM scratch", "files/x": "y"})
if _, err := Build(context.Background(), r.run, r,
"https://forge.invalid/meshboard.git", "", "", workspace, nil, Npmrc{}, nil); err != nil {
"https://forge.invalid/meshboard.git", "", "", workspace, nil, Npmrc{}, GitCredential{}, nil); err != nil {
t.Fatalf("the build failed: %v", err)
}
for _, line := range r.ran {
@@ -127,7 +127,7 @@ func TestAPackageIsBuiltOnAPublicBaseAndPublishedByVersion(t *testing.T) {
})
n := Npmrc{Scope: "@novox", Registry: "https://forge.invalid/api/packages/novox/npm/", Token: "t"}
got, err := Build(context.Background(), r.run, r,
"https://forge.invalid/mesh-sdk.git", "", "", workspace, nil, n, nil)
"https://forge.invalid/mesh-sdk.git", "", "", workspace, nil, n, GitCredential{}, nil)
if err != nil {
t.Fatalf("the package did not build: %v", err)
}
@@ -159,7 +159,7 @@ func TestAPackageWithNoRegistryIsRefused(t *testing.T) {
"package.json": `{"name":"@novox/mesh-sdk","version":"0.1.0"}`,
})
_, err := Build(context.Background(), r.run, r,
"https://forge.invalid/mesh-sdk.git", "", "", workspace, nil, Npmrc{}, nil)
"https://forge.invalid/mesh-sdk.git", "", "", workspace, nil, Npmrc{}, GitCredential{}, nil)
if err == nil {
t.Fatal("a package built with no registry to publish to, silently")
}
@@ -206,7 +206,7 @@ func TestAnImageThatDoesNotAskForTheCredentialDoesNotGetIt(t *testing.T) {
r, workspace := aRepository(t, withBoth, map[string]string{"Dockerfile": "FROM scratch\nCOPY . .", "files/x": "y"})
n := Npmrc{Scope: "@novox", Registry: "https://forge.invalid/api/packages/novox/npm/", Token: "t"}
if _, err := Build(context.Background(), r.run, r,
"https://forge.invalid/meshboard.git", "", "", workspace, nil, n, nil); err != nil {
"https://forge.invalid/meshboard.git", "", "", workspace, nil, n, GitCredential{}, nil); err != nil {
t.Fatalf("the build failed: %v", err)
}
for _, line := range r.ran {
@@ -0,0 +1,38 @@
package catalogue
import (
"strings"
"testing"
)
// The artifact store's seat is one per mesh, read from the catalogue beside this checkout.
//
// **A second store anywhere is refused by name, not discovered as a consumer failure.** The seat
// was node-scoped, so a second `distribution` on another machine resolved cleanly there — and a
// node-scoped requirement with one candidate installs that candidate on the node, so anything that
// required the store's presence beside it would have raised a fresh, empty store on the wrong
// machine. Only afterwards did the mesh notice: `artifact-store` offered by two nodes, and every
// consumer elsewhere refusing to choose. The claim says it first, where the second store is
// assigned.
func TestASecondArtifactStoreAnywhereIsRefusedByName(t *testing.T) {
store := catalogueManifest(t, "distribution")
// The first store resolves as it always has.
if _, err := Resolve(shelf(store), []string{"distribution"}, workstation(), World{}); err != nil {
t.Fatalf("the store alone does not resolve: %v", err)
}
// A second one, on any other machine, is refused — and the refusal names the seat.
elsewhere := World{Held: []Held{{Claim: "the-artifact-store", Scope: ScopeMesh,
Node: "anchor", Module: "distribution"}}}
other := workstation()
other.Name = "laptop"
_, err := Resolve(shelf(store), []string{"distribution"}, other, elsewhere)
if err == nil {
t.Fatal("a second store was accepted on another machine; it would offer artifact-store a " +
"second time and every consumer elsewhere would refuse to choose")
}
if !strings.Contains(err.Error(), "the-artifact-store") || !strings.Contains(err.Error(), "one per mesh") {
t.Fatalf("refused without naming the seat: %v", err)
}
}
+9
View File
@@ -97,6 +97,15 @@ func Rerouted(reference, address string) string {
//
// A kept reference with no store to route it through is refused: sent as it is, the runtime would
// refuse the scheme on the machine, one push away from the reason.
//
// **What this does not reach: the images genesis pinned.** The installer builds the control plane
// and the builder before the mesh exists, pushes them itself and pins their manifests to
// `<registry>:<port>/mesh-controller@…` and `<registry>:<port>/mesh-builder@…` — single-segment
// repositories with no build record, so `with.Built` does not name them and they are left as
// written until each is rebuilt through the mesh, which records it by digest and path. Until then
// a registry that moves strands exactly those two on a recreate, and the control plane's is the
// one that cannot be repaired through the mesh. Rebuild both through `build` before moving the
// store (novox/hq 04-ISSUES/102, finding F4).
func artifactsInto(resource map[string]any, module string, with Rendering) error {
for _, key := range []string{"image", "source"} {
written, ok := resource[key].(string)
+109 -18
View File
@@ -114,6 +114,14 @@ type Rendering struct {
// because which machines exist is a fact about the mesh.
Names map[string]string
// Machines is only the machines, by the same internal name — the subset of Names that is a
// node of this mesh rather than a name it was told to serve. Both matter and they are not the
// same set: a container's hosts wants every name, so a routed name resolves to the proxy that
// serves it, while a resolver told the mesh's suffix is authoritative for it answers from what
// it is given and forwards nothing — so a routed name written there is a name nobody asks for,
// standing beside the machines and looking as real as they do.
Machines map[string]string
Settings SettingsBy
Generators map[string]Generator
// Grants are the credentials this node must create, for the provisions it offers. Passed in
@@ -160,6 +168,13 @@ type Rendering struct {
// with an address — before references were kept without one — from an image a module runs
// straight from a public registry.
Built map[string]bool
// DataRoot is where this node keeps the directories the mesh places for its modules
// (novox/hq ADR 0112, to-be 27): a directory resource that states no path resolves to
// <DataRoot>/<module>/<id>, and ${dir:<id>} names that place from the module's own files,
// mounts and environment. A node setting fixed at installation; empty means the default,
// /var/lib — see dir_into.go.
DataRoot string
}
// machinePort is where a module's port lives on this machine, or the port itself when the mesh has
@@ -209,6 +224,21 @@ func (r Resolution) Compose(with Rendering) (Composed, error) {
}
func (r Resolution) compose(with Rendering, owner map[string]string) ([]map[string]any, error) {
// Every manifest is placed first (novox/hq ADR 0112): the maps naming where its bindings,
// credentials and contributions land are resolved against this node's directories, so every
// reader below — the binding files, the sealed secrets, the grant paths a contribution
// names — sees a concrete place and none learns the vocabulary.
// Into a fresh slice, never the caller's: one resolution may compose for many nodes, and a
// slice element written in place would carry the first node's places into the second's.
placed := make([]Manifest, len(r.Modules))
for i, m := range r.Modules {
var err error
if placed[i], err = placedManifest(m, with); err != nil {
return nil, err
}
}
r.Modules = placed
// Where each provision's credentials land, so a contribution can name the file rather than
// carry a value the mesh does not have.
directories := map[string]string{}
@@ -511,8 +541,11 @@ func (r Resolution) compose(with Rendering, owner map[string]string) ([]map[stri
}
// And what its bindings say, for the half of a connection that is not secret.
known := knownFor(m, r.Needs, r.Node)
// And where this node places the directories the module declared without a path
// (novox/hq ADR 0112) — resolved once per module, named by ${dir:…} from any resource.
dirs := dirsFor(m, with)
// And the machine underneath, which no binding of its own can tell it.
thisMachine := machineFacts(r)
thisMachine := machineFacts(r, with.Names)
// Which of this module's files carry a secret, for the rule that a container may not read
// one of them as its environment without saying so (ADR 0086, issue 041).
@@ -533,6 +566,13 @@ func (r Resolution) compose(with Rendering, owner map[string]string) ([]map[stri
// Said in the catalogue, not on the machine: the host parses strictly and knows no
// such field, and the reason is for a reader of the manifest.
delete(copied, SecretsInEnvironment)
// **Placed before anything reads a path.** A pathless directory receives the path
// this node resolves for it, and every ${dir:…} — in paths, mounts, content and
// environment — becomes that path, so what follows sees only concrete places
// (novox/hq ADR 0112). The host receives paths exactly as it always has.
if err := dirInto(copied, dirs, m.Module); err != nil {
return nil, err
}
// **After settings, and that is the whole reason it is here.** A module's file
// content is where a setting lands, so a placeholder may only exist once the setting
// has been put in — filling secrets first would look at content that is not yet what
@@ -604,7 +644,7 @@ func (r Resolution) compose(with Rendering, owner map[string]string) ([]map[stri
// plane's; making a name resolve is the module's software. Emitted as ordinary files under
// this module's name, so they are applied, reported and removed exactly as anything else
// it declares.
given, err := FactsInto(m, r, with.Names, with.Suffix)
given, err := FactsInto(m, r, with.Names, with.Machines, with.Suffix)
if err != nil {
return nil, err
}
@@ -903,30 +943,53 @@ func (r Resolution) contributions(settings SettingsBy, grants []Grant,
if err != nil {
return nil, fmt.Errorf("%s contributing to %s: %w", m.Module, to, err)
}
composeName(values, r.PublicDomain)
composeName(values, r.PublicDomain, r.At)
out[to] = append(out[to], Contribution{From: m.Module, Values: values})
}
// Several contributions to one requirement (ADR 0094's sibling for `contributes`): an
// object store's data API and its console are two different public names from one module,
// not one. Never in `granted` — a route names a host, not a credential — so every local
// name always reaches the provider from here.
for _, to := range sortedKeys(m.ContributesMany) {
for _, local := range sortedKeys(m.ContributesMany[to]) {
values, err := settle(m.ContributesMany[to][local], settings[m.Module], nil,
m.Module+" contributing "+local+" to "+to)
if err != nil {
return nil, fmt.Errorf("%s contributing %s to %s: %w", m.Module, local, to, err)
}
composeName(values, r.PublicDomain, r.At)
out[to] = append(out[to], Contribution{From: m.Module, Values: values})
}
}
}
return out, nil
}
// composeName joins a contribution's label with a node's public domain, in place (novox/hq ADR
// 0056).
// composeName joins a contribution's label with a node's public domain, and separately with its
// private one, in place (novox/hq ADR 0056).
//
// **The whole of what the mesh does with a route's name: join two given strings.** A contribution
// carries a `label` — the subdomain its operator chose — and the node carries its public domain;
// the granted name is `<label>.<public-domain>` and the mesh interprets neither half. It runs on
// any contribution carrying a label, not only a route's, because the mesh does not know what a
// **The whole of what the mesh does with a route's name: join two given strings — twice.** A
// contribution carries a `label` — the subdomain its operator chose — and the node carries its
// public domain and its own private-network address; the granted names are `<label>.<public-domain>`
// and `<label>.<internal-domain>`, and the mesh interprets none of the halves. It runs on any
// contribution carrying a label, not only a route's, because the mesh does not know what a
// provision means — a name it can compose from parts it was given is the point, whatever the
// provision is called.
//
// **The internal name is not a security boundary.** A predecessor proxy that answered both a
// public and a private-network hostname for the same route did so as a convenience — reaching a
// service over the VPN without a public TLS round trip — not as an access control, and composing
// the same alias here restores that convenience rather than adding one. A route with no internal
// domain to compose against (a node not on the private network) gets no internal name, the same as
// it gets no public one with no public domain.
//
// **Additive, so an unmigrated catalogue still works.** A contribution that already carries a full
// `name` and no `label` is left exactly as it is: the catalogue can migrate module by module while
// the running mesh keeps serving the full names it has. And a labelled contribution on a node with
// no public domain composes nothing — there is nothing to join it to — which reads downstream as a
// route that named no host, the same as it would have before this existed.
func composeName(values map[string]any, publicDomain string) {
if values == nil || publicDomain == "" {
func composeName(values map[string]any, publicDomain, internalDomain string) {
if values == nil {
return
}
if _, already := values["name"]; already {
@@ -939,14 +1002,25 @@ func composeName(values map[string]any, publicDomain string) {
if !ok || strings.TrimSpace(label) == "" {
return
}
if strings.TrimSpace(label) == "@" {
// The apex: a module served at the bare public domain, no subdomain — the zone-file
// convention `@`. Composes to the domain itself, so a node's own site is a label like any
// other rather than the one route that must still carry a full name.
values["name"] = publicDomain
trimmed := strings.TrimSpace(label)
if trimmed == "@" {
// The apex: a module served at the bare domain, no subdomain — the zone-file convention
// `@`. Composes to the domain itself, so a node's own site is a label like any other rather
// than the one route that must still carry a full name.
if publicDomain != "" {
values["name"] = publicDomain
}
if internalDomain != "" {
values["internal-name"] = internalDomain
}
return
}
values["name"] = strings.TrimSpace(label) + "." + publicDomain
if publicDomain != "" {
values["name"] = trimmed + "." + publicDomain
}
if internalDomain != "" {
values["internal-name"] = trimmed + "." + internalDomain
}
}
// receivedFile is the file a provider is given its consumers' contributions in.
@@ -1097,17 +1171,34 @@ func boundFile(n Needed, path, as string) (map[string]any, error) {
// arrangement refused is the ordinary one. A node running eight services against one database is
// not an edge case; it is what a machine looks like. Now each consumer has its own credential and
// there is nothing left to refuse.
//
// **One credential, even where a module contributes several times.** A module may answer one
// requirement more than once (ADR 0094's sibling for `contributes`) — an object store's data API
// and its console are two different names, not one. There is still only one `Needed` for it, one
// credential minted, one grant to settle: a pair credential is not a place to put a label or a
// port. So where several of this module's contributions reach the same requirement, none of them
// is "the" value — settling to the first, arbitrarily, would hand the grant one contribution's
// values under a credential the OTHER contribution's consumer never sees, and would collide with
// that contribution's own entry from contributions() besides. Empty values, still granted: the
// module asked, gets its credential, and each named contribution reaches the provider on its own.
func (r Resolution) ContributionsFrom(requirement, module string, settings SettingsBy) (
map[string]any, bool, error) {
all, err := r.contributions(settings, nil, nil)
if err != nil {
return nil, false, err
}
var mine []map[string]any
for _, g := range all[requirement] {
if g.From == module {
return g.Values, true, nil
mine = append(mine, g.Values)
}
}
if len(mine) == 1 {
return mine[0], true, nil
}
if len(mine) > 1 {
return map[string]any{}, true, nil
}
// It contributes no payload — but a require-only consumer of a parameterless provision (one whose
// `serves` names no consumer-supplied key: `redis-cache`, `amqp`) still ASKS for it and must be
// granted a credential. Keying "asks" on contributions alone marked those grants withdrawn
+322
View File
@@ -0,0 +1,322 @@
package catalogue
import (
"fmt"
"regexp"
"sort"
"strings"
)
// A directory the mesh places (novox/hq ADR 0112, to-be 27, issue 119).
//
// **A module definition names no host path.** A directory resource may omit `path`; the mesh
// resolves where it lands when the declaration is composed — `<root>/<module>/<id>`, the root a
// node's own setting with /var/lib as the default. From then on the module's own files, mounts
// and environment name the place as `${dir:<id>}`, the same shape as `${bound:…}` and
// `${secret:…}`: a fact the module asks for by name and never states.
//
// **A directory that states a path keeps it, and still answers `${dir:<id>}`.** That is the
// placement for an adopted machine: data that must sit where the predecessor already put it is
// declared with the path as the exception it is, and everything else in the module names it by
// id — so moving it later is one line, not a search.
//
// **Resolved here, not on the machine.** The host receives concrete paths exactly as it always
// has; nothing new reaches it and it learns no field. Which also means a resolved path changing
// is a spec change like any other — and the spec comparison must see it (novox/hq issue 126).
// defaultDataRoot is where module data lands when a node states no root of its own.
const defaultDataRoot = "/var/lib"
// dirRef is how a module names one of its placed directories: ${dir:<id>}.
var dirRef = regexp.MustCompile(`\$\{dir:([a-z0-9][a-z0-9-]*)\}`)
// dataRoot is the root this node keeps placed directories under.
func dataRoot(with Rendering) string {
if root := strings.TrimRight(strings.TrimSpace(with.DataRoot), "/"); root != "" {
return root
}
return defaultDataRoot
}
// dirsFor is every placed directory of a module, id → the path it resolves to on this node.
//
// A pathless directory saying `"place": "."` is the assignment's own root, <root>/<module> —
// to-be 27's one directory per assignment, which every other placed thing sits beneath. At most
// one makes sense; nothing enforces one, because two ids resolving to one path is a mistake the
// module's own files make visible immediately.
func dirsFor(m Manifest, with Rendering) map[string]string {
dirs := map[string]string{}
for _, r := range m.Resources {
if fmt.Sprint(r["type"]) != "directory" {
continue
}
id := fmt.Sprint(r["id"])
if path, stated := r["path"].(string); stated && path != "" {
dirs[id] = strings.TrimRight(path, "/")
continue
}
if place, said := r["place"].(string); said && place == "." {
dirs[id] = dataRoot(with) + "/" + m.Module
continue
}
dirs[id] = dataRoot(with) + "/" + m.Module + "/" + id
}
return dirs
}
// placedOrAbsolute says a path is usable where the mesh needs one: absolute already, or
// beginning with a placed reference — resolution makes it absolute before anything reads it.
// (unknownDirRefs is what checks the reference names a real directory.)
func placedOrAbsolute(path string) bool {
return strings.HasPrefix(path, "/") ||
(strings.HasPrefix(path, "${dir:") && dirRef.MatchString(path))
}
// dirFill resolves every ${dir:…} in one string, or refuses a reference naming no directory.
func dirFill(s string, dirs map[string]string, module string) (string, error) {
var missing error
out := dirRef.ReplaceAllStringFunc(s, func(ref string) string {
id := dirRef.FindStringSubmatch(ref)[1]
path, has := dirs[id]
if !has {
missing = fmt.Errorf(
"%s says ${dir:%s}, and %s declares no directory %q. It declares %s",
module, id, module, id, orNothing(namesOfDirs(dirs)))
return ref
}
return path
})
return out, missing
}
// placedManifest is the manifest with every path the mesh resolves already resolved: the maps
// naming where bindings, credentials and contributions land are filled against this node's
// placed directories, so everything downstream — the generated binding files, the sealed
// secrets, the grant directories — reads a concrete place and learns nothing new.
func placedManifest(m Manifest, with Rendering) (Manifest, error) {
dirs := dirsFor(m, with)
fillMap := func(in map[string]string) (map[string]string, error) {
if len(in) == 0 {
return in, nil
}
out := make(map[string]string, len(in))
for key, value := range in {
filled, err := dirFill(value, dirs, m.Module)
if err != nil {
return nil, err
}
out[key] = filled
}
return out, nil
}
var err error
if m.Receives, err = fillMap(m.Receives); err != nil {
return m, err
}
if m.Binds, err = fillMap(m.Binds); err != nil {
return m, err
}
if m.Secrets, err = fillMap(m.Secrets); err != nil {
return m, err
}
if m.OwnSecrets, err = fillMap(m.OwnSecrets); err != nil {
return m, err
}
if m.Grants, err = fillMap(m.Grants); err != nil {
return m, err
}
if len(m.SecretsMany) > 0 {
many := make(map[string]map[string]string, len(m.SecretsMany))
for to, locals := range m.SecretsMany {
if many[to], err = fillMap(locals); err != nil {
return m, err
}
}
m.SecretsMany = many
}
return m, nil
}
// dirInto places a resource: a pathless directory is given the path the mesh resolved for it,
// and every ${dir:…} the resource carries — in its path, its content, its mounts, its
// environment and its env-files — becomes that path.
//
// A reference naming no directory of this module is refused. Left as written, the literal
// `${dir:x}` would reach the machine as a path, and the runtime would create and mount a
// directory called `${dir:x}` — real, wrong, and named after the mistake.
func dirInto(resource map[string]any, dirs map[string]string, module string) error {
fill := func(s string) (string, error) { return dirFill(s, dirs, module) }
if fmt.Sprint(resource["type"]) == "directory" {
id := fmt.Sprint(resource["id"])
if path, stated := resource["path"].(string); !stated || path == "" {
resource["path"] = dirs[id]
}
// Said in the catalogue, not on the machine: the host parses strictly and knows no
// such field — resolved, the place IS the path.
delete(resource, "place")
}
var err error
if path, ok := resource["path"].(string); ok {
if resource["path"], err = fill(path); err != nil {
return err
}
}
if content, ok := resource["content"].(string); ok {
if resource["content"], err = fill(content); err != nil {
return err
}
}
// Nested values are rebuilt, never written into: the resource is a shallow copy of the
// manifest's own map, and the manifest is composed once per node — a fill written in place
// would leave the first node's paths inside every later composition.
if volumes, ok := resource["volumes"].([]any); ok {
filled := make([]any, len(volumes))
for i, v := range volumes {
filled[i] = v
if mount, ok := v.(string); ok {
if filled[i], err = fill(mount); err != nil {
return err
}
}
}
resource["volumes"] = filled
}
if env, ok := resource["env"].(map[string]any); ok {
filled := make(map[string]any, len(env))
for key, v := range env {
filled[key] = v
if value, ok := v.(string); ok {
if filled[key], err = fill(value); err != nil {
return err
}
}
}
resource["env"] = filled
}
if files, ok := resource["env-file"].([]any); ok {
filled := make([]any, len(files))
for i, v := range files {
filled[i] = v
if path, ok := v.(string); ok {
if filled[i], err = fill(path); err != nil {
return err
}
}
}
resource["env-file"] = filled
}
return nil
}
// unknownDirRefs is every ${dir:…} in the definition that names no directory the definition
// declares — refused where the author is, not at composition on some later day (the same
// near-versus-far reasoning as the host's strict parse).
func (m Manifest) unknownDirRefs() []string {
declared := map[string]bool{}
for _, r := range m.Resources {
if fmt.Sprint(r["type"]) == "directory" {
declared[fmt.Sprint(r["id"])] = true
}
}
referenced := func(s string) []string {
var ids []string
for _, match := range dirRef.FindAllStringSubmatch(s, -1) {
ids = append(ids, match[1])
}
return ids
}
var problems []string
for _, r := range m.Resources {
place, said := r["place"].(string)
if !said {
continue
}
if fmt.Sprint(r["type"]) != "directory" {
problems = append(problems, fmt.Sprintf(
"%s says place on %v, which is not a directory — only a directory is placed",
m.Module, r["id"]))
continue
}
if path, stated := r["path"].(string); stated && path != "" {
problems = append(problems, fmt.Sprintf(
"%s states both path and place on %v — a stated path IS the placement",
m.Module, r["id"]))
}
if place != "." {
problems = append(problems, fmt.Sprintf(
"%s says place %q on %v, and the only place is %q — the assignment's own root",
m.Module, place, r["id"], "."))
}
}
seen := map[string]bool{}
refuse := func(id string, where any) {
if declared[id] || seen[id] {
return
}
seen[id] = true
problems = append(problems, fmt.Sprintf(
"%s says ${dir:%s} in %v, and declares no directory %q — a reference the mesh "+
"cannot place would reach the machine as a literal path",
m.Module, id, where, id))
}
for _, r := range m.Resources {
for _, field := range []string{"path", "content"} {
if s, ok := r[field].(string); ok {
for _, id := range referenced(s) {
refuse(id, r["id"])
}
}
}
for _, field := range []string{"volumes", "env-file"} {
if list, ok := r[field].([]any); ok {
for _, v := range list {
if s, ok := v.(string); ok {
for _, id := range referenced(s) {
refuse(id, r["id"])
}
}
}
}
}
if env, ok := r["env"].(map[string]any); ok {
for _, v := range env {
if s, ok := v.(string); ok {
for _, id := range referenced(s) {
refuse(id, r["id"])
}
}
}
}
}
maps := map[string]map[string]string{
"receives": m.Receives, "binds": m.Binds, "secrets": m.Secrets,
"own-secrets": m.OwnSecrets, "grants": m.Grants,
}
for field, entries := range maps {
for _, value := range entries {
for _, id := range referenced(value) {
refuse(id, field)
}
}
}
for to, locals := range m.SecretsMany {
for _, value := range locals {
for _, id := range referenced(value) {
refuse(id, "secrets."+to)
}
}
}
sort.Strings(problems)
return problems
}
func namesOfDirs(dirs map[string]string) []string {
var names []string
for id := range dirs {
names = append(names, fmt.Sprintf("%q", id))
}
sort.Strings(names)
return names
}
+252
View File
@@ -0,0 +1,252 @@
package catalogue
// A directory the mesh places (novox/hq ADR 0112). These tests pin the contract: a pathless
// directory resolves under the node's root, ${dir:…} names it from every field a host path can
// live in, a stated path is the adopted-data placement and wins, an unknown reference refuses at
// the manifest, and filling for one node never leaks into the next composition.
import (
"strings"
"testing"
)
func placedModule() Manifest {
return Manifest{
Module: "photos",
Resources: []map[string]any{
{"id": "data", "type": "directory", "mode": "0700"},
{"id": "server-env", "type": "file", "path": "${dir:data}/server.env",
"content": "STORE=${dir:data}/objects\n"},
{"id": "server", "type": "container", "name": "photos-server",
"volumes": []any{"${dir:data}:/data"},
"env": map[string]any{"DATA": "${dir:data}/objects"},
"env-file": []any{"${dir:data}/server.env"}},
},
}
}
func TestAPathlessDirectoryResolvesUnderTheNodesRoot(t *testing.T) {
m := placedModule()
dirs := dirsFor(m, Rendering{})
if dirs["data"] != "/var/lib/photos/data" {
t.Fatalf("the default root is /var/lib and the shape is <root>/<module>/<id>; got %q", dirs["data"])
}
dirs = dirsFor(m, Rendering{DataRoot: "/tank/nox/"})
if dirs["data"] != "/tank/nox/photos/data" {
t.Fatalf("a node's own root is honoured, trailing slash and all; got %q", dirs["data"])
}
}
func TestDirReferencesBecomeThePlaceInEveryField(t *testing.T) {
m := placedModule()
dirs := dirsFor(m, Rendering{})
directory := shallowCopy(m.Resources[0])
if err := dirInto(directory, dirs, m.Module); err != nil {
t.Fatal(err)
}
if directory["path"] != "/var/lib/photos/data" {
t.Fatalf("a pathless directory receives its resolved path; got %v", directory["path"])
}
file := shallowCopy(m.Resources[1])
if err := dirInto(file, dirs, m.Module); err != nil {
t.Fatal(err)
}
if file["path"] != "/var/lib/photos/data/server.env" {
t.Fatalf("a file's path names the place; got %v", file["path"])
}
if file["content"] != "STORE=/var/lib/photos/data/objects\n" {
t.Fatalf("a file's content names the place; got %v", file["content"])
}
container := shallowCopy(m.Resources[2])
if err := dirInto(container, dirs, m.Module); err != nil {
t.Fatal(err)
}
if container["volumes"].([]any)[0] != "/var/lib/photos/data:/data" {
t.Fatalf("a mount names the place; got %v", container["volumes"])
}
if container["env"].(map[string]any)["DATA"] != "/var/lib/photos/data/objects" {
t.Fatalf("an environment value names the place; got %v", container["env"])
}
if container["env-file"].([]any)[0] != "/var/lib/photos/data/server.env" {
t.Fatalf("an env-file names the place; got %v", container["env-file"])
}
}
func TestAStatedPathIsThePlacementAndStillAnswersByName(t *testing.T) {
m := placedModule()
// The adopted-machine case: data that must sit where the predecessor already put it.
m.Resources[0]["path"] = "/services/mssql/data/"
dirs := dirsFor(m, Rendering{})
if dirs["data"] != "/services/mssql/data" {
t.Fatalf("a stated path wins over the root, trimmed; got %q", dirs["data"])
}
container := shallowCopy(m.Resources[2])
if err := dirInto(container, dirs, m.Module); err != nil {
t.Fatal(err)
}
if container["volumes"].([]any)[0] != "/services/mssql/data:/data" {
t.Fatalf("references follow the placement; got %v", container["volumes"])
}
}
func TestFillingForOneNodeLeaksIntoNoOther(t *testing.T) {
m := placedModule()
first := shallowCopy(m.Resources[2])
if err := dirInto(first, dirsFor(m, Rendering{DataRoot: "/first"}), m.Module); err != nil {
t.Fatal(err)
}
second := shallowCopy(m.Resources[2])
if err := dirInto(second, dirsFor(m, Rendering{DataRoot: "/second"}), m.Module); err != nil {
t.Fatal(err)
}
if got := second["volumes"].([]any)[0]; got != "/second/photos/data:/data" {
t.Fatalf("the second composition must see the manifest, not the first fill; got %v", got)
}
if m.Resources[2]["volumes"].([]any)[0] != "${dir:data}:/data" {
t.Fatalf("the manifest itself stays a template; got %v", m.Resources[2]["volumes"])
}
}
func TestAReferenceToNoDirectoryRefusesAtTheManifest(t *testing.T) {
m := placedModule()
m.Resources[2]["volumes"] = []any{"${dir:date}:/data"} // a typo, the likely shape
problems := m.unknownDirRefs()
if len(problems) != 1 || !strings.Contains(problems[0], `${dir:date}`) {
t.Fatalf("a reference naming no directory is a manifest problem; got %v", problems)
}
if got := placedModule().unknownDirRefs(); len(got) != 0 {
t.Fatalf("a correct definition has none; got %v", got)
}
}
func TestAReferenceToNoDirectoryRefusesAtCompositionToo(t *testing.T) {
m := placedModule()
container := shallowCopy(m.Resources[2])
container["env"] = map[string]any{"DATA": "${dir:date}"}
err := dirInto(container, dirsFor(m, Rendering{}), m.Module)
if err == nil || !strings.Contains(err.Error(), `"date"`) || !strings.Contains(err.Error(), `"data"`) {
t.Fatalf("the refusal names the mistake and what exists; got %v", err)
}
}
func TestTheAssignmentsOwnRootIsAPlace(t *testing.T) {
m := Manifest{Module: "mailu", Resources: []map[string]any{
{"id": "state", "type": "directory", "place": ".", "mode": "0700"},
{"id": "data-mail", "type": "directory"},
}}
dirs := dirsFor(m, Rendering{})
if dirs["state"] != "/var/lib/mailu" {
t.Fatalf("place %q is the assignment's root; got %q", ".", dirs["state"])
}
if dirs["data-mail"] != "/var/lib/mailu/data-mail" {
t.Fatalf("everything else sits beneath it; got %q", dirs["data-mail"])
}
root := shallowCopy(m.Resources[0])
if err := dirInto(root, dirs, m.Module); err != nil {
t.Fatal(err)
}
if root["path"] != "/var/lib/mailu" {
t.Fatalf("the root receives its path; got %v", root["path"])
}
if _, still := root["place"]; still {
t.Fatal("place must never reach the host, which parses strictly")
}
}
func TestTheManifestsMapsArePlaced(t *testing.T) {
m := Manifest{
Module: "photos",
Resources: []map[string]any{
{"id": "state", "type": "directory", "place": "."},
},
Binds: map[string]string{"route": "${dir:state}/route.json"},
Secrets: map[string]string{"mongodb-database": "${dir:state}/database.secret"},
OwnSecrets: map[string]string{"admin-key": "${dir:state}/admin-key.secret"},
Receives: map[string]string{"route": "${dir:state}/grants/mesh.json"},
}
placed, err := placedManifest(m, Rendering{})
if err != nil {
t.Fatal(err)
}
if placed.Binds["route"] != "/var/lib/photos/route.json" {
t.Fatalf("binds are placed; got %v", placed.Binds)
}
if placed.Secrets["mongodb-database"] != "/var/lib/photos/database.secret" {
t.Fatalf("secrets are placed; got %v", placed.Secrets)
}
if placed.OwnSecrets["admin-key"] != "/var/lib/photos/admin-key.secret" {
t.Fatalf("own-secrets are placed; got %v", placed.OwnSecrets)
}
if placed.Receives["route"] != "/var/lib/photos/grants/mesh.json" {
t.Fatalf("receives are placed; got %v", placed.Receives)
}
if m.Binds["route"] != "${dir:state}/route.json" {
t.Fatalf("the manifest itself stays a template; got %v", m.Binds)
}
}
func TestAMapReferenceToNoDirectoryRefusesAtTheManifest(t *testing.T) {
m := Manifest{
Module: "photos",
Resources: []map[string]any{{"id": "state", "type": "directory", "place": "."}},
Binds: map[string]string{"route": "${dir:stat}/route.json"},
}
problems := m.unknownDirRefs()
if len(problems) != 1 || !strings.Contains(problems[0], `${dir:stat}`) {
t.Fatalf("a map naming no directory is a manifest problem; got %v", problems)
}
}
func TestPlaceIsValidatedAtTheManifest(t *testing.T) {
both := Manifest{Module: "x", Resources: []map[string]any{
{"id": "d", "type": "directory", "place": ".", "path": "/somewhere"},
}}
if got := both.unknownDirRefs(); len(got) != 1 || !strings.Contains(got[0], "both path and place") {
t.Fatalf("path beside place refuses; got %v", got)
}
elsewhere := Manifest{Module: "x", Resources: []map[string]any{
{"id": "f", "type": "file", "place": ".", "path": "/somewhere", "content": ""},
}}
if got := elsewhere.unknownDirRefs(); len(got) != 1 || !strings.Contains(got[0], "not a directory") {
t.Fatalf("place on a file refuses; got %v", got)
}
wrong := Manifest{Module: "x", Resources: []map[string]any{
{"id": "d", "type": "directory", "place": "sub/dir"},
}}
if got := wrong.unknownDirRefs(); len(got) != 1 || !strings.Contains(got[0], `the only place is "."`) {
t.Fatalf("a place that is not the root refuses; got %v", got)
}
}
func TestTwoModulesPlacedRootsAreNoCollision(t *testing.T) {
a := Manifest{Module: "gitea", Resources: []map[string]any{
{"id": "state", "type": "directory", "place": "."},
{"id": "env", "type": "file", "path": "${dir:state}/server.env", "content": ""},
}}
b := Manifest{Module: "nextcloud", Resources: []map[string]any{
{"id": "state", "type": "directory", "place": "."},
{"id": "env", "type": "file", "path": "${dir:state}/server.env", "content": ""},
}}
if got := checkResources([]Manifest{a, b}); len(got) != 0 {
t.Fatalf("alike templates are different places; got %v", got)
}
// And the real collision is still caught: a module stating another's placed root.
c := Manifest{Module: "squatter", Resources: []map[string]any{
{"id": "nest", "type": "directory", "path": "/var/lib/gitea"},
}}
got := checkResources([]Manifest{a, c})
if len(got) != 1 || !strings.Contains(got[0], `"/var/lib/gitea"`) {
t.Fatalf("a stated path on a placed root collides; got %v", got)
}
}
func shallowCopy(resource map[string]any) map[string]any {
copied := map[string]any{}
for k, v := range resource {
copied[k] = v
}
return copied
}
+30 -9
View File
@@ -36,16 +36,23 @@ const (
// **A closed list.** A module asking for a fact the mesh does not have is asking for a file nobody
// will write, and finding that out on a machine — as a daemon that starts, reads nothing, and
// answers no queries — is worse than being told where the manifest is.
var facts = map[string]func(Resolution, map[string]string, string) string{
FactNodeNames: nodeNames,
FactNodeZones: nodeZones,
// A fact is written from the names it is about. `every` is every name the mesh serves — machines
// and the names it was told to route; `machines` is only the machines. A fact takes the set it is
// true of, and the two must not be confused (novox/hq 04-ISSUES/111).
var facts = map[string]func(r Resolution, every, machines map[string]string, suffix string) string{
FactNodeNames: func(r Resolution, every, _ map[string]string, suffix string) string {
return nodeNames(r, every, suffix)
},
FactNodeZones: func(r Resolution, _, machines map[string]string, suffix string) string {
return nodeZones(r, machines, suffix)
},
}
// FactsInto renders the facts a module asked for, as files it will be given.
//
// The module owns everything after the file exists: loading it, restarting on it, what a resolver
// does with it. This only puts it there.
func FactsInto(m Manifest, r Resolution, addresses map[string]string, suffix string) ([]map[string]any, error) {
func FactsInto(m Manifest, r Resolution, addresses, machines map[string]string, suffix string) ([]map[string]any, error) {
if len(m.Facts) == 0 {
return nil, nil
}
@@ -70,7 +77,7 @@ func FactsInto(m Manifest, r Resolution, addresses map[string]string, suffix str
}
out = append(out, map[string]any{
"id": "fact-" + name, "type": "file", "path": path, "mode": "0644",
"content": write(r, addresses, suffix),
"content": write(r, addresses, machines, suffix),
})
}
return out, nil
@@ -122,10 +129,18 @@ func nodeNames(r Resolution, addresses map[string]string, suffix string) string
//
// `*.homer.internal` is homer, which is the whole rule: if homer is at an address, so is anything
// homer serves. A module wanting this runs the resolver; the mesh only says what is true.
//
// **And the suffix itself, as a local domain.** A resolver that forwards what it cannot answer
// would otherwise send a mesh name it does not know — a machine that left, a typo — to a public
// resolver, which is a leak of the mesh's names for no answer. `local=` keeps everything under the
// suffix here: answered from the lines below or refused. Written in this file rather than in the
// resolver's own configuration because the suffix is the mesh's choice (the operator may have
// picked another) and this file is the one place the mesh writes what it chose.
func nodeZones(_ Resolution, addresses map[string]string, suffix string) string {
var b strings.Builder
b.WriteString("# Generated by the mesh. Do not edit — this file is replaced whenever a machine\n")
b.WriteString("# joins or leaves, and an edit would survive until then and vanish.\n\n")
fmt.Fprintf(&b, "local=/%s/\n", strings.TrimPrefix(suffixOr(suffix), "."))
for _, name := range sortedNames(addresses) {
internal, _ := meshName(name, suffix)
fmt.Fprintf(&b, "address=/%s/%s\n", internal, addresses[name])
@@ -139,16 +154,22 @@ func nodeZones(_ Resolution, addresses map[string]string, suffix string) string
// suffix is the one the control plane composed those names with, handed down rather than written
// here a second time — the alternative was `homer.internal.internal` on every machine.
func meshName(name, suffix string) (internal, bare string) {
if suffix == "" {
suffix = "internal"
}
dotted := "." + strings.TrimPrefix(suffix, ".")
dotted := "." + strings.TrimPrefix(suffixOr(suffix), ".")
if strings.HasSuffix(name, dotted) {
return name, strings.TrimSuffix(name, dotted)
}
return name + dotted, name
}
// suffixOr is the suffix given, or the one the mesh composes names with when none was handed down.
// The one place the default is written in this file, so a fact and a name cannot disagree about it.
func suffixOr(suffix string) string {
if suffix == "" {
return "internal"
}
return suffix
}
func sortedNames(addresses map[string]string) []string {
out := make([]string, 0, len(addresses))
for name, at := range addresses {
+57 -4
View File
@@ -8,10 +8,14 @@ import (
// Keyed by the internal name, as the control plane hands them (issue 079).
var threeMachines = map[string]string{"homer.internal": "10.42.0.1", "marge.internal": "10.42.0.2", "bart.internal": ""}
// **`*.homer.internal` is homer. That is the whole rule.**
// **`*.homer.internal` is homer. That is the whole rule.** And the suffix itself is local: a
// resolver that forwards what it cannot answer must not send a mesh name it does not know — a
// machine that left, a typo — to a public resolver (hal dnsmasq-app conversion, novox/hq
// 08-connectivity).
func TestEveryMachineIsAWildcardUnderItsOwnName(t *testing.T) {
out := nodeZones(Resolution{Node: "homer"}, threeMachines, "")
for _, want := range []string{
"local=/internal/",
"address=/homer.internal/10.42.0.1",
"address=/marge.internal/10.42.0.2",
} {
@@ -59,7 +63,7 @@ func TestAMachinesOwnNameIsItsMeshAddress(t *testing.T) {
// A module says where it wants a fact, and is given a file.
func TestAModuleIsGivenTheFactsItAskedFor(t *testing.T) {
m := Manifest{Module: "dnsmasq", Facts: map[string]string{FactNodeZones: "/etc/mesh/zones.conf"}}
given, err := FactsInto(m, Resolution{Node: "homer"}, threeMachines, "")
given, err := FactsInto(m, Resolution{Node: "homer"}, threeMachines, threeMachines, "")
if err != nil {
t.Fatal(err)
}
@@ -78,7 +82,7 @@ func TestAModuleIsGivenTheFactsItAskedFor(t *testing.T) {
// starts, reads a file nobody wrote, and answers no queries is a much worse way to find out.
func TestAskingForAFactTheMeshDoesNotHaveIsRefused(t *testing.T) {
m := Manifest{Module: "dnsmasq", Facts: map[string]string{"the-weather": "/etc/weather"}}
_, err := FactsInto(m, Resolution{}, nil, "")
_, err := FactsInto(m, Resolution{}, nil, nil, "")
if err == nil {
t.Fatal("a module asked for something nobody computes and was given nothing, silently")
}
@@ -92,7 +96,7 @@ func TestAskingForAFactTheMeshDoesNotHaveIsRefused(t *testing.T) {
// And a relative path is refused, or a module decides where the mesh writes on a machine.
func TestAFactMustBeAskedForAtAnAbsolutePath(t *testing.T) {
m := Manifest{Module: "dnsmasq", Facts: map[string]string{FactNodeNames: "etc/hosts"}}
if _, err := FactsInto(m, Resolution{}, nil, ""); err == nil {
if _, err := FactsInto(m, Resolution{}, nil, nil, ""); err == nil {
t.Fatal("a relative path was accepted")
}
}
@@ -128,8 +132,57 @@ func TestTheFactsWriteTheSuffixTheNamesWereComposedWith(t *testing.T) {
if !strings.Contains(zones, "address=/homer.lan/10.42.0.1") || strings.Contains(zones, "internal") {
t.Fatalf("the zones do not carry the operator's suffix as given:\n%s", zones)
}
if !strings.Contains(zones, "local=/lan/") {
t.Fatalf("the local domain is not the operator's suffix, so its names would leak upstream:\n%s", zones)
}
hosts := nodeNames(Resolution{Node: "homer"}, names, "lan")
if !strings.Contains(hosts, "10.42.0.1\thomer.lan\thomer\t# this machine") {
t.Fatalf("the hosts line does not carry the operator's suffix as given:\n%s", hosts)
}
}
// novox/hq 04-ISSUES/111: the map the control plane hands a resolution holds every name the mesh
// serves — the machines, and the names it was told to route to whichever machine serves them. A
// container's hosts wants all of it. A resolver's zones want only the machines: told the mesh's
// suffix is its own, it answers authoritatively for everything under it and forwards nothing, so a
// routed name written there with the suffix appended is a name nobody will ever ask for, standing
// beside the machines and looking as real.
func TestTheResolverIsToldTheMachinesAndNotTheNamesTheMeshMerelyServes(t *testing.T) {
machines := map[string]string{"homer.internal": "10.42.0.1", "marge.internal": "10.42.0.2"}
every := map[string]string{
"homer.internal": "10.42.0.1", "marge.internal": "10.42.0.2",
"drive.example.test": "10.42.0.1", "git.example.test": "10.42.0.2",
}
m := Manifest{Module: "resolver", Facts: map[string]string{
FactNodeZones: "/etc/zones.conf", FactNodeNames: "/etc/hosts",
}}
given, err := FactsInto(m, Resolution{Node: "homer"}, every, machines, "")
if err != nil {
t.Fatal(err)
}
by := map[string]string{}
for _, f := range given {
by[f["path"].(string)] = f["content"].(string)
}
zones := by["/etc/zones.conf"]
for _, machine := range []string{"address=/homer.internal/10.42.0.1", "address=/marge.internal/10.42.0.2"} {
if !strings.Contains(zones, machine) {
t.Fatalf("the resolver was not told %q:\n%s", machine, zones)
}
}
for _, served := range []string{"drive.example.test", "git.example.test"} {
if strings.Contains(zones, served) {
t.Fatalf("the resolver was told %q, a name the mesh serves rather than a machine:\n%s", served, zones)
}
}
// And the hosts file is the other way about: every name, so a container reaching a routed name
// finds the machine serving it.
hosts := by["/etc/hosts"]
for _, name := range []string{"homer.internal", "drive.example.test", "git.example.test"} {
if !strings.Contains(hosts, name) {
t.Fatalf("a container would not resolve %q from its hosts:\n%s", name, hosts)
}
}
}
+99 -101
View File
@@ -1,7 +1,6 @@
package catalogue
import (
"encoding/json"
"fmt"
"os"
"reflect"
@@ -85,9 +84,8 @@ func TestTheFilterModuleNeverFlushesTheRuleset(t *testing.T) {
}
// The package registry's port is the node's, like every other foundation port (novox/hq
// 04-ISSUES/085, ADR 0100). Two halves, because the forge is reached two ways: through what the
// module that serves it says it serves, and — for the genesis window, before any module provides
// `package-registry` at all — through the one binding the builder carries instead of resolving.
// 04-ISSUES/085, ADR 0100). The forge is reached through what it says it serves, and consumers —
// the builder among them — are told that, rather than carrying a number of their own.
func TestTheForgesPortIsGivenLikeAnyOtherProvidersPort(t *testing.T) {
forge := catalogueManifest(t, "gitea")
@@ -104,97 +102,67 @@ func TestTheForgesPortIsGivenLikeAnyOtherProvidersPort(t *testing.T) {
// And every consumer of the package registry is told where the machine actually put it,
// because that is read from what the forge serves rather than written in the consumer.
if got := ServedOn(forge, "package-registry", given)["port"]; got != 3100 {
if got := ServedOn(forge, "npm-package-registry", given)["port"]; got != 3100 {
t.Errorf("the package registry is served on %v, not the port this node gave it", got)
}
if got := ServedOn(forge, "package-registry", nil)["port"]; got != float64(3000) {
if got := ServedOn(forge, "npm-package-registry", nil)["port"]; got != float64(3000) {
t.Errorf("without a setting the forge serves %v, not the catalogue's port", got)
}
}
// bindingIn is the package binding the builder carries, as the machine would receive it.
func bindingIn(t *testing.T, m Manifest, layers []Layer) map[string]any {
t.Helper()
for _, r := range m.Resources {
if fmt.Sprint(r["id"]) != "package-binding" {
continue
}
settled, err := ApplySettings(r, layers)
if err != nil {
t.Fatalf("the builder's package binding refused %v: %v", layers, err)
}
if settled["merge"] != nil || settled["protected"] != nil {
t.Fatal("the host would be sent fields it does not know")
}
var out map[string]any
if err := json.Unmarshal([]byte(fmt.Sprint(settled["content"])), &out); err != nil {
t.Fatalf("the builder's package binding is not a binding: %v", err)
}
return out
// And so is where a repository on it is cloned from (novox/hq ADR 0111), for the same reason:
// a build composes the URL from what the forge serves, so a given port is a followed port.
if got := ServedOn(forge, "git", given)["port"]; got != 3100 {
t.Errorf("git is served on %v, not the port this node gave the forge", got)
}
t.Fatal("the builder carries no package binding")
return nil
}
func TestTheBuildersCarriedPackageBindingTakesThePortFromTheNode(t *testing.T) {
// **The builder requires the registry the npm seat delivers, and carries no binding of its own.**
//
// It used to carry a hand-written binding because nothing provided a package registry to resolve
// one from at genesis. The catalogue now requires it like any consumer, and ADR 0110 makes the
// seat's holder the answer when more than one module provides it — so a carried copy would be a
// second answer to the same question, free to drift from the first. Asserted gone, not merely
// unused.
func TestTheBuilderRequiresTheRegistryTheNpmSeatDelivers(t *testing.T) {
builder := catalogueManifest(t, "builder")
// Nothing set: the catalogue's own number, which is what a mesh raised on the defaults uses.
serves := bindingIn(t, builder, nil)["serves"].(map[string]any)
if serves["port"] != float64(3000) {
t.Fatalf("the builder's binding defaults to %v", serves["port"])
seat, _ := SeatNamed("npm-package-registry")
var requires bool
for _, r := range builder.Requires {
requires = requires || r == seat.Delivers
}
// Given a port, the binding dials it — and the rest of what the forge serves survives, because
// a setting is merged into the module's own values rather than replacing them.
moved := bindingIn(t, builder, []Layer{{From: "anchor",
Values: map[string]any{"serves": map[string]any{"port": float64(3100)}}}})
got := moved["serves"].(map[string]any)
if got["port"] != float64(3100) {
t.Errorf("the builder dials %v, not the port this node gave the package registry", got["port"])
if !requires {
t.Fatalf("the builder does not require %q: %v", seat.Delivers, builder.Requires)
}
if got["scheme"] != "http" || got["npm-path"] != "/api/packages/novox/npm/" {
t.Errorf("setting the port lost the rest of what the forge serves: %v", got)
if builder.Binds[seat.Delivers] == "" {
t.Errorf("the builder is not told where the registry is: binds %v", builder.Binds)
}
if moved["as"] != "mesh-builder" || moved["from"] != "gitea" {
t.Errorf("setting the port changed who the binding is with: %v", moved)
}
}
// The two halves are one number. The builder carries a binding because at genesis nothing provides
// `package-registry` to resolve one from; the day the forge is a module, the same consumer is told
// what the forge serves. They have to start from the same port, or a mesh raised on the defaults
// dials one number before the forge is assigned and another after.
func TestTheBuildersCarriedBindingStartsWhereTheForgeServes(t *testing.T) {
forge := ServedOn(catalogueManifest(t, "gitea"), "package-registry", nil)
carried := bindingIn(t, catalogueManifest(t, "builder"), nil)["serves"].(map[string]any)
for _, key := range []string{"port", "scheme", "npm-path"} {
if fmt.Sprint(forge[key]) != fmt.Sprint(carried[key]) {
t.Errorf("the forge serves %s %v and the builder's carried binding says %v — the two "+
"halves of the same registry have drifted apart in the catalogue",
key, forge[key], carried[key])
for _, r := range builder.Resources {
if fmt.Sprint(r["id"]) == "package-binding" {
t.Fatal("the builder carries its own package binding beside the one the mesh resolves")
}
}
}
func TestTheBuildersPackageBindingKeepsItsIdentity(t *testing.T) {
builder := catalogueManifest(t, "builder")
// `at` above all: a setting that moves it points the builder, and the registry password it
// sends as basic auth, at a host somebody else chose.
for _, key := range []string{"provision", "from", "at", "as"} {
var refused error
for _, r := range builder.Resources {
if fmt.Sprint(r["id"]) != "package-binding" {
continue
}
_, refused = ApplySettings(r, []Layer{{From: "anchor",
Values: map[string]any{key: "something else"}}})
}
if refused == nil {
t.Errorf("%q can be set on the builder's package binding, which is not a port but who "+
"the binding is with", key)
// The forge holds the seats it answers for (novox/hq ADR 0110, 0111), parsed by the real parser —
// which refuses a delivering seat claimed by a module that does not provide what it delivers.
func TestTheForgeHoldsTheNpmAndGitSeats(t *testing.T) {
forge := catalogueManifest(t, "gitea")
holds := map[string]bool{}
for _, c := range forge.Claims {
holds[c.Name] = true
}
for _, seat := range []string{"npm-package-registry", "git"} {
if !holds[seat] {
t.Errorf("gitea does not claim the %s seat: %+v", seat, forge.Claims)
}
}
git := ServedOn(forge, "git", nil)
if git["scheme"] != "http" || git["port"] != float64(3000) {
t.Errorf("gitea serves nothing a clone URL can be composed from: %v", git)
}
npm := ServedOn(forge, "npm-package-registry", nil)
if npm["npm-path"] != "/api/packages/novox/npm/" {
t.Errorf("gitea no longer says where its npm registry is: %v", npm)
}
}
// **And the forge's own address follows it**, composed from the manifest in the catalogue beside
@@ -251,27 +219,13 @@ func TestTheForgesOwnAddressFollowsThePortTheNodeGaveIt(t *testing.T) {
}
}
// **And the port the forge publishes the long way is the node's too** (novox/hq ADR 0100).
//
// The forge's ssh port is written `2222:22` — the machine's own daemon holds 22, so the module
// takes 2222 and says so in `listens`. A node whose predecessor served git on another number
// cannot be told to leave it there unless the setting may name the machine side of that mapping,
// which is the number the manifest itself uses everywhere else. Composed from the manifest in the
// catalogue beside this checkout, because what the mesh can move is a fact about what the module
// actually writes.
func TestTheForgesSshPortIsGivenByTheNumberTheForgeCallsIt(t *testing.T) {
// gitea's own sshd is unmodified — the module's own internal port is 22, the number in
// `listens`, the same convention every other module in the catalogue uses (its internal port,
// not an invented identity). Composed from the manifest in the catalogue beside this checkout,
// because what the mesh publishes is a fact about what the module actually writes.
func declaredGiteaSsh(t *testing.T, given map[int]int) map[string]any {
t.Helper()
forge := catalogueManifest(t, "gitea")
given, err := GivenPorts(forge, []Layer{{From: "anchor",
Values: map[string]any{PortsSetting: map[string]any{"2222": float64(222)}}}})
if err != nil {
t.Fatalf("the forge's ssh port cannot be given on a node: %v", err)
}
// Under the number the module listens on — 2222, the machine side of its mapping — which is
// the number the plan, the filter, the openings and the consumer all ask for. One entry.
if want := map[int]int{2222: 222}; !reflect.DeepEqual(given, want) {
t.Fatalf("the forge was given %v, and it names its ssh port %v", given, want)
}
resolved, err := forge.Resolve([]Built{{
Name: "runtime", Kind: ArtifactImage,
Reference: "registry.example/gitea-runtime@sha256:" + strings.Repeat("a", 64),
@@ -286,9 +240,13 @@ func TestTheForgesSshPortIsGivenByTheNumberTheForgeCallsIt(t *testing.T) {
{Name: "secret", For: "gitea", From: "anchor", Local: "internal-token", Sealed: "sealed-token"},
{Name: "secret", For: "gitea", From: "anchor", Local: "admin", Sealed: "sealed-admin"},
}}
givenPorts := map[int]int{3000: 3000}
for k, v := range given {
givenPorts[k] = v
}
out, err := r.Declaration(Rendering{
Needed: map[string]map[string]string{"gitea": {"broker": "sealed-broker"}},
Ports: map[string]map[int]int{"gitea": {3000: 3000, 2222: 222}},
Ports: map[string]map[int]int{"gitea": givenPorts},
Given: map[string]map[int]int{"gitea": given},
})
if err != nil {
@@ -298,8 +256,48 @@ func TestTheForgesSshPortIsGivenByTheNumberTheForgeCallsIt(t *testing.T) {
if server == nil {
t.Fatalf("the forge's own container is not in the declaration: %v", out)
}
if published := fmt.Sprint(server["ports"]); !strings.Contains(published, "222:22") ||
strings.Contains(published, "2222:22") {
return server
}
// **The forge publishes ssh at the mesh's own fixed convention by default** (novox/hq ADR 0100).
//
// `222` is the mesh's own public convention for the forge's ssh, written directly in the
// manifest's `ports` — every node the forge has run on used the same number, so it needs no
// per-node setting to reach it.
func TestTheForgesSshPortIsTheMeshsFixedConventionByDefault(t *testing.T) {
forge := catalogueManifest(t, "gitea")
// Nothing was given — no node moved this port — which is the ordinary answer: the mesh only
// reports what a setting moved, and the manifest's own `222:22` needs no move to be reached.
given, err := GivenPorts(forge, nil)
if err != nil {
t.Fatalf("the forge's ssh port cannot be given on a node: %v", err)
}
if len(given) != 0 {
t.Fatalf("nothing moved the forge's ssh port, yet it was given %v", given)
}
server := declaredGiteaSsh(t, given)
if published := fmt.Sprint(server["ports"]); !strings.Contains(published, "222:22") {
t.Fatalf("the forge is published on %v, not its own fixed convention", server["ports"])
}
}
// **A node whose predecessor served git on a different number can still be told to leave it
// there.** The setting names the port the module itself listens on — 22, gitea's own sshd, the
// same number `listens` uses — not the mesh's own default machine-side number, so moving it does
// not require guessing what the manifest happens to default to.
func TestANodeMayGiveTheForgesSshPortADifferentNumber(t *testing.T) {
forge := catalogueManifest(t, "gitea")
given, err := GivenPorts(forge, []Layer{{From: "anchor",
Values: map[string]any{PortsSetting: map[string]any{"22": float64(9022)}}}})
if err != nil {
t.Fatalf("the forge's ssh port cannot be moved on a node: %v", err)
}
if want := map[int]int{22: 9022}; !reflect.DeepEqual(given, want) {
t.Fatalf("the forge was given %v, and the setting named %v", given, want)
}
server := declaredGiteaSsh(t, given)
if published := fmt.Sprint(server["ports"]); !strings.Contains(published, "9022:22") ||
strings.Contains(published, "222:22") {
t.Fatalf("the forge is published on %v, not the port this node gave it", server["ports"])
}
}
+14 -3
View File
@@ -22,8 +22,11 @@ import (
// provides, it does not require. Written as a literal it would be a manifest carrying one
// deployment's machine name, which is the shape [ADR 0066] exists to remove.
//
// Two facts, both the mesh's own vocabulary — the same `node` and `at` a contribution already
// carries. Nothing about what a machine is *for*: that would be the mesh learning what a module
// Three facts, all the mesh's own vocabulary — the same `node` and `at` a contribution already
// carries, and the address behind `at`, for software that takes an address and not a name. The
// case that found the third is a resolver pointing the container runtime at itself: the runtime's
// list of resolvers is addresses, because a name there would have to be resolved by the resolver
// it names. Nothing about what a machine is *for*: that would be the mesh learning what a module
// means, which it does not do.
// ofMachine is where a module says a fact about the machine underneath it belongs:
@@ -49,10 +52,18 @@ func machineUsed(content string) []string {
// to be reached at an address that does not exist is a misconfiguration, and it is said here —
// where the module and the machine are both named — rather than discovered later as a certificate
// nobody can verify.
func machineFacts(r Resolution) map[string]string {
//
// `address` is what `at` resolves to, read from the names the control plane composed — the same map
// the hosts file and the resolver's wildcards are written from, so a file naming the machine's
// address and the file every other machine reaches it by cannot disagree. Absent, like `at`, when
// the machine is off the network or the mesh has not placed it.
func machineFacts(r Resolution, names map[string]string) map[string]string {
out := map[string]string{"name": r.Node}
if r.At != "" {
out["at"] = r.At
if address := names[r.At]; address != "" {
out["address"] = address
}
}
return out
}
@@ -70,3 +70,36 @@ func TestAnAddressAMachineDoesNotHaveIsRefused(t *testing.T) {
t.Errorf("the refusal does not name what was asked for: %v", err)
}
}
// A module that must give software the machine's ADDRESS rather than its name — a resolver pointing
// the container runtime at itself, whose list of resolvers cannot be a name — says
// ${machine:address}, and gets what the machine's name resolves to on the private network: the same
// address every other machine's hosts file carries for it.
func TestAModuleNamesTheAddressBehindItsMachinesName(t *testing.T) {
pointing := Manifest{Module: "pointing", Version: "1", Resources: []map[string]any{{
"id": "runtime", "type": "file", "path": "/etc/runtime.json",
"content": `{"dns":["${machine:address}"]}`,
}}}
got, err := Resolve(shelf(pointing), []string{"pointing"}, anchored(), World{})
if err != nil {
t.Fatal(err)
}
out, err := got.Declaration(Rendering{Names: map[string]string{
"workstation.internal": "10.42.0.7", "anchor.internal": "10.42.0.1"}})
if err != nil {
t.Fatal(err)
}
if content := plainly(out[0]["content"]); content != `{"dns":["10.42.0.7"]}` {
t.Fatalf("the machine's address was not the one its name resolves to: %q", content)
}
// Off the network there is no such address, and the refusal says what the machine does have —
// rather than a placeholder written into the runtime's file and read as an address.
got, err = Resolve(shelf(pointing), []string{"pointing"}, workstation(), World{})
if err != nil {
t.Fatal(err)
}
if _, err := got.Declaration(Rendering{}); err == nil || !strings.Contains(err.Error(), "${machine:address}") {
t.Fatalf("a machine off the network was given an address, or refused for another reason: %v", err)
}
}
+153 -17
View File
@@ -233,6 +233,18 @@ type Manifest struct {
// module that had to say both would eventually say one.
Contributes map[string]map[string]any `json:"contributes,omitempty"`
// ContributesMany is the same key, `contributes`, where a module tells one provider several
// things under local names — `"route": {"api": {"label": "files-api", "port": 9000}, "console":
// {"label": "files", "port": 9001}}` — because a module may answer one requirement more than
// once: an object store with a data API and a console are two different public names, not one
// (novox/hq ADR 0094's sibling for `contributes` rather than `secrets` — "a module may need more
// than one value from a provider that gives one per pair" applies exactly as well to what a
// module gives a provider as to what it keeps from one). Each local name is a contribution of
// its own, reaching the provider as its own entry in the file it receives.
//
// Filled from the manifest's `contributes` object by UnmarshalJSON; never written by hand.
ContributesMany map[string]map[string]map[string]any `json:"-"`
// Receives is where this module wants its consumers' contributions written, per requirement
// it provides.
//
@@ -435,6 +447,18 @@ type BuildsOn struct {
Image string `json:"image,omitempty"`
}
// ArtifactContext names the repository an image artifact's build context is cloned from, when
// that is not this module's own repository.
type ArtifactContext struct {
// Repository is cloned fresh, the same way the module's own repository is — a working tree
// nothing has touched, so what was built is reproducible from the two commits named rather
// than from whatever a previous build happened to leave behind.
Repository string `json:"repository"`
// Ref is the branch, tag or commit of that repository to build. Empty means its own default
// branch — the same meaning an empty module ref already has.
Ref string `json:"ref,omitempty"`
}
// Artifact is one thing built from a module's source.
type Artifact struct {
// Name is how resources refer to it. Local to the module.
@@ -454,6 +478,17 @@ type Artifact struct {
// Empty means the whole recipe, which is what a module with one image says by saying nothing.
Target string `json:"target,omitempty"`
// Context names a second repository this image's build reaches into for its own source — the
// recipe itself is still read from this module's own directory, at this module's own commit;
// only the build context `docker build`'s final argument names comes from here instead.
//
// **Packaging and source are allowed to live apart.** A module that only ships the recipe for
// source that lives elsewhere — the reference route-proxy in mesh-controller's own repository,
// packaged as a module in the catalogue rather than vendored a second time the two copies
// could drift from — names where that source actually is. Empty means the ordinary case: an
// image built from this same module's own repository, the same as every other artifact.
Context *ArtifactContext `json:"context,omitempty"`
// Language is what this module's code is written in, for a bundle.
//
// **Declared, never guessed.** Inferring it from what files happen to be present makes a
@@ -615,16 +650,24 @@ func AccessID(path string) string { return "access-" + strings.TrimPrefix(path,
// it contributes to.
func (m Manifest) Wants() []string {
out := append([]string{}, m.Requires...)
for to := range m.Contributes {
var already bool
add := func(to string) {
for _, r := range m.Requires {
if r == to {
already = true
return
}
}
if !already {
out = append(out, to)
for _, already := range out {
if already == to {
return
}
}
out = append(out, to)
}
for to := range m.Contributes {
add(to)
}
for to := range m.ContributesMany {
add(to)
}
sort.Strings(out)
return out
@@ -679,6 +722,47 @@ func (m *Manifest) UnmarshalJSON(raw []byte) error {
}
delete(keys, "secrets")
}
contributesPlain := map[string]map[string]any{}
contributesMany := map[string]map[string]map[string]any{}
if contributes, ok := keys["contributes"]; ok && string(contributes) != "null" {
var byTo map[string]json.RawMessage
if err := json.Unmarshal(contributes, &byTo); err != nil {
return fmt.Errorf("contributes: an object of requirement to values, or to {local name: values}: %w", err)
}
for to, v := range byTo {
// Both shapes are JSON objects, unlike secrets' path-vs-object split, so the shapes are
// told apart by what is INSIDE: an ordinary contribution's fields are scalars (a label,
// a port); the several-instance shape is an object of local names, each itself an
// object of fields. Confirmed against the whole catalogue before relying on it — no
// contribution anywhere has an object-valued field.
var fields map[string]json.RawMessage
if err := json.Unmarshal(v, &fields); err != nil {
return fmt.Errorf("contributes.%s: an object of values, or of local name to values: %w", to, err)
}
many := len(fields) > 0
for _, field := range fields {
trimmed := bytes.TrimSpace(field)
if len(trimmed) == 0 || trimmed[0] != '{' {
many = false
break
}
}
if many {
var locals map[string]map[string]any
if err := json.Unmarshal(v, &locals); err != nil {
return fmt.Errorf("contributes.%s: an object of local name to values: %w", to, err)
}
contributesMany[to] = locals
continue
}
var values map[string]any
if err := json.Unmarshal(v, &values); err != nil {
return fmt.Errorf("contributes.%s: an object of values: %w", to, err)
}
contributesPlain[to] = values
}
delete(keys, "contributes")
}
rest, err := json.Marshal(keys)
if err != nil {
return err
@@ -696,22 +780,46 @@ func (m *Manifest) UnmarshalJSON(raw []byte) error {
if len(many) > 0 {
m.SecretsMany = many
}
if len(contributesPlain) > 0 {
m.Contributes = contributesPlain
}
if len(contributesMany) > 0 {
m.ContributesMany = contributesMany
}
return nil
}
// MarshalJSON writes `secrets` back in the shape it was read: paths, and objects of local names.
// MarshalJSON writes `secrets` and `contributes` back in the shape they were read: single values,
// and objects of local names.
func (m Manifest) MarshalJSON() ([]byte, error) {
raw, err := json.Marshal(manifestFields(m))
if err != nil {
return nil, err
}
if len(m.SecretsMany) == 0 {
if len(m.SecretsMany) == 0 && len(m.ContributesMany) == 0 {
return raw, nil
}
var keys map[string]json.RawMessage
if err := json.Unmarshal(raw, &keys); err != nil {
return nil, err
}
if len(m.ContributesMany) > 0 {
mergedContributes := map[string]any{}
for to, values := range m.Contributes {
mergedContributes[to] = values
}
for to, locals := range m.ContributesMany {
mergedContributes[to] = locals
}
contributes, err := json.Marshal(mergedContributes)
if err != nil {
return nil, err
}
keys["contributes"] = contributes
}
if len(m.SecretsMany) == 0 {
return json.Marshal(keys)
}
merged := map[string]any{}
for to, path := range m.Secrets {
merged[to] = path
@@ -842,9 +950,11 @@ func ParseManifest(raw []byte) (Manifest, error) {
problems = append(problems, fmt.Sprintf("%s requires itself", m.Module))
}
}
wellFormed := true
for _, c := range m.Claims {
if !name.MatchString(c.Name) {
problems = append(problems, fmt.Sprintf("%q is not a usable claim name", c.Name))
wellFormed = false
}
switch c.At() {
case ScopeNode, ScopeSite, ScopeMesh:
@@ -852,8 +962,14 @@ func ParseManifest(raw []byte) (Manifest, error) {
problems = append(problems, fmt.Sprintf(
"%s claims %s at scope %q; a claim is held per node, per site or per mesh",
m.Module, c.Name, c.Scope))
wellFormed = false
}
}
// Against the seats the mesh defines (novox/hq ADR 0110), once every claim is at least a name
// and a scope — a malformed claim is refused for that, not a second time for being unknown.
if wellFormed {
problems = append(problems, claimProblems(m)...)
}
if m.Computed != "" && len(m.Resources) > 0 {
// One or the other. A module that both ships files and has them computed would leave
// nobody able to say where a given file came from.
@@ -872,6 +988,25 @@ func ParseManifest(raw []byte) (Manifest, error) {
"%s contributes nothing to %q; if it only needs one, require it", m.Module, to))
}
}
for to, locals := range m.ContributesMany {
if !name.MatchString(to) {
problems = append(problems, fmt.Sprintf("%q is not a usable name to contribute to", to))
}
if len(locals) == 0 {
problems = append(problems, fmt.Sprintf(
"%s contributes nothing to %q; if it only needs one, require it", m.Module, to))
}
for local, values := range locals {
if !name.MatchString(local) {
problems = append(problems, fmt.Sprintf(
"%s contributes to %q under %q, which is not a usable name", m.Module, to, local))
}
if len(values) == 0 {
problems = append(problems, fmt.Sprintf(
"%s contributes nothing to %q under %q", m.Module, to, local))
}
}
}
problems = append(problems, m.Build.problems(m.Module)...)
// **What provides the artifact store cannot be delivered through it** (novox/hq 04-ISSUES/029).
//
@@ -914,9 +1049,9 @@ func ParseManifest(raw []byte) (Manifest, error) {
}
}
for to, where := range m.Binds {
if !strings.HasPrefix(where, "/") {
if !placedOrAbsolute(where) {
problems = append(problems, fmt.Sprintf(
"%s binds %q at %q, which is not an absolute path", m.Module, to, where))
"%s binds %q at %q, which is neither an absolute path nor a placed one", m.Module, to, where))
}
var wanted bool
for _, w := range m.Wants() {
@@ -1048,9 +1183,9 @@ func ParseManifest(raw []byte) (Manifest, error) {
}
}
for name, where := range m.OwnSecrets {
if !strings.HasPrefix(where, "/") {
if !placedOrAbsolute(where) {
problems = append(problems, fmt.Sprintf(
"%s needs %q at %q, which is not an absolute path", m.Module, name, where))
"%s needs %q at %q, which is neither an absolute path nor a placed one", m.Module, name, where))
}
if name == "" {
problems = append(problems, m.Module+" needs a secret with no name")
@@ -1065,9 +1200,9 @@ func ParseManifest(raw []byte) (Manifest, error) {
}
}
for _, f := range m.SecretFiles(to) {
if !strings.HasPrefix(f.Path, "/") {
if !placedOrAbsolute(f.Path) {
problems = append(problems, fmt.Sprintf(
"%s keeps the credential for %q at %q, which is not an absolute path",
"%s keeps the credential for %q at %q, which is neither an absolute path nor a placed one",
m.Module, SecretLocal(to, f.Local), f.Path))
}
if f.Local != "" && !name.MatchString(f.Local) {
@@ -1117,9 +1252,9 @@ func ParseManifest(raw []byte) (Manifest, error) {
}
}
for to, where := range m.Grants {
if !strings.HasPrefix(where, "/") {
if !placedOrAbsolute(where) {
problems = append(problems, fmt.Sprintf(
"%s grants %q into %q, which is not an absolute path", m.Module, to, where))
"%s grants %q into %q, which is neither an absolute path nor a placed one", m.Module, to, where))
}
var offered bool
for _, o := range m.Offers() {
@@ -1140,9 +1275,9 @@ func ParseManifest(raw []byte) (Manifest, error) {
if !name.MatchString(to) {
problems = append(problems, fmt.Sprintf("%q is not a usable name to receive", to))
}
if !strings.HasPrefix(where, "/") {
if !placedOrAbsolute(where) {
problems = append(problems, fmt.Sprintf(
"%s receives %q at %q, which is not an absolute path", m.Module, to, where))
"%s receives %q at %q, which is neither an absolute path nor a placed one", m.Module, to, where))
}
var offered bool
for _, o := range m.Offers() {
@@ -1189,6 +1324,7 @@ func ParseManifest(raw []byte) (Manifest, error) {
// Checked here rather than on the machine because the machine cannot tell the difference: by
// the time it sees the mount it is being asked to create the directory, which it can do.
problems = append(problems, m.undeclaredMounts()...)
problems = append(problems, m.unknownDirRefs()...)
for i, r := range m.Resources {
id, _ := r["id"].(string)
+33 -1
View File
@@ -87,6 +87,10 @@ type Provider struct {
At string
// Serves is what the providing module said a consumer needs to know, settled.
Serves map[string]any
// Module is which module on that node provides it. A provider is a (node, module) pair
// (novox/hq to-be 23), and the pair is what tells the holder of a seat apart from another module
// providing the same thing (ADR 0110).
Module string
}
// Held is a claim somebody already has, used for the scopes wider than one node.
@@ -381,6 +385,15 @@ func Resolve(catalogue map[string]Manifest, assigned []string, node Node, world
default:
chosenNode, pinned := world.Pinned[want]
if !pinned {
// **The seat's holder answers, when a seat delivers this** (novox/hq ADR 0110).
// Not a guess, which ADR 0009 refuses: the choice was made once, mesh-wide, by
// assigning the holder, where a pin makes it again on every consumer's node. A
// pin still wins — it is a consumer coupled to one provider's contents, and has
// said so.
if holder, held := HolderAmong(want, where, world.Held); held {
take(holder)
break
}
problems = append(problems, fmt.Sprintf(
"%d nodes provide %q, wanted by %s — say which with `pin %s %s <node>`: %s",
len(where), want, because[want], node.Name, want,
@@ -692,11 +705,30 @@ func checkResources(modules []Manifest) []string {
ownedPath := map[string]string{} // path → owning module, for the access check below
for _, m := range modules {
// Compared placed, not as written (novox/hq ADR 0112): ${dir:state} is the same six
// characters in every module and a different directory in each — two modules' templates
// being spelled alike is not two modules owning one path. The default root serves the
// comparison: a collision is within one node, and any one root keeps distinct modules'
// places distinct. A reference that cannot be placed is left as written — naming what
// does not exist is the manifest's own problem, refused where it was made.
dirs := dirsFor(m, Rendering{})
for _, r := range m.Resources {
for _, field := range []string{"path", "unit", "name", "package"} {
value, ok := r[field].(string)
if !ok || value == "" {
continue
if field == "path" && fmt.Sprint(r["type"]) == "directory" {
// A pathless directory owns its placed path — a module stating that
// very path is exactly the collision this exists to catch.
value = dirs[fmt.Sprint(r["id"])]
}
if value == "" {
continue
}
}
if field == "path" {
if placed, err := dirFill(value, dirs, m.Module); err == nil {
value = placed
}
}
key := field + " " + value
if other, taken := owner[key]; taken && other != m.Module {
@@ -0,0 +1,191 @@
package catalogue
import (
"encoding/json"
"strings"
"testing"
)
// The catalogue's resolver modules as they are, parsed by the real parser and composed as a
// machine would receive them (hal dnsmasq-app conversion, novox/hq 08-connectivity).
//
// The predecessor's resolver answered every name on a machine: the mesh's own itself, the rest
// forwarded to two fixed upstreams, with the machine's resolv.conf naming it alone and the
// container runtime pointed at its private-network address. These hold the mesh's modules to the
// same arrangement, and to the two things a resolver here must never do — read resolv.conf for
// its upstreams, or take an address systemd-resolved holds.
// resolverShelf is the three resolver modules beside something that answers `mesh-addressing`.
// The networking module that really does is composed in the controller and cannot be imported
// here, so a stand-in offers the same word; what is under test is the manifests, not the network.
func resolverShelf(t *testing.T) map[string]Manifest {
t.Helper()
shelf := map[string]Manifest{
"net": {Module: "net", Version: "1", Provides: []Offer{{Name: "mesh-addressing"}}},
}
for _, name := range []string{"dnsmasq", "resolv-conf", "resolved-split-dns"} {
shelf[name] = catalogueManifest(t, name)
}
return shelf
}
// twoMachines is what the control plane hands a rendering: internal names and their addresses.
var twoMachines = map[string]string{"anchor.internal": "10.42.0.1", "laptop.internal": "10.42.0.2"}
// Its configuration forwards to the upstreams the predecessor's module shipped, and gets them from
// nowhere else: `no-resolv` is what makes the documented loop — the resolver finding its own
// address in resolv.conf and becoming its own upstream — impossible.
func TestTheResolverForwardsToFixedUpstreamsAndNeverReadsResolvConf(t *testing.T) {
m := catalogueManifest(t, "dnsmasq")
var config string
for _, r := range m.Resources {
if r["id"] == "config" {
config, _ = r["content"].(string)
}
}
if config == "" {
t.Fatal("the resolver has no configuration file")
}
for _, want := range []string{
"\nno-resolv\n", "\nserver=1.1.1.1\n", "\nserver=8.8.8.8\n",
"\nlisten-address=127.0.0.1\n", "\ninterface=mesh0\n", "\nbind-dynamic\n",
"\ndomain-needed\n", "\nbogus-priv\n",
"\nconf-file=" + m.Facts[FactNodeZones] + "\n",
} {
if !strings.Contains(config, want) {
t.Errorf("the resolver's configuration lacks %q:\n%s", strings.TrimSpace(want), config)
}
}
// Not .53 or .54, which systemd-resolved holds; and not .55 any more, which was a convention
// beside the one every machine already followed — the predecessor's resolv.conf says .1.
for _, taken := range []string{"127.0.0.53", "127.0.0.54", "127.0.0.55"} {
if strings.Contains(config, "listen-address="+taken) {
t.Errorf("the resolver listens on %s", taken)
}
}
// And the file that decides what the machine asks names it there, alone.
var resolv string
for _, r := range catalogueManifest(t, "resolv-conf").Resources {
if r["path"] == "/etc/resolv.conf" {
resolv, _ = r["content"].(string)
}
}
var nameservers []string
for _, line := range strings.Split(resolv, "\n") {
if strings.HasPrefix(line, "nameserver ") {
nameservers = append(nameservers, strings.TrimPrefix(line, "nameserver "))
}
}
if len(nameservers) != 1 || nameservers[0] != "127.0.0.1" {
t.Errorf("resolv.conf names %v; the predecessor's names the mesh's resolver alone at 127.0.0.1", nameservers)
}
// The split-DNS alternative points at the same address, or a machine that keeps
// systemd-resolved in charge would route the mesh's suffix to nothing.
for _, r := range catalogueManifest(t, "resolved-split-dns").Resources {
if content, _ := r["content"].(string); content != "" && !strings.Contains(content, "DNS=127.0.0.1\n") {
t.Errorf("resolved-split-dns does not point at the resolver's address:\n%s", content)
}
}
}
// The resolver and what points the machine at it compose on one machine, and what arrives is the
// mesh's account of every machine as a wildcard, the suffix kept local, the daemon restarting on
// that file, and the runtime pointed at this machine's own address.
func TestTheResolverAndWhatAsksItComposeOnOneMachine(t *testing.T) {
got, err := Resolve(resolverShelf(t), []string{"dnsmasq", "resolv-conf"},
Node{Name: "anchor", At: "anchor.internal"}, World{})
if err != nil {
t.Fatal(err)
}
if !strings.Contains(strings.Join(named(got), " "), "net") {
t.Fatalf("the resolver's data is the mesh's addresses, and nothing answering them was taken: %v", named(got))
}
out, err := got.Declaration(Rendering{
// Names is every name the mesh serves; Machines is the subset that is a node (novox/hq
// issue 111) — the resolver's zones read only the second, and in this scenario the two
// happen to be the same map, since nothing routed is part of it.
Names: twoMachines, Machines: twoMachines, Suffix: "internal",
Needed: map[string]map[string]string{"dnsmasq": {"broker": "sealed"}},
})
if err != nil {
t.Fatal(err)
}
ids := byID(out)
zones := ids["dnsmasq.fact-node-zones"]
if zones == nil || zones["path"] != "/etc/mesh-resolver/nodes.conf" {
t.Fatalf("the resolver was not given the machines where its configuration reads them: %v", zones)
}
content, _ := zones["content"].(string)
for _, want := range []string{
"local=/internal/", "address=/anchor.internal/10.42.0.1", "address=/laptop.internal/10.42.0.2",
} {
if !strings.Contains(content, want) {
t.Errorf("the machines file lacks %q:\n%s", want, content)
}
}
service := ids["dnsmasq.service"]
if service == nil {
t.Fatal("no resolver service composed")
}
reflects := map[string]bool{}
for _, id := range service["restart-on"].([]any) {
reflects[id.(string)] = true
}
if !reflects["dnsmasq.config"] || !reflects["dnsmasq.fact-node-zones"] {
t.Errorf("the daemon does not restart on its configuration and the machines file both: %v", service["restart-on"])
}
// The runtime's own file, written into (novox/hq ADR 0102) with the one key this module states.
runtime := ids["dnsmasq.runtime-dns"]
if runtime == nil || runtime["path"] != "/etc/docker/daemon.json" || runtime["into"] != "json" {
t.Fatalf("the runtime's dns is not written into its file: %v", runtime)
}
var keys map[string][]string
if err := json.Unmarshal([]byte(runtime["content"].(string)), &keys); err != nil {
t.Fatalf("the runtime's keys are not JSON: %v", err)
}
if len(keys) != 1 || len(keys["dns"]) != 1 || keys["dns"][0] != "10.42.0.1" {
t.Errorf("the runtime is pointed at %v; containers resolve at this machine's own private-network address, and nothing else is written", keys)
}
for _, r := range out {
if r["type"] == "service" && r["unit"] == "docker.service" && r["id"] != "" &&
strings.HasPrefix(r["id"].(string), "dnsmasq.") {
t.Errorf("the resolver orders the runtime restarted or reloaded, which stops every container (ADR 0102) or does nothing for dns: %v", r)
}
}
resolv := ids["resolv-conf.resolv"]
if resolv == nil || !strings.Contains(resolv["content"].(string), "\nnameserver 127.0.0.1\n") {
t.Fatalf("the machine is not pointed at the resolver: %v", resolv)
}
}
// Two modules deciding what a machine asks are refused on one machine, as before — the claim
// exists so they never take turns overwriting each other.
func TestTwoThingsDecidingWhatAMachineAsksAreRefused(t *testing.T) {
_, err := Resolve(resolverShelf(t), []string{"dnsmasq", "resolv-conf", "resolved-split-dns"},
Node{Name: "anchor", At: "anchor.internal"}, World{})
if err == nil {
t.Fatal("resolv-conf and resolved-split-dns were both assigned to one machine")
}
if !strings.Contains(err.Error(), "the-resolver-configuration") {
t.Fatalf("the refusal does not say what was claimed: %v", err)
}
}
// A machine that is not on the private network has no address for the runtime to be pointed at.
// Refused where the module and the machine are both named, rather than a placeholder written into
// the runtime's file and read as an address.
func TestTheResolverOnAMachineOffTheNetworkIsRefused(t *testing.T) {
got, err := Resolve(resolverShelf(t), []string{"dnsmasq"}, Node{Name: "anchor"}, World{})
if err != nil {
t.Fatal(err)
}
_, err = got.Declaration(Rendering{Names: twoMachines, Suffix: "internal",
Needed: map[string]map[string]string{"dnsmasq": {"broker": "sealed"}}})
if err == nil || !strings.Contains(err.Error(), "${machine:address}") {
t.Fatalf("a machine off the network was composed a resolver, or refused for another reason: %v", err)
}
}
+66
View File
@@ -132,6 +132,72 @@ func TestALabelWithNoPublicDomainComposesNothing(t *testing.T) {
}
}
// withPrivateAddress is a workstation on the private network, at the given internal name — the
// same fact a route's own consumers already receive as `${bound:...:at}`.
func withPrivateAddress(at string) Node {
n := workstation()
n.At = at
return n
}
func TestALabelComposesWithTheNodesPrivateAddressToo(t *testing.T) {
// A predecessor proxy answered a route on both a public and a private-network hostname for the
// same convenience the mesh restores here: reaching a service over the VPN without a public TLS
// round trip. Composed independently of the public name, from the node's own `At`.
got, err := Resolve(shelf(proxy(), labelled("board", "git", 8080)),
[]string{"board"}, withPrivateAddress("anchor.internal"), World{})
if err != nil {
t.Fatal(err)
}
given := received(t, mustDeclare(t, got))
if given[0].Values["internal-name"] != "git.anchor.internal" {
t.Fatalf("the label did not compose with the private address: %v", given[0].Values)
}
}
func TestThePublicAndInternalNamesComposeIndependently(t *testing.T) {
// A node with both a public domain and a private address gets both names from one label; a
// node with only one of the two gets only the matching one — neither composition depends on
// the other being possible.
both := withPrivateAddress("anchor.internal")
both.PublicDomain = "example.tld"
got, err := Resolve(shelf(proxy(), labelled("board", "git", 8080)),
[]string{"board"}, both, World{})
if err != nil {
t.Fatal(err)
}
given := received(t, mustDeclare(t, got))
if given[0].Values["name"] != "git.example.tld" {
t.Fatalf("the public name did not compose alongside the internal one: %v", given[0].Values)
}
if given[0].Values["internal-name"] != "git.anchor.internal" {
t.Fatalf("the internal name did not compose alongside the public one: %v", given[0].Values)
}
publicOnly, err := Resolve(shelf(proxy(), labelled("board", "git", 8080)),
[]string{"board"}, withDomain("example.tld"), World{})
if err != nil {
t.Fatal(err)
}
givenPublicOnly := received(t, mustDeclare(t, publicOnly))
if _, has := givenPublicOnly[0].Values["internal-name"]; has {
t.Fatalf("an internal name was composed with no private address to compose it from: %v",
givenPublicOnly[0].Values)
}
}
func TestTheApexLabelComposesToTheBarePrivateAddress(t *testing.T) {
got, err := Resolve(shelf(proxy(), labelled("board", "@", 4000)),
[]string{"board"}, withPrivateAddress("anchor.internal"), World{})
if err != nil {
t.Fatal(err)
}
given := received(t, mustDeclare(t, got))
if given[0].Values["internal-name"] != "anchor.internal" {
t.Fatalf("the apex label did not compose to the bare private address: %v", given[0].Values)
}
}
func TestARoutedNameResolvesToTheServingNode(t *testing.T) {
// novox/hq ADR 0066 propagate: a granted route name is published into internal resolution,
// mapped to the node that serves it, alongside the `<node>.internal` names — so every
+59
View File
@@ -101,6 +101,19 @@ func TestTheControlPlanesOwnAddressesFollowTheNodesPorts(t *testing.T) {
if err != nil {
t.Fatalf("the control plane's own manifest does not parse:\n%v", err)
}
// The manifest itself names them now; withSeatPorts is a no-op on it, and this holds it so.
for _, r := range m.Resources {
if r["type"] != "container" {
continue
}
env, _ := r["env"].(map[string]any)
for key, want := range SeatPorts {
if env[key] != want {
t.Errorf("module.json says %s=%v, not %q", key, env[key], want)
}
}
}
m = withSeatPorts(m)
control, err := m.Resolve([]Built{{
Name: "server", Kind: ArtifactImage,
Reference: ArtifactStoreScheme + "mesh-controller/server@sha256:" + strings.Repeat("c", 64),
@@ -155,3 +168,49 @@ func TestTheControlPlanesOwnAddressesFollowTheNodesPorts(t *testing.T) {
t.Errorf("with no settings, the control plane is told %v", env)
}
}
// SeatPorts is what the control plane's manifest says beside each sealed connection: the port
// this machine put the seat's holder at (novox/hq 04-ISSUES/102).
var SeatPorts = map[string]string{
"MESH_STORE_INVENTORY_PORT": "${seat:mesh-store:5432}",
"MESH_STORE_IDENTITY_PORT": "${seat:mesh-store:5432}",
"MESH_STORE_LICENCES_PORT": "${seat:mesh-store:5432}",
"MESH_BROKER_AMQP_PORT": "${seat:mesh-broker:5672}",
"MESH_BROKER_MANAGEMENT_PORT": "${seat:mesh-broker:15672}",
"MESH_BROKER_ADDRESS_PORT": "${seat:mesh-broker:5671}",
}
// withSeatPorts is the control plane's manifest with SeatPorts in its container's environment.
//
// **The manifest lands one commit after the code that fills it**, deliberately: a control plane
// still running the previous build passes `${seat:…}` through unfilled, and the manifest may only
// name the placeholder once every control plane that could compose it knows it. So the test does
// not depend on module.json carrying these yet, and is a no-op once it does.
func withSeatPorts(m Manifest) Manifest {
out := m
out.Resources = nil
for _, r := range m.Resources {
if r["type"] != "container" {
out.Resources = append(out.Resources, r)
continue
}
copied := map[string]any{}
for k, v := range r {
copied[k] = v
}
env := map[string]any{}
if had, ok := r["env"].(map[string]any); ok {
for k, v := range had {
env[k] = v
}
}
for k, v := range SeatPorts {
if _, said := env[k]; !said {
env[k] = v
}
}
copied["env"] = env
out.Resources = append(out.Resources, copied)
}
return out
}
+149
View File
@@ -0,0 +1,149 @@
package catalogue
import (
"fmt"
"sort"
"strings"
)
// The seats a mesh can have (novox/hq ADR 0110).
//
// **A closed set, defined here rather than by whoever claims one.** Until this, a well-formed name
// became a seat by being claimed, so nothing could say which seats a mesh has or who fills them:
// `the-showcase` and `the-build-machine` were each invented by the module claiming it. The set is
// what a person reads to learn what a mesh can have, so an entry nobody argued for is an entry
// nobody can explain — the same reason every shape in the host's vocabulary names its decision.
//
// A seat is held by a module assignment. What the mesh knows about a holder is what it knows about
// that assignment; nothing about holders is kept here or anywhere else.
// Seat is one role the mesh defines.
type Seat struct {
// Name is what a manifest claims.
Name string
// Scope is where there may be only one holder.
Scope string
// Delivers is the provision the seat's holder answers for, or empty. A seat that delivers a
// provision may only be held by a module providing it at the seat's scope, and its holder is
// what a requirement for that provision resolves to when several modules provide it.
Delivers string
// Decision is the record that made it a seat.
Decision string
}
// seats is the whole set, in the order a person reads it: the mesh's own, then a node's.
var seats = []Seat{
{Name: "mesh-controller", Scope: ScopeMesh, Decision: "novox/hq ADR 0079"},
{Name: "mesh-store", Scope: ScopeMesh, Delivers: "postgres-database", Decision: "novox/hq ADR 0079"},
{Name: "mesh-broker", Scope: ScopeMesh, Delivers: "amqp", Decision: "novox/hq ADR 0079"},
{Name: "the-artifact-store", Scope: ScopeMesh, Delivers: "artifact-store", Decision: "novox/hq ADR 0075"},
{Name: "the-catalogue", Scope: ScopeMesh, Decision: "novox/hq ADR 0110"},
{Name: "npm-package-registry", Scope: ScopeMesh, Delivers: "npm-package-registry", Decision: "novox/hq ADR 0109"},
{Name: "git", Scope: ScopeMesh, Delivers: "git", Decision: "novox/hq ADR 0111"},
{Name: "the-build-machine", Scope: ScopeNode, Decision: "novox/hq ADR 0110"},
{Name: "the-dns-port", Scope: ScopeNode, Decision: "novox/hq ADR 0110"},
{Name: "the-intrusion-prevention", Scope: ScopeNode, Decision: "novox/hq ADR 0110"},
{Name: "the-packet-filter", Scope: ScopeNode, Decision: "novox/hq ADR 0110"},
{Name: "the-private-network", Scope: ScopeNode, Decision: "novox/hq ADR 0110"},
{Name: "the-resolver-configuration", Scope: ScopeNode, Decision: "novox/hq ADR 0110"},
{Name: "the-showcase", Scope: ScopeNode, Decision: "novox/hq ADR 0110"},
}
// Seats is every seat the mesh defines, in reading order.
func Seats() []Seat {
return append([]Seat(nil), seats...)
}
// SeatNamed is the seat a claim names, if the mesh defines one.
func SeatNamed(name string) (Seat, bool) {
for _, s := range seats {
if s.Name == name {
return s, true
}
}
return Seat{}, false
}
// SeatDelivering is the seat whose holder answers for a provision, if there is one.
func SeatDelivering(provision string) (Seat, bool) {
if provision == "" {
return Seat{}, false
}
for _, s := range seats {
if s.Delivers == provision {
return s, true
}
}
return Seat{}, false
}
// claimProblems is what is wrong with a manifest's claims against the set.
//
// Three refusals, each naming the seat: a seat the mesh does not define, a seat claimed at another
// scope, and a seat that delivers a provision claimed by a module that does not provide it — which
// would make the module the mesh's answer for something it cannot answer.
func claimProblems(m Manifest) []string {
var problems []string
for _, c := range m.Claims {
seat, known := SeatNamed(c.Name)
if !known {
problems = append(problems, fmt.Sprintf(
"%s claims %q, which is not a seat this mesh defines (novox/hq ADR 0110) — "+
"the seats are: %s", m.Module, c.Name, seatNames()))
continue
}
if c.At() != seat.Scope {
problems = append(problems, fmt.Sprintf(
"%s claims %s at scope %q, and %s is a %s seat",
m.Module, c.Name, c.At(), c.Name, seat.Scope))
}
if seat.Delivers != "" && !providesAt(m, seat.Delivers, seat.Scope) {
problems = append(problems, fmt.Sprintf(
"%s claims %s, whose holder answers for %q, and %s does not provide %q at %s scope",
m.Module, c.Name, seat.Delivers, m.Module, seat.Delivers, seat.Scope))
}
}
return problems
}
func providesAt(m Manifest, provision, scope string) bool {
for _, o := range m.Provides {
if o.Name == provision && o.At() == scope {
return true
}
}
return false
}
func seatNames() string {
names := make([]string, 0, len(seats))
for _, s := range seats {
names = append(names, s.Name)
}
sort.Strings(names)
return strings.Join(names, ", ")
}
// HolderAmong is which of several providers of a provision holds the seat that delivers it.
//
// Found by the (node, module) pair, because a provider is identified by both (novox/hq to-be 23):
// two modules on one node could both provide a provision, and only the one holding the seat
// answers for it. Nothing when no seat delivers the provision, when nobody holds
// it, or when the holder is not among the providers offered.
func HolderAmong(provision string, providers []Provider, held []Held) (Provider, bool) {
seat, delivered := SeatDelivering(provision)
if !delivered {
return Provider{}, false
}
for _, h := range held {
if h.Claim != seat.Name || h.Scope != seat.Scope {
continue
}
for _, p := range providers {
if p.Node == h.Node && p.Module == h.Module {
return p, true
}
}
}
return Provider{}, false
}
+208
View File
@@ -0,0 +1,208 @@
package catalogue
import (
"encoding/json"
"os"
"path/filepath"
"regexp"
"strings"
"testing"
)
// Defends novox/hq ADR 0110: a seat is a module assignment from a closed set.
// The set is closed, and changing it is a decision.
//
// **The count is asserted, and every entry names the record that made it a seat**, so the next
// person changing the set finds the argument rather than a number to edit — the pattern the host's
// vocabulary test follows. If this fails because a seat was added, the fix is a record in novox/hq
// and a row in to-be 26, not a new number here.
func TestTheSeatsAreAClosedSetAndEachNamesItsDecision(t *testing.T) {
record := regexp.MustCompile(`^novox/hq ADR \d{4}$`)
seen := map[string]bool{}
delivered := map[string]string{}
for _, s := range Seats() {
if seen[s.Name] {
t.Errorf("%s is in the set twice", s.Name)
}
seen[s.Name] = true
if !record.MatchString(s.Decision) {
t.Errorf("%s names %q as its decision; every seat names the record that made it one",
s.Name, s.Decision)
}
switch s.Scope {
case ScopeNode, ScopeSite, ScopeMesh:
default:
t.Errorf("%s is held per %q, which is not a scope", s.Name, s.Scope)
}
if s.Delivers != "" {
// Two seats answering for one provision would put the question "which one?" back,
// which is the question a seat exists to answer.
if other, twice := delivered[s.Delivers]; twice {
t.Errorf("%s and %s both deliver %q", other, s.Name, s.Delivers)
}
delivered[s.Delivers] = s.Name
}
}
if len(Seats()) != 14 {
t.Errorf("the mesh defines %d seats rather than 14; the set is closed, so a change here is "+
"a decision (novox/hq ADR 0110): %s", len(Seats()), seatNames())
}
}
func claimed(claims string) []byte {
return []byte(`{"module":"thing","version":"1","provides":[{"name":"npm-package-registry","scope":"mesh"}],"claims":` + claims + `}`)
}
func TestAClaimOnASeatTheMeshDoesNotDefineIsRefused(t *testing.T) {
_, err := ParseManifest(claimed(`[{"name":"the-anything","scope":"node"}]`))
if err == nil {
t.Fatal("a module invented a seat by claiming it")
}
if !strings.Contains(err.Error(), "the-anything") || !strings.Contains(err.Error(), "not a seat") {
t.Fatalf("the refusal does not say the seat is unknown: %v", err)
}
// And it says what the seats are, because "no" without the list sends somebody reading code.
if !strings.Contains(err.Error(), "the-packet-filter") {
t.Fatalf("the refusal does not list the seats: %v", err)
}
}
func TestASeatClaimedAtAnotherScopeIsRefused(t *testing.T) {
_, err := ParseManifest(claimed(`[{"name":"npm-package-registry","scope":"node"}]`))
if err == nil {
t.Fatal("a mesh seat was held per node")
}
if !strings.Contains(err.Error(), "mesh seat") {
t.Fatalf("the refusal does not say which scope the seat is: %v", err)
}
}
func TestADeliveringSeatIsOnlyHeldByAModuleThatProvides(t *testing.T) {
// Holding it makes the module the mesh's answer for the provision. A module that cannot answer
// would be the answer anyway, and every consumer would be sent to it.
raw := []byte(`{"module":"thing","version":"1","claims":[{"name":"git","scope":"mesh"}]}`)
_, err := ParseManifest(raw)
if err == nil {
t.Fatal("a module holding the git seat need not provide git")
}
if !strings.Contains(err.Error(), `does not provide "git"`) {
t.Fatalf("the refusal does not say what is missing: %v", err)
}
}
func TestAClaimThatIsMalformedIsRefusedOnceForThat(t *testing.T) {
// Not a second time for being unknown: one mistake, one line.
_, err := ParseManifest(claimed(`[{"name":"Not A Name","scope":"node"}]`))
if err == nil {
t.Fatal("a malformed claim was accepted")
}
if strings.Contains(err.Error(), "not a seat") {
t.Fatalf("a malformed claim was also called unknown: %v", err)
}
}
// Every module in use claims a seat in the set, so closing it refuses nothing that runs.
//
// Read from the catalogue beside this checkout and from this repository's own manifest, the two
// places a manifest lives (ADR 0069). The private-network module's manifest is composed in code,
// and its claim is checked where it is composed.
func TestEveryManifestInUseClaimsASeatTheMeshDefines(t *testing.T) {
paths, _ := filepath.Glob("../../../mesh-catalog/modules/*/module.json")
if len(paths) == 0 {
t.Skip("the catalogue is not beside this checkout")
}
paths = append(paths, "../../module.json")
var checked int
for _, path := range paths {
raw, err := os.ReadFile(path)
if err != nil {
t.Fatal(err)
}
// Leniently, so a manifest refused for something unrelated is not reported as a seat
// problem, and the seat check below is the only thing this test holds a module to.
var m Manifest
if err := json.Unmarshal(raw, &m); err != nil {
t.Fatalf("%s: %v", path, err)
}
for _, problem := range claimProblems(m) {
t.Errorf("%s: %s", path, problem)
}
checked += len(m.Claims)
}
if checked == 0 {
t.Fatal("no claims were checked, so this proved nothing")
}
}
// The holder of a seat answers among several providers.
func registryShelf() map[string]Manifest {
return shelf(
Manifest{Module: "gitea", Version: "1", Provides: FromAnywhere("npm-package-registry"),
Claims: []Claim{{Name: "npm-package-registry", Scope: ScopeMesh}}},
Manifest{Module: "verdaccio", Version: "1", Provides: FromAnywhere("npm-package-registry")},
Manifest{Module: "builder", Version: "1", Requires: []string{"npm-package-registry"}},
)
}
func twoRegistries() map[string][]Provider {
return map[string][]Provider{"npm-package-registry": {
{Node: "anchor", At: "anchor.internal", Module: "gitea"},
{Node: "archive", At: "archive.internal", Module: "verdaccio"},
}}
}
func giteaHoldsTheSeat() []Held {
return []Held{{Claim: "npm-package-registry", Scope: ScopeMesh, Node: "anchor", Module: "gitea"}}
}
func TestTheSeatsHolderAnswersWhenSeveralProvide(t *testing.T) {
// The whole point: a second registry beside the holder harms nothing, and nobody pins.
got, err := Resolve(registryShelf(), []string{"builder"}, reachable(),
World{Offered: twoRegistries(), Held: giteaHoldsTheSeat()})
if err != nil {
t.Fatal(err)
}
if len(got.Needs) != 1 || got.Needs[0].From != "anchor" {
t.Fatalf("the seat's holder did not answer: %v", got.Needs)
}
}
func TestAPinStillWinsOverTheSeat(t *testing.T) {
// A consumer coupled to one provider's contents has said so, and the seat does not overrule it.
got, err := Resolve(registryShelf(), []string{"builder"}, reachable(),
World{Offered: twoRegistries(), Held: giteaHoldsTheSeat(),
Pinned: map[string]string{"npm-package-registry": "archive"}})
if err != nil {
t.Fatal(err)
}
if len(got.Needs) != 1 || got.Needs[0].From != "archive" {
t.Fatalf("the pin was overruled by the seat: %v", got.Needs)
}
}
func TestWithTheSeatUnheldSeveralProvidersAreStillRefused(t *testing.T) {
// No seat held is no choice made, and ADR 0009's rule stands: never guessed.
_, err := Resolve(registryShelf(), []string{"builder"}, reachable(),
World{Offered: twoRegistries()})
if err == nil {
t.Fatal("one of two registries was picked with nobody holding the seat")
}
if !strings.Contains(err.Error(), "pin") {
t.Fatalf("the refusal does not say how to choose: %v", err)
}
}
func TestTheHolderIsTheModuleNotTheMachine(t *testing.T) {
// Two modules on one machine could provide the same thing; only the one holding the seat
// answers. A holder matched by node alone would send consumers to whichever came first.
providers := []Provider{
{Node: "anchor", At: "anchor.internal", Module: "verdaccio"},
{Node: "anchor", At: "anchor.internal", Module: "gitea"},
}
holder, held := HolderAmong("npm-package-registry", providers, giteaHoldsTheSeat())
if !held || holder.Module != "gitea" {
t.Fatalf("the holder was not told apart from a neighbour: %+v", holder)
}
}
@@ -0,0 +1,167 @@
package catalogue
import (
"encoding/json"
"testing"
)
// A module may answer one requirement more than once, the sibling of ADR 0094 for `contributes`
// rather than `secrets`: an object store's data API and its console are two different public
// names, not one. `contributes` maps a requirement to several sets of values under local names,
// each reaching the provider as its own entry — the same "several from one" shape ADR 0094 gave
// `secrets`, applied to the other half of an edge.
const twoRoutes = `{"module":"minio","version":"1","requires":["route"],
"contributes":{"route":{"api":{"label":"files-api","port":9000},"console":{"label":"files","port":9001}}}}`
func TestContributesReadsBothShapesAndWritesThemBack(t *testing.T) {
m, err := ParseManifest([]byte(twoRoutes))
if err != nil {
t.Fatal(err)
}
locals := m.ContributesMany["route"]
if len(locals) != 2 || locals["api"]["label"] != "files-api" || locals["console"]["port"] != float64(9001) {
t.Fatalf("two contributions under local names: %+v", locals)
}
plain, err := ParseManifest([]byte(`{"module":"board","version":"1","requires":["route"],
"contributes":{"route":{"label":"board","port":8080}}}`))
if err != nil {
t.Fatal(err)
}
if got := plain.Contributes["route"]; got["label"] != "board" || len(plain.ContributesMany) != 0 {
t.Fatalf("the plain shape is one contribution with no local names: %+v / %+v", got, plain.ContributesMany)
}
// Written back in the shape it was read, so a built manifest keeps its local names.
raw, err := json.Marshal(m)
if err != nil {
t.Fatal(err)
}
again, err := ParseManifest(raw)
if err != nil {
t.Fatalf("what was written does not read: %v\n%s", err, raw)
}
if len(again.ContributesMany["route"]) != 2 {
t.Fatalf("the local names did not survive a round trip:\n%s", raw)
}
}
func TestAContributionLocalNameMustBeUsable(t *testing.T) {
for _, bad := range []string{
// Not a usable name.
`{"module":"minio","version":"1","requires":["route"],
"contributes":{"route":{"Not OK":{"label":"files","port":9000}}}}`,
// A local contribution with nothing in it.
`{"module":"minio","version":"1","requires":["route"],
"contributes":{"route":{"api":{}}}}`,
} {
if _, err := ParseManifest([]byte(bad)); err == nil {
t.Errorf("accepted:\n%s", bad)
}
}
}
func minimalRouteProxy() Manifest {
return Manifest{Module: "route-proxy", Version: "1",
Provides: FromAnywhere("route"),
Receives: map[string]string{"route": "/var/lib/route-proxy/routes/mesh.json"},
}
}
func TestAModuleWithTwoRoutesGivesTheProviderTwoContributions(t *testing.T) {
minio, err := ParseManifest([]byte(twoRoutes))
if err != nil {
t.Fatal(err)
}
got, err := Resolve(shelf(minimalRouteProxy(), minio), []string{"route-proxy", "minio"}, workstation(), World{})
if err != nil {
t.Fatal(err)
}
out, err := got.Declaration(Rendering{})
if err != nil {
t.Fatal(err)
}
var given []Contribution
for _, r := range out {
if r["path"] != "/var/lib/route-proxy/routes/mesh.json" {
continue
}
var parsed struct {
Given []Contribution `json:"given"`
}
if err := json.Unmarshal([]byte(r["content"].(string)), &parsed); err != nil {
t.Fatal(err)
}
given = parsed.Given
}
if len(given) != 2 {
t.Fatalf("two named routes from one module are two contributions: %+v", given)
}
byPort := map[float64]string{}
for _, g := range given {
if g.From != "minio" {
t.Fatalf("both contributions are minio's: %+v", g)
}
port, _ := g.Values["port"].(float64)
label, _ := g.Values["label"].(string)
byPort[port] = label
}
if byPort[9000] != "files-api" || byPort[9001] != "files" {
t.Fatalf("the two routes did not both survive: %+v", given)
}
}
// A module with the ordinary, single-contribution shape resolves exactly as it did before —
// ContributesMany being empty must change nothing about it.
func TestASingleRouteStillResolvesTheOrdinaryWay(t *testing.T) {
got, err := Resolve(shelf(proxy(), published("board", "board", 8080)), []string{"board"}, workstation(), World{})
if err != nil {
t.Fatal(err)
}
given := received(t, mustDeclare(t, got))
if len(given) != 1 || given[0].From != "board" {
t.Fatalf("the plain shape regressed: %+v", given)
}
}
// ContributionsFrom is what mints the ONE pair credential a requiring module is granted
// (cmd/mesh-controller/plan.go's grantsFor) — a separate path from Declaration()'s raw file, and
// the one the two-routes test above never exercised. Where a module contributes several times,
// there is no single "the" value: settling to whichever sorts first would both misrepresent the
// grant and collide with that same contribution's own entry from contributions(), which is
// exactly the duplicate a live plan against minio surfaced (files-api appearing once with a
// credential, once without, while files got neither).
func TestContributionsFromHasNoSingleValueWhenAModuleContributesSeveralTimes(t *testing.T) {
minio, err := ParseManifest([]byte(twoRoutes))
if err != nil {
t.Fatal(err)
}
got, err := Resolve(shelf(minimalRouteProxy(), minio), []string{"route-proxy", "minio"}, workstation(), World{})
if err != nil {
t.Fatal(err)
}
values, asks, err := got.ContributionsFrom("route", "minio", nil)
if err != nil {
t.Fatal(err)
}
if !asks {
t.Fatal("minio still requires route, so it still asks")
}
if len(values) != 0 {
t.Fatalf("no single value represents two contributions, got %+v", values)
}
}
// The ordinary, single-contribution case is unchanged: exactly one match still settles to it.
func TestContributionsFromReturnsTheOneValueForAnOrdinaryContribution(t *testing.T) {
got, err := Resolve(shelf(proxy(), published("board", "board", 8080)), []string{"board"}, workstation(), World{})
if err != nil {
t.Fatal(err)
}
values, asks, err := got.ContributionsFrom("reverse-proxy", "board", nil)
if err != nil {
t.Fatal(err)
}
if !asks || values["host"] != "board" {
t.Fatalf("the ordinary single contribution should still settle to its own value: %+v", values)
}
}
+30 -7
View File
@@ -40,6 +40,20 @@ func Port(name string) (string, error) {
if raw == "" {
return "", nil
}
if unfilled.MatchString(raw) {
// **A placeholder the mesh never filled, and this is the one place it must not be a
// fault.** The control plane composes its own declaration, so a manifest naming
// `${seat:…}` reaches a control plane one build older than the manifest — one that does
// not know the placeholder and passes it through as the value. Refusing it here would
// leave every command and the daemon unable to open a store: headless, on the machine
// that cannot be repaired through the mesh (novox/hq 04-ISSUES/102). So it is what an
// empty answer is — nothing said, the sealed value stands — and it is said aloud, because
// a manifest ahead of its binary is worth a line on stderr and not worth an outage.
fmt.Fprintf(os.Stderr, "%s is %q, a placeholder nothing filled in — ignored; the port in "+
"%s stands. The control plane composing this declaration is older than the manifest "+
"naming it: rebuild and push it\n", PortVar(name), raw, name)
return "", nil
}
n, err := strconv.Atoi(raw)
if err != nil || n < 1 || n > 65535 {
return "", fmt.Errorf("%s is %q, which is not a port", PortVar(name), raw)
@@ -70,6 +84,10 @@ func Placed(name string) (string, error) {
// keywordPort is `port=…` in a `key=value` connection string.
var keywordPort = regexp.MustCompile(`(^|\s)port=\S*`)
// unfilled is a value that is still a placeholder — `${…}` — rather than something a person or the
// mesh wrote as a port.
var unfilled = regexp.MustCompile(`^\$\{[^}]*\}$`)
// WithPort is the value with its port replaced by `port`.
//
// Three shapes, because the control plane's settings come in three: a URL
@@ -84,14 +102,19 @@ func WithPort(value, port string) (string, error) {
return "", fmt.Errorf("the value is empty")
}
if scheme, rest, isURL := strings.Cut(value, "://"); isURL {
end := strings.IndexAny(rest, "/?#")
authority, tail := rest, ""
if end >= 0 {
authority, tail = rest[:end], rest[end:]
// **The password may hold any of `/ ? # @`, so the host begins after the LAST `@`**, and
// the path only after that. Cutting at the first `/` would take a password's slash for the
// path's and put the new port on the user name — a connection string that dials the wrong
// host without a word. Genesis makes passwords that cannot do this; an accepted one can.
// The connection strings this reads — a store's, a broker's, a management API's — carry
// no `@` after their userinfo, which is what makes the last one the boundary.
userinfo, hostAndTail := "", rest
if at := strings.LastIndex(rest, "@"); at >= 0 {
userinfo, hostAndTail = rest[:at+1], rest[at+1:]
}
userinfo := ""
if at := strings.LastIndex(authority, "@"); at >= 0 {
userinfo, authority = authority[:at+1], authority[at+1:]
authority, tail := hostAndTail, ""
if end := strings.IndexAny(hostAndTail, "/?#"); end >= 0 {
authority, tail = hostAndTail[:end], hostAndTail[end:]
}
host, err := hostWithPort(authority, port)
if err != nil {
+12
View File
@@ -16,6 +16,7 @@ func TestAPortTwinMovesTheValuesPort(t *testing.T) {
"amqp://control:p%40ss@127.0.0.1:5672/": "amqp://control:p%40ss@127.0.0.1:6852/",
"amqp://control:p@ss:with@127.0.0.1:5672/": "amqp://control:p@ss:with@127.0.0.1:6852/",
"http://guest:guest@127.0.0.1:15672": "http://guest:guest@127.0.0.1:6852",
"postgres://mesh:a/b?c#d@e@127.0.0.1:5432/inventory": "postgres://mesh:a/b?c#d@e@127.0.0.1:6852/inventory",
"http://[::1]:15672/api": "http://[::1]:6852/api",
"broker.example:5671": "broker.example:6852",
"broker.example": "broker.example:6852",
@@ -64,3 +65,14 @@ func TestAPortTwinThatIsNotAPortIsRefusedWithoutQuotingTheValue(t *testing.T) {
t.Fatal("a port with no value to put it on was accepted")
}
}
// A placeholder nothing filled is nothing said, not a fault: the control plane composing the
// declaration may be one build behind the manifest, and refusing would leave it headless.
func TestAnUnfilledPlaceholderIsNothingSaid(t *testing.T) {
t.Setenv("MESH_R", "postgres://m:p@127.0.0.1:5432/inventory")
t.Setenv("MESH_R_PORT", "${seat:mesh-store:5432}")
got, err := Placed("MESH_R")
if err != nil || got != "postgres://m:p@127.0.0.1:5432/inventory" {
t.Fatalf("an unfilled placeholder was not ignored: %q, %v", got, err)
}
}
+68
View File
@@ -0,0 +1,68 @@
package identity
import (
"context"
"fmt"
"os"
"strings"
"testing"
"time"
"github.com/jackc/pgx/v5"
"github.com/novox/mesh-controller/internal/store"
)
// ForTest is a fresh, migrated identity store in a database of its own, dropped when the test
// ends. Exported for the same reason inventory.ForTest is: the check that a node's signed word
// is verified against the key the mesh recorded lives beside the link, and a second copy of this
// would be a second thing to keep true. It takes a *testing.T, so nothing that is not a test can
// call it.
func ForTest(t *testing.T) *Identity {
t.Helper()
admin := os.Getenv("MESH_TEST_POSTGRES")
if admin == "" {
t.Skip("no MESH_TEST_POSTGRES; run `make check` to raise one")
}
name := fmt.Sprintf("ident_%d_%s", time.Now().UnixNano()%1_000_000,
strings.ToLower(strings.NewReplacer("/", "", "-", "").Replace(t.Name())))
if len(name) > 60 {
name = name[:60]
}
conn, err := pgx.Connect(t.Context(), admin)
if err != nil {
t.Fatalf("cannot reach the test PostgreSQL: %v", err)
}
if _, err := conn.Exec(t.Context(), "create database "+name); err != nil {
t.Fatalf("cannot create %s: %v", name, err)
}
conn.Close(t.Context())
cut := strings.LastIndex(admin, "/")
t.Setenv(store.Variable(Name), admin[:cut]+"/"+name+"?sslmode=disable")
ident, err := Open(t.Context())
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() {
ident.Close()
c, err := pgx.Connect(context.Background(), admin)
if err != nil {
return
}
defer c.Close(context.Background())
_, _ = c.Exec(context.Background(), "drop database if exists "+name+" with (force)")
})
if err := ident.Ready(t.Context(), 20*time.Second); err != nil {
t.Fatal(err)
}
migrations, err := Migrations()
if err != nil {
t.Fatal(err)
}
if _, err := ident.store.Migrate(t.Context(), migrations); err != nil {
t.Fatal(err)
}
return ident
}
+65 -12
View File
@@ -16,25 +16,75 @@ import (
// node's peer list to chase it, and an address that moves is the thing declaring the hub was
// meant to stop.
//
// Allocated in order from the range, taking the lowest free one. Not random: a person reading a
// peer list should be able to guess which node an address belongs to, and reuse of a released
// address is a smaller problem than a list nobody can hold in their head.
// **The adopted tunnel's addresses come first** (novox/hq ADR 0105). The hub that took over a
// tunnel is at the tunnel's own address. A node enrolling with a key the tunnel already routed
// to keeps the address the tunnel had for it — nothing a peer knows changes. And an address the
// tunnel holds for a peer that has not enrolled is never handed to anyone else: that peer is
// still reaching the hub at it.
//
// The rest is allocated in order from the range, taking the lowest free one. Not random: a person
// reading a peer list should be able to guess which node an address belongs to, and reuse of a
// released address is a smaller problem than a list nobody can hold in their head.
func (i *Inventory) AssignAddress(ctx context.Context, node, cidr string) (string, error) {
prefix, err := netip.ParsePrefix(cidr)
if err != nil {
return "", fmt.Errorf("%q is not a network the mesh can allocate from: %w", cidr, err)
}
var existing *string
var existing, key *string
var name string
var hub bool
if err := i.store.Pool().QueryRow(ctx,
`select host(overlay_address) from node where id = $1`, node).Scan(&existing); err != nil {
`select name, host(overlay_address), overlay_key, is_hub from node where id = $1`, node).
Scan(&name, &existing, &key, &hub); err != nil {
return "", err
}
if existing != nil && *existing != "" {
return *existing, nil
}
tunnel, hubName, adopted, err := i.AdoptedTunnel(ctx)
if err != nil {
return "", err
}
if adopted && hub && hubName == name {
address, err := netip.ParsePrefix(tunnel.Address)
if err != nil {
return "", fmt.Errorf("the adopted tunnel's address %q: %w", tunnel.Address, err)
}
return i.place(ctx, node, address.Addr().String())
}
carried, err := i.CarriedPeers(ctx)
if err != nil {
return "", err
}
taken := map[string]bool{}
if adopted {
// The tunnel's own address is the hub's whether or not the hub has been placed yet.
if address, err := netip.ParsePrefix(tunnel.Address); err == nil {
taken[address.Addr().String()] = true
}
}
for _, p := range carried {
if key != nil && p.PublicKey == *key {
// The tunnel already routes to this key: the node keeps that address, and the peer
// notices nothing when its machine enrols.
//
// **Unless the operator named it something else** (novox/hq issue 112). The name is
// what the mesh has been answering for this address in the meantime; a machine
// enrolling under a different one would silently split the two — the name resolving
// here, the node known as that — so it is refused where the operator can read it.
if p.Named != "" && p.Named != name {
return "", fmt.Errorf(
"the carried peer at %s was named %q, and %q is enrolling under its key — "+
"enrol it as %q, or rename the peer first (`overlay name`)",
p.Address, p.Named, name, p.Named)
}
return i.place(ctx, node, p.Address)
}
taken[p.Address] = true
}
rows, err := i.store.Pool().Query(ctx,
`select host(overlay_address) from node where overlay_address is not null`)
if err != nil {
@@ -58,12 +108,7 @@ func (i *Inventory) AssignAddress(ctx context.Context, node, cidr string) (strin
candidate := prefix.Masked().Addr().Next()
for prefix.Contains(candidate) {
if !taken[candidate.String()] {
if _, err := i.store.Pool().Exec(ctx,
`update node set overlay_address = $2::inet where id = $1`,
node, candidate.String()); err != nil {
return "", err
}
return candidate.String(), nil
return i.place(ctx, node, candidate.String())
}
candidate = candidate.Next()
}
@@ -72,5 +117,13 @@ func (i *Inventory) AssignAddress(ctx context.Context, node, cidr string) (strin
// halfway through assigning one node.
return "", fmt.Errorf(
"every address in %s is taken, so %s cannot be given one. The mesh has outgrown its "+
"range and renumbering it is a deliberate act", cidr, node)
"range and renumbering it is a deliberate act", cidr, name)
}
func (i *Inventory) place(ctx context.Context, node, address string) (string, error) {
if _, err := i.store.Pool().Exec(ctx,
`update node set overlay_address = $2::inet where id = $1`, node, address); err != nil {
return "", err
}
return address, nil
}
+2 -2
View File
@@ -65,7 +65,7 @@ func TestTakingIsRefusedOnAConvergedNodeAndForAnUnassignedModule(t *testing.T) {
if _, err := inv.AddNode(t.Context(), "converged"); err != nil {
t.Fatal(err)
}
if err := inv.Assign(t.Context(), "converged", "hello-web"); err != nil {
if _, err := inv.Assign(t.Context(), "converged", "hello-web"); err != nil {
t.Fatal(err)
}
if err := inv.Take(t.Context(), "converged", "hello-web"); !errors.Is(err, ErrNotAdopted) {
@@ -91,7 +91,7 @@ func TestATakenModuleOutlivesItsAssignmentAndReturningToAdopted(t *testing.T) {
t.Fatal(err)
}
for _, m := range []string{"hello-web", "postgres"} {
if err := inv.Assign(t.Context(), "anchor", m); err != nil {
if _, err := inv.Assign(t.Context(), "anchor", m); err != nil {
t.Fatal(err)
}
}
+36 -18
View File
@@ -24,7 +24,12 @@ var ErrStillAssigned = errors.New("that module is still assigned to nodes")
// Source is where a module comes from and what has been built from it.
type Source struct {
// Repository is a URL, cloned exactly as given, unless Seat is set — then it is the
// repository's path on that seat's holder, and never an address (novox/hq ADR 0111).
Repository string
// Seat is the seat the repository lives on: `git` for the mesh's own forge, empty for a
// repository anywhere else.
Seat string
// Path is the module's directory inside that repository (novox/hq ADR 0069). Empty is the
// repository's root, which is a real answer rather than a missing one.
Path string
@@ -62,8 +67,8 @@ func (i *Inventory) RegisterModule(ctx context.Context, m catalogue.Manifest, fr
// record of where the module normally comes from — which is the only thing that would say,
// afterwards, that the machine is running something nobody can rebuild.
_, err = i.store.Pool().Exec(ctx,
`insert into module (name, manifest, version, source, source_path, ref, built_from, source_head)
values ($1, $2, nullif($3,''), nullif($4,''), $7, nullif($5,''), nullif($6,''), nullif($6,''))
`insert into module (name, manifest, version, source, source_path, source_seat, ref, built_from, source_head)
values ($1, $2, nullif($3,''), nullif($4,''), $7, $8, nullif($5,''), nullif($6,''), nullif($6,''))
on conflict (name) do update set
manifest = excluded.manifest,
version = excluded.version,
@@ -71,10 +76,12 @@ func (i *Inventory) RegisterModule(ctx context.Context, m catalogue.Manifest, fr
source = coalesce(excluded.source, module.source),
source_path = case when excluded.source is null then module.source_path
else excluded.source_path end,
source_seat = case when excluded.source is null then module.source_seat
else excluded.source_seat end,
ref = coalesce(excluded.ref, module.ref),
built_from = coalesce(excluded.built_from, module.built_from),
source_head = coalesce(excluded.built_from, module.source_head)`,
m.Module, raw, m.Version, from.Repository, from.Ref, from.BuiltFrom, from.Path)
m.Module, raw, m.Version, from.Repository, from.Ref, from.BuiltFrom, from.Path, from.Seat)
return err
}
@@ -99,10 +106,10 @@ func (i *Inventory) SourceMoved(ctx context.Context, module, head string) error
func (i *Inventory) SourceOf(ctx context.Context, module string) (Source, error) {
var s Source
var repo, ref, built, head *string
var path string
var path, seat string
err := i.store.Pool().QueryRow(ctx,
`select source, source_path, ref, built_from, source_head from module where name = $1`,
module).Scan(&repo, &path, &ref, &built, &head)
`select source, source_path, source_seat, ref, built_from, source_head from module where name = $1`,
module).Scan(&repo, &path, &seat, &ref, &built, &head)
if errors.Is(err, pgx.ErrNoRows) {
return Source{}, fmt.Errorf("%w: %s", ErrNoSuchModule, module)
}
@@ -117,9 +124,10 @@ func (i *Inventory) SourceOf(ctx context.Context, module string) (Source, error)
*pair.to = *pair.from
}
}
// Not in the loop above: the path is never null, because "the repository's root" is an answer
// rather than an absence.
// Not in the loop above: the path and the seat are never null, because "the repository's root"
// and "not on a seat" are answers rather than absences.
s.Path = path
s.Seat = seat
return s, nil
}
@@ -422,27 +430,36 @@ func (i *Inventory) discard(ctx context.Context, name string) error {
return nil
}
// Assign puts a module on a node.
// Assign puts a module on a node, and says whether that is new.
//
// Records the intention and checks nothing. Whether the set of assignments can actually become a
// declaration is resolution's question, asked over the whole set at once — and asking it here,
// one module at a time, would let an assignment look accepted and then refuse when a second
// arrives.
func (i *Inventory) Assign(ctx context.Context, nodeName, module string) error {
//
// **One assignment of a module per node is the rule, not a race lost** (novox/hq ADR 0115). The
// module's name is the assignment's identity — its database user, its broker account, its
// containers and its placed directory are all named by it — so the schema's (node, module) key
// is the decision, and a repeat is absorbed rather than refused. Absorbed audibly: the caller is
// told nothing changed, because "is assigned" printed for a no-op reads as an action.
func (i *Inventory) Assign(ctx context.Context, nodeName, module string) (bool, error) {
node, err := i.NodeByName(ctx, nodeName)
if err != nil {
return err
return false, err
}
if err := i.runsSomewhere(ctx, module); err != nil {
return err
return false, err
}
_, err = i.store.Pool().Exec(ctx,
tag, err := i.store.Pool().Exec(ctx,
`insert into assignment (node, module) values ($1, $2) on conflict do nothing`,
node.ID, module)
if err != nil && strings.Contains(err.Error(), "assignment_module_fkey") {
return fmt.Errorf("%w: %s", ErrNoSuchModule, module)
return false, fmt.Errorf("%w: %s", ErrNoSuchModule, module)
}
return err
if err != nil {
return false, err
}
return tag.RowsAffected() > 0, nil
}
// Unassign takes a module off a node.
@@ -917,13 +934,14 @@ type Entry struct {
func (i *Inventory) Catalogued(ctx context.Context) ([]Entry, error) {
rows, err := i.store.Pool().Query(ctx,
`select m.name, m.manifest,
coalesce(m.source, ''), m.source_path, coalesce(m.ref, ''),
coalesce(m.source, ''), m.source_path, m.source_seat, coalesce(m.ref, ''),
coalesce(m.built_from, ''), coalesce(m.source_head, ''),
coalesce(array_agg(n.name order by n.name) filter (where n.name is not null), '{}')
from module m
left join assignment a on a.module = m.name
left join node n on n.id = a.node
group by m.name, m.manifest, m.source, m.source_path, m.ref, m.built_from, m.source_head
group by m.name, m.manifest, m.source, m.source_path, m.source_seat, m.ref, m.built_from,
m.source_head
order by m.name`)
if err != nil {
return nil, err
@@ -936,7 +954,7 @@ func (i *Inventory) Catalogued(ctx context.Context) ([]Entry, error) {
var name string
var source Source
var on []string
if err := rows.Scan(&name, &raw, &source.Repository, &source.Path, &source.Ref,
if err := rows.Scan(&name, &raw, &source.Repository, &source.Path, &source.Seat, &source.Ref,
&source.BuiltFrom, &source.Head, &on); err != nil {
return nil, err
}
+90 -9
View File
@@ -77,7 +77,7 @@ func TestAModuleAMachineIsRunningCannotBeForgotten(t *testing.T) {
if err := inv.RegisterModule(t.Context(), manifest("thing", nil, nil), Source{}); err != nil {
t.Fatal(err)
}
if err := inv.Assign(t.Context(), "laptop", "thing"); err != nil {
if _, err := inv.Assign(t.Context(), "laptop", "thing"); err != nil {
t.Fatal(err)
}
@@ -104,7 +104,7 @@ func TestRemovingANodeTakesItsAssignments(t *testing.T) {
if err := inv.RegisterModule(t.Context(), manifest("thing", nil, nil), Source{}); err != nil {
t.Fatal(err)
}
if err := inv.Assign(t.Context(), "laptop", "thing"); err != nil {
if _, err := inv.Assign(t.Context(), "laptop", "thing"); err != nil {
t.Fatal(err)
}
if _, err := inv.store.Pool().Exec(t.Context(), `delete from node where id = $1`, node.ID); err != nil {
@@ -136,14 +136,16 @@ func TestAssigningAModuleTheMeshDoesNotKnowIsRefused(t *testing.T) {
if _, err := inv.AddNode(t.Context(), "laptop"); err != nil {
t.Fatal(err)
}
err := inv.Assign(t.Context(), "laptop", "not-a-module")
_, err := inv.Assign(t.Context(), "laptop", "not-a-module")
if !errors.Is(err, ErrNoSuchModule) {
t.Fatalf("assigning an unknown module gave %v", err)
}
}
func TestAssigningTwiceIsNotAnError(t *testing.T) {
// It is a statement of what should be true, and it already is.
func TestAssigningTwiceIsNotAnErrorAndSaysSo(t *testing.T) {
// It is a statement of what should be true, and it already is — one assignment of a module
// per node is the rule (novox/hq ADR 0115), so a repeat is absorbed, audibly: the caller is
// told nothing was new.
inv := fresh(t)
if _, err := inv.AddNode(t.Context(), "laptop"); err != nil {
t.Fatal(err)
@@ -151,10 +153,18 @@ func TestAssigningTwiceIsNotAnError(t *testing.T) {
if err := inv.RegisterModule(t.Context(), manifest("thing", nil, nil), Source{}); err != nil {
t.Fatal(err)
}
for i := 0; i < 3; i++ {
if err := inv.Assign(t.Context(), "laptop", "thing"); err != nil {
first, err := inv.Assign(t.Context(), "laptop", "thing")
if err != nil || !first {
t.Fatalf("the first assignment is the new one; got fresh=%v err=%v", first, err)
}
for i := 0; i < 2; i++ {
again, err := inv.Assign(t.Context(), "laptop", "thing")
if err != nil {
t.Fatalf("assigning again failed: %v", err)
}
if again {
t.Fatal("a repeat must say nothing was new")
}
}
assigned, err := inv.Assigned(t.Context(), "laptop")
if err != nil {
@@ -277,7 +287,7 @@ func TestBeingBehindNamesTheMachinesRunningTheOldOne(t *testing.T) {
t.Fatal(err)
}
for _, n := range []string{"laptop", "workstation"} {
if err := inv.Assign(t.Context(), n, "thing"); err != nil {
if _, err := inv.Assign(t.Context(), n, "thing"); err != nil {
t.Fatal(err)
}
}
@@ -451,7 +461,7 @@ func TestTheCatalogueSaysWhereEachModuleCameFromAndWhoRunsIt(t *testing.T) {
t.Fatal(err)
}
for _, n := range []string{"workstation", "laptop"} {
if err := inv.Assign(ctx, n, "shell"); err != nil {
if _, err := inv.Assign(ctx, n, "shell"); err != nil {
t.Fatal(err)
}
}
@@ -604,3 +614,74 @@ func TestNeverReportedAndReportedNothingAreDifferentProfiles(t *testing.T) {
t.Fatal("a machine that reported nothing looks like one that never reported")
}
}
// Defends novox/hq ADR 0111: a source on a seat is recorded as a path and the seat, never an
// address, and a module recorded before the column existed keeps meaning a URL.
func TestASourceOnASeatIsRecordedAsItsPathAndTheSeat(t *testing.T) {
inv := fresh(t)
ctx := t.Context()
if err := inv.RegisterModule(ctx, manifest("gitea-built", nil, nil), Source{
Repository: "novox/mesh-catalog", Seat: "git", Path: "modules/gitea", Ref: "main",
BuiltFrom: "aaaa1111",
}); err != nil {
t.Fatal(err)
}
if err := inv.RegisterModule(ctx, manifest("external", nil, nil), Source{
Repository: "https://example.invalid/someone/something.git", BuiltFrom: "bbbb2222",
}); err != nil {
t.Fatal(err)
}
own, err := inv.SourceOf(ctx, "gitea-built")
if err != nil {
t.Fatal(err)
}
if own.Seat != "git" || own.Repository != "novox/mesh-catalog" || own.Path != "modules/gitea" {
t.Fatalf("recorded as %+v", own)
}
if strings.Contains(own.Repository, "://") {
t.Fatalf("an address was recorded for a source on a seat: %s", own.Repository)
}
elsewhere, err := inv.SourceOf(ctx, "external")
if err != nil {
t.Fatal(err)
}
if elsewhere.Seat != "" {
t.Fatalf("an external repository was put on a seat: %+v", elsewhere)
}
// And the list every rebuild walks carries the seat, or `build --behind` would clone the path
// as though it were a URL.
all, err := inv.Catalogued(ctx)
if err != nil {
t.Fatal(err)
}
for _, e := range all {
if e.Manifest.Module == "gitea-built" && e.Source.Seat != "git" {
t.Fatalf("the rebuild list lost the seat: %+v", e.Source)
}
}
}
func TestRegisteringWithoutProvenanceKeepsTheSeat(t *testing.T) {
// A manifest handed over by hand keeps the record of where the module normally comes from —
// the seat included, or the next rebuild would treat a path as a URL.
inv := fresh(t)
ctx := t.Context()
if err := inv.RegisterModule(ctx, manifest("thing", nil, nil), Source{
Repository: "novox/thing", Seat: "git", BuiltFrom: "aaaa1111",
}); err != nil {
t.Fatal(err)
}
if err := inv.RegisterModule(ctx, manifest("thing", []string{"a-thing"}, nil), Source{}); err != nil {
t.Fatal(err)
}
got, err := inv.SourceOf(ctx, "thing")
if err != nil {
t.Fatal(err)
}
if got.Seat != "git" || got.Repository != "novox/thing" {
t.Fatalf("a hand-registered manifest erased where the module comes from: %+v", got)
}
}
+1 -1
View File
@@ -212,7 +212,7 @@ func TestAModuleStillAssignedRefusesBeforeAnythingAboutWhatItHolds(t *testing.T)
if err := inv.SetSettings(ctx, "anchor", "step-ca", map[string]any{"a": 1}); err != nil {
t.Fatal(err)
}
if err := inv.Assign(ctx, "anchor", "step-ca"); err != nil {
if _, err := inv.Assign(ctx, "anchor", "step-ca"); err != nil {
t.Fatal(err)
}
for _, forget := range []func() error{
@@ -0,0 +1,27 @@
-- The tunnel a node found on its machine, and the peers it carried (novox/hq ADR 0105).
--
-- On an adopted node that is the hub, the private network takes over the tunnel it finds: its
-- key, its port, its address and range, and every peer. The node presents what it found when it
-- enrols -- the same moment it presents its keys, because the found tunnel's key IS its key on the
-- private network from then on -- and the mesh composes every address from it.
-- What the node presented: interface, unit and configuration path, port, address and range, and
-- the tunnel's public key. The private key never travels; the node keeps it as its own overlay
-- key. Null on a node that found no tunnel, which is every converged one.
alter table node add column tunnel jsonb;
-- The node's last account of carrying it: the found interface down and disabled, the mesh's up
-- in its place. Null until the node says so.
alter table node add column tunnel_carried jsonb;
-- The peers the found tunnel had: a public key and the address the tunnel routed to it. Peers of
-- the tunnel, not nodes of the mesh, until they enrol -- a machine the mesh has no record of, whose
-- identity precedes its enrolment. One row per key, and one address per key on one tunnel.
create table tunnel_peer (
node uuid not null references node(id) on delete cascade,
public_key text not null,
address inet not null,
since timestamptz not null default now(),
primary key (node, public_key),
unique (node, address)
);
@@ -0,0 +1,12 @@
-- Which seat a module's source lives on, when it lives on one.
--
-- novox/hq ADR 0111. A module's repository was recorded exactly as a person typed it, so a
-- self-hosted forge's scheme, host and port were written into every module built from it — and
-- moving the forge made every one of those records stale at once, noticed only when a rebuild
-- failed to clone. A source on the `git` seat is now recorded as its path on the seat's holder, and
-- the clone URL is composed from wherever the holder runs at the moment of building.
--
-- Empty rather than null, and defaulted, because "not on a seat" is a real answer: the repository
-- column is then a URL, cloned exactly as given, which is what every module recorded before this
-- already is. So every existing row keeps exactly the meaning it had.
alter table module add column source_seat text not null default '';
@@ -0,0 +1,10 @@
-- A carried peer may be named before it enrols (novox/hq issue 112).
--
-- The tunnel the hub took over routes to machines the predecessor knows by name and the mesh
-- knows only by address. A name the predecessor answers for must keep resolving until the
-- machine behind it is a node — so the operator may state which machine a carried address is,
-- and everything derived from "the machines the mesh knows" (a container's hosts, the hosts
-- fact, the resolver) answers for it in the meantime. The mesh records the statement as the
-- operator's, unverified: enrolment is what verifies it, and enrolling under a different name
-- than the one stated is refused rather than silently renamed.
alter table tunnel_peer add column named text;
+92
View File
@@ -0,0 +1,92 @@
package inventory
// A carried peer is nameable (novox/hq issue 112): the operator states which machine a carried
// address is, the mesh answers for the name until the machine enrols, and enrolment verifies the
// statement rather than silently renaming it.
import (
"strings"
"testing"
)
func TestACarriedPeerIsNamedAndTheRegistrySaysSo(t *testing.T) {
inv := fresh(t)
anAdoptedHub(t, inv)
if err := inv.NamePeer(t.Context(), "192.0.2.2", "home-server"); err != nil {
t.Fatal(err)
}
carried, err := inv.CarriedPeers(t.Context())
if err != nil {
t.Fatal(err)
}
byKey := map[string]CarriedPeer{}
for _, c := range carried {
byKey[c.PublicKey] = c
}
if byKey[peerTwo].Named != "home-server" || byKey[peerThree].Named != "" {
t.Fatalf("the statement was not recorded where it was made: %+v", carried)
}
}
func TestNamingRefusesWhatWouldCollide(t *testing.T) {
inv := fresh(t)
anAdoptedHub(t, inv)
if err := inv.NamePeer(t.Context(), "192.0.2.9", "ghost"); err == nil ||
!strings.Contains(err.Error(), "no carried peer") {
t.Fatalf("naming an address nothing carries must refuse; got %v", err)
}
if err := inv.NamePeer(t.Context(), "192.0.2.2", "anchor"); err == nil ||
!strings.Contains(err.Error(), "node of this mesh") {
t.Fatalf("naming a peer after a node must refuse; got %v", err)
}
if err := inv.NamePeer(t.Context(), "192.0.2.2", "home-server"); err != nil {
t.Fatal(err)
}
if err := inv.NamePeer(t.Context(), "192.0.2.3", "home-server"); err == nil ||
!strings.Contains(err.Error(), "already named") {
t.Fatalf("one machine per name; got %v", err)
}
}
func TestEnrolmentUnderANamedKeyMustUseTheName(t *testing.T) {
inv := fresh(t)
anAdoptedHub(t, inv)
if err := inv.NamePeer(t.Context(), "192.0.2.3", "home-server"); err != nil {
t.Fatal(err)
}
// The wrong name is refused where the operator can read it…
imposter, err := inv.AddNode(t.Context(), "some-other-name")
if err != nil {
t.Fatal(err)
}
if err := inv.RecordOverlayKey(t.Context(), imposter.ID, peerThree); err != nil {
t.Fatal(err)
}
if _, err := inv.AssignAddress(t.Context(), imposter.ID, "192.0.2.0/24"); err == nil ||
!strings.Contains(err.Error(), `named "home-server"`) {
t.Fatalf("enrolling a named peer under another name must refuse; got %v", err)
}
// …and the stated name enrols cleanly, keeping the carried address.
named, err := inv.AddNode(t.Context(), "home-server")
if err != nil {
t.Fatal(err)
}
if err := inv.RecordOverlayKey(t.Context(), named.ID, peerThree); err != nil {
t.Fatal(err)
}
address, err := inv.AssignAddress(t.Context(), named.ID, "192.0.2.0/24")
if err != nil {
t.Fatal(err)
}
if address != "192.0.2.3" {
t.Fatalf("the named peer keeps its carried address; got %s", address)
}
if err := inv.NamePeer(t.Context(), "192.0.2.3", "renamed"); err == nil ||
!strings.Contains(err.Error(), "enrolled as") {
t.Fatalf("an enrolled peer's name is the node's; got %v", err)
}
}
+2 -2
View File
@@ -221,7 +221,7 @@ func TestWhatAMachineNoLongerHoldsIsAvailableAgain(t *testing.T) {
func TestUnassigningReleasesTheModulesPorts(t *testing.T) {
inv, node := aNodeWithModules(t, "mailu", "other-mail")
ctx := t.Context()
if err := inv.Assign(ctx, node, "mailu"); err != nil {
if _, err := inv.Assign(ctx, node, "mailu"); err != nil {
t.Fatal(err)
}
if _, err := inv.PortFor(ctx, node, "mailu", 25, true); err != nil {
@@ -230,7 +230,7 @@ func TestUnassigningReleasesTheModulesPorts(t *testing.T) {
if err := inv.Unassign(ctx, node, "mailu"); err != nil {
t.Fatal(err)
}
if err := inv.Assign(ctx, node, "other-mail"); err != nil {
if _, err := inv.Assign(ctx, node, "other-mail"); err != nil {
t.Fatal(err)
}
if _, err := inv.PortFor(ctx, node, "other-mail", 25, true); err != nil {
+1 -1
View File
@@ -350,7 +350,7 @@ func TestACredentialGoesWhenTheConsumerStopsAskingForIt(t *testing.T) {
}, Source{}); err != nil {
t.Fatal(err)
}
if err := inv.Assign(ctx, "consumer", "meshboard"); err != nil {
if _, err := inv.Assign(ctx, "consumer", "meshboard"); err != nil {
t.Fatal(err)
}
if _, err := inv.SecretFor(ctx, "postgres-database", "consumer", "gitea", "provider", ""); err != nil {
+421
View File
@@ -0,0 +1,421 @@
package inventory
import (
"context"
"encoding/json"
"errors"
"fmt"
"net/netip"
"strings"
"time"
"github.com/jackc/pgx/v5"
)
// The tunnel a node found on its machine, and the peers it carried (novox/hq ADR 0105).
//
// On an adopted node that is the hub, the private network takes over the tunnel it finds: its
// private key, its port, its address and range, and every peer the found interface had. The node
// presents what it found when it enrols, in the same breath as its keys — the found tunnel's key is
// its key on the private network from then on — and the mesh composes every address from it: the
// hub's is the tunnel's, the range is the tunnel's, and a peer that enrols keeps the address the
// tunnel already had for its key.
// Tunnel is what a node found on its machine, as it presented it. No private key: the node keeps
// it as its own overlay key, sealed like any own secret, and the mesh records only the public half
// — which is then the node's overlay key too.
type Tunnel struct {
// Interface, Unit and Config are what the host takes over: the found interface, the unit
// that raised it, and its configuration file, which is kept like any held file.
Interface string `json:"interface"`
Unit string `json:"unit"`
Config string `json:"config"`
// Port is the port the found interface listened on — one the hosting provider already lets
// through, which is why it is worth taking.
Port int `json:"port"`
// MTU is the found interface's, when it set one; the mesh's interface takes it over so a
// tuned path does not silently regress to the default (novox/hq: a taken tunnel carries its MTU).
MTU int `json:"mtu,omitempty"`
// Address is the interface's own address with its prefix length, 192.0.2.1/24; Range is the
// network that prefix names, 192.0.2.0/24.
Address string `json:"address"`
Range string `json:"range"`
// PublicKey is the found interface's, which every peer knows the tunnel by.
PublicKey string `json:"public_key"`
// Peers are the found interface's peers: each a public key and the address the tunnel routed
// to it.
Peers []TunnelPeer `json:"peers,omitempty"`
// At is when the node presented it; zero on a tunnel not yet recorded.
At time.Time `json:"at,omitempty"`
}
// TunnelPeer is one peer of a found tunnel.
type TunnelPeer struct {
PublicKey string `json:"public_key"`
// Address is the one host address the tunnel routed to the peer, without a prefix.
Address string `json:"address"`
}
// Carried is what a node last said about carrying the tunnel it found: the found interface down
// and disabled, the mesh's up in its place with the found key.
type Carried struct {
Interface string `json:"interface"`
Port int `json:"port"`
Range string `json:"range"`
Peers int `json:"peers"`
// State is one of the CarriedStates: the found interface is still up and the mesh's is not
// (not taken), the found one is down and the mesh's up with its key (taken), or the found one
// is down and the mesh's is not up — the one state where the peers reach nothing. Note is
// what the host did about it, when it did something. Kept is where the found configuration's
// original was kept.
State string `json:"state"`
Note string `json:"note,omitempty"`
Kept string `json:"kept,omitempty"`
At time.Time `json:"at"`
}
// The states a carried tunnel's account can be in, as the host says them.
const (
CarriedNotTaken = "not-taken"
CarriedTaken = "taken"
CarriedDown = "down"
)
// CarriedPeer is one peer of the adopted tunnel as the mesh holds it: a peer of the tunnel, and
// — once a node enrols with that key — a node of the mesh as well.
type CarriedPeer struct {
PublicKey string
Address string
// EnrolledAs names the node that enrolled with this key, or is empty while none has.
EnrolledAs string
// Named is what the operator said this peer is, before it enrolled (novox/hq issue 112) —
// a statement the mesh records and cannot verify, which is why enrolment checks it.
Named string
}
// ErrNoTunnel is asking about a tunnel on a node that presented none.
var ErrNoTunnel = errors.New("that node presented no tunnel")
// RecordTunnel keeps what a node presented, replacing what was there: the question is the tunnel
// as the node found it now. The peers are replaced whole for the same reason.
func (i *Inventory) RecordTunnel(ctx context.Context, nodeID string, t Tunnel) error {
if strings.TrimSpace(t.Interface) == "" || strings.TrimSpace(t.PublicKey) == "" {
return errors.New("a found tunnel names its interface and its public key, and this names neither")
}
if _, err := netip.ParsePrefix(t.Range); err != nil {
return fmt.Errorf("the found tunnel's range %q is not a range: %w", t.Range, err)
}
address, err := netip.ParsePrefix(t.Address)
if err != nil {
return fmt.Errorf("the found tunnel's address %q is not an address with a prefix: %w", t.Address, err)
}
peers := make([]TunnelPeer, 0, len(t.Peers))
for _, p := range t.Peers {
host, single := peerHost(p.Address)
if !single {
// A peer routed a range rather than one address is a spoke's view of its hub — the
// predecessor gives a spoke the whole subnet through the hub — and a hub is not a peer
// the mesh carries. Skipped, not refused: a spoke enrols with what it found, and only
// the hub's peers are ever carried (novox/hq ADR 0105).
continue
}
if !address.Masked().Contains(host) {
return fmt.Errorf("the found tunnel's peer %s is routed at %s, outside the tunnel's %s",
shortKey(p.PublicKey), host, t.Range)
}
peers = append(peers, TunnelPeer{PublicKey: p.PublicKey, Address: host.String()})
}
t.Peers = nil
t.At = time.Now().UTC()
raw, err := json.Marshal(t)
if err != nil {
return err
}
tx, err := i.store.Pool().Begin(ctx)
if err != nil {
return err
}
defer func() { _ = tx.Rollback(context.WithoutCancel(ctx)) }()
if _, err := tx.Exec(ctx, `update node set tunnel = $2 where id = $1`, nodeID, raw); err != nil {
return err
}
if _, err := tx.Exec(ctx, `delete from tunnel_peer where node = $1`, nodeID); err != nil {
return err
}
for _, p := range peers {
if _, err := tx.Exec(ctx,
`insert into tunnel_peer (node, public_key, address) values ($1, $2, $3::inet)`,
nodeID, p.PublicKey, p.Address); err != nil {
return fmt.Errorf("recording the found tunnel's peer %s: %w", shortKey(p.PublicKey), err)
}
}
return tx.Commit(ctx)
}
// peerHost is the one host address a peer's allowed address names — a bare address, or a /32
// (or /128) — and false for anything wider or unreadable: a peer routed a whole range is not a
// machine with an address the mesh could give a node.
func peerHost(allowed string) (netip.Addr, bool) {
allowed = strings.TrimSpace(allowed)
if a, err := netip.ParseAddr(allowed); err == nil {
return a, true
}
p, err := netip.ParsePrefix(allowed)
if err != nil || !p.IsSingleIP() {
return netip.Addr{}, false
}
return p.Addr(), true
}
func shortKey(key string) string {
if len(key) > 8 {
return key[:8] + "…"
}
return key
}
// TunnelOf is the tunnel a node presented, with its peers, or ErrNoTunnel.
func (i *Inventory) TunnelOf(ctx context.Context, name string) (Tunnel, error) {
var raw []byte
var id string
err := i.store.Pool().QueryRow(ctx,
`select id, tunnel from node where name = $1`, name).Scan(&id, &raw)
if errors.Is(err, pgx.ErrNoRows) {
return Tunnel{}, fmt.Errorf("%w: %s", ErrNoSuchNode, name)
}
if err != nil {
return Tunnel{}, err
}
if len(raw) == 0 {
return Tunnel{}, fmt.Errorf("%w: %s", ErrNoTunnel, name)
}
var t Tunnel
if err := json.Unmarshal(raw, &t); err != nil {
return Tunnel{}, err
}
t.Peers, err = i.tunnelPeers(ctx, id)
return t, err
}
func (i *Inventory) tunnelPeers(ctx context.Context, nodeID string) ([]TunnelPeer, error) {
rows, err := i.store.Pool().Query(ctx,
`select public_key, host(address) from tunnel_peer where node = $1 order by address`, nodeID)
if err != nil {
return nil, err
}
defer rows.Close()
var out []TunnelPeer
for rows.Next() {
var p TunnelPeer
if err := rows.Scan(&p.PublicKey, &p.Address); err != nil {
return nil, err
}
out = append(out, p)
}
return out, rows.Err()
}
// AdoptedTunnel is the tunnel the mesh's private network runs over, if the hub adopted one: the
// hub's found tunnel, when the hub's overlay key is the tunnel's. Absent, the mesh runs on its own
// range — and a hub that found a tunnel but holds another key did not adopt it, which `overlay
// show` says.
//
// The condition on the key is the condition of the whole record: a hub raised with a key of its
// own would drop every peer's packets on the found port (novox/hq ADR 0105, option 2), so the
// tunnel is adopted only when the hub answers to the key its peers know. **Not a condition on the
// node's mode**: the range and the carried peers are facts of the mesh once the tunnel is taken,
// and converging the hub — which flips its mode — must not renumber the mesh or drop the peers
// still reaching it.
func (i *Inventory) AdoptedTunnel(ctx context.Context) (Tunnel, string, bool, error) {
var name string
var key *string
err := i.store.Pool().QueryRow(ctx,
`select name, overlay_key from node where is_hub and tunnel is not null`).
Scan(&name, &key)
if errors.Is(err, pgx.ErrNoRows) {
return Tunnel{}, "", false, nil
}
if err != nil {
return Tunnel{}, "", false, err
}
t, err := i.TunnelOf(ctx, name)
if err != nil {
return Tunnel{}, "", false, err
}
if key == nil || *key != t.PublicKey {
return t, name, false, nil
}
return t, name, true, nil
}
// CarriedPeers is every peer of the adopted tunnel, with the node that enrolled under its key
// where one has: peers of the tunnel, and nodes of the mesh once they enrol. Empty when the hub
// adopted no tunnel.
func (i *Inventory) CarriedPeers(ctx context.Context) ([]CarriedPeer, error) {
rows, err := i.store.Pool().Query(ctx,
`select p.public_key, host(p.address), coalesce(n.name, ''), coalesce(p.named, '')
from tunnel_peer p
join node hub on hub.id = p.node and hub.is_hub
and hub.tunnel is not null and hub.overlay_key = hub.tunnel->>'public_key'
left join node n on n.overlay_key = p.public_key
order by p.address`)
if err != nil {
return nil, err
}
defer rows.Close()
var out []CarriedPeer
for rows.Next() {
var p CarriedPeer
if err := rows.Scan(&p.PublicKey, &p.Address, &p.EnrolledAs, &p.Named); err != nil {
return nil, err
}
out = append(out, p)
}
return out, rows.Err()
}
// NamePeer records the operator's statement that a carried address is a particular machine
// (novox/hq issue 112). Refused when nothing carried has that address, when a node of the mesh
// already has the name — the statement would collide with something verified — and when another
// peer was already named it. Naming an enrolled peer is refused too: its name is the node's now.
func (i *Inventory) NamePeer(ctx context.Context, address, name string) error {
peers, err := i.CarriedPeers(ctx)
if err != nil {
return err
}
var at *CarriedPeer
for idx := range peers {
if peers[idx].Address == address {
at = &peers[idx]
continue
}
if peers[idx].Named == name {
return fmt.Errorf("the carried peer at %s is already named %q — one machine per name",
peers[idx].Address, name)
}
}
if at == nil {
return fmt.Errorf("no carried peer has the address %s — `overlay show` lists them", address)
}
if at.EnrolledAs != "" {
return fmt.Errorf("the peer at %s enrolled as %q — its name is the node's now", address, at.EnrolledAs)
}
if _, err := i.NodeByName(ctx, name); err == nil {
return fmt.Errorf("%q is a node of this mesh — a carried peer cannot be named after one", name)
}
tag, err := i.store.Pool().Exec(ctx,
`update tunnel_peer set named = $2 where host(address) = $1`, address, name)
if err != nil {
return err
}
if tag.RowsAffected() == 0 {
return fmt.Errorf("no carried peer has the address %s", address)
}
return nil
}
// FoundTunnel is a node's found tunnel with the node's mode, for composing: the takeover is
// declared to an adopted node only, since only there is a found unit kept to be stopped.
type FoundTunnel struct {
Tunnel
NodeAdopted bool
}
// Tunnels is every node's found tunnel by node name, for the ones whose overlay key is the
// tunnel's — the ones whose private network takes it over. A found tunnel under another key is
// left running beside the mesh's, and ADR 0100's rule that the ranges differ applies to it.
func (i *Inventory) Tunnels(ctx context.Context) (map[string]FoundTunnel, error) {
rows, err := i.store.Pool().Query(ctx,
`select name, tunnel, adopted from node
where tunnel is not null and overlay_key = tunnel->>'public_key'`)
if err != nil {
return nil, err
}
defer rows.Close()
out := map[string]FoundTunnel{}
for rows.Next() {
var name string
var raw []byte
var adopted bool
if err := rows.Scan(&name, &raw, &adopted); err != nil {
return nil, err
}
var t Tunnel
if err := json.Unmarshal(raw, &t); err != nil {
return nil, err
}
out[name] = FoundTunnel{Tunnel: t, NodeAdopted: adopted}
}
return out, rows.Err()
}
// ErrStaleRekey is a rekey that names a previous overlay key other than the one recorded: a
// replay of a rekey already done, or one made against a record that has since moved on.
var ErrStaleRekey = errors.New("the rekey names a previous overlay key that is not the node's current one")
// Rekey records that a node took a found tunnel's key as its overlay key after enrolling (novox/hq
// ADR 0105): the key and the tunnel are recorded as enrolment would have, and a hub is moved to the
// tunnel's address so nothing derived from it is stale. The caller has verified the node signed
// for this; what is checked here is that it follows the record — `previous` is the overlay key the
// node holds now — so the same message cannot be applied twice.
func (i *Inventory) Rekey(ctx context.Context, nodeID, previous, key string, t Tunnel) error {
if key != t.PublicKey {
return errors.New("a rekey takes a tunnel over with the tunnel's own key, and this names another")
}
var current *string
var hub bool
if err := i.store.Pool().QueryRow(ctx,
`select overlay_key, is_hub from node where id = $1`, nodeID).Scan(&current, &hub); err != nil {
return err
}
if (current == nil && previous != "") || (current != nil && *current != previous) {
return ErrStaleRekey
}
if err := i.RecordOverlayKey(ctx, nodeID, key); err != nil {
return err
}
if err := i.RecordTunnel(ctx, nodeID, t); err != nil {
return err
}
if hub {
address, err := netip.ParsePrefix(t.Address)
if err != nil {
return err
}
if _, err := i.place(ctx, nodeID, address.Addr().String()); err != nil {
return err
}
}
return nil
}
// RecordCarriedTunnel keeps what a node last said about carrying its found tunnel.
func (i *Inventory) RecordCarriedTunnel(ctx context.Context, nodeID string, c Carried) error {
c.At = time.Now().UTC()
raw, err := json.Marshal(c)
if err != nil {
return err
}
_, err = i.store.Pool().Exec(ctx, `update node set tunnel_carried = $2 where id = $1`, nodeID, raw)
return err
}
// CarriedTunnelOf is a node's last account of carrying its found tunnel, and whether it ever gave one.
func (i *Inventory) CarriedTunnelOf(ctx context.Context, name string) (Carried, bool, error) {
var raw []byte
err := i.store.Pool().QueryRow(ctx, `select tunnel_carried from node where name = $1`, name).Scan(&raw)
if errors.Is(err, pgx.ErrNoRows) {
return Carried{}, false, fmt.Errorf("%w: %s", ErrNoSuchNode, name)
}
if err != nil {
return Carried{}, false, err
}
if len(raw) == 0 {
return Carried{}, false, nil
}
var c Carried
if err := json.Unmarshal(raw, &c); err != nil {
return Carried{}, false, err
}
return c, true, nil
}
+279
View File
@@ -0,0 +1,279 @@
package inventory
import (
"errors"
"strings"
"testing"
)
// novox/hq ADR 0105: the mesh adopts the predecessor's tunnel in place. The controller reads the
// hub's address and range from the adopted tunnel, assigns an enrolling node the address its key
// already had, and refuses to hand out an address the tunnel already holds.
const (
tunnelKey = "TUNNEL-KEY-the-found-interfaces-public-key="
peerTwo = "PEER-KEY-two============================="
peerThree = "PEER-KEY-three==========================="
)
// theFoundTunnel is what a hub presents at enrolment: the predecessor's interface on a
// documentation range, with two peers each routed one address.
func theFoundTunnel() Tunnel {
return Tunnel{
Interface: "wg0", Unit: "wg-quick@wg0", Config: "/etc/wireguard/wg0.conf",
Port: 51900, Address: "192.0.2.1/24", Range: "192.0.2.0/24", PublicKey: tunnelKey,
Peers: []TunnelPeer{{PublicKey: peerTwo, Address: "192.0.2.2/32"},
{PublicKey: peerThree, Address: "192.0.2.3"}},
}
}
// anAdoptedHub is an adopted node that enrolled with the found tunnel's key and presented the
// tunnel, then was placed as the hub — the order genesis does it in.
func anAdoptedHub(t *testing.T, inv *Inventory) Node {
t.Helper()
hub, err := inv.AddNodeAs(t.Context(), "anchor", true)
if err != nil {
t.Fatal(err)
}
if err := inv.RecordOverlayKey(t.Context(), hub.ID, tunnelKey); err != nil {
t.Fatal(err)
}
if err := inv.RecordTunnel(t.Context(), hub.ID, theFoundTunnel()); err != nil {
t.Fatal(err)
}
if err := inv.SetPlace(t.Context(), "anchor", "anchor.example:51900", "hosting", true, ""); err != nil {
t.Fatal(err)
}
return hub
}
func TestTheHubsAddressAndRangeComeFromTheAdoptedTunnel(t *testing.T) {
inv := fresh(t)
hub := anAdoptedHub(t, inv)
tunnel, name, adopted, err := inv.AdoptedTunnel(t.Context())
if err != nil {
t.Fatal(err)
}
if !adopted || name != "anchor" || tunnel.Range != "192.0.2.0/24" || tunnel.Port != 51900 {
t.Fatalf("the adopted tunnel did not read back: adopted=%t on %s, %+v", adopted, name, tunnel)
}
if len(tunnel.Peers) != 2 || tunnel.Peers[0].Address != "192.0.2.2" || tunnel.Peers[1].Address != "192.0.2.3" {
t.Fatalf("the peers did not read back as one host address each: %+v", tunnel.Peers)
}
// Whatever range the caller would allocate from, the hub is at the tunnel's own address.
address, err := inv.AssignAddress(t.Context(), hub.ID, "10.42.0.0/16")
if err != nil {
t.Fatal(err)
}
if address != "192.0.2.1" {
t.Fatalf("the hub was given %s, not the address the tunnel it took over had", address)
}
}
func TestAnEnrollingNodeKeepsTheAddressTheTunnelHadForItsKey(t *testing.T) {
inv := fresh(t)
anAdoptedHub(t, inv)
// A predecessor machine enrols: its host took its found interface's key as its overlay key,
// which is the key the hub's tunnel already routes to.
peer, err := inv.AddNodeAs(t.Context(), "home-server", true)
if err != nil {
t.Fatal(err)
}
if err := inv.RecordOverlayKey(t.Context(), peer.ID, peerThree); err != nil {
t.Fatal(err)
}
address, err := inv.AssignAddress(t.Context(), peer.ID, "192.0.2.0/24")
if err != nil {
t.Fatal(err)
}
if address != "192.0.2.3" {
t.Fatalf("the enrolling peer was given %s, not the 192.0.2.3 the tunnel had for its key", address)
}
carried, err := inv.CarriedPeers(t.Context())
if err != nil {
t.Fatal(err)
}
byKey := map[string]CarriedPeer{}
for _, c := range carried {
byKey[c.PublicKey] = c
}
if byKey[peerThree].EnrolledAs != "home-server" || byKey[peerTwo].EnrolledAs != "" {
t.Fatalf("the registry cannot say which peer is a node now: %+v", carried)
}
}
func TestAFreshNodeIsNeverGivenAnAddressTheTunnelHolds(t *testing.T) {
inv := fresh(t)
anAdoptedHub(t, inv)
// .1 is the hub, .2 and .3 are peers of the tunnel that have not enrolled: a new machine with
// a key of its own gets the next one, from the same range.
fresh, err := inv.AddNode(t.Context(), "laptop")
if err != nil {
t.Fatal(err)
}
if err := inv.RecordOverlayKey(t.Context(), fresh.ID, "A-KEY-OF-ITS-OWN========================"); err != nil {
t.Fatal(err)
}
address, err := inv.AssignAddress(t.Context(), fresh.ID, "192.0.2.0/24")
if err != nil {
t.Fatal(err)
}
if address != "192.0.2.4" {
t.Fatalf("a fresh node was given %s; 192.0.2.2 and .3 are the tunnel's peers and .1 its hub", address)
}
}
func TestATunnelUnderAnotherKeyIsNotAdopted(t *testing.T) {
// A hub whose overlay key is not the found tunnel's would drop every peer's packets on the
// found port (ADR 0105, option 2). Such a tunnel is recorded and not adopted: the mesh keeps
// its own range, and ADR 0100's non-overlap rule stands for it.
inv := fresh(t)
hub, err := inv.AddNodeAs(t.Context(), "anchor", true)
if err != nil {
t.Fatal(err)
}
if err := inv.RecordOverlayKey(t.Context(), hub.ID, "THE-MESHS-OWN-KEY======================="); err != nil {
t.Fatal(err)
}
if err := inv.RecordTunnel(t.Context(), hub.ID, theFoundTunnel()); err != nil {
t.Fatal(err)
}
if err := inv.SetPlace(t.Context(), "anchor", "anchor.example:51820", "hosting", true, ""); err != nil {
t.Fatal(err)
}
if _, _, adopted, err := inv.AdoptedTunnel(t.Context()); err != nil || adopted {
t.Fatalf("a tunnel under another key was adopted (err %v)", err)
}
if carried, err := inv.CarriedPeers(t.Context()); err != nil || len(carried) != 0 {
t.Fatalf("peers of a tunnel that was not adopted are carried: %+v (err %v)", carried, err)
}
if address, err := inv.AssignAddress(t.Context(), hub.ID, "10.42.0.0/16"); err != nil || address != "10.42.0.1" {
t.Fatalf("the hub was given %s (err %v); it should allocate from the mesh's own range", address, err)
}
}
func TestAPeerRoutedARangeIsNotCarried(t *testing.T) {
// A peer routed a whole range is a spoke's view of its hub, never a machine with an address
// the mesh could carry: skipped, and the single-address peers beside it kept.
inv := fresh(t)
hub, err := inv.AddNodeAs(t.Context(), "anchor", true)
if err != nil {
t.Fatal(err)
}
found := theFoundTunnel()
found.Peers = append(found.Peers, TunnelPeer{PublicKey: "WIDE", Address: "192.0.2.0/24"})
if err := inv.RecordTunnel(t.Context(), hub.ID, found); err != nil {
t.Fatal(err)
}
got, err := inv.TunnelOf(t.Context(), "anchor")
if err != nil || len(got.Peers) != 2 {
t.Fatalf("the range-routed peer was carried, or the others dropped: %+v %v", got.Peers, err)
}
// A single address outside the tunnel's range is still refused: it is not a peer this tunnel
// routes to.
found.Peers = []TunnelPeer{{PublicKey: "ELSEWHERE", Address: "198.51.100.7/32"}}
if err := inv.RecordTunnel(t.Context(), hub.ID, found); err == nil || !strings.Contains(err.Error(), "outside") {
t.Fatalf("a peer outside the range was recorded: %v", err)
}
}
// A predecessor spoke's tunnel has one peer — the hub — routed the whole range. Its enrolment must
// not fail on it: only the hub's peers are ever carried, so a range-routed peer is skipped.
func TestASpokesTunnelEnrolsWithItsHubPeerSkipped(t *testing.T) {
inv := fresh(t)
anAdoptedHub(t, inv)
spoke, err := inv.AddNodeAs(t.Context(), "home-server", true)
if err != nil {
t.Fatal(err)
}
if err := inv.RecordOverlayKey(t.Context(), spoke.ID, peerThree); err != nil {
t.Fatal(err)
}
if err := inv.RecordTunnel(t.Context(), spoke.ID, Tunnel{
Interface: "wg0", Unit: "wg-quick@wg0", Config: "/etc/wireguard/wg0.conf", Port: 51900,
Address: "192.0.2.3/24", Range: "192.0.2.0/24", PublicKey: peerThree,
Peers: []TunnelPeer{{PublicKey: tunnelKey, Address: "192.0.2.0/24"}},
}); err != nil {
t.Fatalf("a spoke-shaped tunnel was refused: %v", err)
}
got, err := inv.TunnelOf(t.Context(), "home-server")
if err != nil || len(got.Peers) != 0 {
t.Fatalf("the spoke's hub was recorded as a peer to carry: %+v %v", got.Peers, err)
}
// Nothing about the hub's carried peers changed: still two, one now enrolled.
carried, err := inv.CarriedPeers(t.Context())
if err != nil || len(carried) != 2 {
t.Fatalf("carried peers: %+v %v", carried, err)
}
if address, err := inv.AssignAddress(t.Context(), spoke.ID, "192.0.2.0/24"); err != nil || address != "192.0.2.3" {
t.Fatalf("the spoke did not keep its address: %s %v", address, err)
}
}
// Converging the hub flips its mode and nothing else: the range stays the tunnel's and the peers
// stay carried, or the mesh would renumber itself and drop the peers still reaching it.
func TestConvergingTheHubKeepsTheRangeAndTheCarriedPeers(t *testing.T) {
inv := fresh(t)
hub := anAdoptedHub(t, inv)
if _, err := inv.AssignAddress(t.Context(), hub.ID, "192.0.2.0/24"); err != nil {
t.Fatal(err)
}
if _, err := inv.Converge(t.Context(), "anchor"); err != nil {
t.Fatal(err)
}
tunnel, _, adopted, err := inv.AdoptedTunnel(t.Context())
if err != nil || !adopted || tunnel.Range != "192.0.2.0/24" {
t.Fatalf("converging renumbered the mesh: adopted=%t %+v %v", adopted, tunnel, err)
}
if carried, err := inv.CarriedPeers(t.Context()); err != nil || len(carried) != 2 {
t.Fatalf("converging dropped the carried peers: %+v %v", carried, err)
}
found, err := inv.Tunnels(t.Context())
if err != nil || found["anchor"].NodeAdopted {
t.Fatalf("a converged hub still reads as adopted for the takeover: %+v %v", found, err)
}
}
// A hub that enrolled with a key of its own takes the tunnel over afterwards by rekeying: the key
// and the tunnel are recorded, the hub moves to the tunnel's address, and the same rekey applied
// again is stale.
func TestARekeyTakesTheTunnelOverAfterEnrolment(t *testing.T) {
inv := fresh(t)
hub, err := inv.AddNodeAs(t.Context(), "anchor", true)
if err != nil {
t.Fatal(err)
}
const own = "THE-MESHS-OWN-KEY======================="
if err := inv.RecordOverlayKey(t.Context(), hub.ID, own); err != nil {
t.Fatal(err)
}
if err := inv.SetPlace(t.Context(), "anchor", "anchor.example:51900", "hosting", true, ""); err != nil {
t.Fatal(err)
}
if address, err := inv.AssignAddress(t.Context(), hub.ID, "10.42.0.0/16"); err != nil || address != "10.42.0.1" {
t.Fatalf("before the rekey the hub is on the mesh's own range: %s %v", address, err)
}
if err := inv.Rekey(t.Context(), hub.ID, own, tunnelKey, theFoundTunnel()); err != nil {
t.Fatal(err)
}
_, _, adopted, err := inv.AdoptedTunnel(t.Context())
if err != nil || !adopted {
t.Fatalf("the tunnel is not adopted after the rekey (%v)", err)
}
placed, err := inv.Overlays(t.Context())
if err != nil || len(placed) != 1 || placed[0].Address != "192.0.2.1" || placed[0].Key != tunnelKey {
t.Fatalf("the hub did not move to the tunnel's address under the tunnel's key: %+v %v", placed, err)
}
if err := inv.Rekey(t.Context(), hub.ID, own, tunnelKey, theFoundTunnel()); !errors.Is(err, ErrStaleRekey) {
t.Fatalf("the same rekey applied again was not refused as stale: %v", err)
}
if err := inv.Rekey(t.Context(), hub.ID, tunnelKey, "ANOTHER-KEY=============================", theFoundTunnel()); err == nil {
t.Fatal("a rekey to a key that is not the tunnel's was accepted")
}
}
+71
View File
@@ -124,6 +124,29 @@ func (e Enrolment) Enrol(ctx context.Context, request EnrolRequest) (reply Enrol
"%s's overlay key could not be recorded: %w", node.Name, err)
}
}
// And the tunnel it found, whose key is the overlay key above (novox/hq ADR 0105). Recorded
// before the token is spent for the same reason as the keys: the first declaration this node
// receives is composed from it, and a hub enrolled without its tunnel would be placed at an
// address of the mesh's choosing rather than the tunnel's.
if request.Tunnel != nil {
if request.Tunnel.PublicKey != request.OverlayKey {
return EnrolReply{}, fmt.Errorf("%s presented a tunnel under key %s and an overlay key "+
"that is not it; a tunnel is taken over with its own key or not at all", node.Name,
request.Tunnel.PublicKey)
}
peers := make([]inventory.TunnelPeer, 0, len(request.Tunnel.Peers))
for _, p := range request.Tunnel.Peers {
peers = append(peers, inventory.TunnelPeer{PublicKey: p.PublicKey, Address: p.Address})
}
if err := e.Inventory.RecordTunnel(ctx, node.ID, inventory.Tunnel{
Interface: request.Tunnel.Interface, Unit: request.Tunnel.Unit,
Config: request.Tunnel.Config, Port: request.Tunnel.Port, MTU: request.Tunnel.MTU,
Address: request.Tunnel.Address, Range: request.Tunnel.Range,
PublicKey: request.Tunnel.PublicKey, Peers: peers,
}); err != nil {
return EnrolReply{}, fmt.Errorf("%s's found tunnel could not be recorded: %w", node.Name, err)
}
}
// Spent once the node is complete in the store.
if err := e.Inventory.Spend(ctx, secret, by); err != nil {
@@ -158,6 +181,34 @@ func (e Enrolment) Enrol(ctx context.Context, request EnrolRequest) (reply Enrol
return reply, nil
}
// rekey applies a verified rekey: the node's overlay key and tunnel are recorded as enrolment
// would have recorded them, and a hub moves to the tunnel's address.
func (e Enrolment) rekey(ctx context.Context, node inventory.Node, r Rekey) error {
if r.Tunnel == nil || r.OverlayKey == "" {
return fmt.Errorf("%s sent a rekey naming no tunnel or no key; refused", node.Name)
}
if e.Identity == nil {
return fmt.Errorf("%s sent a rekey and this mesh has no identity store to verify it against", node.Name)
}
if err := e.Identity.VerifyNode(ctx, node.ID,
RekeyProof(node.Name, r.Previous, r.OverlayKey, r.Tunnel), r.Proof); err != nil {
return fmt.Errorf("%s's rekey is not signed by %s's identity key; refused: %w", node.Name, node.Name, err)
}
peers := make([]inventory.TunnelPeer, 0, len(r.Tunnel.Peers))
for _, p := range r.Tunnel.Peers {
peers = append(peers, inventory.TunnelPeer{PublicKey: p.PublicKey, Address: p.Address})
}
err := e.Inventory.Rekey(ctx, node.ID, r.Previous, r.OverlayKey, inventory.Tunnel{
Interface: r.Tunnel.Interface, Unit: r.Tunnel.Unit, Config: r.Tunnel.Config, Port: r.Tunnel.Port, MTU: r.Tunnel.MTU,
Address: r.Tunnel.Address, Range: r.Tunnel.Range, PublicKey: r.Tunnel.PublicKey, Peers: peers,
})
if err != nil {
return fmt.Errorf("%s's rekey was not recorded: %w", node.Name, err)
}
log.Printf("%s took over the tunnel on %s: its overlay key is the tunnel's now", node.Name, r.Tunnel.Interface)
return nil
}
// claimant names the key presenting a token, so a claim can be held for it alone.
func claimant(public ed25519.PublicKey) string {
sum := sha256.Sum256(public)
@@ -219,6 +270,26 @@ func (e Enrolment) Heard(ctx context.Context, report Report) (err error) {
return err
}
}
// What it says about the tunnel it carried (novox/hq ADR 0105), whenever it says it.
if report.Tunnel != nil {
if err := e.Inventory.RecordCarriedTunnel(ctx, node.ID, inventory.Carried{
Interface: report.Tunnel.Interface, Port: report.Tunnel.Port, Range: report.Tunnel.Range,
Peers: report.Tunnel.Peers, State: report.Tunnel.State, Note: report.Tunnel.Note,
Kept: report.Tunnel.Kept,
}); err != nil {
return err
}
}
// A node taking a found tunnel's key after enrolment (novox/hq ADR 0105). Verified against the
// node's live identity key before anything is written: the broker account authenticates the
// connection, the signature proves the node itself said it. Refused outright when the proof
// does not verify or is stale — a refusal, not "not now", so the node hears why.
if report.Rekey != nil {
if err := e.rekey(ctx, node, *report.Rekey); err != nil {
return err
}
return e.Inventory.Seen(ctx, node.ID)
}
// A bare word that a node is there is not an account of what the machine did or holds: it
// moves last_seen and touches nothing else. This arrives every minute (link.AliveEvery),
+84
View File
@@ -8,6 +8,8 @@ package link
import (
"encoding/base64"
"strconv"
"strings"
"time"
)
@@ -71,12 +73,40 @@ type EnrolRequest struct {
// node's public key could replay the spent token (novox/hq issue 083, on review).
Proof []byte `json:"proof,omitempty"`
// Tunnel is the tunnel this node found on its machine and whose key it took as its overlay
// key (novox/hq ADR 0105): the interface, its port, address and range, and its peers. Presented
// with the keys because it is one of them — OverlayKey above is this tunnel's public key when
// it is set — and the mesh composes the hub's address, the range and every carried peer from
// it. Nil from a node that found none, which is every converged one.
Tunnel *Tunnel `json:"tunnel,omitempty"`
// Redelivered is set by the control plane, never sent: the broker handed this request over a
// second time. Such a request does not finish an enrolment already spent — the first time may
// have answered, and the node holds what it was told.
Redelivered bool `json:"-"`
}
// Tunnel is a found tunnel as a node presents it: everything but its private key, which the node
// keeps as its own overlay key and never sends.
type Tunnel struct {
Interface string `json:"interface"`
Unit string `json:"unit"`
Config string `json:"config"`
Port int `json:"port"`
MTU int `json:"mtu,omitempty"`
Address string `json:"address"`
Range string `json:"range"`
PublicKey string `json:"public_key"`
Peers []TunnelPeer `json:"peers,omitempty"`
}
// TunnelPeer is one peer of a found tunnel: its public key and the address the tunnel routed to
// it.
type TunnelPeer struct {
PublicKey string `json:"public_key"`
Address string `json:"address"`
}
// Signed is a declaration and the signature over it.
//
// The signature is over Declaration exactly as it will arrive, bytes unchanged — a node verifies
@@ -129,6 +159,60 @@ type Report struct {
// Reachable is what can be reached on the machine now: every listening socket and every
// published container port. Only an adopted node reports it; it is what converging previews.
Reachable []Reach `json:"reachable,omitempty"`
// Tunnel is what an adopted node says about the tunnel it found and carried (novox/hq ADR
// 0105): the interface, its port, range and peer count, whether the found interface is down
// and the mesh's up in its place, and where the found configuration's original was kept.
Tunnel *CarriedTunnel `json:"tunnel,omitempty"`
// Rekey is a node taking a found tunnel's key as its overlay key after enrolment (novox/hq
// ADR 0105). A report carrying one is not an account of the machine: it moves the node's
// overlay key and tunnel and nothing else.
Rekey *Rekey `json:"rekey,omitempty"`
}
// CarriedTunnel is a node's account of the tunnel it took over. State is "not-taken" (the found
// interface still up, the mesh's not), "taken" (the found one down, the mesh's up with its key) or
// "down" (the found one down and the mesh's not up: the peers reach nothing); Note is what the host
// did about it.
type CarriedTunnel struct {
Interface string `json:"interface"`
Port int `json:"port"`
Range string `json:"range"`
Peers int `json:"peers"`
State string `json:"state"`
Note string `json:"note,omitempty"`
Kept string `json:"kept,omitempty"`
}
// Rekey is a node saying it took a found tunnel's key as its overlay key after enrolling (novox/hq
// ADR 0105) — the path for a hub that enrolled before the mesh knew to take a tunnel over, since
// re-enrolling would rotate every key the node holds. Carried in a report, on the node's own
// authenticated connection, and signed with its identity key over RekeyProof, so a report forged
// on a stolen broker account cannot move a node's overlay key.
type Rekey struct {
// Previous is the overlay key the node holds now, as the mesh records it. A rekey naming
// another is stale — a replay, or made against a record that moved on — and is refused.
Previous string `json:"previous"`
OverlayKey string `json:"overlay_key"`
Tunnel *Tunnel `json:"tunnel"`
Proof []byte `json:"proof"`
}
// RekeyProof is what a node signs when it rekeys: the node, the key it leaves, the key it takes
// and the tunnel it took it from, so a proof cannot be moved to another node or another tunnel.
func RekeyProof(node, previous, key string, tunnel *Tunnel) []byte {
var t Tunnel
if tunnel != nil {
t = *tunnel
}
peers := make([]string, 0, len(t.Peers))
for _, p := range t.Peers {
peers = append(peers, p.PublicKey+"@"+p.Address)
}
return []byte("novox-mesh-rekey\x00" + node + "\x00" + previous + "\x00" + key + "\x00" +
t.Interface + "\x00" + t.Unit + "\x00" + t.Config + "\x00" + strconv.Itoa(t.Port) + "\x00" +
t.Address + "\x00" + t.Range + "\x00" + t.PublicKey + "\x00" + strings.Join(peers, ","))
}
// Held is one file or container found on an adopted node and kept as it was.
+135
View File
@@ -0,0 +1,135 @@
package link_test
import (
"crypto/ed25519"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/identity"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// novox/hq ADR 0105: a hub that enrolled before the mesh knew to take a tunnel over rekeys onto the
// found tunnel's key without re-enrolling — which would rotate every key it holds and remake every
// credential the mesh sealed to it. The rekey rides in a report and is signed with the node's
// identity key; the mesh verifies it against the key it recorded, and refuses one signed by
// another key or one already applied.
const (
ownKey = "THE-MESHS-OWN-KEY======================="
tunnelKey = "TUNNEL-KEY-the-found-interfaces-public-key="
)
func theTunnel() *link.Tunnel {
return &link.Tunnel{Interface: "wg0", Unit: "wg-quick@wg0", Config: "/etc/wireguard/wg0.conf",
Port: 51900, Address: "192.0.2.1/24", Range: "192.0.2.0/24", PublicKey: tunnelKey,
Peers: []link.TunnelPeer{{PublicKey: "PEER-A=", Address: "192.0.2.2/32"}}}
}
// anEnrolledHub is a hub the way it stands before the feature: adopted, placed, its overlay key its
// own, its identity key recorded — and a mesh holding both stores.
func anEnrolledHub(t *testing.T) (link.Enrolment, inventory.Node, ed25519.PrivateKey) {
t.Helper()
inv := inventory.ForTest(t)
ident := identity.ForTest(t)
ctx := t.Context()
hub, err := inv.AddNodeAs(ctx, "anchor", true)
if err != nil {
t.Fatal(err)
}
public, private, err := ed25519.GenerateKey(nil)
if err != nil {
t.Fatal(err)
}
if _, err := ident.RecordNodeKey(ctx, hub.ID, public); err != nil {
t.Fatal(err)
}
if err := inv.RecordOverlayKey(ctx, hub.ID, ownKey); err != nil {
t.Fatal(err)
}
if err := inv.SetPlace(ctx, "anchor", "anchor.example:51900", "hosting", true, "10.42.0.1"); err != nil {
t.Fatal(err)
}
return link.Enrolment{Inventory: inv, Identity: ident}, hub, private
}
func TestASignedRekeyMovesTheHubOntoItsTunnel(t *testing.T) {
e, hub, private := anEnrolledHub(t)
ctx := t.Context()
rekey := &link.Rekey{Previous: ownKey, OverlayKey: tunnelKey, Tunnel: theTunnel()}
rekey.Proof = ed25519.Sign(private, link.RekeyProof("anchor", ownKey, tunnelKey, theTunnel()))
if err := e.Heard(ctx, link.Report{Node: "anchor", Rekey: rekey}); err != nil {
t.Fatal(err)
}
placed, err := e.Inventory.Overlays(ctx)
if err != nil || len(placed) != 1 {
t.Fatal(placed, err)
}
if placed[0].Key != tunnelKey || placed[0].Address != "192.0.2.1" {
t.Fatalf("the hub is not on the tunnel's key and address: %+v", placed[0])
}
tunnel, _, adopted, err := e.Inventory.AdoptedTunnel(ctx)
if err != nil || !adopted || tunnel.Range != "192.0.2.0/24" || len(tunnel.Peers) != 1 {
t.Fatalf("the tunnel is not adopted after the rekey: %+v %t %v", tunnel, adopted, err)
}
_ = hub
// Replayed, it is stale: the previous key it names is no longer the node's.
err = e.Heard(ctx, link.Report{Node: "anchor", Rekey: rekey})
if err == nil || !strings.Contains(err.Error(), "previous overlay key") {
t.Fatalf("a replayed rekey was accepted: %v", err)
}
}
func TestARekeySignedByAnotherKeyIsRefusedAndChangesNothing(t *testing.T) {
e, _, _ := anEnrolledHub(t)
ctx := t.Context()
_, stranger, err := ed25519.GenerateKey(nil)
if err != nil {
t.Fatal(err)
}
rekey := &link.Rekey{Previous: ownKey, OverlayKey: tunnelKey, Tunnel: theTunnel()}
rekey.Proof = ed25519.Sign(stranger, link.RekeyProof("anchor", ownKey, tunnelKey, theTunnel()))
err = e.Heard(ctx, link.Report{Node: "anchor", Rekey: rekey})
if err == nil || !strings.Contains(err.Error(), "not signed by anchor's identity key") {
t.Fatalf("a rekey signed by a stranger was accepted: %v", err)
}
placed, _ := e.Inventory.Overlays(ctx)
if placed[0].Key != ownKey || placed[0].Address != "10.42.0.1" {
t.Fatalf("a refused rekey changed the record: %+v", placed[0])
}
if _, _, adopted, _ := e.Inventory.AdoptedTunnel(ctx); adopted {
t.Fatal("a refused rekey recorded a tunnel")
}
// And a proof moved to another tunnel — the signature was over one tunnel, the message names
// another — does not verify either.
moved := &link.Rekey{Previous: ownKey, OverlayKey: tunnelKey, Tunnel: theTunnel()}
other := theTunnel()
other.Port = 51820
moved.Proof = ed25519.Sign(mustPrivate(t, e, "anchor"), link.RekeyProof("anchor", ownKey, tunnelKey, other))
if err := e.Heard(ctx, link.Report{Node: "anchor", Rekey: moved}); err == nil {
t.Fatal("a proof over another tunnel was accepted")
}
}
// mustPrivate is a fresh key recorded as the node's live one, for signing in a test that needs
// the node's own signature after the fixture's key is out of scope.
func mustPrivate(t *testing.T, e link.Enrolment, node string) ed25519.PrivateKey {
t.Helper()
public, private, err := ed25519.GenerateKey(nil)
if err != nil {
t.Fatal(err)
}
n, err := e.Inventory.NodeByName(t.Context(), node)
if err != nil {
t.Fatal(err)
}
if _, err := e.Identity.RecordNodeKey(t.Context(), n.ID, public); err != nil {
t.Fatal(err)
}
return private
}
+47 -21
View File
@@ -43,6 +43,40 @@ func Declaration(node Node, peers []Peer, keyPath string) ([]byte, error) {
keyPath = DefaultKeyPath
}
up := Resource{
"id": "overlay-up", "type": "service", "unit": Unit,
"state": "running",
// Enabled, so the node comes back onto the network after a reboot without waiting to
// be told again. A node whose overlay only exists while something is watching is not
// a node that survives being switched off and on.
"boot": "enabled",
// And restarted when the peer list changes, because a running interface does not
// re-read its configuration.
//
// This is the whole of it: a node joins, every existing node's peer list changes,
// each file is replaced — and without this the service is already running, nothing
// reloads it, and every node keeps a network that no longer matches the mesh. It
// reports complete success. The lab found it the moment a third node arrived.
//
// Declared state rather than a command. The service must reflect the file; the host
// works out that it does not. A command to restart would be an action, and the link
// may not carry one (novox/hq ADR 0005) — the host refused exactly that, correctly,
// which is how this shape was arrived at.
"restart-on": []string{"overlay-config"},
}
if node.TakesOver != nil {
// The private network takes over the tunnel it found (novox/hq ADR 0105): before this
// unit starts, the host stops and disables the found one — never flushing it — and keeps
// its configuration like any held file. The key is already the found one: the node took
// it as its own overlay key when it enrolled, which is why the mesh's peer list for it
// carries the found peers under the key they know.
up["takes-over"] = map[string]any{
"interface": node.TakesOver.Interface,
"unit": node.TakesOver.Unit,
"config": node.TakesOver.Config,
}
}
resources := []Resource{
{
"id": "overlay-tools", "type": "package", "package": "wireguard-tools",
@@ -55,27 +89,7 @@ func Declaration(node Node, peers []Peer, keyPath string) ([]byte, error) {
"mode": "0600",
"content": config(node, peers, keyPath),
},
{
"id": "overlay-up", "type": "service", "unit": Unit,
"state": "running",
// Enabled, so the node comes back onto the network after a reboot without waiting to
// be told again. A node whose overlay only exists while something is watching is not
// a node that survives being switched off and on.
"boot": "enabled",
// And restarted when the peer list changes, because a running interface does not
// re-read its configuration.
//
// This is the whole of it: a node joins, every existing node's peer list changes,
// each file is replaced — and without this the service is already running, nothing
// reloads it, and every node keeps a network that no longer matches the mesh. It
// reports complete success. The lab found it the moment a third node arrived.
//
// Declared state rather than a command. The service must reflect the file; the host
// works out that it does not. A command to restart would be an action, and the link
// may not carry one (novox/hq ADR 0005) — the host refused exactly that, correctly,
// which is how this shape was arrived at.
"restart-on": []string{"overlay-config"},
},
up,
}
// The names used to be appended here, on the argument that a node with peers and no names is
@@ -109,6 +123,18 @@ func config(node Node, peers []Peer, keyPath string) string {
if port := portOf(node.Endpoint); port != "" {
fmt.Fprintf(&b, "ListenPort = %s\n", port)
}
} else if node.TakesOver != nil && node.TakesOver.Port != 0 {
// Not dialable from the hub, but a LAN peer dials this node on the tunnel it took over,
// so the mesh's interface must listen on that same port (novox/hq: a taken tunnel brings
// its port). Without this the takeover guard refuses overlay-up, and re-placing the node
// with an endpoint — the guard's suggested remedy — breaks a NAT'd node's path.
fmt.Fprintf(&b, "ListenPort = %d\n", node.TakesOver.Port)
}
// The MTU the found tunnel carried, when it set one: a path tuned to 1380 (say) stalls TLS
// and hangs transfers if the mesh's interface comes up at the 1420 default, and no ping shows
// it (novox/hq: a taken tunnel carries its MTU).
if node.TakesOver != nil && node.TakesOver.MTU != 0 {
fmt.Fprintf(&b, "MTU = %d\n", node.TakesOver.MTU)
}
// The private key is set from a file the node wrote, so it never appears here and never
// travelled. Everything else in this file came from the mesh; this one line is the node's.
+86
View File
@@ -223,3 +223,89 @@ func TestTheHubForwardsAndNobodyElseDoes(t *testing.T) {
"compromised one could do")
}
}
// novox/hq ADR 0105: a node whose private network takes over the tunnel it found is told so on
// the interface's service, and nothing else about the declaration changes — the key is already
// the found one, taken at enrolment.
func TestTakingOverAFoundTunnelIsSaidOnTheInterfacesService(t *testing.T) {
node := Node{Name: "anchor", Key: "PUB", Address: "192.0.2.1", Hub: true,
Endpoint: "198.51.100.1:51900",
TakesOver: &TakeOver{Interface: "wg0", Unit: "wg-quick@wg0", Config: "/etc/wireguard/wg0.conf"}}
config, resources := declarationFor(t, node, nil)
var up map[string]any
for _, r := range resources {
if r["type"] == "service" {
up = r
}
}
takes, ok := up["takes-over"].(map[string]any)
if !ok {
t.Fatalf("the interface's service does not say what it takes over: %+v", up)
}
if takes["unit"] != "wg-quick@wg0" || takes["config"] != "/etc/wireguard/wg0.conf" || takes["interface"] != "wg0" {
t.Errorf("the takeover names the wrong tunnel: %+v", takes)
}
if !strings.Contains(config, "ListenPort = 51900") {
t.Errorf("the hub's interface does not listen on the tunnel's port:\n%s", config)
}
if strings.Contains(config, "PrivateKey") {
t.Error("the found key travelled in the configuration; it is the node's own, set from its key file")
}
_, plain := declarationFor(t, Node{Name: "laptop", Key: "PUB", Address: "192.0.2.4"}, nil)
for _, r := range plain {
if _, says := r["takes-over"]; says {
t.Error("a node taking over nothing was told to take something over")
}
}
}
func TestATakenTunnelBringsItsListenPortEvenWhenNotDialable(t *testing.T) {
// A home node behind NAT (no Endpoint, so not Reachable) that took over a tunnel must still
// listen on that tunnel's port, because its LAN peers dial it there (novox/hq: a taken tunnel
// brings its port). Without this the takeover guard refuses overlay-up.
config, _ := declarationFor(t, Node{
Name: "shanks", Key: "SPOKE", Address: "10.10.0.3",
TakesOver: &TakeOver{Interface: "wg0", Unit: "wg-quick@wg0", Port: 51820},
}, nil)
if !strings.Contains(config, "ListenPort = 51820") {
t.Fatalf("a taken tunnel's port must be the mesh interface's ListenPort:\n%s", config)
}
if strings.Contains(config, "Endpoint =") {
t.Error("a node that only listens for LAN peers must not advertise an endpoint")
}
}
func TestANodeWithNoTunnelAndNoEndpointStillListensOnNothing(t *testing.T) {
// The guard against over-emitting: a plain spoke with neither an endpoint nor a taken tunnel
// writes no ListenPort — it purely dials out.
config, _ := declarationFor(t, Node{Name: "laptop", Key: "K", Address: "10.10.0.9"}, nil)
if strings.Contains(config, "ListenPort") {
t.Fatalf("a dial-only node needs no ListenPort:\n%s", config)
}
}
func TestATakenTunnelCarriesItsMTU(t *testing.T) {
// A path tuned to a smaller MTU (1380 here) must survive the takeover — the mesh interface
// comes up with the same MTU, or transfers hang silently (novox/hq: a taken tunnel carries
// its MTU).
config, _ := declarationFor(t, Node{
Name: "shanks", Key: "SPOKE", Address: "10.10.0.3",
TakesOver: &TakeOver{Interface: "wg0", Port: 51820, MTU: 1380},
}, nil)
if !strings.Contains(config, "MTU = 1380") {
t.Fatalf("the tuned MTU was dropped:\n%s", config)
}
}
func TestNoMTULineWhenTheTunnelSetNone(t *testing.T) {
config, _ := declarationFor(t, Node{
Name: "shanks", Key: "SPOKE", Address: "10.10.0.3",
TakesOver: &TakeOver{Interface: "wg0", Port: 51820},
}, nil)
if strings.Contains(config, "MTU") {
t.Fatalf("no MTU was found, so none should be written:\n%s", config)
}
}
+7 -11
View File
@@ -54,17 +54,13 @@ const Addressing = "mesh-addressing"
// something that needed the mesh's own addresses. Which is exactly what happened, once.
const TheNetwork = "the-private-network"
// Resolver is the module that answers every name under a machine, and the claim it holds.
//
// A claim because a machine has one resolver: two daemons answering the same names on one machine
// is a coin toss about which one a query reaches, and the answer differing between them is the
// kind of fault nobody finds by looking at either.
const (
Resolver = "mesh-resolver"
// ResolverData is what a module running a resolver requires: the mesh's own account of which
// machines exist and where, in a file.
ResolverData = "resolver-data"
)
// **A resolver's data went the same way as the names.** Two constants used to sit here — a
// `mesh-resolver` module that would write the machines as wildcards, and a `resolver-data`
// requirement a daemon would ask for. Nothing ever provided or consumed either: a module that
// answers names asks for the `node-zones` fact in its own manifest (catalogue.FactsInto) and
// requires Addressing, since the file is made of the mesh's addresses and means nothing off the
// network. A requirement nothing provides is refused at resolution, so leaving the names here
// would only have documented a mechanism that does not exist.
// Domain is the module for people who want a network and do not want to choose one.
//
+66
View File
@@ -26,6 +26,55 @@ type Node struct {
Site string
Hub bool
Address string
// Carried are the peers of the tunnel this node took over (novox/hq ADR 0105): machines the
// mesh has no record of, each known by the public key and the address the found tunnel routed
// to it. Only a hub has any. They stay in its peer list until a node enrols with that key —
// from then on the node is the peer.
Carried []Carried
// TakesOver names the found tunnel this node's private network replaces: its unit is stopped
// and disabled, never flushed, and its configuration kept, before the mesh's interface comes
// up with the found key. Nil on a node that raises the mesh's interface beside whatever it has.
TakesOver *TakeOver
}
// Carried is one peer of an adopted tunnel that has not enrolled: a peer of the tunnel, not a
// node of the mesh.
type Carried struct {
Key string
Address string
}
// TakeOver is the found tunnel a node's private network takes over, as the host is told it.
type TakeOver struct {
Interface string
Unit string
Config string
// MTU is the found tunnel's, when it set one — emitted so a tuned path keeps its MTU.
MTU int
// Port is the port the found tunnel listened on. The mesh's interface must listen on it too,
// even on a node that is not dialable from the hub: a home node's LAN peers dial it there
// (novox/hq: a taken tunnel brings its port). Listening is "a peer dials me here"; it is not
// "the hub can dial me", which is Reachable — the two were conflated.
Port int
}
// HostPrefix is one address as a route: /32 for IPv4, /128 for IPv6.
func HostPrefix(address string) string {
if strings.Contains(address, ":") {
return address + "/128"
}
return address + "/32"
}
// CarriedName is how a carried peer is named in a peer list: it has no node name, so it is named
// by the key its packets arrive under.
func CarriedName(key string) string {
short := key
if len(short) > 8 {
short = short[:8] + "…"
}
return "a peer of the tunnel (" + short + ")"
}
// Reachable reports whether other nodes can dial this one. Declared, never inferred.
@@ -145,7 +194,9 @@ func Compute(nodes []Node, overlayCIDR string) (Graph, error) {
// The hub holds every node that does not share a site with it, because those nodes
// route through it and it must know where to send the replies. Ones it cannot dial
// will dial it.
enrolled := map[string]bool{}
for _, other := range usable {
enrolled[other.Key] = true
if other.Name == self.Name || (self.Site != "" && self.Site == other.Site) {
continue
}
@@ -156,6 +207,21 @@ func Compute(nodes []Node, overlayCIDR string) (Graph, error) {
Why: "routes through this hub",
})
}
// And every peer of the tunnel it took over that has not enrolled (novox/hq ADR
// 0105): the same key and the same address the found tunnel had for it, so the
// machine behind it cannot tell the tunnel changed hands. No endpoint — it dials in,
// as it always did. Once a node enrols with that key, the node's entry above is the
// peer, and WireGuard takes one entry per key.
for _, c := range self.Carried {
if enrolled[c.Key] {
continue
}
peers = append(peers, Peer{
Name: CarriedName(c.Key), Key: c.Key,
Allowed: HostPrefix(c.Address),
Why: "carried from the tunnel this hub took over — a peer of the tunnel, not yet a node of the mesh",
})
}
}
sort.Slice(peers, func(i, j int) bool { return peers[i].Name < peers[j].Name })
+36
View File
@@ -304,3 +304,39 @@ func TestOneReachableNodeIsEnoughToPeerDirectly(t *testing.T) {
"nowhere to go")
}
}
// novox/hq ADR 0105: a hub that took over the predecessor's tunnel carries every peer that tunnel
// had, under the key and at the address the peer knows, until a node enrols with that key.
func TestTheHubCarriesTheTunnelsPeersUntilTheyEnrol(t *testing.T) {
hub := at("anchor", "hosting", "192.0.2.1", "198.51.100.1:51900", true)
hub.Carried = []Carried{
{Key: "key-home", Address: "192.0.2.2"},
{Key: "key-workstation", Address: "192.0.2.3"},
}
// The machine behind key-home enrolled: it is a node now, at the address it kept.
home := at("home", "house", "192.0.2.2", "", false)
home.Key = "key-home"
g, err := Compute([]Node{hub, home}, "192.0.2.0/24")
if err != nil {
t.Fatal(err)
}
peers := peersOf(t, g, "anchor")
carried, ok := peers[CarriedName("key-workstation")]
if !ok {
t.Fatalf("the hub does not carry the peer that has not enrolled: %+v", g["anchor"])
}
if carried.Allowed != "192.0.2.3/32" || carried.Endpoint != "" || carried.Key != "key-workstation" {
t.Errorf("a carried peer is not the tunnel's own entry — same key, its one address, no endpoint: %+v", carried)
}
if _, twice := peers[CarriedName("key-home")]; twice {
t.Error("a peer that enrolled is carried as well as listed as a node: WireGuard takes one entry per key")
}
if node, ok := peers["home"]; !ok || node.Key != "key-home" || node.Allowed != "192.0.2.2/32" {
t.Errorf("the enrolled peer is not the node it became: %+v", node)
}
// Carried peers are the hub's business only: a spoke routes everything through the hub.
if _, leaked := peersOf(t, g, "home")[CarriedName("key-workstation")]; leaked {
t.Error("a carried peer appeared in a spoke's peer list")
}
}
+29
View File
@@ -0,0 +1,29 @@
package overlay
import (
"encoding/json"
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
)
// The private network's claim is a seat the mesh defines (novox/hq ADR 0110).
//
// Checked here because this is where the manifest is composed: it ships with the control plane and
// has no module.json for a test reading the catalogue to find. Parsed with the real parser, so a
// set that forgot this seat refuses the control plane's own module here rather than on a machine.
func TestThePrivateNetworksClaimIsASeatTheMeshDefines(t *testing.T) {
for _, composed := range []map[string]any{Manifest(), DomainManifest()} {
raw, err := json.Marshal(composed)
if err != nil {
t.Fatal(err)
}
if _, err := catalogue.ParseManifest(raw); err != nil {
t.Errorf("%s, composed by the control plane, is refused: %v", composed["module"], err)
}
}
seat, defined := catalogue.SeatNamed(TheNetwork)
if !defined || seat.Scope != catalogue.ScopeNode {
t.Fatalf("%s is not a node seat the mesh defines: %+v", TheNetwork, seat)
}
}
+52
View File
@@ -0,0 +1,52 @@
package store
import (
"encoding/json"
"os"
"testing"
)
// **The manifest's placeholder, unfilled, reaches a store reader and changes nothing.**
//
// The control plane composes its own declaration, so the manifest naming `${seat:…}` can be
// composed by a control plane one build older than it — one that passes the literal through as
// the value. That is the state of the live control-node between this manifest landing and its
// next build being pushed, and a reader that refused the literal would leave it headless
// (novox/hq 04-ISSUES/102, finding F1). So the reader is held to ignoring exactly what
// module.json says, not a placeholder shaped like it.
func TestTheManifestsOwnPlaceholderUnfilledLeavesTheStoreWhereTheFileSays(t *testing.T) {
raw, err := os.ReadFile("../../module.json")
if err != nil {
t.Fatal(err)
}
var m struct {
Resources []struct {
Type string `json:"type"`
Env map[string]string `json:"env"`
} `json:"resources"`
}
if err := json.Unmarshal(raw, &m); err != nil {
t.Fatal(err)
}
var written string
for _, r := range m.Resources {
if r.Type == "container" {
written = r.Env["MESH_STORE_INVENTORY_PORT"]
}
}
if written == "" {
t.Fatal("module.json no longer names MESH_STORE_INVENTORY_PORT")
}
alone(t)
t.Setenv(Variable(example), dsn)
t.Setenv(PortVariable(example), written)
opened, err := Open(t.Context(), example)
if err != nil {
t.Fatalf("the unfilled placeholder was refused, which is a headless control plane: %v", err)
}
defer opened.Close()
if got := opened.Pool().Config().ConnConfig.Port; got != 5432 {
t.Fatalf("the store is on %d; with the placeholder unfilled, the file's port stands", got)
}
}
+109
View File
@@ -0,0 +1,109 @@
# Lab bed: the hub adopts the predecessor's tunnel (novox/hq ADR 0105)
A scenario and an integration-test skeleton for the mesh-lab repository, kept here because this
branch changes only the controller and the host. Move `adopt-the-tunnel.yml` to
`mesh-lab/scenarios/` and `adopt-the-tunnel.test.ts` to `mesh-lab/test/integration/` when the
feature lands; neither has been run. The skeleton follows `adoption.test.ts` and reuses its
harness. Documentation addresses throughout; the bed is node-agnostic.
## The bed, precisely
Three machines on one public segment, `hosting` (192.0.2.0/24), inbound allowed on all (the
anchor's firewall is the predecessor's, installed by the bed):
| machine | address | role |
|---|---|---|
| `anchor` | 192.0.2.10 | the machine in use: the predecessor's hub, then the mesh adopted on it |
| `peer-a` | 192.0.2.20 | a predecessor machine: reaches a service on the anchor through the tunnel; later **enrols and keeps its address** |
| `peer-b` | 192.0.2.30 | a second predecessor machine: reaches the same service; **never enrols** — the peer that must notice nothing throughout |
| `fresh` | 192.0.2.40 | a new machine: enrols later and **gets a fresh address from the same range** |
**Prepared the way the predecessor leaves a hub** (before genesis, by the bed, on `anchor`):
- `wireguard-tools` installed; a keypair made on each of `anchor`, `peer-a`, `peer-b`.
- `/etc/wireguard/wg0.conf` on the anchor: `[Interface]` `PrivateKey = <anchor's>`,
`ListenPort = 51900`, `Address = 10.10.0.1/24`; two `[Peer]` sections — `peer-a`'s public
key with `AllowedIPs = 10.10.0.2/32`, `peer-b`'s with `AllowedIPs = 10.10.0.3/32`. Raised with
`systemctl enable --now wg-quick@wg0`. **10.10.0.0/24 is deliberately not the mesh's default
range** (10.42.0.0/16), so a hub address in 10.10.0.0/24 can only have come from the tunnel.
- `wg0.conf` on each peer: its own key, `Address = 10.10.0.2/24` (resp. `.3/24`), one
`[Peer]` — the anchor's public key, `Endpoint = 192.0.2.10:51900`,
`AllowedIPs = 10.10.0.0/24`, `PersistentKeepalive = 25`. Raised the same way.
- A service on the anchor the peers reach **only over the tunnel**: a container publishing
`10.10.0.1:8081:80` (bound to the tunnel address, so a call from 192.0.2.20 to 10.10.0.1:8081
proves the tunnel carried it). Under a name no catalogue module uses — this bed is about the
tunnel, not about taking a service.
- The predecessor's firewall (`ufw`) allowing `51900/udp` and `22/tcp`, denying the rest — as ADR
0100's bed prepares it.
- A record of the anchor's `wg0` public key and of `sha256sum /etc/wireguard/wg0.conf`, taken
before genesis, for the assertions below.
**Genesis**, adopted, on the anchor: `mesh-bootstrap --adopted --node anchor --site hosting
--endpoint 192.0.2.10:51900 …` — no `--hub-port`, no `--overlay-range`, no `--tunnel`: the
installer finds `wg0` itself (one interface besides `mesh0`) and takes its port and range. The
bed asserts genesis **says** it found and took the tunnel.
**Review changes (2026-09-24).** A spoke's `wg0.conf` names one peer — the hub — routed the
whole range; the controller skips range-routed peers, so T2's enrolment carries no peer from the
spoke. A hub that enrolled *before* this feature (a generated key) takes the tunnel over without
re-enrolling: `mesh-host overlay take --tunnel wg0` on the machine rekeys the overlay key only and
sends a signed rekey; the bed adds **R0** for it below. A takeover is composed only for a hub
placed at the tunnel's address on the tunnel's port, and the host stops nothing until the declared
interface matches the found one and the key file holds the found key; a mesh interface that fails
to start gives the found unit back. The host's account has three states: `not-taken`, `taken`,
`down`.
## Assertions, in the record's order
- **R0 — a hub enrolled with its own key takes the tunnel over by rekeying.** Genesis is run
adopted *without* the tunnel being found (the bed stops `wg-quick@wg0` for the run, so the
installer sees no tunnel, then starts it again — the pre-feature shape). Then on the anchor:
`mesh-host overlay take --tunnel wg0`; `node show anchor` says "tunnel found wg0 …"; `overlay
place anchor --hub --endpoint 192.0.2.10:51900 --site hosting` (with `:51820` first, which must
be refused naming 51900); `plan anchor --json` names `Address = 10.10.0.1/32`, `ListenPort =
51900`, two `/32` peers, `takes-over` wg0 and nothing in 10.42.0.0/16; then `push anchor --wait
2m` and T1's assertions hold. `overlay take` run a second time is refused by the controller as
stale and changes nothing.
- **T1 — the tunnel changes hands and the peers notice nothing.** After genesis and the push
that raises the private network on the anchor:
- `wg show interfaces` on the anchor lists `mesh0` and not `wg0`;
`systemctl is-active wg-quick@wg0` is inactive and `is-enabled` disabled;
- `/etc/wireguard/wg0.conf` is on disk with the recorded digest (kept, never flushed), and
`node show anchor` names where its original was kept;
- `wg show mesh0 public-key` is the anchor's recorded `wg0` public key; `wg show mesh0
listen-port` is 51900; `ip -o addr show dev mesh0` carries `10.10.0.1`; `wg show mesh0 peers`
lists both peers' public keys with their `/32` allowed addresses;
- a loop on `peer-a` and `peer-b` calling `http://10.10.0.1:8081/` every second, started before
genesis, records **no window of failure longer than one WireGuard re-handshake** (measure and
assert an upper bound — the switch is one unit stop plus one unit start on the anchor); the
peers' `wg0.conf` digests are unchanged; the peers' `wg show wg0 latest-handshakes` advance
after the switch.
- `overlay show` lists `anchor` as the hub over the tunnel it took over, and both peers under
"peers of the tunnel … not nodes of the mesh", not yet enrolled.
- **T2 — a peer enrols and keeps its address.** On `peer-a`: `node add peer-a --adopted`,
token issued, `mesh-host enrol --token …` **with the broker reached over the tunnel** (the
broker address in the token is `10.10.0.1:<bus>`, which only the tunnel routes); then `overlay
place peer-a --site house` and a push. Assert: `node show peer-a` says a tunnel `wg0` was
found and `overlay show` puts `peer-a` at **10.10.0.2**; the carried-peers list now says
`enrolled as peer-a`; the anchor's `mesh0` still has exactly one entry for `peer-a`'s key;
`peer-a`'s `wg0` is down and `mesh0` up with the same key; `peer-a` still reaches
`10.10.0.1:8081` and `peer-b` still does too, uninterrupted.
- **T3 — a new machine gets a fresh address from the same range.** `fresh` enrols converged
(no tunnel), is placed, pushed. Assert its address is **10.10.0.4** (`.1` hub, `.2` and `.3`
the tunnel's), that it reaches `10.10.0.1` (the hub) and `10.10.0.2` (the enrolled peer) —
`ping -c1` over `mesh0` — and that `peer-b`, never enrolled, is still served.
- **T4 — nothing derived from the address is stale.** `plan anchor --json` and `plan peer-a
--json` (and the rendered `/etc/hosts` on each node) name `10.10.0.1` for the anchor and
`10.10.0.2` for `peer-a`, and no address in `10.42.0.0/16`; the broker address handed to a
module issued on `peer-a` is `anchor.internal:<bus>` resolving to `10.10.0.1`; the same after a
second `push` of every node, byte for byte.
- **N — the narrowed ADR 0100 check.** On `fresh`, a converged genesis dry-run with
`--overlay-range 10.10.0.0/24` while a *second* tunnel the bed raises there (`wg1` at
10.10.0.9/24, not adopted because the node is converged) is up, is refused naming `wg1` —
the non-overlap rule still applies where a tunnel is not adopted.
## What is not asserted here
- Taking a service over the tunnel (ADR 0100's bed does that).
- IPv6 tunnels: the parser reads them, the bed prepares only IPv4.
@@ -0,0 +1,174 @@
/**
* THE HUB ADOPTS THE PREDECESSOR'S TUNNEL (novox/hq ADR 0105). Skeleton — NOT YET RUN.
*
* The bed and every assertion are described in README.md beside this file; the numbered
* assertions here are that document's. Follows adoption.test.ts: same harness, same `on`/`must`
* helpers, same genesis wrapper.
*
* MESH_LAB_INCUS='sudo -n incus'
* MESH_LAB_HOST_BINARY=.../mesh-host/mesh-host
* MESH_LAB_BUNDLE=.../mesh-host/examples/foundation-first-node.lock
* MESH_LAB_CATALOG=.../mesh-catalog/modules
*/
import { test, before, after } from "node:test";
import assert from "node:assert/strict";
import { existsSync } from "node:fs";
import { loadScenario } from "../../src/declaration/parse.ts";
import { raise } from "../../src/lifecycle/raise.ts";
import { destroy, exec } from "../../src/lifecycle/operate.ts";
import { hostBinaryPath } from "../../src/lifecycle/place.ts";
import { labIsUsable, destroyAll, catalogueIsPresent } from "./harness.ts";
import { genesis } from "./genesis.ts";
const capability = await labIsUsable();
const binary = hostBinaryPath();
const bundle = process.env["MESH_LAB_BUNDLE"] ?? "";
const skip = !capability.usable ? `lab not usable: ${capability.why}`
: !binary || !existsSync(binary) ? "MESH_LAB_HOST_BINARY is not set to a built mesh-host"
: !bundle || !existsSync(bundle) ? "MESH_LAB_BUNDLE is not set to a foundation bundle"
: catalogueIsPresent();
const SCENARIO = "adopt-the-tunnel";
const ANCHOR = "anchor", PEER_A = "peer-a", PEER_B = "peer-b", FRESH = "fresh";
const TUNNEL = { port: 51900, range: "10.10.0.0/24", hub: "10.10.0.1", a: "10.10.0.2", b: "10.10.0.3", fresh: "10.10.0.4" };
const SERVICE = `http://${TUNNEL.hub}:8081/`;
let instanceId = "";
let wg0Key = ""; // the anchor's wg0 public key, recorded before genesis
let wg0Digest = ""; // sha256 of /etc/wireguard/wg0.conf before genesis
async function on(machine: string, command: string, timeoutMs?: number): Promise<{ out: string; ok: boolean }> {
const { stdout } = await exec(instanceId, machine, ["sh", "-c", `exec 2>&1\n${command}\necho "__exit=$?"`], timeoutMs);
const marker = stdout.lastIndexOf("__exit=");
if (marker < 0) return { out: stdout, ok: false };
return { out: stdout.slice(0, marker), ok: stdout.slice(marker + 7).trim() === "0" };
}
async function must(machine: string, command: string, timeoutMs?: number): Promise<string> {
const { out, ok } = await on(machine, command, timeoutMs);
if (!ok) throw new Error(`${machine}: ${command}\n${out}`);
return out;
}
/** The controller, a container on the anchor. */
async function control(args: string): Promise<string> {
return must(ANCHOR, `docker exec mesh-controller mesh-controller ${args}`);
}
/** Prepares a machine the way the predecessor leaves it: a keypair and a wg0 — see README.md. */
async function predecessorTunnelOn(machine: string, conf: (keys: Record<string, string>) => string, keys: Record<string, string>): Promise<void> {
await must(machine, "apt-get install -y wireguard-tools >/dev/null 2>&1 || pacman -S --noconfirm wireguard-tools >/dev/null");
await must(machine, `umask 077; printf '%s' '${conf(keys)}' > /etc/wireguard/wg0.conf`);
await must(machine, "systemctl enable --now wg-quick@wg0");
}
before(async () => {
if (skip) return;
await destroyAll(SCENARIO);
const scenario = loadScenario(`scenarios/${SCENARIO}.yml`);
instanceId = (await raise(scenario)).instanceId;
// Keys for the three predecessor machines, made where they live and never moved.
const keys: Record<string, string> = {};
for (const m of [ANCHOR, PEER_A, PEER_B]) {
await must(m, "umask 077; wg genkey > /etc/wireguard/predecessor.key");
keys[m] = (await must(m, "wg pubkey < /etc/wireguard/predecessor.key")).trim();
}
await predecessorTunnelOn(ANCHOR, k => [
"[Interface]", `PrivateKey = $(cat /etc/wireguard/predecessor.key)`, `ListenPort = ${TUNNEL.port}`, `Address = ${TUNNEL.hub}/24`,
"[Peer]", `PublicKey = ${k[PEER_A]}`, `AllowedIPs = ${TUNNEL.a}/32`,
"[Peer]", `PublicKey = ${k[PEER_B]}`, `AllowedIPs = ${TUNNEL.b}/32`,
].join("\n"), keys);
for (const [m, addr] of [[PEER_A, TUNNEL.a], [PEER_B, TUNNEL.b]] as const) {
await predecessorTunnelOn(m, k => [
"[Interface]", `PrivateKey = $(cat /etc/wireguard/predecessor.key)`, `Address = ${addr}/24`,
"[Peer]", `PublicKey = ${k[ANCHOR]}`, `Endpoint = 192.0.2.10:${TUNNEL.port}`, `AllowedIPs = ${TUNNEL.range}`, "PersistentKeepalive = 25",
].join("\n"), keys);
}
// A service reachable only over the tunnel, under a name no catalogue module uses.
await must(ANCHOR, `docker run -d --name predecessor-page -p ${TUNNEL.hub}:8081:80 nginx:alpine`);
// The predecessor's firewall: the tunnel's port and ssh, nothing else (as ADR 0100's bed).
await must(ANCHOR, `ufw --force reset >/dev/null; ufw default deny incoming; ufw allow 22/tcp; ufw allow ${TUNNEL.port}/udp; ufw --force enable`);
for (const m of [PEER_A, PEER_B]) assert.ok((await on(m, `curl -fsS --max-time 3 ${SERVICE}`)).ok, `${m} does not reach the service over the tunnel before genesis`);
wg0Key = (await must(ANCHOR, "wg show wg0 public-key")).trim();
wg0Digest = (await must(ANCHOR, "sha256sum /etc/wireguard/wg0.conf")).split(" ")[0];
// The probe the peers keep running through the switch: one call a second, failures counted.
for (const m of [PEER_A, PEER_B]) await must(m, `nohup sh -c 'while :; do curl -fsS --max-time 1 ${SERVICE} >/dev/null 2>&1 || date +%s >> /tmp/failed; sleep 1; done' >/dev/null 2>&1 &`);
});
after(async () => { if (instanceId) await destroy(instanceId); });
test("T1 — adopted, the tunnel changes hands and the peers notice nothing", { skip }, async () => {
const ran = await genesis({ instanceId, machine: ANCHOR, node: ANCHOR, site: "hosting",
flags: ["--adopted", "--endpoint", `192.0.2.10:${TUNNEL.port}`] } as never);
assert.match(ran.said, /tunnel\s+wg0/i, `genesis did not say it found and took the tunnel:\n${ran.said}`);
const ifaces = await must(ANCHOR, "wg show interfaces");
assert.match(ifaces, /\bmesh0\b/); assert.doesNotMatch(ifaces, /\bwg0\b/);
assert.equal((await on(ANCHOR, "systemctl is-active wg-quick@wg0")).out.trim(), "inactive");
assert.equal((await on(ANCHOR, "systemctl is-enabled wg-quick@wg0")).out.trim(), "disabled");
assert.equal((await must(ANCHOR, "sha256sum /etc/wireguard/wg0.conf")).split(" ")[0], wg0Digest, "the found configuration was changed or flushed");
assert.equal((await must(ANCHOR, "wg show mesh0 public-key")).trim(), wg0Key, "the mesh's interface is not up with the found key");
assert.equal((await must(ANCHOR, "wg show mesh0 listen-port")).trim(), String(TUNNEL.port));
assert.match(await must(ANCHOR, "ip -o addr show dev mesh0"), new RegExp(TUNNEL.hub.replaceAll(".", "\\.")));
const peers = await must(ANCHOR, "wg show mesh0 allowed-ips");
assert.match(peers, new RegExp(`${TUNNEL.a}/32`)); assert.match(peers, new RegExp(`${TUNNEL.b}/32`));
for (const m of [PEER_A, PEER_B]) {
const failed = (await on(m, "cat /tmp/failed 2>/dev/null | wc -l")).out.trim();
assert.ok(Number(failed) <= 5, `${m} lost the service for ${failed} seconds through the switch`);
assert.ok((await on(m, `curl -fsS --max-time 3 ${SERVICE}`)).ok, `${m} does not reach the service after the switch`);
}
const shown = await control("overlay show");
assert.match(shown, /anchor.*hub.*took over on wg0/);
assert.match(shown, /peers of the tunnel/); assert.match(shown, /not yet enrolled/);
assert.match(await control(`node show ${ANCHOR}`), /tunnel found\s+wg0 on port 51900/);
});
test("T2 — a peer enrols over the tunnel and keeps its address", { skip }, async () => {
await control(`node add ${PEER_A} --adopted`);
const token = (await control(`token issue --node ${PEER_A}`)).match(/token\s+(\S+)/)?.[1] ?? "";
// The token's broker address is the hub's tunnel address: only the tunnel routes it.
await must(PEER_A, `mesh-host enrol --token '${token}'`);
await control(`overlay place ${PEER_A} --site house`);
await control(`assign ${PEER_A} networking`);
await control(`push ${PEER_A} --wait 2m`);
assert.match(await control("overlay show"), new RegExp(`${PEER_A}\\s+${TUNNEL.a.replaceAll(".", "\\.")}`));
assert.match(await control("overlay show"), new RegExp(`enrolled as ${PEER_A}`));
const onHub = await must(ANCHOR, "wg show mesh0 allowed-ips");
assert.equal(onHub.split("\n").filter(l => l.includes(`${TUNNEL.a}/32`)).length, 1, "the enrolled peer's key appears twice on the hub");
assert.doesNotMatch(await must(PEER_A, "wg show interfaces"), /\bwg0\b/);
assert.ok((await on(PEER_A, `curl -fsS --max-time 3 ${SERVICE}`)).ok);
assert.ok((await on(PEER_B, `curl -fsS --max-time 3 ${SERVICE}`)).ok, "the peer that never enrols lost the service");
});
test("T3 — a new machine gets a fresh address from the same range", { skip }, async () => {
await control(`node add ${FRESH}`);
const token = (await control(`token issue --node ${FRESH}`)).match(/token\s+(\S+)/)?.[1] ?? "";
await must(FRESH, `mesh-host enrol --token '${token}'`);
await control(`overlay place ${FRESH} --nothing`);
await control(`assign ${FRESH} networking`);
await control(`push ${FRESH} --wait 2m`);
assert.match(await control("overlay show"), new RegExp(`${FRESH}\\s+${TUNNEL.fresh.replaceAll(".", "\\.")}`));
assert.ok((await on(FRESH, `ping -c1 -W2 ${TUNNEL.hub}`)).ok, "the new machine does not reach the hub");
assert.ok((await on(FRESH, `ping -c1 -W2 ${TUNNEL.a}`)).ok, "the new machine does not reach the enrolled peer");
assert.ok((await on(PEER_B, `curl -fsS --max-time 3 ${SERVICE}`)).ok);
});
test("T4 — nothing derived from the address is stale", { skip }, async () => {
for (const n of [ANCHOR, PEER_A, FRESH]) {
const plan = await control(`plan ${n} --json`);
assert.doesNotMatch(plan, /10\.42\./, `${n}'s plan names the mesh's default range`);
assert.match(plan, new RegExp(TUNNEL.hub.replaceAll(".", "\\.")));
assert.match(await must(n, "cat /etc/hosts"), new RegExp(`${TUNNEL.hub.replaceAll(".", "\\.")}\\s+anchor\\.internal`));
}
const first = await control(`plan ${PEER_A} --json`);
await control("push");
assert.equal(await control(`plan ${PEER_A} --json`), first, "a push changed what the plan says");
});
test("N — where a tunnel is not adopted, the ranges must still differ (ADR 0100)", { skip }, async () => {
await must(FRESH, "umask 077; printf '[Interface]\\nPrivateKey = %s\\nAddress = 10.10.0.9/24\\n' \"$(wg genkey)\" > /etc/wireguard/wg1.conf; systemctl start wg-quick@wg1");
const ran = await genesis({ instanceId, machine: FRESH, node: FRESH, flags: ["--dry-run", "--overlay-range", TUNNEL.range], attempts: 1, verify: false, hostService: false } as never);
assert.match(ran.said, /wg1/, `a converged genesis did not refuse the overlapping tunnel it does not adopt:\n${ran.said}`);
});
+50
View File
@@ -0,0 +1,50 @@
# THE HUB ADOPTS THE PREDECESSOR'S TUNNEL (novox/hq ADR 0105). See README.md beside this file.
#
# hosting (public)
# anchor 192.0.2.10 the predecessor's hub: wg0 on udp/51900, 10.10.0.1/24, two peers; then the
# mesh adopted on it, taking the tunnel over
# peer-a 192.0.2.20 a predecessor machine reaching a service on the anchor over the tunnel;
# enrols later and keeps 10.10.0.2
# peer-b 192.0.2.30 a predecessor machine that never enrols: must notice nothing, ever
# fresh 192.0.2.40 a new machine: enrols later and gets 10.10.0.4
#
# inbound: allow on every machine — the anchor's firewall is the predecessor's, installed by the bed.
scenario: adopt-the-tunnel
segments:
hosting:
kind: public
cidr: [192.0.2.0/24]
machines:
anchor:
at: { segment: hosting, address: [192.0.2.10] }
egress: true
inbound: allow
memory: 12GiB
cpus: 6
disk: 60GiB
peer-a:
at: { segment: hosting, address: [192.0.2.20] }
egress: true
inbound: allow
memory: 3GiB
cpus: 2
disk: 20GiB
peer-b:
at: { segment: hosting, address: [192.0.2.30] }
egress: true
inbound: allow
memory: 2GiB
cpus: 2
disk: 15GiB
fresh:
at: { segment: hosting, address: [192.0.2.40] }
egress: true
inbound: allow
memory: 3GiB
cpus: 2
disk: 20GiB
place:
all: [host, runtime]
+7 -1
View File
@@ -11,6 +11,12 @@
"scope": "mesh"
}
],
"accesses": [
{
"path": "/var/lib/mesh-broker-tls",
"mode": "read"
}
],
"own-secrets": {
"inventory": "/var/lib/mesh/mesh-controller/inventory",
"identity": "/var/lib/mesh/mesh-controller/identity",
@@ -51,7 +57,7 @@
"MESH_BROKER_ADDRESS_PORT": "${seat:mesh-broker:5671}"
},
"volumes": [
"mesh-broker-tls:/broker-tls:ro",
"/var/lib/mesh-broker-tls:/broker-tls:ro",
"/var/lib/mesh/mesh-controller/inventory:/run/secrets/inventory:ro",
"/var/lib/mesh/mesh-controller/identity:/run/secrets/identity:ro",
"/var/lib/mesh/mesh-controller/licences:/run/secrets/licences:ro",