Author SHA1 Message Date
jschoubben 506426cf94 Merge pull request 'route-proxy: a route carries the policy applied to a request' (#58) from issue/116-route-proxy-has-no-auth-or-ip-restriction into main 2026-09-25 12:21:43 +00:00
jochen e11e1374bd route-proxy: a priority is an ordering, not a port
Found reviewing my own change before merging it, and it was load-bearing rather than cosmetic.

Priority was read with asPort, which caps at 65535. A rule declared above that silently became
priority 0 and stopped shadowing the route it exists to shadow. The one real rule this has to
reproduce is declared at 100000 — so path scoping and refusal would both have shipped looking
complete, passing their tests, and doing nothing on the only case that motivated them.

A priority is an ordering and has no range. asWhole takes any whole number the mesh wrote and
rejects a non-integral one, which was not meant as a priority.

Also: a host may now be routed on some paths and not others, which made the 404 dishonest — it
said "no route for this name" while listing that very name as served, a contradiction an
operator has to disbelieve the proxy to get past. An uncovered path now says so, and a name
that is genuinely not served still lists what is.

Two regression tests, both through the proxy rather than against the parser, because the parser
was where the bug looked fine.
2026-09-25 14:20:06 +02:00
jochen 008ce39ec0 route-proxy: a route carries the policy applied to a request
Implements novox/hq ADR 0108, closing issue 116. The proxy's request path was a host lookup
and a forward, so it applied nothing — while the ingress it replaces relies on four things it
had none of.

Path scoping came first because it is a prerequisite, not a sibling. The table mapped a host
to one target, so a host could not be routed two ways, and the refusal this issue turns on
matches a path on a host already routed to a workload. No amount of authentication or source
filtering would have made it expressible. The table is now host to an ordered list of rules,
matched on path prefix.

The order is total, not just by priority. Sorting on priority alone leaves rules that share
one in whatever order the map produced, so the same declaration would serve differently
between restarts — a fault that works, and works differently each time, which is the hardest
kind to believe when reported. Within a priority the longer path wins, which is also the
intuitive reading.

auth names a secret and never holds one. A declaration carrying a credential is refused
whole rather than served unprotected, so the option ADR 0108 rejected cannot return by
accident. A secret that cannot be read makes the route refuse and say so, rather than serve
the workload unprotected — a gate that cannot check is not a gate that opens, and the
alternative turns a missing file into a silently public admin surface.

Authentication costs one bcrypt comparison on every path including an unknown user, so an
unknown user is not measurably faster than a known one with a wrong password. That difference
is a way to enumerate a route's users from outside it.

Redirects keep the request's own path and query, or canonicalising one name onto another
would land every deep link on the front page and raise no error doing it.

Eleven tests, four of them for the capabilities and two for the failure modes that rot
quietly: the credential-in-a-declaration refusal, and the unreadable secret failing closed.
Nothing else breaks if those stop working, so nothing else would report it.

No new dependency: bcrypt comes from the x/crypto module already required.
2026-09-25 14:00:57 +02:00
jschoubben 856fabda04 Merge pull request 'contributes: a module's grant carries no value where it contributed several times' (#57) from fix/several-contributions-collide-in-grants-v2 into main 2026-09-24 17:39:55 +00:00
jschoubben 8fa5443862 contributes: a module's grant carries no value where it contributed several times
ContributionsFrom settled to whichever of a module's several contributions to
one requirement sorted first, arbitrarily — the grant minted for it then
carried that contribution's label and port under a credential the OTHER
contribution's consumer never sees, and collided with that same
contribution's own entry from contributions() besides.

Confirmed live: minio's two route contributions (files-api, files) produced
three entries in route-adapter's received file — files-api twice, once
credentialed and once not, files not credentialed at all. Every
single-contribution module (gitea, keycloak, umami) already mints an unused
credential for `route` too — route never needs one, by its own
documentation — but with exactly one contribution to match there was nothing
to collide with, so it never surfaced.

Where a module contributes more than once, there is no single value to
settle on. The module still asks, still gets its one credential — a pair
credential is not a place for a label or a port anyway — and each named
contribution reaches the provider on its own, unchanged.

No cleanup needed for the secret already minted live for minio+route: the
sealed blob is a random pair credential unrelated to Values, which is
recomputed fresh on every plan/push regardless.
2026-09-24 18:54:35 +02:00
jschoubben 3ece1a86d7 Merge pull request 'plan: show what a module would open and why' (#56) from feat/plan-shows-what-a-module-would-open into main 2026-09-24 16:42:24 +00:00
jschoubben dc8839246b Merge pull request 'contributes: a module may answer one requirement several times' (#55) from feat/several-route-contributions-per-module into main 2026-09-24 16:41:58 +00:00
jschoubben 524cc2a3ec plan: show what a module would open and why
Every module.json already declares a why for each port under listens,
but plan only ever used it to build the firewall's rule set — nothing
printed it. An operator deciding whether to assign a module had no way
to see what it would open without reading the manifest by hand.

plan <node> now prints each assigned module's listens entries — port,
protocol, source, and its why — right under the module line, so the
same text that feeds the firewall is visible at the point someone is
actually deciding whether to open it.
2026-09-24 18:40:30 +02:00
jschoubben f4bcb320fe contributes: a module may answer one requirement several times
A module's contributes was map[string]map[string]any — one JSON object key
per requirement, structurally exactly one contribution to "route" ever.
minio needs two public hostnames (the S3 API and the console), which is
two different contributions to route from one module, and nothing let it
say so.

This is the same shape of problem ADR 0094 solved for secrets (a module
needing several values from one provider that gives one per pair):
contributes now accepts either the ordinary {label, port} object, or an
object of local names to several such objects. Detected per requirement
key by what's inside, since (unlike secrets' string-vs-object split) both
shapes are JSON objects: an ordinary contribution's fields are scalars, the
several-instance shape is local-name -> object. Confirmed against every
module.json in mesh-catalog before relying on that split.

Both route-proxy and the migration-era route-adapter already key generated
routers off the composed hostname (Values["name"]), not the module name,
so two contributions with the same From reach them as two independent
routes with no changes needed on the receiving side.
2026-09-24 18:36:08 +02:00
jschoubben 6090953843 Merge pull request 'Tell the resolver the machines, not the names the mesh merely serves' (#53) from fix/the-resolver-is-told-machines-not-routes into main 2026-09-23 23:32:22 +00:00
jschoubben 2277583e99 Tell the resolver the machines, not the names the mesh merely serves
The map the control plane hands a resolution holds both: the machines, and every name
the mesh was told to route to whichever machine serves it. A container's hosts wants all
of it, so a routed name resolves to the proxy. A resolver's zones want only the machines:
told the mesh's suffix is its own it answers authoritatively for everything under it and
forwards none of it, so a routed name with the suffix appended — drive.example.test.internal
— is a name nobody will ever ask for, standing beside the machines and looking as real.

Found composing the resolver's first assignment on a live machine, before pushing it.
hq issue 111.
2026-09-24 01:31:11 +02:00
jschoubben 6bf42025e1 Merge pull request 'Give the resolver the mesh's suffix as a local domain and a module its machine's address' (#52) from convert/dnsmasq-from-hal into main 2026-09-23 23:13:03 +00:00
jschoubben 0d8264ff55 Give the resolver the mesh's suffix as a local domain and a module its machine's address
hal dnsmasq-app conversion, hq 08-connectivity. Converting the resolver from the module it
replaces made it forward what it cannot answer, which is what the predecessor's does, and
that found two things the controller did not say.

A resolver that forwards must not send a mesh name it does not know upstream: the
`node-zones` fact now carries `local=/<suffix>/` beside the wildcards, written here rather
than in the daemon's configuration because the suffix is the mesh's choice and this file is
the one place the mesh writes what it chose. The default lives in one helper now instead of
being spelled in two functions.

The predecessor points the container runtime's `dns` at the machine's own tunnel address —
a container cannot reach the machine's loopback. A module writing that key needs the
address, and `${machine:at}` is the machine's name; a runtime's resolver list cannot be a
name it would need that resolver to look up. So a module may say `${machine:address}`: what
`at` resolves to, read from the same names the hosts file and the wildcards are written
from, absent — and refused — off the network like `at` is.

The `mesh-resolver` and `resolver-data` constants go: nothing provided or consumed either,
the fact and `mesh-addressing` are the mechanism, and a requirement nothing provides is
refused at resolution.

Tests: the catalogue's dnsmasq, resolv-conf and resolved-split-dns manifests are parsed
and composed as a machine would receive them — fixed upstreams, no-resolv, 127.0.0.1, the
machines file, the runtime's key, the pair that decides what a machine asks refused on one
node; and on a real mesh the resolver's machines file is composed with a wildcard per
machine on the network and composed again without one that left, mirroring the hosts fact.
2026-09-24 01:10:15 +02:00
jschoubben 6073e94a4f Merge pull request 'Adopt the predecessor's tunnel in place: its range, its address, its peers (hq ADR 0105)' (#49) from feat/adopt-the-tunnel into main 2026-09-23 22:38:31 +00:00
jschoubben 4566c5c9aa Adopt the tunnel as a mesh fact, refuse a mismatched takeover, and rekey after enrolment
Review of the ADR 0105 build (hq ADR 0105). Four things it got wrong and one
path it lacked:

- A predecessor spoke's tunnel names one peer, the hub, routed the whole
  range; recording refused it and the whole enrolment failed. Range-routed
  peers are skipped now — only the hub's peers are ever carried.
- The range and the carried peers were conditions on the node being adopted,
  so converging the hub would have renumbered the mesh and dropped the peers
  still reaching it. They are facts of the tunnel record now, mode aside; the
  takeover alone is declared to an adopted node. Converging the hub is refused
  while a carried peer has not enrolled, naming it.
- A push composed a takeover for a hub whose address or endpoint disagreed
  with the tunnel, which would have the host stop the found interface and
  raise the mesh's where no peer listens. The graph refuses to compose it,
  naming both and the placement that fixes it.
- The host's account said taken or not; "found down and the mesh's not up"
  read as not taken. Three states now, and an account on every takeover.
- A hub that enrolled before this feature holds a key of its own, and
  re-enrolling would rotate every key the mesh sealed credentials to. A node
  now rekeys in a report, signed with its identity key over the key it
  leaves, the key it takes and the tunnel; the mesh verifies against the live
  key, refuses a stale or foreign proof, records key and tunnel, and moves a
  hub to the tunnel's address. `overlay show` names the path for a hub that
  found no tunnel.

Also: a carried IPv6 peer is routed /128, and identity.ForTest exists so the
link can be tested against a real identity store.
2026-09-24 00:02:07 +02:00
jschoubben 7ef7669c0c Merge pull request 'An address is read from the node's settings where it is used, never recorded with a port (hq issue 102)' (#50) from fix/addresses-follow-the-node into main 2026-09-23 21:55:03 +00:00
jschoubben cdd3638312 The control plane's own manifest says where the node put the store and the broker
Beside each sealed connection genesis wrote, the port this machine put the
seat's holder at: `${seat:mesh-store:5432}` for the three stores,
`${seat:mesh-broker:…}` for the bus, the plain AMQP port and the management API.
Filled from the node's settings when the control plane composes its own
declaration; empty — the sealed value stands — when the mesh has nothing to add.

On its own, after the commit before it is built and running: a control plane
that does not know the placeholder passes it through as the value, and this
manifest is composed by whatever control plane is running when it is pushed.
The reader ignores an unfilled placeholder either way, and a test holds it to
ignoring exactly what this manifest says.

novox/hq 04-ISSUES/102
2026-09-23 23:49:55 +02:00
jschoubben e07b56ce43 An address is read from the node's settings where it is used, never recorded with a port
Three readers did not follow a moved foundation port (novox/hq 04-ISSUES/102),
and each took the control-node down in its own way: the control plane's own
store and broker connections, sealed at genesis with the port inside; and every
build the mesh ever recorded, kept as `<registry>:<port>/<module>/<artifact>@…`.

The control plane cannot open its own sealed connections to move a port, and it
cannot bind the store as a consumer would — a binding mints a credential. So its
settings get a third twin, `NAME_PORT`, read on top of the sealed value by the
store, the broker, the management API and the bus connection, and filled into
its container by a placeholder that names a seat, `${seat:mesh-store:5432}`,
from the node's given or mesh-assigned ports — never the manifest's number, and
empty when the mesh has nothing to add, so what genesis wrote stands. A value
that is still a placeholder is nothing said, aloud: the manifest naming it lands
in the next commit, once every control plane that composes it knows it.

A build is now recorded by digest and path — `artifact-store://<module>/<artifact>@…`
— and the store's address is composed in where a reference is used: the
declaration, the trust file, the bases a build is handed, a replay to the
catalogue. Over the network as `<node>.internal:<port>`; on the store's own node
before any network exists — every genesis push before its "network" step — by
loopback. A reference recorded before this, with an address, is re-routed the
same way when the mesh built it. The trust file and every provider's address
come from one derivation: the node's given port, over the mesh's assignment,
over the manifest's number.

novox/hq 04-ISSUES/102
2026-09-23 23:49:31 +02:00
jschoubben 1b5ccf4165 Merge pull request 'The artifact store's seat is one per mesh, and the test says so from the catalogue' (#51) from fix/the-artifact-store-is-one-per-mesh into main 2026-09-23 21:42:33 +00:00
jschoubben 26690d89f1 The artifact store's seat is one per mesh, and the test says so from the catalogue
Review of the registry work found the seat node-scoped: a second `distribution` on another
machine resolved cleanly there, and only afterwards did the mesh notice `artifact-store`
offered by two nodes, with every consumer elsewhere refusing to choose. A node-scoped
requirement with one candidate installs that candidate, so anything that wanted the store
beside it would have raised a fresh, empty store on the wrong machine first.

The claim is mesh-scoped in mesh-catalog now; this holds the catalogue's manifest to it —
a second store anywhere is refused by name, where it is assigned.
2026-09-23 23:40:53 +02:00
jschoubben 3c836f0abb Adopt the predecessor's tunnel in place: its range, its address, its peers
On an adopted hub the private network takes over the tunnel it finds rather
than running beside it (hq ADR 0105): two tunnels leave the mesh's unreachable
through the provider's filter, so no machine can ever join.

The node presents the found tunnel when it enrols, under the key it took as
its own; the inventory records it (node.tunnel, tunnel_peer — migration 0031)
and the mesh composes from it: the overlay's range is the adopted tunnel's,
the hub is placed at the tunnel's address on the tunnel's port, and every
peer the tunnel had is carried in the hub's peer list as a peer of the
tunnel, not a node of the mesh, until a node enrols with that key — which
then keeps the address the tunnel had for it. A fresh node never gets an
address the tunnel holds. The hub's declaration tells the host which unit to
take over; the host's account of carrying it is recorded and shown.

Every reader of the range follows the setting; nothing stores it. A found
tunnel under another key is recorded and not adopted, so ADR 0100's
non-overlap rule keeps applying where a tunnel is left running beside the
mesh's. A lab bed and test skeleton for "How it is checked" are under lab/.
2026-09-23 23:26:34 +02:00
jschoubben 8fb32d7ee0 Merge pull request 'A node may move a port a module publishes as a mapping's machine side' (#47) from fix/move-a-published-machine-port into main 2026-09-23 00:33:06 +00:00
jschoubben 7d6f37af54 One entry per mapping, under the name the module itself uses
Review found the first pass aliased its answer under both ends of a mapping, which is
wrong wherever two mappings share a number: the alias lands on a key belonging to another
mapping, the later write wins, and the filter and the container then disagree — the very
fault this change exists to close. Two reproduced cases: a module publishing 8080:80 beside
9090:8080 had an explicit setting silently overwritten; a module publishing 4001:80 beside
4002:80 composed both containers onto one machine port, where before it was safely refused.

Now a mapping's answer is filed once, under the end the module names in its listens — the
number the plan, the filter, the openings, the guard and the consumer all ask for — and a
key that names two mappings is refused in the same words as a setting that does.

Also: the guard assertion in the end-to-end test failed open when the resource was absent;
the plan-mirroring helper now says it stands in only where the plan does not allocate, and
the assertions it feeds are narrowed to the port under test.
2026-09-23 02:32:28 +02:00
jschoubben 58644fd282 A node may move a port a module publishes as a mapping's machine side
A module publishing `2222:22` — the machine's own ssh daemon holds 22, so
the module takes 2222 and says so in `listens` — could not be moved. The
setting was read against the last segment of each mapping alone, so the
number the module uses everywhere else was refused as a port it does not
publish, and the node's every push failed for as long as the setting was
stored. The one key that was accepted, the container's own port, was then
read only when the container's mapping was rewritten: the mapping moved
and the ports map, the filter, the adopted node's openings, its guard and
what a consumer is told all stayed on the number the software had left.

Either end of a mapping now names it, and a given port comes back under
both, so every reader finds the same number under the key it holds.
Ambiguity is refused where it is real — one number naming two different
mappings, or the two ends of one mapping given two different numbers.
2026-09-23 02:08:35 +02:00
jschoubben 91a41b7d20 Merge pull request 'A container's environment follows a moved port, as a file already does (hq issue 088)' (#46) from feat/forge-address into main 2026-09-22 23:30:06 +02:00
jschoubben f0049190d7 Prove the untouched environment is the same map, not an equal one 2026-09-22 23:29:28 +02:00
jschoubben 7352c846dd A module is told its port in a container's environment too (hq issue 088)
${port:…} answered only inside a file's content, and the one place a module
routinely writes its own address is a container's `env` — where the literal is
wrong on every node whose assignment differs from the manifest's number, and
wrong again on a node given that port as a setting (ADR 0100). Nothing checked
it: the value is a string like any other, and it fails at runtime, on one node.

Filled by the control plane, like a bound value: a port is not secret, so there
is nothing for the host to be the only witness of and it learns no new field.
That is the line ADR 0086 draws — its objection is to a secret being in an
environment at all, not to who fills one in — so a port crosses it and a
credential still does not. Same guard as before: a port the module never said it
listens on is refused, now naming the container and the variable.

The env map is the catalogue's, shared by every node running the module, and the
resource around it is a shallow copy, so a filled value goes into a fresh map —
otherwise the first node composed writes its own port into the manifest and
every node after it is told that one.

Inert on the catalogue as it stands: ${port:…} is written in one other place in
it, a file. Renamed off _files, which this no longer is.
2026-09-22 22:36:28 +02:00
jschoubben 45425702d5 Merge pull request 'Hold the forge to being an ordinary provider, in tests (hq issue 085)' (#45) from feat/packages-port into main 2026-09-22 21:59:52 +02:00
jschoubben 729537e745 Tie the builder's carried binding to what the forge serves, and hold at shut
The builder carries a binding because at genesis nothing provides
`package-registry` to resolve one from; once the forge is a module the same
consumer is told what the forge serves. Nothing held the two to the same number,
so the catalogue could drift into dialling one port before the forge is assigned
and another after.

And `at` is now protected, for the reason it had to be: a setting that moves it
points the builder, and the registry password it sends, at a host somebody else
chose.

novox/hq 04-ISSUES/085
2026-09-22 21:56:51 +02:00
jschoubben 0e413e3e7a Hold the forge's port to the same rule as every other provider's
The package registry was the one foundation port not resolved from what its
module serves. Nothing in the controller had to change for it — `ports` on the
forge moves its container, what it serves and what consumers are told, and the
builder's carried binding is settable like any other mergeable file — but
nothing said so, which is how it came to be special in the first place.

Two tests over the catalogue's own manifests: the forge's port is given on a
node and reaches what it serves, and the builder's carried binding takes the
port from the node while keeping who the binding is with.

novox/hq 04-ISSUES/085
2026-09-22 21:40:08 +02:00
jschoubben 252186d90c Merge pull request 'Adoption mode: a node in use is adopted before it is converged (hq ADR 0100–0103)' (#44) from feat/adoption-mode into main 2026-09-22 21:01:51 +02:00
jschoubben fad8b30e43 Say that the guard names the runtime's bridges where the filter names their addresses (hq ADR 0103) 2026-09-22 19:51:10 +02:00
jschoubben 1096299e06 Hold a converged declaration to the bytes main sends, captured from it, rather than to re-marshalling itself (hq ADR 0100) 2026-09-22 19:51:10 +02:00
jschoubben 379f459498 Hold the filter module to reloading its rules and restarting only on its units (hq ADR 0102) 2026-09-22 19:47:43 +02:00
jschoubben d05a5e87af Hold the node while an assignment is recorded, so none lands between a preview and the flip (hq ADR 0100) 2026-09-22 19:47:23 +02:00
jschoubben 01814854b8 Refuse the flip on an account naming nothing reachable, and mark such an account partial in the preview (hq ADR 0100) 2026-09-22 19:47:23 +02:00
jschoubben 2cf8739a84 Clear the whole account an adopted node gave when it converges (hq ADR 0100) 2026-09-22 19:45:35 +02:00
jschoubben 87ecc9326e Refuse a port given for the whole mesh where it is set, not at every node's composition (hq ADR 0100) 2026-09-22 19:45:35 +02:00
jschoubben 0f3eedd163 Do not guard a port this node is told to open to everyone (hq ADR 0103) 2026-09-22 19:44:35 +02:00
jschoubben dd6aad4a2f Give a machine port only to a port a module's container publishes, which is the only one the mesh can move (hq ADR 0038) 2026-09-22 19:44:35 +02:00
jschoubben 65187d4de0 Wait for a held node without pinning a pool connection, and give up after a bounded wait naming it (hq ADR 0100) 2026-09-22 18:31:10 +02:00
jschoubben 2e6b9d30bd Give a cascade round's hold back on every return, a body that cannot be marshalled included (hq ADR 0100) 2026-09-22 18:31:00 +02:00
jschoubben cc47330884 Reload the guard on its table rather than restart it, so a change leaves no port unguarded (hq ADR 0103) 2026-09-22 18:27:27 +02:00
jschoubben bfe4991dd7 Name every held kind of each module the flip takes in the converge preview and its digest (hq ADR 0103) 2026-09-22 18:27:19 +02:00
jschoubben 689c2c6d33 Hold a node from composing to sending, so a push composed before converge or adopt is never sent after it (hq ADR 0100) 2026-09-22 18:10:04 +02:00
jschoubben 1eff586a40 Hold the filter module to replacing the stock unit's flushing stop (hq ADR 0100) 2026-09-22 18:06:15 +02:00
jschoubben 4bb19c9e40 Give a machine port one holder: refuse ssh's, another module's and a doubled one, and release the assignment a given port replaces (hq ADR 0100) 2026-09-22 18:05:31 +02:00
jschoubben 9280513afa Refuse token issue --adopted for a converged node and point to adopt, rather than flip it quietly (hq ADR 0100) 2026-09-22 18:03:45 +02:00
jschoubben 41c300eae0 Name every kind of an untaken module's resources in the adoption envelope, not only files and containers (hq ADR 0103) 2026-09-22 18:03:03 +02:00
jschoubben c91fe1a6eb Converge only on the preview the operator saw, named by its digest, and never on an account older than 15 minutes (hq ADR 0100) 2026-09-22 18:02:30 +02:00
jschoubben ba0f44a36d Say in the converge preview that routed traffic is not previewed and is dropped unless declared (hq ADR 0100) 2026-09-22 18:01:04 +02:00
jschoubben 3dc7d0e386 Preview ssh as the derived filter admits it, and say a narrowing to the private network closes (hq ADR 0100) 2026-09-22 18:00:53 +02:00
jschoubben ade6b2bfb6 Declare nftables before the guard's table, so a node joining adopted without nft can load it (hq ADR 0103) 2026-09-22 17:59:32 +02:00
jschoubben a2dfaaf4d1 Guard only packets addressed to this machine, and order the guard's unit before the network and against shutdown (hq ADR 0103) 2026-09-22 17:59:09 +02:00
jschoubben a82bfb41f2 Leave an IPv6 loopback mapping out of what a container publishes, reading ports from the end (hq ADR 0100) 2026-09-22 17:58:50 +02:00
jschoubben 8db66e9532 Derive the guard from taken modules only: their published private-network ports and their manifests' guards (hq ADR 0103) 2026-09-22 17:58:37 +02:00
jschoubben 28b7fb81ba Write the registry's trust into the runtime's file and reload the runtime instead of restarting it; prefix reload-on like restart-on (hq ADR 0102) 2026-09-22 17:51:38 +02:00
jschoubben c93128d82f Hold the catalogue's store, broker and filter manifests to what adoption needs of them (hq ADR 0100) 2026-09-22 17:33:09 +02:00
jschoubben dbf62b5212 Read the foundation's ports from each node's settings, wherever a port is used (hq ADR 0100) 2026-09-22 17:31:07 +02:00
jschoubben 1ea84f8b8f Take a module, converge a node after a preview, and return it to adopted, from the command line and the API (hq ADR 0100) 2026-09-22 17:27:35 +02:00
jschoubben 28894fa5bd Keep what an adopted node reports holding, the firewall it found and what is reachable on it (hq ADR 0100) 2026-09-22 17:23:09 +02:00
jschoubben c3b1617693 Declare openings and a refusal-only guard on an adopted node in place of the filter (hq ADR 0100) 2026-09-22 17:21:44 +02:00
jschoubben a86a6c2974 Carry an adopted node's mode and taken modules in every declaration, from one marshaller (hq ADR 0100) 2026-09-22 17:17:54 +02:00
jschoubben 1c32af6a22 Record whether a node is adopted and which modules were taken on it (hq ADR 0100) 2026-09-22 17:14:05 +02:00
jschoubben 0a39b7df82 Merge pull request 'Nothing the control queue carries is lost while the store restarts (issue 083)' (#43) from multiple-fixes into main 2026-09-22 14:42:57 +02:00
jschoubben 4f3b4e6014 Review of 083: an upgrade handled during shutdown is left for the broker; an enrolment cut short by shutdown is told to try again; a refused redelivery says what it probably is 2026-09-22 14:39:15 +02:00
jschoubben 32b8af6a9b The enrolment proof's message has a known answer, repeated in the host's test 2026-09-22 14:33:33 +02:00
jschoubben 4567fa666c Review of 083: finishing an enrolment whose token was spent takes proof of the key's private half, a live lease and a first delivery — a public key alone cannot replay a spent token; shutdown leaves held messages for the broker; identical builds supersede; what is held leaves room in the prefetch 2026-09-22 14:33:13 +02:00
jschoubben a3b7e830c8 Review of 083: a message the store cannot take is held and retried on a ticker, not slept on, so enrolments are answered meanwhile; a newer one per subject supersedes; the password is replaced after the spend; the same presenter may finish after a lost answer; upgrades retry only on the store 2026-09-22 14:23:03 +02:00
jschoubben 1a41b88ed3 Nothing the control queue carries is lost while the store restarts: an enrolment claims its token and spends it last, and is asked to try again; build results, upgrades and catch-ups are handed back, bounded (novox/hq issue 083) 2026-09-22 14:06:28 +02:00
jschoubben b9cdb96a90 Merge pull request 'A report that arrives while the store restarts is kept, not lost (issue 082)' (#42) from multiple-fixes into main 2026-09-22 13:53:13 +02:00
jschoubben 3fd0c37d22 Review of 082: a store killed mid-conversation is an outage and a wrong password is not; one report holds the queue at most two minutes, then is let go loudly; shutdown does not wait out the pause 2026-09-22 13:34:04 +02:00
jschoubben 047830a6b5 A report the store cannot take yet is handed back to the broker, not acknowledged and lost (novox/hq issue 082) 2026-09-22 13:25:13 +02:00
jschoubben ff26ea959d Merge pull request 'Every machine was named twice over; only a machine on the private network is named (issue 079)' (#41) from multiple-fixes into main 2026-09-22 02:19:13 +02:00
jschoubben 5150c0a0f8 One predicate for the private network: the filter's accept set is the set the names and the resolver mean; the catalogue is read once for the three mesh-wide questions 2026-09-22 02:04:28 +02:00
jschoubben 6a64aba506 Only a machine on the private network is named: the names follow the resolver's rule, one predicate for both (novox/hq issue 079) 2026-09-22 01:41:13 +02:00
jschoubben 8152665298 The facts write the suffix the control plane composed the names with, handed down rather than written twice (review of issue 079); the fixture is keyed as production keys it 2026-09-22 01:27:50 +02:00
jschoubben b529c49cff A machine's names are not suffixed twice: the facts take the internal names the control plane hands them (novox/hq issue 079) 2026-09-22 01:13:28 +02:00
jschoubben 713b43799f Merge pull request 'Multiple fixes: a run-once step may name what it reads (ADR 0099); secret accept refuses an undeclared name (078)' (#40) from multiple-fixes into main 2026-09-21 23:58:47 +02:00
jschoubben 0d1f2a4152 Review: module issue's pre-check factored and tested; a pair delivery for a requirement the module keeps no secret for is refused; builder issue's usage says the module comes first 2026-09-21 23:58:36 +02:00
jschoubben 66332705cd module issue refuses a module with no broker own secret before making the account (issue 078) 2026-09-21 23:46:28 +02:00
jschoubben 35c5c2bb9b secret accept is refused for a name the module does not declare, and a pair delivery for a requirement or local it has not got (novox/hq issue 078) 2026-09-21 23:31:06 +02:00
jschoubben e4da83496f A run-once step may name what it reads: the pair run-once + restart-on is no longer refused (novox/hq ADR 0099) 2026-09-21 23:31:06 +02:00
jschoubben 52d39f0740 Merge pull request 'The control plane's recipe declares its base, and an undeclared FROM is refused (ADR 0097 live)' (#39) from multiple-fixes into main 2026-09-21 22:58:22 +02:00
jschoubben b3486270d1 The control plane's recipe starts FROM the base its manifest declares, and an undeclared base is refused
The mesh's own images declare theirs now, so the refusal ADR 0097 deferred is live.
The builder's own image and the examples take arguments with defaults; make builds
them, not the mesh.
2026-09-21 22:16:10 +02:00
jschoubben d7dab9c09f Merge pull request 'Multiple fixes: several secrets per module (069), ask a module's tool (049), copy an upstream image (046), declare a vendor image (064)' (#38) from multiple-fixes into main 2026-09-21 21:05:11 +02:00
jschoubben 9f3790dcda Review: one local name is still a local name; a local name is unique; recovery knows it; recipes read as instructions; a tag before a digest; ask fails at once when nothing serves
A secrets object with one local name delivered no file. Two requirements could share
a local name. secret recover and the export could not tell two locals apart. The
recipe check missed continued lines and read heredoc bodies as bases. repo:tag@digest
kept the tag in the repository. ask now publishes mandatory, so a tool nothing serves
is said at once rather than after the wait.
2026-09-21 21:03:22 +02:00
jschoubben e81f352979 An undeclared COPY --from is refused; an undeclared FROM is said, not yet refused
The mesh's own images start FROM a public base — the control plane's, the builder's,
the tool runtime's — and refusing those refuses genesis. They declare their bases
next; until then the base is named every build, with the remedy.
2026-09-21 20:48:00 +02:00
jschoubben 6f6e1244d4 A build declares the vendor image it stands on, and a recipe fetches nothing undeclared
build.on takes {arg, image@sha256:…} beside {arg, module, artifact}: the image is
copied into the mesh's registry before the build (ADR 0096) and the recipe reads the
copy from the argument. A FROM or COPY --from naming a registry image the manifest
did not declare is refused before the build, naming it and the remedy; stages,
declared arguments and scratch are not fetches (novox/hq 04-ISSUES/064, ADR 0097).
2026-09-21 20:45:36 +02:00
jschoubben 412599de9a An upstream image is copied between registries, never through a machine's image store
A published image is an index over several architectures; pulled, the runtime's
store keeps the index and refuses to push one platform out of it. The builder now
reads the index and every manifest it names over the registry API, with the
anonymous bearer token the public hub hands out, moves each blob by digest into the
mesh's registry, puts the manifests and then the index under the module's repository,
and pins the index. Genesis, with no registry to copy into, keeps the pull
(novox/hq 04-ISSUES/046, ADR 0096).
2026-09-21 20:42:30 +02:00
jschoubben 5049d201c6 A provider keeps one grant file per holder, with the local name in its id and path
The lab's vault refused a declaration naming two files with one identity: the
two secrets of one consumer. The holder's suffix is in the resource id and the path now.
2026-09-21 20:41:19 +02:00
jschoubben 3e5c010c5e A need is kept once per provision, consumer, local name and provider
Two consumers of one same-node provision produced two raw needs and, fanned out per
consumer, four — the same credential twice for each. Harmless, since a pair is one
row however often it is asked for, and wrong all the same.
2026-09-21 20:38:01 +02:00
jschoubben 76773dfbf5 Several secrets expand where the consumer is known, not on the first module to mention the provision
The lab's two-secrets consumer was given one credential and no file: the expansion
ran on the resolver's walk over names, on whichever module mentioned the provision
first, and the per-consumer pass copied that. It expands in that pass now, and a
test has two consumers of one provision, one keeping one file and one keeping two.
2026-09-21 20:36:19 +02:00
jschoubben 973cda5d76 ask: the control plane calls a module's tool and prints its answer
A module serves tools under an account scoped to exactly that, and nothing else in
the mesh held an account that could ask one. The control plane does: ask publishes
on the RPC exchange with a private reply queue bound under its own name, checks the
correlation, prints the answer, and exits non-zero for a tool that answered with an
error or a module that never answered (novox/hq 04-ISSUES/049, ADR 0095).
2026-09-21 20:34:03 +02:00
jschoubben 6ae4ae1dba A module may hold several secrets from one provider, each a pair of its own
secrets: maps a requirement to several files under local names. Each local name is
its own need, its own pair credential (the pair is keyed on it: migration 0027),
its own file on the consumer, its own holder at the provider (the identity with the
local name after it) and rotates apart from the others. The plain shape is
unchanged and every existing row is the credential it was (novox/hq 04-ISSUES/069,
ADR 0094).
2026-09-21 20:28:16 +02:00
122 changed files with 13609 additions and 761 deletions
+2 -1
View File
@@ -1,3 +1,4 @@
ARG GO_BASE=golang:1.25-alpine
# The control plane's image.
#
# novox/hq ADR 0006: this image is pinned by digest in the bundle the host carries, fetched on a
@@ -11,7 +12,7 @@
# (novox/hq ADR 0004). Adding them "just in case" would put a trust store in the one image whose
# whole argument is that it contains nothing to reason about.
FROM golang:1.25-alpine AS build
FROM ${GO_BASE} AS build
WORKDIR /src
# Dependencies first, so a change to the source does not refetch them.
+4 -2
View File
@@ -1,17 +1,19 @@
ARG ALPINE_BASE=alpine:3
ARG GO_BASE=golang:1.25-alpine
# The builder, as a module ships one.
#
# Not FROM scratch, unlike the control plane: this one runs git and a container client, so it
# needs a filesystem with them on it. That is the honest cost of a machine whose job is to build —
# and it is why building is a MODULE on a machine that has a runtime rather than something the
# control plane does (novox/hq ADR 0005).
FROM golang:1.25-alpine AS build
FROM ${GO_BASE} AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 go build -trimpath -ldflags '-s -w' -o /mesh-builder ./cmd/mesh-builder
FROM alpine:3
FROM ${ALPINE_BASE}
# git to clone what it is asked to build, and the docker client to build and push it. The daemon
# is the machine's, reached through its socket — a build machine shares the runtime it was given
# rather than running one inside itself.
+12
View File
@@ -39,6 +39,13 @@ import (
// It costs a resolution per machine. Assignment is a person typing a command, and being told which
// machines this just blocked is worth more than the milliseconds.
func assign(ctx context.Context, open *stores, node, module string) (string, error) {
// Held while it is recorded, so it cannot land between a converge's preview and its flip and
// be taken without ever having been previewed (novox/hq ADR 0100).
ctx, release, err := holdNodes(ctx, open, []string{node})
if err != nil {
return "", err
}
defer release()
if err := open.inventory.Assign(ctx, node, module); err != nil {
return "", err
}
@@ -71,6 +78,11 @@ func assign(ctx context.Context, open *stores, node, module string) (string, err
// module off one machine is the ordinary way to stop providing something to another, and nothing
// about the command's own output would ever have said so.
func unassign(ctx context.Context, open *stores, node, module string) (string, error) {
ctx, release, err := holdNodes(ctx, open, []string{node})
if err != nil {
return "", err
}
defer release()
if err := open.inventory.Unassign(ctx, node, module); err != nil {
return "", err
}
+335
View File
@@ -0,0 +1,335 @@
package main
import (
"encoding/json"
"os"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/licences"
"github.com/novox/mesh-controller/internal/link"
)
// An address is read from the node's settings where it is used, never recorded with a port
// (novox/hq 04-ISSUES/102). Three readers did not follow the setting; each is held to it here.
var aDigest = "sha256:" + strings.Repeat("e", 64)
// **A build is recorded by digest and path**, whatever address the builder pushed to — and only
// what the build made is rewritten: an image the module runs from elsewhere is left where it says.
func TestABuildIsRecordedWithoutTheStoresAddress(t *testing.T) {
manifest, _ := json.Marshal(map[string]any{
"module": "gitea", "version": "1",
"resources": []map[string]any{
{"id": "server", "type": "container", "name": "mesh-gitea",
"image": "anchor.internal:5100/gitea/server@" + aDigest},
{"id": "config", "type": "archive", "path": "/etc/gitea", "digest": aDigest,
"source": "http://anchor.internal:5100/v2/gitea/config/blobs/" + aDigest},
{"id": "cache", "type": "container", "name": "mesh-gitea-cache",
"image": "valkey/valkey@" + aDigest},
},
})
kept := buildFrom(link.BuildResult{
ID: "b1", Repository: "https://forge.example/gitea.git", Commit: "abc", On: "laptop",
Manifest: manifest,
Made: []link.MadeArtifact{
{Name: "server", Kind: "image", Reference: "anchor.internal:5100/gitea/server@" + aDigest},
{Name: "config", Kind: "archive", Reference: "http://anchor.internal:5100/v2/gitea/config/blobs/" + aDigest},
},
Against: []string{"anchor.internal:5100/mesh-tools/runtime@" + aDigest},
})
if kept.Module != "gitea" {
t.Fatalf("the module was not read from the recorded manifest: %q", kept.Module)
}
if kept.Made[0].Reference != catalogue.ArtifactStoreScheme+"gitea/server@"+aDigest {
t.Errorf("the image is recorded as %q, address and all", kept.Made[0].Reference)
}
if kept.Made[1].Reference != catalogue.ArtifactStoreScheme+"gitea/config/blobs/"+aDigest {
t.Errorf("the archive is recorded as %q, address and all", kept.Made[1].Reference)
}
recorded, err := catalogue.ParseManifest(kept.Manifest)
if err != nil {
t.Fatal(err)
}
if got := recorded.Resources[0]["image"]; got != catalogue.ArtifactStoreScheme+"gitea/server@"+aDigest {
t.Errorf("the recorded manifest's image is %v", got)
}
if got := recorded.Resources[1]["source"]; got != catalogue.ArtifactStoreScheme+"gitea/config/blobs/"+aDigest {
t.Errorf("the recorded manifest's archive is %v", got)
}
if got := recorded.Resources[2]["image"]; got != "valkey/valkey@"+aDigest {
t.Errorf("an image the build did not make was rewritten: %v", got)
}
if strings.Contains(string(kept.Manifest), "anchor.internal:5100") {
t.Errorf("the recorded manifest still carries the store's address:\n%s", kept.Manifest)
}
if kept.Against[0] != "anchor.internal:5100/mesh-tools/runtime@"+aDigest {
t.Errorf("what the build stood on was rewritten: %v", kept.Against)
}
}
// aStore is a module offering the artifact store on 5000, published the long way as the
// distribution module does, so a node may be given another number for it.
func aStore() catalogue.Manifest {
return catalogue.Manifest{Module: "distribution", Version: "1",
Provides: []catalogue.Offer{{Name: catalogue.ArtifactStoreProvision, Scope: catalogue.ScopeMesh}},
Serves: map[string]map[string]any{catalogue.ArtifactStoreProvision: {"port": float64(5000)}},
Listens: []catalogue.Listening{{Port: 5000, From: catalogue.FromMesh}},
Resources: []map[string]any{{"id": "store", "type": "container", "name": "mesh-registry",
"ports": []any{"5000:5000"}, "image": "registry@" + aDigest}}}
}
// **The trust a machine writes for the store, and the address every built image is fetched
// through, say the port the node gave the store** — not the catalogue's number.
func TestTheRegistryTrustAndEveryImageFollowThePortTheNodeGaveTheStore(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
register(t, open, aStore())
if _, err := assign(ctx, open, "anchor", "distribution"); err != nil {
t.Fatal(err)
}
if err := open.inventory.SetSettings(ctx, "anchor", "distribution",
map[string]any{catalogue.PortsSetting: map[string]any{"5000": 5101}}); err != nil {
t.Fatal(err)
}
// A module the mesh built, recorded by digest and path, running on the other machine.
if err := open.inventory.RecordBuild(ctx, inventory.Build{
ID: "b1", Repository: "r", Module: "app", Commit: "abc",
Made: []inventory.Artifact{{Name: "server", Kind: "image",
Reference: catalogue.ArtifactStoreScheme + "app/server@" + aDigest}},
}); err != nil {
t.Fatal(err)
}
register(t, open, catalogue.Manifest{Module: "app", Version: "1",
Resources: []map[string]any{{"id": "server", "type": "container", "name": "mesh-app",
"image": catalogue.ArtifactStoreScheme + "app/server@" + aDigest}}})
if _, err := assign(ctx, open, "laptop", "app"); err != nil {
t.Fatal(err)
}
on := map[string]bool{"anchor": true, "laptop": true}
node, port, found, err := artifactStoreOnNetwork(ctx, open.inventory, on)
if err != nil || !found || node != "anchor" || port != "5101" {
t.Fatalf("the store is found on %q:%q (%v, %v); the node put it on 5101", node, port, found, err)
}
var trust string
for _, r := range composed(t, open, "laptop").Resources {
if r["path"] == "/etc/docker/daemon.json" {
trust, _ = r["content"].(string)
}
if r["id"] == "app.server" && r["image"] != "anchor.internal:5101/app/server@"+aDigest {
t.Errorf("the image the mesh built is fetched as %v", r["image"])
}
}
if !strings.Contains(trust, "anchor.internal:5101") || strings.Contains(trust, ":5000") {
t.Fatalf("the runtime is told to trust %q; the node put the store on 5101", trust)
}
// And a replay to the catalogue says where the store is now.
announced, err := following{open}.Announceable(ctx)
if err != nil {
t.Fatal(err)
}
if len(announced) != 1 || announced[0].Made[0].Reference != "anchor.internal:5101/app/server@"+aDigest {
t.Fatalf("the replay announces %+v", announced)
}
}
// **The control plane's own connections say the port the node gave the store and the broker.**
//
// Composed from the control plane's own manifest against a real inventory: the store's module is
// given 6852 on this node the way genesis or an operator gives it, and the control plane's
// container is told so beside the sealed connection genesis wrote.
func TestTheControlPlaneIsToldWhereTheNodePutTheStoreAndTheBroker(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
raw, err := os.ReadFile("../../module.json")
if err != nil {
t.Fatal(err)
}
m, err := catalogue.ParseManifest(raw)
if err != nil {
t.Fatal(err)
}
control, err := withSeatPorts(m).Resolve([]catalogue.Built{{Name: "server", Kind: catalogue.ArtifactImage,
Reference: "registry.example/control@" + aDigest}})
if err != nil {
t.Fatal(err)
}
register(t, open, control)
register(t, open, catalogue.Manifest{Module: "postgres", Version: "1",
Claims: []catalogue.Claim{{Name: "mesh-store", Scope: catalogue.ScopeMesh}},
Listens: []catalogue.Listening{{Port: 5432, From: catalogue.FromMesh}},
Resources: []map[string]any{{"id": "server", "type": "container", "name": "mesh-store",
"ports": []any{"5432:5432"}, "image": "pg@" + aDigest}}})
register(t, open, catalogue.Manifest{Module: "lavinmq", Version: "1",
Claims: []catalogue.Claim{{Name: "mesh-broker", Scope: catalogue.ScopeMesh}},
Listens: []catalogue.Listening{{Port: 5671, From: catalogue.FromMesh},
{Port: 5672, From: catalogue.FromMesh}},
Guards: []int{15672},
Resources: []map[string]any{{"id": "server", "type": "container", "name": "mesh-broker",
"ports": []any{"5671:5671", "5672:5672", "127.0.0.1:15672:15672"}, "image": "mq@" + aDigest}}})
if _, err := assign(ctx, open, "anchor", "mesh-controller"); err != nil {
t.Fatal(err)
}
// The store's module is registered and given its port, and NOT assigned: the state genesis
// leaves a given-port node in before the foundation is adopted as modules (04-ISSUES/085).
if err := open.inventory.SetSettings(ctx, "anchor", "postgres",
map[string]any{catalogue.PortsSetting: map[string]any{"5432": 6852}}); err != nil {
t.Fatal(err)
}
if _, err := assign(ctx, open, "anchor", "lavinmq"); err != nil {
t.Fatal(err)
}
if err := open.inventory.SetSettings(ctx, "anchor", "lavinmq",
map[string]any{catalogue.PortsSetting: map[string]any{"5672": 5679}}); err != nil {
t.Fatal(err)
}
var env map[string]any
for _, r := range composed(t, open, "anchor").Resources {
if r["id"] == "mesh-controller.server" {
env, _ = r["env"].(map[string]any)
}
}
if env == nil {
t.Fatal("the control plane's container is not in its own node's declaration")
}
for key, want := range map[string]string{
"MESH_STORE_INVENTORY_PORT": "6852",
"MESH_STORE_IDENTITY_PORT": "6852",
"MESH_STORE_LICENCES_PORT": "6852",
"MESH_BROKER_AMQP_PORT": "5679",
// Neither given nor assigned by the mesh: the manifest's own number is NOT the answer,
// because the sealed value beside it carries the port genesis wrote (finding F3).
"MESH_BROKER_ADDRESS_PORT": "",
"MESH_BROKER_MANAGEMENT_PORT": "",
} {
if env[key] != want {
t.Errorf("the control plane is told %s=%v; the node says %q", key, env[key], want)
}
}
}
// withSeatPorts is the control plane's manifest with the seat placeholders in its environment —
// added here until module.json carries them (the manifest lands one commit after the code that
// fills it, so a control plane one build behind never sees a placeholder it cannot fill).
func withSeatPorts(m catalogue.Manifest) catalogue.Manifest {
seatPorts := map[string]string{
"MESH_STORE_INVENTORY_PORT": "${seat:mesh-store:5432}",
"MESH_STORE_IDENTITY_PORT": "${seat:mesh-store:5432}",
"MESH_STORE_LICENCES_PORT": "${seat:mesh-store:5432}",
"MESH_BROKER_AMQP_PORT": "${seat:mesh-broker:5672}",
"MESH_BROKER_MANAGEMENT_PORT": "${seat:mesh-broker:15672}",
"MESH_BROKER_ADDRESS_PORT": "${seat:mesh-broker:5671}",
}
out := m
out.Resources = nil
for _, r := range m.Resources {
if r["type"] != "container" {
out.Resources = append(out.Resources, r)
continue
}
copied := map[string]any{}
for k, v := range r {
copied[k] = v
}
env := map[string]any{}
if had, ok := r["env"].(map[string]any); ok {
for k, v := range had {
env[k] = v
}
}
for k, v := range seatPorts {
if _, said := env[k]; !said {
env[k] = v
}
}
copied["env"] = env
out.Resources = append(out.Resources, copied)
}
return out
}
// aLoneNode is one capable machine with nothing placed on any network — the control-node during
// genesis, before the "network" step, which is after the store, the broker, the vault and the
// catalogue have each been built and pushed (finding F2).
func aLoneNode(t *testing.T) *stores {
t.Helper()
inventory.ForTest(t)
licences.ForTest(t)
open, err := openStores(t.Context())
if err != nil {
t.Fatal(err)
}
t.Cleanup(open.Close)
for _, m := range provided {
if err := open.inventory.Provide(t.Context(), m); err != nil {
t.Fatal(err)
}
}
record, err := open.inventory.AddNode(t.Context(), "anchor")
if err != nil {
t.Fatal(err)
}
reported, _ := json.Marshal(map[string]any{"capabilities": []map[string]any{
{"name": "container-runtime", "present": true}}})
var profile map[string]any
_ = json.Unmarshal(reported, &profile)
if err := open.inventory.RecordProfile(t.Context(), record.ID, profile); err != nil {
t.Fatal(err)
}
if err := open.inventory.RecordSealingKey(t.Context(), record.ID, aPublicKey(t)); err != nil {
t.Fatal(err)
}
return open
}
// **Before the network exists, the store's own node reaches it by loopback** — never refused,
// never handed the scheme: a genesis pushes the store, the broker, the vault and the catalogue to
// a node on no network, and builds the catalogue on a base it must be able to pull.
func TestOnANodeWithNoNetworkTheStoreIsReachedByLoopback(t *testing.T) {
open := aLoneNode(t)
ctx := t.Context()
register(t, open, aStore())
register(t, open, catalogue.Manifest{Module: "builder", Version: "1",
Requires: []string{catalogue.ArtifactStoreProvision},
Resources: []map[string]any{{"id": "server", "type": "container", "name": "mesh-builder",
"image": "registry.example/mesh-builder@" + aDigest}}})
if err := open.inventory.RecordBuild(ctx, inventory.Build{
ID: "b1", Repository: "r", Module: "postgres", Commit: "abc",
Made: []inventory.Artifact{{Name: "runtime", Kind: "image",
Reference: catalogue.ArtifactStoreScheme + "postgres/runtime@" + aDigest}},
}); err != nil {
t.Fatal(err)
}
register(t, open, catalogue.Manifest{Module: "postgres", Version: "1",
Resources: []map[string]any{{"id": "runtime", "type": "container", "name": "mesh-postgres",
"image": catalogue.ArtifactStoreScheme + "postgres/runtime@" + aDigest}}})
for _, module := range []string{"distribution", "builder", "postgres"} {
if _, err := assign(ctx, open, "anchor", module); err != nil {
t.Fatal(err)
}
}
if err := open.inventory.SetSettings(ctx, "anchor", "distribution",
map[string]any{catalogue.PortsSetting: map[string]any{"5000": 5100}}); err != nil {
t.Fatal(err)
}
var image any
for _, r := range composed(t, open, "anchor").Resources {
if r["id"] == "postgres.runtime" {
image = r["image"]
}
}
if image != "127.0.0.1:5100/postgres/runtime@"+aDigest {
t.Fatalf("on the store's own node, off any network, the image is fetched as %v", image)
}
held := heldBy(ctx)
if got := held["postgres/runtime"]; got != "127.0.0.1:5100/postgres/runtime@"+aDigest {
t.Fatalf("a builder beside the store is handed the base %q", got)
}
}
+534
View File
@@ -0,0 +1,534 @@
package main
import (
"context"
"encoding/json"
"errors"
"net/http"
"reflect"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
"github.com/novox/mesh-controller/internal/overlay"
)
// novox/hq ADR 0100: taking a module is its cutover; converging a node is one act, previewed, and
// refused while a found container is held; returning to adopted keeps what was taken.
// anAdoptedAnchor is aMesh with the anchor adopted, running a served module the predecessor also
// runs and a module with only a file, and a filter module in the catalogue.
func anAdoptedAnchor(t *testing.T) (*stores, *[]string) {
t.Helper()
open := aMesh(t)
ctx := t.Context()
register(t, open, catalogue.Manifest{Module: "hello-web", Version: "1",
Listens: []catalogue.Listening{{Port: 8080, From: catalogue.FromEverywhere}},
Resources: []map[string]any{
{"id": "page", "type": "file", "path": "/var/lib/hello-web/index.html", "content": "hi"},
{"id": "server", "type": "container", "name": "hello-web", "ports": []any{"8080:80"},
"image": "registry.example/hello@sha256:" + strings.Repeat("a", 64)},
}})
register(t, open, catalogue.Manifest{Module: "notes", Version: "1",
Resources: []map[string]any{
{"id": "conf", "type": "file", "path": "/etc/notes.conf", "content": "x"},
}})
register(t, open, catalogue.Manifest{Module: "nftables", Version: "1",
Filtering: &catalogue.Filtering{Into: "/etc/nftables.conf"},
Resources: []map[string]any{{"id": "load", "type": "service", "unit": "mesh-filter.service",
"state": "running", "restart-on": []any{"filtering"}}}})
if err := open.inventory.SetAdopted(ctx, "anchor", true); err != nil {
t.Fatal(err)
}
for _, m := range []string{"hello-web", "notes"} {
if _, err := assign(ctx, open, "anchor", m); err != nil {
t.Fatal(err)
}
}
sent := &[]string{}
saved := sendNodes
sendNodes = func(_ context.Context, _ *stores, names []string) error {
*sent = append(*sent, names...)
return nil
}
t.Cleanup(func() { sendNodes = saved })
return open, sent
}
// reportsHolding has the anchor report, on what it was last sent, holding what is given.
func reportsHolding(t *testing.T, open *stores, held ...link.Held) {
t.Helper()
reportsReaching(t, open, []link.Reach{
{Protocol: "tcp", Address: "0.0.0.0", Port: 22, By: "sshd"},
{Protocol: "tcp", Address: "0.0.0.0", Port: 8080, By: "hello-web", Published: true,
ContainerPort: 80},
{Protocol: "tcp", Address: "0.0.0.0", Port: 5000, By: "predecessor-registry",
Published: true, ContainerPort: 5000},
{Protocol: "tcp", Address: "127.0.0.1", Port: 15672, By: "mesh-broker",
Published: true, ContainerPort: 15672},
}, held...)
}
// reportsReaching has the anchor report, on what it was last sent, what is reachable on it and
// holding what is given.
func reportsReaching(t *testing.T, open *stores, reachable []link.Reach, held ...link.Held) {
t.Helper()
ctx := t.Context()
body, err := composed(t, open, "anchor").Body()
if err != nil {
t.Fatal(err)
}
record, err := open.inventory.NodeByName(ctx, "anchor")
if err != nil {
t.Fatal(err)
}
if err := open.inventory.RecordSent(ctx, record.ID, digestOf(body)); err != nil {
t.Fatal(err)
}
if err := (link.Enrolment{Inventory: open.inventory}).Heard(ctx, link.Report{
Node: "anchor", Applied: []string{"hello-web.x"}, Declared: digestOf(body),
Firewall: "ufw", Held: held, Reachable: reachable,
}); err != nil {
t.Fatal(err)
}
}
var (
heldContainer = link.Held{ID: "hello-web.server", Module: "hello-web", Kind: "container",
Target: "hello-web", Since: time.Now()}
heldFile = link.Held{ID: "notes.conf", Module: "notes", Kind: "file",
Target: "/etc/notes.conf", Since: time.Now(), Kept: "/var/lib/mesh-host/kept/abc-notes.conf"}
)
func TestTakingAModuleNotOnTheNodeIsRefused(t *testing.T) {
open, _ := anAdoptedAnchor(t)
if _, err := take(t.Context(), open, "anchor", "nftables"); !errors.Is(err, inventory.ErrNotAssigned) {
t.Fatalf("taking an unassigned module gave %v", err)
}
if _, err := take(t.Context(), open, "laptop", "network"); !errors.Is(err, inventory.ErrNotAdopted) {
t.Fatalf("taking on a converged node gave %v", err)
}
}
func TestConvergingIsRefusedOnAPreviewThatWouldBeStale(t *testing.T) {
open, sent := anAdoptedAnchor(t)
_, err := converge(t.Context(), open, "anchor", false, "", "")
if err == nil || !strings.Contains(err.Error(), "has not reported") {
t.Fatalf("a node that never reported was previewed: %v", err)
}
if len(*sent) != 0 {
t.Fatal("a refused converge sent something")
}
}
func TestTheFlipIsRefusedWhileAFoundContainerIsHeld(t *testing.T) {
open, sent := anAdoptedAnchor(t)
reportsHolding(t, open, heldContainer, heldFile)
_, err := converge(t.Context(), open, "anchor", true, "", "")
if err == nil || !strings.Contains(err.Error(), "take anchor hello-web once its data has moved") {
t.Fatalf("the flip was not refused while hello-web holds its found container: %v", err)
}
if n, _ := open.inventory.NodeByName(t.Context(), "anchor"); !n.Adopted || len(*sent) != 0 {
t.Fatal("a refused flip changed something")
}
}
func TestTakingNamesWhatItReplaces(t *testing.T) {
open, _ := anAdoptedAnchor(t)
reportsHolding(t, open, heldContainer, heldFile)
said, err := take(t.Context(), open, "anchor", "hello-web")
if err != nil {
t.Fatal(err)
}
if !strings.Contains(said, "container hello-web (hello-web.server)") ||
!strings.Contains(said, "push anchor") {
t.Fatalf("taking did not say what it replaces and what to run:\n%s", said)
}
}
func TestConvergingPreviewsThenChangesAndAdoptingKeepsWhatWasTaken(t *testing.T) {
open, sent := anAdoptedAnchor(t)
ctx := t.Context()
if _, err := take(ctx, open, "anchor", "hello-web"); err != nil {
t.Fatal(err)
}
reportsHolding(t, open, heldFile)
preview, err := converge(ctx, open, "anchor", false, "", "")
if err != nil {
t.Fatal(err)
}
for _, want := range []string{
"tcp/8080 hello-web (published, container port 80)",
"declared by hello-web (from anywhere)",
"WILL CLOSE — no module assigned here declares it",
// The anchor faces inward and is on the private network: the derived filter admits ssh
// from the mesh only.
"WILL CLOSE to everything outside the private network — ssh stays open from the mesh",
"notes\n replacing the found file /etc/notes.conf (notes.conf), original kept at",
"assigns nftables",
"the found firewall (ufw) is disabled, never flushed",
// What it routes is not a listener: said not to be previewed, and to be dropped.
"not previewed: traffic the machine routes that is not a published port",
"the derived filter drops it unless a module declares it",
} {
if !strings.Contains(preview, want) {
t.Errorf("the preview does not say %q:\n%s", want, preview)
}
}
if strings.Contains(preview, "15672") {
t.Errorf("a loopback listener is in the preview:\n%s", preview)
}
for _, line := range strings.Split(preview, "\n") {
if strings.Contains(line, "5000") && !strings.Contains(line, "WILL CLOSE") {
t.Errorf("an undeclared published port is not said to close: %s", line)
}
}
if n, _ := open.inventory.NodeByName(ctx, "anchor"); !n.Adopted || len(*sent) != 0 {
t.Fatal("the preview changed something")
}
if _, err := converge(ctx, open, "anchor", true, digestIn(t, preview), ""); err != nil {
t.Fatal(err)
}
n, _ := open.inventory.NodeByName(ctx, "anchor")
if n.Adopted {
t.Fatal("converge --yes left the node adopted")
}
taken, _ := open.inventory.Taken(ctx, "anchor")
if !reflect.DeepEqual(taken, []string{"hello-web", overlay.Name, "nftables", "notes"}) {
t.Fatalf("the flip took %v", taken)
}
if !reflect.DeepEqual(*sent, []string{"anchor"}) {
t.Fatalf("the flip sent %v", *sent)
}
declared := composed(t, open, "anchor")
if declared.Adoption != nil {
t.Fatal("a converged node is still sent an adoption envelope")
}
if !hasID(declared.Resources, "nftables.filtering") {
t.Fatal("the converged node is not declared the mesh's filter")
}
if _, err := adopt(ctx, open, "anchor"); err != nil {
t.Fatal(err)
}
if _, err := adopt(ctx, open, "anchor"); err == nil {
t.Fatal("adopting an adopted node was not refused")
}
again, _ := open.inventory.Taken(ctx, "anchor")
if !reflect.DeepEqual(again, taken) {
t.Fatalf("returning to adopted lost what was taken: %v", again)
}
declared = composed(t, open, "anchor")
if declared.Adoption == nil || len(declared.Adoption.Untaken) != 0 {
t.Fatalf("returned to adopted, the envelope is %+v", declared.Adoption)
}
if hasID(declared.Resources, "nftables.filtering") || hasID(declared.Resources, "nftables.load") {
t.Fatal("returned to adopted, the mesh's filter is still declared")
}
}
func hasID(resources []map[string]any, id string) bool {
for _, r := range resources {
if r["id"] == id {
return true
}
}
return false
}
// The command API refuses a flip exactly as the command line does, in the same words.
func TestTheApiRefusesTheFlipInTheCommandLinesWords(t *testing.T) {
open, _ := anAdoptedAnchor(t)
reportsHolding(t, open, heldContainer)
_, direct := converge(t.Context(), open, "anchor", true, "", "")
if direct == nil {
t.Fatal("the flip was not refused")
}
got := asking(t, letIn{}, "POST", "/converge", `{"node":"anchor","yes":true}`)
if got.Code != http.StatusConflict {
t.Fatalf("got %d: %s", got.Code, got.Body.String())
}
var said map[string]any
if err := json.Unmarshal(got.Body.Bytes(), &said); err != nil {
t.Fatal(err)
}
if said["refused"] != direct.Error() {
t.Fatalf("the API said %q and the command line %q", said["refused"], direct.Error())
}
if got := asking(t, letIn{}, "POST", "/take", `{"node":"anchor"}`); got.Code != http.StatusBadRequest {
t.Fatalf("a take naming no module got %d", got.Code)
}
}
// novox/hq ADR 0100: the preview says what the derived filter does, rendered as it is rendered. On
// a machine that faces inward, ssh is admitted from the private network only, and a port a module
// admits from the mesh only closes to everything outside it: both are said to close.
func TestThePreviewSaysWhatNarrowsToTheMeshCloses(t *testing.T) {
open, _ := anAdoptedAnchor(t)
ctx := t.Context()
register(t, open, catalogue.Manifest{Module: "store", Version: "1",
Listens: []catalogue.Listening{{Port: 5432, From: catalogue.FromMesh}}})
if _, err := assign(ctx, open, "anchor", "store"); err != nil {
t.Fatal(err)
}
reportsReaching(t, open, []link.Reach{
{Protocol: "tcp", Address: "0.0.0.0", Port: 22, By: "sshd"},
{Protocol: "tcp", Address: "0.0.0.0", Port: 5432, By: "postgres"},
{Protocol: "tcp", Address: "10.77.0.1", Port: 5432, By: "postgres"},
{Protocol: "tcp", Address: "0.0.0.0", Port: 8080, By: "hello-web", Published: true,
ContainerPort: 80},
})
preview, err := converge(ctx, open, "anchor", false, "", "")
if err != nil {
t.Fatal(err)
}
lines := map[string]string{}
for _, line := range strings.Split(preview, "\n") {
fields := strings.Fields(line)
if len(fields) > 1 && strings.HasPrefix(fields[0], "tcp/") {
lines[fields[0]+" "+fields[1]] += line + "\n"
}
}
if got := lines["tcp/22 sshd"]; !strings.Contains(got, "WILL CLOSE to everything outside "+
"the private network") {
t.Errorf("ssh on an inward machine is not said to close outside the mesh:\n%s", preview)
}
store := lines["tcp/5432 postgres"]
if strings.Count(store, "WILL CLOSE to everything outside the private network") != 1 ||
!strings.Contains(store, "declared by store (from mesh)") {
t.Errorf("the store's narrowing is not said to close, or its mesh address is:\n%s", preview)
}
if got := lines["tcp/8080 hello-web"]; !strings.Contains(got, "declared by hello-web (from anywhere)") {
t.Errorf("a port open to everywhere is not said to stay:\n%s", preview)
}
}
// digestIn is the digest a converge preview printed.
func digestIn(t *testing.T, preview string) string {
t.Helper()
for _, line := range strings.Split(preview, "\n") {
if fields := strings.Fields(line); len(fields) == 2 && fields[0] == "preview" {
return fields[1]
}
}
t.Fatalf("the preview printed no digest:\n%s", preview)
return ""
}
// The flip acts on the preview the operator saw: it names that preview's digest, and it is refused
// when the digest is missing, when anything the preview says has changed since, or when the node's
// account of itself is too old to be the machine as it is.
func TestTheFlipActsOnlyOnThePreviewTheOperatorSaw(t *testing.T) {
open, sent := anAdoptedAnchor(t)
ctx := t.Context()
if _, err := take(ctx, open, "anchor", "hello-web"); err != nil {
t.Fatal(err)
}
reportsHolding(t, open, heldFile)
preview, err := converge(ctx, open, "anchor", false, "", "")
if err != nil {
t.Fatal(err)
}
saw := digestIn(t, preview)
if !strings.Contains(preview, "converge anchor --yes "+saw) {
t.Fatalf("the preview does not say how to act on it:\n%s", preview)
}
unchanged := func() {
t.Helper()
if n, _ := open.inventory.NodeByName(ctx, "anchor"); !n.Adopted || len(*sent) != 0 {
t.Fatal("a refused flip changed something")
}
}
if _, err := converge(ctx, open, "anchor", true, "", ""); err == nil ||
!strings.Contains(err.Error(), "--yes "+saw) {
t.Fatalf("a flip naming no preview was not refused: %v", err)
}
unchanged()
// Something new is reachable: the preview the operator saw is not what would happen.
reportsReaching(t, open, []link.Reach{
{Protocol: "tcp", Address: "0.0.0.0", Port: 22, By: "sshd"},
{Protocol: "tcp", Address: "0.0.0.0", Port: 8080, By: "hello-web", Published: true,
ContainerPort: 80},
{Protocol: "tcp", Address: "0.0.0.0", Port: 6000, By: "something-new"},
}, heldFile)
_, err = converge(ctx, open, "anchor", true, saw, "")
if err == nil || !strings.Contains(err.Error(), "has changed since preview "+saw) {
t.Fatalf("a flip on a changed preview was not refused: %v", err)
}
unchanged()
// An account older than the flip trusts is refused, whatever digest is named.
again, err := converge(ctx, open, "anchor", false, "", "")
if err != nil {
t.Fatal(err)
}
saved := reportFreshFor
reportFreshFor = time.Nanosecond
_, err = converge(ctx, open, "anchor", true, digestIn(t, again), "")
reportFreshFor = saved
if err == nil || !strings.Contains(err.Error(), "wait for its next report") {
t.Fatalf("a flip on an old account was not refused: %v", err)
}
unchanged()
if _, err := converge(ctx, open, "anchor", true, digestIn(t, again), ""); err != nil {
t.Fatalf("the flip on the preview just seen was refused: %v", err)
}
if n, _ := open.inventory.NodeByName(ctx, "anchor"); n.Adopted {
t.Fatal("the flip did not converge the node")
}
}
// The flip holds the node from its checks to its send, so a push composed meanwhile waits and is
// composed after it — never sent after it with the node still adopted. And the send inside the
// flip is not made to wait on the flip's own hold.
func TestTheFlipHoldsTheNodeWhileItSends(t *testing.T) {
open, _ := anAdoptedAnchor(t)
ctx := t.Context()
if _, err := take(ctx, open, "anchor", "hello-web"); err != nil {
t.Fatal(err)
}
reportsHolding(t, open, heldFile)
preview, err := converge(ctx, open, "anchor", false, "", "")
if err != nil {
t.Fatal(err)
}
var heldElsewhere, heldHere error
sendNodes = func(inner context.Context, open *stores, names []string) error {
// Another caller cannot hold the node while the flip sends it.
waiting, cancel := context.WithTimeout(ctx, 300*time.Millisecond)
defer cancel()
if release, err := open.inventory.HoldNodes(waiting, names); err == nil {
release()
heldElsewhere = errors.New("another caller held the node while the flip sent it")
}
// The flip's own send holds it without waiting on itself.
_, release, err := holdNodes(inner, open, names)
if err != nil {
heldHere = err
return err
}
release()
return nil
}
if _, err := converge(ctx, open, "anchor", true, digestIn(t, preview), ""); err != nil {
t.Fatal(err)
}
if heldElsewhere != nil || heldHere != nil {
t.Fatalf("%v %v", heldElsewhere, heldHere)
}
// And given back once it is done.
release, err := open.inventory.HoldNodes(ctx, []string{"anchor"})
if err != nil {
t.Fatal(err)
}
release()
}
// novox/hq ADR 0103: the preview names every kind of thing a module the flip takes holds as found,
// not only its files, and the digest changes when any of them does.
func TestThePreviewNamesEveryHeldKind(t *testing.T) {
open, _ := anAdoptedAnchor(t)
ctx := t.Context()
if _, err := take(ctx, open, "anchor", "hello-web"); err != nil {
t.Fatal(err)
}
since := time.Now()
held := []link.Held{heldFile,
{ID: "notes.data", Module: "notes", Kind: "directory", Target: "/var/lib/notes", Since: since},
{ID: "notes.daemon", Module: "notes", Kind: "service", Target: "notes.service", Since: since},
{ID: "notes.seed", Module: "notes", Kind: "archive", Target: "/srv/notes", Since: since},
{ID: "notes.worker", Module: "notes", Kind: "process", Target: "notes-worker", Since: since},
{ID: "notes.account", Module: "notes", Kind: "user", Target: "notes", Since: since},
}
reportsHolding(t, open, held...)
preview, err := converge(ctx, open, "anchor", false, "", "")
if err != nil {
t.Fatal(err)
}
for _, want := range []string{
"replacing the found directory /var/lib/notes (notes.data)",
"replacing the found service notes.service (notes.daemon)",
"replacing the found archive /srv/notes (notes.seed)",
"replacing the found process notes-worker (notes.worker)",
"replacing the found user notes (notes.account)",
} {
if !strings.Contains(preview, want) {
t.Errorf("the preview does not say %q:\n%s", want, preview)
}
}
reportsHolding(t, open, held[:len(held)-1]...)
fewer, err := converge(ctx, open, "anchor", false, "", "")
if err != nil {
t.Fatal(err)
}
if digestIn(t, fewer) == digestIn(t, preview) {
t.Fatal("the digest does not change with what is held")
}
}
// novox/hq ADR 0100: the flip acts on what the node said is reachable, so an account naming nothing
// is refused. Every machine that is up answers on ssh; nothing reported means the host's collectors
// did not, and flipping would close ports the preview never named.
func TestTheFlipIsRefusedOnAnAccountNamingNothingReachable(t *testing.T) {
open, sent := anAdoptedAnchor(t)
ctx := t.Context()
if _, err := take(ctx, open, "anchor", "hello-web"); err != nil {
t.Fatal(err)
}
// Only a loopback listener: nothing off the machine, which is the same silence.
reportsReaching(t, open, []link.Reach{
{Protocol: "tcp", Address: "127.0.0.1", Port: 15672, By: "mesh-broker"},
}, heldFile)
preview, err := converge(ctx, open, "anchor", false, "", "")
if err != nil {
t.Fatal(err)
}
if !strings.Contains(preview, "this account looks partial") {
t.Errorf("the preview does not mark a partial account:\n%s", preview)
}
_, err = converge(ctx, open, "anchor", true, digestIn(t, preview), "")
if err == nil || !strings.Contains(err.Error(), "says nothing is reachable on it") {
t.Fatalf("the flip was not refused on an account naming nothing: %v", err)
}
if n, _ := open.inventory.NodeByName(ctx, "anchor"); !n.Adopted || len(*sent) != 0 {
t.Fatal("a refused flip changed something")
}
}
// An assignment cannot land between a preview and the flip that takes every module: assigning
// holds the node, so it waits for whatever is converging it.
func TestAssigningWaitsForWhateverIsConvergingTheNode(t *testing.T) {
open, _ := anAdoptedAnchor(t)
ctx := t.Context()
if _, err := take(ctx, open, "anchor", "hello-web"); err != nil {
t.Fatal(err)
}
reportsHolding(t, open, heldFile)
preview, err := converge(ctx, open, "anchor", false, "", "")
if err != nil {
t.Fatal(err)
}
saved, savedPoll := inventory.HoldWaitFor, inventory.HoldPoll
inventory.HoldWaitFor, inventory.HoldPoll = time.Second, 50*time.Millisecond
defer func() { inventory.HoldWaitFor, inventory.HoldPoll = saved, savedPoll }()
var whileFlipping error
sendNodes = func(context.Context, *stores, []string) error {
_, whileFlipping = assign(ctx, open, "anchor", "notes")
return nil
}
if _, err := converge(ctx, open, "anchor", true, digestIn(t, preview), ""); err != nil {
t.Fatal(err)
}
if !errors.Is(whileFlipping, inventory.ErrNodeBusy) {
t.Fatalf("an assignment landed while the node was being converged: %v", whileFlipping)
}
}
+621
View File
@@ -0,0 +1,621 @@
package main
import (
"context"
"crypto/sha256"
"encoding/hex"
"errors"
"flag"
"fmt"
"slices"
"sort"
"strings"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/overlay"
)
// A node is adopted or converged (novox/hq ADR 0100), and it is said to be adopted wherever the
// mesh reports a node's state: node list, node show, status and the board.
// showMode is the node show lines about a node's mode and what was taken on it.
func showMode(ctx context.Context, inv *inventory.Inventory, node inventory.Node) error {
if !node.Adopted {
fmt.Printf(" mode converged\n")
return nil
}
fmt.Printf(" mode adopted since %s\n",
node.AdoptedSince.Local().Format(time.DateTime))
taken, err := inv.Taken(ctx, node.Name)
if err != nil {
return err
}
if len(taken) == 0 {
fmt.Printf(" taken nothing yet\n")
} else {
fmt.Printf(" taken %s\n", strings.Join(taken, ", "))
}
said, err := inv.AdoptionOf(ctx, node.Name)
if err != nil {
return err
}
if said.At.IsZero() {
fmt.Printf(" it has not yet said what it found\n")
return nil
}
fmt.Printf(" firewall found %s\n", orNone(said.Firewall))
if err := showTunnel(ctx, inv, node.Name); err != nil {
return err
}
if len(said.Held) == 0 {
fmt.Printf(" holding nothing found\n")
}
for _, h := range said.Held {
// A held thing that changed is how a predecessor still writing is caught: said first.
line := fmt.Sprintf(" holds %-11s %s %s, for %s", h.Kind, h.Target, h.ID, h.Module)
if h.Changed != "" {
line += " — " + strings.ToUpper(h.Changed) + " by something other than the mesh"
}
fmt.Println(line)
if h.Kept != "" {
fmt.Printf(" %-17s original kept at %s\n", "", h.Kept)
}
}
fmt.Printf(" as of %s\n", said.At.Local().Format(time.DateTime))
return nil
}
// showTunnel is the node show lines about the tunnel an adopted node found and carried (novox/hq
// ADR 0105): what it presented at enrolment, and what it last said about taking it over.
func showTunnel(ctx context.Context, inv *inventory.Inventory, name string) error {
tunnel, err := inv.TunnelOf(ctx, name)
if errors.Is(err, inventory.ErrNoTunnel) {
return nil
}
if err != nil {
return err
}
fmt.Printf(" tunnel found %s on port %d, %s in %s, %d peer(s)\n",
tunnel.Interface, tunnel.Port, tunnel.Address, tunnel.Range, len(tunnel.Peers))
carried, said, err := inv.CarriedTunnelOf(ctx, name)
if err != nil {
return err
}
switch {
case !said:
fmt.Printf(" %-17s not yet taken over — the node has not said so\n", "")
case carried.State == inventory.CarriedTaken:
fmt.Printf(" %-17s taken over: %s is down and disabled, never flushed; the mesh's interface "+
"runs with its key, port and %d peer(s)\n", "", carried.Interface, carried.Peers)
case carried.State == inventory.CarriedDown:
fmt.Printf(" %-17s TUNNEL DOWN: %s is stopped and the mesh's interface is not up — the peers "+
"reach nothing. On the machine: systemctl start %s\n", "", carried.Interface,
"wg-quick@"+carried.Interface)
default:
fmt.Printf(" %-17s NOT taken over: %s is still the interface the peers reach\n", "", carried.Interface)
}
if said && carried.Note != "" {
fmt.Printf(" %-17s %s\n", "", carried.Note)
}
if said && carried.Kept != "" {
fmt.Printf(" %-17s its configuration's original kept at %s\n", "", carried.Kept)
}
return nil
}
func orNone(s string) string {
if s == "" {
return "none reported"
}
return s
}
// adoptedNodes are the names of every adopted node, in the order given.
func adoptedNodes(nodes []inventory.Node) []string {
var out []string
for _, n := range nodes {
if n.Adopted {
out = append(out, n.Name)
}
}
return out
}
// The operator's acts on an adopted node (novox/hq ADR 0100). Called by the command line and the
// command API alike, so a refusal is the same refusal in the same words at both (ADR 0035).
// sendNodes sends the named machines what they should be now. A variable so a test can see what
// an act would send without a broker.
var sendNodes = sendTo
// DefaultFilter is the module converging a node assigns to load the mesh's derived filter.
const DefaultFilter = "nftables"
// take is a module's cutover on an adopted node: the operator's act, done when that module's data
// has moved. From the next push its resources converge there like any other, replacing what the
// node found and holds for it.
func take(ctx context.Context, open *stores, node, module string) (string, error) {
inv := open.inventory
assigned, err := inv.Assigned(ctx, node)
if err != nil {
return "", err
}
if !slices.Contains(assigned, module) {
if plan, _, err := planFor(ctx, open, node); err == nil {
if why, runs := plan.Because[module]; runs {
return "", fmt.Errorf("%w: %s runs on %s because %s — assign it to %s to take it",
inventory.ErrNotAssigned, module, node, why, node)
}
}
}
if err := inv.Take(ctx, node, module); err != nil {
return "", err
}
said := fmt.Sprintf("%s is taken on %s", module, node)
reported, err := inv.AdoptionOf(ctx, node)
if err != nil {
return "", err
}
var replaces []string
for _, h := range reported.Held {
if h.Module == module {
replaces = append(replaces, " "+heldLine(h))
}
}
if len(replaces) > 0 {
said += "; the next push replaces what the node found and holds for it:\n" +
strings.Join(replaces, "\n")
}
return said + fmt.Sprintf("\n run `push %s` to cut it over", node), nil
}
// reportFreshFor is how old a node's account of itself may be for the flip to act on it. A
// variable so a test can age a report without waiting.
var reportFreshFor = 15 * time.Minute
// converge previews, and with yes makes, the flip of an adopted node to converged: every module it
// runs is taken, the filter module is assigned to load the mesh's derived filter in place of the
// guard, and the found firewall is retired — disabled, never flushed — by the host.
//
// Refused while an assigned module still holds a found container: each service is taken on its
// own, when its data has moved, never by the flip. And refused on a preview that would be stale:
// what is reachable is the node's last account, so that account must be of what it was last sent.
//
// **The flip acts on the preview the operator saw** and on nothing else. The preview ends with a
// short digest of what it said — every reachable thing and its fate, the modules the flip takes and
// the filter — and yes must name that digest: if anything the preview would say has changed since,
// the flip is refused rather than done on a preview nobody read. And it is refused on an account
// older than reportFreshFor: what was reachable then is not evidence of what is reachable now.
func converge(ctx context.Context, open *stores, node string, yes bool, digest string,
filter string) (string, error) {
inv := open.inventory
if filter == "" {
filter = DefaultFilter
}
if yes {
// Held from the checks to the send, so no push composed before the flip is sent after it
// and returns the node to adopted.
held, release, err := holdNodes(ctx, open, []string{node})
if err != nil {
return "", err
}
defer release()
ctx = held
}
record, err := inv.NodeByName(ctx, node)
if err != nil {
return "", err
}
if !record.Adopted {
return "", fmt.Errorf("%s is converged already; there is nothing to flip", node)
}
reports, err := inv.LastReports(ctx)
if err != nil {
return "", err
}
current := false
for _, r := range reports {
if r.Node == node {
current = r.Current
}
}
reported, err := inv.AdoptionOf(ctx, node)
if err != nil {
return "", err
}
if !current || reported.At.IsZero() {
return "", fmt.Errorf("%s has not reported on the declaration it was last sent, so what it "+
"says is reachable may not be the machine as it is: run `push %s --wait 2m` and "+
"converge once it has applied", node, node)
}
assignedWhenPreviewed, err := inv.Assigned(ctx, node)
if err != nil {
return "", err
}
plan, settings, err := planFor(ctx, open, node)
if err != nil {
return "", err
}
runs := map[string]bool{}
for _, m := range plan.Modules {
runs[m.Module] = true
}
var holding []string
for _, h := range reported.Held {
if h.Kind == "container" && runs[h.Module] {
holding = append(holding, fmt.Sprintf(" %s holds the found container %s — take %s %s "+
"once its data has moved", h.Module, h.Target, node, h.Module))
}
}
if len(holding) > 0 {
sort.Strings(holding)
return "", fmt.Errorf("%s still holds what it found, and a service is taken on its own, "+
"never by the flip:\n%s", node, strings.Join(holding, "\n"))
}
// And refused while a peer of the tunnel this hub took over has not enrolled (novox/hq ADR
// 0105): the flip loads the derived filter and retires the found firewall, and a machine the
// mesh has no record of is not one the filter admits — it would go dark.
if _, hubName, adopted, err := inv.AdoptedTunnel(ctx); err != nil {
return "", err
} else if adopted && hubName == node {
carried, err := inv.CarriedPeers(ctx)
if err != nil {
return "", err
}
var waiting []string
for _, c := range carried {
if c.EnrolledAs == "" {
waiting = append(waiting, fmt.Sprintf(" %s at %s", overlay.CarriedName(c.PublicKey), c.Address))
}
}
if len(waiting) > 0 {
return "", fmt.Errorf("%s carries peers of the tunnel it took over that have not enrolled, and "+
"converging would cut them off — enrol each first (`overlay show` says which are enrolled):\n%s",
node, strings.Join(waiting, "\n"))
}
}
shelf, err := inv.Catalogue(ctx)
if err != nil {
return "", err
}
filterModule, known := shelf[filter]
if !known {
return "", fmt.Errorf("%w: %s — converging assigns it to load the mesh's filter; "+
"name another with --filter", inventory.ErrNoSuchModule, filter)
}
if filterModule.Filtering == nil {
return "", fmt.Errorf("%s loads no filter of the mesh's; name a module that does with --filter",
filter)
}
gens, err := generators(ctx, open)
if err != nil {
return "", err
}
with, _, err := renderingFor(ctx, open, node, plan, settings, gens, Reading)
if err != nil {
return "", err
}
rules, err := plan.Rules(with)
if err != nil {
return "", err
}
taken, err := inv.Taken(ctx, node)
if err != nil {
return "", err
}
derived := derivedFilter{rules: rules, foundation: with.Foundation, mesh: with.Mesh,
outward: plan.PublicDomain != ""}
preview, saw := previewOf(node, reported, derived, plan, taken, filter, runs[filter])
preview += "\n\n preview " + saw
if !yes {
return preview + fmt.Sprintf("\n\nNothing has changed. Run `converge %s --yes %s` to do "+
"it.", node, saw), nil
}
// An account naming nothing reachable is not an account of a machine: every machine answers
// on ssh, and the host's collectors failing — `ss` refusing, or the container runtime not
// answering, which drops every published port at once — leaves exactly this. Flipping on it
// would close ports the preview never named.
if yes && countReachable(reported) == 0 {
return preview, fmt.Errorf("%s says nothing is reachable on it, which no machine that is "+
"up ever is: its account looks partial — whatever reads what is listening, or what "+
"the container runtime publishes, did not answer. Fix that on the machine and run "+
"`push %s --wait 2m`, then preview again", node, node)
}
if age := time.Since(reported.At); age > reportFreshFor {
return preview, fmt.Errorf("%s last said what is reachable on it %s ago, and the flip acts "+
"only on an account newer than %s: wait for its next report, or run `push %s --wait 2m`, "+
"then preview again", node, age.Round(time.Second), reportFreshFor, node)
}
if digest == "" {
return preview, fmt.Errorf("converging %s acts on the preview you saw: name its digest, "+
"`converge %s --yes %s`, once you have read it", node, node, saw)
}
if digest != saw {
return preview, fmt.Errorf("what converging %s would do has changed since preview %s "+
"(it is now %s): read the preview above, and run `converge %s --yes %s` if it is "+
"what you want", node, digest, saw, node, saw)
}
// The flip. The filter first, and only kept if the node still resolves with it: a node that
// cannot be worked out would be sent nothing, and would sit with its guard and no filter.
assigned, err := inv.Assigned(ctx, node)
if err != nil {
return "", err
}
// And nothing assigned since the preview was composed: the flip takes every module the node
// runs, and one assigned in between would be taken without ever having been previewed.
if !slices.Equal(assigned, assignedWhenPreviewed) {
return preview, fmt.Errorf("what %s runs changed while this was converging (it is now %s): "+
"the flip takes every module on the node, so read the preview again", node,
strings.Join(assigned, ", "))
}
if !slices.Contains(assigned, filter) {
if err := inv.Assign(ctx, node, filter); err != nil {
return "", err
}
if _, _, err := planFor(ctx, open, node); err != nil {
_ = inv.Unassign(ctx, node, filter)
return "", fmt.Errorf("%s cannot run %s, so it was not converged: %w", node, filter, err)
}
}
took, err := inv.Converge(ctx, node)
if err != nil {
return "", err
}
said := preview + fmt.Sprintf("\n\n%s is converged", node)
if len(took) > 0 {
said += "; took " + strings.Join(took, ", ")
}
if err := sendNodes(ctx, open, []string{node}); err != nil {
return said + "\n and it could not be sent: run `push " + node + "`", err
}
return said + "\n sent: the host loads the mesh's filter and disables the firewall it found", nil
}
// previewOf is what converging a node will change, before it changes it, and a short digest of
// what it said: every reachable thing and its fate, the modules the flip takes and the filter. The
// digest is what the flip is asked to act on, so it changes whenever any of those would.
func previewOf(node string, reported inventory.Adoption, derived derivedFilter,
plan catalogue.Resolution, taken []string, filter string, filterAssigned bool) (string, string) {
var said []string
var b strings.Builder
fmt.Fprintf(&b, "converging %s\n", node)
fmt.Fprintf(&b, "\n reachable on the machine now, as it reported at %s:\n",
reported.At.Local().Format(time.DateTime))
for _, r := range reported.Reachable {
if loopback(r.Address) {
continue
}
what := fmt.Sprintf("%s/%d", r.Protocol, r.Port)
if r.By != "" {
what += " " + r.By
}
if r.Published {
what += fmt.Sprintf(" (published, container port %d)", r.ContainerPort)
}
fate := derived.fate(r)
fmt.Fprintf(&b, " %-44s %s\n", what, fate)
said = append(said, fmt.Sprintf("reach %s %s %s", r.Address, what, fate))
}
if countReachable(reported) == 0 {
// Said as what it is: no machine that is up is reachable on nothing, so this is an
// account that did not come back, not a machine with nothing on it.
b.WriteString(" nothing reported — this account looks partial, and the flip is " +
"refused on it\n")
said = append(said, "reach nothing reported")
}
// What the machine routes for others is not a listener and not a published port, so nothing
// above can show it; the derived filter's forward chain drops it all the same.
b.WriteString(" not previewed: traffic the machine routes that is not a published port " +
"(a tunnel, NAT in the found firewall) — the derived filter drops it unless a module " +
"declares it\n")
isTaken := map[string]bool{}
for _, m := range taken {
isTaken[m] = true
}
var takes []string
for _, m := range plan.Modules {
if !isTaken[m.Module] {
takes = append(takes, m.Module)
}
}
if !filterAssigned && !isTaken[filter] {
takes = append(takes, filter)
}
sort.Strings(takes)
b.WriteString("\n the flip takes:\n")
if len(takes) == 0 {
b.WriteString(" nothing — every module is taken already\n")
}
for _, m := range takes {
fmt.Fprintf(&b, " %s\n", m)
said = append(said, "take "+m)
// Every kind it holds — a directory, a service, an archive, a process, a user as well as a
// file (novox/hq ADR 0103) — each said, and each part of what the flip is asked to act on.
for _, h := range reported.Held {
if h.Module != m {
continue
}
fmt.Fprintf(&b, " replacing the found %s", heldLine(h))
if h.Kept != "" {
fmt.Fprintf(&b, ", original kept at %s", h.Kept)
}
b.WriteString("\n")
said = append(said, "replace "+m+" "+heldLine(h)+" "+h.Kept)
}
}
if !filterAssigned {
fmt.Fprintf(&b, "\n and assigns %s, which loads the mesh's filter in place of its guard\n", filter)
}
fw := reported.Firewall
if fw == "" || fw == "none" {
b.WriteString(" no firewall was found on the machine; the mesh's filter is its first\n")
} else {
fmt.Fprintf(&b, " the found firewall (%s) is disabled, never flushed: its configuration stays on disk\n", fw)
}
said = append(said, fmt.Sprintf("filter %s assigned=%t firewall=%s", filter, filterAssigned, fw))
// Sorted: the same account, reported in another order, is the same preview.
sort.Strings(said)
sum := sha256.Sum256([]byte(strings.Join(said, "\n")))
return strings.TrimRight(b.String(), "\n"), hex.EncodeToString(sum[:])[:12]
}
// derivedFilter is what the filter the flip loads is rendered from, as AsNftables renders it.
type derivedFilter struct {
rules []catalogue.Rule
foundation []int
// mesh is every address on the private network; outward says the machine faces outside.
mesh []string
outward bool
}
// closesOutside is what a narrowing from everywhere to the private network is called: it closes.
const closesOutside = "WILL CLOSE to everything outside the private network"
// fate is what the derived filter does to one reachable thing: which module declares it and from
// where, or that it will close — wholly, or to everything outside the private network. Rendered
// exactly as AsNftables admits it, ssh included.
func (d derivedFilter) fate(r inventory.Reach) string {
// Bound to an address on the private network, it was never reachable from outside it, so
// admitting it from the mesh narrows nothing.
onMesh := slices.Contains(d.mesh, strings.Trim(r.Address, "[]"))
if r.Protocol == "tcp" && r.Port == catalogue.SSHPort {
// From everywhere only when the machine faces outward or the mesh has no addresses to
// narrow it to; otherwise from the private network only.
if d.outward || len(d.mesh) == 0 || onMesh {
return "stays open — ssh is never closed"
}
return closesOutside + " — ssh stays open from the mesh, never closed there"
}
for _, port := range d.foundation {
if r.Protocol == "tcp" && r.Port == port {
return "stays open — the mesh's own, from anywhere"
}
}
for _, rule := range d.rules {
if rule.Port != r.Port || rule.Protocol != r.Protocol {
continue
}
by := strings.Join(rule.Because, ", ")
switch rule.From {
case catalogue.FromMachine:
return fmt.Sprintf("WILL CLOSE to the network — declared by %s for this machine only", by)
case catalogue.FromMesh:
if len(d.mesh) == 0 {
return fmt.Sprintf("WILL CLOSE — declared by %s from the mesh, and this node "+
"knows no mesh addresses", by)
}
if !onMesh {
return fmt.Sprintf("%s — declared by %s from the mesh only", closesOutside, by)
}
}
return fmt.Sprintf("declared by %s (from %s)", by, rule.From)
}
return "WILL CLOSE — no module assigned here declares it"
}
// countReachable is how much of a node's account of itself names something off the machine.
// Loopback is left out for the same reason the preview leaves it out: nothing outside reaches it,
// so a report of loopback alone says nothing about what the filter would close.
func countReachable(reported inventory.Adoption) int {
n := 0
for _, r := range reported.Reachable {
if !loopback(r.Address) {
n++
}
}
return n
}
// heldLine is one thing a node holds as found, as take and the converge preview both say it.
func heldLine(h inventory.Held) string {
return fmt.Sprintf("%s %s (%s)", h.Kind, h.Target, h.ID)
}
// loopback is an address nothing off the machine reaches.
func loopback(address string) bool {
a := strings.Trim(address, "[]")
return strings.HasPrefix(a, "127.") || a == "::1" || a == "localhost"
}
// adopt returns a converged node to adopted: the mesh's filter is unloaded, the guard restored,
// the found firewall enabled again and the openings converged through it once more. What was
// taken stays taken.
func adopt(ctx context.Context, open *stores, node string) (string, error) {
inv := open.inventory
record, err := inv.NodeByName(ctx, node)
if err != nil {
return "", err
}
if record.Adopted {
return "", fmt.Errorf("%s is adopted already", node)
}
held, release, err := holdNodes(ctx, open, []string{node})
if err != nil {
return "", err
}
defer release()
ctx = held
if err := inv.SetAdopted(ctx, node, true); err != nil {
return "", err
}
said := fmt.Sprintf("%s is adopted; what was taken on it stays taken", node)
if err := sendNodes(ctx, open, []string{node}); err != nil {
return said + "\n and it could not be sent: run `push " + node + "`", err
}
return said + "\n sent: the host unloads the mesh's filter and enables the firewall it found", nil
}
// takeCommand, convergeCommand and adoptCommand are the command line's adapters to the acts above.
func takeCommand(ctx context.Context, args []string) error {
if len(args) != 2 {
return errors.New("take <node> <module>")
}
return runAct(ctx, func(open *stores) (string, error) { return take(ctx, open, args[0], args[1]) })
}
func convergeCommand(ctx context.Context, args []string) error {
set := flag.NewFlagSet("converge", flag.ContinueOnError)
yes := set.String("yes", "", "do it, naming the digest the preview printed; without it, only "+
"the preview")
filter := set.String("filter", DefaultFilter, "the module that loads the mesh's filter")
positionals, err := parseAround(set, args)
if err != nil {
return err
}
if len(positionals) != 1 {
return errors.New("converge <node> [--yes <digest>] [--filter nftables]")
}
return runAct(ctx, func(open *stores) (string, error) {
return converge(ctx, open, positionals[0], *yes != "", *yes, *filter)
})
}
func adoptCommand(ctx context.Context, args []string) error {
if len(args) != 1 {
return errors.New("adopt <node>")
}
return runAct(ctx, func(open *stores) (string, error) { return adopt(ctx, open, args[0]) })
}
func runAct(ctx context.Context, act func(*stores) (string, error)) error {
open, err := openStores(ctx)
if err != nil {
return err
}
defer open.Close()
said, err := act(open)
if said != "" {
fmt.Println(said)
}
if err != nil && said != "" {
fmt.Println()
}
return err
}
+26 -6
View File
@@ -94,19 +94,29 @@ func (n notYet) Who(*http.Request) (string, error) {
func commands(who Authenticator) http.Handler {
mux := http.NewServeMux()
mux.HandleFunc("POST /assign", acting(who, func(ctx context.Context, open *stores, in request) (string, error) {
mux.HandleFunc("POST /assign", acting(who, true, func(ctx context.Context, open *stores, in request) (string, error) {
return assign(ctx, open, in.Node, in.Module)
}))
mux.HandleFunc("POST /unassign", acting(who, func(ctx context.Context, open *stores, in request) (string, error) {
mux.HandleFunc("POST /unassign", acting(who, true, func(ctx context.Context, open *stores, in request) (string, error) {
return unassign(ctx, open, in.Node, in.Module)
}))
// Adoption (novox/hq ADR 0100): the same acts as `take`, `converge` and `adopt`.
mux.HandleFunc("POST /take", acting(who, true, func(ctx context.Context, open *stores, in request) (string, error) {
return take(ctx, open, in.Node, in.Module)
}))
mux.HandleFunc("POST /converge", acting(who, false, func(ctx context.Context, open *stores, in request) (string, error) {
return converge(ctx, open, in.Node, in.Yes, in.Digest, in.Filter)
}))
mux.HandleFunc("POST /adopt", acting(who, false, func(ctx context.Context, open *stores, in request) (string, error) {
return adopt(ctx, open, in.Node)
}))
// Anything else is said plainly, because a command surface answering 404 to a verb somebody
// expected is indistinguishable from one that is down.
mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
refuse(w, http.StatusNotFound, fmt.Errorf(
"%s %s is not something this mesh can be asked; it accepts POST /assign and "+
"POST /unassign", r.Method, r.URL.Path))
"%s %s is not something this mesh can be asked; it accepts POST /assign, "+
"POST /unassign, POST /take, POST /converge and POST /adopt", r.Method, r.URL.Path))
})
return mux
}
@@ -114,11 +124,17 @@ func commands(who Authenticator) http.Handler {
type request struct {
Node string `json:"node"`
Module string `json:"module"`
// Yes, Digest and Filter are converge's: do it rather than preview it, the digest of the
// preview it acts on, and which module loads the mesh's filter.
Yes bool `json:"yes,omitempty"`
Digest string `json:"digest,omitempty"`
Filter string `json:"filter,omitempty"`
}
// acting is the shape every route shares: authenticate, read, act, answer.
func acting(
who Authenticator,
needsModule bool,
do func(context.Context, *stores, request) (string, error),
) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
@@ -131,8 +147,12 @@ func acting(
refuse(w, http.StatusBadRequest, fmt.Errorf("this is not a request this understands: %w", err))
return
}
if in.Node == "" || in.Module == "" {
refuse(w, http.StatusBadRequest, errors.New(`both "node" and "module" are needed`))
if in.Node == "" || (needsModule && in.Module == "") {
if needsModule {
refuse(w, http.StatusBadRequest, errors.New(`both "node" and "module" are needed`))
} else {
refuse(w, http.StatusBadRequest, errors.New(`"node" is needed`))
}
return
}
+62
View File
@@ -0,0 +1,62 @@
package main
import (
"context"
"encoding/json"
"errors"
"flag"
"fmt"
"os"
"time"
"github.com/novox/mesh-controller/internal/link"
)
// ask calls one of a module's tools, through the control plane's own broker connection.
//
// A module serves tools under an account scoped to exactly that (novox/hq ADR 0047), and nothing
// else in the mesh held an account that could ask one — not an operator at a terminal, not an agent
// acting for one (novox/hq 04-ISSUES/049). The control plane does, so it is the way in: one
// process, one connection, one place a question can be seen to have been asked (ADR 0095).
func askCommand(ctx context.Context, args []string) error {
positionals, flags := split(args)
set := flag.NewFlagSet("ask", flag.ContinueOnError)
wait := set.Duration("wait", 60*time.Second, "how long to wait for the module's answer")
if err := set.Parse(flags); err != nil {
return err
}
if len(positionals) < 2 || len(positionals) > 3 {
return errors.New("ask <module> <tool> [json arguments] [--wait 60s]")
}
module, tool := positionals[0], positionals[1]
var arguments json.RawMessage
if len(positionals) == 3 {
if !json.Valid([]byte(positionals[2])) {
return fmt.Errorf("the arguments are not JSON: %s", positionals[2])
}
arguments = json.RawMessage(positionals[2])
}
server, err := link.Connect(nil, nil)
if err != nil {
return err
}
defer server.Close()
answer, err := link.Ask(ctx, server.Channel(), module, tool, arguments, *wait)
if err != nil {
return err
}
// The answer as the module gave it, to standard output, for a person or a program. A tool
// that answered with an error has still answered: printed the same way, and the exit status
// says which.
body, err := json.Marshal(answer)
if err != nil {
return err
}
fmt.Fprintln(os.Stdout, string(body))
if answer.Error != "" {
return fmt.Errorf("%s.%s answered with an error: %s", module, tool, answer.Error)
}
return nil
}
+8 -1
View File
@@ -137,6 +137,9 @@ type view struct {
Unresolved []blockedMachine
// Network is why the private network could not be computed, when it could not.
Network string
// Adopted is every node still adopted (novox/hq ADR 0100). Not broken: nothing forces the
// flip, so a node left adopted is shown rather than read as converged.
Adopted []string
At string
}
@@ -181,7 +184,7 @@ type staleModule struct {
func viewOf(asked answers) view {
out := view{Machines: len(asked.nodes), At: time.Now().Format("15:04:05"),
Network: asked.network}
Network: asked.network, Adopted: adoptedNodes(asked.nodes)}
var blocked []string
for name := range asked.refused {
blocked = append(blocked, name)
@@ -311,6 +314,10 @@ new, switched off, or unreachable.</p>
<p class="quiet">Never told is not out of date: nobody has asked that machine to be anything yet.
Both are sent by <code>push --behind</code>.</p>
{{else}}<p class="quiet">Every machine is running what the mesh would send it.</p>{{end}}
{{if .Adopted}}
<h2>Which machines are adopted?</h2>
<ul>{{range .Adopted}}<li><strong>{{.}}</strong> <span class="quiet">adopted — what was found on it is kept until each module is taken</span></li>{{end}}</ul>
{{end}}
{{end}}
<footer>Read at {{.At}}. This page holds nothing and changes nothing.</footer>
+38 -9
View File
@@ -68,6 +68,11 @@ func buildCommand(ctx context.Context, args []string) error {
}
// buildFrom turns what a builder said into what the mesh keeps.
//
// **By digest and path, never by where it was pushed** (novox/hq 04-ISSUES/102). The builder
// says `<registry>:<port>/<module>/<artifact>@sha256:…`; the mesh records the artifact-store
// reference and composes the store's address back in where a reference is used. `against` is kept
// as announced: it is what the build stood on as the builder saw it, and the catalogue's edge.
func buildFrom(result link.BuildResult) inventory.Build {
kept := inventory.Build{
ID: result.ID, Repository: result.Repository, Ref: result.Ref,
@@ -77,16 +82,21 @@ func buildFrom(result link.BuildResult) inventory.Build {
// edges, and it is not always listening when a build happens — on a fresh mesh it cannot
// be, for exactly the modules it needs most. Keeping them is what makes a replay able to
// rebuild the graph rather than a list of names.
Path: result.Path, Manifest: result.Manifest, Against: result.Against,
Path: result.Path, Against: result.Against,
}
var announced []inventory.Artifact
for _, made := range result.Made {
kept.Made = append(kept.Made, inventory.Artifact{
announced = append(announced, inventory.Artifact{
Name: made.Name, Kind: made.Kind, Reference: made.Reference,
})
kept.Made = append(kept.Made, inventory.Artifact{
Name: made.Name, Kind: made.Kind, Reference: catalogue.Recorded(made.Reference),
})
}
kept.Manifest = recordedManifest(result.Manifest, announced)
// The module name comes from the manifest, which only exists when the build got that far.
if len(result.Manifest) > 0 {
if m, err := catalogue.ParseManifest(result.Manifest); err == nil {
if len(kept.Manifest) > 0 {
if m, err := catalogue.ParseManifest(kept.Manifest); err == nil {
kept.Module = m.Module
}
}
@@ -378,7 +388,8 @@ func buildOne(ctx context.Context, repository, path, ref string, wait time.Durat
}
defer open.Close()
inv := open.inventory
if err := inv.RecordBuild(ctx, buildFrom(result)); err != nil {
kept := buildFrom(result)
if err := inv.RecordBuild(ctx, kept); err != nil {
return err
}
@@ -388,13 +399,15 @@ func buildOne(ctx context.Context, repository, path, ref string, wait time.Durat
return fmt.Errorf("%s could not build %s:\n%s", result.On, result.Repository, result.Failed)
}
for _, made := range result.Made {
// Said as recorded: what each artifact is, not where this builder happened to push it.
for _, made := range kept.Made {
fmt.Printf(" %-12s %s %s\n", made.Name, made.Kind, made.Reference)
}
// Parsed with the same parser a hand-written manifest goes through. A second path would be a
// second thing to disagree about what a manifest is.
manifest, err := catalogue.ParseManifest(result.Manifest)
// second thing to disagree about what a manifest is. The manifest as recorded, so the catalogue
// holds references by digest and path and every declaration composes the store's address in.
manifest, err := catalogue.ParseManifest(kept.Manifest)
if err != nil {
return fmt.Errorf("%s built %s and what came back is not a manifest: %w",
result.On, result.Repository, err)
@@ -481,6 +494,9 @@ type answers struct {
// all, and one that does gets a refusal naming exactly what is missing — which is a better sentence
// than a build command refusing to start because a query did not run. So the store not opening is
// reported and the build goes ahead without it.
//
// Routed through the artifact store as the network reaches it now (novox/hq 04-ISSUES/102): a
// base is recorded by digest and path, and a build machine needs something it can pull.
func heldBy(ctx context.Context) map[string]string {
open, err := openStores(ctx)
if err != nil {
@@ -494,5 +510,18 @@ func heldBy(ctx context.Context) map[string]string {
fmt.Fprintf(os.Stderr, "could not read what this mesh has built: %v\n", err)
return nil
}
return held
address, err := whereABuilderReachesTheStore(ctx, open.inventory)
if err != nil {
fmt.Fprintf(os.Stderr, "could not find the artifact store on this mesh's network, so a "+
"module naming a base will be handed a reference nothing can fetch: %v\n", err)
return held
}
if address == "" {
return held
}
routed := make(map[string]string, len(held))
for repository, reference := range held {
routed[repository] = catalogue.Rerouted(reference, address)
}
return routed
}
+37
View File
@@ -0,0 +1,37 @@
package main
import (
"context"
)
// heldKey carries the nodes this call already holds, so an act that holds a node and then sends
// through sendTo does not wait on itself.
type heldKey struct{}
// holdNodes holds the named nodes for composing and sending their declarations (novox/hq ADR
// 0100), skipping any the context already holds, and returns a context that says it holds them.
// Release gives back only what this call took.
func holdNodes(ctx context.Context, open *stores, names []string) (context.Context, func(), error) {
already, _ := ctx.Value(heldKey{}).(map[string]bool)
var take []string
for _, n := range names {
if !already[n] {
take = append(take, n)
}
}
if len(take) == 0 {
return ctx, func() {}, nil
}
release, err := open.inventory.HoldNodes(ctx, take)
if err != nil {
return ctx, nil, err
}
held := map[string]bool{}
for n := range already {
held[n] = true
}
for _, n := range take {
held[n] = true
}
return context.WithValue(ctx, heldKey{}, held), release, nil
}
+45
View File
@@ -0,0 +1,45 @@
package main
import (
"context"
"errors"
"testing"
"time"
)
// A cascade round gives its hold back on every way out: a declaration that cannot be marshalled
// and a send that fails leave nobody waiting for the node.
func TestASendRoundGivesItsHoldBackOnEveryWayOut(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
unmarshallable := func(context.Context, string) (sendable, error) {
return sendable{Resources: []map[string]any{{"id": "x", "bad": make(chan int)}}}, nil
}
plain := func(context.Context, string) (sendable, error) {
return sendable{Resources: []map[string]any{{"id": "x"}}}, nil
}
failing := func(readyNode, []byte) error { return errors.New("the broker went away") }
fine := func(readyNode, []byte) error { return nil }
for name, round := range map[string]func() error{
"a body that cannot be marshalled": func() error {
_, err := sendRound(ctx, open, []string{"anchor"}, unmarshallable, fine)
return err
},
"a send that fails": func() error {
_, err := sendRound(ctx, open, []string{"anchor"}, plain, failing)
return err
},
} {
if err := round(); err == nil {
t.Fatalf("%s was not an error", name)
}
waiting, cancel := context.WithTimeout(ctx, 2*time.Second)
release, err := open.inventory.HoldNodes(waiting, []string{"anchor"})
cancel()
if err != nil {
t.Fatalf("after %s the node is still held: %v", name, err)
}
release()
}
}
+16 -2
View File
@@ -68,6 +68,8 @@ func run() error {
return licenceCommand(ctx, args[1:])
case "rotate":
return rotateCommand(ctx, args[1:])
case "ask":
return askCommand(ctx, args[1:])
case "builds":
return buildsCommand(ctx, args[1:])
case "pin":
@@ -96,6 +98,12 @@ func run() error {
return moduleCommand(ctx, args[1:])
case "assign", "unassign":
return assignCommand(ctx, args[0], args[1:])
case "take":
return takeCommand(ctx, args[1:])
case "converge":
return convergeCommand(ctx, args[1:])
case "adopt":
return adoptCommand(ctx, args[1:])
case "settings":
return settingsCommand(ctx, args[1:])
case "secret":
@@ -124,7 +132,7 @@ func usage() {
fmt.Fprint(os.Stderr, `mesh-controller — the control plane
migrate bring each context's schema up to date
node add <name> create a node record
node add <name> [--adopted] create a node record; --adopted: the machine is in use
node list the nodes this mesh knows about
node show <name> what one machine reported it can do, and why
node public-domain <name> the domain it composes its routed names under
@@ -132,6 +140,7 @@ func usage() {
node public-domain <name> --clear ...it faces the outside no longer
token issue --node <name> a one-time right to join, for an existing record
token issue --new <name> create the record and issue for it
token issue ... --adopted ...for a machine in use, which joins adopted
identity show this control plane's signing key
broker show where the broker is, and what to expect there
serve consume what nodes say, and answer
@@ -152,6 +161,9 @@ func usage() {
api --issuer URL [--listen A] assign and unassign over http, for a surface that is not here
assign <node> <module> put a module on a node
unassign <node> <module> take it off
take <node> <module> cut a module over on an adopted node, once its data has moved
converge <node> [--yes <digest>] [--filter nftables] preview, then make, an adopted node converged
adopt <node> return a converged node to adopted; what was taken stays taken
settings set <module> <file> what a module's config should say, for the whole mesh
settings set <module> <file> --node <n> ...or for one machine
settings clear <module> [--node <n>] take a layer away
@@ -166,11 +178,13 @@ func usage() {
build <repository> [--ref R] have a build machine build it, and record what came out
build --behind build every module the mesh holds older than its source
builds [<module>] what has been built lately, and what came of it
builder issue <name> a broker account for a build machine, scoped to build work
builder issue <name> a broker account for a build machine, scoped to build work,
delivered as the builder module's broker secret (module add it first)
licence add|list|use|key model access, under the name a person calls it
licence manager <name> <node> the node that holds a refreshable licence's refresh token
licence refresh <name> mint a new access token and seal it to every holder
rotate <provision> [--consumer <n>] a new credential for every holder, both ends at once
ask <module> <tool> [json] call one of a module's tools over the broker, and print its answer
pin <node> <provision> <from> which node this one gets a provision from
unpin <node> <provision> put that question back
plan <node> [--files|--json] what that node would run, and why
+15
View File
@@ -253,6 +253,9 @@ func moduleCommand(ctx context.Context, args []string) error {
if !ok {
return fmt.Errorf("this mesh knows no module %q; `module add` it first", module)
}
if err := mayIssue(m); err != nil {
return err
}
management, err := broker.ManagementFromEnvironment()
if err != nil {
@@ -552,3 +555,15 @@ func brokerReachableAt(ctx context.Context, inv *inventory.Inventory, known brok
}
return net.JoinHostPort(brokerAt, port), nil
}
// mayIssue says whether a module can be given a broker account. The account is delivered as the
// module's own secret named broker; a module that declares none has nothing to read it with, and
// an account nothing reads is an orphan on the bus (novox/hq 04-ISSUES/078). Said before the
// account is made, not after.
func mayIssue(m catalogue.Manifest) error {
if _, reads := m.OwnSecrets["broker"]; !reads {
return fmt.Errorf("%s declares no own secret named broker, so there is nowhere to deliver "+
"an account; a module that speaks on the bus declares \"own-secrets\": {\"broker\": <path>}", m.Module)
}
return nil
}
+26
View File
@@ -0,0 +1,26 @@
package main
import (
"strings"
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
)
// `module issue` makes a broker account and delivers it as the module's own secret named broker.
// A module that declares none is refused before the account exists, so the bus never carries an
// account nothing reads (novox/hq 04-ISSUES/078).
func TestAModuleWithNoBrokerSecretCannotBeIssued(t *testing.T) {
err := mayIssue(catalogue.Manifest{Module: "step-ca", OwnSecrets: map[string]string{"password": "/run/password"}})
if err == nil {
t.Fatal("a module with no broker own secret was issued an account")
}
for _, want := range []string{"step-ca", "broker", "own-secrets"} {
if !strings.Contains(err.Error(), want) {
t.Errorf("the refusal does not say %q: %v", want, err)
}
}
if err := mayIssue(catalogue.Manifest{Module: "redis", OwnSecrets: map[string]string{"broker": "/run/broker"}}); err != nil {
t.Errorf("a module declaring its broker secret was refused: %v", err)
}
}
+266 -35
View File
@@ -7,6 +7,7 @@ import (
"fmt"
"os"
"sort"
"strconv"
"strings"
"time"
@@ -21,11 +22,28 @@ import (
// cheapest next step. That is how novox/hq ADR 0001 records `hal/sdk` reaching 34,636:
// nothing in it was wrong, and no one edit was the one that should have been a new file.
func overlayCIDR() string {
if v := strings.TrimSpace(os.Getenv(OverlayCIDRVar)); v != "" {
return v
// DefaultOverlayCIDR is the range the mesh allocates from when nothing says another.
const DefaultOverlayCIDR = "10.42.0.0/16"
// overlayRange is the range the mesh allocates node addresses from.
//
// **The adopted tunnel's range first** (novox/hq ADR 0105): a hub that took over the tunnel it
// found is at that tunnel's address, its peers are at theirs, and every node's address is
// composed from the same range — the hub's, and every binding, hosts entry and endpoint derived
// from it. Those are readers of this; none of them stores the range. Without an adopted tunnel,
// the range genesis was told, or the default.
func overlayRange(ctx context.Context, inv *inventory.Inventory) (string, error) {
tunnel, _, adopted, err := inv.AdoptedTunnel(ctx)
if err != nil {
return "", err
}
return "10.42.0.0/16"
if adopted {
return tunnel.Range, nil
}
if v := strings.TrimSpace(os.Getenv(OverlayCIDRVar)); v != "" {
return v, nil
}
return DefaultOverlayCIDR, nil
}
func overlayCommand(ctx context.Context, args []string) error {
@@ -110,16 +128,46 @@ func overlayPlace(ctx context.Context, inv *inventory.Inventory, args []string)
}
}
// A hub that took over a tunnel listens on that tunnel's port — it is what the peers dial, and
// the reason the port is worth having (novox/hq ADR 0105). An endpoint on another port would
// have the mesh's interface up where no peer is listening for it.
found, err := inv.NodeByName(ctx, node)
if err != nil {
return err
}
var tunnel inventory.Tunnel
adoptsTunnel := false
if *hub && found.Adopted {
if t, err := inv.TunnelOf(ctx, node); err == nil {
tunnel = t
placed, _ := inv.Overlays(ctx)
for _, o := range placed {
if o.Name == node && o.Key == t.PublicKey {
adoptsTunnel = true
}
}
} else if !errors.Is(err, inventory.ErrNoTunnel) {
return err
}
}
if adoptsTunnel {
if port := portOfEndpoint(*endpoint); port != strconv.Itoa(tunnel.Port) {
return fmt.Errorf("%s takes over the tunnel it found on %s, which listens on port %d, and "+
"its endpoint %q names another port: the peers dial the tunnel's port, so the hub's "+
"endpoint must be on it", node, tunnel.Interface, tunnel.Port, *endpoint)
}
}
// Declared, all three. The address is evidence of reachability and is not the fact, and hub
// election by address prefix fails silently (novox/hq ADR 0007).
if err := inv.SetPlace(ctx, node, *endpoint, *site, *hub, ""); err != nil {
return err
}
found, err := inv.NodeByName(ctx, node)
cidr, err := overlayRange(ctx, inv)
if err != nil {
return err
}
address, err := inv.AssignAddress(ctx, found.ID, overlayCIDR())
address, err := inv.AssignAddress(ctx, found.ID, cidr)
if err != nil {
return err
}
@@ -128,6 +176,10 @@ func overlayPlace(ctx context.Context, inv *inventory.Inventory, args []string)
fmt.Println(" credentials issued for it before this placement keep their old broker address —" +
" `module issue` them again and push (novox/hq issue 059)")
switch {
case adoptsTunnel:
fmt.Printf(" the hub — it takes over the tunnel it found on %s: range %s, port %d, "+
"%d peer(s) carried until they enrol\n", tunnel.Interface, tunnel.Range, tunnel.Port,
len(tunnel.Peers))
case *hub:
fmt.Println(" the hub — every node not sharing a site routes through it")
case *endpoint == "":
@@ -154,15 +206,47 @@ func network(ctx context.Context, inv *inventory.Inventory, on map[string]bool,
if err != nil {
return nil, err
}
// The tunnels adopted nodes take over, and the peers the hub's carries (novox/hq ADR 0105).
tunnels, err := inv.Tunnels(ctx)
if err != nil {
return nil, err
}
carried, err := inv.CarriedPeers(ctx)
if err != nil {
return nil, err
}
nodes := make([]overlay.Node, 0, len(places))
for _, p := range places {
if !on[p.Name] {
continue
}
nodes = append(nodes, overlay.Node{
n := overlay.Node{
Name: p.Name, Key: p.Key, Endpoint: p.Endpoint,
Site: p.Site, Hub: p.Hub, Address: p.Address,
})
}
if t, takes := tunnels[p.Name]; takes && t.NodeAdopted {
// Only an adopted node is told to take the found unit over: on a converged one there
// is nothing found to keep, and the host refuses the field. The range and the carried
// peers do not depend on the mode; the takeover does.
//
// **Refused, not composed, when the hub's record disagrees with the tunnel.** A
// declaration that stopped the found unit and raised the mesh's interface on another
// port or address would leave every peer dark while reporting the tunnel taken — so a
// hub placed before it took the tunnel over (or at the wrong port) is named here, and
// nothing is sent until it is re-placed.
if wrong := disagrees(p, t.Tunnel); wrong != "" {
return nil, fmt.Errorf("%s takes over the tunnel on %s and its placement disagrees with it: %s. "+
"Re-place it — `overlay place %s --hub --endpoint <host>:%d …` — and push again; "+
"nothing was composed", p.Name, t.Interface, wrong, p.Name, t.Port)
}
n.TakesOver = &overlay.TakeOver{Interface: t.Interface, Unit: t.Unit, Config: t.Config}
}
if p.Hub {
for _, c := range carried {
n.Carried = append(n.Carried, overlay.Carried{Key: c.PublicKey, Address: c.Address})
}
}
nodes = append(nodes, n)
}
if len(nodes) == 0 {
// Nobody was given it. An empty network is a legitimate mesh, not a broken one, so this
@@ -170,7 +254,11 @@ func network(ctx context.Context, inv *inventory.Inventory, on map[string]bool,
// "no hub" to somebody who never asked for a network would be a lie about the cause.
return overlay.Empty(), nil
}
g, err := overlay.From(nodes, overlayCIDR(), "")
cidr, err := overlayRange(ctx, inv)
if err != nil {
return nil, err
}
g, err := overlay.From(nodes, cidr, "")
if g != nil {
// The artifact store, as this network reaches it. Found rather than configured: the
// provider is whichever module offers it, on whichever machine holds that module — and if
@@ -292,6 +380,17 @@ func overlayShow(ctx context.Context, open *stores) error {
return nil
}
// The tunnel the hub took over, if any, and the peers carried from it (novox/hq ADR 0105):
// listed apart from the nodes, because they are peers of the tunnel and not nodes of the
// mesh until they enrol — and once one has, it is listed as the node it became.
tunnel, hubName, adopted, err := open.inventory.AdoptedTunnel(ctx)
if err != nil {
return err
}
carried, err := open.inventory.CarriedPeers(ctx)
if err != nil {
return err
}
for _, n := range nodes {
place := n.Address
if place == "" {
@@ -301,22 +400,74 @@ func overlayShow(ctx context.Context, open *stores) error {
}
fmt.Printf("%-16s %-14s", n.Name, place)
switch {
case n.Hub && adopted:
fmt.Printf(" hub — over the tunnel it took over on %s (range %s, port %d)",
tunnel.Interface, tunnel.Range, tunnel.Port)
case n.Hub && hubName == n.Name && tunnel.Interface != "":
fmt.Printf(" hub — found a tunnel on %s and did NOT take it over: its key is not the tunnel's; "+
"`mesh-host overlay take --tunnel %s` on the machine takes it", tunnel.Interface, tunnel.Interface)
case n.Hub:
fmt.Print(" hub")
fmt.Print(" hub — found no tunnel; if the machine runs the predecessor's, " +
"`mesh-host overlay take --tunnel <iface>` there adopts it (novox/hq ADR 0105)")
case !n.Reachable():
fmt.Print(" not dialable")
}
if n.Site != "" {
fmt.Printf(" at %s", n.Site)
}
if n.TakesOver != nil && !n.Hub {
fmt.Printf(" takes over %s", n.TakesOver.Interface)
}
fmt.Println()
for _, p := range computed[n.Name] {
fmt.Printf(" → %-14s %-18s %s\n", p.Name, p.Allowed, p.Why)
}
}
if len(carried) > 0 {
fmt.Printf("\npeers of the tunnel %s took over — not nodes of the mesh until they enrol:\n", hubName)
for _, c := range carried {
state := "not yet enrolled"
if c.EnrolledAs != "" {
state = "enrolled as " + c.EnrolledAs + ", which keeps this address"
}
fmt.Printf(" %-16s %-14s %s\n", overlay.CarriedName(c.PublicKey), c.Address, state)
}
}
return nil
}
// disagrees says how a node's placement differs from the tunnel it takes over — its address not
// the tunnel's, its endpoint not on the tunnel's port — or nothing when both agree.
func disagrees(p inventory.Overlay, t inventory.Tunnel) string {
var wrong []string
want := t.Address
if i := strings.Index(want, "/"); i >= 0 {
want = want[:i]
}
if p.Address != want {
wrong = append(wrong, fmt.Sprintf("its address is %s and the tunnel's is %s", orNothing(p.Address), want))
}
if p.Reachable() && portOfEndpoint(p.Endpoint) != strconv.Itoa(t.Port) {
wrong = append(wrong, fmt.Sprintf("its endpoint %s is not on the tunnel's port %d", p.Endpoint, t.Port))
}
return strings.Join(wrong, "; ")
}
func orNothing(s string) string {
if s == "" {
return "unset"
}
return s
}
// portOfEndpoint is the port in host:port, or empty.
func portOfEndpoint(endpoint string) string {
if i := strings.LastIndex(endpoint, ":"); i >= 0 {
return endpoint[i+1:]
}
return ""
}
// SilentFor is how long a node may be quiet before the mesh says so.
//
// A node speaks every minute, so three of them missed is a gap rather than a slow one. The number
@@ -348,11 +499,28 @@ func whereEveryoneIs(ctx context.Context, inv *inventory.Inventory,
return nil, errors.New(
"asked where everyone is without the catalogue, which cannot be answered")
}
places, err := inv.Overlays(ctx)
places, err := onTheNetwork(ctx, inv, shelf)
if err != nil {
return nil, err
}
out := map[string]string{}
for _, p := range places {
out[p.Name] = overlay.InternalName(p.Name)
}
return out, nil
}
// onTheNetwork is every placed machine that resolves the private network — has an address AND
// runs what puts it there. "Has an address" alone was true of every placed machine and told you
// nothing about whether anything could reach it; a name written for such a machine resolves to
// an address that does not answer, and a connection to it hangs (novox/hq issue 079).
func onTheNetwork(ctx context.Context, inv *inventory.Inventory,
shelf map[string]catalogue.Manifest) ([]inventory.Overlay, error) {
places, err := inv.Overlays(ctx)
if err != nil {
return nil, err
}
var out []inventory.Overlay
for _, p := range places {
if p.Address == "" {
continue
@@ -371,7 +539,7 @@ func whereEveryoneIs(ctx context.Context, inv *inventory.Inventory,
for _, m := range got.Modules {
for _, offered := range m.Offers() {
if offered == overlay.Requirement {
out[p.Name] = overlay.InternalName(p.Name)
out = append(out, p)
}
}
}
@@ -379,14 +547,17 @@ func whereEveryoneIs(ctx context.Context, inv *inventory.Inventory,
return out, nil
}
// onThePrivateNetwork is every node's address on the overlay, sorted.
// onThePrivateNetwork is every node's address on the private network, sorted — the same set
// the names and the resolver mean by it (onTheNetwork), so a rule saying "from the mesh" admits
// exactly the machines the mesh names.
//
// A node with no address is left out rather than rendered as an empty source: an empty entry in a
// source set is a syntax error in the rule file, and a rule file that does not load leaves the
// node filtering whatever it was filtering before -- the one outcome worse than a wrong rule,
// because nothing reports it.
func onThePrivateNetwork(ctx context.Context, inv *inventory.Inventory) ([]string, error) {
places, err := inv.Overlays(ctx)
func onThePrivateNetwork(ctx context.Context, inv *inventory.Inventory,
shelf map[string]catalogue.Manifest) ([]string, error) {
places, err := onTheNetwork(ctx, inv, shelf)
if err != nil {
return nil, err
}
@@ -400,29 +571,31 @@ func onThePrivateNetwork(ctx context.Context, inv *inventory.Inventory) ([]strin
return out, nil
}
// namesInTheMesh is every machine's internal name and the address behind it.
// namesInTheMesh is every machine's internal name and the address behind it — every machine
// that is on the private network, the same set the resolver means by that.
//
// A machine with no address has no name: writing one that resolves to nothing is worse than not
// A machine that is not has no name: writing one that resolves to nothing is worse than not
// writing it, because a connection to an address that does not answer hangs where a name that
// does not resolve fails at once and says so. That is the rule the hosts file already follows,
// and this is the same set read the same way.
func namesInTheMesh(ctx context.Context, inv *inventory.Inventory) (map[string]string, error) {
places, err := inv.Overlays(ctx)
// does not resolve fails at once and says so. A machine placed on the overlay but not running
// the module that puts it there is exactly that (novox/hq issue 079).
func namesInTheMesh(ctx context.Context, inv *inventory.Inventory,
shelf map[string]catalogue.Manifest) (map[string]string, error) {
places, err := onTheNetwork(ctx, inv, shelf)
if err != nil {
return nil, err
}
out := map[string]string{}
for _, p := range places {
if strings.TrimSpace(p.Address) == "" {
continue
}
out[overlay.InternalName(p.Name)] = p.Address
}
return out, nil
}
// artifactStoreOnNetwork is the machine and port the mesh's artifact store answers on, when a
// module providing it is assigned to a machine that is on the private network.
// artifactStoreOnNetwork is the machine on this network that offers the artifact store, and the
// port THAT MACHINE put it on — the node's setting when it was given one (novox/hq ADR 0100,
// 04-ISSUES/102), the mesh's assignment when it made one, and the manifest's own number only when
// neither says anything. Read exactly as a consumer's binding is, because the trust a machine
// writes for the store and the address it pulls from are the same fact as what a consumer is told.
//
// A lookup failure is an error, never "not found": collapsing the two composed a declaration
// without the trust whenever the inventory hiccuped, delivered by a push that reported success —
@@ -435,29 +608,87 @@ func artifactStoreOnNetwork(ctx context.Context, inv *inventory.Inventory,
if err != nil {
return "", "", false, fmt.Errorf("reading the catalogue: %w", err)
}
providers := map[string]string{} // module -> served port
providers := map[string]catalogue.Manifest{}
for name, m := range shelf {
served, offers := m.Serves[catalogue.ArtifactStoreProvision]
if !offers {
continue
}
if p, ok := served["port"]; ok {
providers[name] = fmt.Sprintf("%v", p)
if _, offers := m.Serves[catalogue.ArtifactStoreProvision]; offers {
providers[name] = m
}
}
if len(providers) == 0 {
return "", "", false, nil
}
// In a stated order, so two machines offering it would always answer the same one.
machines := make([]string, 0, len(on))
for machine := range on {
machines = append(machines, machine)
}
sort.Strings(machines)
for _, machine := range machines {
assigned, err := inv.Assigned(ctx, machine)
if err != nil {
return "", "", false, fmt.Errorf("reading what %s is assigned: %w", machine, err)
}
for _, a := range assigned {
if p, ok := providers[a]; ok {
return machine, p, true, nil
m, offers := providers[a]
if !offers {
continue
}
serves, err := servedOnNode(ctx, inv, machine, m, catalogue.ArtifactStoreProvision)
if err != nil {
return "", "", false, fmt.Errorf("reading where %s puts the artifact store: %w", machine, err)
}
if p, ok := serves["port"]; ok {
return machine, fmt.Sprintf("%v", p), true, nil
}
}
}
return "", "", false, nil
}
// artifactStoreAddress is the artifact store as `forNode` reaches it: `<node>.internal:<port>`
// over the private network, or — when nothing is on the network yet — `127.0.0.1:<port>` for the
// node that holds the store itself, and "" for any other. The address composed into every
// reference the mesh built, at the moment it is used and never before (novox/hq 04-ISSUES/102).
//
// **Genesis places the network after the store, the broker, the vault and the catalogue.** Each
// of those is built and pushed to a node that is on no network, and the store is on that same
// node; an answer of "no store" there would refuse every one of those pushes and hand every one
// of those builds a base nothing can pull. Loopback is the truth on that machine, and it is the
// address genesis itself reaches the store by.
func artifactStoreAddress(ctx context.Context, inv *inventory.Inventory,
shelf map[string]catalogue.Manifest, forNode string) (string, error) {
onNetwork, err := whereEveryoneIs(ctx, inv, shelf)
if err != nil {
return "", err
}
on := map[string]bool{}
for name := range onNetwork {
on[name] = true
}
node, port, found, err := artifactStoreOnNetwork(ctx, inv, on)
if err != nil {
return "", err
}
if found {
return overlay.InternalName(node) + ":" + port, nil
}
holder, port, found, err := artifactStoreHolder(ctx, inv)
if err != nil || !found || holder != forNode {
return "", err
}
return "127.0.0.1:" + port, nil
}
// artifactStoreHolder is whichever node is assigned a module offering the artifact store, on or
// off the network, and the port that node put it on.
func artifactStoreHolder(ctx context.Context, inv *inventory.Inventory) (node, port string, found bool, err error) {
nodes, err := inv.Nodes(ctx)
if err != nil {
return "", "", false, err
}
all := map[string]bool{}
for _, n := range nodes {
all[n.Name] = true
}
return artifactStoreOnNetwork(ctx, inv, all)
}
+227
View File
@@ -2,10 +2,13 @@ package main
import (
"context"
"os"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/overlay"
)
// placementOf is what the mesh holds about where one node is.
@@ -76,3 +79,227 @@ func TestAPlacementAndNothingTogetherIsRefused(t *testing.T) {
t.Fatal("a placement and --nothing together was accepted")
}
}
// A machine is named for the others only while it is on the private network — placed AND running
// what puts it there — the same set the resolver means by "on the private network". A machine
// that has an address and no networking is not named: a name resolving to an address that does
// not answer hangs where an unknown name fails at once (novox/hq issue 079).
func TestOnlyAMachineOnThePrivateNetworkIsNamed(t *testing.T) {
open := aMesh(t) // two placed machines, both assigned what puts them on the private network
ctx := t.Context()
shelf, err := open.inventory.Catalogue(ctx)
if err != nil {
t.Fatal(err)
}
names, err := namesInTheMesh(ctx, open.inventory, shelf)
if err != nil {
t.Fatal(err)
}
if names["anchor.internal"] != "10.77.0.1" || names["laptop.internal"] != "10.77.0.2" {
t.Fatalf("two machines on the network are not both named: %v", names)
}
// The laptop keeps its place and its address, and stops running the network.
if err := open.inventory.Unassign(ctx, "laptop", overlay.Name); err != nil {
t.Fatal(err)
}
names, err = namesInTheMesh(ctx, open.inventory, shelf)
if err != nil {
t.Fatal(err)
}
if _, still := names["laptop.internal"]; still || names["anchor.internal"] != "10.77.0.1" {
t.Fatalf("a machine that left the network is still named, or the one that stayed is not: %v", names)
}
}
// novox/hq ADR 0105: the range every address is composed from is the adopted tunnel's, read from
// the tunnel the hub holds — never stored anywhere else.
func TestTheOverlaysRangeIsTheAdoptedTunnels(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
t.Setenv(OverlayCIDRVar, "10.99.0.0/16")
before, err := overlayRange(ctx, inv)
if err != nil || before != "10.99.0.0/16" {
t.Fatalf("without an adopted tunnel the range is not what genesis said: %q %v", before, err)
}
// The hub becomes what genesis makes of a machine in use: adopted, enrolled with the found
// tunnel's key, and presenting the tunnel.
if err := inv.SetAdopted(ctx, "anchor", true); err != nil {
t.Fatal(err)
}
hub, err := inv.NodeByName(ctx, "anchor")
if err != nil {
t.Fatal(err)
}
const key = "THE-TUNNELS-KEY========================="
if err := inv.RecordOverlayKey(ctx, hub.ID, key); err != nil {
t.Fatal(err)
}
if err := inv.RecordTunnel(ctx, hub.ID, inventory.Tunnel{
Interface: "wg0", Unit: "wg-quick@wg0", Config: "/etc/wireguard/wg0.conf", Port: 51900,
Address: "192.0.2.1/24", Range: "192.0.2.0/24", PublicKey: key,
Peers: []inventory.TunnelPeer{{PublicKey: "PEER-TWO", Address: "192.0.2.2"}},
}); err != nil {
t.Fatal(err)
}
after, err := overlayRange(ctx, inv)
if err != nil || after != "192.0.2.0/24" {
t.Fatalf("with an adopted tunnel the range is %q (%v), not the tunnel's", after, err)
}
// A placement whose endpoint is on another port than the tunnel's is refused: the peers dial
// the tunnel's port.
err = overlayPlace(ctx, inv, []string{"anchor", "--endpoint", "198.51.100.10:51820", "--site", "hosting", "--hub"})
if err == nil || !strings.Contains(err.Error(), "51900") {
t.Fatalf("an endpoint off the tunnel's port was accepted: %v", err)
}
// On the tunnel's port, the hub is placed at the tunnel's address — whatever it had before.
if err := inv.SetPlace(ctx, "anchor", "", "", false, ""); err != nil {
t.Fatal(err)
}
if err := overlayPlace(ctx, inv, []string{"anchor", "--endpoint", "198.51.100.10:51900", "--site", "hosting", "--hub"}); err != nil {
t.Fatal(err)
}
if placed := placementOf(t, ctx, inv, "anchor"); placed.Address != "192.0.2.1" {
t.Fatalf("the hub was placed at %s, not the tunnel's own address", placed.Address)
}
// And the hub's declaration carries the peer and the takeover.
nodes, computed, err := graph(ctx, open)
if err != nil {
t.Fatal(err)
}
var hubNode overlay.Node
for _, n := range nodes {
if n.Name == "anchor" {
hubNode = n
}
}
if hubNode.TakesOver == nil || hubNode.TakesOver.Unit != "wg-quick@wg0" {
t.Errorf("the hub is not told to take over the found tunnel: %+v", hubNode)
}
carried := false
for _, p := range computed["anchor"] {
if p.Key == "PEER-TWO" && p.Allowed == "192.0.2.2/32" {
carried = true
}
}
if !carried {
t.Errorf("the hub's peer list does not carry the tunnel's peer: %+v", computed["anchor"])
}
}
// A takeover is composed only for a hub whose placement agrees with the tunnel: an address or an
// endpoint port that differs would have the host stop the found interface and raise the mesh's
// where no peer is listening.
func TestATakeoverIsNotComposedForAHubPlacedOffItsTunnel(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
if err := inv.SetAdopted(ctx, "anchor", true); err != nil {
t.Fatal(err)
}
hub, err := inv.NodeByName(ctx, "anchor")
if err != nil {
t.Fatal(err)
}
const key = "THE-TUNNELS-KEY========================="
if err := inv.RecordOverlayKey(ctx, hub.ID, key); err != nil {
t.Fatal(err)
}
if err := inv.RecordTunnel(ctx, hub.ID, inventory.Tunnel{
Interface: "wg0", Unit: "wg-quick@wg0", Config: "/etc/wireguard/wg0.conf", Port: 51900,
Address: "192.0.2.1/24", Range: "192.0.2.0/24", PublicKey: key}); err != nil {
t.Fatal(err)
}
// aMesh placed anchor at 10.77.0.1 on :51820 — the record of a hub placed before it took the
// tunnel over.
_, _, err = graph(ctx, open)
if err == nil {
t.Fatal("a takeover was composed for a hub whose address and port are not the tunnel's")
}
for _, want := range []string{"10.77.0.1", "192.0.2.1", "51820", "51900", "overlay place anchor"} {
if !strings.Contains(err.Error(), want) {
t.Errorf("the refusal does not say %q: %v", want, err)
}
}
// Re-placed on the tunnel, it composes.
if err := inv.SetPlace(ctx, "anchor", "", "", false, ""); err != nil {
t.Fatal(err)
}
if err := overlayPlace(ctx, inv, []string{"anchor", "--endpoint", "198.51.100.10:51900", "--site", "here", "--hub"}); err != nil {
t.Fatal(err)
}
if _, _, err := graph(ctx, open); err != nil {
t.Fatalf("re-placed on the tunnel, the graph still refuses: %v", err)
}
}
// theResolver is the catalogue's dnsmasq module as it is, or the test is skipped where the
// catalogue is not beside this checkout.
func theResolver(t *testing.T) catalogue.Manifest {
t.Helper()
raw, err := os.ReadFile("../../../mesh-catalog/modules/dnsmasq/module.json")
if err != nil {
t.Skipf("the catalogue is not beside this checkout: %v", err)
}
m, err := catalogue.ParseManifest(raw)
if err != nil {
t.Fatalf("dnsmasq does not parse:\n%v", err)
}
return m
}
// The resolver is handed every machine on the private network as a wildcard, the same set and the
// same source as the hosts file, and is handed it again when a machine leaves — through the
// module's own manifest asking for the fact, with no module of the mesh's own in between (hal
// dnsmasq-app conversion, novox/hq 08-connectivity). The runtime on that machine is pointed at the
// machine's own address, where the resolver answers for its containers.
func TestTheResolverIsToldEveryMachineOnTheNetworkAndToldAgainWhenOneLeaves(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
register(t, open, theResolver(t))
if _, err := assign(ctx, open, "anchor", "dnsmasq"); err != nil {
t.Fatal(err)
}
zones := func() string {
t.Helper()
for _, r := range composed(t, open, "anchor").Resources {
if r["id"] == "dnsmasq.fact-node-zones" {
if r["path"] != "/etc/mesh-resolver/nodes.conf" {
t.Fatalf("the machines were written somewhere the resolver does not read: %v", r["path"])
}
return r["content"].(string)
}
}
t.Fatal("the resolver was not handed the machines")
return ""
}
first := zones()
for _, want := range []string{
"local=/internal/", "address=/anchor.internal/10.77.0.1\n", "address=/laptop.internal/10.77.0.2\n",
} {
if !strings.Contains(first, want) {
t.Errorf("the resolver's machines lack %q:\n%s", want, first)
}
}
for _, r := range composed(t, open, "anchor").Resources {
if r["id"] == "dnsmasq.runtime-dns" {
if !strings.Contains(r["content"].(string), `"10.77.0.1"`) || r["into"] != "json" {
t.Errorf("the runtime is not pointed at this machine's own address, written into its file: %v", r)
}
}
}
// The laptop keeps its place and its address, and stops running the network.
if err := open.inventory.Unassign(ctx, "laptop", overlay.Name); err != nil {
t.Fatal(err)
}
after := zones()
if strings.Contains(after, "laptop") || !strings.Contains(after, "address=/anchor.internal/10.77.0.1\n") {
t.Fatalf("a machine that left the network is still a wildcard, or the one that stayed is not:\n%s", after)
}
}
+81 -23
View File
@@ -38,15 +38,7 @@ func nodeCommand(ctx context.Context, args []string) error {
}
return showNode(ctx, inv, args[1])
case "add":
if len(args) != 2 {
return errors.New("node add <name>")
}
node, err := inv.AddNode(ctx, args[1])
if err != nil {
return err
}
fmt.Printf("added %s (%s)\n", node.Name, node.ID)
return nil
return addNode(ctx, inv, args[1:])
case "list":
nodes, err := inv.Nodes(ctx)
@@ -61,7 +53,7 @@ func nodeCommand(ctx context.Context, args []string) error {
return nil
}
for _, n := range nodes {
fmt.Printf("%-20s %-14s %s\n", n.Name, heardFrom(n), n.ID)
fmt.Printf("%-20s %-14s %-9s %s\n", n.Name, heardFrom(n), modeOf(n), n.ID)
}
return nil
@@ -80,6 +72,39 @@ func nodeCommand(ctx context.Context, args []string) error {
}
}
// addNode creates a node record, adopted when the operator says so (novox/hq ADR 0100).
func addNode(ctx context.Context, inv *inventory.Inventory, args []string) error {
set := flag.NewFlagSet("node add", flag.ContinueOnError)
adopted := set.Bool("adopted", false,
"the machine is in use: keep what is found on it until each module is taken")
positionals, err := parseAround(set, args)
if err != nil {
return err
}
if len(positionals) != 1 {
return errors.New("node add <name> [--adopted]")
}
node, err := inv.AddNodeAs(ctx, positionals[0], *adopted)
if err != nil {
return err
}
fmt.Printf("added %s (%s)", node.Name, node.ID)
if node.Adopted {
fmt.Print(", adopted")
}
fmt.Println()
return nil
}
// modeOf is a node's mode as a word (novox/hq ADR 0100): an adopted node is said to be adopted
// wherever the mesh reports a node's state.
func modeOf(n inventory.Node) string {
if n.Adopted {
return "adopted"
}
return "converged"
}
// publicDomainUsage is the one description of the three forms, so a refusal and the help agree.
const publicDomainUsage = "node public-domain <name> — what it is now; " +
"<name> <domain> to set it; <name> --clear to take it away"
@@ -153,6 +178,8 @@ func tokenCommand(ctx context.Context, args []string) error {
existing := set.String("node", "", "issue for a node record that already exists")
fresh := set.String("new", "", "create the node record, then issue for it")
validFor := set.Duration("for", time.Hour, "how long the token may be used")
adopted := set.Bool("adopted", false,
"the machine joining is in use: it is adopted, and keeps what is found on it")
if err := set.Parse(args[1:]); err != nil {
return err
}
@@ -171,16 +198,7 @@ func tokenCommand(ctx context.Context, args []string) error {
defer open.Close()
inv := open.inventory
name := *existing
if *fresh != "" {
node, err := inv.AddNode(ctx, *fresh)
if err != nil {
return err
}
name = node.Name
}
issued, err := inv.IssueToken(ctx, name, *validFor)
issued, err := issueFor(ctx, inv, *existing, *fresh, *adopted, *validFor)
if err != nil {
return err
}
@@ -211,7 +229,8 @@ func tokenCommand(ctx context.Context, args []string) error {
return err
}
made := token.Token{Node: issued.Node.Name, Signer: key.Public, Secret: issued.Secret}
made := token.Token{Node: issued.Node.Name, Signer: key.Public, Secret: issued.Secret,
Adopted: issued.Node.Adopted}
// Absent is a state, not a failure: a control plane can hold records and a key before it has
// a broker. What it cannot do is issue a token anybody could use, and Missing() says so.
@@ -228,8 +247,12 @@ func tokenCommand(ctx context.Context, args []string) error {
return err
}
fmt.Printf("token for %s, usable once, until %s\n\n %s\n\n",
issued.Node.Name, issued.Expires.Format(time.RFC3339), encoded)
joins := ""
if made.Adopted {
joins = ", joining adopted"
}
fmt.Printf("token for %s%s, usable once, until %s\n\n %s\n\n",
issued.Node.Name, joins, issued.Expires.Format(time.RFC3339), encoded)
fmt.Println("This is the only time it is shown. What is stored is a hash of the secret.")
if missing := made.Missing(); len(missing) > 0 {
@@ -243,6 +266,38 @@ func tokenCommand(ctx context.Context, args []string) error {
return nil
}
// issueFor is the inventory's half of issuing a token: the record, made when it is new, adopted
// when the operator says so, and the one-time secret for it. The node in what it returns carries
// its mode, which is what the token says.
func issueFor(ctx context.Context, inv *inventory.Inventory, existing, fresh string, adopted bool,
validFor time.Duration) (inventory.Issued, error) {
name := existing
if fresh != "" {
node, err := inv.AddNodeAs(ctx, fresh, adopted)
if err != nil {
return inventory.Issued{}, err
}
name = node.Name
}
// Not saying adopted leaves the node as it is: re-issuing a token for an adopted node does not
// converge it — converging is its own act, previewed (novox/hq ADR 0100). And saying it for a
// node already converged is refused rather than done quietly: returning a node to adopted is
// its own act too, which unloads the mesh's filter and enables the found firewall again.
if adopted && fresh == "" {
node, err := inv.NodeByName(ctx, name)
if err != nil {
return inventory.Issued{}, err
}
if !node.Adopted {
return inventory.Issued{}, fmt.Errorf("%s is converged, and a token does not change "+
"that: run `adopt %s` to return it to adopted, then issue the token without "+
"--adopted", name, name)
}
}
return inv.IssueToken(ctx, name, validFor)
}
func identityCommand(ctx context.Context, args []string) error {
if len(args) == 0 || args[0] != "show" {
return errors.New("identity show")
@@ -334,6 +389,9 @@ func showNode(ctx context.Context, inv *inventory.Inventory, name string) error
}
fmt.Printf("%s\n", node.Name)
fmt.Printf(" last heard from %s\n", heardFrom(node))
if err := showMode(ctx, inv, node); err != nil {
return err
}
// The domain its routed names are composed under, when it has one (novox/hq ADR 0066). Shown
// only when set: a machine that serves nothing to the outside has no domain, and saying so of
+55
View File
@@ -3,6 +3,7 @@ package main
import (
"strings"
"testing"
"time"
)
// **A read-shaped invocation is never a destructive write.**
@@ -97,3 +98,57 @@ func TestAskingAboutAMachineTheMeshHasNeverHeardOfIsRefused(t *testing.T) {
t.Fatal("a name the mesh does not know was answered as if it were a machine")
}
}
// novox/hq ADR 0100: the operator says a node is adopted — `node add --adopted` for a record, and
// the token for a machine joining.
func TestANodeAddedAdoptedIsAdopted(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
if err := addNode(ctx, open.inventory, []string{"joiner", "--adopted"}); err != nil {
t.Fatal(err)
}
n, err := open.inventory.NodeByName(ctx, "joiner")
if err != nil {
t.Fatal(err)
}
if !n.Adopted {
t.Fatal("node add --adopted made a converged node")
}
if err := addNode(ctx, open.inventory, []string{"plain"}); err != nil {
t.Fatal(err)
}
if n, _ := open.inventory.NodeByName(ctx, "plain"); n.Adopted {
t.Fatal("node add without --adopted made an adopted node")
}
}
func TestATokenIssuedAdoptedSaysSoAndReissuingDoesNotConverge(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
issued, err := issueFor(ctx, open.inventory, "", "joiner", true, time.Hour)
if err != nil {
t.Fatal(err)
}
if !issued.Node.Adopted {
t.Fatal("a token issued --adopted is for a node that is not adopted")
}
again, err := issueFor(ctx, open.inventory, "joiner", "", false, time.Hour)
if err != nil {
t.Fatal(err)
}
if !again.Node.Adopted {
t.Fatal("re-issuing without --adopted converged the node; converging is its own act")
}
// --adopted for a node already converged is refused, and points at the act that does it.
if _, err := issueFor(ctx, open.inventory, "laptop", "", true, time.Hour); err == nil ||
!strings.Contains(err.Error(), "adopt laptop") {
t.Fatalf("--adopted on a converged node was not refused: %v", err)
}
if n, _ := open.inventory.NodeByName(ctx, "laptop"); n.Adopted {
t.Fatal("a refused token flipped the node to adopted")
}
// And said for a node that is adopted already, it is the ordinary re-issue.
if _, err := issueFor(ctx, open.inventory, "joiner", "", true, time.Hour); err != nil {
t.Fatal(err)
}
}
+275 -35
View File
@@ -1,6 +1,7 @@
package main
import (
"bytes"
"context"
"encoding/json"
"errors"
@@ -13,6 +14,7 @@ import (
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/licences"
"github.com/novox/mesh-controller/internal/overlay"
"net"
"strconv"
)
@@ -122,7 +124,7 @@ func planFor(ctx context.Context, open *stores, nodeName string) (catalogue.Reso
}
continue
}
secret, err := inv.SecretFor(ctx, n.Name, nodeName, n.For, n.From)
secret, err := inv.SecretFor(ctx, n.Name, nodeName, n.For, n.From, n.Local)
if err != nil {
// Said rather than skipped. A machine that resolves cleanly and receives no
// credential is one that will fail to authenticate at some later, less obvious
@@ -227,21 +229,10 @@ func theRestOfTheMesh(ctx context.Context, inv *inventory.Inventory,
// What that module says a consumer needs to know, with that node's settings on
// it: a port somebody moved on the provider is a port its consumers must be told
// about, and the two coming from different places is how they come to disagree.
assigned, err := portsOn(ctx, inv, o.node.Name, m.Module)
serves, err := servedOnNode(ctx, inv, o.node.Name, m, name)
if err != nil {
return catalogue.World{}, err
}
serves := catalogue.ServedOn(m, name, assigned)
if len(serves) > 0 {
layers, err := inv.SettingsFor(ctx, o.node.Name, m.Module)
if err != nil {
return catalogue.World{}, err
}
serves, err = catalogue.Settle(serves, layers)
if err != nil {
return catalogue.World{}, err
}
}
offered[name] = append(offered[name], catalogue.Provider{
Node: o.node.Name, At: o.node.At, Serves: serves})
}
@@ -270,10 +261,10 @@ func theRestOfTheMesh(ctx context.Context, inv *inventory.Inventory,
// silently produced a declaration missing them — a difference between what `plan` showed and what
// `plan --json` handed to anything reading it.
func declarationFor(ctx context.Context, open *stores, node string,
plan catalogue.Resolution, settings catalogue.SettingsBy) ([]map[string]any, error) {
plan catalogue.Resolution, settings catalogue.SettingsBy) (sendable, error) {
gens, err := generators(ctx, open)
if err != nil {
return nil, err
return sendable{}, err
}
// **Allocating, because `plan` is the send without the sending.** It is one machine, named by
// a person, who is asking what a push would do — so the port it shows and the secret it seals
@@ -315,11 +306,31 @@ const (
func declarationWith(ctx context.Context, open *stores, node string,
plan catalogue.Resolution, settings catalogue.SettingsBy,
gens map[string]catalogue.Generator, choosing Choosing) ([]map[string]any, error) {
gens map[string]catalogue.Generator, choosing Choosing) (sendable, error) {
with, record, err := renderingFor(ctx, open, node, plan, settings, gens, choosing)
if err != nil {
return sendable{}, err
}
composed, err := plan.Compose(with)
if err != nil {
return sendable{}, err
}
// And what was taken on it, said in every declaration it is sent from this one place.
adoption, err := adoptionOf(ctx, open.inventory, record, plan, composed)
if err != nil {
return sendable{}, err
}
return sendable{Resources: composed.Resources, Adoption: adoption}, nil
}
// renderingFor is everything a node's declaration is composed with, and the node's record.
func renderingFor(ctx context.Context, open *stores, node string,
plan catalogue.Resolution, settings catalogue.SettingsBy,
gens map[string]catalogue.Generator, choosing Choosing) (catalogue.Rendering, inventory.Node, error) {
inv := open.inventory
grants, err := grantsFor(ctx, open, node)
if err != nil {
return nil, err
return catalogue.Rendering{}, inventory.Node{}, err
}
// Where this machine puts what each module needs reachable (novox/hq ADR 0038).
//
@@ -332,7 +343,7 @@ func declarationWith(ctx context.Context, open *stores, node string,
if choosing == Reading {
held, err := inv.PortsFor(ctx, node)
if err != nil {
return nil, err
return catalogue.Rendering{}, inventory.Node{}, err
}
for _, a := range held {
if already[a.Module] == nil {
@@ -342,9 +353,44 @@ func declarationWith(ctx context.Context, open *stores, node string,
}
}
// The machine ports this node was given for its modules (novox/hq ADR 0100): the foundation's
// ports, as genesis chose them. A given port wins over anything assigned and over a manifest's
// own long-form mapping.
given := map[string]map[int]int{}
for _, m := range plan.Modules {
g, err := catalogue.GivenPorts(m, settings[m.Module])
if err != nil {
return catalogue.Rendering{}, inventory.Node{}, err
}
if g != nil {
given[m.Module] = g
}
}
// And one holder per machine port across the node's modules: settings written before this was
// refused where they are set are refused here rather than composed into two containers on
// one port.
holders := map[int]string{}
for _, m := range plan.Modules {
for _, at := range given[m.Module] {
if other, twice := holders[at]; twice && other != m.Module {
return catalogue.Rendering{}, inventory.Node{}, fmt.Errorf("%w: %s and %s are "+
"both given machine port %d on %s", inventory.ErrPortTaken, other, m.Module,
at, node)
}
holders[at] = m.Module
}
}
ports := map[string]map[int]int{}
for _, m := range plan.Modules {
for _, l := range m.Listens {
if at, isGiven := given[m.Module][l.Port]; isGiven {
if ports[m.Module] == nil {
ports[m.Module] = map[int]int{}
}
ports[m.Module][l.Port] = at
continue
}
// **Only a port the module actually publishes is the mesh's to move.** A container's
// mapping is the thing that translates; without one the software binds what it binds,
// and an assignment would not move the service — it would open the wrong number in the
@@ -356,7 +402,7 @@ func declarationWith(ctx context.Context, open *stores, node string,
case mayAssign && choosing == Allocating:
at, err := inv.PortFor(ctx, node, m.Module, l.Port, l.Fixed)
if err != nil {
return nil, fmt.Errorf(
return catalogue.Rendering{}, inventory.Node{}, fmt.Errorf(
"%s needs %d reachable on %s and it could not be assigned: %w",
m.Module, l.Port, node, err)
}
@@ -397,7 +443,7 @@ func declarationWith(ctx context.Context, open *stores, node string,
}
}
if err != nil {
return nil, err
return catalogue.Rendering{}, inventory.Node{}, err
}
if needed[m.Module] == nil {
needed[m.Module] = map[string]string{}
@@ -415,7 +461,7 @@ func declarationWith(ctx context.Context, open *stores, node string,
}
issued, meshCA, err := certificateFor(ctx, open, node)
if err != nil {
return nil, err
return catalogue.Rendering{}, inventory.Node{}, err
}
certificate, authority = issued, meshCA
break
@@ -425,26 +471,53 @@ func declarationWith(ctx context.Context, open *stores, node string,
// resolves to. Every node's address, including this one's: a machine reaching itself by its
// own overlay address rather than by loopback is ordinary, and leaving it out would filter
// the node's own traffic to itself with no rule naming why.
private, err := onThePrivateNetwork(ctx, inv)
// One reading of the catalogue for the three questions below that resolve the whole mesh.
shelf, err := inv.Catalogue(ctx)
if err != nil {
return nil, err
return catalogue.Rendering{}, inventory.Node{}, err
}
private, err := onThePrivateNetwork(ctx, inv, shelf)
if err != nil {
return catalogue.Rendering{}, inventory.Node{}, err
}
// The artifact store as this node reaches it now — the address every image and archive the
// mesh built is fetched through, composed here and recorded nowhere — with what the mesh has
// built, so a reference recorded with an address before that is re-routed too.
artifactStore, err := artifactStoreAddress(ctx, inv, shelf, node)
if err != nil {
return catalogue.Rendering{}, inventory.Node{}, err
}
held, err := inv.Held(ctx)
if err != nil {
return catalogue.Rendering{}, inventory.Node{}, err
}
built := make(map[string]bool, len(held))
for repository := range held {
built[repository] = true
}
// And every machine's name, so a container can reach one. The same set that writes the
// machine's own hosts file — one reading, so a container and its machine cannot disagree
// about where another machine is.
names, err := namesInTheMesh(ctx, inv)
names, err := namesInTheMesh(ctx, inv, shelf)
if err != nil {
return nil, err
return catalogue.Rendering{}, inventory.Node{}, err
}
// And every routed name → the node that serves it (novox/hq ADR 0066). Alongside the
// `<node>.internal` names above, so a container — or an internal ACME validator — resolves a
// routed name to the proxy that serves it, mesh-wide. The mesh publishes the names it was told
// to serve and knows nothing about what they mean.
// Kept apart from the machines, because a fact about the machines must not be handed the names
// the mesh merely serves (novox/hq 04-ISSUES/111).
machines := make(map[string]string, len(names))
for name, at := range names {
machines[name] = at
}
routes, err := routeNamesInTheMesh(ctx, open)
if err != nil {
return nil, err
return catalogue.Rendering{}, inventory.Node{}, err
}
for name, at := range routes {
names[name] = at
@@ -473,15 +546,44 @@ func declarationWith(ctx context.Context, open *stores, node string,
continue
}
if kept, err = inv.OperatorExport(ctx); err != nil {
return nil, err
return catalogue.Rendering{}, inventory.Node{}, err
}
break
}
return plan.Declaration(catalogue.Rendering{
// Whether this node is adopted (novox/hq ADR 0100): then the found firewall stays in force, and
// the declaration carries openings and the mesh's guard in place of a filter.
record, err := inv.NodeByName(ctx, node)
if err != nil {
return catalogue.Rendering{}, inventory.Node{}, err
}
// And, on an adopted node, which modules were taken there: the guard is derived from those
// only (novox/hq ADR 0103).
var taken map[string]bool
if record.Adopted {
list, err := inv.Taken(ctx, node)
if err != nil {
return catalogue.Rendering{}, inventory.Node{}, err
}
taken = map[string]bool{}
for _, m := range list {
taken[m] = true
}
}
// Where this node put the foundation's servers, for the control plane's own connections
// (novox/hq 04-ISSUES/102): read from the node's settings for whatever claims each seat,
// exactly as a consumer's binding is, never from what genesis wrote into a secret.
seats, err := seatsOn(ctx, inv, shelf, node, plan.Modules, record.Adopted, taken)
if err != nil {
return catalogue.Rendering{}, inventory.Node{}, err
}
return catalogue.Rendering{
Settings: settings, Generators: gens, Grants: grants, Needed: needed, Ports: ports,
Certificate: certificate, Authority: authority, Mesh: private, Names: names,
Foundation: foundation, Kept: kept})
Machines: machines,
Suffix: overlay.Suffix(), Foundation: foundation, Kept: kept, Adopted: record.Adopted,
Given: given, Taken: taken, Seats: seats, ArtifactStore: artifactStore, Built: built,
}, record, nil
}
// routeNamesInTheMesh is every routed name and the address of the node that serves it (novox/hq
@@ -693,11 +795,27 @@ func grantsFor(ctx context.Context, open *stores, node string) ([]catalogue.Gran
}
out = append(out, catalogue.Grant{
Provision: s.Name, Consumer: s.Consumer, At: onNetwork[s.Consumer],
From: from, Values: values, Slug: slug, Sealed: s.ForProvider})
From: from, Values: values, Slug: slug, Sealed: s.ForProvider, Local: s.Local})
}
return out, nil
}
// listensLines is what a person is told about what this module would open, and why — the same
// `why` every listens entry already carries for the firewall it also feeds (novox/hq ADR 0007), so
// deciding whether to assign a module can see what it would open before it opens it, not only
// after. A module with nothing to listen on prints nothing extra, same as today.
func listensLines(m catalogue.Manifest) []string {
var out []string
for _, l := range m.Listens {
if l.Why == "" {
out = append(out, fmt.Sprintf(" listens %d/%s from %s", l.Port, l.At(), l.From))
continue
}
out = append(out, fmt.Sprintf(" listens %d/%s from %s — %s", l.Port, l.At(), l.From, l.Why))
}
return out
}
func planCommand(ctx context.Context, args []string) error {
set := flag.NewFlagSet("plan", flag.ContinueOnError)
// Because "one resource" does not tell you whether the settings landed. Being able to read
@@ -730,22 +848,30 @@ func planCommand(ctx context.Context, args []string) error {
return nil
}
if *asJSON {
resources, err := declarationFor(ctx, open, args[0], plan, settings)
declared, err := declarationFor(ctx, open, args[0], plan, settings)
if err != nil {
return err
}
body, err := json.MarshalIndent(
map[string]any{"declaration": 1, "resources": resources}, "", " ")
// The bytes a push would send, indented: one marshaller, so `plan --json` cannot show an
// envelope other than the one sent.
body, err := declared.Body()
if err != nil {
return err
}
fmt.Println(string(body))
var indented bytes.Buffer
if err := json.Indent(&indented, body, "", " "); err != nil {
return err
}
fmt.Println(indented.String())
return nil
}
fmt.Printf("%s would run:\n", args[0])
for _, m := range plan.Modules {
fmt.Printf(" %-20s %s\n", m.Module, plan.Because[m.Module])
for _, line := range listensLines(m) {
fmt.Println(line)
}
}
// What was assigned here and cannot run here. Said with the rest rather than as a refusal: it is
// one module on the wrong machine, the others still run, and the remedy is to move this one.
@@ -763,10 +889,11 @@ func planCommand(ctx context.Context, args []string) error {
for _, n := range plan.Needs {
fmt.Printf(" needs %s from %s, for %s\n", n.Name, n.From, n.For)
}
resources, err := declarationFor(ctx, open, args[0], plan, settings)
declared, err := declarationFor(ctx, open, args[0], plan, settings)
if err != nil {
return err
}
resources := declared.Resources
for module, layers := range settings {
for _, layer := range layers {
fmt.Printf(" %-20s settings from %s\n", module, layer.From)
@@ -871,6 +998,119 @@ func managerPublicKeyFor(
return inv.SealingKeyOf(ctx, node)
}
// servedOnNode is what a module on a node tells a consumer of one of its provisions, with THAT
// node's ports on it: the port the node was given (novox/hq ADR 0100) over the one the mesh
// assigned over the manifest's own, settled with the node's settings layers.
//
// **The one derivation** for every reader of a provider's address — a consumer's binding, the
// artifact store's trust and the references composed through it (04-ISSUES/102). Unreadable
// given ports are that node's refusal to report, not this reader's.
func servedOnNode(ctx context.Context, inv *inventory.Inventory, node string,
m catalogue.Manifest, provision string) (map[string]any, error) {
ports, layers, err := portsGivenOn(ctx, inv, node, m)
if err != nil {
return nil, err
}
serves := catalogue.ServedOn(m, provision, ports)
if len(serves) > 0 {
serves, err = catalogue.Settle(serves, layers)
if err != nil {
return nil, err
}
}
return serves, nil
}
// portsGivenOn is where a node puts a module's ports — assigned, then given over them — and the
// node's settings layers for the module, read once for both.
func portsGivenOn(ctx context.Context, inv *inventory.Inventory, node string,
m catalogue.Manifest) (map[int]int, []catalogue.Layer, error) {
ports, err := portsOn(ctx, inv, node, m.Module)
if err != nil {
return nil, nil, err
}
layers, err := inv.SettingsFor(ctx, node, m.Module)
if err != nil {
return nil, nil, err
}
if given, err := catalogue.GivenPorts(m, layers); err == nil {
for wanted, at := range given {
ports[wanted] = at
}
}
return ports, layers, nil
}
// seatsOn is where a node put the holder of each mesh-scoped seat, by seat and by the port the
// holder's software uses — what ${seat:…} answers with (novox/hq 04-ISSUES/102).
//
// Read for every module in the catalogue that claims a seat, in this node's set or not: the store
// and the broker are given their ports at genesis, as settings on a module that may be registered
// and not yet assigned (04-ISSUES/085), and the control plane must follow that setting from the
// first declaration it composes for itself. A holder in this node's set wins over one that is not.
func seatsOn(ctx context.Context, inv *inventory.Inventory, shelf map[string]catalogue.Manifest,
node string, inSet []catalogue.Manifest, adopted bool, taken map[string]bool) (map[string]map[int]int, error) {
assigned := map[string]bool{}
for _, m := range inSet {
assigned[m.Module] = true
}
names := make([]string, 0, len(shelf))
for name := range shelf {
names = append(names, name)
}
sort.Strings(names)
seats := map[string]map[int]int{}
for _, name := range names {
m := shelf[name]
var claims []string
for _, c := range m.Claims {
if c.At() == catalogue.ScopeMesh {
claims = append(claims, c.Name)
}
}
if len(claims) == 0 {
continue
}
// **Only a port the node was given or the mesh assigned — never the manifest's own
// number.** The sealed value the answer sits beside carries the port genesis wrote, which
// on a given-port node is the predecessor's; a manifest's long-form mapping is the
// catalogue's default, and answering with it would override the right number with one
// the mesh never checked (the contract in seat_into.go). And on an adopted node a holder
// assigned but not yet taken is the found container, on the ports it was found with, not
// the declaration's — so its mesh-assigned ports do not count there either; a given port
// does, because a given port is the found one by construction (ADR 0100).
ports, _, err := portsGivenOn(ctx, inv, node, m)
if err != nil {
return nil, err
}
if adopted && !taken[name] {
layers, err := inv.SettingsFor(ctx, node, m.Module)
if err != nil {
return nil, err
}
ports = map[int]int{}
if given, err := catalogue.GivenPorts(m, layers); err == nil {
ports = given
}
}
if len(ports) == 0 {
continue
}
for _, seat := range claims {
if seats[seat] == nil {
seats[seat] = map[int]int{}
}
for wanted, at := range ports {
if _, said := seats[seat][wanted]; said && !assigned[name] {
continue
}
seats[seat][wanted] = at
}
}
}
return seats, nil
}
// portsOn is one module's assignments on one machine, by the port the software uses.
func portsOn(
ctx context.Context, inv *inventory.Inventory, node, module string,
+37
View File
@@ -0,0 +1,37 @@
package main
import (
"strings"
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
)
// `plan` tells a person what a module would open and why, from the same `why` every listens
// entry already carries for the firewall (novox/hq ADR 0007) — so deciding whether to assign a
// module does not need reading its manifest first.
func TestListensLinesShowWhatAModuleWouldOpenAndWhy(t *testing.T) {
m := catalogue.Manifest{Module: "minio", Listens: []catalogue.Listening{
{Port: 9000, From: catalogue.FromMesh, Why: "the S3 endpoint"},
{Port: 9001, From: catalogue.FromMesh},
}}
got := listensLines(m)
if len(got) != 2 {
t.Fatalf("two listens entries, got %d: %v", len(got), got)
}
if !strings.Contains(got[0], "9000/tcp") || !strings.Contains(got[0], "the S3 endpoint") {
t.Errorf("the port and its why did not both appear: %q", got[0])
}
if strings.Contains(got[1], "—") {
t.Errorf("a listens entry with no why should not print a dash: %q", got[1])
}
if !strings.Contains(got[1], "9001/tcp") {
t.Errorf("the port still appears without a why: %q", got[1])
}
}
func TestListensLinesAreEmptyForAModuleWithNothingToListenOn(t *testing.T) {
if got := listensLines(catalogue.Manifest{Module: "board"}); len(got) != 0 {
t.Errorf("a module with no listens should print nothing, got %v", got)
}
}
+89 -47
View File
@@ -4,7 +4,6 @@ import (
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
"flag"
"fmt"
@@ -283,10 +282,16 @@ func pushCommand(ctx context.Context, args []string) error {
asked = append(asked, n.Name)
}
sending, refusals := composeEach(asked, func(node string) ([]map[string]any, error) {
plan, settings, err := planFor(ctx, open, node)
// Held from composing to sending, so a converge on one of them cannot send between the two
// and be overtaken by what was composed before it (novox/hq ADR 0100).
held, release, err := holdNodes(ctx, open, asked)
if err != nil {
return err
}
sending, refusals := composeEach(asked, func(node string) (sendable, error) {
plan, settings, err := planFor(held, open, node)
if err != nil {
return nil, err
return sendable{}, err
}
// A module assigned here that this machine cannot host is said and left out, not fatal: the
// healthy modules beside it are still resolved and sent. Reported so it is not silently
@@ -295,12 +300,13 @@ func pushCommand(ctx context.Context, args []string) error {
// The private network is in here with everything else. It used to be composed separately
// and prepended, which meant every machine with an address was on it and no machine could
// be kept off. It is a module now, so it arrives the way a module does.
return declarationWith(ctx, open, node, plan, settings, gens, Allocating)
return declarationWith(held, open, node, plan, settings, gens, Allocating)
})
sentDigest := map[string]string{}
defer release()
for _, s := range sending {
body, err := json.Marshal(map[string]any{"declaration": 1, "resources": s.resources})
body, err := s.declared.Body()
if err != nil {
return err
}
@@ -318,8 +324,9 @@ func pushCommand(ctx context.Context, args []string) error {
return err
}
sentDigest[s.node] = digest
fmt.Printf("sent %s %d resource(s)\n", s.node, len(s.resources))
fmt.Printf("sent %s %d resource(s)\n", s.node, len(s.declared.Resources))
}
release()
fmt.Printf("\n%d node(s) told\n", len(sending))
// **A named push leaves the mesh consistent, not just the machine it named** (novox/hq
@@ -374,31 +381,35 @@ func pushCommand(ctx context.Context, args []string) error {
// (novox/hq ADR 0066). The earlier cut routed these through sendTo, which is
// all-or-nothing — so one swept machine's compose error failed the operator's named
// push and skipped its --wait, the very intolerance the main path exists to avoid.
sending, refused := composeEach(also, func(node string) ([]map[string]any, error) {
plan, settings, err := planFor(ctx, open, node)
if err != nil {
return nil, err
}
reportUnhostable(node, plan)
return declarationWith(ctx, open, node, plan, settings, gens, Allocating)
})
// Held for this round only, and after the last round's were given back, so two pushes
// cascading into each other's machines never each wait on the other.
refused, err := sendRound(ctx, open, also,
func(held context.Context, node string) (sendable, error) {
plan, settings, err := planFor(held, open, node)
if err != nil {
return sendable{}, err
}
reportUnhostable(node, plan)
return declarationWith(held, open, node, plan, settings, gens, Allocating)
},
func(s readyNode, body []byte) error {
if err := link.Declare(ctx, server.Channel(), ident, s.node, body,
15*time.Second); err != nil {
return err
}
record, err := inv.NodeByName(ctx, s.node)
if err != nil {
return err
}
if err := inv.RecordSent(ctx, record.ID, digestOf(body)); err != nil {
return err
}
fmt.Printf("sent %s %d resource(s)\n", s.node, len(s.declared.Resources))
return nil
})
refusals = append(refusals, refused...)
for _, s := range sending {
body, err := json.Marshal(map[string]any{"declaration": 1, "resources": s.resources})
if err != nil {
return err
}
if err := link.Declare(ctx, server.Channel(), ident, s.node, body, 15*time.Second); err != nil {
return err
}
record, err := inv.NodeByName(ctx, s.node)
if err != nil {
return err
}
if err := inv.RecordSent(ctx, record.ID, digestOf(body)); err != nil {
return err
}
fmt.Printf("sent %s %d resource(s)\n", s.node, len(s.resources))
if err != nil {
return err
}
// Every candidate this round is marked handled — the sent ones so they are not
// re-listed, and the refused ones so a machine that cannot be composed does not make
@@ -458,8 +469,8 @@ func waitForApplied(ctx context.Context, inv *inventory.Inventory, node, digest
// readyNode is one machine and the declaration it would be sent.
type readyNode struct {
node string
resources []map[string]any
node string
declared sendable
}
// composeEach works out what each named machine should be, and never lets one machine's answer
@@ -480,25 +491,52 @@ type readyNode struct {
// The all-or-nothing rule is kept where it means something — sendTo, which rotates a credential
// across two machines that must agree — and dropped here, where it never did.
func composeEach(names []string,
compose func(node string) ([]map[string]any, error)) ([]readyNode, []string) {
compose func(node string) (sendable, error)) ([]readyNode, []string) {
var sending []readyNode
var refusals []string
for _, name := range names {
resources, err := compose(name)
declared, err := compose(name)
if err != nil {
refusals = append(refusals, fmt.Sprintf("%s:\n%v", name, err))
continue
}
if len(resources) == 0 {
if len(declared.Resources) == 0 {
fmt.Printf("%s is assigned nothing — skipped\n", name)
continue
}
sending = append(sending, readyNode{name, resources})
sending = append(sending, readyNode{name, declared})
}
return sending, refusals
}
// sendRound holds the named nodes, composes each and sends each that composed, and gives the hold
// back on every way out — a body that cannot be marshalled and a send that fails included
// (novox/hq ADR 0100). A node that cannot be composed is a refusal, not an error: the others are
// still sent.
func sendRound(ctx context.Context, open *stores, names []string,
compose func(held context.Context, node string) (sendable, error),
send func(s readyNode, body []byte) error) ([]string, error) {
held, release, err := holdNodes(ctx, open, names)
if err != nil {
return nil, err
}
defer release()
sending, refused := composeEach(names, func(node string) (sendable, error) {
return compose(held, node)
})
for _, s := range sending {
body, err := s.declared.Body()
if err != nil {
return refused, err
}
if err := send(s, body); err != nil {
return refused, err
}
}
return refused, nil
}
// couldNotBeResolved is what a push ends with when some machines could not be worked out.
//
// **After the rest have been sent, never instead of sending them.** It is still an error, because
@@ -533,11 +571,15 @@ func sendTo(ctx context.Context, open *stores, names []string) error {
return err
}
type ready struct {
node string
resources []map[string]any
// Held from composing to sending (novox/hq ADR 0100); a caller that holds them already —
// converge, which flips the node and then sends it — is not made to wait on itself.
ctx, release, err := holdNodes(ctx, open, names)
if err != nil {
return err
}
var sending []ready
defer release()
var sending []readyNode
var refusals []string
for _, name := range names {
plan, settings, err := planFor(ctx, open, name)
@@ -546,12 +588,12 @@ func sendTo(ctx context.Context, open *stores, names []string) error {
continue
}
reportUnhostable(name, plan)
resources, err := declarationWith(ctx, open, name, plan, settings, gens, Allocating)
declared, err := declarationWith(ctx, open, name, plan, settings, gens, Allocating)
if err != nil {
refusals = append(refusals, fmt.Sprintf("%s:\n%v", name, err))
continue
}
sending = append(sending, ready{name, resources})
sending = append(sending, readyNode{name, declared})
}
if len(refusals) > 0 {
return fmt.Errorf("nothing was sent. %d machine(s) could not be resolved:\n\n%s",
@@ -565,7 +607,7 @@ func sendTo(ctx context.Context, open *stores, names []string) error {
defer server.Close()
for _, s := range sending {
body, err := json.Marshal(map[string]any{"declaration": 1, "resources": s.resources})
body, err := s.declared.Body()
if err != nil {
return err
}
@@ -579,7 +621,7 @@ func sendTo(ctx context.Context, open *stores, names []string) error {
if err := inv.RecordSent(ctx, record.ID, digestOf(body)); err != nil {
return err
}
fmt.Printf(" sent %s %d resource(s)\n", s.node, len(s.resources))
fmt.Printf(" sent %s %d resource(s)\n", s.node, len(s.declared.Resources))
}
return nil
}
@@ -610,11 +652,11 @@ func wouldSend(ctx context.Context, open *stores,
if err != nil {
continue
}
resources, err := declarationWith(ctx, open, n.Name, plan, settings, gens, Reading)
declared, err := declarationWith(ctx, open, n.Name, plan, settings, gens, Reading)
if err != nil {
continue
}
body, err := json.Marshal(map[string]any{"declaration": 1, "resources": resources})
body, err := declared.Body()
if err != nil {
return nil, err
}
+4 -4
View File
@@ -18,11 +18,11 @@ import (
func TestOneUnresolvableNodeStillLetsTheRestBeSent(t *testing.T) {
sending, refusals := composeEach(
[]string{"anchor", "home-server", "laptop"},
func(node string) ([]map[string]any, error) {
func(node string) (sendable, error) {
if node == "anchor" {
return nil, errors.New(`nothing provides "acme-ca", wanted by route-proxy`)
return sendable{}, errors.New(`nothing provides "acme-ca", wanted by route-proxy`)
}
return []map[string]any{{"id": node + ".thing"}}, nil
return sendable{Resources: []map[string]any{{"id": node + ".thing"}}}, nil
})
var told []string
@@ -42,7 +42,7 @@ func TestOneUnresolvableNodeStillLetsTheRestBeSent(t *testing.T) {
// And a machine assigned nothing is neither sent nor a refusal — it is nothing to say.
func TestAMachineAssignedNothingIsNotARefusal(t *testing.T) {
sending, refusals := composeEach([]string{"spare"},
func(string) ([]map[string]any, error) { return nil, nil })
func(string) (sendable, error) { return sendable{}, nil })
if len(sending) != 0 || len(refusals) != 0 {
t.Errorf("a machine assigned nothing was treated as something: %v / %v", sending, refusals)
}
+3 -1
View File
@@ -54,6 +54,8 @@ type meshStatus struct {
// Machines is how many the mesh knows about, so a reader can tell "none wrong" from
// "none at all".
Machines int `json:"machines"`
// Adopted is every node still adopted (novox/hq ADR 0100); absent when none is.
Adopted []string `json:"adopted,omitempty"`
}
type machineUnresolved struct {
@@ -133,7 +135,7 @@ func statusAsJSON(asked answers) ([]byte, error) {
out := meshStatus{Machines: len(nodes), Wrong: []machineDoing{},
Quiet: []machineQuiet{}, Behind: []moduleBehind{}, Waiting: []machineWaiting{},
Reported: []machineReported{}, Unresolved: []machineUnresolved{},
Network: asked.network}
Network: asked.network, Adopted: adoptedNodes(nodes)}
for name := range asked.refused {
out.Unresolved = append(out.Unresolved, machineUnresolved{
Node: name, Problem: asked.refused[name]})
+150
View File
@@ -0,0 +1,150 @@
package main
import (
"context"
"encoding/json"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
)
// What a build is recorded as, and what it is announced and handed on as.
//
// **Recorded by what it is; routed where it is used** (novox/hq 04-ISSUES/102). The builder
// announces each artifact by the reference it pushed — `<registry>:<port>/<module>/<artifact>@sha256:…`
// — and the manifest with those references in it. The mesh records the digest and the path
// (catalogue.Recorded) and composes the store's address back in wherever a machine or a builder
// needs a reference it can fetch: a declaration, a replay to the catalogue, the bases a build is
// given. Nothing recorded carries a port, so moving the store is a settings change and not a
// rebuild of everything the mesh has ever built.
// recordedManifest is a build's manifest with the store's address taken off every reference the
// build itself made. Other references — an image a module runs from a public registry — are what
// they were, which is why this rewrites only what `made` names rather than everything that looks
// like an address.
func recordedManifest(raw json.RawMessage, made []inventory.Artifact) json.RawMessage {
announced := map[string]bool{}
for _, a := range made {
announced[a.Reference] = true
}
return withReferences(raw, func(ref string) (string, bool) {
if !announced[ref] {
return ref, false
}
return catalogue.Recorded(ref), true
})
}
// routedManifest is a recorded manifest with the store's address, as this network reaches it now,
// composed into every reference the build made — recorded either way, before or after references
// were kept without their address.
func routedManifest(raw json.RawMessage, made []inventory.Artifact, address string) json.RawMessage {
recorded := map[string]bool{}
for _, a := range made {
recorded[catalogue.Recorded(a.Reference)] = true
}
return withReferences(raw, func(ref string) (string, bool) {
if !recorded[catalogue.Recorded(ref)] {
return ref, false
}
return catalogue.Rerouted(ref, address), true
})
}
// withReferences applies `rewrite` to each resource's `image` and `source`, and hands the manifest
// back untouched — byte for byte — when nothing changed or it could not be read: what a manifest
// is, is the parser's to say, and it says so with a better sentence than anything here would.
func withReferences(raw json.RawMessage, rewrite func(string) (string, bool)) json.RawMessage {
if len(raw) == 0 {
return raw
}
var manifest map[string]any
if err := json.Unmarshal(raw, &manifest); err != nil {
return raw
}
resources, _ := manifest["resources"].([]any)
changed := false
for _, r := range resources {
resource, ok := r.(map[string]any)
if !ok {
continue
}
for _, key := range []string{"image", "source"} {
if written, ok := resource[key].(string); ok {
if rewritten, did := rewrite(written); did && rewritten != written {
resource[key] = rewritten
changed = true
}
}
}
}
if !changed {
return raw
}
out, err := json.Marshal(manifest)
if err != nil {
return raw
}
return out
}
// routedArtifacts is a build's artifacts as something can fetch them now.
func routedArtifacts(made []inventory.Artifact, address string) []inventory.Artifact {
if address == "" {
return made
}
out := make([]inventory.Artifact, 0, len(made))
for _, a := range made {
out = append(out, inventory.Artifact{Name: a.Name, Kind: a.Kind,
Reference: catalogue.Rerouted(a.Reference, address)})
}
return out
}
// whereTheStoreIs is the artifact store's address as something on `forNode` reaches it, or "" —
// read for a caller that has the inventory open and nothing else in hand. With no node named, the
// store's own node: loopback when nothing is on the network yet.
func whereTheStoreIs(ctx context.Context, inv *inventory.Inventory, forNode string) (string, error) {
shelf, err := inv.Catalogue(ctx)
if err != nil {
return "", err
}
if forNode == "" {
if holder, _, found, err := artifactStoreHolder(ctx, inv); err != nil {
return "", err
} else if found {
forNode = holder
}
}
return artifactStoreAddress(ctx, inv, shelf, forNode)
}
// whereABuilderReachesTheStore is the store's address for the machine that builds: the network's
// when there is one, else loopback on the store's own node — when that node also holds a module
// requiring the store, which is what a builder is (genesis: one node holds both).
func whereABuilderReachesTheStore(ctx context.Context, inv *inventory.Inventory) (string, error) {
shelf, err := inv.Catalogue(ctx)
if err != nil {
return "", err
}
holder, _, found, err := artifactStoreHolder(ctx, inv)
if err != nil || !found {
return "", err
}
assigned, err := inv.Assigned(ctx, holder)
if err != nil {
return "", err
}
besideIt := false
for _, a := range assigned {
for _, r := range shelf[a].Requires {
if r == catalogue.ArtifactStoreProvision {
besideIt = true
}
}
}
if !besideIt {
holder = ""
}
return artifactStoreAddress(ctx, inv, shelf, holder)
}
+10 -2
View File
@@ -83,11 +83,11 @@ func rotateCommand(ctx context.Context, args []string) error {
for _, h := range holders {
// The module, because a machine may hold several credentials for one provision and
// rotating "anchor's database password" now means rotating three of them.
fmt.Printf(" %s on %s, from %s\n", h.ConsumerModule, h.Consumer, h.Provider)
fmt.Printf(" %s on %s, from %s%s\n", h.ConsumerModule, h.Consumer, h.Provider, asLocal(h.Local))
}
for _, h := range holders {
if err := inv.RotateSecret(ctx, h.Provision, h.Consumer, h.ConsumerModule, h.Provider); err != nil {
if err := inv.RotateSecret(ctx, h.Provision, h.Consumer, h.ConsumerModule, h.Provider, h.Local); err != nil {
// Partly rotated, and said so plainly. What is gone is remade on the next push, so
// the remedy is to run this again rather than to repair anything — but a machine
// whose secret was discarded and not resent is holding a credential the provider is
@@ -115,3 +115,11 @@ func rotateCommand(ctx context.Context, args []string) error {
"changed cannot authenticate — `status` says who is still behind\n", len(machines))
return nil
}
// asLocal names the credential inside the consumer where it holds several (ADR 0094).
func asLocal(local string) string {
if local == "" {
return ""
}
return " (as " + local + ")"
}
+8 -4
View File
@@ -52,6 +52,9 @@ func secretCommand(ctx context.Context, args []string) error {
provider := set.String("provider", "",
"the node providing <name>: the value becomes the PAIR credential between <module> on <node> "+
"and that provider, sealed to both — the vault's operator-delivered secret (ADR 0092)")
local := set.String("local", "",
"with --provider: the name the credential goes by inside <module>, where its manifest keeps "+
"several for <name> (ADR 0094)")
if err := set.Parse(flags); err != nil {
return err
}
@@ -79,10 +82,10 @@ func secretCommand(ctx context.Context, args []string) error {
// Into the pair, not into the module's own secrets: what the provider is asked to create
// and what the consumer reads are the same value, and neither end can be told a different
// one later without the other (novox/hq 04-ISSUES/070).
if err := open.inventory.AcceptSecretForPair(ctx, name, node, module, *provider, value); err != nil {
if err := open.inventory.AcceptSecretForPair(ctx, name, node, module, *provider, *local, value); err != nil {
return err
}
fmt.Printf("%s on %s now holds %q from %s, sealed to both machines.\n", module, node, name, *provider)
fmt.Printf("%s on %s now holds %q from %s%s, sealed to both machines.\n", module, node, name, *provider, asLocal(*local))
fmt.Printf(" the mesh cannot read it back, will not replace it with one of its own, and will not rotate it\n")
fmt.Printf(" run `push %s` and `push %s` to send it\n", *provider, node)
return nil
@@ -98,7 +101,7 @@ func secretCommand(ctx context.Context, args []string) error {
return nil
}
const secretUsage = "secret accept <node> <module> <name> [--from <file>] [--provider <node>]\n" +
const secretUsage = "secret accept <node> <module> <name> [--from <file>] [--provider <node> [--local <name>]]\n" +
"secret recover <node> <module> <name> --key <operator-key> [--out <file>] [--from-export <file>] [--provider <node>]\n" +
"secret export [--out <file>]"
@@ -120,6 +123,7 @@ func secretRecover(ctx context.Context, args []string) error {
out := set.String("out", "", "where to write the value (0600); - for standard output. Default <node>.<module>.<name>.secret")
fromExport := set.String("from-export", "", "read the sealed copy from this `secret export` file instead of the store")
provider := set.String("provider", "", "for a pair credential held from more than one provider: which one")
local := set.String("local", "", "for a pair credential the module keeps under a local name (ADR 0094): which one")
if err := set.Parse(flags); err != nil {
return err
}
@@ -144,7 +148,7 @@ func secretRecover(ctx context.Context, args []string) error {
return err
}
defer open.Close()
kept, err = open.inventory.KeptSecret(ctx, node, module, name, *provider)
kept, err = open.inventory.KeptSecret(ctx, node, module, name, *provider, *local)
if err != nil {
return err
}
+92
View File
@@ -0,0 +1,92 @@
package main
import (
"context"
"encoding/json"
"sort"
"strings"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
)
// sendable is a declaration as a machine is sent it: its resources and, for an adopted node, its
// mode and which modules were taken on it (novox/hq ADR 0100).
//
// **Body is the only place the envelope is marshalled.** It was written by hand at every send site,
// in the digest the mesh compares, and in `plan --json`; a key added at one and not another would
// make a machine look out of date for ever, or send something `plan` never showed.
type sendable struct {
Resources []map[string]any
// Adoption is nil for a converged node, and then the body is byte for byte what it was before
// adoption existed: an older host parses the envelope strictly and would refuse the key.
Adoption *adoptionEnvelope
}
// adoptionEnvelope is what an adopted node is told about its mode. Taken is every module taken on
// it that it runs; Untaken is, for every module it runs that is not taken, the ids of every one of
// that module's resources — what the host keeps as found until the module is taken (ADR 0103). Ids
// rather than a rule to split them by, because a module's name may contain a dot.
type adoptionEnvelope struct {
Taken []string `json:"taken"`
Untaken map[string][]string `json:"untaken,omitempty"`
}
// Body is the declaration's bytes, as sent and as digested.
func (s sendable) Body() ([]byte, error) {
envelope := map[string]any{"declaration": 1, "resources": s.Resources}
if s.Adoption != nil {
envelope["adoption"] = s.Adoption
}
return json.Marshal(envelope)
}
// adoptionOf is the envelope for a node, nil when it is converged.
func adoptionOf(ctx context.Context, inv *inventory.Inventory, record inventory.Node,
plan catalogue.Resolution, composed catalogue.Composed) (*adoptionEnvelope, error) {
if !record.Adopted {
return nil, nil
}
taken, err := inv.Taken(ctx, record.Name)
if err != nil {
return nil, err
}
return adoptionFor(plan, taken, composed), nil
}
// adoptionFor is the envelope computed from what was taken and who owns each resource.
//
// Every module the node runs that is not taken is untaken — including one pulled in by another
// rather than assigned: what is found is kept until its module is taken, whoever put it there.
func adoptionFor(plan catalogue.Resolution, taken []string,
composed catalogue.Composed) *adoptionEnvelope {
isTaken := map[string]bool{}
for _, m := range taken {
isTaken[m] = true
}
out := &adoptionEnvelope{Taken: []string{}}
runs := map[string]bool{}
for _, m := range plan.Modules {
runs[m.Module] = true
if isTaken[m.Module] {
out.Taken = append(out.Taken, m.Module)
}
}
sort.Strings(out.Taken)
for _, r := range composed.Resources {
id, _ := r["id"].(string)
module, owned := composed.Owner[id]
// Every kind, not only files and containers (novox/hq ADR 0103): a directory, a service, a
// container mounting what was found and an action run in a held container all reach what
// the machine already has. What the mesh declares of its own is never held.
if !owned || !runs[module] || isTaken[module] ||
strings.HasPrefix(id, catalogue.AdoptionPrefix) {
continue
}
if out.Untaken == nil {
out.Untaken = map[string][]string{}
}
out.Untaken[module] = append(out.Untaken[module], id)
}
return out
}
+357
View File
@@ -0,0 +1,357 @@
package main
import (
"bytes"
"encoding/json"
"io"
"os"
"reflect"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/overlay"
)
// novox/hq ADR 0100: every declaration an adopted node is sent says it is adopted and which modules
// were taken on it; a converged node's declaration is byte for byte what it was.
// helloWeb is a module the predecessor also runs: a page and a server under names it uses.
func helloWeb() catalogue.Manifest {
return catalogue.Manifest{Module: "hello-web", Version: "1",
Resources: []map[string]any{
{"id": "page", "type": "file", "path": "/var/lib/hello-web/index.html", "content": "hello"},
{"id": "server", "type": "container", "name": "hello-web",
"image": "registry.example/hello@sha256:" + strings.Repeat("a", 64)},
{"id": "served", "type": "directory", "path": "/var/lib/hello-web"},
}}
}
// composed is what node would be sent now, as push composes it.
func composed(t *testing.T, open *stores, node string) sendable {
t.Helper()
plan, settings, err := planFor(t.Context(), open, node)
if err != nil {
t.Fatal(err)
}
declared, err := declarationFor(t.Context(), open, node, plan, settings)
if err != nil {
t.Fatal(err)
}
return declared
}
// convergedBefore is the envelope a converged node was sent before adoption existed, captured by
// running this same composition at the commit this branch left main (0a39b7d). The mesh here is
// aMesh's laptop with the private network taken off it, so nothing in the declaration is random:
// what changes this string is a change to what a converged machine is sent, which is the thing an
// older host would refuse.
const convergedBefore = `{"declaration":1,"resources":[{"content":"hello","id":"hello-web.page","path":"/var/lib/hello-web/index.html","type":"file"},{"hosts":["anchor.internal:10.77.0.1"],"id":"hello-web.server","image":"registry.example/hello@sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa","name":"hello-web","type":"container"},{"id":"hello-web.served","path":"/var/lib/hello-web","type":"directory"}]}`
func TestAConvergedDeclarationIsByteForByteWhatItWas(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
register(t, open, helloWeb())
if _, err := unassign(ctx, open, "laptop", overlay.Name); err != nil {
t.Fatal(err)
}
if _, err := assign(ctx, open, "laptop", "hello-web"); err != nil {
t.Fatal(err)
}
declared := composed(t, open, "laptop")
if declared.Adoption != nil {
t.Fatal("a converged node was given an adoption envelope")
}
body, err := declared.Body()
if err != nil {
t.Fatal(err)
}
if string(body) != convergedBefore {
t.Fatalf("a converged declaration changed; an older host parses this strictly:\n%s\n%s",
body, convergedBefore)
}
if bytes.Contains(body, []byte(`"adoption"`)) {
t.Fatal("a converged declaration names adoption; an older host would refuse it")
}
}
func TestAnAdoptedDeclarationCarriesItsModeAndWhatWasTaken(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
register(t, open, helloWeb())
if err := open.inventory.SetAdopted(ctx, "anchor", true); err != nil {
t.Fatal(err)
}
if _, err := assign(ctx, open, "anchor", "hello-web"); err != nil {
t.Fatal(err)
}
declared := composed(t, open, "anchor")
if declared.Adoption == nil {
t.Fatal("an adopted node's declaration does not say it is adopted")
}
untaken := declared.Adoption.Untaken["hello-web"]
// Every kind — its directory too (novox/hq ADR 0103).
if !reflect.DeepEqual(untaken, []string{"hello-web.page", "hello-web.server",
"hello-web.served"}) {
t.Fatalf("hello-web's resources are not all named untaken: %v", declared.Adoption)
}
if len(declared.Adoption.Taken) != 0 {
t.Fatalf("nothing was taken, and the declaration says %v", declared.Adoption.Taken)
}
body, err := declared.Body()
if err != nil {
t.Fatal(err)
}
var envelope map[string]any
if err := json.Unmarshal(body, &envelope); err != nil {
t.Fatal(err)
}
adoption, _ := envelope["adoption"].(map[string]any)
if _, ok := adoption["taken"].([]any); !ok {
t.Fatalf("taken is not a list on the wire, even empty: %s", body)
}
// The digest the mesh compares is the digest of what is sent: status and push agree.
would, err := wouldSend(ctx, open, mustNodes(t, open))
if err != nil {
t.Fatal(err)
}
if would["anchor"] != digestOf(body) {
t.Fatal("the digest the mesh compares is not of the declaration push sends")
}
// plan --json prints that same envelope.
printed := stdoutOf(t, func() error { return planCommand(ctx, []string{"anchor", "--json"}) })
var compact bytes.Buffer
if err := json.Compact(&compact, []byte(printed)); err != nil {
t.Fatalf("plan --json is not JSON: %v\n%s", err, printed)
}
if digestOf(compact.Bytes()) != digestOf(body) {
t.Fatalf("plan --json shows something other than what push sends:\n%s", printed)
}
// Taking the module moves its resources out of untaken.
if err := open.inventory.Take(ctx, "anchor", "hello-web"); err != nil {
t.Fatal(err)
}
declared = composed(t, open, "anchor")
if _, still := declared.Adoption.Untaken["hello-web"]; still {
t.Fatalf("a taken module is still untaken: %v", declared.Adoption)
}
if !reflect.DeepEqual(declared.Adoption.Taken, []string{"hello-web"}) {
t.Fatalf("taken is %v", declared.Adoption.Taken)
}
}
func mustNodes(t *testing.T, open *stores) []inventory.Node {
t.Helper()
nodes, err := open.inventory.Nodes(t.Context())
if err != nil {
t.Fatal(err)
}
return nodes
}
// stdoutOf is what run printed.
func stdoutOf(t *testing.T, run func() error) string {
t.Helper()
r, w, err := os.Pipe()
if err != nil {
t.Fatal(err)
}
saved := os.Stdout
os.Stdout = w
done := make(chan string)
go func() {
all, _ := io.ReadAll(r)
done <- string(all)
}()
runErr := run()
os.Stdout = saved
w.Close()
out := <-done
if runErr != nil {
t.Fatalf("%v\n%s", runErr, out)
}
return out
}
// novox/hq ADR 0100: a port a node was given for the store is where its consumers on other
// machines are told to reach it, and a port given for the whole mesh is refused.
func TestConsumersAreToldTheGivenPort(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
register(t, open, catalogue.Manifest{Module: "store", Version: "1",
Provides: []catalogue.Offer{{Name: "database", Scope: catalogue.ScopeMesh}},
Listens: []catalogue.Listening{{Port: 5432, From: catalogue.FromMesh}},
Guards: []int{5432},
Resources: []map[string]any{{"id": "server", "type": "container", "name": "mesh-store",
"ports": []any{"5432:5432"},
"image": "registry.example/pg@sha256:" + strings.Repeat("b", 64)}}})
register(t, open, catalogue.Manifest{Module: "app", Version: "1", Requires: []string{"database"}})
if _, err := assign(ctx, open, "anchor", "store"); err != nil {
t.Fatal(err)
}
if _, err := assign(ctx, open, "laptop", "app"); err != nil {
t.Fatal(err)
}
if err := open.inventory.SetSettings(ctx, "anchor", "store",
map[string]any{catalogue.PortsSetting: map[string]any{"5432": 5433}}); err != nil {
t.Fatal(err)
}
plan, _, err := planFor(ctx, open, "laptop")
if err != nil {
t.Fatal(err)
}
var told any
for _, n := range plan.Needs {
if n.Name == "database" {
told = n.Serves["port"]
}
}
if told != 5433 {
t.Fatalf("the consumer is told the store is on %v", told)
}
for _, r := range composed(t, open, "anchor").Resources {
if r["id"] == "store.server" && !reflect.DeepEqual(r["ports"], []any{"5433:5432"}) {
t.Fatalf("the store publishes %v", r["ports"])
}
}
// A port for the whole mesh is refused where it is set, not stored to refuse every node's
// declaration afterwards.
if err := open.inventory.SetSettings(ctx, "", "store",
map[string]any{catalogue.PortsSetting: map[string]any{"5432": 5434}}); err == nil ||
!strings.Contains(err.Error(), "per node") {
t.Fatalf("a port given for the whole mesh was not refused: %v", err)
}
plan, settings, err := planFor(ctx, open, "anchor")
if err != nil {
t.Fatal(err)
}
if _, err := declarationFor(ctx, open, "anchor", plan, settings); err != nil {
t.Fatalf("the refused mesh-wide layer was stored anyway: %v", err)
}
}
// novox/hq ADR 0103: the guard an adopted node is sent follows what was taken there. A store
// assigned but not taken is not guarded — its port may still be the predecessor's — and taking it
// guards it from the next declaration.
func TestTheGuardIsSentForTakenModulesOnly(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
register(t, open, catalogue.Manifest{Module: "store", Version: "1",
Listens: []catalogue.Listening{{Port: 5432, From: catalogue.FromMesh}},
Guards: []int{5432},
Resources: []map[string]any{{"id": "server", "type": "container", "name": "mesh-store",
"ports": []any{"5432:5432"},
"image": "registry.example/pg@sha256:" + strings.Repeat("b", 64)}}})
if err := open.inventory.SetAdopted(ctx, "anchor", true); err != nil {
t.Fatal(err)
}
if _, err := assign(ctx, open, "anchor", "store"); err != nil {
t.Fatal(err)
}
if hasID(composed(t, open, "anchor").Resources, catalogue.GuardID()) {
t.Fatal("an untaken store is guarded")
}
if err := open.inventory.Take(ctx, "anchor", "store"); err != nil {
t.Fatal(err)
}
for _, r := range composed(t, open, "anchor").Resources {
if r["id"] == catalogue.GuardID() {
if r["content"] != catalogue.AsGuard([]int{5432}) {
t.Fatalf("the taken store's guard is:\n%s", r["content"])
}
return
}
}
t.Fatal("a taken store is not guarded")
}
// novox/hq ADR 0038 and 0100: a module may publish a port the long way — `2222:22`, because the
// machine's own ssh daemon holds 22 — and say it listens on the machine side of that mapping,
// which is the number anything reaching it dials. A node moves that port by naming it, and the
// number has to reach everything derived from it at once: what the runtime is handed, what an
// adopted node is told to open, what its guard refuses, and what a consumer elsewhere dials.
//
// It reached none of them. The setting was refused outright for naming the machine side — so a
// module's port could not be put back where the machine it replaces had it, and, worse, the
// node's every push failed for as long as the setting existed.
func TestTheMachineSideOfAMappingIsMovedEverywhereTheNumberIsUsed(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
register(t, open, catalogue.Manifest{Module: "forge", Version: "1",
Provides: []catalogue.Offer{{Name: "git-over-ssh", Scope: catalogue.ScopeMesh}},
Serves: map[string]map[string]any{"git-over-ssh": {"port": 2222}},
Listens: []catalogue.Listening{{Port: 2222, From: catalogue.FromMesh}},
Resources: []map[string]any{{"id": "server", "type": "container", "name": "forge",
"ports": []any{"2222:22"},
"image": "registry.example/forge@sha256:" + strings.Repeat("c", 64)}}})
register(t, open, catalogue.Manifest{Module: "app", Version: "1",
Requires: []string{"git-over-ssh"}})
if err := open.inventory.SetAdopted(ctx, "anchor", true); err != nil {
t.Fatal(err)
}
if _, err := assign(ctx, open, "anchor", "forge"); err != nil {
t.Fatal(err)
}
if _, err := assign(ctx, open, "laptop", "app"); err != nil {
t.Fatal(err)
}
if err := open.inventory.Take(ctx, "anchor", "forge"); err != nil {
t.Fatal(err)
}
if err := open.inventory.SetSettings(ctx, "anchor", "forge",
map[string]any{catalogue.PortsSetting: map[string]any{"2222": 222}}); err != nil {
t.Fatal(err)
}
resources := composed(t, open, "anchor").Resources
var container, opening map[string]any
for _, r := range resources {
switch r["id"] {
case "forge.server":
container = r
case catalogue.OpeningID("tcp", 222, catalogue.PathForwarded):
opening = r
}
if id, _ := r["id"].(string); strings.HasPrefix(id, "adoption.opening-tcp-2222-") {
t.Errorf("the adopted node is told to open the port the forge was moved off: %s", id)
}
}
if container == nil || !reflect.DeepEqual(container["ports"], []any{"222:22"}) {
t.Fatalf("the forge's container publishes %v", container["ports"])
}
if opening == nil || opening["to"] != 22 || opening["from"] != catalogue.OpeningFromMesh {
t.Fatalf("no opening for the port this node gave the forge: %v", opening)
}
var guard map[string]any
for _, r := range resources {
if r["id"] == catalogue.GuardID() {
guard = r
}
}
if guard == nil || guard["content"] != catalogue.AsGuard([]int{222}) {
t.Fatalf("the guard does not refuse the port the forge is on:\n%v", guard["content"])
}
// And the consumer on the other machine dials the same number.
plan, _, err := planFor(ctx, open, "laptop")
if err != nil {
t.Fatal(err)
}
var told any
for _, n := range plan.Needs {
if n.Name == "git-over-ssh" {
told = n.Serves["port"]
}
}
if told != 222 {
t.Fatalf("the consumer is told the forge answers on %v", told)
}
}
+7
View File
@@ -171,6 +171,13 @@ func statusCommand(ctx context.Context, args []string) error {
fmt.Printf("\n `push --behind` sends them\n\n")
}
if adopted := adoptedNodes(nodes); len(adopted) > 0 {
// Said, because nothing forces the flip: a node left adopted is visible here rather than
// read as converged (novox/hq ADR 0100). Not a fault, so it does not break "all well".
fmt.Printf("%d machine(s) adopted: %s\n", len(adopted), strings.Join(adopted, ", "))
fmt.Printf("\n `converge <node>` previews the flip\n\n")
}
if len(wrong) == 0 && len(quiet) == 0 && len(behind) == 0 && len(asked.waiting) == 0 &&
len(asked.refused) == 0 && asked.network == "" {
// Said plainly. "Nothing to report" and "nothing was checked" must never look the same,
+27 -3
View File
@@ -28,13 +28,15 @@ type following struct{ open *stores }
func (f following) Upgraded(ctx context.Context, u link.Upgraded) error {
inv := f.open.inventory
// The store read first, and an outage there said as one, so the announcement is held and asked
// again (novox/hq issue 083). Only here: a push that fails further down is not asked again.
decision, err := inv.UpgradeOf(ctx, u.Module)
if err != nil {
return err
return notNow(err)
}
on, err := inv.Running(ctx, u.Module)
if err != nil {
return err
return notNow(err)
}
if len(on) == 0 {
fmt.Printf("%s moved to %s; no machine runs it\n", u.Module, shortCommit(u.Commit))
@@ -171,11 +173,21 @@ func sayUpgrade(module string, u inventory.Upgrade) string {
// catalogue misses nothing — but the modules built before it first ran were announced to a queue
// that did not exist, and on a fresh mesh those are always the same three: the shared base, the
// store the catalogue runs on, and the catalogue itself.
//
// **Announced as fetchable, recorded as what it is** (novox/hq 04-ISSUES/102). A build is
// recorded by digest and path; the catalogue hears the builder's own announcements, which name
// the store's address, so a replay composes the address back in — the store's address as the
// network reaches it NOW, which is the whole point of not having recorded the old one. With no
// store on the network yet, the recorded form goes as it is.
func (f following) Announceable(ctx context.Context) ([]link.Announcement, error) {
builds, err := f.open.inventory.Announceable(ctx)
if err != nil {
return nil, err
}
address, err := whereTheStoreIs(ctx, f.open.inventory, "")
if err != nil {
return nil, err
}
out := make([]link.Announcement, 0, len(builds))
for _, b := range builds {
a := link.Announcement{
@@ -184,8 +196,11 @@ func (f following) Announceable(ctx context.Context) ([]link.Announcement, error
}
if len(b.Manifest) > 0 {
a.Manifest = b.Manifest
if address != "" {
a.Manifest = routedManifest(b.Manifest, b.Made, address)
}
}
for _, made := range b.Made {
for _, made := range routedArtifacts(b.Made, address) {
a.Made = append(a.Made, link.MadeArtifact{
Name: made.Name, Kind: made.Kind, Reference: made.Reference,
})
@@ -194,3 +209,12 @@ func (f following) Announceable(ctx context.Context) ([]link.Announcement, error
}
return out, nil
}
// notNow marks a store that could not be read right now, so the announcement is held rather than
// lost; anything else is returned as it was.
func notNow(err error) error {
if inventory.Unreachable(err) {
return fmt.Errorf("%w: %w", link.ErrTryAgain, err)
}
return err
}
+347 -38
View File
@@ -10,10 +10,31 @@
// program. What lives here is that contract, written as something that runs so it can be read
// rather than described.
//
// **A route also carries what a request arriving at it may do** (novox/hq ADR 0108). The grant used
// to say only where to send traffic, so this proxy applied nothing; the four things the ingress it
// replaces actually relies on are now part of the contribution. The set is closed at four, because
// an open middleware surface recreates the thing being replaced and is far harder to narrow later
// than a closed one is to widen.
//
// What it is given, written by the host from an ordinary declaration:
//
// $ROUTES every consumer, the name it asked for, and where the mesh says that machine is
//
// Each contribution's values carry the name and port as before, and optionally:
//
// path the path prefix this rule is scoped to; absent means every path
// priority which rule wins where two match; higher first, and the order is total
// deny refuse the request outright — the shape an incident mitigation needs
// redirect answer with a permanent redirect to this name, keeping the path and query
// auth the *path of a secret* holding `user:hash` lines, never the credential itself
//
// A host may appear more than once, which is what path scoping means: one rule refusing a path
// while another serves everything else on the same name.
//
// **`auth` names a secret and never holds one.** A declaration carrying a credential is refused
// outright rather than served unprotected, and a secret that cannot be read makes the route refuse
// rather than open — a gate that cannot check is not a gate that opens.
//
// It re-reads on change rather than being restarted, for the same reason the provisioner does:
// a route arriving or leaving is an ordinary event and must not drop the connections of every
// other workload.
@@ -23,6 +44,7 @@ import (
"bytes"
"context"
"crypto/sha256"
"crypto/subtle"
"crypto/tls"
"crypto/x509"
"encoding/hex"
@@ -42,6 +64,7 @@ import (
"golang.org/x/crypto/acme"
"golang.org/x/crypto/acme/autocert"
"golang.org/x/crypto/bcrypt"
)
// Where public certificates come from when nothing says otherwise.
@@ -74,7 +97,7 @@ func issuer() string {
// to what it may serve.
func onlyWhatTheMeshSaid(held *table) autocert.HostPolicy {
return func(_ context.Context, host string) error {
if _, known := held.find(host); known {
if held.routed(host) {
return nil
}
return fmt.Errorf("no route for %q in this mesh, so no certificate is asked for", host)
@@ -95,48 +118,145 @@ type contribution struct {
Values map[string]any `json:"values"`
}
// policy is what a rule does with a request that matched it.
//
// **Decided by the mesh, not here** (novox/hq ADR 0108). A route grant used to hand back a name and
// say nothing about what the name admitted, so this proxy admitted everything. The set is closed at
// four — authentication, refusal, path scoping, redirect — because an open middleware surface
// recreates the thing being replaced and is far harder to narrow later than a closed one is to widen.
type policy struct {
// deny refuses the request outright, whatever it is.
deny bool
// redirectTo answers with a permanent redirect instead of proxying. The request's own path and
// query are carried across, which is what canonicalising one public name onto another means.
redirectTo string
// users is what a request must present, read at load time from the secret the declaration
// *named*. A declaration never carries the credential itself.
users map[string]string
// sealed is set when authentication was declared and the secret could not be read. The rule then
// refuses everything and says why.
//
// **Fail closed.** The alternative — serve the route unauthenticated because the gate is
// missing — turns an unreadable file into a silently public admin surface, which is the exact
// outcome ADR 0108 exists to prevent. A gate that cannot check is not a gate that opens.
sealed string
}
// rule is one way a host may be routed. A host may have several, which is what path scoping means.
type rule struct {
path string // "" matches every path
priority int
policy policy
to *httputil.ReverseProxy
target string
}
// table is what the proxy is currently serving, replaced whole whenever the file changes.
//
// Replaced rather than merged: the file is the whole truth about who has a route, so merging
// would keep serving a name whose module was unassigned — which is the stale-route fault
// 08-connectivity lists as open, reintroduced one level down.
//
// Keyed by host to an *ordered* list rather than to one target, because two of the four policies
// need a single host routed more than one way: a refusal on a path the ordinary route also matches,
// and a certificate-challenge path on a host that otherwise serves a workload.
type table struct {
mu sync.RWMutex
to map[string]*httputil.ReverseProxy
targets map[string]string
mu sync.RWMutex
to map[string][]rule
}
func (t *table) set(routes map[string]string) {
made := map[string]*httputil.ReverseProxy{}
for name, target := range routes {
where, err := url.Parse(target)
if err != nil {
log.Printf("route %s points at %q, which is not a URL: %v", name, target, err)
func (t *table) set(routes map[string][]rule) {
made := map[string][]rule{}
for host, rules := range routes {
kept := make([]rule, 0, len(rules))
for _, r := range rules {
// A rule that only refuses or only redirects has nowhere to send anything, and needs
// nowhere: it answers by itself.
if r.policy.deny || r.policy.redirectTo != "" {
kept = append(kept, r)
continue
}
where, err := url.Parse(r.target)
if err != nil {
log.Printf("route %s points at %q, which is not a URL: %v", host, r.target, err)
continue
}
r.to = httputil.NewSingleHostReverseProxy(where)
kept = append(kept, r)
}
if len(kept) == 0 {
continue
}
made[name] = httputil.NewSingleHostReverseProxy(where)
inOrder(kept)
made[host] = kept
}
t.mu.Lock()
t.to, t.targets = made, routes
t.to = made
t.mu.Unlock()
}
func (t *table) find(host string) (*httputil.ReverseProxy, bool) {
// The port is not part of the name. A request to app.example:8080 is for app.example.
// inOrder puts the rules for one host into the order they are matched in, and does so totally.
//
// **Equal priorities must resolve identically every time** (ADR 0108). Sorting only by priority
// leaves rules that share one in whatever order the map produced, so the same declaration would
// serve differently between restarts — a proxy that is not reproducible. Longest path first within a
// priority is also the intuitive reading: the more specific rule wins. The last two keys exist only
// to make the order total.
func inOrder(rules []rule) {
sort.SliceStable(rules, func(i, j int) bool {
a, b := rules[i], rules[j]
if a.priority != b.priority {
return a.priority > b.priority
}
if len(a.path) != len(b.path) {
return len(a.path) > len(b.path)
}
if a.path != b.path {
return a.path < b.path
}
return a.target < b.target
})
}
// find is the rule that answers this request, or nothing if the host is not routed here at all.
func (t *table) find(host, path string) (rule, bool) {
t.mu.RLock()
defer t.mu.RUnlock()
for _, r := range t.to[bareHost(host)] {
if r.path == "" || strings.HasPrefix(path, r.path) {
return r, true
}
}
return rule{}, false
}
// routed says whether this proxy serves the name at all, whatever the path.
//
// Separate from find because certificate issuance is a question about the *name*: a host whose only
// rules are path-scoped is still a name this proxy answers to, and still needs a certificate.
func (t *table) routed(host string) bool {
t.mu.RLock()
defer t.mu.RUnlock()
return len(t.to[bareHost(host)]) > 0
}
// bareHost is the name without the port, lower-cased.
//
// The port is not part of the name: a request to app.example:8080 is for app.example. Lower-cased
// because a Host header is not case-sensitive, and a route that only answers the spelling in the
// manifest answers half the requests made to it.
func bareHost(host string) string {
if h, _, err := net.SplitHostPort(host); err == nil {
host = h
}
t.mu.RLock()
defer t.mu.RUnlock()
p, ok := t.to[strings.ToLower(host)]
return p, ok
return strings.ToLower(host)
}
func (t *table) names() []string {
t.mu.RLock()
defer t.mu.RUnlock()
out := make([]string, 0, len(t.targets))
for name := range t.targets {
out := make([]string, 0, len(t.to))
for name := range t.to {
out = append(out, name)
}
sort.Strings(out)
@@ -285,29 +405,115 @@ func forThisAuthority(cache, directory string, root []byte) string {
// newTable is an empty routing table.
func newTable() *table {
return &table{to: map[string]*httputil.ReverseProxy{}, targets: map[string]string{}}
return &table{to: map[string][]rule{}}
}
// handler is the proxy itself, separated so it can be driven by a test without a listener.
func handler(held *table) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
proxy, known := held.find(r.Host)
matched, known := held.find(r.Host, r.URL.Path)
if !known {
// **Named, not a bare 404.** A route that was withdrawn and a name that never existed
// are different things, and a proxy that says only "not found" makes an operator go
// and read the mesh to tell them apart. What it is serving is the answer to both.
//
// And since a host may now be routed only on some paths, those are a third thing:
// saying "no route for this name" while listing that very name as served is a
// contradiction an operator would have to disbelieve the proxy to get past.
w.Header().Set("Content-Type", "text/plain; charset=utf-8")
w.WriteHeader(http.StatusNotFound)
if held.routed(r.Host) {
fmt.Fprintf(w, "%s is served here, but no route covers %q.\n",
bareHost(r.Host), r.URL.Path)
return
}
fmt.Fprintf(w, "no route for %q in this mesh.\nserving: %s\n",
r.Host, strings.Join(held.names(), ", "))
return
}
proxy.ServeHTTP(w, r)
switch {
case matched.policy.sealed != "":
// Declared a gate, cannot check it. Refused, and says why — an operator reading this
// learns the secret is missing, rather than wondering why a protected name is 503.
w.Header().Set("Content-Type", "text/plain; charset=utf-8")
w.WriteHeader(http.StatusServiceUnavailable)
fmt.Fprintf(w, "this route requires authentication and its credentials cannot be read: %s\n",
matched.policy.sealed)
return
case matched.policy.deny:
http.Error(w, "this path is not served to you", http.StatusForbidden)
return
case matched.policy.redirectTo != "":
http.Redirect(w, r, canonical(matched.policy.redirectTo, r.URL), http.StatusMovedPermanently)
return
case len(matched.policy.users) > 0 && !allowed(matched.policy.users, r):
// The realm is the name asked for, so a browser's prompt says which route it is for.
w.Header().Set("WWW-Authenticate", fmt.Sprintf("Basic realm=%q, charset=\"UTF-8\"", bareHost(r.Host)))
http.Error(w, "unauthorized", http.StatusUnauthorized)
return
}
matched.to.ServeHTTP(w, r)
})
}
// routesFrom reads what the mesh wrote and turns it into name → target.
func routesFrom(path string) (map[string]string, error) {
// canonical is where a redirect sends this request.
//
// The declaration names the destination *name*; the request keeps its own path and query. That is
// what canonicalising one public name onto another means — a link to a page under the old name has
// to arrive at the same page under the new one, or the redirect silently loses every deep link.
func canonical(to string, from *url.URL) string {
where, err := url.Parse(to)
if err != nil {
return to
}
if where.Path == "" || where.Path == "/" {
where.Path = from.Path
}
if where.RawQuery == "" {
where.RawQuery = from.RawQuery
}
return where.String()
}
// allowed says whether the request presented credentials this route accepts.
//
// **Every path costs one bcrypt comparison**, including an unknown user, which is why the miss
// compares against a fixed hash rather than returning early. Returning early would make an unknown
// user measurably faster than a known one with a wrong password, and that difference is a way to
// enumerate the users of a route from outside it.
func allowed(users map[string]string, r *http.Request) bool {
// A hash of nothing anybody knows. Its only job is to cost what a real comparison costs.
const absent = "$2a$10$N9qo8uLOickgx2ZMRZoMyeIjZAgcfl7p92ldGxad68LJZdL17lhWy"
user, password, ok := r.BasicAuth()
if !ok {
return false
}
want, known := users[user]
if !known {
want = absent
}
if err := bcrypt.CompareHashAndPassword([]byte(want), []byte(password)); err != nil {
return false
}
// `known` is checked after the comparison, not instead of it, so the timing is the same either
// way. subtle.ConstantTimeByteEq keeps the branch from being the thing that differs.
return subtle.ConstantTimeByteEq(boolByte(known), 1) == 1
}
func boolByte(b bool) byte {
if b {
return 1
}
return 0
}
// routesFrom reads what the mesh wrote and turns it into host → the rules for that host.
func routesFrom(path string) (map[string][]rule, error) {
raw, err := os.ReadFile(path)
if err != nil {
return nil, err
@@ -317,31 +523,134 @@ func routesFrom(path string) (map[string]string, error) {
return nil, err
}
out := map[string]string{}
out := map[string][]rule{}
for _, c := range said.Given {
name, _ := c.Values["name"].(string)
if name == "" {
log.Printf("%s on %s asked for a route and named nothing; skipped", c.From, c.Node)
continue
}
port, ok := asPort(c.Values["port"])
if !ok {
log.Printf("%s on %s asked for route %q and gave no usable port; skipped",
c.From, c.Node, name)
continue
host := strings.ToLower(name)
made := rule{path: asPath(c.Values["path"])}
if p, ok := asWhole(c.Values["priority"]); ok {
made.priority = p
}
// Where the mesh says that machine is. Empty means it is this one — a workload beside the
// proxy is ordinary, and reaching it over loopback is both correct and the only thing
// that works when there is no private network.
at := c.At
if at == "" {
at = "127.0.0.1"
made.policy.deny, _ = c.Values["deny"].(bool)
made.policy.redirectTo, _ = c.Values["redirect"].(string)
if named, carried := c.Values["auth"].(string); carried && strings.TrimSpace(named) != "" {
// **A declaration names a secret; it never holds one** (ADR 0108). Refused rather than
// tolerated, and the whole rule is dropped rather than served unprotected — the
// rejected option cannot come back by accident, which is the failure this check exists
// to make impossible.
if looksLikeACredential(named) {
log.Printf("%s on %s declared route %q with a credential in the declaration rather "+
"than the name of a secret; the whole route is refused (novox/hq ADR 0108)",
c.From, c.Node, name)
continue
}
users, err := usersFrom(named)
if err != nil {
// Fail closed: the rule is kept so the name stays routed and answers, and it
// answers by refusing. Dropping it instead would make the name 404 and read as a
// withdrawn route rather than an unreadable secret.
made.policy.sealed = err.Error()
}
made.policy.users = users
}
out[strings.ToLower(name)] = fmt.Sprintf("http://%s:%d", at, port)
// Only a rule that actually proxies needs somewhere to send the request.
if !made.policy.deny && made.policy.redirectTo == "" {
port, ok := asPort(c.Values["port"])
if !ok {
log.Printf("%s on %s asked for route %q and gave no usable port; skipped",
c.From, c.Node, name)
continue
}
// Where the mesh says that machine is. Empty means it is this one — a workload beside
// the proxy is ordinary, and reaching it over loopback is both correct and the only
// thing that works when there is no private network.
at := c.At
if at == "" {
at = "127.0.0.1"
}
made.target = fmt.Sprintf("http://%s:%d", at, port)
}
out[host] = append(out[host], made)
}
return out, nil
}
// asWhole is any whole number the mesh wrote, whatever its magnitude.
//
// **Not asPort.** Priority was read with the port reader first, which caps at 65535 — so a rule
// declared at a priority above that silently became priority 0 and stopped shadowing the route it
// exists to shadow. The one real rule this has to reproduce is declared at 100000, so the bug was
// exactly load-bearing. A priority is an ordering, not a port: it has no range.
func asWhole(v any) (int, bool) {
switch n := v.(type) {
case float64:
// JSON makes a float of every number, so a non-integral one was not meant as a priority.
if n != float64(int(n)) {
return 0, false
}
return int(n), true
case int:
return n, true
}
return 0, false
}
// asPath is the path prefix a rule is scoped to, or "" for every path.
func asPath(v any) string {
p, _ := v.(string)
p = strings.TrimSpace(p)
if p == "" {
return ""
}
if !strings.HasPrefix(p, "/") {
p = "/" + p
}
return p
}
// looksLikeACredential is the check that keeps a secret out of a declaration.
//
// It errs towards refusing: a value holding a `:` (the htpasswd separator) or opening with a bcrypt
// identifier is a credential, not a path, and no filesystem path the mesh writes needs either. A
// false refusal is a loud log and a route that does not serve; a false accept is a credential
// committed to a declaration, which is the thing being prevented.
func looksLikeACredential(v string) bool {
v = strings.TrimSpace(v)
return strings.Contains(v, ":") || strings.HasPrefix(v, "$2")
}
// usersFrom reads the credentials the mesh mounted, in the one format every htpasswd already is.
func usersFrom(path string) (map[string]string, error) {
raw, err := os.ReadFile(path)
if err != nil {
return nil, fmt.Errorf("cannot read the secret named for this route: %w", err)
}
users := map[string]string{}
for _, line := range strings.Split(string(raw), "\n") {
line = strings.TrimSpace(line)
if line == "" || strings.HasPrefix(line, "#") {
continue
}
user, hash, ok := strings.Cut(line, ":")
if !ok || user == "" || hash == "" {
continue
}
users[user] = hash
}
if len(users) == 0 {
return nil, fmt.Errorf("the secret named for this route holds no usable credentials")
}
return users, nil
}
// asPort accepts what JSON makes of a number, which is a float even when it was written 8080.
func asPort(v any) (int, bool) {
switch n := v.(type) {
+273
View File
@@ -0,0 +1,273 @@
package main
import (
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"strconv"
"strings"
"testing"
"golang.org/x/crypto/bcrypt"
)
// What a route carries about the requests arriving at it — novox/hq ADR 0108.
//
// Each test here is one of the four capabilities that record closed the set at, plus the negative
// case it promised would be refused. The negative case is the one that rots quietly: nothing fails
// if it stops working, so nothing tells you it has.
// served starts a workload and gives back the host and port the mesh would have recorded for it.
func served(t *testing.T, body string) (string, int) {
t.Helper()
workload := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
_, _ = w.Write([]byte(body))
}))
t.Cleanup(workload.Close)
host, port, _ := strings.Cut(strings.TrimPrefix(workload.URL, "http://"), ":")
n, err := strconv.Atoi(port)
if err != nil {
t.Fatal(err)
}
return host, n
}
// ask makes one request through the proxy for a given name and path, without following redirects.
func ask(t *testing.T, proxy, name, path string, auth [2]string) *http.Response {
t.Helper()
req, err := http.NewRequest(http.MethodGet, proxy+path, nil)
if err != nil {
t.Fatal(err)
}
req.Host = name
if auth[0] != "" {
req.SetBasicAuth(auth[0], auth[1])
}
client := &http.Client{CheckRedirect: func(*http.Request, []*http.Request) error {
return http.ErrUseLastResponse
}}
answer, err := client.Do(req)
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = answer.Body.Close() })
return answer
}
func proxyFor(t *testing.T, routesJSON string) string {
t.Helper()
path := filepath.Join(t.TempDir(), "routes.json")
if err := os.WriteFile(path, []byte(routesJSON), 0o644); err != nil {
t.Fatal(err)
}
routes, err := routesFrom(path)
if err != nil {
t.Fatal(err)
}
held := newTable()
held.set(routes)
server := httptest.NewServer(handler(held))
t.Cleanup(server.Close)
return server.URL
}
// A refusal on a path shadows the ordinary route for that path and leaves every other path alone.
//
// **This is why path scoping is a prerequisite and not a sibling capability.** The rule being
// reproduced matches a path on a host that is already routed to a workload, so a table mapping a
// host to one target cannot express it at all — no amount of authentication or source filtering
// would have helped.
func TestARefusedPathShadowsTheRouteAndLeavesTheRestServed(t *testing.T) {
at, port := served(t, "the workload")
proxy := proxyFor(t, `{"given":[
{"from":"forge","node":"anchor","at":"`+at+`","values":{"name":"forge.example","port":`+strconv.Itoa(port)+`}},
{"from":"forge","node":"anchor","values":{"name":"forge.example","path":"/api/internal","priority":100,"deny":true}}
]}`)
if got := ask(t, proxy, "forge.example", "/api/internal/hook", [2]string{}).StatusCode; got != http.StatusForbidden {
t.Fatalf("the refused path answered %d, so the block that was put in front of it during an "+
"incident is not in front of it any more", got)
}
if got := ask(t, proxy, "forge.example", "/", [2]string{}).StatusCode; got != http.StatusOK {
t.Fatalf("refusing one path took the whole route with it: %d", got)
}
}
// A redirect answers with the redirect, and the request keeps its own path and query.
//
// Losing the path would turn canonicalising one name onto another into "every deep link now lands
// on the front page", which is the kind of breakage that produces no error anywhere.
func TestARedirectKeepsThePathAndQuery(t *testing.T) {
proxy := proxyFor(t, `{"given":[
{"from":"site","node":"anchor","values":{"name":"www.example","redirect":"https://example/"}}
]}`)
answer := ask(t, proxy, "www.example", "/deep/page?ref=1", [2]string{})
if answer.StatusCode != http.StatusMovedPermanently {
t.Fatalf("a declared redirect answered %d", answer.StatusCode)
}
where := answer.Header.Get("Location")
if !strings.Contains(where, "/deep/page") || !strings.Contains(where, "ref=1") {
t.Fatalf("the redirect dropped the path or the query: %q", where)
}
}
// Authentication refuses a request with no credentials, admits one with the right ones, and refuses
// the wrong ones — with the credentials read from the secret the declaration *named*.
func TestAuthenticationAdmitsOnlyWhatTheSecretSays(t *testing.T) {
at, port := served(t, "the console")
hash, err := bcrypt.GenerateFromPassword([]byte("correct horse"), bcrypt.MinCost)
if err != nil {
t.Fatal(err)
}
secret := filepath.Join(t.TempDir(), "console-auth")
if err := os.WriteFile(secret, []byte("# a comment\nadmin:"+string(hash)+"\n"), 0o600); err != nil {
t.Fatal(err)
}
proxy := proxyFor(t, `{"given":[
{"from":"console","node":"anchor","at":"`+at+`","values":{"name":"console.example","port":`+strconv.Itoa(port)+`,"auth":"`+secret+`"}}
]}`)
if got := ask(t, proxy, "console.example", "/", [2]string{}).StatusCode; got != http.StatusUnauthorized {
t.Fatalf("an admin surface with no login of its own answered %d without credentials", got)
}
if got := ask(t, proxy, "console.example", "/", [2]string{"admin", "wrong"}).StatusCode; got != http.StatusUnauthorized {
t.Fatalf("the wrong password answered %d", got)
}
if got := ask(t, proxy, "console.example", "/", [2]string{"admin", "correct horse"}).StatusCode; got != http.StatusOK {
t.Fatalf("the right password answered %d", got)
}
}
// The negative case ADR 0108 promised would be refused: a credential in the declaration.
//
// **Refused whole, not tolerated and not served unprotected.** A hash carried in a declaration was
// the rejected option; nothing in the running system should quietly accept it later, because the
// precedent is far easier to set than to withdraw. If this test is deleted the option returns and
// nothing else notices.
func TestACredentialInTheDeclarationIsRefusedRatherThanServed(t *testing.T) {
inline := []string{
`{"given":[{"from":"c","node":"n","at":"127.0.0.1","values":{"name":"c.example","port":8080,"auth":"admin:$2a$10$abcdefghijklmnopqrstuv"}}]}`,
`{"given":[{"from":"c","node":"n","at":"127.0.0.1","values":{"name":"c.example","port":8080,"auth":"$2a$10$abcdefghijklmnopqrstuv"}}]}`,
}
for _, body := range inline {
path := filepath.Join(t.TempDir(), "routes.json")
if err := os.WriteFile(path, []byte(body), 0o644); err != nil {
t.Fatal(err)
}
routes, err := routesFrom(path)
if err != nil {
t.Fatal(err)
}
if len(routes) != 0 {
t.Fatalf("a declaration carrying a credential was served anyway: %v", routes)
}
}
}
// Authentication declared, secret unreadable: the route refuses. It does not serve unprotected.
//
// **Fail closed.** The alternative turns a missing file into a silently public admin surface, which
// is the outcome the whole record exists to prevent. It answers rather than 404s, so an operator
// sees "cannot read the credentials" instead of concluding the route was withdrawn.
func TestAnUnreadableSecretFailsClosed(t *testing.T) {
at, port := served(t, "the console")
missing := filepath.Join(t.TempDir(), "not-mounted")
proxy := proxyFor(t, `{"given":[
{"from":"console","node":"anchor","at":"`+at+`","values":{"name":"console.example","port":`+strconv.Itoa(port)+`,"auth":"`+missing+`"}}
]}`)
answer := ask(t, proxy, "console.example", "/", [2]string{})
if answer.StatusCode == http.StatusOK {
t.Fatal("a route whose credentials could not be read served the workload unprotected")
}
if answer.StatusCode != http.StatusServiceUnavailable {
t.Fatalf("expected the route to say it cannot check, got %d", answer.StatusCode)
}
}
// Equal priorities resolve the same way every time, so the same declaration serves the same way
// after a restart.
//
// Sorting only by priority leaves rules that share one in whatever order the map produced. The
// proxy would still work, and would work differently between restarts — which is the hardest kind
// of fault to believe when it is reported.
func TestRulesThatShareAPriorityAreStillTotallyOrdered(t *testing.T) {
first := []rule{
{path: "/a", priority: 10, target: "http://x:1"},
{path: "/bb", priority: 10, target: "http://y:2"},
{path: "", priority: 10, target: "http://z:3"},
}
second := []rule{
{path: "", priority: 10, target: "http://z:3"},
{path: "/bb", priority: 10, target: "http://y:2"},
{path: "/a", priority: 10, target: "http://x:1"},
}
inOrder(first)
inOrder(second)
for i := range first {
if first[i].path != second[i].path || first[i].target != second[i].target {
t.Fatalf("two orderings of the same rules disagree at %d: %q vs %q",
i, first[i].path, second[i].path)
}
}
// And the more specific rule is matched first, which is the intuitive reading.
if first[0].path != "/bb" {
t.Fatalf("the longest path is not matched first: %q", first[0].path)
}
}
// Priority decides before path length does, so a rule can be made to win regardless of specificity.
func TestPriorityOutranksPathLength(t *testing.T) {
rules := []rule{
{path: "/very/long/path", priority: 1, target: "http://x:1"},
{path: "", priority: 100, target: "http://y:2"},
}
inOrder(rules)
if rules[0].priority != 100 {
t.Fatalf("a higher priority did not win: %+v", rules[0])
}
}
// A priority above a port number survives, because a priority is an ordering and not a port.
//
// **Found by review, and it was load-bearing.** Priority was first read with the port reader, which
// caps at 65535 — so a rule declared above that silently became priority 0 and stopped shadowing the
// route it exists to shadow. The one real rule this has to reproduce is declared at 100000, so the
// capability would have shipped looking complete and doing nothing.
func TestAPriorityAboveAPortNumberSurvives(t *testing.T) {
at, port := served(t, "the workload")
proxy := proxyFor(t, `{"given":[
{"from":"forge","node":"anchor","at":"`+at+`","values":{"name":"forge.example","port":`+strconv.Itoa(port)+`}},
{"from":"forge","node":"anchor","values":{"name":"forge.example","path":"/api/internal","priority":100000,"deny":true}}
]}`)
if got := ask(t, proxy, "forge.example", "/api/internal/hook", [2]string{}).StatusCode; got != http.StatusForbidden {
t.Fatalf("a rule declared at priority 100000 answered %d instead of refusing", got)
}
}
// A host routed only on some paths says so, rather than claiming the name is not served here.
//
// Saying "no route for this name" while listing that very name as served is a contradiction an
// operator has to disbelieve the proxy to get past — and path scoping makes it reachable, because a
// host can now have rules that none of this request's paths match.
func TestAHostRoutedOnlyOnSomePathsSaysSo(t *testing.T) {
proxy := proxyFor(t, `{"given":[
{"from":"forge","node":"anchor","values":{"name":"forge.example","path":"/api/internal","deny":true}}
]}`)
answer := ask(t, proxy, "forge.example", "/elsewhere", [2]string{})
if answer.StatusCode != http.StatusNotFound {
t.Fatalf("an uncovered path answered %d", answer.StatusCode)
}
body := make([]byte, 256)
n, _ := answer.Body.Read(body)
said := string(body[:n])
if !strings.Contains(said, "is served here") || !strings.Contains(said, "/elsewhere") {
t.Fatalf("the refusal does not distinguish an uncovered path from an unserved name: %q", said)
}
}
+30 -13
View File
@@ -19,6 +19,23 @@ func write(t *testing.T, body string) string {
return path
}
// plain is the table an ordinary set of routes makes: one host, one target, no policy.
func plain(routes map[string]string) map[string][]rule {
out := map[string][]rule{}
for host, target := range routes {
out[host] = []rule{{target: target}}
}
return out
}
// targetOf is where a host's first matching rule sends a request.
func targetOf(routes map[string][]rule, host string) string {
if rules := routes[host]; len(rules) > 0 {
return rules[0].target
}
return ""
}
// A route is a grant: the consumer supplies a target, and where that machine is comes from the
// mesh rather than from a naming convention the proxy has to know.
func TestARouteGoesToWhereTheMeshSaysTheConsumerIs(t *testing.T) {
@@ -30,7 +47,7 @@ func TestARouteGoesToWhereTheMeshSaysTheConsumerIs(t *testing.T) {
}
// Lower-cased, because a Host header is not case-sensitive and a route that only answers the
// spelling in the manifest answers half the requests made to it.
if routes["app.example"] != "http://laptop.internal:8080" {
if targetOf(routes, "app.example") != "http://laptop.internal:8080" {
t.Fatalf("the route does not point at the consumer: %v", routes)
}
}
@@ -44,7 +61,7 @@ func TestAConsumerOnTheProxysOwnMachineIsReachedOverLoopback(t *testing.T) {
if err != nil {
t.Fatal(err)
}
if routes["app.example"] != "http://127.0.0.1:9000" {
if targetOf(routes, "app.example") != "http://127.0.0.1:9000" {
t.Fatalf("a workload on this machine was not reachable: %v", routes)
}
}
@@ -59,7 +76,7 @@ func TestAContributionMissingWhatARouteNeedsIsSkipped(t *testing.T) {
if err != nil {
t.Fatal(err)
}
if len(routes) != 1 || routes["fine.example"] == "" {
if len(routes) != 1 || targetOf(routes, "fine.example") == "" {
t.Fatalf("an unusable contribution was served: %v", routes)
}
}
@@ -75,7 +92,7 @@ func TestTheProxyReachesTheWorkloadAndNamesWhatItServes(t *testing.T) {
host, port, _ := strings.Cut(target, ":")
held := newTable()
held.set(map[string]string{"app.example": "http://" + host + ":" + port})
held.set(plain(map[string]string{"app.example": "http://" + host + ":" + port}))
proxy := httptest.NewServer(handler(held))
defer proxy.Close()
@@ -120,16 +137,16 @@ func TestTheProxyReachesTheWorkloadAndNamesWhatItServes(t *testing.T) {
// nothing fails more visibly than a stale grant, which is exactly why it must not survive.
func TestWithdrawingARouteStopsServingIt(t *testing.T) {
held := newTable()
held.set(map[string]string{
held.set(plain(map[string]string{
"going.example": "http://a.internal:80",
"staying.example": "http://b.internal:80",
})
held.set(map[string]string{"staying.example": "http://b.internal:80"})
}))
held.set(plain(map[string]string{"staying.example": "http://b.internal:80"}))
if _, still := held.find("going.example"); still {
if _, still := held.find("going.example", "/"); still {
t.Fatal("a route whose module was unassigned is still served")
}
if _, kept := held.find("staying.example"); !kept {
if _, kept := held.find("staying.example", "/"); !kept {
t.Fatal("withdrawing one route took another with it")
}
}
@@ -137,8 +154,8 @@ func TestWithdrawingARouteStopsServingIt(t *testing.T) {
// A Host header carries a port and the name does not.
func TestARequestNamingAPortStillFindsItsRoute(t *testing.T) {
held := newTable()
held.set(map[string]string{"app.example": "http://a.internal:8080"})
if _, found := held.find("app.example:8080"); !found {
held.set(plain(map[string]string{"app.example": "http://a.internal:8080"}))
if _, found := held.find("app.example:8080", "/"); !found {
t.Fatal("a request to app.example:8080 did not find the route for app.example")
}
}
@@ -168,7 +185,7 @@ func TestTheIssuerIsStagingUnlessNamed(t *testing.T) {
// rate limit — and the proxy would look healthy throughout.
func TestNoCertificateIsAskedForOnAnUnroutedName(t *testing.T) {
held := newTable()
held.set(map[string]string{"photos.example": "http://127.0.0.1:8080"})
held.set(plain(map[string]string{"photos.example": "http://127.0.0.1:8080"}))
policy := onlyWhatTheMeshSaid(held)
if err := policy(context.Background(), "photos.example"); err != nil {
@@ -184,7 +201,7 @@ func TestNoCertificateIsAskedForOnAnUnroutedName(t *testing.T) {
// A route withdrawn stops being certifiable, without the proxy restarting.
func TestWithdrawingARouteWithdrawsItsCertificate(t *testing.T) {
held := newTable()
held.set(map[string]string{"photos.example": "http://127.0.0.1:8080"})
held.set(plain(map[string]string{"photos.example": "http://127.0.0.1:8080"}))
policy := onlyWhatTheMeshSaid(held)
if err := policy(context.Background(), "photos.example"); err != nil {
t.Fatal(err)
+5 -1
View File
@@ -40,8 +40,12 @@ type Broker struct {
var ErrNotConfigured = errors.New("this control plane has not been told about its broker")
// FromEnvironment reads the two settings, if they are there.
//
// The address's port follows MESH_BROKER_ADDRESS_PORT when the node's settings moved the bus
// (novox/hq 04-ISSUES/102): the address genesis wrote is a public name and the port genesis
// chose, and only the port is the node's to move.
func FromEnvironment() (Broker, error) {
address, err := envfile.Value(AddressVar)
address, err := envfile.Placed(AddressVar)
if err != nil {
return Broker{}, err
}
+29
View File
@@ -178,3 +178,32 @@ func TestBothTogetherGiveABroker(t *testing.T) {
t.Errorf("got %+v", known)
}
}
// The node moved the bus, and the address a token carries follows (novox/hq 04-ISSUES/102).
func TestTheAddressPortFollowsThePortTwin(t *testing.T) {
t.Setenv(AddressVar, "broker.example:5671")
t.Setenv(AddressVar+"_FILE", "")
path, _ := writeCertificate(t)
t.Setenv(CertificateVar, path)
t.Setenv(AddressVar+"_PORT", "5679")
b, err := FromEnvironment()
if err != nil {
t.Fatal(err)
}
if b.Address != "broker.example:5679" {
t.Fatalf("the address is %q; the node put the bus on 5679", b.Address)
}
}
func TestTheManagementPortFollowsThePortTwin(t *testing.T) {
t.Setenv(ManagementVar, "http://guest:guest@127.0.0.1:15672")
t.Setenv(ManagementVar+"_FILE", "")
t.Setenv(ManagementVar+"_PORT", "15673")
m, err := ManagementFromEnvironment()
if err != nil {
t.Fatal(err)
}
if m.base.Host != "127.0.0.1:15673" {
t.Fatalf("the management API is at %q; the node put it on 15673", m.base.Host)
}
}
+4 -1
View File
@@ -41,8 +41,11 @@ type Management struct {
}
// ManagementFromEnvironment reads where the management API is, if it is configured.
//
// On the port MESH_BROKER_MANAGEMENT_PORT names when the node moved it (novox/hq 04-ISSUES/102);
// the URL's own port otherwise.
func ManagementFromEnvironment() (*Management, error) {
raw, err := envfile.Value(ManagementVar)
raw, err := envfile.Placed(ManagementVar)
if err != nil {
return nil, err
}
+209 -6
View File
@@ -14,6 +14,7 @@ import (
"path/filepath"
"regexp"
"sort"
"strconv"
"strings"
"time"
@@ -149,7 +150,20 @@ func Build(ctx context.Context, run Runner, publish Publisher,
// What this module said it stands on, answered with what this mesh actually holds. Done
// before anything is built, so a missing base is refused in front of the person who can
// fix it rather than inside a build that stops on its own first line.
args, err := standingOn(manifest, held)
// An image published elsewhere that the build stands on is copied into the mesh's own
// registry first, like an upstream artifact (ADR 0096), and the recipe is handed the copy.
// Genesis has nowhere to copy to and pulls it into this machine's store instead.
mirror := func(ctx context.Context, from, repository string) (string, error) {
if m, can := publish.(Mirrorer); can {
say("bases", "copying %s into the mesh's registry", from)
return m.MirrorImage(ctx, from, repository)
}
if _, err := run(ctx, tree, "docker", "pull", from); err != nil {
return "", fmt.Errorf("cannot fetch %s: %w", from, err)
}
return from, nil
}
args, err := standingOn(ctx, manifest, held, mirror)
if err != nil {
say("bases", "UNMET: %v", err)
return Result{}, err
@@ -324,14 +338,27 @@ func one(ctx context.Context, run Runner, publish Publisher,
switch a.Kind {
case catalogue.ArtifactUpstream:
// Mirrored, not built. Pulled by the reference the module names and pushed under a name
// of the mesh's own, so what a machine fetches is pinned by a digest this registry
// assigned rather than by a tag somebody else can move.
// Mirrored, not built: copied under a name of the mesh's own, so what a machine fetches is
// pinned by a digest this registry assigned rather than by a tag somebody else can move.
//
// **Between registries, never through this machine's image store** (novox/hq
// 04-ISSUES/046, ADR 0096). A published image is an index over several architectures;
// pulled, the store keeps the index and refuses to push one platform out of it, and
// every variant of pull-then-push failed the same way. A copy moves what is there.
if mirror, can := publish.(Mirrorer); can {
say("mirror", "copying %s into the mesh's registry", a.From)
reference, err := mirror.MirrorImage(ctx, a.From, module+"/"+a.Name)
if err != nil {
return catalogue.Built{}, fmt.Errorf("%s: %w", module, err)
}
return catalogue.Built{Name: a.Name, Kind: a.Kind, Reference: reference}, nil
}
// Genesis has no registry to copy into: the image stays in this machine's store, named by
// its own id, as every artifact does before there is anywhere to publish.
say("mirror", "pulling %s", a.From)
if _, err := run(ctx, tree, "docker", "pull", a.From); err != nil {
return catalogue.Built{}, fmt.Errorf("%s: cannot fetch %s: %w", module, a.From, err)
}
say("mirror", "publishing under the mesh's own name")
reference, err := publish.PublishImage(ctx, a.From, module+"/"+a.Name)
if err != nil {
return catalogue.Built{}, err
@@ -345,6 +372,39 @@ func one(ctx context.Context, run Runner, publish Publisher,
local := fmt.Sprintf("%s-%s:%s", module, a.Name, short(commit))
// The bases this module named, resolved to what this mesh holds. A recipe reads them as
// build arguments, so a module says which module it stands on and never which copy.
// **A recipe fetches nothing the manifest did not declare** (novox/hq 04-ISSUES/064). A FROM
// or a COPY --from naming a registry image that is not a declared base is a build that
// reaches a public registry on its own — and works when that registry answers, which is
// sometimes. Refused here, in front of the person who can declare it, not inside a build
// that fails with "pull access denied" for a reason that is not the mesh's.
recipe, err := os.ReadFile(filepath.Join(tree, a.From))
if err != nil {
return catalogue.Built{}, fmt.Errorf("%s: cannot read the recipe %s: %w", module, a.From, err)
}
declared := map[string]bool{}
for i := 0; i+1 < len(args); i += 2 {
if args[i] == "--build-arg" {
declared[strings.SplitN(args[i+1], "=", 2)[0]] = true
}
}
bases, copies := undeclaredFetches(string(recipe), declared)
if len(copies) > 0 {
return catalogue.Built{}, fmt.Errorf(
"%s: the recipe %s copies out of %s, which the manifest does not declare. A build "+
"reaching a public registry on its own works only when that registry answers; "+
"declare it under build.on as {\"arg\": \"<NAME>\", \"image\": \"<image>@sha256:…\"} "+
"and read it from that argument (novox/hq ADR 0097)",
module, a.From, strings.Join(copies, ", "))
}
if len(bases) > 0 {
// Refused, since the mesh's own images declare theirs (ADR 0097): a base fetched on
// its own is a build that works when a public registry answers, which is sometimes.
return catalogue.Built{}, fmt.Errorf(
"%s: the recipe %s starts FROM %s, which the manifest does not declare. Declare "+
"each under build.on as {\"arg\": \"<NAME>\", \"image\": \"<image>@sha256:…\"} "+
"and start FROM ${<NAME>} (novox/hq ADR 0097)",
module, a.From, strings.Join(bases, ", "))
}
invocation := append([]string{"build", "-f", a.From, "-t", local}, args...)
if a.Target != "" {
invocation = append(invocation, "--target", a.Target)
@@ -565,7 +625,8 @@ var _ io.Writer = (*stringWriter)(nil)
// one a container runtime produces when a recipe's first line refers to an image nobody has.
//
// The order is fixed so two builds of one commit invoke the same command.
func standingOn(manifest catalogue.Manifest, held map[string]string) ([]string, error) {
func standingOn(ctx context.Context, manifest catalogue.Manifest, held map[string]string,
mirror func(ctx context.Context, from, repository string) (string, error)) ([]string, error) {
if manifest.Build == nil || len(manifest.Build.On) == 0 {
return nil, nil
}
@@ -574,6 +635,27 @@ func standingOn(manifest catalogue.Manifest, held map[string]string) ([]string,
var args []string
for _, base := range on {
if base.Image != "" {
// A vendor's image, declared (novox/hq 04-ISSUES/064, ADR 0097). Pinned, because a tag
// is what somebody else can move; copied into the mesh's registry, because a build
// that reaches a public registry on its own is a build that works sometimes.
if base.Arg == "" || base.Module != "" || base.Artifact != "" {
return nil, fmt.Errorf(
"%s stands on the image %s, and a base is either a module's artifact or an "+
"image — never both — read from one build argument", manifest.Module, base.Image)
}
if !strings.Contains(base.Image, "@sha256:") {
return nil, fmt.Errorf(
"%s stands on the image %q, which is not pinned by digest. A tag is what "+
"somebody else can move; name it as <image>@sha256:…", manifest.Module, base.Image)
}
reference, err := mirror(ctx, base.Image, manifest.Module+"/on-"+strings.ToLower(base.Arg))
if err != nil {
return nil, fmt.Errorf("%s stands on %s: %w", manifest.Module, base.Image, err)
}
args = append(args, "--build-arg", base.Arg+"="+reference)
continue
}
if base.Arg == "" || base.Module == "" || base.Artifact == "" {
return nil, fmt.Errorf(
"%s says its build stands on something, and does not say all of what: a base "+
@@ -714,3 +796,124 @@ func sourcesFor(entrypoints []string, out string) []string {
func timeNow() time.Time { return time.Now() }
func since(t time.Time) string { return time.Since(t).Round(time.Millisecond).String() }
// undeclaredFetches is every image a recipe reaches for that is neither a declared build argument
// nor one of its own stages nor `scratch`, in two lists: the bases it starts `FROM`, and the images
// it `COPY --from`s out of — a vendor's tool, the case novox/hq 04-ISSUES/064 is about.
func undeclaredFetches(recipe string, declared map[string]bool) (bases, copies []string) {
stages := map[string]bool{}
seen := map[string]bool{}
var out *[]string
note := func(ref string) {
ref = strings.TrimSpace(ref)
switch {
case ref == "" || ref == "scratch" || stages[strings.ToLower(ref)]:
return
case strings.HasPrefix(ref, "$"):
name := strings.Trim(strings.TrimPrefix(ref, "$"), "{}")
if cut := strings.IndexAny(name, ":-"); cut >= 0 {
name = name[:cut]
}
if !declared[name] {
if !seen[ref] {
seen[ref] = true
*out = append(*out, ref+" (a build argument the manifest does not declare)")
}
}
return
}
// A stage referenced by number (COPY --from=0) is its own recipe's.
if _, err := strconv.Atoi(ref); err == nil {
return
}
if !seen[ref] {
seen[ref] = true
*out = append(*out, ref)
}
}
for _, line := range instructions(recipe) {
fields := strings.Fields(line)
switch strings.ToUpper(fields[0]) {
case "FROM":
// FROM [--platform=…] <ref> [AS <name>]
out = &bases
var ref string
for i := 1; i < len(fields); i++ {
if strings.HasPrefix(fields[i], "--") {
continue
}
ref = fields[i]
if i+2 < len(fields) && strings.EqualFold(fields[i+1], "AS") {
stages[strings.ToLower(fields[i+2])] = true
}
break
}
note(ref)
case "COPY", "ADD":
out = &copies
for _, f := range fields[1:] {
if strings.HasPrefix(f, "--from=") {
note(strings.TrimPrefix(f, "--from="))
}
}
case "RUN":
// RUN --mount=type=bind,from=<image>,… reaches for an image exactly as COPY --from does.
out = &copies
for _, f := range fields[1:] {
if !strings.HasPrefix(f, "--mount=") {
continue
}
for _, opt := range strings.Split(strings.TrimPrefix(f, "--mount="), ",") {
if from, found := strings.CutPrefix(opt, "from="); found {
note(from)
}
}
}
}
}
return bases, copies
}
// instructions is a recipe as its instructions, one per line: continuations joined, comments and
// blank lines dropped, and heredoc bodies (`COPY <<EOF … EOF`) skipped — a Python file written into
// an image is not a list of images to fetch. The review found a `COPY \` continued onto the next
// line slip past the check, and a stage named on a continuation line refused as a fetch.
func instructions(recipe string) []string {
var out []string
var current strings.Builder
var heredoc string
flush := func() {
if line := strings.TrimSpace(current.String()); line != "" && !strings.HasPrefix(line, "#") {
out = append(out, line)
}
current.Reset()
}
for _, raw := range strings.Split(recipe, "\n") {
if heredoc != "" {
if strings.TrimSpace(raw) == heredoc {
heredoc = ""
}
continue
}
line := strings.TrimRight(raw, " \t")
if strings.HasPrefix(strings.TrimSpace(line), "#") && current.Len() == 0 {
continue
}
if strings.HasSuffix(line, "\\") {
current.WriteString(strings.TrimSuffix(line, "\\"))
current.WriteString(" ")
continue
}
current.WriteString(line)
if at := strings.Index(current.String(), "<<"); at >= 0 {
// `<<EOF`, `<<-EOF`, `<<'EOF'`, `<<"EOF"`: the body runs to a line that is the word.
word := strings.Fields(current.String()[at+2:])
if len(word) > 0 {
heredoc = strings.Trim(strings.TrimPrefix(word[0], "-"), `'"`)
}
}
flush()
}
flush()
return out
}
+323
View File
@@ -0,0 +1,323 @@
package builder
import (
"bytes"
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"fmt"
"io"
"net/http"
"strings"
)
// An upstream image is copied between registries, never through a machine's image store
// (novox/hq 04-ISSUES/046, ADR 0096).
//
// A published image is ordinarily an index over several architectures. Pulling it leaves the index
// in the runtime's store, and pushing one platform out of that store is what the runtime refuses —
// every variant of pull-then-push was tried and failed the same way. A copy never needs a platform:
// it moves what is there. Read the index, read each manifest it names, put each blob by digest,
// put the manifests, put the index under the module's repository — the registry API is enough, and
// the runtime's image store is never involved.
// Mirrorer copies an upstream image into the mesh's own registry, whole.
type Mirrorer interface {
MirrorImage(ctx context.Context, from, repository string) (string, error)
}
const (
mediaIndexOCI = "application/vnd.oci.image.index.v1+json"
mediaIndexDocker = "application/vnd.docker.distribution.manifest.list.v2+json"
mediaManifestOCI = "application/vnd.oci.image.manifest.v1+json"
mediaManifestDocker = "application/vnd.docker.distribution.manifest.v2+json"
)
var manifestAccept = strings.Join([]string{mediaIndexOCI, mediaIndexDocker, mediaManifestOCI, mediaManifestDocker}, ", ")
// upstream is where an image lives, as the registry API addresses it.
type upstream struct {
// base is the scheme and host, e.g. https://registry-1.docker.io.
base string
// repository is the path under /v2/, e.g. library/alpine.
repository string
// reference is a tag or a digest.
reference string
}
// parseReference splits `[host/]repo[:tag][@digest]` the way a container runtime does: no host
// means the public hub, and a single-segment repository there lives under `library/`.
func parseReference(ref string) (upstream, error) {
name, reference := ref, "latest"
if at := strings.Index(ref, "@"); at >= 0 {
name, reference = ref[:at], ref[at+1:]
// `repo:tag@digest` is what a runtime prints; the digest names the image and the tag is
// only what it was called. The tag is not part of the repository.
if colon := strings.LastIndex(name, ":"); colon > strings.LastIndex(name, "/") {
name = name[:colon]
}
} else if colon := strings.LastIndex(ref, ":"); colon > strings.LastIndex(ref, "/") {
name, reference = ref[:colon], ref[colon+1:]
}
if name == "" || reference == "" {
return upstream{}, fmt.Errorf("%q is not an image reference", ref)
}
host, repository := "docker.io", name
if slash := strings.Index(name, "/"); slash >= 0 && strings.ContainsAny(name[:slash], ".:") {
host, repository = name[:slash], name[slash+1:]
} else if slash >= 0 && name[:slash] == "localhost" {
host, repository = name[:slash], name[slash+1:]
}
if host == "docker.io" {
host = "registry-1.docker.io"
if !strings.Contains(repository, "/") {
repository = "library/" + repository
}
}
scheme := "https://"
if strings.HasPrefix(host, "localhost") || strings.HasPrefix(host, "127.") {
scheme = "http://"
}
return upstream{base: scheme + host, repository: repository, reference: reference}, nil
}
// source reads from one upstream registry, taking a bearer token where the registry asks for one.
type source struct {
client *http.Client
token string
}
// get fetches a registry URL, answering a bearer challenge once with an anonymous token — which is
// how the public hub serves public images, and every registry the catalogue names does the same.
func (s *source) get(ctx context.Context, url, accept string) (*http.Response, error) {
for attempt := 0; attempt < 2; attempt++ {
request, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
return nil, err
}
if accept != "" {
request.Header.Set("Accept", accept)
}
if s.token != "" {
request.Header.Set("Authorization", "Bearer "+s.token)
}
response, err := s.client.Do(request)
if err != nil {
return nil, err
}
if response.StatusCode != http.StatusUnauthorized || attempt == 1 {
return response, nil
}
challenge := response.Header.Get("WWW-Authenticate")
response.Body.Close()
token, err := s.tokenFor(ctx, challenge)
if err != nil {
return nil, err
}
s.token = token
}
return nil, fmt.Errorf("unreachable")
}
// tokenFor answers `Bearer realm="…",service="…",scope="…"` with an anonymous token request.
func (s *source) tokenFor(ctx context.Context, challenge string) (string, error) {
if !strings.HasPrefix(challenge, "Bearer ") {
return "", fmt.Errorf("the registry asks for %q, and this copies public images anonymously", challenge)
}
fields := map[string]string{}
for _, part := range strings.Split(challenge[len("Bearer "):], ",") {
key, value, found := strings.Cut(strings.TrimSpace(part), "=")
if found {
fields[key] = strings.Trim(value, `"`)
}
}
realm := fields["realm"]
if realm == "" {
return "", fmt.Errorf("the registry's challenge names no realm: %q", challenge)
}
url := realm + "?service=" + fields["service"]
if scope := fields["scope"]; scope != "" {
url += "&scope=" + scope
}
request, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
return "", err
}
response, err := s.client.Do(request)
if err != nil {
return "", fmt.Errorf("cannot get a token from %s: %w", realm, err)
}
defer response.Body.Close()
var issued struct {
Token string `json:"token"`
AccessToken string `json:"access_token"`
}
if err := json.NewDecoder(response.Body).Decode(&issued); err != nil {
return "", fmt.Errorf("%s answered with something that is not a token: %w", realm, err)
}
if issued.Token == "" {
issued.Token = issued.AccessToken
}
if issued.Token == "" {
return "", fmt.Errorf("%s issued no token", realm)
}
return issued.Token, nil
}
// descriptor is what an index or a manifest names: a blob or another manifest, by digest.
type descriptor struct {
MediaType string `json:"mediaType"`
Digest string `json:"digest"`
Size int64 `json:"size"`
}
// MirrorImage copies `from` — an index or a single manifest, by tag or digest — into this registry
// under `repository`, and returns the reference the mesh will pin: this registry, the repository,
// and the digest of the document that was put last, which is the index where there is one.
func (r Registry) MirrorImage(ctx context.Context, from, repository string) (string, error) {
where, err := parseReference(from)
if err != nil {
return "", err
}
src := &source{client: r.client()}
digest, err := r.copyManifest(ctx, src, where, where.reference, repository)
if err != nil {
return "", fmt.Errorf("copying %s into %s/%s: %w", from, r.Address, repository, err)
}
return r.Address + "/" + repository + "@" + digest, nil
}
// copyManifest copies one manifest document and everything it names, and returns its digest. An
// index is copied by copying each manifest it names first, so the index never points at something
// the registry does not hold yet.
func (r Registry) copyManifest(ctx context.Context, src *source, where upstream, reference, repository string) (string, error) {
response, err := src.get(ctx, where.base+"/v2/"+where.repository+"/manifests/"+reference, manifestAccept)
if err != nil {
return "", err
}
defer response.Body.Close()
if response.StatusCode != http.StatusOK {
said, _ := io.ReadAll(io.LimitReader(response.Body, 2048))
return "", fmt.Errorf("%s/%s@%s: %s %s", where.base, where.repository, reference, response.Status, strings.TrimSpace(string(said)))
}
body, err := io.ReadAll(response.Body)
if err != nil {
return "", err
}
mediaType := response.Header.Get("Content-Type")
if semi := strings.Index(mediaType, ";"); semi >= 0 {
mediaType = mediaType[:semi]
}
var document struct {
MediaType string `json:"mediaType"`
Manifests []descriptor `json:"manifests"`
Config *descriptor `json:"config"`
Layers []descriptor `json:"layers"`
}
if err := json.Unmarshal(body, &document); err != nil {
return "", fmt.Errorf("%s is not a manifest: %w", reference, err)
}
if mediaType == "" || mediaType == "application/json" {
mediaType = document.MediaType
}
switch mediaType {
case mediaIndexOCI, mediaIndexDocker:
// The manifests first, each by its digest; the index that names them last.
for _, m := range document.Manifests {
if _, err := r.copyManifest(ctx, src, where, m.Digest, repository); err != nil {
return "", err
}
}
case mediaManifestOCI, mediaManifestDocker:
blobs := append([]descriptor{}, document.Layers...)
if document.Config != nil {
blobs = append(blobs, *document.Config)
}
for _, b := range blobs {
if err := r.copyBlob(ctx, src, where, b.Digest, repository); err != nil {
return "", err
}
}
default:
return "", fmt.Errorf("%s is a %q, which is neither an image index nor an image manifest", reference, mediaType)
}
digest := "sha256:" + hexOf(sha256.Sum256(body))
put, err := http.NewRequestWithContext(ctx, http.MethodPut,
"http://"+r.Address+"/v2/"+repository+"/manifests/"+digest, bytes.NewReader(body))
if err != nil {
return "", err
}
put.Header.Set("Content-Type", mediaType)
done, err := r.client().Do(put)
if err != nil {
return "", fmt.Errorf("cannot put a manifest into %s: %w", r.Address, err)
}
defer done.Body.Close()
if done.StatusCode != http.StatusCreated {
said, _ := io.ReadAll(io.LimitReader(done.Body, 2048))
return "", fmt.Errorf("%s refused the manifest %s: %s %s", r.Address, digest, done.Status, strings.TrimSpace(string(said)))
}
return digest, nil
}
// copyBlob moves one blob by digest, unless the registry already holds it — blobs are immutable
// and content-named, so "already there" is the whole check.
func (r Registry) copyBlob(ctx context.Context, src *source, where upstream, digest, repository string) error {
base := "http://" + r.Address + "/v2/" + repository
if there, err := r.has(ctx, base+"/blobs/"+digest); err != nil {
return err
} else if there {
return nil
}
response, err := src.get(ctx, where.base+"/v2/"+where.repository+"/blobs/"+digest, "")
if err != nil {
return err
}
defer response.Body.Close()
if response.StatusCode != http.StatusOK {
return fmt.Errorf("%s/%s: blob %s: %s", where.base, where.repository, digest, response.Status)
}
start, err := http.NewRequestWithContext(ctx, http.MethodPost, base+"/blobs/uploads/", nil)
if err != nil {
return err
}
begun, err := r.client().Do(start)
if err != nil {
return fmt.Errorf("cannot start an upload to %s: %w", base, err)
}
begun.Body.Close()
if begun.StatusCode != http.StatusAccepted {
return fmt.Errorf("%s answered %s when asked where to put a blob", base, begun.Status)
}
location := begun.Header.Get("Location")
if location == "" {
return fmt.Errorf("%s accepted an upload and said nowhere to put it", base)
}
if strings.HasPrefix(location, "/") {
location = "http://" + r.Address + location
}
put, err := http.NewRequestWithContext(ctx, http.MethodPut, location+separator(location)+"digest="+digest, response.Body)
if err != nil {
return err
}
put.Header.Set("Content-Type", "application/octet-stream")
if response.ContentLength > 0 {
put.ContentLength = response.ContentLength
}
done, err := r.client().Do(put)
if err != nil {
return fmt.Errorf("cannot upload blob %s: %w", digest, err)
}
defer done.Body.Close()
if done.StatusCode != http.StatusCreated {
said, _ := io.ReadAll(io.LimitReader(done.Body, 2048))
return fmt.Errorf("%s refused blob %s: %s %s", base, digest, done.Status, strings.TrimSpace(string(said)))
}
return nil
}
func hexOf(sum [32]byte) string { return hex.EncodeToString(sum[:]) }
+221
View File
@@ -0,0 +1,221 @@
package builder
import (
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"net/http"
"net/http/httptest"
"strings"
"sync"
"testing"
)
// An upstream image is copied between registries, never through a machine's image store
// (novox/hq 04-ISSUES/046, ADR 0096): the index, every manifest it names, every blob — moved by
// digest, and the index put last under the module's repository.
func digestOf(b []byte) string {
sum := sha256.Sum256(b)
return "sha256:" + hex.EncodeToString(sum[:])
}
// anUpstreamRegistry serves one image as an index over two platforms, behind an anonymous bearer
// challenge the way the public hub does, and records what was fetched.
func anUpstreamRegistry(t *testing.T) (*httptest.Server, string, map[string][]byte) {
t.Helper()
blobs := map[string][]byte{}
manifests := map[string][]byte{}
put := func(kind, mediaType string, layer []byte) string {
config := []byte(`{"architecture":"` + kind + `"}`)
blobs[digestOf(config)] = config
blobs[digestOf(layer)] = layer
m, _ := json.Marshal(map[string]any{
"schemaVersion": 2, "mediaType": mediaType,
"config": map[string]any{"mediaType": "application/vnd.oci.image.config.v1+json", "digest": digestOf(config), "size": len(config)},
"layers": []map[string]any{{"mediaType": "application/vnd.oci.image.layer.v1.tar+gzip", "digest": digestOf(layer), "size": len(layer)}},
})
manifests[digestOf(m)] = m
return digestOf(m)
}
amd := put("amd64", mediaManifestOCI, []byte("amd64 layer bytes"))
arm := put("arm64", mediaManifestOCI, []byte("arm64 layer bytes"))
index, _ := json.Marshal(map[string]any{
"schemaVersion": 2, "mediaType": mediaIndexOCI,
"manifests": []map[string]any{
{"mediaType": mediaManifestOCI, "digest": amd, "size": len(manifests[amd]), "platform": map[string]string{"os": "linux", "architecture": "amd64"}},
{"mediaType": mediaManifestOCI, "digest": arm, "size": len(manifests[arm]), "platform": map[string]string{"os": "linux", "architecture": "arm64"}},
},
})
manifests["latest"] = index
manifests[digestOf(index)] = index
var server *httptest.Server
server = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path == "/token" {
_, _ = w.Write([]byte(`{"token":"anonymous-token"}`))
return
}
if r.Header.Get("Authorization") != "Bearer anonymous-token" {
w.Header().Set("WWW-Authenticate", `Bearer realm="`+server.URL+`/token",service="test",scope="repository:library/thing:pull"`)
w.WriteHeader(http.StatusUnauthorized)
return
}
switch {
case strings.HasPrefix(r.URL.Path, "/v2/library/thing/manifests/"):
ref := strings.TrimPrefix(r.URL.Path, "/v2/library/thing/manifests/")
body, ok := manifests[ref]
if !ok {
w.WriteHeader(http.StatusNotFound)
return
}
var typed struct {
MediaType string `json:"mediaType"`
}
_ = json.Unmarshal(body, &typed)
w.Header().Set("Content-Type", typed.MediaType)
_, _ = w.Write(body)
case strings.HasPrefix(r.URL.Path, "/v2/library/thing/blobs/"):
body, ok := blobs[strings.TrimPrefix(r.URL.Path, "/v2/library/thing/blobs/")]
if !ok {
w.WriteHeader(http.StatusNotFound)
return
}
_, _ = w.Write(body)
default:
w.WriteHeader(http.StatusNotFound)
}
}))
return server, digestOf(index), blobs
}
// theMeshsRegistry accepts blobs and manifests the way a registry does, and remembers them.
type theMeshsRegistry struct {
mu sync.Mutex
blobs map[string][]byte
manifests map[string][]byte
uploads int
}
func (m *theMeshsRegistry) handler() http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
m.mu.Lock()
defer m.mu.Unlock()
switch {
case r.Method == http.MethodHead && strings.Contains(r.URL.Path, "/blobs/"):
if _, ok := m.blobs[r.URL.Path[strings.LastIndex(r.URL.Path, "/")+1:]]; ok {
w.WriteHeader(http.StatusOK)
} else {
w.WriteHeader(http.StatusNotFound)
}
case r.Method == http.MethodPost && strings.HasSuffix(r.URL.Path, "/blobs/uploads/"):
w.Header().Set("Location", strings.TrimSuffix(r.URL.Path, "/")+"/one")
w.WriteHeader(http.StatusAccepted)
case r.Method == http.MethodPut && strings.Contains(r.URL.Path, "/blobs/uploads/"):
body, _ := readAll(r)
digest := r.URL.Query().Get("digest")
if digestOf(body) != digest {
w.WriteHeader(http.StatusBadRequest)
return
}
m.blobs[digest] = body
m.uploads++
w.WriteHeader(http.StatusCreated)
case r.Method == http.MethodPut && strings.Contains(r.URL.Path, "/manifests/"):
body, _ := readAll(r)
m.manifests[r.URL.Path[strings.LastIndex(r.URL.Path, "/")+1:]] = body
w.WriteHeader(http.StatusCreated)
default:
w.WriteHeader(http.StatusNotFound)
}
})
}
func readAll(r *http.Request) ([]byte, error) {
var buf strings.Builder
b := make([]byte, 4096)
for {
n, err := r.Body.Read(b)
buf.Write(b[:n])
if err != nil {
break
}
}
return []byte(buf.String()), nil
}
func TestAnUpstreamIndexIsCopiedWholeIntoTheMeshsRegistry(t *testing.T) {
src, indexDigest, srcBlobs := anUpstreamRegistry(t)
defer src.Close()
dst := &theMeshsRegistry{blobs: map[string][]byte{}, manifests: map[string][]byte{}}
dstServer := httptest.NewServer(dst.handler())
defer dstServer.Close()
address := strings.TrimPrefix(dstServer.URL, "http://")
r := Registry{Address: address, HTTP: src.Client()}
from := strings.TrimPrefix(src.URL, "http://") + "/library/thing:latest"
reference, err := r.MirrorImage(context.Background(), from, "hello-web/server")
if err != nil {
t.Fatal(err)
}
// Pinned by the INDEX's digest under the module's own repository: what a machine fetches is
// the whole image, whatever its architecture.
if reference != address+"/hello-web/server@"+indexDigest {
t.Fatalf("pinned as %q, not the index under the module's repository", reference)
}
// Every blob of both platforms, moved by digest, and each only once.
if len(dst.blobs) != len(srcBlobs) || dst.uploads != len(srcBlobs) {
t.Fatalf("%d of %d blobs arrived in %d uploads", len(dst.blobs), len(srcBlobs), dst.uploads)
}
for digest, body := range srcBlobs {
if string(dst.blobs[digest]) != string(body) {
t.Fatalf("blob %s did not arrive intact", digest)
}
}
// Two manifests and the index, each under its digest.
if len(dst.manifests) != 3 {
t.Fatalf("expected two manifests and an index, got %d: %v", len(dst.manifests), dst.manifests)
}
if _, ok := dst.manifests[indexDigest]; !ok {
t.Fatal("the index was not put under its digest")
}
// Copied again, nothing is uploaded twice: blobs are content-named and already there.
if _, err := r.MirrorImage(context.Background(), from, "hello-web/server"); err != nil {
t.Fatal(err)
}
if dst.uploads != len(srcBlobs) {
t.Fatalf("a second copy uploaded blobs the registry already held: %d uploads", dst.uploads)
}
}
func TestAReferenceIsReadTheWayARuntimeReadsIt(t *testing.T) {
for ref, want := range map[string]upstream{
"alpine": {base: "https://registry-1.docker.io", repository: "library/alpine", reference: "latest"},
"alpine@sha256:abc": {base: "https://registry-1.docker.io", repository: "library/alpine", reference: "sha256:abc"},
"minio/minio:RELEASE.2025": {base: "https://registry-1.docker.io", repository: "minio/minio", reference: "RELEASE.2025"},
"quay.io/minio/mc@sha256:def": {base: "https://quay.io", repository: "minio/mc", reference: "sha256:def"},
"lscr.io/linuxserver/sonarr:4": {base: "https://lscr.io", repository: "linuxserver/sonarr", reference: "4"},
"localhost:5000/x/y:1": {base: "http://localhost:5000", repository: "x/y", reference: "1"},
} {
got, err := parseReference(ref)
if err != nil {
t.Fatalf("%s: %v", ref, err)
}
if got != want {
t.Errorf("%s: got %+v want %+v", ref, got, want)
}
}
}
// `repo:tag@digest` is what a runtime prints; the tag is not part of the repository (review C6).
func TestATagBeforeTheDigestIsNotPartOfTheRepository(t *testing.T) {
got, err := parseReference("quay.io/minio/mc:RELEASE.2025@sha256:abc")
if err != nil {
t.Fatal(err)
}
if got.repository != "minio/mc" || got.reference != "sha256:abc" {
t.Fatalf("got %+v", got)
}
}
+87 -4
View File
@@ -1,6 +1,8 @@
package builder
import (
"context"
"fmt"
"strings"
"testing"
@@ -20,7 +22,7 @@ func TestABaseTheMeshHasNotBuiltIsRefused(t *testing.T) {
On: []catalogue.BuildsOn{{Arg: "RUNTIME_BASE", Module: "mesh-tools", Artifact: "runtime"}},
},
}
_, err := standingOn(manifest, map[string]string{})
_, err := standingOn(context.Background(), manifest, map[string]string{}, noMirror)
if err == nil {
t.Fatal("a base nothing has built was accepted; the build would have failed on its first line")
}
@@ -40,7 +42,7 @@ func TestABaseTheMeshHoldsBecomesABuildArgument(t *testing.T) {
},
}
held := map[string]string{"mesh-tools/runtime": "127.0.0.1:5000/mesh-tools/runtime@sha256:" + strings.Repeat("a", 64)}
args, err := standingOn(manifest, held)
args, err := standingOn(context.Background(), manifest, held, noMirror)
if err != nil {
t.Fatalf("a base this mesh holds was refused: %v", err)
}
@@ -52,7 +54,7 @@ func TestABaseTheMeshHoldsBecomesABuildArgument(t *testing.T) {
// A module naming no base asks for nothing, which is most modules.
func TestAModuleNamingNoBaseAddsNoArguments(t *testing.T) {
args, err := standingOn(catalogue.Manifest{Module: "hello-web", Build: &catalogue.Build{}}, nil)
args, err := standingOn(context.Background(), catalogue.Manifest{Module: "hello-web", Build: &catalogue.Build{}}, nil, noMirror)
if err != nil || args != nil {
t.Fatalf("a module naming no base produced %v, %v", args, err)
}
@@ -64,7 +66,88 @@ func TestAnIncompleteBaseIsRefused(t *testing.T) {
Module: "postgres",
Build: &catalogue.Build{On: []catalogue.BuildsOn{{Module: "mesh-tools", Artifact: "runtime"}}},
}
if _, err := standingOn(manifest, map[string]string{"mesh-tools/runtime": "x"}); err == nil {
if _, err := standingOn(context.Background(), manifest, map[string]string{"mesh-tools/runtime": "x"}, noMirror); err == nil {
t.Fatal("a base with no build argument was accepted; nothing would have read it")
}
}
// noMirror is a mirror for tests whose bases are all the mesh's own.
func noMirror(context.Context, string, string) (string, error) {
return "", fmt.Errorf("nothing to copy in this test")
}
// A build may stand on an image published elsewhere, declared and pinned (novox/hq 04-ISSUES/064,
// ADR 0097): it is copied into the mesh's registry first and the recipe is handed the copy.
func TestADeclaredVendorImageIsCopiedInAndHandedToTheRecipe(t *testing.T) {
manifest := catalogue.Manifest{
Module: "minio",
Build: &catalogue.Build{
On: []catalogue.BuildsOn{{Arg: "MC_BASE", Image: "quay.io/minio/mc@sha256:" + strings.Repeat("c", 64)}},
},
}
var asked []string
args, err := standingOn(context.Background(), manifest, nil, func(_ context.Context, from, repository string) (string, error) {
asked = append(asked, from+" -> "+repository)
return "127.0.0.1:5000/" + repository + "@sha256:" + strings.Repeat("d", 64), nil
})
if err != nil {
t.Fatal(err)
}
if len(asked) != 1 || asked[0] != "quay.io/minio/mc@sha256:"+strings.Repeat("c", 64)+" -> minio/on-mc_base" {
t.Fatalf("the image was not copied under the module's repository: %v", asked)
}
if strings.Join(args, " ") != "--build-arg MC_BASE=127.0.0.1:5000/minio/on-mc_base@sha256:"+strings.Repeat("d", 64) {
t.Fatalf("the recipe was not handed the copy: %v", args)
}
// Unpinned, it is refused: a tag is what somebody else can move.
manifest.Build.On[0].Image = "quay.io/minio/mc:latest"
if _, err := standingOn(context.Background(), manifest, nil, noMirror); err == nil || !strings.Contains(err.Error(), "not pinned") {
t.Fatalf("an unpinned vendor image was accepted: %v", err)
}
}
// A recipe reaching for an image the manifest did not declare is named, and its own stages,
// declared arguments and scratch are not.
func TestARecipeFetchingWhatTheManifestDidNotDeclareIsNamed(t *testing.T) {
recipe := `
ARG RUNTIME_BASE
ARG MC_BASE
FROM ${RUNTIME_BASE} AS build
COPY --from=${MC_BASE} /usr/bin/mc /usr/local/bin/mc
COPY --from=build /out /out
COPY --from=0 /x /x
FROM scratch
COPY --from=vendor/tool:latest /tool /tool
FROM golang:1.25-alpine AS go
`
bases, copies := undeclaredFetches(recipe, map[string]bool{"RUNTIME_BASE": true})
if strings.Join(copies, "|") != "${MC_BASE} (a build argument the manifest does not declare)|vendor/tool:latest" {
t.Fatalf("copies out of undeclared images: %v", copies)
}
// A base fetched on its own is named apart, and refused like a copy (ADR 0097).
if strings.Join(bases, "|") != "golang:1.25-alpine" {
t.Fatalf("undeclared bases: %v", bases)
}
if _, copies := undeclaredFetches(recipe, map[string]bool{"RUNTIME_BASE": true, "MC_BASE": true}); len(copies) != 1 {
t.Fatalf("declared arguments are not fetches: %v", copies)
}
}
// Continued lines are one instruction, heredoc bodies are not instructions, and a RUN --mount reaches
// for an image as a COPY --from does (review C4, C5).
func TestARecipeIsReadAsInstructions(t *testing.T) {
recipe := "ARG RUNTIME_BASE\n" +
"FROM ${RUNTIME_BASE} \\\n AS build\n" +
"COPY \\\n --from=docker.io/vendor/one:latest /a /a\n" +
"COPY --from=build /out /out\n" +
"COPY <<EOF /app/x.py\nfrom os import path\nEOF\n" +
"RUN --mount=type=bind,from=docker.io/vendor/two:1,target=/t cp /t/x /x\n" +
"FROM scratch\n"
bases, copies := undeclaredFetches(recipe, map[string]bool{"RUNTIME_BASE": true})
if strings.Join(copies, "|") != "docker.io/vendor/one:latest|docker.io/vendor/two:1" {
t.Fatalf("copies: %v", copies)
}
if len(bases) != 0 {
t.Fatalf("a heredoc line or a continued stage was read as a base: %v", bases)
}
}
+237
View File
@@ -0,0 +1,237 @@
package catalogue
import (
"fmt"
"sort"
"strconv"
"strings"
)
// What an adopted node is declared in place of a filter (novox/hq ADR 0100).
//
// On an adopted node the firewall found on the machine stays in force: the mesh loads no table
// that drops by default or holds an accept. What the mesh needs reachable is declared as
// openings, which the host converges through the found firewall in its own terms; and the mesh
// guards its own foundation ports itself, in a table that only refuses.
// AdoptionPrefix is the id prefix of what the mesh declares of its own on an adopted node. It is
// never a module's, so none of it is ever held as found.
const AdoptionPrefix = "adoption."
// Where an opening admits from, on the wire.
const (
OpeningFromEverywhere = "everywhere"
OpeningFromMesh = "mesh"
)
// The two paths a packet reaches a port by: received by the machine, or forwarded to a
// container that publishes it.
const (
PathIncoming = "incoming"
PathForwarded = "forwarded"
)
// Guard resources: the refusal-only table, the unit that loads it, and that unit running.
const (
GuardPath = "/etc/mesh/guard.nft"
GuardUnit = "mesh-guard.service"
// GuardUnitPath is where the unit is written.
GuardUnitPath = "/etc/systemd/system/" + GuardUnit
)
// GuardID, GuardUnitID and GuardRunningID are the guard's resource identities. The installer
// raises the same three on an adopted genesis, so the first push finds them already there.
func GuardID() string { return AdoptionPrefix + "guard" }
func GuardUnitID() string { return AdoptionPrefix + "guard-unit" }
func GuardRunningID() string { return AdoptionPrefix + "guard-running" }
// GuardPackageID is the tool that loads the guard, declared first: a node joining adopted has
// no filter module and may have no nft at all, and a table nothing can load guards nothing.
func GuardPackageID() string { return AdoptionPrefix + "guard-package" }
// GuardPackage is the package that carries nft.
const GuardPackage = "nftables"
// OpeningID is an opening's resource identity: its protocol, port and path say what it is.
func OpeningID(protocol string, port int, path string) string {
return fmt.Sprintf("%sopening-%s-%d-%s", AdoptionPrefix, protocol, port, path)
}
// Openings are what the mesh needs reachable on an adopted node, from the same inputs as the
// filter it would load were the node converged, each from where that filter would admit it.
//
// `rules` is Filtering's answer — every module's listens, the hub's port, the per-node exposure —
// and `foundation` is the ports the mesh itself needs, from everywhere. A rule for this machine
// only opens nothing. `published` maps a machine port a container publishes to the container's
// port: a published port is forwarded, not received, so its opening names the forwarded path and
// the port the packet is forwarded to.
func Openings(rules []Rule, foundation []int, published map[string]map[int]int) []map[string]any {
type key struct {
protocol string
port int
}
from := map[key]string{}
var order []key
widen := func(k key, f string) {
was, seen := from[k]
if !seen {
order = append(order, k)
}
if !seen || was != OpeningFromEverywhere {
from[k] = f
}
}
for _, rule := range rules {
switch rule.From {
case FromEverywhere:
widen(key{rule.Protocol, rule.Port}, OpeningFromEverywhere)
case FromMesh:
widen(key{rule.Protocol, rule.Port}, OpeningFromMesh)
}
}
for _, port := range foundation {
widen(key{"tcp", port}, OpeningFromEverywhere)
}
sort.Slice(order, func(a, b int) bool {
if order[a].port != order[b].port {
return order[a].port < order[b].port
}
return order[a].protocol < order[b].protocol
})
out := make([]map[string]any, 0, len(order))
for _, k := range order {
opening := map[string]any{"type": "opening", "port": k.port, "protocol": k.protocol,
"from": from[k]}
if to, forwarded := published[k.protocol][k.port]; forwarded {
opening["id"] = OpeningID(k.protocol, k.port, PathForwarded)
opening["path"] = PathForwarded
opening["to"] = to
} else {
opening["id"] = OpeningID(k.protocol, k.port, PathIncoming)
opening["path"] = PathIncoming
}
out = append(out, opening)
}
return out
}
// Published is every port the given containers publish on the machine, by protocol and machine
// port, mapped to the container's own port. A mapping bound to loopback is left out: nothing off
// the machine reaches it, forwarded or not.
func Published(resources []map[string]any) map[string]map[int]int {
out := map[string]map[int]int{}
for _, r := range resources {
if fmt.Sprint(r["type"]) != "container" {
continue
}
listed, _ := r["ports"].([]any)
for _, entry := range listed {
written := strings.TrimSpace(fmt.Sprint(entry))
protocol := "tcp"
if cut := strings.LastIndex(written, "/"); cut >= 0 {
protocol = written[cut+1:]
}
// Indexed from the end, so an IPv6 address's own colons never shift the ports.
outer, inner, address, ok := mapping(written)
if !ok {
continue
}
switch strings.Trim(address, "[]") {
case "127.0.0.1", "localhost", "::1":
continue
}
if out[protocol] == nil {
out[protocol] = map[int]int{}
}
out[protocol][outer] = inner
}
}
return out
}
// AsGuard renders the mesh's refusal-only table for the given machine ports.
//
// It passes everything by default and holds nothing but a refusal, so it cannot close anything
// the machine serves; and it is the mesh's own table, so the found firewall reloading does not
// touch it. It refuses only packets addressed to this machine, and the ports except from the
// machine itself — its loopback and the container runtime's own networks — and from the private
// network, known by the interface a packet arrives on and never by its source address. At
// prerouting, ahead of the runtime's destination translation, so it matches the port the packet
// was sent to; in the inet family, so both address families.
//
// **The machine's own interfaces are named, where the derived filter names address ranges.** The
// filter accepts the container runtime's networks by CIDR; this excludes its bridges by name — lo,
// docker0, the br-* a compose network gets, and the mesh's own mesh0. A runtime whose bridge is
// named anything else (a podman or libvirt bridge, or a docker network created with a fixed name)
// would have its containers' traffic to a guarded port refused, which reads as the port being
// down. Names rather than addresses is deliberate: a source address can be claimed by whoever
// sends the packet, and this table exists to refuse what the found firewall never sees. Widening
// it means adding names here and in the installer's copy together, which the golden test holds to
// one text.
//
// The same text the installer raises on an adopted genesis; a test holds both to it.
func AsGuard(ports []int) string {
sorted := append([]int{}, ports...)
sort.Ints(sorted)
listed := make([]string, len(sorted))
for i, p := range sorted {
listed[i] = strconv.Itoa(p)
}
var b strings.Builder
b.WriteString("table inet mesh_guard {}\n")
b.WriteString("delete table inet mesh_guard\n")
b.WriteString("table inet mesh_guard {\n")
b.WriteString("\tchain prerouting {\n")
b.WriteString("\t\ttype filter hook prerouting priority raw; policy accept;\n")
// Only packets addressed to this machine: traffic it routes for others — a predecessor's hub,
// say — is never the guard's business (novox/hq ADR 0103).
fmt.Fprintf(&b, "\t\tfib daddr type local iifname != \"lo\" iifname != \"docker0\" "+
"iifname != \"br-*\" iifname != \"mesh0\" tcp dport { %s } drop\n",
strings.Join(listed, ", "))
b.WriteString("\t}\n")
b.WriteString("}\n")
return b.String()
}
// GuardUnitText is the unit that loads the guard. Stopping it deletes only its own table: never
// a flush, which would take the container runtime's rules and the found firewall with it.
func GuardUnitText() string {
return "[Unit]\n" +
"Description=The mesh's guard: refuses its own ports from outside (novox/hq ADR 0100)\n" +
// Early, before the network is up, and without the default dependencies that would
// order it after the network; stopped at shutdown like any unit.
"DefaultDependencies=no\n" +
"Wants=network-pre.target\n" +
"Before=network-pre.target shutdown.target\n" +
"Conflicts=shutdown.target\n" +
"\n" +
"[Service]\n" +
"Type=oneshot\n" +
"RemainAfterExit=yes\n" +
"ExecStart=nft -f " + GuardPath + "\n" +
"ExecReload=nft -f " + GuardPath + "\n" +
"ExecStop=nft delete table inet mesh_guard\n" +
"\n" +
"[Install]\n" +
"WantedBy=multi-user.target\n"
}
// GuardResources are the guard as four resources of the existing kinds: the tool that loads it,
// the table, the unit, and the unit running — reloaded when the table changes, so the new table
// replaces the old in one `nft -f` through the unit's ExecReload with no moment unguarded, and
// restarted only when the unit itself changes. Nothing when there is nothing to guard: an empty
// set is not a table nft loads.
func GuardResources(ports []int) []map[string]any {
if len(ports) == 0 {
return nil
}
return []map[string]any{
{"id": GuardPackageID(), "type": "package", "package": GuardPackage},
{"id": GuardID(), "type": "file", "path": GuardPath, "content": AsGuard(ports),
"mode": "0644"},
{"id": GuardUnitID(), "type": "file", "path": GuardUnitPath, "content": GuardUnitText(),
"mode": "0644"},
{"id": GuardRunningID(), "type": "service", "unit": GuardUnit, "state": "running",
"boot": "enabled", "restart-on": []any{GuardUnitID()}, "reload-on": []any{GuardID()}},
}
}
+675
View File
@@ -0,0 +1,675 @@
package catalogue
import (
"encoding/json"
"reflect"
"slices"
"strings"
"testing"
)
// novox/hq ADR 0100: on an adopted node the found firewall stays in force. The mesh declares
// openings where it would have loaded a filter, and guards its own ports in a table that only
// refuses.
// hub is the private network's generator on the hub: it opens the hub's port from anywhere.
type hub struct{}
func (hub) Resources(string) ([]map[string]any, bool, error) {
return []map[string]any{{"id": "config", "type": "file", "path": "/etc/wireguard/mesh0.conf",
"content": "[Interface]\n"}}, true, nil
}
func (hub) Listens(string) ([]Listening, error) {
return []Listening{{Port: 51820, Protocol: "udp", From: FromEverywhere}}, nil
}
// anAdoptedAnchor is the control-node's set: the store, the bus, the registry, the private network,
// a served module and the filter module.
func anAdoptedAnchor() Resolution {
return Resolution{Node: "anchor", Modules: []Manifest{
{Module: "network", Computed: "overlay"},
{Module: "postgres", Guards: []int{5432},
Listens: []Listening{{Port: 5432, From: FromMesh}},
Resources: []map[string]any{{"id": "server", "type": "container", "name": "mesh-store",
"ports": []any{"5432:5432"}}}},
{Module: "lavinmq", Guards: []int{15672},
Listens: []Listening{{Port: 5671, From: FromMesh}, {Port: 5672, From: FromMesh}},
Resources: []map[string]any{{"id": "server", "type": "container", "name": "mesh-broker",
"ports": []any{"5671:5671", "5672:5672", "127.0.0.1:15672:15672"}}}},
{Module: "distribution",
Listens: []Listening{{Port: 5000, From: FromMesh}},
Resources: []map[string]any{{"id": "store", "type": "container", "name": "registry",
"ports": []any{"5000"}}}},
{Module: "hello-web",
Listens: []Listening{{Port: 8080, From: FromEverywhere}},
Resources: []map[string]any{{"id": "server", "type": "container", "name": "hello-web",
"ports": []any{"8080"}}}},
{Module: "helper", Listens: []Listening{{Port: 9000, From: FromMachine}}},
{Module: "nftables", Filtering: &Filtering{Into: "/etc/nftables.conf"},
Resources: []map[string]any{{"id": "load", "type": "service", "unit": "mesh-filter.service",
"state": "running", "restart-on": []any{"filtering"}}}},
}}
}
func anchorRendering(adopted bool) Rendering {
return Rendering{
Generators: map[string]Generator{"overlay": hub{}},
Ports: map[string]map[int]int{"distribution": {5000: 5000}, "hello-web": {8080: 20001}},
Settings: SettingsBy{"distribution": {{From: "node anchor",
Values: map[string]any{ExposeSetting: map[string]any{"5000": FromEverywhere}}}}},
Mesh: []string{"10.42.0.1"},
Foundation: []int{5671},
Adopted: adopted,
// Genesis takes the foundation's modules.
Taken: map[string]bool{"postgres": true, "lavinmq": true},
}
}
func byID(resources []map[string]any) map[string]map[string]any {
out := map[string]map[string]any{}
for _, r := range resources {
out[r["id"].(string)] = r
}
return out
}
func TestAnAdoptedNodeIsDeclaredOpeningsFromTheSameInputsAsTheFilter(t *testing.T) {
composed, err := anAdoptedAnchor().Compose(anchorRendering(true))
if err != nil {
t.Fatal(err)
}
got := byID(composed.Resources)
want := map[string]map[string]any{
// The hub's port, from anywhere, received.
"adoption.opening-udp-51820-incoming": {"port": 51820, "protocol": "udp",
"from": "everywhere", "path": "incoming"},
// The store's port from the private network only, and forwarded: a container publishes it.
"adoption.opening-tcp-5432-forwarded": {"port": 5432, "protocol": "tcp", "from": "mesh",
"path": "forwarded", "to": 5432},
// The bus from anywhere: a node enrols over it before it has a private address.
"adoption.opening-tcp-5671-forwarded": {"port": 5671, "protocol": "tcp",
"from": "everywhere", "path": "forwarded", "to": 5671},
"adoption.opening-tcp-5672-forwarded": {"port": 5672, "protocol": "tcp", "from": "mesh",
"path": "forwarded", "to": 5672},
// The registry from anywhere, by its node's exposure setting.
"adoption.opening-tcp-5000-forwarded": {"port": 5000, "protocol": "tcp",
"from": "everywhere", "path": "forwarded", "to": 5000},
// A published port names the machine port and the container port it is forwarded to.
"adoption.opening-tcp-20001-forwarded": {"port": 20001, "protocol": "tcp",
"from": "everywhere", "path": "forwarded", "to": 8080},
}
for id, fields := range want {
opening, ok := got[id]
if !ok {
t.Errorf("no %s among %v", id, keys(got))
continue
}
if opening["type"] != "opening" {
t.Errorf("%s is a %v", id, opening["type"])
}
for k, v := range fields {
if opening[k] != v {
t.Errorf("%s: %s is %v, want %v", id, k, opening[k], v)
}
}
}
for id := range got {
if strings.HasPrefix(id, "adoption.opening-") && want[id] == nil {
t.Errorf("an opening nothing asked for: %s", id)
}
}
// A port for this machine only opens nothing, and the management port is not opened at all.
for id := range got {
if strings.Contains(id, "-9000-") || strings.Contains(id, "-15672-") {
t.Errorf("%s is opened", id)
}
}
// And openings come first, in the order the machine applies them.
if !strings.HasPrefix(composed.Resources[0]["id"].(string), "adoption.opening-") {
t.Errorf("openings are not first: %v", composed.Resources[0]["id"])
}
}
func TestAnAdoptedNodeLoadsNoFilterOfTheMeshs(t *testing.T) {
composed, err := anAdoptedAnchor().Compose(anchorRendering(true))
if err != nil {
t.Fatal(err)
}
for _, r := range composed.Resources {
if r["path"] == "/etc/nftables.conf" || strings.HasPrefix(r["id"].(string), "nftables.") {
t.Fatalf("an adopted node is declared the filter module's %v", r["id"])
}
if content, _ := r["content"].(string); strings.Contains(content, "policy drop") {
t.Fatalf("an adopted node is declared a table that drops by default: %v", r["id"])
}
// Nothing but refusals: the only accept in the guard is its policy.
if content, _ := r["content"].(string); r["id"] == GuardID() &&
strings.Count(content, "accept") != 1 {
t.Fatalf("the guard holds an accept:\n%s", content)
}
}
got := byID(composed.Resources)
guard := got[GuardID()]
if guard == nil || got[GuardUnitID()] == nil || got[GuardRunningID()] == nil {
t.Fatalf("no guard: %v", keys(got))
}
if guard["content"] != AsGuard([]int{5432, 5672, 15672}) {
t.Fatalf("the guard does not guard the store, the broker and its management port:\n%s",
guard["content"])
}
// The tool that loads it comes first, and the table after it: a node joining adopted has no
// filter module and may have no nft.
pkg, table := -1, -1
for i, r := range composed.Resources {
switch r["id"] {
case GuardPackageID():
pkg = i
if r["type"] != "package" || r["package"] != "nftables" {
t.Fatalf("the guard's package is %v", r)
}
case GuardID():
table = i
}
}
if pkg < 0 || pkg > table {
t.Fatalf("nftables is not declared before the guard's table (%d, %d)", pkg, table)
}
// A changed table is reloaded — one `nft -f`, atomic — never restarted, which would delete the
// table and leave the ports unguarded until it is loaded again. Only a changed unit restarts.
if !reflect.DeepEqual(got[GuardRunningID()]["reload-on"], []any{GuardID()}) ||
!reflect.DeepEqual(got[GuardRunningID()]["restart-on"], []any{GuardUnitID()}) {
t.Fatalf("the guard is not reloaded on its table and restarted on its unit: %v",
got[GuardRunningID()])
}
// Nothing of the mesh's own is anybody's to hold.
for id, module := range composed.Owner {
if strings.HasPrefix(id, AdoptionPrefix) {
t.Fatalf("%s is owned by %s", id, module)
}
}
}
func TestAConvergedNodeIsDeclaredItsFilterAndNoOpenings(t *testing.T) {
composed, err := anAdoptedAnchor().Compose(anchorRendering(false))
if err != nil {
t.Fatal(err)
}
got := byID(composed.Resources)
if got["nftables.filtering"] == nil || got["nftables.load"] == nil {
t.Fatalf("a converged node lost its filter: %v", keys(got))
}
for id := range got {
if strings.HasPrefix(id, AdoptionPrefix) {
t.Fatalf("a converged node is declared %s", id)
}
}
plain, err := anAdoptedAnchor().Declaration(anchorRendering(false))
if err != nil {
t.Fatal(err)
}
a, _ := json.Marshal(plain)
b, _ := json.Marshal(composed.Resources)
if string(a) != string(b) {
t.Fatal("Compose and Declaration disagree on a converged node")
}
}
// The table the installer raises and the controller declares, character for character.
func TestTheGuardIsExactlyThisTable(t *testing.T) {
const golden = `table inet mesh_guard {}
delete table inet mesh_guard
table inet mesh_guard {
chain prerouting {
type filter hook prerouting priority raw; policy accept;
fib daddr type local iifname != "lo" iifname != "docker0" iifname != "br-*" iifname != "mesh0" tcp dport { 5432, 15672 } drop
}
}
`
if got := AsGuard([]int{15672, 5432}); got != golden {
t.Fatalf("the guard changed:\n%s", got)
}
const unit = `[Unit]
Description=The mesh's guard: refuses its own ports from outside (novox/hq ADR 0100)
DefaultDependencies=no
Wants=network-pre.target
Before=network-pre.target shutdown.target
Conflicts=shutdown.target
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=nft -f /etc/mesh/guard.nft
ExecReload=nft -f /etc/mesh/guard.nft
ExecStop=nft delete table inet mesh_guard
[Install]
WantedBy=multi-user.target
`
if got := GuardUnitText(); got != unit {
t.Fatalf("the guard's unit changed:\n%s", got)
}
if GuardResources(nil) != nil {
t.Fatal("a guard with nothing to guard is an empty set nft refuses to load")
}
}
func TestAGuardedPortMustBeAPort(t *testing.T) {
if _, err := ParseManifest([]byte(`{"module":"postgres","guards":[5432]}`)); err != nil {
t.Fatalf("guards is refused: %v", err)
}
if _, err := ParseManifest([]byte(`{"module":"postgres","guards":[0]}`)); err == nil {
t.Fatal("guarding port 0 was accepted")
}
}
func keys[V any](m map[string]V) []string {
return sortedKeys(m)
}
// novox/hq ADR 0100: the foundation's ports are the node's. Given 5433 for the store, every place
// that uses the port reads it from there: the container, the filter, the openings, the guard.
func TestAGivenPortIsUsedEverywhereThePortIs(t *testing.T) {
given := map[string]map[int]int{"postgres": {5432: 5433}, "lavinmq": {15672: 15673}}
for _, adopted := range []bool{true, false} {
with := anchorRendering(adopted)
with.Given = given
with.Ports["postgres"] = map[int]int{5432: 5433}
composed, err := anAdoptedAnchor().Compose(with)
if err != nil {
t.Fatal(err)
}
got := byID(composed.Resources)
if ports := got["postgres.server"]["ports"]; !reflect.DeepEqual(ports, []any{"5433:5432"}) {
t.Fatalf("the store's container publishes %v", ports)
}
if ports := got["lavinmq.server"]["ports"]; !reflect.DeepEqual(ports,
[]any{"5671:5671", "5672:5672", "127.0.0.1:15673:15672"}) {
t.Fatalf("the broker's container publishes %v", ports)
}
if !adopted {
filter, _ := got["nftables.filtering"]["content"].(string)
if !strings.Contains(filter, "tcp dport 5433 accept") || strings.Contains(filter, "5432") {
t.Fatalf("the filter does not use the given port:\n%s", filter)
}
continue
}
if o := got["adoption.opening-tcp-5433-forwarded"]; o == nil || o["to"] != 5432 {
t.Fatalf("no opening for the given port: %v", keys(got))
}
if guard := got[GuardID()]["content"]; guard != AsGuard([]int{5433, 5672, 15673}) {
t.Fatalf("the guard does not guard the given ports:\n%s", guard)
}
}
}
func TestAGivenPortIsTheNodesAndReachesSomething(t *testing.T) {
store := anAdoptedAnchor().Modules[1]
node := func(v any) []Layer {
return []Layer{{From: "anchor", Values: map[string]any{PortsSetting: v}}}
}
if got, err := GivenPorts(store, node(map[string]any{"5432": float64(5433)})); err != nil ||
got[5432] != 5433 {
t.Fatalf("a node's given port was not read: %v %v", got, err)
}
if _, err := GivenPorts(store, []Layer{{From: MeshWideLayer,
Values: map[string]any{PortsSetting: map[string]any{"5432": float64(5433)}}}}); err == nil {
t.Fatal("a port given for the whole mesh was accepted")
}
if _, err := GivenPorts(store, node(map[string]any{"6000": float64(6001)})); err == nil {
t.Fatal("a port the module neither listens on, publishes nor guards was given")
}
if _, err := GivenPorts(store, node(map[string]any{"5432": float64(70000)})); err == nil {
t.Fatal("a machine port that is not a port was given")
}
if _, err := GivenPorts(store, node(map[string]any{"5432": float64(22)})); err == nil {
t.Fatal("ssh's port was given")
}
broker := anAdoptedAnchor().Modules[2]
if _, err := GivenPorts(broker, node(map[string]any{"5671": float64(5700),
"5672": float64(5700)})); err == nil {
t.Fatal("one machine port was given for two of the module's ports")
}
if stray := UnusedSettings(store, node(map[string]any{"5432": float64(5433)})); len(stray) != 0 {
t.Fatalf("a given port is called stray: %v", stray)
}
}
// novox/hq ADR 0103: the guard is derived, and from taken modules only — every machine port a taken
// module publishes that the filter admits from the private network only, and the ports its
// manifest guards. A module assigned but not taken is not guarded: its port may still be the
// predecessor's.
func TestTheGuardIsDerivedFromTakenModulesOnly(t *testing.T) {
guardOf := func(with Rendering) string {
t.Helper()
composed, err := anAdoptedAnchor().Compose(with)
if err != nil {
t.Fatal(err)
}
content, _ := byID(composed.Resources)[GuardID()]["content"].(string)
return content
}
// The broker taken, the store not: the broker's plain port follows from its listens (from
// the mesh, published), its management port from its manifest; the store is not guarded, and
// neither is the bus, which the mesh needs from everywhere.
with := anchorRendering(true)
with.Taken = map[string]bool{"lavinmq": true}
if got := guardOf(with); got != AsGuard([]int{5672, 15672}) {
t.Fatalf("the guard is not the taken broker's ports:\n%s", got)
}
// A taken module publishing a port admitted from everywhere is not guarded; one admitted from
// the mesh is. The registry is exposed everywhere on this node, and hello-web listens from
// everywhere.
with.Taken = map[string]bool{"distribution": true, "hello-web": true}
if got := guardOf(with); got != "" {
t.Fatalf("a port admitted from everywhere is guarded:\n%s", got)
}
with.Settings = nil
if got := guardOf(with); got != AsGuard([]int{5000}) {
t.Fatalf("the registry, from the mesh only, is not guarded:\n%s", got)
}
// Nothing taken, nothing guarded — and no guard at all rather than an empty set.
with = anchorRendering(true)
with.Taken = nil
if got := guardOf(with); got != "" {
t.Fatalf("an untaken store is guarded:\n%s", got)
}
// A given port is followed: where the machine put it is what is refused.
with = anchorRendering(true)
with.Given = map[string]map[int]int{"lavinmq": {5672: 5682, 15672: 15673}}
with.Ports["lavinmq"] = map[int]int{5671: 5671, 5672: 5682}
if got := guardOf(with); got != AsGuard([]int{5432, 5682, 15673}) {
t.Fatalf("the guard does not follow the given ports:\n%s", got)
}
}
// A mapping bound to loopback is not published to anything off the machine — in either address
// family — and an address's own colons never shift the ports.
func TestPublishedLeavesOutLoopbackInBothFamilies(t *testing.T) {
got := Published([]map[string]any{{"type": "container", "ports": []any{
"127.0.0.1:15672:15672", "[::1]:8080:80", "localhost:9090:90",
"[::]:8443:443", "0.0.0.0:5000:5000", "5353:53/udp"}}})
want := map[string]map[int]int{"tcp": {8443: 443, 5000: 5000}, "udp": {5353: 53}}
if !reflect.DeepEqual(got, want) {
t.Fatalf("published is %v, want %v", got, want)
}
}
// novox/hq ADR 0038: only a published port is the mesh's to move. A module that binds the machine
// itself listens where its software was told to, so giving it a machine port is refused.
func TestAGivenPortIsRefusedForAPortNoContainerPublishes(t *testing.T) {
onTheMachine := Manifest{Module: "daemon",
Listens: []Listening{{Port: 9000, From: FromMesh}}, Guards: []int{9000}}
layers := []Layer{{From: "node anchor",
Values: map[string]any{PortsSetting: map[string]any{"9000": float64(9100)}}}}
_, err := GivenPorts(onTheMachine, layers)
if err == nil || !strings.Contains(err.Error(), "does not publish") {
t.Fatalf("a port no container publishes was given: %v", err)
}
}
// novox/hq ADR 0103: a guarded port this node is told to open to everyone is opened, not guarded.
// An opening from everywhere beside a guard dropping it is one statement refusing the other.
func TestAGuardedPortOpenedToEveryoneIsNotGuarded(t *testing.T) {
with := anchorRendering(true)
with.Settings["postgres"] = []Layer{{From: "node anchor",
Values: map[string]any{ExposeSetting: map[string]any{"5432": FromEverywhere}}}}
composed, err := anAdoptedAnchor().Compose(with)
if err != nil {
t.Fatal(err)
}
got := byID(composed.Resources)
if o := got["adoption.opening-tcp-5432-forwarded"]; o == nil || o["from"] != OpeningFromEverywhere {
t.Fatalf("the store's port is not opened to everyone: %v", o)
}
if guard, _ := got[GuardID()]["content"].(string); guard != AsGuard([]int{5672, 15672}) {
t.Fatalf("a port opened to everyone is still guarded:\n%s", guard)
}
}
// A module that publishes its ssh port the long way — `2222:22`, because the machine's own daemon
// holds 22 — and says it listens on the machine side of that mapping, which is what anything
// reaching it dials.
func aForge() Manifest {
return Manifest{Module: "forge",
Provides: []Offer{{Name: "git-over-ssh", Scope: ScopeMesh}},
Serves: map[string]map[string]any{"git-over-ssh": {"port": 2222}},
Listens: []Listening{{Port: 3000, From: FromMesh}, {Port: 2222, From: FromMesh}},
Resources: []map[string]any{{"id": "server", "type": "container", "name": "forge",
"ports": []any{"3000", "2222:22"}}}}
}
// portsAsThePlanWould is where this machine puts each of a module's ports, derived the way
// cmd/mesh-controller/plan.go derives it: a given port first, looked up by the port the module
// says it listens on, and otherwise wherever the manifest's own mapping already put it. Written
// here because everything below — the filter, the openings, the guard, what a consumer is told —
// reads that map, and a given port that the lookup does not find moves the container's mapping
// and nothing else.
//
// It stands in for the plan only where the plan does not allocate: a port the manifest already
// placed, or one a node was given. For a short form with no given port the real plan asks the
// inventory for a machine port and may come back with one from the pool, which needs a store and
// is what cmd/mesh-controller's own tests exercise. So an assertion here about such a port asserts
// this helper, not the mesh; keep the assertions to the ports under test.
func portsAsThePlanWould(m Manifest, given map[int]int) map[int]int {
out := map[int]int{}
for _, l := range m.Listens {
if at, is := given[l.Port]; is {
out[l.Port] = at
continue
}
at, _ := m.MachineSide(l.Port)
out[l.Port] = at
}
return out
}
// novox/hq ADR 0100 and 0038: the machine side of a long-form mapping is a port the module
// publishes, so a node may move it — and a module may name a mapping by either end.
func TestAGivenPortNamesEitherEndOfWhatTheModulePublishes(t *testing.T) {
forge := aForge()
node := func(v any) []Layer {
return []Layer{{From: "node anchor", Values: map[string]any{PortsSetting: v}}}
}
// The machine side — the number this module says it listens on, and the one the predecessor
// had somewhere else. Answered under 2222, the end the module itself names, which is what
// every reader of this map asks for. One entry, not two: a second key for the same answer is
// an entry another mapping's reader could find instead.
given, err := GivenPorts(forge, node(map[string]any{"2222": float64(222)}))
if err != nil {
t.Fatalf("the machine side of a mapping cannot be given a port: %v", err)
}
if want := map[int]int{2222: 222}; !reflect.DeepEqual(given, want) {
t.Fatalf("the forge was given %v; the mapping it names is filed under %v", given, want)
}
// The container's own port names the same mapping and means the same thing — and is filed
// under the same name, because the module's name for it has not changed.
if inside, err := GivenPorts(forge, node(map[string]any{"22": float64(222)})); err != nil ||
!reflect.DeepEqual(inside, map[int]int{2222: 222}) {
t.Fatalf("the container's end of the mapping was given %v: %v", inside, err)
}
// A short form is published on the number it names, and is unchanged by any of this.
if short, err := GivenPorts(forge, node(map[string]any{"3000": float64(2999)})); err != nil ||
short[3000] != 2999 {
t.Fatalf("the short form was given %v: %v", short, err)
}
// A port no container publishes is still refused, in the same words.
if _, err := GivenPorts(forge, node(map[string]any{"9000": float64(9100)})); err == nil ||
!strings.Contains(err.Error(), "does not publish") {
t.Fatalf("a port the forge does not publish was given: %v", err)
}
// And the two ends of one mapping given two different numbers is one setting contradicting
// the other: the machine publishes it once.
if _, err := GivenPorts(forge, node(map[string]any{
"2222": float64(222), "22": float64(300)})); err == nil ||
!strings.Contains(err.Error(), "one mapping") {
t.Fatalf("the two ends of one mapping were given different ports: %v", err)
}
// Said at both ends with the same number, it is still said twice, and refused where every
// other repeated machine port is — as the inventory refuses it when the setting is stored,
// which is the layer that sees it first.
if _, err := GivenPorts(forge, node(map[string]any{
"2222": float64(222), "22": float64(222)})); err == nil ||
!strings.Contains(err.Error(), "to both its 22 and its 2222") {
t.Fatalf("one mapping given one machine port at both ends: %v", err)
}
// A number that names two different mappings names neither: which one to move is not said.
twice := aForge()
twice.Resources[0]["ports"] = []any{"22", "2222:22"}
if _, err := GivenPorts(twice, node(map[string]any{"22": float64(222)})); err == nil ||
!strings.Contains(err.Error(), "twice") {
t.Fatalf("a number naming two of the module's mappings was accepted: %v", err)
}
// And the same refusal when the number naming two mappings is not the one the setting used
// but the one the answer would be filed under. Here `80` is the module's own name for a
// mapping, and two mappings wear it; filing an answer there is one container's port standing
// where the other's is read, and both containers then publish it.
shared := Manifest{Module: "gallery",
Listens: []Listening{{Port: 80, From: FromMesh}},
Resources: []map[string]any{
{"id": "a", "type": "container", "name": "a", "ports": []any{"4001:80"}},
{"id": "b", "type": "container", "name": "b", "ports": []any{"4002:80"}}}}
if _, err := GivenPorts(shared, node(map[string]any{"4001": float64(1234)})); err == nil ||
!strings.Contains(err.Error(), "twice") {
t.Fatalf("two containers were put on one machine port: %v", err)
}
// A module whose mappings chain — one's machine side is another's container port — keeps them
// apart, because each is filed under the port the module names it by and neither name is
// shared. Given both, each moves on its own and neither overwrites the other.
chained := Manifest{Module: "chain",
Listens: []Listening{{Port: 80, From: FromMesh}, {Port: 9090, From: FromMesh}},
Resources: []map[string]any{{"id": "server", "type": "container", "name": "chain",
"ports": []any{"8080:80", "9090:8080"}}}}
both, err := GivenPorts(chained, node(map[string]any{"80": float64(1234), "9090": float64(5678)}))
if err != nil || !reflect.DeepEqual(both, map[int]int{80: 1234, 9090: 5678}) {
t.Fatalf("chained mappings were given %v: %v", both, err)
}
if moved := givenOuter("8080:80", both); moved != "1234:80" {
t.Fatalf("the first mapping moved to %q, and it was given 1234", moved)
}
if moved := givenOuter("9090:8080", both); moved != "5678:8080" {
t.Fatalf("the second mapping moved to %q, and it was given 5678", moved)
}
}
// And the number reaches everything derived from it. The fault this is written against moved the
// container's mapping alone: the filter opened the port the software had left, the adopted node's
// opening named it too, the guard refused it, and a consumer was sent to it.
func TestAGivenMachineSideReachesTheFilterTheOpeningAndTheConsumer(t *testing.T) {
forge := aForge()
given, err := GivenPorts(forge, []Layer{{From: "node anchor",
Values: map[string]any{PortsSetting: map[string]any{"2222": float64(222)}}}})
if err != nil {
t.Fatalf("the machine side of a mapping cannot be given a port: %v", err)
}
r := Resolution{Node: "anchor", Modules: []Manifest{forge}}
with := Rendering{
Ports: map[string]map[int]int{"forge": portsAsThePlanWould(forge, given)},
Given: map[string]map[int]int{"forge": given},
Mesh: []string{"10.77.0.1"},
Adopted: true,
Taken: map[string]bool{"forge": true},
}
// What the runtime is handed: the machine's own port on the outside, the container's within.
composed, err := r.Compose(with)
if err != nil {
t.Fatalf("the forge does not compose: %v", err)
}
got := byID(composed.Resources)
if ports := got["forge.server"]["ports"]; !reflect.DeepEqual(ports, []any{"3000:3000", "222:22"}) {
t.Fatalf("the forge's container publishes %v", ports)
}
// What the filter would open, were the node converged.
rules, err := r.Rules(with)
if err != nil {
t.Fatal(err)
}
var opened []int
for _, rule := range rules {
opened = append(opened, rule.Port)
}
// Only the moved port is asserted: the forge's other port is a short form the real plan would
// allocate rather than read off the manifest, so its number here is portsAsThePlanWould's and
// not the mesh's.
if !slices.Contains(opened, 222) || slices.Contains(opened, 2222) {
t.Fatalf("the filter opens %v, not where the machine puts the forge", opened)
}
// And what it is declared instead, adopted: an opening on the machine's port, naming the
// container's port on the forwarded path — a published port is forwarded, never received.
opening := got[OpeningID("tcp", 222, PathForwarded)]
if opening == nil || opening["to"] != 22 || opening["from"] != OpeningFromMesh {
t.Fatalf("no opening for the port this node gave the forge: %v", keys(got))
}
for id := range got {
if strings.HasPrefix(id, "adoption.opening-tcp-2222-") {
t.Errorf("an opening for the port the forge was moved off: %s", id)
}
}
// The guard refuses it where the machine put it, and nothing where it used to be.
guard, _ := got[GuardID()]["content"].(string)
if !strings.Contains(guard, "222") || strings.Contains(guard, "2222") {
t.Fatalf("the guard does not follow the given port:\n%s", guard)
}
// And a consumer is sent to the same number, which is read from what the module serves.
if told := ServedOn(forge, "git-over-ssh", with.Ports["forge"])["port"]; told != 222 {
t.Fatalf("a consumer is told the forge answers on %v", told)
}
}
// And the mapping itself moves under either name, because Rendering.Given is a map anybody
// composing a declaration hands in: keyed by the machine side, which is what a module declaring
// 2222 calls its port, only the outside moves and the container's own port stays as written.
func TestAMappingIsMovedUnderEitherOfItsNames(t *testing.T) {
for _, c := range []struct {
written string
given map[int]int
want string
}{
{"2222:22", map[int]int{2222: 222}, "222:22"},
{"2222:22", map[int]int{22: 222}, "222:22"},
{"127.0.0.1:15672:15672/tcp", map[int]int{15672: 15673}, "127.0.0.1:15673:15672/tcp"},
{"2222:22", map[int]int{3000: 2999}, "2222:22"},
{"2222:22", nil, "2222:22"},
} {
if got := givenOuter(c.written, c.given); got != c.want {
t.Errorf("%s given %v is published as %s, want %s", c.written, c.given, got, c.want)
}
}
}
// The same on a node that was given nothing, which is where the derivation was never at fault:
// a long-form mapping is published where the manifest put it, and an adopted node opens that port
// on the forwarded path like any other. What broke it was the number reaching only the mapping.
func TestALongFormPortIsOpenedWhereTheManifestPublishesIt(t *testing.T) {
forge := aForge()
r := Resolution{Node: "anchor", Modules: []Manifest{forge}}
composed, err := r.Compose(Rendering{
Ports: map[string]map[int]int{"forge": portsAsThePlanWould(forge, nil)},
Mesh: []string{"10.77.0.1"},
Adopted: true,
})
if err != nil {
t.Fatal(err)
}
got := byID(composed.Resources)
opening := got[OpeningID("tcp", 2222, PathForwarded)]
if opening == nil || opening["to"] != 22 {
t.Fatalf("no opening for the forge's published ssh port: %v", keys(got))
}
}
@@ -0,0 +1,38 @@
package catalogue
import (
"strings"
"testing"
)
// The artifact store's seat is one per mesh, read from the catalogue beside this checkout.
//
// **A second store anywhere is refused by name, not discovered as a consumer failure.** The seat
// was node-scoped, so a second `distribution` on another machine resolved cleanly there — and a
// node-scoped requirement with one candidate installs that candidate on the node, so anything that
// required the store's presence beside it would have raised a fresh, empty store on the wrong
// machine. Only afterwards did the mesh notice: `artifact-store` offered by two nodes, and every
// consumer elsewhere refusing to choose. The claim says it first, where the second store is
// assigned.
func TestASecondArtifactStoreAnywhereIsRefusedByName(t *testing.T) {
store := catalogueManifest(t, "distribution")
// The first store resolves as it always has.
if _, err := Resolve(shelf(store), []string{"distribution"}, workstation(), World{}); err != nil {
t.Fatalf("the store alone does not resolve: %v", err)
}
// A second one, on any other machine, is refused — and the refusal names the seat.
elsewhere := World{Held: []Held{{Claim: "the-artifact-store", Scope: ScopeMesh,
Node: "anchor", Module: "distribution"}}}
other := workstation()
other.Name = "laptop"
_, err := Resolve(shelf(store), []string{"distribution"}, other, elsewhere)
if err == nil {
t.Fatal("a second store was accepted on another machine; it would offer artifact-store a " +
"second time and every consumer elsewhere would refuse to choose")
}
if !strings.Contains(err.Error(), "the-artifact-store") || !strings.Contains(err.Error(), "one per mesh") {
t.Fatalf("refused without naming the seat: %v", err)
}
}
+145
View File
@@ -0,0 +1,145 @@
package catalogue
import (
"fmt"
"strings"
)
// What the mesh built, named by what it is rather than by where it was pushed.
//
// **An image is recorded by its digest and its path; the registry's address is a route to it**
// (novox/hq 04-ISSUES/102). A build used to be recorded as `<registry>:<port>/<module>/<artifact>@sha256:…`
// — the reference the builder pushed to, kept whole — and every declaration carried that literal.
// Move the registry's port, or the registry, and every fresh pull of a mesh image fails: a new
// node, a recreate after eviction. The digest is the identity; the address is the node's setting
// for the module that serves the artifact store, and it is read from there when a reference is
// composed, never written into a record.
//
// So a kept reference has a scheme of the mesh's own, named after the provision that answers it:
//
// artifact-store://<module>/<artifact>@sha256:<hex> an image
// artifact-store://<module>/<artifact>/blobs/sha256:<hex> an archive
//
// No runtime knows the scheme. That is the point of it being one: a reference that leaks to a
// machine uncomposed is refused by the runtime as malformed, in front of whoever sent it, rather
// than pulled from a public registry that happens to have a repository by that name — which is
// what an address-less `<module>/<artifact>@sha256:…` would be.
// ArtifactStoreScheme marks a reference to something in the mesh's artifact store, kept without
// the store's address.
const ArtifactStoreScheme = ArtifactStoreProvision + "://"
// Recorded is a reference as the mesh records it: the artifact store's address, if the builder wrote
// one, taken off.
//
// For a reference the builder announced — one it pushed to the store — which is the only kind
// this is called on. An image the builder named `<host>/<path>@sha256:…` is kept as its path; an
// archive it named `http://<host>/v2/<path>/blobs/<digest>` likewise. A reference with no address
// in it — an image id from a genesis build that had nowhere to publish, a package version — is
// what it was.
func Recorded(reference string) string {
if strings.HasPrefix(reference, ArtifactStoreScheme) {
return reference
}
if rest, isURL := strings.CutPrefix(reference, "http://"); isURL {
if _, path, ok := strings.Cut(rest, "/v2/"); ok && strings.Contains(path, "/blobs/") {
return ArtifactStoreScheme + path
}
return reference
}
host, path, ok := strings.Cut(reference, "/")
if !ok || !isRegistryHost(host) || !strings.Contains(path, "@sha256:") {
return reference
}
return ArtifactStoreScheme + path
}
// isRegistryHost is the runtime's own rule for reading the first component of a reference as a
// registry rather than as a namespace: it has a dot or a port in it, or it is localhost.
func isRegistryHost(component string) bool {
return component == "localhost" || strings.ContainsAny(component, ".:")
}
// InArtifactStore reports whether a reference is a kept one, and what it names there.
func InArtifactStore(reference string) (path string, kept bool) {
return strings.CutPrefix(reference, ArtifactStoreScheme)
}
// Routed is a kept reference as a machine fetches it, through the artifact store at `address`
// (host:port). A reference that is not a kept one is what it was.
func Routed(reference, address string) string {
path, kept := InArtifactStore(reference)
if !kept {
return reference
}
if strings.Contains(path, "/blobs/") {
return "http://" + address + "/v2/" + path
}
return address + "/" + path
}
// Rerouted is a reference the mesh recorded, whichever way it was recorded, as a machine fetches
// it now: a kept one composed with the store's address, and one recorded before references were
// kept without their address — the builder's own `<host>/<path>@sha256:…` — re-routed to where
// the store is now. Only for references that are the mesh's own: everything a build record
// holds is, by construction.
func Rerouted(reference, address string) string {
return Routed(Recorded(reference), address)
}
// artifactsInto composes the artifact store's address into a resource's `image` and `source`.
//
// **Composed here, at the last moment before a machine, and stored nowhere.** A kept reference is
// routed through the store as this network reaches it now. A reference recorded with an address
// before references were kept without one is re-routed the same way — but only when the mesh
// built it (`with.Built` names every `<module>/<artifact>` it has), because a module may run an
// image from a public registry under its own name and that one is exactly where it says.
//
// A kept reference with no store to route it through is refused: sent as it is, the runtime would
// refuse the scheme on the machine, one push away from the reason.
//
// **What this does not reach: the images genesis pinned.** The installer builds the control plane
// and the builder before the mesh exists, pushes them itself and pins their manifests to
// `<registry>:<port>/mesh-controller@…` and `<registry>:<port>/mesh-builder@…` — single-segment
// repositories with no build record, so `with.Built` does not name them and they are left as
// written until each is rebuilt through the mesh, which records it by digest and path. Until then
// a registry that moves strands exactly those two on a recreate, and the control plane's is the
// one that cannot be repaired through the mesh. Rebuild both through `build` before moving the
// store (novox/hq 04-ISSUES/102, finding F4).
func artifactsInto(resource map[string]any, module string, with Rendering) error {
for _, key := range []string{"image", "source"} {
written, ok := resource[key].(string)
if !ok {
continue
}
if _, kept := InArtifactStore(written); kept {
if with.ArtifactStore == "" {
return fmt.Errorf(
"%s's %v names %s, which is in the mesh's artifact store, and this mesh has no "+
"artifact store on its network to fetch it from — nothing assigned offers "+
"%s, or the machine offering it is not on the private network",
module, resource["id"], written, ArtifactStoreProvision)
}
resource[key] = Routed(written, with.ArtifactStore)
continue
}
if with.ArtifactStore == "" {
continue
}
recorded := Recorded(written)
if recorded == written {
continue
}
path, _ := InArtifactStore(recorded)
repository := path
if at := strings.IndexAny(path, "@"); at >= 0 {
repository = path[:at]
} else if blobs := strings.Index(path, "/blobs/"); blobs >= 0 {
repository = path[:blobs]
}
if with.Built[repository] {
resource[key] = Routed(recorded, with.ArtifactStore)
}
}
return nil
}
+137
View File
@@ -0,0 +1,137 @@
package catalogue
import (
"strings"
"testing"
)
// A build is recorded by what it is; where it is pushed is composed where it is used (novox/hq
// 04-ISSUES/102).
var digest = "sha256:" + strings.Repeat("d", 64)
func TestAReferenceIsRecordedWithoutTheStoresAddress(t *testing.T) {
cases := map[string]string{
"anchor.internal:5100/gitea/server@" + digest: ArtifactStoreScheme + "gitea/server@" + digest,
"localhost:5000/gitea/server@" + digest: ArtifactStoreScheme + "gitea/server@" + digest,
"http://anchor.internal:5100/v2/gitea/config/blobs/" + digest: ArtifactStoreScheme + "gitea/config/blobs/" + digest,
ArtifactStoreScheme + "gitea/server@" + digest: ArtifactStoreScheme + "gitea/server@" + digest,
digest: digest,
"@novox/sdk@1.2.3": "@novox/sdk@1.2.3",
"gitea/gitea@" + digest: "gitea/gitea@" + digest,
"https://registry.example/v2/gitea/config/blobs/" + digest: "https://registry.example/v2/gitea/config/blobs/" + digest,
}
for announced, want := range cases {
if got := Recorded(announced); got != want {
t.Errorf("Recorded(%q) = %q, want %q", announced, got, want)
}
}
}
func TestARecordedReferenceIsRoutedThroughTheStoreAsItIsNow(t *testing.T) {
if got := Routed(ArtifactStoreScheme+"gitea/server@"+digest, "anchor.internal:5101"); got != "anchor.internal:5101/gitea/server@"+digest {
t.Errorf("an image is fetched as %q", got)
}
if got := Routed(ArtifactStoreScheme+"gitea/config/blobs/"+digest, "anchor.internal:5101"); got != "http://anchor.internal:5101/v2/gitea/config/blobs/"+digest {
t.Errorf("an archive is fetched as %q", got)
}
if got := Routed("gitea/gitea@"+digest, "anchor.internal:5101"); got != "gitea/gitea@"+digest {
t.Errorf("a reference that is not the store's was routed: %q", got)
}
// One recorded before references were kept without their address follows the store too.
if got := Rerouted("anchor.internal:5100/gitea/server@"+digest, "anchor.internal:5101"); got != "anchor.internal:5101/gitea/server@"+digest {
t.Errorf("a reference recorded with the old address stays there: %q", got)
}
}
// **The address is composed into a declaration, and the record never carries it.**
func TestAnImageTheMeshBuiltIsRoutedThroughTheStoreWhenDeclared(t *testing.T) {
m := Manifest{Module: "gitea", Version: "1", Resources: []map[string]any{
{"id": "server", "type": "container", "name": "mesh-gitea", "artifact": "server"},
{"id": "config", "type": "archive", "path": "/etc/gitea", "artifact": "config"},
{"id": "cache", "type": "container", "name": "mesh-gitea-cache", "image": "valkey/valkey@" + digest},
}, Build: &Build{Artifacts: []Artifact{
{Name: "server", Kind: ArtifactImage, From: "Dockerfile"},
{Name: "config", Kind: ArtifactArchive, From: "config"},
}}}
resolved, err := m.Resolve([]Built{
{Name: "server", Kind: ArtifactImage, Reference: ArtifactStoreScheme + "gitea/server@" + digest},
{Name: "config", Kind: ArtifactArchive, Reference: ArtifactStoreScheme + "gitea/config/blobs/" + digest, Digest: digest},
})
if err != nil {
t.Fatal(err)
}
r := Resolution{Node: "anchor", Modules: []Manifest{resolved}}
out, err := r.Declaration(Rendering{ArtifactStore: "anchor.internal:5101"})
if err != nil {
t.Fatal(err)
}
if got := fileNamed(out, "gitea.server")["image"]; got != "anchor.internal:5101/gitea/server@"+digest {
t.Errorf("the image the mesh built is fetched as %v", got)
}
if got := fileNamed(out, "gitea.config")["source"]; got != "http://anchor.internal:5101/v2/gitea/config/blobs/"+digest {
t.Errorf("the archive the mesh built is fetched from %v", got)
}
if got := fileNamed(out, "gitea.cache")["image"]; got != "valkey/valkey@"+digest {
t.Errorf("an image from a public registry was routed through the store: %v", got)
}
// The manifest the mesh holds still says what it is, not where it was fetched from.
if got := resolved.Resources[0]["image"]; got != ArtifactStoreScheme+"gitea/server@"+digest {
t.Errorf("composing wrote the address into the catalogue's copy: %v", got)
}
// And the store moves: the same record, another address, without a rebuild.
out, err = r.Declaration(Rendering{ArtifactStore: "laptop.internal:5000"})
if err != nil {
t.Fatal(err)
}
if got := fileNamed(out, "gitea.server")["image"]; got != "laptop.internal:5000/gitea/server@"+digest {
t.Errorf("after the store moved, the image is still fetched as %v", got)
}
}
func TestAnImageInTheStoreWithNoStoreToFetchItFromIsRefused(t *testing.T) {
m := Manifest{Module: "gitea", Version: "1", Resources: []map[string]any{
{"id": "server", "type": "container", "name": "mesh-gitea",
"image": ArtifactStoreScheme + "gitea/server@" + digest},
}}
_, err := Resolution{Node: "anchor", Modules: []Manifest{m}}.Declaration(Rendering{})
if err == nil || !strings.Contains(err.Error(), "no artifact store") {
t.Fatalf("a reference nothing can fetch was sent to a machine: %v", err)
}
}
// **A reference recorded with an address before this follows the store too** — when the mesh
// built it. A module running an image straight from a public registry under its own name is left
// exactly where it says: `quay.io/keycloak/keycloak` is not the mesh's, whatever it is called.
func TestAReferenceRecordedWithAnAddressFollowsTheStoreWhenTheMeshBuiltIt(t *testing.T) {
m := Manifest{Module: "keycloak", Version: "1", Resources: []map[string]any{
{"id": "server", "type": "container", "name": "mesh-keycloak",
"image": "anchor.internal:5100/keycloak/server@" + digest},
{"id": "upstream", "type": "container", "name": "mesh-keycloak-upstream",
"image": "quay.io/keycloak/keycloak@" + digest},
}}
r := Resolution{Node: "anchor", Modules: []Manifest{m}}
out, err := r.Declaration(Rendering{
ArtifactStore: "anchor.internal:5101",
Built: map[string]bool{"keycloak/server": true},
})
if err != nil {
t.Fatal(err)
}
if got := fileNamed(out, "keycloak.server")["image"]; got != "anchor.internal:5101/keycloak/server@"+digest {
t.Errorf("an image the mesh built, recorded with the old address, is fetched as %v", got)
}
if got := fileNamed(out, "keycloak.upstream")["image"]; got != "quay.io/keycloak/keycloak@"+digest {
t.Errorf("a public image was re-routed through the store: %v", got)
}
// Nothing known to be built: nothing re-routed, nothing refused.
out, err = r.Declaration(Rendering{ArtifactStore: "anchor.internal:5101"})
if err != nil {
t.Fatal(err)
}
if got := fileNamed(out, "keycloak.server")["image"]; got != "anchor.internal:5100/keycloak/server@"+digest {
t.Errorf("with no build record, a reference was rewritten: %v", got)
}
}
+45
View File
@@ -1,6 +1,7 @@
package catalogue
import (
"fmt"
"strings"
"testing"
)
@@ -157,3 +158,47 @@ func TestTwoWaysToBeOnAPrivateNetworkRefuseAndNameBoth(t *testing.T) {
}
}
}
// reloading answers the runtime's trust as the networking module does (novox/hq ADR 0102): a file
// written into, and the runtime reloaded on it.
type reloading struct{}
func (reloading) Resources(node string) ([]map[string]any, bool, error) {
return []map[string]any{
{"id": "registry-trust", "type": "file", "path": "/etc/docker/daemon.json",
"merge": MergeJSON, "into": "json", "content": `{"insecure-registries":["r:5000"]}`},
{"id": "registry-trust-reload", "type": "service", "unit": "docker.service",
"state": "running", "reload-on": []string{"registry-trust"}},
}, true, nil
}
func TestWhatAServiceIsReloadedOnIsNamedAsTheHostWillSeeIt(t *testing.T) {
// Ids are prefixed with their module on the way out. An unprefixed reload-on would name a
// resource the host never sees, and the runtime would never be reloaded for its trust.
got, err := Resolve(computedShelf(), []string{"mesh-network"}, workstation(), World{})
if err != nil {
t.Fatal(err)
}
out, err := got.Declaration(Rendering{Generators: map[string]Generator{"mesh-network": reloading{}}})
if err != nil {
t.Fatal(err)
}
var file, service map[string]any
for _, r := range out {
switch r["type"] {
case "file":
file = r
case "service":
service = r
}
}
if file == nil || service == nil {
t.Fatalf("got %v", out)
}
if file["into"] != "json" {
t.Errorf("into did not reach the host: %v", file)
}
if want := "[" + file["id"].(string) + "]"; fmt.Sprint(service["reload-on"]) != want {
t.Errorf("reload-on names %v, the file is %v", service["reload-on"], file["id"])
}
}
+369 -57
View File
@@ -60,6 +60,9 @@ type Grant struct {
// Values are what that module contributed — the name it wants, and anything else the
// provision's own vocabulary defines.
Values map[string]any
// Local is the name the credential goes by inside the consumer where it keeps several for one
// provision (ADR 0094); empty for the ordinary one. The provider sees it as a holder of its own.
Local string
// Slug is the consumer module's identity slug, if it declared one — carried on the grant so the
// provider side derives the same login the consumer does, even across nodes where the consumer's
// manifest is not in view (novox/hq ADR 0049). Empty means "use the module name".
@@ -89,6 +92,10 @@ type Rendering struct {
// mesh" resolves to. Passed in for the same reason grants are: who else is on the network is
// a fact about the mesh, and resolution answers questions about one machine.
Mesh []string
// Suffix is what a machine's internal name ends in, as the control plane composed Names —
// `internal` unless the operator chose another — so a fact writing those names does not
// compose it a second time.
Suffix string
// Kept is every operator-sealed secret in the mesh, for a module that `keeps` them. Nil when
// nothing on this node keeps them, or the mesh has no operator key.
@@ -107,6 +114,14 @@ type Rendering struct {
// because which machines exist is a fact about the mesh.
Names map[string]string
// Machines is only the machines, by the same internal name — the subset of Names that is a
// node of this mesh rather than a name it was told to serve. Both matter and they are not the
// same set: a container's hosts wants every name, so a routed name resolves to the proxy that
// serves it, while a resolver told the mesh's suffix is authoritative for it answers from what
// it is given and forwards nothing — so a routed name written there is a name nobody asks for,
// standing beside the machines and looking as real as they do.
Machines map[string]string
Settings SettingsBy
Generators map[string]Generator
// Grants are the credentials this node must create, for the provisions it offers. Passed in
@@ -121,12 +136,47 @@ type Rendering struct {
// set, for what a consumer is told, and for what the runtime publishes — and nothing checked
// that the three agreed. They are all derived from this.
Ports map[string]map[int]int
// Adopted says the node is adopted (novox/hq ADR 0100): the firewall found on it stays in
// force, so no module that loads a filter is declared there, and what the mesh needs
// reachable is declared as openings, with its own ports guarded by a table that only refuses.
Adopted bool
// Given is the machine ports this node was given for its modules' ports, by module and by the
// port the software uses (novox/hq ADR 0100) — the foundation's ports, as genesis chose them.
// They win over anything the mesh would assign and over a manifest's own long-form mapping.
Given map[string]map[int]int
// Taken is the modules taken on this adopted node (novox/hq ADR 0100). The guard is derived
// from these only (ADR 0103): a port of a module assigned but not taken may still be the
// predecessor's.
Taken map[string]bool
// Seats is where this machine put each mesh-scoped seat's holder, by seat and by the port the
// holder's software uses (novox/hq 04-ISSUES/102) — read from the node's settings and
// assignments for whichever module claims the seat, whether or not it is in this node's set.
// What ${seat:…} answers with; see seat_into.go for why the answer may be absent.
Seats map[string]map[int]int
// ArtifactStore is the mesh's artifact store as this network reaches it (host:port) — the
// node holding it and the port that node put it on — or empty when the mesh has none on its
// network yet. Composed into every image and archive the mesh built, at this moment and never
// stored (novox/hq 04-ISSUES/102).
ArtifactStore string
// Built is every `<module>/<artifact>` the mesh has built. What tells a reference recorded
// with an address — before references were kept without one — from an image a module runs
// straight from a public registry.
Built map[string]bool
}
// machinePort is where a module's port lives on this machine, or the port itself when the mesh has
// not been asked. Unassigned is not an error here: a module with no `listens` never needed one,
// and a caller composing a declaration without a store still gets something coherent.
func (r Rendering) machinePort(module string, wanted int) int {
if at, given := r.Given[module][wanted]; given {
return at
}
if at, known := r.Ports[module][wanted]; known {
return at
}
@@ -139,6 +189,34 @@ func (r Rendering) machinePort(module string, wanted int) int {
// both call something "config", and without this the second would silently replace the first —
// the node applying one of them and reporting success.
func (r Resolution) Declaration(with Rendering) ([]map[string]any, error) {
composed, err := r.Compose(with)
if err != nil {
return nil, err
}
return composed.Resources, nil
}
// Composed is a declaration's resources and which module each came from.
//
// Owner is kept beside the resources because a resource id cannot be split back into its module:
// a module's name may itself contain a dot. What the mesh adds of its own — an opening, the guard —
// has no owner.
type Composed struct {
Resources []map[string]any
Owner map[string]string
}
// Compose is Declaration with the owner of every resource said.
func (r Resolution) Compose(with Rendering) (Composed, error) {
owner := map[string]string{}
resources, err := r.compose(with, owner)
if err != nil {
return Composed{}, err
}
return Composed{Resources: resources, Owner: owner}, nil
}
func (r Resolution) compose(with Rendering, owner map[string]string) ([]map[string]any, error) {
// Where each provision's credentials land, so a contribution can name the file rather than
// carry a value the mesh does not have.
directories := map[string]string{}
@@ -204,17 +282,7 @@ func (r Resolution) Declaration(with Rendering) ([]map[string]any, error) {
// Once, from every module's listens -- not per module. A module receiving only its own ports
// would write a rule set that closed every other module on the machine. Each module's per-node
// exposure settings override its listens' source first (novox/hq ADR 0046).
exposure := map[string]map[int]string{}
for _, m := range r.Modules {
e, err := Exposure(m, with.Settings[m.Module])
if err != nil {
return nil, err
}
if e != nil {
exposure[m.Module] = e
}
}
rules, err := r.Filtering(with.Generators, with.Ports, exposure)
rules, err := r.Rules(with)
if err != nil {
return nil, err
}
@@ -222,6 +290,13 @@ func (r Resolution) Declaration(with Rendering) ([]map[string]any, error) {
var out []map[string]any
for _, m := range r.Modules {
if with.Adopted && m.Filtering != nil {
// Nothing of a module that loads a filter, on an adopted node: its table would drop
// by default and hold accepts, and the found firewall stays in force. Every resource,
// not only the rule set — its service must not run, and a node returned to adopted
// stops it by the ordinary removal of what is no longer declared.
continue
}
resources := m.Resources
// What the mesh computes for this module goes FIRST, before the module's own resources.
@@ -285,45 +360,48 @@ func (r Resolution) Declaration(with Rendering) ([]map[string]any, error) {
"id": AccessID(a.Path), "type": "access", "path": a.Path, "mode": a.At(),
})
}
for _, to := range sortedKeys(m.Secrets) {
var found *Needed
for i, n := range r.Needs {
// **This module's need, not the provision's** (novox/hq 04-ISSUES/022). Matching
// on the name alone, every consumer of a provision took whichever credential
// happened to be last in the list — so on a node with two of them, one module
// would be given the other's password and fail to authenticate with a valid
// credential belonging to somebody else.
if n.Name == to && n.For == m.Module {
found = &r.Needs[i]
for _, to := range m.SecretRequirements() {
for _, file := range m.SecretFiles(to) {
var found *Needed
for i, n := range r.Needs {
// **This module's need, not the provision's** (novox/hq 04-ISSUES/022). Matching
// on the name alone, every consumer of a provision took whichever credential
// happened to be last in the list — so on a node with two of them, one module
// would be given the other's password and fail to authenticate with a valid
// credential belonging to somebody else. And this file's local name, where the
// module keeps several (ADR 0094).
if n.Name == to && n.For == m.Module && n.Local == file.Local {
found = &r.Needs[i]
}
}
if found != nil && found.ByRecord && found.Sealed == "" && !found.Manager {
// Answered by a record whose key has not been supplied since this consumer was
// put on it. **Refused, not skipped.** The mesh discarded the plaintext when the
// key was accepted and cannot seal another, so a machine that resolved cleanly
// would receive no file at all and fail at whatever tried to read it — which is
// the outcome ADR 0024 exists to avoid, arrived at politely.
//
// The manager holder is the one exception (novox/hq ADR 0050): an empty refresh token
// is a licence whose manager has not adopted one yet, a real waiting state rather than
// a lost key. It falls through to the skip below — its bound facts (carrying the
// manager's public key) are still delivered, which is what adoption needs to seal the
// first refresh token.
return nil, fmt.Errorf(
"%s on this machine uses the licence %q and no key has been sealed to it. "+
"The mesh cannot make one; supply it again with `licence key %s`",
m.Module, found.From, found.From)
}
if found == nil || found.Sealed == "" {
// Answered on this machine, or answered by a node the mesh could not seal to.
// Nothing to write either way, and writing an empty credential file would be
// worse than none: something would read it and fail authenticating.
continue
}
first = append(first, ownedBy(m.SecretsOwner, map[string]any{
"id": SecretID(SecretLocal(to, file.Local)), "type": "file", "path": file.Path,
"sealed": found.Sealed,
}))
}
if found != nil && found.ByRecord && found.Sealed == "" && !found.Manager {
// Answered by a record whose key has not been supplied since this consumer was
// put on it. **Refused, not skipped.** The mesh discarded the plaintext when the
// key was accepted and cannot seal another, so a machine that resolved cleanly
// would receive no file at all and fail at whatever tried to read it — which is
// the outcome ADR 0024 exists to avoid, arrived at politely.
//
// The manager holder is the one exception (novox/hq ADR 0050): an empty refresh token
// is a licence whose manager has not adopted one yet, a real waiting state rather than
// a lost key. It falls through to the skip below — its bound facts (carrying the
// manager's public key) are still delivered, which is what adoption needs to seal the
// first refresh token.
return nil, fmt.Errorf(
"%s on this machine uses the licence %q and no key has been sealed to it. "+
"The mesh cannot make one; supply it again with `licence key %s`",
m.Module, found.From, found.From)
}
if found == nil || found.Sealed == "" {
// Answered on this machine, or answered by a node the mesh could not seal to.
// Nothing to write either way, and writing an empty credential file would be
// worse than none: something would read it and fail authenticating.
continue
}
first = append(first, ownedBy(m.SecretsOwner, map[string]any{
"id": SecretID(to), "type": "file", "path": m.Secrets[to],
"sealed": found.Sealed,
}))
}
for _, to := range sortedKeys(m.Grants) {
for _, g := range with.Grants {
@@ -341,9 +419,11 @@ func (r Resolution) Declaration(with Rendering) ([]map[string]any, error) {
continue
}
first = append(first, map[string]any{
"id": GrantID(to, g.Consumer+"."+g.From),
// One file per holder — the consumer's module with its local name after it
// where it keeps several (ADR 0094); the lab found two files with one id.
"id": GrantID(to, g.Consumer+"."+holderAs(g.From, g.Local)),
"type": "file",
"path": grantPath(m.Grants[to], g.Consumer, g.From),
"path": grantPath(m.Grants[to], g.Consumer, holderAs(g.From, g.Local)),
"sealed": g.Sealed,
})
}
@@ -440,7 +520,7 @@ func (r Resolution) Declaration(with Rendering) ([]map[string]any, error) {
// And what its bindings say, for the half of a connection that is not secret.
known := knownFor(m, r.Needs, r.Node)
// And the machine underneath, which no binding of its own can tell it.
thisMachine := machineFacts(r)
thisMachine := machineFacts(r, with.Names)
// Which of this module's files carry a secret, for the rule that a container may not read
// one of them as its environment without saying so (ADR 0086, issue 041).
@@ -484,6 +564,11 @@ func (r Resolution) Declaration(with Rendering) ([]map[string]any, error) {
if err := portInto(copied, m.Module, m.Listens, with); err != nil {
return nil, err
}
// And where this machine put the foundation's servers, for the one module that
// reaches them by seat rather than by binding (novox/hq 04-ISSUES/102).
if err := seatInto(copied, m.Module, with); err != nil {
return nil, err
}
if err := machineInto(copied, thisMachine, m.Module); err != nil {
return nil, err
}
@@ -495,6 +580,11 @@ func (r Resolution) Declaration(with Rendering) ([]map[string]any, error) {
if err := built(copied, m.Module); err != nil {
return nil, err
}
// What the mesh built is kept by digest and path; the store's address is this
// network's now, composed here and never recorded (novox/hq 04-ISSUES/102).
if err := artifactsInto(copied, m.Module, with); err != nil {
return nil, err
}
publishedOn(copied, m.Module, with)
copied["id"] = m.Module + "." + fmt.Sprint(resource["id"])
// A service saying what it reflects names resources within its own module, so those
@@ -509,6 +599,12 @@ func (r Resolution) Declaration(with Rendering) ([]map[string]any, error) {
if renamed := reflectsRenamed(m.Module, resource["restart-on"]); renamed != nil {
copied["restart-on"] = renamed
}
// And what it is reloaded on, by the same rule (novox/hq ADR 0102): an id left
// unprefixed matches nothing, and the service is never reloaded.
if renamed := reflectsRenamed(m.Module, resource["reload-on"]); renamed != nil {
copied["reload-on"] = renamed
}
owner[fmt.Sprint(copied["id"])] = m.Module
out = append(out, copied)
}
@@ -516,18 +612,144 @@ func (r Resolution) Declaration(with Rendering) ([]map[string]any, error) {
// plane's; making a name resolve is the module's software. Emitted as ordinary files under
// this module's name, so they are applied, reported and removed exactly as anything else
// it declares.
given, err := FactsInto(m, r, with.Names)
given, err := FactsInto(m, r, with.Names, with.Machines, with.Suffix)
if err != nil {
return nil, err
}
for _, fact := range given {
fact["id"] = m.Module + "." + fmt.Sprint(fact["id"])
owner[fmt.Sprint(fact["id"])] = m.Module
out = append(out, fact)
}
}
if with.Adopted {
// First, before anything a module declares: what the mesh needs reachable, then its guard.
// The order a machine applies is the order written here.
ours := Openings(rules, with.Foundation, Published(out))
ours = append(ours, GuardResources(r.guarded(out, owner, rules, with))...)
out = append(ours, out...)
}
return out, nil
}
// guarded is what the mesh's guard refuses on an adopted node (novox/hq ADR 0103): derived, and
// for taken modules only.
//
// For each taken module, every machine port its containers publish that the filter would admit
// from the private network only — a published port is forwarded, not received, so a found
// firewall filtering only what it receives never sees it — together with the ports the module's
// manifest guards explicitly (the store's port, the broker's management port), wherever the
// machine put them. A port of a module assigned but not taken is not guarded: it may still be the
// predecessor's, serving the predecessor's other machines. The foundation's ports are admitted
// from everywhere and are never guarded.
func (r Resolution) guarded(out []map[string]any, owner map[string]string, rules []Rule,
with Rendering) []int {
meshOnly := map[int]bool{}
fromEverywhere := map[int]bool{}
for _, rule := range rules {
if rule.Protocol != "tcp" {
continue
}
switch rule.From {
case FromMesh:
meshOnly[rule.Port] = true
case FromEverywhere:
fromEverywhere[rule.Port] = true
}
}
for _, port := range with.Foundation {
delete(meshOnly, port)
fromEverywhere[port] = true
}
seen := map[int]bool{}
var ports []int
guard := func(at int) {
if !seen[at] {
seen[at] = true
ports = append(ports, at)
}
}
for _, m := range r.Modules {
if !with.Taken[m.Module] {
continue
}
var mine []map[string]any
for _, resource := range out {
if owner[fmt.Sprint(resource["id"])] == m.Module {
mine = append(mine, resource)
}
}
for outer := range Published(mine)["tcp"] {
if meshOnly[outer] {
guard(outer)
}
}
for _, want := range m.Guards {
at := with.machinePort(m.Module, want)
for _, resource := range mine {
if fmt.Sprint(resource["type"]) != "container" {
continue
}
listed, _ := resource["ports"].([]any)
for _, entry := range listed {
outer, inner, _, ok := mapping(fmt.Sprint(entry))
if ok && inner == want {
at = outer
}
}
}
// Not what this node is told to open to everyone: a per-node exposure setting that
// widens a guarded port is the operator saying so, and declaring an opening for it
// and a guard dropping it would be one statement refusing the other.
if !fromEverywhere[at] {
guard(at)
}
}
}
sort.Ints(ports)
return ports
}
// mapping reads a container's port mapping — `[address:]outer:inner[/protocol]`, the address
// possibly an IPv6 one in brackets — indexing from the end, so an address's own colons never
// shift the ports. Not ok for a short form or anything that is not a mapping.
func mapping(written string) (outer, inner int, address string, ok bool) {
written = strings.TrimSpace(written)
if cut := strings.LastIndex(written, "/"); cut >= 0 {
written = written[:cut]
}
parts := strings.Split(written, ":")
if len(parts) < 2 {
return 0, 0, "", false
}
inner, err := strconv.Atoi(parts[len(parts)-1])
if err != nil {
return 0, 0, "", false
}
outer, err = strconv.Atoi(parts[len(parts)-2])
if err != nil {
return 0, 0, "", false
}
return outer, inner, strings.Join(parts[:len(parts)-2], ":"), true
}
// Rules is the rule set this node's filter is derived from: every module's listens, what was
// computed for this machine, and each module's per-node exposure. The same answer whether the node
// is adopted or converged — the one loads it as a filter, the other declares it as openings.
func (r Resolution) Rules(with Rendering) ([]Rule, error) {
exposure := map[string]map[int]string{}
for _, m := range r.Modules {
e, err := Exposure(m, with.Settings[m.Module])
if err != nil {
return nil, err
}
if e != nil {
exposure[m.Module] = e
}
}
return r.Filtering(with.Generators, with.Ports, exposure)
}
// Contribution is one module telling the answer to a requirement what it needs from it.
type Contribution struct {
// From is the module that said it, so the provider and a person reading the file can tell
@@ -613,6 +835,15 @@ func grantPath(directory, consumer, module string) string {
return strings.TrimRight(directory, "/") + "/" + consumer + "." + module + ".secret"
}
// holderAs is a consumer's name at the provider with a local name after it, where it keeps several
// credentials for one provision (ADR 0094); the name alone otherwise.
func holderAs(as, local string) string {
if local == "" {
return as
}
return as + "_" + local
}
// contributions collects what every module in this set contributes, by requirement.
//
// Ordered by contributing module, because the result becomes a file on a machine and a file whose
@@ -634,7 +865,10 @@ func (r Resolution) contributions(settings SettingsBy, grants []Grant,
if sorted[i].Consumer != sorted[j].Consumer {
return sorted[i].Consumer < sorted[j].Consumer
}
return sorted[i].From < sorted[j].From
if sorted[i].From != sorted[j].From {
return sorted[i].From < sorted[j].From
}
return sorted[i].Local < sorted[j].Local
})
// A consumer already carried by the grants loop, keyed (provision, module). When provider and
// consumer are co-located, `grantsFor` enumerates the same-node consumer too, so without this the
@@ -649,8 +883,11 @@ func (r Resolution) contributions(settings SettingsBy, grants []Grant,
}
out[g.Provision] = append(out[g.Provision], Contribution{
From: g.From, Node: g.Consumer, At: g.At, Values: g.Values,
As: ConsumerIdentity(g.Consumer, IdentitySource(g.Slug, g.From)),
Secret: grantPath(directories[g.Provision], g.Consumer, g.From),
// One holder per local name: the identity the consumer is known by, and the local name
// after it where the module keeps several (ADR 0094). Not a login any backend checks —
// a secret is not a login — so the identity limit does not apply to the suffix.
As: holderAs(ConsumerIdentity(g.Consumer, IdentitySource(g.Slug, g.From)), g.Local),
Secret: grantPath(directories[g.Provision], g.Consumer, holderAs(g.From, g.Local)),
})
if granted[g.Provision] == nil {
granted[g.Provision] = map[string]bool{}
@@ -677,6 +914,21 @@ func (r Resolution) contributions(settings SettingsBy, grants []Grant,
composeName(values, r.PublicDomain)
out[to] = append(out[to], Contribution{From: m.Module, Values: values})
}
// Several contributions to one requirement (ADR 0094's sibling for `contributes`): an
// object store's data API and its console are two different public names from one module,
// not one. Never in `granted` — a route names a host, not a credential — so every local
// name always reaches the provider from here.
for _, to := range sortedKeys(m.ContributesMany) {
for _, local := range sortedKeys(m.ContributesMany[to]) {
values, err := settle(m.ContributesMany[to][local], settings[m.Module], nil,
m.Module+" contributing "+local+" to "+to)
if err != nil {
return nil, fmt.Errorf("%s contributing %s to %s: %w", m.Module, local, to, err)
}
composeName(values, r.PublicDomain)
out[to] = append(out[to], Contribution{From: m.Module, Values: values})
}
}
}
return out, nil
}
@@ -768,6 +1020,8 @@ type Kept struct {
Sealed string `json:"sealed"`
Key string `json:"key"`
MadeAt time.Time `json:"made-at"`
// Local is the credential's name inside the consumer where it holds several (ADR 0094).
Local string `json:"local,omitempty"`
}
// KeptExport is what a person keeps beside the operator key, and what a vault keeps on its disk:
@@ -866,17 +1120,34 @@ func boundFile(n Needed, path, as string) (map[string]any, error) {
// arrangement refused is the ordinary one. A node running eight services against one database is
// not an edge case; it is what a machine looks like. Now each consumer has its own credential and
// there is nothing left to refuse.
//
// **One credential, even where a module contributes several times.** A module may answer one
// requirement more than once (ADR 0094's sibling for `contributes`) — an object store's data API
// and its console are two different names, not one. There is still only one `Needed` for it, one
// credential minted, one grant to settle: a pair credential is not a place to put a label or a
// port. So where several of this module's contributions reach the same requirement, none of them
// is "the" value — settling to the first, arbitrarily, would hand the grant one contribution's
// values under a credential the OTHER contribution's consumer never sees, and would collide with
// that contribution's own entry from contributions() besides. Empty values, still granted: the
// module asked, gets its credential, and each named contribution reaches the provider on its own.
func (r Resolution) ContributionsFrom(requirement, module string, settings SettingsBy) (
map[string]any, bool, error) {
all, err := r.contributions(settings, nil, nil)
if err != nil {
return nil, false, err
}
var mine []map[string]any
for _, g := range all[requirement] {
if g.From == module {
return g.Values, true, nil
mine = append(mine, g.Values)
}
}
if len(mine) == 1 {
return mine[0], true, nil
}
if len(mine) > 1 {
return map[string]any{}, true, nil
}
// It contributes no payload — but a require-only consumer of a parameterless provision (one whose
// `serves` names no consumer-supplied key: `redis-cache`, `amqp`) still ASKS for it and must be
// granted a credential. Keying "asks" on contributions alone marked those grants withdrawn
@@ -1065,7 +1336,11 @@ func publishedOn(resource map[string]any, module string, with Rendering) {
for _, entry := range listed {
written := fmt.Sprint(entry)
if strings.Contains(written, ":") {
out = append(out, written)
// Written the long way, and left alone — unless this node was given a machine port for
// it (novox/hq ADR 0100): the foundation's ports are the node's, and a manifest's
// number is only the default. The outer port only; an address and the software's
// port stay as written.
out = append(out, givenOuter(written, with.Given[module]))
continue
}
wanted, err := strconv.Atoi(strings.TrimSpace(written))
@@ -1080,6 +1355,43 @@ func publishedOn(resource map[string]any, module string, with Rendering) {
resource["ports"] = out
}
// givenOuter rewrites the machine side of a long-form mapping to the port this node was given for
// it, when it was given one.
//
// **Under either of the mapping's names.** A node gives a port by the number the module names it
// by, and a module publishing `"2222:22"` may say it listens on 22 or on 2222 — GivenPorts accepts
// both and answers to both, so looking the machine side up first and the container's port second
// finds the same number either way. Read only by the container's port, this moved nothing for the
// module that declares the machine side, and the setting was refused before it got here.
func givenOuter(written string, given map[int]int) string {
if len(given) == 0 {
return written
}
mapping, protocol := written, ""
if cut := strings.LastIndex(written, "/"); cut >= 0 {
mapping, protocol = written[:cut], written[cut:]
}
parts := strings.Split(mapping, ":")
if len(parts) < 2 {
return written
}
inner, err := strconv.Atoi(strings.TrimSpace(parts[len(parts)-1]))
if err != nil {
return written
}
at, ok := given[inner]
if outer, err := strconv.Atoi(strings.TrimSpace(parts[len(parts)-2])); err == nil {
if machine, named := given[outer]; named {
at, ok = machine, true
}
}
if !ok {
return written
}
parts[len(parts)-2] = strconv.Itoa(at)
return strings.Join(parts, ":") + protocol
}
// ServedOn is what a provider tells a consumer, with the port that machine actually uses.
//
// **The module writes the port once, in `listens`** (novox/hq ADR 0038). It used to write it three
+49 -10
View File
@@ -36,16 +36,23 @@ const (
// **A closed list.** A module asking for a fact the mesh does not have is asking for a file nobody
// will write, and finding that out on a machine — as a daemon that starts, reads nothing, and
// answers no queries — is worse than being told where the manifest is.
var facts = map[string]func(Resolution, map[string]string) string{
FactNodeNames: nodeNames,
FactNodeZones: nodeZones,
// A fact is written from the names it is about. `every` is every name the mesh serves — machines
// and the names it was told to route; `machines` is only the machines. A fact takes the set it is
// true of, and the two must not be confused (novox/hq 04-ISSUES/111).
var facts = map[string]func(r Resolution, every, machines map[string]string, suffix string) string{
FactNodeNames: func(r Resolution, every, _ map[string]string, suffix string) string {
return nodeNames(r, every, suffix)
},
FactNodeZones: func(r Resolution, _, machines map[string]string, suffix string) string {
return nodeZones(r, machines, suffix)
},
}
// FactsInto renders the facts a module asked for, as files it will be given.
//
// The module owns everything after the file exists: loading it, restarting on it, what a resolver
// does with it. This only puts it there.
func FactsInto(m Manifest, r Resolution, addresses map[string]string) ([]map[string]any, error) {
func FactsInto(m Manifest, r Resolution, addresses, machines map[string]string, suffix string) ([]map[string]any, error) {
if len(m.Facts) == 0 {
return nil, nil
}
@@ -70,7 +77,7 @@ func FactsInto(m Manifest, r Resolution, addresses map[string]string) ([]map[str
}
out = append(out, map[string]any{
"id": "fact-" + name, "type": "file", "path": path, "mode": "0644",
"content": write(r, addresses),
"content": write(r, addresses, machines, suffix),
})
}
return out, nil
@@ -92,7 +99,7 @@ func spokenFacts() string {
// and does not yet know where it is, which is the ordinary state between adding a machine and it
// joining. Writing the name anyway would give a name that resolves to nothing, and a connection to
// that hangs; leaving it out fails at once and says the name is unknown.
func nodeNames(r Resolution, addresses map[string]string) string {
func nodeNames(r Resolution, addresses map[string]string, suffix string) string {
var b strings.Builder
b.WriteString("# Generated by the mesh. Do not edit — this file is replaced whenever a machine\n")
b.WriteString("# joins or leaves, and an edit would survive until then and vanish.\n\n")
@@ -106,10 +113,11 @@ func nodeNames(r Resolution, addresses map[string]string) string {
b.WriteString("\n")
for _, name := range sortedNames(addresses) {
at := addresses[name]
internal, bare := meshName(name, suffix)
// Its mesh name resolves to its address on the private network rather than to loopback,
// so a service binding the name it was given stays reachable from everywhere else.
fmt.Fprintf(&b, "%s\t%s.internal\t%s", at, name, name)
if name == r.Node {
fmt.Fprintf(&b, "%s\t%s\t%s", at, internal, bare)
if bare == r.Node {
b.WriteString("\t# this machine")
}
b.WriteString("\n")
@@ -121,16 +129,47 @@ func nodeNames(r Resolution, addresses map[string]string) string {
//
// `*.homer.internal` is homer, which is the whole rule: if homer is at an address, so is anything
// homer serves. A module wanting this runs the resolver; the mesh only says what is true.
func nodeZones(_ Resolution, addresses map[string]string) string {
//
// **And the suffix itself, as a local domain.** A resolver that forwards what it cannot answer
// would otherwise send a mesh name it does not know — a machine that left, a typo — to a public
// resolver, which is a leak of the mesh's names for no answer. `local=` keeps everything under the
// suffix here: answered from the lines below or refused. Written in this file rather than in the
// resolver's own configuration because the suffix is the mesh's choice (the operator may have
// picked another) and this file is the one place the mesh writes what it chose.
func nodeZones(_ Resolution, addresses map[string]string, suffix string) string {
var b strings.Builder
b.WriteString("# Generated by the mesh. Do not edit — this file is replaced whenever a machine\n")
b.WriteString("# joins or leaves, and an edit would survive until then and vanish.\n\n")
fmt.Fprintf(&b, "local=/%s/\n", strings.TrimPrefix(suffixOr(suffix), "."))
for _, name := range sortedNames(addresses) {
fmt.Fprintf(&b, "address=/%s.internal/%s\n", name, addresses[name])
internal, _ := meshName(name, suffix)
fmt.Fprintf(&b, "address=/%s/%s\n", internal, addresses[name])
}
return b.String()
}
// meshName is a machine's internal name and its bare one, from either. The control plane keys
// the names it hands a resolution by the internal name (`homer.internal`), the same map a
// container gets as its hosts; a caller that keys by the bare name gets the same answer. The
// suffix is the one the control plane composed those names with, handed down rather than written
// here a second time — the alternative was `homer.internal.internal` on every machine.
func meshName(name, suffix string) (internal, bare string) {
dotted := "." + strings.TrimPrefix(suffixOr(suffix), ".")
if strings.HasSuffix(name, dotted) {
return name, strings.TrimSuffix(name, dotted)
}
return name + dotted, name
}
// suffixOr is the suffix given, or the one the mesh composes names with when none was handed down.
// The one place the default is written in this file, so a fact and a name cannot disagree about it.
func suffixOr(suffix string) string {
if suffix == "" {
return "internal"
}
return suffix
}
func sortedNames(addresses map[string]string) []string {
out := make([]string, 0, len(addresses))
for name, at := range addresses {
+100 -9
View File
@@ -5,12 +5,17 @@ import (
"testing"
)
var threeMachines = map[string]string{"homer": "10.42.0.1", "marge": "10.42.0.2", "bart": ""}
// Keyed by the internal name, as the control plane hands them (issue 079).
var threeMachines = map[string]string{"homer.internal": "10.42.0.1", "marge.internal": "10.42.0.2", "bart.internal": ""}
// **`*.homer.internal` is homer. That is the whole rule.**
// **`*.homer.internal` is homer. That is the whole rule.** And the suffix itself is local: a
// resolver that forwards what it cannot answer must not send a mesh name it does not know — a
// machine that left, a typo — to a public resolver (hal dnsmasq-app conversion, novox/hq
// 08-connectivity).
func TestEveryMachineIsAWildcardUnderItsOwnName(t *testing.T) {
out := nodeZones(Resolution{Node: "homer"}, threeMachines)
out := nodeZones(Resolution{Node: "homer"}, threeMachines, "")
for _, want := range []string{
"local=/internal/",
"address=/homer.internal/10.42.0.1",
"address=/marge.internal/10.42.0.2",
} {
@@ -27,8 +32,8 @@ func TestEveryMachineIsAWildcardUnderItsOwnName(t *testing.T) {
// unknown, which is a thing somebody can act on.
func TestAMachineWithNoAddressIsNotNamed(t *testing.T) {
for _, out := range []string{
nodeNames(Resolution{Node: "homer"}, threeMachines),
nodeZones(Resolution{Node: "homer"}, threeMachines),
nodeNames(Resolution{Node: "homer"}, threeMachines, ""),
nodeZones(Resolution{Node: "homer"}, threeMachines, ""),
} {
if strings.Contains(out, "bart") {
t.Fatalf("a machine with no address was named, so its name resolves to nothing:\n%s", out)
@@ -39,7 +44,7 @@ func TestAMachineWithNoAddressIsNotNamed(t *testing.T) {
// A machine's own mesh name points at its address on the private network, not at loopback — or a
// service binding the name it was given is unreachable from everywhere else.
func TestAMachinesOwnNameIsItsMeshAddress(t *testing.T) {
out := nodeNames(Resolution{Node: "homer"}, threeMachines)
out := nodeNames(Resolution{Node: "homer"}, threeMachines, "")
var line string
for _, l := range strings.Split(out, "\n") {
if strings.Contains(l, "homer.internal") {
@@ -58,7 +63,7 @@ func TestAMachinesOwnNameIsItsMeshAddress(t *testing.T) {
// A module says where it wants a fact, and is given a file.
func TestAModuleIsGivenTheFactsItAskedFor(t *testing.T) {
m := Manifest{Module: "dnsmasq", Facts: map[string]string{FactNodeZones: "/etc/mesh/zones.conf"}}
given, err := FactsInto(m, Resolution{Node: "homer"}, threeMachines)
given, err := FactsInto(m, Resolution{Node: "homer"}, threeMachines, threeMachines, "")
if err != nil {
t.Fatal(err)
}
@@ -77,7 +82,7 @@ func TestAModuleIsGivenTheFactsItAskedFor(t *testing.T) {
// starts, reads a file nobody wrote, and answers no queries is a much worse way to find out.
func TestAskingForAFactTheMeshDoesNotHaveIsRefused(t *testing.T) {
m := Manifest{Module: "dnsmasq", Facts: map[string]string{"the-weather": "/etc/weather"}}
_, err := FactsInto(m, Resolution{}, nil)
_, err := FactsInto(m, Resolution{}, nil, nil, "")
if err == nil {
t.Fatal("a module asked for something nobody computes and was given nothing, silently")
}
@@ -91,7 +96,93 @@ func TestAskingForAFactTheMeshDoesNotHaveIsRefused(t *testing.T) {
// And a relative path is refused, or a module decides where the mesh writes on a machine.
func TestAFactMustBeAskedForAtAnAbsolutePath(t *testing.T) {
m := Manifest{Module: "dnsmasq", Facts: map[string]string{FactNodeNames: "etc/hosts"}}
if _, err := FactsInto(m, Resolution{}, nil); err == nil {
if _, err := FactsInto(m, Resolution{}, nil, nil, ""); err == nil {
t.Fatal("a relative path was accepted")
}
}
// **The names the control plane hands a resolution are already internal names** — `homer.internal`,
// the same map every container gets as its hosts. Appending the suffix again wrote
// `homer.internal.internal` into every hosts file and every resolver's zones, and the large mesh
// bed's name test was the first to read it back. Either key gives the same files.
func TestNamesKeyedByInternalNameAreNotSuffixedTwice(t *testing.T) {
internal := map[string]string{"homer.internal": "10.42.0.1", "marge.internal": "10.42.0.2"}
bare := map[string]string{"homer": "10.42.0.1", "marge": "10.42.0.2"}
if a, b := nodeZones(Resolution{Node: "homer"}, internal, ""), nodeZones(Resolution{Node: "homer"}, bare, ""); a != b {
t.Fatalf("the zones differ by how the names were keyed:\n%s\n---\n%s", a, b)
}
if a, b := nodeNames(Resolution{Node: "homer"}, internal, ""), nodeNames(Resolution{Node: "homer"}, bare, ""); a != b {
t.Fatalf("the hosts differ by how the names were keyed:\n%s\n---\n%s", a, b)
}
zones := nodeZones(Resolution{Node: "homer"}, internal, "")
if strings.Contains(zones, "internal.internal") || !strings.Contains(zones, "address=/homer.internal/10.42.0.1") {
t.Fatalf("the zones carry a doubled suffix or miss the name:\n%s", zones)
}
hosts := nodeNames(Resolution{Node: "homer"}, internal, "")
if !strings.Contains(hosts, "10.42.0.1\thomer.internal\thomer\t# this machine") {
t.Fatalf("the hosts line for the machine itself is not name, bare name and the mark:\n%s", hosts)
}
}
// The suffix the control plane composed the names with is the one the facts write — an operator
// who chose another does not get `.internal` appended to it.
func TestTheFactsWriteTheSuffixTheNamesWereComposedWith(t *testing.T) {
names := map[string]string{"homer.lan": "10.42.0.1"}
zones := nodeZones(Resolution{Node: "homer"}, names, "lan")
if !strings.Contains(zones, "address=/homer.lan/10.42.0.1") || strings.Contains(zones, "internal") {
t.Fatalf("the zones do not carry the operator's suffix as given:\n%s", zones)
}
if !strings.Contains(zones, "local=/lan/") {
t.Fatalf("the local domain is not the operator's suffix, so its names would leak upstream:\n%s", zones)
}
hosts := nodeNames(Resolution{Node: "homer"}, names, "lan")
if !strings.Contains(hosts, "10.42.0.1\thomer.lan\thomer\t# this machine") {
t.Fatalf("the hosts line does not carry the operator's suffix as given:\n%s", hosts)
}
}
// novox/hq 04-ISSUES/111: the map the control plane hands a resolution holds every name the mesh
// serves — the machines, and the names it was told to route to whichever machine serves them. A
// container's hosts wants all of it. A resolver's zones want only the machines: told the mesh's
// suffix is its own, it answers authoritatively for everything under it and forwards nothing, so a
// routed name written there with the suffix appended is a name nobody will ever ask for, standing
// beside the machines and looking as real.
func TestTheResolverIsToldTheMachinesAndNotTheNamesTheMeshMerelyServes(t *testing.T) {
machines := map[string]string{"homer.internal": "10.42.0.1", "marge.internal": "10.42.0.2"}
every := map[string]string{
"homer.internal": "10.42.0.1", "marge.internal": "10.42.0.2",
"drive.example.test": "10.42.0.1", "git.example.test": "10.42.0.2",
}
m := Manifest{Module: "resolver", Facts: map[string]string{
FactNodeZones: "/etc/zones.conf", FactNodeNames: "/etc/hosts",
}}
given, err := FactsInto(m, Resolution{Node: "homer"}, every, machines, "")
if err != nil {
t.Fatal(err)
}
by := map[string]string{}
for _, f := range given {
by[f["path"].(string)] = f["content"].(string)
}
zones := by["/etc/zones.conf"]
for _, machine := range []string{"address=/homer.internal/10.42.0.1", "address=/marge.internal/10.42.0.2"} {
if !strings.Contains(zones, machine) {
t.Fatalf("the resolver was not told %q:\n%s", machine, zones)
}
}
for _, served := range []string{"drive.example.test", "git.example.test"} {
if strings.Contains(zones, served) {
t.Fatalf("the resolver was told %q, a name the mesh serves rather than a machine:\n%s", served, zones)
}
}
// And the hosts file is the other way about: every name, so a container reaching a routed name
// finds the machine serving it.
hosts := by["/etc/hosts"]
for _, name := range []string{"homer.internal", "drive.example.test", "git.example.test"} {
if !strings.Contains(hosts, name) {
t.Fatalf("a container would not resolve %q from its hosts:\n%s", name, hosts)
}
}
}
+207
View File
@@ -2,6 +2,7 @@ package catalogue
import (
"fmt"
"slices"
"sort"
"strconv"
"strings"
@@ -457,3 +458,209 @@ func byFamily(addresses []string) (four []string, six []string) {
}
return four, six
}
// PortsSetting is the settings key that gives a module's port a machine port on one node (novox/hq
// ADR 0100):
//
// {"ports": {"5432": 5433}}
//
// puts what the software calls 5432 on the machine's 5433. The foundation's ports are the node's:
// every one is an input to genesis, checked free there, and becomes that node's setting for the
// foundation's modules — the catalogue's numbers are only their defaults. Keyed by the port the
// software uses, like expose; the value is where the machine puts it.
const PortsSetting = "ports"
// MeshWideLayer is what a layer set for the whole mesh is called, rather than for one node.
const MeshWideLayer = "the mesh"
// GivenPorts reads a module's given machine ports from its settings: software port → machine port.
//
// Refused from a mesh-wide layer — a port is a fact about one machine, and one number for every
// machine is the collision this exists to avoid — and for a port the module's containers do not
// publish.
//
// **Only a published port is the mesh's to move** (novox/hq ADR 0038). A container's mapping is
// what translates; a module binding the machine's network directly binds the number its software
// was configured with, and moving that number would put it in the filter, in the openings and in
// what consumers are told while the software still listens on the old one — a port that reads as
// moved and is not.
//
// **Either name of a mapping names it; the module's own name answers.** A short form publishes one
// number, which is the container's port and the machine's at once. A mapping written the long way
// — `"2222:22"` — has two, and a module reasonably declares it listens on either: the port its
// software uses, or the port the machine already serves on. A setting may name either end, because
// both are true of the same mapping. The answer comes back under the end the **module** names in
// its `listens`, which is the number every reader of this map holds: the ports map, the filter, the
// openings, what a consumer is told, and the mapping the runtime is handed. Keyed one way and read
// the other, the setting moved the container's mapping and nothing else — a firewall, a set of
// openings and a consumer all pointing at a port the software had left.
//
// One entry per mapping, never two. A second key for the same answer is not a convenience: where
// two mappings share a number, it is an entry one of them writes over the other's, and the reader
// that finds the survivor disagrees with the reader that recomputes it.
func GivenPorts(m Manifest, layers []Layer) (map[int]int, error) {
// Every name a setting may use, and the mapping it names.
names := map[int][]publishing{}
for _, p := range publishedPorts(m) {
names[p.inner] = append(names[p.inner], p)
if p.machine != p.inner {
names[p.machine] = append(names[p.machine], p)
}
}
// And the names the module itself uses. A mapping's answer is filed under these, because they
// are what every reader asks for; a mapping the module names at neither end is filed under its
// machine side, where nothing looks, which is correct — nothing serves it.
declares := map[int]bool{}
for _, l := range m.Listens {
declares[l.Port] = true
}
chose := map[int]int{} // the port a setting named → the machine port it gave it
out := map[int]int{} // the port the module names → the machine port it is on
by := map[int]int{} // and which of the setting's ports put it there, for the refusal
for _, layer := range layers {
raw, ok := layer.Values[PortsSetting]
if !ok {
continue
}
if layer.From == MeshWideLayer {
return nil, fmt.Errorf("%s: %s is given per node — a port is a fact about one "+
"machine; set it with --node", m.Module, PortsSetting)
}
entries, ok := raw.(map[string]any)
if !ok {
return nil, fmt.Errorf("%s: %s is a { port: machine-port } map, and %q set it to "+
"something else", m.Module, PortsSetting, layer.From)
}
for portText, value := range entries {
port, err := strconv.Atoi(portText)
if err != nil {
return nil, fmt.Errorf("%s gives %q a port, which is not a port", m.Module, portText)
}
publishes, known := names[port]
if !known {
return nil, fmt.Errorf("%s gives port %d a machine port, and no container of its "+
"publishes %d — the mesh cannot move a port the module does not publish; the "+
"software would go on listening where it was told to", m.Module, port, port)
}
if err := oneMapping(m.Module, port, publishes); err != nil {
return nil, err
}
at, ok := asPort(value)
if !ok || at < 1 || at > 65535 {
return nil, fmt.Errorf("%s gives port %d the machine port %v, which is not a port",
m.Module, port, value)
}
if at == SSHPort {
return nil, fmt.Errorf("%s gives port %d the machine port %d, which is ssh's — the "+
"one port a machine may never lose", m.Module, port, at)
}
chose[port] = at
}
}
// One holder per machine port, within the module too.
holder := map[int]int{}
for _, port := range sortedPorts(chose) {
at := chose[port]
if other, twice := holder[at]; twice {
return nil, fmt.Errorf("%s gives machine port %d to both its %d and its %d", m.Module,
at, min(port, other), max(port, other))
}
holder[at] = port
}
// Filed under the module's own names for the mapping — every one it uses, so a module that
// says it listens on both ends is answered at both, and under the machine side when it names
// neither.
for _, port := range sortedPorts(chose) {
at, mapping := chose[port], names[port][0]
keys := []int{}
for _, end := range []int{mapping.machine, mapping.inner} {
if declares[end] && !slices.Contains(keys, end) {
keys = append(keys, end)
}
}
if len(keys) == 0 {
keys = []int{mapping.machine}
}
for _, key := range keys {
// The key must name this mapping and no other, or the entry is one mapping's answer
// standing where another's is read.
if err := oneMapping(m.Module, key, names[key]); err != nil {
return nil, err
}
// And one machine port per mapping: `{"22": 222, "2222": 300}` is two numbers for the
// one thing the machine publishes, and neither is more right.
if was, twice := out[key]; twice && was != at {
return nil, fmt.Errorf("%s gives %s the machine ports %d and %d — its %d and its "+
"%d are the two ends of one mapping, and it is published once", m.Module,
mapping, min(was, at), max(was, at), min(by[key], port), max(by[key], port))
}
out[key], by[key] = at, port
}
}
if len(out) == 0 {
return nil, nil
}
return out, nil
}
// oneMapping refuses a number that names two of a module's mappings — `"22"` beside `"2222:22"`,
// say, or `"4001:80"` beside `"4002:80"` read by their shared `80`. Refused rather than picked:
// moving one of them and leaving the other is a mapping the operator did not ask for and cannot
// see, and the two readings differ.
func oneMapping(module string, port int, publishes []publishing) error {
for _, other := range publishes[1:] {
if other != publishes[0] {
return fmt.Errorf("%s gives port %d a machine port, and its containers publish %d "+
"twice — as %s and as %s; which one to move is not said", module, port, port,
publishes[0], other)
}
}
return nil
}
// publishing is one mapping a module's container writes: the port the container itself uses, and
// the machine port the manifest put it on — the same number for a short form, which the runtime
// publishes on the port it names.
type publishing struct{ machine, inner int }
func (p publishing) String() string {
if p.machine == p.inner {
return strconv.Itoa(p.inner)
}
return fmt.Sprintf("%d:%d", p.machine, p.inner)
}
// publishedPorts is every mapping a module's containers publish, whichever form it is written in.
func publishedPorts(m Manifest) []publishing {
var out []publishing
for _, r := range m.Resources {
if fmt.Sprint(r["type"]) != "container" {
continue
}
listed, _ := r["ports"].([]any)
for _, entry := range listed {
written := strings.TrimSpace(fmt.Sprint(entry))
if machine, inner, _, ok := mapping(written); ok {
out = append(out, publishing{machine: machine, inner: inner})
continue
}
if cut := strings.LastIndex(written, "/"); cut >= 0 {
written = written[:cut]
}
if n, err := strconv.Atoi(strings.TrimSpace(written)); err == nil {
out = append(out, publishing{machine: n, inner: n})
}
}
}
return out
}
// sortedPorts is a settings map's ports in order, so a refusal reads the same on every run.
func sortedPorts(of map[int]int) []int {
out := make([]int, 0, len(of))
for port := range of {
out = append(out, port)
}
sort.Ints(out)
return out
}
@@ -0,0 +1,305 @@
package catalogue
import (
"encoding/json"
"fmt"
"os"
"reflect"
"strings"
"testing"
)
// The catalogue's foundation modules as they are, parsed by the real parser (novox/hq ADR 0100):
// the store and the broker say which of their ports the mesh guards on an adopted node, and the
// filter module loads its table through a unit of its own whose stop deletes only that table.
func catalogueManifest(t *testing.T, module string) Manifest {
t.Helper()
raw, err := os.ReadFile("../../../mesh-catalog/modules/" + module + "/module.json")
if err != nil {
t.Skipf("the catalogue is not beside this checkout: %v", err)
}
m, err := ParseManifest(raw)
if err != nil {
t.Fatalf("%s does not parse:\n%v", module, err)
}
return m
}
func TestTheStoreAndTheBrokerSayWhatTheMeshGuards(t *testing.T) {
if got := catalogueManifest(t, "postgres").Guards; !reflect.DeepEqual(got, []int{5432}) {
t.Errorf("postgres guards %v; the store's port must be refused from outside", got)
}
if got := catalogueManifest(t, "lavinmq").Guards; !reflect.DeepEqual(got, []int{15672}) {
t.Errorf("lavinmq guards %v; the management port must be refused from outside", got)
}
}
func TestTheFilterModuleNeverFlushesTheRuleset(t *testing.T) {
m := catalogueManifest(t, "nftables")
var unit, stock, load map[string]any
for _, r := range m.Resources {
switch r["id"] {
case "unit":
unit = r
case "stock-unit-stop":
stock = r
case "load":
load = r
}
}
if load == nil || load["unit"] != "mesh-filter.service" {
t.Fatalf("the filter is not loaded by its own unit: %v", load)
}
content, _ := unit["content"].(string)
if unit == nil || unit["path"] != "/etc/systemd/system/mesh-filter.service" {
t.Fatalf("the filter's unit is not written: %v", unit)
}
if strings.Contains(content, "flush") {
t.Fatalf("stopping the filter flushes the whole ruleset — the runtime's and the found "+
"firewall's with it:\n%s", content)
}
if !strings.Contains(content, "ExecStop=nft delete table inet mesh\n") ||
!strings.Contains(content, "ExecStart=nft -f "+m.Filtering.Into+"\n") {
t.Fatalf("the unit does not load the computed rule set and delete only its own table:\n%s",
content)
}
// A node converged before the filter had its own unit still has the stock nftables.service
// enabled, whose stop flushes the whole ruleset: a drop-in makes it delete only the mesh's
// table, and the load is restarted on it so the host reloads units and the drop-in is read.
if stock == nil || stock["path"] != "/etc/systemd/system/nftables.service.d/mesh.conf" ||
!strings.HasSuffix(fmt.Sprint(stock["content"]),
"[Service]\nExecStop=\nExecStop=nft delete table inet mesh\n") {
t.Fatalf("the stock unit's stop is not replaced with deleting the mesh's table: %v", stock)
}
// A changed rule set is RELOADED — ExecReload replaces the table in one `nft -f`, so the node
// is never unfiltered — and only the units themselves restart it, which is the one change a
// reload cannot carry.
if !reflect.DeepEqual(load["reload-on"], []any{"filtering"}) {
t.Fatalf("the filter is restarted rather than reloaded when its rules change, leaving the "+
"node unfiltered in between: %v", load)
}
if !reflect.DeepEqual(load["restart-on"], []any{"unit", "stock-unit-stop"}) {
t.Fatalf("the filter is not restarted when its unit or the stock unit's drop-in changes: %v",
load["restart-on"])
}
}
// The package registry's port is the node's, like every other foundation port (novox/hq
// 04-ISSUES/085, ADR 0100). Two halves, because the forge is reached two ways: through what the
// module that serves it says it serves, and — for the genesis window, before any module provides
// `package-registry` at all — through the one binding the builder carries instead of resolving.
func TestTheForgesPortIsGivenLikeAnyOtherProvidersPort(t *testing.T) {
forge := catalogueManifest(t, "gitea")
// The catalogue's number is a default and the node's setting moves it.
given, err := GivenPorts(forge, []Layer{{From: "anchor",
Values: map[string]any{PortsSetting: map[string]any{"3000": float64(3100)}}}})
if err != nil {
t.Fatalf("the forge's port cannot be given on a node: %v", err)
}
if given[3000] != 3100 {
t.Fatalf("the forge was given %v", given)
}
// And every consumer of the package registry is told where the machine actually put it,
// because that is read from what the forge serves rather than written in the consumer.
if got := ServedOn(forge, "package-registry", given)["port"]; got != 3100 {
t.Errorf("the package registry is served on %v, not the port this node gave it", got)
}
if got := ServedOn(forge, "package-registry", nil)["port"]; got != float64(3000) {
t.Errorf("without a setting the forge serves %v, not the catalogue's port", got)
}
}
// bindingIn is the package binding the builder carries, as the machine would receive it.
func bindingIn(t *testing.T, m Manifest, layers []Layer) map[string]any {
t.Helper()
for _, r := range m.Resources {
if fmt.Sprint(r["id"]) != "package-binding" {
continue
}
settled, err := ApplySettings(r, layers)
if err != nil {
t.Fatalf("the builder's package binding refused %v: %v", layers, err)
}
if settled["merge"] != nil || settled["protected"] != nil {
t.Fatal("the host would be sent fields it does not know")
}
var out map[string]any
if err := json.Unmarshal([]byte(fmt.Sprint(settled["content"])), &out); err != nil {
t.Fatalf("the builder's package binding is not a binding: %v", err)
}
return out
}
t.Fatal("the builder carries no package binding")
return nil
}
func TestTheBuildersCarriedPackageBindingTakesThePortFromTheNode(t *testing.T) {
builder := catalogueManifest(t, "builder")
// Nothing set: the catalogue's own number, which is what a mesh raised on the defaults uses.
serves := bindingIn(t, builder, nil)["serves"].(map[string]any)
if serves["port"] != float64(3000) {
t.Fatalf("the builder's binding defaults to %v", serves["port"])
}
// Given a port, the binding dials it — and the rest of what the forge serves survives, because
// a setting is merged into the module's own values rather than replacing them.
moved := bindingIn(t, builder, []Layer{{From: "anchor",
Values: map[string]any{"serves": map[string]any{"port": float64(3100)}}}})
got := moved["serves"].(map[string]any)
if got["port"] != float64(3100) {
t.Errorf("the builder dials %v, not the port this node gave the package registry", got["port"])
}
if got["scheme"] != "http" || got["npm-path"] != "/api/packages/novox/npm/" {
t.Errorf("setting the port lost the rest of what the forge serves: %v", got)
}
if moved["as"] != "mesh-builder" || moved["from"] != "gitea" {
t.Errorf("setting the port changed who the binding is with: %v", moved)
}
}
// The two halves are one number. The builder carries a binding because at genesis nothing provides
// `package-registry` to resolve one from; the day the forge is a module, the same consumer is told
// what the forge serves. They have to start from the same port, or a mesh raised on the defaults
// dials one number before the forge is assigned and another after.
func TestTheBuildersCarriedBindingStartsWhereTheForgeServes(t *testing.T) {
forge := ServedOn(catalogueManifest(t, "gitea"), "package-registry", nil)
carried := bindingIn(t, catalogueManifest(t, "builder"), nil)["serves"].(map[string]any)
for _, key := range []string{"port", "scheme", "npm-path"} {
if fmt.Sprint(forge[key]) != fmt.Sprint(carried[key]) {
t.Errorf("the forge serves %s %v and the builder's carried binding says %v — the two "+
"halves of the same registry have drifted apart in the catalogue",
key, forge[key], carried[key])
}
}
}
func TestTheBuildersPackageBindingKeepsItsIdentity(t *testing.T) {
builder := catalogueManifest(t, "builder")
// `at` above all: a setting that moves it points the builder, and the registry password it
// sends as basic auth, at a host somebody else chose.
for _, key := range []string{"provision", "from", "at", "as"} {
var refused error
for _, r := range builder.Resources {
if fmt.Sprint(r["id"]) != "package-binding" {
continue
}
_, refused = ApplySettings(r, []Layer{{From: "anchor",
Values: map[string]any{key: "something else"}}})
}
if refused == nil {
t.Errorf("%q can be set on the builder's package binding, which is not a port but who "+
"the binding is with", key)
}
}
}
// **And the forge's own address follows it**, composed from the manifest in the catalogue beside
// this checkout (novox/hq 04-ISSUES/088).
//
// The forge is reached a third way that neither test above covers: by its own sidecar, over the
// machine's loopback, told where to go in its environment. The `2999:3000` mapping that lets the
// forge go on binding 3000 does nothing for a caller dialling the machine — so a literal there is
// wrong on every node whose assignment differs, and wrong for a second reason on a node given the
// port (ADR 0100). Composed through the whole path, because what proves the placeholder resolves
// in an `env` at all is a declaration, not a manifest.
func TestTheForgesOwnAddressFollowsThePortTheNodeGaveIt(t *testing.T) {
forge, err := catalogueManifest(t, "gitea").Resolve([]Built{{
Name: "runtime", Kind: ArtifactImage,
Reference: "registry.example/gitea-runtime@sha256:" + strings.Repeat("a", 64),
}})
if err != nil {
t.Fatalf("the forge's manifest does not resolve against its own build: %v", err)
}
r := Resolution{Node: "anchor", Modules: []Manifest{forge}, Needs: []Needed{
{Name: "postgres-database", For: "gitea", From: "anchor", At: "127.0.0.1",
Serves: map[string]any{"port": float64(5432)}, Sealed: "sealed-db"},
{Name: "route", For: "gitea", From: "anchor"},
{Name: "secret", For: "gitea", From: "anchor", Local: "internal-token", Sealed: "sealed-token"},
{Name: "secret", For: "gitea", From: "anchor", Local: "admin", Sealed: "sealed-admin"},
}}
// The number this node was given for the forge — the one the machine it is about to run on
// already publishes.
out, err := r.Declaration(Rendering{
Needed: map[string]map[string]string{"gitea": {"broker": "sealed-broker"}},
Given: map[string]map[int]int{"gitea": {3000: 2999}},
})
if err != nil {
t.Fatalf("the forge does not compose: %v", err)
}
// What the machine publishes, and what the forge's sidecar is told to dial: one number.
server := fileNamed(out, "gitea.server")
if server == nil {
t.Fatalf("the forge's own container is not in the declaration: %v", out)
}
if published := fmt.Sprint(server["ports"]); !strings.Contains(published, "2999:3000") {
t.Fatalf("the forge is not published on the port this node gave it: %v", server["ports"])
}
runtime := fileNamed(out, "gitea.runtime")
if runtime == nil {
t.Fatalf("the forge's sidecar is not in the declaration: %v", out)
}
env, _ := runtime["env"].(map[string]any)
if env["MESH_GITEA_URL"] != "http://127.0.0.1:2999" {
t.Fatalf("the forge's sidecar dials %v while the machine publishes the forge on 2999 — "+
"whatever reads it dials a dead port", env["MESH_GITEA_URL"])
}
}
// **And the port the forge publishes the long way is the node's too** (novox/hq ADR 0100).
//
// The forge's ssh port is written `2222:22` — the machine's own daemon holds 22, so the module
// takes 2222 and says so in `listens`. A node whose predecessor served git on another number
// cannot be told to leave it there unless the setting may name the machine side of that mapping,
// which is the number the manifest itself uses everywhere else. Composed from the manifest in the
// catalogue beside this checkout, because what the mesh can move is a fact about what the module
// actually writes.
func TestTheForgesSshPortIsGivenByTheNumberTheForgeCallsIt(t *testing.T) {
forge := catalogueManifest(t, "gitea")
given, err := GivenPorts(forge, []Layer{{From: "anchor",
Values: map[string]any{PortsSetting: map[string]any{"2222": float64(222)}}}})
if err != nil {
t.Fatalf("the forge's ssh port cannot be given on a node: %v", err)
}
// Under the number the module listens on — 2222, the machine side of its mapping — which is
// the number the plan, the filter, the openings and the consumer all ask for. One entry.
if want := map[int]int{2222: 222}; !reflect.DeepEqual(given, want) {
t.Fatalf("the forge was given %v, and it names its ssh port %v", given, want)
}
resolved, err := forge.Resolve([]Built{{
Name: "runtime", Kind: ArtifactImage,
Reference: "registry.example/gitea-runtime@sha256:" + strings.Repeat("a", 64),
}})
if err != nil {
t.Fatalf("the forge's manifest does not resolve against its own build: %v", err)
}
r := Resolution{Node: "anchor", Modules: []Manifest{resolved}, Needs: []Needed{
{Name: "postgres-database", For: "gitea", From: "anchor", At: "127.0.0.1",
Serves: map[string]any{"port": float64(5432)}, Sealed: "sealed-db"},
{Name: "route", For: "gitea", From: "anchor"},
{Name: "secret", For: "gitea", From: "anchor", Local: "internal-token", Sealed: "sealed-token"},
{Name: "secret", For: "gitea", From: "anchor", Local: "admin", Sealed: "sealed-admin"},
}}
out, err := r.Declaration(Rendering{
Needed: map[string]map[string]string{"gitea": {"broker": "sealed-broker"}},
Ports: map[string]map[int]int{"gitea": {3000: 3000, 2222: 222}},
Given: map[string]map[int]int{"gitea": given},
})
if err != nil {
t.Fatalf("the forge does not compose: %v", err)
}
server := fileNamed(out, "gitea.server")
if server == nil {
t.Fatalf("the forge's own container is not in the declaration: %v", out)
}
if published := fmt.Sprint(server["ports"]); !strings.Contains(published, "222:22") ||
strings.Contains(published, "2222:22") {
t.Fatalf("the forge is published on %v, not the port this node gave it", server["ports"])
}
}
+14 -3
View File
@@ -22,8 +22,11 @@ import (
// provides, it does not require. Written as a literal it would be a manifest carrying one
// deployment's machine name, which is the shape [ADR 0066] exists to remove.
//
// Two facts, both the mesh's own vocabulary — the same `node` and `at` a contribution already
// carries. Nothing about what a machine is *for*: that would be the mesh learning what a module
// Three facts, all the mesh's own vocabulary — the same `node` and `at` a contribution already
// carries, and the address behind `at`, for software that takes an address and not a name. The
// case that found the third is a resolver pointing the container runtime at itself: the runtime's
// list of resolvers is addresses, because a name there would have to be resolved by the resolver
// it names. Nothing about what a machine is *for*: that would be the mesh learning what a module
// means, which it does not do.
// ofMachine is where a module says a fact about the machine underneath it belongs:
@@ -49,10 +52,18 @@ func machineUsed(content string) []string {
// to be reached at an address that does not exist is a misconfiguration, and it is said here —
// where the module and the machine are both named — rather than discovered later as a certificate
// nobody can verify.
func machineFacts(r Resolution) map[string]string {
//
// `address` is what `at` resolves to, read from the names the control plane composed — the same map
// the hosts file and the resolver's wildcards are written from, so a file naming the machine's
// address and the file every other machine reaches it by cannot disagree. Absent, like `at`, when
// the machine is off the network or the mesh has not placed it.
func machineFacts(r Resolution, names map[string]string) map[string]string {
out := map[string]string{"name": r.Node}
if r.At != "" {
out["at"] = r.At
if address := names[r.At]; address != "" {
out["address"] = address
}
}
return out
}
@@ -70,3 +70,36 @@ func TestAnAddressAMachineDoesNotHaveIsRefused(t *testing.T) {
t.Errorf("the refusal does not name what was asked for: %v", err)
}
}
// A module that must give software the machine's ADDRESS rather than its name — a resolver pointing
// the container runtime at itself, whose list of resolvers cannot be a name — says
// ${machine:address}, and gets what the machine's name resolves to on the private network: the same
// address every other machine's hosts file carries for it.
func TestAModuleNamesTheAddressBehindItsMachinesName(t *testing.T) {
pointing := Manifest{Module: "pointing", Version: "1", Resources: []map[string]any{{
"id": "runtime", "type": "file", "path": "/etc/runtime.json",
"content": `{"dns":["${machine:address}"]}`,
}}}
got, err := Resolve(shelf(pointing), []string{"pointing"}, anchored(), World{})
if err != nil {
t.Fatal(err)
}
out, err := got.Declaration(Rendering{Names: map[string]string{
"workstation.internal": "10.42.0.7", "anchor.internal": "10.42.0.1"}})
if err != nil {
t.Fatal(err)
}
if content := plainly(out[0]["content"]); content != `{"dns":["10.42.0.7"]}` {
t.Fatalf("the machine's address was not the one its name resolves to: %q", content)
}
// Off the network there is no such address, and the refusal says what the machine does have —
// rather than a placeholder written into the runtime's file and read as an address.
got, err = Resolve(shelf(pointing), []string{"pointing"}, workstation(), World{})
if err != nil {
t.Fatal(err)
}
if _, err := got.Declaration(Rendering{}); err == nil || !strings.Contains(err.Error(), "${machine:address}") {
t.Fatalf("a machine off the network was given an address, or refused for another reason: %v", err)
}
}
+325 -28
View File
@@ -233,6 +233,18 @@ type Manifest struct {
// module that had to say both would eventually say one.
Contributes map[string]map[string]any `json:"contributes,omitempty"`
// ContributesMany is the same key, `contributes`, where a module tells one provider several
// things under local names — `"route": {"api": {"label": "files-api", "port": 9000}, "console":
// {"label": "files", "port": 9001}}` — because a module may answer one requirement more than
// once: an object store with a data API and a console are two different public names, not one
// (novox/hq ADR 0094's sibling for `contributes` rather than `secrets` — "a module may need more
// than one value from a provider that gives one per pair" applies exactly as well to what a
// module gives a provider as to what it keeps from one). Each local name is a contribution of
// its own, reaching the provider as its own entry in the file it receives.
//
// Filled from the manifest's `contributes` object by UnmarshalJSON; never written by hand.
ContributesMany map[string]map[string]map[string]any `json:"-"`
// Receives is where this module wants its consumers' contributions written, per requirement
// it provides.
//
@@ -278,6 +290,14 @@ type Manifest struct {
// makes `restart-on` precise.
Secrets map[string]string `json:"secrets,omitempty"`
// SecretsMany is the same key, `secrets`, where a requirement maps to SEVERAL files under local
// names — `"secret": {"admin": "/…/admin", "token": "/…/token"}` — because a module may need
// more than one value from a provider that gives one per pair (novox/hq 04-ISSUES/069, ADR
// 0094). Each local name is a pair credential of its own, keyed on that name, delivered as its
// own file, served to the provider as its own holder, and rotated with the others. Filled from
// the manifest's `secrets` object by UnmarshalJSON; never written by hand.
SecretsMany map[string]map[string]string `json:"-"`
// OwnSecrets are secrets this module needs in order to be itself, and where to put them.
//
// **Named for whose they are, not how secret they are.** `secrets` above is a credential for
@@ -338,6 +358,14 @@ type Manifest struct {
// that could only see its own ports would write a rule set that closed everything else.
Filtering *Filtering `json:"filtering,omitempty"`
// Guards are ports of this module's the mesh refuses on an adopted node except from the
// private network and from the machine itself (novox/hq ADR 0100) — the store's port and the
// broker's management port. The ports the software uses; the mesh guards where the machine
// publishes them. On an adopted node the found firewall stays in force and the mesh loads no
// filter of its own, so this is what keeps them unreachable from outside whatever that
// firewall does. Ignored on a converged node, whose derived filter already closes them.
Guards []int `json:"guards,omitempty"`
// Facts are things only the mesh knows, written where this module asks for them.
//
// **The graph is the control plane's; how a machine uses it is the module's.** The mesh knows
@@ -403,14 +431,20 @@ type Build struct {
On []BuildsOn `json:"on,omitempty"`
}
// BuildsOn is one base a build needs, and the name the recipe knows it by.
// BuildsOn is one base a build needs, and the name the recipe knows it by: another module's
// artifact, or an image published elsewhere.
type BuildsOn struct {
// Arg is the build argument the recipe reads it from.
Arg string `json:"arg"`
// Module is whose artifact it is.
Module string `json:"module"`
Module string `json:"module,omitempty"`
// Artifact is which of that module's artifacts, by its own name for it.
Artifact string `json:"artifact"`
Artifact string `json:"artifact,omitempty"`
// Image is an image published elsewhere, pinned by digest, that the build copies out of — a
// vendor's tool, a base nobody in the mesh builds. Declared, the mesh copies it into its own
// registry before the build and hands the recipe the copy (novox/hq 04-ISSUES/064, ADR 0097);
// a recipe fetching from a public registry on its own is refused.
Image string `json:"image,omitempty"`
}
// Artifact is one thing built from a module's source.
@@ -593,16 +627,24 @@ func AccessID(path string) string { return "access-" + strings.TrimPrefix(path,
// it contributes to.
func (m Manifest) Wants() []string {
out := append([]string{}, m.Requires...)
for to := range m.Contributes {
var already bool
add := func(to string) {
for _, r := range m.Requires {
if r == to {
already = true
return
}
}
if !already {
out = append(out, to)
for _, already := range out {
if already == to {
return
}
}
out = append(out, to)
}
for to := range m.Contributes {
add(to)
}
for to := range m.ContributesMany {
add(to)
}
sort.Strings(out)
return out
@@ -619,6 +661,199 @@ func ReceivedID(requirement string) string { return "received-" + requirement }
//
// Every problem is reported rather than the first, because somebody writing a manifest fixes
// them in one pass or in four.
// manifestFields is Manifest without its methods, so the JSON methods below can use the ordinary
// field decoding for everything but `secrets`.
type manifestFields Manifest
// UnmarshalJSON reads `secrets` in both of its shapes — a path, or an object of local names to
// paths (ADR 0094) — and everything else exactly as the fields declare, unknown keys refused.
func (m *Manifest) UnmarshalJSON(raw []byte) error {
var keys map[string]json.RawMessage
if err := json.Unmarshal(raw, &keys); err != nil {
return err
}
plain := map[string]string{}
many := map[string]map[string]string{}
if secrets, ok := keys["secrets"]; ok && string(secrets) != "null" {
var byName map[string]json.RawMessage
if err := json.Unmarshal(secrets, &byName); err != nil {
return fmt.Errorf("secrets: an object of requirement to path, or to {local name: path}: %w", err)
}
for to, v := range byName {
switch {
case len(v) > 0 && v[0] == '"':
var path string
if err := json.Unmarshal(v, &path); err != nil {
return err
}
plain[to] = path
case len(v) > 0 && v[0] == '{':
var paths map[string]string
if err := json.Unmarshal(v, &paths); err != nil {
return fmt.Errorf("secrets.%s: an object of local name to path: %w", to, err)
}
many[to] = paths
default:
return fmt.Errorf("secrets.%s: a path, or an object of local name to path, not %s", to, v)
}
}
delete(keys, "secrets")
}
contributesPlain := map[string]map[string]any{}
contributesMany := map[string]map[string]map[string]any{}
if contributes, ok := keys["contributes"]; ok && string(contributes) != "null" {
var byTo map[string]json.RawMessage
if err := json.Unmarshal(contributes, &byTo); err != nil {
return fmt.Errorf("contributes: an object of requirement to values, or to {local name: values}: %w", err)
}
for to, v := range byTo {
// Both shapes are JSON objects, unlike secrets' path-vs-object split, so the shapes are
// told apart by what is INSIDE: an ordinary contribution's fields are scalars (a label,
// a port); the several-instance shape is an object of local names, each itself an
// object of fields. Confirmed against the whole catalogue before relying on it — no
// contribution anywhere has an object-valued field.
var fields map[string]json.RawMessage
if err := json.Unmarshal(v, &fields); err != nil {
return fmt.Errorf("contributes.%s: an object of values, or of local name to values: %w", to, err)
}
many := len(fields) > 0
for _, field := range fields {
trimmed := bytes.TrimSpace(field)
if len(trimmed) == 0 || trimmed[0] != '{' {
many = false
break
}
}
if many {
var locals map[string]map[string]any
if err := json.Unmarshal(v, &locals); err != nil {
return fmt.Errorf("contributes.%s: an object of local name to values: %w", to, err)
}
contributesMany[to] = locals
continue
}
var values map[string]any
if err := json.Unmarshal(v, &values); err != nil {
return fmt.Errorf("contributes.%s: an object of values: %w", to, err)
}
contributesPlain[to] = values
}
delete(keys, "contributes")
}
rest, err := json.Marshal(keys)
if err != nil {
return err
}
decoder := json.NewDecoder(bytes.NewReader(rest))
decoder.DisallowUnknownFields()
var fields manifestFields
if err := decoder.Decode(&fields); err != nil {
return err
}
*m = Manifest(fields)
if len(plain) > 0 {
m.Secrets = plain
}
if len(many) > 0 {
m.SecretsMany = many
}
if len(contributesPlain) > 0 {
m.Contributes = contributesPlain
}
if len(contributesMany) > 0 {
m.ContributesMany = contributesMany
}
return nil
}
// MarshalJSON writes `secrets` and `contributes` back in the shape they were read: single values,
// and objects of local names.
func (m Manifest) MarshalJSON() ([]byte, error) {
raw, err := json.Marshal(manifestFields(m))
if err != nil {
return nil, err
}
if len(m.SecretsMany) == 0 && len(m.ContributesMany) == 0 {
return raw, nil
}
var keys map[string]json.RawMessage
if err := json.Unmarshal(raw, &keys); err != nil {
return nil, err
}
if len(m.ContributesMany) > 0 {
mergedContributes := map[string]any{}
for to, values := range m.Contributes {
mergedContributes[to] = values
}
for to, locals := range m.ContributesMany {
mergedContributes[to] = locals
}
contributes, err := json.Marshal(mergedContributes)
if err != nil {
return nil, err
}
keys["contributes"] = contributes
}
if len(m.SecretsMany) == 0 {
return json.Marshal(keys)
}
merged := map[string]any{}
for to, path := range m.Secrets {
merged[to] = path
}
for to, paths := range m.SecretsMany {
merged[to] = paths
}
secrets, err := json.Marshal(merged)
if err != nil {
return nil, err
}
keys["secrets"] = secrets
return json.Marshal(keys)
}
// SecretFile is one file a module is given a credential in: the local name it goes by inside
// the module (empty for the ordinary one-file case, where the requirement's name serves) and where.
type SecretFile struct {
Local string
Path string
}
// SecretFiles is every file a module wants the credential for one requirement in, in a stable
// order: the plain path as one entry with no local name, or one entry per local name.
func (m Manifest) SecretFiles(to string) []SecretFile {
if path, ok := m.Secrets[to]; ok {
return []SecretFile{{Path: path}}
}
paths := m.SecretsMany[to]
out := make([]SecretFile, 0, len(paths))
for _, local := range sortedKeys(paths) {
out = append(out, SecretFile{Local: local, Path: paths[local]})
}
return out
}
// SecretRequirements is every requirement this module wants a credential file for, sorted.
func (m Manifest) SecretRequirements() []string {
seen := map[string]bool{}
for to := range m.Secrets {
seen[to] = true
}
for to := range m.SecretsMany {
seen[to] = true
}
return sortedKeys(seen)
}
// SecretLocal is the name a credential goes by inside the module: the local name where the
// requirement maps to several, else the requirement itself. It is what `${secret:<name>}` says.
func SecretLocal(to, local string) string {
if local == "" {
return to
}
return local
}
func ParseManifest(raw []byte) (Manifest, error) {
var m Manifest
// Strictly. **An unknown key is refused**, which is the discipline the host's declaration
@@ -722,6 +957,25 @@ func ParseManifest(raw []byte) (Manifest, error) {
"%s contributes nothing to %q; if it only needs one, require it", m.Module, to))
}
}
for to, locals := range m.ContributesMany {
if !name.MatchString(to) {
problems = append(problems, fmt.Sprintf("%q is not a usable name to contribute to", to))
}
if len(locals) == 0 {
problems = append(problems, fmt.Sprintf(
"%s contributes nothing to %q; if it only needs one, require it", m.Module, to))
}
for local, values := range locals {
if !name.MatchString(local) {
problems = append(problems, fmt.Sprintf(
"%s contributes to %q under %q, which is not a usable name", m.Module, to, local))
}
if len(values) == 0 {
problems = append(problems, fmt.Sprintf(
"%s contributes nothing to %q under %q", m.Module, to, local))
}
}
}
problems = append(problems, m.Build.problems(m.Module)...)
// **What provides the artifact store cannot be delivered through it** (novox/hq 04-ISSUES/029).
//
@@ -808,6 +1062,12 @@ func ParseManifest(raw []byte) (Manifest, error) {
"%s listens on %d over %q, which is tcp or udp", m.Module, l.Port, p))
}
}
for _, port := range m.Guards {
if port < 1 || port > 65535 {
problems = append(problems, fmt.Sprintf(
"%s guards port %d, which is not a port", m.Module, port))
}
}
if c := m.Certificate; c != nil {
if !strings.HasPrefix(c.Into, "/") {
problems = append(problems, fmt.Sprintf(
@@ -843,11 +1103,11 @@ func ParseManifest(raw []byte) (Manifest, error) {
}
// **A run-once container is a step the host runs to completion** (novox/hq ADR 0052). It is a
// boolean modifier on the container shape — the host runs the container, requires it to exit 0,
// and starts whatever the declaration places after it only once it has. Two things are refused
// here rather than only on the machine, for the same near-versus-far reason the action ban
// above records: a value that is not a boolean, and the pair run-once + restart-on, which asks
// for two contradictory lifecycles — restart-on brings a *running* container back, and a
// run-once step does not stay running.
// and starts whatever the declaration places after it only once it has. A value that is not a
// boolean is refused here rather than only on the machine, for the same near-versus-far reason
// the action ban above records. A run-once step may name what it reads under restart-on: for a
// step the word means *run again* when one of those changed, which is how a fact fetched from a
// provider is fetched again when the provider moved (novox/hq ADR 0099).
for _, r := range m.Resources {
if fmt.Sprint(r["type"]) != "container" {
continue
@@ -861,14 +1121,6 @@ func ParseManifest(raw []byte) (Manifest, error) {
m.Module, r["id"], raw))
} else {
runOnce = once
if once {
if _, hasRestart := r["restart-on"]; hasRestart {
problems = append(problems, fmt.Sprintf(
"%s declares %v as run-once and with restart-on; a run-once step runs to "+
"completion rather than staying running to be restarted (novox/hq ADR 0052)",
m.Module, r["id"]))
}
}
}
}
// **A scheduled container runs on a recurring cadence** (novox/hq ADR 0053), the recurring
@@ -908,11 +1160,54 @@ func ParseManifest(raw []byte) (Manifest, error) {
problems = append(problems, m.Module+" needs a secret with no name")
}
}
for to, where := range m.Secrets {
if !strings.HasPrefix(where, "/") {
problems = append(problems, fmt.Sprintf(
"%s keeps the credential for %q at %q, which is not an absolute path",
m.Module, to, where))
localOf := map[string]string{}
for _, to := range m.SecretRequirements() {
if _, plain := m.Secrets[to]; plain {
if _, also := m.SecretsMany[to]; also {
problems = append(problems, fmt.Sprintf(
"%s keeps the credential for %q both as one file and as several", m.Module, to))
}
}
for _, f := range m.SecretFiles(to) {
if !strings.HasPrefix(f.Path, "/") {
problems = append(problems, fmt.Sprintf(
"%s keeps the credential for %q at %q, which is not an absolute path",
m.Module, SecretLocal(to, f.Local), f.Path))
}
if f.Local != "" && !name.MatchString(f.Local) {
problems = append(problems, fmt.Sprintf(
"%s keeps a credential for %q under %q, which is not a usable name",
m.Module, to, f.Local))
}
// A local name is what `${secret:<name>}` says, so it may not be another requirement's
// name, another requirement's local name, or one of the module's own secrets — the
// file would hold the wrong credential while every check passed.
if f.Local != "" {
if other, taken := localOf[f.Local]; taken && other != to {
problems = append(problems, fmt.Sprintf(
"%s keeps credentials for %q and %q both under %q — a local name names one",
m.Module, other, to, f.Local))
}
localOf[f.Local] = to
if _, own := m.OwnSecrets[f.Local]; own {
problems = append(problems, fmt.Sprintf(
"%s keeps a credential for %q under %q, which is also one of its own secrets",
m.Module, to, f.Local))
}
for _, w := range m.Wants() {
if w == f.Local {
problems = append(problems, fmt.Sprintf(
"%s keeps a credential for %q under %q, which is also something it requires",
m.Module, to, f.Local))
}
}
}
}
if len(m.SecretsMany[to]) == 0 && m.Secrets[to] == "" {
if _, many := m.SecretsMany[to]; many {
problems = append(problems, fmt.Sprintf(
"%s keeps the credential for %q as several files and names none", m.Module, to))
}
}
var wanted bool
for _, w := range m.Wants() {
@@ -1119,8 +1414,10 @@ func (m Manifest) undeclaredMounts() []string {
for _, where := range m.OwnSecrets {
claim(where)
}
for _, where := range m.Secrets {
claim(where)
for _, to := range m.SecretRequirements() {
for _, f := range m.SecretFiles(to) {
claim(f.Path)
}
}
for _, where := range m.Receives {
claim(where)
+164
View File
@@ -0,0 +1,164 @@
package catalogue
import (
"fmt"
"regexp"
"sort"
"strconv"
"strings"
)
// Telling a module which port it was given.
//
// **The mesh assigns the machine-side port and a module does not choose one**
// ([ADR 0038](../../02-DECISIONS/0038-the-mesh-assigns-the-port.md)); on a node given one for a
// module it is the operator's number rather than the mesh's
// ([ADR 0100](../../02-DECISIONS/0100-a-node-in-use-is-adopted-before-it-is-converged.md)). For a
// container's own listening socket that is invisible: the mesh rewrites `ports` into
// `assigned:wanted`, the software inside binds the number it has always bound, and the machine
// publishes a different one.
//
// **Two kinds of resource have no such layer.**
//
// - **A process** runs on the machine, there is nothing to rewrite, and it binds whatever its
// configuration says — so without this, every process binds the number written in its own
// config, two modules declaring the same one collide, and the mesh's whole reason for
// assigning ports is defeated by the resource kind that most needs it.
// - **A container that DIALS the machine** — a module's own sidecar reaching the service beside
// it over the machine's loopback — is told that address in its environment, and the mapping
// that saves the listener does nothing for the caller: what it must dial is the machine-side
// number, which is exactly the one the module cannot know (novox/hq 04-ISSUES/088).
//
// So a module asks. `${port:8080}` is "the machine-side port you gave me for the 8080 I said I
// listen on", and the module writes that where it would otherwise have written a literal — in a
// file's content, or in a value of a container's `env`.
//
// **The environment is filled by the control plane, exactly as a bound value is.** A port is not
// secret — the mesh holds it in the clear — so there is nothing for the host to be the only
// witness of, and the host learns no new field. That is what separates this from
// [ADR 0086](../../02-DECISIONS/0086-a-secret-reaches-a-process-as-a-file.md), which refuses a
// `${secret:…}` in an `env` outright: the objection there is to the value being in an environment
// at all, not to who fills it in.
//
// **It answers with the machine's number, wherever it is written.** A container reaching a sibling
// over the runtime's own network reaches it on the port inside that container and goes on writing
// that number literally — it is a number the module does control. This is for the machine side,
// which is the side nobody but the mesh can know.
// ofPort is where a module asks which port it was given: ${port:<the port its software uses>}.
var ofPort = regexp.MustCompile(`\$\{port:([0-9]+)\}`)
// portsUsed are the ports a written value asks about, first appearance first.
func portsUsed(content string) []int {
var used []int
seen := map[int]bool{}
for _, m := range ofPort.FindAllStringSubmatch(content, -1) {
n, err := strconv.Atoi(m[1])
if err != nil || seen[n] {
continue
}
seen[n] = true
used = append(used, n)
}
return used
}
// portInto replaces a resource's ${port:…} placeholders with what this machine assigned — in a
// file's content, and in a value of a container's environment.
//
// A port the module did not say it listens on is refused, for the same reason a binding's unknown
// key is: the module is asking about something it never declared, and the answer would be a guess.
// Left alone, the literal would be written into a configuration file, or handed to a process as
// its environment, and read as a port number.
func portInto(resource map[string]any, module string, listens []Listening, with Rendering) error {
switch fmt.Sprint(resource["type"]) {
case "file":
content, ok := resource["content"].(string)
if !ok {
return nil
}
filled, err := portsFilledInto(content,
fmt.Sprintf("%s has a file that", module), module, listens, with)
if err != nil {
return err
}
resource["content"] = filled
case "container":
env, ok := resource["env"].(map[string]any)
if !ok {
return nil
}
// In a stated order, so a container with two bad values always refuses on the same one.
named := make([]string, 0, len(env))
for key := range env {
named = append(named, key)
}
sort.Strings(named)
// **A fresh map, and only when something changes.** This map came out of the module's
// manifest and the resource around it is a shallow copy, so filling a value in place would
// change what the catalogue holds for every other machine running the module — the trap
// withMeshNames is written to avoid, one field along.
var filled map[string]any
for _, key := range named {
written, ok := env[key].(string)
if !ok || len(portsUsed(written)) == 0 {
continue
}
value, err := portsFilledInto(written,
fmt.Sprintf("%s's container %s sets %s to something that",
module, resource["name"], key), module, listens, with)
if err != nil {
return err
}
if filled == nil {
filled = map[string]any{}
for k, v := range env {
filled[k] = v
}
}
filled[key] = value
}
if filled != nil {
resource["env"] = filled
}
}
return nil
}
// portsFilledInto answers every ${port:…} in one written value, or refuses. `where` names the
// place it was written, so a refusal is one edit from right whichever kind of resource it came
// out of.
func portsFilledInto(written, where, module string, listens []Listening, with Rendering) (
string, error) {
for _, wanted := range portsUsed(written) {
var declared bool
for _, l := range listens {
if l.Port == wanted {
declared = true
}
}
if !declared {
return "", fmt.Errorf(
"%s says ${port:%d}, and %s does not say it listens on %d. A module is told the "+
"port it was given for something it declared, and %s",
where, wanted, module, wanted, orNoListens(listens))
}
written = strings.ReplaceAll(written, fmt.Sprintf("${port:%d}", wanted),
strconv.Itoa(with.machinePort(module, wanted)))
}
return written, nil
}
// orNoListens says what would have worked, so a refusal is one edit from right.
func orNoListens(listens []Listening) string {
if len(listens) == 0 {
return "it declares no ports at all"
}
said := make([]string, 0, len(listens))
for _, l := range listens {
said = append(said, strconv.Itoa(l.Port))
}
return "it declares " + strings.Join(said, ", ")
}
-87
View File
@@ -1,87 +0,0 @@
package catalogue
import (
"fmt"
"regexp"
"strconv"
"strings"
)
// Telling a module which port it was given.
//
// **The mesh assigns the machine-side port and a module does not choose one**
// ([ADR 0038](../../02-DECISIONS/0038-the-mesh-assigns-the-port.md)). For a container that is
// invisible: the mesh rewrites `ports` into `assigned:wanted`, the software inside binds the number
// it has always bound, and the machine publishes a different one.
//
// **A process has no such layer.** It runs on the machine, there is nothing to rewrite, and it
// binds whatever its configuration says — so without this, every process binds the number written
// in its own config, two modules declaring the same one collide, and the mesh's whole reason for
// assigning ports is defeated by the resource kind that most needs it.
//
// So a module asks. `${port:8080}` is "the machine-side port you gave me for the 8080 I said I
// listen on", and the module writes that into its own configuration exactly as it writes an
// address it was bound to.
// ofPort is where a module asks which port it was given: ${port:<the port its software uses>}.
var ofPort = regexp.MustCompile(`\$\{port:([0-9]+)\}`)
// portsUsed are the ports a file's content asks about, first appearance first.
func portsUsed(content string) []int {
var used []int
seen := map[int]bool{}
for _, m := range ofPort.FindAllStringSubmatch(content, -1) {
n, err := strconv.Atoi(m[1])
if err != nil || seen[n] {
continue
}
seen[n] = true
used = append(used, n)
}
return used
}
// portInto replaces a file's ${port:…} placeholders with what this machine assigned.
//
// A port the module did not say it listens on is refused, for the same reason a binding's unknown
// key is: the module is asking about something it never declared, and the answer would be a guess.
// Left alone, the literal would be written into a configuration file and read as a port number.
func portInto(resource map[string]any, module string, listens []Listening, with Rendering) error {
if fmt.Sprint(resource["type"]) != "file" {
return nil
}
content, ok := resource["content"].(string)
if !ok {
return nil
}
for _, wanted := range portsUsed(content) {
var declared bool
for _, l := range listens {
if l.Port == wanted {
declared = true
}
}
if !declared {
return fmt.Errorf(
"%s has a file that says ${port:%d}, and %s does not say it listens on %d. A "+
"module is told the port it was given for something it declared, and %s",
module, wanted, module, wanted, orNoListens(listens))
}
content = strings.ReplaceAll(content, fmt.Sprintf("${port:%d}", wanted),
strconv.Itoa(with.machinePort(module, wanted)))
resource["content"] = content
}
return nil
}
// orNoListens says what would have worked, so a refusal is one edit from right.
func orNoListens(listens []Listening) string {
if len(listens) == 0 {
return "it declares no ports at all"
}
said := make([]string, 0, len(listens))
for _, l := range listens {
said = append(said, strconv.Itoa(l.Port))
}
return "it declares " + strings.Join(said, ", ")
}
@@ -1,66 +0,0 @@
package catalogue
import (
"strings"
"testing"
)
// **A process binds the port the mesh gave it, not the one it wrote down.**
//
// A container never needed this: the mesh rewrites its `ports` into assigned:wanted, so the
// software binds the number it always bound and the machine publishes another. A process runs on
// the machine with nothing to rewrite, so without a way to ask, every process binds the number in
// its own configuration and two modules declaring the same one collide — which is the whole
// problem ADR 0038 exists to prevent, reintroduced by the resource kind that most needs it.
func TestAModuleIsToldWhichPortItWasGiven(t *testing.T) {
file := map[string]any{
"type": "file", "id": "settings",
"content": "LISTEN=${port:8080}\n",
}
listens := []Listening{{Port: 8080, From: FromMesh}}
with := Rendering{Ports: map[string]map[int]int{"showcase": {8080: 21000}}}
if err := portInto(file, "showcase", listens, with); err != nil {
t.Fatal(err)
}
if got := file["content"].(string); got != "LISTEN=21000\n" {
t.Fatalf("the module was not told its assigned port: %q", got)
}
}
// With nothing assigned yet, it is told the port it asked about — so a mesh that has not made an
// assignment still composes something coherent rather than writing a zero.
func TestWithNoAssignmentAModuleIsToldWhatItAskedFor(t *testing.T) {
file := map[string]any{"type": "file", "content": "LISTEN=${port:8080}\n"}
if err := portInto(file, "showcase", []Listening{{Port: 8080}}, Rendering{}); err != nil {
t.Fatal(err)
}
if got := file["content"].(string); got != "LISTEN=8080\n" {
t.Fatalf("an unassigned port did not fall back to what was declared: %q", got)
}
}
// **Asking about a port it never declared is refused**, and the refusal says what it did declare.
// The module is asking about something the mesh has no opinion on, and answering would be a guess
// written into a configuration file as a port number.
func TestAskingAboutAnUndeclaredPortIsRefused(t *testing.T) {
file := map[string]any{"type": "file", "content": "LISTEN=${port:9999}\n"}
err := portInto(file, "showcase", []Listening{{Port: 8080}}, Rendering{})
if err == nil {
t.Fatal("a module was told a port it never said it listens on")
}
if !strings.Contains(err.Error(), "8080") {
t.Fatalf("the refusal does not say what would have worked: %v", err)
}
}
// And a file mentioning no port is left exactly as it was.
func TestAFileWithNoPortIsUntouched(t *testing.T) {
file := map[string]any{"type": "file", "content": "GREETING=hello\n"}
if err := portInto(file, "showcase", nil, Rendering{}); err != nil {
t.Fatal(err)
}
if got := file["content"].(string); got != "GREETING=hello\n" {
t.Fatalf("a file with no port was changed: %q", got)
}
}
+170
View File
@@ -0,0 +1,170 @@
package catalogue
import (
"reflect"
"strconv"
"strings"
"testing"
)
// **A process binds the port the mesh gave it, not the one it wrote down.**
//
// A container never needed this: the mesh rewrites its `ports` into assigned:wanted, so the
// software binds the number it always bound and the machine publishes another. A process runs on
// the machine with nothing to rewrite, so without a way to ask, every process binds the number in
// its own configuration and two modules declaring the same one collide — which is the whole
// problem ADR 0038 exists to prevent, reintroduced by the resource kind that most needs it.
func TestAModuleIsToldWhichPortItWasGiven(t *testing.T) {
file := map[string]any{
"type": "file", "id": "settings",
"content": "LISTEN=${port:8080}\n",
}
listens := []Listening{{Port: 8080, From: FromMesh}}
with := Rendering{Ports: map[string]map[int]int{"showcase": {8080: 21000}}}
if err := portInto(file, "showcase", listens, with); err != nil {
t.Fatal(err)
}
if got := file["content"].(string); got != "LISTEN=21000\n" {
t.Fatalf("the module was not told its assigned port: %q", got)
}
}
// With nothing assigned yet, it is told the port it asked about — so a mesh that has not made an
// assignment still composes something coherent rather than writing a zero.
func TestWithNoAssignmentAModuleIsToldWhatItAskedFor(t *testing.T) {
file := map[string]any{"type": "file", "content": "LISTEN=${port:8080}\n"}
if err := portInto(file, "showcase", []Listening{{Port: 8080}}, Rendering{}); err != nil {
t.Fatal(err)
}
if got := file["content"].(string); got != "LISTEN=8080\n" {
t.Fatalf("an unassigned port did not fall back to what was declared: %q", got)
}
}
// **Asking about a port it never declared is refused**, and the refusal says what it did declare.
// The module is asking about something the mesh has no opinion on, and answering would be a guess
// written into a configuration file as a port number.
func TestAskingAboutAnUndeclaredPortIsRefused(t *testing.T) {
file := map[string]any{"type": "file", "content": "LISTEN=${port:9999}\n"}
err := portInto(file, "showcase", []Listening{{Port: 8080}}, Rendering{})
if err == nil {
t.Fatal("a module was told a port it never said it listens on")
}
if !strings.Contains(err.Error(), "8080") {
t.Fatalf("the refusal does not say what would have worked: %v", err)
}
}
// And a file mentioning no port is left exactly as it was.
func TestAFileWithNoPortIsUntouched(t *testing.T) {
file := map[string]any{"type": "file", "content": "GREETING=hello\n"}
if err := portInto(file, "showcase", nil, Rendering{}); err != nil {
t.Fatal(err)
}
if got := file["content"].(string); got != "GREETING=hello\n" {
t.Fatalf("a file with no port was changed: %q", got)
}
}
// **A container that dials the machine is told the same thing, in its environment.**
//
// The forge's own sidecar reaches the forge over the machine's loopback (novox/hq 04-ISSUES/088).
// The `assigned:wanted` mapping that lets the forge itself go on binding 3000 does nothing for the
// caller: the caller dials the machine, so it must be given the machine's number — here one the
// node was given rather than one the mesh assigned, which is the case that makes the literal wrong
// even on a mesh that never allocates (ADR 0100).
func TestAContainerIsToldWhichPortItWasGiven(t *testing.T) {
sidecar := map[string]any{
"type": "container", "id": "runtime", "name": "mesh-gitea",
"env": map[string]any{
"MESH_GITEA_URL": "http://127.0.0.1:${port:3000}",
"MESH_ADMIN_USER": "mesh-admin",
},
}
listens := []Listening{{Port: 3000, From: FromMesh}}
with := Rendering{Given: map[string]map[int]int{"gitea": {3000: 2999}}}
if err := portInto(sidecar, "gitea", listens, with); err != nil {
t.Fatal(err)
}
env := sidecar["env"].(map[string]any)
if env["MESH_GITEA_URL"] != "http://127.0.0.1:2999" {
t.Fatalf("the sidecar dials %v, not the port this machine puts the forge on",
env["MESH_GITEA_URL"])
}
if env["MESH_ADMIN_USER"] != "mesh-admin" {
t.Fatalf("filling one value changed another: %v", env)
}
}
// **And the manifest it came out of is left alone.**
//
// An `env` map is the catalogue's, shared by every node running the module, and the resource
// around it is a shallow copy. Filled in place, the first node composed would write its own port
// into the catalogue and every node composed after it would be told that one — a fault that is
// invisible in a single composition and wrong in every mesh with two machines.
func TestFillingAContainersEnvironmentLeavesTheManifestAlone(t *testing.T) {
env := map[string]any{"MESH_GITEA_URL": "http://127.0.0.1:${port:3000}"}
manifest := map[string]any{"type": "container", "id": "runtime", "name": "mesh-gitea", "env": env}
for _, at := range []int{2999, 3100} {
copied := map[string]any{}
for k, v := range manifest {
copied[k] = v
}
with := Rendering{Given: map[string]map[int]int{"gitea": {3000: at}}}
if err := portInto(copied, "gitea", []Listening{{Port: 3000}}, with); err != nil {
t.Fatal(err)
}
want := "http://127.0.0.1:" + strconv.Itoa(at)
if got := copied["env"].(map[string]any)["MESH_GITEA_URL"]; got != want {
t.Fatalf("the second node was told %v, not %v — the first composition wrote its own "+
"port into the catalogue", got, want)
}
}
if env["MESH_GITEA_URL"] != "http://127.0.0.1:${port:3000}" {
t.Fatalf("the module's own manifest was edited: %v", env)
}
}
// Asking in an environment about a port it never declared is refused exactly as a file's is, and
// the refusal names the container and the variable — there is no line number to go on.
func TestAContainerAskingAboutAnUndeclaredPortIsRefused(t *testing.T) {
sidecar := map[string]any{
"type": "container", "id": "runtime", "name": "mesh-gitea",
"env": map[string]any{"MESH_GITEA_URL": "http://127.0.0.1:${port:9999}"},
}
err := portInto(sidecar, "gitea", []Listening{{Port: 3000}}, Rendering{})
if err == nil {
t.Fatal("a container was told a port its module never said it listens on")
}
for _, said := range []string{"mesh-gitea", "MESH_GITEA_URL", "3000"} {
if !strings.Contains(err.Error(), said) {
t.Errorf("the refusal does not say %q: %v", said, err)
}
}
}
// And a container naming no port keeps the environment it was written with — the same map, so
// nothing is copied and no other node's composition is disturbed.
func TestAContainerWithNoPortPlaceholderKeepsItsEnvironment(t *testing.T) {
env := map[string]any{"MESH_GITEA_URL": "http://gitea:3000", "PORT": 3000}
sidecar := map[string]any{"type": "container", "name": "mesh-gitea", "env": env}
if err := portInto(sidecar, "gitea", []Listening{{Port: 3000}}, Rendering{
Given: map[string]map[int]int{"gitea": {3000: 2999}},
}); err != nil {
t.Fatal(err)
}
// A sibling reached over the runtime's own network is reached on the port INSIDE it, which the
// module does control: a literal there is right, and moving it would break the one address the
// mapping does not touch.
if got := sidecar["env"].(map[string]any)["MESH_GITEA_URL"]; got != "http://gitea:3000" {
t.Fatalf("an address on the runtime's own network was moved to the machine's port: %v", got)
}
// The same map, not a copy of it: a container that asks for nothing is left alone, and
// comparing the contents would say yes even to a rebuilt map.
if reflect.ValueOf(sidecar["env"]).Pointer() != reflect.ValueOf(env).Pointer() {
t.Fatalf("a container that asked for nothing had its environment rebuilt: %v", sidecar["env"])
}
}
+41 -2
View File
@@ -157,6 +157,10 @@ type Needed struct {
Sealed string
// For is the module that wanted it.
For string
// Local is the name this credential goes by inside that module, where the module wants several
// for one requirement (ADR 0094); empty for the ordinary one. Part of what identifies the pair
// credential, so two secrets from one provider to one module are two secrets.
Local string
// Manager is set when this holder is a refreshable-grant licence's MANAGER, delivered the refresh
// token rather than an access token (novox/hq ADR 0050). It changes one thing downstream: an empty
// Sealed is tolerated — the manager has not adopted a refresh token yet, which is a real waiting
@@ -830,6 +834,19 @@ func providersFirst(order []string, shelf map[string]Manifest) []string {
// losing the one it depends on.
func perConsumer(needs []Needed, order []string, catalogue map[string]Manifest) []Needed {
out := make([]Needed, 0, len(needs))
// Once per (provision, consumer, local name, provider). The walk over names visits a
// same-node provision once per module that mentions it, so two consumers of one produced two
// raw needs and, fanned out below, four — the same credential twice for each. Harmless
// downstream, since a pair is one row however often it is asked for, and wrong all the same.
seen := map[[4]string]bool{}
keep := func(n Needed) {
key := [4]string{n.Name, n.For, n.Local, n.From}
if seen[key] {
return
}
seen[key] = true
out = append(out, n)
}
for _, n := range needs {
var wanted bool
for _, name := range order {
@@ -843,14 +860,36 @@ func perConsumer(needs []Needed, order []string, catalogue map[string]Manifest)
}
copied := n
copied.For = m.Module
out = append(out, copied)
// And once per file THIS module keeps the credential in, where it keeps several
// (ADR 0094) — here, where the consumer is finally known, not on the walk above.
for _, one := range eachLocal(nil, catalogue, copied) {
keep(one)
}
wanted = true
break
}
}
if !wanted {
out = append(out, n)
keep(n)
}
}
return out
}
// eachLocal appends the need once per file the wanting module keeps the credential in: once, with
// no local name, in the ordinary case; once per local name where the module wants several values
// from one provider (ADR 0094). Each is its own pair credential downstream.
func eachLocal(needs []Needed, catalogue map[string]Manifest, n Needed) []Needed {
files := catalogue[n.For].SecretFiles(n.Name)
// One file under a local name is still a local name: the review found a module keeping ONE
// named secret given a need with no local, and so no file, while everything reported success.
if len(files) == 0 || (len(files) == 1 && files[0].Local == "") {
return append(needs, n)
}
for _, f := range files {
one := n
one.Local = f.Local
needs = append(needs, one)
}
return needs
}
@@ -0,0 +1,188 @@
package catalogue
import (
"encoding/json"
"strings"
"testing"
)
// The catalogue's resolver modules as they are, parsed by the real parser and composed as a
// machine would receive them (hal dnsmasq-app conversion, novox/hq 08-connectivity).
//
// The predecessor's resolver answered every name on a machine: the mesh's own itself, the rest
// forwarded to two fixed upstreams, with the machine's resolv.conf naming it alone and the
// container runtime pointed at its private-network address. These hold the mesh's modules to the
// same arrangement, and to the two things a resolver here must never do — read resolv.conf for
// its upstreams, or take an address systemd-resolved holds.
// resolverShelf is the three resolver modules beside something that answers `mesh-addressing`.
// The networking module that really does is composed in the controller and cannot be imported
// here, so a stand-in offers the same word; what is under test is the manifests, not the network.
func resolverShelf(t *testing.T) map[string]Manifest {
t.Helper()
shelf := map[string]Manifest{
"net": {Module: "net", Version: "1", Provides: []Offer{{Name: "mesh-addressing"}}},
}
for _, name := range []string{"dnsmasq", "resolv-conf", "resolved-split-dns"} {
shelf[name] = catalogueManifest(t, name)
}
return shelf
}
// twoMachines is what the control plane hands a rendering: internal names and their addresses.
var twoMachines = map[string]string{"anchor.internal": "10.42.0.1", "laptop.internal": "10.42.0.2"}
// Its configuration forwards to the upstreams the predecessor's module shipped, and gets them from
// nowhere else: `no-resolv` is what makes the documented loop — the resolver finding its own
// address in resolv.conf and becoming its own upstream — impossible.
func TestTheResolverForwardsToFixedUpstreamsAndNeverReadsResolvConf(t *testing.T) {
m := catalogueManifest(t, "dnsmasq")
var config string
for _, r := range m.Resources {
if r["id"] == "config" {
config, _ = r["content"].(string)
}
}
if config == "" {
t.Fatal("the resolver has no configuration file")
}
for _, want := range []string{
"\nno-resolv\n", "\nserver=1.1.1.1\n", "\nserver=8.8.8.8\n",
"\nlisten-address=127.0.0.1\n", "\ninterface=mesh0\n", "\nbind-dynamic\n",
"\ndomain-needed\n", "\nbogus-priv\n",
"\nconf-file=" + m.Facts[FactNodeZones] + "\n",
} {
if !strings.Contains(config, want) {
t.Errorf("the resolver's configuration lacks %q:\n%s", strings.TrimSpace(want), config)
}
}
// Not .53 or .54, which systemd-resolved holds; and not .55 any more, which was a convention
// beside the one every machine already followed — the predecessor's resolv.conf says .1.
for _, taken := range []string{"127.0.0.53", "127.0.0.54", "127.0.0.55"} {
if strings.Contains(config, "listen-address="+taken) {
t.Errorf("the resolver listens on %s", taken)
}
}
// And the file that decides what the machine asks names it there, alone.
var resolv string
for _, r := range catalogueManifest(t, "resolv-conf").Resources {
if r["path"] == "/etc/resolv.conf" {
resolv, _ = r["content"].(string)
}
}
var nameservers []string
for _, line := range strings.Split(resolv, "\n") {
if strings.HasPrefix(line, "nameserver ") {
nameservers = append(nameservers, strings.TrimPrefix(line, "nameserver "))
}
}
if len(nameservers) != 1 || nameservers[0] != "127.0.0.1" {
t.Errorf("resolv.conf names %v; the predecessor's names the mesh's resolver alone at 127.0.0.1", nameservers)
}
// The split-DNS alternative points at the same address, or a machine that keeps
// systemd-resolved in charge would route the mesh's suffix to nothing.
for _, r := range catalogueManifest(t, "resolved-split-dns").Resources {
if content, _ := r["content"].(string); content != "" && !strings.Contains(content, "DNS=127.0.0.1\n") {
t.Errorf("resolved-split-dns does not point at the resolver's address:\n%s", content)
}
}
}
// The resolver and what points the machine at it compose on one machine, and what arrives is the
// mesh's account of every machine as a wildcard, the suffix kept local, the daemon restarting on
// that file, and the runtime pointed at this machine's own address.
func TestTheResolverAndWhatAsksItComposeOnOneMachine(t *testing.T) {
got, err := Resolve(resolverShelf(t), []string{"dnsmasq", "resolv-conf"},
Node{Name: "anchor", At: "anchor.internal"}, World{})
if err != nil {
t.Fatal(err)
}
if !strings.Contains(strings.Join(named(got), " "), "net") {
t.Fatalf("the resolver's data is the mesh's addresses, and nothing answering them was taken: %v", named(got))
}
out, err := got.Declaration(Rendering{
Names: twoMachines, Suffix: "internal",
Needed: map[string]map[string]string{"dnsmasq": {"broker": "sealed"}},
})
if err != nil {
t.Fatal(err)
}
ids := byID(out)
zones := ids["dnsmasq.fact-node-zones"]
if zones == nil || zones["path"] != "/etc/mesh-resolver/nodes.conf" {
t.Fatalf("the resolver was not given the machines where its configuration reads them: %v", zones)
}
content, _ := zones["content"].(string)
for _, want := range []string{
"local=/internal/", "address=/anchor.internal/10.42.0.1", "address=/laptop.internal/10.42.0.2",
} {
if !strings.Contains(content, want) {
t.Errorf("the machines file lacks %q:\n%s", want, content)
}
}
service := ids["dnsmasq.service"]
if service == nil {
t.Fatal("no resolver service composed")
}
reflects := map[string]bool{}
for _, id := range service["restart-on"].([]any) {
reflects[id.(string)] = true
}
if !reflects["dnsmasq.config"] || !reflects["dnsmasq.fact-node-zones"] {
t.Errorf("the daemon does not restart on its configuration and the machines file both: %v", service["restart-on"])
}
// The runtime's own file, written into (novox/hq ADR 0102) with the one key this module states.
runtime := ids["dnsmasq.runtime-dns"]
if runtime == nil || runtime["path"] != "/etc/docker/daemon.json" || runtime["into"] != "json" {
t.Fatalf("the runtime's dns is not written into its file: %v", runtime)
}
var keys map[string][]string
if err := json.Unmarshal([]byte(runtime["content"].(string)), &keys); err != nil {
t.Fatalf("the runtime's keys are not JSON: %v", err)
}
if len(keys) != 1 || len(keys["dns"]) != 1 || keys["dns"][0] != "10.42.0.1" {
t.Errorf("the runtime is pointed at %v; containers resolve at this machine's own private-network address, and nothing else is written", keys)
}
for _, r := range out {
if r["type"] == "service" && r["unit"] == "docker.service" && r["id"] != "" &&
strings.HasPrefix(r["id"].(string), "dnsmasq.") {
t.Errorf("the resolver orders the runtime restarted or reloaded, which stops every container (ADR 0102) or does nothing for dns: %v", r)
}
}
resolv := ids["resolv-conf.resolv"]
if resolv == nil || !strings.Contains(resolv["content"].(string), "\nnameserver 127.0.0.1\n") {
t.Fatalf("the machine is not pointed at the resolver: %v", resolv)
}
}
// Two modules deciding what a machine asks are refused on one machine, as before — the claim
// exists so they never take turns overwriting each other.
func TestTwoThingsDecidingWhatAMachineAsksAreRefused(t *testing.T) {
_, err := Resolve(resolverShelf(t), []string{"dnsmasq", "resolv-conf", "resolved-split-dns"},
Node{Name: "anchor", At: "anchor.internal"}, World{})
if err == nil {
t.Fatal("resolv-conf and resolved-split-dns were both assigned to one machine")
}
if !strings.Contains(err.Error(), "the-resolver-configuration") {
t.Fatalf("the refusal does not say what was claimed: %v", err)
}
}
// A machine that is not on the private network has no address for the runtime to be pointed at.
// Refused where the module and the machine are both named, rather than a placeholder written into
// the runtime's file and read as an address.
func TestTheResolverOnAMachineOffTheNetworkIsRefused(t *testing.T) {
got, err := Resolve(resolverShelf(t), []string{"dnsmasq"}, Node{Name: "anchor"}, World{})
if err != nil {
t.Fatal(err)
}
_, err = got.Declaration(Rendering{Names: twoMachines, Suffix: "internal",
Needed: map[string]map[string]string{"dnsmasq": {"broker": "sealed"}}})
if err == nil || !strings.Contains(err.Error(), "${machine:address}") {
t.Fatalf("a machine off the network was composed a resolver, or refused for another reason: %v", err)
}
}
+7 -8
View File
@@ -77,17 +77,16 @@ func TestARunOnceMustBeABoolean(t *testing.T) {
}
}
func TestARunOnceContainerCannotAlsoDeclareRestartOn(t *testing.T) {
// restart-on brings a running container back; a run-once step does not stay running. The pair
// is a contradiction, refused at the manifest rather than surfacing far away on the host.
func TestARunOnceStepMayNameWhatItReads(t *testing.T) {
// For a step, restart-on means *run again* when what it reads changed: a gate that fetches a
// provider's root names the binding file it reads, so a provider that moved is fetched again
// (novox/hq ADR 0099). Accepted here, and the host's digest does the rest.
digest := "@sha256:" + strings.Repeat("a", 64)
bad := []byte(`{"module":"m","resources":[
good := []byte(`{"module":"m","resources":[
{"id":"conf","type":"file","path":"/x","content":"y"},
{"id":"seed","type":"container","name":"seed","image":"registry.example/x` + digest + `","run-once":true,"restart-on":["conf"]}
]}`)
if _, err := ParseManifest(bad); err == nil {
t.Error("a run-once container that also declared restart-on was accepted")
} else if !strings.Contains(err.Error(), "restart-on") {
t.Errorf("refused for the wrong reason: %v", err)
if _, err := ParseManifest(good); err != nil {
t.Errorf("a run-once step naming what it reads was refused: %v", err)
}
}
+120
View File
@@ -0,0 +1,120 @@
package catalogue
import (
"fmt"
"regexp"
"sort"
"strconv"
"strings"
)
// Telling a module where this machine put the holder of a seat.
//
// **The foundation's ports are the node's** (novox/hq ADR 0100): the port a foundation server was
// given at genesis becomes that node's setting for the module that serves it, and every reader
// follows the setting. Every consumer's binding did. The control plane's own connections did not
// (04-ISSUES/102): they are written at genesis, before any module exists to bind to — full
// connection strings, sealed, with the port inside — so when the node moved the store, the
// control plane went on dialling where genesis had written and the mesh was headless.
//
// The control plane cannot open its own sealed connection to move the port, and it cannot bind
// the store as a consumer would: a binding mints a credential, and what the control plane holds
// is the foundation's superuser, made before the mesh. What it can do is read the node's settings
// when it composes its own declaration — it is the thing that composes every other module's — and
// say in its own environment which port this machine put the store at.
//
// So a module may ask about a **seat** (ADR 0079: a foundation seat is named after the server it
// guards — `mesh-store`, `mesh-broker`). `${seat:mesh-store:5432}` is "the port this machine put
// the holder of the mesh-store seat's 5432 at". Not a provision: nothing is required, nothing is
// granted, no credential is minted. A seat is the mesh's own vocabulary for the store and the
// broker, which is what makes this the control plane's way of naming them and not a way for a
// module to reach a server it was not granted — the answer is a port number the mesh holds in the
// clear, and the credential to use it is still the module's own to have.
//
// **The answer may be empty, and that is the one place a placeholder answers with nothing.** The
// store and the broker are raised at genesis, before the mesh knows them as modules; a mesh raised
// on the catalogue's own ports never gives them a setting at all. In both, the port genesis wrote
// into the connection string is the right one, and the mesh has nothing to add. An empty answer
// says exactly that, and what reads it — the control plane's `_PORT` twin — treats an empty value
// as no value. Answering with the software's own port instead would override what genesis wrote
// with a number the mesh never checked, on the one machine where that is a headless mesh.
// ofSeat is where a module asks about a seat: ${seat:<seat>:<the port its holder's software uses>}.
var ofSeat = regexp.MustCompile(`\$\{seat:([a-z0-9][a-z0-9-]*):([0-9]+)\}`)
// seatInto replaces a resource's ${seat:…} placeholders with where this machine put each seat's
// holder — in a file's content, and in a value of a container's environment. The same two places
// portInto fills, for the same reason: they are where a process reads a number from.
func seatInto(resource map[string]any, module string, with Rendering) error {
switch fmt.Sprint(resource["type"]) {
case "file":
content, ok := resource["content"].(string)
if !ok || !ofSeat.MatchString(content) {
return nil
}
filled, err := seatsFilledInto(content, fmt.Sprintf("%s has a file that", module), with)
if err != nil {
return err
}
resource["content"] = filled
case "container":
env, ok := resource["env"].(map[string]any)
if !ok {
return nil
}
named := make([]string, 0, len(env))
for key := range env {
named = append(named, key)
}
sort.Strings(named)
// A fresh map, and only when something changes — this map is the catalogue's, shared by
// every node running the module (see portInto).
var filled map[string]any
for _, key := range named {
written, ok := env[key].(string)
if !ok || !ofSeat.MatchString(written) {
continue
}
value, err := seatsFilledInto(written,
fmt.Sprintf("%s's container %s sets %s to something that",
module, resource["name"], key), with)
if err != nil {
return err
}
if filled == nil {
filled = map[string]any{}
for k, v := range env {
filled[k] = v
}
}
filled[key] = value
}
if filled != nil {
resource["env"] = filled
}
}
return nil
}
// seatsFilledInto answers every ${seat:…} in one written value.
//
// A port the seat's holder does not publish on this machine — or a seat nothing on it holds —
// answers with nothing, for the reason the package comment gives. A port that is not one is
// refused: it was written by a person and it is wrong.
func seatsFilledInto(written, where string, with Rendering) (string, error) {
for _, m := range ofSeat.FindAllStringSubmatch(written, -1) {
seat, port := m[1], m[2]
wanted, err := strconv.Atoi(port)
if err != nil || wanted < 1 || wanted > 65535 {
return "", fmt.Errorf("%s says ${seat:%s:%s}, and %s is not a port", where, seat, port, port)
}
answer := ""
if at, known := with.Seats[seat][wanted]; known {
answer = strconv.Itoa(at)
}
written = strings.ReplaceAll(written, m[0], answer)
}
return written, nil
}
+216
View File
@@ -0,0 +1,216 @@
package catalogue
import (
"os"
"strings"
"testing"
)
// The control plane's own addresses follow the node's ports (novox/hq 04-ISSUES/102).
func TestASeatPlaceholderAnswersWhereThisMachinePutTheHolder(t *testing.T) {
control := map[string]any{
"type": "container", "id": "server", "name": "mesh-controller",
"env": map[string]any{
"MESH_STORE_INVENTORY_PORT": "${seat:mesh-store:5432}",
"MESH_BROKER_AMQP_PORT": "${seat:mesh-broker:5672}",
"MESH_BROKER_ADDRESS_PORT": "${seat:mesh-broker:5671}",
"MESH_STORE_INVENTORY_FILE": "/run/secrets/inventory",
},
}
with := Rendering{Seats: map[string]map[int]int{
"mesh-store": {5432: 6852}, "mesh-broker": {5672: 5679, 5671: 5671},
}}
if err := seatInto(control, "mesh-controller", with); err != nil {
t.Fatal(err)
}
env := control["env"].(map[string]any)
for key, want := range map[string]string{
"MESH_STORE_INVENTORY_PORT": "6852",
"MESH_BROKER_AMQP_PORT": "5679",
"MESH_BROKER_ADDRESS_PORT": "5671",
"MESH_STORE_INVENTORY_FILE": "/run/secrets/inventory",
} {
if env[key] != want {
t.Errorf("%s = %v, want %q", key, env[key], want)
}
}
}
// A seat nothing on this machine holds — or one whose holder the mesh has given no port — answers
// with nothing, so the port genesis wrote into the connection string stands. Not the software's
// own port: on a node given a port at genesis before the store's module exists, that would
// override the right number with the catalogue's.
func TestASeatTheMeshCannotPlaceAnswersWithNothing(t *testing.T) {
control := map[string]any{
"type": "container", "id": "server", "name": "mesh-controller",
"env": map[string]any{"MESH_STORE_INVENTORY_PORT": "${seat:mesh-store:5432}"},
}
if err := seatInto(control, "mesh-controller", Rendering{}); err != nil {
t.Fatal(err)
}
if got := control["env"].(map[string]any)["MESH_STORE_INVENTORY_PORT"]; got != "" {
t.Fatalf("with nothing known, the seat answered %q", got)
}
file := map[string]any{"type": "file", "content": "port=${seat:mesh-store:5432}\n"}
if err := seatInto(file, "x", Rendering{Seats: map[string]map[int]int{"mesh-store": {5433: 1}}}); err != nil {
t.Fatal(err)
}
if got := file["content"]; got != "port=\n" {
t.Fatalf("a port the holder does not publish answered %q", got)
}
}
func TestASeatPlaceholderNamingNoPortIsRefused(t *testing.T) {
file := map[string]any{"type": "file", "content": "${seat:mesh-store:99999}"}
if err := seatInto(file, "x", Rendering{}); err == nil {
t.Fatal("99999 was accepted as a port")
}
}
func TestFillingASeatLeavesTheManifestAlone(t *testing.T) {
env := map[string]any{"MESH_STORE_INVENTORY_PORT": "${seat:mesh-store:5432}"}
manifest := map[string]any{"type": "container", "id": "server", "env": env}
for _, at := range []int{6852, 5432} {
copied := map[string]any{}
for k, v := range manifest {
copied[k] = v
}
with := Rendering{Seats: map[string]map[int]int{"mesh-store": {5432: at}}}
if err := seatInto(copied, "mesh-controller", with); err != nil {
t.Fatal(err)
}
}
if env["MESH_STORE_INVENTORY_PORT"] != "${seat:mesh-store:5432}" {
t.Fatalf("the module's own manifest was edited: %v", env)
}
}
// **The control plane's own manifest, composed through the whole path.**
//
// The store was given 6852 and the broker's plain port 5679 (the control-node's migration, novox/hq
// 04-ISSUES/102). The control plane's own connections are sealed at genesis with the ports genesis
// wrote; what its container is told beside them is where this machine put the store and the
// broker now, read from the node's settings exactly as every consumer's binding is.
func TestTheControlPlanesOwnAddressesFollowTheNodesPorts(t *testing.T) {
raw, err := os.ReadFile("../../module.json")
if err != nil {
t.Fatal(err)
}
m, err := ParseManifest(raw)
if err != nil {
t.Fatalf("the control plane's own manifest does not parse:\n%v", err)
}
// The manifest itself names them now; withSeatPorts is a no-op on it, and this holds it so.
for _, r := range m.Resources {
if r["type"] != "container" {
continue
}
env, _ := r["env"].(map[string]any)
for key, want := range SeatPorts {
if env[key] != want {
t.Errorf("module.json says %s=%v, not %q", key, env[key], want)
}
}
}
m = withSeatPorts(m)
control, err := m.Resolve([]Built{{
Name: "server", Kind: ArtifactImage,
Reference: ArtifactStoreScheme + "mesh-controller/server@sha256:" + strings.Repeat("c", 64),
}})
if err != nil {
t.Fatal(err)
}
r := Resolution{Node: "anchor", Modules: []Manifest{control}}
needed := map[string]map[string]string{"mesh-controller": {}}
for name := range m.OwnSecrets {
needed["mesh-controller"][name] = "sealed-" + name
}
out, err := r.Declaration(Rendering{
Needed: needed,
ArtifactStore: "anchor.internal:5100",
Seats: map[string]map[int]int{
"mesh-store": {5432: 6852},
"mesh-broker": {5671: 5671, 5672: 5679, 15672: 15673},
},
})
if err != nil {
t.Fatalf("the control plane does not compose: %v", err)
}
server := fileNamed(out, "mesh-controller.server")
if server == nil {
t.Fatalf("the control plane's container is not in the declaration: %v", out)
}
env, _ := server["env"].(map[string]any)
for key, want := range map[string]string{
"MESH_STORE_INVENTORY_PORT": "6852",
"MESH_STORE_IDENTITY_PORT": "6852",
"MESH_STORE_LICENCES_PORT": "6852",
"MESH_BROKER_AMQP_PORT": "5679",
"MESH_BROKER_MANAGEMENT_PORT": "15673",
"MESH_BROKER_ADDRESS_PORT": "5671",
} {
if env[key] != want {
t.Errorf("the control plane is told %s=%v; the node put it on %s", key, env[key], want)
}
}
if got := server["image"]; got != "anchor.internal:5100/mesh-controller/server@sha256:"+strings.Repeat("c", 64) {
t.Errorf("the control plane's own image is %v, not routed through the store", got)
}
// And on a mesh where the foundation is where genesis raised it, nothing is added.
out, err = r.Declaration(Rendering{Needed: needed, ArtifactStore: "anchor.internal:5100"})
if err != nil {
t.Fatal(err)
}
env, _ = fileNamed(out, "mesh-controller.server")["env"].(map[string]any)
if env["MESH_STORE_INVENTORY_PORT"] != "" || env["MESH_BROKER_AMQP_PORT"] != "" {
t.Errorf("with no settings, the control plane is told %v", env)
}
}
// SeatPorts is what the control plane's manifest says beside each sealed connection: the port
// this machine put the seat's holder at (novox/hq 04-ISSUES/102).
var SeatPorts = map[string]string{
"MESH_STORE_INVENTORY_PORT": "${seat:mesh-store:5432}",
"MESH_STORE_IDENTITY_PORT": "${seat:mesh-store:5432}",
"MESH_STORE_LICENCES_PORT": "${seat:mesh-store:5432}",
"MESH_BROKER_AMQP_PORT": "${seat:mesh-broker:5672}",
"MESH_BROKER_MANAGEMENT_PORT": "${seat:mesh-broker:15672}",
"MESH_BROKER_ADDRESS_PORT": "${seat:mesh-broker:5671}",
}
// withSeatPorts is the control plane's manifest with SeatPorts in its container's environment.
//
// **The manifest lands one commit after the code that fills it**, deliberately: a control plane
// still running the previous build passes `${seat:…}` through unfilled, and the manifest may only
// name the placeholder once every control plane that could compose it knows it. So the test does
// not depend on module.json carrying these yet, and is a no-op once it does.
func withSeatPorts(m Manifest) Manifest {
out := m
out.Resources = nil
for _, r := range m.Resources {
if r["type"] != "container" {
out.Resources = append(out.Resources, r)
continue
}
copied := map[string]any{}
for k, v := range r {
copied[k] = v
}
env := map[string]any{}
if had, ok := r["env"].(map[string]any); ok {
for k, v := range had {
env[k] = v
}
}
for k, v := range SeatPorts {
if _, said := env[k]; !said {
env[k] = v
}
}
copied["env"] = env
out.Resources = append(out.Resources, copied)
}
return out
}
+20 -17
View File
@@ -61,23 +61,26 @@ func sealedFor(m Manifest, needs []Needed, with Rendering) (map[string]string, e
sealed[name] = value
}
}
for _, to := range sortedKeys(m.Secrets) {
if _, taken := sealed[to]; taken {
// A module whose own secret and whose requirement share a name. Refused rather than
// settled by precedence: whichever won, the manifest would read as though the other
// had, and the file would hold the credential for the wrong thing while every check
// passed.
return nil, fmt.Errorf(
"%s has a secret of its own called %q and also requires %q, so a file saying "+
"${secret:%s} could mean either — rename one of them", m.Module, to, to, to)
}
for i := range needs {
// `For == m.Module`, not name alone: on a node with two modules requiring the same
// provision, both appear in `needs`, and matching by name would fill ${secret:X} with
// whichever came last — the other module's credential (novox/hq 04-ISSUES/022). The
// `secrets:`-map path already guards this way; the ${secret:…} placeholder path did not.
if needs[i].Name == to && needs[i].For == m.Module && needs[i].Sealed != "" {
sealed[to] = needs[i].Sealed
for _, to := range m.SecretRequirements() {
for _, file := range m.SecretFiles(to) {
key := SecretLocal(to, file.Local)
if _, taken := sealed[key]; taken {
// A module whose own secret and whose requirement share a name. Refused rather than
// settled by precedence: whichever won, the manifest would read as though the other
// had, and the file would hold the credential for the wrong thing while every check
// passed.
return nil, fmt.Errorf(
"%s has a secret of its own called %q and also requires %q, so a file saying "+
"${secret:%s} could mean either — rename one of them", m.Module, key, key, key)
}
for i := range needs {
// `For == m.Module`, not name alone: on a node with two modules requiring the same
// provision, both appear in `needs`, and matching by name would fill ${secret:X} with
// whichever came last — the other module's credential (novox/hq 04-ISSUES/022). And
// the local name, where the module keeps several (ADR 0094).
if needs[i].Name == to && needs[i].For == m.Module && needs[i].Local == file.Local && needs[i].Sealed != "" {
sealed[key] = needs[i].Sealed
}
}
}
}
+10
View File
@@ -101,6 +101,11 @@ func settle(base map[string]any, layers []Layer, protected map[string]bool, what
merged := deepCopy(base)
for _, layer := range layers {
for key, value := range layer.Values {
if key == PortsSetting {
// Where the machine puts a port is the mesh's to apply, not a value for a file or
// for what a consumer is told (novox/hq ADR 0100); it reaches both as the port.
continue
}
if protected[key] {
// The module said it must own this one. Refused rather than ignored: a setting
// that is quietly dropped is somebody believing they changed something.
@@ -176,6 +181,11 @@ func UnusedSettings(m Manifest, layers []Layer) []string {
if key == ExposeSetting && len(m.Listens) > 0 {
continue
}
// `ports` gives a module's port a machine port on one node (novox/hq ADR 0100),
// validated in GivenPorts, so it is not stray here either.
if key == PortsSetting {
continue
}
unused = append(unused, fmt.Sprintf(
"%s sets %q, and %s has no file or contribution to merge it into",
layer.From, key, m.Module))
@@ -0,0 +1,167 @@
package catalogue
import (
"encoding/json"
"testing"
)
// A module may answer one requirement more than once, the sibling of ADR 0094 for `contributes`
// rather than `secrets`: an object store's data API and its console are two different public
// names, not one. `contributes` maps a requirement to several sets of values under local names,
// each reaching the provider as its own entry — the same "several from one" shape ADR 0094 gave
// `secrets`, applied to the other half of an edge.
const twoRoutes = `{"module":"minio","version":"1","requires":["route"],
"contributes":{"route":{"api":{"label":"files-api","port":9000},"console":{"label":"files","port":9001}}}}`
func TestContributesReadsBothShapesAndWritesThemBack(t *testing.T) {
m, err := ParseManifest([]byte(twoRoutes))
if err != nil {
t.Fatal(err)
}
locals := m.ContributesMany["route"]
if len(locals) != 2 || locals["api"]["label"] != "files-api" || locals["console"]["port"] != float64(9001) {
t.Fatalf("two contributions under local names: %+v", locals)
}
plain, err := ParseManifest([]byte(`{"module":"board","version":"1","requires":["route"],
"contributes":{"route":{"label":"board","port":8080}}}`))
if err != nil {
t.Fatal(err)
}
if got := plain.Contributes["route"]; got["label"] != "board" || len(plain.ContributesMany) != 0 {
t.Fatalf("the plain shape is one contribution with no local names: %+v / %+v", got, plain.ContributesMany)
}
// Written back in the shape it was read, so a built manifest keeps its local names.
raw, err := json.Marshal(m)
if err != nil {
t.Fatal(err)
}
again, err := ParseManifest(raw)
if err != nil {
t.Fatalf("what was written does not read: %v\n%s", err, raw)
}
if len(again.ContributesMany["route"]) != 2 {
t.Fatalf("the local names did not survive a round trip:\n%s", raw)
}
}
func TestAContributionLocalNameMustBeUsable(t *testing.T) {
for _, bad := range []string{
// Not a usable name.
`{"module":"minio","version":"1","requires":["route"],
"contributes":{"route":{"Not OK":{"label":"files","port":9000}}}}`,
// A local contribution with nothing in it.
`{"module":"minio","version":"1","requires":["route"],
"contributes":{"route":{"api":{}}}}`,
} {
if _, err := ParseManifest([]byte(bad)); err == nil {
t.Errorf("accepted:\n%s", bad)
}
}
}
func minimalRouteProxy() Manifest {
return Manifest{Module: "route-proxy", Version: "1",
Provides: FromAnywhere("route"),
Receives: map[string]string{"route": "/var/lib/route-proxy/routes/mesh.json"},
}
}
func TestAModuleWithTwoRoutesGivesTheProviderTwoContributions(t *testing.T) {
minio, err := ParseManifest([]byte(twoRoutes))
if err != nil {
t.Fatal(err)
}
got, err := Resolve(shelf(minimalRouteProxy(), minio), []string{"route-proxy", "minio"}, workstation(), World{})
if err != nil {
t.Fatal(err)
}
out, err := got.Declaration(Rendering{})
if err != nil {
t.Fatal(err)
}
var given []Contribution
for _, r := range out {
if r["path"] != "/var/lib/route-proxy/routes/mesh.json" {
continue
}
var parsed struct {
Given []Contribution `json:"given"`
}
if err := json.Unmarshal([]byte(r["content"].(string)), &parsed); err != nil {
t.Fatal(err)
}
given = parsed.Given
}
if len(given) != 2 {
t.Fatalf("two named routes from one module are two contributions: %+v", given)
}
byPort := map[float64]string{}
for _, g := range given {
if g.From != "minio" {
t.Fatalf("both contributions are minio's: %+v", g)
}
port, _ := g.Values["port"].(float64)
label, _ := g.Values["label"].(string)
byPort[port] = label
}
if byPort[9000] != "files-api" || byPort[9001] != "files" {
t.Fatalf("the two routes did not both survive: %+v", given)
}
}
// A module with the ordinary, single-contribution shape resolves exactly as it did before —
// ContributesMany being empty must change nothing about it.
func TestASingleRouteStillResolvesTheOrdinaryWay(t *testing.T) {
got, err := Resolve(shelf(proxy(), published("board", "board", 8080)), []string{"board"}, workstation(), World{})
if err != nil {
t.Fatal(err)
}
given := received(t, mustDeclare(t, got))
if len(given) != 1 || given[0].From != "board" {
t.Fatalf("the plain shape regressed: %+v", given)
}
}
// ContributionsFrom is what mints the ONE pair credential a requiring module is granted
// (cmd/mesh-controller/plan.go's grantsFor) — a separate path from Declaration()'s raw file, and
// the one the two-routes test above never exercised. Where a module contributes several times,
// there is no single "the" value: settling to whichever sorts first would both misrepresent the
// grant and collide with that same contribution's own entry from contributions(), which is
// exactly the duplicate a live plan against minio surfaced (files-api appearing once with a
// credential, once without, while files got neither).
func TestContributionsFromHasNoSingleValueWhenAModuleContributesSeveralTimes(t *testing.T) {
minio, err := ParseManifest([]byte(twoRoutes))
if err != nil {
t.Fatal(err)
}
got, err := Resolve(shelf(minimalRouteProxy(), minio), []string{"route-proxy", "minio"}, workstation(), World{})
if err != nil {
t.Fatal(err)
}
values, asks, err := got.ContributionsFrom("route", "minio", nil)
if err != nil {
t.Fatal(err)
}
if !asks {
t.Fatal("minio still requires route, so it still asks")
}
if len(values) != 0 {
t.Fatalf("no single value represents two contributions, got %+v", values)
}
}
// The ordinary, single-contribution case is unchanged: exactly one match still settles to it.
func TestContributionsFromReturnsTheOneValueForAnOrdinaryContribution(t *testing.T) {
got, err := Resolve(shelf(proxy(), published("board", "board", 8080)), []string{"board"}, workstation(), World{})
if err != nil {
t.Fatal(err)
}
values, asks, err := got.ContributionsFrom("reverse-proxy", "board", nil)
if err != nil {
t.Fatal(err)
}
if !asks || values["host"] != "board" {
t.Fatalf("the ordinary single contribution should still settle to its own value: %+v", values)
}
}
+205
View File
@@ -0,0 +1,205 @@
package catalogue
import (
"encoding/json"
"strings"
"testing"
)
// A module may need several values from one provider that gives one per pair (novox/hq
// 04-ISSUES/069, ADR 0094): `secrets` maps a requirement to several files under local names, and
// each local name is a pair credential of its own — its own need, its own file, its own holder.
const twoSecrets = `{"module":"ca","version":"1","requires":["secret"],
"secrets":{"secret":{"root-key":"/var/lib/ca/root.key","root-pass":"/var/lib/ca/root.pass"}},
"resources":[{"id":"state","type":"directory","path":"/var/lib/ca","mode":"0700"}]}`
func TestSecretsReadBothShapesAndWriteThemBack(t *testing.T) {
m, err := ParseManifest([]byte(twoSecrets))
if err != nil {
t.Fatal(err)
}
files := m.SecretFiles("secret")
if len(files) != 2 || files[0].Local != "root-key" || files[1].Path != "/var/lib/ca/root.pass" {
t.Fatalf("two files under local names, in order: %+v", files)
}
plain, err := ParseManifest([]byte(`{"module":"app","version":"1","requires":["secret"],"secrets":{"secret":"/var/lib/app/secret"}}`))
if err != nil {
t.Fatal(err)
}
if got := plain.SecretFiles("secret"); len(got) != 1 || got[0].Local != "" || got[0].Path != "/var/lib/app/secret" {
t.Fatalf("the plain shape is one file with no local name: %+v", got)
}
// Written back in the shape it was read, so a built manifest keeps its local names.
raw, err := json.Marshal(m)
if err != nil {
t.Fatal(err)
}
again, err := ParseManifest(raw)
if err != nil {
t.Fatalf("what was written does not read: %v\n%s", err, raw)
}
if len(again.SecretFiles("secret")) != 2 {
t.Fatalf("the local names did not survive a round trip:\n%s", raw)
}
}
func TestALocalNameMayNotCollideWithWhatTheModuleAlreadyCallsSomething(t *testing.T) {
for _, bad := range []string{
// One of the module's own secrets.
`{"module":"ca","version":"1","requires":["secret"],"own-secrets":{"root-key":"/var/lib/ca/own"},
"secrets":{"secret":{"root-key":"/var/lib/ca/root.key"}}}`,
// Something it requires.
`{"module":"ca","version":"1","requires":["secret","postgres-database"],
"secrets":{"secret":{"postgres-database":"/var/lib/ca/x"}}}`,
// Not a usable name.
`{"module":"ca","version":"1","requires":["secret"],"secrets":{"secret":{"Root Key":"/var/lib/ca/x"}}}`,
// A relative path.
`{"module":"ca","version":"1","requires":["secret"],"secrets":{"secret":{"root-key":"root.key"}}}`,
} {
if _, err := ParseManifest([]byte(bad)); err == nil {
t.Errorf("accepted:\n%s", bad)
}
}
}
func vaultAndCA() map[string]Manifest {
ca, _ := ParseManifest([]byte(twoSecrets))
vault := Manifest{Module: "mesh-vault", Version: "1", Provides: FromAnywhere("secret"),
Grants: map[string]string{"secret": "/var/lib/vault/grants"},
Receives: map[string]string{"secret": "/var/lib/vault/grants/mesh.json"}}
// A second consumer of the same provision that keeps ONE file, mentioned before the one that
// keeps two: the lab found the expansion done on the first module to mention the provision,
// and the second consumer given one credential and no file.
cache := Manifest{Module: "cache", Version: "1", Requires: []string{"secret"},
Secrets: map[string]string{"secret": "/var/lib/cache/secret"}}
return shelf(vault, cache, ca)
}
func TestEachLocalNameIsANeedAFileAndAHolderOfItsOwn(t *testing.T) {
got, err := Resolve(vaultAndCA(), []string{"mesh-vault", "cache", "ca"}, workstation(), World{})
if err != nil {
t.Fatal(err)
}
var locals, cacheLocals []string
for _, n := range got.Needs {
if n.Name == "secret" && n.For == "ca" {
locals = append(locals, n.Local)
}
if n.Name == "secret" && n.For == "cache" {
cacheLocals = append(cacheLocals, n.Local)
}
}
if strings.Join(locals, ",") != "root-key,root-pass" {
t.Fatalf("two secrets from one provider are two needs: %v", got.Needs)
}
if len(cacheLocals) != 1 || cacheLocals[0] != "" {
t.Fatalf("the one-file consumer keeps one need with no local name: %v", got.Needs)
}
for i := range got.Needs {
got.Needs[i].Sealed = "sealed-" + got.Needs[i].Local
}
out, err := got.Declaration(Rendering{})
if err != nil {
t.Fatal(err)
}
seen := map[string]string{}
for _, r := range out {
if r["type"] == "file" && strings.HasPrefix(r["path"].(string), "/var/lib/ca/root.") {
seen[r["id"].(string)] = r["sealed"].(string)
}
}
if seen["ca."+SecretID("root-key")] != "sealed-root-key" || seen["ca."+SecretID("root-pass")] != "sealed-root-pass" {
t.Fatalf("each local name is its own file with its own credential: %v", seen)
}
}
func TestAProviderSeesEachLocalNameAsAHolderOfItsOwn(t *testing.T) {
r := Resolution{Modules: []Manifest{vaultAndCA()["mesh-vault"]}}
got, err := r.contributions(SettingsBy{}, []Grant{
{Provision: "secret", Consumer: "workstation", From: "ca", Local: "root-key", Sealed: "x"},
{Provision: "secret", Consumer: "workstation", From: "ca", Local: "root-pass", Sealed: "y"},
}, map[string]string{"secret": "/var/lib/vault/grants"})
if err != nil {
t.Fatal(err)
}
given := got["secret"]
if len(given) != 2 {
t.Fatalf("two holders: %+v", given)
}
if given[0].As != "mesh_workstation_ca_root_key" && given[0].As != "mesh_workstation_ca_root-key" {
t.Fatalf("the holder is the consumer's identity with the local name after it: %q", given[0].As)
}
if given[0].Secret == given[1].Secret {
t.Fatalf("two holders share one file on the provider: %q", given[0].Secret)
}
}
// And on the provider's machine, two files with two ids — the lab's first run had the declaration
// refused for two resources with one identity.
func TestAProviderKeepsOneFilePerHolder(t *testing.T) {
got, err := Resolve(vaultAndCA(), []string{"mesh-vault", "cache", "ca"}, workstation(), World{})
if err != nil {
t.Fatal(err)
}
out, err := got.Declaration(Rendering{Grants: []Grant{
{Provision: "secret", Consumer: "workstation", From: "ca", Local: "root-key", Sealed: "x"},
{Provision: "secret", Consumer: "workstation", From: "ca", Local: "root-pass", Sealed: "y"},
}})
if err != nil {
t.Fatal(err)
}
ids := map[string]string{}
for _, r := range out {
if id, _ := r["id"].(string); strings.Contains(id, "grant-secret") {
ids[id] = r["path"].(string)
}
}
if len(ids) != 2 {
t.Fatalf("two holders are two grant files: %v", ids)
}
}
// One file under a local name is still a local name (review C1): the need carries it, the file
// is written, and ${secret:<name>} is filled.
func TestOneLocalNameIsStillALocalName(t *testing.T) {
only, _ := ParseManifest([]byte(`{"module":"one","version":"1","requires":["secret"],
"secrets":{"secret":{"only":"/var/lib/one/only"}}}`))
vault := vaultAndCA()["mesh-vault"]
got, err := Resolve(shelf(vault, only), []string{"mesh-vault", "one"}, workstation(), World{})
if err != nil {
t.Fatal(err)
}
var found *Needed
for i, n := range got.Needs {
if n.For == "one" && n.Name == "secret" {
found = &got.Needs[i]
}
}
if found == nil || found.Local != "only" {
t.Fatalf("the one named file did not become a need under its name: %v", got.Needs)
}
found.Sealed = "sealed-only"
out, err := got.Declaration(Rendering{})
if err != nil {
t.Fatal(err)
}
var written bool
for _, r := range out {
if r["path"] == "/var/lib/one/only" && r["sealed"] == "sealed-only" {
written = true
}
}
if !written {
t.Fatal("the file under the one local name was not written")
}
}
// A local name names one credential: two requirements may not share it (review C2).
func TestALocalNameIsUniqueAcrossRequirements(t *testing.T) {
_, err := ParseManifest([]byte(`{"module":"x","version":"1","requires":["secret","postgres-database"],
"secrets":{"secret":{"x":"/var/lib/x/a"},"postgres-database":{"x":"/var/lib/x/b"}}}`))
if err == nil || !strings.Contains(err.Error(), "both under") {
t.Fatalf("two requirements under one local name were accepted: %v", err)
}
}
+146
View File
@@ -0,0 +1,146 @@
package envfile
import (
"fmt"
"net"
"os"
"regexp"
"strconv"
"strings"
)
// The port a machine put something on, said beside the value that names it.
//
// **An address written down when a port was decided does not follow the port** (novox/hq
// 04-ISSUES/102). The control plane's own store and broker connections are written at genesis —
// full connection strings, password and all — and sealed, so the mesh cannot open them to move the
// port inside. When the node's settings move that port, every consumer's binding follows and the
// control plane's own connection does not: it goes on dialling the number genesis wrote, and the
// mesh is headless.
//
// So a setting has a third twin. `NAME_FILE` says where the value is; `NAME_PORT` says which port
// this machine put the thing at, composed into the control plane's environment from the node's
// settings exactly as a consumer's binding is. The value's own port is what stands when the twin
// says nothing — a mesh whose foundation is still where genesis raised it needs no override, and
// an empty answer is the mesh saying it has nothing to add, not the mesh saying zero.
//
// Only the port. The host inside the value is the machine's own loopback, or the broker's public
// name — a fact of the genesis, not of any node's settings — and the mesh has no better opinion
// of it. What moves under ADR 0100 is the port.
// PortVar is the twin saying which port this machine put the named thing at.
func PortVar(name string) string { return name + "_PORT" }
// Port is what NAME_PORT says, checked to be a port, or "" when it says nothing.
//
// Blank counts as unset, as it does for every other variable here: a container given an empty
// string was given nothing, and refusing it as ambiguous would turn an omission into a puzzle.
func Port(name string) (string, error) {
raw := strings.TrimSpace(os.Getenv(PortVar(name)))
if raw == "" {
return "", nil
}
if unfilled.MatchString(raw) {
// **A placeholder the mesh never filled, and this is the one place it must not be a
// fault.** The control plane composes its own declaration, so a manifest naming
// `${seat:…}` reaches a control plane one build older than the manifest — one that does
// not know the placeholder and passes it through as the value. Refusing it here would
// leave every command and the daemon unable to open a store: headless, on the machine
// that cannot be repaired through the mesh (novox/hq 04-ISSUES/102). So it is what an
// empty answer is — nothing said, the sealed value stands — and it is said aloud, because
// a manifest ahead of its binary is worth a line on stderr and not worth an outage.
fmt.Fprintf(os.Stderr, "%s is %q, a placeholder nothing filled in — ignored; the port in "+
"%s stands. The control plane composing this declaration is older than the manifest "+
"naming it: rebuild and push it\n", PortVar(name), raw, name)
return "", nil
}
n, err := strconv.Atoi(raw)
if err != nil || n < 1 || n > 65535 {
return "", fmt.Errorf("%s is %q, which is not a port", PortVar(name), raw)
}
return strconv.Itoa(n), nil
}
// Placed is Value, with its port moved to where NAME_PORT says this machine put it.
func Placed(name string) (string, error) {
value, err := Value(name)
if err != nil {
return "", err
}
port, err := Port(name)
if err != nil || port == "" {
return value, err
}
placed, err := WithPort(value, port)
if err != nil {
// Where it came from is said; what it says is not. The value is a connection string with
// a password in it, and every error here is written on the assumption it will be logged.
return "", fmt.Errorf("%s names a port to move %s to, and %s could not be read as "+
"something with a port in it: %w", PortVar(name), name, name, err)
}
return placed, nil
}
// keywordPort is `port=…` in a `key=value` connection string.
var keywordPort = regexp.MustCompile(`(^|\s)port=\S*`)
// unfilled is a value that is still a placeholder — `${…}` — rather than something a person or the
// mesh wrote as a port.
var unfilled = regexp.MustCompile(`^\$\{[^}]*\}$`)
// WithPort is the value with its port replaced by `port`.
//
// Three shapes, because the control plane's settings come in three: a URL
// (`scheme://[user:password@]host[:port][/…]`), a bare `host[:port]` (the broker's address) and
// a `key=value` connection string (`host=… port=…`), which is what the store's driver accepts
// beside a URL. An IPv6 host stays in its brackets. Nothing here parses the URL properly — a
// password with a character a URL parser dislikes is still a working connection string, and
// refusing it would refuse a value that has been dialling fine since genesis.
func WithPort(value, port string) (string, error) {
value = strings.TrimSpace(value)
if value == "" {
return "", fmt.Errorf("the value is empty")
}
if scheme, rest, isURL := strings.Cut(value, "://"); isURL {
// **The password may hold any of `/ ? # @`, so the host begins after the LAST `@`**, and
// the path only after that. Cutting at the first `/` would take a password's slash for the
// path's and put the new port on the user name — a connection string that dials the wrong
// host without a word. Genesis makes passwords that cannot do this; an accepted one can.
// The connection strings this reads — a store's, a broker's, a management API's — carry
// no `@` after their userinfo, which is what makes the last one the boundary.
userinfo, hostAndTail := "", rest
if at := strings.LastIndex(rest, "@"); at >= 0 {
userinfo, hostAndTail = rest[:at+1], rest[at+1:]
}
authority, tail := hostAndTail, ""
if end := strings.IndexAny(hostAndTail, "/?#"); end >= 0 {
authority, tail = hostAndTail[:end], hostAndTail[end:]
}
host, err := hostWithPort(authority, port)
if err != nil {
return "", err
}
return scheme + "://" + userinfo + host + tail, nil
}
if strings.Contains(value, "=") && !strings.ContainsAny(value, "/") {
if keywordPort.MatchString(value) {
return keywordPort.ReplaceAllString(value, "${1}port="+port), nil
}
return value + " port=" + port, nil
}
return hostWithPort(value, port)
}
// hostWithPort is `host[:port]` with the port set, brackets kept around an IPv6 host.
func hostWithPort(authority, port string) (string, error) {
if authority == "" {
return "", fmt.Errorf("there is no host to put a port on")
}
host, _, err := net.SplitHostPort(authority)
if err != nil {
// No port yet. A bracketed IPv6 host without a port is a shape SplitHostPort refuses, and
// a bare one carries colons of its own — both are a host, not an error.
host = strings.TrimSuffix(strings.TrimPrefix(authority, "["), "]")
}
return net.JoinHostPort(host, port), nil
}
+78
View File
@@ -0,0 +1,78 @@
package envfile
import (
"os"
"path/filepath"
"strings"
"testing"
)
// The control plane's own connections follow the port the node moved (novox/hq 04-ISSUES/102).
func TestAPortTwinMovesTheValuesPort(t *testing.T) {
cases := map[string]string{
"postgres://mesh:s3cret@127.0.0.1:5432/inventory?sslmode=disable": "postgres://mesh:s3cret@127.0.0.1:6852/inventory?sslmode=disable",
"postgres://mesh:s3cret@127.0.0.1/inventory": "postgres://mesh:s3cret@127.0.0.1:6852/inventory",
"amqp://control:p%40ss@127.0.0.1:5672/": "amqp://control:p%40ss@127.0.0.1:6852/",
"amqp://control:p@ss:with@127.0.0.1:5672/": "amqp://control:p@ss:with@127.0.0.1:6852/",
"http://guest:guest@127.0.0.1:15672": "http://guest:guest@127.0.0.1:6852",
"postgres://mesh:a/b?c#d@e@127.0.0.1:5432/inventory": "postgres://mesh:a/b?c#d@e@127.0.0.1:6852/inventory",
"http://[::1]:15672/api": "http://[::1]:6852/api",
"broker.example:5671": "broker.example:6852",
"broker.example": "broker.example:6852",
"host=127.0.0.1 port=5432 dbname=inventory": "host=127.0.0.1 port=6852 dbname=inventory",
"host=127.0.0.1 dbname=inventory": "host=127.0.0.1 dbname=inventory port=6852",
}
for value, want := range cases {
got, err := WithPort(value, "6852")
if err != nil || got != want {
t.Errorf("WithPort(%q) = %q, %v; want %q", value, got, err, want)
}
}
}
func TestPlacedLeavesTheValueAloneWhenNothingSaysAPort(t *testing.T) {
path := filepath.Join(t.TempDir(), "value")
if err := os.WriteFile(path, []byte("postgres://m:p@127.0.0.1:5432/inventory\n"), 0o600); err != nil {
t.Fatal(err)
}
t.Setenv("MESH_P_FILE", path)
t.Setenv("MESH_P_PORT", "")
got, err := Placed("MESH_P")
if err != nil || got != "postgres://m:p@127.0.0.1:5432/inventory" {
t.Fatalf("got %q, %v", got, err)
}
t.Setenv("MESH_P_PORT", " 6852 ")
got, err = Placed("MESH_P")
if err != nil || got != "postgres://m:p@127.0.0.1:6852/inventory" {
t.Fatalf("got %q, %v", got, err)
}
}
func TestAPortTwinThatIsNotAPortIsRefusedWithoutQuotingTheValue(t *testing.T) {
t.Setenv("MESH_Q", "postgres://m:hunter2@127.0.0.1:5432/inventory")
t.Setenv("MESH_Q_PORT", "many")
_, err := Placed("MESH_Q")
if err == nil {
t.Fatal("a port that is not a number was accepted")
}
if strings.Contains(err.Error(), "hunter2") {
t.Fatalf("the value was quoted back: %v", err)
}
t.Setenv("MESH_Q", "")
t.Setenv("MESH_Q_PORT", "6852")
if _, err := Placed("MESH_Q"); err == nil {
t.Fatal("a port with no value to put it on was accepted")
}
}
// A placeholder nothing filled is nothing said, not a fault: the control plane composing the
// declaration may be one build behind the manifest, and refusing would leave it headless.
func TestAnUnfilledPlaceholderIsNothingSaid(t *testing.T) {
t.Setenv("MESH_R", "postgres://m:p@127.0.0.1:5432/inventory")
t.Setenv("MESH_R_PORT", "${seat:mesh-store:5432}")
got, err := Placed("MESH_R")
if err != nil || got != "postgres://m:p@127.0.0.1:5432/inventory" {
t.Fatalf("an unfilled placeholder was not ignored: %q, %v", got, err)
}
}
+68
View File
@@ -0,0 +1,68 @@
package identity
import (
"context"
"fmt"
"os"
"strings"
"testing"
"time"
"github.com/jackc/pgx/v5"
"github.com/novox/mesh-controller/internal/store"
)
// ForTest is a fresh, migrated identity store in a database of its own, dropped when the test
// ends. Exported for the same reason inventory.ForTest is: the check that a node's signed word
// is verified against the key the mesh recorded lives beside the link, and a second copy of this
// would be a second thing to keep true. It takes a *testing.T, so nothing that is not a test can
// call it.
func ForTest(t *testing.T) *Identity {
t.Helper()
admin := os.Getenv("MESH_TEST_POSTGRES")
if admin == "" {
t.Skip("no MESH_TEST_POSTGRES; run `make check` to raise one")
}
name := fmt.Sprintf("ident_%d_%s", time.Now().UnixNano()%1_000_000,
strings.ToLower(strings.NewReplacer("/", "", "-", "").Replace(t.Name())))
if len(name) > 60 {
name = name[:60]
}
conn, err := pgx.Connect(t.Context(), admin)
if err != nil {
t.Fatalf("cannot reach the test PostgreSQL: %v", err)
}
if _, err := conn.Exec(t.Context(), "create database "+name); err != nil {
t.Fatalf("cannot create %s: %v", name, err)
}
conn.Close(t.Context())
cut := strings.LastIndex(admin, "/")
t.Setenv(store.Variable(Name), admin[:cut]+"/"+name+"?sslmode=disable")
ident, err := Open(t.Context())
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() {
ident.Close()
c, err := pgx.Connect(context.Background(), admin)
if err != nil {
return
}
defer c.Close(context.Background())
_, _ = c.Exec(context.Background(), "drop database if exists "+name+" with (force)")
})
if err := ident.Ready(t.Context(), 20*time.Second); err != nil {
t.Fatal(err)
}
migrations, err := Migrations()
if err != nil {
t.Fatal(err)
}
if _, err := ident.store.Migrate(t.Context(), migrations); err != nil {
t.Fatal(err)
}
return ident
}
+54 -12
View File
@@ -16,25 +16,64 @@ import (
// node's peer list to chase it, and an address that moves is the thing declaring the hub was
// meant to stop.
//
// Allocated in order from the range, taking the lowest free one. Not random: a person reading a
// peer list should be able to guess which node an address belongs to, and reuse of a released
// address is a smaller problem than a list nobody can hold in their head.
// **The adopted tunnel's addresses come first** (novox/hq ADR 0105). The hub that took over a
// tunnel is at the tunnel's own address. A node enrolling with a key the tunnel already routed
// to keeps the address the tunnel had for it — nothing a peer knows changes. And an address the
// tunnel holds for a peer that has not enrolled is never handed to anyone else: that peer is
// still reaching the hub at it.
//
// The rest is allocated in order from the range, taking the lowest free one. Not random: a person
// reading a peer list should be able to guess which node an address belongs to, and reuse of a
// released address is a smaller problem than a list nobody can hold in their head.
func (i *Inventory) AssignAddress(ctx context.Context, node, cidr string) (string, error) {
prefix, err := netip.ParsePrefix(cidr)
if err != nil {
return "", fmt.Errorf("%q is not a network the mesh can allocate from: %w", cidr, err)
}
var existing *string
var existing, key *string
var name string
var hub bool
if err := i.store.Pool().QueryRow(ctx,
`select host(overlay_address) from node where id = $1`, node).Scan(&existing); err != nil {
`select name, host(overlay_address), overlay_key, is_hub from node where id = $1`, node).
Scan(&name, &existing, &key, &hub); err != nil {
return "", err
}
if existing != nil && *existing != "" {
return *existing, nil
}
tunnel, hubName, adopted, err := i.AdoptedTunnel(ctx)
if err != nil {
return "", err
}
if adopted && hub && hubName == name {
address, err := netip.ParsePrefix(tunnel.Address)
if err != nil {
return "", fmt.Errorf("the adopted tunnel's address %q: %w", tunnel.Address, err)
}
return i.place(ctx, node, address.Addr().String())
}
carried, err := i.CarriedPeers(ctx)
if err != nil {
return "", err
}
taken := map[string]bool{}
if adopted {
// The tunnel's own address is the hub's whether or not the hub has been placed yet.
if address, err := netip.ParsePrefix(tunnel.Address); err == nil {
taken[address.Addr().String()] = true
}
}
for _, p := range carried {
if key != nil && p.PublicKey == *key {
// The tunnel already routes to this key: the node keeps that address, and the peer
// notices nothing when its machine enrols.
return i.place(ctx, node, p.Address)
}
taken[p.Address] = true
}
rows, err := i.store.Pool().Query(ctx,
`select host(overlay_address) from node where overlay_address is not null`)
if err != nil {
@@ -58,12 +97,7 @@ func (i *Inventory) AssignAddress(ctx context.Context, node, cidr string) (strin
candidate := prefix.Masked().Addr().Next()
for prefix.Contains(candidate) {
if !taken[candidate.String()] {
if _, err := i.store.Pool().Exec(ctx,
`update node set overlay_address = $2::inet where id = $1`,
node, candidate.String()); err != nil {
return "", err
}
return candidate.String(), nil
return i.place(ctx, node, candidate.String())
}
candidate = candidate.Next()
}
@@ -72,5 +106,13 @@ func (i *Inventory) AssignAddress(ctx context.Context, node, cidr string) (strin
// halfway through assigning one node.
return "", fmt.Errorf(
"every address in %s is taken, so %s cannot be given one. The mesh has outgrown its "+
"range and renumbering it is a deliberate act", cidr, node)
"range and renumbering it is a deliberate act", cidr, name)
}
func (i *Inventory) place(ctx context.Context, node, address string) (string, error) {
if _, err := i.store.Pool().Exec(ctx,
`update node set overlay_address = $2::inet where id = $1`, node, address); err != nil {
return "", err
}
return address, nil
}
+246
View File
@@ -0,0 +1,246 @@
package inventory
import (
"context"
"encoding/json"
"errors"
"fmt"
"sort"
"time"
"github.com/jackc/pgx/v5"
)
// A node is adopted or converged (novox/hq ADR 0100).
//
// Adopted: what is found on the machine is kept until its module is taken, and the firewall found
// there stays in force. Converged: the machine is what the mesh declares, as every node was before
// adoption existed. The controller is authoritative, and every declaration it sends says which.
// ErrNotAdopted is taking a module on a node that is converged. On a converged node every assigned
// module converges already; there is nothing to take.
var ErrNotAdopted = errors.New("the node is converged, so every module on it is taken already")
// ErrNotAssigned is taking a module that is not on the node. Taking is the cutover of a module
// the node runs; one it does not run has nothing to cut over.
var ErrNotAssigned = errors.New("that module is not assigned to the node")
// SetAdopted makes a node adopted or converged. Becoming adopted stamps when; converging stamps
// when too. Neither touches what was taken: what was taken stays taken when a node returns to
// adopted, and converging takes the rest by its own act.
func (i *Inventory) SetAdopted(ctx context.Context, name string, adopted bool) error {
node, err := i.NodeByName(ctx, name)
if err != nil {
return err
}
if node.Adopted == adopted {
return nil
}
if adopted {
_, err = i.store.Pool().Exec(ctx,
`update node set adopted = true, adopted_since = now() where id = $1`, node.ID)
return err
}
_, err = i.store.Pool().Exec(ctx,
`update node set adopted = false, adopted_since = null, converged_at = now() where id = $1`,
node.ID)
return err
}
// Take records that a module has been taken on an adopted node: its cutover. From then on the
// module's resources converge on that node like any other, replacing what was found.
//
// Refused on a converged node and for a module not assigned there. Taking again is not an error;
// the first time it was taken is kept.
func (i *Inventory) Take(ctx context.Context, nodeName, module string) error {
node, err := i.NodeByName(ctx, nodeName)
if err != nil {
return err
}
if !node.Adopted {
return fmt.Errorf("%w: %s", ErrNotAdopted, nodeName)
}
return i.take(ctx, node, module)
}
// take is Take without the adopted check, for converging, which takes every assigned module in
// the same act that makes the node converged.
func (i *Inventory) take(ctx context.Context, node Node, module string) error {
var assigned bool
if err := i.store.Pool().QueryRow(ctx,
`select exists (select 1 from assignment where node = $1 and module = $2)`,
node.ID, module).Scan(&assigned); err != nil {
return err
}
if !assigned {
return fmt.Errorf("%w: %s is not on %s; assign it first", ErrNotAssigned, module, node.Name)
}
_, err := i.store.Pool().Exec(ctx,
`insert into taken (node, module) values ($1, $2) on conflict do nothing`, node.ID, module)
return err
}
// Converge makes an adopted node converged in one act: every module assigned there is taken, and
// the node is recorded converged. Returned is what this act took, in name order.
func (i *Inventory) Converge(ctx context.Context, nodeName string) ([]string, error) {
node, err := i.NodeByName(ctx, nodeName)
if err != nil {
return nil, err
}
if !node.Adopted {
return nil, fmt.Errorf("%s is converged already", nodeName)
}
tx, err := i.store.Pool().Begin(ctx)
if err != nil {
return nil, err
}
defer func() { _ = tx.Rollback(context.WithoutCancel(ctx)) }()
rows, err := tx.Query(ctx,
`insert into taken (node, module)
select node, module from assignment where node = $1
on conflict do nothing
returning module`, node.ID)
if err != nil {
return nil, err
}
var took []string
for rows.Next() {
var m string
if err := rows.Scan(&m); err != nil {
rows.Close()
return nil, err
}
took = append(took, m)
}
rows.Close()
if err := rows.Err(); err != nil {
return nil, err
}
if _, err := tx.Exec(ctx,
`update node set adopted = false, adopted_since = null, converged_at = now(),
held = null, reachable = null, firewall = null, adoption_reported = null
where id = $1`,
node.ID); err != nil {
return nil, err
}
if err := tx.Commit(ctx); err != nil {
return nil, err
}
sort.Strings(took)
return took, nil
}
// Taken is every module taken on a node, in name order — including one no longer assigned there:
// unassigning does not un-take.
func (i *Inventory) Taken(ctx context.Context, nodeName string) ([]string, error) {
node, err := i.NodeByName(ctx, nodeName)
if err != nil {
return nil, err
}
rows, err := i.store.Pool().Query(ctx,
`select module from taken where node = $1 order by module`, node.ID)
if err != nil {
return nil, err
}
defer rows.Close()
var out []string
for rows.Next() {
var m string
if err := rows.Scan(&m); err != nil {
return nil, err
}
out = append(out, m)
}
return out, rows.Err()
}
// Held is one file or container an adopted node found and keeps as it was until its module is
// taken. The node's own account, kept as it said it.
type Held struct {
ID string `json:"id"`
Module string `json:"module"`
Kind string `json:"kind"`
Target string `json:"target"`
Since time.Time `json:"since"`
Changed string `json:"changed,omitempty"`
Kept string `json:"kept,omitempty"`
}
// Reach is one thing reachable on an adopted node: a listening socket or a published port.
type Reach struct {
Protocol string `json:"protocol"`
Address string `json:"address"`
Port int `json:"port"`
By string `json:"by,omitempty"`
Published bool `json:"published,omitempty"`
ContainerPort int `json:"container-port,omitempty"`
}
// Adoption is what an adopted node last said about adoption, and when.
type Adoption struct {
Held []Held
Firewall string
Reachable []Reach
// At is when it said so; zero when it never has.
At time.Time
}
// RecordAdoption keeps what a node last reported about adoption, replacing what was there: the
// question is the machine as it is now.
func (i *Inventory) RecordAdoption(ctx context.Context, node string, held []Held, firewall string,
reachable []Reach) error {
heldRaw, err := json.Marshal(nonNil(held))
if err != nil {
return err
}
reachRaw, err := json.Marshal(nonNil(reachable))
if err != nil {
return err
}
_, err = i.store.Pool().Exec(ctx,
`update node set held = $2, firewall = nullif($3, ''), reachable = $4,
adoption_reported = now(), last_seen = now()
where id = $1`, node, heldRaw, firewall, reachRaw)
return err
}
func nonNil[T any](s []T) []T {
if s == nil {
return []T{}
}
return s
}
// AdoptionOf is what a node last reported about adoption.
func (i *Inventory) AdoptionOf(ctx context.Context, name string) (Adoption, error) {
var heldRaw, reachRaw []byte
var firewall *string
var at *time.Time
err := i.store.Pool().QueryRow(ctx,
`select held, firewall, reachable, adoption_reported from node where name = $1`, name).
Scan(&heldRaw, &firewall, &reachRaw, &at)
if errors.Is(err, pgx.ErrNoRows) {
return Adoption{}, fmt.Errorf("%w: %s", ErrNoSuchNode, name)
}
if err != nil {
return Adoption{}, err
}
var out Adoption
if firewall != nil {
out.Firewall = *firewall
}
if at != nil {
out.At = *at
}
if len(heldRaw) > 0 {
if err := json.Unmarshal(heldRaw, &out.Held); err != nil {
return Adoption{}, err
}
}
if len(reachRaw) > 0 {
if err := json.Unmarshal(reachRaw, &out.Reachable); err != nil {
return Adoption{}, err
}
}
return out, nil
}
+161
View File
@@ -0,0 +1,161 @@
package inventory
import (
"errors"
"reflect"
"testing"
)
// novox/hq ADR 0100: a node is adopted or converged, and the controller records which.
func TestANodeAddedWithoutSayingIsConverged(t *testing.T) {
inv := fresh(t)
if _, err := inv.AddNode(t.Context(), "anchor"); err != nil {
t.Fatal(err)
}
n, err := inv.NodeByName(t.Context(), "anchor")
if err != nil {
t.Fatal(err)
}
if n.Adopted || !n.AdoptedSince.IsZero() {
t.Fatalf("a node nobody said anything about reads as adopted: %+v", n)
}
}
func TestAnAdoptedNodeRoundTripsThroughEveryReading(t *testing.T) {
inv := fresh(t)
made, err := inv.AddNodeAs(t.Context(), "anchor", true)
if err != nil {
t.Fatal(err)
}
if !made.Adopted || made.AdoptedSince.IsZero() {
t.Fatalf("added adopted, got %+v", made)
}
byName, err := inv.NodeByName(t.Context(), "anchor")
if err != nil {
t.Fatal(err)
}
all, err := inv.Nodes(t.Context())
if err != nil {
t.Fatal(err)
}
if !byName.Adopted || len(all) != 1 || !all[0].Adopted {
t.Fatalf("adoption did not survive reading back: %+v %+v", byName, all)
}
// And through a token: enrolment reads the node from the token it spends.
issued, err := inv.IssueToken(t.Context(), "anchor", 60e9)
if err != nil {
t.Fatal(err)
}
claimed, err := inv.Claim(t.Context(), issued.Secret, "a-key", false)
if err != nil {
t.Fatal(err)
}
if !claimed.Adopted {
t.Fatal("the node a token claims lost its mode")
}
}
func TestTakingIsRefusedOnAConvergedNodeAndForAnUnassignedModule(t *testing.T) {
inv := fresh(t)
if err := inv.RegisterModule(t.Context(), manifest("hello-web", nil, nil), Source{}); err != nil {
t.Fatal(err)
}
if _, err := inv.AddNode(t.Context(), "converged"); err != nil {
t.Fatal(err)
}
if err := inv.Assign(t.Context(), "converged", "hello-web"); err != nil {
t.Fatal(err)
}
if err := inv.Take(t.Context(), "converged", "hello-web"); !errors.Is(err, ErrNotAdopted) {
t.Fatalf("taking on a converged node gave %v", err)
}
if _, err := inv.AddNodeAs(t.Context(), "anchor", true); err != nil {
t.Fatal(err)
}
if err := inv.Take(t.Context(), "anchor", "hello-web"); !errors.Is(err, ErrNotAssigned) {
t.Fatalf("taking an unassigned module gave %v", err)
}
}
func TestATakenModuleOutlivesItsAssignmentAndReturningToAdopted(t *testing.T) {
inv := fresh(t)
for _, m := range []string{"hello-web", "postgres"} {
if err := inv.RegisterModule(t.Context(), manifest(m, nil, nil), Source{}); err != nil {
t.Fatal(err)
}
}
if _, err := inv.AddNodeAs(t.Context(), "anchor", true); err != nil {
t.Fatal(err)
}
for _, m := range []string{"hello-web", "postgres"} {
if err := inv.Assign(t.Context(), "anchor", m); err != nil {
t.Fatal(err)
}
}
if err := inv.Take(t.Context(), "anchor", "postgres"); err != nil {
t.Fatal(err)
}
if err := inv.Take(t.Context(), "anchor", "postgres"); err != nil {
t.Fatalf("taking twice is not an error: %v", err)
}
if err := inv.Unassign(t.Context(), "anchor", "postgres"); err != nil {
t.Fatal(err)
}
taken, err := inv.Taken(t.Context(), "anchor")
if err != nil {
t.Fatal(err)
}
if !reflect.DeepEqual(taken, []string{"postgres"}) {
t.Fatalf("unassigning un-took it: %v", taken)
}
took, err := inv.Converge(t.Context(), "anchor")
if err != nil {
t.Fatal(err)
}
if !reflect.DeepEqual(took, []string{"hello-web"}) {
t.Fatalf("converging took %v; it takes every assigned module not yet taken", took)
}
n, _ := inv.NodeByName(t.Context(), "anchor")
if n.Adopted {
t.Fatal("converged node still reads adopted")
}
if err := inv.SetAdopted(t.Context(), "anchor", true); err != nil {
t.Fatal(err)
}
taken, _ = inv.Taken(t.Context(), "anchor")
if !reflect.DeepEqual(taken, []string{"hello-web", "postgres"}) {
t.Fatalf("returning to adopted lost what was taken: %v", taken)
}
}
// Converging clears the whole of a node's account of itself: what it held, what was reachable, the
// firewall it found and when it said so. Keeping any of it would have `node show` report an
// adopted machine's account of a converged one.
func TestConvergingClearsTheAccountTheNodeGave(t *testing.T) {
inv := fresh(t)
ctx := t.Context()
made, err := inv.AddNodeAs(ctx, "anchor", true)
if err != nil {
t.Fatal(err)
}
if err := inv.RecordAdoption(ctx, made.ID,
[]Held{{ID: "notes.conf", Module: "notes", Kind: "file", Target: "/etc/notes.conf"}},
"ufw", []Reach{{Protocol: "tcp", Address: "0.0.0.0", Port: 22, By: "sshd"}}); err != nil {
t.Fatal(err)
}
if _, err := inv.Converge(ctx, "anchor"); err != nil {
t.Fatal(err)
}
said, err := inv.AdoptionOf(ctx, "anchor")
if err != nil {
t.Fatal(err)
}
if len(said.Held) != 0 || len(said.Reachable) != 0 || said.Firewall != "" || !said.At.IsZero() {
t.Fatalf("converging kept the adopted machine's account: %+v", said)
}
}
+162 -3
View File
@@ -5,6 +5,8 @@ import (
"encoding/json"
"errors"
"fmt"
"sort"
"strconv"
"strings"
"github.com/jackc/pgx/v5"
@@ -583,6 +585,17 @@ func (i *Inventory) SetSettings(ctx context.Context, nodeName, module string, va
return err
}
if nodeName == "" {
// A port is a fact about one machine (novox/hq ADR 0100). Refused here, in composition's
// words: stored, it refuses every node running the module at composition, and the mesh
// cannot be pushed at all until somebody finds the layer that did it.
given, err := givenIn(module, raw)
if err != nil {
return err
}
if len(given) > 0 {
return fmt.Errorf("%s: %s is given per node — a port is a fact about one machine; "+
"set it with --node", module, catalogue.PortsSetting)
}
_, err = i.store.Pool().Exec(ctx,
`insert into settings (node, module, values) values (null, $1, $2)
on conflict (module) where node is null
@@ -593,11 +606,157 @@ func (i *Inventory) SetSettings(ctx context.Context, nodeName, module string, va
if err != nil {
return err
}
_, err = i.store.Pool().Exec(ctx,
given, err := givenIn(module, raw)
if err != nil {
return err
}
tx, err := i.store.Pool().Begin(ctx)
if err != nil {
return err
}
defer func() { _ = tx.Rollback(context.WithoutCancel(ctx)) }()
if len(given) > 0 {
if err := refuseGivenCollisions(ctx, tx, node.ID, nodeName, module, given); err != nil {
return err
}
}
_, err = tx.Exec(ctx,
`insert into settings (node, module, values) values ($1, $2, $3)
on conflict (node, module) where node is not null
do update set values = excluded.values, set_at = now()`, node.ID, module, raw)
return wrapModule(err, module)
if err != nil {
return wrapModule(err, module)
}
// A given port replaces what the mesh assigned for that port: the assignment is given back,
// so the number is free for the next module rather than held for ever in the name of a port
// that now lives elsewhere.
for wanted := range given {
if _, err := tx.Exec(ctx,
`delete from port_assignment where node = $1 and module = $2 and wanted = $3`,
node.ID, module, wanted); err != nil {
return err
}
}
return tx.Commit(ctx)
}
// givenIn is the machine ports a node-level settings layer gives a module, software port →
// machine port (novox/hq ADR 0100). Nothing when the layer gives none; what is not a port is left
// for composition to refuse in its own words.
func givenIn(module string, raw []byte) (map[int]int, error) {
var layer map[string]any
if err := json.Unmarshal(raw, &layer); err != nil {
return nil, err
}
entries, ok := layer[catalogue.PortsSetting].(map[string]any)
if !ok {
return nil, nil
}
out := map[int]int{}
by := map[int]int{}
for text, value := range entries {
wanted, err := strconv.Atoi(text)
if err != nil {
continue
}
at, ok := value.(float64)
if !ok {
continue
}
machine := int(at)
if machine == catalogue.SSHPort {
return nil, fmt.Errorf("%s cannot be given port %d for its %d: that is ssh's, the one "+
"port a machine may never lose", module, machine, wanted)
}
if other, twice := by[machine]; twice {
return nil, fmt.Errorf("%s gives machine port %d to both its %d and its %d; a machine "+
"port has one holder", module, machine, min(other, wanted), max(other, wanted))
}
by[machine] = wanted
out[wanted] = machine
}
return out, nil
}
// refuseGivenCollisions refuses a given machine port another module on the node already has —
// assigned by the mesh or given by its own setting — or that this module has for another of its
// ports. A port it was assigned for the same software port is not a collision: the given one
// replaces it.
func refuseGivenCollisions(ctx context.Context, tx pgx.Tx, nodeID any, node, module string,
given map[int]int) error {
type holder struct {
module string
wanted int
}
held := map[int]holder{}
rows, err := tx.Query(ctx,
`select machine, module, wanted from port_assignment where node = $1`, nodeID)
if err != nil {
return err
}
for rows.Next() {
var h holder
var machine int
if err := rows.Scan(&machine, &h.module, &h.wanted); err != nil {
rows.Close()
return err
}
held[machine] = h
}
rows.Close()
if err := rows.Err(); err != nil {
return err
}
rows, err = tx.Query(ctx,
`select module, values->'ports' from settings
where node = $1 and module <> $2 and jsonb_typeof(values->'ports') = 'object'`,
nodeID, module)
if err != nil {
return err
}
for rows.Next() {
var other string
var raw []byte
if err := rows.Scan(&other, &raw); err != nil {
rows.Close()
return err
}
var theirs map[string]any
if err := json.Unmarshal(raw, &theirs); err != nil {
rows.Close()
return err
}
for text, v := range theirs {
wanted, _ := strconv.Atoi(text)
if at, ok := v.(float64); ok {
held[int(at)] = holder{module: other, wanted: wanted}
}
}
}
rows.Close()
if err := rows.Err(); err != nil {
return err
}
wanted := make([]int, 0, len(given))
for w := range given {
wanted = append(wanted, w)
}
sort.Ints(wanted)
for _, w := range wanted {
machine := given[w]
h, taken := held[machine]
if !taken || (h.module == module && h.wanted == w) {
continue
}
if _, moving := given[h.wanted]; h.module == module && moving {
// Its own port for another of its software ports, which this same layer moves away.
continue
}
return fmt.Errorf("%w: %s cannot be given %d on %s for its %d — %s already has it for "+
"its %d", ErrPortTaken, module, machine, node, w, h.module, h.wanted)
}
return nil
}
func wrapModule(err error, module string) error {
@@ -654,7 +813,7 @@ func (i *Inventory) SettingsFor(ctx context.Context, nodeName, module string) ([
}
from := nodeName
if meshWide {
from = "the mesh"
from = catalogue.MeshWideLayer
}
layers = append(layers, catalogue.Layer{From: from, Values: values})
}
+11 -4
View File
@@ -32,7 +32,7 @@ func TestRegisteringAModuleAgainKeepsWhatTheMeshHoldsForIt(t *testing.T) {
t.Fatal(err)
}
m := catalogue.Manifest{Module: "step-ca", Version: "1",
Provides: catalogue.Offers("acme-ca")}
Provides: catalogue.Offers("acme-ca"), OwnSecrets: map[string]string{"password": "/run/password"}}
if err := inv.RegisterModule(ctx, m, Source{}); err != nil {
t.Fatal(err)
}
@@ -101,7 +101,7 @@ func TestForgettingAModuleRefusesRatherThanDiscardingWhatTheMeshHolds(t *testing
if err := inv.RecordSealingKey(ctx, node.ID, key); err != nil {
t.Fatal(err)
}
if err := inv.RegisterModule(ctx, manifest("step-ca", nil, nil), Source{}); err != nil {
if err := inv.RegisterModule(ctx, withOwnSecret(manifest("step-ca", nil, nil), "password"), Source{}); err != nil {
t.Fatal(err)
}
if err := inv.SetSettings(ctx, "anchor", "step-ca", map[string]any{"port": 9000}); err != nil {
@@ -156,7 +156,7 @@ func TestDiscardingAModuleSaysWhatWentWithIt(t *testing.T) {
if err := inv.RecordSealingKey(ctx, node.ID, key); err != nil {
t.Fatal(err)
}
if err := inv.RegisterModule(ctx, manifest("step-ca", nil, nil), Source{}); err != nil {
if err := inv.RegisterModule(ctx, withOwnSecret(manifest("step-ca", nil, nil), "password"), Source{}); err != nil {
t.Fatal(err)
}
if err := inv.SetSettings(ctx, "", "step-ca", map[string]any{"issuer": "the mesh"}); err != nil {
@@ -206,7 +206,7 @@ func TestAModuleStillAssignedRefusesBeforeAnythingAboutWhatItHolds(t *testing.T)
if _, err := inv.AddNode(ctx, "anchor"); err != nil {
t.Fatal(err)
}
if err := inv.RegisterModule(ctx, manifest("step-ca", nil, nil), Source{}); err != nil {
if err := inv.RegisterModule(ctx, withOwnSecret(manifest("step-ca", nil, nil), "password"), Source{}); err != nil {
t.Fatal(err)
}
if err := inv.SetSettings(ctx, "anchor", "step-ca", map[string]any{"a": 1}); err != nil {
@@ -224,3 +224,10 @@ func TestAModuleStillAssignedRefusesBeforeAnythingAboutWhatItHolds(t *testing.T)
}
}
}
// withOwnSecret gives a fixture manifest an own secret, so a delivery to it is one the module
// declares (novox/hq 04-ISSUES/078).
func withOwnSecret(m catalogue.Manifest, name string) catalogue.Manifest {
m.OwnSecrets = map[string]string{name: "/run/" + name}
return m
}
+89
View File
@@ -0,0 +1,89 @@
package inventory
import (
"context"
"errors"
"fmt"
"slices"
"strings"
"sync"
"time"
)
// ErrNodeBusy is a node another act kept holding for longer than a caller waits.
var ErrNodeBusy = errors.New("another act is composing or sending that node's declaration")
// HoldWaitFor is how long HoldNodes waits for a node another act holds, and HoldPoll how often it
// looks again. Variables so a test need not wait minutes.
var (
HoldWaitFor = 2 * time.Minute
HoldPoll = 250 * time.Millisecond
)
// HoldNodes serialises composing and sending a declaration per node (novox/hq ADR 0100): while
// one caller holds a node, another asking for it waits. Without it a push that composed a node as
// adopted could send that declaration after `converge --yes` sent the converged one, and the node
// would return to adopted with nobody having asked.
//
// Session-level advisory locks on one connection, all or none: a set not wholly free is given back
// at once, so two callers holding overlapping sets never each wait on the other. **A waiter pins
// no connection.** It looks again every HoldPoll with a connection borrowed for the look, and gives
// up after HoldWaitFor with ErrNodeBusy naming the node — so a stuck holder costs the pool one
// connection, never one per caller queued behind it. Release gives every one back, and may be
// called more than once.
func (i *Inventory) HoldNodes(ctx context.Context, names []string) (func(), error) {
sorted := slices.Clone(names)
slices.Sort(sorted)
sorted = slices.Compact(sorted)
deadline := time.Now().Add(HoldWaitFor)
for {
release, busy, err := i.tryHold(ctx, sorted)
if err != nil || busy == "" {
return release, err
}
if time.Now().After(deadline) {
return nil, fmt.Errorf("%w: %s has been held for over %s — try again once it is done",
ErrNodeBusy, busy, HoldWaitFor)
}
select {
case <-ctx.Done():
return nil, ctx.Err()
case <-time.After(HoldPoll):
}
}
}
// tryHold takes every named node's lock or none, and says which node was busy when it took none.
func (i *Inventory) tryHold(ctx context.Context, sorted []string) (func(), string, error) {
conn, err := i.store.Pool().Acquire(ctx)
if err != nil {
return nil, "", err
}
var once sync.Once
release := func() {
once.Do(func() {
// Unlocking all of this session's advisory locks, then handing the connection back: a
// connection returned still holding one would hold it for whoever borrows it next.
_, err := conn.Exec(context.WithoutCancel(ctx), `select pg_advisory_unlock_all()`)
if err != nil {
// The session's locks die with the session: close it rather than return it.
_ = conn.Conn().Close(context.WithoutCancel(ctx))
}
conn.Release()
})
}
for _, name := range sorted {
var took bool
if err := conn.QueryRow(ctx,
`select pg_try_advisory_lock(hashtext('mesh-node-declaration:' || $1)::bigint)`,
name).Scan(&took); err != nil {
release()
return nil, "", err
}
if !took {
release()
return nil, strings.TrimSpace(name), nil
}
}
return release, "", nil
}
+88
View File
@@ -0,0 +1,88 @@
package inventory
import (
"errors"
"strings"
"testing"
"time"
)
// Composing and sending one node's declaration is serialised: a second holder waits for the first.
func TestHoldingANodeMakesTheNextHolderWait(t *testing.T) {
inv := fresh(t)
ctx := t.Context()
release, err := inv.HoldNodes(ctx, []string{"anchor", "laptop"})
if err != nil {
t.Fatal(err)
}
got := make(chan func(), 1)
go func() {
second, err := inv.HoldNodes(ctx, []string{"laptop"})
if err != nil {
t.Error(err)
got <- func() {}
return
}
got <- second
}()
select {
case <-got:
t.Fatal("a node held by one caller was held by another at the same time")
case <-time.After(300 * time.Millisecond):
}
// Another node is not held up.
other, err := inv.HoldNodes(ctx, []string{"joiner"})
if err != nil {
t.Fatal(err)
}
other()
release()
select {
case second := <-got:
second()
case <-time.After(5 * time.Second):
t.Fatal("releasing the node did not let the next holder in")
}
}
// A waiter pins no pool connection while it waits, and gives up after a bounded wait saying which
// node is busy.
func TestAWaiterPinsNoConnectionAndGivesUp(t *testing.T) {
inv := fresh(t)
ctx := t.Context()
release, err := inv.HoldNodes(ctx, []string{"anchor"})
if err != nil {
t.Fatal(err)
}
defer release()
savedWait, savedPoll := HoldWaitFor, HoldPoll
HoldWaitFor, HoldPoll = 1500*time.Millisecond, 50*time.Millisecond
defer func() { HoldWaitFor, HoldPoll = savedWait, savedPoll }()
pool := inv.store.Pool()
base := pool.Stat().AcquiredConns()
const waiters = 3
done := make(chan error, waiters)
for range waiters {
go func() {
_, err := inv.HoldNodes(ctx, []string{"anchor"})
done <- err
}()
}
// While they wait, the pool lends nothing to them for longer than a look.
pinned := 0
for range 10 {
time.Sleep(60 * time.Millisecond)
if n := int(pool.Stat().AcquiredConns() - base); n > pinned {
pinned = n
}
}
if pinned >= waiters {
t.Fatalf("%d connections were held by %d waiters", pinned, waiters)
}
for range waiters {
if err := <-done; !errors.Is(err, ErrNodeBusy) || !strings.Contains(err.Error(), "anchor") {
t.Fatalf("a waiter did not give up naming the busy node: %v", err)
}
}
}
+85
View File
@@ -383,3 +383,88 @@ func TestBeingHeardFromDoesNotChangeWhatANodeOwns(t *testing.T) {
t.Errorf("after a bare word that the node is here, the mesh believes it owns %v", owned)
}
}
// **A token is claimed, then spent** (novox/hq issue 083). A presenter may claim it again — an
// enrolment interrupted by a restarting store asks again with the same key — while another is held
// off until the lease lapses; and it is spent only by the one holding the claim.
func TestATokenIsClaimedByOnePresenterAndSpentOnlyByIt(t *testing.T) {
inv := fresh(t)
ctx := t.Context()
if _, err := inv.AddNode(ctx, "laptop"); err != nil {
t.Fatal(err)
}
issued, err := inv.IssueToken(ctx, "laptop", time.Hour)
if err != nil {
t.Fatal(err)
}
node, err := inv.Claim(ctx, issued.Secret, "key-a", false)
if err != nil || node.Name != "laptop" {
t.Fatalf("a fresh token was not claimed for its node: %v %q", err, node.Name)
}
if _, err := inv.Claim(ctx, issued.Secret, "key-a", false); err != nil {
t.Fatalf("the presenter holding the claim could not claim again after an interruption: %v", err)
}
if _, err := inv.Claim(ctx, issued.Secret, "key-b", false); !errors.Is(err, ErrTokenInUse) {
t.Fatalf("a second presenter was not held off while the claim is live: %v", err)
}
if err := inv.Spend(ctx, issued.Secret, "key-b"); !errors.Is(err, ErrTokenRefused) {
t.Fatalf("a presenter not holding the claim spent the token: %v", err)
}
if err := inv.Spend(ctx, issued.Secret, "key-a"); err != nil {
t.Fatalf("the presenter holding the claim could not spend it: %v", err)
}
// Spent: nobody else may claim it, ever.
if _, err := inv.Claim(ctx, issued.Secret, "key-b", false); !errors.Is(err, ErrTokenRefused) {
t.Fatalf("a spent token was claimed by another presenter: %v", err)
}
// Nor the presenter that spent it, without proof it holds the key: a public key is no secret.
if _, err := inv.Claim(ctx, issued.Secret, "key-a", false); !errors.Is(err, ErrTokenRefused) {
t.Fatalf("a spent token was claimed again with no proof of the key: %v", err)
}
// With it, and inside the lease, it may, and spend it again: its spend reached the store and
// the answer did not reach the node, which asked again — refusing it would lock out a machine
// the mesh holds as enrolled.
if _, err := inv.Claim(ctx, issued.Secret, "key-a", true); err != nil {
t.Fatalf("the proven presenter whose answer was lost after its spend was refused: %v", err)
}
if err := inv.Spend(ctx, issued.Secret, "key-a"); err != nil {
t.Fatalf("spending again by the same presenter failed: %v", err)
}
// And not once the lease is over: then a spent token is spent to everyone, proof or not.
if _, err := inv.store.Pool().Exec(ctx,
`update enrolment_token set claimed_until = now() - interval '1 second' where secret = $1`,
hashSecret(issued.Secret)); err != nil {
t.Fatal(err)
}
if _, err := inv.Claim(ctx, issued.Secret, "key-a", true); !errors.Is(err, ErrTokenRefused) {
t.Fatalf("a spent token was claimed again after its lease: %v", err)
}
}
// A claim lapses: a host that gave up and was started over, with keys of its own, is not held off
// for longer than the lease.
func TestAClaimThatLapsedCanBeTakenByAnotherPresenter(t *testing.T) {
inv := fresh(t)
ctx := t.Context()
if _, err := inv.AddNode(ctx, "laptop"); err != nil {
t.Fatal(err)
}
issued, err := inv.IssueToken(ctx, "laptop", time.Hour)
if err != nil {
t.Fatal(err)
}
if _, err := inv.Claim(ctx, issued.Secret, "key-a", false); err != nil {
t.Fatal(err)
}
if _, err := inv.store.Pool().Exec(ctx,
`update enrolment_token set claimed_until = now() - interval '1 second' where secret = $1`,
hashSecret(issued.Secret)); err != nil {
t.Fatal(err)
}
if _, err := inv.Claim(ctx, issued.Secret, "key-b", false); err != nil {
t.Fatalf("a lapsed claim held off a new presenter: %v", err)
}
if err := inv.Spend(ctx, issued.Secret, "key-a"); !errors.Is(err, ErrTokenRefused) {
t.Fatalf("the presenter whose claim lapsed could still spend the token: %v", err)
}
}
@@ -0,0 +1,12 @@
-- A module may need several values from one provider that gives one per pair
-- (novox/hq 04-ISSUES/069, ADR 0094).
--
-- A pair credential was keyed on (provision, consumer node, consumer module, provider): one value
-- per module per provider. Seven catalogue modules hold two to four independent secrets of their
-- own -- a root certificate, its key and that key's password -- and the vault could serve each
-- module one. The pair now carries the LOCAL name the credential goes by inside the module; empty
-- for the ordinary one, so every existing row is the credential it was.
alter table secret add column local text not null default '';
alter table secret drop constraint secret_pkey;
alter table secret add primary key (name, local, consumer, consumer_module, provider);
@@ -0,0 +1,11 @@
-- A token is claimed by the enrolment presenting it, and spent only when that enrolment has
-- written everything it needs (novox/hq 04-ISSUES/083).
--
-- Spending came first and the node's keys after, in another database: a store that went away
-- between the two left a spent token and a node with no key, and the host — which makes new keys
-- on every attempt — could not try again. The claim holds the token for one presenter for a short
-- lease, so an attempt that failed part-way can be made again by the same presenter, and a second
-- presenter cannot interleave with the first.
alter table enrolment_token add column claimed_by text;
alter table enrolment_token add column claimed_until timestamptz;
@@ -0,0 +1,21 @@
-- A node is adopted or converged, and the mesh records which (novox/hq ADR 0100).
--
-- A machine already serving a predecessor's services is adopted: what is found on it is kept
-- until its module is taken, and the firewall found on it stays in force. It stays adopted until
-- the operator converges it. False by default, so every node that exists is converged, as it was.
alter table node add column adopted boolean not null default false;
-- When it was last made adopted, and when it last converged. Both kept: a node returned to adopted
-- after converging is a different history from one that never converged.
alter table node add column adopted_since timestamptz;
alter table node add column converged_at timestamptz;
-- The modules taken on a node: its cutover, the operator's act, done when the module's data has
-- moved. A row per node and module, and it outlives the assignment on purpose -- unassigning a
-- module does not un-take it, and what was taken stays taken when a node returns to adopted.
create table taken (
node uuid not null references node(id) on delete cascade,
module text not null references module(name) on delete cascade,
taken_at timestamptz not null default now(),
primary key (node, module)
);
@@ -0,0 +1,12 @@
-- What an adopted node last reported about adoption (novox/hq ADR 0100): the files and containers
-- it found and holds until their module is taken, the firewall it found, and what is reachable on
-- the machine now — which converging it previews, so nothing closes without being named first.
--
-- The last report, replaced, like what a node owns: the question is the machine as it is now.
-- Kept apart from node_report because a node reports it on its own schedule, when what it holds
-- changes, and not only after an apply.
alter table node add column held jsonb;
alter table node add column firewall text;
alter table node add column reachable jsonb;
alter table node add column adoption_reported timestamptz;
@@ -0,0 +1,27 @@
-- The tunnel a node found on its machine, and the peers it carried (novox/hq ADR 0105).
--
-- On an adopted node that is the hub, the private network takes over the tunnel it finds: its
-- key, its port, its address and range, and every peer. The node presents what it found when it
-- enrols -- the same moment it presents its keys, because the found tunnel's key IS its key on the
-- private network from then on -- and the mesh composes every address from it.
-- What the node presented: interface, unit and configuration path, port, address and range, and
-- the tunnel's public key. The private key never travels; the node keeps it as its own overlay
-- key. Null on a node that found no tunnel, which is every converged one.
alter table node add column tunnel jsonb;
-- The node's last account of carrying it: the found interface down and disabled, the mesh's up
-- in its place. Null until the node says so.
alter table node add column tunnel_carried jsonb;
-- The peers the found tunnel had: a public key and the address the tunnel routed to it. Peers of
-- the tunnel, not nodes of the mesh, until they enrol -- a machine the mesh has no record of, whose
-- identity precedes its enrolment. One row per key, and one address per key on one tunnel.
create table tunnel_peer (
node uuid not null references node(id) on delete cascade,
public_key text not null,
address inet not null,
since timestamptz not null default now(),
primary key (node, public_key),
unique (node, address)
);
+111 -17
View File
@@ -49,6 +49,12 @@ type Node struct {
// month's assignments, and until this existed those looked the same as a node that is
// current (novox/hq 09-the-node-lifecycle).
LastSeen time.Time
// Adopted is whether this node is adopted rather than converged (novox/hq ADR 0100): what is
// found on it is kept until its module is taken, and the firewall found on it stays in force.
// AdoptedSince is when it last became so; zero for a converged node.
Adopted bool
AdoptedSince time.Time
}
// Silent is how long since this node was last heard from, and whether it ever was.
@@ -74,28 +80,59 @@ var ErrNameTaken = errors.New("a node of that name already exists")
// (novox/hq 09-the-node-lifecycle), so the record is what a token binds to and must exist before
// there is anything to join.
func (i *Inventory) AddNode(ctx context.Context, name string) (Node, error) {
return i.AddNodeAs(ctx, name, false)
}
// AddNodeAs creates a node record, adopted or converged (novox/hq ADR 0100). The operator says
// which; a node added without saying is converged, as every node was before adoption existed.
func (i *Inventory) AddNodeAs(ctx context.Context, name string, adopted bool) (Node, error) {
name = strings.TrimSpace(name)
if name == "" {
return Node{}, errors.New("a node needs a name: it is how a token is issued for it")
}
var n Node
var since *time.Time
err := i.store.Pool().QueryRow(ctx,
`insert into node (name) values ($1) returning id, name, created`,
name).Scan(&n.ID, &n.Name, &n.Created)
`insert into node (name, adopted, adopted_since)
values ($1, $2, case when $2 then now() end)
returning id, name, created, adopted, adopted_since`,
name, adopted).Scan(&n.ID, &n.Name, &n.Created, &n.Adopted, &since)
if err != nil {
if strings.Contains(err.Error(), "node_name_key") {
return Node{}, fmt.Errorf("%w: %s", ErrNameTaken, name)
}
return Node{}, err
}
if since != nil {
n.AdoptedSince = *since
}
return n, nil
}
// nodeColumns and scanNode are the one reading of a node row, so every way of finding a node
// says whether it is adopted.
const nodeColumns = `id, name, created, last_seen, adopted, adopted_since`
func scanNode(row pgx.Row) (Node, error) {
var n Node
var seen, since *time.Time
if err := row.Scan(&n.ID, &n.Name, &n.Created, &seen, &n.Adopted, &since); err != nil {
return Node{}, err
}
if seen != nil {
n.LastSeen = *seen
}
if since != nil {
n.AdoptedSince = *since
}
return n, nil
}
// Nodes are every node record, oldest first.
func (i *Inventory) Nodes(ctx context.Context) ([]Node, error) {
rows, err := i.store.Pool().Query(ctx,
`select id, name, created, last_seen from node order by created, name`)
`select `+nodeColumns+` from node order by created, name`)
if err != nil {
return nil, err
}
@@ -103,14 +140,10 @@ func (i *Inventory) Nodes(ctx context.Context) ([]Node, error) {
var nodes []Node
for rows.Next() {
var n Node
var seen *time.Time
if err := rows.Scan(&n.ID, &n.Name, &n.Created, &seen); err != nil {
n, err := scanNode(rows)
if err != nil {
return nil, err
}
if seen != nil {
n.LastSeen = *seen
}
nodes = append(nodes, n)
}
return nodes, rows.Err()
@@ -118,9 +151,8 @@ func (i *Inventory) Nodes(ctx context.Context) ([]Node, error) {
// NodeByName finds one node record.
func (i *Inventory) NodeByName(ctx context.Context, name string) (Node, error) {
var n Node
err := i.store.Pool().QueryRow(ctx,
`select id, name, created from node where name = $1`, name).Scan(&n.ID, &n.Name, &n.Created)
n, err := scanNode(i.store.Pool().QueryRow(ctx,
`select `+nodeColumns+` from node where name = $1`, name))
if errors.Is(err, pgx.ErrNoRows) {
return Node{}, fmt.Errorf("%w: %s", ErrNoSuchNode, name)
}
@@ -203,7 +235,72 @@ func (i *Inventory) IssueToken(ctx context.Context, nodeName string, validFor ti
// token that had expired.
var ErrTokenRefused = errors.New("that token cannot be used")
// Redeem spends a token and reports which node it was for.
// ClaimLease is how long a claimed token is held for the one presenter that claimed it. Long
// enough for an enrolment to be tried again through a store restart; short enough that a host
// which gave up and was started over, with keys of its own, is not kept waiting long.
const ClaimLease = 2 * time.Minute
// ErrTokenInUse is a token another presenter holds a claim on right now. Not a refusal: the claim
// lapses, and asking again after it is the answer.
var ErrTokenInUse = errors.New("the token is being used by another enrolment")
// Claim takes a token for one presenter — `by`, which names the key presenting it — for the length
// of a lease, and says which node it enrols. The same presenter may claim it again, as may anyone
// once the lease has lapsed; nothing is spent until Spend (novox/hq 04-ISSUES/083).
//
// A token this same presenter already spent may be claimed again when `again` says so — the
// caller has proof the presenter holds the key's private half — and only while its claim's lease
// is live: its spend reached the store and the answer did not reach the node, which asked again.
// Refusing it then would lock out a machine the mesh holds as enrolled. Without the proof a spent
// token stays spent to everyone, as ADR 0004 says.
func (i *Inventory) Claim(ctx context.Context, secret, by string, again bool) (Node, error) {
var id string
err := i.store.Pool().QueryRow(ctx,
`update enrolment_token set claimed_by = $2,
claimed_until = case when redeemed is null then now() + $3::interval else claimed_until end
where secret = $1 and expires > now()
and ((redeemed is null and (claimed_by is null or claimed_by = $2 or claimed_until < now()))
or (redeemed is not null and $4 and claimed_by = $2 and claimed_until > now()))
returning node`, hashSecret(secret), by, ClaimLease.String(), again).Scan(&id)
if errors.Is(err, pgx.ErrNoRows) {
// Unusable, or held by someone else — told apart, because the second passes.
var held bool
probe := i.store.Pool().QueryRow(ctx,
`select true from enrolment_token
where secret = $1 and redeemed is null and expires > now()`, hashSecret(secret)).Scan(&held)
switch {
case probe == nil && held:
return Node{}, ErrTokenInUse
case probe != nil && !errors.Is(probe, pgx.ErrNoRows):
// The store went away between the two questions: "not now", not a refusal.
return Node{}, probe
}
return Node{}, ErrTokenRefused
}
if err != nil {
return Node{}, err
}
return scanNode(i.store.Pool().QueryRow(ctx, `select `+nodeColumns+` from node where id = $1`, id))
}
// Spend makes a claimed token used, only for the presenter holding the claim. The last write to the
// store in an enrolment, so a token is spent exactly when the node it enrolled is complete. Spent
// again by the same presenter is not an error: an answer lost after the first spend.
func (i *Inventory) Spend(ctx context.Context, secret, by string) error {
tag, err := i.store.Pool().Exec(ctx,
`update enrolment_token set redeemed = coalesce(redeemed, now())
where secret = $1 and claimed_by = $2`, hashSecret(secret), by)
if err != nil {
return err
}
if tag.RowsAffected() == 0 {
return ErrTokenRefused
}
return nil
}
// Redeem spends a token in one step and reports which node it was for. Enrolment claims and then
// spends (Claim, Spend); this is the one-step form, kept for what spends a token outright.
//
// It does not issue an identity. What a node presents afterwards to prove it is that node is not
// decided anywhere (novox/hq ADR 0004 names the property, not the mechanism), and guessing at it
@@ -225,10 +322,7 @@ func (i *Inventory) Redeem(ctx context.Context, secret string) (Node, error) {
return Node{}, err
}
var n Node
err = i.store.Pool().QueryRow(ctx,
`select id, name, created from node where id = $1`, id).Scan(&n.ID, &n.Name, &n.Created)
return n, err
return scanNode(i.store.Pool().QueryRow(ctx, `select `+nodeColumns+` from node where id = $1`, id))
}
// RecordProfile keeps the last thing a node said about what it can do.
+11 -10
View File
@@ -124,20 +124,20 @@ func (i *Inventory) KeptForOperator(ctx context.Context) (kept, earlier, unrecov
}
rows, err := i.store.Pool().Query(ctx,
`select 'own', n.name, s.module, s.name, '', s.origin, coalesce(s.operator_sealed, ''),
coalesce(s.operator_key, ''), s.made_at
coalesce(s.operator_key, ''), s.made_at, ''
from module_secret s join node n on n.id = s.node
union all
select 'pair', c.name, s.consumer_module, s.name, p.name, 'made', coalesce(s.operator_sealed, ''),
coalesce(s.operator_key, ''), s.created_at
select 'pair', c.name, s.consumer_module, s.name, p.name, s.origin, coalesce(s.operator_sealed, ''),
coalesce(s.operator_key, ''), s.created_at, s.local
from secret s join node c on c.id = s.consumer join node p on p.id = s.provider
order by 1, 2, 3, 4`)
order by 1, 2, 3, 4, 10`)
if err != nil {
return nil, nil, nil, err
}
defer rows.Close()
for rows.Next() {
var k Kept
if err := rows.Scan(&k.Kind, &k.Node, &k.Module, &k.Name, &k.Provider, &k.Origin, &k.Sealed, &k.Key, &k.MadeAt); err != nil {
if err := rows.Scan(&k.Kind, &k.Node, &k.Module, &k.Name, &k.Provider, &k.Origin, &k.Sealed, &k.Key, &k.MadeAt, &k.Local); err != nil {
return nil, nil, nil, err
}
switch {
@@ -159,7 +159,7 @@ func (i *Inventory) KeptForOperator(ctx context.Context) (kept, earlier, unrecov
// credential is keyed by provider as well, and a consumer whose provision moved leaves the old
// provider's row behind: two rows is refused with both providers named, never answered with
// whichever came first, unless `provider` says which.
func (i *Inventory) KeptSecret(ctx context.Context, node, module, name, provider string) (Kept, error) {
func (i *Inventory) KeptSecret(ctx context.Context, node, module, name, provider, local string) (Kept, error) {
var k Kept
err := i.store.Pool().QueryRow(ctx,
`select 'own', n.name, s.module, s.name, '', s.origin, coalesce(s.operator_sealed, ''),
@@ -169,11 +169,12 @@ func (i *Inventory) KeptSecret(ctx context.Context, node, module, name, provider
Scan(&k.Kind, &k.Node, &k.Module, &k.Name, &k.Provider, &k.Origin, &k.Sealed, &k.Key, &k.MadeAt)
if errors.Is(err, pgx.ErrNoRows) {
rows, qerr := i.store.Pool().Query(ctx,
`select 'pair', c.name, s.consumer_module, s.name, p.name, 'made', coalesce(s.operator_sealed, ''),
coalesce(s.operator_key, ''), s.created_at
`select 'pair', c.name, s.consumer_module, s.name, p.name, s.origin, coalesce(s.operator_sealed, ''),
coalesce(s.operator_key, ''), s.created_at, s.local
from secret s join node c on c.id = s.consumer join node p on p.id = s.provider
where c.name = $1 and s.consumer_module = $2 and s.name = $3 and ($4 = '' or p.name = $4)
order by p.name`, node, module, name, provider)
and s.local = $5
order by p.name`, node, module, name, provider, local)
if qerr != nil {
return Kept{}, qerr
}
@@ -181,7 +182,7 @@ func (i *Inventory) KeptSecret(ctx context.Context, node, module, name, provider
var found []Kept
for rows.Next() {
var row Kept
if err := rows.Scan(&row.Kind, &row.Node, &row.Module, &row.Name, &row.Provider, &row.Origin, &row.Sealed, &row.Key, &row.MadeAt); err != nil {
if err := rows.Scan(&row.Kind, &row.Node, &row.Module, &row.Name, &row.Provider, &row.Origin, &row.Sealed, &row.Key, &row.MadeAt, &row.Local); err != nil {
return Kept{}, err
}
found = append(found, row)
+12 -11
View File
@@ -13,7 +13,8 @@ import (
// opens exactly the value the node was given.
func TestAnOwnSecretIsSealedToTheOperatorToo(t *testing.T) {
inv, ctx := twoNodesWithKeys(t)
if err := inv.RegisterModule(ctx, catalogue.Manifest{Module: "postgres", Version: "1"}, Source{}); err != nil {
if err := inv.RegisterModule(ctx, catalogue.Manifest{Module: "postgres", Version: "1",
OwnSecrets: map[string]string{"superuser": "/run/superuser", "replication": "/run/replication"}}, Source{}); err != nil {
t.Fatal(err)
}
@@ -21,7 +22,7 @@ func TestAnOwnSecretIsSealedToTheOperatorToo(t *testing.T) {
if _, err := inv.SecretForModule(ctx, "consumer", "postgres", "superuser"); err != nil {
t.Fatal(err)
}
if _, err := inv.KeptSecret(ctx, "consumer", "postgres", "superuser", ""); err == nil {
if _, err := inv.KeptSecret(ctx, "consumer", "postgres", "superuser", "", ""); err == nil {
t.Fatal("a secret made before the operator key was reported recoverable")
}
kept, _, unrecoverable, err := inv.KeptForOperator(ctx)
@@ -54,7 +55,7 @@ func TestAnOwnSecretIsSealedToTheOperatorToo(t *testing.T) {
if len(kept) != 2 || len(unrecoverable) != 1 {
t.Fatalf("after a key: %d kept, %d unrecoverable", len(kept), len(unrecoverable))
}
got, err := inv.KeptSecret(ctx, "provider", "postgres", "replication", "")
got, err := inv.KeptSecret(ctx, "provider", "postgres", "replication", "", "")
if err != nil {
t.Fatal(err)
}
@@ -68,7 +69,7 @@ func TestAnOwnSecretIsSealedToTheOperatorToo(t *testing.T) {
if got.Origin != "accepted" || got.Key != pub {
t.Fatalf("kept as %+v", got)
}
minted, err := inv.KeptSecret(ctx, "provider", "postgres", "superuser", "")
minted, err := inv.KeptSecret(ctx, "provider", "postgres", "superuser", "", "")
if err != nil {
t.Fatal(err)
}
@@ -81,7 +82,7 @@ func TestAnOwnSecretIsSealedToTheOperatorToo(t *testing.T) {
if _, err := inv.SecretForModule(ctx, "consumer", "postgres", "superuser"); err != nil {
t.Fatal(err)
}
if _, err := inv.KeptSecret(ctx, "consumer", "postgres", "superuser", ""); err == nil {
if _, err := inv.KeptSecret(ctx, "consumer", "postgres", "superuser", "", ""); err == nil {
t.Fatal("asking again did not remake, yet it became recoverable")
}
// Until the node rejoins with a new sealing key: then the secret is remade, and the remake is
@@ -97,7 +98,7 @@ func TestAnOwnSecretIsSealedToTheOperatorToo(t *testing.T) {
if _, err := inv.SecretForModule(ctx, "consumer", "postgres", "superuser"); err != nil {
t.Fatal(err)
}
remade, err := inv.KeptSecret(ctx, "consumer", "postgres", "superuser", "")
remade, err := inv.KeptSecret(ctx, "consumer", "postgres", "superuser", "", "")
if err != nil {
t.Fatalf("the remade secret is not recoverable: %v", err)
}
@@ -164,11 +165,11 @@ func TestAPairCredentialIsSealedToTheOperatorToo(t *testing.T) {
if _, err := inv.SetOperatorKey(ctx, pub); err != nil {
t.Fatal(err)
}
made, err := inv.SecretFor(ctx, "secret", "consumer", "gitea", "provider")
made, err := inv.SecretFor(ctx, "secret", "consumer", "gitea", "provider", "")
if err != nil {
t.Fatal(err)
}
kept, err := inv.KeptSecret(ctx, "consumer", "gitea", "secret", "")
kept, err := inv.KeptSecret(ctx, "consumer", "gitea", "secret", "", "")
if err != nil {
t.Fatal(err)
}
@@ -191,13 +192,13 @@ func TestAPairCredentialIsSealedToTheOperatorToo(t *testing.T) {
// A second provider of the same provision: two rows, refused rather than the first one taken,
// unless the provider is named. And replacing the key counts pair credentials as orphaned.
if _, err := inv.SecretFor(ctx, "secret", "consumer", "gitea", "consumer"); err != nil {
if _, err := inv.SecretFor(ctx, "secret", "consumer", "gitea", "consumer", ""); err != nil {
t.Fatal(err)
}
if _, err := inv.KeptSecret(ctx, "consumer", "gitea", "secret", ""); err == nil || !strings.Contains(err.Error(), "--provider") {
if _, err := inv.KeptSecret(ctx, "consumer", "gitea", "secret", "", ""); err == nil || !strings.Contains(err.Error(), "--provider") {
t.Fatalf("two providers were not refused: %v", err)
}
if byName, err := inv.KeptSecret(ctx, "consumer", "gitea", "secret", "provider"); err != nil || byName.Provider != "provider" {
if byName, err := inv.KeptSecret(ctx, "consumer", "gitea", "secret", "provider", ""); err != nil || byName.Provider != "provider" {
t.Fatalf("naming the provider did not select it: %+v %v", byName, err)
}
pub2, _, _ := secrets.Keypair()
+42
View File
@@ -2,6 +2,7 @@ package inventory
import (
"context"
"encoding/json"
"errors"
"fmt"
@@ -132,6 +133,17 @@ func (i *Inventory) assignPort(
taken[port] = "something this machine already runs"
}
}
// And every port this machine was given for a module (novox/hq ADR 0100): the foundation's
// ports, as genesis chose them, are the node's settings and never the mesh's to hand out.
given, err := i.givenOn(ctx, nodeID)
if err != nil {
return Assigned{}, err
}
for port, by := range given {
if _, mine := taken[port]; !mine {
taken[port] = by
}
}
machine := wanted
if !fixed {
@@ -258,3 +270,33 @@ func (i *Inventory) ReleasePorts(ctx context.Context, node, module string) error
`delete from port_assignment where node = $1 and module = $2`, record.ID, module)
return err
}
// givenOn is every machine port a module was given on this node by its `ports` setting, and which
// module it was given to.
func (i *Inventory) givenOn(ctx context.Context, nodeID any) (map[int]string, error) {
rows, err := i.store.Pool().Query(ctx,
`select module, values->'ports' from settings
where node = $1 and jsonb_typeof(values->'ports') = 'object'`, nodeID)
if err != nil {
return nil, err
}
defer rows.Close()
out := map[int]string{}
for rows.Next() {
var module string
var raw []byte
if err := rows.Scan(&module, &raw); err != nil {
return nil, err
}
var given map[string]any
if err := json.Unmarshal(raw, &given); err != nil {
return nil, err
}
for _, v := range given {
if at, ok := v.(float64); ok {
out[int(at)] = module
}
}
}
return out, rows.Err()
}
+108
View File
@@ -2,6 +2,7 @@ package inventory
import (
"errors"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
@@ -236,3 +237,110 @@ func TestUnassigningReleasesTheModulesPorts(t *testing.T) {
t.Fatalf("port 25 is still held in the name of a module that was unassigned: %v", err)
}
}
// novox/hq ADR 0100: a port a node was given for a module is the node's, and the mesh never hands
// it to another.
func TestAGivenPortIsNeverAssigned(t *testing.T) {
inv, node := aNodeWithModules(t, "postgres", "web")
ctx := t.Context()
if err := inv.SetSettings(ctx, node, "postgres",
map[string]any{catalogue.PortsSetting: map[string]any{"5432": 20000}}); err != nil {
t.Fatal(err)
}
got, err := inv.PortFor(ctx, node, "web", 8080, false)
if err != nil {
t.Fatal(err)
}
if got.Machine == 20000 {
t.Fatal("a port given to postgres was assigned to web")
}
if _, err := inv.PortFor(ctx, node, "web", 20000, true); !errors.Is(err, ErrPortTaken) {
t.Fatalf("a fixed port given to another module was handed over: %v", err)
}
}
// novox/hq ADR 0100: a given machine port has one holder. It is refused when another module has it,
// assigned or given, when the same layer gives it twice, and when it is ssh's; and when it replaces
// what the mesh assigned for that port, the assignment is given back.
func TestAGivenPortHasOneHolderAndReplacesTheAssignment(t *testing.T) {
inv, node := aNodeWithModules(t, "postgres", "web", "cache")
ctx := t.Context()
give := func(module string, ports map[string]any) error {
return inv.SetSettings(ctx, node, module, map[string]any{catalogue.PortsSetting: ports})
}
web, err := inv.PortFor(ctx, node, "web", 8080, false)
if err != nil {
t.Fatal(err)
}
if err := give("postgres", map[string]any{"5432": web.Machine}); !errors.Is(err, ErrPortTaken) {
t.Fatalf("a port the mesh assigned to web was given to postgres: %v", err)
}
if err := give("postgres", map[string]any{"5432": 22}); err == nil {
t.Fatal("ssh's port was given")
}
if err := give("postgres", map[string]any{"5432": 5433, "5433": 5433}); err == nil {
t.Fatal("one machine port was given for two ports")
}
if err := give("cache", map[string]any{"6379": 6380}); err != nil {
t.Fatal(err)
}
if err := give("postgres", map[string]any{"5432": 6380}); !errors.Is(err, ErrPortTaken) {
t.Fatalf("a port given to cache was given to postgres: %v", err)
}
// Postgres was assigned a port for 5432; given one, the assignment is released and the number
// is free again.
assigned, err := inv.PortFor(ctx, node, "postgres", 5432, false)
if err != nil {
t.Fatal(err)
}
if err := give("postgres", map[string]any{"5432": 5433}); err != nil {
t.Fatal(err)
}
// Given again, the same: its own given port is not a collision with itself.
if err := give("postgres", map[string]any{"5432": 5433}); err != nil {
t.Fatal(err)
}
held, err := inv.PortsFor(ctx, node)
if err != nil {
t.Fatal(err)
}
for _, a := range held {
if a.Module == "postgres" {
t.Fatalf("the assignment a given port replaced is still held: %+v", a)
}
}
other, err := inv.PortFor(ctx, node, "cache", 11211, false)
if err != nil {
t.Fatal(err)
}
if other.Machine != assigned.Machine {
t.Fatalf("the released port %d was not free again (got %d)", assigned.Machine, other.Machine)
}
}
// novox/hq ADR 0100: a port is a fact about one machine, so a layer for the whole mesh cannot give
// one. Refused where it is set — stored, it refuses every node running the module at composition,
// and the mesh cannot be pushed until somebody finds the layer that did it.
func TestAPortGivenForTheWholeMeshIsRefusedWhereItIsSet(t *testing.T) {
inv, node := aNodeWithModules(t, "postgres")
ctx := t.Context()
err := inv.SetSettings(ctx, "", "postgres",
map[string]any{catalogue.PortsSetting: map[string]any{"5432": 5433}})
if err == nil || !strings.Contains(err.Error(), "per node") {
t.Fatalf("a port given for the whole mesh was accepted: %v", err)
}
layers, err := inv.SettingsFor(ctx, node, "postgres")
if err != nil {
t.Fatal(err)
}
if len(layers) != 0 {
t.Fatalf("the refused layer was stored: %v", layers)
}
// The same values for one machine are the ordinary setting.
if err := inv.SetSettings(ctx, node, "postgres",
map[string]any{catalogue.PortsSetting: map[string]any{"5432": 5433}}); err != nil {
t.Fatal(err)
}
}
+52
View File
@@ -0,0 +1,52 @@
package inventory
import (
"errors"
"io"
"net"
"github.com/jackc/pgx/v5/pgconn"
)
// Unreachable says whether an error means the store could not be asked right now, rather than
// that it answered no.
//
// The difference decides whether something worth keeping is kept or lost. The store is recreated
// when the foundation is adopted (ADR 0078), and for those seconds every write fails; a caller
// that treats that like a refusal throws away what it was writing — a node's report of the apply
// that caused the restart was lost exactly so, and the mesh never heard from the node again
// (novox/hq issue 082).
//
// Unreachable is the connection failing and the server saying "not now". The connection failing
// is a network error, a connection that ended mid-conversation (a store killed rather than
// stopped gives a bare unexpected EOF), a timeout, or anything pgx marks safe to retry. The server
// saying "not now" is a connection exception (class 08) or it shutting down or starting up (57P01,
// 57P02, 57P03). Everything else is an answer, and asking again gets the same one: a wrong
// password, a database that does not exist, a statement cancelled, a constraint. Those are
// checked first, even inside a failed connection, because a connection that failed on a wrong
// password would otherwise read as a network fault and be asked again for ever.
func Unreachable(err error) bool {
if err == nil {
return false
}
var pg *pgconn.PgError
if errors.As(err, &pg) {
switch pg.Code {
case "57P01", "57P02", "57P03":
return true
}
return len(pg.Code) == 5 && pg.Code[:2] == "08"
}
if errors.Is(err, io.ErrUnexpectedEOF) || errors.Is(err, io.EOF) {
return true
}
if pgconn.Timeout(err) || pgconn.SafeToRetry(err) {
return true
}
var network net.Error
if errors.As(err, &network) {
return true
}
var connect *pgconn.ConnectError
return errors.As(err, &connect)
}
+76
View File
@@ -0,0 +1,76 @@
package inventory
import (
"context"
"errors"
"fmt"
"io"
"os"
"strings"
"testing"
"time"
"github.com/jackc/pgx/v5/pgconn"
)
// The store restarting or dying is "not now"; an answer the store gave is not (issue 082).
func TestAStoreThatIsGoneForNowIsUnreachableAndAnAnswerIsNot(t *testing.T) {
for _, c := range []struct {
what string
err error
want bool
}{
{"starting up", &pgconn.PgError{Code: "57P03"}, true},
{"stopped by its administrator, wrapped", fmt.Errorf("recording: %w", &pgconn.PgError{Code: "57P01"}), true},
{"crashed", &pgconn.PgError{Code: "57P02"}, true},
{"a connection exception", &pgconn.PgError{Code: "08006"}, true},
{"killed under a live connection", io.ErrUnexpectedEOF, true},
{"killed, wrapped", fmt.Errorf("recording: %w", io.ErrUnexpectedEOF), true},
{"a statement cancelled", &pgconn.PgError{Code: "57014"}, false},
{"the database dropped", &pgconn.PgError{Code: "57P04"}, false},
{"a wrong password", &pgconn.PgError{Code: "28P01"}, false},
{"a constraint", &pgconn.PgError{Code: "23503"}, false},
{"a plain error", errors.New("a report named no node"), false},
{"nothing", nil, false},
} {
if got := Unreachable(c.err); got != c.want {
t.Errorf("%s: Unreachable(%v) = %v, want %v", c.what, c.err, got, c.want)
}
}
}
// A connection refused is the shape a store that is down gives — a real one, to a port on
// loopback nothing listens on.
func TestAStoreThatRefusesTheConnectionIsUnreachable(t *testing.T) {
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
_, err := pgconn.Connect(ctx, "postgres://nobody@127.0.0.1:1/nothing?sslmode=disable&connect_timeout=2")
if err == nil {
t.Skip("something listens on port 1")
}
if !Unreachable(err) {
t.Fatalf("a refused connection is not unreachable: %T %v", err, err)
}
}
// A connection the store refused on a wrong password is an answer, even though it arrives as a
// failed connection — asked again for ever, it would hold every message behind it.
func TestAWrongPasswordIsAnAnswerNotAnOutage(t *testing.T) {
dsn := os.Getenv("MESH_TEST_POSTGRES")
if dsn == "" {
t.Skip("MESH_TEST_POSTGRES is not set")
}
wrong := strings.Replace(dsn, "postgres:check@", "postgres:not-the-password@", 1)
if wrong == dsn {
t.Skip("the test store's address does not carry the expected credential")
}
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
_, err := pgconn.Connect(ctx, wrong)
if err == nil {
t.Fatal("a wrong password connected")
}
if Unreachable(err) {
t.Fatalf("a wrong password was taken for an outage and would be retried for ever: %v", err)
}
}
+123 -35
View File
@@ -4,9 +4,13 @@ import (
"context"
"errors"
"fmt"
"slices"
"sort"
"strings"
"github.com/jackc/pgx/v5"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/secrets"
)
@@ -24,11 +28,14 @@ type Secret struct {
// **Part of the key, not a label** (novox/hq 04-ISSUES/022). Two modules on one node wanting
// the same provision are two consumers, and were one credential until this.
ConsumerModule string
Provider string
ForConsumer string
ForProvider string
ConsumerKey string
ProviderKey string
// Local is the name the credential goes by inside the consumer where it keeps several for one
// provision (novox/hq ADR 0094); empty for the ordinary one. Part of the key.
Local string
Provider string
ForConsumer string
ForProvider string
ConsumerKey string
ProviderKey string
// Origin is `made` — the mesh generated it — or `accepted` — a person supplied it, for
// something outside the mesh, and the mesh cannot make another (novox/hq 04-ISSUES/070).
Origin string
@@ -51,7 +58,7 @@ const (
// can no longer open what was sealed to the old one, so keeping the blob would deliver something
// unreadable for ever. The new secret reaches both ends in the same push, which is the only
// moment they can be changed together.
func (i *Inventory) SecretFor(ctx context.Context, name, consumer, consumerModule, provider string) (
func (i *Inventory) SecretFor(ctx context.Context, name, consumer, consumerModule, provider, local string) (
Secret, error) {
consumerKey, err := i.SealingKeyOf(ctx, consumer)
if err != nil {
@@ -74,12 +81,12 @@ func (i *Inventory) SecretFor(ctx context.Context, name, consumer, consumerModul
var held Secret
err = i.store.Pool().QueryRow(ctx,
`select for_consumer, for_provider, consumer_key, provider_key, origin from secret
where name = $1 and consumer = $2 and consumer_module = $3 and provider = $4`,
name, consumerNode.ID, consumerModule, providerNode.ID).
where name = $1 and consumer = $2 and consumer_module = $3 and provider = $4 and local = $5`,
name, consumerNode.ID, consumerModule, providerNode.ID, local).
Scan(&held.ForConsumer, &held.ForProvider, &held.ConsumerKey, &held.ProviderKey, &held.Origin)
if err == nil && held.ConsumerKey == consumerKey && held.ProviderKey == providerKey {
held.Name, held.Consumer, held.Provider = name, consumer, provider
held.ConsumerModule = consumerModule
held.ConsumerModule, held.Local = consumerModule, local
return held, nil
}
if err == nil && held.Origin == OriginAccepted {
@@ -90,8 +97,8 @@ func (i *Inventory) SecretFor(ctx context.Context, name, consumer, consumerModul
return Secret{}, fmt.Errorf(
"%s's %q credential from %s was accepted from a person, and a sealing key at one end "+
"has changed since. The mesh cannot re-seal a value it does not hold: accept it "+
"again with `secret accept %s %s %s --provider %s`",
consumerModule, name, provider, consumer, consumerModule, name, provider)
"again with `secret accept %s %s %s --provider %s%s`",
consumerModule, name, provider, consumer, consumerModule, name, provider, localFlag(local))
}
// And to the operator, when the mesh has one (novox/hq ADR 0085, amended): the third copy that
@@ -107,19 +114,19 @@ func (i *Inventory) SecretFor(ctx context.Context, name, consumer, consumerModul
forOperator, operatorKey := operatorColumns(operator, blob)
_, err = i.store.Pool().Exec(ctx,
`insert into secret (name, consumer, consumer_module, provider, for_consumer, for_provider,
consumer_key, provider_key, operator_sealed, operator_key)
values ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10)
on conflict (name, consumer, consumer_module, provider) do update set
consumer_key, provider_key, operator_sealed, operator_key, local)
values ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11)
on conflict (name, local, consumer, consumer_module, provider) do update set
for_consumer = excluded.for_consumer, for_provider = excluded.for_provider,
consumer_key = excluded.consumer_key, provider_key = excluded.provider_key,
created_at = now(),
operator_sealed = excluded.operator_sealed, operator_key = excluded.operator_key`,
name, consumerNode.ID, consumerModule, providerNode.ID,
made.ForConsumer, made.ForProvider, made.ConsumerKey, made.ProviderKey, forOperator, operatorKey)
made.ForConsumer, made.ForProvider, made.ConsumerKey, made.ProviderKey, forOperator, operatorKey, local)
if err != nil {
return Secret{}, err
}
return Secret{Name: name, Consumer: consumer, ConsumerModule: consumerModule,
return Secret{Name: name, Consumer: consumer, ConsumerModule: consumerModule, Local: local,
Provider: provider,
ForConsumer: made.ForConsumer, ForProvider: made.ForProvider,
ConsumerKey: made.ConsumerKey, ProviderKey: made.ProviderKey, Origin: OriginMade}, nil
@@ -133,7 +140,31 @@ func (i *Inventory) SecretFor(ctx context.Context, name, consumer, consumerModul
// person can supply. It is the counterpart to AcceptSecretForModule for a module's own secret;
// what differs is that both ends of the pair are sealed to, and that the record says `accepted`
// so a later read never replaces it with a minted one. The plaintext is discarded here.
func (i *Inventory) AcceptSecretForPair(ctx context.Context, name, consumer, consumerModule, provider, value string) error {
func (i *Inventory) AcceptSecretForPair(ctx context.Context, name, consumer, consumerModule, provider, local, value string) error {
// Refused for a requirement the module does not have, or a local it does not keep under it
// (novox/hq 04-ISSUES/078): the credential would sit in the pair unread.
m, err := i.declared(ctx, consumerModule)
if err != nil {
return err
}
if !slices.Contains(m.Requires, name) {
return fmt.Errorf("%s does not require %q; it requires: %s", consumerModule, name, orNone(m.Requires))
}
if locals := m.SecretsMany[name]; len(locals) > 0 {
if local == "" {
return fmt.Errorf("%s keeps several secrets for %q; name one with --local: %s",
consumerModule, name, orNone(sortedNames(locals)))
}
if _, kept := locals[local]; !kept {
return fmt.Errorf("%s does not keep %q for %q; it keeps: %s",
consumerModule, local, name, orNone(sortedNames(locals)))
}
} else if m.Secrets[name] == "" {
return fmt.Errorf("%s requires %q but keeps no secret for it, so a delivered value would sit unread; "+
"it keeps secrets for: %s", consumerModule, name, orNone(sortedNames(m.Secrets)))
} else if local != "" {
return fmt.Errorf("%s keeps one secret for %q, not several; drop --local", consumerModule, name)
}
consumerKey, err := i.SealingKeyOf(ctx, consumer)
if err != nil {
return err
@@ -165,16 +196,16 @@ func (i *Inventory) AcceptSecretForPair(ctx context.Context, name, consumer, con
}
_, err = i.store.Pool().Exec(ctx,
`insert into secret (name, consumer, consumer_module, provider, for_consumer, for_provider,
consumer_key, provider_key, operator_sealed, operator_key, origin)
values ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11)
on conflict (name, consumer, consumer_module, provider) do update set
consumer_key, provider_key, operator_sealed, operator_key, origin, local)
values ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12)
on conflict (name, local, consumer, consumer_module, provider) do update set
for_consumer = excluded.for_consumer, for_provider = excluded.for_provider,
consumer_key = excluded.consumer_key, provider_key = excluded.provider_key,
created_at = now(), origin = excluded.origin,
operator_sealed = excluded.operator_sealed, operator_key = excluded.operator_key`,
name, consumerNode.ID, consumerModule, providerNode.ID,
sealed.ForConsumer, sealed.ForProvider, sealed.ConsumerKey, sealed.ProviderKey,
forOperator, operatorKey, OriginAccepted)
forOperator, operatorKey, OriginAccepted, local)
return err
}
@@ -190,7 +221,7 @@ func (i *Inventory) AcceptSecretForPair(ctx context.Context, name, consumer, con
// **An accepted credential is not rotated.** The mesh did not make it and cannot make its
// replacement; deleting it would have the next read mint one, which is exactly the wrong value
// delivered with the mesh insisting it was (novox/hq 04-ISSUES/070). Refused, and the remedy named.
func (i *Inventory) RotateSecret(ctx context.Context, name, consumer, consumerModule, provider string) error {
func (i *Inventory) RotateSecret(ctx context.Context, name, consumer, consumerModule, provider, local string) error {
consumerNode, err := i.NodeByName(ctx, consumer)
if err != nil {
return err
@@ -202,19 +233,19 @@ func (i *Inventory) RotateSecret(ctx context.Context, name, consumer, consumerMo
var origin string
err = i.store.Pool().QueryRow(ctx,
`select origin from secret where name = $1 and consumer = $2 and consumer_module = $3
and provider = $4`,
name, consumerNode.ID, consumerModule, providerNode.ID).Scan(&origin)
and provider = $4 and local = $5`,
name, consumerNode.ID, consumerModule, providerNode.ID, local).Scan(&origin)
if err == nil && origin == OriginAccepted {
return fmt.Errorf(
"%s's %q credential from %s was accepted from a person, and the mesh cannot make "+
"its replacement. Accept the new value instead: `secret accept %s %s %s "+
"--provider %s --from <file>`",
consumerModule, name, provider, consumer, consumerModule, name, provider)
"--provider %s%s --from <file>`",
consumerModule, name, provider, consumer, consumerModule, name, provider, localFlag(local))
}
_, err = i.store.Pool().Exec(ctx,
`delete from secret where name = $1 and consumer = $2 and consumer_module = $3
and provider = $4`,
name, consumerNode.ID, consumerModule, providerNode.ID)
and provider = $4 and local = $5`,
name, consumerNode.ID, consumerModule, providerNode.ID, local)
return err
}
@@ -225,9 +256,9 @@ func (i *Inventory) SecretsFrom(ctx context.Context, provider string) ([]Secret,
return nil, err
}
rows, err := i.store.Pool().Query(ctx,
`select s.name, c.name, s.consumer_module, s.for_provider from secret s
`select s.name, c.name, s.consumer_module, s.local, s.for_provider from secret s
join node c on c.id = s.consumer
where s.provider = $1 order by s.name, c.name, s.consumer_module`, providerNode.ID)
where s.provider = $1 order by s.name, c.name, s.consumer_module, s.local`, providerNode.ID)
if err != nil {
return nil, err
}
@@ -236,7 +267,7 @@ func (i *Inventory) SecretsFrom(ctx context.Context, provider string) ([]Secret,
var out []Secret
for rows.Next() {
s := Secret{Provider: provider}
if err := rows.Scan(&s.Name, &s.Consumer, &s.ConsumerModule, &s.ForProvider); err != nil {
if err := rows.Scan(&s.Name, &s.Consumer, &s.ConsumerModule, &s.Local, &s.ForProvider); err != nil {
return nil, err
}
out = append(out, s)
@@ -359,6 +390,16 @@ func (i *Inventory) SecretForModule(ctx context.Context, node, module, name stri
// Sealed on the way in and the plaintext discarded, exactly as a generated one is — so the only
// difference between the two is where the value came from.
func (i *Inventory) AcceptSecretForModule(ctx context.Context, node, module, name, value string) error {
// Refused for a name the module does not declare. A value stored under a name nothing reads
// is a delivery that changed nothing and reported success — the shape of failure the mesh
// is built to refuse (novox/hq 04-ISSUES/078).
m, err := i.declared(ctx, module)
if err != nil {
return err
}
if _, own := m.OwnSecrets[name]; !own {
return fmt.Errorf("%s does not declare %q as an own secret; %s", module, name, declaresOwn(m))
}
key, err := i.SealingKeyOf(ctx, node)
if err != nil {
return err
@@ -401,7 +442,9 @@ type Holder struct {
// ConsumerModule is which module on that machine holds it. Part of what identifies a
// credential (novox/hq 04-ISSUES/022), so rotating one consumer's does not touch another's.
ConsumerModule string
Provider string
// Local is the credential's name inside the consumer where it holds several (ADR 0094).
Local string
Provider string
}
// HoldersOf is every pair sharing a credential for one provision.
@@ -415,11 +458,11 @@ type Holder struct {
// Empty consumer means all of them.
func (i *Inventory) HoldersOf(ctx context.Context, provision, consumer string) ([]Holder, error) {
rows, err := i.store.Pool().Query(ctx,
`select s.name, c.name, s.consumer_module, p.name from secret s
`select s.name, c.name, s.consumer_module, s.local, p.name from secret s
join node c on c.id = s.consumer
join node p on p.id = s.provider
where s.name = $1 and ($2 = '' or c.name = $2)
order by c.name, s.consumer_module, p.name`, provision, consumer)
order by c.name, s.consumer_module, s.local, p.name`, provision, consumer)
if err != nil {
return nil, err
}
@@ -428,10 +471,55 @@ func (i *Inventory) HoldersOf(ctx context.Context, provision, consumer string) (
var out []Holder
for rows.Next() {
var h Holder
if err := rows.Scan(&h.Provision, &h.Consumer, &h.ConsumerModule, &h.Provider); err != nil {
if err := rows.Scan(&h.Provision, &h.Consumer, &h.ConsumerModule, &h.Local, &h.Provider); err != nil {
return nil, err
}
out = append(out, h)
}
return out, rows.Err()
}
// localFlag is the `--local` a remedy has to name where a credential has a local name.
func localFlag(local string) string {
if local == "" {
return ""
}
return " --local " + local
}
// declared is the manifest the mesh holds for a module — what a delivered value is checked
// against, so a delivery for a name the module does not have is refused rather than stored.
func (i *Inventory) declared(ctx context.Context, module string) (catalogue.Manifest, error) {
known, err := i.Catalogue(ctx)
if err != nil {
return catalogue.Manifest{}, err
}
m, ok := known[module]
if !ok {
return catalogue.Manifest{}, fmt.Errorf("%s is not a module the mesh knows; `module add` it first", module)
}
return m, nil
}
func declaresOwn(m catalogue.Manifest) string {
if len(m.OwnSecrets) == 0 {
return "it declares no own secrets"
}
return "it declares: " + strings.Join(sortedNames(m.OwnSecrets), ", ")
}
func sortedNames(of map[string]string) []string {
names := make([]string, 0, len(of))
for name := range of {
names = append(names, name)
}
sort.Strings(names)
return names
}
func orNone(names []string) string {
if len(names) == 0 {
return "none"
}
return strings.Join(names, ", ")
}
+198 -37
View File
@@ -6,6 +6,7 @@ import (
"crypto/rand"
"encoding/base64"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/secrets"
"strings"
"testing"
@@ -51,8 +52,11 @@ func twoNodesWithKeys(t *testing.T) (*Inventory, context.Context) {
// before it can (novox/hq 04-ISSUES/022). Two of them, because "two consumers on one node"
// is the case that key exists for.
for _, m := range []string{"gitea", "keycloak"} {
if err := inv.RegisterModule(ctx, catalogue.Manifest{Module: m, Version: "1"},
Source{}); err != nil {
// Each requires what a test delivers to it: a pair credential is refused for a
// requirement the module does not have (novox/hq 04-ISSUES/078).
if err := inv.RegisterModule(ctx, catalogue.Manifest{Module: m, Version: "1",
Requires: []string{"secret", "postgres-database", "object-store"},
Secrets: map[string]string{"secret": "/run/secret", "postgres-database": "/run/pg"}}, Source{}); err != nil {
t.Fatal(err)
}
}
@@ -63,11 +67,11 @@ func TestASecretIsMadeOnceAndKept(t *testing.T) {
// Regenerating on every declaration would restart both ends on every push, and — worse — the
// password a provider was told to create would never be the one its consumer was given.
inv, ctx := twoNodesWithKeys(t)
first, err := inv.SecretFor(ctx, "postgres-database", "consumer", "gitea", "provider")
first, err := inv.SecretFor(ctx, "postgres-database", "consumer", "gitea", "provider", "")
if err != nil {
t.Fatal(err)
}
second, err := inv.SecretFor(ctx, "postgres-database", "consumer", "gitea", "provider")
second, err := inv.SecretFor(ctx, "postgres-database", "consumer", "gitea", "provider", "")
if err != nil {
t.Fatal(err)
}
@@ -81,7 +85,7 @@ func TestTheStoredSecretIsNotTheSecret(t *testing.T) {
// what an encrypted column does not achieve, because whoever runs the control plane can read
// through it.
inv, ctx := twoNodesWithKeys(t)
got, err := inv.SecretFor(ctx, "postgres-database", "consumer", "gitea", "provider")
got, err := inv.SecretFor(ctx, "postgres-database", "consumer", "gitea", "provider", "")
if err != nil {
t.Fatal(err)
}
@@ -114,7 +118,7 @@ func TestANewSealingKeyMeansANewSecret(t *testing.T) {
// A node that rejoined generated a new key and can no longer open what was sealed to the old
// one. Keeping the blob would deliver something unreadable for ever, reported as configured.
inv, ctx := twoNodesWithKeys(t)
before, err := inv.SecretFor(ctx, "postgres-database", "consumer", "gitea", "provider")
before, err := inv.SecretFor(ctx, "postgres-database", "consumer", "gitea", "provider", "")
if err != nil {
t.Fatal(err)
}
@@ -126,7 +130,7 @@ func TestANewSealingKeyMeansANewSecret(t *testing.T) {
if err := inv.RecordSealingKey(ctx, node.ID, fresh); err != nil {
t.Fatal(err)
}
after, err := inv.SecretFor(ctx, "postgres-database", "consumer", "gitea", "provider")
after, err := inv.SecretFor(ctx, "postgres-database", "consumer", "gitea", "provider", "")
if err != nil {
t.Fatal(err)
}
@@ -142,14 +146,14 @@ func TestANewSealingKeyMeansANewSecret(t *testing.T) {
func TestRotatingReachesBothEnds(t *testing.T) {
inv, ctx := twoNodesWithKeys(t)
before, err := inv.SecretFor(ctx, "postgres-database", "consumer", "gitea", "provider")
before, err := inv.SecretFor(ctx, "postgres-database", "consumer", "gitea", "provider", "")
if err != nil {
t.Fatal(err)
}
if err := inv.RotateSecret(ctx, "postgres-database", "consumer", "gitea", "provider"); err != nil {
if err := inv.RotateSecret(ctx, "postgres-database", "consumer", "gitea", "provider", ""); err != nil {
t.Fatal(err)
}
after, err := inv.SecretFor(ctx, "postgres-database", "consumer", "gitea", "provider")
after, err := inv.SecretFor(ctx, "postgres-database", "consumer", "gitea", "provider", "")
if err != nil {
t.Fatal(err)
}
@@ -184,7 +188,7 @@ func TestAProviderIsToldEveryCredentialItMustCreate(t *testing.T) {
t.Fatal(err)
}
for _, who := range []string{"consumer", "second-consumer"} {
if _, err := inv.SecretFor(ctx, "postgres-database", who, "gitea", "provider"); err != nil {
if _, err := inv.SecretFor(ctx, "postgres-database", who, "gitea", "provider", ""); err != nil {
t.Fatal(err)
}
}
@@ -215,7 +219,7 @@ func TestANodeWithNoSealingKeyCannotBeGivenASecret(t *testing.T) {
t.Fatal(err)
}
}
_, err := inv.SecretFor(ctx, "postgres-database", "consumer", "gitea", "provider")
_, err := inv.SecretFor(ctx, "postgres-database", "consumer", "gitea", "provider", "")
if err == nil {
t.Fatal("a credential was made for nodes that cannot open one")
}
@@ -226,7 +230,7 @@ func TestANodeWithNoSealingKeyCannotBeGivenASecret(t *testing.T) {
func TestSecretsGoWhenANodeLeaves(t *testing.T) {
inv, ctx := twoNodesWithKeys(t)
if _, err := inv.SecretFor(ctx, "postgres-database", "consumer", "gitea", "provider"); err != nil {
if _, err := inv.SecretFor(ctx, "postgres-database", "consumer", "gitea", "provider", ""); err != nil {
t.Fatal(err)
}
if _, err := inv.store.Pool().Exec(ctx, `delete from node where name = 'consumer'`); err != nil {
@@ -349,7 +353,7 @@ func TestACredentialGoesWhenTheConsumerStopsAskingForIt(t *testing.T) {
if err := inv.Assign(ctx, "consumer", "meshboard"); err != nil {
t.Fatal(err)
}
if _, err := inv.SecretFor(ctx, "postgres-database", "consumer", "gitea", "provider"); err != nil {
if _, err := inv.SecretFor(ctx, "postgres-database", "consumer", "gitea", "provider", ""); err != nil {
t.Fatal(err)
}
@@ -379,7 +383,7 @@ func TestACredentialGoesWhenEitherMachineDoes(t *testing.T) {
// The case that must not leave a live login behind: a machine removed from the mesh. Its
// credentials go with it, and the provider stops being told to keep them.
inv, ctx := twoNodesWithKeys(t)
if _, err := inv.SecretFor(ctx, "postgres-database", "consumer", "gitea", "provider"); err != nil {
if _, err := inv.SecretFor(ctx, "postgres-database", "consumer", "gitea", "provider", ""); err != nil {
t.Fatal(err)
}
if _, err := inv.store.Pool().Exec(ctx, `delete from node where name = 'consumer'`); err != nil {
@@ -403,8 +407,8 @@ func TestACredentialGoesWhenEitherMachineDoes(t *testing.T) {
// secret was delivered — which it was.
func TestASecretTheMeshWasGivenIsNotReinventedWhenTheMachineRejoins(t *testing.T) {
inv, ctx := twoNodesWithKeys(t)
if err := inv.RegisterModule(ctx, catalogue.Manifest{Module: "builder", Version: "1"},
Source{}); err != nil {
if err := inv.RegisterModule(ctx, catalogue.Manifest{Module: "builder", Version: "1",
OwnSecrets: map[string]string{"broker": "/run/broker"}}, Source{}); err != nil {
t.Fatal(err)
}
const url = "amqps://builder:the-password-the-broker-was-told@broker/"
@@ -435,8 +439,8 @@ func TestASecretTheMeshWasGivenIsNotReinventedWhenTheMachineRejoins(t *testing.T
// that would make the refusal above useless if it were wrong.
func TestASecretTheMeshWasGivenSurvivesAnOrdinaryPush(t *testing.T) {
inv, ctx := twoNodesWithKeys(t)
if err := inv.RegisterModule(ctx, catalogue.Manifest{Module: "builder", Version: "1"},
Source{}); err != nil {
if err := inv.RegisterModule(ctx, catalogue.Manifest{Module: "builder", Version: "1",
OwnSecrets: map[string]string{"broker": "/run/broker"}}, Source{}); err != nil {
t.Fatal(err)
}
if err := inv.AcceptSecretForModule(ctx, "consumer", "builder", "broker",
@@ -473,12 +477,12 @@ func TestEveryHolderOfACredentialCanBeNamed(t *testing.T) {
t.Fatal(err)
}
for _, consumer := range []string{"consumer", "third"} {
if _, err := inv.SecretFor(ctx, "postgres-database", consumer, "gitea", "provider"); err != nil {
if _, err := inv.SecretFor(ctx, "postgres-database", consumer, "gitea", "provider", ""); err != nil {
t.Fatal(err)
}
}
// And one for a different provision, which must not be swept up.
if _, err := inv.SecretFor(ctx, "cache", "consumer", "gitea", "provider"); err != nil {
if _, err := inv.SecretFor(ctx, "cache", "consumer", "gitea", "provider", ""); err != nil {
t.Fatal(err)
}
@@ -509,14 +513,14 @@ func TestEveryHolderOfACredentialCanBeNamed(t *testing.T) {
// And rotating gives both ends a new credential, together — the same one.
func TestRotatingGivesBothEndsTheSameNewCredential(t *testing.T) {
inv, ctx := twoNodesWithKeys(t)
before, err := inv.SecretFor(ctx, "postgres-database", "consumer", "gitea", "provider")
before, err := inv.SecretFor(ctx, "postgres-database", "consumer", "gitea", "provider", "")
if err != nil {
t.Fatal(err)
}
if err := inv.RotateSecret(ctx, "postgres-database", "consumer", "gitea", "provider"); err != nil {
if err := inv.RotateSecret(ctx, "postgres-database", "consumer", "gitea", "provider", ""); err != nil {
t.Fatal(err)
}
after, err := inv.SecretFor(ctx, "postgres-database", "consumer", "gitea", "provider")
after, err := inv.SecretFor(ctx, "postgres-database", "consumer", "gitea", "provider", "")
if err != nil {
t.Fatal(err)
}
@@ -543,14 +547,14 @@ func TestRotatingGivesBothEndsTheSameNewCredential(t *testing.T) {
if err := inv.RecordSealingKey(ctx, third.ID, key); err != nil {
t.Fatal(err)
}
untouched, err := inv.SecretFor(ctx, "postgres-database", "third", "gitea", "provider")
untouched, err := inv.SecretFor(ctx, "postgres-database", "third", "gitea", "provider", "")
if err != nil {
t.Fatal(err)
}
if err := inv.RotateSecret(ctx, "postgres-database", "consumer", "gitea", "provider"); err != nil {
if err := inv.RotateSecret(ctx, "postgres-database", "consumer", "gitea", "provider", ""); err != nil {
t.Fatal(err)
}
again, err := inv.SecretFor(ctx, "postgres-database", "third", "gitea", "provider")
again, err := inv.SecretFor(ctx, "postgres-database", "third", "gitea", "provider", "")
if err != nil {
t.Fatal(err)
}
@@ -565,17 +569,17 @@ func TestRotatingGivesBothEndsTheSameNewCredential(t *testing.T) {
// because it cannot make the replacement.
func TestAnAcceptedPairCredentialIsKeptAndNeverRemade(t *testing.T) {
inv, ctx := twoNodesWithKeys(t)
if err := inv.AcceptSecretForPair(ctx, "secret", "consumer", "gitea", "provider", "hunter2"); err != nil {
if err := inv.AcceptSecretForPair(ctx, "secret", "consumer", "gitea", "provider", "", "hunter2"); err != nil {
t.Fatal(err)
}
got, err := inv.SecretFor(ctx, "secret", "consumer", "gitea", "provider")
got, err := inv.SecretFor(ctx, "secret", "consumer", "gitea", "provider", "")
if err != nil {
t.Fatal(err)
}
if got.Origin != OriginAccepted {
t.Fatalf("an accepted credential reads back as %q", got.Origin)
}
again, err := inv.SecretFor(ctx, "secret", "consumer", "gitea", "provider")
again, err := inv.SecretFor(ctx, "secret", "consumer", "gitea", "provider", "")
if err != nil {
t.Fatal(err)
}
@@ -584,15 +588,15 @@ func TestAnAcceptedPairCredentialIsKeptAndNeverRemade(t *testing.T) {
}
// Rotation is refused, and says what to do instead.
err = inv.RotateSecret(ctx, "secret", "consumer", "gitea", "provider")
err = inv.RotateSecret(ctx, "secret", "consumer", "gitea", "provider", "")
if err == nil || !strings.Contains(err.Error(), "secret accept") {
t.Fatalf("rotating an accepted credential was not refused with the remedy: %v", err)
}
// And a made one still rotates.
if _, err := inv.SecretFor(ctx, "postgres-database", "consumer", "gitea", "provider"); err != nil {
if _, err := inv.SecretFor(ctx, "postgres-database", "consumer", "gitea", "provider", ""); err != nil {
t.Fatal(err)
}
if err := inv.RotateSecret(ctx, "postgres-database", "consumer", "gitea", "provider"); err != nil {
if err := inv.RotateSecret(ctx, "postgres-database", "consumer", "gitea", "provider", ""); err != nil {
t.Fatalf("a made credential no longer rotates: %v", err)
}
}
@@ -601,7 +605,7 @@ func TestAnAcceptedPairCredentialIsKeptAndNeverRemade(t *testing.T) {
// re-seal what it does not hold: refused aloud, never quietly replaced by a minted one.
func TestAnAcceptedPairCredentialIsNotRemadeWhenAKeyChanges(t *testing.T) {
inv, ctx := twoNodesWithKeys(t)
if err := inv.AcceptSecretForPair(ctx, "secret", "consumer", "gitea", "provider", "hunter2"); err != nil {
if err := inv.AcceptSecretForPair(ctx, "secret", "consumer", "gitea", "provider", "", "hunter2"); err != nil {
t.Fatal(err)
}
node, err := inv.NodeByName(ctx, "consumer")
@@ -612,15 +616,172 @@ func TestAnAcceptedPairCredentialIsNotRemadeWhenAKeyChanges(t *testing.T) {
if err := inv.RecordSealingKey(ctx, node.ID, fresh); err != nil {
t.Fatal(err)
}
_, err = inv.SecretFor(ctx, "secret", "consumer", "gitea", "provider")
_, err = inv.SecretFor(ctx, "secret", "consumer", "gitea", "provider", "")
if err == nil || !strings.Contains(err.Error(), "accept it again") {
t.Fatalf("an accepted credential was remade, or refused without the remedy: %v", err)
}
// Accepting it again is the remedy, and it works.
if err := inv.AcceptSecretForPair(ctx, "secret", "consumer", "gitea", "provider", "hunter3"); err != nil {
if err := inv.AcceptSecretForPair(ctx, "secret", "consumer", "gitea", "provider", "", "hunter3"); err != nil {
t.Fatal(err)
}
if _, err := inv.SecretFor(ctx, "secret", "consumer", "gitea", "provider"); err != nil {
if _, err := inv.SecretFor(ctx, "secret", "consumer", "gitea", "provider", ""); err != nil {
t.Fatal(err)
}
}
// Two secrets from one provider to one module are two credentials (novox/hq 04-ISSUES/069, ADR
// 0094): keyed on the local name, made and rotated apart, and listed apart for the provider.
func TestTwoLocalNamesAreTwoCredentials(t *testing.T) {
inv, ctx := twoNodesWithKeys(t)
key, err := inv.SecretFor(ctx, "secret", "consumer", "gitea", "provider", "root-key")
if err != nil {
t.Fatal(err)
}
pass, err := inv.SecretFor(ctx, "secret", "consumer", "gitea", "provider", "root-pass")
if err != nil {
t.Fatal(err)
}
if key.ForConsumer == pass.ForConsumer {
t.Fatal("two local names were given one credential")
}
if err := inv.RotateSecret(ctx, "secret", "consumer", "gitea", "provider", "root-key"); err != nil {
t.Fatal(err)
}
keyAgain, err := inv.SecretFor(ctx, "secret", "consumer", "gitea", "provider", "root-key")
if err != nil {
t.Fatal(err)
}
passAgain, err := inv.SecretFor(ctx, "secret", "consumer", "gitea", "provider", "root-pass")
if err != nil {
t.Fatal(err)
}
if keyAgain.ForConsumer == key.ForConsumer || passAgain.ForConsumer != pass.ForConsumer {
t.Fatal("rotating one local name touched the other, or neither")
}
holders, err := inv.HoldersOf(ctx, "secret", "")
if err != nil {
t.Fatal(err)
}
var locals []string
for _, h := range holders {
locals = append(locals, h.Local)
}
if strings.Join(locals, ",") != "root-key,root-pass" {
t.Fatalf("the holders are listed apart, by local name: %v", holders)
}
from, err := inv.SecretsFrom(ctx, "provider")
if err != nil {
t.Fatal(err)
}
if len(from) != 2 || from[0].Local == from[1].Local {
t.Fatalf("the provider is told two credentials to create: %+v", from)
}
}
// The operator can recover either of two local names apart (review C3).
func TestTheOperatorRecoversEachLocalNameApart(t *testing.T) {
inv, ctx := twoNodesWithKeys(t)
pub, _, err := secrets.Keypair()
if err != nil {
t.Fatal(err)
}
if _, err := inv.SetOperatorKey(ctx, pub); err != nil {
t.Fatal(err)
}
for _, local := range []string{"root-key", "root-pass"} {
if _, err := inv.SecretFor(ctx, "secret", "consumer", "gitea", "provider", local); err != nil {
t.Fatal(err)
}
}
key, err := inv.KeptSecret(ctx, "consumer", "gitea", "secret", "provider", "root-key")
if err != nil {
t.Fatal(err)
}
pass, err := inv.KeptSecret(ctx, "consumer", "gitea", "secret", "provider", "root-pass")
if err != nil {
t.Fatal(err)
}
if key.Local != "root-key" || pass.Local != "root-pass" || key.Kind != "pair" {
t.Fatalf("recovery does not tell the two apart: %+v %+v", key, pass)
}
kept, _, _, err := inv.KeptForOperator(ctx)
if err != nil {
t.Fatal(err)
}
var locals []string
for _, k := range kept {
if k.Kind == "pair" {
locals = append(locals, k.Local)
}
}
if strings.Join(locals, ",") != "root-key,root-pass" {
t.Fatalf("the export does not name the local names: %v", kept)
}
}
// **A delivered secret is accepted only under a name the module declares** (novox/hq
// 04-ISSUES/078). A value stored under a name nothing reads is a delivery that changed nothing
// and reported success; refused, naming what the module does declare.
func TestADeliveredSecretIsRefusedUnderANameTheModuleDoesNotDeclare(t *testing.T) {
inv, ctx := twoNodesWithKeys(t)
if err := inv.RegisterModule(ctx, catalogue.Manifest{Module: "step-ca", Version: "1",
OwnSecrets: map[string]string{"password": "/run/password"}}, Source{}); err != nil {
t.Fatal(err)
}
err := inv.AcceptSecretForModule(ctx, "consumer", "step-ca", "root-key", "not-a-key")
if err == nil {
t.Fatal("a secret delivered under a name the module does not declare was accepted")
}
for _, want := range []string{"root-key", "password"} {
if !strings.Contains(err.Error(), want) {
t.Errorf("the refusal does not say %q: %v", want, err)
}
}
if err := inv.AcceptSecretForModule(ctx, "consumer", "step-ca", "password", "hunter2"); err != nil {
t.Fatalf("a secret delivered under a declared name was refused: %v", err)
}
// A module the mesh does not know is said, not stored.
err = inv.AcceptSecretForModule(ctx, "consumer", "nobody", "password", "hunter2")
if err == nil || !strings.Contains(err.Error(), "module add") {
t.Errorf("a delivery to an unknown module was not refused with the remedy: %v", err)
}
}
func TestADeliveredPairCredentialIsRefusedForARequirementTheModuleDoesNotHave(t *testing.T) {
inv, ctx := twoNodesWithKeys(t)
// gitea requires secret, postgres-database and object-store (the fixture); it keeps a secret
// for the first two only, and requires no cache at all.
err := inv.AcceptSecretForPair(ctx, "redis-cache", "consumer", "gitea", "provider", "", "hunter2")
if err == nil || !strings.Contains(err.Error(), "postgres-database") {
t.Fatalf("a pair credential for a requirement the module does not have was not refused naming what it requires: %v", err)
}
err = inv.AcceptSecretForPair(ctx, "object-store", "consumer", "gitea", "provider", "", "hunter2")
if err == nil || !strings.Contains(err.Error(), "keeps no secret") {
t.Fatalf("a pair credential for a requirement the module keeps no secret for was not refused: %v", err)
}
err = inv.AcceptSecretForPair(ctx, "secret", "consumer", "gitea", "provider", "extra", "hunter2")
if err == nil || !strings.Contains(err.Error(), "drop --local") {
t.Fatalf("a local named where the module keeps one secret was not refused: %v", err)
}
if err := inv.AcceptSecretForPair(ctx, "secret", "consumer", "gitea", "provider", "", "hunter2"); err != nil {
t.Fatalf("a delivery for a requirement the module keeps one secret for was refused: %v", err)
}
// A module keeping several secrets for one requirement (ADR 0094) takes a delivery only
// under one of its locals.
m := catalogue.Manifest{Module: "mailu", Version: "1", Requires: []string{"secret"},
SecretsMany: map[string]map[string]string{"secret": {"admin": "/run/admin", "api-token": "/run/api-token"}}}
if err := inv.RegisterModule(ctx, m, Source{}); err != nil {
t.Fatal(err)
}
err = inv.AcceptSecretForPair(ctx, "secret", "consumer", "mailu", "provider", "", "hunter2")
if err == nil || !strings.Contains(err.Error(), "--local") {
t.Errorf("a delivery to a module keeping several secrets, with no local named, was not refused: %v", err)
}
err = inv.AcceptSecretForPair(ctx, "secret", "consumer", "mailu", "provider", "secret-key", "hunter2")
if err == nil || !strings.Contains(err.Error(), "api-token") {
t.Errorf("a delivery under a local the module does not keep was not refused naming the ones it keeps: %v", err)
}
if err := inv.AcceptSecretForPair(ctx, "secret", "consumer", "mailu", "provider", "admin", "hunter2"); err != nil {
t.Errorf("a delivery under a kept local was refused: %v", err)
}
}
+375
View File
@@ -0,0 +1,375 @@
package inventory
import (
"context"
"encoding/json"
"errors"
"fmt"
"net/netip"
"strings"
"time"
"github.com/jackc/pgx/v5"
)
// The tunnel a node found on its machine, and the peers it carried (novox/hq ADR 0105).
//
// On an adopted node that is the hub, the private network takes over the tunnel it finds: its
// private key, its port, its address and range, and every peer the found interface had. The node
// presents what it found when it enrols, in the same breath as its keys — the found tunnel's key is
// its key on the private network from then on — and the mesh composes every address from it: the
// hub's is the tunnel's, the range is the tunnel's, and a peer that enrols keeps the address the
// tunnel already had for its key.
// Tunnel is what a node found on its machine, as it presented it. No private key: the node keeps
// it as its own overlay key, sealed like any own secret, and the mesh records only the public half
// — which is then the node's overlay key too.
type Tunnel struct {
// Interface, Unit and Config are what the host takes over: the found interface, the unit
// that raised it, and its configuration file, which is kept like any held file.
Interface string `json:"interface"`
Unit string `json:"unit"`
Config string `json:"config"`
// Port is the port the found interface listened on — one the hosting provider already lets
// through, which is why it is worth taking.
Port int `json:"port"`
// Address is the interface's own address with its prefix length, 192.0.2.1/24; Range is the
// network that prefix names, 192.0.2.0/24.
Address string `json:"address"`
Range string `json:"range"`
// PublicKey is the found interface's, which every peer knows the tunnel by.
PublicKey string `json:"public_key"`
// Peers are the found interface's peers: each a public key and the address the tunnel routed
// to it.
Peers []TunnelPeer `json:"peers,omitempty"`
// At is when the node presented it; zero on a tunnel not yet recorded.
At time.Time `json:"at,omitempty"`
}
// TunnelPeer is one peer of a found tunnel.
type TunnelPeer struct {
PublicKey string `json:"public_key"`
// Address is the one host address the tunnel routed to the peer, without a prefix.
Address string `json:"address"`
}
// Carried is what a node last said about carrying the tunnel it found: the found interface down
// and disabled, the mesh's up in its place with the found key.
type Carried struct {
Interface string `json:"interface"`
Port int `json:"port"`
Range string `json:"range"`
Peers int `json:"peers"`
// State is one of the CarriedStates: the found interface is still up and the mesh's is not
// (not taken), the found one is down and the mesh's up with its key (taken), or the found one
// is down and the mesh's is not up — the one state where the peers reach nothing. Note is
// what the host did about it, when it did something. Kept is where the found configuration's
// original was kept.
State string `json:"state"`
Note string `json:"note,omitempty"`
Kept string `json:"kept,omitempty"`
At time.Time `json:"at"`
}
// The states a carried tunnel's account can be in, as the host says them.
const (
CarriedNotTaken = "not-taken"
CarriedTaken = "taken"
CarriedDown = "down"
)
// CarriedPeer is one peer of the adopted tunnel as the mesh holds it: a peer of the tunnel, and
// — once a node enrols with that key — a node of the mesh as well.
type CarriedPeer struct {
PublicKey string
Address string
// EnrolledAs names the node that enrolled with this key, or is empty while none has.
EnrolledAs string
}
// ErrNoTunnel is asking about a tunnel on a node that presented none.
var ErrNoTunnel = errors.New("that node presented no tunnel")
// RecordTunnel keeps what a node presented, replacing what was there: the question is the tunnel
// as the node found it now. The peers are replaced whole for the same reason.
func (i *Inventory) RecordTunnel(ctx context.Context, nodeID string, t Tunnel) error {
if strings.TrimSpace(t.Interface) == "" || strings.TrimSpace(t.PublicKey) == "" {
return errors.New("a found tunnel names its interface and its public key, and this names neither")
}
if _, err := netip.ParsePrefix(t.Range); err != nil {
return fmt.Errorf("the found tunnel's range %q is not a range: %w", t.Range, err)
}
address, err := netip.ParsePrefix(t.Address)
if err != nil {
return fmt.Errorf("the found tunnel's address %q is not an address with a prefix: %w", t.Address, err)
}
peers := make([]TunnelPeer, 0, len(t.Peers))
for _, p := range t.Peers {
host, single := peerHost(p.Address)
if !single {
// A peer routed a range rather than one address is a spoke's view of its hub — the
// predecessor gives a spoke the whole subnet through the hub — and a hub is not a peer
// the mesh carries. Skipped, not refused: a spoke enrols with what it found, and only
// the hub's peers are ever carried (novox/hq ADR 0105).
continue
}
if !address.Masked().Contains(host) {
return fmt.Errorf("the found tunnel's peer %s is routed at %s, outside the tunnel's %s",
shortKey(p.PublicKey), host, t.Range)
}
peers = append(peers, TunnelPeer{PublicKey: p.PublicKey, Address: host.String()})
}
t.Peers = nil
t.At = time.Now().UTC()
raw, err := json.Marshal(t)
if err != nil {
return err
}
tx, err := i.store.Pool().Begin(ctx)
if err != nil {
return err
}
defer func() { _ = tx.Rollback(context.WithoutCancel(ctx)) }()
if _, err := tx.Exec(ctx, `update node set tunnel = $2 where id = $1`, nodeID, raw); err != nil {
return err
}
if _, err := tx.Exec(ctx, `delete from tunnel_peer where node = $1`, nodeID); err != nil {
return err
}
for _, p := range peers {
if _, err := tx.Exec(ctx,
`insert into tunnel_peer (node, public_key, address) values ($1, $2, $3::inet)`,
nodeID, p.PublicKey, p.Address); err != nil {
return fmt.Errorf("recording the found tunnel's peer %s: %w", shortKey(p.PublicKey), err)
}
}
return tx.Commit(ctx)
}
// peerHost is the one host address a peer's allowed address names — a bare address, or a /32
// (or /128) — and false for anything wider or unreadable: a peer routed a whole range is not a
// machine with an address the mesh could give a node.
func peerHost(allowed string) (netip.Addr, bool) {
allowed = strings.TrimSpace(allowed)
if a, err := netip.ParseAddr(allowed); err == nil {
return a, true
}
p, err := netip.ParsePrefix(allowed)
if err != nil || !p.IsSingleIP() {
return netip.Addr{}, false
}
return p.Addr(), true
}
func shortKey(key string) string {
if len(key) > 8 {
return key[:8] + "…"
}
return key
}
// TunnelOf is the tunnel a node presented, with its peers, or ErrNoTunnel.
func (i *Inventory) TunnelOf(ctx context.Context, name string) (Tunnel, error) {
var raw []byte
var id string
err := i.store.Pool().QueryRow(ctx,
`select id, tunnel from node where name = $1`, name).Scan(&id, &raw)
if errors.Is(err, pgx.ErrNoRows) {
return Tunnel{}, fmt.Errorf("%w: %s", ErrNoSuchNode, name)
}
if err != nil {
return Tunnel{}, err
}
if len(raw) == 0 {
return Tunnel{}, fmt.Errorf("%w: %s", ErrNoTunnel, name)
}
var t Tunnel
if err := json.Unmarshal(raw, &t); err != nil {
return Tunnel{}, err
}
t.Peers, err = i.tunnelPeers(ctx, id)
return t, err
}
func (i *Inventory) tunnelPeers(ctx context.Context, nodeID string) ([]TunnelPeer, error) {
rows, err := i.store.Pool().Query(ctx,
`select public_key, host(address) from tunnel_peer where node = $1 order by address`, nodeID)
if err != nil {
return nil, err
}
defer rows.Close()
var out []TunnelPeer
for rows.Next() {
var p TunnelPeer
if err := rows.Scan(&p.PublicKey, &p.Address); err != nil {
return nil, err
}
out = append(out, p)
}
return out, rows.Err()
}
// AdoptedTunnel is the tunnel the mesh's private network runs over, if the hub adopted one: the
// hub's found tunnel, when the hub's overlay key is the tunnel's. Absent, the mesh runs on its own
// range — and a hub that found a tunnel but holds another key did not adopt it, which `overlay
// show` says.
//
// The condition on the key is the condition of the whole record: a hub raised with a key of its
// own would drop every peer's packets on the found port (novox/hq ADR 0105, option 2), so the
// tunnel is adopted only when the hub answers to the key its peers know. **Not a condition on the
// node's mode**: the range and the carried peers are facts of the mesh once the tunnel is taken,
// and converging the hub — which flips its mode — must not renumber the mesh or drop the peers
// still reaching it.
func (i *Inventory) AdoptedTunnel(ctx context.Context) (Tunnel, string, bool, error) {
var name string
var key *string
err := i.store.Pool().QueryRow(ctx,
`select name, overlay_key from node where is_hub and tunnel is not null`).
Scan(&name, &key)
if errors.Is(err, pgx.ErrNoRows) {
return Tunnel{}, "", false, nil
}
if err != nil {
return Tunnel{}, "", false, err
}
t, err := i.TunnelOf(ctx, name)
if err != nil {
return Tunnel{}, "", false, err
}
if key == nil || *key != t.PublicKey {
return t, name, false, nil
}
return t, name, true, nil
}
// CarriedPeers is every peer of the adopted tunnel, with the node that enrolled under its key
// where one has: peers of the tunnel, and nodes of the mesh once they enrol. Empty when the hub
// adopted no tunnel.
func (i *Inventory) CarriedPeers(ctx context.Context) ([]CarriedPeer, error) {
rows, err := i.store.Pool().Query(ctx,
`select p.public_key, host(p.address), coalesce(n.name, '')
from tunnel_peer p
join node hub on hub.id = p.node and hub.is_hub
and hub.tunnel is not null and hub.overlay_key = hub.tunnel->>'public_key'
left join node n on n.overlay_key = p.public_key
order by p.address`)
if err != nil {
return nil, err
}
defer rows.Close()
var out []CarriedPeer
for rows.Next() {
var p CarriedPeer
if err := rows.Scan(&p.PublicKey, &p.Address, &p.EnrolledAs); err != nil {
return nil, err
}
out = append(out, p)
}
return out, rows.Err()
}
// FoundTunnel is a node's found tunnel with the node's mode, for composing: the takeover is
// declared to an adopted node only, since only there is a found unit kept to be stopped.
type FoundTunnel struct {
Tunnel
NodeAdopted bool
}
// Tunnels is every node's found tunnel by node name, for the ones whose overlay key is the
// tunnel's — the ones whose private network takes it over. A found tunnel under another key is
// left running beside the mesh's, and ADR 0100's rule that the ranges differ applies to it.
func (i *Inventory) Tunnels(ctx context.Context) (map[string]FoundTunnel, error) {
rows, err := i.store.Pool().Query(ctx,
`select name, tunnel, adopted from node
where tunnel is not null and overlay_key = tunnel->>'public_key'`)
if err != nil {
return nil, err
}
defer rows.Close()
out := map[string]FoundTunnel{}
for rows.Next() {
var name string
var raw []byte
var adopted bool
if err := rows.Scan(&name, &raw, &adopted); err != nil {
return nil, err
}
var t Tunnel
if err := json.Unmarshal(raw, &t); err != nil {
return nil, err
}
out[name] = FoundTunnel{Tunnel: t, NodeAdopted: adopted}
}
return out, rows.Err()
}
// ErrStaleRekey is a rekey that names a previous overlay key other than the one recorded: a
// replay of a rekey already done, or one made against a record that has since moved on.
var ErrStaleRekey = errors.New("the rekey names a previous overlay key that is not the node's current one")
// Rekey records that a node took a found tunnel's key as its overlay key after enrolling (novox/hq
// ADR 0105): the key and the tunnel are recorded as enrolment would have, and a hub is moved to the
// tunnel's address so nothing derived from it is stale. The caller has verified the node signed
// for this; what is checked here is that it follows the record — `previous` is the overlay key the
// node holds now — so the same message cannot be applied twice.
func (i *Inventory) Rekey(ctx context.Context, nodeID, previous, key string, t Tunnel) error {
if key != t.PublicKey {
return errors.New("a rekey takes a tunnel over with the tunnel's own key, and this names another")
}
var current *string
var hub bool
if err := i.store.Pool().QueryRow(ctx,
`select overlay_key, is_hub from node where id = $1`, nodeID).Scan(&current, &hub); err != nil {
return err
}
if (current == nil && previous != "") || (current != nil && *current != previous) {
return ErrStaleRekey
}
if err := i.RecordOverlayKey(ctx, nodeID, key); err != nil {
return err
}
if err := i.RecordTunnel(ctx, nodeID, t); err != nil {
return err
}
if hub {
address, err := netip.ParsePrefix(t.Address)
if err != nil {
return err
}
if _, err := i.place(ctx, nodeID, address.Addr().String()); err != nil {
return err
}
}
return nil
}
// RecordCarriedTunnel keeps what a node last said about carrying its found tunnel.
func (i *Inventory) RecordCarriedTunnel(ctx context.Context, nodeID string, c Carried) error {
c.At = time.Now().UTC()
raw, err := json.Marshal(c)
if err != nil {
return err
}
_, err = i.store.Pool().Exec(ctx, `update node set tunnel_carried = $2 where id = $1`, nodeID, raw)
return err
}
// CarriedTunnelOf is a node's last account of carrying its found tunnel, and whether it ever gave one.
func (i *Inventory) CarriedTunnelOf(ctx context.Context, name string) (Carried, bool, error) {
var raw []byte
err := i.store.Pool().QueryRow(ctx, `select tunnel_carried from node where name = $1`, name).Scan(&raw)
if errors.Is(err, pgx.ErrNoRows) {
return Carried{}, false, fmt.Errorf("%w: %s", ErrNoSuchNode, name)
}
if err != nil {
return Carried{}, false, err
}
if len(raw) == 0 {
return Carried{}, false, nil
}
var c Carried
if err := json.Unmarshal(raw, &c); err != nil {
return Carried{}, false, err
}
return c, true, nil
}
+279
View File
@@ -0,0 +1,279 @@
package inventory
import (
"errors"
"strings"
"testing"
)
// novox/hq ADR 0105: the mesh adopts the predecessor's tunnel in place. The controller reads the
// hub's address and range from the adopted tunnel, assigns an enrolling node the address its key
// already had, and refuses to hand out an address the tunnel already holds.
const (
tunnelKey = "TUNNEL-KEY-the-found-interfaces-public-key="
peerTwo = "PEER-KEY-two============================="
peerThree = "PEER-KEY-three==========================="
)
// theFoundTunnel is what a hub presents at enrolment: the predecessor's interface on a
// documentation range, with two peers each routed one address.
func theFoundTunnel() Tunnel {
return Tunnel{
Interface: "wg0", Unit: "wg-quick@wg0", Config: "/etc/wireguard/wg0.conf",
Port: 51900, Address: "192.0.2.1/24", Range: "192.0.2.0/24", PublicKey: tunnelKey,
Peers: []TunnelPeer{{PublicKey: peerTwo, Address: "192.0.2.2/32"},
{PublicKey: peerThree, Address: "192.0.2.3"}},
}
}
// anAdoptedHub is an adopted node that enrolled with the found tunnel's key and presented the
// tunnel, then was placed as the hub — the order genesis does it in.
func anAdoptedHub(t *testing.T, inv *Inventory) Node {
t.Helper()
hub, err := inv.AddNodeAs(t.Context(), "anchor", true)
if err != nil {
t.Fatal(err)
}
if err := inv.RecordOverlayKey(t.Context(), hub.ID, tunnelKey); err != nil {
t.Fatal(err)
}
if err := inv.RecordTunnel(t.Context(), hub.ID, theFoundTunnel()); err != nil {
t.Fatal(err)
}
if err := inv.SetPlace(t.Context(), "anchor", "anchor.example:51900", "hosting", true, ""); err != nil {
t.Fatal(err)
}
return hub
}
func TestTheHubsAddressAndRangeComeFromTheAdoptedTunnel(t *testing.T) {
inv := fresh(t)
hub := anAdoptedHub(t, inv)
tunnel, name, adopted, err := inv.AdoptedTunnel(t.Context())
if err != nil {
t.Fatal(err)
}
if !adopted || name != "anchor" || tunnel.Range != "192.0.2.0/24" || tunnel.Port != 51900 {
t.Fatalf("the adopted tunnel did not read back: adopted=%t on %s, %+v", adopted, name, tunnel)
}
if len(tunnel.Peers) != 2 || tunnel.Peers[0].Address != "192.0.2.2" || tunnel.Peers[1].Address != "192.0.2.3" {
t.Fatalf("the peers did not read back as one host address each: %+v", tunnel.Peers)
}
// Whatever range the caller would allocate from, the hub is at the tunnel's own address.
address, err := inv.AssignAddress(t.Context(), hub.ID, "10.42.0.0/16")
if err != nil {
t.Fatal(err)
}
if address != "192.0.2.1" {
t.Fatalf("the hub was given %s, not the address the tunnel it took over had", address)
}
}
func TestAnEnrollingNodeKeepsTheAddressTheTunnelHadForItsKey(t *testing.T) {
inv := fresh(t)
anAdoptedHub(t, inv)
// A predecessor machine enrols: its host took its found interface's key as its overlay key,
// which is the key the hub's tunnel already routes to.
peer, err := inv.AddNodeAs(t.Context(), "home-server", true)
if err != nil {
t.Fatal(err)
}
if err := inv.RecordOverlayKey(t.Context(), peer.ID, peerThree); err != nil {
t.Fatal(err)
}
address, err := inv.AssignAddress(t.Context(), peer.ID, "192.0.2.0/24")
if err != nil {
t.Fatal(err)
}
if address != "192.0.2.3" {
t.Fatalf("the enrolling peer was given %s, not the 192.0.2.3 the tunnel had for its key", address)
}
carried, err := inv.CarriedPeers(t.Context())
if err != nil {
t.Fatal(err)
}
byKey := map[string]CarriedPeer{}
for _, c := range carried {
byKey[c.PublicKey] = c
}
if byKey[peerThree].EnrolledAs != "home-server" || byKey[peerTwo].EnrolledAs != "" {
t.Fatalf("the registry cannot say which peer is a node now: %+v", carried)
}
}
func TestAFreshNodeIsNeverGivenAnAddressTheTunnelHolds(t *testing.T) {
inv := fresh(t)
anAdoptedHub(t, inv)
// .1 is the hub, .2 and .3 are peers of the tunnel that have not enrolled: a new machine with
// a key of its own gets the next one, from the same range.
fresh, err := inv.AddNode(t.Context(), "laptop")
if err != nil {
t.Fatal(err)
}
if err := inv.RecordOverlayKey(t.Context(), fresh.ID, "A-KEY-OF-ITS-OWN========================"); err != nil {
t.Fatal(err)
}
address, err := inv.AssignAddress(t.Context(), fresh.ID, "192.0.2.0/24")
if err != nil {
t.Fatal(err)
}
if address != "192.0.2.4" {
t.Fatalf("a fresh node was given %s; 192.0.2.2 and .3 are the tunnel's peers and .1 its hub", address)
}
}
func TestATunnelUnderAnotherKeyIsNotAdopted(t *testing.T) {
// A hub whose overlay key is not the found tunnel's would drop every peer's packets on the
// found port (ADR 0105, option 2). Such a tunnel is recorded and not adopted: the mesh keeps
// its own range, and ADR 0100's non-overlap rule stands for it.
inv := fresh(t)
hub, err := inv.AddNodeAs(t.Context(), "anchor", true)
if err != nil {
t.Fatal(err)
}
if err := inv.RecordOverlayKey(t.Context(), hub.ID, "THE-MESHS-OWN-KEY======================="); err != nil {
t.Fatal(err)
}
if err := inv.RecordTunnel(t.Context(), hub.ID, theFoundTunnel()); err != nil {
t.Fatal(err)
}
if err := inv.SetPlace(t.Context(), "anchor", "anchor.example:51820", "hosting", true, ""); err != nil {
t.Fatal(err)
}
if _, _, adopted, err := inv.AdoptedTunnel(t.Context()); err != nil || adopted {
t.Fatalf("a tunnel under another key was adopted (err %v)", err)
}
if carried, err := inv.CarriedPeers(t.Context()); err != nil || len(carried) != 0 {
t.Fatalf("peers of a tunnel that was not adopted are carried: %+v (err %v)", carried, err)
}
if address, err := inv.AssignAddress(t.Context(), hub.ID, "10.42.0.0/16"); err != nil || address != "10.42.0.1" {
t.Fatalf("the hub was given %s (err %v); it should allocate from the mesh's own range", address, err)
}
}
func TestAPeerRoutedARangeIsNotCarried(t *testing.T) {
// A peer routed a whole range is a spoke's view of its hub, never a machine with an address
// the mesh could carry: skipped, and the single-address peers beside it kept.
inv := fresh(t)
hub, err := inv.AddNodeAs(t.Context(), "anchor", true)
if err != nil {
t.Fatal(err)
}
found := theFoundTunnel()
found.Peers = append(found.Peers, TunnelPeer{PublicKey: "WIDE", Address: "192.0.2.0/24"})
if err := inv.RecordTunnel(t.Context(), hub.ID, found); err != nil {
t.Fatal(err)
}
got, err := inv.TunnelOf(t.Context(), "anchor")
if err != nil || len(got.Peers) != 2 {
t.Fatalf("the range-routed peer was carried, or the others dropped: %+v %v", got.Peers, err)
}
// A single address outside the tunnel's range is still refused: it is not a peer this tunnel
// routes to.
found.Peers = []TunnelPeer{{PublicKey: "ELSEWHERE", Address: "198.51.100.7/32"}}
if err := inv.RecordTunnel(t.Context(), hub.ID, found); err == nil || !strings.Contains(err.Error(), "outside") {
t.Fatalf("a peer outside the range was recorded: %v", err)
}
}
// A predecessor spoke's tunnel has one peer — the hub — routed the whole range. Its enrolment must
// not fail on it: only the hub's peers are ever carried, so a range-routed peer is skipped.
func TestASpokesTunnelEnrolsWithItsHubPeerSkipped(t *testing.T) {
inv := fresh(t)
anAdoptedHub(t, inv)
spoke, err := inv.AddNodeAs(t.Context(), "home-server", true)
if err != nil {
t.Fatal(err)
}
if err := inv.RecordOverlayKey(t.Context(), spoke.ID, peerThree); err != nil {
t.Fatal(err)
}
if err := inv.RecordTunnel(t.Context(), spoke.ID, Tunnel{
Interface: "wg0", Unit: "wg-quick@wg0", Config: "/etc/wireguard/wg0.conf", Port: 51900,
Address: "192.0.2.3/24", Range: "192.0.2.0/24", PublicKey: peerThree,
Peers: []TunnelPeer{{PublicKey: tunnelKey, Address: "192.0.2.0/24"}},
}); err != nil {
t.Fatalf("a spoke-shaped tunnel was refused: %v", err)
}
got, err := inv.TunnelOf(t.Context(), "home-server")
if err != nil || len(got.Peers) != 0 {
t.Fatalf("the spoke's hub was recorded as a peer to carry: %+v %v", got.Peers, err)
}
// Nothing about the hub's carried peers changed: still two, one now enrolled.
carried, err := inv.CarriedPeers(t.Context())
if err != nil || len(carried) != 2 {
t.Fatalf("carried peers: %+v %v", carried, err)
}
if address, err := inv.AssignAddress(t.Context(), spoke.ID, "192.0.2.0/24"); err != nil || address != "192.0.2.3" {
t.Fatalf("the spoke did not keep its address: %s %v", address, err)
}
}
// Converging the hub flips its mode and nothing else: the range stays the tunnel's and the peers
// stay carried, or the mesh would renumber itself and drop the peers still reaching it.
func TestConvergingTheHubKeepsTheRangeAndTheCarriedPeers(t *testing.T) {
inv := fresh(t)
hub := anAdoptedHub(t, inv)
if _, err := inv.AssignAddress(t.Context(), hub.ID, "192.0.2.0/24"); err != nil {
t.Fatal(err)
}
if _, err := inv.Converge(t.Context(), "anchor"); err != nil {
t.Fatal(err)
}
tunnel, _, adopted, err := inv.AdoptedTunnel(t.Context())
if err != nil || !adopted || tunnel.Range != "192.0.2.0/24" {
t.Fatalf("converging renumbered the mesh: adopted=%t %+v %v", adopted, tunnel, err)
}
if carried, err := inv.CarriedPeers(t.Context()); err != nil || len(carried) != 2 {
t.Fatalf("converging dropped the carried peers: %+v %v", carried, err)
}
found, err := inv.Tunnels(t.Context())
if err != nil || found["anchor"].NodeAdopted {
t.Fatalf("a converged hub still reads as adopted for the takeover: %+v %v", found, err)
}
}
// A hub that enrolled with a key of its own takes the tunnel over afterwards by rekeying: the key
// and the tunnel are recorded, the hub moves to the tunnel's address, and the same rekey applied
// again is stale.
func TestARekeyTakesTheTunnelOverAfterEnrolment(t *testing.T) {
inv := fresh(t)
hub, err := inv.AddNodeAs(t.Context(), "anchor", true)
if err != nil {
t.Fatal(err)
}
const own = "THE-MESHS-OWN-KEY======================="
if err := inv.RecordOverlayKey(t.Context(), hub.ID, own); err != nil {
t.Fatal(err)
}
if err := inv.SetPlace(t.Context(), "anchor", "anchor.example:51900", "hosting", true, ""); err != nil {
t.Fatal(err)
}
if address, err := inv.AssignAddress(t.Context(), hub.ID, "10.42.0.0/16"); err != nil || address != "10.42.0.1" {
t.Fatalf("before the rekey the hub is on the mesh's own range: %s %v", address, err)
}
if err := inv.Rekey(t.Context(), hub.ID, own, tunnelKey, theFoundTunnel()); err != nil {
t.Fatal(err)
}
_, _, adopted, err := inv.AdoptedTunnel(t.Context())
if err != nil || !adopted {
t.Fatalf("the tunnel is not adopted after the rekey (%v)", err)
}
placed, err := inv.Overlays(t.Context())
if err != nil || len(placed) != 1 || placed[0].Address != "192.0.2.1" || placed[0].Key != tunnelKey {
t.Fatalf("the hub did not move to the tunnel's address under the tunnel's key: %+v %v", placed, err)
}
if err := inv.Rekey(t.Context(), hub.ID, own, tunnelKey, theFoundTunnel()); !errors.Is(err, ErrStaleRekey) {
t.Fatalf("the same rekey applied again was not refused as stale: %v", err)
}
if err := inv.Rekey(t.Context(), hub.ID, tunnelKey, "ANOTHER-KEY=============================", theFoundTunnel()); err == nil {
t.Fatal("a rekey to a key that is not the tunnel's was accepted")
}
}
+96
View File
@@ -0,0 +1,96 @@
package link
import (
"context"
"encoding/json"
"fmt"
"time"
amqp "github.com/rabbitmq/amqp091-go"
)
// RPCExchange is where a module's tools are asked over the broker, keyed `<module>.<tool>`, and
// where the answer comes back, keyed by the asker's reply queue (novox/hq ADR 0047).
const RPCExchange = "mesh.rpc"
// Answer is what a module's tool replies: one of the two, never both.
type Answer struct {
Result json.RawMessage `json:"result,omitempty"`
Error string `json:"error,omitempty"`
}
// Ask calls one of a module's tools over the broker and waits for its answer.
//
// **The control plane is the way in** (novox/hq 04-ISSUES/049, ADR 0095). A module's broker
// account is scoped to what it emits, consumes and serves, and a tool call needs a reply queue the
// caller creates and a publish to the serving module's request key — which no module's scope
// grants, and should not. The control plane already holds a connection that may, so a person or
// an agent asks through it, and every question passes one process where an audit belongs.
//
// The reply queue is the caller's own, server-named and exclusive, bound to the RPC exchange under
// its own name: a serving module answers through that exchange and never the default one, whose
// permission is per exchange rather than per queue. The correlation is checked rather than
// assumed, as every RPC here is.
func Ask(ctx context.Context, channel *amqp.Channel, module, tool string, args json.RawMessage,
timeout time.Duration) (Answer, error) {
if len(args) == 0 {
args = json.RawMessage(`{}`)
}
replies, err := channel.QueueDeclare("", false, true, true, false, nil)
if err != nil {
return Answer{}, err
}
if err := channel.QueueBind(replies.Name, replies.Name, RPCExchange, false, nil); err != nil {
return Answer{}, fmt.Errorf("cannot bind a reply queue to %s: %w", RPCExchange, err)
}
answers, err := channel.ConsumeWithContext(ctx, replies.Name, "", true, true, false, false, nil)
if err != nil {
return Answer{}, err
}
// Mandatory, so a request nothing consumes comes straight back: a module that is down, or a
// tool that does not exist, is said at once rather than after the whole wait.
returned := channel.NotifyReturn(make(chan amqp.Return, 1))
id := fmt.Sprintf("ask-%d", time.Now().UnixNano())
key := module + "." + tool
if err := channel.PublishWithContext(ctx, RPCExchange, key, true, false, amqp.Publishing{
ContentType: "application/json",
CorrelationId: id,
ReplyTo: replies.Name,
Body: args,
}); err != nil {
return Answer{}, fmt.Errorf("cannot ask %s: %w", key, err)
}
waiting, cancel := context.WithTimeout(ctx, timeout)
defer cancel()
for {
select {
case back := <-returned:
if back.CorrelationId == id {
return Answer{}, fmt.Errorf(
"nothing serves %s: no runtime has bound %q on the broker. The module is not "+
"assigned, its runtime is not up, or it serves no such tool — `status` "+
"says whether the machine carrying it has applied", module, key)
}
case <-waiting.Done():
return Answer{}, fmt.Errorf(
"%s did not answer within %s. Its runtime serves %q when it is up and has bound "+
"the broker — `status` says whether the machine carrying it has applied",
module, timeout, key)
case delivery, ok := <-answers:
if !ok {
return Answer{}, fmt.Errorf("the connection closed while waiting for %s", key)
}
if delivery.CorrelationId != id {
continue
}
var answer Answer
if err := json.Unmarshal(delivery.Body, &answer); err != nil {
return Answer{}, fmt.Errorf("%s answered with something unreadable: %w", key, err)
}
return answer, nil
}
}
}
+113
View File
@@ -0,0 +1,113 @@
package link_test
import (
"crypto/ed25519"
"crypto/rand"
"crypto/sha256"
"encoding/hex"
"errors"
"testing"
"time"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// A token another enrolment holds for the moment is "not now", not a refusal: the node asks again
// with the same request, and nothing is spent (novox/hq issue 083).
func TestAnEnrolmentMetByAHeldTokenIsAskedToTryAgain(t *testing.T) {
inv := inventory.ForTest(t)
ctx := t.Context()
if _, err := inv.AddNode(ctx, "laptop"); err != nil {
t.Fatal(err)
}
issued, err := inv.IssueToken(ctx, "laptop", time.Hour)
if err != nil {
t.Fatal(err)
}
if _, err := inv.Claim(ctx, issued.Secret, "another enrolment's key", false); err != nil {
t.Fatal(err)
}
public, _, err := ed25519.GenerateKey(rand.Reader)
if err != nil {
t.Fatal(err)
}
_, err = link.Enrolment{Inventory: inv}.Enrol(ctx, link.EnrolRequest{
Node: "laptop", Secret: issued.Secret, PublicKey: public})
if !errors.Is(err, link.ErrTryAgain) {
t.Fatalf("an enrolment met by a held token was not asked to try again: %v", err)
}
// And a token that cannot be used at all is still refused outright.
_, err = link.Enrolment{Inventory: inv}.Enrol(ctx, link.EnrolRequest{
Node: "laptop", Secret: "not-a-token", PublicKey: public})
if err == nil || errors.Is(err, link.ErrTryAgain) {
t.Fatalf("a token that cannot be used was not refused outright: %v", err)
}
}
// **A spent token cannot be replayed with a node's public key** (novox/hq issue 083, on review). A
// public key is no secret: finishing an enrolment whose token that key spent takes proof the
// presenter holds its private half, and a proof made with another key — or for another request —
// is refused outright.
func TestASpentTokenCannotBeReplayedWithAPublicKeyAlone(t *testing.T) {
inv := inventory.ForTest(t)
ctx := t.Context()
if _, err := inv.AddNode(ctx, "laptop"); err != nil {
t.Fatal(err)
}
issued, err := inv.IssueToken(ctx, "laptop", time.Hour)
if err != nil {
t.Fatal(err)
}
victim, victimPrivate, err := ed25519.GenerateKey(rand.Reader)
if err != nil {
t.Fatal(err)
}
// The victim's enrolment spent the token.
by := claimantOf(victim)
if _, err := inv.Claim(ctx, issued.Secret, by, false); err != nil {
t.Fatal(err)
}
if err := inv.Spend(ctx, issued.Secret, by); err != nil {
t.Fatal(err)
}
// Someone with the leaked token and the victim's public key, and no proof.
_, err = link.Enrolment{Inventory: inv}.Enrol(ctx, link.EnrolRequest{
Node: "laptop", Secret: issued.Secret, PublicKey: victim, SealingKey: "the attacker's"})
if err == nil || errors.Is(err, link.ErrTryAgain) {
t.Fatalf("a spent token was taken again with a public key alone: %v", err)
}
// With a proof made by another key.
_, forger, err := ed25519.GenerateKey(rand.Reader)
if err != nil {
t.Fatal(err)
}
forged := ed25519.Sign(forger, link.EnrolProof(issued.Secret, victim, "", "the attacker's", ""))
_, err = link.Enrolment{Inventory: inv}.Enrol(ctx, link.EnrolRequest{
Node: "laptop", Secret: issued.Secret, PublicKey: victim, SealingKey: "the attacker's", Proof: forged})
if err == nil {
t.Fatal("a spent token was taken again with a proof made by another key")
}
// With the victim's own proof, but for another request — keys swapped in.
theirs := ed25519.Sign(victimPrivate, link.EnrolProof(issued.Secret, victim, "", "the victim's", ""))
_, err = link.Enrolment{Inventory: inv}.Enrol(ctx, link.EnrolRequest{
Node: "laptop", Secret: issued.Secret, PublicKey: victim, SealingKey: "the attacker's", Proof: theirs})
if err == nil {
t.Fatal("a spent token was taken again with a proof made for another request")
}
// And the victim's own, proven request is not honoured when the broker redelivered it: the
// first delivery may already have been answered.
_, err = link.Enrolment{Inventory: inv}.Enrol(ctx, link.EnrolRequest{
Node: "laptop", Secret: issued.Secret, PublicKey: victim, SealingKey: "the victim's",
Proof: theirs, Redelivered: true})
if err == nil {
t.Fatal("a redelivered request finished an enrolment already spent")
}
}
// claimantOf is the claimant the enrolment derives from a key, recomputed here.
func claimantOf(public ed25519.PublicKey) string {
sum := sha256.Sum256(public)
return hex.EncodeToString(sum[:])
}
+13
View File
@@ -0,0 +1,13 @@
package link
import "testing"
// The bytes a node signs when it enrols, built here to check its signature. The host builds the
// same bytes in another repository; this known answer is repeated in its test, so the two cannot
// drift apart without one of them failing (novox/hq issue 083).
func TestWhatAnEnrollingNodeSignsIsFixed(t *testing.T) {
got := string(EnrolProof("s", []byte{1, 2, 3}, "o", "e", "v"))
if want := "novox-mesh-enrol\x00s\x00AQID\x00o\x00e\x00v"; got != want {
t.Fatalf("the enrolment proof's message changed: %q, want %q", got, want)
}
}
+1 -1
View File
@@ -85,7 +85,7 @@ func TestWhatANodeSaysWhenItJoinsIsWhatThisMeshReads(t *testing.T) {
t.Fatal(err)
}
secret, err := inv.SecretFor(ctx, "postgres-database", request.Node, "gitea", "the-other-end")
secret, err := inv.SecretFor(ctx, "postgres-database", request.Node, "gitea", "the-other-end", "")
if err != nil {
t.Fatalf("nothing could be sealed to a key that arrived from a real node: %v", err)
}
+174 -36
View File
@@ -4,7 +4,9 @@ import (
"context"
"crypto/ed25519"
"crypto/rand"
"crypto/sha256"
"encoding/base64"
"encoding/hex"
"errors"
"fmt"
"log"
@@ -28,32 +30,57 @@ type Enrolment struct {
Broker broker.Broker
}
// Enrol spends the token and records what the node presented.
// Enrol records what the node presented and spends the token.
//
// Order matters and it is the order things become irreversible. The token is spent first, in a
// single statement that both finds and marks it, so two machines racing on one secret produce one
// winner. Only then is a key recorded — because recording a key for a node whose token turned out
// to be spent would leave the mesh believing a machine that never had the right to join.
func (e Enrolment) Enrol(ctx context.Context, request EnrolRequest) (EnrolReply, error) {
// Order matters and it is the order things become irreversible (novox/hq issue 083). The token is
// claimed first, in a single statement that both finds it and holds it for this presenter's key, so
// two machines racing on one secret produce one holder. Then everything the node presented is
// written — each write an overwrite, so an attempt interrupted by the store going away can be made
// again by the same presenter. Then the token is spent. Last, the token's secret stops being the
// node's broker password: done after the spend, because a password replaced by an attempt that
// then failed would be one nobody holds, and the node could not even log in to ask again.
func (e Enrolment) Enrol(ctx context.Context, request EnrolRequest) (reply EnrolReply, err error) {
secret, public, profile := request.Secret, ed25519.PublicKey(request.PublicKey), request.Profile
// A store that could not be asked right now, or a token another presenter holds for the
// moment, is "not now": the node asks again with the same request (novox/hq issue 083).
defer func() {
if inventory.Unreachable(err) || errors.Is(err, inventory.ErrTokenInUse) ||
errors.Is(err, context.Canceled) {
err = fmt.Errorf("%w: %w", ErrTryAgain, err)
}
}()
if len(public) != ed25519.PublicKeySize {
return EnrolReply{}, fmt.Errorf("a node presented a %d-byte key, and an identity is %d",
len(public), ed25519.PublicKeySize)
}
node, err := e.Inventory.Redeem(ctx, secret)
// Claimed, not spent: the token is held for this presenter while the node is written, and
// spent only as the last write. The node's identity lives in another database than the
// token, so the two cannot be one transaction; a failure between them used to leave a spent
// token and a node with no key, which the host — making new keys on every attempt — could not
// recover from. Every write below overwrites, so an attempt made again is safe.
// A proof that does not verify is refused outright: it was made with another key, or for
// another request. One that verifies lets this presenter finish an enrolment whose token it
// already spent — never a request the broker handed over a second time, which may already
// have been answered.
proven := false
if len(request.Proof) > 0 {
if !ed25519.Verify(public, EnrolProof(secret, public, request.OverlayKey, request.SealingKey,
request.ServingKey), request.Proof) {
return EnrolReply{}, errors.New("the enrolment's proof does not match the key it presents")
}
proven = true
}
by := claimant(public)
node, err := e.Inventory.Claim(ctx, secret, by, proven && !request.Redelivered)
if err != nil {
return EnrolReply{}, err
}
// From here the token is gone whatever happens next, so anything that fails leaves a node
// record with no live key — which is visible and fixable with a new token, where a spent
// token believed to be unspent is neither.
if _, err := e.Identity.RecordNodeKey(ctx, node.ID, public); err != nil {
return EnrolReply{}, fmt.Errorf(
"the token was spent and the key could not be recorded, so %s has no identity and "+
"needs a new token: %w", node.Name, err)
return EnrolReply{}, fmt.Errorf("%s's key could not be recorded: %w", node.Name, err)
}
key, err := e.Identity.Active(ctx)
@@ -61,7 +88,7 @@ func (e Enrolment) Enrol(ctx context.Context, request EnrolRequest) (EnrolReply,
return EnrolReply{}, err
}
reply := EnrolReply{
reply = EnrolReply{
Accepted: true,
Node: node.Name,
Queue: QueueFor(node.Name),
@@ -70,22 +97,6 @@ func (e Enrolment) Enrol(ctx context.Context, request EnrolRequest) (EnrolReply,
Signer: key.Public,
}
// The token's secret was the broker password up to this moment, which is what let this
// connection exist at all. It is replaced now, so the one-time thing stays one-time and the
// credential the node keeps for years is not the one that was pasted into a terminal.
if e.Management != nil {
password, err := freshPassword()
if err != nil {
return EnrolReply{}, err
}
if err := e.Management.CreateNodeAccount(ctx, node.Name, password); err != nil {
return EnrolReply{}, fmt.Errorf(
"the token was spent and %s's broker password could not be replaced: %w",
node.Name, err)
}
reply.Password = password
}
// Recorded before the profile because the overlay is the first declaration this node will
// receive, and without this key the mesh cannot compose one. A node enrolled with no overlay
// key is a node the graph skips — an ordinary in-between state, and one worth leaving as
@@ -98,21 +109,66 @@ func (e Enrolment) Enrol(ctx context.Context, request EnrolRequest) (EnrolReply,
if request.ServingKey != "" {
if err := e.Identity.RecordServingKey(ctx, node.ID, request.ServingKey); err != nil {
return EnrolReply{}, fmt.Errorf(
"the token was spent and %s's serving key could not be recorded: %w",
node.Name, err)
"%s's serving key could not be recorded: %w", node.Name, err)
}
}
if request.SealingKey != "" {
if err := e.Inventory.RecordSealingKey(ctx, node.ID, request.SealingKey); err != nil {
return EnrolReply{}, fmt.Errorf(
"the token was spent and %s's sealing key could not be recorded: %w",
node.Name, err)
"%s's sealing key could not be recorded: %w", node.Name, err)
}
}
if request.OverlayKey != "" {
if err := e.Inventory.RecordOverlayKey(ctx, node.ID, request.OverlayKey); err != nil {
return EnrolReply{}, fmt.Errorf(
"the token was spent and %s's overlay key could not be recorded: %w", node.Name, err)
"%s's overlay key could not be recorded: %w", node.Name, err)
}
}
// And the tunnel it found, whose key is the overlay key above (novox/hq ADR 0105). Recorded
// before the token is spent for the same reason as the keys: the first declaration this node
// receives is composed from it, and a hub enrolled without its tunnel would be placed at an
// address of the mesh's choosing rather than the tunnel's.
if request.Tunnel != nil {
if request.Tunnel.PublicKey != request.OverlayKey {
return EnrolReply{}, fmt.Errorf("%s presented a tunnel under key %s and an overlay key "+
"that is not it; a tunnel is taken over with its own key or not at all", node.Name,
request.Tunnel.PublicKey)
}
peers := make([]inventory.TunnelPeer, 0, len(request.Tunnel.Peers))
for _, p := range request.Tunnel.Peers {
peers = append(peers, inventory.TunnelPeer{PublicKey: p.PublicKey, Address: p.Address})
}
if err := e.Inventory.RecordTunnel(ctx, node.ID, inventory.Tunnel{
Interface: request.Tunnel.Interface, Unit: request.Tunnel.Unit,
Config: request.Tunnel.Config, Port: request.Tunnel.Port,
Address: request.Tunnel.Address, Range: request.Tunnel.Range,
PublicKey: request.Tunnel.PublicKey, Peers: peers,
}); err != nil {
return EnrolReply{}, fmt.Errorf("%s's found tunnel could not be recorded: %w", node.Name, err)
}
}
// Spent once the node is complete in the store.
if err := e.Inventory.Spend(ctx, secret, by); err != nil {
return EnrolReply{}, fmt.Errorf("%s was written and its token could not be spent: %w", node.Name, err)
}
// The token's secret was the broker password up to this moment, which is what let this
// connection exist at all. It is replaced now, so the one-time thing stays one-time and the
// credential the node keeps for years is not the one that was pasted into a terminal. After
// the spend and not before: a replaced password on an attempt that failed would be held by
// nobody. If the broker will not take it now, the enrolment still stands — the node keeps
// the token's secret as its password, which it is told, and which is said here.
if e.Management != nil {
password, err := freshPassword()
if err != nil {
return EnrolReply{}, err
}
if err := e.Management.CreateNodeAccount(ctx, node.Name, password); err != nil {
log.Printf("%s is enrolled and its broker password could not be replaced, so it keeps "+
"the token's secret as its password: %v", node.Name, err)
} else {
reply.Password = password
}
}
@@ -125,6 +181,40 @@ func (e Enrolment) Enrol(ctx context.Context, request EnrolRequest) (EnrolReply,
return reply, nil
}
// rekey applies a verified rekey: the node's overlay key and tunnel are recorded as enrolment
// would have recorded them, and a hub moves to the tunnel's address.
func (e Enrolment) rekey(ctx context.Context, node inventory.Node, r Rekey) error {
if r.Tunnel == nil || r.OverlayKey == "" {
return fmt.Errorf("%s sent a rekey naming no tunnel or no key; refused", node.Name)
}
if e.Identity == nil {
return fmt.Errorf("%s sent a rekey and this mesh has no identity store to verify it against", node.Name)
}
if err := e.Identity.VerifyNode(ctx, node.ID,
RekeyProof(node.Name, r.Previous, r.OverlayKey, r.Tunnel), r.Proof); err != nil {
return fmt.Errorf("%s's rekey is not signed by %s's identity key; refused: %w", node.Name, node.Name, err)
}
peers := make([]inventory.TunnelPeer, 0, len(r.Tunnel.Peers))
for _, p := range r.Tunnel.Peers {
peers = append(peers, inventory.TunnelPeer{PublicKey: p.PublicKey, Address: p.Address})
}
err := e.Inventory.Rekey(ctx, node.ID, r.Previous, r.OverlayKey, inventory.Tunnel{
Interface: r.Tunnel.Interface, Unit: r.Tunnel.Unit, Config: r.Tunnel.Config, Port: r.Tunnel.Port,
Address: r.Tunnel.Address, Range: r.Tunnel.Range, PublicKey: r.Tunnel.PublicKey, Peers: peers,
})
if err != nil {
return fmt.Errorf("%s's rekey was not recorded: %w", node.Name, err)
}
log.Printf("%s took over the tunnel on %s: its overlay key is the tunnel's now", node.Name, r.Tunnel.Interface)
return nil
}
// claimant names the key presenting a token, so a claim can be held for it alone.
func claimant(public ed25519.PublicKey) string {
sum := sha256.Sum256(public)
return hex.EncodeToString(sum[:])
}
// freshPassword is the node's own broker credential from enrolment onward.
func freshPassword() (string, error) {
raw := make([]byte, 32)
@@ -144,7 +234,14 @@ var ErrNoBrokerManagement = errors.New("no broker management configured")
// A node states; the owning context writes (novox/hq ADR 0006). What a node says it applied is
// its own account of its own machine, kept as a copy for recovery — so this writes it down and
// decides nothing from it.
func (e Enrolment) Heard(ctx context.Context, report Report) error {
func (e Enrolment) Heard(ctx context.Context, report Report) (err error) {
// A store that could not be asked right now is said as such, so the report is kept for
// another attempt rather than acknowledged and lost (novox/hq issue 082).
defer func() {
if inventory.Unreachable(err) {
err = fmt.Errorf("%w: %w", ErrTryAgain, err)
}
}()
if report.Node == "" {
return errors.New("a report named no node")
}
@@ -153,6 +250,47 @@ func (e Enrolment) Heard(ctx context.Context, report Report) error {
return err
}
// What an adopted node holds, which firewall it found, and what is reachable on it (novox/hq
// ADR 0100). Recorded whenever a report carries any of it — a node reports these on its own
// schedule, when what it holds changes, not only after an apply — and never cleared by a
// report that carries none, which is every bare word that the node is there. An adopted node
// always names its firewall, so a report from one replaces all three, emptied held included.
if len(report.Held) > 0 || report.Firewall != "" || len(report.Reachable) > 0 {
held := make([]inventory.Held, 0, len(report.Held))
for _, h := range report.Held {
held = append(held, inventory.Held{ID: h.ID, Module: h.Module, Kind: h.Kind,
Target: h.Target, Since: h.Since, Changed: h.Changed, Kept: h.Kept})
}
reachable := make([]inventory.Reach, 0, len(report.Reachable))
for _, r := range report.Reachable {
reachable = append(reachable, inventory.Reach{Protocol: r.Protocol, Address: r.Address,
Port: r.Port, By: r.By, Published: r.Published, ContainerPort: r.ContainerPort})
}
if err := e.Inventory.RecordAdoption(ctx, node.ID, held, report.Firewall, reachable); err != nil {
return err
}
}
// What it says about the tunnel it carried (novox/hq ADR 0105), whenever it says it.
if report.Tunnel != nil {
if err := e.Inventory.RecordCarriedTunnel(ctx, node.ID, inventory.Carried{
Interface: report.Tunnel.Interface, Port: report.Tunnel.Port, Range: report.Tunnel.Range,
Peers: report.Tunnel.Peers, State: report.Tunnel.State, Note: report.Tunnel.Note,
Kept: report.Tunnel.Kept,
}); err != nil {
return err
}
}
// A node taking a found tunnel's key after enrolment (novox/hq ADR 0105). Verified against the
// node's live identity key before anything is written: the broker account authenticates the
// connection, the signature proves the node itself said it. Refused outright when the proof
// does not verify or is stale — a refusal, not "not now", so the node hears why.
if report.Rekey != nil {
if err := e.rekey(ctx, node, *report.Rekey); err != nil {
return err
}
return e.Inventory.Seen(ctx, node.ID)
}
// A bare word that a node is there is not an account of what the machine did or holds: it
// moves last_seen and touches nothing else. This arrives every minute (link.AliveEvery),
// while a real report is rare, so recording it as one would overwrite the node's last real

Some files were not shown because too many files have changed in this diff Show More