Proxy configuration beside insecure, not a fifth policy — ADR 0108 closed that set at four, and both
of these tune how a request is carried rather than deciding what a name admits. A registry is the
case that needs it: image layers arrive as single requests of gigabytes and a proxy's own default
refuses them long before the workload is reached.
Absent is no limit, which is what every route already got. A limit that is not a whole positive
number of bytes takes the route with it, named in the log like a port that is not one — serving it
without the limit would carry exactly what the module said not to carry. Enforced on the declared
length where there is one, and while reading for a chunked body, which declares none: without the
second, a limit is advice.
The swap from a named volume to the host directory left the mount undeclared, which main's own
manifest check now refuses: a bind the module did not declare is created by the runtime as root, so
the module's owner and mode never reach it and ADR 0030's data rule does not cover it.
The directory is the broker's — lavinmq declares it as its own, mode 0700 — so from here it is an
access, read-only: a pre-existing path this module is granted use of and does not own.
autocert checks the host policy before the token and answers 403 — the
internal authority does this for every public name, so mail.novox.be's
challenge died on the internal manager's probe one commit after it
stopped dying on the public one's 404. Both shapes of refusal now fall
through to routing; a fifth test pins the 403 case with a refusing
policy.
autocert's HTTPHandler answers 404 itself for a token it does not hold
and never consults its fallback on the challenge path — the
predecessor's exact fault, rediscovered live when Mailu's renewal died
behind this proxy on cutover day. tokenOrRoute probes each authority
against a buffered writer and hands a token none of them holds to plain
routing, so a consumer's own ACME client answers its own challenge
through an ordinary path-scoped route. Four tests pin it, including the
cache-key shape a restart-surviving token actually has.
A private repository could not be built: the builder clones anonymously,
and had no way to say who it is. It already holds exactly one credential
to exactly the right place — the package-registry binding and its sealed
secret, one gitea user whose password answers npm and git alike — so a
clone now offers that, and nothing new is minted or carried.
Offered, never pushed: the credential is written as a git
credential-store file (0600, in the workspace, never argv) and named
with -c credential.helper, so git itself decides when it applies — only
on an authentication challenge, and only for the URL it was written
for, scheme, host and port included. A public repository clones exactly
as before; a repository on any other host is never shown it. The same
store rides along on an artifact's own context clone, so a private
module with a private context builds too.
Implements novox/hq ADR 0110 and 0111.
The seat set lives in internal/catalogue/seats.go: fourteen seats, each with a scope, what occupying
it delivers, and the record that made it one. A test asserts the count and a decision per entry, so
changing the set means finding the argument, as the host's vocabulary test does. The first set is
every seat already claimed — including the-private-network, which the network module claims from a
manifest composed in this repository's code, not from any module.json — plus npm-package-registry
(ADR 0109) and git (ADR 0111). A test parses every catalogue manifest and this repository's own and
fails on any refused claim, so closing the set refuses nothing in use.
ParseManifest now refuses a claim on a seat the mesh does not define, a seat claimed at another
scope, and a delivering seat claimed by a module that does not provide what it delivers. A
malformed claim is refused once, for being malformed.
Resolution: among several providers of a mesh provision, a pin still wins; then the holder of the
seat that delivers it; then the only provider; otherwise refused as before. ADR 0009's "never
guessed" holds — the seat is the choice made once, mesh-wide, rather than a pin per consumer node.
A provider now carries the module it came from, because a provider is a (node, module) pair and the
pair is what tells a holder from a neighbour on the same machine.
The planner's second pass is now given the first pass's holdings. Without them, a node consuming a
seat-delivered provision was refused there, and a refused node's own claims dropped out of what the
mesh holds — letting a second holder of one of its seats pass unrefused.
`seats [--json]` lists every seat, what it delivers, and each holder, derived from assignments
every time and never stored. Unheld seats are listed. A stored claim outside the set — possible
for a manifest registered before the set closed, since stored manifests are not re-validated — is
shown rather than hidden.
`build --self <owner>/<repo>` builds from a repository on the git seat's holder. The clone URL is
composed at build time from the holder's node and what it serves for git; the recorded source is the
path and the seat (migration 0032), never an address, so a moved forge changes nothing recorded.
Nobody holding the seat refuses self-hosted builds and says so; external URLs are unchanged. An
address passed with --self is refused rather than recorded as a path.
Replaces three foundation tests that defended the builder's carried package binding. The catalogue
removed that binding when the builder began requiring the registry through a real grant, so the
tests were already failing on main; they now assert the builder requires what the npm seat delivers
and carries no copy of its own, and that the forge holds the npm and git seats.
Verified: go vet clean; the whole suite passes against a throwaway Postgres (make postgres), the new
inventory tests included; gofmt clean apart from cmd/mesh-builder/stdout_test.go, which fails on
main too.
Internal aliases were served over plain HTTP only — correctly refused a
public certificate (no public CA can validate a private name), and then
left with nothing. The mesh has two authorities for its two name spaces
(08-connectivity §2), so the proxy now takes an optional internal ACME
directory and dispatches at the handshake by the same question HostPolicy
already answers: which authority may certify this name at all.
A route may also say its target speaks https, with insecure for a backend
whose own certificate nothing would trust — the shape Mailu's webmail
front needs, and the exception: everything else the mesh hands this proxy
stays plain http on the private network.
route-proxy's own Dockerfile documents the shape it has always needed and
never had: 'the proxy source is not vendored here... the build context is
the mesh-controller repository root, and this Dockerfile compiles
./examples/route-proxy from it.' Nothing in the mesh could do that — the
build command clones one repository and builds every artifact from
within it, so route-proxy has never once been built through the pipeline,
consistent with it never having been assigned anywhere. Found attempting
exactly that build tonight: 'stat go.mod: file does not exist', because
the context was mesh-catalog, which does not have one.
An image artifact may now carry a context: {repository, ref}, cloned
fresh alongside the module's own tree. The recipe (Dockerfile) is still
read from the module's own directory, at the module's own commit — only
docker build's own context argument moves. Packaging and source stay
exactly as separate as route-proxy's own comment already said they were,
now for real.
A route now consumed with two hosts when the mesh composed both — the
same host under internal-name reaches the same rule as its public name,
restoring the convenience a predecessor proxy gave for reaching a service
over the VPN without a public TLS round trip (the field composeName now
writes, feat/route-carries-internal-alias — this branch depends on that
one landing for internal-name to ever be populated; builds and tests
clean without it, just serves nothing extra).
Never certified: onlyWhatTheMeshSaid used routed(), which answered yes
for any host in the table regardless of how it got there. A new
eligibleForACME() checks a parallel 'public' set instead — every host
reached through a route's own name, never one reached only through its
internal-name — so an internal alias is proxied but never given its own
failing ACME order. routed() is unchanged and still used for the 404
message, which legitimately wants 'is this host served at all.'
TestTheForgesSshPortIsGivenByTheNumberTheForgeCallsIt exercised a settings
override from '2222' to 222 — but 2222 was never a real port anywhere,
just a mistake in gitea's own manifest (fixed alongside this: listens.port
is now 22, the container's real internal sshd port, matching every other
module's convention, and ports declares 222:22 directly — 222 has always
been the real, fixed public git-ssh port, needing no per-node override).
Split into two tests: the fixed default with no override, and a genuine
override case for a hypothetical node whose predecessor used a different
number, keyed correctly by 22.
TestTheResolverAndWhatAsksItComposeOnOneMachine set Rendering.Names but
FactNodeZones reads Rendering.Machines (novox/hq issue 111 split the two
apart: every name the mesh serves vs. the machines subset) — a loose end
from that merge, not exercised until now. Both are the same map in this
test's scenario, so both fields are set.
Every cutover done on novox tonight (drive, files, files-api, git,
keycloak, umami) dropped the <label>.<node>.internal alias HAL always
paired with the public hostname — found only when the operator tested it
by hand. Not a security boundary (a predecessor proxy served both as a
convenience, reaching a service over the VPN without a public TLS round
trip, not as access control), so restoring it is composing the same
convenience the same way the public name already is: <label> joined to
the node's own private address (r.At), independently of whether a public
domain exists to join the other half to.
composeName's signature changes (publicDomain, internalDomain) but its
shape does not — additive, label-gated, apex-aware, exactly mirroring the
public half it already did. A contribution the mesh writes both names
into is the entire fix; route-adapter and route-proxy pick up internal-
name whenever they're updated to serve it, not before, so this alone
changes nothing about what is live on any node yet.
Found reviewing my own change before merging it, and it was load-bearing rather than cosmetic.
Priority was read with asPort, which caps at 65535. A rule declared above that silently became
priority 0 and stopped shadowing the route it exists to shadow. The one real rule this has to
reproduce is declared at 100000 — so path scoping and refusal would both have shipped looking
complete, passing their tests, and doing nothing on the only case that motivated them.
A priority is an ordering and has no range. asWhole takes any whole number the mesh wrote and
rejects a non-integral one, which was not meant as a priority.
Also: a host may now be routed on some paths and not others, which made the 404 dishonest — it
said "no route for this name" while listing that very name as served, a contradiction an
operator has to disbelieve the proxy to get past. An uncovered path now says so, and a name
that is genuinely not served still lists what is.
Two regression tests, both through the proxy rather than against the parser, because the parser
was where the bug looked fine.
Implements novox/hq ADR 0108, closing issue 116. The proxy's request path was a host lookup
and a forward, so it applied nothing — while the ingress it replaces relies on four things it
had none of.
Path scoping came first because it is a prerequisite, not a sibling. The table mapped a host
to one target, so a host could not be routed two ways, and the refusal this issue turns on
matches a path on a host already routed to a workload. No amount of authentication or source
filtering would have made it expressible. The table is now host to an ordered list of rules,
matched on path prefix.
The order is total, not just by priority. Sorting on priority alone leaves rules that share
one in whatever order the map produced, so the same declaration would serve differently
between restarts — a fault that works, and works differently each time, which is the hardest
kind to believe when reported. Within a priority the longer path wins, which is also the
intuitive reading.
auth names a secret and never holds one. A declaration carrying a credential is refused
whole rather than served unprotected, so the option ADR 0108 rejected cannot return by
accident. A secret that cannot be read makes the route refuse and say so, rather than serve
the workload unprotected — a gate that cannot check is not a gate that opens, and the
alternative turns a missing file into a silently public admin surface.
Authentication costs one bcrypt comparison on every path including an unknown user, so an
unknown user is not measurably faster than a known one with a wrong password. That difference
is a way to enumerate a route's users from outside it.
Redirects keep the request's own path and query, or canonicalising one name onto another
would land every deep link on the front page and raise no error doing it.
Eleven tests, four of them for the capabilities and two for the failure modes that rot
quietly: the credential-in-a-declaration refusal, and the unreadable secret failing closed.
Nothing else breaks if those stop working, so nothing else would report it.
No new dependency: bcrypt comes from the x/crypto module already required.
ContributionsFrom settled to whichever of a module's several contributions to
one requirement sorted first, arbitrarily — the grant minted for it then
carried that contribution's label and port under a credential the OTHER
contribution's consumer never sees, and collided with that same
contribution's own entry from contributions() besides.
Confirmed live: minio's two route contributions (files-api, files) produced
three entries in route-adapter's received file — files-api twice, once
credentialed and once not, files not credentialed at all. Every
single-contribution module (gitea, keycloak, umami) already mints an unused
credential for `route` too — route never needs one, by its own
documentation — but with exactly one contribution to match there was nothing
to collide with, so it never surfaced.
Where a module contributes more than once, there is no single value to
settle on. The module still asks, still gets its one credential — a pair
credential is not a place for a label or a port anyway — and each named
contribution reaches the provider on its own, unchanged.
No cleanup needed for the secret already minted live for minio+route: the
sealed blob is a random pair credential unrelated to Values, which is
recomputed fresh on every plan/push regardless.
Every module.json already declares a why for each port under listens,
but plan only ever used it to build the firewall's rule set — nothing
printed it. An operator deciding whether to assign a module had no way
to see what it would open without reading the manifest by hand.
plan <node> now prints each assigned module's listens entries — port,
protocol, source, and its why — right under the module line, so the
same text that feeds the firewall is visible at the point someone is
actually deciding whether to open it.
A module's contributes was map[string]map[string]any — one JSON object key
per requirement, structurally exactly one contribution to "route" ever.
minio needs two public hostnames (the S3 API and the console), which is
two different contributions to route from one module, and nothing let it
say so.
This is the same shape of problem ADR 0094 solved for secrets (a module
needing several values from one provider that gives one per pair):
contributes now accepts either the ordinary {label, port} object, or an
object of local names to several such objects. Detected per requirement
key by what's inside, since (unlike secrets' string-vs-object split) both
shapes are JSON objects: an ordinary contribution's fields are scalars, the
several-instance shape is local-name -> object. Confirmed against every
module.json in mesh-catalog before relying on that split.
Both route-proxy and the migration-era route-adapter already key generated
routers off the composed hostname (Values["name"]), not the module name,
so two contributions with the same From reach them as two independent
routes with no changes needed on the receiving side.
Found checking whether the named volumes mesh-catalog PR #54/#55 replaced
are actually unused before considering them safe to remove -- this repo
has its own independent volumes declaration for the same TLS material
(mesh-controller reads it directly, not through lavinmq's own resource),
and it still named the old mesh-broker-tls volume.
Right now the content is identical -- copied once during the conversion.
If the cert ever rotates, lavinmq writes the new directory and this would
keep reading stale content from the volume nothing else updates.
Checked both repos for any other reference to the four converted volume
names (mesh-store-data, mesh-broker-data, mesh-broker-tls,
mesh-registry-data): this was the only one.
The map the control plane hands a resolution holds both: the machines, and every name
the mesh was told to route to whichever machine serves it. A container's hosts wants all
of it, so a routed name resolves to the proxy. A resolver's zones want only the machines:
told the mesh's suffix is its own it answers authoritatively for everything under it and
forwards none of it, so a routed name with the suffix appended — drive.example.test.internal
— is a name nobody will ever ask for, standing beside the machines and looking as real.
Found composing the resolver's first assignment on a live machine, before pushing it.
hq issue 111.
hal dnsmasq-app conversion, hq 08-connectivity. Converting the resolver from the module it
replaces made it forward what it cannot answer, which is what the predecessor's does, and
that found two things the controller did not say.
A resolver that forwards must not send a mesh name it does not know upstream: the
`node-zones` fact now carries `local=/<suffix>/` beside the wildcards, written here rather
than in the daemon's configuration because the suffix is the mesh's choice and this file is
the one place the mesh writes what it chose. The default lives in one helper now instead of
being spelled in two functions.
The predecessor points the container runtime's `dns` at the machine's own tunnel address —
a container cannot reach the machine's loopback. A module writing that key needs the
address, and `${machine:at}` is the machine's name; a runtime's resolver list cannot be a
name it would need that resolver to look up. So a module may say `${machine:address}`: what
`at` resolves to, read from the same names the hosts file and the wildcards are written
from, absent — and refused — off the network like `at` is.
The `mesh-resolver` and `resolver-data` constants go: nothing provided or consumed either,
the fact and `mesh-addressing` are the mechanism, and a requirement nothing provides is
refused at resolution.
Tests: the catalogue's dnsmasq, resolv-conf and resolved-split-dns manifests are parsed
and composed as a machine would receive them — fixed upstreams, no-resolv, 127.0.0.1, the
machines file, the runtime's key, the pair that decides what a machine asks refused on one
node; and on a real mesh the resolver's machines file is composed with a wildcard per
machine on the network and composed again without one that left, mirroring the hosts fact.
Review of the ADR 0105 build (hq ADR 0105). Four things it got wrong and one
path it lacked:
- A predecessor spoke's tunnel names one peer, the hub, routed the whole
range; recording refused it and the whole enrolment failed. Range-routed
peers are skipped now — only the hub's peers are ever carried.
- The range and the carried peers were conditions on the node being adopted,
so converging the hub would have renumbered the mesh and dropped the peers
still reaching it. They are facts of the tunnel record now, mode aside; the
takeover alone is declared to an adopted node. Converging the hub is refused
while a carried peer has not enrolled, naming it.
- A push composed a takeover for a hub whose address or endpoint disagreed
with the tunnel, which would have the host stop the found interface and
raise the mesh's where no peer listens. The graph refuses to compose it,
naming both and the placement that fixes it.
- The host's account said taken or not; "found down and the mesh's not up"
read as not taken. Three states now, and an account on every takeover.
- A hub that enrolled before this feature holds a key of its own, and
re-enrolling would rotate every key the mesh sealed credentials to. A node
now rekeys in a report, signed with its identity key over the key it
leaves, the key it takes and the tunnel; the mesh verifies against the live
key, refuses a stale or foreign proof, records key and tunnel, and moves a
hub to the tunnel's address. `overlay show` names the path for a hub that
found no tunnel.
Also: a carried IPv6 peer is routed /128, and identity.ForTest exists so the
link can be tested against a real identity store.
Beside each sealed connection genesis wrote, the port this machine put the
seat's holder at: `${seat:mesh-store:5432}` for the three stores,
`${seat:mesh-broker:…}` for the bus, the plain AMQP port and the management API.
Filled from the node's settings when the control plane composes its own
declaration; empty — the sealed value stands — when the mesh has nothing to add.
On its own, after the commit before it is built and running: a control plane
that does not know the placeholder passes it through as the value, and this
manifest is composed by whatever control plane is running when it is pushed.
The reader ignores an unfilled placeholder either way, and a test holds it to
ignoring exactly what this manifest says.
novox/hq 04-ISSUES/102
Three readers did not follow a moved foundation port (novox/hq 04-ISSUES/102),
and each took the control-node down in its own way: the control plane's own
store and broker connections, sealed at genesis with the port inside; and every
build the mesh ever recorded, kept as `<registry>:<port>/<module>/<artifact>@…`.
The control plane cannot open its own sealed connections to move a port, and it
cannot bind the store as a consumer would — a binding mints a credential. So its
settings get a third twin, `NAME_PORT`, read on top of the sealed value by the
store, the broker, the management API and the bus connection, and filled into
its container by a placeholder that names a seat, `${seat:mesh-store:5432}`,
from the node's given or mesh-assigned ports — never the manifest's number, and
empty when the mesh has nothing to add, so what genesis wrote stands. A value
that is still a placeholder is nothing said, aloud: the manifest naming it lands
in the next commit, once every control plane that composes it knows it.
A build is now recorded by digest and path — `artifact-store://<module>/<artifact>@…`
— and the store's address is composed in where a reference is used: the
declaration, the trust file, the bases a build is handed, a replay to the
catalogue. Over the network as `<node>.internal:<port>`; on the store's own node
before any network exists — every genesis push before its "network" step — by
loopback. A reference recorded before this, with an address, is re-routed the
same way when the mesh built it. The trust file and every provider's address
come from one derivation: the node's given port, over the mesh's assignment,
over the manifest's number.
novox/hq 04-ISSUES/102
Review of the registry work found the seat node-scoped: a second `distribution` on another
machine resolved cleanly there, and only afterwards did the mesh notice `artifact-store`
offered by two nodes, with every consumer elsewhere refusing to choose. A node-scoped
requirement with one candidate installs that candidate, so anything that wanted the store
beside it would have raised a fresh, empty store on the wrong machine first.
The claim is mesh-scoped in mesh-catalog now; this holds the catalogue's manifest to it —
a second store anywhere is refused by name, where it is assigned.
On an adopted hub the private network takes over the tunnel it finds rather
than running beside it (hq ADR 0105): two tunnels leave the mesh's unreachable
through the provider's filter, so no machine can ever join.
The node presents the found tunnel when it enrols, under the key it took as
its own; the inventory records it (node.tunnel, tunnel_peer — migration 0031)
and the mesh composes from it: the overlay's range is the adopted tunnel's,
the hub is placed at the tunnel's address on the tunnel's port, and every
peer the tunnel had is carried in the hub's peer list as a peer of the
tunnel, not a node of the mesh, until a node enrols with that key — which
then keeps the address the tunnel had for it. A fresh node never gets an
address the tunnel holds. The hub's declaration tells the host which unit to
take over; the host's account of carrying it is recorded and shown.
Every reader of the range follows the setting; nothing stores it. A found
tunnel under another key is recorded and not adopted, so ADR 0100's
non-overlap rule keeps applying where a tunnel is left running beside the
mesh's. A lab bed and test skeleton for "How it is checked" are under lab/.
Review found the first pass aliased its answer under both ends of a mapping, which is
wrong wherever two mappings share a number: the alias lands on a key belonging to another
mapping, the later write wins, and the filter and the container then disagree — the very
fault this change exists to close. Two reproduced cases: a module publishing 8080:80 beside
9090:8080 had an explicit setting silently overwritten; a module publishing 4001:80 beside
4002:80 composed both containers onto one machine port, where before it was safely refused.
Now a mapping's answer is filed once, under the end the module names in its listens — the
number the plan, the filter, the openings, the guard and the consumer all ask for — and a
key that names two mappings is refused in the same words as a setting that does.
Also: the guard assertion in the end-to-end test failed open when the resource was absent;
the plan-mirroring helper now says it stands in only where the plan does not allocate, and
the assertions it feeds are narrowed to the port under test.
A module publishing `2222:22` — the machine's own ssh daemon holds 22, so
the module takes 2222 and says so in `listens` — could not be moved. The
setting was read against the last segment of each mapping alone, so the
number the module uses everywhere else was refused as a port it does not
publish, and the node's every push failed for as long as the setting was
stored. The one key that was accepted, the container's own port, was then
read only when the container's mapping was rewritten: the mapping moved
and the ports map, the filter, the adopted node's openings, its guard and
what a consumer is told all stayed on the number the software had left.
Either end of a mapping now names it, and a given port comes back under
both, so every reader finds the same number under the key it holds.
Ambiguity is refused where it is real — one number naming two different
mappings, or the two ends of one mapping given two different numbers.
jochen
jschoubben