The jail.local [DEFAULT] gains ignoreip = 127.0.0.1/8 ::1 ${machine:mesh-range}
— localhost plus the mesh's own private range, named through the placeholder
rather than hardcoded (data is the mesh's, ADR 0112). Without it fail2ban could
ban the mesh's own nodes on 10.10.0.0/24; on novox that rule survived only in
memory from a now-deleted HAL file and would be lost on the next restart.
Claims renamed to match the controller's seat set: node-dns-resolver (dnsmasq),
node-intrusion-prevention (fail2ban), node-packet-filter (nftables),
node-resolver-config (resolv-conf, resolved-split-dns), node-uplink
(networkmanager, systemd-networkd, dhcpcd), mesh-build-machine (builder, +mesh
scope), mesh-catalog (mesh-catalog). showcase now declares its own seat and
claims it. verdaccio removed — the mesh keeps distribution as its registry and
gitea already serves npm, so a second npm registry is redundant.
The node-zones fact was a path; the local=/address= syntax lived in the
control plane. It is dnsmasq's configuration language, so it moves into
dnsmasq's manifest as a template over the roster. The mesh renders it; it
reads none of it. Output is unchanged.
Lands with mesh-controller's ADR 0120 change — the two are one schema step.
A workstation's job includes names that are neither a mesh machine nor
a routed name (novox/hq 122): shanks carries 13 Mediahuis entries in
/etc/hosts, and mesh-wireguard replaces /etc/hosts whole when taken —
so without this they vanish, and the take gates the node. Two homes,
neither the mesh's to own: conf-dir=/etc/dnsmasq.d/,*.conf (drop-in
directives, HAL's dnsmasq-app used exactly this) and
addn-hosts=/etc/hosts.local (plain host lines). The mesh creates and
rewrites neither; a machine with none loses nothing. The migration
moves such names here BEFORE the /etc/hosts take, closing the window.
The stated path was the adopted-data exception; with the take done and
the placement vocabulary live, the exception has no reason left. The
landing window renames the directory and recreates the container, since
a changed volume path does not do that by itself (hq 126).
The spec is the working system: HAL's 99-hal.conf, restated as
10-mesh.conf so lexical include order makes the mesh's answer the one
that wins while the predecessor's file is still on disk. Subsystem
stays the stock config's — first-set wins and it sits before the
Include. Port 22 from anywhere, said in listens with its reason: the
machines that need the door are exactly the ones not on the mesh yet,
and locking the operator out is the one failure a firewall must never
arrange.
The file is shared — the operator's insecure-registries for the mesh's
own store live there — and replacing it whole would break every pull
from that store the moment the module is taken (the 098 class, caught
in the pre-take diff this time). ADR 0102's verb is merge.
A consumer connecting by the binding's address meets a certificate for
mail.novox.be and refuses it — found live by the forwarder's cutover
proof, one send before production would have. The TLS name is mailu's
own fact (HOSTNAMES), so the binding carries it; consumers say
${bound:smtp:name} and verification holds.
The state directories say place "." — the assignment's own root — and
every bind, secret, own-secret, receives and grants path references it
as ${dir:state}/…; grants directories that are their own resources are
placed by id. gitea's two coincidence strings from the first pass
(${dir:data}base.json — resolving correctly by pure concatenation) are
spelled honestly now. What still says /var/lib is inside containers —
the software's contract — or under /var/lib/mesh, the mesh's own
plumbing, which the requirements unification absorbs next. Every
resolved path is byte-identical to what runs; landing this is a no-op
on the node, and the converter checks its own boundaries this time.
Each resolves to <root>/mailu/<id> — the maildir at
/var/lib/mailu/data-mail, certs at data-certs, and so on. Landing this
is a window, not an edit: seventeen renames on the node (the nested
data/ tree flattens to the ids), then the full stack recreated, because
a changed volume path does not recreate a container by itself (hq 126).
Ids are untouched on purpose — a renamed id orphans its held record,
and the mail spool is the wrong place to learn what a removal step does
with one.
The module is nextcloud; its tree was /var/lib/nextcloud-module — a
historic spelling nothing depends on. The root moves to
/var/lib/nextcloud (a rename on the node, done in this change's
window), and html drops its path: the mesh resolves it to
<root>/nextcloud/html. Landing this requires the window: rename the
tree, push, recreate the container — a changed volume path does not
recreate one by itself (hq 126).
The mesh resolves both to <root>/gitea/<id> — where the 5.8G forge and
its grant files already sit, so the roll-out its upgrade policy makes
of this build changes no byte of the spec. The module root and the
mesh's plumbing stay stated.
The mesh resolves it to <root>/mongodb/data — where the granted
databases already sit. The provider's own state, grants and the mesh's
plumbing stay stated.
Seven data directories drop their paths; the mesh resolves each to
<root>/only-office/<id>, which is exactly where the data already sits —
a textual no-op on this node, and the first module speaking ADR 0112's
vocabulary. The module root and the mesh's own state stay stated.
The /services paths were the adopted-node pattern doing its job: take
replaced containers over the predecessor's data without moving a byte
(gitea set it — 'its data never moved'). With every cutover done the
exception has no reason left, and the operator called it: a nox
module's world is /var/lib/<module>, data included. Six modules
repathed; mssql keeps its /services path deliberately — it is still
held, HAL-run, and moves at its own take. Both trees are one
filesystem, so each move is a rename.
The controller now accesses /var/lib/mesh-broker-tls (mesh-controller
#54) and the push refused whole: lavinmq declared the directory as an
owned resource, and shared data is the operator's, owned by no module
(ADR 0051). lavinmq only ever reads the certs — genesis laid them down
— so it declares a read access like the controller does, and the
directory belongs to nobody.
The env block carried the key twice — mailu-webmail from #79's address
sweep, and a stray =webmail further down that survived it. Last write
wins in an env file, so the front resolved a name that answers nowhere
on the mesh's network and 502'd every logged-in webmail request. Latent
since the cutover: the SSO redirect the checks watched never touches
the upstream; the operator's first real login did.
Take one (#90) died on two real edge bugs, both fixed and pinned by
tests in mesh-controller (#66: autocert 404s unknown tokens itself;
#67: the internal authority 403s every public name before the token
lookup). The challenge path verified end to end reaching mailu's own
nginx before this flip.
The letsencrypt flavor served certbot's April-expired state to live IMAPS
users within minutes: autocert's HTTPHandler answers 404 itself for
tokens it does not hold and never consults the fallback for challenge
paths, so mailu's own client cannot answer through the path-scoped
route. cert flavor (valid to Nov 27) until route-proxy's handler
actually falls through.
PR #82 set TLS_FLAVOR=cert as the honest interim while the predecessor's
proxy owned /.well-known/acme-challenge outright. route-proxy took port
80 today and its handler passes unknown tokens through to routed paths
by design — the one line #82 promised, made now. The copied cert (valid
to Nov 27) stays on disk untouched; mailu's own certbot takes over from
here.
The manifest predated the working deployment on three axes: it declared a
data directory the running portainer never used (taking it would have
started empty), pinned an image digest the machine has moved past (issue
099), and contributed no route while portainer.novox.be rides a traefik
container label today. Now: the predecessor's portainer_data path, the
running image's digest, 9090:9000 kept as the predecessor's machine port
with the route contribution naming it, and 9443 kept for the runtime
sidecar's own TLS conversation.
A bare '80' tried to bind the node's port 80 — the edge's — instead of
auto-allocating. 9070 is the predecessor's number and the one the route
contribution already names.
The app reads MONGO_DB (default 'invoicing') for every operation and
uses the URL only to connect — listCollections ran against a database
the granted user cannot see. Same fault and same fix as photos' MONGO_DB,
found by the API's own logs at take.
The mongo credential authenticates against its own database and the
database is the granted one (mesh_novox_invoice), not the contributed
name the provisioner ignores. Same for the store: the key is sealed to
the derived bucket (mesh-novox-invoice) — the data mirrors in during the
window, the ncloud/photos pattern. And the api gets the route
contribution it always needed: invoicing-api.novox.be is today a traefik
container label, invisible to every file survey, and it must be a grant
before the edge can ever flip.
The adopted node still runs the predecessor's mongo container, and it must
keep running: invoicing points at novox.be:27017 and is not migrating in
this window. A module container named 'mongo' would be held at assign and
would replace the predecessor at take, cutting invoicing off its database.
The mesh's server coexists instead — fresh data directory, its own name,
auto-allocated machine port — and the predecessor retires with its last
consumer.
The declared 0700 was applied at take and broke mail quietly: postfix's
master runs as root but pickup and smtpd drop to uid postfix, and a
spool root they cannot traverse is a maildrop they cannot scan and a
rewrite socket they cannot open — auth succeeded and MAIL FROM hung.
0755 root is exactly what postfix's own set-permissions makes of
/var/spool/postfix. Fixed live by chmod first; declared here so the
next push stops undoing it.
letsencrypt was the aspiration and cannot work yet, proven live: the
predecessor's own ACME machinery owns /.well-known/acme-challenge on
port 80 outright (unknown tokens get its 404) and its entrypoint
redirect owns every other path — the hand-authored passthrough never
matched anything, which is why mailu's certbot state had quietly
expired in April while the copied files carried the name. cert flavor
serves those files (valid to Nov 27). Mailu certifying itself becomes
possible the day route-proxy takes port 80, whose handler falls through
unknown tokens by design — that flip is one line here, made then.
The hand-written seed predates automx2's prio column, so every
config-v1.1.xml request 500'd against a table the seed had just made —
and the predecessor's own database had the same gap: client
autoconfiguration has been silently broken on the old stack for a long
time, behind a root page that answered 200. The live database gained
the column by ALTER; a fresh mesh now seeds it right.
An unpinned pip install took the latest automx2, whose schema grew a
column (server.prio) the module's own seeding SQL predates — a 500 on
every autoconfig request against a table the seed had just written.
2021.6 is what the proven image runs; the seed and the software agree
again. Regenerating the seed for a newer automx2 is its own change,
made deliberately, not by whatever pip resolved this week.
The front resolves its upstreams from *_ADDRESS, defaulting to the bare
compose service names — admin, antispam — and reads the 1.9-era HOST_*
not at all. Phase 1 never noticed because the predecessor's service
names WERE the defaults; the mesh's containers are mailu-*, and the
front answered 502 asking docker for a name nothing carries. The dead
vocabulary goes; every upstream is named as the container actually is.
2024.06's admin refuses to serve behind a resolver that does not
validate DNSSEC — found live as an unhealthy admin, 454s on submission
and a 500 webmail, with the runtime's forwarder validating nothing. The
unbound this module always shipped becomes reachable: pinned at the
predecessor's own address on the module network (the subnet the mesh
adopted), and named as dns by the nine containers that resolve anything.
Stands on mesh-host #26, which gave the vocabulary these two fields.
1.9's admin served on 80; 2024.06's gunicorn listens on 8080, and the
runtime's fetch failed against the old port the moment the broker
credential let it try.
The predecessor's image carried .venv/scripts/flask.sh, created by a
build step that never made it into the files this module holds — the
image worked and its recipe could not reproduce it, caught the moment
the mesh built it from source (exit 127 crash loop at cutover). The
wrapper is now written explicitly, verbatim from the proven image, so
the recipe is the whole truth about the image.
Phase 1 (MAILU-CUTOVER.md) upgraded the live stack 1.9→2024.06 and its
lessons land here as pins: every image is the digest running and
verified tonight — webmail under the name 2024.06 actually uses (the
draft's roundcube pin misled a whole hop), antivirus on the upstream
clamav image with the signature DB in its own directory (the mailu-built
image ended at 2.0), and the front trusting the proxy address the live
config actually names. The welcome-mail texts ride along for parity.
TLS_FLAVOR stays letsencrypt deliberately where the live .env says cert:
the cert files are copies whose HAL-era renewal hook died with HAL
(expiry Nov 27); mailu managing its own issuance through the existing
ACME passthrough is the fix, and the files remain on disk as the
fallback flavor if first issuance misbehaves during the window.
PORTS defaults to 25,80,443,465,993,995,4190 in 2024.06 — submission on
587 among the closed, which is what every client of this server uses.
The same parity decision the listens already state, now stated where the
software reads it.
Five gaps between the draft and what actually runs, each verified live
before being written down:
- front published bare 80 — the machine port Traefik holds; now the
predecessor's own mappings (7080:80, 7443:443) plus the 110/143/995
parity ports the draft dropped. Pruning legacy protocols is its own
deliberate change, not a cutover side effect.
- TLS_FLAVOR said cert, which nothing supplies; live is letsencrypt —
mailu runs its own certbot, state already on disk, HTTP-01 answered
through a path-scoped route contribution (priority above the web one).
- the web route said http:7080, the redirect-loop shape; it now says
what the hand-authored file always knew: https 7443, insecure.
- automx was absent entirely: the autoconfig responder is now a second
artifact (its Containerfile moved in from the predecessor's images
dir, base declared per ADR 0097), a container on a real data dir —
the anonymous-volume loss of 2026-08-10 stays fixed — and the three
public names are route contributions.
- and the reason this moved ahead of de-spiegel: mailu now provides
smtp. A consumer contributes the account it sends as; the provisioner
creates <account>@<domain> via the admin API and applies the minted
password every reconcile (ADR 0048). The domain is served on the
binding so a consumer composes its own login from mesh facts.
route-adapter learns to say no: a contribution over https, scoped to a
path, or carrying a policy is skipped aloud rather than written into a
file shape that cannot say it — plain http into a TLS listener was the
concrete wrong file this prevents. The hand-authored files keep covering
those routes until the mesh's own proxy takes over, exactly as today.
The builder's first credentialed clone of a private repository answered
'not found': the packages team named only repo.packages in its
units_map, which is exhaustive — so members had no code unit at all, and
gitea hides what a user cannot read. One credential answering npm and
git alike was the whole design of the builder's grant; the team now says
so.
And found teams are patched, not just returned: a team is configuration
the reconcile loop owns, the same as a user's password, so a unit this
code gains reaches the team that already exists rather than only the
next mesh raised from scratch.
The builder's package-registry grant — the first this provider ever
received — retried for a day saying only that an edit 404'd. Two faults
under it: the API refuses an email without a dotted domain, so
`@localhost` failed validation at create (the CLI that made mesh-admin
accepts it, which is why the admin exists and no consumer did); and
ensureUser read that 422 as 'already exists' and went on to edit a user
that was never made, burying the create's own message. The address is now
gitea's own hidden-address shape, and the edit path is taken only for a
user that is actually there.
The 2026-09-12 incident response blocked /api/internal by hand in the
predecessor's dynamic directory, with a note that its durable home is the
mesh's routing. A route carries the policy applied to a request (ADR
0108), so the refusal now travels with the grant: route-proxy enforces
it on both the public name and the internal alias the moment it serves
this route, and the adapter skips it aloud (no port, nothing to write)
while the predecessor's own file still stands. The hand-authored file
retires with the proxy it configures.
Two name spaces, two authorities (08-connectivity §2): a public name is
certified by a public CA, an internal one by the mesh's own. step-ca now
offers that second seat as internal-acme-ca beside its existing acme-ca,
and route-proxy requires both — the server dispatches by which authority
may certify the name at all, so an .internal alias stops being plain-HTTP
only without ever asking a public CA for a name it cannot validate.
files-api.novox.be and files.novox.be worked earlier tonight from route-
adapter-generated files, but minio's module.json on main never actually
carried a route requirement — that capability has been sitting in PR #58
the whole time, bundled with an unrelated network rename and console
redirect URL that need their own calmer review. This is just the two
routes: requires: route, contributes.route.api/.console (ContributesMany,
proven working via mesh-controller #55/#57), and the console port (9001)
actually published and declared in listens.
Found assigning route-proxy for the first time tonight: its own routes
file, generated the identical way route-adapter's always was, had four
hostnames in it instead of six — nothing served files-api/files at all,
which would have been a real, silent outage the moment Traefik stopped.
Composed ACME_ROOTS unconditionally from ${bound:acme-ca:roots} even when
that field is empty — public-acme's own case, where an empty roots means
'the system trust store', not 'fetch from the bare authority host'. The
run-once step wget'd https://acme-v02.api.letsencrypt.org:443 (host, no
path) for two minutes every apply and failed, blocking every resource
after it — found live tonight, assigning route-proxy for the first time.
Carries the raw, uncomposed roots value alongside the composed URL
(ACME_ROOTS_PATH) so the step can tell 'nothing to fetch' apart from 'the
authority didn't answer' — a distinction the composed URL alone cannot
make. Empty copies the image's own system CA bundle to /ca/root.crt
instead of fetching one, so ACME_CA_BUNDLE stays the one path it has
always been rather than needing to become conditional itself.
Was still pinned to the scaffold's placeholder digest (mesh-route-
proxy@sha256:0000...0000) even after the build+context work landed —
never caught because nothing had assigned route-proxy before tonight.
artifact: server, matching trust's own reference a few lines up and
every other built module in the catalogue.
step-ca (the other acme-ca provider) already spells it correctly; route-
proxy's own template reads ${bound:acme-ca:roots}. Found live, assigning
public-acme for the first time tonight: the mesh refused the push outright
rather than composing a broken binding — 'route-proxy asks its acme-ca
binding for roots, and what answers it says ... root'. Empty stays empty:
a public CA's root is the system trust store already, per route-proxy's
own design (an empty ACME_CA_BUNDLE means exactly that).
jschoubben