ADR 0121. The builder declared `built` as its own event, so every consumer depended
on which module happens to be the build machine today. It is the build-machine
role's event now: the builder declares none of its own, and the catalogue listens
for `mesh-build-machine.built` rather than `builder.built`.
Nothing changes about what reaches the catalogue. What changes is that it survives
the build machine being a different module, which is the whole reason the mesh has a
word for a role.
Every module named its events the way the old bus spelled a routing key —
`module.<module>.<verb>`. Design 29 says a module names an event locally and the
mesh works out where it lands, so all 37 were stale against a rule already
decided. On the new bus that derives into a namespace belonging to a module
called "module", so no cross-module subscription in the mesh matched anything:
nothing failed, nothing reacted (novox/hq 04-ISSUES/127).
36 manifests converted, and 43 files of module code with them. The code mattered
as much as the manifests: the runtime builds the subject from what `emit()` is
handed, so a converted manifest with unconverted code would have had the
permission and the subject disagree.
Three things the new check found on the way:
- `photos` emitted an event its manifest never declared, which the new bus refuses
outright. Declared.
- `showcase` waited for an event nothing emits, so its demo could never be
triggered — only `showcase` may publish under its own name. It emits both halves
now.
- `distribution` declared an event named after a different module. It emits
`image.pushed` under its own name. An event about a *role* belongs on the seat,
where the name outlives whoever holds it, but the sdk has no way to publish on a
seat yet, so that stays recorded rather than declared.
The audit logger's "everything" pattern is `**` rather than the old bus's `#`.
The split the controller now makes, from this side. The module's own
configuration — ports, TLS, JetStream — is a declared file resource, because those
are properties of this container and change when its image does. `bus-users` names
where the mesh writes every account and permission, in the same directory, and the
module's configuration includes it.
**Both files in one directory because they have to be.** An absolute include path
is resolved relative to the including file's directory: nats-server given
`include /etc/nats/accounts.conf` from /etc/nats-server/nats.conf looks for
/etc/nats-server/etc/nats/accounts.conf and refuses to start. Verified against the
server, and recorded in the configuration itself where somebody moving a file will
read it.
**`verify: true` is gone, and it was refusing every connection in the mesh.** It
makes the server demand a client certificate; a host pins this server's exact
certificate and authenticates with the password the mesh minted, and presents none.
Found by building this image and connecting to it as a host would.
The entrypoint now waits for both files and watches the mesh's half: the module's
own does not change without a new declaration, and that recreates the container
anyway. Verified end to end against this image — the mesh's user list rewritten,
the module noticing and reloading the server itself with no signal from outside,
and the connection the mesh already had still working afterwards.
Ten manifests claim mesh-* names now. What they PROVIDE is unchanged: gitea
still provides git and npm-package-registry, and a consumer requires the
interface, not the seat.
It claims no seat: mesh-broker is the NATS server's (novox/hq ADR 0119).
The amqp interface stays exactly as it is — a backing service a module may
require, like a database.
Step 1.1 and 1.2 of novox/hq ADR 0116. The server is a built artifact rather
than the upstream image directly, because it needs an entrypoint of its own:
the host can only recreate a container, and recreating the bus for every
permission change drops every connection and every in-flight ack. nats-server
reloads on SIGHUP by itself, so the config is mounted as a directory (not
digest-tracked, hq issue 103) and the entrypoint watches the one file.
Verified against the real server, not assumed: a user added to the config
connects, a revoked one is refused, both within one poll interval, with the
container's PID and restart count unchanged and "Reloaded: accounts" in its
log.
Two corrections found by checking rather than reading:
- the seat delivers nothing now (hq ADR 0117), and the controller's parser
refused the manifest until it did — "nats claims mesh-broker, whose holder
answers for amqp, and nats does not provide amqp"
- pinned to the multi-arch index digest; the first pin was the amd64
manifest, which builds here and fails on any other architecture
The state directories say place "." — the assignment's own root — and
every bind, secret, own-secret, receives and grants path references it
as ${dir:state}/…; grants directories that are their own resources are
placed by id. gitea's two coincidence strings from the first pass
(${dir:data}base.json — resolving correctly by pure concatenation) are
spelled honestly now. What still says /var/lib is inside containers —
the software's contract — or under /var/lib/mesh, the mesh's own
plumbing, which the requirements unification absorbs next. Every
resolved path is byte-identical to what runs; landing this is a no-op
on the node, and the converter checks its own boundaries this time.
Each resolves to <root>/mailu/<id> — the maildir at
/var/lib/mailu/data-mail, certs at data-certs, and so on. Landing this
is a window, not an edit: seventeen renames on the node (the nested
data/ tree flattens to the ids), then the full stack recreated, because
a changed volume path does not recreate a container by itself (hq 126).
Ids are untouched on purpose — a renamed id orphans its held record,
and the mail spool is the wrong place to learn what a removal step does
with one.
The module is nextcloud; its tree was /var/lib/nextcloud-module — a
historic spelling nothing depends on. The root moves to
/var/lib/nextcloud (a rename on the node, done in this change's
window), and html drops its path: the mesh resolves it to
<root>/nextcloud/html. Landing this requires the window: rename the
tree, push, recreate the container — a changed volume path does not
recreate one by itself (hq 126).
The mesh resolves both to <root>/gitea/<id> — where the 5.8G forge and
its grant files already sit, so the roll-out its upgrade policy makes
of this build changes no byte of the spec. The module root and the
mesh's plumbing stay stated.
The mesh resolves it to <root>/mongodb/data — where the granted
databases already sit. The provider's own state, grants and the mesh's
plumbing stay stated.
Seven data directories drop their paths; the mesh resolves each to
<root>/only-office/<id>, which is exactly where the data already sits —
a textual no-op on this node, and the first module speaking ADR 0112's
vocabulary. The module root and the mesh's own state stay stated.
The /services paths were the adopted-node pattern doing its job: take
replaced containers over the predecessor's data without moving a byte
(gitea set it — 'its data never moved'). With every cutover done the
exception has no reason left, and the operator called it: a nox
module's world is /var/lib/<module>, data included. Six modules
repathed; mssql keeps its /services path deliberately — it is still
held, HAL-run, and moves at its own take. Both trees are one
filesystem, so each move is a rename.
The provisioner derives a consumer's bucket from the login the mesh minted — 'derived from the
login, so teardown recomputes it with nothing to persist' — and never reads the bucket a manifest
contributed. Three modules contributed one anyway, and the value was decorative in two and wrong in
the third: photos told its container MINIO_BUCKET=photos, the predecessor's bucket, while its minted
key is scoped to mesh-novox-photos. Deployed as it stood, it would have authenticated and then been
denied on every object.
photos now names the bucket the mesh actually provisions, and the contributed bucket is gone from
all three: a value nothing reads, that reads as though it decides.
Verified against the live store before changing anything: the derived names are the populated ones —
mesh-novox-ncloud (77,886 objects, 174.9 GiB), mesh-novox-photos and mesh-novox-invoice. Nothing has
to move.
The controller now accesses /var/lib/mesh-broker-tls (mesh-controller
#54) and the push refused whole: lavinmq declared the directory as an
owned resource, and shared data is the operator's, owned by no module
(ADR 0051). lavinmq only ever reads the certs — genesis laid them down
— so it declares a read access like the controller does, and the
directory belongs to nobody.
The adapter skips what its one file shape cannot say. A body limit is the exception: the predecessor
has a buffering middleware and served its own registry name with exactly it, so this is written
rather than skipped, named after the router so the two halves cannot drift.
A limit that is not a whole positive number of bytes takes the route with it. Written without the
limit, the predecessor would carry what the module said not to carry and this module would report
success. Silence stays silence — no middleware, the predecessor's default.
The env block carried the key twice — mailu-webmail from #79's address
sweep, and a stray =webmail further down that survived it. Last write
wins in an env file, so the front resolved a name that answers nowhere
on the mesh's network and 502'd every logged-in webmail request. Latent
since the cutover: the SSO redirect the checks watched never touches
the upstream; the operator's first real login did.
Take one (#90) died on two real edge bugs, both fixed and pinned by
tests in mesh-controller (#66: autocert 404s unknown tokens itself;
#67: the internal authority 403s every public name before the token
lookup). The challenge path verified end to end reaching mailu's own
nginx before this flip.
The letsencrypt flavor served certbot's April-expired state to live IMAPS
users within minutes: autocert's HTTPHandler answers 404 itself for
tokens it does not hold and never consults the fallback for challenge
paths, so mailu's own client cannot answer through the path-scoped
route. cert flavor (valid to Nov 27) until route-proxy's handler
actually falls through.
PR #82 set TLS_FLAVOR=cert as the honest interim while the predecessor's
proxy owned /.well-known/acme-challenge outright. route-proxy took port
80 today and its handler passes unknown tokens through to routed paths
by design — the one line #82 promised, made now. The copied cert (valid
to Nov 27) stays on disk untouched; mailu's own certbot takes over from
here.
The manifest predated the working deployment on three axes: it declared a
data directory the running portainer never used (taking it would have
started empty), pinned an image digest the machine has moved past (issue
099), and contributed no route while portainer.novox.be rides a traefik
container label today. Now: the predecessor's portainer_data path, the
running image's digest, 9090:9000 kept as the predecessor's machine port
with the route contribution naming it, and 9443 kept for the runtime
sidecar's own TLS conversation.
A bare '80' tried to bind the node's port 80 — the edge's — instead of
auto-allocating. 9070 is the predecessor's number and the one the route
contribution already names.
The app reads MONGO_DB (default 'invoicing') for every operation and
uses the URL only to connect — listCollections ran against a database
the granted user cannot see. Same fault and same fix as photos' MONGO_DB,
found by the API's own logs at take.
The mongo credential authenticates against its own database and the
database is the granted one (mesh_novox_invoice), not the contributed
name the provisioner ignores. Same for the store: the key is sealed to
the derived bucket (mesh-novox-invoice) — the data mirrors in during the
window, the ncloud/photos pattern. And the api gets the route
contribution it always needed: invoicing-api.novox.be is today a traefik
container label, invisible to every file survey, and it must be a grant
before the edge can ever flip.
The adopted node still runs the predecessor's mongo container, and it must
keep running: invoicing points at novox.be:27017 and is not migrating in
this window. A module container named 'mongo' would be held at assign and
would replace the predecessor at take, cutting invoicing off its database.
The mesh's server coexists instead — fresh data directory, its own name,
auto-allocated machine port — and the predecessor retires with its last
consumer.
ALTER USER ... WITH LOGIN runs only when the user's SID is not the
login's, so an already-mapped user is left alone. The provisioner
enables a mailbox through its own method; the password tool an
operator uses keeps changing the password only.
create re-enables what holds refuses (mssql login, mosquitto client,
mailu mailbox, gitea user) and clears an expired postgres password, so
no disabled account loops. mssql and mongodb checks take the password
from the environment, never argv; mosquitto_ctrl failures no longer
repeat -P. mosquitto reads 'could not ask' as an error, not absence.
mailu checks existence and enabled only: its imap passdb cannot verify
a password. mssql checks the user's SID; gitea pages teams at 50.
holds() for postgres, mssql, mongodb, minio, lavinmq, mosquitto, mailu
and gitea, so the harness makes again a login the backend lost (hq issue
120). Each checks the mesh's password as the consumer presents it, or
compares it read-only, and returns false only when the backend says the
credential is absent or wrong; an unreachable backend throws.
The server keeps ACL users in memory only, so a restart forgets every
consumer while the provisioner keeps running (hq issue 120). holds()
checks ACL GETUSER for the user, enabled, with the mesh's password, so
the harness makes a forgotten user again. Needs mesh-sdk 0.1.1.
The declared 0700 was applied at take and broke mail quietly: postfix's
master runs as root but pickup and smtpd drop to uid postfix, and a
spool root they cannot traverse is a maildrop they cannot scan and a
rewrite socket they cannot open — auth succeeded and MAIL FROM hung.
0755 root is exactly what postfix's own set-permissions makes of
/var/spool/postfix. Fixed live by chmod first; declared here so the
next push stops undoing it.
letsencrypt was the aspiration and cannot work yet, proven live: the
predecessor's own ACME machinery owns /.well-known/acme-challenge on
port 80 outright (unknown tokens get its 404) and its entrypoint
redirect owns every other path — the hand-authored passthrough never
matched anything, which is why mailu's certbot state had quietly
expired in April while the copied files carried the name. cert flavor
serves those files (valid to Nov 27). Mailu certifying itself becomes
possible the day route-proxy takes port 80, whose handler falls through
unknown tokens by design — that flip is one line here, made then.
The hand-written seed predates automx2's prio column, so every
config-v1.1.xml request 500'd against a table the seed had just made —
and the predecessor's own database had the same gap: client
autoconfiguration has been silently broken on the old stack for a long
time, behind a root page that answered 200. The live database gained
the column by ALTER; a fresh mesh now seeds it right.
An unpinned pip install took the latest automx2, whose schema grew a
column (server.prio) the module's own seeding SQL predates — a 500 on
every autoconfig request against a table the seed had just written.
2021.6 is what the proven image runs; the seed and the software agree
again. Regenerating the seed for a newer automx2 is its own change,
made deliberately, not by whatever pip resolved this week.
The front resolves its upstreams from *_ADDRESS, defaulting to the bare
compose service names — admin, antispam — and reads the 1.9-era HOST_*
not at all. Phase 1 never noticed because the predecessor's service
names WERE the defaults; the mesh's containers are mailu-*, and the
front answered 502 asking docker for a name nothing carries. The dead
vocabulary goes; every upstream is named as the container actually is.
2024.06's admin refuses to serve behind a resolver that does not
validate DNSSEC — found live as an unhealthy admin, 454s on submission
and a 500 webmail, with the runtime's forwarder validating nothing. The
unbound this module always shipped becomes reachable: pinned at the
predecessor's own address on the module network (the subnet the mesh
adopted), and named as dns by the nine containers that resolve anything.
Stands on mesh-host #26, which gave the vocabulary these two fields.
1.9's admin served on 80; 2024.06's gunicorn listens on 8080, and the
runtime's fetch failed against the old port the moment the broker
credential let it try.
The predecessor's image carried .venv/scripts/flask.sh, created by a
build step that never made it into the files this module holds — the
image worked and its recipe could not reproduce it, caught the moment
the mesh built it from source (exit 127 crash loop at cutover). The
wrapper is now written explicitly, verbatim from the proven image, so
the recipe is the whole truth about the image.
Phase 1 (MAILU-CUTOVER.md) upgraded the live stack 1.9→2024.06 and its
lessons land here as pins: every image is the digest running and
verified tonight — webmail under the name 2024.06 actually uses (the
draft's roundcube pin misled a whole hop), antivirus on the upstream
clamav image with the signature DB in its own directory (the mailu-built
image ended at 2.0), and the front trusting the proxy address the live
config actually names. The welcome-mail texts ride along for parity.
TLS_FLAVOR stays letsencrypt deliberately where the live .env says cert:
the cert files are copies whose HAL-era renewal hook died with HAL
(expiry Nov 27); mailu managing its own issuance through the existing
ACME passthrough is the fix, and the files remain on disk as the
fallback flavor if first issuance misbehaves during the window.
PORTS defaults to 25,80,443,465,993,995,4190 in 2024.06 — submission on
587 among the closed, which is what every client of this server uses.
The same parity decision the listens already state, now stated where the
software reads it.
Five gaps between the draft and what actually runs, each verified live
before being written down:
- front published bare 80 — the machine port Traefik holds; now the
predecessor's own mappings (7080:80, 7443:443) plus the 110/143/995
parity ports the draft dropped. Pruning legacy protocols is its own
deliberate change, not a cutover side effect.
- TLS_FLAVOR said cert, which nothing supplies; live is letsencrypt —
mailu runs its own certbot, state already on disk, HTTP-01 answered
through a path-scoped route contribution (priority above the web one).
- the web route said http:7080, the redirect-loop shape; it now says
what the hand-authored file always knew: https 7443, insecure.
- automx was absent entirely: the autoconfig responder is now a second
artifact (its Containerfile moved in from the predecessor's images
dir, base declared per ADR 0097), a container on a real data dir —
the anonymous-volume loss of 2026-08-10 stays fixed — and the three
public names are route contributions.
- and the reason this moved ahead of de-spiegel: mailu now provides
smtp. A consumer contributes the account it sends as; the provisioner
creates <account>@<domain> via the admin API and applies the minted
password every reconcile (ADR 0048). The domain is served on the
binding so a consumer composes its own login from mesh facts.
route-adapter learns to say no: a contribution over https, scoped to a
path, or carrying a policy is skipped aloud rather than written into a
file shape that cannot say it — plain http into a TLS listener was the
concrete wrong file this prevents. The hand-authored files keep covering
those routes until the mesh's own proxy takes over, exactly as today.
The builder's first credentialed clone of a private repository answered
'not found': the packages team named only repo.packages in its
units_map, which is exhaustive — so members had no code unit at all, and
gitea hides what a user cannot read. One credential answering npm and
git alike was the whole design of the builder's grant; the team now says
so.
And found teams are patched, not just returned: a team is configuration
the reconcile loop owns, the same as a user's password, so a unit this
code gains reaches the team that already exists rather than only the
next mesh raised from scratch.
jschoubben
jochen