mesh/merge-gate pass: builds minio, sent nowhere; no bus step; 1 wait(s) for a person; every machine composes with the change as it did without (4 of 4 com…
- dnsmasq holds mesh-dns-resolver: provides wildcard-resolution mesh-wide, forwards every declared
zone (zones fact), listens on the private address and loopback only, reads no hosts file and no
operator's files, and no longer writes the container runtime's dns.
- resolv-conf names the mesh's resolver by address, then 1.1.1.1, timeout 1, one attempt; it now
holds the runtime's live-restore, which dnsmasq held and every node needs.
- resolved-split-dns routes the suffix to the mesh's resolver by address, not 127.0.0.1.
- hosts: new module holding node-hosts-file — the machine's own lines in its block of /etc/hosts,
the operator's lines kept, changed by entries/add/remove through sudo -n.
Withdrawn 2026-10-04 to unblock novox: while-stopped named the module-local
id and the host refuses a declaration naming a container it does not have,
whole. mesh-controller#259 fixed the namespacing and it has been live since,
so the window composes as distribution.store and the host will take it.
Unchanged from before: plain garbage-collect at 03:30 with the server held
still. The store is at 40G and nothing reclaims it until this runs.
The first restore of an arr app refused its settings file with "not a directory": restic restores
a snapshot's subfolder, not a file. A file is restored with its full path into a scratch directory
beside the target, moved into place, and the scratch removed.
The arr apps keep SQLite databases; a consistent copy of a live one is sqlite3's .backup. Declared
once, by the holder that runs the copies, rather than by each app.
The first run on the control node called postgres's dumps missing: the holder looked as the
runtime's account, which cannot see inside a store's 0700 data directory. Every other act already
runs as root; the existence checks do too now.
Go is the default for module code. One binary, fail2ban-tools, serving the node-intrusion-prevention
seat's four verbs and fail2ban_settings over the SDK, with the same parsing and the same tests; read
back against the control node's live daemon.
Go is the default for new module code; the holder is one binary like the licence manager, serving
the seat's verbs over the SDK and running the nights beside them. Same behaviour and tests.
ADR 0214 / to-be 43. restic keeps one repository per machine and takes a nightly snapshot per
module — 14 daily, 8 weekly, 6 monthly — and restores beside the live data, never over it. postgres,
mssql and mongodb contribute a consistent dump; minio, influxdb, the vault, mailu, gitea and
nextcloud the directories that hold their data.
mssql disables the login, mongodb takes the user's roles, minio revokes the key and keeps the bucket,
mailu disables the mailbox, gitea prohibits the login instead of purging the user and their
repositories, umami keeps the website. Each provider's create already enables what this locks.
Two definitions held one module name. Rebuilt to this catalogue's main on 2026-10-04, the mesh took
this stub — a server and an admin client — over the app's definition in photos.git, and five of its
six sites lost their routes. The app's repository is the source, as de-spiegel's and link2pay's are.
The provider creates each consumer's user in that consumer's own database.
photos asked for admin, where no such user exists; invoicing, against the
same provider, already asked for the name the mesh gave it and worked.
Invisible until hq 225 was fixed: while the provisioner could not read its
secrets, no user existed anywhere, so 'UserNotFound for db admin' was a true
and complete account of that fault. A fault that explains the symptom is not
evidence there is only one.
Each declares a web endpoint — 4001, 4012, 4013 — and published a bare 80,
which the mesh has nothing to assign for, so 80 reached the machine and
collided with the reverse proxy. Written the long way, the software's 80 is
published at the port the module declares and the mesh rewrites the outer
half to whatever it assigned.
photos is the one that failed on the control node; the other two are the same
fault waiting for a machine that runs a proxy.
mesh-controller#259 fixes while-stopped to name the container as the machine
knows it — `distribution.store`, not `store`. Until that controller is the
one composing, novox refuses its whole declaration and takes nothing at all.
The step comes out; deletion stays on, already applied and harmless on its
own. A collect step without its window would be worse than none: garbage
collection against a live registry can sweep a blob a build is pushing.
Put back once the fixed controller is deployed and stays.
The bundles refactor took ADR 0188 on main, so minio's comments cite 0201.
The sdk is 0.1.7 after the same rebase, and minio needs the `derived` field
it carries.
REGISTRY_STORAGE_DELETE_ENABLED on the server — the door already accepts a
push — and a scheduled step running the registry's own collector over the
volume at 03:30 with the server held still. Plain garbage-collect: what the
mesh keeps is still a manifest, so --delete-untagged is not needed and would
delete images machines are running.
serves.s3-bucket.bucket is ${consumer:as:dns}; the provisioner uses what it
is given. nextcloud, invoicing and photos ask for ${bound:s3-bucket:bucket}
instead of naming mesh-novox-* literals, which also named this node.
bucketFor and the long-dead accessKeyFor are gone.
The names region of /etc/hosts is written by mesh-wireguard and dnsmasq answers from it, but read
it once at start: after the mesh stopped publishing public names (hq ADR 0191) every machine's
hosts file was right and every resolver still answered mail.novox.be with a tunnel address. The
config comment that says so changes the file, which restarts dnsmasq once everywhere.
A package has one owning module, and sshd already owns openssh on every node — assigning
ssh-client was refused for both declaring it, and the refusal left every node unresolvable
until it was unassigned. Owning it was wrong besides: unassigning ssh-client would have
removed the package sshd serves from. The client binary ships in the package sshd holds.
Requires openssh; creates ~/.ssh (0700, owned by the operator account via
${machine:account}); writes every other node's Host block (HostName + User
<account>) into a marked region of ~/.ssh/config (home-scoped, into:block), so
`ssh <node>` reaches each peer as the right account and the operator's own
config is kept. Universal-tier: assigned wherever a person logs in; a node with
no account gets no config.
step-ca and public-acme both offered acme-ca on novox, and route-proxy's pin names a node, not
a module — so which one certified the public names depended on provider order. After the
controller restart on 2026-10-03 it came out as step-ca, and every public site served a
certificate no browser trusts. step-ca's own names are already certified through
internal-acme-ca; acme-ca is the public authority's alone.
The jail read gitea's container journal, which carries its sshd's lines,
but matched only the web login. 167 ssh attempts an hour from the
internet went unbanned. Two patterns, one per attempt: an unknown user,
and a user sshd refuses; tested against a day of the real log, 946
matches and none on an accepted login.
The home server banned the house's own router within an hour of the first public jail: the router
reflects local traffic, so every client in the building arrives as the gateway's address. Every
private range joins the mesh's own in the never-ban list.
fail2ban expands <HOST> to a named group, so two in one pattern is a duplicate group name and
the daemon refuses to start at all -- every jail on the machine, not just this one. Two patterns,
one <HOST> each: the certificate refused for an unserved name, and the request refused for one.
Caught live on the control node (hq ADR 0179).
The pattern ended at the line's end, which only the certificate refusal does; a request for
an unserved name carries trailing text and never matched. Caught against the live lines
before the jail counted anything (hq ADR 0179).
The module gains a runtime carrying only the fail2ban client with the daemon's socket shared in,
serving status/banned/ban/unban and its own fail2ban_settings. It declares jailing, so the
controller's composition lands in jail.d/mesh.conf and filter.d; mailu, route-proxy and gitea log to
the journal and declare a jail reading it by container name. The base is strict: three in a day for
a day, twice banned in two weeks for four; the mesh's range stays never banned.
Both sources sit in tools/, so tsc took tools/ as the root and wrote
dist/index.js, while the runtime loads dist/tools/index.js: the module
started and served nothing.
Five tools on the machine the lab runs on: check, run beds against
branches on the forge, a run's status, its log, and stop. A run checks
out every repository the lab builds, side by side, and runs the suite;
one at a time, answered at once with an id (novox/hq ADR 0172).
The seat's three verbs over the machine's own tools: the filter as enforced
(nftables and the legacy filter), the mesh's own table reloaded from its file,
and one rule set the mesh did not write removed by the name the host reports
it under (ADR 0168) — a predecessor's chain loses its jumps and goes, the
runtime's user chain is emptied back to its return, a table of the machine's
own goes whole; the mesh's tables, the runtime's chains, a built-in chain and
an active found firewall's chains are refused. Tested over the shapes two
machines of the first mesh reported live. The module's own tool stays.
Which DNS server the home network's DHCP hands out could be changed only
in the controller's own interface or by hand against its API (novox/hq
issue 198).
The addresses dnsmasq listens on beside the machine's are a setting, so a
machine that answers its own LAN can say so (novox/hq issue 198). Docker's
daemon.json no longer merges the module's settings: it needs none, and a
setting reaching it is a key dockerd refuses. The host still merges it
into the existing file.
A module's settings merge into every mergeable file it owns, so the
setting reached docker's daemon.json beside dnsmasq's config, where
dockerd would refuse it (novox/hq issue 198). Back to the fixed
loopback line until settings can be kept out of files they are not for.
Loopback by default, as before. A machine that answers its own LAN adds
its LAN address, and its DNS endpoints' reach opens the filter (novox/hq
issue 198). The mesh-wide default must be set before this lands.
Proven on a throwaway server: as the administrator a caller's $(SQLCMDPASSWORD) returned
the sa password, and a line beginning ':!!' ran a program in the tools container. The
statement now runs as mesh_mssql_reader (CONNECT ANY DATABASE, SELECT ALL USER SECURABLES),
with substitution off (-x), after the module's own text on the first line, and a line break
is refused. go-sqlcmd v1.10.0 is installed at a pinned digest: the image never had sqlcmd,
so every mssql tool failed with spawn sqlcmd ENOENT.
The verb wrapped the caller's text in BEGIN READ ONLY ... ROLLBACK as the superuser, so
'COMMIT; ...' left the transaction and, proven on a throwaway server, COPY TO PROGRAM ran a
shell command on the database host. The statement now runs as mesh_store_reader:
pg_read_all_data, no other grant, read-only transactions by role and session, its password
an own-secret the mesh mints. Without that password the call is refused. -q drops the
command tags that came back as rows keyed by BEGIN.
The same split lost them the other way round: the merge base held both changes, each branch had reset
the other's files, and the three-way merge kept neither. Both halves of hq ADR 0161 are on main again
with this.
The uplink branch was split from the vault's with the vault's files reset to a main that did not yet
hold #205; merging it afterwards took the older vault definition along (no claim, the refused event
names), and the vault could not be built. Restored to #205's state.
A module publishes under its own name only; secret.provisioned read as another module's event and the
builder refused the vault's definition today, so the seat claim could not be built. The three events lose
the prefix; nothing outside the vault listens for the old names.
The vault's claim makes a second provider of secret a second claimant, refused by name. The three
uplink definitions declare uplink-networkmanager, uplink-systemd-networkd and uplink-dhcpcd, which the
host reports for the manager it finds active, so the holder for a manager the machine does not run
is refused the way any missing capability is. Merges after the controller holds the seat and the host
reports the capability.
The store's databases and query are registered under mesh-store, so the runtime serves them on the
seat's subjects wherever postgres holds the seat and never lists them as postgres's own; the claim
names them, so the mesh can judge the holder without postgres listing the seat's verbs among its
tools. Scoped to what the store enables: creating a database stays postgres's tool.
bazarr, bookshelf, lidarr, nzbget, ombi, plex, qbittorrent, radarr,
sonarr and tautulli live in novox/mesh-media-catalog (#195 moved jackett
and left these behind). A build of this repository at a commit today
registered plex and nzbget from these copies, which carry no provides,
and ace's plan stopped resolving. Nothing here depends on the directories:
home-assistant consumes their provisions by name.
Named as the seat names them so the runtime finds them by name; the same calls as its own tools.
Its definition now lists its tools, which is what holding a seat with verbs demands at registration.
Merge before the controller declares the verbs on the mesh-store seat.
Both land in settings.js and the runtime's config file, and the containers that read them restart
on those files; a rotation is a new value and a restart. The broker credential says nothing yet: its
other party is the bus, and that rotation is the two-party form.
Ten tools the console lacked for the actions a review and a merge leave behind: close or reopen a
pull request whose work landed elsewhere, change its title or body, read its files, its diff and its
comments, reopen an issue, read one file at a ref, list branches, delete the branch a closed pull
request leaves. Each is the client's own call; `gitea_api` stays the escape hatch for the rest.
Tested against the fake forge through the compiled tools, the way the console calls them (13/13).
The container mounted /services/media literally — one installation's path
(ADR 0112). The access is now declared by id and mounted as ${access:media};
the assignment says where the library is (ace: /storage/media, hq 153).
The module named /var/lib/n8n, /services/n8n/n8n-data and n8n.novox.be -
paths and a domain no definition may carry (ADR 0112). State and data are
placed directories; the public name is ${bound:route:name} (depends on
mesh-controller #149), for N8N_HOST and WEBHOOK_URL alike.
The endpoint said 5682 while the container publishes 5678. 5682 was one
machine's host port; the endpoint is the software's port and the mesh
assigns the machine's (ADR 0038).
n8n had been run from an image in a registry that no longer exists: the
upstream image plus shadow, a `media` group (2000) with `node` in it, and
a global `uuid`. That recipe is now this module's Dockerfile, built on the
upstream 1.71.3 image named in build.on by digest, with uuid pinned to the
version the running image carries (14.0.1) - Code nodes require() it. The
media group is how the container writes into the shared media library, a
read-write `access` (ADR 0051), mounted where workflows expect it,
/media-library.
The workflows also use a redis (the Redis nodes of the chat workflows) and a
Selenium Chrome (the scraper), which the previous deployment ran beside n8n.
Both are containers on the module's own network, publishing nothing, pinned
to the digests in use; redis keeps its append-only file in a placed
directory.
The basic-auth secret is gone: N8N_BASIC_AUTH_* was removed in n8n 1.0 and
did nothing. The grant's password is a 0400 file owned by `node`, read
through DB_POSTGRESDB_PASSWORD_FILE, so nothing secret is in the
environment. The credentials' encryption key is n8n's own, in the data
directory (config), and moves with it - nothing to mint or accept.
Verified: catalogue tests with MESH_CATALOGUE set; the Dockerfile built
against the pinned base gives n8n 1.71.3, uid 1000 in group 2000, uuid
14.0.1 - the running image's shape. Throwaway containers: an instance on
PostgreSQL 15 with an owner, a workflow and an encrypted credential;
stopped, copied, dumped from the copy, restored (--no-owner --role, the
uuid-ossp extension pre-made by the superuser) into a grant-shaped
database on the postgres module's pgvector image (PG17); the new shape
(password from the file, data dir copied) serves /healthz, the owner logs
in, the workflow is listed, and the credential decrypts with the carried
key. The node user writes into a root:2000 0775 library through the media
group; redis and Selenium resolve by name on the module network and
Selenium reports ready. Test containers and data removed.
Ten definitions name each access by id; the path stays as the default an assignment may replace,
and the host side of every mount says ${access:<id>}. Resolved with no placement, every definition
names exactly the paths it named before (TestPlacedDirectoriesKeepTheirPaths, extended). On an
adopted machine the assignment now says `accesses: {<id>: <path>}` and the mount follows.
Needs the controller that knows an access id (mesh-controller #176); the running one refuses the
field at registration.
Consumers read `${bound:smtp:domain}` and `${bound:oidc-client:issuer}`, and both keys reached
them only because a module's settings were laid over everything it served. Issue 173 stops that: a
setting overrides a key a served fact declares and adds none. So the two providers declare the keys
their consumers read, as the operator's value (`${setting:…}`, ADR 0155), and the setting that
already carries each fills it. Nothing a consumer reads changes.
Merges first: under the controller that still merges settings over served facts this is the same
value, and the controller that stops merging (mesh-controller, feat/the-mesh-places-its-own-files)
needs these declared before it rolls out.
48 definitions stop naming /var/lib/mesh/<module>: the directory says `place: "mesh"`, the two
subdirectories beneath it (gitea's runtime state, anthropic-manager's output) state their path
beneath it, and every credential, binding, merged file and mount names it as ${dir:mesh-state}.
Resolved on the default root, every definition names exactly the paths it named before —
TestPlacedDirectoriesKeepTheirPaths in mesh-controller, run over both checkouts. Needs the
controller that knows the word (mesh-controller, same branch) one release ahead.
jackett leaves: it is registered from novox/mesh-media-catalog with sonarr,
radarr, lidarr, bazarr, nzbget, qbittorrent, bookshelf, plex, tautulli,
kometa and ombi (PRs 145-168 consolidated there). What those branches
changed outside the chain stays here: home-assistant's provisions
(sonarr-api, radarr-api, mqtt-topic — from #147) and searxng's sidecar
dialling the port it was given (#154).
Twenty-eight modules' data directories are placed: the root as place ".", a sub-directory named by
its id, and every host-side reference — binds, secrets, own secrets, grants, receives, file paths,
mounts, env-files — as ${dir:<id>}. Resolved on the default root every path is the one the manifest
named before, which the controller's TestPlacedDirectoriesKeepTheirPaths proves over both checkouts;
so no data moves and no machine sees a change. Five directories whose id is not their last segment
keep their path as a placement (novox/hq issue 119, ADR 0112, design 27).
The forge's own users are the mesh's to settle — making the builder's login
a site admin so private repos build (hq 229) — and the tools' token had no
write:admin. A token kept from before a scope was added lacks it, so the
client now treats the forge's 403 "required scope" like a 401: the source
re-mints by name with the whole list and retries once. The fake forge in the
tests learns /repos/search, which the client has used since 2026-09-28 and
which had left 9 of the 11 token tests failing on main.
listens said 4000 while the container publishes 8080; the old assignment's port setting hid it, and
the rename lost the setting, so the proxy dialled a port nothing answered (2026-09-30).
The rename changed the network resource's name and not the container's network, so the container
looked for a network that no longer exists and the site answered 502 (2026-09-30).
9090:9000 and 9443:9443 were the predecessor's machine numbers written into the
definition. The manifest now says 9000 and 9443 and the mesh assigns the
machine ports on each node (the portainer slice of #173, which is stale).
keycloak, minio and n8n are told their names from their route bindings; mailu takes its domain, site
name, website and proxy address as settings and its front's name from its route, and the provisioner
reads the domain from the merged config; builder and route-proxy package the controller from the git
seat; the applications built outside the mesh say so per container; matrix says which of the world's
servers it means; the site module is named website, and the why prose no longer names a name (novox/hq
ADR 0155, issues 122 and 134). module check passes over all 77.
#186 moved settings.js to /data for the image's health check but left
settings.json at /config; settings.js requires ./settings.json, so node-red
crashed at start. Both now mount under /data.
The image's /healthcheck.js requires /data/settings.js, so with the mesh's
settings mounted at /config the container ran fine but reported unhealthy
forever. Mount the same file at /data/settings.js and point --settings there.
The record is prose wrapped at a hundred columns; matched line by line, the first live search for a
sentence of ADR 0025 found nothing. A line is matched together with the next, emphasis marks are
ignored, and a hit still names the line it starts on.
The manifest declared its secrets as secrets.secret.<name> and required a
"secret" provision, so the controller saw no own secrets and every accept
was refused (the influxdb defect of #180). The directories functions and
storage shared their ids with the containers of the same name, which the
host refuses as two resources with one identity. The containers are now
edge-functions and storage-api; the placed directories keep their ids and
paths.
A module keeping a checkout of a repository of decisions, designs and issues from the git seat,
current on every announced merge and on a timer, answering records_search / records_read /
records_list / records_status / records_sync at the commit it read (novox/hq ADR 0025, ADR 0153).
The repository is a setting; it names no mesh.
The received grants file names each pair secret by its HOST path. A sidecar
that mounts the placed grants directory at another path inside its
container sees mesh.json and not the secrets beside it — mosquitto's
provisioner reported ENOENT for a file that was there, and nodered's mqtt
step failed against a login that was never created. Mounted at its own
path now, as postgres does.
The tool runtime's own client, mesh serve, started by the mesh on the credential it sealed to the
machine (novox/hq ADR 0152, design 34): invokes every tool, listens from the machine only, holds no
state. Checked with module check before it was ever registered (hq issue 148).
jschoubben