/user/repos lists what the token's user owns, which for the mesh's administrator is nothing — so
the forge module watched an empty list and never announced a merge. It reads the forge's whole
view through the search endpoint, every page.
The merge tool emitted at the instant it acted; a merge made in the forge's own
pages or over its API emitted nothing, and the mesh went on believing every module
current with its source (novox/hq 04-ISSUES/131). Merged pull requests are now
watched the way repositories are: what the forge holds, asked for on a tick,
announced once, with the merge commit and the clone URL a build needs. What has
been announced is kept beside the module's state, so a restart does not announce
the whole history again, and a first tick with no record announces nothing.
The mesh's broker certificate is the operator's, kept outside any module and
mounted read-only by whatever serves the bus; the controller declares that
access and now so does nats, the same way. A mount nothing declares is refused
at registration (ADR 0030), which is how this was found.
The module mounted a TLS directory nothing fills, so the server could not start
on a mesh that was not raised by the genesis template. The mesh already has a
broker certificate every machine pins by fingerprint and the controller trusts;
serving the new bus with it means no pin changes when a machine moves and there
is no second certificate to be wrong about. No ca_file: the directory has none,
and a pinning client checks the leaf and nothing else.
The recipe started FROM the upstream server's digest directly, and the build machine
refuses that: every base is declared under build.on and copied into the mesh's own
store before a build, so a build never reaches out to a registry the mesh does not
run (novox/hq ADR 0097). Found the first time the module was built on a real mesh.
Same digest, now declared as NATS_BASE and arriving as a build argument; the
Dockerfile says where it comes from and why the digest is the index's.
AMQP is not a provision (novox/hq ADR 0131, design 28 task 5.4). These were the
only three manifests that named it: the broker that provided it, a proof that a
grant worked end to end, and a forwarder reading mail off a queue. Removed, not
converted — a module that wants messaging wants the mesh's bus, reached through
the sdk and named by the mesh-broker seat, and either of the last two is re-done
against that if wanted, as a new module under the record.
The controller refuses a manifest naming amqp from its next release, so these
could not be re-registered anyway. Nothing else in the catalogue referenced them.
The claim comes back for the third and last time. The store's row says the bus seat
answers for `amqp`, lavinmq provides `amqp`, and with mesh-controller#89 the check
that judges a claim reads that row instead of a copy compiled into the build
machine. So this is accepted for the reason it should have been all along.
Restores the holder the controller composes its own bus address through, which is
what ends tonight's crash loop. nats takes the seat over when the cutover is done
deliberately, not because the seat emptied itself.
Putting the claim back was wrong on its own terms. `mesh-broker` delivers
`mesh-bus`, and a seat that delivers a provision may only be held by a module that
provides it — so the claim is refused at registration:
lavinmq claims mesh-broker, whose holder answers for "mesh-bus",
and lavinmq does not provide "mesh-bus" at mesh scope
Which means the merged claim does not restore the holder, it stops lavinmq being
built at all. Removed again.
The seat being empty is still the live fault, and it has only one valid answer: the
holder must provide `mesh-bus`, and the module that does is nats. Recorded against
the rollout, because it moves a step that was optional into the critical path.
Taking the claim off made the seat unheld, and the controller dereferences that
seat to find its own bus (to-be 26, "the one exception is the controller itself").
Unheld, the composed address fell back to a default port nothing serves, and the
control plane crash-looped: "cannot reach the broker named in MESH_BROKER_AMQP:
dial tcp 127.0.0.1:5672". The broker itself never stopped — it is healthy on the
port the mesh actually assigned it.
lavinmq becoming an ordinary provider is right, and it is still a provider of amqp
here. What was wrong is the order: the seat has to pass from one holder to the next,
and it cannot be empty in between, because the thing that reads it is the thing that
would have to fix it.
builder and route-proxy both build from the mesh-controller repository's context,
so its go.mod is theirs, and `nats.go v1.54.0` puts that at `go >= 1.26`. Pinned at
1.25.14 they cannot compile it: the build machine's own build failed with "go.mod
requires go >= 1.26.0 (running go 1.25.14)".
Each moves to the 1.26.8 digest of the flavour it already used — alpine for
builder, debian for route-proxy — so nothing changes but the compiler version.
Two lines of work renamed the same seats differently. The trunk named them for their
scope — node-scoped ones `node-*`, leaving `the-artifact-store`, `npm-package-registry`
and `git` as they were — and this branch had renamed ten of them to `mesh-*`. The trunk's
set is what the live controller loads and what the live seats were actually renamed to, so
a manifest claiming this branch's name is one the running mesh refuses. Three of them
needed reverting by hand: git had auto-merged this branch's names where the trunk had not
touched those lines, which is the quiet kind of merge result.
Event names are this branch's, because the trunk has not converted them and they are what
issue 127 was about.
Verdaccio goes with the trunk's removal of it. The template work on dnsmasq's roster fact
is the trunk's, sitting beside this branch's local event names in the same file — the one
hunk where both changes landed together.
75 manifests, all parsing, no claim outside the trunk's set and no event name left in the
old bus's form.
The jail.local [DEFAULT] gains ignoreip = 127.0.0.1/8 ::1 ${machine:mesh-range}
— localhost plus the mesh's own private range, named through the placeholder
rather than hardcoded (data is the mesh's, ADR 0112). Without it fail2ban could
ban the mesh's own nodes on 10.10.0.0/24; on novox that rule survived only in
memory from a now-deleted HAL file and would be lost on the next restart.
ADR 0121. The builder declared `built` as its own event, so every consumer depended
on which module happens to be the build machine today. It is the build-machine
role's event now: the builder declares none of its own, and the catalogue listens
for `mesh-build-machine.built` rather than `builder.built`.
Nothing changes about what reaches the catalogue. What changes is that it survives
the build machine being a different module, which is the whole reason the mesh has a
word for a role.
Every module named its events the way the old bus spelled a routing key —
`module.<module>.<verb>`. Design 29 says a module names an event locally and the
mesh works out where it lands, so all 37 were stale against a rule already
decided. On the new bus that derives into a namespace belonging to a module
called "module", so no cross-module subscription in the mesh matched anything:
nothing failed, nothing reacted (novox/hq 04-ISSUES/127).
36 manifests converted, and 43 files of module code with them. The code mattered
as much as the manifests: the runtime builds the subject from what `emit()` is
handed, so a converted manifest with unconverted code would have had the
permission and the subject disagree.
Three things the new check found on the way:
- `photos` emitted an event its manifest never declared, which the new bus refuses
outright. Declared.
- `showcase` waited for an event nothing emits, so its demo could never be
triggered — only `showcase` may publish under its own name. It emits both halves
now.
- `distribution` declared an event named after a different module. It emits
`image.pushed` under its own name. An event about a *role* belongs on the seat,
where the name outlives whoever holds it, but the sdk has no way to publish on a
seat yet, so that stays recorded rather than declared.
The audit logger's "everything" pattern is `**` rather than the old bus's `#`.
Claims renamed to match the controller's seat set: node-dns-resolver (dnsmasq),
node-intrusion-prevention (fail2ban), node-packet-filter (nftables),
node-resolver-config (resolv-conf, resolved-split-dns), node-uplink
(networkmanager, systemd-networkd, dhcpcd), mesh-build-machine (builder, +mesh
scope), mesh-catalog (mesh-catalog). showcase now declares its own seat and
claims it. verdaccio removed — the mesh keeps distribution as its registry and
gitea already serves npm, so a second npm registry is redundant.
The split the controller now makes, from this side. The module's own
configuration — ports, TLS, JetStream — is a declared file resource, because those
are properties of this container and change when its image does. `bus-users` names
where the mesh writes every account and permission, in the same directory, and the
module's configuration includes it.
**Both files in one directory because they have to be.** An absolute include path
is resolved relative to the including file's directory: nats-server given
`include /etc/nats/accounts.conf` from /etc/nats-server/nats.conf looks for
/etc/nats-server/etc/nats/accounts.conf and refuses to start. Verified against the
server, and recorded in the configuration itself where somebody moving a file will
read it.
**`verify: true` is gone, and it was refusing every connection in the mesh.** It
makes the server demand a client certificate; a host pins this server's exact
certificate and authenticates with the password the mesh minted, and presents none.
Found by building this image and connecting to it as a host would.
The entrypoint now waits for both files and watches the mesh's half: the module's
own does not change without a new declaration, and that recreates the container
anyway. Verified end to end against this image — the mesh's user list rewritten,
the module noticing and reloading the server itself with no signal from outside,
and the connection the mesh already had still working afterwards.
The node-zones fact was a path; the local=/address= syntax lived in the
control plane. It is dnsmasq's configuration language, so it moves into
dnsmasq's manifest as a template over the roster. The mesh renders it; it
reads none of it. Output is unchanged.
Lands with mesh-controller's ADR 0120 change — the two are one schema step.
Ten manifests claim mesh-* names now. What they PROVIDE is unchanged: gitea
still provides git and npm-package-registry, and a consumer requires the
interface, not the seat.
A workstation's job includes names that are neither a mesh machine nor
a routed name (novox/hq 122): shanks carries 13 Mediahuis entries in
/etc/hosts, and mesh-wireguard replaces /etc/hosts whole when taken —
so without this they vanish, and the take gates the node. Two homes,
neither the mesh's to own: conf-dir=/etc/dnsmasq.d/,*.conf (drop-in
directives, HAL's dnsmasq-app used exactly this) and
addn-hosts=/etc/hosts.local (plain host lines). The mesh creates and
rewrites neither; a machine with none loses nothing. The migration
moves such names here BEFORE the /etc/hosts take, closing the window.
It claims no seat: mesh-broker is the NATS server's (novox/hq ADR 0119).
The amqp interface stays exactly as it is — a backing service a module may
require, like a database.
The stated path was the adopted-data exception; with the take done and
the placement vocabulary live, the exception has no reason left. The
landing window renames the directory and recreates the container, since
a changed volume path does not do that by itself (hq 126).
The spec is the working system: HAL's 99-hal.conf, restated as
10-mesh.conf so lexical include order makes the mesh's answer the one
that wins while the predecessor's file is still on disk. Subsystem
stays the stock config's — first-set wins and it sits before the
Include. Port 22 from anywhere, said in listens with its reason: the
machines that need the door are exactly the ones not on the mesh yet,
and locking the operator out is the one failure a firewall must never
arrange.
The file is shared — the operator's insecure-registries for the mesh's
own store live there — and replacing it whole would break every pull
from that store the moment the module is taken (the 098 class, caught
in the pre-take diff this time). ADR 0102's verb is merge.
Step 1.1 and 1.2 of novox/hq ADR 0116. The server is a built artifact rather
than the upstream image directly, because it needs an entrypoint of its own:
the host can only recreate a container, and recreating the bus for every
permission change drops every connection and every in-flight ack. nats-server
reloads on SIGHUP by itself, so the config is mounted as a directory (not
digest-tracked, hq issue 103) and the entrypoint watches the one file.
Verified against the real server, not assumed: a user added to the config
connects, a revoked one is refused, both within one poll interval, with the
container's PID and restart count unchanged and "Reloaded: accounts" in its
log.
Two corrections found by checking rather than reading:
- the seat delivers nothing now (hq ADR 0117), and the controller's parser
refused the manifest until it did — "nats claims mesh-broker, whose holder
answers for amqp, and nats does not provide amqp"
- pinned to the multi-arch index digest; the first pin was the amd64
manifest, which builds here and fails on any other architecture
A consumer connecting by the binding's address meets a certificate for
mail.novox.be and refuses it — found live by the forwarder's cutover
proof, one send before production would have. The TLS name is mailu's
own fact (HOSTNAMES), so the binding carries it; consumers say
${bound:smtp:name} and verification holds.
The state directories say place "." — the assignment's own root — and
every bind, secret, own-secret, receives and grants path references it
as ${dir:state}/…; grants directories that are their own resources are
placed by id. gitea's two coincidence strings from the first pass
(${dir:data}base.json — resolving correctly by pure concatenation) are
spelled honestly now. What still says /var/lib is inside containers —
the software's contract — or under /var/lib/mesh, the mesh's own
plumbing, which the requirements unification absorbs next. Every
resolved path is byte-identical to what runs; landing this is a no-op
on the node, and the converter checks its own boundaries this time.
Each resolves to <root>/mailu/<id> — the maildir at
/var/lib/mailu/data-mail, certs at data-certs, and so on. Landing this
is a window, not an edit: seventeen renames on the node (the nested
data/ tree flattens to the ids), then the full stack recreated, because
a changed volume path does not recreate a container by itself (hq 126).
Ids are untouched on purpose — a renamed id orphans its held record,
and the mail spool is the wrong place to learn what a removal step does
with one.
The module is nextcloud; its tree was /var/lib/nextcloud-module — a
historic spelling nothing depends on. The root moves to
/var/lib/nextcloud (a rename on the node, done in this change's
window), and html drops its path: the mesh resolves it to
<root>/nextcloud/html. Landing this requires the window: rename the
tree, push, recreate the container — a changed volume path does not
recreate one by itself (hq 126).
The mesh resolves both to <root>/gitea/<id> — where the 5.8G forge and
its grant files already sit, so the roll-out its upgrade policy makes
of this build changes no byte of the spec. The module root and the
mesh's plumbing stay stated.
The mesh resolves it to <root>/mongodb/data — where the granted
databases already sit. The provider's own state, grants and the mesh's
plumbing stay stated.
Seven data directories drop their paths; the mesh resolves each to
<root>/only-office/<id>, which is exactly where the data already sits —
a textual no-op on this node, and the first module speaking ADR 0112's
vocabulary. The module root and the mesh's own state stay stated.
The /services paths were the adopted-node pattern doing its job: take
replaced containers over the predecessor's data without moving a byte
(gitea set it — 'its data never moved'). With every cutover done the
exception has no reason left, and the operator called it: a nox
module's world is /var/lib/<module>, data included. Six modules
repathed; mssql keeps its /services path deliberately — it is still
held, HAL-run, and moves at its own take. Both trees are one
filesystem, so each move is a rename.
The controller now accesses /var/lib/mesh-broker-tls (mesh-controller
#54) and the push refused whole: lavinmq declared the directory as an
owned resource, and shared data is the operator's, owned by no module
(ADR 0051). lavinmq only ever reads the certs — genesis laid them down
— so it declares a read access like the controller does, and the
directory belongs to nobody.
The env block carried the key twice — mailu-webmail from #79's address
sweep, and a stray =webmail further down that survived it. Last write
wins in an env file, so the front resolved a name that answers nowhere
on the mesh's network and 502'd every logged-in webmail request. Latent
since the cutover: the SSO redirect the checks watched never touches
the upstream; the operator's first real login did.
Take one (#90) died on two real edge bugs, both fixed and pinned by
tests in mesh-controller (#66: autocert 404s unknown tokens itself;
#67: the internal authority 403s every public name before the token
lookup). The challenge path verified end to end reaching mailu's own
nginx before this flip.
The letsencrypt flavor served certbot's April-expired state to live IMAPS
users within minutes: autocert's HTTPHandler answers 404 itself for
tokens it does not hold and never consults the fallback for challenge
paths, so mailu's own client cannot answer through the path-scoped
route. cert flavor (valid to Nov 27) until route-proxy's handler
actually falls through.
PR #82 set TLS_FLAVOR=cert as the honest interim while the predecessor's
proxy owned /.well-known/acme-challenge outright. route-proxy took port
80 today and its handler passes unknown tokens through to routed paths
by design — the one line #82 promised, made now. The copied cert (valid
to Nov 27) stays on disk untouched; mailu's own certbot takes over from
here.
The manifest predated the working deployment on three axes: it declared a
data directory the running portainer never used (taking it would have
started empty), pinned an image digest the machine has moved past (issue
099), and contributed no route while portainer.novox.be rides a traefik
container label today. Now: the predecessor's portainer_data path, the
running image's digest, 9090:9000 kept as the predecessor's machine port
with the route contribution naming it, and 9443 kept for the runtime
sidecar's own TLS conversation.
A bare '80' tried to bind the node's port 80 — the edge's — instead of
auto-allocating. 9070 is the predecessor's number and the one the route
contribution already names.
The app reads MONGO_DB (default 'invoicing') for every operation and
uses the URL only to connect — listCollections ran against a database
the granted user cannot see. Same fault and same fix as photos' MONGO_DB,
found by the API's own logs at take.
The mongo credential authenticates against its own database and the
database is the granted one (mesh_novox_invoice), not the contributed
name the provisioner ignores. Same for the store: the key is sealed to
the derived bucket (mesh-novox-invoice) — the data mirrors in during the
window, the ncloud/photos pattern. And the api gets the route
contribution it always needed: invoicing-api.novox.be is today a traefik
container label, invisible to every file survey, and it must be a grant
before the edge can ever flip.
The adopted node still runs the predecessor's mongo container, and it must
keep running: invoicing points at novox.be:27017 and is not migrating in
this window. A module container named 'mongo' would be held at assign and
would replace the predecessor at take, cutting invoicing off its database.
The mesh's server coexists instead — fresh data directory, its own name,
auto-allocated machine port — and the predecessor retires with its last
consumer.
The declared 0700 was applied at take and broke mail quietly: postfix's
master runs as root but pickup and smtpd drop to uid postfix, and a
spool root they cannot traverse is a maildrop they cannot scan and a
rewrite socket they cannot open — auth succeeded and MAIL FROM hung.
0755 root is exactly what postfix's own set-permissions makes of
/var/spool/postfix. Fixed live by chmod first; declared here so the
next push stops undoing it.
letsencrypt was the aspiration and cannot work yet, proven live: the
predecessor's own ACME machinery owns /.well-known/acme-challenge on
port 80 outright (unknown tokens get its 404) and its entrypoint
redirect owns every other path — the hand-authored passthrough never
matched anything, which is why mailu's certbot state had quietly
expired in April while the copied files carried the name. cert flavor
serves those files (valid to Nov 27). Mailu certifying itself becomes
possible the day route-proxy takes port 80, whose handler falls through
unknown tokens by design — that flip is one line here, made then.
The hand-written seed predates automx2's prio column, so every
config-v1.1.xml request 500'd against a table the seed had just made —
and the predecessor's own database had the same gap: client
autoconfiguration has been silently broken on the old stack for a long
time, behind a root page that answered 200. The live database gained
the column by ALTER; a fresh mesh now seeds it right.
An unpinned pip install took the latest automx2, whose schema grew a
column (server.prio) the module's own seeding SQL predates — a 500 on
every autoconfig request against a table the seed had just written.
2021.6 is what the proven image runs; the seed and the software agree
again. Regenerating the seed for a newer automx2 is its own change,
made deliberately, not by whatever pip resolved this week.
jschoubben