Review of the fix for hq issue 104 found three faults in it. A file applied
on an enrolled node — the mesh's own last declaration included — is applied
as the bundle is, so its resources are recorded as the machine's own and
what the mesh declared reads as undeclared: the plan removed the foundation.
`apply FILE` is for a machine the mesh has not spoken to, and is now refused
saying so whenever declared.json exists. The plan looked at what is held
before what the declaration says is taken, so the one cutover ADR 0100 says
must be previewed read as a hold; it now decides in holdOnAdopted's order,
models a step run inside a held container, and a test holds the plan's
sequence to the apply's outcomes. Genesis wrote the mode on every run, so a
re-run after `converge` left the state saying adopted while the kept,
signed declaration said converged, and the reconcile loop refused every five
minutes with no delivery coming to end it: genesis now writes the mode only
when none is recorded, and where the state and the verified kept declaration
disagree, the kept declaration wins and the repair is said.
Also: a file lock beside the state, taken by the link service, the host's
own commands and the installer alike, so a `reconcile` run by hand no
longer races the loop's save — chosen over refusing while a named service is
active, which would miss a `mesh-host run` started by hand; `--json
--dry-run` emits {plan} like an apply emits {plan, report}; the README's
duplicate flag line; and the bundle refusal is about the digest, not a claim
the carried bytes can never match what genesis applied.
An operator ran `mesh-host reconcile` on an adopted control-node with twelve
modules assigned. It applied the bundle the host carries — the genesis
declaration, foundation only, converged: recreated the store, failed on the
broker's held port, wrote the converged base filter and started its service,
and stopped at the first failing action. The filter closed the machine for
forty-five minutes. The host reported the node adopted in every report, the
declaration said converged, and nothing compared the two; nothing was printed
before acting (hq issue 104).
The host now records the node's mode — from every declaration the mesh sends,
and at genesis from what the operator said — and refuses, at the point of
application, a declaration that says the other mode, naming both and the act
that changes it. Only a declaration the link delivers, signed, changes the
mode: that is how `converge` and `adopt` arrive, so the flip still works and
nothing else can do it. Genesis marks the bundle consumed, with the digest of
what it applied, so `reconcile` holds a node the mesh has spoken to against
what the mesh last said and never the bundle, and refuses the carried bytes
when they are not what genesis applied. A file is refused when it is not what
the mesh last said: a declaration carries no sequence and no issued-at, so the
host cannot tell older from newer, and says so. Both commands print what they
would change — a hold, a removal, an action named as one — before touching
anything, and --dry-run is that list and nothing more.
Registering a module is an overwrite. Recording the forge's port ran `module add`
every time, so a genesis re-run pointed at an older catalogue would replace the
manifest of a forge that is built and assigned — with a push a few lines later.
Registering is only here so a settings row has a module row to hang on, and that
row is already there on a mesh that knows the forge. So: ask first, and skip.
Two comments narrowed to what is true. What follows the node's setting is what
the mesh derives from a module's ports — its container mapping, its filter rule,
its opening and what it serves. The forge's own address in its runtime's
environment (hq 088) and its route contribution's port do not, and are already
wrong for any port the mesh assigned. And a settings layer is the module's, not
one resource's: a second mergeable file on the builder would be given `serves`
too.
novox/hq 04-ISSUES/085
Every foundation port given at genesis became a per-node setting of the module
that binds it, except the package registry's: that one was fixed by rewriting
the builder's manifest when the installer registered it. Registering the builder
again from the catalogue undid it, and the forge's own module, when it took the
bootstrap forge over, came up on the catalogue's port — which on a machine where
a predecessor holds 3000 points the builder at the predecessor's forge.
So the rewrite is gone, and the port is recorded twice as a setting, both from
the one input:
- the forge's module is registered at genesis — not assigned, nothing of it runs
— so the controller has something to hold `{"ports": {"3000": <given>}}`
against. Assigning the forge later raises it on the port this machine was
given, and its container, its filter rule, its opening, what it serves and
what consumers are told all read it from there.
- the builder is given `{"serves": {"port": <given>}}`, which merges into the
binding it carries in place of one nothing can resolve yet.
A genesis on the catalogue's port records nothing and registers nothing, so it
does exactly what it did before.
novox/hq 04-ISSUES/085, ADR 0100
The control plane's manifest existed twice: at the root of its repository, read
whenever the mesh rebuilds it from source, and as a copy in the catalogue, read by
genesis. Nothing kept them equal, and the first rebuild replaced the mesh's record
with the repository's shape while every later push was refused (novox/hq
04-ISSUES/072). The builder's one-shot result already carries the manifest it built,
artifact resolved to the image; step 3 keeps it and step 9 registers it, re-pinning
the built image's bare id to the reference the registry assigned. The catalogue is
still read for the registry's and the builder's manifests and for phase two.
A container on the machine dialling a port the machine publishes reaches it
through the runtime's proxy — input, not forward — and the builder could not
reach the broker. The derived ruleset opens the mesh's own ports in both
chains; the base one now does the same.
031: a window of unacknowledged declarations is drained to the newest; the
rest are set aside and reported as superseded. 035: a file resource may say
create-once — written when absent, kept untouched when present (ADR 0087).
054: the bundle installs nftables and loads a base ruleset before the store
and broker, in the table the filter module later replaces (ADR 0088).
The verify reads the marker with the shell's read, which fails at end of file
without a line ending; the action ran and its verify said no. And the applier
reports each action with its command line, two of which now carry the real
store and broker passwords — the installer masks the values it made in
everything it says.
From review: the store and broker passwords genesis makes were carried into
the controller through a world-readable file in /tmp, a bundle left at 0644 by
an earlier installer kept that mode while now holding them, a mesh raised by
the old installer would have been handed new passwords its servers do not have,
and the broker-admin action's marker did not depend on the value. Secrets now
stage in a 0700 directory owned by the controller's account; the bundle is
chmod'd; an existing store or broker volume with no credential file is refused
by name; the marker holds the password's fingerprint. Also: one install path
for the store, broker and vault, no error-string matching for the operator
key, and no unreachable fallback for the superuser.
The template raises the store with the password 'bootstrap' and the broker
with its image's default administrator, and the installer carried both into
the mesh as accepted secrets — permanent, and not secret (novox/hq issue 071).
Now the installer makes both credentials, once, at the paths the postgres and
lavinmq modules declare as their own secrets, rewrites the produced bundle to
use them (the store reads its password from a file; the broker's default
account is given the new password by an action before anything dials it), and
writes the bundle at 0600 since it now carries them.
Before the first secret is accepted it makes the operator's sealing key beside
the bundle and gives the mesh the public half, so everything minted from there
is sealed to it too (ADR 0085, amended). Phase three adopts the broker as the
lavinmq module beside the store and installs mesh-vault as a foundation module;
the run ends by writing the operator-sealed export beside the key.
Like the store, the amqp provision is plaintext 5672 (vhost-per-login), so a
consumer must reach it — bind 0.0.0.0 (firewall-gated to `mesh`, WireGuard-
encrypted on the wire) instead of loopback. amqps (5671) was already mesh-wide;
management (15672) stays loopback for the host-networked provisioner.
Issue 051 (WBS 3.2).
Claude-Session: https://claude.ai/code/session_01D6qtiYU3P9jk3pnAXyAFyx
Not "every module's database"; a module requests one via requires
postgres-database. The one server holds the controller's contexts and the
database of each module that asks for one.
Claude-Session: https://claude.ai/code/session_01D6qtiYU3P9jk3pnAXyAFyx
InstallStore turns the mesh-store the foundation raised at genesis into the
postgres module, adopted in place: it verifies the module's server names the
same container and the same image the foundation is running (fail-fast on a
drift, rather than tearing down the mesh's store), then registers, builds the
provisioner, and carries the superuser in via secret accept — the mesh cannot
invent a credential that already made the databases (mirroring the control
plane's store-connection delivery, control.go). pinImage generalised to any
module for reuse.
Issue 051 (WBS 3.1). One server holds the controller's contexts and every
module's database.
Claude-Session: https://claude.ai/code/session_01D6qtiYU3P9jk3pnAXyAFyx
One name per thing, per the HQ glossary: the module/container/image/binary/repo
becomes mesh-controller, the seat the-controller, and the store+broker pair the
foundation (embedded base bundles, default template and example lock renamed with
their go:embed directives). No behaviour change — a pure vocabulary rename.
Claude-Session: https://claude.ai/code/session_01D6qtiYU3P9jk3pnAXyAFyx
Fixes found raising the package registry end-to-end in the lab: seed gitea's DB
with plain psql statements (no \gexec, no $$ DO-blocks that clash with the
shell); run gitea on the host network so it reaches the substrate store and
answers where the builder looks; set gitea ROOT_URL to the machine's loopback so
npm's stored credential matches the tarball host; keep the pivot's passwords so a
re-run is the same run; create the admin without re-enabling must-change-password.
Claude-Session: https://claude.ai/code/session_01D6qtiYU3P9jk3pnAXyAFyx
The base (mesh-tools) resolves the SDK by version from the mesh's package
registry rather than cloning it from a git URL (hq ADR 0076, issue 053), so the
registry has to answer and the SDK has to be in it before the base build runs.
New steps, before base: seed gitea's database in the substrate store, raise
gitea's server on it, create the admin/org/team and the builder's account, seal
the builder its registry credential, and publish the SDK on a public base. gitea
is adopted as an ordinary module after the base, so its provisioner image can be
built. A minimal Go gitea admin client stands in until that module exists.
Claude-Session: https://claude.ai/code/session_01D6qtiYU3P9jk3pnAXyAFyx
Twelve steps made a mesh that RUNS and then said "what remains is somebody
else's". The seven things that turn it into a mesh that WORKS — the shared base,
a database provider, the catalogue, the private network, the packet filter — were
typed afterwards, which is how they went missing for weeks without anything
complaining.
Six more steps now: base, store, catalogue, network, filter, extras. Everything
in them is module add, build, assign and push — the same verbs a person types,
through the same commands, so the installer and an operator remain one act.
Where a human must choose, the installer asks. A choice resolves in the order a
person expects: the flag wins; a lone option answers itself ALOUD, because "it
chose for me" and "there was nothing to choose" read identically afterwards
unless one speaks; a terminal is asked; a default fills in; and a required
choice nothing answered refuses naming its flag — a guessed packet filter is a
machine somebody else configured. The filter is required, so the question is
which, not whether. A run without a terminal (the lab, --json) is never left
waiting on a prompt nobody will answer.
Placement is part of the network step, not a separate act — a lesson paid for:
the module installed, the names file was written with no names in it, and
everything reported success because nobody had said where the machine IS. The
hub endpoint derives from the broker address when unsaid: the host other
machines dial is one fact, not two that drift.
Extras fail the run rather than soft-fail: somebody asked for them by name, and
a mesh reporting success minus one thing is reporting the wrong thing.
Claude-Session: https://claude.ai/code/session_01D6qtiYU3P9jk3pnAXyAFyx
The module that provides artifact-store runs Distribution, the OCI reference
implementation. It was called registry, which named neither the software nor the
provision.
Claude-Session: https://claude.ai/code/session_01D6qtiYU3P9jk3pnAXyAFyx
Found by being asked whether processes and containers handle environment the
same way. They do not, and the difference is not cosmetic.
Docker passes --env through literally. A unit file reads three things out of a
value that nothing else does, and a module's environment routinely contains all
three because a generated password is arbitrary bytes:
- % begins a specifier. %H is the hostname. A password containing one is
silently replaced, and it fails later as an authentication error nobody can
explain by reading the declaration.
- whitespace separates assignments. Unquoted, K=a b sets K to "a" and reads
"b" as another assignment.
- a newline ends the line, and what follows is read as a unit DIRECTIVE.
The first two are escaped: quoted, with quotes and backslashes escaped and
percent doubled. The third cannot be — a unit's environment has no way to carry
a line break — so it is refused in validation, near whoever wrote it. Without
that, an environment value could write ExecStart= and have the machine run
something nobody declared.
Ordinary awkward values stay accepted, because refusing those too would leave a
module unable to hold a generated password.
Claude-Session: https://claude.ai/code/session_01D6qtiYU3P9jk3pnAXyAFyx
The first cut of this added a `daemon` for the long-running case alone. That
would have meant a new vocabulary entry for each of the others — a scheduled
task, a run-once migration, a health check — when they are one thing run at
different cadences. That is a field, not four entries in a vocabulary where every
entry widens what a compromised control plane can express.
So it mirrors a container exactly, because it IS a container's twin: the same
intent, hosted by the machine's own supervisor instead of a runtime. Stays up,
runs once, or runs on a schedule.
Tools, hooks and event consumers are not further modes. They are loaded by a tool
host, which is itself a process that stays up — so the generic case already
covers them, which is the test of whether it is generic.
A scheduled process gets a timer and a unit that finishes; a long-running one
gets a unit that is restarted when it exits. Getting that wrong either way is a
second copy running continuously between fires, or a schedule that never fires.
The modes are exclusive and validation says so near the author: something that
runs once does not run on a schedule, and something not running between fires
cannot be restarted when a file changes.
A missed fire happens when the machine comes back rather than being skipped,
which is the difference between a machine that was down and a schedule that
quietly stopped.
Claude-Session: https://claude.ai/code/session_01D6qtiYU3P9jk3pnAXyAFyx
The mechanism was leaking into every module. Code of one's own meant a container
and therefore an image; a script meant a service and a unit somebody else had to
install. One intent — run this and keep it running — expressed two unrelated
ways, with the hosting chosen before anything could be declared.
A daemon names a bundle and a command. The host fetches it, refuses it unless it
hashes to what was declared, unpacks it where the mesh keeps such things, writes
the unit and puts it in the state asked for. The unit is the mesh's, generated
whole and saying so, because an edit that survives until the next declaration and
then vanishes is worse than one that is refused.
Its identity is the bytes AND how it is run: two daemons from one bundle
differing only in their command are different daemons, and tracking the digest
alone would call the second unchanged and leave the first running. The unit is
rendered deterministically for the same reason — environment from a map would be
written in Go's iteration order, so every apply would see a different unit and
restart an unchanged daemon for ever.
restart-on is honoured as a service's is: a running process does not re-read its
configuration, so replacing a file and finding the daemon already up leaves the
machine behaving as before while every check passes.
A full-host shape, not a portable one: it needs a process supervisor to install
into. It does NOT need a container runtime, which is the point.
Two guards caught this properly and both were updated deliberately rather than
silenced: the vocabulary count, which exists because every addition widens what a
compromised control plane can express, and the shape test that catches a kind the
language has and a host cannot apply — added after `network` did exactly that.
Claude-Session: https://claude.ai/code/session_01D6qtiYU3P9jk3pnAXyAFyx
Two faults that both reported success while being wrong, found while proving
the firewall module actually delivers.
A unit whose job is to apply something and exit — load a rule set, set a
sysctl — is inactive the instant it succeeds. Reading that as stopped made it
permanently unsatisfiable: the host started it, it worked, the host read back
stopped and reported the machine as not doing what it was told, on every apply,
for ever, with the rules correctly in place the whole time. That is what the
firewall has been doing on every machine it was assigned to, and why the
four-machine bed was red.
And a container took its identity from its own fields, not from the files it
reads. A file written in an earlier apply — or before the container declared it
as a dependency — left a process holding a credential the mesh had already
replaced, with everything reporting success (novox/hq 04-ISSUES/045). What a
container reads is now part of what it is, so the comparison is a standing one
rather than a tripwire that fires during one apply and never again.
It defaulted to mesh-host.service. What packaging/ ships is nox-mesh-host.service,
so on any machine with the packaged unit installed the installer looked for
something absent and told the operator a real machine needs it installed — when
it was installed, under the name the installer was not using.
Only the lab missed it, because the lab passes --host-in-background and never
names a unit at all.
Genesis ended with a mesh that runs and cannot make anything: every module in
the catalogue names artifacts and nothing had built them, so the first thing
anybody had to do was install a builder by hand.
The installer already carries one — it is what built the control plane — so
this is the same two acts the control plane goes through, in the same order:
publish it, so the mesh names it by a digest its own registry assigned rather
than a local identity nothing else can fetch, then install it as an ordinary
module pinned to that. And then the part only it needs, a broker account, issued
before the push so it arrives with the declaration rather than after it.
Verified on a bare machine: the install ends with a builder running, and that
mesh then built the shared base images and a module on top of them with nobody
helping it.
The carried image is the builder now. The publish step still pushed it, so the
registry got a builder under the control plane's name and the mesh installed it
as the control plane — which presented as a control plane that started, printed
a builder's usage, exited cleanly, and did it again. Caught by the lab on the
first genesis run, at the step that waits for it to answer.
jschoubben