Review of the fix for hq issue 104 found three faults in it. A file applied
on an enrolled node — the mesh's own last declaration included — is applied
as the bundle is, so its resources are recorded as the machine's own and
what the mesh declared reads as undeclared: the plan removed the foundation.
`apply FILE` is for a machine the mesh has not spoken to, and is now refused
saying so whenever declared.json exists. The plan looked at what is held
before what the declaration says is taken, so the one cutover ADR 0100 says
must be previewed read as a hold; it now decides in holdOnAdopted's order,
models a step run inside a held container, and a test holds the plan's
sequence to the apply's outcomes. Genesis wrote the mode on every run, so a
re-run after `converge` left the state saying adopted while the kept,
signed declaration said converged, and the reconcile loop refused every five
minutes with no delivery coming to end it: genesis now writes the mode only
when none is recorded, and where the state and the verified kept declaration
disagree, the kept declaration wins and the repair is said.
Also: a file lock beside the state, taken by the link service, the host's
own commands and the installer alike, so a `reconcile` run by hand no
longer races the loop's save — chosen over refusing while a named service is
active, which would miss a `mesh-host run` started by hand; `--json
--dry-run` emits {plan} like an apply emits {plan, report}; the README's
duplicate flag line; and the bundle refusal is about the digest, not a claim
the carried bytes can never match what genesis applied.
An operator ran `mesh-host reconcile` on an adopted control-node with twelve
modules assigned. It applied the bundle the host carries — the genesis
declaration, foundation only, converged: recreated the store, failed on the
broker's held port, wrote the converged base filter and started its service,
and stopped at the first failing action. The filter closed the machine for
forty-five minutes. The host reported the node adopted in every report, the
declaration said converged, and nothing compared the two; nothing was printed
before acting (hq issue 104).
The host now records the node's mode — from every declaration the mesh sends,
and at genesis from what the operator said — and refuses, at the point of
application, a declaration that says the other mode, naming both and the act
that changes it. Only a declaration the link delivers, signed, changes the
mode: that is how `converge` and `adopt` arrive, so the flip still works and
nothing else can do it. Genesis marks the bundle consumed, with the digest of
what it applied, so `reconcile` holds a node the mesh has spoken to against
what the mesh last said and never the bundle, and refuses the carried bytes
when they are not what genesis applied. A file is refused when it is not what
the mesh last said: a declaration carries no sequence and no issued-at, so the
host cannot tell older from newer, and says so. Both commands print what they
would change — a hold, a removal, an action named as one — before touching
anything, and --dry-run is that list and nothing more.
One name per thing, per the HQ glossary: the module/container/image/binary/repo
becomes mesh-controller, the seat the-controller, and the store+broker pair the
foundation (embedded base bundles, default template and example lock renamed with
their go:embed directives). No behaviour change — a pure vocabulary rename.
Claude-Session: https://claude.ai/code/session_01D6qtiYU3P9jk3pnAXyAFyx
The enrolment request is a struct in each repository, and this node now
reports a third key — the one its secrets are sealed to. That wiring had
tests on each side and had never been run across the join, where a
renamed field fails silently: enrolment succeeds, the key is absent, and
the node looks joined until the first thing sealed to it cannot be
opened.
So this writes a real one — keys generated the way enrolment generates
them, not typed as literals — and the private half of the sealing key
beside it, so the other side can prove what it sealed is openable rather
than merely present.
The mirror of the declaration check that already runs the other way.
The host side of enrolment. It parses a token the control plane issued, dials
the broker, refuses anything but the pinned certificate, and generates an
Ed25519 keypair whose private half never leaves the machine.
Verified against a real LavinMQ serving a real certificate: the pin matched and
the node proceeded. Then against a second broker with a different certificate
on another port, which was refused -- with an error that says retrying will not
help, because it does not mean the network is down, it means the mesh was
substituted.
InsecureSkipVerify is set and that is the point rather than a weakening. At
bootstrap the broker is self-signed and reached at an address, so there is no
authority to trace and no name to match. Chain and hostname checks are replaced
with something stricter: this exact certificate or nothing, checked in
VerifyPeerCertificate, which runs before the handshake completes -- so nothing
is sent to the wrong broker. There is a test that counts the bytes an impostor
receives, and it is zero.
The token format is defined separately here and in the control plane, because
this binary requires nothing present and does not import it. They are held
together by a test on each side asserting the exact field names, so a rename
breaks both immediately rather than at enrolment on a real machine.
Two distinctions the identity file has to keep. A machine that never joined has
no identity, which is an ordinary state and not a fault. A machine whose
identity cannot be read is a different thing entirely, and must not take the
same path -- re-enrolling would discard the identity the mesh still believes and
need a person with a new token. Fault injection found the second case untested:
the corrupt-file test was passing on the parse check, so the read-error path had
nothing defending it. It does now.
An already-enrolled machine refuses to enrol again rather than quietly
acquiring a second identity.
What is not built is the link. Enrolment stops after verifying the broker and
generating the identity, having saved nothing, so it can be run again unchanged.
132 tests, plus 32 launcher and 9 rollback.
96 comments across the two repos named records that no longer exist. Each now
points at the consolidated record that holds its reasoning -- ADR 0034 (a test
defends a decision) is 0017, the eight host records are 0005, the four lab
records are 0016.
Worth noting for next time: these are references from outside HQ, so renumbering
there is not free. It cost 38 files here.
novox/hq ADR 0038: one behaviour, two sources of declaration. This is the source
that does not need a mesh — the first node's path.
The bundle is embedded in the binary rather than shipped beside it, because
"copy it onto a machine and run it is the whole installation" stops being true
the moment a second file has to arrive with it. `make host BUNDLE=...` builds a
host carrying one; `mesh-host reconcile` applies it; `mesh-host bundle` shows it.
A default build carries nothing and REFUSES to reconcile, saying why. A host
that applied nothing and reported success would look exactly like one that
raised a first node, and the difference would surface later as a mesh that never
came up with nothing to point at.
Proved on a sealed machine: no route out, no name resolution, one binary copied
on, and it configured itself from what it carried. Idempotent on the second run.
One bug found by running rather than reasoning, and it is a shape worth naming:
`mesh-host bundle` validated the carried bundle through a path that strips
comments, while `reconcile` handed the raw bytes to the parser. So the command
whose whole job is to check the bundle said yes, and the command that uses it
said no. Two paths to one artefact, disagreeing. There is one path now, and a
test asserts that what validates is what is applied.
What this does NOT prove is stated in the README rather than left implied: the
claim under stage 2 is that one host can raise the substrate alone, and the
substrate is four container services. There is no container type, because a
container needs an image and where images come from is open; what belongs in a
substrate is not known, because the closure for a one-node mesh is what research
011 and 012 exist to answer; and the machine used to test this cannot install a
container runtime through a sealed network.
The mechanism is finished. The claim is not, and shipping a host that claimed a
substrate it has never raised would be the fault this whole project is about.
65 tests.
A declaration is JSON, versioned, and an ordered list of resources with stable
identities (novox/hq ADR 0043). The vocabulary is directory, file and service,
and anything outside it — an unknown version, type or field — refuses the WHOLE
declaration. A host that skipped what it did not understand would apply most of
what it was sent and report success.
It converges rather than executes: applying twice changes nothing the second
time, and applying to a drifted machine returns it. A mode is maintained rather
than set, because a permission applied at creation is not a permission held —
this repository has paid for that once already.
It owns a footprint and only that. What it applied and is no longer declared is
removed; what it did not create is never touched. Removal runs FIRST, because a
resource leaving a declaration while another arrives at the same path is an
ordinary rename, and removing afterwards would delete the file just written.
The store arrives here rather than at stage 3, as ADR 0043 predicted: nothing
can be removed without knowing what was applied. It is written atomically,
refuses to start empty when it exists and cannot be read — believing it owns
nothing would leave everything behind forever — and is saved even when an apply
fails, because what was applied before the failure is on the machine either way.
Three faults found by running inside a raised machine rather than by reasoning:
A unit that DOES NOT EXIST reads as `inactive` from `systemctl is-active`,
exactly as a stopped one does. So declaring a unit stopped reported success for
a unit the host cannot manage at all — absence read as satisfaction, which is
04-ISSUES/007 wearing a different hat. LoadState separates them.
Removing an orphaned service whose unit has since been uninstalled failed the
whole apply, and a host holding such a record could then apply NOTHING, ever,
with no way out but editing its state by hand. Removal is now idempotent for the
same reason os.RemoveAll is.
And the flag parser was wrong in the same way twice: fixing `mesh-host inventory
--json` by taking the subcommand off the front left `mesh-host apply decl.json
--dry-run` broken identically, because the standard library stops at the first
non-flag argument wherever that argument is. Parsed in a loop now.
30 new tests, 55 in total.
Tier 0's first slice, per novox/hq 03-DESIGN/01-to-be/05-the-node-host.md. It
applies nothing, connects to nothing, listens on nothing. 2.9 MB, static, no
dynamic dependencies: copy it onto a machine and run it is the whole install,
which is the property ADR 0041 rests on.
A capability is detected, never assumed. Every detector runs something that only
succeeds if the thing FUNCTIONS — the daemon is asked for its version, the
package database is queried, the firewall is asked to list a ruleset, which
needs the privilege as well as the tool. 04-ISSUES/007 is the fault this
prevents: a client on disk with its daemon down looks exactly like a working
runtime, and a node assigned work on that basis fails when the work arrives.
Every verdict carries the reason and the method. A capability reported absent
with no reason is the same fault in a new place: something nobody can act on.
Two bugs found by running rather than reasoning, both silent:
systemctl is-system-running exits non-zero for every state except `running` —
including `degraded`, which means units failed and the init is emphatically
there. Reading the exit code reported NO service manager on a machine whose init
it was. That is 007 in the mirror, and both directions place work wrongly. A
verdict now reads what a tool says about itself, not only how it exited.
And `mesh-host inventory --json` printed text: the standard library stops
parsing at the first non-flag argument, so the flag sat unread and the command
exited 0 having ignored what was asked. The parser now takes the subcommand off
the front, and a stray or mistyped argument is refused rather than dropped.
Detection deliberately does NOT follow ADR 0008. That rule governs applying
state, where a failed step means the machine is not what was asked for. A failed
probe is a finding — "absent, because the probe failed" — and aborting would
replace one legible absence with total ignorance of the rest.
25 tests: structure and logic with a fake runner, and the same detectors against
this machine, because a test that fakes the system under detection asserts only
that the fake behaves as expected.
jschoubben