mesh/check being carried to 5dc3a8e17f97 on 998d8f737008 | from fabfdf04f78a
The journal of an unpack sat at <target>.unpacked.json, in a directory
the module's account may write: a forged one would make the root engine
remove or adopt any path it named. It now lives under
<statedir>/unpacked/<sha256 of the target>.json, mode 0600, names its
target, and is refused whole when an entry is not filepath.IsLocal or a
parent is not above the target. It goes when the archive is taken off,
whether its state record still said what it unpacked or not.
The launcher's stop grace for a host on trial rises from 20 s to 75 s and
the unit's TimeoutStopSec is 120 s (openrc the same), past the host's own
60 s wait for the apply in flight, so neither cuts a save off again.
Tests folded as rows into existing ones (the repository is over its unit
budget): the journal's rows, a forged entry and parent refused, the
journal gone with a record that lost its Unpacked; the test that passed
on main dropped; the stop test waits on its own line, not a sleep, and no
longer consumes the save it waits for.
Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
On novox the node-engine's state had lost the records of nats's two
archives, which only a temporary save file of 2026-10-24 still held, and
about 260 such files (~80 MB) lay beside it. The cause: an apply runs
detached from the engine's stop (applyContext, WithoutCancel), and
runLink returned, and the process ended, without waiting for it; a save
cut between its write and its rename left its temporary file and the
state before it, and whatever the apply did was never recorded.
Quick fix: remove the leftover files by hand and restore the records.
Thorough fix, chosen: the engine ends between applies whichever way it
ends (a stop, standing aside, a move to another bus): it waits up to 60 s
for the apply in flight, then lets no save start (store.Quiesce). Save is
one writer per process, flushes the directory after its rename, and says
loudly each record the state saved before held that this save drops and
nothing forgot (Forget, Release mark what is dropped on purpose, and no
longer write into an array an earlier copy reads). The engine's start
removes leftover temporary saves under the apply lock and says how many.
Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
A delivery and the five-minute reconcile were two paths that applied, ordered only by a lock, and
each order it allowed was met live (issues 257, 261, 267). Now both only enqueue: one worker takes
the newest declaration held when it starts, applies it once and makes one report, and reports leave
in the order they are made.
A declaration may carry the controller's lease epoch beside its sequence; one older than what this
node applied is refused before anything is touched, counted, logged and reported. A report carries
the declaration's epoch and sequence and the host's own report sequence, kept on disk so it goes on
increasing across restarts and self-updates. Without an epoch, today's behaviour stands.
The five-minute reconcile read the kept declaration and then waited for the link's apply; the
link keeps a declaration only once its apply ends, so the reconcile applied the older one over
it. A module assigned a moment earlier was given back, and the report named a declaration the
mesh no longer recorded as sent, which held a plan at its first machine.
Every table and chain that refuses traffic is reported with whose it is:
the mesh's, the found firewall's, the container runtime's own, a ban, or
other — the runtime's user chain is other, which is where both predecessors
kept their rules, in the legacy filter on one machine and invisible to the
mesh. Adoption's threshold does not move; a converged machine's report
grows by its filters and its found firewall's state.
Convergence is a state the host keeps: a found firewall enabled again is
retired again and said; a reconcile that finds it inactive records that it
was found so, never that the mesh did it; a step skipped after a failed
apply is said. A retirement the mesh began and did not finish is finished.
Fixtures are rulesets captured from three machines of the first mesh.
The serving path passed nil where the apply writes its detail. Nil is silence, so
everything the apply says — a file held, a container replaced, the found firewall
retired — was visible when a person ran the one-shot command and discarded in the
way the host actually runs, which is always.
Measured: after a machine was converged and its found firewall was not retired,
what the host decided was unrecoverable, because it had said it to nobody. That is
why issue 143 has candidates instead of a cause.
say already reaches stdout and the unit sends that to the journal, so this needed
no new mechanism — only for the argument to be passed. Both paths now reach the
apply through one named helper, so a reader asking where the apply's output goes
finds one answer.
The test asserts the log is never nil and cannot catch the fault it was written
for, which is wiring; that is proved by a deployed host whose journal carries the
detail.
A reconcile is otherwise silent, and the condition deciding when it speaks asked
only what an adopted node reports: what it holds, and the firewall it found. A
converged node has neither, so it could speak only in reply to a declaration —
and the mesh composes no declaration for a node that has not said which links
face outside (ADR 0140). A machine waiting for a push that was waiting for the
machine.
Measured, not predicted: after the control plane learnt to read the links, the
control node recorded its own and the three converged machines sat silent while
their filters were refused.
The condition is now a named predicate, because as an inline expression nothing
could test it — which is why the gap shipped.
The mesh composes no filter for a machine that has not said (ADR 0140), and a
converged machine only speaks unasked when its adoption fingerprint changes. The
links were not in that fingerprint, so a machine that had just learnt to say
could only speak when a declaration arrived — and a declaration cannot be
composed until it has spoken. A machine waiting for a push that is waiting for
the machine.
Found before it bit: every machine in this mesh is in exactly that state right
now, having just been given a host that reports the links to a control plane that
does not yet read them.
Review of the fix for hq issue 104 found three faults in it. A file applied
on an enrolled node — the mesh's own last declaration included — is applied
as the bundle is, so its resources are recorded as the machine's own and
what the mesh declared reads as undeclared: the plan removed the foundation.
`apply FILE` is for a machine the mesh has not spoken to, and is now refused
saying so whenever declared.json exists. The plan looked at what is held
before what the declaration says is taken, so the one cutover ADR 0100 says
must be previewed read as a hold; it now decides in holdOnAdopted's order,
models a step run inside a held container, and a test holds the plan's
sequence to the apply's outcomes. Genesis wrote the mode on every run, so a
re-run after `converge` left the state saying adopted while the kept,
signed declaration said converged, and the reconcile loop refused every five
minutes with no delivery coming to end it: genesis now writes the mode only
when none is recorded, and where the state and the verified kept declaration
disagree, the kept declaration wins and the repair is said.
Also: a file lock beside the state, taken by the link service, the host's
own commands and the installer alike, so a `reconcile` run by hand no
longer races the loop's save — chosen over refusing while a named service is
active, which would miss a `mesh-host run` started by hand; `--json
--dry-run` emits {plan} like an apply emits {plan, report}; the README's
duplicate flag line; and the bundle refusal is about the digest, not a claim
the carried bytes can never match what genesis applied.
An operator ran `mesh-host reconcile` on an adopted control-node with twelve
modules assigned. It applied the bundle the host carries — the genesis
declaration, foundation only, converged: recreated the store, failed on the
broker's held port, wrote the converged base filter and started its service,
and stopped at the first failing action. The filter closed the machine for
forty-five minutes. The host reported the node adopted in every report, the
declaration said converged, and nothing compared the two; nothing was printed
before acting (hq issue 104).
The host now records the node's mode — from every declaration the mesh sends,
and at genesis from what the operator said — and refuses, at the point of
application, a declaration that says the other mode, naming both and the act
that changes it. Only a declaration the link delivers, signed, changes the
mode: that is how `converge` and `adopt` arrive, so the flip still works and
nothing else can do it. Genesis marks the bundle consumed, with the digest of
what it applied, so `reconcile` holds a node the mesh has spoken to against
what the mesh last said and never the bundle, and refuses the carried bytes
when they are not what genesis applied. A file is refused when it is not what
the mesh last said: a declaration carries no sequence and no issued-at, so the
host cannot tell older from newer, and says so. Both commands print what they
would change — a hold, a removal, an action named as one — before touching
anything, and --dry-run is that list and nothing more.
A declaration is JSON, versioned, and an ordered list of resources with stable
identities (novox/hq ADR 0043). The vocabulary is directory, file and service,
and anything outside it — an unknown version, type or field — refuses the WHOLE
declaration. A host that skipped what it did not understand would apply most of
what it was sent and report success.
It converges rather than executes: applying twice changes nothing the second
time, and applying to a drifted machine returns it. A mode is maintained rather
than set, because a permission applied at creation is not a permission held —
this repository has paid for that once already.
It owns a footprint and only that. What it applied and is no longer declared is
removed; what it did not create is never touched. Removal runs FIRST, because a
resource leaving a declaration while another arrives at the same path is an
ordinary rename, and removing afterwards would delete the file just written.
The store arrives here rather than at stage 3, as ADR 0043 predicted: nothing
can be removed without knowing what was applied. It is written atomically,
refuses to start empty when it exists and cannot be read — believing it owns
nothing would leave everything behind forever — and is saved even when an apply
fails, because what was applied before the failure is on the machine either way.
Three faults found by running inside a raised machine rather than by reasoning:
A unit that DOES NOT EXIST reads as `inactive` from `systemctl is-active`,
exactly as a stopped one does. So declaring a unit stopped reported success for
a unit the host cannot manage at all — absence read as satisfaction, which is
04-ISSUES/007 wearing a different hat. LoadState separates them.
Removing an orphaned service whose unit has since been uninstalled failed the
whole apply, and a host holding such a record could then apply NOTHING, ever,
with no way out but editing its state by hand. Removal is now idempotent for the
same reason os.RemoveAll is.
And the flag parser was wrong in the same way twice: fixing `mesh-host inventory
--json` by taking the subcommand off the front left `mesh-host apply decl.json
--dry-run` broken identically, because the standard library stops at the first
non-flag argument wherever that argument is. Parsed in a loop now.
30 new tests, 55 in total.
Tier 0's first slice, per novox/hq 03-DESIGN/01-to-be/05-the-node-host.md. It
applies nothing, connects to nothing, listens on nothing. 2.9 MB, static, no
dynamic dependencies: copy it onto a machine and run it is the whole install,
which is the property ADR 0041 rests on.
A capability is detected, never assumed. Every detector runs something that only
succeeds if the thing FUNCTIONS — the daemon is asked for its version, the
package database is queried, the firewall is asked to list a ruleset, which
needs the privilege as well as the tool. 04-ISSUES/007 is the fault this
prevents: a client on disk with its daemon down looks exactly like a working
runtime, and a node assigned work on that basis fails when the work arrives.
Every verdict carries the reason and the method. A capability reported absent
with no reason is the same fault in a new place: something nobody can act on.
Two bugs found by running rather than reasoning, both silent:
systemctl is-system-running exits non-zero for every state except `running` —
including `degraded`, which means units failed and the init is emphatically
there. Reading the exit code reported NO service manager on a machine whose init
it was. That is 007 in the mirror, and both directions place work wrongly. A
verdict now reads what a tool says about itself, not only how it exited.
And `mesh-host inventory --json` printed text: the standard library stops
parsing at the first non-flag argument, so the flag sat unread and the command
exited 0 having ignored what was asked. The parser now takes the subcommand off
the front, and a stray or mistyped argument is refused rather than dropped.
Detection deliberately does NOT follow ADR 0008. That rule governs applying
state, where a failed step means the machine is not what was asked for. A failed
probe is a finding — "absent, because the probe failed" — and aborting would
replace one legible absence with total ignorance of the rest.
25 tests: structure and logic with a fake runner, and the same detectors against
this machine, because a test that fakes the system under detection asserts only
that the fake behaves as expected.
jschoubben
mesh-tools
jochen