novox/hq 04-ISSUES/107. A declaration's only identity was the digest of
its bytes: a host could say "not the last" and could not say "older". On
the link, nothing refused an older one at all, and the drain picked the
last to arrive — wrong exactly when it mattered, a backlog drained out of
order or a broker that split a burst.
A declaration may now carry a sequence, one higher per send. A host
refuses one lower than what it kept, whole, and says why. The drain keeps
the highest sequence in a batch rather than the last to arrive.
Only when both sides claim an order. Absent reads as zero — "no ordering
claimed", not "first" — so a controller that sends none is still
understood and a host that kept one before it understood them compares
nothing. That is what lets hosts go first and the controller follow, which
is the order 087 says a new field needs.
novox/hq 04-ISSUES/163. The host asks after every apply whether a newer
host is delivered than the one running, and asked with the link-time
version stamp — which every delivered host carries as "development
build", because the version comes from where the binary sits now (0142).
So a delivered host never matched the newest delivered version, stood
aside on every push for ever, and because standing aside cancels the
report, the mesh never heard from it again.
Measured on two machines: each push produced "host <v> is delivered;
standing aside" for the version already running, then "applied, and could
not tell the mesh: reporting: context canceled". A machine restarting its
host on every push and reporting nothing, reading as healthy.
Asked with the running version now. Half of 0142 was applied to the
report and the known-good record and not here; this is the other half.
novox/hq ADR 0142, which decided this and was not implemented: "It is
unpacked into a directory named for its version, so it can read its own
version from its path. The stamp goes, and with it the need for a build to
know what it will be called."
The mesh's toolchain stamps no version, on purpose, so a delivered host
called itself "development build" and the mesh could not tell which host
any machine ran — which is the whole of what 087 added. A delivered host
lives at <libexec>/versions/<version>/<binary>, and that directory is the
answer.
A host placed by hand keeps its stamp, which is the honest answer for one
the mesh did not deliver, and is every machine until a delivery reaches
it. A binary sitting anywhere else is not read as a version at all.
The decision is split from the reading so a test can ask about a path
without being that binary.
novox/hq ADR 0141 and 04-ISSUES/142. A version was being delivered to a
machine and nothing started it: the launcher on these machines predates
the versions mechanism and runs the fixed binary path, so the delivery was
correct and inert.
Delivered as a FILE resource, not as part of an archive, and the
difference is the whole reason this is safe. A file is written atomically —
temp file in the same directory, then rename — so the running launcher
keeps the inode it was started from and the next start picks up the new
one. An archive writes in place with truncate, which would cut the file a
running shell is reading halfway through.
The manifest therefore carries a second copy of the script, and a test
refuses any difference between it and packaging/nox-mesh-host-launch.
Proven by drifting one and watching it fail. Two copies of a script is a
bad thing to accept, and the alternative was writing over a running
supervisor.
Together the two resources complete the loop: the version lands, the
running host stands aside because it sees one delivered, and the launcher
that starts next is the one that looks in versions/ and picks the newest
by arrival.
nox-mesh-host, not mesh-host. The command directory is cmd/mesh-host and
the launcher looks inside a delivered version for nox-mesh-host — the name
this is installed at and the name in its unit. The first delivery landed
the package's name, reported success, and would have been invisible.
novox/hq ADR 0141 and 04-ISSUES/142. The host half of the delivery has
been built and tested since 0141 and has never had a version to work on:
versions side by side, the newest runs, the running one stands aside
between reconciles, rollback picks a directory. This is the declaration
that gives it one.
One archive, unpacked to /usr/lib/nox-mesh-host/versions/${version}. The
version resolves to the artifact's digest, so an unchanged build lands at
the path it already had and re-composing a declaration moves nothing.
Not circular: the host applying this is a different version from the one
being written, and neither writes over the other — the kernel refuses to
truncate a running executable, which is the reason the path carries the
version rather than a link pointing at "current".
**The launcher is deliberately not delivered here.** The one on these
machines predates the versions mechanism and runs the fixed binary path,
so a delivered version is inert until it is replaced — and replacing it
from the mesh means writing over a running shell script, which sh reads
incrementally. That wants a designed swap rather than a file resource, and
it is the last piece rather than this one.
novox/hq 04-ISSUES/142 and ADR 0142. Nothing delivered the host because
nothing could compile it, and nothing could compile it partly because the
host was not a thing the mesh builds at all — it had no manifest.
One bundle in Go, for arch, built from cmd/mesh-host. A system is
required for a compiled artifact because a binary is pinned at link time
so a host refuses to touch a machine it was not built for (ADR 0005), and
`arch` is what all four of this mesh's machines report themselves to be.
Another system is another artifact and another build, which is what ADR
0142 means by one per target.
No resources yet. What places a version into a directory named for it
needs an archive resource whose path carries the version, and nothing
interpolates one — the second half of 0141's insight, and the next piece.
novox/hq 04-ISSUES/125. An adopted node keeps what it found until its
module is taken, which is correct and was recorded only in the node's own
state file. On the edge cut-over the mesh sent 346 resources, the journal
said it applied 330, and nothing anywhere said which sixteen or why —
reading it meant opening state.json by hand, and not reading it took every
public name on the machine down.
The line that reports the apply now carries it, grouped by module and
ordered by name, because the sentence an operator needs is "route-proxy is
assigned and not taken" and the module is the thing `take` acts on. An
apply that held nothing says nothing extra: a line that reports "0 held"
on every converged apply is a line that stops being read.
novox/hq 04-ISSUES/146. Each was right while the mesh ran on the previous
broker, and nothing has raised a foundation since it changed.
The bus's certificate is made by the program that needs it rather than by
openssl inside the broker's image — the bus's image is Alpine with a shell and
no openssl, so the step exited 127 and no mesh could be raised. Self-signed as
before and on purpose; --user 0:0 because the volume is root's and the control
plane's image runs as nobody.
Enrolment no longer opens a raw TLS connection to check the pin: NATS speaks
its own protocol and upgrades afterwards, so the handshake met a plaintext
greeting. The client that presents the token carries the same pinned config
and verifies inside its own handshake, so the secret still leaves only after
the certificate is checked. The raw dial stays as what its tests prove, and is
no longer a path anything takes.
And the token says which bus it is for. Empty meant 'whatever the mesh runs
today' while two buses existed and became a refusal the moment one did.
It now stops at the bus's user list, which is the genesis half of 146.
The serving path passed nil where the apply writes its detail. Nil is silence, so
everything the apply says — a file held, a container replaced, the found firewall
retired — was visible when a person ran the one-shot command and discarded in the
way the host actually runs, which is always.
Measured: after a machine was converged and its found firewall was not retired,
what the host decided was unrecoverable, because it had said it to nobody. That is
why issue 143 has candidates instead of a cause.
say already reaches stdout and the unit sends that to the journal, so this needed
no new mechanism — only for the argument to be passed. Both paths now reach the
apply through one named helper, so a reader asking where the apply's output goes
finds one answer.
The test asserts the log is never nil and cannot catch the fault it was written
for, which is wiring; that is proved by a deployed host whose journal carries the
detail.
A reconcile is otherwise silent, and the condition deciding when it speaks asked
only what an adopted node reports: what it holds, and the firewall it found. A
converged node has neither, so it could speak only in reply to a declaration —
and the mesh composes no declaration for a node that has not said which links
face outside (ADR 0140). A machine waiting for a push that was waiting for the
machine.
Measured, not predicted: after the control plane learnt to read the links, the
control node recorded its own and the three converged machines sat silent while
their filters were refused.
The condition is now a named predicate, because as an inline expression nothing
could test it — which is why the gap shipped.
The mesh composes no filter for a machine that has not said (ADR 0140), and a
converged machine only speaks unasked when its adoption fingerprint changes. The
links were not in that fingerprint, so a machine that had just learnt to say
could only speak when a declaration arrived — and a declaration cannot be
composed until it has spoken. A machine waiting for a push that is waiting for
the machine.
Found before it bit: every machine in this mesh is in exactly that state right
now, having just been given a host that reports the links to a control plane that
does not yet read them.
The supervision was already right: a clean exit means the host stood aside, and
the launcher's next turn runs what is on disk. Two things made it dead code —
nothing told the running host a successor was waiting, and the rollback resolved
its known-good version through pacman, which no machine here uses and which two
of three operating systems do not have.
Keeping a version rather than a path was the clue. Versions now live in
directories named for them:
- the launcher picks the newest delivered one every time round the loop, or the
one a rollback pinned, or the host placed by hand when nothing is delivered;
- the running host stands aside between reconciles, never inside one, by exiting
cleanly — and returns nil so the launcher does not count it as a crash;
- a completed reconcile retires what is older than the predecessor, keeping the
predecessor because that is what a rollback starts, and never the running one;
- rollback pins the predecessor instead of reinstalling a package: no package
manager, no cache anyone may clean, same script on every operating system;
- the report says which host version produced it, so 'behind' is answerable.
Newest is when it arrived, never how the name sorts: '1.10' orders before '1.9',
and ordering by name would start an older host and call it an upgrade.
novox/hq ADR 0141. The delivery half — a module carrying the next host — follows;
until then nothing delivers a version and every machine takes the fallback, which
is what it does today.
The filter blocks everything passing through the machine and then allows the
machine's own containers back by naming the address ranges they sit on — two
ranges fixed in the control plane and the rest typed after a flip had already
cut a workstation off. A range describes one machine and goes stale in silence.
Read the links carrying a default route instead, from /proc rather than by
asking a program, and report them on every apply. A machine with no route off
itself reports nothing, and the mesh composes no filter for it rather than
writing a rule around a link with no name.
novox/hq ADR 0140. The control plane does not read this yet.
A container resolves every machine and public name through the entries it is given when it is created,
and nothing re-reads them. The host compared everything about a container except those, so one whose
image and files never changed was left alone holding an overlay address five days out of date — it
restarted 2286 times against a database it could no longer find, and the mesh reported the machine as
doing what it was told (novox/hq 04-ISSUES/135, the same fault as 045 in the field left out).
Sorted, so the digest does not move for a reordering nobody made. The first apply after this recreates
every container that carries mesh names, once.
A mesh raised from nothing must be able to say what it applied and what a machine refused (novox/hq
ADR 0134), and the first user list is what permits it. Kept in step with the controller's own
composition by the test that reads this file.
novox.be is a module on this mesh. Reading a resource's owner to the first dot made its resources
belong to something called "novox", and a step's gate would then skip whatever else happened to start
that way — silently. A resource's own id never contains a dot, which is what makes the last one the
boundary; the mesh's derived step was changed to add a hyphen rather than a dot for the same reason
(mesh-controller #128).
A run-once container that does not complete stops the rest of that module's resources; everything
else on the machine is attempted, as every other shape already is (novox/hq ADR 0136). An action
still gates the machine — genesis is a row of them and they belong to no module. What was not
attempted is reported as skipped, because that and 'nothing to do' are different answers.
Without this, ADR 0135's derived preparation would let one module's unreachable database hold a
machine hostage — the fault 04-ISSUES/011 removed for everything else, and the reason the catalogue
migrates itself at start.
The controller is the way in for tool calls (novox/hq ADR 0095); the first user list must say so,
or a mesh raised from nothing refuses its own first ask. Kept in step with the controller's
composition by the test that reads this file.
The mesh runs on the seat's bus alone (novox/hq ADR 0131, design 28 task 5.5). The host's old
dialling and enrolment paths are deleted with the switch that chose between them; a membership or
a token naming another bus is refused before anything is sent, rather than dialled on a transport
that no longer exists.
The rescue path: an operator writes the membership file by hand on a machine no
bus can reach — rotated while it still held the old password — and restarts the
host. Same file, same check as the declaration path.
Two things the controller's own composition now derives and the installer's
carried list did not: the delivery subjects of the controller's consumers, and
JetStream enabled on the account — without which the first bound consumer is
refused. Found live on a mesh moved rather than raised; a fresh genesis would
have met both at first start.
jschoubben
mesh-admin