A seat's protocol lives in the store (migration 0047; seeded additively), a served verb carries its
description and schema, holding a mesh seat requires serving its verbs, a node-scoped seat's tool
carries the node, and the control plane serves status, nodes, node, modules, seats, builds, plan,
assign, unassign, push, build and tools on its seat by running the same commands (novox/hq ADR 0132,
ADR 0154, design 33). A grant of * reaches a role's tools; seat:<seat>.<verb> grants one.
A module contributes a label; the mesh joins it with the node's domains and
the provider serves the result — and the module itself was never told.
Software that must know its own address (a login redirect, a canonical URL,
an issuer) had it written into the manifest as a literal: a domain in a
definition, wrong on every other machine (ADR 0112). Found converting
grafana's keycloak login for ace, where it forced GF_SERVER_ROOT_URL and
keycloak's issuer back into manifests.
The binding for a requirement a module contributes to now carries `name`
and `internal-name` (or `names` by local name for several contributions),
and `${bound:<requirement>:name}` / `:internal-name` (`:name-<local>`) fill
files from it. Both come from the one function the provider's received
file is composed by, so the proxy and the module cannot disagree about the
name. Absent when nothing was composed, so a file asking for a name on a
node with no public domain is refused, not rendered empty.
Also: `${bound:…}` could not name a requirement answered by a node-scoped
provider on the same machine — its binding file was written (from `here`)
but the placeholders only looked at the mesh's needs. Filled from the same
answer now.
"3478/udp" read as one token was not a port, so it passed through and the
runtime published it wherever it liked: on ace, unifi's STUN and discovery
landed on random machine ports while every TCP pin beside them held. The
protocol is split off, the number is assigned as for any short form, and
the suffix rides along on the outside.
invokes: a manifest word that becomes exactly the publish grant a person's account gets (ADR 0152),
derived by the same composition; refused at parse when it names no tool. module check <file|dir>...
runs what registration runs with no store, for a manifest in any repository (hq issue 148).
novox/hq ADR 0151 (issues 139, 157). <label>.<node>.internal is answered
by every resolver as 'anything under that node goes to that node', so
the node in a route's internal name must be the one whose proxy answers
it; composed under the consumer's own name it sent a client to a machine
with nothing listening whenever the proxy ran elsewhere. Composed under
the serving node now — the same machine wherever the proxy runs beside
the module, so nothing changes on a mesh with one hub.
A routed public name gets no .internal alias any more: the roster
publishes it as itself, once. The alias resolved and nothing served it.
`assign ace postgres` made the control plane's own store unresolvable:
postgres's manifest claims mesh-store, nobody was ever recorded as its
holder, and with two eligible assignments and nothing on record both
claimed and both were refused — novox's included. ADR 0110 says the
assignment holds, by a deliberate act; ADR 0131 gave the record its force
but left a seat nobody handed over held by whichever assignment happened
to be alone.
Before acting on an assignment the controller now writes the derived
answer down: every mesh-scoped seat the store knows, resolved to exactly
one holder with nothing on record, gets that holder recorded — the same
record `seat <name> --to <node>/<module>` makes by hand. The next
assignment able to hold the seat then stands beside the holder, eligible
and silent. A seat with two derived claimants is left for a person; a
seat on record is never rewritten; seats the store does not list stay
held by derivation as before. And the refusal, when it still happens,
names the handover that records the holder.
Verified: catalogue tests against the real catalogue; the cmd suite
against a store (the only failure, TestConvergingPreviewsThenChanges…,
fails identically on main).
novox/hq ADR 0148, step 3. Every container got the whole roster as
--add-host entries at creation and nothing re-read them (issues 109,
135); once the roster was in the digest so that could be caught, one
name moving anywhere replaced every container in the mesh (issue 151).
A container resolves through its machine's resolver, which the resolver
module tells the runtime about once per machine. A module's own hosts
entries stay exactly as declared.
Also brings the resolver tests up to the catalogue as it now is: the
runtime is reloaded (never restarted) and given live-restore, and the
resolver answers by address, not by interface (issue 110).
novox/hq 04-ISSUES/107. The controller already held a per-node lock while
it composed and recorded each send; the order existed and was thrown away
at the wire. Each send now takes the next number for its node, one
higher than the last, under that hold and before the body exists — so
the number is inside what the mesh signs, and a replayed older
declaration cannot borrow a newer one's.
Zero is not sent. A host reads absence as "no order claimed", which is
the shape of every declaration before this, so nothing that worked
before changes for a machine sent nothing since numbering existed.
One subtlety, and it is the one that would have read every machine as
behind for ever: the mesh decides a machine is behind by comparing the
digest of what it WOULD send against what it DID send, and a number
changes the bytes. The read-only comparison composes with the number the
machine was LAST sent, not a fresh one, so it is byte for byte what was
sent when nothing else changed.
Hosts went first and every machine runs one that understands the field.
novox/hq 04-ISSUES/161. A repeated flag is not a merged one. The Go
command takes the last -ldflags and drops the first, so passing the
toolchain's flags and then the system stamp as a second one produced a
binary that knew its system and had lost -s -w: 12.2MB against 8.5MB,
with its debug info intact.
My own comment said the linker "accepts and merges" them. It does not,
and I found out by reading the file the build produced rather than by
reading the comment again.
Linker flags are now the toolchain's own list, composed into one flag with
the stamp. A test refuses a compile line that carries -ldflags itself,
because that is what makes two.
novox/hq 04-ISSUES/161. The mesh compiled the host, published it,
delivered it, and the launcher started it — and it would have refused the
first declaration it was asked to apply, because it asks which system it
was built for before applying anything and the answer was empty.
The Makefile links that in. The mesh's toolchain deliberately takes
nothing from the module, so it linked in nothing.
The system is the stated exception, and ADR 0142 says why: the target is a
property of the artifact rather than of the recipe, because a compiled
binary is per system and a toolchain accepting it from the module would be
accepting a build instruction. So the toolchain names the variable it
fills and the artifact supplies the value.
Named in the toolchain rather than inferred, and empty for a language
whose output is not pinned to a system — which is every interpreted one,
and a manifest declaring a system for those is already refused.
novox/hq 04-ISSUES/142. The name a machine runs a binary by is not always
the name of the package that built it. The host's command is cmd/mesh-host
and every machine runs it as nox-mesh-host — the path it is installed at,
the name in its unit, and the name its launcher looks for inside a
delivered version.
So the first delivered host version landed as `mesh-host`, the host
reported "created /usr/lib/nox-mesh-host/versions/2681d936b949: 1 file(s)",
everything said success, and the launcher would never have seen it. Found
by listing the directory instead of believing the line.
An artifact may now say what its executable is called. Saying nothing
keeps what the compiler would have chosen — the package's name — so
nothing that worked before changes.
novox/hq 04-ISSUES/142, and the second of the two things ADR 0141's own
insight named: "a version cannot reach the path". A component is unpacked
into a directory named for its version so it can read its own version from
its path — and an archive named a fixed path in the manifest with nothing
interpolating the build into it, so nothing could ask for
.../versions/<version>/ and every machine took a hand-placed fallback.
A resource using an archive or a bundle may now say ${version} in any of
its values. No artifact name in the reference: the resource already says
which artifact it is for, and a second name is a second thing to keep in
step.
The version is the artifact's digest, short, and not the commit. Two
builds of one commit are meant to be the same bytes — every toolchain here
is -trimpath for that reason — so a content-addressed version means an
unchanged build resolves to the path it already had. A commit-named path
would move for an identical binary and recreate everything that reads it.
An image is refused one, with a reason: an image is not unpacked, so it
has no versioned place. Left alone it would reach a machine as literal
text and be created as a directory called ${version}.
novox/hq 04-ISSUES/142. A bundle is refused if it names what it is built
from, because a bundle is the module's own directory compiled whole and
naming a source would be describing its own build. That reason holds for
an interpreted language and cannot hold for a compiled one.
A repository written in Go carries several commands — the host and its
bootstrap live in one — and "the module's own directory" is then not a
package at all. So a compiled bundle may say which package, and says the
module root by saying nothing. The refusal stands for every interpreted
bundle, which is what it was written for.
Found by writing the host's manifest, which is the first bundle in a
compiled language this mesh has had.
novox/hq 04-ISSUES/142, and ADR 0141's own progressive insight naming
this as the first of two things missing: "nothing can compile it". The
toolchain list was a closed set of typescript and python, whose warning —
every language is another implementation of the contracts modules share —
does not attach to Go. Go is how the host, the control plane and the
builder are written, and none of them is a module in that sense: the host
is what APPLIES modules.
The toolchain names mesh-tools-go as its base rather than pinning an
upstream release here (ADR 0142, 0044): named and not pinned means the
mesh answers with the copy it holds, and moving compiler is a build
instead of an edit to this file.
Two things beyond the list also assumed one language, and both would have
failed after the entry was added:
sourcesFor turned every entrypoint into a `.ts` file. The extension is
the toolchain's now — one language's file extension written into the
code that serves every language is a wall the next one hits.
The output directory was the compiler's to create. tsc --outDir makes
one; go build -o writes into a directory and does not make it, failing
with a message about a path rather than about a build. Made here for
every toolchain, because which compilers are forgiving is not something
a reader should have to know.
And a toolchain now says what it is pointed at: a file list from the
module's entrypoints, or the one package the artifact is built `from`.
Pointing `go build` at a file list builds a program out of exactly those
files and ignores the rest of the package — a missing symbol rather than a
legible refusal.
Static and -trimpath: what a machine holds is a file, not a container, so
a binary needing a libc it did not bring is a delivery that works until a
machine differs; and a version comes from where a component sits rather
than from its linker, so two builds of one commit are the same bytes.
novox/hq 04-ISSUES/087. The version I shipped this morning said "N
machine(s) run an older host than another machine does" and worked it out
by comparing versions as strings. A host reports its version as a commit.
Commits have no order.
On the live mesh it named the three machines running the NEWER host as the
ones behind: `ced54d4` sorts above `04a27ca` and means nothing. An
arbitrary lexicographic result, presented as a fact, about the one thing
this was built to make trustworthy.
It now reports the split — which machines run which version — and claims
no ordering:
4 machine(s) do not all run the same host:
04a27ca g14, novox, shanks
ced54d4 ace
a host refuses a declaration carrying a field it does not know, whole
— so the mesh may send only what every one of these understands. Which
of them is newer is not readable from a commit; that needs a version
the host reports as ordered
More useful as well as more honest: the reader sees who is on which side
of the split, which is what decides whether a field can be sent.
A report that confidently says the opposite of the truth is worse than one
that says less — which is the subject of 04-ISSUES/145, arriving by my own
door within an hour of my closing it.
novox/hq 04-ISSUES/145. "N machine(s), all doing what they were told, all
heard from, running what the mesh would send them, and every module
current with its source" was true for eleven hours of a mesh in which no
module could reach another. An operator read it, and every routine check
they made afterwards — ports from outside, routed services, egress —
passed, because the broken path was module-to-module over the machine's
own name and nothing exercises that.
Every question the sentence answers is about the mesh and a machine
agreeing: applied what it was sent, matches what would be sent, built from
what the source has. None dials a provision, and the mesh composes every
one of those grants itself. So the sentence now says so, in the reader's
way, immediately below it.
This is not the check ADR 0146 describes and does not pretend to be. It
closes the distance between "the machines are as the mesh described them"
and "it works" by naming it, which is where the eleven hours went.
Also: printStatus is separated from the asking, so its exact words can be
read by a test with no store, bus or machine. Those words have been acted
on and been misleading twice — here, and a held module reading as a
machine doing what it was told (04-ISSUES/125) — which makes them the
thing worth holding still.
novox/hq 04-ISSUES/087. A host refuses a declaration carrying a field it
does not know, and refuses it WHOLE — deliberately, because that keeps a
half-understood declaration off a machine. It makes every new declaration
field a flag day: hosts first, then the controller. The mesh had no record
of which host any machine ran, so that order was kept by somebody
remembering it, and a machine that refused for this reason reported a
failure with nothing saying why.
The machine has reported its host version since ADR 0141. The
controller's own copy of the report did not have the field, so it was
unmarshalled into nothing and thrown away on arrival. It has it now,
records it, and shows it in `node show` — "not reported" rather than
blank, because a machine that has not said is not a machine running
nothing.
Status says which machines run an older host than another machine does,
and which is newest. Deliberately disagreement rather than staleness:
nothing delivers a host version yet (ADR 0141, accepted and not built), so
the mesh holds no canonical current version and cannot honestly say a
machine is behind THE host. What it can say is that the oldest host in the
mesh is what the mesh may send.
A machine that has reported nothing is left out rather than called
behind. Versions compare as strings, which suits the timestamps and
commits this mesh uses and is wrong for a scheme where "10" sorts before
"9" — said in the code, at the place that would have to learn.
novox/hq 04-ISSUES/125. A module assigned to a machine and never taken
runs none of what it declares. Status had no vocabulary for it: the
machine was heard from, current, and doing what it was told, so the mesh
printed "all doing what they were told" — which was true, and was acted
on, and every public name on the machine went dark.
Status now names each module a machine is holding rather than running,
per machine and with a count, read from what the MACHINE reported rather
than from the mesh's take-time listing — the machine is the only thing
that knows what it found. The JSON form carries the same rows, absent
rather than empty when nothing is held.
And a hold suppresses the all-well sentence, where being adopted does
not: adopted is a mode somebody chose, a module assigned and never taken
is a half-finished action with nothing left to finish it. The condition
is now a named function so the rule lives in one place and a test binds
to the real thing rather than a copy of it.
untakenModules raises a read it cannot make rather than answering "holding
nothing" from a failed query, which is the shape this whole issue is.
novox/hq 04-ISSUES/156. Issue 146 put the stream into a push consumer's
delivery subject. The server will not move that subject while a
subscriber is bound, and answers `consumer name already in use` — a
message about the name, for a conflict about the subject. A node is bound
to its declaration consumer the whole time it is up: that IS a node
listening. So every node consumer in a running mesh became one the
assertion could not bring to match, and the control plane crash-looped on
the assertion it makes before it serves. A fresh mesh showed nothing,
because nothing was bound.
Kept rather than deleted and re-made. Re-making moves the subject, and a
holder may not be allowed to subscribe to the new one yet: the wider
grant travels in the bus's user list, which this same control plane
composes and a machine applies minutes later. On the live mesh the nodes
are granted `_DELIVER.<node>` and not `_DELIVER.<node>.>`, so re-making
would have silenced every machine — worse than the collision it fixes,
and harder to undo.
Kept rather than fatal, which is what 146's change intended and did not
do. The bare subject still delivers, and collides only where one holder
has two consumers of one name. That is the controller's own pair, and the
controller is not bound to them while it asserts, so those do move.
Also: an existing consumer's deliver policy is carried across rather than
reasserted, because the server refuses to change it and where a consumer
starts is its history.
Two tests against a real server: a consumer with a subscriber bound keeps
its subject, is reported, and still delivers; one with nothing bound
moves, so 146's fix still applies where it matters.
Three gatherers walk every node and pass over one whose plan will not
compose, so that one broken set does not cost the rest. They read a
plain error to mean that, and so read a store that was briefly
unreachable as a machine running nothing.
On the roster of routed names that is not a degraded answer but a false
one: it states to every machine at once that another machine's names do
not exist. Because the roster is part of every container's identity, a
control node replaced every container it ran — its own store, the
registry, the edge, mail, the bus — on a six-minute cycle for hours. The
loop closed through the store this is read from: each pass restarted it,
the read failed, one name left the roster, and the roster changing is
every container changing.
planFor now marks the two failures that really are the node's own — its
set not composing, and a setting that reaches nothing — and the three
gatherers pass over those and only those. Every other failure is raised,
naming the machine and the read, because a mesh-wide refusal with
nothing named in it is the other way to lose an evening.
novox/hq 04-ISSUES/152, and 151 for why a changed roster is a changed
container.
novox/hq 04-ISSUES/146. A push consumer delivers onto an ordinary subject and
everything subscribed to it gets a copy. The controller holds a consumer called
'controller' on CONTROL and another called 'controller' on EVENTS, and both were
given _DELIVER.controller — so the one process, holding both subscriptions,
acted on every message twice.
Measured: one enrolment published, one message in the stream, one delivery, no
redelivery, and the controller enrolled the machine twice — the second minting a
credential that replaced the one the machine had just been handed, which is why
it then reconnected for ever as a user whose password the mesh had rotated. Every
report and every followed event doubled the same way, silently.
The stream goes in the subject because the pair is what identifies a consumer.
A subscriber's permission gains the same shape, keeping the bare name so an
existing consumer keeps working until the next assertion moves it.
It was broken and stayed broken: the Dockerfile's fallback base is a Go older
than go.mod asks for, so every hand build died at 'go mod download' with
'go.mod requires go >= 1.26.0'. The pipeline never saw it because the pipeline
passes the declared base in, so the cost fell entirely on whoever built the
image themselves and had to find the digest by hand (novox/hq 04-ISSUES/146).
Read from module.json rather than written here as well, so the two cannot
disagree, and refused outright if the manifest declares none.
novox/hq 04-ISSUES/146. The composed user list names an enrolment user for
every machine with a live token and nothing minted a credential for it, so the
composer left it out as a user with no password — and every enrolment since the
mesh moved to this bus was refused before the mesh heard of it. The comment
above the issuing code already said the account is created before the token is
handed over; now it is. Recorded rather than minted, because the token's secret
is the password.
And 'broker accounts', which composes the same list the declaration carries and
writes it to standard output. For genesis, where no declaration can reach the
machine running the bus because that machine is not yet a node. It says what it
composed; whoever is raising the machine places it. A control plane that wrote
the file itself would have to learn where the bus keeps its configuration and
how to make it reload, which is the module's knowledge.
novox/hq 04-ISSUES/146. The foundation made it by running openssl inside the
broker's image, which worked while the broker was one that carried it and
stopped the day the bus changed: the new one has a shell and no openssl, so
the step exited 127 and no mesh could be raised. No other image the bundle
names has it either, so there was nothing to substitute.
broker certificate --into <dir> writes the pair, --check is the step's verify.
Self-signed on purpose — a host pins this server's exact certificate (ADR
0004) and at genesis there is no authority to ask — and made once, because a
second certificate is one every host that pinned the first no longer believes.
The key is written before the certificate, so an interruption never leaves
something that looks finished.
novox/hq ADR 0147. ca-trust carries a script and a unit; the one thing
neither can state is where the authority is, because that is a fact about
the mesh. This checks the rendering — the script fetches from the bound
address and is executable, and the unit runs it both ways, install and
remove. The verification itself is the lab's.
Local is not a boundary this mesh draws. A service here is callable by everything
else here, whatever form either takes — a package with a unit, a binary, a
container. Whether a caller sits in a container was never meant to change the
answer, and the only reason it did was that this chain asked about addresses: a
caller on the machine carries the machine's address, a caller in one of its
containers carries a bridge address, and a rule naming the former silently refused
the latter.
One rule for every service here, replacing the line-per-port added an hour ago,
which only ever covered the ports somebody remembered to think about. The three
reaches are now three lines: on this machine, over the private network, from
anywhere.
The tests assert per chain body, because the forward chain carries the same line in
the same words and an assertion on the whole file passed with the input chain's copy
deleted — which is what ADR 0137's own tests say to do and this file was not doing.
A port declared from the mesh admitted the machines' own addresses on the private
network. A container reaching a port on the machine it runs on comes from a bridge,
matching none of them — and where the container runtime routes directly, that packet
is delivered to this machine rather than forwarded, so the forward chain's allowance
never saw it either.
ADR 0100 requires this to work: the store is reachable 'from a container on the node
itself'. It was, through a rule the predecessor left, which allowed the private
ranges wholesale. Converging the machine replaced that with the four overlay
addresses and closed it.
Measured, and it was an outage: every module reaching another by its machine's own
name timed out for eleven hours while the mesh reported the machine healthy. A web
application logged 'connection to server at novox.internal (10.10.0.1), port 6852
failed: timeout expired' throughout.
Asked for by the link it arrives on, for the reason the forward chain no longer
names an address: a range describes one machine and goes stale in silence. A port
open to everything needs no such line.
jschoubben
mesh-admin