mesh/check failed (13): ace: novox/mesh-catalog#612: modules/minio/object-store: a compiled program added …; and 12 more | 32c5df952cbf on e73aae3686a6
mesh/delivery-group group feat/registry-milestone rejected: composed together: the checker's verdict at the members' heads (3a038b9f8b4d): ace
The store's front is distribution's program, a bundle the mesh builds, so nothing serves it before the builder: the
bootstrap's /v2/ wait and its pushes now go to the loopback port the store's manifest publishes, as the publish contract
allows on the store's own node. The one-off credential and its file go: the runtime could not carry it on a push, and no
push needs it. The tags list is asked for the repository being published, not always the control plane's.
Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
waitIdle returns once every change sent to the loop is taken in and applied and no backoff wait runs, so a retry the
clock or a wrong settled would start is counted, with no real sleep.
Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
The owner is the account the manifest names for the directory that holds the file; the bootstrap no longer makes that
directory itself, and a run that reaches it first is refused naming what it waits on.
Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
The image, the seat and its holder, when the pull began, and the last retry with what triggered it ride in the failed
have's pull-wait (mesh-sdk delivery.Have.PullWait, asked of the lead in j1-sdk-requests.patch), from the agent through
the link's module fact to the have bucket; the agent's own read view of them goes with the verb. Rows in the agent's
pull test and the have bucket's test; each failed without the code.
Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
ADR 0627 §1, §4: mesh-sdk keeps no asks, asker, vault answers or compose-candidate checks, so the node-engine takes them
from mesh-contracts: operator-channel's contract and client (asks, warrants, the asker's client), mesh-vault's sign verb
and answer, and node-engine's own contract and client for compose-candidate's ask, answer, verdicts and bound. The
lead's ruling on pull-waits (no record gives the seat that verb): the verb and its handler go; the row that read the
waits now holds the seat refusing the verb, and failed before the change. The go.mod requires of mesh-contracts' modules
come with the group's pin.
Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
mesh-sdk's ComposeCandidateAnswer and ModuleComposed are now the generated nodeengine types (ADR 0613 §3, §5), so the
input sent for the checker's sandbox is decoded once into the object the contract carries, its numbers kept as written
(json.Number), and its size still counted from the bytes it was marshalled to.
Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
The verb's answer and the event's body are mesh-contracts' nodeengine types, not mesh-sdk's, and the seat's verb and event
names are the contract's. The two reasons of the event, started and returned, are the engine's own words beside it.
Requires mesh-contracts and the registry milestone's mesh-sdk (a go.work, uncommitted, until the group pins them).
Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
mesh/check being carried to e2380404a1fa on e73aae3686a6 | from 998d8f737008
nats on novox, 2026-11-04 11:07: after mesh-host #207 the module stayed held
although its want carried "Back up again", and the engine said nothing. The
engine's composition verb showed it running what its want names (taken from
its held have at the start), so when the answer was in a revision of the want
read after the hold, settleOne returned on "it runs what the want names"
before it looked at the hold: the answer was swallowed, silently. The start
path #207 fixed was right but never met; no test ran the live shape.
- settleOne: the hold is looked at before what runs. An answer (again or
skip) in a want revision after the one the hold took is acted on whatever
runs; with no answer the hold is kept and said with both revisions.
- baseline: the hold kept at the start is said (it was the silent branch).
- composition verb: the want as the engine holds it now carries self, backup,
words and its key's revision, beside the engine's attempt (held, answered
revision, want and composition revisions, transient) and whether the module
was looked at since the start, so which want the engine read is read, never
guessed.
Quick fix: answer once more by hand (a new want revision), which the guard
still swallows while what runs equals the want, so none exists. Thorough fix,
chosen: the order of the guards and the read tool, proved by rows of the live
shape (a self module of policy record, its held have naming the want's own
composition, the answer before the start, and again and skip arriving while
held); the two later rows fail with the old guard order.
Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
mesh/check being carried to 5dc3a8e17f97 on 998d8f737008 | from fabfdf04f78a
The journal of an unpack sat at <target>.unpacked.json, in a directory
the module's account may write: a forged one would make the root engine
remove or adopt any path it named. It now lives under
<statedir>/unpacked/<sha256 of the target>.json, mode 0600, names its
target, and is refused whole when an entry is not filepath.IsLocal or a
parent is not above the target. It goes when the archive is taken off,
whether its state record still said what it unpacked or not.
The launcher's stop grace for a host on trial rises from 20 s to 75 s and
the unit's TimeoutStopSec is 120 s (openrc the same), past the host's own
60 s wait for the apply in flight, so neither cuts a save off again.
Tests folded as rows into existing ones (the repository is over its unit
budget): the journal's rows, a forged entry and parent refused, the
journal gone with a record that lost its Unpacked; the test that passed
on main dropped; the stop test waits on its own line, not a sleep, and no
longer consumes the save it waits for.
Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
nats on novox, 2026-11-04: the operator answered "Back up again", the
engine restarted, and baseline() took the hold with AnsweredRev = the
want's current revision, so the answer in that very revision was never
acted on and compose refused "held for a failed backup".
Quick fix: answer once more (a new want revision). Thorough fix, chosen:
at baseline, a held module whose want carries again or skip is applied
(again backs it up first, skip goes on without one); a hold with no
answer stays held. The cost said in the code: an answer already acted on
before a stop, whose backup failed again, is tried once more per start.
Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
On novox the node-engine's state had lost the records of nats's two
archives, which only a temporary save file of 2026-10-24 still held, and
about 260 such files (~80 MB) lay beside it. The cause: an apply runs
detached from the engine's stop (applyContext, WithoutCancel), and
runLink returned, and the process ended, without waiting for it; a save
cut between its write and its rename left its temporary file and the
state before it, and whatever the apply did was never recorded.
Quick fix: remove the leftover files by hand and restore the records.
Thorough fix, chosen: the engine ends between applies whichever way it
ends (a stop, standing aside, a move to another bus): it waits up to 60 s
for the apply in flight, then lets no save start (store.Quiesce). Save is
one writer per process, flushes the directory after its rename, and says
loudly each record the state saved before held that this save drops and
nothing forgot (Forget, Release mark what is dropped on purpose, and no
longer write into an array an earlier copy reads). The engine's start
removes leftover temporary saves under the apply lock and says how many.
Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
mesh/check being carried to 966093d8d067 on fabfdf04f78a | from 4d47ff5ce743, mesh-catalog dd458cb155f5
The vault judged a module handed only when its assignment already carried a handing plan, and the node-engine followed
that word: a module with no plan was composed the old way, refused for a key only the provider's helper answers
(forge-account's ssh_alias), and so never got its plan. resolution.Handing is now the manifest, its build and the
machine alone; the vault's word is read only to say where the two disagree. A refusal of a file or an own secret the
provisioner's client should fill now says "handed module not handed" and why, never only the keys a binding lacks or a
secret to make.
Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig
mesh/check being carried to 8e325830891c on 4d47ff5ce743 | from 6019c6d514a7
The declaration named the account only beside its filled files (node.go), so a node whose every filled-file module
was held this pass named none: on ace from 2026-11-04 01:29 its build modules and ssh-client were held for their
forge binding, and the node-engine refused every write of the files they keep, and of every other module's. The
account is the node's, named whenever the rendering has one. A row holds restic, the provisioned mesh's one
filled-file module, with the client placed: the node's own still names the account; it named none before.
Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
mesh/check being carried to 054754482cf4 on 6019c6d514a7 | from 83e135f10187
The review removed the guard and the suite still passed. The guard is now provisionerHeld, a placed client held as
its stand-in, rather than any placed client, so one placed and not held that composes no account keeps none stale.
Rows in the held client's table: the client taken out of the node's placement while what runs names the account, read
from the running own composition and from the client's running process, names no account; a placed client whose
process names no user names none. With the guard removed the first three fail; with it reduced to "placed", the last.
Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
Reading ace's node composition gave four resources twice and no way to see that one revision named no provisioner
client's account. The verb now shows, for module node, the revision's provisioner account beside what runs names, its
adoption, how many identities, groups and libraries it names, and each module the composer's last pass held with why.
Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
Applied, a node's own composition naming no account refuses every filled write and hands no module, which on ace
went unsaid from 01:29 to 01:33 but for the refusals themselves. The pass now says it once, naming the account what
runs names. A placed client keeps its account by the previous commit; this says the case that is left, a client no
longer composed here.
Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
provisionerAccount reads the user of the client's process from this pass's plan, and a client held as its stub (or
carried as it runs) composes no process: on ace, 2026-11-04 01:29 to 01:33, the node's own composition named no
account, the node-engine refused every filled write, and no module was handed (postgres's consumers, restic, power).
A node now carries the account its client runs as (Node.RunningProvisioner: its own composition's account as it runs,
else the user of the client's running process), and a placed client that composes none this pass keeps it. A row holds
the client in the provisioned mesh, both ways: the account stays and restic keeps its filled files; both were lost
before.
Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
frombus skipped reading the build of every module in Scope.InOwn, while compose.compositions folds such a module only
at no src2: fingerprint: a want at src2: (nats on novox, its have waiting for a yes) composed it from the indexer's
unresolved manifest and built() refused it, "no build has published" its server artifact. compose.Folds is now the one
rule; frombus takes the build in, or keeps the module at its wanted build, exactly where it says the module is not
folded. A row composes spiegel, in InOwn and wanted at src2:, from its build's upstream image; it read the indexer's
manifest before.
Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
mesh/check checking 30b86ea602a8 on 83e135f10187 | novox/mesh-host#203 in group feat/l3-final
The stub's mark (Manifest.Stubbed) is no field of a manifest's JSON, so an input a node-engine sent the check's
sandbox lost it: node-tools, held by its manifest's verdict, read as a tool runner with no bundle and refused every
module of ace, g14 and shanks for "its build produced 0 bundle(s)". compose.Input now writes its held modules (catalogue
and each node's kept) beside its fields and marks them again on reading; an input from an engine before the mark is
read by its contract, every module Refused names held as its stub. The sandbox composer also says, in the module's
read, why the input's node-engine holds a module the case carries, so the check names the cause, not a rule about the
stub. One subtest round-trips an input holding a refused tool runner, in both shapes; it refused for 0 bundles before.
Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
mesh/check failed (2): unit: mesh-provisioner-client/cmd/mesh-provisioner-client.TestNatsANodeComingBackFil…; and 1 more | 32a99144f65e on 798d73eb11e9
recordOf writes catalogue.ManifestReads in NodeRecord.ManifestReads, once per build, so the assigner moves a want only
to an engine that reads its manifest. A row holds the engine's own manifest to the baseline's words, so its new build is
never held for a word only it reads.
Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
mesh/delivery-group group feat/l3b-bus rejected: composed together: the checker's verdict at the members' heads (a1cb9beeb97a): unit: modules/mesh-provisioner.…
mesh/check failed (4): unit: modules/mesh-provisioner.gofmt: not as gofmt writes them (run gofmt -w): modul…; and 3 more | a24a1d72ad45 on 798d73eb11e9
The second sandbox fault of the same kind on mesh-host #200 (postgres's secret
reader handed, the module not): the engine stripped the whole catalogue from
the input it answers, so the sandbox composed the node's own modules from the
indexer's source manifests while the bus reader composes them taken in from
their builds (artifacts, templates archive texts, what is handed). Both sides
now go through composer.SentInput: the node's own taken-in manifests and kept
builds stay in the input, and the sandbox composes them over the cases'. An
input past what one bus answer carries is refused in words. A conformance row
composes one node through the bus reader and through the sandbox and requires
the same declaration.
Issue 1126.
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
mesh/check failed: novox: not composed on novox: postgres: its own secret reader is handed to the provisioner and the m… | 61a3bd996dd4 on 27a956448c0a
A check that composes with a candidate's own composer (any mesh-host change)
gave it the indexer's manifests alone, and the input carries no catalogue, so
mesh-wireguard (shipped with mesh-sdk, no repository) was missing: "nothing
provides mesh-wireguard / mesh-addressing" on novox for mesh-host #200. Each case
now lays the provided module's manifest over its shelf, as the bus reader does.
Issue 1126.
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
mesh/check failed: novox: not composed on novox: these assignments cannot be applied: | b8158affa7af on 27a956448c0a
An endpoint's subscription hands its asks over one at a time: two checks asking
one node at once queued, and the second answered at about 100 s, past the
checker's bound and the bus's one-minute reply window. compose-candidate is now
answered in a goroutine of its own, at most four at once, one more refused in
words. The new cases are rows of the existing tests (test budget).
Issue 1126, review round 1.
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
Off the control-node an ask of compose-candidate ran into its 60 s bound: each of
its two compositions read every manifest in the mesh one key at a time over the
bus, on scratch copies of the shadow that kept nothing, so each ask read them all
twice again (about 35 s per composition on ace, g14 and shanks; 2 s on novox).
The answer at 60 s then came after the bus's one-minute reply window
(allow_responses ttl), so the bus refused it and the checker heard nothing.
Now the manifests are read eight at a time into the shadow before either
composition, a scratch copy keeps what never changes (manifest, archive, done
build) in the shadow too, the candidate is laid over a copy taken after the
composition without it, and the engine answers within 50 s: unready, naming what
it waits on, when not done. Every answer is said in one journal line with its
reads and its longest wait.
Issue 1126.
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
The checker asks by the composition key, <group>.<heads hash> (ADR 0593 §7), but the
indexer keys a candidate by its group alone (delivery.CandidatesKey). The node-engine
read the ask's key as the bucket key, so every check's composition was refused on
every node with "the indexer holds no candidate" (mesh-catalog #599, 2026-11-02).
It now parses the ask, reads CandidatesKey(group), and answers unready when the
group's candidate has moved to other heads. Both tests' fixtures had captured the
wrong key; they now key by group, with a slash in the group name.
Quick fix: this lookup. Thorough fix (a shared SDK reader that takes a composition
key and returns the candidate at those heads) not taken: one reader, one repository.
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
mesh/check failed: integration: meshbuildgo/cmd/mesh-build-go.TestAComposeAskIsBuiltOnceAndAnsweredOnTheBus: NOT CHECKED: the node-engine's composer c…
The test read mesh-controller/module.json beside the check, which no longer exists after the
controller's removal (issue 1126, ADR 0593 §11), so it would fail every check of the node-engine.
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
mesh/check failed: integration: meshbuildgo/cmd/mesh-build-go.TestAComposeAskIsBuiltOnceAndAnsweredOnTheBus: NOT CHECKED: the node-engine's composer c…
ADR 0593 §7: the environment's rendering left mesh-sdk for the node-engine,
so its two integration tests (the POSIX environment sourced twice, and
the service manager reading the systemd rendering) move with it.
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
ADR 0593 §7: the composition is the node-engine's, so its tests hold
what the indexer's judge tests and route-proxy's tests proved by
composing: a bundle's words per machine, a Go bundle's delivery, the
credential form in a handed plan, settings refusals, placeholder and
vault refusals, the session's files, a field set aside composed whole,
route-proxy's manifest composing for a machine, a route endpoint's
machine port in six cases and the plan's route entry end to end. Two
table tests, one subtest per item; route-proxy's manifest is captured
beside the catalogue.
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
ADR 0593 §7: the composer with its refusals leaves mesh-sdk, so mesh-sdk
carries no composition rule. The catalogue functions only the
node-engine's composition reaches (Resolve, Resolution's Compose,
Declaration, DataItems, LeftOut, Rules and the rest, the settings,
placeholder and template refusals, the rendering helpers) are copied
with their tests into internal/compose/resolution; a method of a
mesh-sdk shape becomes a function taking it first. The composition
calls its own copies, and the captured catalogue its tests read lives
here now, shared by mesh-compose's tests.
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
ADR 0593 §6: a composition read from composed-<node> is a record never
applied, and the engine applies only what it composes itself, so no
composition carries a signature and nothing is verified. The agent's
trusted signers, its signed-composition test and the unused signing
helper of the integration bed go, so mesh-sdk can drop SignComposed,
VerifyComposed and Composed.Signature.
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
The test budget: mesh-host is over 500 unit tests, so the three tests
added for ADR 0593 §6 and §7 are paid for by folding the private
network's hub refusals, its MTU cases and its no-self-peer check into
table-driven tests, each case kept.
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
ADR 0593 §6, §7, §9: the integration test runs the node agent against
the real bus under the grant the grants part composes for a node: it
publishes web's composition in composed-<node> with the digest its have
names, and answers compose-candidate for the indexer's candidate,
changing nothing. A node that cannot compose now answers unready, so
the checker counts it as no answer by the verdict and never by words.
The golden grant reads the indexer's candidates and verdicts as it
reads the builder's builds.
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
ADR 0593 §6: composed-<node> is written again, by the node's engine,
each composition with its plan. The verb composition finds a published
revision by the digest the have and want name, its resources', so a plan
on the record does not hide it. The golden node grant also reads the
indexer's verdicts, which compose-candidate takes for a candidate's
manifests.
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
ADR 0593 §7: a check's composition is the checker's, made from each
placed node's answer, since only a node can compose with its own sealed
values. The node composes the indexer's candidate in shadow on a copy of
its shadow's inputs, the candidate's modules laid over the indexer's
with their manifest verdicts, and answers per module composed (changed
or not), refused or held, and its own composition's verdict, within the
answer bound; it applies, writes and hands nothing. With input it also
answers the composition input, for the sandbox path.
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
ADR 0593 §6: the provisioner, the assigner and the view read compositions
instead of composing, so the node's engine writes each one it makes to
its composition state: the wanted builds, every placed module at its
newest done build (so the plan exists before the want moves), each with
its provisioning plan, and a module refused alone with its refusal. A
key is written only when it differs, and kept while among the module's
five newest or named by a want. A placed module's trunk build landing
now makes a pass.
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
ADR 0593 §1 and §7: compose, its bus reader and runtime reader, takein,
validate, the overlay generator, the composer's run and mesh-compose, and
the judgement of a move leave mesh-sdk for the node-engine's internal
packages, so no other module composes and a change to them rebuilds only
the node-engine. The shapes others read (the composer's request and
verdict, the overlay's names, compositions and answers) stay in mesh-sdk.
The node's grant is now a golden file of what the grants part composes,
since the composer of the bus's user list leaves mesh-sdk too.
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
The bus module on the bus node runs from what its agent holds (its own composition, or the declaration kept before the
switch: every novox have read 2026-11-02 names its own composition or none, and nats names a build alone), while
composed-<node> was written by the controller, which ended (issue 1126). The follower now looks at the agent's record
of what runs whenever it changed, writes again only when the file it names moved, and trusts no composition signer.
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
Every node composes its own since group A (issue 1126), so the resync that handed the vault an own secret's value
while a node applied the controller's compositions never runs, the shadow's answers read no want, and nothing follows
the heard vault key any more.
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
nox now calls the seats' verbs through the console itself (mesh-catalog #593), and the controller the engine forwarded
its lines to ended (issue 1126), so the socket, its forward and the subject it asked on go. The kernel's word on a
peer moves into the filled-file socket, its one remaining reader.
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
The CONTROL stream the facts, the enrolment and the overlay rekey were published into was the controller's, which ended
by its unassignment (issue 1126) and whose stream is removed after group B's delivery: a publish there would fail every
facts event before its haves are written. The facts go into the haves as they did after the report, the enrolment goes
through mesh-enrolment alone, and overlay take is refused because no seat takes a rekey now.
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
The controller that signed the hand-over, search, judgement, access and move asks ended (issue 1126), so a link
believes none of them by having no signer, and the per-ask key it was told to follow goes with the lease.
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
The controller's declared-packages mirror ended with the controller (issue 1126) and is deleted after group B's
delivery, so the verb no longer reads or compares it.
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
The controller ended by its unassignment (issue 1126, ADR 0590), so no build is judged by the lease: the lease
witness, its contract and the direct get the witness asked it with are removed, and its tests now prove each rollback
path on the node tools' PING.
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
mesh/check being carried to 38d666f22390 on 84737f4cb26d | from 14a12cc7b6f9
The controller ended by its unassignment, so a node that falls back to the
compositions in its bucket follows what nobody writes (issue 1126, A7). The
node agent now always runs its own composer: without a placement key it
composes nothing and keeps what runs, a placement it cannot watch stops the
agent and is said, and a candidate taking its placement away is refused
instead of answered as a return to the controller. The start-time placement
read, the end-when-placed watchers and the returning answer go, with their
tests and the three integration tests of the controller-composed path.
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
The controller ended by its unassignment and its lease is never handed, so
the engine's lease gate kept trusting the controller's key and the bus node
kept the controller's list as a fallback nobody writes (issue 1126, A6).
The engine now trusts no controller signature, its hand-over line names the
node seat's hand-over, and the user-list follower writes only the grants
part's list: a key unreadable or deleted keeps the list loaded, and with no
key heard nothing is written. Compositions in the node's bucket name the
file's place alone. The guard no longer asks for the controller's user.
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
A fall back to the old login during a rotation is now delivery.NodeHealth's Fallback (the node, the login fallen back from,
why, when), written with the node's health report, from which the watchman raises node-account-fallback; it is said no more
once the vault's key of the node moves on (blocker 7, ADR 0583 §6). mesh-host enrol's fall back to the controller's
enrolment subject says it is the path before the lease write, which the bus refuses once the lease is written. Tests: rows
in the existing rotation test; none added.
(issue 774)
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
A stand-in vault reloads a nats-server's users as the grants part's list is loaded, and the engine's own rotation code runs
against it with real test-dials and a real confirmation request: at every step at least one login authenticates; the engine
switches only once the list holds both logins and confirms in the new login's name; at rotation-done the old login is
retired and its value dropped; with the list never loaded the test-dial is refused and the engine never switches; and an
abandon after the switch brings the engine back to the old login before the new value goes (ADR 0583 §2-§6). One integration
test; mesh-host holds seven of fifty. The real vault and the real engine together are proven live on g14 first.
(issue 774, 774/S65)
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
The engine dials as enrol.<node> on the pinned certificate, calls redeem with the public halves of its keys and a proof over
them (the SDK's RedeemProof, which binds the node and the keys and never the token's secret), asks again while the holder
says to, and verifies the vault's answer against the key its token carries before it keeps anything: an answer signed by
another key, naming another signer or node, or altered is refused and nothing is kept. A signed one is kept (membership,
login, keys), then the node dials as the login it was given, reads its key of node-logins, and calls confirm; nothing is
applied until its placement names a module. When the bus does not serve the seat to the token (the controller still holds its
lease) the controller's enrolment subject is used as before. The token's shape is mesh-sdk's now: identity keeps its names as
aliases. Tests: one added, paid by merging three identity token tests into one (net two fewer).
(issue 774, 774/S64)
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
On node-logins' ready the engine fetches the new value through the vault's own hand-out (own/<node>/mesh-host/<login>, opened
with its sealing key), dials a second connection as the new login on the pinned certificate, and switches only if that
authenticates, then calls mesh-vault.node-account-confirm over the link as the new login (the vault is asked in the name of
the login the open link authenticated as, and only once that link is open). It keeps the old value until the key says the
rotation is done and drops it then, never before; an abandon, after the switch before or after the confirmation, switches
back before the new value is dropped; repeated refusals of the new login's credentials before rotation-done bring it back
to the old one, said as the finding node-account-fallback (in the journal: no route for an engine's finding to the watchman
is decided), and the same rotation is not tried again. Tests: two added, paid by merging two identity sealing tests into one;
more to be paid by deleting the identity token tests that move to mesh-sdk with the token (774/S64).
(issue 774, 774/S62, 774/S63)
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
A membership may name its Logins (ADR 0583 §1): the login in use and the password of each login held, kept beside the
identity, so a restart dials as the login in use. The engine follows its own key of node-logins as a watch on every link
and dials as the login it names when it holds its value, with the node's one inbox; with no key, or a key naming the first
login, it dials node.<n> with the membership's password as before. A change of the login in use drops the link so it is
opened again as the new one, and the vault is asked in the name of the login dialled (the subject names the caller).
Tests: one added, paid by merging the pinned-broker test with the malformed-pin test (two into one).
(issue 774, 774/S61)
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
mesh/check failed: mesh-build-go: the check of novox/mesh-catalog#590: the ask mvyhumdm860c7f's step on shanks did not end within 30m0s of its start |…
A key that later cannot be read or is deleted is said, the key last heard stands and the list loaded stays. A key without
the flag still falls back, as before.
Tests: three rows added to the existing table test of the follower; no new test function. Built against the group's
mesh-sdk through a go.work outside the tree; go.mod is not re-pinned here.
(issue 774, 774/S31)
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
mesh/check checking 74ecda8ea2cf on bc960c51dbf0 | novox/mesh-host#193 in group feat/node-packages
The delivery group feat/node-packages pins one mesh-sdk head in every member: it carries DeclaredPackagesOf and the engine's read of the mirror this change uses.
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
pacman still reads the controller's mirror because no engine wrote its key: the engine
now keeps each handed composition's package resources and, after a pass that handed
something new, writes mesh-host_packages.<node> only when the value differs, owed after
the growing bound when refused. The manifest declares the state so the grants part
creates it. The verb packages answers the key beside the mirror's key for the node and
their difference, and with write writes the key again from what was composed. H8's
integration test proves the write under the grant the grants part composes.
Needs mesh-sdk feat/node-packages (0ef7c0c) for DeclaredPackagesOf and the engine's read of the
mirror (go.mod pin left for the lead: the SDK commit is not pushed).
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
Issue 1585, S17: the engine decodes manifests leniently; a module whose manifest the indexer refused is held and named,
its data key stands, and every other module's items are still published. Policy, HoldsData and the certificate check no
longer parse with the judge.
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
mesh/check pass e031758e4a37 on 6c1f7cc7f50f | unit 4510 run, 4259 kept, 49 skipped | integration 4589 run, 4361 kept, 51 skipped | over bound: unit, …
The builder claims a done key again as a new revision, so a build kept as done is dropped on any
write of its key. Data items that could not be worked out or written stay due and owe a pass after
the growing bound, so a first write refused before the state or grant exists is tried again
without waiting for an unrelated composition.
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
The data items after a pass re-read each placed module's build over the bus (about 55 per pass on
ace, g14 and shanks); a done build never changes, so the composer keeps it. The items are now
worked out with the pass's own kept builds and modules run inside the node's own, as composed.
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
After a pass that handed a new composition, the node's composer works out its modules' data
items from the inputs it composed from and writes each module's key <node>.<module> of
mesh-host_data only when it differs, deleting the keys of modules that left. Pins mesh-sdk
feat/data-items for the contract and the engine's grant. Two tool-answer tests are merged so
the unit count stays where it was.
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
mesh/check being carried to 71afdf3ad234 on 6c1f7cc7f50f | from ffa63d25a066
novox's postgres stayed held after mesh-host #189: its have, written before that fix,
names no build, so the node's own composition still took it in and was refused for its
secret reader before the start could back it up again. Only that exact record leaves
the node's own; every other have naming nothing keeps main's rule. Completes #189 for a
have written before it (issue 1289).
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
mesh/check being carried to 76c7ddfc5227 on ffa63d25a066 | from a4713e2d28d7
Review round 1: a have naming no build in a phase but running is also written for a
module that runs inside the node's own (waiting for a yes), so taking it out would
drop its resources before the yes. The fp the held have now keeps already fixes
postgres (issue 1289); a test drives an interrupted backup end to end.
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
On novox (2026-10-31) postgres's repeat backup was interrupted by the node-engine's
restart: its held have named no build, so the node's own composition took postgres in
and was refused for its secret reader, and nothing ran the backup again. Issue 1289
(R4c, its delivery held).
- A held have names what runs: read from the module's have before backing-up when
the agent has no running entry, as the not-composed mark does.
- A have in any phase but running that names nothing is not run inside the node's own.
- A backup interrupted by the agent's stop is no failure: the next start backs the
module up again and applies while the want asks a backup. A real failure stays
held for the operator's answer.
Quick fix: a new backup ask by hand. Thorough, chosen: the fault fixed where it lives
in the node agent, so every data module's interrupted backup resumes by itself.
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
mesh/check failed: shanks: the check of novox/mesh-controller#487 failed: go mod download in modules/route-proxy: go: git.novox.be/novox/mesh-controll…
mesh/check being carried to da0f8f1cfdb4 on bb8383b065a1 | from 62b211bb6040
Every put under want.<node>.>, every module's take-in and every placement write
made a pass, and each pass asked the vault for the node's own secrets. The
shadow's change check moves onto the composer, gains the want (by its
fingerprint, the only part composed) and serves both loops.
On the bus node restic's manifest replaced its process measure by awake (mesh-catalog 35ff67bd). The record of restic.measure was the node's, made while restic was folded, so restic's own apply never saw it, and the guard against removing a placed module's resources refused the node's own apply whole every two minutes from 2026-10-30 19:42: nothing of the node's own applied.
Quick fix considered: forget the one record by hand (a hand act on the node; refused). Thorough fix (this): the node's own apply keeps such a record and says so, never refusing or removing it; a module's own apply owns a node record whose id names it, so it alone removes it once neither composition declares it.
mesh/check being carried to b0d6684d3731 on 91fb323ce0cf | from mesh-sdk dec3bfe7a8cb | novox/mesh-host#185 in group fix/compose-retries-transient
The verb compose has no warrant (ADR 0563 §2): Agent.Retry released a module held for a failed backup and re-applied
one already running its want. RetryFailed tries again only a failed apply not held, and refuses in words otherwise.
A hand ask no longer restarts the growing bound, and a pass less than askedWithin old is the answer, so a caller in a
loop drives no passes against the vault. A composer whose loop ended refuses at once. The retry line said try N+1.
The test's first shadow now ends before slowAfter is set, a race the race detector found under the new timing.
On 2026-10-31 ace's route-proxy said only that the vault could not be asked, and ace said nothing for five minutes
while a pass waited after the vault restarted; the retry that already existed was invisible. A refusal for a seat or
a read that did not answer now says when the node composes again ("the node composes again at HH:MM:SS (try N)"), a
retry pass says it starts, a pass that runs on says every minute what it waits on, and the node-engine's verb
compose makes a pass now and says how it went, so a person recovers without a restart.
Four providers waited for a yes on 2026-10-31 and nobody could read what waited: the have said only
"where its data is kept". The read verb composition answers it from the engine: the latest revision of the
want's build or the one asked by digest, every revision the bucket keeps, what runs and where that is known from,
the want and the have, the move's judgement and whether it waits for a yes, and with diff the resources that
differ, a received file naming its consumers. revisionNamed now reads the same revisions.
waitsForYes returned at once for a composition equal to what runs, and settleOne then found nothing to apply, so
the have kept phase waiting-yes for a move that no longer existed (ace.postgres, novox.postgres, novox.keycloak and
novox.mongodb on 2026-10-31). The have is now rewritten to running.
Quick fix: rewrite the have where the equal composition is met. Thorough fix (not here): one place that writes a
module's have from the settled state after every settle, so no early return can leave a stale phase.
The review of d5e8458 found that mesh-wireguard, provided by the mesh and never wanted at a fingerprint, kept the kept
declaration read on every node for ever, against ADR 0550 §2 once the lease is written. A selfcompose row shows the
node's own composes it from delivery.ProvidedManifest with nothing running, so the kept read holds nothing of it.
Quick fix: name mesh-wireguard in allHave. Thorough fix: allHave counts every module delivery.ProvidedByMesh says the
mesh provides as covered. Chose thorough: one list of provided modules, mesh-sdk's.
At 19:43:01 on 2026-10-30 shanks' new engine said "the kept declaration is no longer read: each of the 77 modules
placed here has a have of its own", and from 19:43:57 every apply of the node's own was refused: it would remove
systemd-resolved's resources (novox the same for nats and restic). systemd-resolved's have names only a controller
build (build-1791404191702719760), no fingerprint, so the assigner wants it nowhere and it runs folded in the node's
own, of which the kept declaration is the one record; retired, the folded module was composed as nothing.
Quick fix: never retire the kept declaration. Thorough fix: allHave counts only a have naming a fingerprint of the new
form (or removed); a module at a build of the old form keeps the read, named with why, until it has one. Chose
thorough: the read retires by itself once every placed module is wanted.
mesh/check checking 33131dcbfe77 on 2ae7c53376e8 | novox/mesh-host#181 in group fix/build-not-done-holds-one-module
At 19:30 on 2026-10-30 node-tools' want moved to a build not yet done; once it landed, ace, g14 and shanks stayed
refused until some other input moved (g14 and shanks at 19:40, ace after a restart), since the composer watched no
build.
Quick fix: compose again on a timer while anything is held. Thorough fix: the composer watches the builder's builds
(within the node's compose reads, NodeComposeReads) and makes a pass when a done build of a module the node wants, at
its want's or the trunk's build, lands; nothing else moves it. With mesh-sdk 9685990a (pinned here) a running module
whose build is not done is held, keeping what runs, so the pass that waited composes it as soon as it lands. Chose
thorough: event-driven, no poll.
mesh/check checking 4055e0094b48 on ee71ba4dea81 | novox/mesh-host#180 in group fix/one-bad-item-holds-one-module
At 19:13 on 2026-10-30 power's bar item (mesh-catalog ca8d6f33) made frombus.Compose refuse the whole node, and the
composer marked every module not composed (32 of 81 failed on shanks). The composer already holds what Compose names
held (NotComposedEach, issue 1126); the SDK now names a refused manifest's module held instead of refusing the node.
Quick fix: none in mesh-host; thorough fix is the SDK's input (one rule for Compose, Answer and PlansHeld). The test
replays the captured power manifest and fails on the old pin with the live words.
mesh/check checking b9c24a7b42cb on 737bce9c9448 | novox/mesh-host#179 in group fix/sweep-2026-10-30
mesh-vault (mesh-catalog f3b13de2) answers a move it refuses with `refused: {reason, held}` and `why`, and
no read-back. The engine read only the read-back, so every such row failed "answered with no read-back"
and the controller asked it again on every pass.
Quick fix: treat any answer without a read-back as done. Thorough fix (chosen): read `refused`. For
holds-another-value the vault's value is the live one: the move answers done, superseded, at the held
generation with the held record in its words, so the controller records the row and stops asking; the
journal says it once per secret. Any other reason fails in the vault's own words.
restic's composition dropped its process restic.measure on g14 and shanks at 16:14; the engine removed it
and said it down on every look after. The judges were set only when facts were said, and the facts said
inside an apply read the module's composition from before it (the agent records the new one only after
the apply returns), so the removed resource was put back. Judge.Set itself prunes what is not declared.
Quick fix: set the judges after the restic apply. Thorough fix (chosen): one function hands every judge
(liveness, resolver, units, accounts, libraries) what is declared; it runs from the facts as before and
now also whenever a module settles (applied or removed) or is taken as running, so every removed health
entry of every module is pruned at recompose.
On 2026-10-30 ace's redis, postgres and restic (14:03), step-ca (13:55) and mesh-vault on novox (15:22)
were held on a single backup ask that met no responder while the tool runner restarted.
Quick fix: ask node-backup.now twice. Thorough fix (chosen): every node-backup ask (backed-up and now)
is asked again with the engine's backoff while it is known never to have been delivered (ErrNotAsked),
bounded at three minutes; an ask that may have been delivered (a timeout, a refusal) is never asked
twice, so no backup starts twice. Past the bound the backup fails saying how often and how long it was
asked, the module is held and its have carries those words for the assigner's ask and condition. This
was the engine's only one-shot seat ask.
Every engine's node-seat subscriptions were refused at 12:10 by a user list composed without them. nats
asks a refused subscription again only on a new connection, and a user-list reload makes none, so the
seat answered nobody all afternoon and the messenger held every Telegram ask. The endpoints also lacked
the kind and scope discovery reads, so the console said nothing in the mesh is called node-engine.
Quick fix: restart the engines. Thorough fix (chosen, the second time a stale user list left an engine
deaf): a refused subscription of any kind is said and the link is opened again after a pause (1 min,
doubling to 30 min), so every subscription the engine makes is asked again; the seat's endpoints carry
kind, seat, scope, module, description and schema like every tool runner's.
mesh/check checking 015334020ad2 on a3c098a5c618 | novox/mesh-host#177 in group feat/774-l3-reland
The conflicts were go.mod, go.sum and testdata/beside/CAPTURED alone, taken from the release. 2da0de7 is 6e4a301's code;
its conformance events are unchanged.
mesh/check pass 2fd165f5af8e on 8c5af3e4a3a3 | unit 9174 run, 546 kept, 51 skipped | integration 9952 run, 659 kept, 60 skipped | over bound: unit, in…
78c9ec8 is sdk main 307b623, whose tree is main's pin dbcaa29, with one commit that parses the new words (a credential's
form, an artifact's bus, a log's max-age-days) and names the new seats; an offer with no form still means pair. No code
of ours changes. Unit and integration suites pass against a local bus and main's catalogue and controller beside: 1260
tests, 7 skipped for a container or a throwaway machine.
No code of ours changes. Whole unit and integration suites pass at main's code against a local bus: 1260 tests, 7
skipped for a container or a throwaway machine.
The whole integration suite passes at this head against a local bus (JetStream, and a bus with the witness grants) and
the release's mesh-catalog and mesh-controller beside it: 1260 tests, 7 skipped for a container or a throwaway machine.
Every account resource marked root never goes into NodeHealth.Root; the statement's contract into Contract. Relogin,
root and found are judged each on its own, so an account may be both. Pins mesh-sdk f3677eb. Test: the report's row in
TestAHealthStatementIsSaidOnTheLinkOpenNow carries the agent account's verdict and the contract.
delivery.NetworkPart.Targets (mesh-sdk 6eae076) is filled where the report is made, by the controller's own rule:
- the hub's node for "hub";
- each resolver's node by its address on the private network;
- none when the part points at nothing, at this machine, or at an address no other mesh machine has.
The names come from the placements the node's own composer holds (Decided.Address, Decided.Hub). A spoke's tunnel names
only the hub, so the tunnel cannot name a resolver on another spoke. A node that does not compose its own names no
target.
Test: rows in TestAHealthStatementIsSaidOnTheLinkOpenNow (hub, two resolvers, a VPN's resolver, its own, a healthy
part).
delivery.NodeHealth and its parts replace this package's copies. The message's network and units parts are aliases of
the same types, so its JSON is unchanged. The engine keeps the derivation from its statement and the write on a change.
Test: the key's bytes, decoded and encoded again by the sdk, are the same bytes.
Until now the six machine alerts came from a health message passing to the controller. The report is now kept in the
engine's declared state `health`, key `<node>`, created by the grants part as `acts` is. Only this node's engine
writes it, after comparing it on every look and only when what it says changed; it is read first after a restart.
It carries:
- the network parts (machine-network; network-rewritten through the resolver file part's writer and owner);
- the tunnel's peers, each seen within five minutes or not (network-unreachable);
- the failed units no module owns (machine-units);
- the accounts marked relogin needed (relogin-needed);
- the directories used as found (directory-used-as-found).
The streaks and the time of the look are left out, so a look alone writes nothing. A peer is reported seen or not,
never its handshake time, so a renewed handshake writes nothing.
The layout is internal/link's (NodeHealth) until it moves into mesh-sdk. Pins mesh-sdk 58a8324.
Tests:
- rows in TestOnTheHubAnyFreshPeerIsAHealthyTunnel: the peers, and a change only when one is seen or lost;
- a case in TestAHealthStatementIsSaidOnTheLinkOpenNow: what the report takes, and one write per change across a
restart;
- the manifest's states in TestTheManifestConsumesTheBackupEvent.
- A placeholder is `${secret:<name>}`, the name the module calls the credential (catalogue.SecretLocal): the local
name where it keeps several values of one provision, else the provision. The `<provision>.<local>` form goes.
- The engine finds the one key under `<node>.<module>.` of mesh-provisioner's `provisions` going by that name. That is
what node.<node> may read, by a watch to its caught-up mark; plans are not read.
- The reference's ID must be a credential name of that provision (provisions.VaultName). The vault's answer must carry
exactly that name and generation. Anything else refuses the write, naming both.
Pins mesh-sdk 66ff33d. Tests: rows in TestAWriteAndAnApplyOfTheSameFileSerialise (a local name, a name no provision
goes by, a reference to no credential of the provision, a mismatched answer).
The value written is the one the provision names: a1-47 makes Reference.ID the vault's credential name. A mismatch in
either refuses the write, naming both. Test: a row in TestAWriteAndAnApplyOfTheSameFileSerialise.
The provisioner's client fills a handed module's file but for each `${secret:<provision>}` (or a whole `secrets` file
that is the placeholder alone).
- The node-engine reads the provision's key `<node>.<consumer>.<provision>[.<local>]` in mesh-provisioner's
`provisions`. The key must be `provided` and carry the vault's reference.
- It asks the vault's `own` for that pair credential sealed to this machine, at least at the reference's generation,
and opens it with the node's sealing key.
- It writes the file as declared (its owner), 0600 when the declaration gives no mode.
- Anything short of that refuses the write, naming the provision and why.
A module the node composes itself already has its `secrets` files written so: its composition carries the vault's
value from `own`, written 0600 with the declared owner.
Pins mesh-sdk 4a96793 (go/provisions). Tests: rows in TestAWriteAndAnApplyOfTheSameFileSerialise.
The reviewer showed how 76e214a still lets the 07:54 outage happen at the re-landed engine's first start on novox.
applied/ is empty there, so loadApplied loads nothing. The kept declaration retires, because every placed module has a
have. Nothing then carries the node's own composition. The guard misses nats, which has no want there, so its
nats.conf is guessed as nats's and removed.
- At the start, when the store has no node's own composition, the node's own runs as the kept declaration has it, and
the store keeps it now. The kept declaration retires only once the store holds `node`.
- What this engine applied runs until loadHad reads the composition a differing have names. The bucket no longer keeps
the one novox's have names, so the stored one stays.
- In the node's own apply, a resource of another module goes only when that module's want says removed
(Scope.RemovedByWant). A module the node's own composition carried has no want, so Placed never named it.
Cost: the node's own apply cannot drop one of its own dotted resources without such a want. It is refused whole and
said, never removed silently.
Tests, with no new test functions:
- A first-start case in the agent's Everything test (empty store, then a start with the declaration retired). It fails
on 76e214a.
- Two rows in the guard's table: no want refuses; a want saying removed removes.
- A case for the store in the authority test.
On novox on 2026-10-29 at 07:54 UTC the kept declaration was retired and the node's own composition was in no bucket.
The composer's first pass reads what runs before the agent reads any have (the agent waits for the composer's caught,
sent after that pass), so the node's own was composed with its folded modules running nothing, and the bus server and
four held modules were removed.
- Quick fix: read the node's own have in loadHad too. It still waits for the composer's first pass, and still finds
nothing when the bucket no longer keeps the composition the have names.
- Thorough fix, chosen: the engine keeps each composition it applied (the node's own too) in `applied/` beside its
state, 0600. The agent reads it at Run's start, before any watch, as what runs ahead of the bucket and the kept
declaration. A have naming another running composition wins. An apply keeps; a removal forgets.
Test: a case in TestEverythingIsWhatTheAgentRunsAndWhatIsKeptForTheRest (no new test function). It fails on L3's
engine 4f9227a (0 compositions at the first own composition) and passes here.
novox's mesh-vault has failed since 01:16 on 2026-10-29: its have says failed, so loadHad (fbea777) takes no
composition from it, and what runs falls back to the kept declaration, whose export (mesh-vault.kept) names
f099eaa1, an object the controller's sweep removed. The node wrote a newer export since. The composer carries that
file as it runs (mesh-sdk frombus keptOf), so what runs now names the digest and size of what is on disk, but only
where the node's record says it wrote exactly that content at that path (Applied.Wrote, Target). The apply then
finds the file in place and needs no object.
Quick fix weighed: have the controller put the object back. Thorough (chosen, in the node): what runs is what the node
wrote, never a guess from an older declaration. The rows join the existing whole-apply test; the test count stays.
On novox at 01:16 on 2026-10-29, after the node began composing its own modules, mesh-vault's apply failed while the
vault ran: its own composition carries the export as it runs (mesh-sdk frombus KeptRuns) by a digest whose object the
controller's sweep (mesh-controller cmd/mesh-controller/declaration_sweep.go, sweepNode and sweepDeclarationFiles)
had removed, since the sweep keeps only what the controller's compositions in composed-<node> and the wants name.
The node may only read its bucket (ADR 0486 §2), so the node keeps what it had: a file here that verifies, or an
object fetched, goes to <state dir>/objects under the object's name, and a kept one is used before the bus is asked.
An object unused for 14 days is let go. A fetch that still fails says the keep never held it, before its cause.
Quick fix weighed: re-put the object from the controller's store. Thorough (chosen): the node never depends on the
bucket for content it once had, for every module, since the sweep and a node composing its own can always disagree.
The whole-apply test gains three rows that fail on the old code; the test count stays.
The kept-declaration fallback built a composition without Data, so after a restart a data module (mongodb, mssql
on novox) was judged to move its data against its first composition and the operator was asked a false yes. Now
every such composition takes Data from the manifest (agent.Env.HoldsData, the composer's DataItems), and a module
whose manifest cannot be read is said to be unjudgeable rather than taken to hold none. The composition a have
names comes before the kept declaration in the judgement too.
loadHad takes a have only in the phases baseline takes as running (running, held, a composer's mark): one backing
up, applying, checking, failed or waiting names a composition not applied (review of 031e106). Two rows added to
the same table test; the count stays.
A held module on a node composing its own never gets a composition, so the agent never took it and the filling
judged its files by the kept declaration, older than power's three new drop-ins: refused from every engine start
on ace, g14 and shanks until some module settled (2026-10-29 00:09 UTC). The second meeting of issue 924.
Quick fix: knock the filling once the haves are read at the start. Thorough fix (chosen, the kind recurs for every
file added since the switch): what runs of such a module is the composition its have names, from the bucket or
its revisions, ahead of the kept declaration; the filling is knocked on every change of what runs, not only on a
settle. One unit test replaced by a table of four cases; the count stays.
mesh-sdk's composer no longer holds a module behind a held one it reads from; the selfcompose test now expects
proxy handed, composed without de-spiegel's routes, and de-spiegel told alone. Needs mesh-sdk pinned at the fix.
mesh/check pass a6c70803ab72 on bb6e47293909 | unit 3054 run, 234 kept, 36 skipped | integration 3075 run, 266 kept, 29 skipped | over bound: unit (sl…
novox's seed p-20261028t231141-ec84b4 refused 11 modules behind de-spiegel's hold and gitea's handed credential. The
node composes and answers through mesh-sdk's frombus, so the fix is the pin.
- The quick fix: follow one more set of words in the answer. It moves the error again.
- The thorough fix, chosen (mesh-sdk fix/1126-held-as-runs): a held module composes only as what runs of it; a consumer
or collector composes against that and is never held for the held one's stub, and a stub's handed credentials and own
secrets are left out as they run rather than refused.
mesh/delivery-group group feat/774-vault-mints-l2 checking: the checker's verdict is out of date: novox/mesh-delivery@64d364a7686c was judged on trunk fa075dc8…
mesh/check failed: ace: the check of novox/mesh-host#168 failed: the pin of git.novox.be/novox/mesh-sdk/go at commit d4c61ee8ffd0 is on no pull reques…
mesh/delivery-group group feat/774-vault-mints-l2 checking: the checker's verdict is out of date: novox/mesh-host@e0c7f36ca90c was judged on trunk d2a868f99f05…
mesh/check failed: integration: meshbuildgo/cmd/mesh-build-go.TestAComposeAskIsBuiltOnceAndAnsweredOnTheBus: the compose ask mvsg85b34d25b4 never ende…
The answer names its proposal and the revision of the node's own candidate key; a node with no key in a proposal is not asked. The watch for its own placement key is tried again until the seat's state exists, so the node switches when its key is written.
mesh/check being carried to bf667a602d21 on 58b4f8b6bc07 | from 9a8b3870bad0
A reboot left the have running, so a unit that failed to start at boot was never started again and nothing said so. Baseline now asks the machine; the judge's look calls Agent.Reconcile once per down edge, leaving held and unwanted modules alone.
mesh/check pass 24c23a1d066f on a548a61e86d7 | unit 2667 run, 590 kept, 36 skipped | integration 2676 run, 633 kept, 29 skipped | over bound: unit (sl…
frombus.Compose now holds a module refused alone (mesh-sdk 0de3006, the rule
its answer follows too): composed as what runs of it, with what depends on
how it is held held too. The composer hands every other module, keeps what it
handed of a held one, and tells the agent which modules it holds and why;
agent.NotComposedEach marks those alone and clears the rest. Only a refusal
of no single module still marks every module (NotComposed).
heldOut also holds out, of the node's own composition as it runs, the
resources of each module held alone, until a composition of its own is
handed: the node's own apply never removes a module leaving the fold with its
names unset (review of the apply, round 1; issue 1945 by a new path).
Pins mesh-sdk fix/1126-apply-culprit at 0de3006 (a group).
mesh/check pass bc28606d4566 on 690aa662a837 | unit 2192 run, 3794 kept, 57 skipped | integration 2208 run, 3829 kept, 57 skipped | over bound: unit, …
A skipped or held file comes back on every apply; counting it would reload the
service manager on every apply of a node with a gated or held module. The same
allowlist as MovesRoot: created, updated, corrected, removed.
A drop-in that no service reflects, like the power module's bound on
reboot.target, was written but never loaded: the node-engine reloaded units only
for a file named in a service's restart-on, so the next reboot still ran the
target as loaded. Any file written or removed under a system unit directory, or
a drop-in directory below one, now gets one daemon-reload per apply.
mesh/check pass 8bd0b5db15a3 on b4fd2cc210bf | unit 1639 run, 4201 kept, 46 skipped | integration 1674 run, 4215 kept, 47 skipped | over bound: unit, …
Each answer's line now reads 'answering placement candidate G (revision N; woken by X)', X being its first answer,
a retry after a read the bus did not answer, the candidate, or the inputs that changed (placement, records, modules,
values, haves, wants). The single input counter becomes one per input, which also decides whether a group is answered
again, as before.
Quick fix weighed: a counter word in the line. Thorough fix chosen: one count per input, so the line names exactly what
moved since the group's last answer.
mesh/check being carried to 1a021052fe07 on b4fd2cc210bf | from 7a2af2376405
Review round 1 measured it on systemd 262.1: a finished one-shot nothing references is
unloaded, and loaded again it reads inactive, ExecMainStatus=0 and a start timestamp of
0, as one that never ran does. The previous commit's rule read such a unit stopped, so
a one-shot declared running would be started, read stopped and fail on every apply, and
the hold, the take-over and the giving back would judge it wrongly.
ServiceState keeps main's reading. ServiceAtRest and declaredState stay: they alone
satisfy a stopped declaration for a one-shot at rest, which is the kernel cleanup's
case. A one-shot declared running that must be seen to have run lingers
(RemainAfterExit) or is enabled at boot. Unit tests: the arch table has 5 rows, the
apply test 1; 1262 in all.
The kernel module's boot-time cleanup, linux-modules-cleanup.service (Type=oneshot,
declared stopped and enabled at boot), failed its delivery on ace with 'was asked to
be stopped and is running': ServiceState read every inactive one-shot with
ExecMainStatus=0 as running, and systemd says 0 for one that never ran too. Stopping
an inactive unit changes nothing, so the read back said running again; after a boot
that ran it, the same declaration would fail every apply.
ServiceState now reads ExecMainStartTimestampMonotonic: 0 means never ran, read as
stopped (a package's one-shot declared running is then started, and the hold no
longer counts it as in use). The service apply reads a one-shot at rest as stopped
to a declaration that it be stopped (ServiceAtRest), whatever its last run did.
Quick fix weighed: declare the cleanup running in the catalogue. Thorough chosen:
the reading was wrong for every one-shot, and this is its second fault of the kind
(the firewall's was the first). Unit tests: three arch one-shot tests merged into one
table (7 rows), one apply test (2 rows); 1263 to 1262.
Met twice on 2026-10-28: shanks kept 'want.shanks.brother could not be read ... no
responders' as the own-secret step's last word, and ace kept 'the vault's own: not yet
... no responders' as its answer to seed-7, each until another input moved. Every bus
read of the composer (manifests, builds, archives, served values, the vault's own and
credentials, the have of what runs) is noted (BusFailed); an answer or a pass that met
one is made again after a growing bound (5 s to 5 min), and a pass is not handed when
the have of its sealed values did not answer. The own-secret step tells a read or an
ask that did not answer (Retry, taken again) from a key absent or the vault's word.
On shanks probe/1126-seed-7 was answered at revision 84 every 13-17 s: each look rewrote
the haves of its unhealthy modules with new words, and the shadow took any write of its
own module haves, placement, modules or config values as a change. Each answer loaded
the bus, the health tools timed out, the words moved again. Now only what an answer is
composed from wakes it (what runs, folded modules, a changed value), and each group is
answered again only when its keys' revisions or the node's inputs moved.
A new step, internal/ownresync, after each shadow answer: for each own secret masked as another value, while the node
applies the controller's compositions and runs the one its want names, it asks the vault's move-in with
replace_running: true and the running value sealed to the call's key, once per value, and says what it asked and what
the vault answered. The answer written to the node's have key carries values_differ. Pins the group's mesh-sdk head
1062b33 (go.sum is filled once that head is pushed).
A box is reused by slot (resource, field) after the composition, from what runs or the
have's slots; a have written before slots is reused within its module alone. Config
values are laid by mesh-sdk's one rule (never a default). The shadow answers the
newest candidate first and hands each answer as it lands, says when an answer starts,
every minute what it waits on, and when it ends, and fits an answer to the node's key.
Needs mesh-sdk fix/1126-seed-same (16b8d8a1).
mesh/check pass 047237c911a4 on 449cda1b003e | unit 2980 run, 270 kept, 36 skipped | integration 2999 run, 302 kept, 29 skipped | over bound: unit (sl…
ace, g14 and shanks answered nothing to seed #531 and said nothing either. The
shadow now says, after a minute not ready, which inputs it has not read, and
answers each group it knows refused for that; an answer running over a minute
is said while it runs and noted in the answer. The answer's differing resources
carry both bodies, masked, the node saying whether two sealed values open to the
same value (Composer.sameValue: only the comparison leaves the node).
mesh/check pass a6e53a485813 on 199f6b48b1df | unit 2992 run, 1008 kept, 36 skipped | integration 3025 run, 1102 kept, 29 skipped | over bound: unit (…
NotComposed marked from a read have whatever its phase: an apply that failed,
naming the want's build and digest, became a composer's mark, which baseline
then took for what runs, and never applied again. It now marks only a have that
runs, says nothing, or is a composer's mark already. And its loop, whose reads
can take seconds, checks the run's context at each module, so a run that ended
writes no have.
Live 2026-10-28: the own run's last pass, failing on its run's end, marked
every module of the controller-mode agent that replaced it (ace 60, shanks 63),
and the marks dropped the build and digest because the have read failed in the
bus's reload pause. Now: the refusal hook is bound to the run's agent and ctx;
a pass whose context ended returns nothing; JetStream not enabled or no
responders are tried again after 1, 2 and 4 s before any mark; NotComposed reads
the have again while the bus does not answer and marks nothing it cannot read;
baseline takes a have a composer marked as what runs, so a mark it did not make
clears; each run's shadow lives with the run.
mesh/check checking 463626143567 on bb15aab50eda | novox/mesh-host#154 in group fix/1126-shadow-all
selfComposer took a KVHistory error for 'not placed': at any restart a timeout
or a bus not ready flipped a self-composing node to the controller's
compositions for a whole run. selfcompose.Placed chooses only on a read that
succeeds (a missing placement state is one), and tries again otherwise, said
once. The shadow's answer to a head deleting the placement reads the
controller's composition at each want in the node's bucket and compares it with
what runs; a want with none refuses, without returning.
The seed's revert (mesh-catalog #526) was refused on 'would stop running node',
while a self-composing node would have refused its placement's absence, kept
what runs and marked every have failed until a restart. endWhenUnplaced ends the
agent's run when the key goes, so it begins again on the controller's
compositions, as at a start without a key; the composer refuses nothing while
unplaced; the node's have says whose compositions it applies (Composes); the
shadow answers a deleted placement Returns, verdict same.
novox refused its own composition and its answer on de-spiegel, a module it runs
folded into its own composition at a build only the controller holds. The
composer now gives frombus what the node runs (Scope.Running), so such a module
is carried as it runs. The shadow's answer names every refusal (mesh-sdk). A
templates archive read is bounded at 30 s: nothing else held an unanswered store.
H8 found it: a module the node could not compose yet got a failed have naming
no fingerprint, which counted as run inside the node's own composition, so the
next composition took it from its manifest past its build.
The node read a behind module's source manifest digest and composed every module
from the indexer's raw manifest; frombus now takes the build itself (Scope.KeptAt,
a fingerprint). The reader gives frombus the builder's key, a templates archive
fetched as every archive is (apply.Fetch) once per digest and checked against it,
and another node's served keys one by one, as the node's grant allows them.
The fold's watch of every have key ('>') was refused (permissions violation on
CONSUMER.CREATE KV_node-engine_have), and every self-composing agent stopped
after the seed. The composer now watches '*.node' (records) and '<node>.>' (its
own haves), as buscompose grants a node; the shadow watches '<node>.>' too. The
selfcompose tests check every watch and get against PermissionsFor(KindNode),
and the agent's integration tests run the agent as the node under that grant.
mesh/check checking 69b845c0f7b2 on 0d22776b20ef | novox/mesh-host#152 in group fix/1126-fold
A fresh agent composes before it has applied anything (Running empty), so
InOwn of Running folded only mesh-wireguard and the switch would drop nats.
The composer now watches the node's haves: a placed module whose have names no
new-form fingerprint is folded; one with a new-form have whose want is removed
is not folded. The shadow uses the same set, over the candidate's placement.
Pins mesh-sdk 05b67831, whose compositions fold a module no new-form build
names into the node's own. The shadow composed a placed module it runs inside
its own composition (nats, photos) at the trunk's build, as if the candidate
added it; shadowBuilds now leaves those to the fold, and both paths pass
frombus.InOwn of what runs.
mesh/check checking b225c7bb4cf5 on be94d2689e2f | novox/mesh-host#150 in group fix/1496-start-health
On ace, 2026-10-26 21:19, mesh-build-node's cache directory was there before the
mesh and used as found. The health statement says such a directory unhealthy, and
the have folded it into the module's health as failed, with no words, while the
module's own check answered healthy; the assigner read it as its canary failing
and held the version on all four nodes.
The directory waits for the operator's hand-over and keeps its own condition,
raised from the same statement; it no longer counts in the module's have. A have
whose health is failed or waits and whose apply said nothing carries the words of
the resource that made it so, so the next one is readable.
mesh/check failed: unit: ..test: ✔ a transient empty list for four passes retires nothing; five stable passes retire, data kept (290.408814ms) | 58de6…
mesh/check failed: ace: novox/mesh-catalog#505 does not merge into the trunk: files in conflict: .gitignore | cdd3f3fd4f63 on ff9386dcccc2 | unit 1245…
mesh/check failed: ace: novox/mesh-catalog#505: checks/placement/go.mod names mesh-sdk 91413de33516, which is neither on mesh-sdk's trunk nor the grou…
mesh/check failed: shanks: novox/mesh-catalog#505: checks/placement/go.mod names mesh-sdk 91413de33516, which is neither on mesh-sdk's trunk nor the g…
mesh/check failed: shanks: novox/mesh-catalog#505: checks/placement/go.mod names mesh-sdk f1d7c7377dc4, which is neither on mesh-sdk's trunk nor the g…
mesh/check pass 65cd7b47873a on bba9081ca127 | carried from 94f9aa298cb8, mesh-sdk 21252194d530 | unit 971 run, 423 kept, 32 skipped | integration 976…
A process no witness judges had its directory removed before the new bundle
was unpacked and the unit restarted, so the old process ran on in a deleted
working directory: mesh-build-go's `go version` probe failed with "getwd: no
such file or directory" and refused three checks on ace on 2026-10-26, each in
the second a new build of it arrived.
Every process is now replaced the way a judged one is: its build is renamed
under .witness/<name>/retired/ and deleted only once nothing runs from it.
"Runs from it" now counts a process's working directory as well as its image,
so the children a daemon starts keep the directory too. Forget keeps a retired
build that still drains, and the witness's sweep covers every process it keeps
a directory for, judged or not.
Two unit-file tests are folded into one, so the unit count goes down by one.
The share belongs to the part that enforces it: build-cpu-share, default 50,
set per node through the node's settings layer and read by the composer for
the build seats' runtimes. Pin mesh-sdk at fix/build-cpu-yields' head.
Review round 1: one quota per unit let three build modules take three shares.
The engine now writes meshyield.slice, top-level so its idle weight binds
against the operator's session, with the node's share as its CPUQuota; each
yielding unit joins it at nice 19. Comments say the IO controls bind only under
bfq or io.cost, not under the none scheduler NVMe disks use.
mesh/check pass 43f2d6020e11 on 94f9aa298cb8 | carried from 7f04d1204765, mesh-catalog a4f5df797b68, mesh-sdk b4cd95e7c4a4 | unit 972 run, 423 kept, 3…
On the bus node the bus module has no want and has left the declaration, so no
pass of this engine held its fact and the follower's user list was kept in
memory and never written: the vault and the grants part read no hashes and no
key. A module's user list now reaches its key on every pass, merged into what
the key says with every other field kept; a missing key is said once, never
made up (ADR 0547 §8).
A build seat's runtime took 1650% of shanks' 24 processors and slowed the
operator's desktop. The unit of a process declared with yields now carries
CPUWeight=idle, IOWeight=1, Nice=19, the lowest best-effort IO level, and
CPUQuota for its share of the processors this machine has.
Merges fix/1945-adopt-hashes 1a190090. A new follower (one per link) waits for the grants part's key to catch up before
its first write, so a restart after take-over never writes the controller's list first. A refused or ended watch is
tried again on link.Backoff, the have says why it is not watched, and the key last heard, kept beside the node's state,
holds meanwhile. While the grants part's list is followed, the bus module's composition is applied with the list on the
machine in place of the controller's. The grants list is never handed to the vault. Pins mesh-sdk c2a410bb.
On the node whose compositions carry the bus's list, the engine also watches mesh-grants' key user-list.sealed. While it
says follow grants, the object it names is read by name, opened with this machine's sealing key, checked against the
key's digest and the guard, and written to the same file at 0600 only when its digest differs; the bus reloads it
itself. The have says which list is followed and this machine's sealing key, which the grants part seals to.
The follower's test waits for its watch to catch up before a later put, which raced under load. Pins mesh-sdk 175a5567.
The vault holds no value for the controller's own account, the node accounts and the enrolled devices, so it cannot
hash them, and take-over waits on their hashes. The bus node's engine is the one part that has them, in the list the
bus loaded. Its follower of the user list now puts every loaded user's bcrypt hash in the bus module's have, sealed to
the vault's own key (mesh-sdk delivery.AccountsHanded), so of every reader of the have bucket the vault alone opens
it; the clear part says only the list's digest, the key's fingerprint and how many users. The vault's key is heard from
its answer to node-key, which the engine already asks at start; a first or changed key looks at the list again. It is
sealed again only when the list on the machine or the vault's key changes; a refused list hands the list kept on the
machine. The vault takes from it only the users it holds no value for.
Also: the user list follower's test sent its later puts before the watch had handed the keys there before it, so two
of its cases failed on main whenever the goroutine started late; the feed now says when the watch began.
mesh/check pass 99285cb080db on 1195cf45971e | unit 303 run, 1080 kept, 5 skipped | integration 306 run, 1082 kept, 4 skipped | over bound: unit (slow…
The bus node's engine writes the bus user list (ADR 0547 bridge); each write makes the bus reload its accounts, and
for about three seconds every JetStream request meets no responders. An apply that failed then was kept failed until a
key changed (mesh-vault on novox, 2026-10-25 23:04:24), holding every module that needs the secret seat. A transient
bus failure is now retried with a backoff (2 s, 5 s, 15 s, 60 s, then every 5 min), the have saying retrying meanwhile;
after the engine writes the user list, the transiently failed modules are tried again once the reload is over. A have
write that met it is written again the same way. shorten keeps the head and the tail, the cause.
mesh/check pass 5e71ab6ae612 on d75c66dc7e27 | unit 303 run, 1077 kept, 5 skipped | integration 306 run, 1079 kept, 4 skipped | over bound: unit, inte…
B1: the controller sends the user list by digest (over 64 KiB), so its users are counted from the list as resolved, not
from the composition's inline content, which is empty then.
R1: a composition signed for another node or module than its key names is no source.
R2: a list without the controller's own user, or one that drops more than half the users of the file on the machine,
is not written; the have says why and the file on the machine is kept.
The list reached the bus only through a want of the bus module, which nobody writes (ADR 0539 §15): on 2026-10-25 it
stood from 11:23 to 20:50 and was put back by hand. The node agent now follows its composition bucket for the newest
signed composition carrying the user list, writes that file alone through the apply's file step when it differs
(mode 0600, read back, recorded as the module's), moves nothing that reads it (the bus module's entrypoint reloads the
server in place), never goes back to an older list, looks again after every apply of the bus module, and says source,
revision, wanted and written digests, users and any error in the bus module's have (mesh-sdk UserListWritten).
Read with mesh-delivery-view.node_haves (module nats); the bus's own nats_user_can says in_force once it reloaded.
mesh/check pass 338797a52703 on d75c66dc7e27 | unit 2301 run, 3410 kept, 10 skipped | integration 2326 run, 3431 kept, 13 skipped | over bound: unit, …
A write judged by the declaration before the apply settled could record its
refusal just after Knock found nothing owed: the original fault in a small
window. Knock now holds the apply's lock as a write does, reads no declaration
while nothing is owed, and its comment names Settled as its caller.
ace's filling refused mesh-build-go's files as not named as filled 32 s before
its want arrived; after the apply that named them nothing asked again, and the
runtime crash-looped on its missing broker credential. The filling now keeps
what it refused or failed and, when the node agent says a module's apply or
removal settled (agent.Env.Settled, called once what runs holds it), touches its
write socket if the declaration now names such a path, names the provisioner's
account, or a write had failed; the client watches that socket and fills every
module again. Said in the journal each time.
mesh/check pass b32a27c115a2 on 0dfbe1a179f2 | unit 292 run, 1060 kept, 5 skipped | integration 295 run, 1062 kept, 4 skipped | over bound: unit (slow…
The facts look inside an apply runs while the module is still applying and judges nothing; the
running write then left xorg running and unknown until the five-minute statement, holding every
x11-display consumer behind it. The running write now nudges the health loop, which looks and says
at once. A declaration's report gets the same motion and composition check, and a removed module
leaves nothing marked unjudged (review of fix 15).
A have said applying and failed for the controller on novox at 15:27:36, from a look at the
container being replaced, two seconds before its launch said healthy; the assigner held the
canary on it and set novox back. A look now judges only a module that is not in motion, and a
module whose composition moved carries unknown until a look after the move.
xorg's have on g14 and shanks kept the unknown an engine before 958ccbd wrote: the
baseline believed the have and never handed the module to the session, so no look
rewrote its health, and the assigner held every module requiring x11-display back.
mesh/check pass 1bdbf47ffa83 on 0dfbe1a179f2 | unit 302 run, 1044 kept, 5 skipped | integration 305 run, 1046 kept, 4 skipped | over bound: unit (slow…
A unit or container of the node's own that shares only its name with a
module's resource is not the module's: dropped, its record became an orphan
the apply removes, stopping the module's unit (1945/AC17).
The node's own composition and a module's both declared the same files, and
the joined declaration was refused whole ("2 resources declare X"): the
node's own copy is now left out when joining and at the node's own apply, and
said once (1945/AC17). A module still running from the kept declaration now
carries its filled paths, so the filling no longer refuses them as not named
as filled (1945/AC18).
mesh/delivery-group group fix/r6-node-composition held: a member is held: novox/mesh-sdk@10b6e202320f (merged without a passing check: only a person decides wh…
The build agent left mesh-controller with R4g (ADR 0524 section 5); genesis still installs it until issue 1152 decides where a new mesh's builder comes from. The tests that read its manifest say NOT JUDGED and are judged again with 1152.
mesh/merge-gate head 0d12fb11a48b merged into trunk 59e6a6fa2a4a — pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes wi…
mesh/check failed: mesh-build-go: no go build node qualifies: builders disagree: go1.27.1-X:nodwarf5 recipe 74744709d277ded9 on ace, g14, shanks; go1.…
On ace the postgres container was replaced at 22:11:51 mounting superuser.secret, a
file the provisioner's client fills only after the apply. The runtime made the
missing source a directory, and every fill after it failed with "is a directory".
A container whose mount source is a file the declaration declares, not yet on the
machine, now waits and is neither created nor replaced. The write of that file
(WriteFilled) applies what reads it, as before, so the container is made once the
file is there.
What the runtime already left is cured by the mesh: a declared file whose path is
an empty directory in the mesh's own tree (by its path and by where it leads),
owned by the engine's account, has that directory removed before the file is
written, and the outcome says so. A directory with anything in it, or outside the
tree, is never removed.
A root-only link inside /var/lib/mesh pointing out of it would have made a
directory outside the tree, owned by the engine's account, the mesh's to give
away: the fault of issue 1873 again. The path and the tree's root are now both
resolved, links followed, and the directory is the tree's only when the resolved
path lies below the resolved root as well.
postgres's data directory moved to /var/lib/mesh/postgres. On ace and novox that
directory was there, root:root 0700, before the directory resource was applied, so
the engine used it as found (issue 339's rule) and left it to root: the postgres
provider, running as the operator's account, cannot reach its own secrets,
consumers and grants there.
/var/lib/mesh is the engine's tree (root, 0755): only the engine's account makes a
directory in it. A directory below it owned by that account is now the mesh's and
is given its declared owner and mode. Anywhere else, or owned by another account,
issue 339's rule stands.
mesh/merge-gate head 8c0bb9d4531d merged into trunk 4065b45ae84f — pass: group fix/1873-system-state-dirs composed: every machine composes with the change …
The nats module takes its server as an upstream artifact, pinned by digest
(issue 1641), and builds no image on a NATS_BASE argument any more. The test of
the bundle's bus read that argument, found none, and failed every mesh-host check
against the catalogue's main. Bootstrap's own code never read it: the bundle's
bus image comes from the bundle, so genesis was not affected. The test now
compares the bundle's image with the module's server by repository and digest.
The postgres module's own secrets move from /var/lib/postgres, the postgresql
package's directory, to its directory under /var/lib/mesh (mesh-catalog,
fix/1873-system-state-dirs). Genesis writes the store's superuser file at the
module's own-secret path, so the store it raises is the one the module adopts in
place: StoreSuperuserFile follows, to /var/lib/mesh/postgres/superuser.secret.
A directory genesis makes on the way is 0755, the mode of the mesh's directories,
and only the credential's own is 0700: MkdirAll at 0700 would have closed
/var/lib/mesh on a machine that had none yet.
mesh/merge-gate head 09c77d57b823 merged into trunk c231d1031fc7 — pass: group fix/1732-held-moves-asked composed: every machine composes with the change a…
mesh/merge-gate head e597f9791c62 merged into trunk 7e83137aa822 — pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes wi…
mesh/delivery superseded: its walk was superseded and names no walk that took it over: a walk that took it over answers its merge, and it follows that wa…
mesh/merge-gate head d9f8bbec0c0b merged into trunk fd3259a3f5d7 — pass: group feat/r4d-groups composed: every machine composes with the change as it did w…
mesh/merge-gate head fdb59bcf4eb7 merged into trunk c5c4caa2a4ab — pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes wi…
mesh/repo-check head fdb59bcf4eb7 merged into trunk c5c4caa2a4ab — pass: THE CHANGE ALTERS ITS OWN CHECK (merge-check.sh): main's version judged it; the ch…
mesh/merge-gate head 5a2bc0391054 merged into trunk ce1b97ad973d — pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes wi…
mesh/merge-gate head 52600ec7bd61 merged into trunk cdcba9ca7375 — pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes wi…
The asking helper registered a.Close, which closes the shared test connection; the stand-in's cleanup ran after it on a closed connection, so its consumer was never deleted and blocked the next test on the workqueue stream.
The stand-in shared one durable name across tests and let go with Unsubscribe alone, so a test starting before the server dropped the binding was refused with "consumer is already bound to a subscription".
mesh/merge-gate head 150fadfe8bed merged into trunk 619e14aa8e57 — pass: group fix/r3e-before-the-gate composed: every machine composes with the change as …
The set of modules this process wrote lives in memory, so a node-engine restarted before the apply that drops a module never wrote that module's removal. The report now names the modules the kept declaration held before, and they are removal candidates.
A module that left a node kept its have key at running for ever, so nothing could tell it stopped and the gate would never take the seat. The key is written once with the phase removed over the revision read, never deleted. Pins mesh-sdk to the commit that adds the phase.
mesh/merge-gate head 64dfa90c3a6c merged into trunk 454be9b2693d — pass: group feat/r4a-sdk-first composed: every machine composes with the change as it di…
A build module's sandbox needs one, and a module can require only a capability the node-engine detects (ADR 0448 section 5). The probe drops to an unprivileged uid first when run as root, since root always can.
mesh/merge-gate head 671f2840046f merged into trunk 89025ad004d4 — pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes wi…
mesh/merge-gate head aab65b81f65c merged into trunk 6469e8238443 — pass: group feat/delivery-have composed: every machine composes with the change as it di…
The node-engine reads a resource's module and a declaration's builds, says it does in every report (reads_modules), groups the apply's outcome per module with every failing resource named, and writes have.<node>.<module> and have.<node>.node to the bucket only when a value differs, never ahead of or in place of the report (novox/hq ADR 0429 section 4).
mesh/merge-gate head dae79c76434c merged into trunk cd13f7726ef0 — pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes wi…
The merge check's build container has no pacman, so the acceptance test now stubs it and names the host it was built for, as other command tests do; nothing in the node-engine itself changes.
mesh/merge-gate head d237dee4c9e3 merged into trunk cd13f7726ef0 — pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes wi…
A directory or archive resource carries `unknown` (move-out, module, program) and `knows`. For move-out under a home, every look moves each path the declaration does not know into ~/.local/state/mesh-moved/<module>/<time>/, the list written before each move, without following a link, a young path waiting, a socket or pipe left and reported, nothing made when nothing moved, and another file system refused. Reviewed in two rounds.
ADR 0380's secret-given announcement asks operator-channel.tell over the
bus; the genesis bundle's first user list must already carry that
publish subject, in the same place the controller composes it
(internal/broker, mesh-controller ms-835 435468c1), or a first node's
controller starts without the grant until its own first composition.
mesh/merge-gate head 62af89f85d0d merged into trunk 1a2677882380 — pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes wi…
Every 15 s the failed-units look read the own manager of every account
the declaration creates, and reading one is a login that starts it: six
module accounts' managers on the control node started and stopped all
day, most of its journal. A look only reads (ADR 0240 rule 6) and never
asks an account's manager (ADR 0177).
An account's manager is read now only when the declaration places a
user-scoped unit in it, and only after the machine's own manager says
user@<uid>.service is active, as the apply asks before it. A manager not
running is said away in the units statement, with no failed units; in
liveness its units read unknown with the apply's words, never down. The
operator's account on a desktop is still read while logged in, since a
module places a user unit there (issue 315's Razer daemon stays said).
mesh/merge-gate head b9a293f8c8bc merged into trunk 0e92ef3d1b21 — pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes wi…
A restart-on trigger that is no file counts as changed only in its own pass,
so after a busy wait the container read unchanged and was never replaced,
while the report stopped listing it (review of #98).
The broker step sent the nats module with a plain push, and the controller refuses a push that would
replace the bus: that is a planned step, bus upgrade, which snapshots the streams first and checks
them after (ADR 0236). Genesis stopped at step 18.
The step now snapshots every stream with the module's own snapshot program (no backup holder exists
yet to be asked), asks bus upgrade with that snapshot and why, says the one restart, and waits until
the bus runs the module's image, answers, and the controller ends the step done. The bundle's bus runs
the release the module's image is built on (2.11.17, by the same digest, where it ran 2.10.29), so the
step changes no server release and is said reversible.
mesh/merge-gate head cb7b71dcc568 merged into trunk 12bc6a56a38a — pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes wi…
A port reached from local-network is admitted from the private networks the machine is attached to, and only the machine can say which those are. Every network on a link facing outside is said among its facts, and a change is said at the next reconcile.
Genesis's base step built only mesh-tools, the TypeScript toolchain. The store's and the bus's modules
carry Go bundles, compiled in mesh-tools-go, so genesis stopped at the store step: "this mesh holds no
copy of it. Build mesh-tools-go first". The base step now builds mesh-tools-go too, from
--tools-go-source or the repository beside --tools-source, and a test holds every toolchain a genesis
module's bundles compile in, read from the catalogue, to the ones the step builds.
The packages step sealed the build agent its genesis document and pushed, and the push returned as soon
as it was sent. The sdk step then asked for the SDK's build at once, and the agent reads the document
from the file the node writes when it applies that push: one genesis found it in time, the next was
refused for having neither a binding nor the document. That one push now waits until the node reports
it applied it (push --wait); every other push still returns on sending.
The bundle raised the store as mesh-store, on upstream postgres, in a docker volume; the postgres
module keeps it as postgres, on its pgvector image, in /var/lib/mesh-store owned 999:70, reading its
superuser password from a file. The store step refused, and genesis stopped at step 17 of 24.
The bundle now raises the store with the module's name, image, data directory and password file, and
every step of the bundle and the installer reaches it by that name. The refusal to raise a store over
data it cannot open looks for a cluster in the store's directory, no longer for the old volume.
The bus reads its user list from a file that, until the broker step adopts it as the nats module,
only the installer writes, and the installer wrote it only at enrolment. The build agent's account,
minted with step 13's push, never reached the bus; the agent was refused (Authorization Violation)
and the sdk step's build waited until it was killed. Every push genesis makes now places the composed
list and has the bus re-read it, until the nats module is registered (ADR 0331 §5, ADR 0333 §4).
The build agent declares neither the retired builder's npm-password own-secret
nor a serves setting, so the packages step handed its credential to nothing
and genesis's sdk step had no way to publish. The packages step now seals the
genesis-package-registry own-secret to build-agent: the forge on this node's
loopback, the port given for the package registry, its npm path, and the
mesh-builder account with its password. setFoundationSettings no longer gives
the builder a serves.port.
Genesis's builder step read modules/builder from the catalogue, which retired it on 2026-10-03 for
the build agent (node-build-agent on every machine), so no genesis got past step 13. The step now
installs build-agent from the catalogue, puts the builder image it carried and published where the
manifest names the image it builds, and issues its bus account as any module's.
The controller reads its merge window's settings from a file its manifest declares from its
settings (MESH_MERGE_WINDOW_FILE, ADR 0276). Step 11 took every MESH_..._FILE for a secret and
refused this one as a file nobody writes, so no genesis got past step 11. A _FILE naming a file
resource of the module is now the module's own; one naming neither an own-secret nor a file of the
module is still refused.
The bundle raised the bus as mesh-broker on docker volumes; the catalogue's nats module keeps it as
mesh-broker-nats on directories of the machine, so the broker step refused rather than raise a second
bus, and genesis stopped at step 18 of 24.
The bundle now raises mesh-broker-nats with its configuration and user list in /var/lib/nats-module/conf,
its streams in /var/lib/mesh-broker-nats and its certificate in /var/lib/mesh-broker-tls, published on the
node's bus port. The broker step holds that layout to the module's manifest field by field before
registering anything, says the one restart onto the module's image, and waits until the bus runs that image
and its monitor answers. The installer's bridge writes the composed users at the module's bus-users path and
signals mesh-broker-nats. The bundle's bus configuration and user list are seeds, so a re-run after the
adoption does not put the bundle's one user back over the module's.
mesh/delivery-group group feat/335-a-busy-container-is-replaced-when-idle rejected: its order contradicts itself: novox/mesh-catalog@190608193764, novox/mesh-c…
A send that replaced the build agent killed the check it ran, and the queue
started it again elsewhere (issue 594). A container may now declare busy; the
node-engine asks it before replacing it, applies everything else, says the
wait, and replaces it on the first idle pass or at the bound, kept across
restarts (novox/hq ADR 0335).
mesh/merge-gate head c7b534a08c98 merged into trunk 6fe82c61bd32 — pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes wi…
A module's own directories need setgid so what is made beneath keeps its own group,
and a mode of 2750 refused the whole declaration while every apply dropped the bit:
a directory's mode now carries setgid (never setuid or sticky), set, kept and said.
A tool runner's account that is missing or root's failed the whole apply and dropped
the library's outcome: it is now a warning, said as runner:<module> or
runner:library:<name>. And a runner that leaves a group is taken out before the
resources apply, so its unit is restarted in that apply to lose it.
A joining module's own directories belong to its own group, and its tools run in the
tool runner, which could not read them. An identity may now name the tool runner's
account: the engine puts it in mesh-<module> as it puts it in a library's group,
says the change as identity:<module> for the runner's unit to restart once, takes
it back when no identity names it, and refuses an account that is missing or root's.
The engine states contract 7.
A re-run's apply writes the bundle's one-entry user list over the list placed at
enrolment, and an enrolled machine skipped placing it: enrolled was taken for adopted,
against ADR 0331 §5. Until the controller knows the nats module, the installer places
the composed list again. Also drop the previous broker's AMQP and management ports,
which nothing raises any more, and say why the bus always carries a ports setting.
The installer could not raise a first node: the old bundle's control plane had no
MESH_BUS_NATS and crash-looped, and the NATS bundle was refused because rootsecrets
only knew the previous broker's guest:guest@ template.
One bundle now, the NATS one. rootsecrets makes the controller's bus account and puts
it in both placeholders (the address and the bcrypt hash in the carried user list),
kept with its hash so a re-run writes the same bundle. Preflight refuses a template
naming no bus connection it knows, naming the placeholder, instead of waiting out the
verify step. The broker step adopts the bus as the nats module and refuses, by name,
a catalogue without it or a server that is not the foundation's container.
mesh/merge-gate head 281837847e03 merged into trunk 303d4037e84e — pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes wi…
An earlier change dropped alpine's RemoveAccount and RemoveGroup, so alpine made no
module identity at all. The sample beneath a root in shape had no test of its own.
And an apply that found the library's lock held read the library as kept even when
what held it was a take act about to be refused for a missing disk; it now does so
only while a take walk is running, and otherwise waits and judges as always.
usermod --gid regroups every file under an account's home, and accounts were made
with / as home: they now have /nonexistent, and an older account's home is moved
there (without its files) before its group changes. A sub-path asked for later that
holds data not in the group waits for the operator as the root would. The take act
gives the group to hardlinked files an account owns (the *arr apps' downloads) and
leaves root's; a file's setuid or setgid bit the regroup takes is never put back, so
no setgid file hands out the library's group; each is counted. An apply no longer
waits for a running take walk.
A root in the group with setgid was kept without a warrant even when the data beneath
it was not in the group, which §7 says the operator is asked about. A root is now
kept without asking only when it is empty, or it and a bounded sample beneath it are
all in the group with group write (and setgid on folders). On busybox, an account the
mesh made is given its added groups with addgroup and delgroup; a change of its
primary group is refused, saying why.
ADR 0328 supersedes ADR 0325: shared data is a library with one home, kept usable by
its group rather than by ACL entries. The engine now makes each library's group and
each module's user in its libraries' groups; keeps a library's root and requested
sub-paths in its group with group write and setgid on its home, making missing
sub-paths as the root is; waits for the operator's word before touching a root that
holds data, and on the take act walks it by descriptor (no link, same filesystem,
bounded and resumed, files with other names left); holds the containers of a library
whose disk or mount is not there; and puts the tool runner's account in the group.
The ACL keeping, mask repair and watch are gone; an access is again as before ADR 0325.
mesh/merge-gate head 5c8e768ef520 merged into trunk ef26dba2753a — pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes wi…
getent passwd reads an all-digit key as a uid, so a tool runner's account named
"1000" resolved to whoever has uid 1000 and the entry went to them. Only an answer
naming the account asked for is taken.
The controller cannot know the operator's account's uid, names an access by its
declared id (which holds a slash), and should not parse words. A tool runner's
account may now come by name alone and is read here (refused when absent or root's,
its uid said in the access's listing); the take act matches the declared id exactly;
an access wait says its reason (keep or disk) and, for a filesystem that refuses
ACLs, whether the take act can enable them.
mesh/merge-gate head d12bba01090d merged into trunk dd9ab7927d96 — pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes wi…
The build seat runs merge-check as root, so every folder a test made was root's
data, which is never kept (§12), and 28 tests failed for that alone. A test's
folders now stand for an operator's: run as root, they are given to nobody before
they are judged. The root rule is unchanged.
mesh/merge-gate head c85c2804a010 merged into trunk dd9ab7927d96 — pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes wi…
mesh/repo-check head c85c2804a010 merged into trunk dd9ab7927d96 — fail: its merge-check.sh failed: --- FAIL: TestAnAccessTheMeshMayNotKeepYetIsSaidAsAWait…
The engine delivers before the controller half, and today's controller sends no
identities, users or taken. Keeping now starts only for an access whose folder has
users, or a declaration with identities; every other access is mounted when present
and refused when absent as on main, with nothing made, kept, held or judged. A
leftover whose walk could not run says why, not always that its disk is away.
A name read that timed out gave an empty name, which differed from the recorded one
and so reset the record: a folder kept while empty asked again, a warranted one was
walked again. An unnamed filesystem now leaves the folder exactly as it is, the
access is said not known for that look, and it resumes once it is named.
NFS and FUSE report a zero f_fsid, so another export or another FUSE mount at the
same point was named alike and kept without a warrant; xfs's f_fsid follows the
device number. A filesystem is now named by its UUID (its device matched in
/dev/disk/by-uuid), else a nonzero f_fsid, else mountinfo's source; one that cannot
be named is never kept or taken. Entries left on a filesystem not back within 30 days
are forgotten, said once, and the naming read is shared per mount.
An ext4 or xfs library mounted by UUID can come back from a reboot as another device
(/dev/sdb1 as /dev/sdc1); naming the filesystem by mountinfo's source read that as a
move, stripped the warranted library's entries and asked for a new Approve. A
filesystem is now named by statfs's f_fsid (from the UUID, or the ZFS dataset), its
type and its mount point. Entries left on the filesystem a folder is on now are
merged back or held, never walked away, and a TakenOn naming a mount point is not
taken.
A folder kept because it was empty, then covered by its disk, was said as unhealthy
leftover entries for ever: its entries sit on a hidden empty folder and grant nothing,
so they are now forgotten, and only entries walked onto data under a warrant are kept
to be taken out. A warrant and a kept folder now name their filesystem by mount point,
source and type, so another pool at the same point is a move; records naming a mount
point are read as its name. A record with no access ids is said under a path-free name.
A container was still created over an unmounted mount point, so its writes landed on
the filesystem below; a folder moved to another disk was kept on the old warrant; and
entries left by a drop or a move that could not run lived only in a log line. Every
container mounting an access whose disk is away now waits, the warrant holds only for
the filesystem it was carried out on (the wait names the new one, path-free), entries
left on a filesystem are recorded per mount and taken out when it is back, and every
leftover is said as unhealthy until it is. A hung stat is shared per path, and a
widened mask takes from other entries only the bits it adds.
A walk run while a dataset was unmounted wrote onto root's empty mount point, or
"dropped" entries from it and forgot the ones on the data; a folder kept while empty
below an unnamed mount was then swept whole once the disk mounted, with no warrant.
Every walk is now judged as an apply is, a changed mount judges the folder afresh, and
systemd .mount units and mounts seen are read for the first-sight rule. Widening a
mask caps the entries it limited, so the owning group gains nothing. The watch lets
go of removed folders, re-arms on lost notice, reads users under its lock and ends
with its context. A drop that cannot read an ACL is counted, a broken folder's words
carry no path, a grant's group that could not be left is tried again, and a folder's
stat is bounded.
A module's containers ran as whatever their image chose and shared folders were
made writable by putting accounts in the folder's group (ADR 0312), which ADR 0325
supersedes. The engine now makes each declared module identity as a system account
with the controller's numbers, runs a container declaring `identity` as it, and keeps
each placed shared folder usable through one POSIX ACL entry per user, set through
the kernel's xattrs on descriptors it opened itself. A folder holding data is kept
only after the operator's warrant (the take act); a missing folder is made where its
data lives and waits where its disk is not; root's data is never kept. A watch and a
bounded sweep put back what an app narrows. The group grant is removed, and what it
put an account in is taken back.
Reuses the descriptor walks, the create act, the missing-path wait and the warrant-
backed act message from the paused issue 567 branch.
mesh/merge-gate head c7a4bf5c11d6 merged into trunk d8e785bb4b4b — pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes wi…
Go's monotonic clock stops while a laptop is suspended, so a day judged by it could pass a day and a half
of wall time before the machine said what it is. And a hung statfs left a goroutine and thread blocked per
apply; now one check runs at a time and reports after it say the sizes unknown until it ends.
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
The mesh knew a machine's architecture and kernel family and nothing it is made of, and heard even
that only when it sent the machine something. The profile now carries the machine's memory, each local
filesystem's size and free space, processor model, cores and threads, distribution and version, kernel
release, virtual or physical, boot time and timezone; a fact it cannot read is named unknown with why,
never zeroed. The reconcile says the profile unasked once the mesh last heard it 23 hours ago, so on a
machine nobody pushed to nothing the mesh holds of it is older than a day.
The fix is an order, and nothing failed if the trust call moved after the
push or went: steps 7 and 8 are now one function a test drives against a
runtime that refuses an untrusted push. Preflight now refuses, before
anything is raised, a runtime file that is no JSON object, a docker.service
that is not loaded and active, and a dockerd started with
--insecure-registry, which would refuse the reload. The comment names the
order the docker module's takeover rests on.
Genesis pushed the control plane's image to the mesh's plain-HTTP registry
before anything told the container runtime to trust it; the docker module
that writes that trust is applied long after. Genesis now writes the trust
itself at the registry step, with the same into-json writer and a reload,
for the address given and the node's mesh name on the node's registry
port, and refuses before any push unless the runtime says it trusts each.
The docker module later finds the mesh name there and takes it over
without a change.
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
A mesh directory outside every home was chmodded and chowned by path, so a
parent turned into a link between two applies carried root's chmod and chown
to the link's target. It is now made and changed through the descriptor of
the walk from / that follows a link only root alone could have made.
A directory that walk refuses is a refusal like the placement guard's
(review 47): judged before anything is applied, and again when the
directory is applied, used as found or not; refused at either point it gates
its module and refuses what is below it, so no file is written into the
link's target and the report does not say everything was attempted.
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
Once a mesh directory's mode and owner go only through a walk that follows no
link another account could make, a directory behind such a link is refused
and its module gated: on the control-node that could stop the controller.
The operator is to be told which directories that walk would refuse on every
node before it lands. So the engine judges the declaration it keeps with the
same walk, making and changing nothing: on a signed ask from the controller
(mesh.node.<node>.ask.judge-directories, held to the hand-over's checks) and
as `nox-mesh-host judge-directories` at the machine.
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
The engine is built without cgo, so a check by name missed a root group a name
service holds or one number carrying another name: the root groups are resolved
to numbers (now with kmem and shadow) and a number named nothing is refused for
containers too. A group without write is granted nothing (§2). The look reads
each path off the lock within five seconds, so a dead network mount cannot stop
every health statement. A grant's account may not start with a dash. A group a
record leaves for another asker is handed to it, so it is still taken back when
the last asker goes (§5). The scheduler skips paths issue 339's guard refuses.
The engine reads the group owning the accessed path at apply and gives it to the
containers and the tools' account an access's new `grant` names: --group-add for a
container (recreated with it), an account group for the account (recorded, taken
back as ADR 0252 takes back). The path is never changed; an access whose group
cannot write is stated unhealthy for the operator to decide; group 0 and the
root-granting groups are refused by name. The tool runner restarts through the
restart-on the controller gives it, because the access reports updated exactly
when the account's groups moved. Health says GrantContract (4), which tells the
controller it may send `grant`.
The controller is to ask mesh-delivery whether a merge's delivery
group still has a member to merge, so that a merge in no waiting group
is cut at once (ADR 0304, ADR 0306); its bus grant names the verb, and
the installer's first user list must too, as for times (212a3cc).
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
A module-less action that stopped the apply after a module was gated was reported as
everything else attempted. The failure now names the gated modules and the stop, counts
skipped resources apart from applied ones, and the tests cover an archive below a refused
path, a gated module tried again by the next apply, and a refused file among the failures.
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
On ace the engine refused Docker's data root as not placed, said nothing was touched, then
wrote a file into it and ran the module's step. A refused directory now gates the rest of
its module as a failed step does, a file is refused in a forbidden tree or below a refused
path, and the failure names the modules whose rest was not attempted.
Both fail on main: the shape seen on ace, a refused directory, a file below it and a
step after it, writes the file, runs the step and reports that everything was attempted.
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
A newer sequence carrying an older view of the assignments undeclared four
assigned modules on 2026-10-04; the sequence cannot tell it, the generation
can. The engine keeps the highest generation it applied, refuses an older one
before reading the machine unless the declaration is a gate's put-back, and
says the refusal in its report so the controller can name the sender. A
declaration without a generation is applied as before, and every report says
it reads one, so the controller sends it only to engines that do.
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
A failure recorded nothing, so the record of the step's last success stayed and
the next apply skipped it, opening ADR 0136's gate on what it had just refused.
The record is kept, for removal, with what it was run as cleared.
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
A refusal waiting for its request parsed it into a 64 KiB reader for up to 5 s,
on a socket every account may open. It now drains at most 4 KiB through a small
buffer for at most 1 s and never past the accept-time deadline, only 32 wait at
once, and an unknown account is refused before it takes a slot. The busy tests
wait for the first line with a timeout and say what came of it (novox/hq issue 434).
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
The engine answered busy and closed before the client's request had landed;
the client's write then broke and it never read the refusal waiting for it.
Under the whole suite's load this failed the busy test (novox/hq issue 434).
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
A file's own details (lstat) may be read from the disk as a directory is,
so a directory of tens of thousands of files was read without a look and
the header's promise that no burst is longer than PaceLook did not hold.
The walk now asks the pacer before each file as well; the 250 ms gate
makes that a clock read per file. A test with one directory of twenty
files fails without it. The clock is read once after a run of pauses,
and the constants test says that it guards the constants only.
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
The review of #79 found that pacing alone brought back issue 361's
fault. The control-node's own I/O pressure sits near the 10 % bound with
no walk running, so a walk that paused 30 s for every 250 ms it read
would read 0.8 % of the time: a 16-minute walk would take some 32 hours,
outlive the age a search judges for, and be begun again at an engine
restart, leaving the machine unjudged.
A run of pauses is now at most nine times as long as the walk read since
the one before, so the walk keeps at least a tenth of the time whatever
the pressure, and takes at most ten times as long as unpaced: about
2.7 hours for the 16-minute walk, inside the 12 hours the controller
waits quietly. A test ties the bound to rootsearch.Quiet.
The pressure is the share of the time since the last look in which some
task waited on I/O, from PSI's cumulative counter, not its ten-second
average, which the kernel moves only every two seconds. A pause is
500 ms, then the pressure is looked at again.
The engine's walker is built in one place, engineWalker, and a test
fails if it is not paced by /proc/pressure/io. What the walk cannot read
is said through the search's own channel (the cache's Say), as the rest
of the search is, with what it means for the disk.
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
The walk for setuid programs is put in the idle I/O class so it yields
to everything else, but only a scheduler that schedules by class (BFQ)
honours that. The control-node's disks run mq-deadline, which does not,
so the walk read at full priority. On 2026-10-11 the walk started at
01:27, right after the engine applied a new controller, and ran sixteen
minutes; it kept two spinning disks saturated, the bus whose store
shares them stalled for half a minute twice, and the controller lost
its lease and restarted each time.
The walk now also paces itself by PSI, which reads the same whatever
the scheduler: before it reads a directory, while some task has waited
on I/O for more than 10 % of the last ten seconds, it pauses two
seconds at a time. It looks at most every 250 ms, pauses at most 30 s
in a row and then reads one more directory, so a machine that is never
quiet is walked slowly but still judged. Its own reads count in the
pressure, so on an otherwise idle machine it holds the disk below the
bound. Pressure it cannot read pauses nothing, and is said once.
A paced walk finds what an unpaced one finds; a walk ended while it
pauses is an incomplete walk, never one that found nothing.
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
A resource that passed, then waited, then failed once after its grace was
still stated healthy from the old pass; and a change of what it waits for
alone raised no change event, so the controller could miss it.
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
A module whose only fault was a secret the operator had not given yet was
stated unhealthy, failed its first-node gate and was put back. The engine
now reads the tool's answer through mesh-sdk go/health and states such a
resource waiting, with what it waits for, on one look; a malformed wait
stays unhealthy.
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
The two finds on the control-node sit in the operator's home of mode 0700, which the agent account cannot
search. The search keeps every find; each judgement reads the modes on the way again, since an account can
open its own home without root. A find in a home is said by count, its path in the journal alone. Also
from review: a checkpoint racing SearchNow forgets the walk it just kept, and a walk kept by an older engine
says its counts as lower bounds.
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
The walk kept its progress every minute and nobody could read it: root-free said only that a search ran.
The account verdict now carries the progress (counts, times, mount points; never a path) as fields and in
its reason, outside what makes a verdict changed. A signed ask from the controller's terminal throws the last
search away and starts a full one, after the operator removed by hand what it found.
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
The review found a cursor of a name that is not UTF-8 came back changed
from JSON, so a resumed walk skipped what it stood before; an ask to
search again was lost to a restart, which carried the older walk on; a
fresh search judged a filesystem it never walked; a mount point root
cannot read failed or hung the walk; a found file gone before the package
manager was asked failed every resume; and skipped or kept outcomes
started a search on every reconcile, back to back with a long walk.
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
The control-node's walk takes far longer than the engine runs between
updates, so a search that starts over at every restart never completes.
The walk is the engine's own, in lexical order, so its progress is one
cursor per filesystem, kept every minute beside the engine's state and
carried on after a restart while younger than rootsearch.FreshFor. It runs
on a thread of its own at nice 19 and the idle I/O class.
A machine with millions of files never finished the walk within fifteen
minutes, so its agent account was never judged and the router refused to
link. Only root can make a setuid-root file, so a complete search stays
sound until root acts: it now runs at idle priority with no bound, is kept
beside the engine's state, judges for rootsearch.FreshFor from its start,
runs again daily and after an apply that moves accounts, packages, actions
or sudo rules, and an incomplete or stale search is never free.
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/delivery covered after its failure: a later walk that contains it was delivered: plan-1791657606681868190 (novox/mesh-host at 9d8aa9d6 into main, do…
SetuidCache compared the caller's now with a time read from the wall clock when the search finished. When the caller's clock ran behind the wall clock, a search was never due again: TestTheSetuidSearchRunsInTheBackgroundAndIsKeptForItsInterval, whose clock is a fixed date, has failed on main and every branch since 2026-10-09 18:00 UTC. A search's time is now the now it was started at; a result reads at most one search older than it is, which errs towards searching again.
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
After the first apply the record is no longer used as found, so the directory read as the mesh's own and later applies set its mode and owner by path: a parent swapped for a link after the hand-over took carried root's chmod and chown to its target. The hand-over is now checked before that shortcut, so a handed-over directory always goes through applyHandedOver. The test swaps the parent and the grandparent after an apply already gave the directory its mode; both cases fail with the old order.
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
Two holes the review proved. The ask's subject proved nothing — a responder can reach it through a reply — so the engine now verifies the mesh's signature over a fixed context with the key it verifies declarations with, before reading anything, and refuses an ask for another node, expired, good too long, without a nonce, or replayed. And the apply followed a link the hand-over had refused: a component swapped for a link after the record carried root's chmod and chown to its target. A handed-over directory must now still resolve to itself, and is changed only through one descriptor opened from / with no link followed (fchmod, fchown). Tests for the reply guard, the run loop's hook, the fallback renewal on proof and a non-regular fallback; 27 checks each removed in turn, each failing a test.
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
The file on PATH was the engine a person copied at installation, and every delivered engine ran from the versions directory: the operator typed hand-over and a build from before it answered "unexpected argument". A proved engine now copies itself over that file when the bytes differ (never a link). And the hand-over of a directory used as found (issue 339) gets its channel: the controller's terminal asks this node's engine on mesh.node.<node>.ask.hand-over; the engine judges every value — the ask's size and shape, who asked, the path as it states it, a directory and no link, through no link — and records it or refuses and records nothing. Every text the engine addresses to the operator names the nox line (ADR 0272); the engine's own subcommands are install-time and rescue, and its usage says so.
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
A secret given at the controller's terminal through mesh-cli (secret accept … --from -) never reached the
line: the engine forwarded the words only. It now forwards what mesh-cli's standard input held (at most
64 KiB, the SDK's bound) on the same core request to the controller, says it nowhere, and sends none
with a follow of the same call. mesh-sdk moves to f047d0a.
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
- UID, CLASS and SCOPE are read at their first occurrence: logind writes
each once, and a later line saying otherwise is not the record's word.
- A comment says /run/systemd/sessions is systemd's private format, and
that a format change fails closed: no session, so nobody is the terminal.
- Tests that hold, each proved by the mutation it guards failing:
- a record naming UID=1001 for session-8.scope under user-1000.slice,
asked as uid 1000 (the record's UID check alone);
- session-a.b.scope with a passing record (the id's characters alone);
- session-9.scope/x.scope (the scope being the slice's direct child);
- a key said twice, either way round (first occurrence);
- the client sends its line only after the session was read, so reading
the session in the forward fails the test.
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh-controller #156 (give) grants the controller mesh.seat.node-launcher.tool.secret.*; the
controller's test holds the installer's list equal to what it composes, and failed without it.
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/delivery covered: a later merge that contains it was delivered: novox/mesh-host@4ba6d0864e3d (merged as d8ff1540 into main, walk plan-17915554870866…
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
The cgroup names a scope; only logind's record of it says whose session it
is. The engine now takes a session only when /run/systemd/sessions/<id>
says it is this account's (UID), of class user (not a greeter's or the
user manager's), and of this very scope. And it reads the session the
moment the connection is accepted, with the namespaces, before the line,
so the peer is judged as it was when it connected.
Tests: session-9.scope with a user record is taken; x.scope, a different
uid, another account's session, a greeter's, a record naming another
scope, no record, and a path with dots are not; the session is read
before the line is forwarded.
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
A service running as the operator — the tool runner, a unit of the
account's user manager — shares the operator's user id, and was as much the
terminal on the control-node as a person typing. The engine now reads the
asking process's cgroup and sends its login session's scope
(session-<id>.scope under the account's own user-<uid>.slice), or nothing;
the controller makes nothing else the terminal. The SDK is pinned at its
tag now that go/v0.1.9 exists.
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
- Any failure of a request the link may have delivered now says the line
may have run; "nothing ran" is said only of a request the link knows it
never asked (no link open, nobody listening: link.ErrNotAsked).
- A test holds that a connection saying nothing is answered within its
deadline, set at accept, and gives its slot back.
- The README says the socket refuses a process outside the machine's own
namespaces, so flatpak, toolbox and distrobox shells are refused.
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
- A request that timed out was said as "the controller was not reached,
nothing ran", though the line may have run. The engine now follows a line
the controller answers as running, by its call, until it ends; when no
answer comes in time it says the line may have run and names the call.
"Nothing ran" is said only of a request no controller took.
- One account could hold the socket: accepting waited for a free slot, and
a connection could stall before its deadline was set. The deadline is set
at accept, before any lookup; each user id has two lines at once and the
machine eight, and one over is answered busy at once.
- A container running with the operator's user id shares the number, not
the machine: a peer whose mount or user namespace is not the engine's is
refused, and nothing is asked.
SDK pinned at the go/cli branch's newest commit until go/v0.1.9 is tagged.
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
The homes come from a map of the user database, and the test compared them in
the order it happened to iterate: it failed one run in four, the repo check
of 2026-10-09 among them.
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
The search kept a two-minute limit of its own inside its 15-minute bound, so
a long walk always ended incomplete; pacman was asked once per path; and find
exiting with an error but some output was read as a complete walk. The bound
and the not-judged reasons move to the public rootsearch package so the
controller reads the same values (the review of 2026-10-09).
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
The records send the operator to "the controller's terminal", and nothing
reaches it (hq issue 343). mesh-cli holds no credential: it asks the engine
on its own machine over a socket. The engine reads the asking account from
the kernel (SO_PEERCRED), never from the program, and asks the controller on
mesh.control.<node>.cli, its own node's subject, so the bus server says which
node asked. The controller decides what the line may do; nothing received
on the socket is run or applied here, and the socket has no network address
(ADR 0004 stands).
The local protocol is the SDK's go/cli package, at a pseudo-version of its
branch until that merges and is tagged go/v0.1.8.
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
A search over a large disk ran inside the look, holding the judge's lock and
stalling the health statement; it now runs apart, serving its last result,
backing off after a failure, and saying not judged until it has one. And a
rule naming a user, or a comment, no longer hides another rule's
unconditional yes (hq ADR 0266).
Issue 339's guard (#63/#64/#65) now refuses the machine's own trees, another account's
directory below a home, and uses a found directory as found; this branch's own copy of
those rules goes. What stays is the one rule only an account that never becomes root
needs: below its home it owns every parent and could swap a link in between the
engine's check and an unpack or a write-into (hq ADR 0266 §7). One reader of the user
database serves both.
A hard link swapped in for ~/.claude would have had root chown another
account's file; fstat on the descriptor now refuses a second link, a fifo or
an unexpected kind before anything is changed (hq ADR 0266, the re-review).
The judge also finds polkit rules for every account, a runtime's API on TCP,
setgid-to-root programs whoever owns them, setuid programs on every suid
filesystem, and unprotected links; the rest is listed as not judged.
The controller cannot read a machine's user database, so a name it is given
for the agents' account (hq ADR 0266) may be a service's own; taking it over
would hand agents that service's files. Refuse it, touching nothing.
An account could replace ~/.claude with a link to /etc and have the
node-engine chown, chmod or write through it on the next apply. Below a
person's or an agent's home every component is now opened without
following a link, and a link refuses the resource in words.
The root judge also reads doas and polkit rules, the container runtimes'
sockets with their ACLs, ACLs on the secrets, and setuid-root programs no
package owns, in the C locale; Judged and NotJudged write down exactly
what it covers (hq ADR 0266 review).
An account an agent runs as (novox/hq ADR 0266) is read on every look for a
uid of 0, a group that grants root, any sudo rule and a mesh secret it can
read; any way found is unhealthy and said, a read that fails is unknown.
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
#64/#65 and #66 each added Applied.Declared: a directory's declared owner and
mode (string) and a service's held state (*FoundUnit), both under the JSON key
"declared". The directory's is already on nodes, written by d28ca95, so it
keeps the name and key; the service's becomes DeclaredUnit, "declared_unit".
#66 never deployed, so no node holds a record under the new key.
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/delivery covered after its failure: a later walk that contains it was delivered: plan-1791657606681868190 (novox/mesh-host at 9d8aa9d6 into main, do…
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
Third review of mesh-catalog #147.
A unit's name can now come from a setting (a template's instance), so every systemctl call that names a
unit passes `--` before it, and a name can never be read as an option.
On removal, a unit is started or enabled again only when it is exactly the stopped and disabled the mesh
held it to. A masked unit, or one somebody started or enabled since, is left as it is, and the report
says why. A restore that fails (the start refused, or the unit not running after it) puts back what this
attempt changed and is reported as waiting, so no apply is made fatal by it and the next one tries again.
The tests' fake service managers read systemctl's arguments in the shape they were written for
(oldShape); internal/system's own test holds the `--`.
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
The fourth review (hq issue 339): `trusted` is the catalogue's word and the
controller takes it out, but a controller that passes it on, an older one or
one rolled back to, must not cost a node its whole declaration. A file now
accepts it and nothing reads it; any other resource still refuses it. And a
podman that hangs no longer eats docker's ten seconds.
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
A unit found running and enabled that a declaration held stopped and disabled stayed stopped and
disabled when the declaration went: ADR 0118 never starts a unit on the way out. Unassigning the zfs
module would so have left the home server's pool with no scrub schedule at all, said nowhere.
The host now records what it declared for a service (Declared, beside Found). On removal, a unit found
running and declared stopped is started again, and one found enabled and declared disabled is enabled
again, each read back; the report says restored, and the plan says it first. A record from before the
field starts nothing.
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
The third review of #64 (hq issue 339): an ExecStart continued over lines, a
quoted path, a variable from Environment= or EnvironmentFile= across the unit
and its drop-ins, dockerd's --config-file and podman's TOML literal strings
each hid a data root from the guard. The running runtimes are now asked first
(docker info, podman info) on a runner of the guard's own, and the files are
read beside them in all those forms.
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/delivery covered: a later merge that contains it was delivered: novox/mesh-host@7c046a8a1d84 (merged as d28ca959 into main, walk plan-17915049347305…
The review of #64 asked for containerd's tree, dockerd's --data-root from its
unit and drop-ins, and podman's graphroot to be refused like docker's own
(hq issue 339); and the controller now tells a directory a send found from
one found before it, which needs the time it was first found kept across
applies instead of the time of the last apply. A hand-over now names the
resource it was for, so it lapses if another declaration points at the path.
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
A directory the engine used as found kept its owner and mode in silence, so
a module that cannot write it would just fail (hq issue 339 review). Each
one is now a WARNING in every apply's log and its module's resource of kind
directory, stated unhealthy on every look with its owner and what was
declared, which the controller raises as the module's condition; `mesh-host
hand-over <dir>` at the machine hands it to the mesh, and the next apply
gives it the declared owner and mode. The guard also refuses the spool, /opt,
the container runtimes' data (its configured data-root too), and an account's
.ssh as an access, a mount, or a directory anywhere but its own home.
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
A module's places setting could point a directory anywhere, with an owner it
named, and the engine chowned and chmodded whatever it was pointed at as root,
through a link too; its accesses setting could mount any path into a container
(hq issue 339). The engine now refuses, as a failed resource with its reason,
a directory, access or mount source that is or holds one of the machine's
roots, anything placed in the kernel's, the boot loader's, root's or the
engine's own trees, a non-root directory below /etc, /usr or /run, an access
there, another account's directory below a home, and a container mounting
anything it refused, judged with every link resolved. A directory whose last
component is a link is refused, and one found here that the mesh did not make
keeps its owner and mode and is never removed.
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: THE CHANGE ALTERS ITS OWN CHECK (merge-check.sh): main's version judged it; the change's judges the pull requests after it merges; it…
The evidence read "asking it: asking <subject>": the words are now the link's own.
A deadline is said as the time the check gave it. A refusal is said only when the bus
refused this question, not an earlier one under a grant since widened. merge-check.sh
runs the tests that need a bus against a throwaway nats-server when the toolchain has
one, and says so when it does not (hq issue 331).
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
The queue asks a tool through a ToolBus assertion that only OverNATS satisfied; the
link the engine actually holds (natsLink) had no Ask, so every tool check failed as
"no link to the bus is open" while the link was up (hq issue 331). Give natsLink Ask,
assert at build time every optional interface the queue expects of it, and tell a link
that cannot ask apart from no link.
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
An APU's integrated graphics and Intel's second (MMIO) view of a package made a machine
report a power meter the sampler would not read, so its bar could only say not
measured.
The power module is on every machine and its bar blocks apply only where the
hardware is. Two capabilities read from the kernel let the controller compose them
there and nowhere else: battery (a Battery supply not scoped to a device) and
power-meter (a system battery, a powercap package counter, or a graphics device
reporting its power).
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
A module puts the operator's account in a group by declaring the account with that group alone.
The node-engine now records each group it added, takes back only those when nothing declared still
asks for them, refuses a group the machine lacks before usermod runs, and states each such account
as its module's resource of kind account: relogin needed while the running session lacks the group.
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
Liveness judged only what a module runs long-lived, so a module whose
daemon is a package's unit started by D-Bus activation failed at every
start while its machine read healthy, and a degraded service manager was
said by nothing but the profile.
The engine now reads the failed units of the machine's manager and of
every account manager the declaration names, on the two-look rule, and
says whose each is: a declared service or process, a unit file the mesh
writes, or a package the mesh installs makes it that module's, said as
an unhealthy resource of kind unit; anything else is the machine's own,
said in the statement's new units field. It reads; it never acts.
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
The installer placed them between making the enrolment's deadline and
using it, so a failure returned without cancelling it, and go vet refused
the package.
Issuing the token sends the hub its new peer, and the hub applies it on
its own time; a bus dialled before then timed out naming the bus. The
first tunnel now waits for a handshake with the hub, and says so in the
tunnel's words when there is none (novox/hq ADR 0169).
At genesis the bus's users reach it in no declaration, because the
machine running it has not enrolled. The installer, which raised the
bus from its bundle, places the control plane's composed list beside it
and makes it re-read it: before the machine enrols, for the token's
account, and after, for the node's own (novox/hq issue 146).
nox-mesh-host key makes the tunnel key, or reads the one made, and
prints its public half for the token to be issued for. enrol with a
token that carries a tunnel takes that key, refuses another, writes
mesh0 with the hub as its one peer and starts it, then reaches the bus
over it (novox/hq ADR 0169). Tokens without a tunnel enrol as before.
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
The judge's tests run against files and fakes; the drill runs it against a
real resolver file and real resolvers, rewritten as the VPN client does,
and records the statements the controller's replay raises and clears from.
A VPN client rewrote the laptop's resolver file and every mesh name failed
while each module read healthy: nothing asked the machine. The engine now
looks every 30 s at the resolver file the uplink holder declared (naming
the program that rewrote it), the names through each listed resolver
(NXDOMAIN for a mesh name's IPv6 address is a finding, issue 262), the
tunnel's handshake with the hub, the bus and the default route; a part is
unhealthy on its second failing look, and the statement carries it.
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
Liveness alone could not see a web application whose port was open and whose
program ran while every request hung for eleven hours (issue 145). A resource
now carries the `health` its module declared: the engine makes http and tcp
looks itself from the machine to the endpoint's published port, reads a unit's
readiness from the show it already makes, hands an exec command or the image's
own check to the runtime as the container's check with the declared timing and
reads its state from the inspect it already makes, and asks a module's tool on
its own node tools. Starting until the check passed, unhealthy once its looks
after the grace fail the declared number of times; never more looks than the
measured budget; nothing restarted. The statement says contract 2, which tells
the controller this engine may be sent the field.
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
At 03:30 the store's collector held the registry still while the
node-engine's reconcile on that machine was fetching bundle blobs from
it: every archive failed 'connection refused' and the machine was held
until the next pass. Other machines can meet the same window.
A scheduled step now opens its window only once no apply is in flight
here (the apply lock is taken just to write the record, so a push still
never queues behind the window). An apply whose fetch the store does
not answer waits for a window open on its own machine to close, and
elsewhere retries with backoff within one bounded budget per apply,
well past the window's length; an answer such as 404 still fails at
once.
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
The witness moved a running controller's build into a 0700 directory and deleted it
on proof, while the old process could still be serving. Its directories are now
0711, and a build without a reader is retired and swept once /proc shows nothing
runs from it.
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
A container that crash-looped after its compose applied passed every check the
gate had: nothing looked at what a module runs. The node-engine now judges every
long-running resource on every look — one read of the runtime, one per service
manager — keeps the restarts it counts across recreates and its own restarts,
and says the state in every report and as an event on change, again every minute
while not healthy. It reads only; nothing is restarted for being unhealthy.
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (0 of 4 compose)
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (0 of 4 compose)
The toolchain the build seat runs merge-check.sh in (Go 1.26) and a newer local Go format a return of
several multi-line composite literals differently; the seat's is the one that judges, and it failed every
pull request on this file (novox/hq issue 283).
mesh/merge-gate pass: builds mesh-host → ace, g14, novox, shanks; no bus step; every machine composes with the change as it did without (0 of 4 compose)
mesh/repo-check fail: its merge-check.sh failed: internal/bootstrap/publish_test.go
The build seat now runs the gate itself, judging a node-engine change with the change's
validator in the running controller because the module graph builds mesh-host from here.
This script is the repository's own layer (mesh/repo-check): format, vet, and the suite
under the race detector when the toolchain can.
merge-check.sh runs the suite, then builds the controller the mesh runs with this change's
validator in place of the one it vendors and runs the merge gate: a change to the node-engine
that would refuse what the mesh sends today fails its pull request, naming the machine. The
installer's first user list lets the controller say a check's verdict and hear a pull request's
head, as the controller now composes it.
The controller says a rollback as rolled-back and asks the bus machine's backup holder
for a snapshot before the planned bus step; the installer's first user list must grant
both, or a fresh mesh's controller is refused the first time it uses them.
The launcher trusted a counter only a by-hand reconcile ever cleared and a
known-good nothing in the daemon wrote, so no machine could roll its host back;
the controller and the node tools were replaced in place with nothing kept.
- The launcher runs a delivered host that is not known-good on trial: one that
crashes, stops for nothing, or does not report within ten minutes goes back
to known-good, once per version, recorded in rolled-back. The host proves
itself when the mesh takes a report under its own build, says every standing
verdict on its reports, never stands aside for a rolled-back version, and
restarts its service once when its launcher was replaced on disk.
- The engine keeps the controller's and the node tools' previous build beside
the new one and judges the new one: the lease taken by the controller it
started (read-only direct get of mesh-controller_lease/holder), or this
machine's runtime answering $SRV.PING.node-tools.<node>, within sixty seconds
of time it could ask. Not healthy: the previous restored, once, said. Proved:
the previous deleted. A build declared not-reversible is never rolled back.
- Retire never removes a version newer than the running one.
The installer's first user list must match the controller's composed grant, which now reads every
machine's backup holder; and a kept directory's report pointed at the one act that loses data.
The controller now derives a subscription to provisioner.retirement; the first
user list the installer carries must match it, or the genesis controller is
refused the subject its composition expects.
The controller's healer H1 answers a send that went unreported by asking the
machine first: a report lost on its way (issue 264) needs no second send. The
node-engine now hears mesh.node.<self>.ask.report on core NATS and enqueues a
reconcile whose account is said whether or not it is news; a delivery waiting
meanwhile is applied and reported instead. The answer is an ordinary report on
its own subject, so the node publishes nothing new and answers nobody's inbox.
The genesis lock grants the controller the healer-acted seat event, which it
now composes; the genesis test in mesh-controller holds the two equal.
The controller now composes a publish grant for its lease bucket
($KV.mesh-controller_lease.>), which it must write before it acts, and for
the seat event secret-replaced (hq ADR 0228, mesh-controller #82); and it no
longer publishes mesh.control.>: a machine's report has one writer, its
node-engine, and the writers table refuses a second at composition. The
installer's first user list must say what the controller derives, or a new
mesh's first controller is refused its lease and serves without one.
A delivery and the five-minute reconcile were two paths that applied, ordered only by a lock, and
each order it allowed was met live (issues 257, 261, 267). Now both only enqueue: one worker takes
the newest declaration held when it starts, applies it once and makes one report, and reports leave
in the order they are made.
A declaration may carry the controller's lease epoch beside its sequence; one older than what this
node applied is refused before anything is touched, counted, logged and reported. A report carries
the declaration's epoch and sequence and the host's own report sequence, kept on disk so it goes on
increasing across restarts and self-updates. Without an epoch, today's behaviour stands.
The controller now composes a publish grant for
mesh.seat.node-intrusion-prevention.tool.banned.*, which D8 asks every
machine; the installer's first user list must say what the controller
derives, or a new mesh's first self-check is refused it.
The controller's watchdog of a machine's heartbeat (S1) is bound to three
of its intervals, and a bound the controller guessed would not move when
the interval does: the heartbeat now carries interval_seconds.
Its self-check (D1) must judge every composed declaration as the host
does, and a second validator written from the host's rules would drift
from them: the host's own parsing is exported, unchanged, as
github.com/novox/mesh-host/validate.
The installer's first user list grants what the controller now composes
for itself: its condition buckets, the condition and doctor-heartbeat
events, the bus's two consumer advisories and $SRV.INFO — or the first
controller would be refused them until the broker's machine is pushed.
The controller keeps its calls and the hand-act log in two key-value buckets
of its own, and composes a grant to write them. The installer's first user
list must say what the controller derives, or the first controller writes
nothing until the broker's machine is pushed.
A reconcile that held the machine to the kept declaration just before a
delivery arrived queued its report while the delivery's apply waited for
it; the link published the apply's report and then the reconcile's, so the
mesh's last word from the machine named the older declaration and the
release plan waited on a report it had already been given. The link now
sets aside an unasked report about a declaration other than the one it
has applied since.
A host that delivered its own successor stood aside before the report of
that apply was published, so it failed with 'context canceled' and the
release plan waited for a report that never came (novox/hq issue 264).
Reports are now published on a context the stand-aside does not cancel,
and the last apply's report is kept until the broker takes it and said
again on the next link, so a crash between apply and report is covered too.
The carried user list must equal what the controller derives (its test reads
this file): add the provisioner.failing/recovered subscriptions (hq ADR 0224)
and the build seats' tool publishes it already derived (hq ADR 0219).
The resolver file moves from resolv-conf to the uplink's holder in one apply.
Orphans go first, so the file was deleted or given back its pre-mesh original
until the new owner's turn came. Now the old record is forgotten once the new
one is recorded at the same path, and the kept original goes with it.
Removing one record of a unit gave back what that record found, even while another declared
service holds the unit: when the private network's docker.service record goes and the docker
module's stays, a record that found the runtime stopped would stop it, and every container with
it, only for the docker module to start it again in the same apply. The record is forgotten
instead, and the plan says so. A test pins the registry member moving from the network's record
to the docker module's in one apply without leaving the list.
A service was restarted where it was declared, so one declared ahead of a
file in its restart-on was restarted before that file existed (the
resolver's zones file, a fact appended after its module's resources), and
a later change to such a file was never acted on. Each service is now
applied right after the last resource it names under restart-on or
reload-on; nothing else moves.
The five-minute reconcile read the kept declaration and then waited for the link's apply; the
link keeps a declaration only once its apply ends, so the reconcile applied the older one over
it. A module assigned a moment earlier was given back, and the report named a declaration the
mesh no longer recorded as sent, which held a plan at its first machine.
A while-stopped step stops its module's containers, and an apply arriving
mid-window read the stopped server as broken and recreated it running under
the step - for the store's collector, a registry taking an upload the sweep
then deletes. The scheduler now records each window under the node's state
directory before its first stop and erases it after its last start, so every
apply on the machine (daemon, reconcile by hand, installer) reports a held
container held-still and leaves it for the first apply after the window.
A window whose host died, or past six hours, holds nothing; the report says
which windows are open.
The host keeps the original of a file before writing over it (ADR 0102), but
removing the file's record deleted the file and never put the original back,
although ADR 0118 and the comment on meshMadeUnits say it does. A module writing
/etc/pacman.conf, logrotate.conf, locale.conf or vconsole.conf whole would, once
unassigned, leave the machine without the file.
removeWhole now decides, in order: no kept original (the mesh made it) is
removed as before; a file gone since is not brought back; a file changed since
the mesh last wrote it is left as it stands, as a block or JSON write-into stays
the machine's; an unreadable kept copy leaves the mesh's file in place. Otherwise
the original goes back atomically with the mode and owner it was found with,
now recorded beside Kept, and the outcome is "restored". None of it is fatal.
The plan says "restore" for such a file.
A kept original is carried only for the path it was kept from, and a file whose
path moved keeps the original at its new path first, so a moved file is never
given another path's original.
An account's manager runs only while it is logged in or lingers. A user-scoped unit
whose manager is not running is now "waiting" rather than failed, its record kept as
it was; its removal is never fatal (kept recorded, retried) and an account that is
gone is forgotten. Whether the manager runs is asked of user@<uid>.service in the
machine's manager: asking the account's own, through --machine, logs it in.
The user shape gains `linger`, set with loginctl, read back from logind's record,
and given back on removal like the shell. Unit files the mesh writes under
~/.config/systemd/user or /etc/systemd/user make that unit the mesh's, and made,
holds and found units are keyed by manager and name, so an account's unit and the
machine's of one name are two units. A service moved between managers gives the old
one back through the manager it was in. OpenRC refuses both.
A workstation's per-user daemons — a window manager's reload watcher, an
audio mask, a memory guard — are units in the operator account's own service
manager, and until now had no form the mesh could send (to-be 29). The
`service` shape gains `scope` ("system", the default, or "user") and `user`
(the account, named ${machine:account} by a module); a user-scoped unit
without an account, or a system unit naming one, is refused at parse.
The host reaches the account's manager as `systemctl --user --machine=<account>@`
from its own process: no environment to forge, no user to switch to. Done on
the runner rather than per system, since every system's reading of a unit
already goes through systemctl. Apply, reflect-only and removal all go through
the same manager, and the applied record carries scope and user so removal
gives the unit back to the manager it came from. It answers only while that
manager runs — a login, or lingering enabled for the account; declaring
lingering is a follow-up.
Tests: a user-scoped unit is started and enabled in the account's manager and
recorded with its scope; a system unit never sees --user; the validation of
scope and user.
An archive had no removal, so an unassigned one failed as an orphan and
aborted every apply after: a module with tools could not be unassigned,
and a race between two pushes froze a machine against every change.
The record now keeps what an archive unpacked: its files, the
directories the host made inside its path, whether the host made the
path itself, and the parents it made to reach it. Removal takes exactly
that away, directories only once empty, never one that was there
before; a directory that is the host's alone is renamed aside first so a
reader sees the whole bundle or none of it. Whatever cannot be removed
is said and forgotten, never fatal.
A directory found before the archive is no longer swapped away with
what was in it: the archive is moved in file by file, and one that would
write over a file the mesh did not put there is refused before anything
moves. A record from before this change learns its files from the
archive's bytes on the next apply; one already orphaned is left in
place, said and forgotten. A former target is still left in place: the
version before is what a rollback starts (ADR 0141).
A file or archive placed under a fresh account's home with an owner left
the parents it created, such as ~/.config or ~/.local/share, owned by
root, so the person's own programs could not write there. Parents that
already existed, and any outside the owner's home, are left as before.
A user had no removal, so an undeclared one failed as an orphan and
aborted every apply after. Removal now keeps the account, gives back
the shell recorded when the mesh first changed it if it is still the
mesh's and still usable, and says why otherwise (hq ADR 0176 §2).
A shell is refused before it is set unless it is executable and listed
in /etc/shells, since usermod succeeds on a missing one.
containerSpec says a changed cadence moves the marker so the install is
reported updated and re-established. Which containers are held still is the
same kind of statement, and a declaration that changed it while the machine
reported no change would be a machine quietly holding yesterday's containers.
while-stopped names resource ids of the same module's containers; the host
stops them before the run and starts them again after it, in reverse order,
whatever the step did. The restart is deferred before the first stop and runs
on its own context, because the one real risk of this field is a window that
never closes.
Scheduled steps only: at apply the declaration is applied in order and a
run-once step already gates what follows.
A user that does not exist yet was matched as Go's 'exit status 2', while the host's runner says
'getent exited 2', so it read as a user database that did not answer: the controller's account was
never created and the handover to its process stopped there. The runner keeps the exit underneath
its words, and a caller asks the code.
The controller's manifest now declares a Go bundle the host runs as a
process (novox/hq issue 213). Genesis cannot run that: the bundle is
fetched from the artifact store and compiled in a toolchain, and the mesh
makes both long after the controller. The builder, asked to build the
manifest at genesis, refuses for lack of the Go toolchain. So genesis
raises the controller as before, as a container, and the first push hands
it over to the process through `replaces` (issue 223, option b).
- Step 3 clones the controller at the commit with the carried builder's
git and reads its manifest. In the image form (an older controller) it
builds through the builder as before. In the process form it builds the
repository's own Dockerfile and hands step 9 a manifest of its own
shape: the process becomes a container with the id the process
`replaces`, the image genesis built, host network, and every host path
the process's env names mounted at that same path read-only. Secrets
belong to the image's user (65534) until the process's account takes
them over. `prepares` is dropped: the temporary controller from the
same commit already migrated the stores, and a pinned image is
nothing the controller can derive a step from.
- Steps 4 to 9 are unchanged: they take the manifest as they did.
- apply.ForTests lets the bootstrap's test apply a process.
The first composed declaration from the process manifest names
`mesh-controller.server`, which is what the host recorded for the genesis
container, so the first apply hands over and leaves one controller.
The controller moves from a container to a process on the one machine
that runs it (novox/hq issue 213). Every orphan is removed before anything
is applied, so the container would go first and nothing would answer the
mesh's verbs while the process was fetched, unpacked and started — and
never again, if it did not start.
- a process may say what it `replaces`: resources the declaration no
longer declares. Such an orphan is kept through the up-front sweep and
removed right after the process applied and is up: active and running
at two looks ten seconds apart, the same main process, no restart in
between (stricter than ADR 0184's second look, which reads a unit
waiting to restart as running). If the process failed, was skipped
behind its module's step, or is not running, the orphan stays running
and recorded, reported kept, and the next apply hands it over.
Refused: naming something still declared, itself, an empty id, one
thing named by two processes, and `replaces` on a step or a schedule.
- beyond #85's oneshot unit for a step: a step written ./name runs its
own bundle's binary (tested), and is started, never enabled.
- a run-once process that fails gates its module, as a run-once
container already did, so a version whose preparation failed is not
started.
- an unchanged run-once process is not run again, and an unchanged
scheduled one is kept up by its timer: both were "a daemon that had
stopped" and were started on every apply.
A step was run directly: in the host's own working directory, without its env, env files or
user. A module step moved out of its container (node bootstrap/index.js) could find neither its
code nor its words. A oneshot unit carries all four as a daemon's does, and starting it waits.
Unpacked over the previous tree, a file the new archive no longer has stayed: a bundle rebuilt as
one file per entrypoint kept the old package directory. Unpack into a fresh directory and swap it
in, so a refused archive also leaves the old tree whole.
The controller announces the mesh-controller seat on $SRV.PING/$SRV.INFO from genesis; the
carried account is the composed one, which the controller's test compares.
A bundle compiled to a binary runs itself, but only the host knows where it unpacked it, and the
service manager takes no path relative to the working directory. A command written ./name is that
file in the process's own unpacked bundle, made absolute in the unit.
The process applier's unchanged path returned an outcome that said nothing about what was
written; the loop recorded it like any other, erasing the digest. The next cycle found no
record and re-created the daemon, the one after found a record again, and so on: the
node's runtime restarted every ten minutes on every machine since it arrived. The outcome
now carries the digest forward, as a file's does. The test applies one process three
times and asserts the record survives an unchanged apply and no restart is asked.
pacman writes its errors to stderr, which the runner folds into the error rather than the output;
the stale-index classifier read the output alone and never saw a single 'failed retrieving file',
so a ten-week-old database on the control node reported as a wall of 404s from the mirrors. The
classifier now reads everything pacman said, knows a failed signature as the same staleness, tells a
mirror outage with a fresh database apart from it, and names the database's date and age beside the
remedy: a full upgrade by the operator, never a one-package sync, which on this distribution is a
partial upgrade. Whose job keeping the database current is stays issue 205's question.
Until the retired mesh-build-machine row is dropped the controller asks and hears both roles, so
the genesis list grants both; the controller's own test of this list says so.
The controller composes the build seat's subjects as node-build-agent's now; the installer's
genesis list must say the same, or the controller's own test of that list disagrees with it.
The legacy reader required every path into a chain of refusals to come from a built-in whose policy
accepts. The home server's ban chain hangs off the container runtime's user chain, whose forward
policy the runtime set to DROP, so the machine reported the mesh's own intrusion prevention as a
rule set the mesh did not write. A chain is a ban when every refusal names its sources and the
chain accepts nothing — the rule the nftables side already used. A chain that accepts anything is
still not a ban. Fixture captured from the machine. The citations for the uninstalled front end move
to ADR 0180, which another session's renumber had left pointing at an unrelated record.
The read-back raced the failure: a service manager returns when it has started the process, and a
daemon that refuses its configuration exits a fraction of a second later, so one look saw it alive.
fail2ban took 221ms on the control node and the apply reported "restarted" onto a dead daemon while
both public machines kept no bans at all. The host looks again, after that moment. A unit still
starting is accepted at both looks; a service asked to stop is not waited on.
A jail reads a log; a container's output went to a file of the runtime's own under a path that
changes on recreate, so no jail could read a container's service. logging: journald runs the
container with the journal as its driver, named in the spec so moving it recreates it; any other
place is refused.
absent: true on a package has the host remove it through the machine's own
package manager when it is installed and leave alone a machine that never had
it; read back either way. Undeclaring a package still removes nothing. A
found firewall whose command is gone is recorded as removed, said once, and
asked nothing of.
The lab module needs the virtualisation daemon (novox/hq ADR 0172); the
capability is detected by asking the daemon about itself, not by finding
a client on disk.
Exactly the names declared reach the runtime, named in the spec so a change
recreates the container; a name that is not a capability's is refused and a
privileged container stays undeclarable. For a seat holder whose runtime
changes the machine's packet filter.
Every table and chain that refuses traffic is reported with whose it is:
the mesh's, the found firewall's, the container runtime's own, a ban, or
other — the runtime's user chain is other, which is where both predecessors
kept their rules, in the legacy filter on one machine and invisible to the
mesh. Adoption's threshold does not move; a converged machine's report
grows by its filters and its found firewall's state.
Convergence is a state the host keeps: a found firewall enabled again is
retired again and said; a reconcile that finds it inactive records that it
was found so, never that the mesh did it; a step skipped after a failed
apply is said. A retirement the mesh began and did not finish is finished.
Fixtures are rulesets captured from three machines of the first mesh.
The filter accepts what does not arrive on a link the machine names as
outward, and the host named only links carrying a default route. An
unplugged wired port was left unfiltered for whenever it was plugged in
(novox/hq issue 197). A link backed by a physical device is now named
whether or not it is up.
The host delivers its own successor as an archive whose target is a new
directory each version, and since mesh-host 63 the record keeps a resource's
former target for the next apply to remove. An archive has no removal (issue
162), so the first host that replaced itself under that rule refused its own
former version at the first step of every apply, and all four machines applied
nothing from then on. A former target nobody dropped is forgotten and said;
an archive the declaration dropped still refuses.
A container may name networks it also joins once created, for the per-machine
setting that keeps a found network while a neighbour still resolves it there:
joined after the run, part of the spec, refused when it cannot be joined.
A declaration may say which modules the mesh left out because a stored setting
cannot compose with its definition. Absence used to read as removal; a left-out
module's records are kept and said, and its holds are not released.
Genesis raises the bootstrap forge under the gitea module's container name, with
its image digest and its data directory mounted at /data, so the module holds it
by the found rule instead of raising a second forge beside it (issue 090). The
network is the one difference left for a take to say. Before this the forge had
no volume: its repositories were the container's, lost with it.
Every held thing carries what a take compares: for a found container its image and the image's date,
the networks it is on and the other containers on each, its mounts and published ports, beside the
declared image (and its date once pulled), ports and volumes, with the downgrade decided when both
dates are known; for a found file whether the declared content differs and how, as lines lost and
lines new. A resource whose target moved keeps the former target on record as an orphan, so the next
apply removes the container or file the host wrote under the old name (issue 097). Every apply reports
the strays: containers the mesh neither wrote nor holds.
The controller's own composition (mesh-controller internal/broker, 2026-10-01)
publishes memberships into the assignments stream after each push and
subscribes to its seat's tool subjects; the list the installer carries did
not say so, and a controller on it is refused on the first thing it tries:
"Permissions Violation for Publish to mesh.assignment.novox.builder" (hq #251),
which is why every build asked through the console was lost. The controller's
test that compares the two (TestTheInstallersFirstUserListIsWhatTheControllerWouldCompose,
run with this checkout beside it) named exactly these two subjects, and passes.
One capability per manager — uplink-networkmanager, uplink-systemd-networkd, uplink-dhcpcd — from
systemctl is-active, so the uplink seat's holder for a manager this machine does not run is refused
the way any missing capability is, naming it (issue 138). The apply that reports detects the profile
again and sends it, the same shape enrolment sends, so a machine that switched managers reaches the
mesh at its next push.
novox/hq 04-ISSUES/107. A declaration's only identity was the digest of
its bytes: a host could say "not the last" and could not say "older". On
the link, nothing refused an older one at all, and the drain picked the
last to arrive — wrong exactly when it mattered, a backlog drained out of
order or a broker that split a burst.
A declaration may now carry a sequence, one higher per send. A host
refuses one lower than what it kept, whole, and says why. The drain keeps
the highest sequence in a batch rather than the last to arrive.
Only when both sides claim an order. Absent reads as zero — "no ordering
claimed", not "first" — so a controller that sends none is still
understood and a host that kept one before it understood them compares
nothing. That is what lets hosts go first and the controller follow, which
is the order 087 says a new field needs.
novox/hq 04-ISSUES/163. The host asks after every apply whether a newer
host is delivered than the one running, and asked with the link-time
version stamp — which every delivered host carries as "development
build", because the version comes from where the binary sits now (0142).
So a delivered host never matched the newest delivered version, stood
aside on every push for ever, and because standing aside cancels the
report, the mesh never heard from it again.
Measured on two machines: each push produced "host <v> is delivered;
standing aside" for the version already running, then "applied, and could
not tell the mesh: reporting: context canceled". A machine restarting its
host on every push and reporting nothing, reading as healthy.
Asked with the running version now. Half of 0142 was applied to the
report and the known-good record and not here; this is the other half.
novox/hq ADR 0142, which decided this and was not implemented: "It is
unpacked into a directory named for its version, so it can read its own
version from its path. The stamp goes, and with it the need for a build to
know what it will be called."
The mesh's toolchain stamps no version, on purpose, so a delivered host
called itself "development build" and the mesh could not tell which host
any machine ran — which is the whole of what 087 added. A delivered host
lives at <libexec>/versions/<version>/<binary>, and that directory is the
answer.
A host placed by hand keeps its stamp, which is the honest answer for one
the mesh did not deliver, and is every machine until a delivery reaches
it. A binary sitting anywhere else is not read as a version at all.
The decision is split from the reading so a test can ask about a path
without being that binary.
novox/hq ADR 0141 and 04-ISSUES/142. A version was being delivered to a
machine and nothing started it: the launcher on these machines predates
the versions mechanism and runs the fixed binary path, so the delivery was
correct and inert.
Delivered as a FILE resource, not as part of an archive, and the
difference is the whole reason this is safe. A file is written atomically —
temp file in the same directory, then rename — so the running launcher
keeps the inode it was started from and the next start picks up the new
one. An archive writes in place with truncate, which would cut the file a
running shell is reading halfway through.
The manifest therefore carries a second copy of the script, and a test
refuses any difference between it and packaging/nox-mesh-host-launch.
Proven by drifting one and watching it fail. Two copies of a script is a
bad thing to accept, and the alternative was writing over a running
supervisor.
Together the two resources complete the loop: the version lands, the
running host stands aside because it sees one delivered, and the launcher
that starts next is the one that looks in versions/ and picks the newest
by arrival.
nox-mesh-host, not mesh-host. The command directory is cmd/mesh-host and
the launcher looks inside a delivered version for nox-mesh-host — the name
this is installed at and the name in its unit. The first delivery landed
the package's name, reported success, and would have been invisible.
novox/hq ADR 0141 and 04-ISSUES/142. The host half of the delivery has
been built and tested since 0141 and has never had a version to work on:
versions side by side, the newest runs, the running one stands aside
between reconciles, rollback picks a directory. This is the declaration
that gives it one.
One archive, unpacked to /usr/lib/nox-mesh-host/versions/${version}. The
version resolves to the artifact's digest, so an unchanged build lands at
the path it already had and re-composing a declaration moves nothing.
Not circular: the host applying this is a different version from the one
being written, and neither writes over the other — the kernel refuses to
truncate a running executable, which is the reason the path carries the
version rather than a link pointing at "current".
**The launcher is deliberately not delivered here.** The one on these
machines predates the versions mechanism and runs the fixed binary path,
so a delivered version is inert until it is replaced — and replacing it
from the mesh means writing over a running shell script, which sh reads
incrementally. That wants a designed swap rather than a file resource, and
it is the last piece rather than this one.
novox/hq 04-ISSUES/142 and ADR 0142. Nothing delivered the host because
nothing could compile it, and nothing could compile it partly because the
host was not a thing the mesh builds at all — it had no manifest.
One bundle in Go, for arch, built from cmd/mesh-host. A system is
required for a compiled artifact because a binary is pinned at link time
so a host refuses to touch a machine it was not built for (ADR 0005), and
`arch` is what all four of this mesh's machines report themselves to be.
Another system is another artifact and another build, which is what ADR
0142 means by one per target.
No resources yet. What places a version into a directory named for it
needs an archive resource whose path carries the version, and nothing
interpolates one — the second half of 0141's insight, and the next piece.
novox/hq 04-ISSUES/125. An adopted node keeps what it found until its
module is taken, which is correct and was recorded only in the node's own
state file. On the edge cut-over the mesh sent 346 resources, the journal
said it applied 330, and nothing anywhere said which sixteen or why —
reading it meant opening state.json by hand, and not reading it took every
public name on the machine down.
The line that reports the apply now carries it, grouped by module and
ordered by name, because the sentence an operator needs is "route-proxy is
assigned and not taken" and the module is the thing `take` acts on. An
apply that held nothing says nothing extra: a line that reports "0 held"
on every converged apply is a line that stops being read.
novox/hq 04-ISSUES/146, the layers behind the three already fixed.
A new membership says which bus it is for. Empty meant 'whatever the mesh runs
today' while two buses existed, and became a refusal the moment one did: an
enrolled node came up and reconnected for ever against its own record.
The enrolling client takes its inboxes in the space its user may listen in. A
JetStream publish waits for the stream's acknowledgement on an inbox the client
picks, and its default is one this user may not subscribe to — so the enrolment
failed with a permissions violation on a subject nobody had chosen.
And the enrolment publish carries a message id, so the client's own retry is
discarded by the stream rather than enrolling the machine twice. That one is
not finished: the duplicate survives it, and the issue says where the trail
stops.
novox/hq 04-ISSUES/146. Each was right while the mesh ran on the previous
broker, and nothing has raised a foundation since it changed.
The bus's certificate is made by the program that needs it rather than by
openssl inside the broker's image — the bus's image is Alpine with a shell and
no openssl, so the step exited 127 and no mesh could be raised. Self-signed as
before and on purpose; --user 0:0 because the volume is root's and the control
plane's image runs as nobody.
Enrolment no longer opens a raw TLS connection to check the pin: NATS speaks
its own protocol and upgrades afterwards, so the handshake met a plaintext
greeting. The client that presents the token carries the same pinned config
and verifies inside its own handshake, so the secret still leaves only after
the certificate is checked. The raw dial stays as what its tests prove, and is
no longer a path anything takes.
And the token says which bus it is for. Empty meant 'whatever the mesh runs
today' while two buses existed and became a refusal the moment one did.
It now stops at the bus's user list, which is the genesis half of 146.
The serving path passed nil where the apply writes its detail. Nil is silence, so
everything the apply says — a file held, a container replaced, the found firewall
retired — was visible when a person ran the one-shot command and discarded in the
way the host actually runs, which is always.
Measured: after a machine was converged and its found firewall was not retired,
what the host decided was unrecoverable, because it had said it to nobody. That is
why issue 143 has candidates instead of a cause.
say already reaches stdout and the unit sends that to the journal, so this needed
no new mechanism — only for the argument to be passed. Both paths now reach the
apply through one named helper, so a reader asking where the apply's output goes
finds one answer.
The test asserts the log is never nil and cannot catch the fault it was written
for, which is wiring; that is proved by a deployed host whose journal carries the
detail.
A reconcile is otherwise silent, and the condition deciding when it speaks asked
only what an adopted node reports: what it holds, and the firewall it found. A
converged node has neither, so it could speak only in reply to a declaration —
and the mesh composes no declaration for a node that has not said which links
face outside (ADR 0140). A machine waiting for a push that was waiting for the
machine.
Measured, not predicted: after the control plane learnt to read the links, the
control node recorded its own and the three converged machines sat silent while
their filters were refused.
The condition is now a named predicate, because as an inline expression nothing
could test it — which is why the gap shipped.
The mesh composes no filter for a machine that has not said (ADR 0140), and a
converged machine only speaks unasked when its adoption fingerprint changes. The
links were not in that fingerprint, so a machine that had just learnt to say
could only speak when a declaration arrived — and a declaration cannot be
composed until it has spoken. A machine waiting for a push that is waiting for
the machine.
Found before it bit: every machine in this mesh is in exactly that state right
now, having just been given a host that reports the links to a control plane that
does not yet read them.
The supervision was already right: a clean exit means the host stood aside, and
the launcher's next turn runs what is on disk. Two things made it dead code —
nothing told the running host a successor was waiting, and the rollback resolved
its known-good version through pacman, which no machine here uses and which two
of three operating systems do not have.
Keeping a version rather than a path was the clue. Versions now live in
directories named for them:
- the launcher picks the newest delivered one every time round the loop, or the
one a rollback pinned, or the host placed by hand when nothing is delivered;
- the running host stands aside between reconciles, never inside one, by exiting
cleanly — and returns nil so the launcher does not count it as a crash;
- a completed reconcile retires what is older than the predecessor, keeping the
predecessor because that is what a rollback starts, and never the running one;
- rollback pins the predecessor instead of reinstalling a package: no package
manager, no cache anyone may clean, same script on every operating system;
- the report says which host version produced it, so 'behind' is answerable.
Newest is when it arrived, never how the name sorts: '1.10' orders before '1.9',
and ordering by name would start an older host and call it an upgrade.
novox/hq ADR 0141. The delivery half — a module carrying the next host — follows;
until then nothing delivers a version and every machine takes the fallback, which
is what it does today.
The filter blocks everything passing through the machine and then allows the
machine's own containers back by naming the address ranges they sit on — two
ranges fixed in the control plane and the rest typed after a flip had already
cut a workstation off. A range describes one machine and goes stale in silence.
Read the links carrying a default route instead, from /proc rather than by
asking a program, and report them on every apply. A machine with no route off
itself reports nothing, and the mesh composes no filter for it rather than
writing a rule around a link with no name.
novox/hq ADR 0140. The control plane does not read this yet.
A container resolves every machine and public name through the entries it is given when it is created,
and nothing re-reads them. The host compared everything about a container except those, so one whose
image and files never changed was left alone holding an overlay address five days out of date — it
restarted 2286 times against a database it could no longer find, and the mesh reported the machine as
doing what it was told (novox/hq 04-ISSUES/135, the same fault as 045 in the field left out).
Sorted, so the digest does not move for a reordering nobody made. The first apply after this recreates
every container that carries mesh names, once.
A mesh raised from nothing must be able to say what it applied and what a machine refused (novox/hq
ADR 0134), and the first user list is what permits it. Kept in step with the controller's own
composition by the test that reads this file.
novox.be is a module on this mesh. Reading a resource's owner to the first dot made its resources
belong to something called "novox", and a step's gate would then skip whatever else happened to start
that way — silently. A resource's own id never contains a dot, which is what makes the last one the
boundary; the mesh's derived step was changed to add a hyphen rather than a dot for the same reason
(mesh-controller #128).
A run-once container that does not complete stops the rest of that module's resources; everything
else on the machine is attempted, as every other shape already is (novox/hq ADR 0136). An action
still gates the machine — genesis is a row of them and they belong to no module. What was not
attempted is reported as skipped, because that and 'nothing to do' are different answers.
Without this, ADR 0135's derived preparation would let one module's unreachable database hold a
machine hostage — the fault 04-ISSUES/011 removed for everything else, and the reason the catalogue
migrates itself at start.
The controller is the way in for tool calls (novox/hq ADR 0095); the first user list must say so,
or a mesh raised from nothing refuses its own first ask. Kept in step with the controller's
composition by the test that reads this file.
The mesh runs on the seat's bus alone (novox/hq ADR 0131, design 28 task 5.5). The host's old
dialling and enrolment paths are deleted with the switch that chose between them; a membership or
a token naming another bus is refused before anything is sent, rather than dialled on a transport
that no longer exists.
The rescue path: an operator writes the membership file by hand on a machine no
bus can reach — rotated while it still held the old password — and restarts the
host. Same file, same check as the declaration path.
Two things the controller's own composition now derives and the installer's
carried list did not: the delivery subjects of the controller's consumers, and
JetStream enabled on the account — without which the first bound consumer is
refused. Found live on a mesh moved rather than raised; a fresh genesis would
have met both at first start.
The mesh names a machine's bus user node.<name>; the host dialled with the bare
name and every machine was refused the first time it reached the server:
"authentication error - User". Same string as the composed user list, or nothing
connects.
Until now a membership — bus address, fingerprint, password — was written once,
at enrolment, and nothing ever rewrote it, so a machine already enrolled could not
be moved to another bus at all (novox/hq design 28, task 5.2). The mesh now
delivers a membership for the new bus as a sealed file in the declaration, like
any secret; the host reads it after the declaration has applied, saves it as its
identity, and exits cleanly so the service manager restarts it dialling the bus it
names. The same path a machine takes after a reboot — so nothing new has to be
right for it to work. A membership carries its transport; empty means the bus the
mesh ran on before, so nothing written earlier reads as unset.
The same twelve steps, with the difference that matters: the mesh composes its own user
list and at genesis there is none, so this carries the first one — the controller's
account at a bootstrap password, rotated with the store's and replaced by the
controller's own composition from its first start.
The server's settings and the user list are separate files in one directory. Separate
because the settings belong to whoever raises the server and the users belong to the
mesh; in one directory of necessity, because an include path resolves relative to the
including file's own directory, so an absolute one sends the server looking underneath
that directory and it refuses to start.
No `verify` in the TLS block. That makes the server demand a client certificate and
nothing in the mesh presents one — a host pins this server's exact certificate and
authenticates with a password.
The controller's permissions here are checked against what the controller derives, by a
test in its own repository reading this file. They are two statements of one fact, and a
template that granted less than the controller needs would produce a mesh that comes up
and is refused on its first act.
The last of the host's link that still named a transport. `Asking` is one
enrolment conversation — a connection made with the token, a question asked, and
an answer waited for — and it is its own seam rather than part of `Link` because
almost nothing about it is the same: the credential is a one-time secret, there
is no declaration to hear, and a node that fails here is not in the mesh at all,
where a node that fails in `Link` has merely lost touch with one it belongs to.
`Enrol`'s thirteen arguments became an `Approach` — where, which certificate,
which bus — and the request it already had. The token says nothing about which
bus, and does not need to: every token names the one the mesh runs on today until
the rollout.
**The reply address is the whole of what changes on the new bus**, and it is
forced rather than preferred. Verified against a running server, both halves: the
answer reaches the node at the address its request carried in the payload, and
the transport's own reply field held something else entirely by the time the
consumer saw it — the consumer's ack address, exactly as design 25 §2 says. The
test asserts the field is *not* the node's inbox, so a future server that stopped
claiming it would fail this rather than let the reason quietly become folklore.
The inbox is under `_INBOX.enrol.<node>.`, which is exactly what the enrolling
user may subscribe and no wider, with a random tail per attempt: a reply left
over from an attempt that timed out is not the answer to this question, which is
what the correlation id does on the other transport. Subscribed before anything
is published, because a node that published first could miss an answer to a
question nobody was listening for.
The outbound half went behind `Bus` and a node's two statements stopped naming a
transport. This is the other half, and where the transport reached furthest: the
run loop selected on a channel of the client library's own delivery type, so
every part of holding a node in its mesh knew which bus it was on.
`Link` is dialling, hearing and saying in one interface, because dialling is
where the transport is chosen and choosing it twice is how one half of a node
ends up on a different bus from the other. `Declaration` has one way of being
done rather than two: a declaration set aside for a newer one is settled exactly
as an applied one is, on both buses, and the difference is a fact the report
carries.
Four things this settled.
**The host declares nothing on the new bus.** On the bus the mesh has it declares
its own queue, because a queue that is not there means a node that hears
nothing. Here it binds to a consumer the mesh made when the node enrolled, and a
missing one is said as the mesh's to answer rather than quietly created with
whatever this client happens to default to.
**The pin is easier here than in the tool runtime, not harder.** The Go client
takes a *tls.Config, so the same PinnedConfig with the same VerifyPeerCertificate
does the work — the subject-alternative-name constraint recorded against the
runtime's client is that client's, because it takes PEM strings with no verify
hook. A host checks the fingerprint and nothing else.
**Binding needs the subject as well as the consumer.** An empty subject is
refused rather than taken to mean "whatever that consumer delivers", which the
server said plainly and only when asked.
**Reconnection stays the caller's.** Hold already decides when to try again and
how long to wait; a client reconnecting underneath it would make that reasoning
a duplicate of the library's.
The drain keeps its live half and loses its catch-up half, as it said it would:
verified that three declarations pushed to an absent node leave one on the
stream, and it is the newest.
One test-harness lesson worth the comment it got: delete-then-add is not a reset.
A test that did that inherited the previous test's messages, and the symptom was
a declaration counted as delivered twice — which reads as a redelivery bug in the
code under test rather than as a dirty stream.
Put back by hand, it was found afresh and its copy became the hold's original, so a machine that
kept restoring it kept growing copies and lost which one was first. The first original now stays
the record's, content that differs is kept once beside it, and the note says a rollback means
unassigning the private network. Nothing is removed unless it is <wireguard dir>/<iface>.conf,
not a path the mesh writes, and not a link, which would leave the key-bearing target behind.
Kept on disk it was the take's fallback; once the mesh's interface is up in its place and a peer
has handshaken with it, it is an unmaintained way back onto the network, held for ever. It is now
removed from where its unit reads it, its kept original verified first and left as it is, and the
hold ends. Until proven — no handshake, or wg not answering — it is kept and the report says why.
The retirement is recorded apart from holds, so later applies, an undeclare, and a reassignment
find it retired rather than missing, and nothing writes it back.
Records written before Found existed left the adoption guard and the converge filter loaded on
undeclare, then deleted their unit files from under them; a unit whose own file the mesh created
is now the mesh's, whatever its record says. Found is kept apart the moment it is read, so a
first apply that enabled and then failed is not read back as the machine's; boot is found the
first time the mesh sets it; a service once stateless, or moved to another unit, is found afresh
(the old unit given back). The unit is read after the reload that loads a file written in the
same apply, and removal reports what it actually did.
removeProcess deletes filepath.Join(daemonRoot, name) whole; a process named ".." made that
/var/lib/mesh. The declaration and the removal now hold the name to one rule.
A service undeclared used to be stopped: unassigning the private network stopped the container
runtime, unassigning sshd would stop ssh, an uplink module would take the machine offline. The
host now records the unit's state when it first applies it and restores that on undeclare —
found running stays running; started by the mesh (the converge filter) is stopped again; nothing
is started on the way out; a pre-existing record leaves the unit alone.
An undeclared process had no removal at all and failed every apply on its node; its unit, timer
and bundle are now removed.
Step 3.5's first half, mirroring the controller's. A host says exactly two
things unprompted, and the difference between them is the whole interface:
a report must arrive, and a heartbeat must not be insisted on. So a report
goes through JetStream — it is the message the store-window guarantee is
about — and a heartbeat stays on core, because a heartbeat in a stream is
the mesh's least valuable message competing for retention with its most
valuable.
The host still imports nothing of the mesh's own (ADR 0005): this is its
own interface over its own libraries. It agrees with the controller because
a fixture holds both to one envelope, which is the only agreement that
survives two repositories.
Also recorded, where the next person reads it rather than in a plan: the
"newest wins" window narrows at the rollout and does not disappear. Last-
per-subject makes the catch-up half the stream's, and sequence orders them
definitively — but three pushes to a connected node are still three
deliveries. Saying which half goes is worth more than "can probably be
removed", which is how a load-bearing window gets deleted in a hurry.
The empty-resources guard refused every empty body as a likely mistake,
with no way to say emptiness was meant — so the control plane could
never tell a node to drop its last resource. The envelope gains
owns_nothing: with it, an empty declaration is applied (the node drops
what the mesh owned); without it, empty is still refused, so a
truncated or mis-composed body cannot silently strip a machine. One
test, both directions.
The host parses MTU from the found [Interface] and reports it, so the
mesh's interface can come up with the same MTU when it takes the tunnel
over. A path tuned to 1380 regresses to the 1420 default otherwise —
invisible to ping, fatal to TLS handshakes and transfers over that path
(novox/hq: the mesh had no MTU concept). Zero when the config named
none, and the mesh writes no MTU line then.
dns and ip were declared, validated, handed to the runtime — and part of
no comparison, so their first deployment compared every container equal
and changed nothing, silently. The same shape as 04-ISSUES/045: a field
that is not in the spec is a field that can never reach a container that
already runs.
Mailu's 2024.06 admin refuses to serve behind a resolver that does not
validate DNSSEC, and the runtime's own forwarder (127.0.0.11) validates
nothing — so a module shipping its own validating resolver had a
resolver nothing could be pointed at. Found live, blocking a cutover:
the admin sat unhealthy, submission answered 454, and the declaration
language had no words for the fix.
Two fields on a container, both handed to the runtime verbatim: dns —
the resolvers it asks — and ip, its static address on its user-defined
network, which exists for exactly one shape: a container others must
reach before name resolution works, the resolver itself being the case
that forced it. Both take only addresses and are refused on arrival
otherwise — a name here would reach the runtime verbatim and be refused
at create, after the old container was already gone.
docker inspect <name> resolves across every object kind, not just
containers. A module regularly names a network the same as the
container that joins it (keycloak does this today, ordinarily) — so
when the container does not exist yet but the same-named network
already does, the bare form answers with the network's JSON instead
of reporting the container absent, and the template these callers use
(.State.Running) fails to execute against it entirely.
Live on novox tonight: minio's LB container, named the same as its
network ("minio"), could never be created — every apply crashed on
"the container runtime could not say whether minio is here", stuck
since first push, because the check itself never got a clean answer.
Fixed at every call site asking a container's state by name
(containerState, inspectFound, NamesFree, raiseGiteaServer,
containerRunning) by scoping to `docker container inspect`, matching
the type-scoped form this codebase already uses correctly for
networks, volumes and images elsewhere. Also scoped the one image
inspect that was still bare (publish.go), for the same reason.
mesh-host runs as a host-level service (nox-mesh-host.service), not a
Docker module — merging this does not redeploy it. The live novox
failure persists until the service itself is rebuilt and updated.
Review of the ADR 0105 build (hq ADR 0105). The takeover stopped the found
unit and then found out whether the mesh's interface would do; a start that
failed left the machine with no tunnel at all.
Now nothing is stopped until the declared interface listens on the found port
at the found address and the key file it names holds the found key — the
refusal names the remedy — and a mesh interface that fails to start after the
takeover has the found unit started again, with the account saying so. The
account has three states (not taken, taken, down) and is given on every
takeover, failure included. An interface raised by hand is looked at again
for a moment and then refused naming `wg-quick down`. A found unit started
again by hand beside the mesh's is said, not stopped: on the hub it cannot
hold the port, and on a spoke two interfaces with one key would fight.
`mesh-host overlay take --tunnel <iface>` is the path for a node that
enrolled before the mesh knew to take a tunnel over: the found key becomes its
overlay key — identity, sealing and serving keys untouched, so nothing sealed
to the node is remade — and the mesh is told with a rekey signed by the
identity key, over the key left, the key taken and the tunnel. Told first,
written second, so a run again puts right whichever half did not happen.
The plan says what an apply would change from the declaration and the
record, before the machine is touched. The apply now recreates a container
when the content of a file it reads at creation changed, and the plan said
"check" for every recorded container — true, but a preview that hides the
one step somebody asked about.
So a recorded container whose record of what it read differs from what this
apply will hand it — a plain file declared here, by its declared content;
otherwise what this host last wrote at that path — is planned as an update
naming the file, the same comparison applyContainer makes. What the record
cannot settle stays a check: a file neither declared nor recorded is read
from the machine by the apply, not by the plan; and a container with no
record of what it read was labelled before the host kept that record and is
accepted as it is.
novox/hq 04-ISSUES/103, 104
Review of the first cut found four things.
A directory mounted into a container is no longer looked inside, not even
for the files this host wrote there. The controller records every
provider's received and contributions file as a plain file under a mounted
directory, so folding those in would have recreated the route proxy — which
re-reads its routes live, by design — on every route change, and killed
every provisioner sidecar, which polls what it receives, mid-reconcile on
every grant. Whether a service reads a file under its directory once or
watches it is the service's; restart-on is how a module says "once", and it
stays the opt-in. Env-files and files mounted directly remain by content.
Genesis wrote the superuser secret as `value\n`; `secret accept` strips the
line ending by design, so the postgres module declared `value` — and with
a mounted file's content in the spec, phase three would have recreated the
store it meant to adopt in place, with the temporary control plane
connected to it. Genesis now writes the value alone. readCredentialFile
tolerated both endings already. Pinned with the bytes the genesis code
path writes, then the module's declaration of the same container: it must
reconcile.
A container carrying a label from before the host folded in what it reads
is accepted rather than recreated, when that label matches the spec as it
used to be computed: what it reads is recorded then, a change is caught
from that record from the next apply on, and the label is renewed at the
next genuine recreate. Recreating them all would have been a restart storm
across the mesh in declaration order, the store first. The trade-off is
stated in the code: a container already stale at upgrade time is not
caught, and could not have been either way.
The record of what a container read is looked up by its name when its
declared id has none — the bundle's `store` becomes `postgres.server` for
the same container — so a change on the day it is adopted still names the
file. The by-target lookup takes the most recently applied record, since
the bundle's record for the same target is never removed by the mesh's.
novox/hq 04-ISSUES/103
The host decided whether a container was still the one declared by a digest
of its declaration, and the declaration names an env-file's path and a
mount's path — never what is in them. So when the store was given a new
port, the host rewrote the forge's and the analytics service's environment
files, correctly, and left both containers running with the old port in
their environment: a container reads its env-file when it is CREATED, and
`docker restart` hands it the same environment again. Both looked healthy
until they answered 502.
What a running container takes in at creation is now part of its spec, by
content: every env-file, a file bind-mounted into it, and every file this
host wrote at or under a directory bind-mounted into it — the secrets,
bindings and configs under a module's state directories. The digest is the
one the store already records for a file the host wrote (`wrote`), read
from the state as it stands when the container is reached, so a file
rewritten earlier in the same apply is already the new one; a file the host
has no record of — an env-file a predecessor left, the superuser secret
genesis writes before any declaration names it — is read from disk, which
is what keeps adopting a running store in place a reconcile and not a
recreate.
Deliberately not part of it: what else is in a bind-mounted directory,
which is the service's own data and changes while it runs; a named volume;
a seed created once, which digests as the seed the host wrote and not as
what has grown in it; and a step — a run-once or scheduled container reads
its files when it runs and runs fresh each time. On an adopted node a held
container is held before any of this is looked at.
The host records what each container was created reading, per file, so
the recreate can say which file changed — "recreated: <file> changed" in
the report and, now with its detail, in the log. A container made before
this record existed is recreated once and says so.
novox/hq 04-ISSUES/103
Review of the fix for hq issue 104 found three faults in it. A file applied
on an enrolled node — the mesh's own last declaration included — is applied
as the bundle is, so its resources are recorded as the machine's own and
what the mesh declared reads as undeclared: the plan removed the foundation.
`apply FILE` is for a machine the mesh has not spoken to, and is now refused
saying so whenever declared.json exists. The plan looked at what is held
before what the declaration says is taken, so the one cutover ADR 0100 says
must be previewed read as a hold; it now decides in holdOnAdopted's order,
models a step run inside a held container, and a test holds the plan's
sequence to the apply's outcomes. Genesis wrote the mode on every run, so a
re-run after `converge` left the state saying adopted while the kept,
signed declaration said converged, and the reconcile loop refused every five
minutes with no delivery coming to end it: genesis now writes the mode only
when none is recorded, and where the state and the verified kept declaration
disagree, the kept declaration wins and the repair is said.
Also: a file lock beside the state, taken by the link service, the host's
own commands and the installer alike, so a `reconcile` run by hand no
longer races the loop's save — chosen over refusing while a named service is
active, which would miss a `mesh-host run` started by hand; `--json
--dry-run` emits {plan} like an apply emits {plan, report}; the README's
duplicate flag line; and the bundle refusal is about the digest, not a claim
the carried bytes can never match what genesis applied.
On an adopted machine the private network takes the predecessor's tunnel
over in place (hq ADR 0105). Genesis finds the one interface up besides the
mesh's own, settles the hub's port and the mesh's range on it, and skips
ADR 0100's non-overlap check for a range that is now the tunnel's; a
--hub-port or --overlay-range that disagrees is refused naming the tunnel's.
At enrolment the found interface's private key becomes this node's overlay
key — the one credential the mesh takes rather than mints — stored where a
generated one is stored, never printed and never sent; the tunnel (port,
address, range, peers) travels with the keys so the mesh composes from it
before the first declaration.
The interface's service may say what it takes over. Before the mesh's unit
starts, the found configuration is kept like any held file and the found
unit is stopped and disabled; nothing is flushed, and an interface still up
after its unit stopped refuses the takeover rather than half-working. The
report says what was carried: interface, port, range, peer count, taken or
not, and where the original was kept.
An operator ran `mesh-host reconcile` on an adopted control-node with twelve
modules assigned. It applied the bundle the host carries — the genesis
declaration, foundation only, converged: recreated the store, failed on the
broker's held port, wrote the converged base filter and started its service,
and stopped at the first failing action. The filter closed the machine for
forty-five minutes. The host reported the node adopted in every report, the
declaration said converged, and nothing compared the two; nothing was printed
before acting (hq issue 104).
The host now records the node's mode — from every declaration the mesh sends,
and at genesis from what the operator said — and refuses, at the point of
application, a declaration that says the other mode, naming both and the act
that changes it. Only a declaration the link delivers, signed, changes the
mode: that is how `converge` and `adopt` arrive, so the flip still works and
nothing else can do it. Genesis marks the bundle consumed, with the digest of
what it applied, so `reconcile` holds a node the mesh has spoken to against
what the mesh last said and never the bundle, and refuses the carried bytes
when they are not what genesis applied. A file is refused when it is not what
the mesh last said: a declaration carries no sequence and no issued-at, so the
host cannot tell older from newer, and says so. Both commands print what they
would change — a hold, a removal, an action named as one — before touching
anything, and --dry-run is that list and nothing more.
Registering a module is an overwrite. Recording the forge's port ran `module add`
every time, so a genesis re-run pointed at an older catalogue would replace the
manifest of a forge that is built and assigned — with a push a few lines later.
Registering is only here so a settings row has a module row to hang on, and that
row is already there on a mesh that knows the forge. So: ask first, and skip.
Two comments narrowed to what is true. What follows the node's setting is what
the mesh derives from a module's ports — its container mapping, its filter rule,
its opening and what it serves. The forge's own address in its runtime's
environment (hq 088) and its route contribution's port do not, and are already
wrong for any port the mesh assigned. And a settings layer is the module's, not
one resource's: a second mergeable file on the builder would be given `serves`
too.
novox/hq 04-ISSUES/085
Every foundation port given at genesis became a per-node setting of the module
that binds it, except the package registry's: that one was fixed by rewriting
the builder's manifest when the installer registered it. Registering the builder
again from the catalogue undid it, and the forge's own module, when it took the
bootstrap forge over, came up on the catalogue's port — which on a machine where
a predecessor holds 3000 points the builder at the predecessor's forge.
So the rewrite is gone, and the port is recorded twice as a setting, both from
the one input:
- the forge's module is registered at genesis — not assigned, nothing of it runs
— so the controller has something to hold `{"ports": {"3000": <given>}}`
against. Assigning the forge later raises it on the port this machine was
given, and its container, its filter rule, its opening, what it serves and
what consumers are told all read it from there.
- the builder is given `{"serves": {"port": <given>}}`, which merges into the
binding it carries in place of one nothing can resolve yet.
A genesis on the catalogue's port records nothing and registers nothing, so it
does exactly what it did before.
novox/hq 04-ISSUES/085, ADR 0100
"a checkout of the mesh's catalogue, for the registry's and the builder's manifests and what phase two installs; without it this stops after the foundation")
"a checkout of the mesh's catalogue, for the registry's, the builder's and the forge's manifests and what phase two installs; without it this stops after the foundation")
These tests read files outside this directory (the whole catalogue, the repository, or its root files): mesh-build-go never keeps this package's verdict (issue 1935).
{"container not running",fakeRuntime{units:map[string]liveness.Observed{"dnsmasq.service":up},containers:map[string]liveness.Observed{"box":{Found:true}}},nil,true},
{"container held by a window",fakeRuntime{units:map[string]liveness.Observed{"dnsmasq.service":up},containers:map[string]liveness.Observed{"box":{Found:true}}},map[string]bool{"box":true},false},
{"unread",fakeRuntime{err:errors.New("the service manager could not be read")},nil,false},
ifline:="this node's login is not followed yet ("+werr.Error()+"): it dials as it last did, and looks again in "+loginsFollowAgain.String();line!=said{
said=line
say(line)
}
}else{
said=""
foru:=rangeupdates{
ifu.Caught||u.Deleted||len(u.Value)==0{
continue
}
nl,err:=delivery.DecodeNodeLogin(u.Value)
iferr!=nil{
say("the key "+node+" of "+delivery.NodeLoginsRead+" is not read: "+err.Error())
continue
}
hear(nl)
}
stop()
}
select{
case<-linked.Done():
case<-time.After(loginsFollowAgain):
}
cancel()
}
}
// loginSecretModule is the module the vault names this engine's login values under: own/<node>/mesh-host/<login> (ADR 0583 §3,
// insight).
constloginSecretModule="mesh-host"
// fetchLogin asks the vault, through its existing hand-out of a machine's own secrets, for the value of login, sealed to this
// node's key, and opens it with the sealing key; no new verb is asked for it.
// Issue 1126 (ADR 0547 §2, the milestone's H8): the node agent started whole against a real bus, once the catalogue
// places its node (the placement written after it started), composes a module from its want, the indexer's manifest, its record and its mesh-config value, and applies it,
// with no key in the controller's compositions; a changed config value is composed again on its event and applied as a
// revision of what runs. And a catalogue pull request's placement candidate equal to its placement is answered "same"
// in the node's own key, and nothing is applied for it (ADR 0559 §4, subtask 10 of the milestone).
//
// And (issue 1126, ADR 0560) it reads, through its own session under the grant the grants part composes for it
// (nodegrant.For with the served key, as the grants part grants it), the build of a placed module that declares one and the one config value
// a provider on another node serves from (a mail domain); under a grant without that served key the bus refuses the
These tests read files outside this directory (the whole catalogue, the repository, or its root files): mesh-build-go never keeps this package's verdict (issue 1935).
// Over TCP, not the socket. While the store initialises it runs a temporary server on the
// socket ONLY, then stops it and starts the real one — so a socket check passes, the action
@@ -99,21 +114,21 @@
{
"id": "store-ready",
"type": "action",
"in": "mesh-store",
"in": "postgres",
"command": ["sh", "-c", "for i in $(seq 1 180); do pg_isready -h 127.0.0.1 -U postgres >/dev/null 2>&1 && exit 0; sleep 1; done; echo 'the store did not answer within 180s; its own last words follow'; pg_isready -h 127.0.0.1 -U postgres; tail -n 20 /var/lib/postgresql/data/log/*.log 2>/dev/null; exit 1"],
// Now is the clock the cache is kept by; nil is the machine's.
Nowfunc()time.Time
}
// FileMode is what decides whether an account reads a file: its owner, its group and its permission bits.
typeFileModestruct{
UID,GIDint
Permfs.FileMode
}
// Escalation is every way account can become root without a person (novox/hq ADR 0266) that the judge
// looks for — Judged lists them, NotJudged what it does not: its uid, a group of RootGroups the user database
// lists it in, any sudo rule naming it or a group of it, any of secrets it can read by owner, group or other
// bits, and the ways of ways.go: doas, polkit, a runtime's socket, an ACL, a setuid program no package owns.
// sudo absent is no sudo rule; doas and polkit (pkexec's grants) are judged from their own rules. A secret not there yet is skipped: there is nothing to read.
// The parent directories are not walked, so a file the bits allow and a directory hides is still said:
// the judge errs toward saying a way that is not, never toward missing one that is.
rules,err:=SudoRules("Matching Defaults entries for agent on box:\n env_reset\n\nUser agent may run the following commands on box:\n (ALL) /usr/bin/systemctl\n (root) NOPASSWD: /usr/bin/true\n",nil)
t.Fatalf("index.js is %q after a refused archive",got)
}
ifentries,_:=os.ReadDir(dir);len(entries)!=1{
t.Fatalf("%d entries beside the tree after a refused archive",len(entries))
}
}
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.