mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
Two incidents had one shape: a change took effect that nobody saw first (hq issues 241, 304).
Three guards, each silent when nothing is at stake:
- settings show [--history], and a set that answers each key it adds, changes and removes, and
refuses a removal unless --replace; the replaced or cleared layer is kept in settings_history,
in the same transaction as the write (migration 0078).
- every send keeps a summary of what it sent (no file content), plan <node> --diff compares with
it, and push with no machine is refused unless --all.
- a push that would give a running container's mount another host directory holds that machine,
naming the module, mount and both directories, until push <node> --move <module>; a named push's
cascade is held the same way.
Rebased onto main and fitted to it: the hold runs before the push says what it recreates, a whole
push still says so first and leaves machines waiting for a gate, the verb's push with no machine is
still --behind and a push still needs why. The verbs take plan diff, settings replace and history,
and push move beside a machine, each refused where it cannot take effect.
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
Only gated sends said which containers a move recreates; a push moved
mail to a new build and recreated a container with a new image without a
word. The push now lists, per machine, every module it moves and what
that recreates, with the same Recreates the gated send uses.
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
A seat's verb names the shell commands it is the mesh's way to do, and a module says it for its own
tools in its manifest; the tools verb and module list --json carry both, for the mesh MCP server's
search, the agent's instructions and the guard on its shell.
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
A plan's gated send for mail carried postgres's and mongodb's new builds
(policy record) to the control node and the anchor on 2026-10-07: a send
composes the machine's whole declaration from the builds the mesh holds,
and the gate only ever looked at modules that roll out. Every send but a
person's push now composes a recorded module from the manifest of the build
the machine was last sent and records that it still carries it; the bus
step moves the bus alone. And each move a gated send carries says how many
of the module's containers it recreates, and whether with a new image or
only their declaration.
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
The witness moves a running build aside into a directory the controller's user
cannot enter, then deletes it; a verb exec'd from os.Executable() in that window
failed with permission denied. /proc/self/exe stays valid while the process lives.
A verb that still cannot start, or arrives while the controller stops, is refused
with link.ErrHandingOver and marked retry: handing-over.
The gate judged a module by what the mesh saw from outside, so a container that
crash-looped after it applied passed it. Each machine's node-engine now states
the health of every long-running resource it runs; the controller keeps the
newest statement per machine, raises module.<module>.<machine>.unhealthy on the
second statement in a row, clears it on the first that does not say it, and the
gate passes a module only when every long-running resource of it is stated
healthy since the send. An engine that states nothing is judged as before.
While the mesh-delivery seat has a holder on record, a merge that moves no
core module opens its walk and asks nothing until mesh-delivery or a person
says go; nothing of it is registered before its turn, so no other send
carries it. The controller keeps the planner, the gate, sending and the
walk, and gains the verbs the owner asks with: delivery-plan, -order,
-check (a group composed as one future state), deliver, delivery-stop,
delivery-walks; every walk kept is said as plan-moved.
Every check the mesh had ran after a merge, on a machine: a manifest the node-engine refused
(236), an identity a real machine's name made too long (263). merge-gate raises the mesh as the
facts snapshot says it is and the mesh with the change, each in a throwaway store through the
controller's own records, composes every machine twice and validates it with the node-engine's
validator, and fails what the change breaks, naming the machine's roles and the module - plus a
manifest the judging controller cannot read, a consumer left out of its grant, a module removed
while a machine runs it, a new module the node-engine would refuse; it warns on a wide rebuild.
The forge's new head of a pull request becomes a check the controller asks of the build seat:
the head and, beside it, the controller the mesh runs, the catalogue, the host and the lab; a
throwaway store and bus of the versions the mesh runs; the repository's merge-check.sh in the
mesh's Go toolchain with no container runtime socket; then mesh-lab's replays. The verdict is
said as checked, an error never a pass, and nothing is recorded or registered.
Every check the mesh had was right about the world it was given and none was given
the mesh's: a real machine's name made an identity too long (263), the node-engine
refused what the catalogue check passed (236). The controller now composes what a
check needs - every machine under a pseudonym of its name's length, its roles,
system, builds, capabilities, assignments, pins, settings and how its declaration
composes; every seat, module and source; the bus, store and node-engine versions it
runs - with no secret, no address and no name, and keeps it in the artifact store
as facts:latest when it moved, or daily. The replaced snapshot's manifest is let go
of, so the nightly collector takes it. S14 raises facts-stale past two days.
A send carries the machine's whole declaration, so at the switch to roll the next send of
anything would have carried the old default's backlog, unjudged, to every machine. A gated
send now carries and judges everything waiting on its machine; every other send is refused
or leaves the machine; a release plan walks what waits one machine at a time, the control
node last, and one that fails holds the next until a person releases it.
A plan's send to the bus's machine for another module carried the bus's new build and
restarted it under every machine with nobody asking (2026-10-06). The guard is in the one
send everything uses; only the bus step passes it. A rebuild that made the same artifacts
is no move.
A build that reported applied was sent everywhere; one that then did nothing, served
no tools or broke its machine's word reached every machine. Now the first machine is
judged by the component's health (the core's definitions, as doctor probes H-*, or a
module's own) three times over two minutes within ten; a failing gate puts the previous
build back there once, marks the build, and says it as a condition and an event.
Upgrades roll out by default; the bus is a planned step; a module deleted at its
source is not built (the public-acme plan failure).
Issue 258's fix let a mesh seat's holder elsewhere answer before this machine's own provider. Right
for the resolver, which any provider answers alike; for the store's seat it re-bound every database
consumer on a machine running its own store to the holder on another, each was given a fresh, empty
database there, and nothing said so for twenty hours.
- An offer says whether it keeps its consumers' data (`keeps-consumer-data`); unsaid, a provider
that grants each consumer a credential does. For such a provision the seat's holder no longer
overrules a provider beside the consumer; a pin still does.
- Where each such consumer was sent is recorded (migration 0071). A resolution that would bind it
elsewhere keeps the recorded provider and says the move; one whose provider is gone is refused,
never answered by another.
- A push says a kept move and raises it as an urgent condition at once; the self-check's D12 raises
it every run, with a pinned move not yet sent as a warning and any unasked move as urgent.
A provider now waits for a person before retiring more than three consumers
or half of what it holds, and deletes only when asked. The controller is that
person's way in: it keeps waiting and rejected sets as conditions, answers them
with retire approve|reject, lists and deletes retired consumers through the
provider's own tools on its machine, records each act in the hand-act log, and
probes for anything retired longer than thirty days (D11).
Two controllers could both act (issue 204), a reconcile's report could
overtake the apply after it and the digest decided (issue 267), and a grant
could make a second writer of a machine's report.
- The lease (internal/lease, ADR 0229): mesh-controller_lease key `holder`,
15 s age, renewed every 5 s by compare-and-set; the epoch is the revision
it was taken at. The gate is the clock (stops 3 s before expiry); a refused
renewal is a loss and the process exits; a holder that stops gives it back.
serve takes it before asserting the bus. Epochs kept in the store
(migration 0068 controller_epoch) as a floor: a bucket raised from nothing
is compacted past it. Unleased (no epoch, S12 urgent) only when nobody
holds it and the bus will not let it be written. A shell command acts
under the holder's epoch, or its own lease when none.
- Declarations carry `epoch` inside the signed envelope, only to a machine
whose latest account carried a report_sequence (mesh-host #35); would-send
is composed with the epoch last sent. Allot and the send both pass the gate.
- Reports: contract in internal/link/order.go (epoch, sequence,
report_sequence, older_than, refused_older). Accounts kept by epoch, then
sequence, then report sequence; older refused, counted; unordered reports
keep the digest rule. Plans by compare-and-set on a revision, with epoch.
Conditions and calls carry the epoch and are not written off the lease.
- S12 and S13 (naming the writer by epoch) watched, D5 run; reset of the
bucket said. Writers table compiled in and enforced in PermissionsFor; the
controller no longer publishes mesh.control.>. A contract per consumed
kind, and the empty-on-error lint over the repository.
- mesh-host pinned to its main with the epoch in the validator (D1 validates
the envelope as sent).
Needs mesh-host's genesis lock with the lease grant (mesh-host PR) for
TestTheInstallersFirstUserListIsWhatTheControllerWouldCompose.
A given own secret the module reads at start is held by nobody but that
module, so the mesh need not read it to replace it: secret rotate now
works on it, and a value given through secret accept is replaced on its
own after the module's first good start under the mesh. Only a value an
outside party issues (own-secrets "issued-by": "outside") or one the
module applies stays as given, refused with the reason.
A module or seat holder assigned after the controller started was sent
its declaration and found nothing to bind: messenger on novox
("consumer novox_messenger not found", 2026-10-06) and every first
build-agent holder (2026-10-03). assertBusObjects ran only in the start
raise; a push ensured a module's consumer only when a bus credential was
minted, which a module carried by the runtime never is.
The send's grant now runs the same derivation (assertOnSend) before the
memberships, on every push, cascade, plan send and rotation. A failure
is said in the send's output and raised as bus.objects.unasserted, which
the next send that asserts everything clears; the send itself goes on,
because the objects are the mesh's and holding every machine back for
one would turn one fault into all. Module consumers are each tried and
every failure named. The start raise stays as it was.
Every one of the 48 core failures of research 031 was found by a person
looking; the mesh's answers carried the fact for whoever asked and told
nobody.
- The condition store (to-be 45 §2): mesh-controller_conditions, one key
per open condition, written by compare-and-set so a person's silence
and the watchdogs never lose each other's word; every transition kept
ninety days in mesh-controller_condition-history and said as the
seat's events condition-raised / condition-changed / condition-cleared
(the condition at the top level, with event, at, change, why, show),
offered again while the bus is away. Raised and cleared by observation
only; a clearing reopened within ten minutes is the same condition with
its count up, its silence kept. Verbs: conditions, conditions show,
conditions silence (a hand act, at most a week), conditions history.
- ADR 0224's provider standing is the first kind, provider-failing, held
by the provider's events; the provider_standing table is no longer read
or written (left in place: dropping it is the operator's word).
- status leads with the open conditions, urgent first, and says all well
only with none open; conditions it cannot read are said and not well.
- The signals table compiled in, one watchdog loop over it every 30s: S1
heartbeat (3 intervals, asleep machines excepted, control node urgent
after 30 min), S2 report after a send, S3 plan tier, S4 event loop deaf,
S5 merge not acted, S6 ask lost, S7 call hung, S8 provider silent, S9
advisories, S10 self-check silent, S11 node tools silent, S13 stale
refusals; S12, S14, S15 deferred with their reasons. A row that cannot
see raises probe-failed and clears nothing. A test generated from the
table suppresses each signal inside and past its bound.
- The bus's advisories (maximum deliveries, a mesh consumer deleted) and
the controller's own slow consumer and refused subjects, said in the
mesh's words.
- doctor: the probe registry D1-D10 (D5 deferred) and DW, every five
minutes, each in thirty seconds; a probe that cannot run is never a
pass. D1 validates with mesh-host's own validator. Every run ends with
the doctor-heartbeat event mesh-watcher listens for.
- The controller is granted its new buckets, events, the two advisories
and $SRV.INFO; the node tools their tools-alive heartbeat. The streams
and consumers the controller asserts and the ones D6/D7 expect are one
derivation.
A controller restart lost every call's outcome, `status` composed the mesh
while its caller waited (18.6s live on 2026-10-06, past the 10s window), a
repair by hand left no trace, and the core's bounds had nothing measured to
be set from.
- calls: kept in the controller's bucket mesh-controller_calls (last 1000 or
14 days, answers bounded to 64 KiB), read by id across a restart; a
controller starting marks a stopped one's running calls abandoned; each
call names its caller from the inbox its answer goes to.
- status: the serving controller composes it at start, after news from a
machine, a build or an acting verb, and every minute; the verb answers the
last composition at once with when and how long it took. Composing resolves
each machine once instead of twice.
- hand-act log in mesh-controller_hand-acts: push (required through the seat),
plans stop/close, broker consumer-reset and the new hand-act record take
--why/--cause/--condition; `hand-acts` lists them and repeated causes;
status counts the week's.
- durations (migration 0066): apply (send to first report), heartbeat gap,
plan tier and build, recorded as heard; `durations` summarises them.
- the controller's seat row takes this binary's definition of its own verbs,
so the console no longer judges calls against an older build's schema.
- the controller is granted its two buckets' subjects.
The controller acted only on what its events consumer handed it, so a merge
the bus skipped left modules behind with nothing said. The stream is now read
back every five minutes on a single-filter consumer, and any merge that would
still move a module after ten minutes is said and acted on.
A push outlasted the console's 30s wait and, when it sent the bus its
changed user list, the broker's reload forgot the reply it may send:
the push happened and its caller was told it did not answer. Calls now
answer in full or as running with an id, a push answers before it
sends, refused answers are recorded on their call, and 'calls' reads
them back.
A push naming one machine reached the verb without it and pushed every
machine behind (hq issue 244). The controller now refuses any argument a
verb does not declare, any it composed its command line without, and a
switch that is not true or false; a push that names no machine says first
that it is the whole mesh. Tests walk every served verb: no argument is
ever ignored, and every flag of a verb's command, read from the source, is
in its schema or accounted for. plan gains files, push behind, builds and
plans limit.
The identity provider failed every consumer for a day and status called the
mesh well (hq issue 179). The controller now follows every provider's
provisioner.failing/recovered, keeps the newest failing word per provider,
machine and consumer (migration 0065), and status, its JSON and node show
name it until it recovers. Every module that receives contributions is
granted the two events, so no manifest can forget them.
A named push flushed every other machine whose declaration differed from
what it was last sent (hq ADR 0083). Under an upgrade policy of `record`,
or a plan still waiting on its first machine (ADR 0218), every machine
running the module differs, so `push <one>` sent the held build to all of
them (hq issue 259).
Each send now records which build of each module it carried
(node.sent_builds, migration 0061). The cascade, and the bus holder added
to a named push, skip a machine any of whose modules would move to a
build its policy records or an open plan has not sent it, and say which
module, which build, why, and that `push <node>` sends it. A machine
whose last send was not recorded is held until it is named. The named
machine itself, a whole-mesh push and `push --behind` are unchanged.
A queued ask could only be waited out and a running build only ended by
stopping the machine, which redelivered it elsewhere. The holder now serves
current, kill, pause and resume on its own machine's subjects; a kill ends
the build's process group and labelled containers and settles the ask as
failed, killed by hand; pause is kept in the workspace across a restart and
said on the bus. The controller writes cancelled ids to a cancelled set the
holder reads on taking an ask, closing the race a delete alone leaves.
Grants first could hold back the very declaration that lets the controller
issue them. The holder now goes first, then buckets and memberships, then
the rest; a membership that fails holds back only its own machine, and an
announcement whose send stopped at its grants is asked again. Whether the
holder must go first is read from a digest of the user list it was last
sent, not its whole declaration. Migration renumbered to 0058.
A module's new state reached every machine before the permissions to use
it, which came only with a later push. Memberships and buckets now go
before declarations, the machine holding mesh-broker goes first when its
user list must change, and a grant that fails sends nothing and is an
error so the rollout is retried.
Every assigned module was composed as a bus user, though only one declaring
a broker secret can ever be issued an account; the rest were named on every
status, plan and push as credentials never minted (137 now), burying the
real gaps. Their durable consumers are now derived from what the runtime
carries, so nothing they hear changes. The composed file is unchanged:
those users had no password and were already left out. Fixes hq issue 195.
assign and unassign say only what they changed on their node; push <node>
lists that node's, push to many counts each and points at status. The
once-per-change log is the serving controller's alone: a one-shot command
starts with no memory, so it logged every node on every call.
The raise at start was the only place buckets were asserted, so a module
registered and assigned since had none until the control plane restarted —
found on the first module to declare state.
A manifest names the state it keeps (state) and reads (reads); the controller
asserts a key-value bucket per name on every raise, grants owners write and
readers read (measured against a running server), issues each assignment its
buckets in the membership, and reports buckets nothing declares without
removing them.
Grants: a principal that serves tools subscribes $SRV.PING/$SRV.INFO and those questions under
each name it serves — its own and no other's; the tool runtime and people may ask. The controller
answers discovery for the mesh-controller seat in NATS's services format, one endpoint per verb it
serves, with the seat's description and schema. module list --json says which modules declare tools,
so the console expects an announcement only from those.
On 2026-10-02 a runtime assigned and applied on two machines was undone two seconds later by a
declaration that had the assignments of a minute earlier. Every path composes from the records at
compose time and holds the machines it sends — but the number went on at SEND time, after
composing, so a declaration composed before an assignment changed and sent after a newer one
carried the higher number, and the host, which rightly refuses a lower number, took the older
content as the mesh's newest word. The record of that send was never written either: it is written
after the declaration is away, on the sender's context, and the controller sending it was being
replaced in that very second — status read "applied, current" over a machine just told otherwise.
Now the number is taken before the composition reads anything, in every path, so what was composed
earlier is numbered lower however late it goes out and the host's refusal does what it is for; and
what was sent is written down on a context that outlives the sender, bounded, so a dying controller
still records what it told a machine. The `declare` command — a declaration a person sends by hand —
records its send too. Proven: compositions in one order and sends in the other keep the numbers in
composition order; a send is recorded after the sender's context is cancelled.
One verb in the row that this binary cannot run aborted the start, and a stale push that put an
older control plane back took the whole mesh off the bus for ten minutes — recoverable only by a
person running the binary outside its service, because the push that repairs it is one of the verbs
that had stopped being served. Now the verbs it knows are served, the ones it does not answer the
reason, and the start names them once.
take ends its preview with a digest and --yes names it, as the flip does; a
changed preview or an account older than the flip allows is refused. A module
the machine holds nothing for has nothing to compare, and --yes suffices. A
published port's reach is said as the machine reported it. Every secret the
module holds on the machine is listed with where it came from, and one the
mesh minted for a service whose data was found refuses unless --mint names it.
One judgement of a module's settings against its definition, in the catalogue:
settings set refuses what cannot compose or reaches nothing, naming node,
module, layer and key; Compose leaves out a module whose definition moved
under a stored setting, the envelope says so (left_out), plan and push say it
by name, and the machine is told everything else. A stray setting no longer
refuses the whole machine where it is read (issue 096).
The per-machine setting networks keeps a found network for a taken container,
on an adopted machine only; the container's declaration carries it and the
preview names it (rule 4).
The proxy answered every routed name to any request carrying it, so an
internal-only route would have been public under its internal name. Each
membership now carries what its module receives, from the same
composition as its received file, and every machine's private-network
address, the list the packet filter's "from the mesh" is. The proxy
follows its membership, serves internal names only to those machines and
itself, and keeps the file until the bus has spoken (novox/hq ADR 0167,
issue 191).
A tier whose module a later tier is built by waited for the machines running it to report after the
build — and relied on the catalogue's moved event to send them. A rebuild from the same source commit is
not a move the catalogue announces: the build machine rebuilt for a controller change kept its commit,
nothing sent it, and the plan waited on a report that would never come (2026-10-01, 19:45Z). The plan
now sends the machines running a gated module once, records when, and waits for the reports after that.
A module's dependencies are one relation in the catalogue — stands-on, packages, built-by, declared —
answered by one call. A merge takes what moved and everything reachable from it, sorts the set into
tiers (a code dependency in the same tier, a build dependency after its base is built, a runtime
dependency after the build machine is built and running; the build machine's own base comes first,
built by the one that runs), writes the plan to the store, asks the first tier and returns. Every
outcome advances the plan; a ticker advances what outcomes cannot; a controller replaced mid-plan
resumes it. status lists open plans and names one that has waited too long.
sendTo — the path a roll-out, a rotation and a secret change take — issued memberships after its
sends (ADR 0160); the push command, which sends the same declarations through its own loop, did
not, so the one command operators run issued none. Said once in the same words after the sends.
A stream publish waits for its acknowledgement as long as its context lives, and the server never
acknowledges a publish it refuses. Issuing memberships after a push used the daemon's own context, so
the one refused membership of 2026-10-01 (hq issue 183) held the controller's receive loop for good:
no report, no build outcome, no merge was heard until a restart (hq issue 185). Issuing one
membership is now bounded to ten seconds, and a push says how many could not be issued and stands —
the machines keep the shape they derive until the next push.
For every module on every machine the controller composes what that instance serves — its machine's
address always, the module's plain address in a queue when it is alone or its definition says its
instances are interchangeable — the verbs of the seats it holds at the seats' subjects, where its
events land, and what it may reach, resolved the same way for the modules it invokes. Published
beside the node's declaration on `mesh.assignment.<node>.<module>`, last per subject in a stream
that allows direct reads, and the account may read exactly its own. Composed from the same records
the bus's accounts are, so what a runtime serves and what its account may are one composition.
`instances: interchangeable` is the one fact a definition states for it.
The shape issued is the shape the mesh already had, so nothing moves when the membership arrives;
the runtime that reads it instead of deriving it is the next piece.
A seat's protocol lives in the store (migration 0047; seeded additively), a served verb carries its
description and schema, holding a mesh seat requires serving its verbs, a node-scoped seat's tool
carries the node, and the control plane serves status, nodes, node, modules, seats, builds, plan,
assign, unassign, push, build and tools on its seat by running the same commands (novox/hq ADR 0132,
ADR 0154, design 33). A grant of * reaches a role's tools; seat:<seat>.<verb> grants one.
novox/hq 04-ISSUES/107. The controller already held a per-node lock while
it composed and recorded each send; the order existed and was thrown away
at the wire. Each send now takes the next number for its node, one
higher than the last, under that hold and before the body exists — so
the number is inside what the mesh signs, and a replayed older
declaration cannot borrow a newer one's.
Zero is not sent. A host reads absence as "no order claimed", which is
the shape of every declaration before this, so nothing that worked
before changes for a machine sent nothing since numbering existed.
One subtlety, and it is the one that would have read every machine as
behind for ever: the mesh decides a machine is behind by comparing the
digest of what it WOULD send against what it DID send, and a number
changes the bytes. The read-only comparison composes with the number the
machine was LAST sent, not a fresh one, so it is byte for byte what was
sent when nothing else changed.
Hosts went first and every machine runs one that understands the field.
jochen
mesh-admin
jschoubben