The live tests reached one shared bus and assert, read and remove the mesh's own objects by
their fixed names, so packages run in parallel deleted what each other read and the suite
passed only one package at a time; a red suite read as noise. internal/testbus starts a server
per test, linked in at the nats-server release go.mod pins, and a test holds that pin to the
catalogue's bus image and to the facts snapshot's bus when there is one, so the tests never run
a bus the mesh does not. The waiter test read a timing (the most connections held at one look)
and now reads the state it means (the fewest held across the wait). make check runs the packages
in parallel under the race detector, with a timeout.
Every check the mesh had ran after a merge, on a machine: a manifest the node-engine refused
(236), an identity a real machine's name made too long (263). merge-gate raises the mesh as the
facts snapshot says it is and the mesh with the change, each in a throwaway store through the
controller's own records, composes every machine twice and validates it with the node-engine's
validator, and fails what the change breaks, naming the machine's roles and the module - plus a
manifest the judging controller cannot read, a consumer left out of its grant, a module removed
while a machine runs it, a new module the node-engine would refuse; it warns on a wide rebuild.
The forge's new head of a pull request becomes a check the controller asks of the build seat:
the head and, beside it, the controller the mesh runs, the catalogue, the host and the lab; a
throwaway store and bus of the versions the mesh runs; the repository's merge-check.sh in the
mesh's Go toolchain with no container runtime socket; then mesh-lab's replays. The verdict is
said as checked, an error never a pass, and nothing is recorded or registered.
A build that reported applied was sent everywhere; one that then did nothing, served
no tools or broke its machine's word reached every machine. Now the first machine is
judged by the component's health (the core's definitions, as doctor probes H-*, or a
module's own) three times over two minutes within ten; a failing gate puts the previous
build back there once, marks the build, and says it as a condition and an event.
Upgrades roll out by default; the bus is a planned step; a module deleted at its
source is not built (the public-acme plan failure).
The night's backup of the bus takes each stream through JetStream's snapshot
API, run by the nats module under its own account. The module holding
mesh-broker is composed that account: stream names and info, the snapshot
request, its flow-control acks, its own inbox — no write, which the writers
table checks. A bus module declaring anything else to say on the bus is
refused by module check rather than silently granted nothing. The genesis
user list is unchanged: the controller's grants are.
A module's data section says what it keeps and how precious it is; the backup holder's lines,
binding stickiness, retirement on unassign and D13's conditions follow from it, so issue 273's
empty replacement is said and an unassigned module's data is remembered, not forgotten.
A provider now waits for a person before retiring more than three consumers
or half of what it holds, and deletes only when asked. The controller is that
person's way in: it keeps waiting and rejected sets as conditions, answers them
with retire approve|reject, lists and deletes retired consumers through the
provider's own tools on its machine, records each act in the hand-act log, and
probes for anything retired longer than thirty days (D11).
Research 031 counted the repairs people made by hand: a push to unstick a
plan waiting on a report, a controller restarted to make an object again, a
plan closed, a consumer re-made from now. Each was the ordinary path taken
again by someone who noticed. The healer registry makes each a registered
response to one condition kind, with a budget, a settle and its event:
- H1 sent-not-reported: ask the machine's node-engine to report again
(mesh.node.<n>.ask.report); if it does not report what it was sent, send
it again, never moving a build a policy or a plan holds back
- H2 stalled: close a plan whose wait is superseded or finished
- H3 holder-silent / consumer-lost: the send's own assertion of the bus's
objects (issue 208's note)
- H4 consumer-behind: consumer-reset, only for a consumer the stream table
marks resettable (the controller's own events consumer)
- H5 is the identity provider's own repair (ADR 0224 §5), registered only
Success is the observation clearing the condition, never the healer; a spent
budget hands the condition to the operator, urgent, with what was tried, and
no healer touches it again. Every act is begun in the store before it is made
(migration 0070), kept in the condition's tried as "healer Hn" and said as
the seat event healer-acted; a heal is never a hand act. More than twelve acts
in an hour stop every healer until an hour after the last, said urgently.
Only the lease holder heals.
S15 is live: a cause repaired by hand twice in a fortnight raises
healer-wanted, naming the healer that was not enough where one exists. D6's
far-behind finding has its own kind, consumer-behind. Nodes are granted the
question; the controller's grant gains healer-acted (genesis lock in
mesh-host). `healers` lists the registry, the acts and the brake; status
counts the week's heals.
Two controllers could both act (issue 204), a reconcile's report could
overtake the apply after it and the digest decided (issue 267), and a grant
could make a second writer of a machine's report.
- The lease (internal/lease, ADR 0229): mesh-controller_lease key `holder`,
15 s age, renewed every 5 s by compare-and-set; the epoch is the revision
it was taken at. The gate is the clock (stops 3 s before expiry); a refused
renewal is a loss and the process exits; a holder that stops gives it back.
serve takes it before asserting the bus. Epochs kept in the store
(migration 0068 controller_epoch) as a floor: a bucket raised from nothing
is compacted past it. Unleased (no epoch, S12 urgent) only when nobody
holds it and the bus will not let it be written. A shell command acts
under the holder's epoch, or its own lease when none.
- Declarations carry `epoch` inside the signed envelope, only to a machine
whose latest account carried a report_sequence (mesh-host #35); would-send
is composed with the epoch last sent. Allot and the send both pass the gate.
- Reports: contract in internal/link/order.go (epoch, sequence,
report_sequence, older_than, refused_older). Accounts kept by epoch, then
sequence, then report sequence; older refused, counted; unordered reports
keep the digest rule. Plans by compare-and-set on a revision, with epoch.
Conditions and calls carry the epoch and are not written off the lease.
- S12 and S13 (naming the writer by epoch) watched, D5 run; reset of the
bucket said. Writers table compiled in and enforced in PermissionsFor; the
controller no longer publishes mesh.control.>. A contract per consumed
kind, and the empty-on-error lint over the repository.
- mesh-host pinned to its main with the epoch in the validator (D1 validates
the envelope as sent).
Needs mesh-host's genesis lock with the lease grant (mesh-host PR) for
TestTheInstallersFirstUserListIsWhatTheControllerWouldCompose.
A given own secret the module reads at start is held by nobody but that
module, so the mesh need not read it to replace it: secret rotate now
works on it, and a value given through secret accept is replaced on its
own after the module's first good start under the mesh. Only a value an
outside party issues (own-secrets "issued-by": "outside") or one the
module applies stays as given, refused with the reason.
Live on 2026-10-06, two of the first self-check's findings were its own:
- D8 asked every machine's node-intrusion-prevention.banned, and the
controller's grant did not name the subject: the bus refused it 24 times
and D8 timed out after thirty seconds instead of saying so. The verbs the
self-check asks are named in broker.VerbsTheSelfCheckAsks and granted
(mesh.seat.<seat>.tool.<verb>.*); each probe declares the seat verbs it
calls, askSeatTool refuses an undeclared one, and a test over the
registry fails a probe whose question the controller is not granted.
AskSeatTool now returns a refused publish at once ("the bus refused…")
instead of waiting out its timeout; D8 asks the machines in parallel.
- D10 read every machine as behind right after a push: a node-engine says
its version as the directory it is delivered into, the archive's digest
(31045596c83a, catalogue versionOf), and D10 compared that with the
build's commit (1545b00a). It now compares with the versions the
registered build is delivered as, and a hand-placed engine's commit.
Every one of the 48 core failures of research 031 was found by a person
looking; the mesh's answers carried the fact for whoever asked and told
nobody.
- The condition store (to-be 45 §2): mesh-controller_conditions, one key
per open condition, written by compare-and-set so a person's silence
and the watchdogs never lose each other's word; every transition kept
ninety days in mesh-controller_condition-history and said as the
seat's events condition-raised / condition-changed / condition-cleared
(the condition at the top level, with event, at, change, why, show),
offered again while the bus is away. Raised and cleared by observation
only; a clearing reopened within ten minutes is the same condition with
its count up, its silence kept. Verbs: conditions, conditions show,
conditions silence (a hand act, at most a week), conditions history.
- ADR 0224's provider standing is the first kind, provider-failing, held
by the provider's events; the provider_standing table is no longer read
or written (left in place: dropping it is the operator's word).
- status leads with the open conditions, urgent first, and says all well
only with none open; conditions it cannot read are said and not well.
- The signals table compiled in, one watchdog loop over it every 30s: S1
heartbeat (3 intervals, asleep machines excepted, control node urgent
after 30 min), S2 report after a send, S3 plan tier, S4 event loop deaf,
S5 merge not acted, S6 ask lost, S7 call hung, S8 provider silent, S9
advisories, S10 self-check silent, S11 node tools silent, S13 stale
refusals; S12, S14, S15 deferred with their reasons. A row that cannot
see raises probe-failed and clears nothing. A test generated from the
table suppresses each signal inside and past its bound.
- The bus's advisories (maximum deliveries, a mesh consumer deleted) and
the controller's own slow consumer and refused subjects, said in the
mesh's words.
- doctor: the probe registry D1-D10 (D5 deferred) and DW, every five
minutes, each in thirty seconds; a probe that cannot run is never a
pass. D1 validates with mesh-host's own validator. Every run ends with
the doctor-heartbeat event mesh-watcher listens for.
- The controller is granted its new buckets, events, the two advisories
and $SRV.INFO; the node tools their tools-alive heartbeat. The streams
and consumers the controller asserts and the ones D6/D7 expect are one
derivation.
A cancel writes the ask's id into the seat's cancelled set before deleting
the ask, and a write is a publish to the bucket's subject, which the
controller was not granted: against a server holding exactly the
controller's composed list, every cancel timed out.
A controller restart lost every call's outcome, `status` composed the mesh
while its caller waited (18.6s live on 2026-10-06, past the 10s window), a
repair by hand left no trace, and the core's bounds had nothing measured to
be set from.
- calls: kept in the controller's bucket mesh-controller_calls (last 1000 or
14 days, answers bounded to 64 KiB), read by id across a restart; a
controller starting marks a stopped one's running calls abandoned; each
call names its caller from the inbox its answer goes to.
- status: the serving controller composes it at start, after news from a
machine, a build or an acting verb, and every minute; the verb answers the
last composition at once with when and how long it took. Composing resolves
each machine once instead of twice.
- hand-act log in mesh-controller_hand-acts: push (required through the seat),
plans stop/close, broker consumer-reset and the new hand-act record take
--why/--cause/--condition; `hand-acts` lists them and repeated causes;
status counts the week's.
- durations (migration 0066): apply (send to first report), heartbeat gap,
plan tier and build, recorded as heard; `durations` summarises them.
- the controller's seat row takes this binary's definition of its own verbs,
so the console no longer judges calls against an older build's schema.
- the controller is granted its two buckets' subjects.
A push outlasted the console's 30s wait and, when it sent the bus its
changed user list, the broker's reload forgot the reply it may send:
the push happened and its caller was told it did not answer. Calls now
answer in full or as running with an id, a push answers before it
sends, refused answers are recorded on their call, and 'calls' reads
them back.
The identity provider failed every consumer for a day and status called the
mesh well (hq issue 179). The controller now follows every provider's
provisioner.failing/recovered, keeps the newest failing word per provider,
machine and consumer (migration 0065), and status, its JSON and node show
name it until it recovers. Every module that receives contributions is
granted the two events, so no manifest can forget them.
The seat now covers /etc/hostname too. The migration keeps the old name as
an alias so hosts, still assigned while machines move, holds the same seat.
Claims were compared by spelling, so the old and new module would both have
held it on one machine; they are now compared by the seat they resolve to.
Nothing has claimed node-dns-resolver since the mesh moved to one resolver
(hq ADR 0194); seeding never removes a row, so a migration deletes it.
resolv.conf stays the mesh's only while the network manager is told to keep
off it, which the node-uplink holder does (ADR 0117). A seat's Needs makes
that a dependency checked at assignment by the ADR 0207 mechanism (hq ADR
0220). The two-claimants test keeps its intent with a synthetic module now
that resolved-split-dns leaves the catalogue.
A registered replay asked now outranked newer asks of its module, and a plan
took any later outcome as its answer. A replay is now refused while the module
is asked anywhere, a plan module asked under an id is answered by that id
alone, and a rebuild of a commit asks what the module follows. An ask handed
back after a restart no longer reads as dead; a cancel that meets a start is
withdrawn; pause holds for an ask fetched as it lands; kill removes containers
before and after the build ends and says whether its outcome went out; a
holder may say only its own machine is paused.
A queued ask could only be waited out and a running build only ended by
stopping the machine, which redelivered it elsewhere. The holder now serves
current, kill, pause and resume on its own machine's subjects; a kill ends
the build's process group and labelled containers and settles the ask as
failed, killed by hand; pause is kept in the workspace across a restart and
said on the bus. The controller writes cancelled ids to a cancelled set the
holder reads on taking an ask, closing the race a delete alone leaves.
A consumer made with the server's default replays everything a stream that
keeps history holds: the controller's EVENTS consumer, re-made that way,
replayed a week of merges and builds one at a time and held every new one
behind them. FromNow makes it start at the end; broker consumer-reset
re-makes a stuck one from now, refusing a work queue. hq issue 244.
Every assigned module was composed as a bus user, though only one declaring
a broker secret can ever be issued an account; the rest were named on every
status, plan and push as credentials never minted (137 now), burying the
real gaps. Their durable consumers are now derived from what the runtime
carries, so nothing they hear changes. The composed file is unchanged:
those users had no password and were already left out. Fixes hq issue 195.
The raise at start was the only place buckets were asserted, so a module
registered and assigned since had none until the control plane restarted —
found on the first module to declare state.
A manifest names the state it keeps (state) and reads (reads); the controller
asserts a key-value bucket per name on every raise, grants owners write and
readers read (measured against a running server), issues each assignment its
buckets in the membership, and reports buckets nothing declares without
removing them.
A module's long-running code is a bundle the runtime launches, and the runtime is its bus: it binds
the module's own durable consumer — EVENTS, <node>_<module>, still the controller's to make from the
module's principal — and acknowledges what the module's code took. So the runtime principal is
granted, for each carried module that consumes, exactly what that module's own principal has for
its consumer: its info, its next message, its ack subject. Nothing is pushed to it; it pulls. ADR
0175's "consumes nothing" no longer holds. Memberships need nothing new: the consumer's name is
derived, as the module's own runtime derived it.
Endpoints named <seat>__<verb> with the metadata the console identifies them by (kind, module,
tool, seat, scope); $SRV.STATS answered with its identity and endpoints, nothing counted. Grants:
STATS beside PING and INFO, and the tool runtime may answer under its own name, since it announces
everything it carries as one service — the bus lets it answer each request once.
A manifest names its toolchain by language, not in build.on, so the planner did not know a bundle
depends on the module that publishes its toolchain and built the two in one tier: the bundle
against the old toolchain, recorded as built from the new commit. The edge is read from the
manifest, so it holds before any build recorded it, and a toolchain that moves rebuilds every
bundle compiled in it.
Grants: a principal that serves tools subscribes $SRV.PING/$SRV.INFO and those questions under
each name it serves — its own and no other's; the tool runtime and people may ask. The controller
answers discovery for the mesh-controller seat in NATS's services format, one endpoint per verb it
serves, with the seat's description and schema. module list --json says which modules declare tools,
so the console expects an announcement only from those.
Left for a hand, the hand re-made it with the server's default — everything the stream holds — and
on 2026-10-03 that replayed every build ask since 1 October into the catalogue. Re-made with
deliver-new instead: nothing acknowledged comes back; what was in flight is said and asked again.
A holder built for a pull worker cannot bind a push one — `cannot pull subscribe to push based
consumer` — and on 2026-10-03 the build machine rolled before the controller that would have
redefined its worker, restarted on that for an hour, and nothing could build the controller that
would have ended it. The server cannot change a consumer's type in place, so the assertion re-makes
one of the wrong type: on a work queue nothing is lost, because what was acknowledged is gone from the
stream and what was not is delivered again from the start. On a stream that keeps its history it is
said and left, since a re-made consumer replays what this one acknowledged (issue 156), and that is a
person's call. Proven against a real bus: a push worker with one ask acknowledged and two pending is
re-made as pull, a pull subscription binds, and takes exactly the two.
A controller that asked node-build-agent from its first run would queue every build where nothing
pulls, and the build that registers build-agent — the first holder — would be among them. So the
role is chosen at ask time from the catalogue: the current role when any assigned module claims it,
the retired one while only the builder does, the current one when neither. Outcomes are followed on
both seats, the controller may publish to both, and a build's log is read under whichever role did
it; a machine on the retired role is proven on the bus to take that role's asks. The switch order
is written where the role is named, and the retired half is marked for removal with the seat row.
The worker a holder bound was a push consumer in a queue group with one ask in flight: right for
one holder, and with two it would still be a queue of one — the server hands a pushed ask to
whichever subscriber it picks, busy or not, and the in-flight cap is per consumer, not per holder.
Now the worker is pulled: every machine holding the seat binds the same durable and fetches one
ask when it has finished the last, so an idle machine is the one that takes the next, the asks in
flight are bounded by the holders working, and nothing is delivered that nobody asked for — which
is also what ended the race issue 186 describes. A holder's grants trade the delivery subject for
MSG.NEXT on the worker; the ack grant and the heartbeat that keeps a long build alive stay.
Proven against a real bus: the build round trip, a backlog taken by a machine that arrives later,
and work handed back by one machine coming round again.
One build machine built everything, in a queue of one, because the seat was mesh-scoped and a
mesh seat has one holder. ADR 0190 makes building a node role: node-build-agent, held on every
machine that builds, with the work asked of the role and taken by whichever holder is idle. The
work subject of a node-scoped seat carries no node — that token is for a seat's tools, asked of
one machine (design 33 §4) — so holders on several machines read one queue; a test now says so.
The retired mesh-build-machine row stays while the builder module's registered manifest claims
it: a claim to a seat the mesh no longer defines is refused, and the machine holding it would be
unresolvable until build-agent replaces it. Removed once no manifest claims it.
The installer's genesis template (in the host's repository) still grants the controller the old
seat's subjects; its test here says so until that template names node-build-agent.
Where node-tools is in a node's set, the declaration ends with one process: the runtime module's own
bundle, run from its one entrypoint by its language's interpreter, told in MESH_TOOL_MODULES every
<module>=<file> the machine's bundles load, where its credential is (the module's own broker secret
as this node places it), and — on a machine with an operator account — who the operator is, running
as that account so a tool that needs root can escalate as the operator would. Restarted when any
bundle it loads or the credential changes. A machine with no account runs it as root without the two
operator words; a machine without the runtime is sent nothing new.
A bundle says which of its entrypoints the runtime LOADS (`loads`), because one bundle may carry a
daemon beside its tools and importing the daemon into the runtime would start it there; absent, a
module declaring tools has every entrypoint loaded. And the TypeScript toolchain is rooted at the
module, so an entrypoint lands at the path it is named by — the runtime loading bundles by their
declared paths is what made the compiler's common-directory default visible.
Where the node-tools module is assigned, the machine's bus user list gains one principal of kind
node-tools in place of that module's own: it may subscribe every carried module's tool namespace
and every held seat's verbs on its node, read and follow every membership on its node, call any
tool anywhere, answer what it is asked — and consume nothing, because tools are what it runs.
Every other module keeps its own principal, so a module still serving tools from its container
holds its own credential until it moves.
Named exactly as the module it stands for, so `module issue` and `rollout mint` deliver its
credential through the path a module's already takes, into node-tools' own `broker` secret. The
runtime module's name is one constant in each of the broker and catalogue packages, held to one
string by the agreement test, because a rule turns on it.
The proxy answered every routed name to any request carrying it, so an
internal-only route would have been public under its internal name. Each
membership now carries what its module receives, from the same
composition as its received file, and every machine's private-network
address, the list the packet filter's "from the mesh" is. The proxy
follows its membership, serves internal names only to those machines and
itself, and keeps the file until the bus has spoken (novox/hq ADR 0167,
issue 191).
With the worker consumer's default of many deliveries in flight, every ask behind the one being
built was delivered at once, left unacknowledged for the length of the build, redelivered after the
ack wait and dropped after the fifth time: on 2026-10-01 twenty-six of forty-three builds asked in two
minutes were never built and the queue read as empty (hq issue 186). The holder's worker now has one
in flight, and a running build tells the bus it is still working, as the controller's long handlers
do, so a build longer than the ack wait is neither redelivered nor counted out.
The server refused every membership the controller published after a push (2026-10-01, Permissions
Violation for Publish to mesh.assignment.<node>.<module>): the controller's own grant named the
control, node and JetStream subjects and not the assignments it alone issues (hq ADR 0160). Broker
golden regenerated; one line differs.
For every module on every machine the controller composes what that instance serves — its machine's
address always, the module's plain address in a queue when it is alone or its definition says its
instances are interchangeable — the verbs of the seats it holds at the seats' subjects, where its
events land, and what it may reach, resolved the same way for the modules it invokes. Published
beside the node's declaration on `mesh.assignment.<node>.<module>`, last per subject in a stream
that allows direct reads, and the account may read exactly its own. Composed from the same records
the bus's accounts are, so what a runtime serves and what its account may are one composition.
`instances: interchangeable` is the one fact a definition states for it.
The shape issued is the shape the mesh already had, so nothing moves when the membership arrives;
the runtime that reads it instead of deriving it is the next piece.
`invokes: [<module>.<tool>]` now grants `mesh.mod.<module>.tool.<tool>` and the same with the
machine as its last token, which is how a call reaches one machine's instance. The broker
credential the mesh writes carries `claims`: each seat the module claims, its scope, and the verbs
the seat promises, so the runtime serves them on the seat's subjects; the holder's grant, composed
from the holding, is what admits the subscription.
The build-machine seat emits `started` and `log.<build id>` beside `built`. Every line the builder
speaks — each step, each command with its duration, and on failure the command's own output — goes
to stderr as before and onto the bus under the build's id, one subject per build, kept a week in
EVENTS with every other event. `builds --log <id>` reads it back from the stream with a consumer
that is gone when the reading is done, on the command line and as the controller's seat verb;
`builds` lists each build's id and `build` says the id it asked with.
Lines are core publishes with a sequence number, so a build is not slowed by an ack per line and a
gap is visible; `started` and `built` are awaited into the stream. The seat protocol widens
additively at the controller's next start; the holder's grant follows on the broker node's next
composition.
A merge's handler builds for minutes and keeps its own delivery alive; the announcements handed over
behind it timed out on the client and came back, and a merge that came back rebuilt what it had just
built, five times over (novox/hq issue 175). MaxAckPending 1 on the events consumer: the server holds
the rest.
ADR 0121 decided it and deferred it as a delivering-seat migration; ADR 0122's aliases made it one
update and one alias (migration 0048). The former name resolves to it forever (novox/hq ADR 0156,
issue 123).
A seat's protocol lives in the store (migration 0047; seeded additively), a served verb carries its
description and schema, holding a mesh seat requires serving its verbs, a node-scoped seat's tool
carries the node, and the control plane serves status, nodes, node, modules, seats, builds, plan,
assign, unassign, push, build and tools on its seat by running the same commands (novox/hq ADR 0132,
ADR 0154, design 33). A grant of * reaches a role's tools; seat:<seat>.<verb> grants one.
mesh-admin
jochen
jschoubben