With no plan left to name a build or a send to keep the plans, a want names what a collection must keep, and the
composition is where the provisioner's plans are composed for the assignments mirror (ADR 0539 clauses 1 and 5);
the assignments no longer name a node's build, so the controller replaced is the newest earlier build of it.
A sweep read what it keeps before what was ever recorded, so a take-in's
build recorded between the two reads, by a concurrent take-in off the
plans' hold, was eligible and let go of seconds after the take-in read
it; a new module's build is unkept until it is registered. Now the sweep
reads the records first and keeps what a build recorded within the hour
or an open plan names, and names every reference it lets go of; a take-in
holds every artifact in the store before recording, failing on one the
store lacks; and RecordBuild clears artifact_collected as RecordMirrored
does.
A found ended or pending key is skipped and a pulled-in key is created only where no key exists. RecordSentWith and RecordSentUnder set sent_builds = null in their own update of the node row, so a machine delivery 2 has sent is never copied afterwards. These two updates and the copy's own null are the only writes to the column.
Every method of builds, build requests, rollouts, merges and assignments reads and writes the chain alone: the else branches that read the tables, the movers and MoveChain with their tests, chain_moved reads, ChainReads and the flags that switched a kind, the store's assignments mirror (KeepAssignmentPlans needs none of it: it uses only the AssignmentMirrored type, mirrors.go, and AssignmentValues, provisioning.go) and its tick, the move code of the controller's start, and ForTest's chain_moved row. The listed store tests of the move go with their functions; the tests of the readers, requests, assignments and plans that lived beside them stay in chain_store_test.go. Tests that replayed old data migrations over the build, assignment and pin tables are removed with the tables. The static test that no plan is written but through SavePlansMoving now reads the plan's chain path.
SentBuilds answers from the keys of the node's assignments that name a commit, known when the node was sent anything and no key says its commit unknown. A send that names its builds writes the commit, build and rollout on each carried module's key (a module nothing assigns gets a standing key marked pulled-in, never a state) and clears what every other key of the node named; a send by hand writes sent-unknown, a time, on every key and clears their commits. A failed write of these fails the record of the send. node.sent_builds is no longer written. The plan pass ends a pulled-in key only when no send carries it.
A send that records what a machine was sent writes, in the assignment of each module it carried a build of, the build, what it published, the commit, the rollout that sent it and since when: filled where the assignment names nothing, replaced when the commit moved, never blanked by a push by hand. A plan's sends are made as the plan, so they name it, and the plan lists the assignments each send changed with the build and commit before and after, the builds it carried and the steps taken. The builds, plans and node verbs say the links. node.sent_builds is written and read as before; SentBuilds from the assignments follows with the delivery that drops the column. Acceptance 10 is a test that enters by a merge heard and cut into a plan and writes no key by hand.
Once chain_moved records the kind, Assign, Unassign and Assigned write and read the keys <node>.<module>, and every join of the store that read assignments is made from the bucket: where a module runs, what is behind, what a machine awaits, a given value due for replacement, taking a module, and whether a module may be forgotten (refused while a standing assignment names it). An unassignment ends the key with a marker and keeps its earlier values, as it removed the row; no ended or pending mark is written yet. The generation rises with each change, and a seat held by an assignment goes with it: the foreign key from seat_holding is dropped by migration 0104 and the controller says both. The provisioner's plan is kept in the values by a pass of the tick that writes that one field by compare-and-set; the assignment-changed event is said from what the bucket holds, whoever wrote it. The start moves the rows with what each machine was last sent, keeping a plan the mirror put in a value.
Once chain_moved records the kind, a plan is a key of the rollouts bucket under its id and the open batch is the same key in state assembling. A save refuses a copy another save moved since (the plan's own Revision, kept in its value, counts saves) and keeps the merges the record holds, so a copy read before a merge moved cannot lay it back. The merges a batch or walk answers are the record's held field, with the columns the table had; a merge no plan answers stays, flagged, in the record that held it last, or in the record kept for those. A merge moved to a plan is written to the receiving record first and then taken from the giving one; one found in both is read from the one it entered last and repaired at the next cut. Plans and merges are read from a view kept by one watch of the bucket. A bus that does not answer, a lease lost and a key moved are now not-now like the store being away, so a merge whose plan could not be written is held and handed over again, never acknowledged. The start moves release_plan and batched_merge into the bucket, one kind, counted against both tables.
Once chain_moved records the kind, RecordBuild and every build read go through the chain (reuse key and published added); before that the tables are read as before. The serving controller copies the rows at its start, holding the lease: idempotent per key, rows counted against keys, the largest value measured, a refusal (urgent condition, nothing switched) on a count that differs or a value over the bus's largest, and a re-read when the table took a row meanwhile. ForTest attaches the chain in memory with builds moved; ForTestBefore leaves it unmoved. Build requests, gates and the other kinds are still in their tables.
Review of #283: a drain whose replacement was withdrawn held the agent for
ever with its health green; the gate trusted the engine's until; a machine
of the rest waiting at the bound read as silent and its apply time held the
wait; an unknown mesh-builder word exited 0; plan said nothing of a busy an
older engine is not sent; and a node with a report and no send broke the
reading of every last report (the store test's NULL).
mesh/delivery-group group feat/335-a-busy-container-is-replaced-when-idle rejected: its order contradicts itself: novox/mesh-catalog@190608193764, novox/mesh-c…
A merge into mesh-controller replaced every build agent and threw away the
checks in flight (issue 594). The manifest's container busy is checked and
sent only to an engine that reads it; a report's busy waits are kept, the
gate starts at the report that says the container replaced, and the wait is
a phase of the delivery time. mesh-builder busy drains the agent and answers
while a build is in hand (novox/hq ADR 0335).
The walk records its send to the rest only after every machine was sent,
which takes 20-28 s, so the machine sent first fell outside the 2 s slack,
its report was missed and the phase ended at the next walk's send to it.
Keep each machine's own send time with what it was sent, match its apply
duration by that send and digest, and show each machine's report in the
walk's phases (ADR 0282 decision 6).
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
On 2026-10-09 a release's gate on the control node read the machine's
report against a newer send another plan had just made there, failed
three builds the machine had reported healthy, and put them back on
every machine to a controller older than the store's schema; that
controller then passed the newer plan's gate from its own health.
- A gate keeps what its send carried (digest, sequence) and reads the
report against it; a report on the last send is on it too.
- A gate judges only the build the machine was last sent: another build
there supersedes the judging — no verdict, nothing put back.
- A controller is told its build (MESH_CONTROLLER_VERSION, ${version}
in a process's env) and records how far it reads the store's schema;
a put-back to a build that reaches less, or never said, is refused
and the current build kept, said as urgent.
- A release's open gate holds other sends of its modules there, and a
plan's own first send waits on it.
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/delivery covered after its failure: a later walk that contains it was delivered: plan-1791658180097457523 (novox/mesh-controller at a2a671ad into ma…
The node-engine's own since starts again at every restart, so an agent that
restarted the engine in a loop kept agent-can-become-root quiet for ever (the
review of 2026-10-09). The controller now keeps when it first saw the verdict
waiting for the setuid search (migration 0085), forgets it at the next
complete verdict, and raises once the engine's own bound has passed since;
the bound and the pending reason are read from mesh-host's rootsearch.
The generic command verb ran node account and node agent-account, so an agent
could name itself the operator account and have the next send grant it root
(hq ADR 0266 review). Refuse every node subcommand but list and show through
any verb; refuse the operator account as the agent account in both
directions and well-known service accounts as an agent account; and count a
verdict heard more than 15 minutes ago as not judged, so stopping the
node-engine cannot freeze a healthy one. Re-pin mesh-host to its review head.
On the control node every agent ran as the operator's account, which has
passwordless sudo, so an agent could become root without a person (hq ADR
0266). A node now names an agent account at the controller's terminal only;
the agent's module declares it never to become root, the node-engine judges
that, and the self-check (DA) raises agent-can-become-root while it does not
hold, so ADR 0259's router can rest on it.
token issue --overlay-key records the key the machine made, binds the
token to it, gives the machine its address and makes it a peer of the
hub, pushing the hub before the token is shown. The token carries the
hub's tunnel and the bus at its holder's address on the private network,
and enrolment refuses any other key (novox/hq ADR 0169). The bus is no
longer public, so a machine outside the mesh can join only this way; a
token without a key is still what the machine running the bus joins its
own mesh with. Also a token verb, which says it replaces running the
command by hand and adding a peer to the hub with wg.
Between assign and push a module's own secrets are not made yet; D1 composed
without making them and raised an urgent 'nothing can be sent' that the next
push resolved silently. D1 now composes as the push would (Foreseeing): a
secret the push makes gets a stand-in and is named, one the push is refused on
is refused with the push's words. Waiting is said only past 30 minutes, as a
warning. D3 and D13 expect a holder only once its machine was sent it and
reported or had ten minutes to.
Two controllers could both act (issue 204), a reconcile's report could
overtake the apply after it and the digest decided (issue 267), and a grant
could make a second writer of a machine's report.
- The lease (internal/lease, ADR 0229): mesh-controller_lease key `holder`,
15 s age, renewed every 5 s by compare-and-set; the epoch is the revision
it was taken at. The gate is the clock (stops 3 s before expiry); a refused
renewal is a loss and the process exits; a holder that stops gives it back.
serve takes it before asserting the bus. Epochs kept in the store
(migration 0068 controller_epoch) as a floor: a bucket raised from nothing
is compacted past it. Unleased (no epoch, S12 urgent) only when nobody
holds it and the bus will not let it be written. A shell command acts
under the holder's epoch, or its own lease when none.
- Declarations carry `epoch` inside the signed envelope, only to a machine
whose latest account carried a report_sequence (mesh-host #35); would-send
is composed with the epoch last sent. Allot and the send both pass the gate.
- Reports: contract in internal/link/order.go (epoch, sequence,
report_sequence, older_than, refused_older). Accounts kept by epoch, then
sequence, then report sequence; older refused, counted; unordered reports
keep the digest rule. Plans by compare-and-set on a revision, with epoch.
Conditions and calls carry the epoch and are not written off the lease.
- S12 and S13 (naming the writer by epoch) watched, D5 run; reset of the
bucket said. Writers table compiled in and enforced in PermissionsFor; the
controller no longer publishes mesh.control.>. A contract per consumed
kind, and the empty-on-error lint over the repository.
- mesh-host pinned to its main with the epoch in the validator (D1 validates
the envelope as sent).
Needs mesh-host's genesis lock with the lease grant (mesh-host PR) for
TestTheInstallersFirstUserListIsWhatTheControllerWouldCompose.
The last report stored per node decides whether a release plan moves on,
and it was whichever arrived last. A report about a declaration the mesh
has moved past now records what it says about the machine but leaves the
account of the apply alone, so arrival order cannot undo the newer.
A named push flushed every other machine whose declaration differed from
what it was last sent (hq ADR 0083). Under an upgrade policy of `record`,
or a plan still waiting on its first machine (ADR 0218), every machine
running the module differs, so `push <one>` sent the held build to all of
them (hq issue 259).
Each send now records which build of each module it carried
(node.sent_builds, migration 0061). The cascade, and the bus holder added
to a named push, skip a machine any of whose modules would move to a
build its policy records or an open plan has not sent it, and say which
module, which build, why, and that `push <node>` sends it. A machine
whose last send was not recorded is held until it is named. The named
machine itself, a whole-mesh push and `push --behind` are unchanged.
Grants first could hold back the very declaration that lets the controller
issue them. The holder now goes first, then buckets and memberships, then
the rest; a membership that fails holds back only its own machine, and an
announcement whose send stopped at its grants is asked again. Whether the
holder must go first is read from a digest of the user list it was last
sent, not its whole declaration. Migration renumbered to 0058.
novox/hq 04-ISSUES/107. The controller already held a per-node lock while
it composed and recorded each send; the order existed and was thrown away
at the wire. Each send now takes the next number for its node, one
higher than the last, under that hold and before the body exists — so
the number is inside what the mesh signs, and a replayed older
declaration cannot borrow a newer one's.
Zero is not sent. A host reads absence as "no order claimed", which is
the shape of every declaration before this, so nothing that worked
before changes for a machine sent nothing since numbering existed.
One subtlety, and it is the one that would have read every machine as
behind for ever: the mesh decides a machine is behind by comparing the
digest of what it WOULD send against what it DID send, and a number
changes the bytes. The read-only comparison composes with the number the
machine was LAST sent, not a fresh one, so it is byte for byte what was
sent when nothing else changed.
Hosts went first and every machine runs one that understands the field.
novox/hq 04-ISSUES/087. A host refuses a declaration carrying a field it
does not know, and refuses it WHOLE — deliberately, because that keeps a
half-understood declaration off a machine. It makes every new declaration
field a flag day: hosts first, then the controller. The mesh had no record
of which host any machine ran, so that order was kept by somebody
remembering it, and a machine that refused for this reason reported a
failure with nothing saying why.
The machine has reported its host version since ADR 0141. The
controller's own copy of the report did not have the field, so it was
unmarshalled into nothing and thrown away on arrival. It has it now,
records it, and shows it in `node show` — "not reported" rather than
blank, because a machine that has not said is not a machine running
nothing.
Status says which machines run an older host than another machine does,
and which is newest. Deliberately disagreement rather than staleness:
nothing delivers a host version yet (ADR 0141, accepted and not built), so
the mesh holds no canonical current version and cannot honestly say a
machine is behind THE host. What it can say is that the oldest host in the
mesh is what the mesh may send.
A machine that has reported nothing is left out rather than called
behind. Versions compare as strings, which suits the timestamps and
commits this mesh uses and is wrong for a scheme where "10" sorts before
"9" — said in the code, at the place that would have to learn.
novox/hq 04-ISSUES/146. The composed user list names an enrolment user for
every machine with a live token and nothing minted a credential for it, so the
composer left it out as a user with no password — and every enrolment since the
mesh moved to this bus was refused before the mesh heard of it. The comment
above the issuing code already said the account is created before the token is
handed over; now it is. Recorded rather than minted, because the token's secret
is the password.
And 'broker accounts', which composes the same list the declaration carries and
writes it to standard output. For genesis, where no declaration can reach the
machine running the bus because that machine is not yet a node. It says what it
composed; whoever is raising the machine places it. A control plane that wrote
the file itself would have to learn where the bus keeps its configuration and
how to make it reload, which is the module's knowledge.
The forward chain blocked everything passing through the machine and then allowed
the machine's own containers back by naming their address ranges: 172.16.0.0/12 and
192.168.128.0/17 fixed here, the rest recorded per machine by 0043. Every way of
keeping that list correct fails — a constant describes one machine, a recorded range
goes stale in silence and cannot tell a network the mesh made from one a predecessor
left behind, and generating it from the modules would put half the rule set on the
machine.
The mesh has no position on a container reaching outward: that is not a port opened
to anybody. So both chains are written around the links traffic arrives on. What did
not arrive from outside is accepted in one line; what did meets the declared rules.
The tunnel is named beside the outward links rather than treated as inside, or a port
nothing declares would be reachable from every machine in the mesh.
A machine that has not reported an outward link is sent no filter and keeps the one
it has, refused where a person reads it rather than as a rule set that will not load.
Removes the two constants, `node networks`, and the column behind it. novox/hq ADR
0140, superseding 0137 and 0139.
The derived filter denies forwarding by default and then allows the container
runtime's two default pools, named in this code with a comment saying a machine
configured otherwise needs to say so -- and no way to say it. So the filter was
right on a machine using the defaults and silently wrong on any other.
Measured today: flipping a workstation to the derived filter cut egress for five
of its container networks and for every network its test beds create, because
those come from ranges the defaults do not cover. Nothing reported a fault; the
guests just could not reach anything, while the machine reported it had applied
what it was told.
A node-level fact beside the public domain, because the machine routes them and
the module that loads the filter may be replaced. Added to the defaults, never
replacing them. Their guests also keep address and name service, without which a
network does not work at all, and the converge preview now says what a machine
routes instead of leaving it to a sentence about what it cannot preview.
The pipeline was observable from a merge to an artifact and went dark where it touched a machine: a
node's report is control traffic only the control plane reads, so nothing said which version a
machine runs, or that it refused to (novox/hq ADR 0134). The control plane now states both under the
seat it holds — a role's events belong to the role and keep their address when the holder is
replaced — and only when the report is news, because a machine reconciles every minute and a fact per
report would be a fact per minute per machine.
Whether a report is news is the store's answer: it holds the previous one, so the listener returns it
and the server states the fact. That also gives the catch-up replay a subject the controller may
publish: it was published as a module's event from a module called "control-plane", which does not
exist, so the controller's own account refused it and every catalogue that asked what it missed was
answered with nothing.
Both branches changed the seat set from the same starting point, so every number
collided and every `mesh-*` name existed twice. The trunk's numbers and names win:
this branch's records became 0129/0130 and its migrations 0037/0038, and the
hardcoded rename map gave way to the trunk's `seat_alias` table — a rename is a
row now (ADR 0122), not a recompile.
Three of my checks were wrong and the merge is what showed it:
A seat with an empty protocol is a marker, not an incomplete declaration. Most
node-scoped seats are markers — which module is this machine's packet filter —
and refusing one refused most of the set, the showcase module included. A
mistyped field name is already refused by the parser, so an empty protocol was
written as one deliberately.
A claim on a seat this manifest does not declare is not the parser's to judge. A
module may hold a seat another module declared; that is the whole reason ADR 0126
has callers name the seat and not its provider. Whether the seat exists is a fact
about the catalogue, so the refusal is at registration, where every declaration
is in view.
And a seat may share a name with the provision it delivers. `git`, the npm
registry and the artifact store still do, because renaming a delivering seat
cascades to every consumer requiring it, with a window where a holder stops
resolving mid-flight. The trunk deferred exactly those three on purpose.
Full suite green against a real NATS and store.
A node carries its operator account (name + home; migration 0036, Node.Account,
SetAccount, 'node account' CLI). The account and its home are offered as
machine facts ${machine:account} / ${machine:account-home}, and machineInto
now resolves placeholders in a resource's path and owner (not just content), so
a module writes into a person's home naming what it cannot know. A RosterFile
gains Home: the file is placed under the account's home and chowned to it, its
template sees each node's Account, and a machine with no account gets none —
this is how the ssh Host blocks for every node reach a person's ~/.ssh. Roster
carries per-node accounts (Rendering.Accounts). Tested, including ssh-client
composed end-to-end. Not deployed.
The outbound half went behind `Bus` and the transport stopped reaching its
callers; this is the other half, and the larger one. Every handler took
`amqp.Delivery`, so the serving loop could not move to another bus without
moving enrolment, reports, builds, upgrades and catch-up with it in one breath.
`Control` states one message in the mesh's words — took it, dropped it, or held
it for the store — and `Inbound` is where messages come from. The AMQP
implementation is today's loop moved rather than changed: same queues, same
prefetch, same holding, because the mesh is running on it and a bus nothing
speaks yet is no reason to alter the one every node is on.
The window (window.go) is now what decides, instead of the conditions that were
inlined in the loop. Two things that surfaced in the wiring:
**Supersession is asked before the store, not after.** A report about a
declaration the mesh has moved past would otherwise wait out a restarting store
to be written and then overwrite what the node is doing now.
**Half of a report is not about a declaration, and that half is never stale.**
What the machine *is* — the tunnel it took over, the ports its own bundle
holds, what an adopted node found, a node moving its overlay key — reaches the
mesh on a report and nowhere else. A rekey set aside as stale is a node whose
overlay key never moves, and no retry is coming, because the node said it once.
So staleness is asked only of a report that is purely an apply's account.
The one thing holding-in-memory can do that holding-in-the-server cannot is
named rather than hidden: `About` sets aside a held message when a newer one
about the same thing arrives, and the bus being built ignores it because the
digest answers the same question.
The host's error text may carry a duration or a counter, and a resource looping on
it would never have read as stuck. The previous row is read and compared here.
Stuck needs a start to say. A container may mount the file a binding lands in; the
runtime socket is declared under both of its spellings; the catalogue-wide test
takes MESH_CATALOG.
The mesh kept one report per machine, replaced, so a resource nothing can ever apply
looked like a failure that had just happened, every reconcile interval, for ever.
The row now keeps when the current failure began and how many reports in a row have
said it — the same outcome, refusal and failed resources; anything different starts
again and a clean apply clears it. Three make the machine stuck, and status says so
beside the failure, in words and in JSON (novox/hq 04-ISSUES/065, ADR 0090).
One name per thing, per the HQ glossary: the module/container/image/binary/repo
becomes mesh-controller, the seat the-controller, and the store+broker pair the
foundation (embedded base bundles, default template and example lock renamed with
their go:embed directives). No behaviour change — a pure vocabulary rename.
Claude-Session: https://claude.ai/code/session_01D6qtiYU3P9jk3pnAXyAFyx
0056 is 'the authority is the control plane, not a database'. A citation
pointing at the wrong decision is worse than none: it reads as corroboration.
Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
A public route used to carry its whole hostname as a literal in the module
manifest, so running the same catalogue against a different domain meant
overriding that literal on every routed module, per node. The mesh was, in
effect, holding a map of names to services: the one thing it should never hold,
because the subdomain is the operator's choice and the domain is the node's.
Compose instead. A route contribution carries a `label` (the subdomain); a node
carries its `public_domain` as node-level configuration; the mesh joins
`<label>.<public-domain>` and grants exactly that, interpreting neither half.
Held as a node property beside the node's other node-level facts (endpoint,
site, overlay address), not in a module's settings — the ADR calls it
node-level, and the settings table is keyed per module.
Additive, so an unmigrated catalogue keeps working: a contribution that still
carries a full `name` and no `label` passes through unchanged, and the catalogue
can migrate module by module. A labelled contribution on a node with no public
domain composes nothing, reading downstream as a route that named no host.
And propagate: each granted route name is published into internal resolution
mesh-wide, mapped to the node that serves it, alongside the `<node>.internal`
names every container already gets. So a container — and an internal ACME
validator, which cannot complete a challenge for a name it cannot reach —
resolves a routed name to the proxy that serves it. Name-agnostic throughout:
the mesh propagates whatever names it was told to serve and knows nothing about
what they mean.
novox/hq 02-DECISIONS/0056
Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
The report carries the digest of the declaration it applied (mesh-host
8211d8b), and the mesh stores it beside the outcome. `reported` rows in
the status JSON now say `current`: whether the machine's last word
names the declaration last sent.
Not derivable from the timestamps beside it, which is why they were
not enough: an apply begun under the previous declaration reports
after the next send — newer, and still about the old words. The lab
lost exactly that race between one test's closing push and the next
test's opening one.
Empty digests — every host from before reports carried one — read as
not current, which errs toward waiting rather than toward asserting on
files that are not there yet.
"Not waiting" says the declaration is current, not that the machine
finished applying it: the sent digest is recorded at send. So a test
that pushed, saw waiting clear, and asked the machine what it was
running found containers that did not exist yet — the certificate fix
made compositions stable, and the settling that used to fail first had
been hiding the gap behind it.
The mesh already held the missing half: every machine's last report,
with its time. It just was not in the JSON. `reported` now sets each
machine's last word beside when the current declaration went to it, and
"has it caught up" becomes a comparison of two timestamps the mesh
recorded itself — a report newer than the send means the machine acted
on what was sent; older means it is still working, which waiting alone
cannot distinguish.
jschoubben
mesh-tools
jochen