80 KiB
layer, status, code, updated, decisions
| layer | status | code | updated | decisions | ||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| to-be | in-progress |
|
2026-10-14 |
|
45 — A core that cannot fail silently
The core says when it is wrong, refuses what is stale or unreadable, repairs what it knows how to repair, replaces itself one machine at a time with something other than itself watching, and is checked against the mesh's real facts before a change merges (ADR 0227, from research 031).
The core, here: the controller, the node-engine and its launcher, the bus server, the tool runner and the mesh MCP server, the build seat, and the forge's announcer of merges.
This document is the build's specification. §1–§9 are the parts; §10 is the order they are built in, each phase with what it delivers, in which repository, and when it is done. Every bound marked provisional is set from the durations Phase 0 records and corrected in Phase 1's first live week.
The parts, and how a fact reaches the operator
signals ──────────► watchdogs ─┐ ┌─► status (open conditions first)
(heartbeats, reports, │ │
plan progress, advisories) ├─► CONDITION STORE ───┼─► condition events ─► operator-channel ─► Telegram
doctor probes ─────────────────┤ (controller is │ holder desktop notifier
(live invariants, every 5 min) │ its only writer) └─► healers ─► repair, braked ─► event
events (provider failing, …) ──┘ │
└─► hand-act log (people)
second machine: watcher ── hears doctor's heartbeat? ── no, past bound ──► Telegram directly (not the bus)
Owning repositories: mesh-controller (the condition store, watchdogs, doctor, healers, the lease,
calls, the hand-act log, the facts snapshot), mesh-host (the node-engine: the apply queue, report
order, epoch refusal, the report verb, rollback witnessing), mesh-tools (the tool runner and the
mesh MCP server: their heartbeat, passing every argument, health answers), mesh-catalog (the
operator-channel seat's holder and channels, the watcher, the providers' retirement, the catalogue's
merge gate), mesh-sdk (the TypeScript providers' loop and its retirement), mesh-lab (the
replays and the induced-failure scenarios).
1. The writers table (rule 1)
Every kind of state the core keeps, and the one component that writes it. Anyone else asks.
| State | Writer | Kept in | Others |
|---|---|---|---|
| a machine's declaration | controller (lease holder) | the bus, last per subject | read |
| a hand-over asked of a machine (issue 356) | controller, at its terminal | the machine, beside its state | the engine verifies the mesh's signature and records it, or refuses |
| a fresh setuid search asked of a machine (issue 361) | controller, at its terminal | the machine, which throws its last search away | the engine verifies the mesh's signature and starts it, or refuses |
the take of a shared folder asked of a machine (ADR 0325 §8, issue 567; subject mesh.node.<node>.ask.access-act) |
controller, on the operator's warrant | the machine, which records the act | the engine verifies the mesh's signature and takes it, or refuses |
| a machine's applied state and its report | the node-engine's apply queue (§6) | the machine; the report on the bus | the reconcile and a delivery enqueue, never apply |
| the controller lease | the controller instance holding it | key-value mesh-controller_lease |
a candidate waits |
| plans and their tiers | controller (lease holder), compare-and-set on the plan's revision | the controller's store | read through plans |
| conditions | controller | key-value mesh-controller_conditions, and their history mesh-controller_condition-history; each open condition's looks (how often observed, when last, its newest evidence) in the controller's store (issue 420) |
raise or clear only through observations the controller reads |
| calls and their outcomes | controller | key-value mesh-controller_calls |
read by id |
| the hand-act log | controller, through the verbs that act | key-value mesh-controller_hand-acts |
— |
| the healers' acts and their brake | controller (lease holder), each act begun before it is made | the controller's store | read through healers; each act said as healer-acted and in its condition's tried |
| stream definitions and bus permissions | controller | the bus | — |
| builds and their outcomes | the build seat's holder | its own state | the controller asks |
| a merge announced | one announcer per forge (the hook, or the poll when the hook is absent — never both) | the bus | — |
| a pull request's head announced | the forge's announcer, the one that announces its merges | the bus | the controller asks the build seat to check it |
| a pull request's merge check | the build seat's holder that ran it, said as the controller's checked |
the bus | the forge's holder sets it as the pull request's status |
| a provider's standing | the provider | the provider's events | the controller keeps the newest word as a condition |
| a consumer's retirement: active, retired (when, why), deleted | the provider | its own backend's mark; said on provisioner.retirement |
the controller asks the provider's tools; a person approves, rejects and deletes through the controller's verbs |
| the operator-channel's open messages | the seat's holder | its own key-value state | — |
| a machine's declared data: what it measured, what is retired, what was deleted (ADR 0233) | controller, from what each node-backup holder answers |
the controller's store, data_item and data_reading |
the holder measures and says; it records nothing the controller keeps |
| the facts snapshot | controller | the artifact store, facts/latest |
the build seat reads |
| the record mirrors: nodes, assignments, rollouts, builds, settings (ADR 0289, added 2026-10-10) | controller (lease holder), after each change commits in its store, only when the value differs | key-value mesh-controller_nodes, _assignments, _walks, _builds, _settings; one event per write |
read through reads (ADR 0201); D17 compares them with the store; the controller never reads them to decide (to-be 54) |
| a message a consumer gave up on (ADR 0264) | controller, from the server's maximum-deliveries notice | the bus, the stream DEAD_LETTERS |
read, delivered again or dropped through the controller's dead-letters verb, with why; an ask is delivered again only when the controller made it itself, every other ask is only dropped (ADR 0293) |
A bus subject two components may publish on is refused when the controller composes grants, unless this table marks it shared. Changing a writer is a change to this table, through a decision.
How it is enforced (ADR 0229):
the table is compiled into the controller, each row the bus carries with the subjects a write of it
publishes to — a declaration mesh.node.*.declare, a hand-over asked mesh.node.*.ask.hand-over (signed as
well, because a responder can reach a subject through a reply, which no grant refuses), a report mesh.control.*.report (its own machine's
only), the controller's buckets, stream creation, update and deletion and durable consumer creation, a
build's outcome, a merge announced, a provider's standing — and its writer among the principals. A
grant that overlaps another writer's subject is refused at composition, naming the state and its
writer. A test holds the compiled table to this one, row for row. The controller does not publish
mesh.control.>: it would be a second writer of every machine's report.
2. The condition store (rules 5, 6)
A condition is a durable fact about something the mesh owns that is wrong. It is raised and cleared by observation only.
Where. A key-value bucket, mesh-controller_conditions, written by the controller alone, one entry
per open condition, with no age. Every transition — raised, changed, silenced, cleared — is also
appended to a history kept ninety days in a bucket of its own, mesh-controller_condition-history,
read through conditions history: a bucket has one age for every key, and an open condition must
outlive the history.
The key names the thing and the kind, so the same fault said again is the same condition:
<scope>.<id>.<kind>, where scope is one of machine, plan, call, build, merge, provider,
seat, bus, core, probe, mesh. Examples of the shape: machine.<node>.silent,
plan.<id>.stalled, provider.<module>.<node>.<consumer>.failing, core.controller.<node>.rolled-back,
bus.<consumer>.slow-consumer. The last token is the kind's short word where it has one (failing for
provider-failing), the kind elsewhere; a key is opaque to every reader, and the kind is the field.
Added 2026-10-07, ADR 0240:
the scope module, for a module's own health on a machine — module.<module>.<machine>.unhealthy, raised by
the controller on the second statement from the node-engine that a long-running resource is unhealthy, and
cleared on the first that does not. Designed in to-be 48.
The fields:
| Field | Holds |
|---|---|
| kind | the condition kind, from the signals table, the probe registry or an event kind |
| subject | scope, id, and the machine it concerns when there is one |
| severity | urgent (needs the operator now) or warning (when they can) — two levels, no more |
| summary | one line in the mesh's words, for whoever looks closer: it may name plans, commits and verbs |
| headline, needs, explanation, resolved | what the operator reads, in plain words (ADR 0253): a few words naming the thing and what is wrong; the one thing the operator can do themselves, or nothing; the explanation, opened by its verdict ("Nothing for you to do." or "Needs you:" and that thing); the line said when it clears |
| actions | the answers a notification offers, each a label and the seat verb the operator channel calls when it is chosen (ADR 0253); a desk click performs only an acknowledgement — Details, Silence for a week — until answers are authorised (ADR 0258) |
| evidence | the newest observations, at most ten, each with its time — a look, kept in the controller's store |
| source | the signals-table row, probe or event that raised it |
| raised, last observed | times; and how many observations since raised — the last observed and the count are looks, kept in the controller's store |
| tried | each healer attempt: when, what, outcome |
| resolver | self (clears on observation), healer:<name>, operator, or agent |
| silenced | until when, by whom, why — empty when not silenced |
| epoch | the controller lease epoch that last wrote it |
Added 2026-10-12, issue 420, under ADR 0289 §1 and §4:
a condition's looks are kept beside the bucket, not in it. The count of observations, the last observed
time and the evidence change on every look, so a value that carried them was rewritten every half minute
for a condition that had not changed, and a watcher of the bucket heard each look as a write. The bucket
holds what a condition says — kind, subject, severity, summary, the plain words and actions, source, raised,
count, tried, resolver, silenced, epoch — and is written when it is raised, when one of those changes, and
when it clears. Its looks are a row per open condition in the controller's store, written under the lease on
every look that brings something new and removed when it clears. The conditions verb and the three
condition events carry the whole condition, looks included, as before; where the looks could not be read,
the condition says so in looks-unread rather than a count of none. Checked by the controller's tests:
repeated looks at an unchanged condition write the bucket once (counted by the fake bucket, and on a real
bus by the bucket stream's sequence), the bucket's value holds none of the looks, and the verb run as a
process of its own reads them back from the store.
Added 2026-10-08, ADR 0253: a condition says itself to the operator in plain words, beside its summary, and says whether the operator is needed. The headline, needs, explanation, resolved line and actions are made where the condition is made: by the producer when it knows more than the subject (a rollout's modules, a failed unit's name, a hand-act's cause), otherwise by the wording registered for the kind, otherwise from the scope, which is reported. The explanation opens with its verdict. All of it is held to the plain rule when the controller takes an observation: no hash, numbered id, dotted name, duration in code, clock time, flag, markup or line break; nothing that sends the operator to an agent or a session; and the content rule besides. Words that break the rule are replaced by the scope's words, never refused. A condition a healer gave up on that needed nothing now needs the operator. Checked by the plain rule's test and by a lint over every condition the controller's suite raises, which fails a kind without words.
The life of one:
(absent) ──observation past bound──► OPEN ──healer tries──► OPEN (tried += …)
│ ▲ │ budget spent
│ └─ seen again ◄───────┘──► OPEN, resolver: operator, severity: urgent
│
silence (verb, ≤ 7 days, reason, hand act) ──► OPEN, silenced (no messages)
│
observation says resolved ──► CLEARED: entry removed, transition kept in history
-
Nobody resolves a condition by hand. It clears when the signal returns or the probe passes.
-
A condition cleared and raised again within ten minutes reopens with its count increased; it is not a new message.
-
Added 2026-10-10, ADR 0289 §1: a bucket is written only when what it describes changed, and a value carries no field that changes on every look. This bucket breaks the second part:
last observed, the count of observations and the rolling evidence rewrite an open condition on every observation (131 647 writes in 3 days 21 hours for a dozen open conditions, measured 2026-10-10). Moving them out of the bucket is tracked by issue 420 (the conditions bucket is rewritten on every observation). -
The verbs:
conditions(open ones, filtered by scope, severity or machine),conditions show <key>,conditions silence <key> --for <duration> --why <text>,conditions history. -
The events, emitted by the controller on its own subjects:
condition-raised,condition-changed(severity or resolver changed; not every observation),condition-cleared. The operator-channel's holder and any other surface consume them; the controller learns nothing about telling. Each carries the condition at the top level, withevent,at,change(raised, reopened, severity, resolver, silenced, silence-ended, cleared),why,was,clearedandshowbeside it; a silence and its ending arecondition-changed, a reopeningcondition-raisedwith changereopened. -
statuslists open conditions first, urgent before warning, oldest first, silenced ones marked with their expiry. The all-well sentence requires none open, silenced included.Changed 2026-10-08 (ADR 0261):
statusalso lists every open pending assignment (one kept while its module's build runs) and every one ended in the last day, with what ended it. The all-well sentence requires none open. An assignment that was not made (expired or refused) is the conditionassignment-not-made, a warning the operator resolves, open until the module is assigned there, assigned again, or taken back withunassign. The controller settles pending assignments on its own one-minute tick and on each build's outcome; readingstatussettles nothing. -
ADR 0224's provider standing is the first kind:
provider-failing, raised by the provider's event, cleared by its recovery, its silence after thirty minutes the row S8 below. -
Retirement (ADR 0230) raises three, keyed by the provider module and its machine:
retire-waiting(urgent) while a stable set too large to retire alone waits forretire approveorretire reject, listing it;retire-rejected(warning) while a rejected set is kept active though the mesh no longer asks for it; andcleanup-waiting(warning), from the probe D11, while anything retired is older than thirty days. The first two clear on the provider'sretired,settledorapprovedword, and all three when the provider is no longer assigned there. -
Data (ADR 0233) raises, from D13, each urgent for irreplaceable data and a warning for valuable data — the operator's ranking — and none for rebuildable data or caches:
data-shrank(less than half its largest size in seven days, and 16 MiB less),data-missing,empty-replacement(a copy in use less than half a copy of the same thing kept elsewhere — issue 273's shape — for an item or for a consumer's data at a provider),data-quiet(an item that saysactive, unwritten past it),backup-stale,array-degraded(one per array under watched data) andprotection-missing(an item said to be on redundancy, found on plain storage); and as warningsdata-held-twice,data-unmeasuredandcleanup-waitingfor an item retired over thirty days. Keyed by machine, module and item, or by the array. A shrink a person meant is silenced with why; everything else clears on observation.
3. The signals table (rule 5)
Compiled into the controller. A watchdog per row; the test generated from the table suppresses each
signal and asserts its condition. doctor signals shows, for every row, the age of its newest signal.
| Row | Signal | Emitter | Trigger | Bound | Condition kind | Severity | Healer |
|---|---|---|---|---|---|---|---|
| S1 | machine heartbeat | node-engine | its interval | 3 × interval, provisional | silent — not raised while the machine has declared itself asleep or shut down (ADR 0211) |
warning; urgent after 30 min for the control node | — |
| S2 | report after a send | node-engine | each declaration sent | max(2 min, 3 × that machine's last apply duration), provisional | sent-not-reported |
warning | H1 |
| S3 | plan tier progress | controller's plan | each tier entered | from the tier's build and apply durations, provisional | stalled (names the tier and what it waits on) |
warning | H2 |
| S4 | the controller's event loop takes a message | controller | while its consumer has pending messages | 2 min | controller-deaf |
urgent | — |
| S5 | a merge announced becomes a plan or nothing reads it | announcer → controller | each merge | 10 min (exists, issue 266) | merge-not-acted |
urgent | — |
| S6 | a build asked → its outcome | build seat | each ask | max(20 min, 3 × the p90 of measured builds), 1 h while nothing is measured, provisional (the build seat declares no timeout) | ask-lost |
warning | — |
| S7 | a call running → finished | controller | each call | the verb's bound: push, rotate and command 30 min; assign and unassign 15 min; doctor 3 min; others 10 min |
call-hung |
warning | — |
| S8 | a provider's failing word repeated | provider | every 15 min while failing (ADR 0224) | 30 min | provider-silent |
warning | — |
| S9 | bus advisories: slow consumer, maximum deliveries, permission violation, consumer deleted; every user's connection the server closed as a slow consumer or for breaking the bus's rules (ADR 0357) | bus server's advisory subjects; the controller's connection; the bus module's watch of its server (bus.slow-consumer, bus.connection-closed) |
any | any occurrence; a message given up on for as long as DEAD_LETTERS holds it (ADR 0264) |
slow-consumer, connection-closed (one per bus user), max-deliveries, refused, consumer-lost — each naming the call, consumer or module in the mesh's words |
warning | H3 for consumer-lost |
| S10 | the self-check's heartbeat | controller's doctor |
every run | 2 × its interval, watched from the second machine (§5) | self-check-silent |
urgent | — |
| S11 | tool runner heartbeat | tool runner | its interval | 3 × interval, provisional | tools-silent |
warning | — |
| S12 | the controller lease renewed | controller | every 5 s | 15 s; a holder that lost the lease or was found expired, and a lease bucket found raised again from nothing, said for an hour; a controller serving without the lease, while it does | lease-lost |
urgent | — |
| S13 | stale refusals | every receiver (rule 2) | each refusal | more than 5 from one writer in 5 min | stale-writer (names the writer: the controller epoch a refused declaration claimed, with its instance and how its lease ended; or the machine whose older accounts the controller refused) |
warning | — |
| S14 | facts snapshot exported | controller | when it moved, and daily | 2 days, or none kept by a controller up that long (Phase 5) | facts-stale |
warning | — |
| S15 | a hand act with a cause already recorded | hand-act log | each act | the second within 14 days; clears when fewer than two remain within 14 days | healer-wanted (names the cause, and the healer that was not enough where one answers it) |
warning | — |
| S16 | a rollout waiting for its delivery's word is let go | mesh-delivery, through deliver (ADR 0239) |
each merge whose rollout waits | 30 min; urgent after 4 h | waiting (names the rollout and plans go) |
warning | — |
| S17 | a send held for the bus's planned step is told to a person (ADR 0271) | controller's plan | each send refused because it would replace the bus outside its step | none: the first refusal; cleared once the bus's machine has been sent the build the mesh holds | bus-step-waiting, for the operator (names the machines, what waits, the bus build from and to, since when and the mesh-controller.bus call); S3 leaves such a rollout out |
warning | — |
| S18 | a batch of merges is cut into its rollout (ADR 0276) | controller's merge window | each batch | a minute past merge-window-at-most while no rollout is open: the controller failed to cut it |
batch-not-cut (names the batch and what it holds) |
warning | — |
| S19 | a batch waiting behind an open rollout is cut when it ends (ADR 0276) | controller's merge window | each batch closed while a rollout is open | that rollout's bound, a tier's bound for each of its tiers, from when the batch closed | batch-behind-walk (names the rollout) |
warning | — |
The bus advisories (S9) cost one read-only subscription: the server already publishes them. The controller translates each into a condition naming the thing in the mesh's words, as the refused reply of issue 265 is translated today. An advisory clears after an hour without another; a deleted consumer is said only for one the mesh names and expects, and clears when it exists again.
Every user's slow consumer, from the bus module (ADR 0357). The server closes every client it judges
a slow consumer and keeps its user and reason in its record of closed connections; it publishes neither
in the mesh's account. So the nats module's tool bundle reads that record on its own machine every 30
seconds and publishes bus.slow-consumer for each connection closed as a slow consumer, and
bus.connection-closed for each the server closed for something its client did against the bus's
rules or limits — naming the bus user, the connection and the reason, never a credential. A closure by
the client, by the network or by the server's own shutdown is not said: D15 says a user dropped in a
loop. A closure — its user, connection id and opening time — is said at least once: again when publishing
it failed, and again for the last hour when the bundle starts. The controller's watchdogs read the last
hour of both events back from the events stream on every tick, count each closure once, and raise one
condition per bus user and kind, naming the module and its machine; it clears an hour after the newest
closure. A closure is a condition at most about a minute after the server closed it.
A refused subject is still heard for the controller's own connection only. The server tells the client and writes its log, and says it nowhere else — not in its monitoring endpoint and not to a system account (issue 702, open).
4. The self-check: doctor (rule 6)
A probe registry in the controller: each probe is a live invariant of a design, with an id, an
interval, a timeout and the condition kind it raises. The controller runs the registry every five
minutes; each probe has thirty seconds. A probe that errors or times out is said in that run's
verdict at once — an unanswered probe is never a pass — and raises probe-failed for itself when the
next run cannot run it either.
A finding one look can be wrong about is raised on the second look in a row (issue 277): a question over the network that went unanswered, a time measured once, a reading a burst can move. Within a run such a question is asked again before it counts; across runs the finding is raised when the previous run saw it too, kept while it is open however it is seen, and cleared by the run that no longer sees it. Held back, it is listed in the verdict as unconfirmed. A definite answer — an address that is wrong, a stream that is not there — is raised at once. The watchdogs hold a row that cannot read its facts to the same rule over two ticks. Checked by the controller's tests of the self-check over runs and of a blind row over ticks.
A condition's summary names machines and says the rest in words; an address, a domain, a path or a raw error is its evidence, which stays inside the mesh. The condition store holds every summary to the operator channel's content rule (ADR 0234 §6) and says one that would be withheld in words, keeping it whole in the evidence; a lint over every condition raised in the controller's test suite fails the producer that wrote it.
| Probe | Asserts | From |
|---|---|---|
| D1 | every machine's declaration composes, and passes the node-engine's validation (the validator is a package of mesh-host the controller and the merge gate import — one validator) |
236, 263 |
| D2 | every holder of the mesh's resolver answers a machine name for IPv4, and NODATA for IPv6 | 262 |
| D3 | every seat on record has a live holder that answers (holder-silent, H3) |
208, 218 |
| D4 | every kept archive is held by a manifest | 253 |
| D5 | exactly one lease holder — the key names this controller at the epoch it acts under, and the controller's record holds no other epoch open; no message from a stale epoch refused in the last interval | 204 |
| D6 | every durable consumer exists with its definition and is near its stream's head (within 1000 messages, provisional); a missing one is consumer-lost (H3), one far behind consumer-behind (H4) |
248, 266 |
| D7 | every stream the controller defines exists with its definition | 208 |
| D8 | no address the mesh owns is in a ban list | 238 |
| D9 | status answers in full within ten seconds |
265 |
| D10 | every machine runs the node-engine and tool runner builds its plan says, or is inside a plan's window | version split |
| D11 | no provider holds a consumer retired more than thirty days: each provider assigned is asked provisioner_retirement on its machine (ADR 0230); one that does not serve it yet is named, not failed |
ADR 0230 |
| D12 | every consumer of a provision that keeps its data is bound where it was last sent, or moves by a pin (binding-kept, binding-moving, binding-moved) |
ADR 0232 |
| D13 | every item of data a machine declares is measured, is there, holds what it held, is written where it says it is, is backed up within its bound or sits on healthy redundant storage, and is no empty replacement of a copy kept elsewhere; an irreplaceable or valuable item a machine no longer declares is retired, not forgotten. Each machine's node-backup holder is asked backed-up; every provider of kept consumer data provisioner_retirement (with each held consumer's size) — see §2 for the kinds |
ADR 0233, issue 273 |
| D15 | no user of the bus had its connection dropped more than twelve times in the last hour (provisional): the bus module's nats_closed_connections, from the server's record of closed connections; bus-reconnects (ADR 0265) |
327 |
| D16 | every delivery runs healthy on every node within its class's budget (five minutes for a leaf module, ten for a core one); delivery.<class>.over-budget, naming the delivery and its longest phase. Listed 2026-10-10: the controller already runs it under this id |
ADR 0282 |
| D17 | every record mirror holds exactly what its store says: no key missing, none differing, none without a record, a difference counted only when a second read 30 seconds later still finds it and the record did not change between them; mirror-drifted (H6), and mirror-event-lost for an event lost after its write. Added 2026-10-10 (to-be 54 §4) |
ADR 0289 |
| DW | the watchdogs of §3 ran within three of their intervals: the watchers are watched, and raise S10 when the self-check stops | rule 6 |
| H-* | the health probes of §8, run for every core component on every machine: H-controller, H-engine, H-tools, H-bus (as built, ADR 0236) | rule 8 |
| DG | every build that failed its gate, and every core build a witness put back, is said — build.<module>.<machine>.rolled-back, core.<component>.<machine>.<outcome>, rollback-failed — until a newer build passes or the reports stop saying it (ADR 0236) |
rule 8 |
| DB | a bus upgrade a person started is said as bus-maintenance while it runs, and ends healthy within its bound or is said bus-upgrade-failed, with its snapshot (ADR 0236) |
rule 8 |
doctoranswers the last run's verdict at once: per probe, pass, fail or failed-to-run, and age.doctor runruns now under a call id.doctor probeslists the registry;doctor signalsthe table's ages.- Every run ends with a heartbeat event carrying the run's id and counts. That is S10.
- The registry is the design's live form. A check over the to-be designs counts invariants that name a probe against those that do not; the number without may only go down.
5. The output channel, minimal form (rule 6)
From research 028, the smallest form that works. Amended 2026-10-06 by ADR 0234, 028's graduation: this form is the first step of to-be 46, the conversation with the operator, which replaces the points marked below. Phase 1 builds this section; to-be 46 §13 builds what follows from it.
- One mesh seat,
operator-channel, held once. It acceptsnotifyas a work queue, so a message waits for a holder; it keeps its open messages in its own key-value state, so a restart forgets nothing; it servesopenandhistory. - The holder consumes the controller's condition events and decides what is sent. The controller calls nobody.
- Two channels: Telegram (a bot to the operator's chat) and the desktop notifier of the
machine the operator is at. Amended: a channel is not a contribution to this seat. Each is the
holder of a kind on the kinded benches
channelandintake, a module of its own holding its own secrets, and this seat's holder becomes the router (to-be 46 §2, §3, §8). - A message is: the condition's key, its subject (a machine's role, a module, a plan), kind,
severity, the one-line summary, since when, and the verb that shows more. Deduplicated by the key.
Amended 2026-10-08, ADR 0253:
a message reads in one glance. Its title is the condition's headline, with "Urgent:", "Still open:" or
"Now urgent:" before it where that applies. Its body is the explanation, opened by its verdict, and since
when, in the operator's time zone (the messenger's setting
time-zone). It offers the condition's actions and Details; the desk takes the answer (node-notifier.sendgains actions, and the dunst holder says the chosen one as its event), and the holder calls the action's verb with a why saying it was a desk click. Amended 2026-10-08, ADR 0258: only an acknowledgement (Details, Silence for a week) is offered or performed; each answer is a one-time token bound to its condition and machine, good for a day; a change of words to "Needs you" is acondition-changed, and a quiet warning is said once after a day. It carries no key, id, commit, verb or markup: those stay inconditions, the holder's history and Details. A warning that needs nothing is not sent: it is kept, and said when it becomes urgent or needs the operator. A clearance is one line, the resolved line and how long it was open. A digest lists titles, and a message that offers answers is never in one. Still deduplicated by the key. Amended 2026-10-10, ADR 0285: a condition's message says which issue tracks it. Below the explanation, the router says the tracking line, from the issue tracker'stracking(to-be 53 § 12), asked outside its work at most once a minute and used ten minutes: "Tracked as issue N:" and where the issue stands in plain words (nobody is working on it; who is, on which machine, since when; its fix merged and on its way, held, failed or delivered; it waits for your answer or for a decision; resolved, yet this still happens), at most three issues and then a count; "No issue tracks this yet."; or "Whether an issue tracks this could not be checked." when the tracker gave no answer in ten minutes. "Nothing for you to do." becomes "Needs you: answer issue N." while an issue tracking it waits for the operator, and is left out while nobody works on any issue tracking it. The issue number is the one numbered id a message may carry. A clearing and a silence say no tracking line; what is sent, and when, does not change. - When: on
condition-raised; once more if still open after 1 hour (urgent) or 12 hours (warning); oncondition-cleared, by editing the first message where the channel can. A silenced condition sends nothing. Urgent goes to both channels; warning to the desktop notifier when the operator's session is there, otherwise to Telegram. Amended 2026-10-12, ADR 0318: history is never news. On start, every ten minutes, and at once when an event older than its last reading arrives, the holder reads the open conditions from the controller and takes them as its state; an event is acted on only when it is newer than that reading (five seconds of clock slack) and its ownatis no more than ten minutes old — otherwise it is recorded as state and never said as its own message. While the controller cannot be read, theattest alone decides, and the reading is asked for again until it is had. Every close, drop, fold into a digest and hold under the cap is recorded in the holder's own history. A condition open in the holder's state but absent from the controller's open set ended while the holder was away, and is closed quietly: edited where a channel can, never sent to wake a desktop. A clearing of something never said says nothing, and a message still waiting is dropped the moment its clearing is read. Every urgent condition a reading finds open and unsaid is named once, together, in one digest; when the grace minute ends, or at once when the reading is not a start's, each still open and unacknowledged is then sent as its own message with its answers, under the cap. A grace minute follows every start or reassignment: nothing is sent in it except that one digest, not even a fresh urgent event. - Rate: at most twenty messages an hour; the excess is folded into one message naming them all. Amended 2026-10-12, ADR 0318: a channel holds what it has to say for thirty seconds from the first item and sends it together — alone as itself, several as one digest of headlines, one line each, with no answers; a message that offers answers is never folded into a digest but sent on its own, and an urgent item breaks a thirty-second quiet at once, except inside the grace minute. The desktop keeps a gentler rhythm than Telegram: a warning reaches it at most once every fifteen minutes; an urgent message is never held to that rhythm, and still goes to both channels. Reaching the cap is said once, in its own message; what it holds back, urgent included, waits until the hour allows, when each message that offers answers is sent on its own with them and only the rest goes as one digest.
- What may leave the mesh: roles and words. A message carrying an address, a path or anything
shaped like a secret is refused by the holder and raises
channel-refusedinstead. Amended: the mesh's own machine names may appear too; a concrete detail travels as a reference only a private surface opens, and a refusal names the offending part to the sender (to-be 46 §9). - Answering back. Amended: the minimal form had none, and acknowledging was
conditions silence, through the mesh. The operator now answers and is asked over the intake seat; an answer that performs an action is checked and performed by the controller (to-be 46 §5, §6, §10).conditions silencestays callable. - The watcher's watcher. A module,
mesh-watcher, assigned to one machine that is not the control node, holds the Telegram channel's secret too. It listens for the self-check heartbeat (S10) and for the bus itself; when either has been silent past its bound it sends to Telegram directly over HTTPS, not through the bus, and says so again when they return. It is the only sender that does not pass through the control node.
6. Order: lease, epoch, report sequence, one apply queue (rules 1, 2)
The lease. A controller instance acts — sends a declaration, writes a plan, a condition or a call
— only while it holds the key holder in mesh-controller_lease: written with compare-and-set, a
fifteen-second time to live, renewed every five seconds. The epoch is the bucket revision at which
it was taken. A starting controller waits for the key to be absent or expired. One that fails a renewal
stops acting at once and exits, so its service manager restarts it as a waiting candidate.
As built (ADR 0229):
- The gate is the clock. A holder stops acting three seconds before its key could expire unrenewed, whatever its renewing goroutine is doing; every act passes that gate at the moment it is made — a declaration at its send as well as where it was composed.
- The epoch never goes backwards. The controller's store keeps every epoch issued, its instance,
and how it ended (
released,lost,expired); a lease bucket at or under the highest is moved past it before the key is taken, and S12 says so. - A holder that stops gives the key back, so the next takes it at once.
- Without the bucket's grant, nobody holding it, a controller serves unleased — no epoch on what it sends, S12 urgent — and takes the lease once the user list that grants it reaches the bus.
- A command at a shell acts under the holder's epoch, read as it acts, or under a lease of its own while nobody holds one.
controller A (epoch 41) ──renew──renew──╳ (renewal refused)──► stops sending, exits
controller B ──wait──────────────take (epoch 57)──► acts; marks A's running calls abandoned
node-engine accepts 41 … then 57; refuses anything from 41 after 57, counted, reported
What carries the order:
| Message | Carries | The receiver keeps, per writer | Refuses |
|---|---|---|---|
| declaration | epoch, sequence | highest epoch, then highest sequence | older epoch; same epoch, lower sequence — only where both claim an epoch |
| report | machine, the declaration's epoch and sequence it is about, the node-engine's own report sequence (kept on disk, increasing across restarts and self-updates) | highest epoch where both claim one, then declaration sequence, then report sequence | an account of an older declaration; an older report |
| plan write | the plan's revision, the epoch | — (compare-and-set) | a write against a revision already moved |
| build outcome | the build's ask id and order | newest per module | an older build finishing later (exists, 219) |
| call | call id, epoch | — | a finish from an epoch that is not the holder's, recorded as abandoned |
| merge announcement | forge, repository, commit, the announcer's sequence | newest per repository | a duplicate (one announcer) |
Every refusal is one line in the receiver's log in the mesh's words, a counter, and — from the node-engine — a report naming the refused declaration, so the controller sees it. The counter feeds S13.
The wire — the contract, written once on each side: a declaration's epoch and sequence are
top-level keys of its signed envelope; a report carries back the epoch and sequence of the
declaration it is about, its own report_sequence, older_than (the order held) on a refusal of an
older declaration, and refused_older (how many ever) on every report. Every key is optional, zero is
"no order claimed". A machine is sent epoch only while its latest account carried a
report_sequence, because an older node-engine refuses an unknown key, whole; a declaration without an
epoch is taken by its sequence, so a controller rolled back to a build without the lease is never
stranded. A report without a report_sequence is judged by the digest it names (issue 267). What the
mesh would send a machine is composed with the epoch it was last sent: a new holder is not a change of
the machine.
One apply queue on every machine. The node-engine has one worker that applies; a delivery, the
five-minute reconcile, a self-update hand-over and a manual apply are four reasons to enqueue.
The queue holds at most one pending request, coalesced. When the worker starts it takes the newest
declaration held at that moment, applies it once, and sends one report naming the sequence it applied.
Nothing else in the node-engine applies.
The report verb. The node-engine answers, on request, the report of the last declaration it
applied, from what it keeps on disk. It is what healer H1 asks. As built (ADR 0231):
asked on mesh.node.<machine>.ask.report, its own only, the node-engine enqueues a reconcile whose
account is said whether or not it is news — a delivery waiting is applied and reported instead — and the
answer is its ordinary report; nobody's inbox is answered.
Durable calls. Every call's record — verb, arguments with secrets removed, caller, started, epoch,
state (running, finished, failed, abandoned), the answer bounded in size, finished — lives in
mesh-controller_calls, the last thousand or fourteen days. A new lease holder marks the previous
holder's running calls abandoned, which is said.
7. Healers and the hand-act log (rule 7)
A healer is a registered response to one condition kind: its repair (the ordinary path again), its
budget, its back-off, its brake, and the healer-acted event naming the condition, the act and the
outcome. A healer may not withdraw, delete or recreate data; such a repair is a condition for the
operator.
| Healer | Condition | Repair | Budget, then |
|---|---|---|---|
| H1 | sent-not-reported |
ask the machine's node-engine for report; if it does not then report the declaration it was sent, send it again — never moving a build a policy or a plan holds back |
twice per condition in 6 h, then resolver operator, urgent |
| H2 | stalled on a wait that is superseded (a newer plan of its repository and branch) or already finished (every module built or failed, every one that rolls out sent) |
close the plan with its note, as plans close does: superseded or done |
once |
| H3 | a seat holder that does not answer (D3, holder-silent), or a consumer the mesh expects missing (D6, S9, consumer-lost) |
assert the bus's objects again — the assertion every send makes (208) | once per object per hour |
| H4 | consumer-behind (D6) on a consumer the stream table marks resettable — the controller's own events consumer alone |
broker consumer-reset (248) |
once a day per consumer, then condition |
| H5 | provider-failing with the administrator refusing the mesh's secret |
ADR 0224 §5 (exists, in the module) | as ADR 0224 |
| H6 | mirror-drifted (D17) |
write the store's value for each key the condition names, purge a key with no record, each write said as its mirror's event marked repaired; never writes the store (ADR 0289, added 2026-10-10) |
twice per mirror in 24 h, then resolver operator, urgent |
As built (ADR 0231): the registry is compiled into the controller with a test generated
from it; each act is begun in the controller's store before it is made, under the lease, and the
budgets are counted from there; a healer that does not apply to a condition leaves it alone; only
observation says a repair worked — the act is kept in the condition's tried as healer Hn and said
as healer-acted, and the condition clears when what raised it no longer sees it; a spent budget hands it
to the operator, urgent, and no healer touches it again. The mesh-wide brake: twelve acts in an hour,
all healers together, stop every healer — mesh.healers.braked, urgent — until an hour after the last.
A heal is never a hand act. healers lists the registry, the acts and the brake.
The hand-act log. Every verb that repairs by hand — a named push outside a plan, plans close,
broker consumer-reset, conditions silence, retire approve, retire reject and cleanup delete
(ADR 0230; and an approval, rejection or deletion a provider reports that did not come through the
controller), and hand-act record for an act done outside the mesh —
takes a required --why and writes an entry: who, which verb and arguments, why, when, the condition
key it addresses if any, and a cause (the condition kind, or a word the person gives). status
shows the week's count. A cause recorded twice within fourteen days raises healer-wanted (S15) —
except an act recorded by a verb that is a person's decision by design, which is no repair and never
counts, whatever cause it gives. The controller keeps one table of the verbs that write the log, and
each says whether it records a repair or a decision, and why: approving or rejecting a retirement and
deleting what was retired (ADR 0230); bus upgrade, because the bus is never rolled by the mesh, and
upgrade release-backlog, because after a failed rollout the next opens only on a person's word
(ADR 0236); and secret rotate when its cause is a leak (leaked-in-logs) — a person judges what was
disclosed, one leak rotates several values, and a leak that recurs is a defect of the module that
prints them, an issue against it, not a healer that rotates. A rotation for any other cause counts. A
push by hand, plans stop and close, broker consumer-reset, conditions silence and hand-act record (an act outside the mesh, whose kind the mesh cannot tell) count; so does a verb the table does
not list. The table's test finds every verb the controller records and fails on one it does not list.
A healer-wanted already open for a decision clears on the next watchdog tick, as any condition its
row no longer sees.
A drill is no repair. Something broken on purpose to see the mesh raise and clear it is recorded by
hand-act drill (the seat's drill), cause drill, which the table lists as a person's decision;
hand-act record refuses that cause, so a repair cannot pass for a drill by the word it gives (issue
292; checked by the controller's S15 tests).
A push of recorded builds is no repair. A build of a module whose upgrade policy records moves only by a person's push (ADR 0242), and that push is the word the policy asks for. So a push says what it was, read by the controller from what it carries and never from a word the person gives. It is a push of recorded builds when it names one machine, that machine's last send is known and its last report is no failure, and every build it moves records rather than rolls out: a held new build, or a recorded module's first build on a machine newly assigned it. The act is written with its kind and what it carried; the table lists a push of that kind as a person's decision, and S15 never counts it. Any other push by hand still counts (issue 301; checked by the controller's replay R301 and its tests of the rule).
Progressive insight — 2026-10-06. This paragraph said before: "except
retire-waitingandcleanup-waiting: approving a retirement and deleting what was retired are a person's decision by design (ADR 0230), and no healer may take them over." The exception was keyed on two causes, and the first planned bus upgrades (two within a day, one recorded after the fact and one throughbus upgrade) raisedhealer-wantedforbus-upgrade, as did two rotations after one leak forleaked-in-logs. What makes an act a decision is the verb that recorded it, not the word it gives as its cause, so the exception is read from the verbs. Nothing decided changes: ADR 0230's acts stay exempt, and pushes by hand — what roll-out by default (ADR 0236) exists to end — still count.
A condition no healer can fix becomes an issue (2026-10-12, ADR 0314, extending ADR 0285; the build is tracked by issue 409). A condition no healer can fix is one whose kind no healer answers, one a healer passes over because it does not apply, or one handed to the operator after a healer's budget was spent. When such a condition is raised, passed over or handed on, and no issue links its key (ADR 0285), the issue tracker opens one itself, from the controller's condition events, with the condition's summary, its evidence and its key, and links the key at once; the controller sets no link. Every message about the condition then names the issue through its tracking line. Agent sessions take such an issue as any other: a claiming move that names the worker, a branch, review and the required checks; no standing agent acts on a condition by itself. What an agent does about it is written in the issue's notes, and every act it makes by hand on the mesh is also written to the hand-act log, which S15 reads. ADR 0275 §9's week stands: the tracker opens no issue for a condition before 2026-10-17 20:17 UTC, and until then such a condition is linked by whoever opens an issue for it. How it is checked: the tracker's tests feeding condition events (one issue opened and linked for each of the three cases, none for a condition a healer acts on or one already linked, none before the week ends); live, the messenger's preview of such a condition says "Tracked as issue N", and the issue's notes and the hand-act log hold what the agent did.
8. Staged core upgrades and rollback (rule 8)
Health, per core component, as probes in the registry (§4):
| Component | Healthy when | Witness that rolls it back |
|---|---|---|
| controller | holds the lease within 60 s of starting; status answers in full within 10 s; doctor ran once |
the node-engine on the control node, which keeps the previous controller build installed beside the new one and reads the lease bucket |
| node-engine | has reported its current declaration under its own build | its launcher (ADR 0141), which keeps the known-good |
| tool runner | announced, and answer a ping within 5 s | the node-engine, which keeps the known-good |
| bus | every stream and durable consumer present (D6, D7); a request/reply round trip from every machine | none — a planned step, below |
The gate. ADR 0218's first machine for a core component is judged by that component's health probes passing three consecutive times over two minutes, within ten minutes of the apply. Only then do the other machines follow. "Reported applied" is not enough.
merge ─► plan ─► first machine applies new build ─► health probes ×3 within 10 min?
├─ yes ─► the rest follow, one tier at a time
└─ no ──► witness restores previous build
─► condition core.<component>.<node>.rolled-back (urgent)
─► plan halts that component, records why
- Every core upgrade leaves a record in its plan: component, first machine, from and to build,
verdict, time to verdict, rolled back or not. Read through
plans. - A rolled-back build is not retried by the same plan. A newer merge makes a new plan.
As built (ADR 0236):
- Every rollout is gated, not only the core's. A module is judged by its own health: it reported applied, no witness put it back, no condition was raised since its send about the machine or about the module there, and its tools are served on that machine where it has tools. Three healthy judgings at least forty seconds apart and two minutes after the send, within ten minutes of it. A policy of together is not gated.
- The rollback is the ordinary path once per build. The build is marked failed at its gate (never
registered again,
plans retryrefuses it); the module's registered build goes back to the build the first machine ran before, still kept by ADR 0189; that machine is sent it. The verdict is written before the send. Said asbuild.<module>.<machine>.rolled-back(warning) orcore.<component>.<machine>. rolled-back(urgent),rollback-failed(urgent, the operator's) when nothing could be put back, and as the eventrolled-back. - The witness, as built on the node-engine (
mesh-hostinternal/witness/contract.go, the controller's halfinternal/lease/witness.go): the controller's is the lease alone — the holder on this machine, taken since the start, renewed within fifteen seconds, within sixty seconds of the start — and the tool runner' is their answer to the services protocol's ping within five seconds, within sixty. The controller's "status in bound, doctor ran once" is the controller's own word in the lease's value, which the gate reads and the node-engine does not: a controller that holds the lease and never becomes ready is put back by the gate sending the previous build. A witness says its verdict inrollbackson every report while it stands; the controller raisescore.<component>.<machine>.<outcome>from it — urgent for rolled-back, not-reversible, restore-failed and halted — and clears it with the first report without it. - No build reaches a machine without a gate: a gated send carries and judges everything waiting on
its machine; every other send — a plan's rest, a cascade, a healer's, a whole-mesh push — is refused
or leaves the machine while a build no gate has seen waits there; a rebuild with the same artifacts and
manifest is no move. What waits is moved by a rollout, one machine at a time, the control
node last, each judged; one that fails holds the next until
upgrade release-backlog --why. - A wait for a person passes, and each module of a send keeps its own pass
(ADR 0254): a module whose only fault is a new login the operator owes
is passed with the wait carried and said as its condition
relogin-needed; when a send fails, a module healthy on its own for the passes asked keeps a pass and is not put back. Designed in to-be 48 §5. - A new controller that passes its gate sends the bus's machine the user list it composes, when that changed and nothing held back would go with it.
- The default policy is to roll (ADR 0236 §4);
recordstays where a module says why, keeps irreplaceable data, or is the bus.
The bus is a planned step. A bus upgrade is a maintenance step a person starts through the
controller: the streams are snapshotted; a bus-maintenance condition is open for the step's duration;
the bus is replaced; afterwards D6, D7 and the round trip must pass, or the step is reported failed and
the snapshot is the way back. A step whose new version cannot be reverted (the bus's 2.10 → 2.11 is one)
says so before it starts and runs only on a person's explicit word, recorded as a hand act. Whether the
bus becomes a cluster that can be upgraded live is left to its own effort.
As built (ADR 0236): the verb is bus upgrade --why … --reversible|--irreversible; the snapshot is the
bus machine's backup holder backing the bus module up now (ADR 0235),
or one a person took and names with --snapshot-taken; the step's bound is fifteen minutes; no send
reaches the bus's machine while a new bus build waits for it — a push, a plan's send for another
module, a healer's — after a plan's send restarted the bus unasked on 2026-10-06; a change of the bus's
user list is reloaded in place, not a restart. A send held for the step is told at once (ADR 0271):
S17 raises bus.<module>.step-waiting for the operator on the first refusal, and a change that builds the bus
says in its delivery plan, and so in mesh/merge-gate's description, that merging it needs a person's bus
upgrade. 2026-10-06: the snapshot
and the way back from it exist — the bus image's own snapshot program, the same one the night's backup
runs, and a restore that builds a new store beside the live one for a person to swap in
(ADR 0235,
to-be 43).
9. Before merge: facts and replays (rule 9)
The facts snapshot. The controller exports daily, and after any change of machines, assignments or
seats: every machine (a stable pseudonym of the same length as its name, its role, operating system,
C library, architecture, node-engine and tool runner builds), assignments, settings keys and their
non-secret values, seats and their holders, the catalogue commit, and the versions the mesh runs of
the bus server, the store and the node-engine. No secret and no address: an address is replaced by one
from a documentation range. It is kept in the artifact store as facts/latest, where the build seat
reads it.
The merge gate. In mesh-controller, mesh-host and mesh-catalog, a check composes every machine of
the snapshot with the change applied and runs the node-engine's validator over each. A change that
makes any machine fail to compose or validate fails its check, naming the machine's role and the
module. The resolver module's tests run under both C libraries the snapshot lists. A test in each core
repository asserts that a pinned dependency's version equals the version the snapshot says runs.
Revision, ADR 0238
(2026-10-06). The commit is the build at hand. A commit off its module's trunk is only checked; a
commit on it is published and pushed, and the controller refuses to register a build of one off it. The
module graph, not the repository, decides what a pull request's check runs: every pull request the
forge holds is announced, and the controller asks the planner — the one function that maps a changed file
onto modules, and the one answer of what a merge moves and builds — what a merge of the head would reach.
A changed file touches exactly the modules whose build reads it; a file no build reads touches nothing. A
change reaching a module, or adding one, runs the gate on the build seat (mesh/merge-gate): the
touched manifests through module check, failing only what the change brings; every machine composed with
the definitions of the modules the plan moves or adds; the replays. Every repository of the mesh runs its
own merge-check.sh (mesh/repo-check) in the toolchain it declares. A repository in two languages
declares each further part of its check and the toolchain that part runs in, and the build seat runs each
part in that toolchain's own container: the layer passes only when every part does (issue 302; checked by
the builder's TestARepositoryInTwoLanguagesIsCheckedInBoth). The check that judges a change is the base
branch's: where the branch a pull request merges into holds a merge-check.sh, its script, its parts and
its toolchain run against the change's tree, and a head that lacks or alters any of them is said so in the
summary. A change to the check is judged by the one it replaces (issue 310; checked by the builder's
TestAHeadWithoutTheScriptIsJudgedByMains and TestAHeadThatGutsTheScriptIsStillJudgedByMains, replayed as
R310). A repository with none has "no repository
check defined", a success where the branch does not require mesh/repo-check and a failure for a person
where it does. A note is never a warning on a required status: the forge combines warning as a
failure, so a note — a wide rebuild, a problem already so on the base — is a success that says it, and only
what a person must decide fails (issue 293; checked by the forge module's status tests). The
check's result is the commit's delivery plan — the build plan, the deploy plan machine by machine with
what is not an ordinary send, and the verdict — posted on the pull request. The plan is one object with
a state machine, kept, followed by the release and written to the commit as a note; which parts are built
is Phase 5's.
Revision, ADR 0323 (2026-10-12). The gate judges what merging makes, not the head. The build seat takes
the trunk the pull request merges into, its head at that moment as the check's trunk commit, and makes in
its own clone the merge of the head into it, as the forge's default merge would; both layers judge that
merge, and a head that does not merge cleanly fails, naming the files in conflict. The merge is only checked:
never pushed, never registered. The verdict names the head and the trunk commit in both statuses' words
("head … merged into trunk …"), in mesh-delivery's checks and show and in the kept delivery plan, and
names the commits of the other repositories the check read; their move is issue 568's question. When the
trunk moves, the forge's holder puts both statuses of every open pull request into it back to pending with
the reason and runs nothing; the merge tool refuses a verdict whose trunk commit is not the trunk's head,
asks that head's check once, says how long the last check of the repository took, and never merges by
itself (checked by the builder's TestReplay341, replayed as R341, and the forge module's trunk and
guard tests).
The replays. mesh-lab carries a scripted scenario for each core incident, asserting the rule's
outcome, run on every merge to mesh-controller, mesh-host and mesh-tools:
| Replay | Incident | Asserts |
|---|---|---|
| R1 | two controllers at once (204) | the second waits; nothing from the stale epoch is applied |
| R2 | a reconcile due during a push (257, 261, 267) | one apply, one report, the newest sequence |
| R3 | the node-engine self-updates during its report (230, 264) | the report arrives under the new build |
| R4 | the bus's authorization reloads during a call (265) | the call's outcome is readable by id |
| R5 | a consumer with several filter subjects under mixed traffic (266) | no announcement is skipped; S5 fires if one is |
| R6 | an unreadable contributions file (241) | refused by name; nothing retired; a condition |
| R7 | the controller rebuilds itself mid-plan (214) | the plan continues under the new epoch |
| R8 | a broken controller, node-engine and tool runner build | each rolled back with no hand; condition and message |
| R9 | each signal of §3 suppressed | its condition within its bound, cleared on return |
A new core issue resolves with its replay added, or with a stated reason none is possible. The live mesh stays the test bed (ADR 0149): a replay covers what must not be done to it on purpose, and every rule keeps a live check.
As built (ADR 0237):
- The snapshot is composed every ten minutes and kept as
facts:latestwhen it moved or is a day old; the one it replaced is let go of, so the store keeps one. It carries, beyond the list above, each machine's roles in words, its reported capabilities, pins, settings, the names of secrets a person gave, and how its declaration composes today; every module the mesh holds as its manifest, with its source and build edges.facts,facts compose,facts export. - The gate is the controller's
merge-gate: the mesh as the snapshot says it is and the mesh with the change, each raised in a throwaway store through the controller's own records and every machine composed twice and validated; what composes in the first and not the second is the change's. It also fails a manifest the judging controller cannot read (version skew), a consumer left out of a grant, a module removed while a machine runs it, two definitions of one name, a stored setting the change cannot keep, and a new module the node-engine would refuse on the first machine that could run it; it warns when a merge would rebuild more than twelve modules, saying when shared code is why (issue 278). - It runs on the build seat, asked by the controller when the forge's announcer says a pull request's
head moved: the repository's own
merge-check.shin the mesh's Go toolchain with no container runtime socket, beside a throwaway store and bus of the versions the snapshot says run, then mesh-lab's replays. The verdict is the head commit's statusmesh/merge-gate, an error never a pass. A catalogue change is judged by the controller the mesh runs; a node-engine change by it built with the change's validator. - Replays live where their incident is: a component's logic in its repository as
TestReplay<issue>, what the mesh runs (the bus's release, the resolver under musl and glibc) in mesh-lab'sreplays/, all named in its register and proved byreplays/cmd/prove. A core issue opened from 2026-10-07 resolves only withreplayorreplay-none, refused without one by the issue tracker'smove(ADR 0275;cycle.pyheld it until then). - The controller's tests run a bus of their own per test, linked in at the release the mesh runs and held to it by a test, so the suite runs in parallel and under the race detector.
10. The phases
Ordered by risk removed per day: detection first, because it covers every class including those not met yet. A phase is done when its done when holds; this section records the date when it does.
Phase 0 — Finish what is in flight
| Repository | Delivers |
|---|---|
| mesh-controller | the located fixes of 244, 265, 266, 267 rolled out; calls moved into mesh-controller_calls; status answered from a summary the event loop keeps current, inside ten seconds; the hand-act log with --why on the repairing verbs and hand-act record; recording the durations the bounds come from (apply duration per machine, heartbeat gaps, plan tier durations, build durations) |
mesh-host |
the fixes of 264 and 257/261 rolled out to every machine |
| mesh-tools | the mesh MCP server passing a mesh seat's node (244) rolled out |
| mesh-catalog | the bus's 2.10 → 2.11 upgrade, done as the first planned bus step by hand (snapshot, announced, checked after), recorded as a hand act |
Done when: the four located core issues resolve with their live checks; a controller restart keeps
every call's outcome; status answers in full within ten seconds five times in a row; the hand-act log
has a week of entries; the durations are recorded for every machine.
Phase 1 — The mesh says when it is wrong
| Repository | Delivers |
|---|---|
| mesh-controller | the condition store, its verbs, history and events (§2); ADR 0224's standing moved into it; watchdogs for S1–S11 and S13 with bounds set from Phase 0's durations (S12 is Phase 2's, S14 Phase 5's); the bus advisories subscribed and translated (S9), and every user's slow consumer and closed connection read back from the bus module's events (ADR 0357); doctor with D1–D4, D6–D10, DW and its heartbeat (§4; D5 is Phase 2's); status led by open conditions; the test generated from the signals table |
mesh-host |
the heartbeat carries its interval; D1's validator published as a package the controller imports |
| mesh-tools | the tool runner's heartbeat (S11) |
| mesh-catalog | the operator-channel seat and its holder; the Telegram channel and the desktop notifier contributing to it; mesh-watcher on a machine other than the control node (§5); the nats module's watch of its server's closed connections, publishing bus.slow-consumer and bus.connection-closed (S9, ADR 0357) |
| mesh-lab | R9: each signal suppressed in turn |
Done when: on a lab mesh, suppressing each signal raises its condition within its bound and sends a message; restoring it clears both. Stopping the controller makes the watcher send self-check silent within twice the self-check's interval. Live: a week of conditions read back, every one real or its bound corrected in the table.
Phase 2 — Order and one writer
| Repository | Delivers |
|---|---|
| mesh-controller | the lease and epoch (§6); plans written by compare-and-set; a report kept by sequence; abandoned calls marked; S12, S13, D5; the writers table enforced at grant composition; contract tests for every consumed subject and the check listing them; the empty-on-error lint |
mesh-host |
one apply queue; the report sequence kept on disk; epoch refusal reported; the report verb; the contract tests and lint |
| mesh-tools | refusing an unreadable or unknown input by name; the lint |
| mesh-catalog, mesh-sdk | retirement in the providers' loop (ADR 0230, replacing ADR 0229's brake): a consumer no longer asked for is retired — disabled, marked, its data kept; or, by a provider that cannot disable, only marked, its access kept — only once the same result holds for five passes and ten minutes; remove is never called to retire; a set of more than three, or more than half of those held, waits for a person and raises a condition; the four retirement tools on every provider |
| mesh-lab | R1, R2, R6 |
Done when: R1 ends with nothing from the stale epoch applied, R2 with one report of the newest sequence, R6 with nothing retired; every consumed subject has its contract test.
Built 2026-10-06 (ADR 0229):
the controller's lease and epoch, plans by compare-and-set, accounts kept by order, S12, S13 by writer,
D5, the writers table enforced at composition, a contract for every consumed kind, the empty-on-error
lint; the node-engine's one apply queue, report order and epoch refusal; the withdrawal brake in the Go
providers' loop and the SDK's. Not yet: the replays R1, R2 and R6 in mesh-lab; mesh-tools' refusals
and lint; the lint in mesh-host; the SDK's loop announcing a provider's standing, without which its
brake is said in its journal only.
Amended 2026-10-06 (ADR 0230,
the operator's decision): the brake's hourly release is replaced by retirement — three states (active,
retired, deleted), five stable passes and ten minutes before a consumer is retired, a provider that
cannot disable marking only (every TypeScript provider today), a person for more than three or more
than half, retire approve|reject, cleanup list|delete, retirement events and D11. Built on branches
in mesh-catalog (both Go providers), mesh-sdk (0.1.12) and mesh-controller, not yet merged. Not yet:
the TypeScript providers giving their adapters a retire that disables (until then each is mark-only:
its retired consumers keep their access), passing their module name and an announcer to the loop,
without which a set over the bound there waits with no way to approve it, and adapters that list and
delete what they retired.
Phase 3 — Healers
| Repository | Delivers |
|---|---|
| mesh-controller | the healer registry and H1–H4, each braked and said; S15 |
| mesh-lab | an induced failure per healer |
Done when: a lab mesh recovers from each induced failure with no hand, says so, and brakes after its budget. Live: a week with no cause repeated in the hand-act log.
Built 2026-10-06 (ADR 0231): the healer registry
and H1–H4 in the controller, H5 registered as the provider's own; the mesh-wide brake; healer-acted;
healers; S15 live; the node-engine's report verb. Each healer's induced failure runs in the
controller's tests against a real store, and H3 and H4 against a real bus: induced, repaired, cleared by
the probe's own next run, and handed to the operator after the budget. Not yet: the induced failures
on a lab mesh (mesh-lab), and so the phase's done when; the live week.
Amended 2026-10-06 (ADR 0233,
the operator's decision, before Phase 4): a module declares the data it holds; D13 and its conditions;
the data verb; cleanup extended to a module's own retired data, deleted by the machine's backup
holder after a last restore point; the controller's grant names node-backup.backed-up. Built on
branches in mesh-controller (migration 0072), mesh-host (the installer's user list; the kept-directory
report), mesh-catalog (the holder's measuring and deletion, every module's data, the providers' held
sizes), the media catalogue and the photo application, not yet merged. Not yet: the TypeScript
providers saying their consumers' sizes (until then a consumer at two of them is data-held-twice, a
warning, not compared), and SMART read under an array.
Phase 4 — Core upgrades that roll back
| Repository | Delivers |
|---|---|
| mesh-controller | the health probes of §8; the gate on a core component's first machine — as built, on every plan's first machine (ADR 0236); the upgrade record in the plan; the bus maintenance step as a verb; the default upgrade policy roll (ADR 0236) |
mesh-host |
keeping the previous controller and tool runner builds; restoring one when its health is not met in bound, watching the lease bucket for the controller |
| mesh-tools | answering the health ping |
| mesh-lab | R3, R7, R8 |
Done when: on a lab mesh, a broken build of the controller, the node-engine and the tool runner — one that starts and does nothing, one that crashes, one that cannot reach the bus — is each rolled back with no hand, the mesh ends on the previous build, and a condition and a message say so. Live: the next three core builds to roll out each record a health verdict.
Built 2026-10-06 (ADR 0236,
the operator's "start phase 4"): in mesh-controller (migration 0073), the probes H-controller, H-engine,
H-tools, H-bus, DG and DB; the gate on every plan's first machine, its record in the plan (plans <id>)
and in the store; the rollback by the ordinary path, once per build; the witnesses' verdicts as
conditions; the grants the witness reads; bus and bus upgrade; the default policy roll, upgrade
listing every module's policy and where it comes from; the user list carried after a controller passes;
a module deleted at its source forgotten, not built. In mesh-host, the witness (keeping the previous
controller and tool runner, restoring one not healthy in bound, the launcher's for the node-engine) and the
installer's grants. In mesh-catalog, the modules that keep record say why, and the forge's announcer
says which files a merge deleted. On branches, not yet merged. Not yet: R3, R7, R8 on a lab mesh, and
so the done when; container state in the node-engine's report, without which a container that
crash-loops after its compose applied is seen only through what it breaks; (designed 2026-10-07 by
ADR 0240 as
to-be 48, whose phases track it) composing not-reversible for
a build whose migration cannot be undone; the verdicts of the next three core builds to roll out.
Phase 5 — Checks before merge, and the replays
| Repository | Delivers |
|---|---|
| mesh-controller | the facts snapshot and S14 |
mesh-controller, mesh-host, mesh-catalog |
the compose-and-validate merge gate; versions tested as run; the resolver's tests under both C libraries |
| mesh-lab | R4, R5 and the rest of the window's incidents; running the replays on every core merge |
Done when: the replays of 236, 262, 263 and 266 fail on the commit before their fix and pass after; a new core issue cannot resolve without a replay or a stated reason.
Built 2026-10-06 (ADR 0237,
the operator's "start Phase 5"), on branches, not yet merged: in mesh-controller the facts snapshot and
S14, merge-gate, the check asked of and run by the build seat, merge-check.sh, the replays of 263 and 273,
and a bus per test at the mesh's release; in mesh-catalog the forge's announcer of pull requests' heads and
the verdict as their status, and merge-check.sh; in mesh-host merge-check.sh and the grants in the
installer's user list; in mesh-lab the replays of 262 and 266, the register and the prover; in hq the
replay rule in cycle.py. The prover ran the replays of 236, 262, 263, 266 and 273: each fails on the
commit before its fix (236's check did not exist there) and passes on it. Not yet: R4, R5 and the
rest of the window's incidents as replays; the merge gate's first runs against the live mesh's snapshot;
the status required to merge, which is the operator's setting on the forge; mesh-tools' and mesh-sdk's
merge-check.sh.
Revised 2026-10-06 (ADR 0238, the operator's "enable the checks on every repository of the mesh", "the logic acts on our module graph" and "the commit is the build at hand"), on branches, not yet merged:
| Repository | Delivers |
|---|---|
| mesh-controller | the planner's one mapping of a changed file onto modules and its one answer of a merge's reach, asked by the merge handler, the what-if, the gate and the check; a file no build reads touches nothing (issue 280's left open); every pull request answered — the gate when it reaches a module, the repository's own check for the mesh's repositories, a pass that says so otherwise; the gate run by the build seat with its judge chosen from the graph, composing the plan's definitions only, failing only a manifest problem the change brings; the delivery plan computed and carried with the verdict; a build of a commit off its module's trunk recorded and never registered; its own manifest naming every verb of its seat |
| mesh-catalog | the forge's announcer saying, with every pull request, the directories holding a module at its head, the files it deletes and whether it has a merge-check.sh; both statuses and the delivery plan on the pull request; gitea_branch_protection_get and gitea_branch_protection_set; the catalogue's own check |
mesh-host, mesh-tools, mesh-sdk, mesh-lab, mesh-media-catalog, and the applications built from their own repositories |
each its own merge-check.sh; mesh-tools' tests each on a bus of their own |
| mesh-tools-go, mesh-tools | a C compiler and Python in the Go toolchain, git in the TypeScript one |
| hq | its own merge-check.sh, running records.py, index.py and cycle.py |
Not yet: the delivery plan kept (id, inputs, result) and the release comparing its plan with it; the
state machine replacing the rollouts' states, with its table, its test, its conditions and healer H2;
the status's link to the kept plan; the notes under refs/notes/mesh-plan; check builds in a scratch
namespace (the check builds no module today, so there is none to keep apart yet); the statuses made
required — prepared as calls of the forge module's tool, applied by the operator.
Given an owner 2026-10-06 (ADR 0239): the state machine, the
notes and the status's link are built as a delivery's, owned by the module mesh-delivery. They are designed
in to-be 47, and their progress is tracked there.
What is not decided here
- The bus as a cluster of three, to upgrade it live.
- Routing by presence, quiet hours, answering back through a channel, and an external dead-man service — decided by ADR 0234, designed in to-be 46.