301: S15 counted the operator's pushes of recorded builds, the word ADR 0242 asks for, as repairs and wanted a healer for three causes. 302: the lab's replays register was never compiled before a merge, because its check ran in the TypeScript toolchain. To-be 45 §7 and §9 say how each is now decided and checked.
66 KiB
layer, status, code, updated, decisions
| layer | status | code | updated | decisions | |||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| to-be | in-progress |
|
2026-10-08 |
|
45 — A core that cannot fail silently
The core says when it is wrong, refuses what is stale or unreadable, repairs what it knows how to repair, replaces itself one machine at a time with something other than itself watching, and is checked against the mesh's real facts before a change merges (ADR 0227, from research 031).
The core, here: the controller, the node-engine and its launcher, the bus server, the tool runner and the mesh MCP server, the build seat, and the forge's announcer of merges.
This document is the build's specification. §1–§9 are the parts; §10 is the order they are built in, each phase with what it delivers, in which repository, and when it is done. Every bound marked provisional is set from the durations Phase 0 records and corrected in Phase 1's first live week.
The parts, and how a fact reaches the operator
signals ──────────► watchdogs ─┐ ┌─► status (open conditions first)
(heartbeats, reports, │ │
plan progress, advisories) ├─► CONDITION STORE ───┼─► condition events ─► operator-channel ─► Telegram
doctor probes ─────────────────┤ (controller is │ holder desktop notifier
(live invariants, every 5 min) │ its only writer) └─► healers ─► repair, braked ─► event
events (provider failing, …) ──┘ │
└─► hand-act log (people)
second machine: watcher ── hears doctor's heartbeat? ── no, past bound ──► Telegram directly (not the bus)
Owning repositories: mesh-controller (the condition store, watchdogs, doctor, healers, the lease,
calls, the hand-act log, the facts snapshot), mesh-host (the node-engine: the apply queue, report
order, epoch refusal, the report verb, rollback witnessing), mesh-tools (the tool runner and the
mesh MCP server: their heartbeat, passing every argument, health answers), mesh-catalog (the
operator-channel seat's holder and channels, the watcher, the providers' retirement, the catalogue's
merge gate), mesh-sdk (the TypeScript providers' loop and its retirement), mesh-lab (the
replays and the induced-failure scenarios).
1. The writers table (rule 1)
Every kind of state the core keeps, and the one component that writes it. Anyone else asks.
| State | Writer | Kept in | Others |
|---|---|---|---|
| a machine's declaration | controller (lease holder) | the bus, last per subject | read |
| a machine's applied state and its report | the node-engine's apply queue (§6) | the machine; the report on the bus | the reconcile and a delivery enqueue, never apply |
| the controller lease | the controller instance holding it | key-value mesh-controller_lease |
a candidate waits |
| plans and their tiers | controller (lease holder), compare-and-set on the plan's revision | the controller's store | read through plans |
| conditions | controller | key-value mesh-controller_conditions, and their history mesh-controller_condition-history |
raise or clear only through observations the controller reads |
| calls and their outcomes | controller | key-value mesh-controller_calls |
read by id |
| the hand-act log | controller, through the verbs that act | key-value mesh-controller_hand-acts |
— |
| the healers' acts and their brake | controller (lease holder), each act begun before it is made | the controller's store | read through healers; each act said as healer-acted and in its condition's tried |
| stream definitions and bus permissions | controller | the bus | — |
| builds and their outcomes | the build seat's holder | its own state | the controller asks |
| a merge announced | one announcer per forge (the hook, or the poll when the hook is absent — never both) | the bus | — |
| a pull request's head announced | the forge's announcer, the one that announces its merges | the bus | the controller asks the build seat to check it |
| a pull request's merge check | the build seat's holder that ran it, said as the controller's checked |
the bus | the forge's holder sets it as the pull request's status |
| a provider's standing | the provider | the provider's events | the controller keeps the newest word as a condition |
| a consumer's retirement: active, retired (when, why), deleted | the provider | its own backend's mark; said on provisioner.retirement |
the controller asks the provider's tools; a person approves, rejects and deletes through the controller's verbs |
| the operator-channel's open messages | the seat's holder | its own key-value state | — |
| a machine's declared data: what it measured, what is retired, what was deleted (ADR 0233) | controller, from what each node-backup holder answers |
the controller's store, data_item and data_reading |
the holder measures and says; it records nothing the controller keeps |
| the facts snapshot | controller | the artifact store, facts/latest |
the build seat reads |
A bus subject two components may publish on is refused when the controller composes grants, unless this table marks it shared. Changing a writer is a change to this table, through a decision.
How it is enforced (ADR 0229):
the table is compiled into the controller, each row the bus carries with the subjects a write of it
publishes to — a declaration mesh.node.*.declare, a report mesh.control.*.report (its own machine's
only), the controller's buckets, stream creation, update and deletion and durable consumer creation, a
build's outcome, a merge announced, a provider's standing — and its writer among the principals. A
grant that overlaps another writer's subject is refused at composition, naming the state and its
writer. A test holds the compiled table to this one, row for row. The controller does not publish
mesh.control.>: it would be a second writer of every machine's report.
2. The condition store (rules 5, 6)
A condition is a durable fact about something the mesh owns that is wrong. It is raised and cleared by observation only.
Where. A key-value bucket, mesh-controller_conditions, written by the controller alone, one entry
per open condition, with no age. Every transition — raised, changed, silenced, cleared — is also
appended to a history kept ninety days in a bucket of its own, mesh-controller_condition-history,
read through conditions history: a bucket has one age for every key, and an open condition must
outlive the history.
The key names the thing and the kind, so the same fault said again is the same condition:
<scope>.<id>.<kind>, where scope is one of machine, plan, call, build, merge, provider,
seat, bus, core, probe, mesh. Examples of the shape: machine.<node>.silent,
plan.<id>.stalled, provider.<module>.<node>.<consumer>.failing, core.controller.<node>.rolled-back,
bus.<consumer>.slow-consumer. The last token is the kind's short word where it has one (failing for
provider-failing), the kind elsewhere; a key is opaque to every reader, and the kind is the field.
Added 2026-10-07, ADR 0240:
the scope module, for a module's own health on a machine — module.<module>.<machine>.unhealthy, raised by
the controller on the second statement from the node-engine that a long-running resource is unhealthy, and
cleared on the first that does not. Designed in to-be 48.
The fields:
| Field | Holds |
|---|---|
| kind | the condition kind, from the signals table, the probe registry or an event kind |
| subject | scope, id, and the machine it concerns when there is one |
| severity | urgent (needs the operator now) or warning (when they can) — two levels, no more |
| summary | one line in the mesh's words |
| evidence | the newest observations, at most ten, each with its time |
| source | the signals-table row, probe or event that raised it |
| raised, last observed | times; and how many observations since raised |
| tried | each healer attempt: when, what, outcome |
| resolver | self (clears on observation), healer:<name>, operator, or agent |
| silenced | until when, by whom, why — empty when not silenced |
| epoch | the controller lease epoch that last wrote it |
The life of one:
(absent) ──observation past bound──► OPEN ──healer tries──► OPEN (tried += …)
│ ▲ │ budget spent
│ └─ seen again ◄───────┘──► OPEN, resolver: operator, severity: urgent
│
silence (verb, ≤ 7 days, reason, hand act) ──► OPEN, silenced (no messages)
│
observation says resolved ──► CLEARED: entry removed, transition kept in history
- Nobody resolves a condition by hand. It clears when the signal returns or the probe passes.
- A condition cleared and raised again within ten minutes reopens with its count increased; it is not a new message.
- The verbs:
conditions(open ones, filtered by scope, severity or machine),conditions show <key>,conditions silence <key> --for <duration> --why <text>,conditions history. - The events, emitted by the controller on its own subjects:
condition-raised,condition-changed(severity or resolver changed; not every observation),condition-cleared. The operator-channel's holder and any other surface consume them; the controller learns nothing about telling. Each carries the condition at the top level, withevent,at,change(raised, reopened, severity, resolver, silenced, silence-ended, cleared),why,was,clearedandshowbeside it; a silence and its ending arecondition-changed, a reopeningcondition-raisedwith changereopened. statuslists open conditions first, urgent before warning, oldest first, silenced ones marked with their expiry. The all-well sentence requires none open, silenced included.- ADR 0224's provider standing is the first kind:
provider-failing, raised by the provider's event, cleared by its recovery, its silence after thirty minutes the row S8 below. - Retirement (ADR 0230)
raises three, keyed by the provider module and its machine:
retire-waiting(urgent) while a stable set too large to retire alone waits forretire approveorretire reject, listing it;retire-rejected(warning) while a rejected set is kept active though the mesh no longer asks for it; andcleanup-waiting(warning), from the probe D11, while anything retired is older than thirty days. The first two clear on the provider'sretired,settledorapprovedword, and all three when the provider is no longer assigned there. - Data (ADR 0233)
raises, from D13, each urgent for irreplaceable data and a warning for valuable data — the
operator's ranking — and none for rebuildable data or caches:
data-shrank(less than half its largest size in seven days, and 16 MiB less),data-missing,empty-replacement(a copy in use less than half a copy of the same thing kept elsewhere — issue 273's shape — for an item or for a consumer's data at a provider),data-quiet(an item that saysactive, unwritten past it),backup-stale,array-degraded(one per array under watched data) andprotection-missing(an item said to be on redundancy, found on plain storage); and as warningsdata-held-twice,data-unmeasuredandcleanup-waitingfor an item retired over thirty days. Keyed by machine, module and item, or by the array. A shrink a person meant is silenced with why; everything else clears on observation.
3. The signals table (rule 5)
Compiled into the controller. A watchdog per row; the test generated from the table suppresses each
signal and asserts its condition. doctor signals shows, for every row, the age of its newest signal.
| Row | Signal | Emitter | Trigger | Bound | Condition kind | Severity | Healer |
|---|---|---|---|---|---|---|---|
| S1 | machine heartbeat | node-engine | its interval | 3 × interval, provisional | silent — not raised while the machine has declared itself asleep or shut down (ADR 0211) |
warning; urgent after 30 min for the control node | — |
| S2 | report after a send | node-engine | each declaration sent | max(2 min, 3 × that machine's last apply duration), provisional | sent-not-reported |
warning | H1 |
| S3 | plan tier progress | controller's plan | each tier entered | from the tier's build and apply durations, provisional | stalled (names the tier and what it waits on) |
warning | H2 |
| S4 | the controller's event loop takes a message | controller | while its consumer has pending messages | 2 min | controller-deaf |
urgent | — |
| S5 | a merge announced becomes a plan or nothing reads it | announcer → controller | each merge | 10 min (exists, issue 266) | merge-not-acted |
urgent | — |
| S6 | a build asked → its outcome | build seat | each ask | max(20 min, 3 × the p90 of measured builds), 1 h while nothing is measured, provisional (the build seat declares no timeout) | ask-lost |
warning | — |
| S7 | a call running → finished | controller | each call | the verb's bound: push, rotate and command 30 min; assign and unassign 15 min; doctor 3 min; others 10 min |
call-hung |
warning | — |
| S8 | a provider's failing word repeated | provider | every 15 min while failing (ADR 0224) | 30 min | provider-silent |
warning | — |
| S9 | bus advisories: slow consumer, maximum deliveries, permission violation, consumer deleted | bus server's system subjects | any | any occurrence | slow-consumer, max-deliveries, refused, consumer-lost — each naming the call, consumer or module in the mesh's words |
warning | H3 for consumer-lost |
| S10 | the self-check's heartbeat | controller's doctor |
every run | 2 × its interval, watched from the second machine (§5) | self-check-silent |
urgent | — |
| S11 | tool runner heartbeat | tool runner | its interval | 3 × interval, provisional | tools-silent |
warning | — |
| S12 | the controller lease renewed | controller | every 5 s | 15 s; a holder that lost the lease or was found expired, and a lease bucket found raised again from nothing, said for an hour; a controller serving without the lease, while it does | lease-lost |
urgent | — |
| S13 | stale refusals | every receiver (rule 2) | each refusal | more than 5 from one writer in 5 min | stale-writer (names the writer: the controller epoch a refused declaration claimed, with its instance and how its lease ended; or the machine whose older accounts the controller refused) |
warning | — |
| S14 | facts snapshot exported | controller | when it moved, and daily | 2 days, or none kept by a controller up that long (Phase 5) | facts-stale |
warning | — |
| S15 | a hand act with a cause already recorded | hand-act log | each act | the second within 14 days; clears when fewer than two remain within 14 days | healer-wanted (names the cause, and the healer that was not enough where one answers it) |
warning | — |
| S16 | a walk waiting for its delivery's word is let go | mesh-delivery, through deliver (ADR 0239) |
each merge whose walk waits | 30 min; urgent after 4 h | waiting (names the walk and plans go) |
warning | — |
The bus advisories (S9) cost one read-only subscription: the server already publishes them. The controller translates each into a condition naming the thing in the mesh's words, as the refused reply of issue 265 is translated today. An advisory clears after an hour without another; a deleted consumer is said only for one the mesh names and expects, and clears when it exists again. Until the bus has a system account the controller hears a slow consumer and a refused subject for its own connection only (issue 270, open question 2).
4. The self-check: doctor (rule 6)
A probe registry in the controller: each probe is a live invariant of a design, with an id, an
interval, a timeout and the condition kind it raises. The controller runs the registry every five
minutes; each probe has thirty seconds. A probe that errors or times out is said in that run's
verdict at once — an unanswered probe is never a pass — and raises probe-failed for itself when the
next run cannot run it either.
A finding one look can be wrong about is raised on the second look in a row (issue 277): a question over the network that went unanswered, a time measured once, a reading a burst can move. Within a run such a question is asked again before it counts; across runs the finding is raised when the previous run saw it too, kept while it is open however it is seen, and cleared by the run that no longer sees it. Held back, it is listed in the verdict as unconfirmed. A definite answer — an address that is wrong, a stream that is not there — is raised at once. The watchdogs hold a row that cannot read its facts to the same rule over two ticks. Checked by the controller's tests of the self-check over runs and of a blind row over ticks.
A condition's summary names machines and says the rest in words; an address, a domain, a path or a raw error is its evidence, which stays inside the mesh. The condition store holds every summary to the operator channel's content rule (ADR 0234 §6) and says one that would be withheld in words, keeping it whole in the evidence; a lint over every condition raised in the controller's test suite fails the producer that wrote it.
| Probe | Asserts | From |
|---|---|---|
| D1 | every machine's declaration composes, and passes the node-engine's validation (the validator is a package of mesh-host the controller and the merge gate import — one validator) |
236, 263 |
| D2 | every holder of the mesh's resolver answers a machine name for IPv4, and NODATA for IPv6 | 262 |
| D3 | every seat on record has a live holder that answers (holder-silent, H3) |
208, 218 |
| D4 | every kept archive is held by a manifest | 253 |
| D5 | exactly one lease holder — the key names this controller at the epoch it acts under, and the controller's record holds no other epoch open; no message from a stale epoch refused in the last interval | 204 |
| D6 | every durable consumer exists with its definition and is near its stream's head (within 1000 messages, provisional); a missing one is consumer-lost (H3), one far behind consumer-behind (H4) |
248, 266 |
| D7 | every stream the controller defines exists with its definition | 208 |
| D8 | no address the mesh owns is in a ban list | 238 |
| D9 | status answers in full within ten seconds |
265 |
| D10 | every machine runs the node-engine and tool runner builds its plan says, or is inside a plan's window | version split |
| D11 | no provider holds a consumer retired more than thirty days: each provider assigned is asked provisioner_retirement on its machine (ADR 0230); one that does not serve it yet is named, not failed |
ADR 0230 |
| D12 | every consumer of a provision that keeps its data is bound where it was last sent, or moves by a pin (binding-kept, binding-moving, binding-moved) |
ADR 0232 |
| D13 | every item of data a machine declares is measured, is there, holds what it held, is written where it says it is, is backed up within its bound or sits on healthy redundant storage, and is no empty replacement of a copy kept elsewhere; an irreplaceable or valuable item a machine no longer declares is retired, not forgotten. Each machine's node-backup holder is asked backed-up; every provider of kept consumer data provisioner_retirement (with each held consumer's size) — see §2 for the kinds |
ADR 0233, issue 273 |
| DW | the watchdogs of §3 ran within three of their intervals: the watchers are watched, and raise S10 when the self-check stops | rule 6 |
| H-* | the health probes of §8, run for every core component on every machine: H-controller, H-engine, H-tools, H-bus (as built, ADR 0236) | rule 8 |
| DG | every build that failed its gate, and every core build a witness put back, is said — build.<module>.<machine>.rolled-back, core.<component>.<machine>.<outcome>, rollback-failed — until a newer build passes or the reports stop saying it (ADR 0236) |
rule 8 |
| DB | a bus upgrade a person started is said as bus-maintenance while it runs, and ends healthy within its bound or is said bus-upgrade-failed, with its snapshot (ADR 0236) |
rule 8 |
doctoranswers the last run's verdict at once: per probe, pass, fail or failed-to-run, and age.doctor runruns now under a call id.doctor probeslists the registry;doctor signalsthe table's ages.- Every run ends with a heartbeat event carrying the run's id and counts. That is S10.
- The registry is the design's live form. A check over the to-be designs counts invariants that name a probe against those that do not; the number without may only go down.
5. The output channel, minimal form (rule 6)
From research 028, the smallest form that works. Amended 2026-10-06 by ADR 0234, 028's graduation: this form is the first step of to-be 46, the conversation with the operator, which replaces the points marked below. Phase 1 builds this section; to-be 46 §13 builds what follows from it.
- One mesh seat,
operator-channel, held once. It acceptsnotifyas a work queue, so a message waits for a holder; it keeps its open messages in its own key-value state, so a restart forgets nothing; it servesopenandhistory. - The holder consumes the controller's condition events and decides what is sent. The controller calls nobody.
- Two channels: Telegram (a bot to the operator's chat) and the desktop notifier of the
machine the operator is at. Amended: a channel is not a contribution to this seat. Each is the
holder of a kind on the kinded benches
channelandintake, a module of its own holding its own secrets, and this seat's holder becomes the router (to-be 46 §2, §3, §8). - A message is: the condition's key, its subject (a machine's role, a module, a plan), kind, severity, the one-line summary, since when, and the verb that shows more. Deduplicated by the key.
- When: on
condition-raised; once more if still open after 1 hour (urgent) or 12 hours (warning); oncondition-cleared, by editing the first message where the channel can. A silenced condition sends nothing. Urgent goes to both channels; warning to the desktop notifier when the operator's session is there, otherwise to Telegram. - Rate: at most twenty messages an hour; the excess is folded into one message naming them all.
- What may leave the mesh: roles and words. A message carrying an address, a path or anything
shaped like a secret is refused by the holder and raises
channel-refusedinstead. Amended: the mesh's own machine names may appear too; a concrete detail travels as a reference only a private surface opens, and a refusal names the offending part to the sender (to-be 46 §9). - Answering back. Amended: the minimal form had none, and acknowledging was
conditions silence, through the mesh. The operator now answers and is asked over the intake seat; an answer that performs an action is checked and performed by the controller (to-be 46 §5, §6, §10).conditions silencestays callable. - The watcher's watcher. A module,
mesh-watcher, assigned to one machine that is not the control node, holds the Telegram channel's secret too. It listens for the self-check heartbeat (S10) and for the bus itself; when either has been silent past its bound it sends to Telegram directly over HTTPS, not through the bus, and says so again when they return. It is the only sender that does not pass through the control node.
6. Order: lease, epoch, report sequence, one apply queue (rules 1, 2)
The lease. A controller instance acts — sends a declaration, writes a plan, a condition or a call
— only while it holds the key holder in mesh-controller_lease: written with compare-and-set, a
fifteen-second time to live, renewed every five seconds. The epoch is the bucket revision at which
it was taken. A starting controller waits for the key to be absent or expired. One that fails a renewal
stops acting at once and exits, so its service manager restarts it as a waiting candidate.
As built (ADR 0229):
- The gate is the clock. A holder stops acting three seconds before its key could expire unrenewed, whatever its renewing goroutine is doing; every act passes that gate at the moment it is made — a declaration at its send as well as where it was composed.
- The epoch never goes backwards. The controller's store keeps every epoch issued, its instance,
and how it ended (
released,lost,expired); a lease bucket at or under the highest is moved past it before the key is taken, and S12 says so. - A holder that stops gives the key back, so the next takes it at once.
- Without the bucket's grant, nobody holding it, a controller serves unleased — no epoch on what it sends, S12 urgent — and takes the lease once the user list that grants it reaches the bus.
- A command at a shell acts under the holder's epoch, read as it acts, or under a lease of its own while nobody holds one.
controller A (epoch 41) ──renew──renew──╳ (renewal refused)──► stops sending, exits
controller B ──wait──────────────take (epoch 57)──► acts; marks A's running calls abandoned
node-engine accepts 41 … then 57; refuses anything from 41 after 57, counted, reported
What carries the order:
| Message | Carries | The receiver keeps, per writer | Refuses |
|---|---|---|---|
| declaration | epoch, sequence | highest epoch, then highest sequence | older epoch; same epoch, lower sequence — only where both claim an epoch |
| report | machine, the declaration's epoch and sequence it is about, the node-engine's own report sequence (kept on disk, increasing across restarts and self-updates) | highest epoch where both claim one, then declaration sequence, then report sequence | an account of an older declaration; an older report |
| plan write | the plan's revision, the epoch | — (compare-and-set) | a write against a revision already moved |
| build outcome | the build's ask id and order | newest per module | an older build finishing later (exists, 219) |
| call | call id, epoch | — | a finish from an epoch that is not the holder's, recorded as abandoned |
| merge announcement | forge, repository, commit, the announcer's sequence | newest per repository | a duplicate (one announcer) |
Every refusal is one line in the receiver's log in the mesh's words, a counter, and — from the node-engine — a report naming the refused declaration, so the controller sees it. The counter feeds S13.
The wire — the contract, written once on each side: a declaration's epoch and sequence are
top-level keys of its signed envelope; a report carries back the epoch and sequence of the
declaration it is about, its own report_sequence, older_than (the order held) on a refusal of an
older declaration, and refused_older (how many ever) on every report. Every key is optional, zero is
"no order claimed". A machine is sent epoch only while its latest account carried a
report_sequence, because an older node-engine refuses an unknown key, whole; a declaration without an
epoch is taken by its sequence, so a controller rolled back to a build without the lease is never
stranded. A report without a report_sequence is judged by the digest it names (issue 267). What the
mesh would send a machine is composed with the epoch it was last sent: a new holder is not a change of
the machine.
One apply queue on every machine. The node-engine has one worker that applies; a delivery, the
five-minute reconcile, a self-update hand-over and a manual apply are four reasons to enqueue.
The queue holds at most one pending request, coalesced. When the worker starts it takes the newest
declaration held at that moment, applies it once, and sends one report naming the sequence it applied.
Nothing else in the node-engine applies.
The report verb. The node-engine answers, on request, the report of the last declaration it
applied, from what it keeps on disk. It is what healer H1 asks. As built (ADR 0231):
asked on mesh.node.<machine>.ask.report, its own only, the node-engine enqueues a reconcile whose
account is said whether or not it is news — a delivery waiting is applied and reported instead — and the
answer is its ordinary report; nobody's inbox is answered.
Durable calls. Every call's record — verb, arguments with secrets removed, caller, started, epoch,
state (running, finished, failed, abandoned), the answer bounded in size, finished — lives in
mesh-controller_calls, the last thousand or fourteen days. A new lease holder marks the previous
holder's running calls abandoned, which is said.
7. Healers and the hand-act log (rule 7)
A healer is a registered response to one condition kind: its repair (the ordinary path again), its
budget, its back-off, its brake, and the healer-acted event naming the condition, the act and the
outcome. A healer may not withdraw, delete or recreate data; such a repair is a condition for the
operator.
| Healer | Condition | Repair | Budget, then |
|---|---|---|---|
| H1 | sent-not-reported |
ask the machine's node-engine for report; if it does not then report the declaration it was sent, send it again — never moving a build a policy or a plan holds back |
twice per condition in 6 h, then resolver operator, urgent |
| H2 | stalled on a wait that is superseded (a newer plan of its repository and branch) or already finished (every module built or failed, every one that rolls out sent) |
close the plan with its note, as plans close does: superseded or done |
once |
| H3 | a seat holder that does not answer (D3, holder-silent), or a consumer the mesh expects missing (D6, S9, consumer-lost) |
assert the bus's objects again — the assertion every send makes (208) | once per object per hour |
| H4 | consumer-behind (D6) on a consumer the stream table marks resettable — the controller's own events consumer alone |
broker consumer-reset (248) |
once a day per consumer, then condition |
| H5 | provider-failing with the administrator refusing the mesh's secret |
ADR 0224 §5 (exists, in the module) | as ADR 0224 |
As built (ADR 0231): the registry is compiled into the controller with a test generated
from it; each act is begun in the controller's store before it is made, under the lease, and the
budgets are counted from there; a healer that does not apply to a condition leaves it alone; only
observation says a repair worked — the act is kept in the condition's tried as healer Hn and said
as healer-acted, and the condition clears when what raised it no longer sees it; a spent budget hands it
to the operator, urgent, and no healer touches it again. The mesh-wide brake: twelve acts in an hour,
all healers together, stop every healer — mesh.healers.braked, urgent — until an hour after the last.
A heal is never a hand act. healers lists the registry, the acts and the brake.
The hand-act log. Every verb that repairs by hand — a named push outside a plan, plans close,
broker consumer-reset, conditions silence, retire approve, retire reject and cleanup delete
(ADR 0230; and an approval, rejection or deletion a provider reports that did not come through the
controller), and hand-act record for an act done outside the mesh —
takes a required --why and writes an entry: who, which verb and arguments, why, when, the condition
key it addresses if any, and a cause (the condition kind, or a word the person gives). status
shows the week's count. A cause recorded twice within fourteen days raises healer-wanted (S15) —
except an act recorded by a verb that is a person's decision by design, which is no repair and never
counts, whatever cause it gives. The controller keeps one table of the verbs that write the log, and
each says whether it records a repair or a decision, and why: approving or rejecting a retirement and
deleting what was retired (ADR 0230); bus upgrade, because the bus is never rolled by the mesh, and
upgrade release-backlog, because after a failed walk the next opens only on a person's word
(ADR 0236); and secret rotate when its cause is a leak (leaked-in-logs) — a person judges what was
disclosed, one leak rotates several values, and a leak that recurs is a defect of the module that
prints them, an issue against it, not a healer that rotates. A rotation for any other cause counts. A
push by hand, plans stop and close, broker consumer-reset, conditions silence and hand-act record (an act outside the mesh, whose kind the mesh cannot tell) count; so does a verb the table does
not list. The table's test finds every verb the controller records and fails on one it does not list.
A healer-wanted already open for a decision clears on the next watchdog tick, as any condition its
row no longer sees.
A drill is no repair. Something broken on purpose to see the mesh raise and clear it is recorded by
hand-act drill (the seat's drill), cause drill, which the table lists as a person's decision;
hand-act record refuses that cause, so a repair cannot pass for a drill by the word it gives (issue
292; checked by the controller's S15 tests).
A push of recorded builds is no repair. A build of a module whose upgrade policy records moves only by a person's push (ADR 0242), and that push is the word the policy asks for. So a push says what it was, read by the controller from what it carries and never from a word the person gives. It is a push of recorded builds when it names one machine, that machine's last send is known and its last report is no failure, and every build it moves records rather than rolls out: a held new build, or a recorded module's first build on a machine newly assigned it. The act is written with its kind and what it carried; the table lists a push of that kind as a person's decision, and S15 never counts it. Any other push by hand still counts (issue 301; checked by the controller's replay R301 and its tests of the rule).
Progressive insight — 2026-10-06. This paragraph said before: "except
retire-waitingandcleanup-waiting: approving a retirement and deleting what was retired are a person's decision by design (ADR 0230), and no healer may take them over." The exception was keyed on two causes, and the first planned bus upgrades (two within a day, one recorded after the fact and one throughbus upgrade) raisedhealer-wantedforbus-upgrade, as did two rotations after one leak forleaked-in-logs. What makes an act a decision is the verb that recorded it, not the word it gives as its cause, so the exception is read from the verbs. Nothing decided changes: ADR 0230's acts stay exempt, and pushes by hand — what roll-out by default (ADR 0236) exists to end — still count.
8. Staged core upgrades and rollback (rule 8)
Health, per core component, as probes in the registry (§4):
| Component | Healthy when | Witness that rolls it back |
|---|---|---|
| controller | holds the lease within 60 s of starting; status answers in full within 10 s; doctor ran once |
the node-engine on the control node, which keeps the previous controller build installed beside the new one and reads the lease bucket |
| node-engine | has reported its current declaration under its own build | its launcher (ADR 0141), which keeps the known-good |
| tool runner | announced, and answer a ping within 5 s | the node-engine, which keeps the known-good |
| bus | every stream and durable consumer present (D6, D7); a request/reply round trip from every machine | none — a planned step, below |
The gate. ADR 0218's first machine for a core component is judged by that component's health probes passing three consecutive times over two minutes, within ten minutes of the apply. Only then do the other machines follow. "Reported applied" is not enough.
merge ─► plan ─► first machine applies new build ─► health probes ×3 within 10 min?
├─ yes ─► the rest follow, one tier at a time
└─ no ──► witness restores previous build
─► condition core.<component>.<node>.rolled-back (urgent)
─► plan halts that component, records why
- Every core upgrade leaves a record in its plan: component, first machine, from and to build,
verdict, time to verdict, rolled back or not. Read through
plans. - A rolled-back build is not retried by the same plan. A newer merge makes a new plan.
As built (ADR 0236):
- Every walk is gated, not only the core's. A module is judged by its own health: it reported applied, no witness put it back, no condition was raised since its send about the machine or about the module there, and its tools are served on that machine where it has tools. Three healthy judgings at least forty seconds apart and two minutes after the send, within ten minutes of it. A policy of together is not gated.
- The rollback is the ordinary path once per build. The build is marked failed at its gate (never
registered again,
plans retryrefuses it); the module's registered build goes back to the build the first machine ran before, still kept by ADR 0189; that machine is sent it. The verdict is written before the send. Said asbuild.<module>.<machine>.rolled-back(warning) orcore.<component>.<machine>. rolled-back(urgent),rollback-failed(urgent, the operator's) when nothing could be put back, and as the eventrolled-back. - The witness, as built on the node-engine (
mesh-hostinternal/witness/contract.go, the controller's halfinternal/lease/witness.go): the controller's is the lease alone — the holder on this machine, taken since the start, renewed within fifteen seconds, within sixty seconds of the start — and the tool runner' is their answer to the services protocol's ping within five seconds, within sixty. The controller's "status in bound, doctor ran once" is the controller's own word in the lease's value, which the gate reads and the node-engine does not: a controller that holds the lease and never becomes ready is put back by the gate sending the previous build. A witness says its verdict inrollbackson every report while it stands; the controller raisescore.<component>.<machine>.<outcome>from it — urgent for rolled-back, not-reversible, restore-failed and halted — and clears it with the first report without it. - No build reaches a machine without a gate: a gated send carries and judges everything waiting on
its machine; every other send — a plan's rest, a cascade, a healer's, a whole-mesh push — is refused
or leaves the machine while a build no gate has seen waits there; a rebuild with the same artifacts and
manifest is no move. What waits is moved by a walk, one machine at a time, the control
node last, each judged; one that fails holds the next until
upgrade release-backlog --why. - A new controller that passes its gate sends the bus's machine the user list it composes, when that changed and nothing held back would go with it.
- The default policy is to roll (ADR 0236 §4);
recordstays where a module says why, keeps irreplaceable data, or is the bus.
The bus is a planned step. A bus upgrade is a maintenance step a person starts through the
controller: the streams are snapshotted; a bus-maintenance condition is open for the step's duration;
the bus is replaced; afterwards D6, D7 and the round trip must pass, or the step is reported failed and
the snapshot is the way back. A step whose new version cannot be reverted (the bus's 2.10 → 2.11 is one)
says so before it starts and runs only on a person's explicit word, recorded as a hand act. Whether the
bus becomes a cluster that can be upgraded live is left to its own effort.
As built (ADR 0236): the verb is bus upgrade --why … --reversible|--irreversible; the snapshot is the
bus machine's backup holder backing the bus module up now (ADR 0235),
or one a person took and names with --snapshot-taken; the step's bound is fifteen minutes; no send
reaches the bus's machine while a new bus build waits for it — a push, a plan's send for another
module, a healer's — after a plan's send restarted the bus unasked on 2026-10-06; a change of the bus's
user list is reloaded in place, not a restart. 2026-10-06: the snapshot
and the way back from it exist — the bus image's own snapshot program, the same one the night's backup
runs, and a restore that builds a new store beside the live one for a person to swap in
(ADR 0235,
to-be 43).
9. Before merge: facts and replays (rule 9)
The facts snapshot. The controller exports daily, and after any change of machines, assignments or
seats: every machine (a stable pseudonym of the same length as its name, its role, operating system,
C library, architecture, node-engine and tool runner builds), assignments, settings keys and their
non-secret values, seats and their holders, the catalogue commit, and the versions the mesh runs of
the bus server, the store and the node-engine. No secret and no address: an address is replaced by one
from a documentation range. It is kept in the artifact store as facts/latest, where the build seat
reads it.
The merge gate. In mesh-controller, mesh-host and mesh-catalog, a check composes every machine of
the snapshot with the change applied and runs the node-engine's validator over each. A change that
makes any machine fail to compose or validate fails its check, naming the machine's role and the
module. The resolver module's tests run under both C libraries the snapshot lists. A test in each core
repository asserts that a pinned dependency's version equals the version the snapshot says runs.
Revision, ADR 0238
(2026-10-06). The commit is the build at hand. A commit off its module's trunk is only checked; a
commit on it is published and pushed, and the controller refuses to register a build of one off it. The
module graph, not the repository, decides what a pull request's check runs: every pull request the
forge holds is announced, and the controller asks the planner — the one function that maps a changed file
onto modules, and the one answer of what a merge moves and builds — what a merge of the head would reach.
A changed file touches exactly the modules whose build reads it; a file no build reads touches nothing. A
change reaching a module, or adding one, runs the gate on the build seat (mesh/merge-gate): the
touched manifests through module check, failing only what the change brings; every machine composed with
the definitions of the modules the plan moves or adds; the replays. Every repository of the mesh runs its
own merge-check.sh (mesh/repo-check) in the toolchain it declares. A repository in two languages
declares each further part of its check and the toolchain that part runs in, and the build seat runs each
part in that toolchain's own container: the layer passes only when every part does (issue 302; checked by
the builder's TestARepositoryInTwoLanguagesIsCheckedInBoth). A repository with none has "no repository
check defined", a success where the branch does not require mesh/repo-check and a failure for a person
where it does. A note is never a warning on a required status: the forge combines warning as a
failure, so a note — a wide rebuild, a problem already so on the base — is a success that says it, and only
what a person must decide fails (issue 293; checked by the forge module's status tests). The
check's result is the commit's delivery plan — the build plan, the deploy plan machine by machine with
what is not an ordinary send, and the verdict — posted on the pull request. The plan is one object with
a state machine, kept, followed by the release and written to the commit as a note; which parts are built
is Phase 5's.
The replays. mesh-lab carries a scripted scenario for each core incident, asserting the rule's
outcome, run on every merge to mesh-controller, mesh-host and mesh-tools:
| Replay | Incident | Asserts |
|---|---|---|
| R1 | two controllers at once (204) | the second waits; nothing from the stale epoch is applied |
| R2 | a reconcile due during a push (257, 261, 267) | one apply, one report, the newest sequence |
| R3 | the node-engine self-updates during its report (230, 264) | the report arrives under the new build |
| R4 | the bus's authorization reloads during a call (265) | the call's outcome is readable by id |
| R5 | a consumer with several filter subjects under mixed traffic (266) | no announcement is skipped; S5 fires if one is |
| R6 | an unreadable contributions file (241) | refused by name; nothing retired; a condition |
| R7 | the controller rebuilds itself mid-plan (214) | the plan continues under the new epoch |
| R8 | a broken controller, node-engine and tool runner build | each rolled back with no hand; condition and message |
| R9 | each signal of §3 suppressed | its condition within its bound, cleared on return |
A new core issue resolves with its replay added, or with a stated reason none is possible. The live mesh stays the test bed (ADR 0149): a replay covers what must not be done to it on purpose, and every rule keeps a live check.
As built (ADR 0237):
- The snapshot is composed every ten minutes and kept as
facts:latestwhen it moved or is a day old; the one it replaced is let go of, so the store keeps one. It carries, beyond the list above, each machine's roles in words, its reported capabilities, pins, settings, the names of secrets a person gave, and how its declaration composes today; every module the mesh holds as its manifest, with its source and build edges.facts,facts compose,facts export. - The gate is the controller's
merge-gate: the mesh as the snapshot says it is and the mesh with the change, each raised in a throwaway store through the controller's own records and every machine composed twice and validated; what composes in the first and not the second is the change's. It also fails a manifest the judging controller cannot read (version skew), a consumer left out of a grant, a module removed while a machine runs it, two definitions of one name, a stored setting the change cannot keep, and a new module the node-engine would refuse on the first machine that could run it; it warns when a merge would rebuild more than twelve modules, saying when shared code is why (issue 278). - It runs on the build seat, asked by the controller when the forge's announcer says a pull request's
head moved: the repository's own
merge-check.shin the mesh's Go toolchain with no container runtime socket, beside a throwaway store and bus of the versions the snapshot says run, then mesh-lab's replays. The verdict is the head commit's statusmesh/merge-gate, an error never a pass. A catalogue change is judged by the controller the mesh runs; a node-engine change by it built with the change's validator. - Replays live where their incident is: a component's logic in its repository as
TestReplay<issue>, what the mesh runs (the bus's release, the resolver under musl and glibc) in mesh-lab'sreplays/, all named in its register and proved byreplays/cmd/prove. A core issue opened from 2026-10-07 resolves only withreplay:orreplay-none:(cycle.py). - The controller's tests run a bus of their own per test, linked in at the release the mesh runs and held to it by a test, so the suite runs in parallel and under the race detector.
10. The phases
Ordered by risk removed per day: detection first, because it covers every class including those not met yet. A phase is done when its done when holds; this section records the date when it does.
Phase 0 — Finish what is in flight
| Repository | Delivers |
|---|---|
| mesh-controller | the located fixes of 244, 265, 266, 267 rolled out; calls moved into mesh-controller_calls; status answered from a summary the event loop keeps current, inside ten seconds; the hand-act log with --why on the repairing verbs and hand-act record; recording the durations the bounds come from (apply duration per machine, heartbeat gaps, plan tier durations, build durations) |
mesh-host |
the fixes of 264 and 257/261 rolled out to every machine |
| mesh-tools | the mesh MCP server passing a mesh seat's node (244) rolled out |
| mesh-catalog | the bus's 2.10 → 2.11 upgrade, done as the first planned bus step by hand (snapshot, announced, checked after), recorded as a hand act |
Done when: the four located core issues resolve with their live checks; a controller restart keeps
every call's outcome; status answers in full within ten seconds five times in a row; the hand-act log
has a week of entries; the durations are recorded for every machine.
Phase 1 — The mesh says when it is wrong
| Repository | Delivers |
|---|---|
| mesh-controller | the condition store, its verbs, history and events (§2); ADR 0224's standing moved into it; watchdogs for S1–S11 and S13 with bounds set from Phase 0's durations (S12 is Phase 2's, S14 Phase 5's); the bus advisories subscribed and translated (S9); doctor with D1–D4, D6–D10, DW and its heartbeat (§4; D5 is Phase 2's); status led by open conditions; the test generated from the signals table |
mesh-host |
the heartbeat carries its interval; D1's validator published as a package the controller imports |
| mesh-tools | the tool runner's heartbeat (S11) |
| mesh-catalog | the operator-channel seat and its holder; the Telegram channel and the desktop notifier contributing to it; mesh-watcher on a machine other than the control node (§5) |
| mesh-lab | R9: each signal suppressed in turn |
Done when: on a lab mesh, suppressing each signal raises its condition within its bound and sends a message; restoring it clears both. Stopping the controller makes the watcher send self-check silent within twice the self-check's interval. Live: a week of conditions read back, every one real or its bound corrected in the table.
Phase 2 — Order and one writer
| Repository | Delivers |
|---|---|
| mesh-controller | the lease and epoch (§6); plans written by compare-and-set; a report kept by sequence; abandoned calls marked; S12, S13, D5; the writers table enforced at grant composition; contract tests for every consumed subject and the check listing them; the empty-on-error lint |
mesh-host |
one apply queue; the report sequence kept on disk; epoch refusal reported; the report verb; the contract tests and lint |
| mesh-tools | refusing an unreadable or unknown input by name; the lint |
| mesh-catalog, mesh-sdk | retirement in the providers' loop (ADR 0230, replacing ADR 0229's brake): a consumer no longer asked for is retired — disabled, marked, its data kept; or, by a provider that cannot disable, only marked, its access kept — only once the same result holds for five passes and ten minutes; remove is never called to retire; a set of more than three, or more than half of those held, waits for a person and raises a condition; the four retirement tools on every provider |
| mesh-lab | R1, R2, R6 |
Done when: R1 ends with nothing from the stale epoch applied, R2 with one report of the newest sequence, R6 with nothing retired; every consumed subject has its contract test.
Built 2026-10-06 (ADR 0229):
the controller's lease and epoch, plans by compare-and-set, accounts kept by order, S12, S13 by writer,
D5, the writers table enforced at composition, a contract for every consumed kind, the empty-on-error
lint; the node-engine's one apply queue, report order and epoch refusal; the withdrawal brake in the Go
providers' loop and the SDK's. Not yet: the replays R1, R2 and R6 in mesh-lab; mesh-tools' refusals
and lint; the lint in mesh-host; the SDK's loop announcing a provider's standing, without which its
brake is said in its journal only.
Amended 2026-10-06 (ADR 0230,
the operator's decision): the brake's hourly release is replaced by retirement — three states (active,
retired, deleted), five stable passes and ten minutes before a consumer is retired, a provider that
cannot disable marking only (every TypeScript provider today), a person for more than three or more
than half, retire approve|reject, cleanup list|delete, retirement events and D11. Built on branches
in mesh-catalog (both Go providers), mesh-sdk (0.1.12) and mesh-controller, not yet merged. Not yet:
the TypeScript providers giving their adapters a retire that disables (until then each is mark-only:
its retired consumers keep their access), passing their module name and an announcer to the loop,
without which a set over the bound there waits with no way to approve it, and adapters that list and
delete what they retired.
Phase 3 — Healers
| Repository | Delivers |
|---|---|
| mesh-controller | the healer registry and H1–H4, each braked and said; S15 |
| mesh-lab | an induced failure per healer |
Done when: a lab mesh recovers from each induced failure with no hand, says so, and brakes after its budget. Live: a week with no cause repeated in the hand-act log.
Built 2026-10-06 (ADR 0231): the healer registry
and H1–H4 in the controller, H5 registered as the provider's own; the mesh-wide brake; healer-acted;
healers; S15 live; the node-engine's report verb. Each healer's induced failure runs in the
controller's tests against a real store, and H3 and H4 against a real bus: induced, repaired, cleared by
the probe's own next run, and handed to the operator after the budget. Not yet: the induced failures
on a lab mesh (mesh-lab), and so the phase's done when; the live week.
Amended 2026-10-06 (ADR 0233,
the operator's decision, before Phase 4): a module declares the data it holds; D13 and its conditions;
the data verb; cleanup extended to a module's own retired data, deleted by the machine's backup
holder after a last restore point; the controller's grant names node-backup.backed-up. Built on
branches in mesh-controller (migration 0072), mesh-host (the installer's user list; the kept-directory
report), mesh-catalog (the holder's measuring and deletion, every module's data, the providers' held
sizes), the media catalogue and the photo application, not yet merged. Not yet: the TypeScript
providers saying their consumers' sizes (until then a consumer at two of them is data-held-twice, a
warning, not compared), and SMART read under an array.
Phase 4 — Core upgrades that roll back
| Repository | Delivers |
|---|---|
| mesh-controller | the health probes of §8; the gate on a core component's first machine — as built, on every plan's first machine (ADR 0236); the upgrade record in the plan; the bus maintenance step as a verb; the default upgrade policy roll (ADR 0236) |
mesh-host |
keeping the previous controller and tool runner builds; restoring one when its health is not met in bound, watching the lease bucket for the controller |
| mesh-tools | answering the health ping |
| mesh-lab | R3, R7, R8 |
Done when: on a lab mesh, a broken build of the controller, the node-engine and the tool runner — one that starts and does nothing, one that crashes, one that cannot reach the bus — is each rolled back with no hand, the mesh ends on the previous build, and a condition and a message say so. Live: the next three core builds to roll out each record a health verdict.
Built 2026-10-06 (ADR 0236,
the operator's "start phase 4"): in mesh-controller (migration 0073), the probes H-controller, H-engine,
H-tools, H-bus, DG and DB; the gate on every plan's first machine, its record in the plan (plans <id>)
and in the store; the rollback by the ordinary path, once per build; the witnesses' verdicts as
conditions; the grants the witness reads; bus and bus upgrade; the default policy roll, upgrade
listing every module's policy and where it comes from; the user list carried after a controller passes;
a module deleted at its source forgotten, not built. In mesh-host, the witness (keeping the previous
controller and tool runner, restoring one not healthy in bound, the launcher's for the node-engine) and the
installer's grants. In mesh-catalog, the modules that keep record say why, and the forge's announcer
says which files a merge deleted. On branches, not yet merged. Not yet: R3, R7, R8 on a lab mesh, and
so the done when; container state in the node-engine's report, without which a container that
crash-loops after its compose applied is seen only through what it breaks; (designed 2026-10-07 by
ADR 0240 as
to-be 48, whose phases track it) composing not-reversible for
a build whose migration cannot be undone; the verdicts of the next three core builds to roll out.
Phase 5 — Checks before merge, and the replays
| Repository | Delivers |
|---|---|
| mesh-controller | the facts snapshot and S14 |
mesh-controller, mesh-host, mesh-catalog |
the compose-and-validate merge gate; versions tested as run; the resolver's tests under both C libraries |
| mesh-lab | R4, R5 and the rest of the window's incidents; running the replays on every core merge |
Done when: the replays of 236, 262, 263 and 266 fail on the commit before their fix and pass after; a new core issue cannot resolve without a replay or a stated reason.
Built 2026-10-06 (ADR 0237,
the operator's "start Phase 5"), on branches, not yet merged: in mesh-controller the facts snapshot and
S14, merge-gate, the check asked of and run by the build seat, merge-check.sh, the replays of 263 and 273,
and a bus per test at the mesh's release; in mesh-catalog the forge's announcer of pull requests' heads and
the verdict as their status, and merge-check.sh; in mesh-host merge-check.sh and the grants in the
installer's user list; in mesh-lab the replays of 262 and 266, the register and the prover; in hq the
replay rule in cycle.py. The prover ran the replays of 236, 262, 263, 266 and 273: each fails on the
commit before its fix (236's check did not exist there) and passes on it. Not yet: R4, R5 and the
rest of the window's incidents as replays; the merge gate's first runs against the live mesh's snapshot;
the status required to merge, which is the operator's setting on the forge; mesh-tools' and mesh-sdk's
merge-check.sh.
Revised 2026-10-06 (ADR 0238, the operator's "enable the checks on every repository of the mesh", "the logic acts on our module graph" and "the commit is the build at hand"), on branches, not yet merged:
| Repository | Delivers |
|---|---|
| mesh-controller | the planner's one mapping of a changed file onto modules and its one answer of a merge's reach, asked by the merge handler, the what-if, the gate and the check; a file no build reads touches nothing (issue 280's left open); every pull request answered — the gate when it reaches a module, the repository's own check for the mesh's repositories, a pass that says so otherwise; the gate run by the build seat with its judge chosen from the graph, composing the plan's definitions only, failing only a manifest problem the change brings; the delivery plan computed and carried with the verdict; a build of a commit off its module's trunk recorded and never registered; its own manifest naming every verb of its seat |
| mesh-catalog | the forge's announcer saying, with every pull request, the directories holding a module at its head, the files it deletes and whether it has a merge-check.sh; both statuses and the delivery plan on the pull request; gitea_branch_protection_get and gitea_branch_protection_set; the catalogue's own check |
mesh-host, mesh-tools, mesh-sdk, mesh-lab, mesh-media-catalog, and the applications built from their own repositories |
each its own merge-check.sh; mesh-tools' tests each on a bus of their own |
| mesh-tools-go, mesh-tools | a C compiler and Python in the Go toolchain, git in the TypeScript one |
| hq | its own merge-check.sh, running records.py, index.py and cycle.py |
Not yet: the delivery plan kept (id, inputs, result) and the release comparing its plan with it; the
state machine replacing the walks' states, with its table, its test, its conditions and healer H2;
the status's link to the kept plan; the notes under refs/notes/mesh-plan; check builds in a scratch
namespace (the check builds no module today, so there is none to keep apart yet); the statuses made
required — prepared as calls of the forge module's tool, applied by the operator.
Given an owner 2026-10-06 (ADR 0239): the state machine, the
notes and the status's link are built as a delivery's, owned by the module mesh-delivery. They are designed
in to-be 47, and their progress is tracked there.