novox/hq
6
0
Fork 0
Code Issues Pull Requests 4 Releases Wiki Activity

ADR 0229: the core's order is a lease the store remembers, and an epoch a machine is sent once it reads one

Browse Source 
Phase 2 of to-be 45 met questions its paragraphs do not answer: a strict
node-engine refuses an unknown key, a lease bucket can be raised again from
nothing, a command at a shell sends declarations too, and the bus's grant
is composed by the controller that needs it. The answers, the wire contract
and the withdrawal brake go into to-be 45 with issue 270's Phase 1
decisions; issue 272 records that the SDK's provider loop says nothing on
the bus.
This commit is contained in:
jochen
2026-10-06 12:26:33 +02:00
parent c99eb82e0e
commit c6e7f2c911
5 changed files with 298 additions and 17 deletions
Download Patch File Download Diff File Expand all files Collapse all files
02-DECISIONS/0229-the-cores-order-is-a-lease-the-store-remembers-and-an-epoch-a-machine-is-sent-once-it-reads-one.md
+170
View File
@@ -0,0 +1,170 @@
---
topic: the mesh
status: accepted
date: 2026-10-06
deciders: jochen
reconstructed: false
extends: 02-DECISIONS/0227-the-core-holds-nine-rules-each-checked-and-is-built-to-them-in-six-phases.md
---
# 229. The core's order is a lease the store remembers, and an epoch a machine is sent once it reads one
## Context
Phase 2 of [to-be 45](../03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md) — *order and one
writer* — was built on 2026-10-06 in the controller, the node-engine, the providers' loop and the
SDK's. The design states the lease, the epoch, the report's order and the brake in a paragraph each;
building them met questions the paragraphs do not answer, and each was answered in code. This record
is those answers, so the design can say them and the next build does not answer them again.
Four of them could not be left to taste:
- **The node-engine decodes a declaration strictly** and refuses a key it does not know, whole. An
epoch sent to every machine from the first controller that has one is refused by every machine
not yet updated — and a refused declaration is also the one that would have updated its
node-engine, so a machine asleep through the rollout could never catch up.
- **The epoch is a bucket revision, and a bucket can be raised again from nothing.** A bus whose data
directory was replaced starts its revisions at one, and every machine that heard epoch 57 would
refuse the next controller for ever.
- **A command at a shell sends declarations too** — the installer's first pushes, a lab step, a
person repairing a mesh whose controller is down ([issue 201](../04-ISSUES/201-a-push-recreated-the-controller-behind-the-row-its-successor-wrote/00-report.md)).
"Only the holder acts" read strictly would stop the installer at its first push.
- **A controller's grant on the bus is composed by the controller, and the live bus holds the list the
previous build composed.** A controller that needs a grant for its lease's bucket before it may act
cannot send the list that grants it.
## Considered Options
1. **Send the epoch to every machine and roll the node-engine out first**, as ADR 0227's consequences
suggest. Rejected: it relies on every machine being up for the node-engine's rollout, and strands
one that is not.
2. **The epoch outside the signed bytes**, where a strict decoder does not look. Rejected: a broker
could then give an old declaration a new epoch, and the order of declarations is the one property
their signature does not already protect.
3. **The epoch inside the signed bytes, sent to a machine only once its node-engine has said it reads
one.** Chosen.
4. **A one-off command takes no part in the lease**, and is refused while a controller serves.
Rejected for the installer above.
5. **A controller that cannot take the lease refuses to act.** Rejected for the grant above: it could
never send the user list that grants it.
## Decision
**The lease.** The controller's lease is the key `holder` in the bucket `mesh-controller_lease`, whose
age is fifteen seconds; the holder renews it every five by compare-and-set at the revision it last
wrote. Its value names the instance (machine, process, start), its build, its epoch, and when it was
taken and renewed. **A holder stops acting three seconds before its key could expire unrenewed**, by
its own clock, whatever its renewing goroutine is doing; a renewal refused or failed is the lease lost,
said, the instance's epoch recorded as lost, and the process exits to be started again as a candidate.
A holder that stops gives the key back, so the next takes it at once. The serving controller takes the
lease before it asserts the bus's objects, and a candidate waits, said once per holder.
**The epoch never goes backwards.** Every epoch issued is kept in the controller's store with its
instance, when it was taken and how it ended: given back (`released`), lost by its own renewal
(`lost`), or found gone by the next holder (`expired`). The highest is a floor: a lease bucket whose
revisions are at or under it was raised again from nothing, and its stream is compacted past the floor
before the key is taken — said as a condition, S12.
**A controller the bus refuses the lease, with nobody holding it, serves without one**, as every
controller did before: its declarations carry no epoch, which no node-engine refuses; it says so as
the urgent condition S12 names, and tries again every five seconds. The first push of the machine
holding the bus sends the user list that grants it. One that finds another took the lease meanwhile
stops.
**A command run at a shell** acts under the holder's epoch, read at the moment it acts, while a
controller holds the lease — it is the same mesh's word, composed under the same hold of each machine —
and under a lease of its own, given back as it ends, while none does. A process with no bus configured
claims no epoch.
**What carries the order on the wire** — the contract, written once on each side (the controller's
`internal/link/order.go`, the node-engine's `internal/link/messages.go`):
| Where | Key | Holds |
|---|---|---|
| a declaration, inside the signed envelope | `epoch` | the lease epoch it was composed under; absent claims none |
| | `sequence` | its number for that machine (issue 107); absent claims none |
| a report | `epoch`, `sequence` | the order of the declaration the report is about |
| | `report_sequence` | the node-engine's own number for the report, kept on disk, growing across restarts and self-updates; **its presence says the node-engine reads `epoch`** |
| | `older_than` | on a refusal of a declaration older than one applied: the order of the one held |
| | `refused_older` | how many declarations the node-engine has refused as older, ever, on every report |
- **A machine is sent `epoch` only while its latest account carried a `report_sequence`.** One that
stops carrying it — a node-engine rolled back — is sent none again.
- **A node-engine refuses by epoch only when both declarations claim one**: an older epoch, or the
same epoch and a lower sequence. A declaration with no epoch is taken by its sequence, so a
controller rolled back to a build without the lease is never stranded; that gives up the epoch's
protection for as long as such a build runs.
- **The controller keeps the account of each machine by its order**: by epoch where both claim one,
then by sequence, then by report sequence, and refuses an older account — said, counted, and the
machine's facts in it kept as before. An account without a report sequence, from an older
node-engine, is judged by the digest it names (issue 267), and clears the order kept.
- **What the mesh would send a machine is composed with the epoch it was last sent**, as with its
sequence: a new holder of the lease is not a change of the machine.
**Stale refusals name their writer** (S13): a declaration refused as older is counted against the
controller epoch it claimed, named from the store's record of epochs with how that epoch ended; an
account the controller refused is counted against the machine's node-engine; and what a machine's
`refused_older` rose by beyond the refusals heard is counted as refusals whose reports were lost.
**One writer is enforced where grants are composed.** The writers table is compiled into the
controller with, for each state the bus carries, the subjects a write of it publishes to and who its
writer is; a principal whose grant overlaps another writer's subject is refused at composition, naming
the state and its writer. The controller's own grant loses `mesh.control.>`, which it never published
and which made it a second writer of every machine's report. The stream-definition row names stream
creation, update and deletion and durable consumer creation — not every consumer creation, because a
module watching its own bucket makes an ordered consumer that defines nothing the mesh keeps.
**A plan is written by compare-and-set** on a revision the store keeps beside it, and carries the epoch
that wrote it; a write against a plan moved since is refused, and nothing is written by a process that
may not act.
**The empty-on-error lint** is a test over the repository's own source: a `return` inside an
`if err != nil` branch that answers an empty collection and no error fails it, unless a comment
`empty-on-error: <why>` on the line or above says why empty is the truth there.
**Every consumed message kind has a contract**: how an older one is refused, with the tests that deliver
the newer and then the older, or why none is needed. A check fails a kind the controller can be handed
without one, and one naming a test that does not exist.
**The withdrawal brake** (the providers' loop, Go and the SDK's alike): a pass that would withdraw more
than one consumer, or more than half of those held where it holds more than one, withdraws nothing;
each consumer it keeps is announced `provisioner.failing` with the class `withdrawal-braked`, so the
controller raises it as a condition (ADR 0224); and while the mesh goes on not asking for them, one is
let go every hour, said. A consumer asked for again is kept. **The brake stops the bulk, not the
intention**: an unassignment of many completes without a hand, a mistaken one costs at most one consumer
an hour while the operator is told — and withdrawal destroys no data since issue 241.
**What Phase 1 decided while building** — [issue 270](../04-ISSUES/270-phase-1-watches-signals-that-come-later-and-hears-only-the-controllers-faults/00-report.md)'s
decisions 1 to 6: the condition history as a bucket of its own, the events' shape, a key's last token,
the probe DW, which deleted consumers are said, and the bounds the design left to the build — is taken
into to-be 45 as built, and S12 and D5 move to Phase 2, S14 to Phase 5, as that issue asked.
## Consequences
- **The rollout is the node-engine first, then the controller**, as ADR 0227 says — but the order no
longer has to be exact. A machine on an older node-engine is sent no epoch; one that updates later
is sent it from its first ordered report.
- **The first controller with a lease on the live mesh serves without it** until the user list that
grants its bucket reaches the bus, and S12 says so, urgent. A push of the machine holding the bus
ends it.
- **Two controllers at once are now the lease's to settle**, not the consumers' binding (issue 213's
standing by), which stays as a second guard.
- **The TypeScript providers' brake is said in their journal only**: the SDK's loop does not announce a
provider's standing at all (ADR 0224 is in the Go loop), so a braked withdrawal there is not a
condition until it does.
- **The lint runs in the controller's repository.** The node-engine's and the node tools' carry their
own copy when they take it up; until then they are not covered.
## How it is checked
| What | Checked by |
|---|---|
| the lease: one holder, a waiting second, a handover at a higher epoch, a loss acting on nothing | `internal/lease` tests and the controller's `TestTwoControllersOneActs`, against a real bus and store |
| an epoch never issued twice | `TestAnEpochIsNeverIssuedTwiceWhenTheBucketStartsOver`; live, D5 and S12 |
| the contract | `internal/link/order_test.go`, beside the node-engine's own tests of the same cases |
| an account kept by its order | the report's contract tests (`heard_order_test.go`); live, S13 |
| one writer at composition | `internal/broker/writers_test.go`: the table is the design's, a whole mesh composes, a second writer is refused |
| a plan by compare-and-set | `TestAPlanIsWrittenByCompareAndSetUnderTheLease` |
| empty on error | `internal/lint`, over the whole repository on every test run |
| every consumed kind | `TestEveryConsumedKindHasAContract` |
| the brake | the providers' `brake_test.go` (issue 241 replayed with seven consumers) and the SDK's test |
02-DECISIONS/README.md
+1
View File
@@ -200,6 +200,7 @@ python3 00-META/checks/index.py fail if stale
- **0222** — [A module is told where a mesh seat's holder is reached, and the controller writes no file a seat's holder owns](0222-a-module-is-told-where-a-mesh-seats-holder-is-reached-and-the-controller-writes-no-file-a-seats-holder-owns.md)
- **0224** — [A provider that keeps failing a consumer is a problem the controller reports](0224-a-provider-that-keeps-failing-a-consumer-is-a-problem-the-controller-reports.md)
- **0227** — [The core holds nine rules, each checked, and is built to them in six phases](0227-the-core-holds-nine-rules-each-checked-and-is-built-to-them-in-six-phases.md)
- **0229** — [The core's order is a lease the store remembers, and an epoch a machine is sent once it reads one](0229-the-cores-order-is-a-lease-the-store-remembers-and-an-epoch-a-machine-is-sent-once-it-reads-one.md)
### Its tiers, from the bottom up
03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md
+70 -16
View File
@@ -5,6 +5,7 @@ code: [mesh-controller, mesh-host, mesh-tools, mesh-catalog, mesh-lab]
updated: 2026-10-06
decisions:
- 02-DECISIONS/0227-the-core-holds-nine-rules-each-checked-and-is-built-to-them-in-six-phases.md
- 02-DECISIONS/0229-the-cores-order-is-a-lease-the-store-remembers-and-an-epoch-a-machine-is-sent-once-it-reads-one.md
- 02-DECISIONS/0224-a-provider-that-keeps-failing-a-consumer-is-a-problem-the-controller-reports.md
- 02-DECISIONS/0218-a-plan-sends-grants-before-code-rolls-out-one-machine-first-and-a-newer-merge-takes-over-an-older-plan.md
- 02-DECISIONS/0141-the-host-delivers-its-own-successor.md
@@ -58,7 +59,7 @@ Every kind of state the core keeps, and the one component that writes it. Anyone
| a machine's applied state and its report | the node-engine's apply queue (§6) | the machine; the report on the bus | the reconcile and a delivery *enqueue*, never apply |
| the controller lease | the controller instance holding it | key-value `mesh-controller_lease` | a candidate waits |
| plans and their tiers | controller (lease holder), compare-and-set on the plan's revision | the controller's store | read through `plans` |
| conditions | controller | key-value `mesh-controller_conditions` | raise or clear only through observations the controller reads |
| conditions | controller | key-value `mesh-controller_conditions`, and their history `mesh-controller_condition-history` | raise or clear only through observations the controller reads |
| calls and their outcomes | controller | key-value `mesh-controller_calls` | read by id |
| the hand-act log | controller, through the verbs that act | key-value `mesh-controller_hand-acts` | — |
| stream definitions and bus permissions | controller | the bus | — |
@@ -71,20 +72,32 @@ Every kind of state the core keeps, and the one component that writes it. Anyone
A bus subject two components may publish on is refused when the controller composes grants, unless
this table marks it shared. Changing a writer is a change to this table, through a decision.
**How it is enforced** ([ADR 0229](../../02-DECISIONS/0229-the-cores-order-is-a-lease-the-store-remembers-and-an-epoch-a-machine-is-sent-once-it-reads-one.md)):
the table is compiled into the controller, each row the bus carries with the subjects a write of it
publishes to — a declaration `mesh.node.*.declare`, a report `mesh.control.*.report` (its own machine's
only), the controller's buckets, stream creation, update and deletion and durable consumer creation, a
build's outcome, a merge announced, a provider's standing — and its writer among the principals. A
grant that overlaps another writer's subject is refused at composition, naming the state and its
writer. A test holds the compiled table to this one, row for row. The controller does not publish
`mesh.control.>`: it would be a second writer of every machine's report.
## 2. The condition store (rules 5, 6)
**A condition is a durable fact about something the mesh owns that is wrong.** It is raised and cleared
by observation only.
**Where.** A key-value bucket, `mesh-controller_conditions`, written by the controller alone, one entry
per open condition. Every transition — raised, changed, silenced, cleared — is also appended to a
history kept ninety days, read through `conditions history`.
per open condition, with no age. Every transition — raised, changed, silenced, cleared — is also
appended to a history kept ninety days in a bucket of its own, `mesh-controller_condition-history`,
read through `conditions history`: a bucket has one age for every key, and an open condition must
outlive the history.
**The key** names the thing and the kind, so the same fault said again is the same condition:
`<scope>.<id>.<kind>`, where scope is one of `machine`, `plan`, `call`, `build`, `merge`, `provider`,
`seat`, `bus`, `core`, `probe`, `mesh`. Examples of the shape: `machine.<node>.silent`,
`plan.<id>.stalled`, `provider.<module>.<node>.<consumer>.failing`, `core.controller.<node>.rolled-back`,
`bus.<consumer>.slow-consumer`.
`bus.<consumer>.slow-consumer`. The last token is the kind's short word where it has one (`failing` for
`provider-failing`), the kind elsewhere; a key is opaque to every reader, and the kind is the field.
**The fields:**
@@ -122,7 +135,10 @@ history kept ninety days, read through `conditions history`.
- **The events**, emitted by the controller on its own subjects: `condition-raised`,
`condition-changed` (severity or resolver changed; not every observation), `condition-cleared`. The
operator-channel's holder and any other surface consume them; the controller learns nothing about
telling.
telling. Each carries the condition at the top level, with `event`, `at`, `change` (raised,
reopened, severity, resolver, silenced, silence-ended, cleared), `why`, `was`, `cleared` and `show`
beside it; a silence and its ending are `condition-changed`, a reopening `condition-raised` with
change `reopened`.
- **`status`** lists open conditions first, urgent before warning, oldest first, silenced ones marked
with their expiry. The all-well sentence requires none open, silenced included.
- **ADR 0224's provider standing** is the first kind: `provider-failing`, raised by the provider's
@@ -140,20 +156,24 @@ signal and asserts its condition. `doctor signals` shows, for every row, the age
| S3 | plan tier progress | controller's plan | each tier entered | from the tier's build and apply durations, *provisional* | `stalled` (names the tier and what it waits on) | warning | H2 |
| S4 | the controller's event loop takes a message | controller | while its consumer has pending messages | 2 min | `controller-deaf` | urgent | — |
| S5 | a merge announced becomes a plan or *nothing reads it* | announcer → controller | each merge | 10 min (exists, issue 266) | `merge-not-acted` | urgent | — |
| S6 | a build asked → its outcome | build seat | each ask | the build's declared timeout | `ask-lost` | warning | — |
| S7 | a call running → finished | controller | each call | the verb's declared bound | `call-hung` | warning | — |
| S6 | a build asked → its outcome | build seat | each ask | max(20 min, 3 × the p90 of measured builds), 1 h while nothing is measured, *provisional* (the build seat declares no timeout) | `ask-lost` | warning | — |
| S7 | a call running → finished | controller | each call | the verb's bound: push, rotate and command 30 min; assign and unassign 15 min; doctor 3 min; others 10 min | `call-hung` | warning | — |
| S8 | a provider's failing word repeated | provider | every 15 min while failing (ADR 0224) | 30 min | `provider-silent` | warning | — |
| S9 | bus advisories: slow consumer, maximum deliveries, permission violation, consumer deleted | bus server's system subjects | any | any occurrence | `slow-consumer`, `max-deliveries`, `refused`, `consumer-lost` — each naming the call, consumer or module in the mesh's words | warning | H4 for `slow-consumer` on a resettable consumer |
| S10 | the self-check's heartbeat | controller's `doctor` | every run | 2 × its interval, watched **from the second machine** (§5) | `self-check-silent` | urgent | — |
| S11 | node tools heartbeat | node tools | its interval | 3 × interval, *provisional* | `tools-silent` | warning | — |
| S12 | the controller lease renewed | controller | every 5 s | 15 s | `lease-lost` | urgent | — |
| S13 | stale refusals | every receiver (rule 2) | each refusal | more than 5 from one writer in 5 min | `stale-writer` (names the writer) | warning | — |
| S14 | facts snapshot exported | controller | daily | 2 days | `facts-stale` | warning | — |
| S12 | the controller lease renewed | controller | every 5 s | 15 s; a holder that lost the lease or was found expired, and a lease bucket found raised again from nothing, said for an hour; a controller serving without the lease, while it does | `lease-lost` | urgent | — |
| S13 | stale refusals | every receiver (rule 2) | each refusal | more than 5 from one writer in 5 min | `stale-writer` (names the writer: the controller epoch a refused declaration claimed, with its instance and how its lease ended; or the machine whose older accounts the controller refused) | warning | — |
| S14 | facts snapshot exported | controller | daily | 2 days (Phase 5) | `facts-stale` | warning | — |
| S15 | a hand act with a cause already recorded | hand-act log | each act | the second within 14 days | `healer-wanted` | warning | — |
The bus advisories (S9) cost one read-only subscription: the server already publishes them. The
controller translates each into a condition naming the thing in the mesh's words, as the refused reply
of issue 265 is translated today.
of issue 265 is translated today. An advisory clears after an hour without another; a deleted consumer
is said only for one the mesh names and expects, and clears when it exists again. Until the bus has a
system account the controller hears a slow consumer and a refused subject for its own connection only
([issue 270](../../04-ISSUES/270-phase-1-watches-signals-that-come-later-and-hears-only-the-controllers-faults/00-report.md),
open question 2).
## 4. The self-check: `doctor` (rule 6)
@@ -168,12 +188,13 @@ itself — an unanswered probe is never a pass.
| D2 | every holder of the mesh's resolver answers a machine name for IPv4, and NODATA for IPv6 | 262 |
| D3 | every seat on record has a live holder that answers | 208, 218 |
| D4 | every kept archive is held by a manifest | 253 |
| D5 | exactly one lease holder; no message from a stale epoch in the last interval | 204 |
| D6 | every durable consumer exists with its definition and is near its stream's head | 248, 266 |
| D5 | exactly one lease holder — the key names this controller at the epoch it acts under, and the controller's record holds no other epoch open; no message from a stale epoch refused in the last interval | 204 |
| D6 | every durable consumer exists with its definition and is near its stream's head (within 1000 messages, *provisional*) | 248, 266 |
| D7 | every stream the controller defines exists with its definition | 208 |
| D8 | no address the mesh owns is in a ban list | 238 |
| D9 | `status` answers in full within ten seconds | 265 |
| D10 | every machine runs the node-engine and node tools builds its plan says, or is inside a plan's window | version split |
| DW | the watchdogs of §3 ran within three of their intervals: the watchers are watched, and raise S10 when the self-check stops | rule 6 |
| H-* | the health probes of §8, run for every core component on every machine | rule 8 |
- **`doctor`** answers the last run's verdict at once: per probe, pass, fail or failed-to-run, and age.
@@ -220,6 +241,20 @@ fifteen-second time to live, renewed every five seconds. The **epoch** is the bu
it was taken. A starting controller waits for the key to be absent or expired. One that fails a renewal
stops acting at once and exits, so its service manager restarts it as a waiting candidate.
As built ([ADR 0229](../../02-DECISIONS/0229-the-cores-order-is-a-lease-the-store-remembers-and-an-epoch-a-machine-is-sent-once-it-reads-one.md)):
- **The gate is the clock.** A holder stops acting three seconds before its key could expire
unrenewed, whatever its renewing goroutine is doing; every act passes that gate at the moment it is
made — a declaration at its send as well as where it was composed.
- **The epoch never goes backwards.** The controller's store keeps every epoch issued, its instance,
and how it ended (`released`, `lost`, `expired`); a lease bucket at or under the highest is moved past
it before the key is taken, and S12 says so.
- **A holder that stops gives the key back**, so the next takes it at once.
- **Without the bucket's grant, nobody holding it,** a controller serves unleased — no epoch on what it
sends, S12 urgent — and takes the lease once the user list that grants it reaches the bus.
- **A command at a shell** acts under the holder's epoch, read as it acts, or under a lease of its own
while nobody holds one.
```
controller A (epoch 41) ──renew──renew──╳ (renewal refused)──► stops sending, exits
controller B ──wait──────────────take (epoch 57)──► acts; marks A's running calls abandoned
@@ -230,8 +265,8 @@ stops acting at once and exits, so its service manager restarts it as a waiting
| Message | Carries | The receiver keeps, per writer | Refuses |
|---|---|---|---|
| declaration | epoch, sequence | highest epoch, then highest sequence | older epoch; same epoch, lower sequence |
| report | machine, the declaration's epoch and sequence it is about, the node-engine's own report sequence (kept on disk, increasing across restarts and self-updates) | highest declaration sequence, then report sequence | an account of an older declaration; an older report |
| declaration | epoch, sequence | highest epoch, then highest sequence | older epoch; same epoch, lower sequence — only where both claim an epoch |
| report | machine, the declaration's epoch and sequence it is about, the node-engine's own report sequence (kept on disk, increasing across restarts and self-updates) | highest epoch where both claim one, then declaration sequence, then report sequence | an account of an older declaration; an older report |
| plan write | the plan's revision, the epoch | — (compare-and-set) | a write against a revision already moved |
| build outcome | the build's ask id and order | newest per module | an older build finishing later (exists, 219) |
| call | call id, epoch | — | a finish from an epoch that is not the holder's, recorded as abandoned |
@@ -241,6 +276,17 @@ stops acting at once and exits, so its service manager restarts it as a waiting
node-engine — a report naming the refused declaration, so the controller sees it. The counter feeds
S13.
**The wire** — the contract, written once on each side: a declaration's `epoch` and `sequence` are
top-level keys of its signed envelope; a report carries back the `epoch` and `sequence` of the
declaration it is about, its own `report_sequence`, `older_than` (the order held) on a refusal of an
older declaration, and `refused_older` (how many ever) on every report. Every key is optional, zero is
"no order claimed". **A machine is sent `epoch` only while its latest account carried a
`report_sequence`**, because an older node-engine refuses an unknown key, whole; a declaration without an
epoch is taken by its sequence, so a controller rolled back to a build without the lease is never
stranded. A report without a `report_sequence` is judged by the digest it names (issue 267). What the
mesh would send a machine is composed with the epoch it was last sent: a new holder is not a change of
the machine.
**One apply queue on every machine.** The node-engine has one worker that applies; a delivery, the
five-minute reconcile, a self-update hand-over and a manual `apply` are four reasons to **enqueue**.
The queue holds at most one pending request, coalesced. When the worker starts it takes the newest
@@ -369,7 +415,7 @@ has a week of entries; the durations are recorded for every machine.
| Repository | Delivers |
|---|---|
| mesh-controller | the condition store, its verbs, history and events (§2); ADR 0224's standing moved into it; watchdogs for S1–S10 and S12–S14 with bounds set from Phase 0's durations; the bus advisories subscribed and translated (S9); `doctor` with D1–D10 and its heartbeat (§4); `status` led by open conditions; the test generated from the signals table |
| mesh-controller | the condition store, its verbs, history and events (§2); ADR 0224's standing moved into it; watchdogs for S1–S11 and S13 with bounds set from Phase 0's durations (S12 is Phase 2's, S14 Phase 5's); the bus advisories subscribed and translated (S9); `doctor` with D1–D4, D6–D10, DW and its heartbeat (§4; D5 is Phase 2's); `status` led by open conditions; the test generated from the signals table |
| mesh-host | the heartbeat carries its interval; D1's validator published as a package the controller imports |
| mesh-tools | the node tools' heartbeat (S11) |
| mesh-catalog | the `operator-channel` seat and its holder; the Telegram channel and the desktop notifier contributing to it; `mesh-watcher` on a machine other than the control node (§5) |
@@ -393,6 +439,14 @@ bound corrected in the table.
**Done when:** R1 ends with nothing from the stale epoch applied, R2 with one report of the newest
sequence, R6 with the withdrawal braked; every consumed subject has its contract test.
**Built 2026-10-06** ([ADR 0229](../../02-DECISIONS/0229-the-cores-order-is-a-lease-the-store-remembers-and-an-epoch-a-machine-is-sent-once-it-reads-one.md)):
the controller's lease and epoch, plans by compare-and-set, accounts kept by order, S12, S13 by writer,
D5, the writers table enforced at composition, a contract for every consumed kind, the empty-on-error
lint; the node-engine's one apply queue, report order and epoch refusal; the withdrawal brake in the Go
providers' loop and the SDK's. **Not yet:** the replays R1, R2 and R6 in mesh-lab; mesh-tools' refusals
and lint; the lint in mesh-host; the SDK's loop announcing a provider's standing, without which its
brake is said in its journal only.
### Phase 3 — Healers
| Repository | Delivers |
04-ISSUES/270-phase-1-watches-signals-that-come-later-and-hears-only-the-controllers-faults/00-report.md
+13 -1
View File
@@ -3,7 +3,7 @@ status: located
opened: 2026-10-06
located-in: [mesh-controller]
fixed-by:
amended-design:
amended-design: 03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md
---
# 270. Phase 1 watches signals that come later, and hears only the controller's own bus faults
@@ -67,3 +67,15 @@ but the design says Phase 1 delivers them, and a reader of the design believes i
2. Does the bus gain a system account — or the controller read the server's monitoring endpoint — so
S9 hears every principal's slow consumer and refused subject? Either changes the foundation's shape.
3. Do decisions 1–6 above go into §1–§4 as written, through an amendment of to-be 45?
## Phase 2, 2026-10-06
Open questions 1 and 3 are answered by [ADR 0229](../../02-DECISIONS/0229-the-cores-order-is-a-lease-the-store-remembers-and-an-epoch-a-machine-is-sent-once-it-reads-one.md)
and the amendment of to-be 45 it authorises: S12 and D5 are Phase 2's and S14 Phase 5's in the phase
table, and decisions 1–6 above are in §1–§4 as built. Phase 2 built S12 (the lease renewed, a holder that
lost it, a bucket raised again from nothing, a controller serving without it), D5, and S13 naming its
writer — the controller epoch a refused declaration claimed, with its instance and how its lease ended.
**Open question 2 stays open**: S9 still hears a slow consumer and a refused subject for the controller's
own connection only, until the bus has a system account or the controller reads the server's monitoring
endpoint.
04-ISSUES/272-the-sdks-provider-loop-says-nothing-on-the-bus/00-report.md
+44
View File
@@ -0,0 +1,44 @@
---
status: located
opened: 2026-10-06
located-in: [mesh-sdk src/provisioner]
fixed-by:
amended-design:
---
# 272. The SDK's provider loop says nothing on the bus
## Symptom
Found building Phase 2 of [to-be 45](../../03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md)
on 2026-10-06, adding the withdrawal brake to both copies of the providers' loop. The Go loop (carried by
the postgres and identity-provider modules) announces a consumer it keeps failing as
`provisioner.failing`, and its recovery as `provisioner.recovered`
([ADR 0224](../../02-DECISIONS/0224-a-provider-that-keeps-failing-a-consumer-is-a-problem-the-controller-reports.md));
the controller raises each as a condition. **The SDK's TypeScript loop announces neither.** Every provider
built on it — the object store, the SQL server, the document store, the mail server, the forge, the
analytics service — fails a consumer, and is braked from withdrawing several at once, in its journal
alone.
## Why it matters
ADR 0224 is written about every provider: *a provider that keeps failing a consumer is a problem the
controller reports*. ADR 0227 rule 4 has a reconcile that would withdraw more than its bound **raise a
condition**. For half the mesh's providers neither reaches the controller, so neither reaches `status` or
the operator — the shape of issue 179, where a provider failed every consumer for a day and said so only
in its journal.
## Located
`mesh-sdk src/provisioner/index.ts`: `runProvisioner` keeps no standing per consumer and has no way to
emit an event. The Go loop's `failed`, `succeeded`, `recovered` and `Announce` are the behaviour to
carry over (it says it follows the SDK's loop "line for line in behaviour"; since ADR 0224 it does more).
The SDK's stdio module can already emit a module's events, which is what the Go providers announce
through.
## Fix direction
Give `runProvisioner` the Go loop's standing — a run of failures announced after five minutes, again
every fifteen, recovered on the first success or when the consumer goes — and announce a braked
withdrawal through it with the class `withdrawal-braked`, as the Go loop does. Each TypeScript provider
then bumps its SDK. A test like the Go loop's `standing_test.go` and `brake_test.go`.