diff --git a/02-DECISIONS/0229-the-cores-order-is-a-lease-the-store-remembers-and-an-epoch-a-machine-is-sent-once-it-reads-one.md b/02-DECISIONS/0229-the-cores-order-is-a-lease-the-store-remembers-and-an-epoch-a-machine-is-sent-once-it-reads-one.md new file mode 100644 index 0000000..2a58d1b --- /dev/null +++ b/02-DECISIONS/0229-the-cores-order-is-a-lease-the-store-remembers-and-an-epoch-a-machine-is-sent-once-it-reads-one.md @@ -0,0 +1,170 @@ +--- +topic: the mesh +status: accepted +date: 2026-10-06 +deciders: jochen +reconstructed: false +extends: 02-DECISIONS/0227-the-core-holds-nine-rules-each-checked-and-is-built-to-them-in-six-phases.md +--- + +# 229. The core's order is a lease the store remembers, and an epoch a machine is sent once it reads one + +## Context + +Phase 2 of [to-be 45](../03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md) — *order and one +writer* — was built on 2026-10-06 in the controller, the node-engine, the providers' loop and the +SDK's. The design states the lease, the epoch, the report's order and the brake in a paragraph each; +building them met questions the paragraphs do not answer, and each was answered in code. This record +is those answers, so the design can say them and the next build does not answer them again. + +Four of them could not be left to taste: + +- **The node-engine decodes a declaration strictly** and refuses a key it does not know, whole. An + epoch sent to every machine from the first controller that has one is refused by every machine + not yet updated — and a refused declaration is also the one that would have updated its + node-engine, so a machine asleep through the rollout could never catch up. +- **The epoch is a bucket revision, and a bucket can be raised again from nothing.** A bus whose data + directory was replaced starts its revisions at one, and every machine that heard epoch 57 would + refuse the next controller for ever. +- **A command at a shell sends declarations too** — the installer's first pushes, a lab step, a + person repairing a mesh whose controller is down ([issue 201](../04-ISSUES/201-a-push-recreated-the-controller-behind-the-row-its-successor-wrote/00-report.md)). + "Only the holder acts" read strictly would stop the installer at its first push. +- **A controller's grant on the bus is composed by the controller, and the live bus holds the list the + previous build composed.** A controller that needs a grant for its lease's bucket before it may act + cannot send the list that grants it. + +## Considered Options + +1. **Send the epoch to every machine and roll the node-engine out first**, as ADR 0227's consequences + suggest. Rejected: it relies on every machine being up for the node-engine's rollout, and strands + one that is not. +2. **The epoch outside the signed bytes**, where a strict decoder does not look. Rejected: a broker + could then give an old declaration a new epoch, and the order of declarations is the one property + their signature does not already protect. +3. **The epoch inside the signed bytes, sent to a machine only once its node-engine has said it reads + one.** Chosen. +4. **A one-off command takes no part in the lease**, and is refused while a controller serves. + Rejected for the installer above. +5. **A controller that cannot take the lease refuses to act.** Rejected for the grant above: it could + never send the user list that grants it. + +## Decision + +**The lease.** The controller's lease is the key `holder` in the bucket `mesh-controller_lease`, whose +age is fifteen seconds; the holder renews it every five by compare-and-set at the revision it last +wrote. Its value names the instance (machine, process, start), its build, its epoch, and when it was +taken and renewed. **A holder stops acting three seconds before its key could expire unrenewed**, by +its own clock, whatever its renewing goroutine is doing; a renewal refused or failed is the lease lost, +said, the instance's epoch recorded as lost, and the process exits to be started again as a candidate. +A holder that stops gives the key back, so the next takes it at once. The serving controller takes the +lease before it asserts the bus's objects, and a candidate waits, said once per holder. + +**The epoch never goes backwards.** Every epoch issued is kept in the controller's store with its +instance, when it was taken and how it ended: given back (`released`), lost by its own renewal +(`lost`), or found gone by the next holder (`expired`). The highest is a floor: a lease bucket whose +revisions are at or under it was raised again from nothing, and its stream is compacted past the floor +before the key is taken — said as a condition, S12. + +**A controller the bus refuses the lease, with nobody holding it, serves without one**, as every +controller did before: its declarations carry no epoch, which no node-engine refuses; it says so as +the urgent condition S12 names, and tries again every five seconds. The first push of the machine +holding the bus sends the user list that grants it. One that finds another took the lease meanwhile +stops. + +**A command run at a shell** acts under the holder's epoch, read at the moment it acts, while a +controller holds the lease — it is the same mesh's word, composed under the same hold of each machine — +and under a lease of its own, given back as it ends, while none does. A process with no bus configured +claims no epoch. + +**What carries the order on the wire** — the contract, written once on each side (the controller's +`internal/link/order.go`, the node-engine's `internal/link/messages.go`): + +| Where | Key | Holds | +|---|---|---| +| a declaration, inside the signed envelope | `epoch` | the lease epoch it was composed under; absent claims none | +| | `sequence` | its number for that machine (issue 107); absent claims none | +| a report | `epoch`, `sequence` | the order of the declaration the report is about | +| | `report_sequence` | the node-engine's own number for the report, kept on disk, growing across restarts and self-updates; **its presence says the node-engine reads `epoch`** | +| | `older_than` | on a refusal of a declaration older than one applied: the order of the one held | +| | `refused_older` | how many declarations the node-engine has refused as older, ever, on every report | + +- **A machine is sent `epoch` only while its latest account carried a `report_sequence`.** One that + stops carrying it — a node-engine rolled back — is sent none again. +- **A node-engine refuses by epoch only when both declarations claim one**: an older epoch, or the + same epoch and a lower sequence. A declaration with no epoch is taken by its sequence, so a + controller rolled back to a build without the lease is never stranded; that gives up the epoch's + protection for as long as such a build runs. +- **The controller keeps the account of each machine by its order**: by epoch where both claim one, + then by sequence, then by report sequence, and refuses an older account — said, counted, and the + machine's facts in it kept as before. An account without a report sequence, from an older + node-engine, is judged by the digest it names (issue 267), and clears the order kept. +- **What the mesh would send a machine is composed with the epoch it was last sent**, as with its + sequence: a new holder of the lease is not a change of the machine. + +**Stale refusals name their writer** (S13): a declaration refused as older is counted against the +controller epoch it claimed, named from the store's record of epochs with how that epoch ended; an +account the controller refused is counted against the machine's node-engine; and what a machine's +`refused_older` rose by beyond the refusals heard is counted as refusals whose reports were lost. + +**One writer is enforced where grants are composed.** The writers table is compiled into the +controller with, for each state the bus carries, the subjects a write of it publishes to and who its +writer is; a principal whose grant overlaps another writer's subject is refused at composition, naming +the state and its writer. The controller's own grant loses `mesh.control.>`, which it never published +and which made it a second writer of every machine's report. The stream-definition row names stream +creation, update and deletion and durable consumer creation — not every consumer creation, because a +module watching its own bucket makes an ordered consumer that defines nothing the mesh keeps. + +**A plan is written by compare-and-set** on a revision the store keeps beside it, and carries the epoch +that wrote it; a write against a plan moved since is refused, and nothing is written by a process that +may not act. + +**The empty-on-error lint** is a test over the repository's own source: a `return` inside an +`if err != nil` branch that answers an empty collection and no error fails it, unless a comment +`empty-on-error: ` on the line or above says why empty is the truth there. + +**Every consumed message kind has a contract**: how an older one is refused, with the tests that deliver +the newer and then the older, or why none is needed. A check fails a kind the controller can be handed +without one, and one naming a test that does not exist. + +**The withdrawal brake** (the providers' loop, Go and the SDK's alike): a pass that would withdraw more +than one consumer, or more than half of those held where it holds more than one, withdraws nothing; +each consumer it keeps is announced `provisioner.failing` with the class `withdrawal-braked`, so the +controller raises it as a condition (ADR 0224); and while the mesh goes on not asking for them, one is +let go every hour, said. A consumer asked for again is kept. **The brake stops the bulk, not the +intention**: an unassignment of many completes without a hand, a mistaken one costs at most one consumer +an hour while the operator is told — and withdrawal destroys no data since issue 241. + +**What Phase 1 decided while building** — [issue 270](../04-ISSUES/270-phase-1-watches-signals-that-come-later-and-hears-only-the-controllers-faults/00-report.md)'s +decisions 1 to 6: the condition history as a bucket of its own, the events' shape, a key's last token, +the probe DW, which deleted consumers are said, and the bounds the design left to the build — is taken +into to-be 45 as built, and S12 and D5 move to Phase 2, S14 to Phase 5, as that issue asked. + +## Consequences + +- **The rollout is the node-engine first, then the controller**, as ADR 0227 says — but the order no + longer has to be exact. A machine on an older node-engine is sent no epoch; one that updates later + is sent it from its first ordered report. +- **The first controller with a lease on the live mesh serves without it** until the user list that + grants its bucket reaches the bus, and S12 says so, urgent. A push of the machine holding the bus + ends it. +- **Two controllers at once are now the lease's to settle**, not the consumers' binding (issue 213's + standing by), which stays as a second guard. +- **The TypeScript providers' brake is said in their journal only**: the SDK's loop does not announce a + provider's standing at all (ADR 0224 is in the Go loop), so a braked withdrawal there is not a + condition until it does. +- **The lint runs in the controller's repository.** The node-engine's and the node tools' carry their + own copy when they take it up; until then they are not covered. + +## How it is checked + +| What | Checked by | +|---|---| +| the lease: one holder, a waiting second, a handover at a higher epoch, a loss acting on nothing | `internal/lease` tests and the controller's `TestTwoControllersOneActs`, against a real bus and store | +| an epoch never issued twice | `TestAnEpochIsNeverIssuedTwiceWhenTheBucketStartsOver`; live, D5 and S12 | +| the contract | `internal/link/order_test.go`, beside the node-engine's own tests of the same cases | +| an account kept by its order | the report's contract tests (`heard_order_test.go`); live, S13 | +| one writer at composition | `internal/broker/writers_test.go`: the table is the design's, a whole mesh composes, a second writer is refused | +| a plan by compare-and-set | `TestAPlanIsWrittenByCompareAndSetUnderTheLease` | +| empty on error | `internal/lint`, over the whole repository on every test run | +| every consumed kind | `TestEveryConsumedKindHasAContract` | +| the brake | the providers' `brake_test.go` (issue 241 replayed with seven consumers) and the SDK's test | diff --git a/02-DECISIONS/README.md b/02-DECISIONS/README.md index 2b64cba..7bd46c4 100644 --- a/02-DECISIONS/README.md +++ b/02-DECISIONS/README.md @@ -200,6 +200,7 @@ python3 00-META/checks/index.py fail if stale - **0222** — [A module is told where a mesh seat's holder is reached, and the controller writes no file a seat's holder owns](0222-a-module-is-told-where-a-mesh-seats-holder-is-reached-and-the-controller-writes-no-file-a-seats-holder-owns.md) - **0224** — [A provider that keeps failing a consumer is a problem the controller reports](0224-a-provider-that-keeps-failing-a-consumer-is-a-problem-the-controller-reports.md) - **0227** — [The core holds nine rules, each checked, and is built to them in six phases](0227-the-core-holds-nine-rules-each-checked-and-is-built-to-them-in-six-phases.md) +- **0229** — [The core's order is a lease the store remembers, and an epoch a machine is sent once it reads one](0229-the-cores-order-is-a-lease-the-store-remembers-and-an-epoch-a-machine-is-sent-once-it-reads-one.md) ### Its tiers, from the bottom up diff --git a/03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md b/03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md index 1555321..5fce151 100644 --- a/03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md +++ b/03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md @@ -5,6 +5,7 @@ code: [mesh-controller, mesh-host, mesh-tools, mesh-catalog, mesh-lab] updated: 2026-10-06 decisions: - 02-DECISIONS/0227-the-core-holds-nine-rules-each-checked-and-is-built-to-them-in-six-phases.md + - 02-DECISIONS/0229-the-cores-order-is-a-lease-the-store-remembers-and-an-epoch-a-machine-is-sent-once-it-reads-one.md - 02-DECISIONS/0224-a-provider-that-keeps-failing-a-consumer-is-a-problem-the-controller-reports.md - 02-DECISIONS/0218-a-plan-sends-grants-before-code-rolls-out-one-machine-first-and-a-newer-merge-takes-over-an-older-plan.md - 02-DECISIONS/0141-the-host-delivers-its-own-successor.md @@ -58,7 +59,7 @@ Every kind of state the core keeps, and the one component that writes it. Anyone | a machine's applied state and its report | the node-engine's apply queue (§6) | the machine; the report on the bus | the reconcile and a delivery *enqueue*, never apply | | the controller lease | the controller instance holding it | key-value `mesh-controller_lease` | a candidate waits | | plans and their tiers | controller (lease holder), compare-and-set on the plan's revision | the controller's store | read through `plans` | -| conditions | controller | key-value `mesh-controller_conditions` | raise or clear only through observations the controller reads | +| conditions | controller | key-value `mesh-controller_conditions`, and their history `mesh-controller_condition-history` | raise or clear only through observations the controller reads | | calls and their outcomes | controller | key-value `mesh-controller_calls` | read by id | | the hand-act log | controller, through the verbs that act | key-value `mesh-controller_hand-acts` | — | | stream definitions and bus permissions | controller | the bus | — | @@ -71,20 +72,32 @@ Every kind of state the core keeps, and the one component that writes it. Anyone A bus subject two components may publish on is refused when the controller composes grants, unless this table marks it shared. Changing a writer is a change to this table, through a decision. +**How it is enforced** ([ADR 0229](../../02-DECISIONS/0229-the-cores-order-is-a-lease-the-store-remembers-and-an-epoch-a-machine-is-sent-once-it-reads-one.md)): +the table is compiled into the controller, each row the bus carries with the subjects a write of it +publishes to — a declaration `mesh.node.*.declare`, a report `mesh.control.*.report` (its own machine's +only), the controller's buckets, stream creation, update and deletion and durable consumer creation, a +build's outcome, a merge announced, a provider's standing — and its writer among the principals. A +grant that overlaps another writer's subject is refused at composition, naming the state and its +writer. A test holds the compiled table to this one, row for row. The controller does not publish +`mesh.control.>`: it would be a second writer of every machine's report. + ## 2. The condition store (rules 5, 6) **A condition is a durable fact about something the mesh owns that is wrong.** It is raised and cleared by observation only. **Where.** A key-value bucket, `mesh-controller_conditions`, written by the controller alone, one entry -per open condition. Every transition — raised, changed, silenced, cleared — is also appended to a -history kept ninety days, read through `conditions history`. +per open condition, with no age. Every transition — raised, changed, silenced, cleared — is also +appended to a history kept ninety days in a bucket of its own, `mesh-controller_condition-history`, +read through `conditions history`: a bucket has one age for every key, and an open condition must +outlive the history. **The key** names the thing and the kind, so the same fault said again is the same condition: `..`, where scope is one of `machine`, `plan`, `call`, `build`, `merge`, `provider`, `seat`, `bus`, `core`, `probe`, `mesh`. Examples of the shape: `machine..silent`, `plan..stalled`, `provider....failing`, `core.controller..rolled-back`, -`bus..slow-consumer`. +`bus..slow-consumer`. The last token is the kind's short word where it has one (`failing` for +`provider-failing`), the kind elsewhere; a key is opaque to every reader, and the kind is the field. **The fields:** @@ -122,7 +135,10 @@ history kept ninety days, read through `conditions history`. - **The events**, emitted by the controller on its own subjects: `condition-raised`, `condition-changed` (severity or resolver changed; not every observation), `condition-cleared`. The operator-channel's holder and any other surface consume them; the controller learns nothing about - telling. + telling. Each carries the condition at the top level, with `event`, `at`, `change` (raised, + reopened, severity, resolver, silenced, silence-ended, cleared), `why`, `was`, `cleared` and `show` + beside it; a silence and its ending are `condition-changed`, a reopening `condition-raised` with + change `reopened`. - **`status`** lists open conditions first, urgent before warning, oldest first, silenced ones marked with their expiry. The all-well sentence requires none open, silenced included. - **ADR 0224's provider standing** is the first kind: `provider-failing`, raised by the provider's @@ -140,20 +156,24 @@ signal and asserts its condition. `doctor signals` shows, for every row, the age | S3 | plan tier progress | controller's plan | each tier entered | from the tier's build and apply durations, *provisional* | `stalled` (names the tier and what it waits on) | warning | H2 | | S4 | the controller's event loop takes a message | controller | while its consumer has pending messages | 2 min | `controller-deaf` | urgent | — | | S5 | a merge announced becomes a plan or *nothing reads it* | announcer → controller | each merge | 10 min (exists, issue 266) | `merge-not-acted` | urgent | — | -| S6 | a build asked → its outcome | build seat | each ask | the build's declared timeout | `ask-lost` | warning | — | -| S7 | a call running → finished | controller | each call | the verb's declared bound | `call-hung` | warning | — | +| S6 | a build asked → its outcome | build seat | each ask | max(20 min, 3 × the p90 of measured builds), 1 h while nothing is measured, *provisional* (the build seat declares no timeout) | `ask-lost` | warning | — | +| S7 | a call running → finished | controller | each call | the verb's bound: push, rotate and command 30 min; assign and unassign 15 min; doctor 3 min; others 10 min | `call-hung` | warning | — | | S8 | a provider's failing word repeated | provider | every 15 min while failing (ADR 0224) | 30 min | `provider-silent` | warning | — | | S9 | bus advisories: slow consumer, maximum deliveries, permission violation, consumer deleted | bus server's system subjects | any | any occurrence | `slow-consumer`, `max-deliveries`, `refused`, `consumer-lost` — each naming the call, consumer or module in the mesh's words | warning | H4 for `slow-consumer` on a resettable consumer | | S10 | the self-check's heartbeat | controller's `doctor` | every run | 2 × its interval, watched **from the second machine** (§5) | `self-check-silent` | urgent | — | | S11 | node tools heartbeat | node tools | its interval | 3 × interval, *provisional* | `tools-silent` | warning | — | -| S12 | the controller lease renewed | controller | every 5 s | 15 s | `lease-lost` | urgent | — | -| S13 | stale refusals | every receiver (rule 2) | each refusal | more than 5 from one writer in 5 min | `stale-writer` (names the writer) | warning | — | -| S14 | facts snapshot exported | controller | daily | 2 days | `facts-stale` | warning | — | +| S12 | the controller lease renewed | controller | every 5 s | 15 s; a holder that lost the lease or was found expired, and a lease bucket found raised again from nothing, said for an hour; a controller serving without the lease, while it does | `lease-lost` | urgent | — | +| S13 | stale refusals | every receiver (rule 2) | each refusal | more than 5 from one writer in 5 min | `stale-writer` (names the writer: the controller epoch a refused declaration claimed, with its instance and how its lease ended; or the machine whose older accounts the controller refused) | warning | — | +| S14 | facts snapshot exported | controller | daily | 2 days (Phase 5) | `facts-stale` | warning | — | | S15 | a hand act with a cause already recorded | hand-act log | each act | the second within 14 days | `healer-wanted` | warning | — | The bus advisories (S9) cost one read-only subscription: the server already publishes them. The controller translates each into a condition naming the thing in the mesh's words, as the refused reply -of issue 265 is translated today. +of issue 265 is translated today. An advisory clears after an hour without another; a deleted consumer +is said only for one the mesh names and expects, and clears when it exists again. Until the bus has a +system account the controller hears a slow consumer and a refused subject for its own connection only +([issue 270](../../04-ISSUES/270-phase-1-watches-signals-that-come-later-and-hears-only-the-controllers-faults/00-report.md), +open question 2). ## 4. The self-check: `doctor` (rule 6) @@ -168,12 +188,13 @@ itself — an unanswered probe is never a pass. | D2 | every holder of the mesh's resolver answers a machine name for IPv4, and NODATA for IPv6 | 262 | | D3 | every seat on record has a live holder that answers | 208, 218 | | D4 | every kept archive is held by a manifest | 253 | -| D5 | exactly one lease holder; no message from a stale epoch in the last interval | 204 | -| D6 | every durable consumer exists with its definition and is near its stream's head | 248, 266 | +| D5 | exactly one lease holder — the key names this controller at the epoch it acts under, and the controller's record holds no other epoch open; no message from a stale epoch refused in the last interval | 204 | +| D6 | every durable consumer exists with its definition and is near its stream's head (within 1000 messages, *provisional*) | 248, 266 | | D7 | every stream the controller defines exists with its definition | 208 | | D8 | no address the mesh owns is in a ban list | 238 | | D9 | `status` answers in full within ten seconds | 265 | | D10 | every machine runs the node-engine and node tools builds its plan says, or is inside a plan's window | version split | +| DW | the watchdogs of §3 ran within three of their intervals: the watchers are watched, and raise S10 when the self-check stops | rule 6 | | H-* | the health probes of §8, run for every core component on every machine | rule 8 | - **`doctor`** answers the last run's verdict at once: per probe, pass, fail or failed-to-run, and age. @@ -220,6 +241,20 @@ fifteen-second time to live, renewed every five seconds. The **epoch** is the bu it was taken. A starting controller waits for the key to be absent or expired. One that fails a renewal stops acting at once and exits, so its service manager restarts it as a waiting candidate. +As built ([ADR 0229](../../02-DECISIONS/0229-the-cores-order-is-a-lease-the-store-remembers-and-an-epoch-a-machine-is-sent-once-it-reads-one.md)): + +- **The gate is the clock.** A holder stops acting three seconds before its key could expire + unrenewed, whatever its renewing goroutine is doing; every act passes that gate at the moment it is + made — a declaration at its send as well as where it was composed. +- **The epoch never goes backwards.** The controller's store keeps every epoch issued, its instance, + and how it ended (`released`, `lost`, `expired`); a lease bucket at or under the highest is moved past + it before the key is taken, and S12 says so. +- **A holder that stops gives the key back**, so the next takes it at once. +- **Without the bucket's grant, nobody holding it,** a controller serves unleased — no epoch on what it + sends, S12 urgent — and takes the lease once the user list that grants it reaches the bus. +- **A command at a shell** acts under the holder's epoch, read as it acts, or under a lease of its own + while nobody holds one. + ``` controller A (epoch 41) ──renew──renew──╳ (renewal refused)──► stops sending, exits controller B ──wait──────────────take (epoch 57)──► acts; marks A's running calls abandoned @@ -230,8 +265,8 @@ stops acting at once and exits, so its service manager restarts it as a waiting | Message | Carries | The receiver keeps, per writer | Refuses | |---|---|---|---| -| declaration | epoch, sequence | highest epoch, then highest sequence | older epoch; same epoch, lower sequence | -| report | machine, the declaration's epoch and sequence it is about, the node-engine's own report sequence (kept on disk, increasing across restarts and self-updates) | highest declaration sequence, then report sequence | an account of an older declaration; an older report | +| declaration | epoch, sequence | highest epoch, then highest sequence | older epoch; same epoch, lower sequence — only where both claim an epoch | +| report | machine, the declaration's epoch and sequence it is about, the node-engine's own report sequence (kept on disk, increasing across restarts and self-updates) | highest epoch where both claim one, then declaration sequence, then report sequence | an account of an older declaration; an older report | | plan write | the plan's revision, the epoch | — (compare-and-set) | a write against a revision already moved | | build outcome | the build's ask id and order | newest per module | an older build finishing later (exists, 219) | | call | call id, epoch | — | a finish from an epoch that is not the holder's, recorded as abandoned | @@ -241,6 +276,17 @@ stops acting at once and exits, so its service manager restarts it as a waiting node-engine — a report naming the refused declaration, so the controller sees it. The counter feeds S13. +**The wire** — the contract, written once on each side: a declaration's `epoch` and `sequence` are +top-level keys of its signed envelope; a report carries back the `epoch` and `sequence` of the +declaration it is about, its own `report_sequence`, `older_than` (the order held) on a refusal of an +older declaration, and `refused_older` (how many ever) on every report. Every key is optional, zero is +"no order claimed". **A machine is sent `epoch` only while its latest account carried a +`report_sequence`**, because an older node-engine refuses an unknown key, whole; a declaration without an +epoch is taken by its sequence, so a controller rolled back to a build without the lease is never +stranded. A report without a `report_sequence` is judged by the digest it names (issue 267). What the +mesh would send a machine is composed with the epoch it was last sent: a new holder is not a change of +the machine. + **One apply queue on every machine.** The node-engine has one worker that applies; a delivery, the five-minute reconcile, a self-update hand-over and a manual `apply` are four reasons to **enqueue**. The queue holds at most one pending request, coalesced. When the worker starts it takes the newest @@ -369,7 +415,7 @@ has a week of entries; the durations are recorded for every machine. | Repository | Delivers | |---|---| -| mesh-controller | the condition store, its verbs, history and events (§2); ADR 0224's standing moved into it; watchdogs for S1–S10 and S12–S14 with bounds set from Phase 0's durations; the bus advisories subscribed and translated (S9); `doctor` with D1–D10 and its heartbeat (§4); `status` led by open conditions; the test generated from the signals table | +| mesh-controller | the condition store, its verbs, history and events (§2); ADR 0224's standing moved into it; watchdogs for S1–S11 and S13 with bounds set from Phase 0's durations (S12 is Phase 2's, S14 Phase 5's); the bus advisories subscribed and translated (S9); `doctor` with D1–D4, D6–D10, DW and its heartbeat (§4; D5 is Phase 2's); `status` led by open conditions; the test generated from the signals table | | mesh-host | the heartbeat carries its interval; D1's validator published as a package the controller imports | | mesh-tools | the node tools' heartbeat (S11) | | mesh-catalog | the `operator-channel` seat and its holder; the Telegram channel and the desktop notifier contributing to it; `mesh-watcher` on a machine other than the control node (§5) | @@ -393,6 +439,14 @@ bound corrected in the table. **Done when:** R1 ends with nothing from the stale epoch applied, R2 with one report of the newest sequence, R6 with the withdrawal braked; every consumed subject has its contract test. +**Built 2026-10-06** ([ADR 0229](../../02-DECISIONS/0229-the-cores-order-is-a-lease-the-store-remembers-and-an-epoch-a-machine-is-sent-once-it-reads-one.md)): +the controller's lease and epoch, plans by compare-and-set, accounts kept by order, S12, S13 by writer, +D5, the writers table enforced at composition, a contract for every consumed kind, the empty-on-error +lint; the node-engine's one apply queue, report order and epoch refusal; the withdrawal brake in the Go +providers' loop and the SDK's. **Not yet:** the replays R1, R2 and R6 in mesh-lab; mesh-tools' refusals +and lint; the lint in mesh-host; the SDK's loop announcing a provider's standing, without which its +brake is said in its journal only. + ### Phase 3 — Healers | Repository | Delivers | diff --git a/04-ISSUES/270-phase-1-watches-signals-that-come-later-and-hears-only-the-controllers-faults/00-report.md b/04-ISSUES/270-phase-1-watches-signals-that-come-later-and-hears-only-the-controllers-faults/00-report.md index ea101f2..71924cc 100644 --- a/04-ISSUES/270-phase-1-watches-signals-that-come-later-and-hears-only-the-controllers-faults/00-report.md +++ b/04-ISSUES/270-phase-1-watches-signals-that-come-later-and-hears-only-the-controllers-faults/00-report.md @@ -3,7 +3,7 @@ status: located opened: 2026-10-06 located-in: [mesh-controller] fixed-by: -amended-design: +amended-design: 03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md --- # 270. Phase 1 watches signals that come later, and hears only the controller's own bus faults @@ -67,3 +67,15 @@ but the design says Phase 1 delivers them, and a reader of the design believes i 2. Does the bus gain a system account — or the controller read the server's monitoring endpoint — so S9 hears every principal's slow consumer and refused subject? Either changes the foundation's shape. 3. Do decisions 1–6 above go into §1–§4 as written, through an amendment of to-be 45? + +## Phase 2, 2026-10-06 + +Open questions 1 and 3 are answered by [ADR 0229](../../02-DECISIONS/0229-the-cores-order-is-a-lease-the-store-remembers-and-an-epoch-a-machine-is-sent-once-it-reads-one.md) +and the amendment of to-be 45 it authorises: S12 and D5 are Phase 2's and S14 Phase 5's in the phase +table, and decisions 1–6 above are in §1–§4 as built. Phase 2 built S12 (the lease renewed, a holder that +lost it, a bucket raised again from nothing, a controller serving without it), D5, and S13 naming its +writer — the controller epoch a refused declaration claimed, with its instance and how its lease ended. + +**Open question 2 stays open**: S9 still hears a slow consumer and a refused subject for the controller's +own connection only, until the bus has a system account or the controller reads the server's monitoring +endpoint. diff --git a/04-ISSUES/272-the-sdks-provider-loop-says-nothing-on-the-bus/00-report.md b/04-ISSUES/272-the-sdks-provider-loop-says-nothing-on-the-bus/00-report.md new file mode 100644 index 0000000..3a861cf --- /dev/null +++ b/04-ISSUES/272-the-sdks-provider-loop-says-nothing-on-the-bus/00-report.md @@ -0,0 +1,44 @@ +--- +status: located +opened: 2026-10-06 +located-in: [mesh-sdk src/provisioner] +fixed-by: +amended-design: +--- + +# 272. The SDK's provider loop says nothing on the bus + +## Symptom + +Found building Phase 2 of [to-be 45](../../03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md) +on 2026-10-06, adding the withdrawal brake to both copies of the providers' loop. The Go loop (carried by +the postgres and identity-provider modules) announces a consumer it keeps failing as +`provisioner.failing`, and its recovery as `provisioner.recovered` +([ADR 0224](../../02-DECISIONS/0224-a-provider-that-keeps-failing-a-consumer-is-a-problem-the-controller-reports.md)); +the controller raises each as a condition. **The SDK's TypeScript loop announces neither.** Every provider +built on it — the object store, the SQL server, the document store, the mail server, the forge, the +analytics service — fails a consumer, and is braked from withdrawing several at once, in its journal +alone. + +## Why it matters + +ADR 0224 is written about every provider: *a provider that keeps failing a consumer is a problem the +controller reports*. ADR 0227 rule 4 has a reconcile that would withdraw more than its bound **raise a +condition**. For half the mesh's providers neither reaches the controller, so neither reaches `status` or +the operator — the shape of issue 179, where a provider failed every consumer for a day and said so only +in its journal. + +## Located + +`mesh-sdk src/provisioner/index.ts`: `runProvisioner` keeps no standing per consumer and has no way to +emit an event. The Go loop's `failed`, `succeeded`, `recovered` and `Announce` are the behaviour to +carry over (it says it follows the SDK's loop "line for line in behaviour"; since ADR 0224 it does more). +The SDK's stdio module can already emit a module's events, which is what the Go providers announce +through. + +## Fix direction + +Give `runProvisioner` the Go loop's standing — a run of failures announced after five minutes, again +every fifteen, recovered on the first success or when the consumer goes — and announce a braked +withdrawal through it with the class `withdrawal-braked`, as the Go loop does. Each TypeScript provider +then bumps its SDK. A test like the Go loop's `standing_test.go` and `brake_test.go`.