Issues 348 and 349: the review's answers — gaps, parts, D2 unchanged, the newest merge in any state
This commit is contained in:
+26
-18
@@ -126,50 +126,58 @@ waits for the private address (above).
|
|||||||
TypeScript tools were never served on any machine). Its new long-running process, the watch, runs as root
|
TypeScript tools were never served on any machine). Its new long-running process, the watch, runs as root
|
||||||
and every 15 seconds asks the machine's own name — from the roster — on every address the configuration
|
and every 15 seconds asks the machine's own name — from the roster — on every address the configuration
|
||||||
listens on, over UDP and TCP, and checks the answer is that machine's address. After two failing looks in
|
listens on, over UDP and TCP, and checks the answer is that machine's address. After two failing looks in
|
||||||
a row it restarts the unit, at most once every two minutes and three times an hour, and the looks after it
|
a row it restarts the unit — only one that runs: a unit stopped or disabled on purpose, still starting
|
||||||
|
(waiting at boot for the private address) or failed is the service manager's or a person's, and is held
|
||||||
|
and said — without waiting for the start, at most once every two minutes and three times an hour (a
|
||||||
|
budget it reads back when it restarts itself), and the looks after it
|
||||||
say whether the restart helped. Its tool `dnsmasq_health` is the watch's declared check (ADR 0240): it asks
|
say whether the restart helped. Its tool `dnsmasq_health` is the watch's declared check (ADR 0240): it asks
|
||||||
again and reads what the watch did, so a restart that helped leaves the module healthy and the operator
|
again and reads what the watch did, so a restart that helped leaves the module healthy and the operator
|
||||||
hears nothing, and one that did not becomes `module.dnsmasq.<machine>.unhealthy`, saying "restarted at …,
|
hears nothing, and one that did not becomes `module.dnsmasq.<machine>.unhealthy`, saying "restarted at …,
|
||||||
which did not help". The unit itself is judged by its unit (`health: unit`). The bundle emits `restarted`
|
which did not help". The unit itself is judged by its unit (`health: unit`). The bundle emits `restarted`
|
||||||
on the bus beside the names it answers. The repair is the module's own, as the identity provider repairs
|
and, once the looks decide it, `restart-judged` (helped or not) on the bus beside the names it answers,
|
||||||
|
each remembered as said only once emitted. The repair is the module's own, as the identity provider repairs
|
||||||
its administrator (ADR 0224 §5), because a resolver that stops answering can cut its machine off from the
|
its administrator (ADR 0224 §5), because a resolver that stops answering can cut its machine off from the
|
||||||
bus by name, and then nothing outside the machine reaches it. **This departs from ADR 0240 rule 6**
|
bus by name, and then nothing outside the machine reaches it. **This departs from ADR 0240 rule 6**
|
||||||
("nothing restarts on health"), which left a restart on health to a healer's own record under ADR 0231.
|
("nothing restarts on health"), which left a restart on health to a healer's own record under ADR 0231.
|
||||||
That record is not written; whether this module-local repair stands, or becomes a row of the healer
|
That record is not written; whether this module-local repair stands, or becomes a row of the healer
|
||||||
registry, is the operator's decision.
|
registry, is the operator's decision.
|
||||||
|
|
||||||
**mesh-controller — a fault is judged by when it began.**
|
**mesh-controller — a fault is judged by whether it was there at the send.**
|
||||||
|
|
||||||
- A condition keeps `first`, when the fault was first raised, across a reopening within `ReopenWithin`
|
- A condition reopened within `ReopenWithin` keeps when the fault was first raised (`first`) and its
|
||||||
(and across a controller restart, from the history); `Began()` reads it. The gate compares `Began()`,
|
gaps, each from a clearing to the reopening after it, across a controller restart too (from the
|
||||||
not the raised time, with the send.
|
history). The gate asks `OpenAt(send)`: the fault began at or before the send and the send fell in none
|
||||||
- An undecided network statement — a part `unknown` or `starting`, as a node-engine just started says —
|
of its gaps. So a fault there at the send that flapped after it is not the send's, and a machine that
|
||||||
clears no network condition about that machine. The next statement decides.
|
had recovered before the send and broke after it fails the gate, as it must (review of the controller's
|
||||||
|
pull request: a silent pass is worse than a false blame).
|
||||||
|
- An undecided part of a network statement — `unknown` or `starting`, as a node-engine just started says
|
||||||
|
— holds the open network conditions whose newest evidence names that part, and only those. A statement
|
||||||
|
undecided as a whole holds every part. The next statement decides.
|
||||||
- D10 counts what a release's tier names as rolling out, so a walked node-engine is not `core-behind` on
|
- D10 counts what a release's tier names as rolling out, so a walked node-engine is not `core-behind` on
|
||||||
its own first machine.
|
its own first machine.
|
||||||
- D2 raises a resolver at once when every try of every question it asked was refused: a refusal is the
|
- D2 is unchanged: a resolver that answered nothing, refused or silent, is held for the next run, because
|
||||||
machine's answer that nothing listens there, not a loaded resolver. The detection is then at most one
|
a few hundred milliseconds of refusals is a resolver restarting as well as one that stopped. D2 says it
|
||||||
self-check interval (five minutes). The node-engine's own network check stays the fast one (48 seconds
|
five to ten minutes after; the node-engine's own network check is the fast one (48 seconds on the day).
|
||||||
on the day).
|
|
||||||
|
|
||||||
**mesh-catalog — the tools that were missing.**
|
**mesh-catalog — the tools that were missing.**
|
||||||
|
|
||||||
- `netcheck_dns` asks one named resolver directly (`server`, `tcp`), and `netcheck_listening` lists what
|
- `netcheck_dns` asks one named resolver directly (`server`, `tcp`), and `netcheck_listening` lists what
|
||||||
listens on the machine (`ss -lunt`). netcheck is assigned on the home server and the laptop only;
|
listens on the machine (`ss -lunt`). netcheck is assigned on the home server and the laptop only;
|
||||||
assigning it on the control node and the workstation is the operator's.
|
assigning it on the control node and the workstation is the operator's.
|
||||||
- `docker_events` says how far back the runtime's memory reaches, and says plainly when the window asked
|
- `docker_events` reads the runtime's whole buffer of events: the window is complete when the buffer is
|
||||||
reaches further back, instead of answering an empty list.
|
not full (it then holds every event since the runtime started) or reaches past the window's start, and
|
||||||
|
otherwise says where the history begins, instead of answering an empty list.
|
||||||
- `docker_resolv_conf` reads a container's resolver file as the container sees it.
|
- `docker_resolv_conf` reads a container's resolver file as the container sees it.
|
||||||
|
|
||||||
## How it is checked
|
## How it is checked
|
||||||
|
|
||||||
| Rule | Check |
|
| Rule | Check |
|
||||||
|---|---|
|
|---|---|
|
||||||
| A reopened fault from before a send fails no gate | controller `TestReplay348` (the day's statements: raised, an undecided restart, raised again; the gate on a send between), `TestAReopenedFaultFromBeforeTheSendIsNotRaisedSinceIt`, conditions `TestAReopeningKeepsWhenTheFaultBegan` — the first two fail on the commit before the fix |
|
| A fault there at the send fails no gate; one recovered at the send and back after it does | controller `TestReplay348` (the day's statements: raised, an undecided restart, raised again; the gate on a send between), `TestAFaultThatFlappedAfterTheSendIsNotTheSendsAndOneThatRecoveredBeforeItIs`, conditions `TestAReopeningKeepsWhenTheFaultBegan`, `TestAFaultKeepsItsNewestGapsAndForgetsWhatWasBefore` |
|
||||||
| An undecided statement clears nothing | `TestReplay348`, `TestAnUndecidedStatementKeepsTheMachinesNetworkConditions` |
|
| An undecided part holds only what names it | `TestReplay348`, `TestAnUndecidedPartHoldsOnlyWhatNamesIt` |
|
||||||
| A release's modules are rolling | `TestAReleaseRollsOutWhatItsTierNames` |
|
| A release's modules are rolling | `TestAReleaseRollsOutWhatItsTierNames` |
|
||||||
| A refusing resolver is raised at once | `TestAResolverThatRefusesEveryTryIsSaidAtOnce` — fails on the commit before the fix |
|
| The resolver is asked a real question, restarted, and said when it did not help | dnsmasq `TestALookFindsAResolverThatStoppedAnsweringOnOneAddress`, `TestTheWatchRestartsOnASecondFailingLookWithinItsBudget`, `TestTheWatchRestartsAResolverThatStoppedAnsweringAndHealthSaysWhetherItHelped`, `TestTheWatchRestartsOnlyAUnitThatRuns`, `TestTheWatchLoopKeepsItsBudgetAcrossItsOwnRestart`, `TestHealthSaysWhenTheWatchIsNotLookingOrTheUnitIsNotRunning`, `TestTheAnnouncerSaysWhatChangedAndRemembersOnlyWhatWasSaid` |
|
||||||
| The resolver is asked a real question, restarted, and said when it did not help | dnsmasq `TestALookFindsAResolverThatStoppedAnsweringOnOneAddress`, `TestTheWatchRestartsOnASecondFailingLookWithinItsBudget`, `TestTheWatchRestartsAResolverThatStoppedAnsweringAndHealthSaysWhetherItHelped` |
|
| Every changed condition is tested | the controller's and the modules' changes were mutated one at a time (28 in the controller, 31 in the modules); each failed a test |
|
||||||
| live | after delivery: `dnsmasq_health` on both resolvers healthy; a restart by the watch said as `restarted` on the bus |
|
| live | after delivery: `dnsmasq_health` on both resolvers healthy; a restart by the watch said as `restarted` on the bus |
|
||||||
|
|
||||||
A replay in mesh-lab's register is still owed before this core issue resolves (ADR 0237).
|
A replay in mesh-lab's register is still owed before this core issue resolves (ADR 0237).
|
||||||
|
|||||||
+15
-6
@@ -39,17 +39,26 @@ left it, so a later merge's commit contains every earlier one's, and the forge's
|
|||||||
|
|
||||||
In mesh-controller, in the same pull request as issue 348 (PR 179):
|
In mesh-controller, in the same pull request as issue 348 (PR 179):
|
||||||
|
|
||||||
- A plan keeps when its merge was made (`merged_at`, migration 0086; null for a release and for a plan kept
|
- A plan keeps when its merge was made, to the nanosecond (`merged_at`, migration 0086; null for a release
|
||||||
before it).
|
and for a plan kept before it).
|
||||||
- `supersededBy` orders two plans of a branch by their merge times where both are known, and by when they
|
- `supersededBy` orders two plans of a branch by their merge times where both are known, and by when they
|
||||||
were made only where one is not.
|
were made only where one is not.
|
||||||
- A merge older than an open plan of its branch is planned at that plan's commit, which contains it, and
|
- A merge older than the newest planned merge of its branch — that plan in any state, open or already done
|
||||||
that plan is superseded by it as by any newer plan. What both merges moved is built once, from the newest
|
— is planned at that merge's commit, which contains it, with that merge's time. A module the newer merge
|
||||||
commit of the branch.
|
already looked at then reads as history and is not built again; what only the older merge moved, and its
|
||||||
|
dependents, are built from the newest commit. Comparing with open plans alone left a gap the review
|
||||||
|
found: a late merge arriving after the newer plan was done rebuilt its dependents at the older commit.
|
||||||
|
That gap is closed.
|
||||||
|
- Not covered: a newer merge that moved nothing the mesh builds made no plan, so it is not known here; a
|
||||||
|
late older merge after it is then planned at its own commit, as before. Its modules are those the newer
|
||||||
|
merge did not change, so what it builds from the older commit is what the newer one holds too, unless a
|
||||||
|
dependent's build reads shared code the newer merge changed.
|
||||||
|
|
||||||
## How it is checked
|
## How it is checked
|
||||||
|
|
||||||
`TestTwoMergesActedOnInReverseOrderBuildTheNewerCommit` replays the day's case: the later merge acted on
|
`TestTwoMergesActedOnInReverseOrderBuildTheNewerCommit` replays the day's case: the later merge acted on
|
||||||
first, the earlier one second; one plan is left open, at the later commit, building what both moved.
|
first, the earlier one second; one plan is left open, at the later commit, building what both moved.
|
||||||
`TestTheBranchOrderIsTheMerges` holds the order and the lookup. Both fail on the commit before the fix.
|
`TestALateMergeAfterTheNewerPlanEndedBuildsTheNewerCommit` acts on the late merge after the newer plan is
|
||||||
|
done. `TestTheBranchOrderIsTheMerges` holds the order and the lookup, to the nanosecond. The first and the
|
||||||
|
last fail on the commit before the fix, and every condition of the change was mutated and caught.
|
||||||
A replay in mesh-lab's register is still owed before this core issue resolves (ADR 0237).
|
A replay in mesh-lab's register is still owed before this core issue resolves (ADR 0237).
|
||||||
|
|||||||
Reference in New Issue
Block a user