Hold a consumer's findings under its unhealthy provider, and say them once there (hq ADR 0240, to-be 48 Phase C)
With twelve consumers of the database provision, one provider down would be twelve conditions for one fault and twelve gates failed for something none of them did. A consumer's check names the provision it exercises; while the provider composed for it — its recorded binding, or the machine its credential comes from — is unhealthy on the record, what that check finds raises nothing of its own: the provider's condition lists it as waiting and is urgent, and the consumer's gate waits, past its bound too, rather than putting a build back. Liveness findings and checks naming no provision stay the consumer's own, and once the provider is healthy a consumer still failing is raised at once.
This commit is contained in:
@@ -77,6 +77,10 @@ type health int
|
||||
|
||||
const (
|
||||
healthGood health = iota
|
||||
// healthWaiting is not a pass and not a fault: what the module's checks find waits on an unhealthy
|
||||
// provider (ADR 0240 rule 5), so the judging waits — past the bound too — rather than putting back a
|
||||
// build for something it did not do.
|
||||
healthWaiting
|
||||
healthNotYet
|
||||
healthBroken
|
||||
)
|
||||
@@ -109,6 +113,8 @@ type gateFacts struct {
|
||||
// healthErr is why they could not be read.
|
||||
health map[string]inventory.NodeHealth
|
||||
healthErr error
|
||||
// heldOn is, per "<module>@<machine>", the provider its findings are held under (ADR 0240 rule 5).
|
||||
heldOn map[string]string
|
||||
}
|
||||
|
||||
// gatherGateFacts reads what a judging needs, from the store, the bus and this controller's memory. A
|
||||
@@ -147,6 +153,17 @@ var gatherGateFacts = func(ctx context.Context, open *stores, component string)
|
||||
} else {
|
||||
f.servedErr = errors.New("this process does not serve the mesh, so it cannot ask the bus who serves what")
|
||||
}
|
||||
// Whose findings wait on an unhealthy provider (ADR 0240 rule 5): their gates wait, not fail.
|
||||
if f.healthErr == nil && f.openErr == nil {
|
||||
if hold, err := readHolding(ctx, inv, f.open); err == nil {
|
||||
f.heldOn = map[string]string{}
|
||||
for machine := range f.health {
|
||||
for module, p := range hold.heldModules(machine) {
|
||||
f.heldOn[module+"@"+machine] = p.Module + " on " + p.Node
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
if theLease != nil {
|
||||
h, found, err := theLease.holder(ctx)
|
||||
switch {
|
||||
@@ -435,6 +452,10 @@ func judgeMoves(ctx context.Context, open *stores, g *inventory.PlanGate, pairs
|
||||
// What broke is put back; what was only not yet healthy beside it is too — they moved together.
|
||||
g.Failing = failing
|
||||
decide(g, inventory.GateFailed, why, now)
|
||||
case worst == healthWaiting:
|
||||
// Waiting on a provider that is unhealthy: not a pass, and not a failure at the bound either —
|
||||
// the provider's own condition says what is wrong (ADR 0240 rule 5).
|
||||
g.Passes, g.LastPass, g.Last, g.Failing = 0, nil, why, failing
|
||||
case worst == healthNotYet:
|
||||
g.Passes, g.LastPass, g.Last, g.Failing = 0, nil, why, failing
|
||||
if now.Sub(*g.Since) > gateBound {
|
||||
|
||||
Reference in New Issue
Block a user