Report a provider that keeps failing a consumer in status (hq ADR 0224)

The identity provider failed every consumer for a day and status called the
mesh well (hq issue 179). The controller now follows every provider's
provisioner.failing/recovered, keeps the newest failing word per provider,
machine and consumer (migration 0065), and status, its JSON and node show
name it until it recovers. Every module that receives contributions is
granted the two events, so no manifest can forget them.
This commit is contained in:
jochen
2026-10-06 00:13:23 +02:00
parent cc25baa563
commit 8d9d33ae85
23 changed files with 913 additions and 11 deletions
+7
View File
@@ -79,6 +79,8 @@ type Server struct {
recorder Recorder
upgrader Upgrader
replayer Replayer
// standings keeps what providers say about their consumers (novox/hq ADR 0224).
standings Standings
log *log.Logger
// giveUp is how long one message is held for the store; zero means GiveUpAfter.
@@ -153,6 +155,9 @@ func (s *Server) Serve(ctx context.Context) error {
if s.replayer != nil {
s.log.Printf("answering %s", KindCatchUp)
}
if s.standings != nil {
s.log.Printf("keeping every provider's %s", KindProvisioner)
}
return s.inbound.Receive(ctx, s.act)
}
@@ -173,6 +178,8 @@ func (s *Server) act(ctx context.Context, m Control) {
s.sourceMoved(ctx, m)
case KindCatchUp:
s.catchingUp(ctx, m)
case KindProvisioner:
s.provisioner(ctx, m)
default:
// Dropped: a message nothing understands will not be understood on the next attempt
// either, and asking for it again would spin.