novox/mesh-catalog
13
0
Fork 0
Code Issues Pull Requests 4 Actions Packages Projects Releases Wiki Activity

Retire a consumer the mesh stops asking for, and delete only on a person's word (hq ADR 0230)

Browse Source 
The hourly release of ADR 0229's brake still ended in the mesh acting alone on
a mistake. A consumer now stays active until the same unasked set holds for
five passes, waits for a person past three or half of those held, is disabled
and marked rather than withdrawn, comes back as it was when asked again, and is
deleted only through the provider's delete tool. The backend keeps the mark, so
a restart forgets nothing and finds what was withdrawn before.
This commit is contained in:
jochen
2026-10-06 13:54:10 +02:00
parent 4554e18279
commit e68ef88333
26 changed files with 3105 additions and 494 deletions
Download Patch File Download Diff File Expand all files Collapse all files
modules/postgres/cmd/postgres-provider/harness.go
+55 -119
View File
@@ -1,12 +1,12 @@
package main
// The reconcile loop every provider shares, as the TypeScript SDK's runProvisioner runs it
// (@novox/mesh-sdk/provisioner, 0.1.11). The Go SDK has no provisioner yet, so this module carries
// (@novox/mesh-sdk/provisioner, 0.1.12). The Go SDK has no provisioner yet, so this module carries
// the loop itself, line for line in behaviour; when the Go SDK grows one, this file is what moves
// there (novox/hq ADR 0039: the loop is the SDK's, the adapter is the module's).
//
// Read the contributions the mesh delivered; bring each consumer's resource into being through the
// adapter, under the login and password the mesh minted; withdraw what the mesh no longer asks for.
// adapter, under the login and password the mesh minted; retire what the mesh no longer asks for.
// **A provider creates the credential the mesh minted, and seals nothing (novox/hq ADR 0048).**
//
// **A provider that keeps failing a consumer says so on the bus (novox/hq ADR 0224).** A consumer
@@ -17,18 +17,15 @@ package main
// identity provider failed every consumer 31,000 times in a day and said so only in its journal
// (novox/hq issue 179).
//
// **A reconcile that would withdraw more than its bound stops, and says so (novox/hq to-be 45 Phase 2,
// ADR 0227 rule 4).** Withdrawing more than WithdrawAtOnce consumers in one pass — or more than
// WithdrawFraction of those this process holds — is the shape of issue 241, where one misread file
// withdrew seven at once. Such a pass withdraws nothing: each consumer it would have withdrawn is kept,
// announced `provisioner.failing` with the class `withdrawal-braked` (the controller raises it as a
// condition), and said. While the same consumers stay unasked for, one is released every ReleaseEvery,
// said and announced as it goes, so an intended unassignment of many completes without a hand and a
// mistaken one costs at most one consumer an hour while the operator is told. Withdrawal never destroys
// data (issue 241's second half), so a release is a login locked, not a database dropped.
// **A consumer the mesh stops asking for is retired, not withdrawn, and deleted only by a person
// (novox/hq ADR 0230, the operator's model of 2026-10-06).** Retiring disables its access — reversibly —
// and marks its login and data "to delete" with when and why; nothing is deleted. Asked for again, the
// ordinary create re-enables it as it was. It is retired only once the same set has gone unasked in
// StablePasses consecutive passes, and a set larger than the bound waits for a person. retirement.go.
//
// Carried, identical, by every Go provider until the Go SDK has the loop: postgres and keycloak.
// Each module's `harness_same_test.go` fails when its copy and the other's differ.
// Each module's `harness_same_test.go` fails when its copy and the other's differ (this file and
// retirement.go).
import (
"context"
@@ -37,8 +34,8 @@ import (
"fmt"
"net/url"
"os"
"sort"
"strings"
"sync"
"time"
)
@@ -59,11 +56,21 @@ type Provision struct {
// Adapter is the per-service half.
type Adapter interface {
// Create brings the consumer into being under the mesh's login and password — and, for one that
// was retired, re-enables it as it was and clears its mark to delete.
Create(ctx context.Context, p Provision) error
// Remove withdraws what Create made; derived is what the mesh last derived, remembered here.
Remove(ctx context.Context, as string, derived map[string]any) error
// Retire disables the consumer's access, reversibly, and marks its login and data to delete with
// when and why. It deletes nothing (novox/hq ADR 0230). derived is what the mesh last derived,
// remembered here; nil for a consumer this process did not make.
Retire(ctx context.Context, as string, derived map[string]any, why string, at time.Time) error
// Holds says whether the backend still holds the consumer exactly as p says. Read-only.
Holds(ctx context.Context, p Provision) (bool, error)
// Inventory is what the backend holds that the mesh made: consumers active and retired, read from
// the backend itself so a restart forgets nothing. Read-only.
Inventory(ctx context.Context) (Inventory, error)
// Delete removes one retired consumer — its login and its data — for good. Only ever asked by a
// person, through `cleanup delete`; the harness has checked it is retired and not asked for.
Delete(ctx context.Context, r Retired) (freedBytes int64, err error)
}
// Harness is the loop's settings and memory.
@@ -85,19 +92,18 @@ type Harness struct {
// missed the first hears the next, and a standing nobody repeats can be told from one that holds.
FailingAfter time.Duration
SayAgainEvery time.Duration
// WithdrawAtOnce (1) and WithdrawFraction (0.5) bound what one pass may withdraw: more consumers
// than WithdrawAtOnce, or a larger share of those held than WithdrawFraction, brakes the pass.
// ReleaseEvery (1h) is how often a braked withdrawal lets one consumer go.
WithdrawAtOnce int
WithdrawFraction float64
ReleaseEvery time.Duration
// StablePasses (5) is how many consecutive passes must see the same set no longer asked for before
// it is retired. RetireAtOnce (3) and RetireFraction (0.5) bound what is retired without a person:
// more consumers than RetireAtOnce, or — where more than one is held — a larger share of those held
// than RetireFraction, waits for `retire approve` (novox/hq ADR 0230).
StablePasses int
RetireAtOnce int
RetireFraction float64
verifiedAt time.Time
// braked is every consumer a braked pass kept, by when it was first kept; releasedAt is when the
// brake last let one go, and brakeSaid the set it last said, so a pass repeats nothing.
braked map[string]time.Time
releasedAt time.Time
brakeSaid string
// mu is held by a pass and by every tool a person asks, so the two never interleave.
mu sync.Mutex
verifiedAt time.Time
r retirement
applied map[string]appliedEntry
lost map[string]brake
waiting map[string]int
@@ -131,9 +137,6 @@ const (
ClassUnreachable = "unreachable"
ClassSecret = "secret-unreadable"
ClassRefused = "refused"
// ClassWithdrawalBraked is a consumer the mesh no longer asks for, kept because the pass that would
// withdraw it would withdraw more than its bound (ADR 0227 rule 4).
ClassWithdrawalBraked = "withdrawal-braked"
)
// Classifier is an adapter that can say what class an error of its own is.
@@ -196,14 +199,14 @@ func (h *Harness) init() {
if h.SayAgainEvery == 0 {
h.SayAgainEvery = 15 * time.Minute
}
if h.WithdrawAtOnce == 0 {
h.WithdrawAtOnce = 1
if h.StablePasses == 0 {
h.StablePasses = 5
}
if h.WithdrawFraction == 0 {
h.WithdrawFraction = 0.5
if h.RetireAtOnce == 0 {
h.RetireAtOnce = 3
}
if h.ReleaseEvery == 0 {
h.ReleaseEvery = time.Hour
if h.RetireFraction == 0 {
h.RetireFraction = 0.5
}
if h.Log == nil {
h.Log = func(format string, args ...any) { fmt.Fprintf(os.Stderr, format+"\n", args...) }
@@ -215,7 +218,7 @@ func (h *Harness) init() {
h.failing = map[string]failure{}
h.trouble = map[string]*standing{}
h.cleared = map[string]bool{}
h.braked = map[string]time.Time{}
h.r.retired = map[string]retiredEntry{}
}
}
@@ -249,7 +252,7 @@ func (h *Harness) warn(why string) {
}
// readContributions answers the consumers asked for, or nil when the file says nothing usable.
// **Nothing read is not nobody asking** (novox/hq issue 241): only a file that was read can withdraw.
// **Nothing read is not nobody asking** (novox/hq issue 241): only a file that was read can retire anything.
func (h *Harness) readContributions() []contribution {
raw, err := os.ReadFile(h.Receives)
if err != nil {
@@ -299,15 +302,20 @@ func orEmpty(m map[string]any) map[string]any {
// Reconcile is one pass.
func (h *Harness) Reconcile(ctx context.Context) {
h.mu.Lock()
defer h.mu.Unlock()
h.init()
given := h.readContributions()
if given == nil {
// Not a result: the passes that must agree before anything is retired start again.
h.r.key, h.r.count = "", 0
return
}
want := map[string]bool{}
for _, g := range given {
want[g.As] = true
}
h.seed(ctx, want)
verifying := h.Now().Sub(h.verifiedAt) >= h.VerifyEvery
if verifying {
h.verifiedAt = h.Now()
@@ -396,6 +404,7 @@ func (h *Harness) Reconcile(ctx context.Context) {
}
h.succeeded(g.As)
h.applied[g.As] = appliedEntry{hash: hash, derived: p.Derived, node: g.Node}
h.reenabled(g.As, g.Node)
if reapplying == 0 {
delete(h.lost, g.As)
} else {
@@ -410,97 +419,24 @@ func (h *Harness) Reconcile(ctx context.Context) {
}
}
// Withdraw every login this process made that the mesh no longer asks for — within the bound.
var withdrawing []string
for as := range h.applied {
if !want[as] {
withdrawing = append(withdrawing, as)
}
}
sort.Strings(withdrawing)
for as := range h.braked {
if want[as] {
// Asked for again: the brake held what the mesh still wanted. Its standing was ended by this
// pass's success above, as any consumer's is.
delete(h.braked, as)
}
}
if h.overTheBound(len(withdrawing), len(h.applied)) {
withdrawing = h.brakeWithdrawal(withdrawing)
} else if len(h.braked) > 0 {
h.say("the withdrawal is within its bound again: %s withdrawn as asked", strings.Join(withdrawing, ", "))
h.braked, h.brakeSaid = map[string]time.Time{}, ""
}
for _, as := range withdrawing {
was := h.applied[as]
h.say("%s: no longer in %s; withdrawing it from the backend", as, h.Receives)
if err := h.Adapter.Remove(ctx, as, was.derived); err != nil {
h.say("%s: remove failed, will retry: %v", as, err)
continue
}
delete(h.applied, as)
delete(h.lost, as)
delete(h.braked, as)
}
// Retire what the mesh has stably stopped asking for — within the bound, or with a person.
h.retireUnasked(ctx, want)
h.r.lastWant = want
for as := range h.failing {
if !want[as] {
delete(h.failing, as)
}
}
// A consumer the mesh stopped asking for is no longer failed by anyone: said, so a standing
// the controller keeps for it is cleared rather than left naming a consumer that is gone. One the
// brake holds is still kept, and its standing stays.
// A consumer the mesh stopped asking for that this provider holds nothing for is no longer failed by
// anyone: said, so a standing the controller keeps for it is cleared rather than left naming a
// consumer that is gone. One still held is said recovered when it is retired.
for as := range h.trouble {
if _, held := h.braked[as]; !want[as] && !held {
h.recovered(as, "withdrawn")
if _, held := h.applied[as]; !want[as] && !held {
h.recovered(as, "no longer asked for")
}
}
}
// overTheBound says a pass withdrawing n of the held consumers would withdraw more than it may.
func (h *Harness) overTheBound(n, held int) bool {
if n == 0 {
return false
}
return n > h.WithdrawAtOnce || (held > 1 && float64(n) > h.WithdrawFraction*float64(held))
}
// brakeWithdrawal keeps every consumer a pass over its bound would withdraw, announces each as failing
// with the class withdrawal-braked, and answers the one it releases now, if one is due.
func (h *Harness) brakeWithdrawal(withdrawing []string) []string {
now := h.Now()
set := strings.Join(withdrawing, ", ")
if set != h.brakeSaid {
h.say("WITHDRAWAL BRAKED: this pass would withdraw %d of the %d consumer(s) this provider holds (%s), "+
"more than %d at once or %.0f%% of them. Nothing is withdrawn; each is announced as %s (%s), and one "+
"is let go every %s while the mesh goes on not asking for them (novox/hq ADR 0227 rule 4)",
len(withdrawing), len(h.applied), set, h.WithdrawAtOnce, h.WithdrawFraction*100, EventFailing,
ClassWithdrawalBraked, h.ReleaseEvery)
h.brakeSaid = set
}
if len(h.braked) == 0 {
// The release clock starts with the brake, not at the last release of an earlier one.
h.releasedAt = now
}
for _, as := range withdrawing {
if _, kept := h.braked[as]; !kept {
h.braked[as] = now
}
text := fmt.Sprintf("the mesh no longer asks for it, and the pass that would withdraw it would withdraw %d "+
"consumers at once: kept until released (%s)", len(withdrawing), set)
h.failed(as, h.applied[as].node, ClassWithdrawalBraked, text)
}
if now.Sub(h.releasedAt) < h.ReleaseEvery {
return nil
}
h.releasedAt = now
release := withdrawing[0]
h.say("%s: released by the withdrawal brake after %s; %d more kept", release,
now.Sub(h.braked[release]).Round(time.Second), len(withdrawing)-1)
h.recovered(release, "withdrawn")
return []string{release}
}
// failed counts one more failure in a consumer's unbroken run, and announces the run once it has
// lasted FailingAfter — then again every SayAgainEvery while it lasts.
func (h *Harness) failed(as, node, class, text string) {