mesh-controller/internal/link/window.go

package link

import (
	"errors"
	"time"

	"github.com/novox/mesh-controller/internal/inventory"
)

// The store window, as a decision rather than a mechanism.
//
// The guarantee (novox/hq ADR 0083): a push the controller cannot record because its store is
// restarting is **held and retried**, not dropped and not falsely acknowledged. On the bus the
// mesh runs on today that is done by keeping the delivery unacknowledged in memory and settling
// it later. On the bus being built it is a `nak` with a delay: the server holds it and redelivers,
// so the controller keeps no list of parked messages and a controller that restarts mid-window
// loses nothing it was holding.
//
// **The decision is the same either way, and the mechanism is not the interesting part.** What is
// interesting is that moving the holding into the server introduces a problem the in-memory
// version did not have, and the answer was already in the message.

// Verdict is what to do with one control message.
type Verdict int

const (
	// Take it: apply, then acknowledge.
	Take Verdict = iota
	// Hold it: the store cannot record this yet. Nak with a delay and let the server redeliver.
	Hold
	// Stale: this is about a declaration the node has already moved past, and applying it would
	// undo what came after. Acknowledge without acting — redelivering forever is worse.
	Stale
	// GiveUp: the store has not come back in time. Settle it and say so, loudly.
	GiveUp
)

// StoreWindow decides. Pure, so the guarantee is testable without a bus, a store or a clock.
type StoreWindow struct {
	// GiveUpAfter is how long one message may be held before it is let go with a line saying so.
	GiveUpAfter time.Duration
}

// Decide answers for one delivery.
//
//   - err is what the store said, or nil.
//   - declaredIn is the digest of the declaration this message is about, empty when it is not
//     about one (an enrolment, a build result).
//   - outstanding is the digest the mesh last sent that node, empty when it has sent none.
//   - heldFor is how long this message has already been held; zero on first delivery.
func (w StoreWindow) Decide(err error, declaredIn, outstanding string, heldFor time.Duration) Verdict {
	// **Staleness is checked before the store, not after.** A redelivery that lost its race is
	// not worth waiting on a store for, and asking the store first would mean a message about a
	// superseded declaration holding a slot in the window that a current one needs.
	if declaredIn != "" && outstanding != "" && declaredIn != outstanding {
		return Stale
	}
	if err == nil {
		return Take
	}
	if !errors.Is(err, ErrTryAgain) && !inventory.Unreachable(err) {
		// Not the store being away: a refusal is an answer, and holding it would turn a message
		// the mesh understood into one it retries forever.
		return Take
	}
	if heldFor >= w.GiveUpAfter {
		return GiveUp
	}
	return Hold
}

// RedeliverAfter is how long the server should hold a naked message before trying again.
//
// Backed off, and bounded. A store restarting is back in seconds; a store that is gone is not
// helped by being asked every second, and the delay is what keeps a window of held messages from
// becoming a spin.
func RedeliverAfter(heldFor time.Duration) time.Duration {
	switch {
	case heldFor < 5*time.Second:
		return time.Second
	case heldFor < 30*time.Second:
		return 5 * time.Second
	default:
		return 15 * time.Second
	}
}

// The problem holding-in-the-server introduces, and why the answer was already in the message.
//
// Holding a delivery in memory let the controller do something a server cannot: when a newer
// report for the same node arrived, it dropped the older one, "because acting on it after the
// newer would undo the newer". A `nak`ed message is the server's, and the server will redeliver
// it whatever else has happened in the meantime — so the older report comes back *after* the
// newer was applied, and applying it would undo exactly what that comment describes.
//
// **A report already says which declaration it is about.** `Declared` is the digest of the exact
// bytes the mesh sent, and it exists because an earlier attempt to order reports by time lost the
// race it invited — an apply that started under the previous declaration finishes after the next
// is sent, and the report reads as newer than the send. Clocks cannot answer *which*; the digest
// is the answer itself.
//
// So supersession stops being a thing the controller remembers and becomes a thing it checks: a
// report whose digest is not the one outstanding for that node is stale, and is acknowledged
// without being acted on. Which is the same shape as a node refusing a superseded declaration by
// sequence (novox/hq issue 107) — ordering settled by what the message says, not by when it
// happened to arrive.