Keep refused keys from holding the take-in's slots, and judge its stall by acknowledgements (issue 1945, ADR 0544, review)

A refused key stays unacknowledged until its delay ends, so with 8 a consumer
a handful refused for long (a manifest word this controller does not read
yet, a node that keeps failing) stopped that consumer for good, the build that
reads the word among what it held back. Each consumer now leaves up to 1000
unacknowledged; the store's bound is the one loop's batch of 16. A refusal
waits 30 s, doubling each time up to 10 minutes.

The stall watch judged movement by deliveries, which rise with every
redelivery: it now counts the keys this process acknowledged. A consumer that
cannot be made raises take-in-stalled instead of a log line. On stop the
gathered batch and a key in hand are refused with no delay, so the next
controller has them at once. A trunk value with no fingerprint of the new form
is said once, as every wait for a write.
Research 072: the take-ins on a bounded pull consumer.

cmd/mesh-controller/composed_keys.go

@@ -57,6 +57,8 @@ type takeInSteps struct {
 type delivered struct {
 	module string
 	again  bool
+	// deliveries is how often the bus delivered it, this time included: what a refusal's delay grows with.
+	deliveries uint64
 }
 
 // verdict is what becomes of a module's delivered keys.
@@ -123,7 +125,18 @@ func (s takeInSteps) batch(ctx context.Context, keys []delivered, memo map[strin
 			continue
 		}
 		trunk, err := s.trunk(ctx, m)
-		if errors.Is(err, builderkeys.ErrNoValue) || (err == nil && !strings.HasPrefix(trunk.FP2, sdkdelivery.SourcePrefix2)) {
+		if errors.Is(err, builderkeys.ErrNoValue) {
+			out = append(out, r)
+			continue
+		}
+		if err == nil && !strings.HasPrefix(trunk.FP2, sdkdelivery.SourcePrefix2) {
+			// The indexer's next write names one: said once, as every wait for a write (ADR 0544 §4).
+			why := trunk.None2
+			if why == "" {
+				why = "the indexer wrote no fingerprint of the new form"
+			}
+			r.waits = fmt.Sprintf("%v: %s at %.8s has no fingerprint to take a build by: %s", errWaitsForAWrite, m,
+				trunk.Commit, why)
 			out = append(out, r)
 			continue
 		}
@@ -253,9 +266,23 @@ func (s takeInSteps) batch(ctx context.Context, keys []delivered, memo map[strin
 // doneKeysSay says a line of the take-ins; a variable so a test may hear it.
 var doneKeysSay = func(s string) { fmt.Println(s) }
 
-// doneKeyRetry is how long the bus waits before it delivers again a key whose take-in was refused.
+// doneKeyRetry is how long the bus waits before it delivers again a key whose take-in was refused the first time; each
+// refusal after doubles it, up to doneKeyRetryMost, so a key refused for long (a manifest word this controller does not
+// read yet, a node whose composition keeps failing) costs a take-in every few minutes, not every half minute.
 var doneKeyRetry = 30 * time.Second
 
+// doneKeyRetryMost is the longest a refused key waits before it is delivered again.
+const doneKeyRetryMost = 10 * time.Minute
+
+// refusalDelay is how long the bus waits before it delivers again a key refused on its deliveries-th delivery.
+func refusalDelay(deliveries uint64) time.Duration {
+	d := doneKeyRetry
+	for i := uint64(1); i < deliveries && d < doneKeyRetryMost; i++ {
+		d *= 2
+	}
+	return min(d, doneKeyRetryMost)
+}
+
 // kindNotComposed is the condition of a module's new fingerprint whose compositions or grant are not written: until
 // they are, no want can name it.
 const kindNotComposed = "fingerprint-not-composed"