Share the controller's stores with its take-ins and cap take-ins at 16
mesh/delivery held for a person: merged without a passing check: only a person decides what follows
mesh/check checking e9a885cda925 on 62802cd96602
mesh/delivery held for a person: merged without a passing check: only a person decides what follows
mesh/check checking e9a885cda925 on 62802cd96602
At 16:07:36 the builder rewrote about 120 done keys at once. Each woke a take-in of its own with no limit, and each take-in read the rest of the mesh through restOfTheMeshOf, which opened a licences and an identity pool per call. novox's store ran out of connections at 16:08 and refused everyone, the controller's retries keeping it full. - The serving controller lends its stores by inventory (shareStores); restOfTheMeshOf reads on them instead of opening pools per call, and a verb run as its own process still opens its own. stores opens each store once under a lock, since readers now share it. serve uses the shared identity store instead of a second pool. - The watch's take-ins and the start's look share one cap of 16 take-ins at once (takeInSlots); a take-in waiting for its next try holds none. - store.Ready backs off on a full store (SQLSTATE 53300), doubling its wait up to 5 s, and says once that the store is full. Issues 1945 and 1277.
This commit is contained in:
@@ -163,7 +163,7 @@ func followDoneKey(ctx context.Context, open *stores, module string) {
|
||||
saidWait := ""
|
||||
for {
|
||||
doneKeysLookAgain.Delete(module) // what this look reads covers every wake before it
|
||||
err := doneKeyTakeOnce(ctx, open, module)
|
||||
err := takeOnceInSlot(ctx, open, module)
|
||||
o := doneKeyObservation(module)
|
||||
if err == nil {
|
||||
if keeper := conditionsFrom; keeper != nil && failed {
|
||||
@@ -348,9 +348,25 @@ func doneKeysAtStart(ctx context.Context, modules func(context.Context) ([]strin
|
||||
wg.Wait()
|
||||
}
|
||||
|
||||
// startTakeIns is how many take-ins the start's look runs at once.
|
||||
// startTakeIns is how many take-ins the start's look runs at once, and how many run at once in all.
|
||||
const startTakeIns = 16
|
||||
|
||||
// takeInSlots are the take-ins that may run at once in this process, the watch's and the start's look's together: a
|
||||
// burst of done keys (about 120 at 16:07:36 on 2026-10-25) once started a take-in each, all at once, and their reads
|
||||
// filled the store until it refused every connection (issues 1945 and 1277).
|
||||
var takeInSlots = make(chan struct{}, startTakeIns)
|
||||
|
||||
// takeOnceInSlot is one take-in once a slot is free; a take-in that waits for its next try holds none.
|
||||
func takeOnceInSlot(ctx context.Context, open *stores, module string) error {
|
||||
select {
|
||||
case takeInSlots <- struct{}{}:
|
||||
case <-ctx.Done():
|
||||
return ctx.Err()
|
||||
}
|
||||
defer func() { <-takeInSlots }()
|
||||
return doneKeyTakeOnce(ctx, open, module)
|
||||
}
|
||||
|
||||
// takeDoneKeyAtStart is one take-in of the start's look, under the module's lock; one that fails is handed to takeDoneKey,
|
||||
// which tries again every doneKeyRetry, loudly, so a failing module never holds up the look at the others.
|
||||
func takeDoneKeyAtStart(ctx context.Context, open *stores, module string) {
|
||||
@@ -361,7 +377,7 @@ func takeDoneKeyAtStart(ctx context.Context, open *stores, module string) {
|
||||
return
|
||||
}
|
||||
doneKeysLookAgain.Delete(module) // what this look reads covers every wake before it
|
||||
err := doneKeyTakeOnce(ctx, open, module)
|
||||
err := takeOnceInSlot(ctx, open, module)
|
||||
mu.(*sync.Mutex).Unlock()
|
||||
// Read after letting go, as every holder of a module's lock does: a wake during this look only marked the module
|
||||
// (the review of issue 1945).
|
||||
|
||||
Reference in New Issue
Block a user