Share the controller's stores and take in at most 16 done keys at once, so a burst of builds cannot fill the store (issue 1277, issue 1945) #463

2026-10-25T15:26:31Z

mesh-tools commented

2026-10-25 15:26:31 +00:00

Issue 1277 (met again, a second kind) and issue 1945.

The store on novox (max_connections 100) refused every new connection from about 16:08 to 16:18 today. At 16:07:36 the builder rewrote about 120 done keys. watchDoneKeys started a take-in per key with no cap (only the start pass was capped at 16), and each take-in's forge-address read (restOfTheMeshOf) opened a new licences pool and a new identity pool. store.Ready's 250 ms pings and the retry loops kept the store full. A restart at 16:18 (on the operator's yes) freed it, and it refilled to 88 of 100 within four minutes.

Shared stores (stores.go, compose_sdk.go, push.go): serve opens its stores once and lends them; Identity and Licences open each store once under a mutex. restOfTheMeshOf reads on the shared stores in serve; a one-shot verb process still opens its own.
One cap (composed_keys.go): takeInSlots, 16 slots shared by the watch's take-ins and the start pass. A take-in waiting for its next retry holds no slot.
Back-off on a full store (internal/store/store.go): on SQLSTATE 53300 Ready's wait doubles from 250 ms to 5 s, and the journal says once that the store is full.

Tests, each written first and seen to fail: composed_keys_cap_test.go (110 at once before, 16 after), shared_stores_test.go (120 opens before, 1 after), internal/store/ready_test.go.

Checked: gofmt, go vet with and without -tags integration, the unit suite with -race. No try: the operator decided no checks run during the recovery (issue 1945 note 27). This goes out before fix/r6-mirror-on-change, the same module.

Follows as its own pull request: fix/store-forge-address (one shared reading of the forge's address across take-ins).

Reviewed-by: reviewer-store@shanks at e9a885cda9

Issue 1277 (met again, a second kind) and issue 1945. The store on novox (max_connections 100) refused every new connection from about 16:08 to 16:18 today. At 16:07:36 the builder rewrote about 120 done keys. watchDoneKeys started a take-in per key with no cap (only the start pass was capped at 16), and each take-in's forge-address read (restOfTheMeshOf) opened a new licences pool and a new identity pool. store.Ready's 250 ms pings and the retry loops kept the store full. A restart at 16:18 (on the operator's yes) freed it, and it refilled to 88 of 100 within four minutes. - **Shared stores** (stores.go, compose_sdk.go, push.go): serve opens its stores once and lends them; Identity and Licences open each store once under a mutex. restOfTheMeshOf reads on the shared stores in serve; a one-shot verb process still opens its own. - **One cap** (composed_keys.go): takeInSlots, 16 slots shared by the watch's take-ins and the start pass. A take-in waiting for its next retry holds no slot. - **Back-off on a full store** (internal/store/store.go): on SQLSTATE 53300 Ready's wait doubles from 250 ms to 5 s, and the journal says once that the store is full. Tests, each written first and seen to fail: composed_keys_cap_test.go (110 at once before, 16 after), shared_stores_test.go (120 opens before, 1 after), internal/store/ready_test.go. Checked: gofmt, go vet with and without -tags integration, the unit suite with -race. No try: the operator decided no checks run during the recovery (issue 1945 note 27). This goes out before fix/r6-mirror-on-change, the same module. Follows as its own pull request: fix/store-forge-address (one shared reading of the forge's address across take-ins). Reviewed-by: reviewer-store@shanks at e9a885cda925cd34f11671bcb5e7075648729658

mesh-tools added 1 commit 2026-10-25 15:26:31 +00:00

Share the controller's stores with its take-ins and cap take-ins at 16

mesh/check checking e9a885cda925 on 62802cd96602

mesh/delivery held for a person: merged without a passing check: only a person decides what follows

Details

e9a885cda9

At 16:07:36 the builder rewrote about 120 done keys at once. Each woke a
take-in of its own with no limit, and each take-in read the rest of the
mesh through restOfTheMeshOf, which opened a licences and an identity
pool per call. novox's store ran out of connections at 16:08 and refused
everyone, the controller's retries keeping it full.

- The serving controller lends its stores by inventory (shareStores);
  restOfTheMeshOf reads on them instead of opening pools per call, and a
  verb run as its own process still opens its own. stores opens each
  store once under a lock, since readers now share it. serve uses the
  shared identity store instead of a second pool.
- The watch's take-ins and the start's look share one cap of 16 take-ins
  at once (takeInSlots); a take-in waiting for its next try holds none.
- store.Ready backs off on a full store (SQLSTATE 53300), doubling its
  wait up to 5 s, and says once that the store is full.

Issues 1945 and 1277.

mesh-tools commented

2026-10-25 15:26:34 +00:00

Delivery novox/mesh-controller@e9a885cda925 — held since 2026-10-25T15:30:01Z

Held for a person: merged without a passing check: only a person decides what follows — the merge reached the trunk without a passing check.

Source — source.main moved to novox/mesh-controller@62802cd966. Whether it reached the machines is mesh-delivery-view's, derived from source.main, the wants and the haves.

Transitions

2026-10-25 15:26 (new) → proposed (announced): novox/mesh-controller#463's head announced
2026-10-25 15:30 proposed → held (merged-unchecked): merged without a passing check — a verdict known with it is taken first: only a person decides what follows

The commit's note under refs/notes/mesh-plan keeps every transition of this change up to its merge: git log --notes=mesh-plan.

**Delivery** `novox/mesh-controller@e9a885cda925` — **held** since 2026-10-25T15:30:01Z Held for a person: merged without a passing check: only a person decides what follows — the merge reached the trunk without a passing check. **Source** — source.main moved to novox/mesh-controller@62802cd96602. Whether it reached the machines is `mesh-delivery-view`'s, derived from `source.main`, the wants and the haves. **Transitions** - 2026-10-25 15:26 (new) → proposed (announced): novox/mesh-controller#463's head announced - 2026-10-25 15:30 proposed → held (merged-unchecked): merged without a passing check — a verdict known with it is taken first: only a person decides what follows The commit's note under `refs/notes/mesh-plan` keeps every transition of this change up to its merge: `git log --notes=mesh-plan`.

jschoubben merged commit 62802cd966 into main

2026-10-25 15:29:58 +00:00

jschoubben deleted branch fix/store-pool-flood

2026-10-25 15:29:58 +00:00

jschoubben referenced this issue from a commit

2026-10-25 15:36:52 +00:00

Count the take-ins and every other background work in the serving group, so serve's stop waits for them too (issue 1945)

mesh-tools referenced this pull request

2026-10-25 16:11:44 +00:00

Compose the packages mirror only on a change, end every serve goroutine before serve returns, and raise no alarm for a cancellation (issue 1945) #464

Sign in to join this conversation.