Let a planned maintenance window fail no apply (hq issue 291) #49

Merged

mesh-admin merged 1 commits from fix/a-planned-window-fails-no-apply into main 2026-10-07 11:20:20 +00:00

Conversation 2 Commits 1 Files Changed 8

+559 -9

mesh-admin commented 2026-10-07 11:16:02 +00:00

Contributor

Copy Link

Copy Source

hq issue 291 (gap found live 2026-10-07): at 03:30 distribution.collect held mesh-registry still (~12 s, ADR 0189 window) while the node-engine's reconcile on the same machine was already in flight; that apply then fetched every bundle blob from the store and failed each with dial tcp …:5100: connect: connection refused (postgres, keycloak, minio, gitea, mesh-vault, node-tools, …), holding the machine until its next pass. Other machines can meet the same window.

Approach (the simplest that is right on every machine)

• Order on the store's machine (maintenance_window.go, schedule.go): a scheduled step opens its window only once no apply is in flight here — it polls the apply lock (store.TryLockIn, new) and holds it only while writing the window record, so an apply that started before the window never finds the server stopped under it, and a push arriving mid-window still never queues behind the window (issue 224's reason not to take the lock). Bounded (15 min); past it the window opens beside the apply, said.
• An apply during a window waits instead of failing (store_away.go, archive + process fetches): a fetch the store does not answer (refused, reset, EOF, timeout, 502/503/504) waits while a window is open on this machine (the record every apply there already reads; polled, ≤ 20 min) and fetches again.
• Elsewhere the store's window is not published: such a fetch is retried with backoff within one budget per apply (3 min, many times the measured window; shared by every fetch, so a store really down costs one bounded wait, not one per bundle). The engine's next pass retries anything still failed, so the bound only decides whether one pass fails.
• An answer (404, wrong bytes) still fails at once.

Tests (store_away_test.go, the 03:30 sequence against a pretend registry/runtime and a pretend store whose address refuses while held still): an apply on the store's machine during the window waits and applies the bundle once it closes; a collector due while an apply holds the lock opens its window only after that apply ended (stop/step/start in order); an apply elsewhere waits for a store that comes back; six bundles from a store that stays down cost one bounded wait; a 404 is asked once and fails at once. Full suite + -race on the new tests pass.

mesh-controller check-here: merge gate PASS (4 of 4 compose), repo check PASS.

No change is needed in modules/distribution: the window, its holds and its step are already declared there; the node-engine now honours them in its own order.

hq issue 291 (gap found live 2026-10-07): at 03:30 `distribution.collect` held `mesh-registry` still (~12 s, ADR 0189 window) while the node-engine's reconcile **on the same machine** was already in flight; that apply then fetched every bundle blob from the store and failed each with `dial tcp …:5100: connect: connection refused` (postgres, keycloak, minio, gitea, mesh-vault, node-tools, …), holding the machine until its next pass. Other machines can meet the same window. **Approach (the simplest that is right on every machine)** - **Order on the store's machine** (`maintenance_window.go`, `schedule.go`): a scheduled step opens its window only once no apply is in flight here — it polls the apply lock (`store.TryLockIn`, new) and holds it only while writing the window record, so an apply that started before the window never finds the server stopped under it, and a push arriving mid-window still never queues behind the window (issue 224's reason not to take the lock). Bounded (15 min); past it the window opens beside the apply, said. - **An apply during a window waits instead of failing** (`store_away.go`, archive + process fetches): a fetch the store does *not answer* (refused, reset, EOF, timeout, 502/503/504) waits while a window is open on this machine (the record every apply there already reads; polled, ≤ 20 min) and fetches again. - **Elsewhere** the store's window is not published: such a fetch is retried with backoff within **one budget per apply** (3 min, many times the measured window; shared by every fetch, so a store really down costs one bounded wait, not one per bundle). The engine's next pass retries anything still failed, so the bound only decides whether one pass fails. - An answer (404, wrong bytes) still fails at once. **Tests** (`store_away_test.go`, the 03:30 sequence against a pretend registry/runtime and a pretend store whose address refuses while held still): an apply on the store's machine during the window waits and applies the bundle once it closes; a collector due while an apply holds the lock opens its window only after that apply ended (stop/step/start in order); an apply elsewhere waits for a store that comes back; six bundles from a store that stays down cost one bounded wait; a 404 is asked once and fails at once. Full suite + `-race` on the new tests pass. `mesh-controller check-here`: merge gate PASS (4 of 4 compose), repo check PASS. No change is needed in modules/distribution: the window, its holds and its step are already declared there; the node-engine now honours them in its own order.

mesh-admin added 1 commit 2026-10-07 11:16:02 +00:00

Let a planned maintenance window fail no apply (hq issue 291) ... 3c1ac6aef2

At 03:30 the store's collector held the registry still while the
node-engine's reconcile on that machine was fetching bundle blobs from
it: every archive failed 'connection refused' and the machine was held
until the next pass. Other machines can meet the same window.

A scheduled step now opens its window only once no apply is in flight
here (the apply lock is taken just to write the record, so a push still
never queues behind the window). An apply whose fetch the store does
not answer waits for a window open on its own machine to close, and
elsewhere retries with backoff within one bounded budget per apply,
well past the window's length; an answer such as 404 still fails at
once.

mesh-admin commented 2026-10-07 11:16:31 +00:00

Author

Contributor

Copy Link

Copy Source

Delivery novox/mesh-host@3c1ac6aef2a6 — delivered since 2026-10-07T11:24:45Z

Delivery plan — builds mesh-host → ace, g14, novox, shanks; no bus step

• tier 0: mesh-host
• ace: receives mesh-host
• g14: receives mesh-host
• novox: receives mesh-host
• shanks: receives mesh-host

Machines

• mesh-host on ace: passed — healthy 3 times over 2m29s
• mesh-host on novox: judging — carried by plan-1791372045144630550
• mesh-host on the rest running it: sent

Transitions

• 2026-10-07 11:16 (new) → proposed (announced): novox/mesh-host#49's head announced
• 2026-10-07 11:18 proposed → checked (checked): the verdict names this commit
• 2026-10-07 11:18 checked → ready (accepted): the gate passed or warned, and the repository's own check did not fail
• 2026-10-07 11:20 ready → published (merged): on the trunk its modules follow: merged there, and its walk opened — or nothing for a walk to move
• 2026-10-07 11:20 published → delivering (go): its walk started: let go by this owner in its group's order, by a person, or on the controller's own path
• 2026-10-07 11:24 delivering → delivered (done): its walk is done: every machine of its deploy plan passed, or was left as its policy says

The commit's note under refs/notes/mesh-plan keeps every transition: git log --notes=mesh-plan.

<!-- mesh-delivery:view --> **Delivery** `novox/mesh-host@3c1ac6aef2a6` — **delivered** since 2026-10-07T11:24:45Z **Delivery plan** — builds mesh-host → ace, g14, novox, shanks; no bus step - tier 0: mesh-host - ace: receives mesh-host - g14: receives mesh-host - novox: receives mesh-host - shanks: receives mesh-host **Machines** - mesh-host on ace: passed — healthy 3 times over 2m29s - mesh-host on novox: judging — carried by plan-1791372045144630550 - mesh-host on the rest running it: sent **Transitions** - 2026-10-07 11:16 (new) → proposed (announced): novox/mesh-host#49's head announced - 2026-10-07 11:18 proposed → checked (checked): the verdict names this commit - 2026-10-07 11:18 checked → ready (accepted): the gate passed or warned, and the repository's own check did not fail - 2026-10-07 11:20 ready → published (merged): on the trunk its modules follow: merged there, and its walk opened — or nothing for a walk to move - 2026-10-07 11:20 published → delivering (go): its walk started: let go by this owner in its group's order, by a person, or on the controller's own path - 2026-10-07 11:24 delivering → delivered (done): its walk is done: every machine of its deploy plan passed, or was left as its policy says The commit's note under `refs/notes/mesh-plan` keeps every transition: `git log --notes=mesh-plan`.

mesh-admin commented 2026-10-07 11:18:09 +00:00

Author

Contributor

Copy Link

Copy Source

Merge check at 3c1ac6ae

• mesh/merge-gate: PASS — every machine composes with the change as it did without (4 of 4 compose) (modules: mesh-host)
• mesh/repo-check: PASS — its merge-check.sh passed

Change plan — builds mesh-host → ace, g14, novox, shanks; no bus step

• tier 0: mesh-host
• ace: receives mesh-host
• g14: receives mesh-host
• novox: receives mesh-host
• shanks: receives mesh-host

Run by the build seat on ace as build-1791371773125500548 (builds --log build-1791371773125500548).

--- the gate: mesh-host
mesh-host: ok
1 manifest(s) checked. Judged against the seats this binary carries; a claim on one of the mesh's own seats is judged fully at registration, and a seat declared by a module not given here reads as unknown. Identities judged on a 6-character machine name, against the bounds of the providers given here
judging 0 module(s) changed against 4 machine(s), as the snapshot of 2026-10-07T01:26:48Z says they run
composing every machine as the mesh is
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
composing every machine with the change
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
merge gate: PASS — every machine composes with the change as it did without (4 of 4 compose)
judged by controller development build against the facts of 2026-10-07T01:26:48Z

machines:
  - a machine (f35): composes
  - holds mesh-dns-resolver (idt): composes
  - holds git, holds mesh-artifact-store, holds mesh-broker, holds mesh-catalog, holds mesh-dns-resolver, holds mesh-store, holds mesh-vault, holds npm-package-registry, the control node, the hub (ygcyr): composes
  - a machine (yvbpsm): composes

a merge would rebuild 1 module(s) in 1 tier(s): mesh-host
--- its merge-check.sh (go toolchain)
ok  	github.com/novox/mesh-host/cmd/mesh-bootstrap	1.016s
ok  	github.com/novox/mesh-host/cmd/mesh-host	1.310s
ok  	github.com/novox/mesh-host/examples	1.022s
ok  	github.com/novox/mesh-host/internal/apply	4.392s
ok  	github.com/novox/mesh-host/internal/bootstrap	1.521s
ok  	github.com/novox/mesh-host/internal/bundle	1.033s
ok  	github.com/novox/mesh-host/internal/declaration	1.058s
ok  	github.com/novox/mesh-host/internal/firewall	1.637s
ok  	github.com/novox/mesh-host/internal/identity	1.118s
ok  	github.com/novox/mesh-host/internal/image	1.035s
ok  	github.com/novox/mesh-host/internal/inventory	1.031s
ok  	github.com/novox/mesh-host/internal/link	10.336s
ok  	github.com/novox/mesh-host/internal/liveness	1.058s
ok  	github.com/novox/mesh-host/internal/outward	1.040s
ok  	github.com/novox/mesh-host/internal/profile	1.494s
ok  	github.com/novox/mesh-host/internal/reachable	1.037s
ok  	github.com/novox/mesh-host/internal/store	1.231s
ok  	github.com/novox/mesh-host/internal/system	1.039s
ok  	github.com/novox/mesh-host/internal/tunnel	1.042s
ok  	github.com/novox/mesh-host/internal/upgrade	1.147s
ok  	github.com/novox/mesh-host/internal/witness	4.617s
ok  	github.com/novox/mesh-host/packaging	1.034s
ok  	github.com/novox/mesh-host/validate	1.032s

**Merge check** at `3c1ac6ae` - `mesh/merge-gate`: **PASS** — every machine composes with the change as it did without (4 of 4 compose) (modules: mesh-host) - `mesh/repo-check`: **PASS** — its merge-check.sh passed **Change plan** — builds mesh-host → ace, g14, novox, shanks; no bus step - tier 0: mesh-host - ace: receives mesh-host - g14: receives mesh-host - novox: receives mesh-host - shanks: receives mesh-host Run by the build seat on ace as `build-1791371773125500548` (`builds --log build-1791371773125500548`). ``` --- the gate: mesh-host mesh-host: ok 1 manifest(s) checked. Judged against the seats this binary carries; a claim on one of the mesh's own seats is judged fully at registration, and a seat declared by a module not given here reads as unknown. Identities judged on a 6-character machine name, against the bounds of the providers given here judging 0 module(s) changed against 4 machine(s), as the snapshot of 2026-10-07T01:26:48Z says they run composing every machine as the mesh is the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued composing every machine with the change the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued merge gate: PASS — every machine composes with the change as it did without (4 of 4 compose) judged by controller development build against the facts of 2026-10-07T01:26:48Z machines: - a machine (f35): composes - holds mesh-dns-resolver (idt): composes - holds git, holds mesh-artifact-store, holds mesh-broker, holds mesh-catalog, holds mesh-dns-resolver, holds mesh-store, holds mesh-vault, holds npm-package-registry, the control node, the hub (ygcyr): composes - a machine (yvbpsm): composes a merge would rebuild 1 module(s) in 1 tier(s): mesh-host --- its merge-check.sh (go toolchain) ok github.com/novox/mesh-host/cmd/mesh-bootstrap 1.016s ok github.com/novox/mesh-host/cmd/mesh-host 1.310s ok github.com/novox/mesh-host/examples 1.022s ok github.com/novox/mesh-host/internal/apply 4.392s ok github.com/novox/mesh-host/internal/bootstrap 1.521s ok github.com/novox/mesh-host/internal/bundle 1.033s ok github.com/novox/mesh-host/internal/declaration 1.058s ok github.com/novox/mesh-host/internal/firewall 1.637s ok github.com/novox/mesh-host/internal/identity 1.118s ok github.com/novox/mesh-host/internal/image 1.035s ok github.com/novox/mesh-host/internal/inventory 1.031s ok github.com/novox/mesh-host/internal/link 10.336s ok github.com/novox/mesh-host/internal/liveness 1.058s ok github.com/novox/mesh-host/internal/outward 1.040s ok github.com/novox/mesh-host/internal/profile 1.494s ok github.com/novox/mesh-host/internal/reachable 1.037s ok github.com/novox/mesh-host/internal/store 1.231s ok github.com/novox/mesh-host/internal/system 1.039s ok github.com/novox/mesh-host/internal/tunnel 1.042s ok github.com/novox/mesh-host/internal/upgrade 1.147s ok github.com/novox/mesh-host/internal/witness 4.617s ok github.com/novox/mesh-host/packaging 1.034s ok github.com/novox/mesh-host/validate 1.032s ```

mesh-admin merged commit 41f908b803 into main 2026-10-07 11:20:20 +00:00

mesh-admin deleted branch fix/a-planned-window-fails-no-apply 2026-10-07 11:20:20 +00:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No labels

Milestone

No items

No Milestone

Assignees

Clear assignees

jschoubben miel.truyen

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: novox/mesh-host#49