Apply through one queue, and order what is applied and reported (hq to-be 45 Phase 2) #35

2026-10-06T09:54:33Z

mesh-admin commented

2026-10-06 09:54:33 +00:00

hq to-be 45 §6, ADR 0227 rules 1 and 2. This is the host side of Phase 2. The controller side (lease and epoch, refusing older reports) is a separate PR in mesh-controller.

What changes

One apply queue (internal/link/queue.go). A delivery and the five-minute reconcile both only enqueue. One worker takes the newest declaration held when it starts (by epoch, then sequence, then arrival), sets the rest aside (each reported as set aside, then settled), applies once and makes one report. A reconcile that is due while a delivery waits is that delivery's apply. Reports leave in the order they are made, from one place. This removes the race class of issues 257, 261 and 267, and replaces the outbox and overtaken digest check.
The order on the wire (link.Order). A declaration may carry epoch beside sequence, inside what is signed. A report carries back the declaration's epoch and sequence, plus report_sequence, older_than and refused_older.
Refusing an older epoch. In the host's own apply path, before the machine is read: an older epoch is refused, and so is a lower sequence under the same epoch. Both sides must claim an epoch, otherwise today's behaviour stands. Each refusal is counted (kept on disk), gets one log line, and is reported naming what was refused and what is held. It does not replace the kept unsaid account. The same rule applies to mesh-host reconcile.
The report sequence is kept on disk (numbers.json beside the state), so it goes on increasing across restarts and self-updates. If it cannot be read, the host says so and continues from the current time in ms. It never restarts from 1.
What is kept: the unsaid report (264) is said first on every link, exactly as it was made, with its original report sequence. A link is not let go while a delivery is in hand, so an apply that stands aside is still reported on its link (264). After standing aside nothing further is applied, and the process waits for the worker before it exits.
Smaller changes: a host that starts enqueues a reconcile, so schedules are armed at start rather than after 5 minutes. A set-aside report now names the declaration by its real digest (before, it was always empty). A delivery that arrives twice while the worker is busy is settled without a set-aside report. The link keeps sending heartbeats during long applies.

Compatibility

New host with an old controller: no epoch means no epoch refusal and no new refusal of a lower sequence on the link. Reports carry extra keys, which the old controller ignores.
Old host with a new controller: an old host decodes the declaration strictly and refuses epoch. The controller must send epoch only to a machine whose latest report carries report_sequence > 0. That key is how the host says it reads epoch.
The controller's D1 probe imports validate from this repo, so it needs this version to accept epoch.

Tests

Replays as tests: a reconcile that is due during a delivery; a reconcile running when a delivery arrives; two deliveries in quick succession; deliveries while the worker is busy; a self-update during a report; an older epoch refused, counted and said; a restart with an unsaid report; unreadable numbers; a reconcile report made offline. Plus a wire contract test, Older/Supersedes tables, and a NATS integration test (TestNatsTheQueueAppliesTheNewestOnceAndReportsInOrder). make check is green and go test -race ./... passes. The NATS tests also pass against nats 2.11.

Not in this PR: the report verb (it needs bus grants; H1 is Phase 3), and the Phase 2 contract tests and lint.

hq to-be 45 §6, ADR 0227 rules 1 and 2. This is the host side of Phase 2. The controller side (lease and epoch, refusing older reports) is a separate PR in mesh-controller. ## What changes - **One apply queue** (`internal/link/queue.go`). A delivery and the five-minute reconcile both only *enqueue*. One worker takes the newest declaration held when it starts (by epoch, then sequence, then arrival), sets the rest aside (each reported as set aside, then settled), applies once and makes one report. A reconcile that is due while a delivery waits is that delivery's apply. Reports leave in the order they are made, from one place. This removes the race class of issues 257, 261 and 267, and replaces the `outbox` and `overtaken` digest check. - **The order on the wire** (`link.Order`). A declaration may carry `epoch` beside `sequence`, inside what is signed. A report carries back the declaration's `epoch` and `sequence`, plus `report_sequence`, `older_than` and `refused_older`. - **Refusing an older epoch.** In the host's own apply path, before the machine is read: an older epoch is refused, and so is a lower sequence under the same epoch. Both sides must claim an epoch, otherwise today's behaviour stands. Each refusal is counted (kept on disk), gets one log line, and is reported naming what was refused and what is held. It does not replace the kept unsaid account. The same rule applies to `mesh-host reconcile`. - **The report sequence is kept on disk** (`numbers.json` beside the state), so it goes on increasing across restarts and self-updates. If it cannot be read, the host says so and continues from the current time in ms. It never restarts from 1. - **What is kept:** the unsaid report (264) is said first on every link, exactly as it was made, with its original report sequence. A link is not let go while a *delivery* is in hand, so an apply that stands aside is still reported on its link (264). After standing aside nothing further is applied, and the process waits for the worker before it exits. - **Smaller changes:** a host that starts enqueues a reconcile, so schedules are armed at start rather than after 5 minutes. A set-aside report now names the declaration by its real digest (before, it was always empty). A delivery that arrives twice while the worker is busy is settled without a set-aside report. The link keeps sending heartbeats during long applies. ## Compatibility - New host with an old controller: no epoch means no epoch refusal and no new refusal of a lower sequence on the link. Reports carry extra keys, which the old controller ignores. - **Old host with a new controller: an old host decodes the declaration strictly and refuses `epoch`.** The controller must send `epoch` only to a machine whose latest report carries `report_sequence` > 0. That key is how the host says it reads `epoch`. - The controller's D1 probe imports `validate` from this repo, so it needs this version to accept `epoch`. ## Tests Replays as tests: a reconcile that is due during a delivery; a reconcile running when a delivery arrives; two deliveries in quick succession; deliveries while the worker is busy; a self-update during a report; an older epoch refused, counted and said; a restart with an unsaid report; unreadable numbers; a reconcile report made offline. Plus a wire contract test, `Older`/`Supersedes` tables, and a NATS integration test (`TestNatsTheQueueAppliesTheNewestOnceAndReportsInOrder`). `make check` is green and `go test -race ./...` passes. The NATS tests also pass against nats 2.11. Not in this PR: the `report` verb (it needs bus grants; H1 is Phase 3), and the Phase 2 contract tests and lint.

mesh-admin added 1 commit 2026-10-06 09:54:33 +00:00

Apply through one queue, and order what is applied and reported (hq to-be 45 Phase 2) 31804bd8c2

A delivery and the five-minute reconcile were two paths that applied, ordered only by a lock, and
each order it allowed was met live (issues 257, 261, 267). Now both only enqueue: one worker takes
the newest declaration held when it starts, applies it once and makes one report, and reports leave
in the order they are made.

A declaration may carry the controller's lease epoch beside its sequence; one older than what this
node applied is refused before anything is touched, counted, logged and reported. A report carries
the declaration's epoch and sequence and the host's own report sequence, kept on disk so it goes on
increasing across restarts and self-updates. Without an epoch, today's behaviour stands.

mesh-admin merged commit 3e80b7ae32 into main

2026-10-06 09:55:19 +00:00

mesh-admin deleted branch feat/a-core-that-cannot-fail-silently-phase-2

2026-10-06 09:55:19 +00:00

mesh-admin commented

2026-10-06 23:10:37 +00:00

Delivery novox/mesh-host@3e80b7ae325e — held since 2026-10-06T23:05:28Z

Held for a person: merged without a passing check: only a person decides that it goes on — mesh-delivery.release with why lets it go on.

Delivery plan — builds mesh-host → ace, g14, novox, shanks; no bus step

tier 0: mesh-host
ace: receives mesh-host
g14: receives mesh-host
novox: receives mesh-host
shanks: receives mesh-host

Transitions

2026-10-06 23:05 (new) → proposed (announced): novox/mesh-host#35 merged; its head was not heard by this owner, so it is made from the forge's word
2026-10-06 23:05 proposed → held (merged-unchecked): merged without a passing check — a verdict known with it is taken first: only a person decides that it goes on

The commit's note under refs/notes/mesh-plan keeps every transition: git log --notes=mesh-plan.

**Delivery** `novox/mesh-host@3e80b7ae325e` — **held** since 2026-10-06T23:05:28Z Held for a person: merged without a passing check: only a person decides that it goes on — `mesh-delivery.release` with why lets it go on. **Delivery plan** — builds mesh-host → ace, g14, novox, shanks; no bus step - tier 0: mesh-host - ace: receives mesh-host - g14: receives mesh-host - novox: receives mesh-host - shanks: receives mesh-host **Transitions** - 2026-10-06 23:05 (new) → proposed (announced): novox/mesh-host#35 merged; its head was not heard by this owner, so it is made from the forge's word - 2026-10-06 23:05 proposed → held (merged-unchecked): merged without a passing check — a verdict known with it is taken first: only a person decides that it goes on The commit's note under `refs/notes/mesh-plan` keeps every transition: `git log --notes=mesh-plan`.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: novox/mesh-host#35