Keep calls and hand acts on the bus, answer status at once, record durations (hq to-be 45 Phase 0) #78

2026-10-06T01:04:26Z

mesh-admin commented

2026-10-06 01:04:26 +00:00

Phase 0 of hq to-be 45 (ADR 0227), the mesh-controller row, plus hq issue 269.

What it delivers

Durable calls (§6). Every call is written to the key-value bucket mesh-controller_calls, the controller's alone: record under <id>, answer under <id>.answer (bounded to 64 KiB, cut and said beyond). The stream under the bucket keeps 2000 messages (= 1000 calls) and 14 days. Writes go through one goroutine, retried, never blocking a call; a bus that does not take them is said. calls lists memory + bucket; calls <id> reads the bucket when this process did not serve it. A starting controller marks a previous controller's running calls abandoned and says so. Each call records its caller (the principal named by the reply inbox) and its holder process.

status within ten seconds (D9, §8). The serving controller composes status --json at start, after news from a machine (the listener nudges), after a build is registered, after any verb that acts, and every minute; the status verb answers the last composition at once with composed, age, composedIn, and lastAttemptFailed when a later attempt failed. Composing also resolves each machine once rather than twice (wouldSendFrom reuses the plans the unheld/overflow loop computed). The CLI status at a shell still composes when asked.

The hand-act log (§7), bucket mesh-controller_hand-acts (90 days). --why/--cause/--condition on: push (required through the seat and through command; recorded when given at a shell, because the installer and the lab push by command line), plans stop|close (required everywhere), broker consumer-reset (required), and new hand-act record <what> --why --cause [--condition]. New hand-acts [--days N] lists who/what/why/cause and the causes repeated within a fortnight (the S15 signal). status shows the week's count (handActsThisWeek). The entry is written before the act and never instead of it: an unwritable log is said loudly and the repair still happens.

Durations (migration 0066, highest on main and open branches was 0065): table duration and release_plan.tier_entered. Recorded as heard: apply (send → first report of that send, per machine, once per send), heartbeat-gap (previous word → this word, per machine), plan-tier (measured in SavePlan's transaction when a plan leaves a tier or ends, per repository), build (asked → outcome, per module). durations [--kind] [--days] [--json|--all] summarises count/median/p90/max with the S1/S2 bound each would give. Kept 30 days, pruned daily.

Seat verbs: push gains why (required) and cause; plans gains why/cause; new hand-act, hand-acts, durations.

The seat row takes this binary's definition of its own verbs. The live console describes push as {node} and plan without files (checked tonight with mesh_describe): SeedSeats only ever added verbs by name, so the row kept an older build's schemas and the console judges calls against the row — behind, files and now why were unreachable. Re-seeding now replaces a controller verb the binary defines; a verb only the row has (a newer build's, ADR 0185) is kept.

Issue 269: the controller is granted $KV.<cancelled set>.> for each work queue it asks. Reproduced against a nats 2.11 server holding exactly the controller's composed grant: MarkCancelled timed out before, succeeds after; the same for the two new buckets.

Tests

gofmt -l . clean, go vet ./... clean, go test -p 1 ./... with MESH_TEST_POSTGRES (postgres:17) and MESH_TEST_NATS (nats:2.11) — 20/20 packages ok; new code also under -race. New: TestNatsACallsOutcomeOutlivesItsController, TestNatsTheCallsBucketIsBounded, TestNatsAnAnswerLargerThanTheBoundIsCut, TestTheCallerIsTheInboxsPrincipal, TestNatsAnActByHandIsKeptWithWhyAndARepeatIsFound, TestStatusAnswersAtOnceHoweverLongComposingTakes (five in a row while every composition after the first hangs), TestStatusBeforeItsFirstCompositionSaysSo, TestANudgeComposesAgainAndAFailureKeepsTheLastSummary, TestTheStatusVerbAnswersFromTheSummary, TestARepairByHandWithoutAReasonIsRefused, TestARepairByHandCarriesItsReason, TestAnApplyIsMeasuredOncePerSend, TestASilenceIsMeasuredFromTheLastWord, TestAPlansTierIsMeasuredWhenItIsLeft, TestDurationsAreSummarisedPerSubject, TestTheControllersVerbsInTheRowAreTheBinarys, TestTheControllerMayWriteEveryBucketItWrites; golden composition updated.

Measured

Before, live 2026-10-06 02:33:49: mesh-controller.status answered "running" at 10 s and finished at 18.6 s (call-1791246829460871617-88). After: the verb reads memory; the console's round trip for in-process verbs (calls) is ~10–40 ms in the same log. The background composition's own time is in every answer as composedIn.

Merge order and rollout

Merge novox/mesh-host#32 (template only), then this.
The plan rolls the controller out; prepare runs migration 0066 and re-seeds the seat row.
Until the machine holding the bus is sent the new user list, the bus refuses the controller's bucket writes: calls are then kept in memory only and said so in the journal. Push that machine once (with why), then check calls survives a controller restart.
Record the bus's 2.10 → 2.11 step with hand-act record (Phase 0's mesh-catalog row).

Phase 0 of hq to-be 45 (ADR 0227), the mesh-controller row, plus hq issue 269. ## What it delivers **Durable `calls`** (§6). Every call is written to the key-value bucket `mesh-controller_calls`, the controller's alone: record under `<id>`, answer under `<id>.answer` (bounded to 64 KiB, cut and said beyond). The stream under the bucket keeps 2000 messages (= 1000 calls) and 14 days. Writes go through one goroutine, retried, never blocking a call; a bus that does not take them is said. `calls` lists memory + bucket; `calls <id>` reads the bucket when this process did not serve it. A starting controller marks a previous controller's `running` calls `abandoned` and says so. Each call records its caller (the principal named by the reply inbox) and its holder process. **`status` within ten seconds** (D9, §8). The serving controller composes `status --json` at start, after news from a machine (the listener nudges), after a build is registered, after any verb that acts, and every minute; the `status` verb answers the last composition at once with `composed`, `age`, `composedIn`, and `lastAttemptFailed` when a later attempt failed. Composing also resolves each machine once rather than twice (`wouldSendFrom` reuses the plans the unheld/overflow loop computed). The CLI `status` at a shell still composes when asked. **The hand-act log** (§7), bucket `mesh-controller_hand-acts` (90 days). `--why/--cause/--condition` on: `push` (required through the seat and through `command`; recorded when given at a shell, because the installer and the lab push by command line), `plans stop|close` (required everywhere), `broker consumer-reset` (required), and new `hand-act record <what> --why --cause [--condition]`. New `hand-acts [--days N]` lists who/what/why/cause and the causes repeated within a fortnight (the S15 signal). `status` shows the week's count (`handActsThisWeek`). The entry is written before the act and never instead of it: an unwritable log is said loudly and the repair still happens. **Durations** (migration **0066**, highest on main and open branches was 0065): table `duration` and `release_plan.tier_entered`. Recorded as heard: `apply` (send → first report of that send, per machine, once per send), `heartbeat-gap` (previous word → this word, per machine), `plan-tier` (measured in `SavePlan`'s transaction when a plan leaves a tier or ends, per repository), `build` (asked → outcome, per module). `durations [--kind] [--days] [--json|--all]` summarises count/median/p90/max with the S1/S2 bound each would give. Kept 30 days, pruned daily. **Seat verbs**: `push` gains `why` (required) and `cause`; `plans` gains `why`/`cause`; new `hand-act`, `hand-acts`, `durations`. **The seat row takes this binary's definition of its own verbs.** The live console describes `push` as `{node}` and `plan` without `files` (checked tonight with `mesh_describe`): `SeedSeats` only ever added verbs by name, so the row kept an older build's schemas and the console judges calls against the row — `behind`, `files` and now `why` were unreachable. Re-seeding now replaces a controller verb the binary defines; a verb only the row has (a newer build's, ADR 0185) is kept. **Issue 269**: the controller is granted `$KV.<cancelled set>.>` for each work queue it asks. Reproduced against a nats 2.11 server holding exactly the controller's composed grant: `MarkCancelled` timed out before, succeeds after; the same for the two new buckets. ## Tests `gofmt -l .` clean, `go vet ./...` clean, `go test -p 1 ./...` with `MESH_TEST_POSTGRES` (postgres:17) and `MESH_TEST_NATS` (nats:2.11) — 20/20 packages ok; new code also under `-race`. New: `TestNatsACallsOutcomeOutlivesItsController`, `TestNatsTheCallsBucketIsBounded`, `TestNatsAnAnswerLargerThanTheBoundIsCut`, `TestTheCallerIsTheInboxsPrincipal`, `TestNatsAnActByHandIsKeptWithWhyAndARepeatIsFound`, `TestStatusAnswersAtOnceHoweverLongComposingTakes` (five in a row while every composition after the first hangs), `TestStatusBeforeItsFirstCompositionSaysSo`, `TestANudgeComposesAgainAndAFailureKeepsTheLastSummary`, `TestTheStatusVerbAnswersFromTheSummary`, `TestARepairByHandWithoutAReasonIsRefused`, `TestARepairByHandCarriesItsReason`, `TestAnApplyIsMeasuredOncePerSend`, `TestASilenceIsMeasuredFromTheLastWord`, `TestAPlansTierIsMeasuredWhenItIsLeft`, `TestDurationsAreSummarisedPerSubject`, `TestTheControllersVerbsInTheRowAreTheBinarys`, `TestTheControllerMayWriteEveryBucketItWrites`; golden composition updated. ## Measured Before, live 2026-10-06 02:33:49: `mesh-controller.status` answered "running" at 10 s and finished at 18.6 s (call-1791246829460871617-88). After: the verb reads memory; the console's round trip for in-process verbs (`calls`) is ~10–40 ms in the same log. The background composition's own time is in every answer as `composedIn`. ## Merge order and rollout 1. Merge novox/mesh-host#32 (template only), then this. 2. The plan rolls the controller out; `prepare` runs migration 0066 and re-seeds the seat row. 3. Until the machine holding the bus is sent the new user list, the bus refuses the controller's bucket writes: calls are then kept in memory only and said so in the journal. Push that machine once (with `why`), then check `calls` survives a controller restart. 4. Record the bus's 2.10 → 2.11 step with `hand-act record` (Phase 0's mesh-catalog row).

mesh-admin added 2 commits 2026-10-06 01:04:26 +00:00

Keep calls and hand acts on the bus, answer status at once, record durations (hq to-be 45 Phase 0) e74c32ed50

A controller restart lost every call's outcome, `status` composed the mesh
while its caller waited (18.6s live on 2026-10-06, past the 10s window), a
repair by hand left no trace, and the core's bounds had nothing measured to
be set from.

- calls: kept in the controller's bucket mesh-controller_calls (last 1000 or
  14 days, answers bounded to 64 KiB), read by id across a restart; a
  controller starting marks a stopped one's running calls abandoned; each
  call names its caller from the inbox its answer goes to.
- status: the serving controller composes it at start, after news from a
  machine, a build or an acting verb, and every minute; the verb answers the
  last composition at once with when and how long it took. Composing resolves
  each machine once instead of twice.
- hand-act log in mesh-controller_hand-acts: push (required through the seat),
  plans stop/close, broker consumer-reset and the new hand-act record take
  --why/--cause/--condition; `hand-acts` lists them and repeated causes;
  status counts the week's.
- durations (migration 0066): apply (send to first report), heartbeat gap,
  plan tier and build, recorded as heard; `durations` summarises them.
- the controller's seat row takes this binary's definition of its own verbs,
  so the console no longer judges calls against an older build's schema.
- the controller is granted its two buckets' subjects.

Grant the controller its work queues' cancelled sets (hq issue 269) 9d8cbe7b81

A cancel writes the ask's id into the seat's cancelled set before deleting
the ask, and a write is a publish to the bucket's subject, which the
controller was not granted: against a server holding exactly the
controller's composed list, every cancel timed out.

mesh-admin referenced this pull request from novox/hq

2026-10-06 01:04:42 +00:00

To-be 45 in progress; issue 269: the controller cannot write a cancelled set #129

mesh-admin merged commit cf4834a36c into main

2026-10-06 07:13:39 +00:00

mesh-admin deleted branch feat/a-core-that-cannot-fail-silently-phase-0

2026-10-06 07:13:40 +00:00

mesh-admin commented

2026-10-06 23:10:18 +00:00

Delivery novox/mesh-controller@cf4834a36c71 — held since 2026-10-06T23:05:28Z

Held for a person: merged without a passing check: only a person decides that it goes on — mesh-delivery.release with why lets it go on.

Delivery plan — builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step

tier 0: mesh-controller
tier 1: build-agent
tier 2: route-proxy
ace: receives build-agent, route-proxy
g14: receives build-agent
novox: receives mesh-controller, build-agent, route-proxy
shanks: receives build-agent
build-agent keeps data (ADR 0233): what it holds is backed up before it moves
route-proxy provides route: its consumers are sent again after it
route-proxy keeps data (ADR 0233): what it holds is backed up before it moves

Transitions

2026-10-06 23:05 (new) → proposed (announced): novox/mesh-controller#78 merged; its head was not heard by this owner, so it is made from the forge's word
2026-10-06 23:05 proposed → held (merged-unchecked): merged without a passing check — a verdict known with it is taken first: only a person decides that it goes on

The commit's note under refs/notes/mesh-plan keeps every transition: git log --notes=mesh-plan.

**Delivery** `novox/mesh-controller@cf4834a36c71` — **held** since 2026-10-06T23:05:28Z Held for a person: merged without a passing check: only a person decides that it goes on — `mesh-delivery.release` with why lets it go on. **Delivery plan** — builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step - tier 0: mesh-controller - tier 1: build-agent - tier 2: route-proxy - ace: receives build-agent, route-proxy - g14: receives build-agent - novox: receives mesh-controller, build-agent, route-proxy - shanks: receives build-agent - build-agent keeps data (ADR 0233): what it holds is backed up before it moves - route-proxy provides route: its consumers are sent again after it - route-proxy keeps data (ADR 0233): what it holds is backed up before it moves **Transitions** - 2026-10-06 23:05 (new) → proposed (announced): novox/mesh-controller#78 merged; its head was not heard by this owner, so it is made from the forge's word - 2026-10-06 23:05 proposed → held (merged-unchecked): merged without a passing check — a verdict known with it is taken first: only a person decides that it goes on The commit's note under `refs/notes/mesh-plan` keeps every transition: `git log --notes=mesh-plan`.

This repo is archived. You cannot comment on pull requests.

1 Participants

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: novox/mesh-controller#78