Say when the mesh is wrong: conditions, watchdogs, the bus's advisories, doctor (hq to-be 45 Phase 1) #79

2026-10-06T08:22:40Z

mesh-admin commented

2026-10-06 08:22:40 +00:00

hq to-be 45 Phase 1 (ADR 0227), the controller's part. Depends on mesh-host#33 (imports github.com/novox/mesh-host/validate, fetched by a replace to git.novox.be at that PR's commit, which stays reachable after a merge commit).

What

Condition store (§2): KV mesh-controller_conditions (no age, CAS writes) + mesh-controller_condition-history (90 days). Raised/cleared by observation only; reopened within 10 min = same condition, count+1, silence kept. Verbs conditions (list/filter), show, silence (hand act, ≤7d), history.
Events on mesh.seat.mesh-controller.event.condition-raised|condition-changed|condition-cleared: the condition at top level (key, kind, subject{scope,id,machine,also}, severity, summary, evidence, source, raised, last-observed, observations, count, tried, resolver, silenced|null, epoch) + event, at, change, was, why, cleared, show. Self-check heartbeat doctor-heartbeat {run, at, interval-seconds, counts, probes, controller, why}.
ADR 0224 standing is kind provider-failing (provider.<module>.<node>.<consumer>.failing); provider_standing table no longer read/written (left in place — dropping is the operator's call).
status leads with open conditions; all-well requires none open; unreadable conditions are said and not well.
Signals table compiled in (signals.go), watchdogs every 30s: S1–S11, S13; S12/S14/S15 deferred with reasons. Blind row → probe.<row>.failed, clears nothing. Only the controller holding its consumers watches (standby says nothing). Test generated from the table (signals_test.go).
S9: JetStream advisories (max deliveries, mesh-named consumer deleted) + the controller's own slow consumer / refused subjects.
doctor: D1–D10 (D5 deferred) + DW (watchdogs watched), every 5 min, 30 s each; failed-to-run is never a pass.
Grants: controller gets its new buckets, events, two advisories, $SRV.INFO; node tools get mesh.control.<node>.tools-alive. Streams/consumers asserted at start and expected by D6/D7 are one derivation (busobjects.go).

Tests

Full go test -count=1 -p 1 ./... green with MESH_TEST_POSTGRES (postgres:17) and MESH_TEST_NATS (fresh nats:2.11); -race on conditions, link and the new cmd tests; vet/gofmt clean. Controller's composed permissions verified against an authenticated nats:2.11 holding exactly them (KV writes, history read, advisories, $SRV.INFO, events). Smoke: serve against throwaway stores — doctor ran 10/10 pass, a silenced heartbeat raised machine.anchor.silent (event seen on the bus), silence recorded as hand act, cleared on heartbeat.

Rollout

mesh-host#33 merged and rolled out (or at least merged — the genesis lock matters only for new meshes).
This PR merged → the controller rebuilds; the broker's machine must be pushed after the plan so the composed user list grants the new subjects (until then the condition events / bucket writes are refused and said in the controller log; standings are held and retried).
mesh-tools PR (node tools heartbeat) after this one; until it rolls out, S11 raises tools-silent on every machine with node-tools assigned (3 min after the controller starts). If that is unwanted, roll mesh-tools out right after.

hq: issue 270 (hq PR) records what Phase 1 deferred and decided.

hq to-be 45 Phase 1 (ADR 0227), the controller's part. Depends on mesh-host#33 (imports `github.com/novox/mesh-host/validate`, fetched by a `replace` to git.novox.be at that PR's commit, which stays reachable after a merge commit). ## What - **Condition store** (§2): KV `mesh-controller_conditions` (no age, CAS writes) + `mesh-controller_condition-history` (90 days). Raised/cleared by observation only; reopened within 10 min = same condition, count+1, silence kept. Verbs `conditions` (list/filter), `show`, `silence` (hand act, ≤7d), `history`. - **Events** on `mesh.seat.mesh-controller.event.condition-raised|condition-changed|condition-cleared`: the condition at top level (key, kind, subject{scope,id,machine,also}, severity, summary, evidence, source, raised, last-observed, observations, count, tried, resolver, silenced|null, epoch) + event, at, change, was, why, cleared, show. Self-check heartbeat `doctor-heartbeat` {run, at, interval-seconds, counts, probes, controller, why}. - **ADR 0224 standing** is kind `provider-failing` (`provider.<module>.<node>.<consumer>.failing`); `provider_standing` table no longer read/written (left in place — dropping is the operator's call). - **status** leads with open conditions; all-well requires none open; unreadable conditions are said and not well. - **Signals table** compiled in (`signals.go`), watchdogs every 30s: S1–S11, S13; S12/S14/S15 deferred with reasons. Blind row → `probe.<row>.failed`, clears nothing. Only the controller holding its consumers watches (standby says nothing). Test generated from the table (`signals_test.go`). - **S9**: JetStream advisories (max deliveries, mesh-named consumer deleted) + the controller's own slow consumer / refused subjects. - **doctor**: D1–D10 (D5 deferred) + DW (watchdogs watched), every 5 min, 30 s each; failed-to-run is never a pass. - **Grants**: controller gets its new buckets, events, two advisories, `$SRV.INFO`; node tools get `mesh.control.<node>.tools-alive`. Streams/consumers asserted at start and expected by D6/D7 are one derivation (`busobjects.go`). ## Tests Full `go test -count=1 -p 1 ./...` green with MESH_TEST_POSTGRES (postgres:17) and MESH_TEST_NATS (fresh nats:2.11); `-race` on conditions, link and the new cmd tests; vet/gofmt clean. Controller's composed permissions verified against an authenticated nats:2.11 holding exactly them (KV writes, history read, advisories, $SRV.INFO, events). Smoke: `serve` against throwaway stores — doctor ran 10/10 pass, a silenced heartbeat raised `machine.anchor.silent` (event seen on the bus), silence recorded as hand act, cleared on heartbeat. ## Rollout 1. mesh-host#33 merged and rolled out (or at least merged — the genesis lock matters only for new meshes). 2. This PR merged → the controller rebuilds; **the broker's machine must be pushed after the plan** so the composed user list grants the new subjects (until then the condition events / bucket writes are refused and said in the controller log; standings are held and retried). 3. mesh-tools PR (node tools heartbeat) after this one; until it rolls out, S11 raises `tools-silent` on every machine with node-tools assigned (3 min after the controller starts). If that is unwanted, roll mesh-tools out right after. hq: issue 270 (hq PR) records what Phase 1 deferred and decided.

mesh-admin added 1 commit 2026-10-06 08:22:40 +00:00

Say when the mesh is wrong: conditions, watchdogs, the bus's advisories, doctor (hq to-be 45 Phase 1) bb1607e424

Every one of the 48 core failures of research 031 was found by a person
looking; the mesh's answers carried the fact for whoever asked and told
nobody.

- The condition store (to-be 45 §2): mesh-controller_conditions, one key
  per open condition, written by compare-and-set so a person's silence
  and the watchdogs never lose each other's word; every transition kept
  ninety days in mesh-controller_condition-history and said as the
  seat's events condition-raised / condition-changed / condition-cleared
  (the condition at the top level, with event, at, change, why, show),
  offered again while the bus is away. Raised and cleared by observation
  only; a clearing reopened within ten minutes is the same condition with
  its count up, its silence kept. Verbs: conditions, conditions show,
  conditions silence (a hand act, at most a week), conditions history.
- ADR 0224's provider standing is the first kind, provider-failing, held
  by the provider's events; the provider_standing table is no longer read
  or written (left in place: dropping it is the operator's word).
- status leads with the open conditions, urgent first, and says all well
  only with none open; conditions it cannot read are said and not well.
- The signals table compiled in, one watchdog loop over it every 30s: S1
  heartbeat (3 intervals, asleep machines excepted, control node urgent
  after 30 min), S2 report after a send, S3 plan tier, S4 event loop deaf,
  S5 merge not acted, S6 ask lost, S7 call hung, S8 provider silent, S9
  advisories, S10 self-check silent, S11 node tools silent, S13 stale
  refusals; S12, S14, S15 deferred with their reasons. A row that cannot
  see raises probe-failed and clears nothing. A test generated from the
  table suppresses each signal inside and past its bound.
- The bus's advisories (maximum deliveries, a mesh consumer deleted) and
  the controller's own slow consumer and refused subjects, said in the
  mesh's words.
- doctor: the probe registry D1-D10 (D5 deferred) and DW, every five
  minutes, each in thirty seconds; a probe that cannot run is never a
  pass. D1 validates with mesh-host's own validator. Every run ends with
  the doctor-heartbeat event mesh-watcher listens for.
- The controller is granted its new buckets, events, the two advisories
  and $SRV.INFO; the node tools their tools-alive heartbeat. The streams
  and consumers the controller asserts and the ones D6/D7 expect are one
  derivation.

jschoubben added 1 commit 2026-10-06 08:29:14 +00:00

Vendor every dependency, so no build fetches the host's validator (hq to-be 45 D1) e1f5d4fdf0

The controller imports mesh-host/validate through a replace onto the forge
that holds it, and every build — the build agent's go build in a fresh
toolchain container, the Dockerfile's go mod download — would have fetched
it through the public proxy and checksum database at build time: a merge
breaking main on the network, the class Phase 1 removes. vendor/ is
committed; go builds from it with nothing fetched, and refuses to build
when it and go.mod disagree, so a pin moved without go mod vendor fails at
once. The Dockerfile copies vendor/ and builds with GOPROXY=off.

mesh-admin merged commit fab6b0059e into main

2026-10-06 08:29:32 +00:00

mesh-admin deleted branch feat/a-core-that-cannot-fail-silently-phase-1

2026-10-06 08:29:32 +00:00

mesh-admin commented

2026-10-06 23:10:31 +00:00

Delivery novox/mesh-controller@fab6b0059eaa — held since 2026-10-06T23:05:28Z

Held for a person: merged without a passing check: only a person decides that it goes on — mesh-delivery.release with why lets it go on.

Delivery plan — builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step

tier 0: mesh-controller
tier 1: build-agent
tier 2: route-proxy
ace: receives build-agent, route-proxy
g14: receives build-agent
novox: receives mesh-controller, build-agent, route-proxy
shanks: receives build-agent
build-agent keeps data (ADR 0233): what it holds is backed up before it moves
route-proxy provides route: its consumers are sent again after it
route-proxy keeps data (ADR 0233): what it holds is backed up before it moves

Transitions

2026-10-06 23:05 (new) → proposed (announced): novox/mesh-controller#79 merged; its head was not heard by this owner, so it is made from the forge's word
2026-10-06 23:05 proposed → held (merged-unchecked): merged without a passing check — a verdict known with it is taken first: only a person decides that it goes on

The commit's note under refs/notes/mesh-plan keeps every transition: git log --notes=mesh-plan.

**Delivery** `novox/mesh-controller@fab6b0059eaa` — **held** since 2026-10-06T23:05:28Z Held for a person: merged without a passing check: only a person decides that it goes on — `mesh-delivery.release` with why lets it go on. **Delivery plan** — builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step - tier 0: mesh-controller - tier 1: build-agent - tier 2: route-proxy - ace: receives build-agent, route-proxy - g14: receives build-agent - novox: receives mesh-controller, build-agent, route-proxy - shanks: receives build-agent - build-agent keeps data (ADR 0233): what it holds is backed up before it moves - route-proxy provides route: its consumers are sent again after it - route-proxy keeps data (ADR 0233): what it holds is backed up before it moves **Transitions** - 2026-10-06 23:05 (new) → proposed (announced): novox/mesh-controller#79 merged; its head was not heard by this owner, so it is made from the forge's word - 2026-10-06 23:05 proposed → held (merged-unchecked): merged without a passing check — a verdict known with it is taken first: only a person decides that it goes on The commit's note under `refs/notes/mesh-plan` keeps every transition: `git log --notes=mesh-plan`.

This repo is archived. You cannot comment on pull requests.

2 Participants

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: novox/mesh-controller#79