Judge a send by when a fault began, not when it was last raised (hq issue 348) #179

Merged

mesh-admin merged 5 commits from fix/a-fault-from-before-a-send-fails-no-gate into main 2026-10-09 13:25:21 +00:00

Conversation 6 Commits 5 Files Changed 13

+732 -26

mesh-admin commented 2026-10-09 11:39:13 +00:00

Contributor

Copy Link

Copy Source

hq issue 348 (a resolver that stopped answering ran on unnoticed, and builds sent after it were blamed).

On 2026-10-09 the control node's resolver refused from 10:57:57 UTC. The node-engine sent at 10:59:34 restarted; its first statement said the names part unknown (one look failed), which cleared machine.<control node>.network while its evidence still said "connection refused". The next look reopened it at 11:00:23 with a new raised time, after the send, and the gate put back the controller and the node-engine at 11:10 for a fault older than them.

• internal/conditions: a condition keeps first across a reopening within ReopenWithin (and across a controller restart, from the history); Began() reads it. omitzero, so no condition grows on the wire.
• gate.go: judgeHealth and aboutTheMachine compare Began() with the send.
• machine_network.go: an undecided statement (a part unknown or starting) clears no network condition about that machine.
• probes.go D10: a release's tier counts as rolling out, so a walked node-engine is not core-behind on its own first machine (seen 11:21 today: the release waited five minutes on it).
• probes.go D2: a resolver that refused every try of every question is raised at once, not held for the next run (a refusal is not a loaded resolver, issue 277).

Tests: TestReplay348, TestAReopenedFaultFromBeforeTheSendIsNotRaisedSinceIt, TestAResolverThatRefusesEveryTryIsSaidAtOnce fail on the commit before the fix and pass on it; TestAReopeningKeepsWhenTheFaultBegan, TestAnUndecidedStatementKeepsTheMachinesNetworkConditions, TestAReleaseRollsOutWhatItsTierNames. merge-check.sh passes locally (postgres 17, race detector).

Not merged; a person's approval decides.

hq issue 348 (a resolver that stopped answering ran on unnoticed, and builds sent after it were blamed). On 2026-10-09 the control node's resolver refused from 10:57:57 UTC. The node-engine sent at 10:59:34 restarted; its first statement said the names part `unknown` (one look failed), which cleared `machine.<control node>.network` while its evidence still said "connection refused". The next look reopened it at 11:00:23 with a new raised time, after the send, and the gate put back the controller and the node-engine at 11:10 for a fault older than them. - `internal/conditions`: a condition keeps `first` across a reopening within `ReopenWithin` (and across a controller restart, from the history); `Began()` reads it. `omitzero`, so no condition grows on the wire. - `gate.go`: `judgeHealth` and `aboutTheMachine` compare `Began()` with the send. - `machine_network.go`: an undecided statement (a part `unknown` or `starting`) clears no network condition about that machine. - `probes.go` D10: a release's tier counts as rolling out, so a walked node-engine is not `core-behind` on its own first machine (seen 11:21 today: the release waited five minutes on it). - `probes.go` D2: a resolver that refused every try of every question is raised at once, not held for the next run (a refusal is not a loaded resolver, issue 277). Tests: `TestReplay348`, `TestAReopenedFaultFromBeforeTheSendIsNotRaisedSinceIt`, `TestAResolverThatRefusesEveryTryIsSaidAtOnce` fail on the commit before the fix and pass on it; `TestAReopeningKeepsWhenTheFaultBegan`, `TestAnUndecidedStatementKeepsTheMachinesNetworkConditions`, `TestAReleaseRollsOutWhatItsTierNames`. `merge-check.sh` passes locally (postgres 17, race detector). Not merged; a person's approval decides.

mesh-admin commented 2026-10-09 11:39:37 +00:00

Author

Contributor

Copy Link

Copy Source

Delivery novox/mesh-controller@58cb586c37f8 — delivered since 2026-10-09T13:40:21Z

Delivery plan — builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step

• tier 0: mesh-controller
• tier 1: build-agent
• tier 2: route-proxy
• ace: receives build-agent, route-proxy
• g14: receives build-agent
• novox: receives mesh-controller, build-agent, route-proxy
• shanks: receives build-agent
• build-agent keeps data (ADR 0233): what it holds is backed up before it moves
• route-proxy provides route: its consumers are sent again after it
• route-proxy keeps data (ADR 0233): what it holds is backed up before it moves

Machines

• mesh-controller on novox: passed — healthy 3 times over 3m39s
• mesh-controller on the rest running it: sent
• build-agent on ace: passed — healthy 3 times over 3m23s
• build-agent on the rest running it: sent
• route-proxy on ace: passed — healthy 3 times over 3m36s
• route-proxy on the rest running it: sent

Transitions

• 2026-10-09 13:01 (new) → proposed (announced): novox/mesh-controller#179's head announced
• 2026-10-09 13:07 proposed → checked (checked): the verdict names this commit
• 2026-10-09 13:07 checked → ready (accepted): the gate passed or warned, and the repository's own check did not fail
• 2026-10-09 13:25 ready → published (merged): on the trunk its modules follow: merged there, and its walk opened — or nothing for a walk to move
• 2026-10-09 13:25 published → delivering (go): its walk started: let go by this owner in its group's order, by a person, or on the controller's own path
• 2026-10-09 13:40 delivering → delivered (done): its walk is done: every machine of its deploy plan passed, or was left as its policy says

The commit's note under refs/notes/mesh-plan keeps every transition: git log --notes=mesh-plan.

<!-- mesh-delivery:view --> **Delivery** `novox/mesh-controller@58cb586c37f8` — **delivered** since 2026-10-09T13:40:21Z **Delivery plan** — builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step - tier 0: mesh-controller - tier 1: build-agent - tier 2: route-proxy - ace: receives build-agent, route-proxy - g14: receives build-agent - novox: receives mesh-controller, build-agent, route-proxy - shanks: receives build-agent - build-agent keeps data (ADR 0233): what it holds is backed up before it moves - route-proxy provides route: its consumers are sent again after it - route-proxy keeps data (ADR 0233): what it holds is backed up before it moves **Machines** - mesh-controller on novox: passed — healthy 3 times over 3m39s - mesh-controller on the rest running it: sent - build-agent on ace: passed — healthy 3 times over 3m23s - build-agent on the rest running it: sent - route-proxy on ace: passed — healthy 3 times over 3m36s - route-proxy on the rest running it: sent **Transitions** - 2026-10-09 13:01 (new) → proposed (announced): novox/mesh-controller#179's head announced - 2026-10-09 13:07 proposed → checked (checked): the verdict names this commit - 2026-10-09 13:07 checked → ready (accepted): the gate passed or warned, and the repository's own check did not fail - 2026-10-09 13:25 ready → published (merged): on the trunk its modules follow: merged there, and its walk opened — or nothing for a walk to move - 2026-10-09 13:25 published → delivering (go): its walk started: let go by this owner in its group's order, by a person, or on the controller's own path - 2026-10-09 13:40 delivering → delivered (done): its walk is done: every machine of its deploy plan passed, or was left as its policy says The commit's note under `refs/notes/mesh-plan` keeps every transition: `git log --notes=mesh-plan`.

mesh-admin commented 2026-10-09 11:46:01 +00:00

Author

Contributor

Copy Link

Copy Source

Merge check at e7fcebcf

• mesh/merge-gate: PASS — every machine composes with the change as it did without (4 of 4 compose) (modules: build-agent, mesh-controller, route-proxy)
• mesh/repo-check: PASS — its merge-check.sh passed

Delivery plan — builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step

• tier 0: mesh-controller
• tier 1: build-agent
• tier 2: route-proxy
• ace: receives build-agent, route-proxy
• g14: receives build-agent
• novox: receives mesh-controller, build-agent, route-proxy
• shanks: receives build-agent
• build-agent keeps data (ADR 0233): what it holds is backed up before it moves
• route-proxy provides route: its consumers are sent again after it
• route-proxy keeps data (ADR 0233): what it holds is backed up before it moves

Run by the build seat on g14 as build-1791546002303064592 (builds --log build-1791546002303064592).

--- the gate: build-agent, mesh-controller, route-proxy
mesh-controller: ok, 52 tool(s); WARNING: controller stay(s) up and say(s) not how it is ready — judged by liveness alone, refused from 2026-11-18 (ADR 0240 rule 8)
long-running resources without health: 1
files asking for a setting without saying whether it is trusted: 0
1 manifest(s) checked. Judged against the seats this binary carries; a claim on one of the mesh's own seats is judged fully at registration, and a seat declared by a module not given here reads as unknown. Identities judged on a 6-character machine name, against the bounds of the providers given here
judging 0 module(s) changed against 4 machine(s), as the snapshot of 2026-10-09T11:35:23Z says they run
composing every machine as the mesh is
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
composing every machine with the change
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
merge gate: PASS — every machine composes with the change as it did without (4 of 4 compose)
judged by controller development build against the facts of 2026-10-09T11:35:23Z

machines:
  - a machine (f35): composes
  - holds mesh-dns-resolver (idt): composes
  - holds git, holds mesh-artifact-store, holds mesh-broker, holds mesh-catalog, holds mesh-delivery, holds mesh-dns-resolver, holds mesh-store, holds mesh-vault, holds npm-package-registry, the control node, the hub (ygcyr): composes
  - a machine (yvbpsm): composes

a merge would rebuild 3 module(s) in 3 tier(s): build-agent, mesh-controller, route-proxy
--- its merge-check.sh (go toolchain)
ok  	github.com/novox/mesh-controller/cmd/mesh-builder	1.060s
ok  	github.com/novox/mesh-controller/cmd/mesh-controller	227.144s
ok  	github.com/novox/mesh-controller/examples/objectstore-provisioner	1.112s
ok  	github.com/novox/mesh-controller/examples/postgres-provisioner	1.017s
ok  	github.com/novox/mesh-controller/examples/redis-provisioner	1.025s
ok  	github.com/novox/mesh-controller/examples/route-proxy	1.208s
ok  	github.com/novox/mesh-controller/internal/artifacts	1.058s
ok  	github.com/novox/mesh-controller/internal/broker	3.192s
ok  	github.com/novox/mesh-controller/internal/builder	3.500s
ok  	github.com/novox/mesh-controller/internal/catalogue	6.550s
ok  	github.com/novox/mesh-controller/internal/conditions	2.598s
ok  	github.com/novox/mesh-controller/internal/envfile	1.115s
ok  	github.com/novox/mesh-controller/internal/facts	1.030s
ok  	github.com/novox/mesh-controller/internal/identity	7.413s
ok  	github.com/novox/mesh-controller/internal/inventory	102.179s
ok  	github.com/novox/mesh-controller/internal/lease	17.669s
ok  	github.com/novox/mesh-controller/internal/licences	6.740s
ok  	github.com/novox/mesh-controller/internal/licences/adapters	1.083s
ok  	github.com/novox/mesh-controller/internal/link	56.137s
ok  	github.com/novox/mesh-controller/internal/lint	2.496s
ok  	github.com/novox/mesh-controller/internal/outward	1.049s
ok  	github.com/novox/mesh-controller/internal/overlay	1.041s
ok  	github.com/novox/mesh-controller/internal/secrets	1.654s
ok  	github.com/novox/mesh-controller/internal/store	4.116s
ok  	github.com/novox/mesh-controller/internal/testbus	1.253s
ok  	github.com/novox/mesh-controller/internal/token	1.042s

**Merge check** at `e7fcebcf` - `mesh/merge-gate`: **PASS** — every machine composes with the change as it did without (4 of 4 compose) (modules: build-agent, mesh-controller, route-proxy) - `mesh/repo-check`: **PASS** — its merge-check.sh passed **Delivery plan** — builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step - tier 0: mesh-controller - tier 1: build-agent - tier 2: route-proxy - ace: receives build-agent, route-proxy - g14: receives build-agent - novox: receives mesh-controller, build-agent, route-proxy - shanks: receives build-agent - build-agent keeps data (ADR 0233): what it holds is backed up before it moves - route-proxy provides route: its consumers are sent again after it - route-proxy keeps data (ADR 0233): what it holds is backed up before it moves Run by the build seat on g14 as `build-1791546002303064592` (`builds --log build-1791546002303064592`). ``` --- the gate: build-agent, mesh-controller, route-proxy mesh-controller: ok, 52 tool(s); WARNING: controller stay(s) up and say(s) not how it is ready — judged by liveness alone, refused from 2026-11-18 (ADR 0240 rule 8) long-running resources without health: 1 files asking for a setting without saying whether it is trusted: 0 1 manifest(s) checked. Judged against the seats this binary carries; a claim on one of the mesh's own seats is judged fully at registration, and a seat declared by a module not given here reads as unknown. Identities judged on a 6-character machine name, against the bounds of the providers given here judging 0 module(s) changed against 4 machine(s), as the snapshot of 2026-10-09T11:35:23Z says they run composing every machine as the mesh is the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued composing every machine with the change the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued merge gate: PASS — every machine composes with the change as it did without (4 of 4 compose) judged by controller development build against the facts of 2026-10-09T11:35:23Z machines: - a machine (f35): composes - holds mesh-dns-resolver (idt): composes - holds git, holds mesh-artifact-store, holds mesh-broker, holds mesh-catalog, holds mesh-delivery, holds mesh-dns-resolver, holds mesh-store, holds mesh-vault, holds npm-package-registry, the control node, the hub (ygcyr): composes - a machine (yvbpsm): composes a merge would rebuild 3 module(s) in 3 tier(s): build-agent, mesh-controller, route-proxy --- its merge-check.sh (go toolchain) ok github.com/novox/mesh-controller/cmd/mesh-builder 1.060s ok github.com/novox/mesh-controller/cmd/mesh-controller 227.144s ok github.com/novox/mesh-controller/examples/objectstore-provisioner 1.112s ok github.com/novox/mesh-controller/examples/postgres-provisioner 1.017s ok github.com/novox/mesh-controller/examples/redis-provisioner 1.025s ok github.com/novox/mesh-controller/examples/route-proxy 1.208s ok github.com/novox/mesh-controller/internal/artifacts 1.058s ok github.com/novox/mesh-controller/internal/broker 3.192s ok github.com/novox/mesh-controller/internal/builder 3.500s ok github.com/novox/mesh-controller/internal/catalogue 6.550s ok github.com/novox/mesh-controller/internal/conditions 2.598s ok github.com/novox/mesh-controller/internal/envfile 1.115s ok github.com/novox/mesh-controller/internal/facts 1.030s ok github.com/novox/mesh-controller/internal/identity 7.413s ok github.com/novox/mesh-controller/internal/inventory 102.179s ok github.com/novox/mesh-controller/internal/lease 17.669s ok github.com/novox/mesh-controller/internal/licences 6.740s ok github.com/novox/mesh-controller/internal/licences/adapters 1.083s ok github.com/novox/mesh-controller/internal/link 56.137s ok github.com/novox/mesh-controller/internal/lint 2.496s ok github.com/novox/mesh-controller/internal/outward 1.049s ok github.com/novox/mesh-controller/internal/overlay 1.041s ok github.com/novox/mesh-controller/internal/secrets 1.654s ok github.com/novox/mesh-controller/internal/store 4.116s ok github.com/novox/mesh-controller/internal/testbus 1.253s ok github.com/novox/mesh-controller/internal/token 1.042s ```

jschoubben force-pushed fix/a-fault-from-before-a-send-fails-no-gate from e7fcebcfa1 to 7a22419771 2026-10-09 12:11:49 +00:00 Compare

mesh-admin commented 2026-10-09 12:18:30 +00:00

Author

Contributor

Copy Link

Copy Source

Merge check at 7a224197

• mesh/merge-gate: PASS — every machine composes with the change as it did without (4 of 4 compose) (modules: build-agent, mesh-controller, route-proxy)
• mesh/repo-check: FAIL — its merge-check.sh failed: --- FAIL: TestTheResolverForwardsToFixedUpstreamsAndNeverReadsResolvConf (0.00s)

Delivery plan — builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step

• tier 0: mesh-controller
• tier 1: build-agent
• tier 2: route-proxy
• ace: receives build-agent, route-proxy
• g14: receives build-agent
• novox: receives mesh-controller, build-agent, route-proxy
• shanks: receives build-agent
• build-agent keeps data (ADR 0233): what it holds is backed up before it moves
• route-proxy provides route: its consumers are sent again after it
• route-proxy keeps data (ADR 0233): what it holds is backed up before it moves

Run by the build seat on ace as build-1791547936508087350 (builds --log build-1791547936508087350).

--- the gate: build-agent, mesh-controller, route-proxy
mesh-controller: ok, 52 tool(s); WARNING: controller stay(s) up and say(s) not how it is ready — judged by liveness alone, refused from 2026-11-18 (ADR 0240 rule 8)
long-running resources without health: 1
files asking for a setting without saying whether it is trusted: 0
1 manifest(s) checked. Judged against the seats this binary carries; a claim on one of the mesh's own seats is judged fully at registration, and a seat declared by a module not given here reads as unknown. Identities judged on a 6-character machine name, against the bounds of the providers given here
judging 0 module(s) changed against 4 machine(s), as the snapshot of 2026-10-09T12:05:01Z says they run
composing every machine as the mesh is
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
composing every machine with the change
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
merge gate: PASS — every machine composes with the change as it did without (4 of 4 compose)
judged by controller development build against the facts of 2026-10-09T12:05:01Z

machines:
  - a machine (f35): composes
  - holds mesh-dns-resolver (idt): composes
  - holds git, holds mesh-artifact-store, holds mesh-broker, holds mesh-catalog, holds mesh-delivery, holds mesh-dns-resolver, holds mesh-store, holds mesh-vault, holds npm-package-registry, the control node, the hub (ygcyr): composes
  - a machine (yvbpsm): composes

a merge would rebuild 3 module(s) in 3 tier(s): build-agent, mesh-controller, route-proxy
--- its merge-check.sh (go toolchain)
ok  	github.com/novox/mesh-controller/cmd/mesh-builder	1.033s
ok  	github.com/novox/mesh-controller/cmd/mesh-controller	183.826s
ok  	github.com/novox/mesh-controller/examples/objectstore-provisioner	1.036s
ok  	github.com/novox/mesh-controller/examples/postgres-provisioner	1.044s
ok  	github.com/novox/mesh-controller/examples/redis-provisioner	1.024s
ok  	github.com/novox/mesh-controller/examples/route-proxy	1.131s
ok  	github.com/novox/mesh-controller/internal/artifacts	1.073s
ok  	github.com/novox/mesh-controller/internal/broker	2.554s
ok  	github.com/novox/mesh-controller/internal/builder	1.799s
--- FAIL: TestTheResolverForwardsToFixedUpstreamsAndNeverReadsResolvConf (0.00s)
    resolver_manifests_test.go:65: the resolver's configuration lacks "bind-interfaces":
        # Managed by the mesh. dnsmasq's own defaults are replaced whole rather than
        # patched, because this module owns the file and a patch would leave whatever
        # was there before to be discovered later.
        #
        # This is one of the mesh's resolvers (novox/hq ADR 0194, 0223): it holds the
        # `mesh-dns-resolver` seat, which more than one machine may hold, each answering
        # the same names from the same roster. Every node and every container lists every
        # holder and no public resolver, so whichever answers first gives the one answer.
        # It holds the mesh's own names and nothing else (ADR 0191) — the roster is the
        # mesh's, rendered into each holder; no other copy lives on any machine.
        
        # What the mesh computed: one wildcard per machine — its name and everything
        # under it — and the mesh's own suffix as a local domain, so a name under it is
        # answered here or not at all and is never asked upstream. Rewritten whenever a
        # machine joins or leaves, which is why the service below restarts on it: a
        # reload makes dnsmasq re-read hosts files, not its configuration, and a
        # wildcard is configuration.
        conf-file=/etc/mesh-resolver/nodes.conf
        
        # Every zone a module answers itself (ADR 0199): one forwarding line per zone, to
        # the private address of the machine its module runs on and the port its
        # answerer is published on. Nothing in a zone is answered here; what names a zone
        # holds is the module's. Rewritten whenever a zone is declared or moves, and the
        # service restarts on it for the same reason as above.
        conf-file=/etc/mesh-resolver/zones.conf
        
        # Where it answers: this machine's private address, so every node — and every
        # container on every node, which copies its machine's resolvers — can ask; and
        # loopback, for this machine's own use. Never a LAN address: a device that is
        # not a member cannot reach what the mesh's names point at, and a LAN's resolver
        # is its router's (ADR 0194).
        #
        # Named as an ADDRESS and not as the interface that carries it, on purpose. A
        # container's query is addressed to this address but arrives on the runtime's
        # bridge, and dnsmasq checks every query against what it was told to answer on:
        # an `interface=` line admits queries by the interface they arrive on, a
        # `listen-address=` line by the address they are sent to. With `interface=mesh0`
        # alone, a query from a container was received and dropped without a word
        # (novox/hq 04-ISSUES/110). The address admits it whatever bridge it comes in on.
        #
        # bind-dynamic rather than bind-interfaces: the private address does not exist
        # until the machine is on the private network, and binding an address that is
        # not there yet fails to start rather than waiting for it.
        bind-dynamic
        listen-address=${machine:address}
        listen-address=127.0.0.1
        
        # **It must never read resolv.conf to find out where to forward.** This
        # machine's resolv.conf names this resolver — so a resolver that read it for
        # upstreams would find itself, and every query it could not answer locally would
        # loop until its receive queue filled. That is not theoretical: it filled with
        # 15KB of queries and every lookup on the machine hung. no-resolv makes that loop
        # impossible: the upstreams are the two lines below, and nothing on the machine
        # can redirect them. The mesh's own names never reach them: the local= line in
        # the file above stops them here, answered or refused.
        no-resolv
        server=1.1.1.1
        server=8.8.8.8
        
        # Both upstreams validate DNSSEC and say so with the Authenticated Data bit,
        # and this passes that bit down rather than dropping it, which dnsmasq does
        # unless told. It does not validate itself: an answer's trust is the upstream's
        # and the path to it, which is what a forwarding resolver's trust always was.
        # Without it a program that refuses to run behind a resolver that does not
        # validate — a mail system's admin does exactly that check at start — cannot
        # use this resolver (novox/hq 04-ISSUES/171).
        proxy-dnssec
        
        # A name without a dot is never forwarded, and reverse lookups of private ranges
        # are answered here rather than asking the world who 10.x is.
        domain-needed
        bogus-priv
        
        # **No hosts file, and no operator's files.** This machine's /etc/hosts is its
        # own — the hosts module's block and the operator's lines (ADR 0199) — and the
        # mesh's resolver answers every node, so a line written for one machine's own
        # programs must not become an answer for all of them. Every per-node resolver
        # went wrong exactly here: a hosts file read once at start, an operator's old
        # line beside the mesh's, a drop-in read from a directory nobody owned.
        no-hosts
    resolver_manifests_test.go:69: the resolver binds dynamically, and closes a listener whenever a re-read of the addresses fails (issue 348)
FAIL
FAIL	github.com/novox/mesh-controller/internal/catalogue	3.347s
ok  	github.com/novox/mesh-controller/internal/conditions	2.413s
ok  	github.com/novox/mesh-controller/internal/envfile	1.038s
ok  	github.com/novox/mesh-controller/internal/facts	1.040s
ok  	github.com/novox/mesh-controller/internal/identity	4.143s
ok  	github.com/novox/mesh-controller/internal/inventory	71.835s
ok  	github.com/novox/mesh-controller/internal/lease	17.670s
ok  	github.com/novox/mesh-controller/internal/licences	3.750s
ok  	github.com/novox/mesh-controller/internal/licences/adapters	1.037s
ok  	github.com/novox/mesh-controller/internal/link	51.064s
ok  	github.com/novox/mesh-controller/internal/lint	1.955s
ok  	github.com/novox/mesh-controller/internal/outward	1.066s
ok  	github.com/novox/mesh-controller/internal/overlay	1.059s
ok  	github.com/novox/mesh-controller/internal/secrets	1.435s
ok  	github.com/novox/mesh-controller/internal/store	3.285s
ok  	github.com/novox/mesh-controller/internal/testbus	1.208s
ok  	github.com/novox/mesh-controller/internal/token	1.015s
FAIL

**Merge check** at `7a224197` - `mesh/merge-gate`: **PASS** — every machine composes with the change as it did without (4 of 4 compose) (modules: build-agent, mesh-controller, route-proxy) - `mesh/repo-check`: **FAIL** — its merge-check.sh failed: --- FAIL: TestTheResolverForwardsToFixedUpstreamsAndNeverReadsResolvConf (0.00s) **Delivery plan** — builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step - tier 0: mesh-controller - tier 1: build-agent - tier 2: route-proxy - ace: receives build-agent, route-proxy - g14: receives build-agent - novox: receives mesh-controller, build-agent, route-proxy - shanks: receives build-agent - build-agent keeps data (ADR 0233): what it holds is backed up before it moves - route-proxy provides route: its consumers are sent again after it - route-proxy keeps data (ADR 0233): what it holds is backed up before it moves Run by the build seat on ace as `build-1791547936508087350` (`builds --log build-1791547936508087350`). ``` --- the gate: build-agent, mesh-controller, route-proxy mesh-controller: ok, 52 tool(s); WARNING: controller stay(s) up and say(s) not how it is ready — judged by liveness alone, refused from 2026-11-18 (ADR 0240 rule 8) long-running resources without health: 1 files asking for a setting without saying whether it is trusted: 0 1 manifest(s) checked. Judged against the seats this binary carries; a claim on one of the mesh's own seats is judged fully at registration, and a seat declared by a module not given here reads as unknown. Identities judged on a 6-character machine name, against the bounds of the providers given here judging 0 module(s) changed against 4 machine(s), as the snapshot of 2026-10-09T12:05:01Z says they run composing every machine as the mesh is the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued composing every machine with the change the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued merge gate: PASS — every machine composes with the change as it did without (4 of 4 compose) judged by controller development build against the facts of 2026-10-09T12:05:01Z machines: - a machine (f35): composes - holds mesh-dns-resolver (idt): composes - holds git, holds mesh-artifact-store, holds mesh-broker, holds mesh-catalog, holds mesh-delivery, holds mesh-dns-resolver, holds mesh-store, holds mesh-vault, holds npm-package-registry, the control node, the hub (ygcyr): composes - a machine (yvbpsm): composes a merge would rebuild 3 module(s) in 3 tier(s): build-agent, mesh-controller, route-proxy --- its merge-check.sh (go toolchain) ok github.com/novox/mesh-controller/cmd/mesh-builder 1.033s ok github.com/novox/mesh-controller/cmd/mesh-controller 183.826s ok github.com/novox/mesh-controller/examples/objectstore-provisioner 1.036s ok github.com/novox/mesh-controller/examples/postgres-provisioner 1.044s ok github.com/novox/mesh-controller/examples/redis-provisioner 1.024s ok github.com/novox/mesh-controller/examples/route-proxy 1.131s ok github.com/novox/mesh-controller/internal/artifacts 1.073s ok github.com/novox/mesh-controller/internal/broker 2.554s ok github.com/novox/mesh-controller/internal/builder 1.799s --- FAIL: TestTheResolverForwardsToFixedUpstreamsAndNeverReadsResolvConf (0.00s) resolver_manifests_test.go:65: the resolver's configuration lacks "bind-interfaces": # Managed by the mesh. dnsmasq's own defaults are replaced whole rather than # patched, because this module owns the file and a patch would leave whatever # was there before to be discovered later. # # This is one of the mesh's resolvers (novox/hq ADR 0194, 0223): it holds the # `mesh-dns-resolver` seat, which more than one machine may hold, each answering # the same names from the same roster. Every node and every container lists every # holder and no public resolver, so whichever answers first gives the one answer. # It holds the mesh's own names and nothing else (ADR 0191) — the roster is the # mesh's, rendered into each holder; no other copy lives on any machine. # What the mesh computed: one wildcard per machine — its name and everything # under it — and the mesh's own suffix as a local domain, so a name under it is # answered here or not at all and is never asked upstream. Rewritten whenever a # machine joins or leaves, which is why the service below restarts on it: a # reload makes dnsmasq re-read hosts files, not its configuration, and a # wildcard is configuration. conf-file=/etc/mesh-resolver/nodes.conf # Every zone a module answers itself (ADR 0199): one forwarding line per zone, to # the private address of the machine its module runs on and the port its # answerer is published on. Nothing in a zone is answered here; what names a zone # holds is the module's. Rewritten whenever a zone is declared or moves, and the # service restarts on it for the same reason as above. conf-file=/etc/mesh-resolver/zones.conf # Where it answers: this machine's private address, so every node — and every # container on every node, which copies its machine's resolvers — can ask; and # loopback, for this machine's own use. Never a LAN address: a device that is # not a member cannot reach what the mesh's names point at, and a LAN's resolver # is its router's (ADR 0194). # # Named as an ADDRESS and not as the interface that carries it, on purpose. A # container's query is addressed to this address but arrives on the runtime's # bridge, and dnsmasq checks every query against what it was told to answer on: # an `interface=` line admits queries by the interface they arrive on, a # `listen-address=` line by the address they are sent to. With `interface=mesh0` # alone, a query from a container was received and dropped without a word # (novox/hq 04-ISSUES/110). The address admits it whatever bridge it comes in on. # # bind-dynamic rather than bind-interfaces: the private address does not exist # until the machine is on the private network, and binding an address that is # not there yet fails to start rather than waiting for it. bind-dynamic listen-address=${machine:address} listen-address=127.0.0.1 # **It must never read resolv.conf to find out where to forward.** This # machine's resolv.conf names this resolver — so a resolver that read it for # upstreams would find itself, and every query it could not answer locally would # loop until its receive queue filled. That is not theoretical: it filled with # 15KB of queries and every lookup on the machine hung. no-resolv makes that loop # impossible: the upstreams are the two lines below, and nothing on the machine # can redirect them. The mesh's own names never reach them: the local= line in # the file above stops them here, answered or refused. no-resolv server=1.1.1.1 server=8.8.8.8 # Both upstreams validate DNSSEC and say so with the Authenticated Data bit, # and this passes that bit down rather than dropping it, which dnsmasq does # unless told. It does not validate itself: an answer's trust is the upstream's # and the path to it, which is what a forwarding resolver's trust always was. # Without it a program that refuses to run behind a resolver that does not # validate — a mail system's admin does exactly that check at start — cannot # use this resolver (novox/hq 04-ISSUES/171). proxy-dnssec # A name without a dot is never forwarded, and reverse lookups of private ranges # are answered here rather than asking the world who 10.x is. domain-needed bogus-priv # **No hosts file, and no operator's files.** This machine's /etc/hosts is its # own — the hosts module's block and the operator's lines (ADR 0199) — and the # mesh's resolver answers every node, so a line written for one machine's own # programs must not become an answer for all of them. Every per-node resolver # went wrong exactly here: a hosts file read once at start, an operator's old # line beside the mesh's, a drop-in read from a directory nobody owned. no-hosts resolver_manifests_test.go:69: the resolver binds dynamically, and closes a listener whenever a re-read of the addresses fails (issue 348) FAIL FAIL github.com/novox/mesh-controller/internal/catalogue 3.347s ok github.com/novox/mesh-controller/internal/conditions 2.413s ok github.com/novox/mesh-controller/internal/envfile 1.038s ok github.com/novox/mesh-controller/internal/facts 1.040s ok github.com/novox/mesh-controller/internal/identity 4.143s ok github.com/novox/mesh-controller/internal/inventory 71.835s ok github.com/novox/mesh-controller/internal/lease 17.670s ok github.com/novox/mesh-controller/internal/licences 3.750s ok github.com/novox/mesh-controller/internal/licences/adapters 1.037s ok github.com/novox/mesh-controller/internal/link 51.064s ok github.com/novox/mesh-controller/internal/lint 1.955s ok github.com/novox/mesh-controller/internal/outward 1.066s ok github.com/novox/mesh-controller/internal/overlay 1.059s ok github.com/novox/mesh-controller/internal/secrets 1.435s ok github.com/novox/mesh-controller/internal/store 3.285s ok github.com/novox/mesh-controller/internal/testbus 1.208s ok github.com/novox/mesh-controller/internal/token 1.015s FAIL ```

mesh-admin commented 2026-10-09 12:26:54 +00:00

Author

Contributor

Copy Link

Copy Source

Merge check at f0f71069

• mesh/merge-gate: PASS — every machine composes with the change as it did without (4 of 4 compose) (modules: build-agent, mesh-controller, route-proxy)
• mesh/repo-check: PASS — its merge-check.sh passed

Delivery plan — builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step

• tier 0: mesh-controller
• tier 1: build-agent
• tier 2: route-proxy
• ace: receives build-agent, route-proxy
• g14: receives build-agent
• novox: receives mesh-controller, build-agent, route-proxy
• shanks: receives build-agent
• build-agent keeps data (ADR 0233): what it holds is backed up before it moves
• route-proxy provides route: its consumers are sent again after it
• route-proxy keeps data (ADR 0233): what it holds is backed up before it moves

Run by the build seat on ace as build-1791548543399875627 (builds --log build-1791548543399875627).

--- the gate: build-agent, mesh-controller, route-proxy
mesh-controller: ok, 52 tool(s); WARNING: controller stay(s) up and say(s) not how it is ready — judged by liveness alone, refused from 2026-11-18 (ADR 0240 rule 8)
long-running resources without health: 1
files asking for a setting without saying whether it is trusted: 0
1 manifest(s) checked. Judged against the seats this binary carries; a claim on one of the mesh's own seats is judged fully at registration, and a seat declared by a module not given here reads as unknown. Identities judged on a 6-character machine name, against the bounds of the providers given here
judging 0 module(s) changed against 4 machine(s), as the snapshot of 2026-10-09T12:14:01Z says they run
composing every machine as the mesh is
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
composing every machine with the change
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
merge gate: PASS — every machine composes with the change as it did without (4 of 4 compose)
judged by controller development build against the facts of 2026-10-09T12:14:01Z

machines:
  - a machine (f35): composes
  - holds mesh-dns-resolver (idt): composes
  - holds git, holds mesh-artifact-store, holds mesh-broker, holds mesh-catalog, holds mesh-delivery, holds mesh-dns-resolver, holds mesh-store, holds mesh-vault, holds npm-package-registry, the control node, the hub (ygcyr): composes
  - a machine (yvbpsm): composes

a merge would rebuild 3 module(s) in 3 tier(s): build-agent, mesh-controller, route-proxy
--- its merge-check.sh (go toolchain)
ok  	github.com/novox/mesh-controller/cmd/mesh-builder	1.035s
ok  	github.com/novox/mesh-controller/cmd/mesh-controller	183.780s
ok  	github.com/novox/mesh-controller/examples/objectstore-provisioner	1.013s
ok  	github.com/novox/mesh-controller/examples/postgres-provisioner	1.018s
ok  	github.com/novox/mesh-controller/examples/redis-provisioner	1.014s
ok  	github.com/novox/mesh-controller/examples/route-proxy	1.097s
ok  	github.com/novox/mesh-controller/internal/artifacts	1.041s
ok  	github.com/novox/mesh-controller/internal/broker	2.455s
ok  	github.com/novox/mesh-controller/internal/builder	1.729s
ok  	github.com/novox/mesh-controller/internal/catalogue	4.390s
ok  	github.com/novox/mesh-controller/internal/conditions	2.386s
ok  	github.com/novox/mesh-controller/internal/envfile	1.019s
ok  	github.com/novox/mesh-controller/internal/facts	1.019s
ok  	github.com/novox/mesh-controller/internal/identity	4.118s
ok  	github.com/novox/mesh-controller/internal/inventory	72.202s
ok  	github.com/novox/mesh-controller/internal/lease	17.655s
ok  	github.com/novox/mesh-controller/internal/licences	3.762s
ok  	github.com/novox/mesh-controller/internal/licences/adapters	1.053s
ok  	github.com/novox/mesh-controller/internal/link	51.016s
ok  	github.com/novox/mesh-controller/internal/lint	1.978s
ok  	github.com/novox/mesh-controller/internal/outward	1.074s
ok  	github.com/novox/mesh-controller/internal/overlay	1.070s
ok  	github.com/novox/mesh-controller/internal/secrets	1.431s
ok  	github.com/novox/mesh-controller/internal/store	3.254s
ok  	github.com/novox/mesh-controller/internal/testbus	1.234s
ok  	github.com/novox/mesh-controller/internal/token	1.061s

**Merge check** at `f0f71069` - `mesh/merge-gate`: **PASS** — every machine composes with the change as it did without (4 of 4 compose) (modules: build-agent, mesh-controller, route-proxy) - `mesh/repo-check`: **PASS** — its merge-check.sh passed **Delivery plan** — builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step - tier 0: mesh-controller - tier 1: build-agent - tier 2: route-proxy - ace: receives build-agent, route-proxy - g14: receives build-agent - novox: receives mesh-controller, build-agent, route-proxy - shanks: receives build-agent - build-agent keeps data (ADR 0233): what it holds is backed up before it moves - route-proxy provides route: its consumers are sent again after it - route-proxy keeps data (ADR 0233): what it holds is backed up before it moves Run by the build seat on ace as `build-1791548543399875627` (`builds --log build-1791548543399875627`). ``` --- the gate: build-agent, mesh-controller, route-proxy mesh-controller: ok, 52 tool(s); WARNING: controller stay(s) up and say(s) not how it is ready — judged by liveness alone, refused from 2026-11-18 (ADR 0240 rule 8) long-running resources without health: 1 files asking for a setting without saying whether it is trusted: 0 1 manifest(s) checked. Judged against the seats this binary carries; a claim on one of the mesh's own seats is judged fully at registration, and a seat declared by a module not given here reads as unknown. Identities judged on a 6-character machine name, against the bounds of the providers given here judging 0 module(s) changed against 4 machine(s), as the snapshot of 2026-10-09T12:14:01Z says they run composing every machine as the mesh is the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued composing every machine with the change the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued merge gate: PASS — every machine composes with the change as it did without (4 of 4 compose) judged by controller development build against the facts of 2026-10-09T12:14:01Z machines: - a machine (f35): composes - holds mesh-dns-resolver (idt): composes - holds git, holds mesh-artifact-store, holds mesh-broker, holds mesh-catalog, holds mesh-delivery, holds mesh-dns-resolver, holds mesh-store, holds mesh-vault, holds npm-package-registry, the control node, the hub (ygcyr): composes - a machine (yvbpsm): composes a merge would rebuild 3 module(s) in 3 tier(s): build-agent, mesh-controller, route-proxy --- its merge-check.sh (go toolchain) ok github.com/novox/mesh-controller/cmd/mesh-builder 1.035s ok github.com/novox/mesh-controller/cmd/mesh-controller 183.780s ok github.com/novox/mesh-controller/examples/objectstore-provisioner 1.013s ok github.com/novox/mesh-controller/examples/postgres-provisioner 1.018s ok github.com/novox/mesh-controller/examples/redis-provisioner 1.014s ok github.com/novox/mesh-controller/examples/route-proxy 1.097s ok github.com/novox/mesh-controller/internal/artifacts 1.041s ok github.com/novox/mesh-controller/internal/broker 2.455s ok github.com/novox/mesh-controller/internal/builder 1.729s ok github.com/novox/mesh-controller/internal/catalogue 4.390s ok github.com/novox/mesh-controller/internal/conditions 2.386s ok github.com/novox/mesh-controller/internal/envfile 1.019s ok github.com/novox/mesh-controller/internal/facts 1.019s ok github.com/novox/mesh-controller/internal/identity 4.118s ok github.com/novox/mesh-controller/internal/inventory 72.202s ok github.com/novox/mesh-controller/internal/lease 17.655s ok github.com/novox/mesh-controller/internal/licences 3.762s ok github.com/novox/mesh-controller/internal/licences/adapters 1.053s ok github.com/novox/mesh-controller/internal/link 51.016s ok github.com/novox/mesh-controller/internal/lint 1.978s ok github.com/novox/mesh-controller/internal/outward 1.074s ok github.com/novox/mesh-controller/internal/overlay 1.070s ok github.com/novox/mesh-controller/internal/secrets 1.431s ok github.com/novox/mesh-controller/internal/store 3.254s ok github.com/novox/mesh-controller/internal/testbus 1.234s ok github.com/novox/mesh-controller/internal/token 1.061s ```

jschoubben added 5 commits 2026-10-09 13:01:00 +00:00

Judge a send by when a fault began, not when it was last raised (hq issue 348) ... 077ddf0eb8

On 2026-10-09 the control node's resolver refused from 10:57:57 UTC. A
node-engine restarted by the 10:59:34 send said its names undecided, that
statement cleared the network condition, the next look raised it again
after the send, and the gate put back two builds for a fault older than
them.

- A condition keeps First across a reopening; the gate reads Began.
- An undecided network statement (unknown, starting) clears nothing.
- D10 counts what a release's tier names as rolling, so a walked
  node-engine is not core-behind on its own first machine.
- D2 raises a resolver that refuses every try at once: a refusal is an
  answer, not a loaded resolver (issue 277).

Order a branch's plans by its merges, and build the newest commit (hq issue 349) ... 997a4925b0

A merge acted on late by the catch-up made its plan after the plan of the
merge that followed it, superseded it by creation time, and folded its
unbuilt modules into a plan at the older commit: on 2026-10-09 the
forge's security fix (a082615b) was superseded by 8ff8197a. A plan now
keeps its merge time (migration 0086), supersession follows it, and a
merge older than an open plan of its branch is planned at that plan's
commit, which contains it.

Hold the resolver to bind-interfaces, as mesh-catalog PR 161 makes it (hq issue 348) 63b87b6f7b

Take either binding until the catalogue's main binds once (hq issue 348) ... c3c56a69e2

The test reads the catalogue beside it, which the build seat checks out at
main; mesh-catalog PR 161 changes the binding, so the two land in either
order.

Answer the review of hq issues 348 and 349: gaps, parts, the newest merge in any state ... 58cb586c37

- A reopened fault keeps its gaps: it was there at a send unless the send
  fell in one, so a send that breaks a machine recovered before it still
  fails its gate (A2).
- An undecided part holds only the conditions that name it (A4).
- D2 holds a silent resolver for the next run again, refused or not: a
  burst of refusals is also a restart (A3).
- A late merge is planned at the newest planned merge of its branch in any
  state, not only an open one (A1); merge times to the nanosecond (A5).

jschoubben force-pushed fix/a-fault-from-before-a-send-fails-no-gate from f1df33e15b to 58cb586c37 2026-10-09 13:01:00 +00:00 Compare

mesh-admin commented 2026-10-09 13:05:38 +00:00

Author

Contributor

Copy Link

Copy Source

Merge check at f1df33e1

• mesh/merge-gate: PASS — every machine composes with the change as it did without (4 of 4 compose) (modules: build-agent, mesh-controller, route-proxy)
• mesh/repo-check: PASS — its merge-check.sh passed

Delivery plan — builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step

• tier 0: mesh-controller
• tier 1: build-agent
• tier 2: route-proxy
• ace: receives build-agent, route-proxy
• g14: receives build-agent
• novox: receives mesh-controller, build-agent, route-proxy
• shanks: receives build-agent
• build-agent keeps data (ADR 0233): what it holds is backed up before it moves
• route-proxy provides route: its consumers are sent again after it
• route-proxy keeps data (ADR 0233): what it holds is backed up before it moves

Run by the build seat on ace as build-1791550854116150831 (builds --log build-1791550854116150831).

--- the gate: build-agent, mesh-controller, route-proxy
mesh-controller: ok, 52 tool(s); WARNING: controller stay(s) up and say(s) not how it is ready — judged by liveness alone, refused from 2026-11-18 (ADR 0240 rule 8)
long-running resources without health: 1
files asking for a setting without saying whether it is trusted: 0
1 manifest(s) checked. Judged against the seats this binary carries; a claim on one of the mesh's own seats is judged fully at registration, and a seat declared by a module not given here reads as unknown. Identities judged on a 6-character machine name, against the bounds of the providers given here
judging 0 module(s) changed against 4 machine(s), as the snapshot of 2026-10-09T12:52:20Z says they run
composing every machine as the mesh is
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
composing every machine with the change
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
merge gate: PASS — every machine composes with the change as it did without (4 of 4 compose)
judged by controller development build against the facts of 2026-10-09T12:52:20Z

machines:
  - a machine (f35): composes
  - holds mesh-dns-resolver (idt): composes
  - holds git, holds mesh-artifact-store, holds mesh-broker, holds mesh-catalog, holds mesh-delivery, holds mesh-dns-resolver, holds mesh-store, holds mesh-vault, holds npm-package-registry, the control node, the hub (ygcyr): composes
  - a machine (yvbpsm): composes

a merge would rebuild 3 module(s) in 3 tier(s): build-agent, mesh-controller, route-proxy
--- its merge-check.sh (go toolchain)
ok  	github.com/novox/mesh-controller/cmd/mesh-builder	1.034s
ok  	github.com/novox/mesh-controller/cmd/mesh-controller	184.526s
ok  	github.com/novox/mesh-controller/examples/objectstore-provisioner	1.011s
ok  	github.com/novox/mesh-controller/examples/postgres-provisioner	1.016s
ok  	github.com/novox/mesh-controller/examples/redis-provisioner	1.013s
ok  	github.com/novox/mesh-controller/examples/route-proxy	1.095s
ok  	github.com/novox/mesh-controller/internal/artifacts	1.033s
ok  	github.com/novox/mesh-controller/internal/broker	2.560s
ok  	github.com/novox/mesh-controller/internal/builder	1.831s
ok  	github.com/novox/mesh-controller/internal/catalogue	4.347s
ok  	github.com/novox/mesh-controller/internal/conditions	2.426s
ok  	github.com/novox/mesh-controller/internal/envfile	1.059s
ok  	github.com/novox/mesh-controller/internal/facts	1.097s
ok  	github.com/novox/mesh-controller/internal/identity	4.082s
ok  	github.com/novox/mesh-controller/internal/inventory	72.150s
ok  	github.com/novox/mesh-controller/internal/lease	17.735s
ok  	github.com/novox/mesh-controller/internal/licences	3.729s
ok  	github.com/novox/mesh-controller/internal/licences/adapters	1.093s
ok  	github.com/novox/mesh-controller/internal/link	51.183s
ok  	github.com/novox/mesh-controller/internal/lint	2.019s
ok  	github.com/novox/mesh-controller/internal/outward	1.079s
ok  	github.com/novox/mesh-controller/internal/overlay	1.108s
ok  	github.com/novox/mesh-controller/internal/secrets	1.482s
ok  	github.com/novox/mesh-controller/internal/store	3.275s
ok  	github.com/novox/mesh-controller/internal/testbus	1.268s
ok  	github.com/novox/mesh-controller/internal/token	1.016s

**Merge check** at `f1df33e1` - `mesh/merge-gate`: **PASS** — every machine composes with the change as it did without (4 of 4 compose) (modules: build-agent, mesh-controller, route-proxy) - `mesh/repo-check`: **PASS** — its merge-check.sh passed **Delivery plan** — builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step - tier 0: mesh-controller - tier 1: build-agent - tier 2: route-proxy - ace: receives build-agent, route-proxy - g14: receives build-agent - novox: receives mesh-controller, build-agent, route-proxy - shanks: receives build-agent - build-agent keeps data (ADR 0233): what it holds is backed up before it moves - route-proxy provides route: its consumers are sent again after it - route-proxy keeps data (ADR 0233): what it holds is backed up before it moves Run by the build seat on ace as `build-1791550854116150831` (`builds --log build-1791550854116150831`). ``` --- the gate: build-agent, mesh-controller, route-proxy mesh-controller: ok, 52 tool(s); WARNING: controller stay(s) up and say(s) not how it is ready — judged by liveness alone, refused from 2026-11-18 (ADR 0240 rule 8) long-running resources without health: 1 files asking for a setting without saying whether it is trusted: 0 1 manifest(s) checked. Judged against the seats this binary carries; a claim on one of the mesh's own seats is judged fully at registration, and a seat declared by a module not given here reads as unknown. Identities judged on a 6-character machine name, against the bounds of the providers given here judging 0 module(s) changed against 4 machine(s), as the snapshot of 2026-10-09T12:52:20Z says they run composing every machine as the mesh is the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued composing every machine with the change the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued merge gate: PASS — every machine composes with the change as it did without (4 of 4 compose) judged by controller development build against the facts of 2026-10-09T12:52:20Z machines: - a machine (f35): composes - holds mesh-dns-resolver (idt): composes - holds git, holds mesh-artifact-store, holds mesh-broker, holds mesh-catalog, holds mesh-delivery, holds mesh-dns-resolver, holds mesh-store, holds mesh-vault, holds npm-package-registry, the control node, the hub (ygcyr): composes - a machine (yvbpsm): composes a merge would rebuild 3 module(s) in 3 tier(s): build-agent, mesh-controller, route-proxy --- its merge-check.sh (go toolchain) ok github.com/novox/mesh-controller/cmd/mesh-builder 1.034s ok github.com/novox/mesh-controller/cmd/mesh-controller 184.526s ok github.com/novox/mesh-controller/examples/objectstore-provisioner 1.011s ok github.com/novox/mesh-controller/examples/postgres-provisioner 1.016s ok github.com/novox/mesh-controller/examples/redis-provisioner 1.013s ok github.com/novox/mesh-controller/examples/route-proxy 1.095s ok github.com/novox/mesh-controller/internal/artifacts 1.033s ok github.com/novox/mesh-controller/internal/broker 2.560s ok github.com/novox/mesh-controller/internal/builder 1.831s ok github.com/novox/mesh-controller/internal/catalogue 4.347s ok github.com/novox/mesh-controller/internal/conditions 2.426s ok github.com/novox/mesh-controller/internal/envfile 1.059s ok github.com/novox/mesh-controller/internal/facts 1.097s ok github.com/novox/mesh-controller/internal/identity 4.082s ok github.com/novox/mesh-controller/internal/inventory 72.150s ok github.com/novox/mesh-controller/internal/lease 17.735s ok github.com/novox/mesh-controller/internal/licences 3.729s ok github.com/novox/mesh-controller/internal/licences/adapters 1.093s ok github.com/novox/mesh-controller/internal/link 51.183s ok github.com/novox/mesh-controller/internal/lint 2.019s ok github.com/novox/mesh-controller/internal/outward 1.079s ok github.com/novox/mesh-controller/internal/overlay 1.108s ok github.com/novox/mesh-controller/internal/secrets 1.482s ok github.com/novox/mesh-controller/internal/store 3.275s ok github.com/novox/mesh-controller/internal/testbus 1.268s ok github.com/novox/mesh-controller/internal/token 1.016s ```

mesh-admin commented 2026-10-09 13:07:12 +00:00

Author

Contributor

Copy Link

Copy Source

Merge check at 58cb586c

• mesh/merge-gate: PASS — every machine composes with the change as it did without (4 of 4 compose) (modules: build-agent, mesh-controller, route-proxy)
• mesh/repo-check: PASS — its merge-check.sh passed

Delivery plan — builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step

• tier 0: mesh-controller
• tier 1: build-agent
• tier 2: route-proxy
• ace: receives build-agent, route-proxy
• g14: receives build-agent
• novox: receives mesh-controller, build-agent, route-proxy
• shanks: receives build-agent
• build-agent keeps data (ADR 0233): what it holds is backed up before it moves
• route-proxy provides route: its consumers are sent again after it
• route-proxy keeps data (ADR 0233): what it holds is backed up before it moves

Run by the build seat on g14 as build-1791550888739170184 (builds --log build-1791550888739170184).

--- the gate: build-agent, mesh-controller, route-proxy
mesh-controller: ok, 52 tool(s); WARNING: controller stay(s) up and say(s) not how it is ready — judged by liveness alone, refused from 2026-11-18 (ADR 0240 rule 8)
long-running resources without health: 1
files asking for a setting without saying whether it is trusted: 0
1 manifest(s) checked. Judged against the seats this binary carries; a claim on one of the mesh's own seats is judged fully at registration, and a seat declared by a module not given here reads as unknown. Identities judged on a 6-character machine name, against the bounds of the providers given here
judging 0 module(s) changed against 4 machine(s), as the snapshot of 2026-10-09T12:52:20Z says they run
composing every machine as the mesh is
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
composing every machine with the change
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
merge gate: PASS — every machine composes with the change as it did without (4 of 4 compose)
judged by controller development build against the facts of 2026-10-09T12:52:20Z

machines:
  - a machine (f35): composes
  - holds mesh-dns-resolver (idt): composes
  - holds git, holds mesh-artifact-store, holds mesh-broker, holds mesh-catalog, holds mesh-delivery, holds mesh-dns-resolver, holds mesh-store, holds mesh-vault, holds npm-package-registry, the control node, the hub (ygcyr): composes
  - a machine (yvbpsm): composes

a merge would rebuild 3 module(s) in 3 tier(s): build-agent, mesh-controller, route-proxy
--- its merge-check.sh (go toolchain)
ok  	github.com/novox/mesh-controller/cmd/mesh-builder	1.029s
ok  	github.com/novox/mesh-controller/cmd/mesh-controller	204.537s
ok  	github.com/novox/mesh-controller/examples/objectstore-provisioner	1.012s
ok  	github.com/novox/mesh-controller/examples/postgres-provisioner	1.023s
ok  	github.com/novox/mesh-controller/examples/redis-provisioner	1.013s
ok  	github.com/novox/mesh-controller/examples/route-proxy	1.159s
ok  	github.com/novox/mesh-controller/internal/artifacts	1.050s
ok  	github.com/novox/mesh-controller/internal/broker	2.528s
ok  	github.com/novox/mesh-controller/internal/builder	1.804s
ok  	github.com/novox/mesh-controller/internal/catalogue	4.291s
ok  	github.com/novox/mesh-controller/internal/conditions	2.338s
ok  	github.com/novox/mesh-controller/internal/envfile	1.028s
ok  	github.com/novox/mesh-controller/internal/facts	1.031s
ok  	github.com/novox/mesh-controller/internal/identity	3.997s
ok  	github.com/novox/mesh-controller/internal/inventory	72.473s
ok  	github.com/novox/mesh-controller/internal/lease	17.704s
ok  	github.com/novox/mesh-controller/internal/licences	3.629s
ok  	github.com/novox/mesh-controller/internal/licences/adapters	1.008s
ok  	github.com/novox/mesh-controller/internal/link	51.456s
ok  	github.com/novox/mesh-controller/internal/lint	1.908s
ok  	github.com/novox/mesh-controller/internal/outward	1.030s
ok  	github.com/novox/mesh-controller/internal/overlay	1.019s
ok  	github.com/novox/mesh-controller/internal/secrets	1.414s
ok  	github.com/novox/mesh-controller/internal/store	3.203s
ok  	github.com/novox/mesh-controller/internal/testbus	1.157s
ok  	github.com/novox/mesh-controller/internal/token	1.011s

**Merge check** at `58cb586c` - `mesh/merge-gate`: **PASS** — every machine composes with the change as it did without (4 of 4 compose) (modules: build-agent, mesh-controller, route-proxy) - `mesh/repo-check`: **PASS** — its merge-check.sh passed **Delivery plan** — builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step - tier 0: mesh-controller - tier 1: build-agent - tier 2: route-proxy - ace: receives build-agent, route-proxy - g14: receives build-agent - novox: receives mesh-controller, build-agent, route-proxy - shanks: receives build-agent - build-agent keeps data (ADR 0233): what it holds is backed up before it moves - route-proxy provides route: its consumers are sent again after it - route-proxy keeps data (ADR 0233): what it holds is backed up before it moves Run by the build seat on g14 as `build-1791550888739170184` (`builds --log build-1791550888739170184`). ``` --- the gate: build-agent, mesh-controller, route-proxy mesh-controller: ok, 52 tool(s); WARNING: controller stay(s) up and say(s) not how it is ready — judged by liveness alone, refused from 2026-11-18 (ADR 0240 rule 8) long-running resources without health: 1 files asking for a setting without saying whether it is trusted: 0 1 manifest(s) checked. Judged against the seats this binary carries; a claim on one of the mesh's own seats is judged fully at registration, and a seat declared by a module not given here reads as unknown. Identities judged on a 6-character machine name, against the bounds of the providers given here judging 0 module(s) changed against 4 machine(s), as the snapshot of 2026-10-09T12:52:20Z says they run composing every machine as the mesh is the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued composing every machine with the change the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued merge gate: PASS — every machine composes with the change as it did without (4 of 4 compose) judged by controller development build against the facts of 2026-10-09T12:52:20Z machines: - a machine (f35): composes - holds mesh-dns-resolver (idt): composes - holds git, holds mesh-artifact-store, holds mesh-broker, holds mesh-catalog, holds mesh-delivery, holds mesh-dns-resolver, holds mesh-store, holds mesh-vault, holds npm-package-registry, the control node, the hub (ygcyr): composes - a machine (yvbpsm): composes a merge would rebuild 3 module(s) in 3 tier(s): build-agent, mesh-controller, route-proxy --- its merge-check.sh (go toolchain) ok github.com/novox/mesh-controller/cmd/mesh-builder 1.029s ok github.com/novox/mesh-controller/cmd/mesh-controller 204.537s ok github.com/novox/mesh-controller/examples/objectstore-provisioner 1.012s ok github.com/novox/mesh-controller/examples/postgres-provisioner 1.023s ok github.com/novox/mesh-controller/examples/redis-provisioner 1.013s ok github.com/novox/mesh-controller/examples/route-proxy 1.159s ok github.com/novox/mesh-controller/internal/artifacts 1.050s ok github.com/novox/mesh-controller/internal/broker 2.528s ok github.com/novox/mesh-controller/internal/builder 1.804s ok github.com/novox/mesh-controller/internal/catalogue 4.291s ok github.com/novox/mesh-controller/internal/conditions 2.338s ok github.com/novox/mesh-controller/internal/envfile 1.028s ok github.com/novox/mesh-controller/internal/facts 1.031s ok github.com/novox/mesh-controller/internal/identity 3.997s ok github.com/novox/mesh-controller/internal/inventory 72.473s ok github.com/novox/mesh-controller/internal/lease 17.704s ok github.com/novox/mesh-controller/internal/licences 3.629s ok github.com/novox/mesh-controller/internal/licences/adapters 1.008s ok github.com/novox/mesh-controller/internal/link 51.456s ok github.com/novox/mesh-controller/internal/lint 1.908s ok github.com/novox/mesh-controller/internal/outward 1.030s ok github.com/novox/mesh-controller/internal/overlay 1.019s ok github.com/novox/mesh-controller/internal/secrets 1.414s ok github.com/novox/mesh-controller/internal/store 3.203s ok github.com/novox/mesh-controller/internal/testbus 1.157s ok github.com/novox/mesh-controller/internal/token 1.011s ```

mesh-admin merged commit 9ca7952d5e into main 2026-10-09 13:25:21 +00:00

mesh-admin deleted branch fix/a-fault-from-before-a-send-fails-no-gate 2026-10-09 13:25:21 +00:00

mesh-admin referenced this pull request 2026-10-09 13:29:25 +00:00

Test the newest merge's order, a second reopening's gaps and sub-second merge times (hq issues 348, 349) #183

jschoubben referenced this issue from a commit 2026-10-13 03:21:57 +00:00

Add the replay of issue 349: a merge acted on late superseded the merge after it (hq issue 349)

mesh-admin referenced this pull request 2026-10-13 03:22:07 +00:00

Add the replay of issue 349: a merge acted on late superseded the merge after it (hq issue 349) #290

mesh-admin referenced this pull request 2026-10-13 03:34:49 +00:00

Add the replay of issue 349: a merge acted on late superseded the merge after it (hq issue 349) #290

mesh-admin referenced this pull request 2026-10-13 03:53:31 +00:00

Add the replay of issue 349: a merge acted on late superseded the merge after it (hq issue 349) #290

jschoubben referenced this issue from a commit 2026-10-13 04:09:12 +00:00

Add the replay of issue 348, and refuse bind-dynamic now the catalogue binds once (hq issue 348)

mesh-admin referenced this pull request 2026-10-13 04:09:59 +00:00

Add the replay of issue 348, and refuse bind-dynamic now the catalogue binds once (hq issue 348) #291

This repo is archived. You cannot comment on pull requests.

Reviewers

No Reviewers

Labels

Clear labels

mesh/ready

Reviewed and ready: mesh-delivery merges at the marked head once its checks pass (hq ADR 0364)

No labels

Milestone

No items

No Milestone

Assignees

Clear assignees

agent-ace agent-g14 agent-novox agent-shanks jschoubben mesh-tools miel.truyen nox-agent

No Assignees

2 Participants

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: novox/mesh-controller#179