Fixes novox/hq issue 477 (a bus stall of tens of seconds ends the controller, though no other controller could take its lease).
What was seen
2026-10-14 13:04:10 UTC: LOST the controller lease (epoch 136861): the renewal could not be made: context deadline exceeded — this controller stops acting at once, then exit; systemd restarted it and the new process took epoch 140306 at 13:04:19. The bus had stalled only about 3 s ($JS.API.STREAM.INFO... took too long: 3.16s). The operator got the urgent "The controller lost its lease" on every channel. Only one controller ever ran. Same on 2026-10-11 01:31-01:35.
The rule (decided, the operator's standing rule)
A renewal that fails is not a lost lease. The controller keeps the lease until the key's TTL has passed without a successful renewal. It starts no new act once half the TTL has passed unrenewed, and resumes on the next successful renewal. It is lost only when the TTL has passed, or another holder is seen. The urgent lease-lost condition is raised only on a real loss.
The change
internal/lease: Renew answers ErrDelayed for a renewal the bus did not answer, keeps the lease and says it once per run of failures; Keep carries on and loses the lease when the TTL passes unrenewed (a timer, so a hanging renewal cannot stretch it; one renewal's timeout is capped at the TTL left).
The gate (Epoch, which every act passes) now closes at half the TTL unrenewed (or TTL less the margin, if sooner) instead of TTL less the margin, and opens again on the next renewal made.
A refused renewal (key moved) reads the key: still this instance at this epoch means a stalled renewal that was written though its answer was lost, so it renews at the key's revision; another holder, or the key gone, is a loss at once. A key that cannot be read is a delayed renewal, not a loss.
Seam for issue 478 (the bus-stall condition, not built here): Options.Delayed(why, renewed, lostBy) is told of each delayed renewal; nothing sets it yet.
lease-lost stays raised only from an epoch ended as lost (now only a real loss) or a lease unrenewed past its TTL; its "lost" wording says the two real causes.
Timeline in production (TTL 15 s, renewal every 5 s): renewal at 0 made; a stall from 1 s; the renewal at 5 s fails, said once, lease kept; at 7.5 s no new act starts; a renewal made before 15 s opens the gate again and says so; none made by 15 s → lost and exit as today.
Tests
internal/lease (real bus, design bounds divided by five): TestAFailedRenewalKeepsTheLeaseUntilItsTTL, TestARenewalMadeAfterAStallOpensTheGateAgain, TestARenewalWrittenThoughItsAnswerWasLostIsStillTheHolders, TestAnotherHolderSeenAfterAStallIsALossAtOnce; the existing tests (refusal is a loss at once, a holder that stops renewing stops acting before it is taken over, handover) still pass. go test -race -count=3 ./internal/lease/ passes; go vet ./... clean. TestTwoControllersOneActs needs the store and runs in the merge check.
Acceptance
A bus stall shorter than the lease's 15 s TTL no longer ends the controller or raises the urgent lease-lost condition; the controller pauses new acts after 7.5 s unrenewed and acts again on its next renewal, and it still exits at once when another controller writes the key.
Fixes novox/hq issue 477 (a bus stall of tens of seconds ends the controller, though no other controller could take its lease).
## What was seen
2026-10-14 13:04:10 UTC: `LOST the controller lease (epoch 136861): the renewal could not be made: context deadline exceeded — this controller stops acting at once`, then exit; systemd restarted it and the new process took epoch 140306 at 13:04:19. The bus had stalled only about 3 s (`$JS.API.STREAM.INFO... took too long: 3.16s`). The operator got the urgent "The controller lost its lease" on every channel. Only one controller ever ran. Same on 2026-10-11 01:31-01:35.
## The rule (decided, the operator's standing rule)
A renewal that fails is not a lost lease. The controller keeps the lease until the key's TTL has passed without a successful renewal. It starts no new act once half the TTL has passed unrenewed, and resumes on the next successful renewal. It is lost only when the TTL has passed, or another holder is seen. The urgent lease-lost condition is raised only on a real loss.
## The change
- `internal/lease`: `Renew` answers `ErrDelayed` for a renewal the bus did not answer, keeps the lease and says it once per run of failures; `Keep` carries on and loses the lease when the TTL passes unrenewed (a timer, so a hanging renewal cannot stretch it; one renewal's timeout is capped at the TTL left).
- The gate (`Epoch`, which every act passes) now closes at half the TTL unrenewed (or TTL less the margin, if sooner) instead of TTL less the margin, and opens again on the next renewal made.
- A refused renewal (key moved) reads the key: still this instance at this epoch means a stalled renewal that was written though its answer was lost, so it renews at the key's revision; another holder, or the key gone, is a loss at once. A key that cannot be read is a delayed renewal, not a loss.
- Seam for issue 478 (the bus-stall condition, not built here): `Options.Delayed(why, renewed, lostBy)` is told of each delayed renewal; nothing sets it yet.
- `lease-lost` stays raised only from an epoch ended as lost (now only a real loss) or a lease unrenewed past its TTL; its "lost" wording says the two real causes.
Timeline in production (TTL 15 s, renewal every 5 s): renewal at 0 made; a stall from 1 s; the renewal at 5 s fails, said once, lease kept; at 7.5 s no new act starts; a renewal made before 15 s opens the gate again and says so; none made by 15 s → lost and exit as today.
## Tests
`internal/lease` (real bus, design bounds divided by five): TestAFailedRenewalKeepsTheLeaseUntilItsTTL, TestARenewalMadeAfterAStallOpensTheGateAgain, TestARenewalWrittenThoughItsAnswerWasLostIsStillTheHolders, TestAnotherHolderSeenAfterAStallIsALossAtOnce; the existing tests (refusal is a loss at once, a holder that stops renewing stops acting before it is taken over, handover) still pass. `go test -race -count=3 ./internal/lease/` passes; `go vet ./...` clean. TestTwoControllersOneActs needs the store and runs in the merge check.
## Acceptance
A bus stall shorter than the lease's 15 s TTL no longer ends the controller or raises the urgent lease-lost condition; the controller pauses new acts after 7.5 s unrenewed and acts again on its next renewal, and it still exits at once when another controller writes the key.
A bus stall of a few seconds made a renewal time out, and the controller
called that a lost lease, exited and raised the urgent lease-lost alert,
though no other controller could take a key that still lived. Now only
another holder seen, or the key's TTL passed unrenewed, loses the lease;
new acts stop at half the TTL unrenewed and resume on the next renewal
(novox/hq issue 477).
build-agent keeps data (ADR 0233): what it holds is backed up before it moves
route-proxy provides route: its consumers are sent again after it
route-proxy keeps data (ADR 0233): what it holds is backed up before it moves
grants step: if this changes what the bus's user list says (a new tool, subject or user), novox is sent that list alone, every build there kept, before build-agent, route-proxy is judged on its first machine
Machines
mesh-controller on novox: passed — healthy 1 times over 1m55s, by the controller's rule: one reading once it is ready, after the node-engine's witness window (novox/hq ADR 0351)
mesh-controller on the rest running it: sent
build-agent on ace: passed — healthy 1 times over 16s, by a leaf module's rule: one reading (novox/hq ADR 0304)
build-agent on the rest running it: sent
route-proxy on ace: passed — healthy 1 times over 17s, by a leaf module's rule: one reading (novox/hq ADR 0304)
route-proxy on the rest running it: sent
Transitions
2026-10-14 13:21 (new) → proposed (announced): novox/mesh-controller#331's head announced
2026-10-14 13:34 proposed → checked (checked): a verdict names this commit and the trunk's head as its trunk commit
2026-10-14 13:34 checked → ready (accepted): the gate passed or warned, and the repository's own check did not fail
2026-10-14 13:35 ready → published (merged): on the trunk its modules follow: merged there, and its walk opened — or nothing for a walk to move
2026-10-14 13:35 published → delivering (go): its walk started: let go by this owner in its group's order, by a person, or on the controller's own path — not a failed walk that carried a later commit for it, after which it waits for a walk of its own commit
2026-10-14 13:45 delivering → delivered (done): its walk is done, carrying its own commit: every machine of its deploy plan passed, or was left as its policy says
The commit's note under refs/notes/mesh-plan keeps every transition: git log --notes=mesh-plan.
<!-- mesh-delivery:view -->
**Delivery** `novox/mesh-controller@98e856874913` — **delivered** since 2026-10-14T13:45:18Z
**Delivery plan** — builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step
- tier 0: mesh-controller
- tier 1: build-agent
- tier 2: route-proxy
- ace: receives build-agent, route-proxy
- g14: receives build-agent
- novox: receives mesh-controller, build-agent, route-proxy
- shanks: receives build-agent
- build-agent keeps data (ADR 0233): what it holds is backed up before it moves
- route-proxy provides route: its consumers are sent again after it
- route-proxy keeps data (ADR 0233): what it holds is backed up before it moves
- grants step: if this changes what the bus's user list says (a new tool, subject or user), novox is sent that list alone, every build there kept, before build-agent, route-proxy is judged on its first machine
**Machines**
- mesh-controller on novox: passed — healthy 1 times over 1m55s, by the controller's rule: one reading once it is ready, after the node-engine's witness window (novox/hq ADR 0351)
- mesh-controller on the rest running it: sent
- build-agent on ace: passed — healthy 1 times over 16s, by a leaf module's rule: one reading (novox/hq ADR 0304)
- build-agent on the rest running it: sent
- route-proxy on ace: passed — healthy 1 times over 17s, by a leaf module's rule: one reading (novox/hq ADR 0304)
- route-proxy on the rest running it: sent
**Transitions**
- 2026-10-14 13:21 (new) → proposed (announced): novox/mesh-controller#331's head announced
- 2026-10-14 13:34 proposed → checked (checked): a verdict names this commit and the trunk's head as its trunk commit
- 2026-10-14 13:34 checked → ready (accepted): the gate passed or warned, and the repository's own check did not fail
- 2026-10-14 13:35 ready → published (merged): on the trunk its modules follow: merged there, and its walk opened — or nothing for a walk to move
- 2026-10-14 13:35 published → delivering (go): its walk started: let go by this owner in its group's order, by a person, or on the controller's own path — not a failed walk that carried a later commit for it, after which it waits for a walk of its own commit
- 2026-10-14 13:45 delivering → delivered (done): its walk is done, carrying its own commit: every machine of its deploy plan passed, or was left as its policy says
The commit's note under `refs/notes/mesh-plan` keeps every transition: `git log --notes=mesh-plan`.
build-agent keeps data (ADR 0233): what it holds is backed up before it moves
route-proxy provides route: its consumers are sent again after it
route-proxy keeps data (ADR 0233): what it holds is backed up before it moves
grants step: if this changes what the bus's user list says (a new tool, subject or user), novox is sent that list alone, every build there kept, before build-agent, route-proxy is judged on its first machine
read by no module's build: internal/lease/lease_test.go
Run by the build seat on shanks as build-1791984061260125618 (builds --log build-1791984061260125618).
--- the gate: build-agent, mesh-controller, route-proxy
mesh-controller: ok, 60 tool(s); WARNING: merge-window ask(s) for a setting and do(es) not say whether it is trusted, so it counts as trusted: set at the terminal alone; say "trusted" false where nothing trusts it (novox/hq issue 339); WARNING: controller stay(s) up and say(s) not how it is ready — judged by liveness alone, refused from 2026-11-18 (ADR 0240 rule 8)
long-running resources without health: 1
files asking for a setting without saying whether it is trusted: 1
1 manifest(s) checked. Judged against the seats this binary carries; a claim on one of the mesh's own seats is judged fully at registration, and a seat declared by a module not given here reads as unknown. Identities judged on a 6-character machine name, against the bounds of the providers given here
judging 1 module(s) changed against 4 machine(s), as the snapshot of 2026-10-14T13:14:20Z says they run
composing every machine as the mesh is
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
composing every machine with the change
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
merge gate: PASS — every machine composes with the change as it did without (4 of 4 compose)
judged by controller development build against the facts of 2026-10-14T13:14:20Z
modules the change moves:
- mesh-controller (changed)
machines:
- a machine (f35): composes
- holds mesh-dns-resolver (idt): composes
- holds git, holds mesh-artifact-store, holds mesh-broker, holds mesh-catalog, holds mesh-delivery, holds mesh-dns-resolver, holds mesh-store, holds mesh-vault, holds npm-package-registry, the control node, the hub (ygcyr): composes
- a machine (yvbpsm): composes
a merge would rebuild 3 module(s) in 3 tier(s): build-agent, mesh-controller, route-proxy
notes:
- internal/lease/lease_test.go read by no module's build, and rebuild nothing (issue 280)
--- its merge-check.sh (go toolchain)
ok github.com/novox/mesh-controller/cmd/mesh-builder 1.094s
ok github.com/novox/mesh-controller/cmd/mesh-controller 183.910s
ok github.com/novox/mesh-controller/examples/objectstore-provisioner 1.012s
ok github.com/novox/mesh-controller/examples/postgres-provisioner 1.022s
ok github.com/novox/mesh-controller/examples/redis-provisioner 1.059s
ok github.com/novox/mesh-controller/examples/route-proxy 1.198s
ok github.com/novox/mesh-controller/internal/artifacts 1.080s
? github.com/novox/mesh-controller/internal/beside [no test files]
ok github.com/novox/mesh-controller/internal/broker 4.651s
ok github.com/novox/mesh-controller/internal/builder 41.865s
ok github.com/novox/mesh-controller/internal/catalogue 8.849s
ok github.com/novox/mesh-controller/internal/conditions 2.708s
ok github.com/novox/mesh-controller/internal/envfile 1.065s
ok github.com/novox/mesh-controller/internal/facts 1.072s
ok github.com/novox/mesh-controller/internal/identity 5.240s
ok github.com/novox/mesh-controller/internal/inventory 35.996s
ok github.com/novox/mesh-controller/internal/lease 30.921s
ok github.com/novox/mesh-controller/internal/licences 4.179s
ok github.com/novox/mesh-controller/internal/licences/adapters 1.065s
ok github.com/novox/mesh-controller/internal/link 59.726s
ok github.com/novox/mesh-controller/internal/lint 2.655s
ok github.com/novox/mesh-controller/internal/outward 1.085s
ok github.com/novox/mesh-controller/internal/overlay 1.070s
ok github.com/novox/mesh-controller/internal/secrets 1.525s
ok github.com/novox/mesh-controller/internal/store 3.484s
ok github.com/novox/mesh-controller/internal/testbus 1.236s
ok github.com/novox/mesh-controller/internal/token 1.016s
store tests, each run against the store:
./cmd/mesh-controller --- PASS: TestAStopWithoutAMergeWalksTheOthersAgain (0.22s)
./cmd/mesh-controller --- PASS: TestAStopWithoutAMergeEndsTheMergesThatContainIt (0.23s)
./cmd/mesh-controller --- PASS: TestAStopWalksTheOthersAgainInTheOpenBatch (0.22s)
./cmd/mesh-controller --- PASS: TestAStopNamesWhatStaysOnTheMachinesAndWhatStillCarriesIt (0.22s)
./cmd/mesh-controller --- PASS: TestAStopWithoutAMergeLeavesItOutOfABatch (0.20s)
./cmd/mesh-controller --- PASS: TestSavingPlansAndMovingMergesIsOneAct (0.18s)
./cmd/mesh-controller --- PASS: TestAPlanFollowedFromTheStoreAnswersWhenItMoves (1.97s)
./cmd/mesh-controller --- PASS: TestNodeShowAndStatusReadTheLastSendFromTheStore (0.25s)
./cmd/mesh-controller --- PASS: TestTheConditionsVerbReadsTheLooksTheBucketNoLongerHolds (0.13s)
./cmd/mesh-controller --- PASS: TestAWalkLetGoAndStoppedByVerbsIsSaidStartedAndEnded (0.25s)
./cmd/mesh-controller --- PASS: TestAWalkStoppedOrClosedByAPersonIsSaidEndedStopped (0.25s)
./cmd/mesh-controller --- PASS: TestAWalkRetriedByAPersonIsSaidStartedAgain (0.21s)
./cmd/mesh-controller --- PASS: TestABacklogReleasedByAPersonIsSaidStarted (0.21s)
./cmd/mesh-controller --- PASS: TestAWalkEndedByCancelOrClearIsSaidEnded (0.27s)
./cmd/mesh-controller --- PASS: TestAWalkRestartedByARebuildIsSaidStarted (0.22s)
./cmd/mesh-controller --- PASS: TestAWalkStoppedByHandAndRetriedEndsFailedWhenItFailsAgain (0.26s)
./cmd/mesh-controller --- PASS: TestReplay237 (0.20s)
./cmd/mesh-controller --- PASS: TestAnUnassignThatLeavesItsNodeUnresolvedSaysSoOnceInItsAnswer (0.20s)
./cmd/mesh-controller --- PASS: TestAnUnassignThatLeavesItsNodeResolvedSaysNothingOfIt (0.21s)
./cmd/mesh-controller --- PASS: TestAnAssignThatLeavesItsNodeUnresolvedSaysOnlyThePlansRefusal (0.21s)
./cmd/mesh-controller --- PASS: TestADirectoryOwnersKeyIsRefusedThroughAVerb (0.23s)
./cmd/mesh-controller --- PASS: TestTheSendMakesNoValueForAnOutsideSecretNobodyGave (0.29s)
./cmd/mesh-controller --- PASS: TestAQuestionComposesWithoutAnOutsideSecretNobodyGave (0.27s)
./cmd/mesh-controller --- PASS: TestReplay201 (0.17s)
./cmd/mesh-controller --- PASS: TestARollbackWhoseSendWasRefusedIsCompletedByTheNextSend (0.15s)
./cmd/mesh-controller --- PASS: TestAGateJudgesTheOtherMovesBesideAKeptCore (0.16s)
./cmd/mesh-controller --- PASS: TestReplay401 (0.31s)
./cmd/mesh-controller --- PASS: TestReplay201ThePlansSend (0.23s)
./cmd/mesh-controller --- PASS: TestPlanAndNodeListAnOptionalRequirementLeftUnbound (0.29s)
./cmd/mesh-controller --- PASS: TestReplay340 (0.26s)
./cmd/mesh-controller --- PASS: TestOnlyATrustedFilesNamedKeyIsRefusedThroughAVerb (0.19s)
./cmd/mesh-controller --- PASS: TestReplay349 (0.22s)
./cmd/mesh-controller --- PASS: TestReplay348AFaultFromBeforeTheSendFailsNoGate (0.19s)
./cmd/mesh-controller --- PASS: TestReplay239AModuleNameIsNotTakenOverByAnotherRepository (0.16s)
./cmd/mesh-controller --- PASS: TestAWalkThatMovedNothingSaysWhichModulesMatchedWhatRuns (0.19s)
./cmd/mesh-controller --- PASS: TestReplay614ANewSDKPlansTheToolchainAndThenItsBundles (0.08s)
./cmd/mesh-controller --- PASS: TestOneAcceptForAProviderIsPlacedForEveryConsumer (0.50s)
./cmd/mesh-controller --- PASS: TestReplay164 (0.29s)
./cmd/mesh-controller --- PASS: TestAnApprovedLayerOpensTheWalkThatSendsIt (0.20s)
./cmd/mesh-controller --- PASS: TestASettingsWalkSendsOneMachineAtATimeEachJudged (0.20s)
./cmd/mesh-controller --- PASS: TestAFailedSettingsWalkStopsAndHoldsNothingBack (0.19s)
./cmd/mesh-controller --- PASS: TestASettingsWalkSendsTheLayerAloneEveryBuildKept (0.19s)
./cmd/mesh-controller --- PASS: TestASettingsWalkStopsBeforeADataMoveAndPassesOverAMachineAway (0.54s)
./cmd/mesh-controller --- PASS: TestACrossingSendSupersedesASettingsGate (0.18s)
./cmd/mesh-controller --- PASS: TestASettingsWalkFailsOnARefusalThatCannotClear (0.17s)
./cmd/mesh-controller --- PASS: TestAProviderElsewhereServesItsOwnComposedName (0.24s)
./cmd/mesh-controller --- PASS: TestAReusedCheckBuildIsRegisteredByItsDigestAndGatedAsABuiltOne (0.29s)
./cmd/mesh-controller --- PASS: TestAReleaseSendsTheRestAtOnceAfterTheFirstPasses (0.25s)
./cmd/mesh-controller --- PASS: TestARestMachineThatFailsIsPutBackAloneAndTheOthersAreJudgedOn (0.23s)
./cmd/mesh-controller --- PASS: TestAWalkTakesOverTheReadingsOfAGateJudgingTheSameBuildThere (0.20s)
./cmd/mesh-controller --- PASS: TestAFirstSendMadeAndNotKeptIsNotSentAgain (0.25s)
./cmd/mesh-controller --- PASS: TestAReleaseSendKeptAndMadeIsNotMadeAgain (0.18s)
./cmd/mesh-controller --- PASS: TestAFirstMachineThatFailsStillStopsTheWalk (0.20s)
./cmd/mesh-controller --- PASS: TestTheControlNodeIsSentLastInAStep (0.22s)
./cmd/mesh-controller --- PASS: TestARestMachineSilentPastItsBoundIsLeftAndSaid (0.22s)
./cmd/mesh-controller --- PASS: TestANoMoveTierWaitsForNoNewReport (0.24s)
./cmd/mesh-controller --- PASS: TestOneProviderDownAndThreeConsumersFailingAreOneCondition (0.17s)
./cmd/mesh-controller --- PASS: TestAConsumerFailingBesideAHealthyProviderIsItsOwn (0.16s)
./internal/inventory --- PASS: TestTheLastSendSaysWhatTheMachineAnswered (0.08s)
./internal/inventory --- PASS: TestAReportIsTheLastSendsAnswerOnlyWhenItIsAboutThatSend (0.08s)
./internal/inventory --- PASS: TestAnIssuedValueIsSealedToEveryHolderOnceAndNotAgain (0.09s)
./internal/inventory --- PASS: TestAnIssuedValueIsSealedAgainAfterAKeyChangeOrAnAcceptance (0.09s)
./internal/inventory --- PASS: TestAConditionsLooksAreKeptBesideTheBucket (0.07s)
./internal/inventory --- PASS: TestAPlanIdHoldingAQuoteIsReadAsAPlainId (0.07s)
./internal/inventory --- PASS: TestAWalksStartIsKeptAndARefusedSaveLeavesItUnsaid (0.07s)
./internal/inventory --- PASS: TestAWalksRestIsReadFromEachMachinesOwnSend (0.08s)
./internal/inventory --- PASS: TestAStoredBuildWithASilentMergeableFileIsStillReadAndComposed (0.07s)
./internal/inventory --- PASS: TestACredentialIssuedOutsideIsNeverMade (0.08s)
./internal/inventory --- PASS: TestOneAcceptForAProviderIsSealedToEveryConsumer (0.08s)
./internal/inventory --- PASS: TestASharedOwnSecretIssuedOutsideIsNeverMade (0.07s)
./internal/inventory --- PASS: TestAReportOfABusyContainerIsKeptAndReadBack (0.08s)
Out of date when it landed: out of date: judged against trunk 535c7641ac, the trunk is now c975dbf7b053; judged again on a recheck or when it is merged
**Merge check** at `98e85687`
Judged: head `98e856874913` merged into trunk `535c7641ac41`; read beside it: mesh-catalog `50861c74f8b4`, mesh-controller-main `535c7641ac41`, mesh-host `1a2677882380`, mesh-tools `954e1a53eef9`
- `mesh/merge-gate`: **PASS** — every machine composes with the change as it did without (4 of 4 compose) (modules: build-agent, mesh-controller, route-proxy)
- `mesh/repo-check`: **PASS** — its merge-check.sh passed
**Delivery plan** — builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step
- tier 0: mesh-controller
- tier 1: build-agent
- tier 2: route-proxy
- ace: receives build-agent, route-proxy
- g14: receives build-agent
- novox: receives mesh-controller, build-agent, route-proxy
- shanks: receives build-agent
- build-agent keeps data (ADR 0233): what it holds is backed up before it moves
- route-proxy provides route: its consumers are sent again after it
- route-proxy keeps data (ADR 0233): what it holds is backed up before it moves
- grants step: if this changes what the bus's user list says (a new tool, subject or user), novox is sent that list alone, every build there kept, before build-agent, route-proxy is judged on its first machine
- read by no module's build: internal/lease/lease_test.go
Run by the build seat on shanks as `build-1791984061260125618` (`builds --log build-1791984061260125618`).
```
--- the gate: build-agent, mesh-controller, route-proxy
mesh-controller: ok, 60 tool(s); WARNING: merge-window ask(s) for a setting and do(es) not say whether it is trusted, so it counts as trusted: set at the terminal alone; say "trusted" false where nothing trusts it (novox/hq issue 339); WARNING: controller stay(s) up and say(s) not how it is ready — judged by liveness alone, refused from 2026-11-18 (ADR 0240 rule 8)
long-running resources without health: 1
files asking for a setting without saying whether it is trusted: 1
1 manifest(s) checked. Judged against the seats this binary carries; a claim on one of the mesh's own seats is judged fully at registration, and a seat declared by a module not given here reads as unknown. Identities judged on a 6-character machine name, against the bounds of the providers given here
judging 1 module(s) changed against 4 machine(s), as the snapshot of 2026-10-14T13:14:20Z says they run
composing every machine as the mesh is
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
composing every machine with the change
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
merge gate: PASS — every machine composes with the change as it did without (4 of 4 compose)
judged by controller development build against the facts of 2026-10-14T13:14:20Z
modules the change moves:
- mesh-controller (changed)
machines:
- a machine (f35): composes
- holds mesh-dns-resolver (idt): composes
- holds git, holds mesh-artifact-store, holds mesh-broker, holds mesh-catalog, holds mesh-delivery, holds mesh-dns-resolver, holds mesh-store, holds mesh-vault, holds npm-package-registry, the control node, the hub (ygcyr): composes
- a machine (yvbpsm): composes
a merge would rebuild 3 module(s) in 3 tier(s): build-agent, mesh-controller, route-proxy
notes:
- internal/lease/lease_test.go read by no module's build, and rebuild nothing (issue 280)
--- its merge-check.sh (go toolchain)
ok github.com/novox/mesh-controller/cmd/mesh-builder 1.094s
ok github.com/novox/mesh-controller/cmd/mesh-controller 183.910s
ok github.com/novox/mesh-controller/examples/objectstore-provisioner 1.012s
ok github.com/novox/mesh-controller/examples/postgres-provisioner 1.022s
ok github.com/novox/mesh-controller/examples/redis-provisioner 1.059s
ok github.com/novox/mesh-controller/examples/route-proxy 1.198s
ok github.com/novox/mesh-controller/internal/artifacts 1.080s
? github.com/novox/mesh-controller/internal/beside [no test files]
ok github.com/novox/mesh-controller/internal/broker 4.651s
ok github.com/novox/mesh-controller/internal/builder 41.865s
ok github.com/novox/mesh-controller/internal/catalogue 8.849s
ok github.com/novox/mesh-controller/internal/conditions 2.708s
ok github.com/novox/mesh-controller/internal/envfile 1.065s
ok github.com/novox/mesh-controller/internal/facts 1.072s
ok github.com/novox/mesh-controller/internal/identity 5.240s
ok github.com/novox/mesh-controller/internal/inventory 35.996s
ok github.com/novox/mesh-controller/internal/lease 30.921s
ok github.com/novox/mesh-controller/internal/licences 4.179s
ok github.com/novox/mesh-controller/internal/licences/adapters 1.065s
ok github.com/novox/mesh-controller/internal/link 59.726s
ok github.com/novox/mesh-controller/internal/lint 2.655s
ok github.com/novox/mesh-controller/internal/outward 1.085s
ok github.com/novox/mesh-controller/internal/overlay 1.070s
ok github.com/novox/mesh-controller/internal/secrets 1.525s
ok github.com/novox/mesh-controller/internal/store 3.484s
ok github.com/novox/mesh-controller/internal/testbus 1.236s
ok github.com/novox/mesh-controller/internal/token 1.016s
store tests, each run against the store:
./cmd/mesh-controller --- PASS: TestAStopWithoutAMergeWalksTheOthersAgain (0.22s)
./cmd/mesh-controller --- PASS: TestAStopWithoutAMergeEndsTheMergesThatContainIt (0.23s)
./cmd/mesh-controller --- PASS: TestAStopWalksTheOthersAgainInTheOpenBatch (0.22s)
./cmd/mesh-controller --- PASS: TestAStopNamesWhatStaysOnTheMachinesAndWhatStillCarriesIt (0.22s)
./cmd/mesh-controller --- PASS: TestAStopWithoutAMergeLeavesItOutOfABatch (0.20s)
./cmd/mesh-controller --- PASS: TestSavingPlansAndMovingMergesIsOneAct (0.18s)
./cmd/mesh-controller --- PASS: TestAPlanFollowedFromTheStoreAnswersWhenItMoves (1.97s)
./cmd/mesh-controller --- PASS: TestNodeShowAndStatusReadTheLastSendFromTheStore (0.25s)
./cmd/mesh-controller --- PASS: TestTheConditionsVerbReadsTheLooksTheBucketNoLongerHolds (0.13s)
./cmd/mesh-controller --- PASS: TestAWalkLetGoAndStoppedByVerbsIsSaidStartedAndEnded (0.25s)
./cmd/mesh-controller --- PASS: TestAWalkStoppedOrClosedByAPersonIsSaidEndedStopped (0.25s)
./cmd/mesh-controller --- PASS: TestAWalkRetriedByAPersonIsSaidStartedAgain (0.21s)
./cmd/mesh-controller --- PASS: TestABacklogReleasedByAPersonIsSaidStarted (0.21s)
./cmd/mesh-controller --- PASS: TestAWalkEndedByCancelOrClearIsSaidEnded (0.27s)
./cmd/mesh-controller --- PASS: TestAWalkRestartedByARebuildIsSaidStarted (0.22s)
./cmd/mesh-controller --- PASS: TestAWalkStoppedByHandAndRetriedEndsFailedWhenItFailsAgain (0.26s)
./cmd/mesh-controller --- PASS: TestReplay237 (0.20s)
./cmd/mesh-controller --- PASS: TestAnUnassignThatLeavesItsNodeUnresolvedSaysSoOnceInItsAnswer (0.20s)
./cmd/mesh-controller --- PASS: TestAnUnassignThatLeavesItsNodeResolvedSaysNothingOfIt (0.21s)
./cmd/mesh-controller --- PASS: TestAnAssignThatLeavesItsNodeUnresolvedSaysOnlyThePlansRefusal (0.21s)
./cmd/mesh-controller --- PASS: TestADirectoryOwnersKeyIsRefusedThroughAVerb (0.23s)
./cmd/mesh-controller --- PASS: TestTheSendMakesNoValueForAnOutsideSecretNobodyGave (0.29s)
./cmd/mesh-controller --- PASS: TestAQuestionComposesWithoutAnOutsideSecretNobodyGave (0.27s)
./cmd/mesh-controller --- PASS: TestReplay201 (0.17s)
./cmd/mesh-controller --- PASS: TestARollbackWhoseSendWasRefusedIsCompletedByTheNextSend (0.15s)
./cmd/mesh-controller --- PASS: TestAGateJudgesTheOtherMovesBesideAKeptCore (0.16s)
./cmd/mesh-controller --- PASS: TestReplay401 (0.31s)
./cmd/mesh-controller --- PASS: TestReplay201ThePlansSend (0.23s)
./cmd/mesh-controller --- PASS: TestPlanAndNodeListAnOptionalRequirementLeftUnbound (0.29s)
./cmd/mesh-controller --- PASS: TestReplay340 (0.26s)
./cmd/mesh-controller --- PASS: TestOnlyATrustedFilesNamedKeyIsRefusedThroughAVerb (0.19s)
./cmd/mesh-controller --- PASS: TestReplay349 (0.22s)
./cmd/mesh-controller --- PASS: TestReplay348AFaultFromBeforeTheSendFailsNoGate (0.19s)
./cmd/mesh-controller --- PASS: TestReplay239AModuleNameIsNotTakenOverByAnotherRepository (0.16s)
./cmd/mesh-controller --- PASS: TestAWalkThatMovedNothingSaysWhichModulesMatchedWhatRuns (0.19s)
./cmd/mesh-controller --- PASS: TestReplay614ANewSDKPlansTheToolchainAndThenItsBundles (0.08s)
./cmd/mesh-controller --- PASS: TestOneAcceptForAProviderIsPlacedForEveryConsumer (0.50s)
./cmd/mesh-controller --- PASS: TestReplay164 (0.29s)
./cmd/mesh-controller --- PASS: TestAnApprovedLayerOpensTheWalkThatSendsIt (0.20s)
./cmd/mesh-controller --- PASS: TestASettingsWalkSendsOneMachineAtATimeEachJudged (0.20s)
./cmd/mesh-controller --- PASS: TestAFailedSettingsWalkStopsAndHoldsNothingBack (0.19s)
./cmd/mesh-controller --- PASS: TestASettingsWalkSendsTheLayerAloneEveryBuildKept (0.19s)
./cmd/mesh-controller --- PASS: TestASettingsWalkStopsBeforeADataMoveAndPassesOverAMachineAway (0.54s)
./cmd/mesh-controller --- PASS: TestACrossingSendSupersedesASettingsGate (0.18s)
./cmd/mesh-controller --- PASS: TestASettingsWalkFailsOnARefusalThatCannotClear (0.17s)
./cmd/mesh-controller --- PASS: TestAProviderElsewhereServesItsOwnComposedName (0.24s)
./cmd/mesh-controller --- PASS: TestAReusedCheckBuildIsRegisteredByItsDigestAndGatedAsABuiltOne (0.29s)
./cmd/mesh-controller --- PASS: TestAReleaseSendsTheRestAtOnceAfterTheFirstPasses (0.25s)
./cmd/mesh-controller --- PASS: TestARestMachineThatFailsIsPutBackAloneAndTheOthersAreJudgedOn (0.23s)
./cmd/mesh-controller --- PASS: TestAWalkTakesOverTheReadingsOfAGateJudgingTheSameBuildThere (0.20s)
./cmd/mesh-controller --- PASS: TestAFirstSendMadeAndNotKeptIsNotSentAgain (0.25s)
./cmd/mesh-controller --- PASS: TestAReleaseSendKeptAndMadeIsNotMadeAgain (0.18s)
./cmd/mesh-controller --- PASS: TestAFirstMachineThatFailsStillStopsTheWalk (0.20s)
./cmd/mesh-controller --- PASS: TestTheControlNodeIsSentLastInAStep (0.22s)
./cmd/mesh-controller --- PASS: TestARestMachineSilentPastItsBoundIsLeftAndSaid (0.22s)
./cmd/mesh-controller --- PASS: TestANoMoveTierWaitsForNoNewReport (0.24s)
./cmd/mesh-controller --- PASS: TestOneProviderDownAndThreeConsumersFailingAreOneCondition (0.17s)
./cmd/mesh-controller --- PASS: TestAConsumerFailingBesideAHealthyProviderIsItsOwn (0.16s)
./internal/inventory --- PASS: TestTheLastSendSaysWhatTheMachineAnswered (0.08s)
./internal/inventory --- PASS: TestAReportIsTheLastSendsAnswerOnlyWhenItIsAboutThatSend (0.08s)
./internal/inventory --- PASS: TestAnIssuedValueIsSealedToEveryHolderOnceAndNotAgain (0.09s)
./internal/inventory --- PASS: TestAnIssuedValueIsSealedAgainAfterAKeyChangeOrAnAcceptance (0.09s)
./internal/inventory --- PASS: TestAConditionsLooksAreKeptBesideTheBucket (0.07s)
./internal/inventory --- PASS: TestAPlanIdHoldingAQuoteIsReadAsAPlainId (0.07s)
./internal/inventory --- PASS: TestAWalksStartIsKeptAndARefusedSaveLeavesItUnsaid (0.07s)
./internal/inventory --- PASS: TestAWalksRestIsReadFromEachMachinesOwnSend (0.08s)
./internal/inventory --- PASS: TestAStoredBuildWithASilentMergeableFileIsStillReadAndComposed (0.07s)
./internal/inventory --- PASS: TestACredentialIssuedOutsideIsNeverMade (0.08s)
./internal/inventory --- PASS: TestOneAcceptForAProviderIsSealedToEveryConsumer (0.08s)
./internal/inventory --- PASS: TestASharedOwnSecretIssuedOutsideIsNeverMade (0.07s)
./internal/inventory --- PASS: TestAReportOfABusyContainerIsKeptAndReadBack (0.08s)
```
**Out of date when it landed**: out of date: judged against trunk 535c7641ac41, the trunk is now c975dbf7b053; judged again on a recheck or when it is merged
build-agent keeps data (ADR 0233): what it holds is backed up before it moves
route-proxy provides route: its consumers are sent again after it
route-proxy keeps data (ADR 0233): what it holds is backed up before it moves
grants step: if this changes what the bus's user list says (a new tool, subject or user), novox is sent that list alone, every build there kept, before build-agent, route-proxy is judged on its first machine
read by no module's build: internal/lease/lease_test.go
Run by the build seat on ace as build-1791984564984525209 (builds --log build-1791984564984525209).
--- the gate: build-agent, mesh-controller, route-proxy
mesh-controller: ok, 60 tool(s); WARNING: merge-window ask(s) for a setting and do(es) not say whether it is trusted, so it counts as trusted: set at the terminal alone; say "trusted" false where nothing trusts it (novox/hq issue 339); WARNING: controller stay(s) up and say(s) not how it is ready — judged by liveness alone, refused from 2026-11-18 (ADR 0240 rule 8)
long-running resources without health: 1
files asking for a setting without saying whether it is trusted: 1
1 manifest(s) checked. Judged against the seats this binary carries; a claim on one of the mesh's own seats is judged fully at registration, and a seat declared by a module not given here reads as unknown. Identities judged on a 6-character machine name, against the bounds of the providers given here
judging 1 module(s) changed against 4 machine(s), as the snapshot of 2026-10-14T13:26:29Z says they run
composing every machine as the mesh is
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
composing every machine with the change
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
merge gate: PASS — every machine composes with the change as it did without (4 of 4 compose)
judged by controller development build against the facts of 2026-10-14T13:26:29Z
modules the change moves:
- mesh-controller (changed)
machines:
- a machine (f35): composes
- holds mesh-dns-resolver (idt): composes
- holds git, holds mesh-artifact-store, holds mesh-broker, holds mesh-catalog, holds mesh-delivery, holds mesh-dns-resolver, holds mesh-store, holds mesh-vault, holds npm-package-registry, the control node, the hub (ygcyr): composes
- a machine (yvbpsm): composes
a merge would rebuild 3 module(s) in 3 tier(s): build-agent, mesh-controller, route-proxy
--- its merge-check.sh (go toolchain)
ok github.com/novox/mesh-controller/cmd/mesh-builder 1.088s
ok github.com/novox/mesh-controller/cmd/mesh-controller 163.939s
ok github.com/novox/mesh-controller/examples/objectstore-provisioner 1.012s
ok github.com/novox/mesh-controller/examples/postgres-provisioner 1.079s
ok github.com/novox/mesh-controller/examples/redis-provisioner 1.073s
ok github.com/novox/mesh-controller/examples/route-proxy 1.120s
ok github.com/novox/mesh-controller/internal/artifacts 1.035s
? github.com/novox/mesh-controller/internal/beside [no test files]
ok github.com/novox/mesh-controller/internal/broker 4.036s
ok github.com/novox/mesh-controller/internal/builder 41.619s
ok github.com/novox/mesh-controller/internal/catalogue 7.558s
ok github.com/novox/mesh-controller/internal/conditions 2.596s
ok github.com/novox/mesh-controller/internal/envfile 1.075s
ok github.com/novox/mesh-controller/internal/facts 1.122s
ok github.com/novox/mesh-controller/internal/identity 4.372s
ok github.com/novox/mesh-controller/internal/inventory 37.462s
ok github.com/novox/mesh-controller/internal/lease 32.752s
ok github.com/novox/mesh-controller/internal/licences 3.613s
ok github.com/novox/mesh-controller/internal/licences/adapters 1.115s
ok github.com/novox/mesh-controller/internal/link 61.330s
ok github.com/novox/mesh-controller/internal/lint 2.353s
ok github.com/novox/mesh-controller/internal/outward 1.100s
ok github.com/novox/mesh-controller/internal/overlay 1.084s
ok github.com/novox/mesh-controller/internal/secrets 1.451s
ok github.com/novox/mesh-controller/internal/store 3.366s
ok github.com/novox/mesh-controller/internal/testbus 1.255s
ok github.com/novox/mesh-controller/internal/token 1.016s
store tests, each run against the store:
./cmd/mesh-controller --- PASS: TestAStopWithoutAMergeWalksTheOthersAgain (0.19s)
./cmd/mesh-controller --- PASS: TestAStopWithoutAMergeEndsTheMergesThatContainIt (0.20s)
./cmd/mesh-controller --- PASS: TestAStopWalksTheOthersAgainInTheOpenBatch (0.21s)
./cmd/mesh-controller --- PASS: TestAStopNamesWhatStaysOnTheMachinesAndWhatStillCarriesIt (0.19s)
./cmd/mesh-controller --- PASS: TestAStopWithoutAMergeLeavesItOutOfABatch (0.17s)
./cmd/mesh-controller --- PASS: TestSavingPlansAndMovingMergesIsOneAct (0.15s)
./cmd/mesh-controller --- PASS: TestAPlanFollowedFromTheStoreAnswersWhenItMoves (1.95s)
./cmd/mesh-controller --- PASS: TestNodeShowAndStatusReadTheLastSendFromTheStore (0.22s)
./cmd/mesh-controller --- PASS: TestTheConditionsVerbReadsTheLooksTheBucketNoLongerHolds (0.12s)
./cmd/mesh-controller --- PASS: TestAWalkLetGoAndStoppedByVerbsIsSaidStartedAndEnded (0.22s)
./cmd/mesh-controller --- PASS: TestAWalkStoppedOrClosedByAPersonIsSaidEndedStopped (0.22s)
./cmd/mesh-controller --- PASS: TestAWalkRetriedByAPersonIsSaidStartedAgain (0.19s)
./cmd/mesh-controller --- PASS: TestABacklogReleasedByAPersonIsSaidStarted (0.19s)
./cmd/mesh-controller --- PASS: TestAWalkEndedByCancelOrClearIsSaidEnded (0.25s)
./cmd/mesh-controller --- PASS: TestAWalkRestartedByARebuildIsSaidStarted (0.19s)
./cmd/mesh-controller --- PASS: TestAWalkStoppedByHandAndRetriedEndsFailedWhenItFailsAgain (0.22s)
./cmd/mesh-controller --- PASS: TestReplay237 (0.18s)
./cmd/mesh-controller --- PASS: TestAnUnassignThatLeavesItsNodeUnresolvedSaysSoOnceInItsAnswer (0.17s)
./cmd/mesh-controller --- PASS: TestAnUnassignThatLeavesItsNodeResolvedSaysNothingOfIt (0.18s)
./cmd/mesh-controller --- PASS: TestAnAssignThatLeavesItsNodeUnresolvedSaysOnlyThePlansRefusal (0.18s)
./cmd/mesh-controller --- PASS: TestADirectoryOwnersKeyIsRefusedThroughAVerb (0.20s)
./cmd/mesh-controller --- PASS: TestTheSendMakesNoValueForAnOutsideSecretNobodyGave (0.26s)
./cmd/mesh-controller --- PASS: TestAQuestionComposesWithoutAnOutsideSecretNobodyGave (0.24s)
./cmd/mesh-controller --- PASS: TestReplay201 (0.13s)
./cmd/mesh-controller --- PASS: TestARollbackWhoseSendWasRefusedIsCompletedByTheNextSend (0.13s)
./cmd/mesh-controller --- PASS: TestAGateJudgesTheOtherMovesBesideAKeptCore (0.14s)
./cmd/mesh-controller --- PASS: TestReplay401 (0.27s)
./cmd/mesh-controller --- PASS: TestReplay201ThePlansSend (0.20s)
./cmd/mesh-controller --- PASS: TestPlanAndNodeListAnOptionalRequirementLeftUnbound (0.25s)
./cmd/mesh-controller --- PASS: TestReplay340 (0.22s)
./cmd/mesh-controller --- PASS: TestOnlyATrustedFilesNamedKeyIsRefusedThroughAVerb (0.18s)
./cmd/mesh-controller --- PASS: TestReplay349 (0.18s)
./cmd/mesh-controller --- PASS: TestReplay348AFaultFromBeforeTheSendFailsNoGate (0.16s)
./cmd/mesh-controller --- PASS: TestReplay239AModuleNameIsNotTakenOverByAnotherRepository (0.13s)
./cmd/mesh-controller --- PASS: TestAWalkThatMovedNothingSaysWhichModulesMatchedWhatRuns (0.17s)
./cmd/mesh-controller --- PASS: TestReplay614ANewSDKPlansTheToolchainAndThenItsBundles (0.07s)
./cmd/mesh-controller --- PASS: TestOneAcceptForAProviderIsPlacedForEveryConsumer (0.41s)
./cmd/mesh-controller --- PASS: TestReplay164 (0.25s)
./cmd/mesh-controller --- PASS: TestAnApprovedLayerOpensTheWalkThatSendsIt (0.17s)
./cmd/mesh-controller --- PASS: TestASettingsWalkSendsOneMachineAtATimeEachJudged (0.18s)
./cmd/mesh-controller --- PASS: TestAFailedSettingsWalkStopsAndHoldsNothingBack (0.17s)
./cmd/mesh-controller --- PASS: TestASettingsWalkSendsTheLayerAloneEveryBuildKept (0.17s)
./cmd/mesh-controller --- PASS: TestASettingsWalkStopsBeforeADataMoveAndPassesOverAMachineAway (0.48s)
./cmd/mesh-controller --- PASS: TestACrossingSendSupersedesASettingsGate (0.16s)
./cmd/mesh-controller --- PASS: TestASettingsWalkFailsOnARefusalThatCannotClear (0.16s)
./cmd/mesh-controller --- PASS: TestAProviderElsewhereServesItsOwnComposedName (0.21s)
./cmd/mesh-controller --- PASS: TestAReusedCheckBuildIsRegisteredByItsDigestAndGatedAsABuiltOne (0.26s)
./cmd/mesh-controller --- PASS: TestAReleaseSendsTheRestAtOnceAfterTheFirstPasses (0.22s)
./cmd/mesh-controller --- PASS: TestARestMachineThatFailsIsPutBackAloneAndTheOthersAreJudgedOn (0.20s)
./cmd/mesh-controller --- PASS: TestAWalkTakesOverTheReadingsOfAGateJudgingTheSameBuildThere (0.17s)
./cmd/mesh-controller --- PASS: TestAFirstSendMadeAndNotKeptIsNotSentAgain (0.22s)
./cmd/mesh-controller --- PASS: TestAReleaseSendKeptAndMadeIsNotMadeAgain (0.17s)
./cmd/mesh-controller --- PASS: TestAFirstMachineThatFailsStillStopsTheWalk (0.17s)
./cmd/mesh-controller --- PASS: TestTheControlNodeIsSentLastInAStep (0.19s)
./cmd/mesh-controller --- PASS: TestARestMachineSilentPastItsBoundIsLeftAndSaid (0.19s)
./cmd/mesh-controller --- PASS: TestANoMoveTierWaitsForNoNewReport (0.19s)
./cmd/mesh-controller --- PASS: TestOneProviderDownAndThreeConsumersFailingAreOneCondition (0.14s)
./cmd/mesh-controller --- PASS: TestAConsumerFailingBesideAHealthyProviderIsItsOwn (0.14s)
./internal/inventory --- PASS: TestTheLastSendSaysWhatTheMachineAnswered (0.07s)
./internal/inventory --- PASS: TestAReportIsTheLastSendsAnswerOnlyWhenItIsAboutThatSend (0.07s)
./internal/inventory --- PASS: TestAnIssuedValueIsSealedToEveryHolderOnceAndNotAgain (0.08s)
./internal/inventory --- PASS: TestAnIssuedValueIsSealedAgainAfterAKeyChangeOrAnAcceptance (0.07s)
./internal/inventory --- PASS: TestAConditionsLooksAreKeptBesideTheBucket (0.06s)
./internal/inventory --- PASS: TestAPlanIdHoldingAQuoteIsReadAsAPlainId (0.06s)
./internal/inventory --- PASS: TestAWalksStartIsKeptAndARefusedSaveLeavesItUnsaid (0.06s)
./internal/inventory --- PASS: TestAWalksRestIsReadFromEachMachinesOwnSend (0.06s)
./internal/inventory --- PASS: TestAStoredBuildWithASilentMergeableFileIsStillReadAndComposed (0.06s)
./internal/inventory --- PASS: TestACredentialIssuedOutsideIsNeverMade (0.07s)
./internal/inventory --- PASS: TestOneAcceptForAProviderIsSealedToEveryConsumer (0.07s)
./internal/inventory --- PASS: TestASharedOwnSecretIssuedOutsideIsNeverMade (0.07s)
./internal/inventory --- PASS: TestAReportOfABusyContainerIsKeptAndReadBack (0.07s)
**Merge check** at `98e85687`
Judged: head `98e856874913` merged into trunk `c975dbf7b053`; read beside it: mesh-catalog `02427a60ef78`, mesh-controller-main `c975dbf7b053`, mesh-host `1a2677882380`, mesh-tools `954e1a53eef9`
- `mesh/merge-gate`: **PASS** — every machine composes with the change as it did without (4 of 4 compose) (modules: build-agent, mesh-controller, route-proxy)
- `mesh/repo-check`: **PASS** — its merge-check.sh passed
**Delivery plan** — builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step
- tier 0: mesh-controller
- tier 1: build-agent
- tier 2: route-proxy
- ace: receives build-agent, route-proxy
- g14: receives build-agent
- novox: receives mesh-controller, build-agent, route-proxy
- shanks: receives build-agent
- build-agent keeps data (ADR 0233): what it holds is backed up before it moves
- route-proxy provides route: its consumers are sent again after it
- route-proxy keeps data (ADR 0233): what it holds is backed up before it moves
- grants step: if this changes what the bus's user list says (a new tool, subject or user), novox is sent that list alone, every build there kept, before build-agent, route-proxy is judged on its first machine
- read by no module's build: internal/lease/lease_test.go
Run by the build seat on ace as `build-1791984564984525209` (`builds --log build-1791984564984525209`).
```
--- the gate: build-agent, mesh-controller, route-proxy
mesh-controller: ok, 60 tool(s); WARNING: merge-window ask(s) for a setting and do(es) not say whether it is trusted, so it counts as trusted: set at the terminal alone; say "trusted" false where nothing trusts it (novox/hq issue 339); WARNING: controller stay(s) up and say(s) not how it is ready — judged by liveness alone, refused from 2026-11-18 (ADR 0240 rule 8)
long-running resources without health: 1
files asking for a setting without saying whether it is trusted: 1
1 manifest(s) checked. Judged against the seats this binary carries; a claim on one of the mesh's own seats is judged fully at registration, and a seat declared by a module not given here reads as unknown. Identities judged on a 6-character machine name, against the bounds of the providers given here
judging 1 module(s) changed against 4 machine(s), as the snapshot of 2026-10-14T13:26:29Z says they run
composing every machine as the mesh is
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
composing every machine with the change
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
merge gate: PASS — every machine composes with the change as it did without (4 of 4 compose)
judged by controller development build against the facts of 2026-10-14T13:26:29Z
modules the change moves:
- mesh-controller (changed)
machines:
- a machine (f35): composes
- holds mesh-dns-resolver (idt): composes
- holds git, holds mesh-artifact-store, holds mesh-broker, holds mesh-catalog, holds mesh-delivery, holds mesh-dns-resolver, holds mesh-store, holds mesh-vault, holds npm-package-registry, the control node, the hub (ygcyr): composes
- a machine (yvbpsm): composes
a merge would rebuild 3 module(s) in 3 tier(s): build-agent, mesh-controller, route-proxy
--- its merge-check.sh (go toolchain)
ok github.com/novox/mesh-controller/cmd/mesh-builder 1.088s
ok github.com/novox/mesh-controller/cmd/mesh-controller 163.939s
ok github.com/novox/mesh-controller/examples/objectstore-provisioner 1.012s
ok github.com/novox/mesh-controller/examples/postgres-provisioner 1.079s
ok github.com/novox/mesh-controller/examples/redis-provisioner 1.073s
ok github.com/novox/mesh-controller/examples/route-proxy 1.120s
ok github.com/novox/mesh-controller/internal/artifacts 1.035s
? github.com/novox/mesh-controller/internal/beside [no test files]
ok github.com/novox/mesh-controller/internal/broker 4.036s
ok github.com/novox/mesh-controller/internal/builder 41.619s
ok github.com/novox/mesh-controller/internal/catalogue 7.558s
ok github.com/novox/mesh-controller/internal/conditions 2.596s
ok github.com/novox/mesh-controller/internal/envfile 1.075s
ok github.com/novox/mesh-controller/internal/facts 1.122s
ok github.com/novox/mesh-controller/internal/identity 4.372s
ok github.com/novox/mesh-controller/internal/inventory 37.462s
ok github.com/novox/mesh-controller/internal/lease 32.752s
ok github.com/novox/mesh-controller/internal/licences 3.613s
ok github.com/novox/mesh-controller/internal/licences/adapters 1.115s
ok github.com/novox/mesh-controller/internal/link 61.330s
ok github.com/novox/mesh-controller/internal/lint 2.353s
ok github.com/novox/mesh-controller/internal/outward 1.100s
ok github.com/novox/mesh-controller/internal/overlay 1.084s
ok github.com/novox/mesh-controller/internal/secrets 1.451s
ok github.com/novox/mesh-controller/internal/store 3.366s
ok github.com/novox/mesh-controller/internal/testbus 1.255s
ok github.com/novox/mesh-controller/internal/token 1.016s
store tests, each run against the store:
./cmd/mesh-controller --- PASS: TestAStopWithoutAMergeWalksTheOthersAgain (0.19s)
./cmd/mesh-controller --- PASS: TestAStopWithoutAMergeEndsTheMergesThatContainIt (0.20s)
./cmd/mesh-controller --- PASS: TestAStopWalksTheOthersAgainInTheOpenBatch (0.21s)
./cmd/mesh-controller --- PASS: TestAStopNamesWhatStaysOnTheMachinesAndWhatStillCarriesIt (0.19s)
./cmd/mesh-controller --- PASS: TestAStopWithoutAMergeLeavesItOutOfABatch (0.17s)
./cmd/mesh-controller --- PASS: TestSavingPlansAndMovingMergesIsOneAct (0.15s)
./cmd/mesh-controller --- PASS: TestAPlanFollowedFromTheStoreAnswersWhenItMoves (1.95s)
./cmd/mesh-controller --- PASS: TestNodeShowAndStatusReadTheLastSendFromTheStore (0.22s)
./cmd/mesh-controller --- PASS: TestTheConditionsVerbReadsTheLooksTheBucketNoLongerHolds (0.12s)
./cmd/mesh-controller --- PASS: TestAWalkLetGoAndStoppedByVerbsIsSaidStartedAndEnded (0.22s)
./cmd/mesh-controller --- PASS: TestAWalkStoppedOrClosedByAPersonIsSaidEndedStopped (0.22s)
./cmd/mesh-controller --- PASS: TestAWalkRetriedByAPersonIsSaidStartedAgain (0.19s)
./cmd/mesh-controller --- PASS: TestABacklogReleasedByAPersonIsSaidStarted (0.19s)
./cmd/mesh-controller --- PASS: TestAWalkEndedByCancelOrClearIsSaidEnded (0.25s)
./cmd/mesh-controller --- PASS: TestAWalkRestartedByARebuildIsSaidStarted (0.19s)
./cmd/mesh-controller --- PASS: TestAWalkStoppedByHandAndRetriedEndsFailedWhenItFailsAgain (0.22s)
./cmd/mesh-controller --- PASS: TestReplay237 (0.18s)
./cmd/mesh-controller --- PASS: TestAnUnassignThatLeavesItsNodeUnresolvedSaysSoOnceInItsAnswer (0.17s)
./cmd/mesh-controller --- PASS: TestAnUnassignThatLeavesItsNodeResolvedSaysNothingOfIt (0.18s)
./cmd/mesh-controller --- PASS: TestAnAssignThatLeavesItsNodeUnresolvedSaysOnlyThePlansRefusal (0.18s)
./cmd/mesh-controller --- PASS: TestADirectoryOwnersKeyIsRefusedThroughAVerb (0.20s)
./cmd/mesh-controller --- PASS: TestTheSendMakesNoValueForAnOutsideSecretNobodyGave (0.26s)
./cmd/mesh-controller --- PASS: TestAQuestionComposesWithoutAnOutsideSecretNobodyGave (0.24s)
./cmd/mesh-controller --- PASS: TestReplay201 (0.13s)
./cmd/mesh-controller --- PASS: TestARollbackWhoseSendWasRefusedIsCompletedByTheNextSend (0.13s)
./cmd/mesh-controller --- PASS: TestAGateJudgesTheOtherMovesBesideAKeptCore (0.14s)
./cmd/mesh-controller --- PASS: TestReplay401 (0.27s)
./cmd/mesh-controller --- PASS: TestReplay201ThePlansSend (0.20s)
./cmd/mesh-controller --- PASS: TestPlanAndNodeListAnOptionalRequirementLeftUnbound (0.25s)
./cmd/mesh-controller --- PASS: TestReplay340 (0.22s)
./cmd/mesh-controller --- PASS: TestOnlyATrustedFilesNamedKeyIsRefusedThroughAVerb (0.18s)
./cmd/mesh-controller --- PASS: TestReplay349 (0.18s)
./cmd/mesh-controller --- PASS: TestReplay348AFaultFromBeforeTheSendFailsNoGate (0.16s)
./cmd/mesh-controller --- PASS: TestReplay239AModuleNameIsNotTakenOverByAnotherRepository (0.13s)
./cmd/mesh-controller --- PASS: TestAWalkThatMovedNothingSaysWhichModulesMatchedWhatRuns (0.17s)
./cmd/mesh-controller --- PASS: TestReplay614ANewSDKPlansTheToolchainAndThenItsBundles (0.07s)
./cmd/mesh-controller --- PASS: TestOneAcceptForAProviderIsPlacedForEveryConsumer (0.41s)
./cmd/mesh-controller --- PASS: TestReplay164 (0.25s)
./cmd/mesh-controller --- PASS: TestAnApprovedLayerOpensTheWalkThatSendsIt (0.17s)
./cmd/mesh-controller --- PASS: TestASettingsWalkSendsOneMachineAtATimeEachJudged (0.18s)
./cmd/mesh-controller --- PASS: TestAFailedSettingsWalkStopsAndHoldsNothingBack (0.17s)
./cmd/mesh-controller --- PASS: TestASettingsWalkSendsTheLayerAloneEveryBuildKept (0.17s)
./cmd/mesh-controller --- PASS: TestASettingsWalkStopsBeforeADataMoveAndPassesOverAMachineAway (0.48s)
./cmd/mesh-controller --- PASS: TestACrossingSendSupersedesASettingsGate (0.16s)
./cmd/mesh-controller --- PASS: TestASettingsWalkFailsOnARefusalThatCannotClear (0.16s)
./cmd/mesh-controller --- PASS: TestAProviderElsewhereServesItsOwnComposedName (0.21s)
./cmd/mesh-controller --- PASS: TestAReusedCheckBuildIsRegisteredByItsDigestAndGatedAsABuiltOne (0.26s)
./cmd/mesh-controller --- PASS: TestAReleaseSendsTheRestAtOnceAfterTheFirstPasses (0.22s)
./cmd/mesh-controller --- PASS: TestARestMachineThatFailsIsPutBackAloneAndTheOthersAreJudgedOn (0.20s)
./cmd/mesh-controller --- PASS: TestAWalkTakesOverTheReadingsOfAGateJudgingTheSameBuildThere (0.17s)
./cmd/mesh-controller --- PASS: TestAFirstSendMadeAndNotKeptIsNotSentAgain (0.22s)
./cmd/mesh-controller --- PASS: TestAReleaseSendKeptAndMadeIsNotMadeAgain (0.17s)
./cmd/mesh-controller --- PASS: TestAFirstMachineThatFailsStillStopsTheWalk (0.17s)
./cmd/mesh-controller --- PASS: TestTheControlNodeIsSentLastInAStep (0.19s)
./cmd/mesh-controller --- PASS: TestARestMachineSilentPastItsBoundIsLeftAndSaid (0.19s)
./cmd/mesh-controller --- PASS: TestANoMoveTierWaitsForNoNewReport (0.19s)
./cmd/mesh-controller --- PASS: TestOneProviderDownAndThreeConsumersFailingAreOneCondition (0.14s)
./cmd/mesh-controller --- PASS: TestAConsumerFailingBesideAHealthyProviderIsItsOwn (0.14s)
./internal/inventory --- PASS: TestTheLastSendSaysWhatTheMachineAnswered (0.07s)
./internal/inventory --- PASS: TestAReportIsTheLastSendsAnswerOnlyWhenItIsAboutThatSend (0.07s)
./internal/inventory --- PASS: TestAnIssuedValueIsSealedToEveryHolderOnceAndNotAgain (0.08s)
./internal/inventory --- PASS: TestAnIssuedValueIsSealedAgainAfterAKeyChangeOrAnAcceptance (0.07s)
./internal/inventory --- PASS: TestAConditionsLooksAreKeptBesideTheBucket (0.06s)
./internal/inventory --- PASS: TestAPlanIdHoldingAQuoteIsReadAsAPlainId (0.06s)
./internal/inventory --- PASS: TestAWalksStartIsKeptAndARefusedSaveLeavesItUnsaid (0.06s)
./internal/inventory --- PASS: TestAWalksRestIsReadFromEachMachinesOwnSend (0.06s)
./internal/inventory --- PASS: TestAStoredBuildWithASilentMergeableFileIsStillReadAndComposed (0.06s)
./internal/inventory --- PASS: TestACredentialIssuedOutsideIsNeverMade (0.07s)
./internal/inventory --- PASS: TestOneAcceptForAProviderIsSealedToEveryConsumer (0.07s)
./internal/inventory --- PASS: TestASharedOwnSecretIssuedOutsideIsNeverMade (0.07s)
./internal/inventory --- PASS: TestAReportOfABusyContainerIsKeptAndReadBack (0.07s)
```
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Fixes novox/hq issue 477 (a bus stall of tens of seconds ends the controller, though no other controller could take its lease).
What was seen
2026-10-14 13:04:10 UTC:
LOST the controller lease (epoch 136861): the renewal could not be made: context deadline exceeded — this controller stops acting at once, then exit; systemd restarted it and the new process took epoch 140306 at 13:04:19. The bus had stalled only about 3 s ($JS.API.STREAM.INFO... took too long: 3.16s). The operator got the urgent "The controller lost its lease" on every channel. Only one controller ever ran. Same on 2026-10-11 01:31-01:35.The rule (decided, the operator's standing rule)
A renewal that fails is not a lost lease. The controller keeps the lease until the key's TTL has passed without a successful renewal. It starts no new act once half the TTL has passed unrenewed, and resumes on the next successful renewal. It is lost only when the TTL has passed, or another holder is seen. The urgent lease-lost condition is raised only on a real loss.
The change
internal/lease:RenewanswersErrDelayedfor a renewal the bus did not answer, keeps the lease and says it once per run of failures;Keepcarries on and loses the lease when the TTL passes unrenewed (a timer, so a hanging renewal cannot stretch it; one renewal's timeout is capped at the TTL left).Epoch, which every act passes) now closes at half the TTL unrenewed (or TTL less the margin, if sooner) instead of TTL less the margin, and opens again on the next renewal made.Options.Delayed(why, renewed, lostBy)is told of each delayed renewal; nothing sets it yet.lease-loststays raised only from an epoch ended as lost (now only a real loss) or a lease unrenewed past its TTL; its "lost" wording says the two real causes.Timeline in production (TTL 15 s, renewal every 5 s): renewal at 0 made; a stall from 1 s; the renewal at 5 s fails, said once, lease kept; at 7.5 s no new act starts; a renewal made before 15 s opens the gate again and says so; none made by 15 s → lost and exit as today.
Tests
internal/lease(real bus, design bounds divided by five): TestAFailedRenewalKeepsTheLeaseUntilItsTTL, TestARenewalMadeAfterAStallOpensTheGateAgain, TestARenewalWrittenThoughItsAnswerWasLostIsStillTheHolders, TestAnotherHolderSeenAfterAStallIsALossAtOnce; the existing tests (refusal is a loss at once, a holder that stops renewing stops acting before it is taken over, handover) still pass.go test -race -count=3 ./internal/lease/passes;go vet ./...clean. TestTwoControllersOneActs needs the store and runs in the merge check.Acceptance
A bus stall shorter than the lease's 15 s TTL no longer ends the controller or raises the urgent lease-lost condition; the controller pauses new acts after 7.5 s unrenewed and acts again on its next renewal, and it still exits at once when another controller writes the key.
Delivery
novox/mesh-controller@98e856874913— delivered since 2026-10-14T13:45:18ZDelivery plan — builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step
Machines
Transitions
The commit's note under
refs/notes/mesh-plankeeps every transition:git log --notes=mesh-plan.Merge check at
98e85687Judged: head
98e856874913merged into trunk535c7641ac41; read beside it: mesh-catalog50861c74f8b4, mesh-controller-main535c7641ac41, mesh-host1a2677882380, mesh-tools954e1a53eef9mesh/merge-gate: PASS — every machine composes with the change as it did without (4 of 4 compose) (modules: build-agent, mesh-controller, route-proxy)mesh/repo-check: PASS — its merge-check.sh passedDelivery plan — builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step
Run by the build seat on shanks as
build-1791984061260125618(builds --log build-1791984061260125618).Out of date when it landed: out of date: judged against trunk
535c7641ac, the trunk is now c975dbf7b053; judged again on a recheck or when it is mergedMerge check at
98e85687Judged: head
98e856874913merged into trunkc975dbf7b053; read beside it: mesh-catalog02427a60ef78, mesh-controller-mainc975dbf7b053, mesh-host1a2677882380, mesh-tools954e1a53eef9mesh/merge-gate: PASS — every machine composes with the change as it did without (4 of 4 compose) (modules: build-agent, mesh-controller, route-proxy)mesh/repo-check: PASS — its merge-check.sh passedDelivery plan — builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step
Run by the build seat on ace as
build-1791984564984525209(builds --log build-1791984564984525209).