Design 47's last milestones: L3's engine re-landed with held modules never removed, provision secrets, the node's own health report (issues 774, 1126, 1954, 1952) #177

2026-10-30T00:21:42Z

mesh-tools commented

2026-10-30 00:21:42 +00:00

Delivery 2 of design 47's single release (group feat/774-l3-reland). L3 re-land: the engine keeps every composition it applied (applied/), fills it at first start from what runs, and refuses removing a resource of a module no want removes — so the 2026-10-29 removal of nats on novox cannot recur; the object keep (issue 924); provision secrets filled from the vault with name and generation checked; the node's own health report in mesh-host_health (network with targets, units, relogin, found folders, root verdicts). Pins mesh-sdk 2da0de75; unit and integration 1260 pass.

Reviewed-by: rev-47@shanks at 015334020a

Delivery 2 of design 47's single release (group feat/774-l3-reland). L3 re-land: the engine keeps every composition it applied (applied/), fills it at first start from what runs, and refuses removing a resource of a module no want removes — so the 2026-10-29 removal of nats on novox cannot recur; the object keep (issue 924); provision secrets filled from the vault with name and generation checked; the node's own health report in mesh-host_health (network with targets, units, relogin, found folders, root verdicts). Pins mesh-sdk 2da0de75; unit and integration 1260 pass. Reviewed-by: rev-47@shanks at 015334020ad2f06e8dede19c0a1175d0c1563f7a

mesh-tools added 22 commits 2026-10-30 00:21:42 +00:00

Revert "Merge pull request 'Revert L3: its node-engine removed every held module on novox, the bus server among them (issue 774)' (#174 ) from revert/774-l3 into main" 45d0348044

This reverts commit 1ec73915b8, reversing
changes made to 0a4c35f731.

Judge a module the agent has not taken yet by the composition its have names, and knock the filling at the start and whenever a module is taken as running (issue 924) d8de0a8853

A held module on a node composing its own never gets a composition, so the agent never took it and the filling
judged its files by the kept declaration, older than power's three new drop-ins: refused from every engine start
on ace, g14 and shanks until some module settled (2026-10-29 00:09 UTC). The second meeting of issue 924.

Quick fix: knock the filling once the haves are read at the start. Thorough fix (chosen, the kind recurs for every
file added since the switch): what runs of such a module is the composition its have names, from the bucket or
its revisions, ahead of the kept declaration; the filling is knocked on every change of what runs, not only on a
settle. One unit test replaced by a table of four cases; the count stays.

What runs from the kept declaration says Data as the module's manifest does, and only a have that runs counts at the start (issue 924) f38ac87ac9

The kept-declaration fallback built a composition without Data, so after a restart a data module (mongodb, mssql
on novox) was judged to move its data against its first composition and the operator was asked a false yes. Now
every such composition takes Data from the manifest (agent.Env.HoldsData, the composer's DataItems), and a module
whose manifest cannot be read is said to be unjudgeable rather than taken to hold none. The composition a have
names comes before the kept declaration in the judgement too.

loadHad takes a have only in the phases baseline takes as running (running, held, a composer's mark): one backing
up, applying, checking, failed or waiting names a composition not applied (review of 031e106). Two rows added to
the same table test; the count stays.

Keep the content of every file given by digest on the node, so an object the controller's sweep removed never refuses a composition the node already had (issue 924) 7d39510450

On novox at 01:16 on 2026-10-29, after the node began composing its own modules, mesh-vault's apply failed while the
vault ran: its own composition carries the export as it runs (mesh-sdk frombus KeptRuns) by a digest whose object the
controller's sweep (mesh-controller cmd/mesh-controller/declaration_sweep.go, sweepNode and sweepDeclarationFiles)
had removed, since the sweep keeps only what the controller's compositions in composed-<node> and the wants name.
The node may only read its bucket (ADR 0486 §2), so the node keeps what it had: a file here that verifies, or an
object fetched, goes to <state dir>/objects under the object's name, and a kept one is used before the bus is asked.
An object unused for 14 days is let go. A fetch that still fails says the keep never held it, before its cause.

Quick fix weighed: re-put the object from the controller's store. Thorough (chosen): the node never depends on the
bucket for content it once had, for every module, since the sweep and a node composing its own can always disagree.
The whole-apply test gains three rows that fail on the old code; the test count stays.

Compose a file that runs as the node last wrote it, so a stale kept declaration's export no longer names a swept object (issue 924) d2645a27e9

novox's mesh-vault has failed since 01:16 on 2026-10-29: its have says failed, so loadHad (fbea777) takes no
composition from it, and what runs falls back to the kept declaration, whose export (mesh-vault.kept) names
f099eaa1, an object the controller's sweep removed. The node wrote a newer export since. The composer carries that
file as it runs (mesh-sdk frombus keptOf), so what runs now names the digest and size of what is on disk, but only
where the node's record says it wrote exactly that content at that path (Applied.Wrote, Target). The apply then
finds the file in place and needs no object.

Quick fix weighed: have the controller put the object back. Thorough (chosen, in the node): what runs is what the node
wrote, never a guess from an older declaration. The rows join the existing whole-apply test; the test count stays.

Refuse an apply that would remove a resource of a module a want still places here, so a composition that lost what runs can never remove the bus server or a held module again (issue 774) a7acee3eed

Pin mesh-sdk 22c9ddd, so a carried or held module serves its tools from what runs and a data flag alone moves no data (issues 1126, 924) acf02d52ff

Keep every composition the engine applied on the machine, so a module whose composition the bus no longer keeps runs as applied and is never composed as nothing (issue 774) 76e214a8b9

On novox on 2026-10-29 at 07:54 UTC the kept declaration was retired and the node's own composition was in no bucket.
The composer's first pass reads what runs before the agent reads any have (the agent waits for the composer's caught,
sent after that pass), so the node's own was composed with its folded modules running nothing, and the bus server and
four held modules were removed.

- Quick fix: read the node's own have in loadHad too. It still waits for the composer's first pass, and still finds
  nothing when the bucket no longer keeps the composition the have names.
- Thorough fix, chosen: the engine keeps each composition it applied (the node's own too) in `applied/` beside its
  state, 0600. The agent reads it at Run's start, before any watch, as what runs ahead of the bucket and the kept
  declaration. A have naming another running composition wins. An apply keeps; a removal forgets.

Test: a case in TestEverythingIsWhatTheAgentRunsAndWhatIsKeptForTheRest (no new test function). It fails on L3's
engine 4f9227a (0 compositions at the first own composition) and passes here.

Keep the node's own as it runs from an engine's first start, and refuse the node's own apply removing a module no want removed (issue 774, review round 1) 81672b183d

The reviewer showed how 76e214a still lets the 07:54 outage happen at the re-landed engine's first start on novox.
applied/ is empty there, so loadApplied loads nothing. The kept declaration retires, because every placed module has a
have. Nothing then carries the node's own composition. The guard misses nats, which has no want there, so its
nats.conf is guessed as nats's and removed.

- At the start, when the store has no node's own composition, the node's own runs as the kept declaration has it, and
  the store keeps it now. The kept declaration retires only once the store holds `node`.
- What this engine applied runs until loadHad reads the composition a differing have names. The bucket no longer keeps
  the one novox's have names, so the stored one stays.
- In the node's own apply, a resource of another module goes only when that module's want says removed
  (Scope.RemovedByWant). A module the node's own composition carried has no want, so Placed never named it.
  Cost: the node's own apply cannot drop one of its own dotted resources without such a want. It is refused whole and
  said, never removed silently.

Tests, with no new test functions:
- A first-start case in the agent's Everything test (empty store, then a start with the declaration retired). It fails
  on 76e214a.
- Two rows in the guard's table: no want refuses; a want saying removed removes.
- A case for the store in the authority test.

Pin mesh-sdk 7000bd5, the re-land group's sdk head: a tools bundle declares its own bus account (issue 1952 S5, issue 774) 58aacaad2d

Pin mesh-sdk 2e5cc4f, the re-land group's sdk head (issue 774) f7814853f9

Fill a provision's ${secret:} at the write from the vault's value sealed to this machine, so the provisioner's client never holds it (issue 1954 L6-E1, E2; ADR 0570 clause 6) 6b32e07164

The provisioner's client fills a handed module's file but for each `${secret:<provision>}` (or a whole `secrets` file
that is the placeholder alone).
- The node-engine reads the provision's key `<node>.<consumer>.<provision>[.<local>]` in mesh-provisioner's
  `provisions`. The key must be `provided` and carry the vault's reference.
- It asks the vault's `own` for that pair credential sealed to this machine, at least at the reference's generation,
  and opens it with the node's sealing key.
- It writes the file as declared (its owner), 0600 when the declaration gives no mode.
- Anything short of that refuses the write, naming the provision and why.

A module the node composes itself already has its `secrets` files written so: its composition carries the vault's
value from `own`, written 0600 with the declared owner.

Pins mesh-sdk 4a96793 (go/provisions). Tests: rows in TestAWriteAndAnApplyOfTheSameFileSerialise.

Refuse a provision's value unless the vault's credential name and generation are the ones its reference names (issue 1954 L6-E1; ADR 0570 clauses 1, 6) 5497d362dd

The value written is the one the provision names: a1-47 makes Reference.ID the vault's credential name. A mismatch in
either refuses the write, naming both. Test: a row in TestAWriteAndAnApplyOfTheSameFileSerialise.

Resolve ${secret:<name>} through this node's provision keys, and check the reference by the vault's own name (issue 1954 L6-E1, E2; ADR 0570 clauses 1, 6; ADR 0094) cbe8528f91

- A placeholder is `${secret:<name>}`, the name the module calls the credential (catalogue.SecretLocal): the local
  name where it keeps several values of one provision, else the provision. The `<provision>.<local>` form goes.
- The engine finds the one key under `<node>.<module>.` of mesh-provisioner's `provisions` going by that name. That is
  what node.<node> may read, by a watch to its caught-up mark; plans are not read.
- The reference's ID must be a credential name of that provision (provisions.VaultName). The vault's answer must carry
  exactly that name and generation. Anything else refuses the write, naming both.

Pins mesh-sdk 66ff33d. Tests: rows in TestAWriteAndAnApplyOfTheSameFileSerialise (a local name, a name no provision
goes by, a reference to no credential of the provision, a mismatched answer).

Write each node's health report into its own key of mesh-host_health on a change, so the watchman raises the machine's alerts without the controller (issue 774, the operator's alert table) 65fbc1e6f8

Until now the six machine alerts came from a health message passing to the controller. The report is now kept in the
engine's declared state `health`, key `<node>`, created by the grants part as `acts` is. Only this node's engine
writes it, after comparing it on every look and only when what it says changed; it is read first after a restart.

It carries:
- the network parts (machine-network; network-rewritten through the resolver file part's writer and owner);
- the tunnel's peers, each seen within five minutes or not (network-unreachable);
- the failed units no module owns (machine-units);
- the accounts marked relogin needed (relogin-needed);
- the directories used as found (directory-used-as-found).

The streaks and the time of the look are left out, so a look alone writes nothing. A peer is reported seen or not,
never its handshake time, so a renewed handshake writes nothing.

The layout is internal/link's (NodeHealth) until it moves into mesh-sdk. Pins mesh-sdk 58a8324.

Tests:
- rows in TestOnTheHubAnyFreshPeerIsAHealthyTunnel: the peers, and a change only when one is seen or lost;
- a case in TestAHealthStatementIsSaidOnTheLinkOpenNow: what the report takes, and one write per change across a
  restart;
- the manifest's states in TestTheManifestConsumesTheBackupEvent.

Take the node's health report layout from mesh-sdk 94d0de5, so the engine and the watchman read one form (issue 774) 552f97c919

delivery.NodeHealth and its parts replace this package's copies. The message's network and units parts are aliases of
the same types, so its JSON is unchanged. The engine keeps the derivation from its statement and the write on a change.
Test: the key's bytes, decoded and encoded again by the sdk, are the same bytes.

Format the health report's aliases as gofmt does (issue 774) 3ed9abf658

Name the machines a failing network part points at in the node's health report, so the watchman holds it where the controller does (issue 774) c3ff18de4e

delivery.NetworkPart.Targets (mesh-sdk 6eae076) is filled where the report is made, by the controller's own rule:
- the hub's node for "hub";
- each resolver's node by its address on the private network;
- none when the part points at nothing, at this machine, or at an address no other mesh machine has.

The names come from the placements the node's own composer holds (Decided.Address, Decided.Hub). A spoke's tunnel names
only the hub, so the tunnel cannot name a resolver on another spoke. A node that does not compose its own names no
target.

Test: rows in TestAHealthStatementIsSaidOnTheLinkOpenNow (hub, two resolvers, a VPN's resolver, its own, a healthy
part).

Say each account's root verdict and the engine's contract in the node's health report, so the self-check's DA and D-root judge without the controller (issue 1952, ADR 0266) e5e8ff5783

Every account resource marked root never goes into NodeHealth.Root; the statement's contract into Contract. Relogin,
root and found are judged each on its own, so an account may be both. Pins mesh-sdk f3677eb. Test: the report's row in
TestAHealthStatementIsSaidOnTheLinkOpenNow carries the agent account's verdict and the contract.

Pin mesh-sdk 007b6fd, the release's final sdk (issue 774) 192633edb8

Pin mesh-sdk 6e4a301, the release's final sdk (issue 774) 02f78c9949

The whole integration suite passes at this head against a local bus (JetStream, and a bus with the witness grants) and
the release's mesh-catalog and mesh-controller beside it: 1260 tests, 7 skipped for a container or a throwaway machine.

Merge main (delivery 1, the parse-only sdk) into L3's re-land, and pin mesh-sdk 2da0de7, the release's final head (issue 774)

mesh/check checking 015334020ad2 on a3c098a5c618 | novox/mesh-host#177 in group feat/774-l3-reland

mesh/delivery-group group feat/774-l3-reland merged: every member is merged

Details

mesh/delivery merged: source.main moved to novox/mesh-host@a9edd2951c46

Details

015334020a

The conflicts were go.mod, go.sum and testdata/beside/CAPTURED alone, taken from the release. 2da0de7 is 6e4a301's code;
its conformance events are unchanged.

mesh-tools commented

2026-10-30 00:22:15 +00:00

Delivery novox/mesh-host@015334020ad2 — merged since 2026-10-30T11:02:18Z

Source — source.main moved to novox/mesh-host@a9edd2951c. Whether it reached the machines is mesh-delivery-view's, derived from source.main, the wants and the haves.

Group feat/774-l3-reland: novox/hq@0c450cad25, novox/mesh-catalog@f3a57dcc56, novox/mesh-controller@8b32e04e2f, novox/mesh-delivery@8edc83950c, novox/mesh-host@015334020a, novox/mesh-sdk@2da0de75c3, novox/mesh-tools@250aa73a56

Transitions

2026-10-30 00:21 (new) → proposed (announced): novox/mesh-host#177's head announced
2026-10-30 00:22 proposed → ready (accepted): a verdict names this commit and the trunk's head as its trunk commit, the gate passed or warned, and the repository's own check did not fail; or the check gates no merge (check-gates-merge false)
2026-10-30 11:02 ready → merged (merged): on the trunk its modules follow: merged there

The commit's note under refs/notes/mesh-plan keeps every transition of this change up to its merge: git log --notes=mesh-plan.

**Delivery** `novox/mesh-host@015334020ad2` — **merged** since 2026-10-30T11:02:18Z **Source** — source.main moved to novox/mesh-host@a9edd2951c46. Whether it reached the machines is `mesh-delivery-view`'s, derived from `source.main`, the wants and the haves. **Group** `feat/774-l3-reland`: novox/hq@0c450cad258e, novox/mesh-catalog@f3a57dcc56fb, novox/mesh-controller@8b32e04e2f8b, novox/mesh-delivery@8edc83950cea, novox/mesh-host@015334020ad2, novox/mesh-sdk@2da0de75c3f0, novox/mesh-tools@250aa73a5631 **Transitions** - 2026-10-30 00:21 (new) → proposed (announced): novox/mesh-host#177's head announced - 2026-10-30 00:22 proposed → ready (accepted): a verdict names this commit and the trunk's head as its trunk commit, the gate passed or warned, and the repository's own check did not fail; or the check gates no merge (check-gates-merge false) - 2026-10-30 11:02 ready → merged (merged): on the trunk its modules follow: merged there The commit's note under `refs/notes/mesh-plan` keeps every transition of this change up to its merge: `git log --notes=mesh-plan`.

mesh-tools commented

2026-10-30 08:58:52 +00:00

mesh-delivery: the operator's Approve of this pull request's merge into main is ended: a head it covers moved on: the heads it covers are no longer the group's heads. Nothing is merged on it. The operator is asked again once the checks pass at the new head.

mesh-tools commented

2026-10-30 09:36:07 +00:00

mesh-delivery: not merged. The merge of this pull request into main at 015334020a was declined: the operator, via telegram (user id verified), chose Decline. A new head asks the operator again.

mesh-delivery: **not merged.** The merge of this pull request into main at 015334020ad2f06e8dede19c0a1175d0c1563f7a was declined: the operator, via telegram (user id verified), chose Decline. A new head asks the operator again.

mesh-tools merged commit a9edd2951c into main

2026-10-30 11:02:16 +00:00

mesh-tools deleted branch feat/774-l3-reland

2026-10-30 11:02:16 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: novox/mesh-host#177