--- status: accepted date: 2026-08-27 deciders: jochen reconstructed: false extends: 0014-build-publish-and-deploy-are-three-silos.md --- # 58. Delivery ends in a declaration, not in a push to a node ## Context Today a push produces a pipeline with three silos, and the third — **deploy** — runs *once per module per node*, sending a command to every assigned node telling it to install, configure, start and verify ([`00-as-is/04`](../03-DESIGN/00-as-is/04-delivery.md)). That third silo is where the as-is document records the most damage: - *"A green pipeline proves transport, not effect."* The stages report a message was dispatched and accepted, not that anything is running. - A service reported started when the container command merely returned. An image pull failure that did not fail the deploy. **A package install that 404ed from every mirror while the job went green** ([04-ISSUES/001](../04-ISSUES/001-failed-package-install-reports-success/00-report.md)). - A node left on old code after a failed download, with a version marker that had already advanced. - A verify stage built to close the gap, and *never scheduled*, because the coordinator's stage list did not include it. Meanwhile [ADR 0037](0037-the-host-applies-it-does-not-decide.md) has given every node a component that does exactly what deploy does — applies state, reads back, reports — and does it continuously rather than once per pipeline. **Two mechanisms now change a node**, and only one of them checks its work. ## Decision **A pipeline ends when the declaration is updated. The node applies it.** The three silos become: | Silo | Runs | Ends with | |---|---|---| | **build** | once per module | a self-contained artifact | | **publish** | once per module | that artifact addressable — an image by digest, a package in the mesh's repository | | **deploy** | **once, not once per node** | the affected nodes' **declarations updated** to name the new version | **Deploy stops sending commands to nodes.** It changes what the control plane says each node should be, which is one write. What happens on the machines is the host's ordinary reconcile, on whatever schedule each node is on. ### Why this fixes the failure class rather than patching it The as-is faults share one shape: **the thing that reported success was not the thing that did the work.** A coordinator dispatching a command can only report on dispatch. Under this decision the reporter *is* the applier. The host already refuses to record a resource until it read it back ([ADR 0035](0035-a-picture-is-read-from-what-runs.md)), and already fails the whole apply on one failed step ([ADR 0008](0008-a-failed-step-fails-the-job.md)). A package that 404s cannot go green, because nothing between the package manager and the report has an opportunity to be optimistic. **The verify stage disappears as a stage**, which is the strongest evidence for this shape: verification stops being a step that can be omitted from a list, and becomes a property of applying at all. ### What a pipeline result now means The honest answer, and it is different from today's: > **The declaration is updated, and here is which nodes have applied it.** A pipeline **does not wait for every node**, because a node may be legitimately switched off for a week ([ADR 0036](0036-a-node-is-a-managed-machine.md)) and a delivery mechanism that blocks on a sleeping laptop is one nobody will use. It reports what landed and what has not landed *yet*: ``` delivered declaration updated for 5 nodes applied 3 of 5 outstanding 2 — last seen 4 days ago, 20 minutes ago ``` **Outstanding is not failure**, and conflating them is how the old system got a stall with no error anywhere. A node that has not applied yet is a fact with a timestamp, and it resolves itself when the node comes back. ### The host is delivered the same way as everything else The host is tier 0, which makes it tempting to treat as special. It is not: 1. a push to `mesh-host` builds a binary; 2. publish packages it and puts it in **the mesh's own package repository** — which is a directory of files behind the object store and the proxy, so it needs no new machinery; 3. deploy updates each node's declaration to name the new version; 4. **the new declaration is pushed to each node**, and the host applies `package: nox-mesh-host` on arrival, exactly as it applies any other package. Step 4 is a push and not a poll. The link is already open ([ADR 0001](0001-nodes-communicate-over-a-broker.md)), so a node learns of a change when it is made — a node that is offline learns on reconnect, which is what makes *outstanding* a real category rather than a euphemism for lost. **No new resource type is needed**, which is the test of whether this is uniform or a special case wearing a uniform. The repository is reachable because a `file` resource put its address in the package manager's configuration — an ordinary declaration, applied by the same host. ## Consequences - **The most consequential boundary in the mesh gets smaller.** The as-is calls the fan-out *"the most consequential boundary in the mesh"* and documents a defect class from the build node having passed through two silos while others had not. **There is no fan-out**: deploy is one write, and the asymmetry it created cannot arise. - **Detection stays the fragile input, and this does not fix it.** *A merge that created no pipeline, and nothing said so* is upstream of everything here and is untouched. - **Rollback becomes a declaration change**, which is a real gain — the previous version is still named in the previous declaration — but nothing here designs how a previous declaration is retained or chosen. - **"Deployed" needs redefining wherever it is used**, because it now means *told*, and the useful fact is *applied on node X at time T*. Anything reporting deployment state has to move to the second, or it will report success for work that has not happened — the exact fault this record is closing, reintroduced at the reporting layer. - **A node offline for a long time applies a large jump at once**, having missed intermediate versions. That is correct — the declaration is a desired state, not a queue of changes — but a machine returning after months applies a very different declaration than it left with, and nothing tests that path. - **The pipeline stops being able to lie and starts being able to be incomplete.** That is a better failure mode and it is still a failure mode: a result that is honest about two outstanding nodes is only useful if somebody looks at it. ## References - [ADR 0014](0014-build-publish-and-deploy-are-three-silos.md) — the silos this keeps and redefines the third of. - [ADR 0037](0037-the-host-applies-it-does-not-decide.md) — the applier that makes this possible. - [ADR 0008](0008-a-failed-step-fails-the-job.md), [ADR 0035](0035-a-picture-is-read-from-what-runs.md) — why the host cannot report success it did not verify. - [`00-as-is/04`](../03-DESIGN/00-as-is/04-delivery.md) — the faults this addresses.