04-ISSUES/010. The store now records where each resource came from -- carried, or declared -- and each origin removes only its own. A declaration removes what the mesh previously declared and never what the bundle raised. State written before the field existed reads as carried, because everything a host had applied by then came from its bundle: there was no other way to tell it anything. Guessing the other way would have the first upgrade remove the substrate, which is this fault arriving through the change that fixes it. Verified on the scenario that caused it, and on the property that had to survive it: a later declaration dropping a resource still removes that resource, so removal by omission still means what it meant. Also stops swallowing a publish failure. A node that applied a declaration and could not tell the mesh looked exactly like one that had -- the mesh believing it never answered, the node believing it did, and nothing anywhere saying so. Reports are published mandatory now, so anything the broker cannot route comes back and is said out loud rather than dropped in silence.
Examples
substrate-first-node.lock
What a machine must be before a mesh exists — steps 0 to 4 of the bootstrap in
novox/hq 07-the-substrate.md:
0 a container runtime
1 the store runs
2 a database per context one today, `inventory`
3 that context's schema mesh-control migrate
4 the broker runs
It stops there, and the file says why. Step 5 is a virtual host, a credential and a certificate; step 6 is the control plane running. Nothing consumes any of them yet, and a bundle whose last step cannot be checked is worse than a shorter one.
Build a host carrying it:
make host SYSTEM=arch BUNDLE=examples/substrate-first-node.lock
The registry address and digests have to be replaced before this is useful. They are written
as 192.0.2.250:5000/…@sha256:… because a digest belongs to whatever registry serves it — here,
one a lab scenario raises, which reports its digests when it comes up. That is not a placeholder
to be tidied away: a bundle is built for a target, and which registry that target pulls from is
part of the target.
What was verified, and how
On a lab machine confirmed sealed — curl https://example.com times out, the lab registry answers
200 — the whole bundle applied from bare: eight resources, inventory created and mesh nowhere,
the node table present with its indexes, the migration recorded, and LavinMQ answering
lavinmqctl status with AMQP listening on 5672.
Three consecutive reconciles after that: already matches — 8 resource(s) checked, each time.
Then the machine was rebooted, and everything came back: docker from boot: enabled, both
containers because the host creates every container --restart unless-stopped
(internal/apply/apply.go), the schema intact in its named volume, and reconcile still finding
nothing to do.
The reboot is worth doing rather than assuming. Nothing in the declaration asks for a container to return, so that it does is a property of the host, and the only way to know it holds is to take the machine away and give it back.