bc5b6e2143ed41705334eacab89698dfacb4b411
35
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 jschoubben
|
bc5b6e2143 |
One reader for a declaration file, because there were three
Found raising two machines: `apply <file>` refused the bundle example in this repository with `invalid character '/'`. The bundle strips whole-line comments; apply handed the raw bytes to the parser. So a file this repo ships could be built into a binary and not applied from disk. This is the third instance of one fault. There is already a test here named "what validates is what is applied", written when `mesh-host bundle` said yes and `reconcile` said no about the same artefact — two paths to one thing, disagreeing. Fixing that instance left the shape intact, so it came back somewhere else. So the fix is structural rather than local: `declaration.ParseFileTrusted` is the one way to read a declaration from disk, and the bundle and apply both use it. Comment handling and its test now live in one place, since having them in two is how it came to be done in two. The wire format is untouched — over the link it stays exactly JSON, because a format with a second thing to strip is a format with a second thing to disagree about. Asserted, and confirmed to fail if the link starts stripping. |
||
|
jschoubben
|
bdc9c436b4 |
The token says what the mesh calls this machine
Found by raising a mesh end to end for the first time. Enrolment's own help says the token "is the only thing it needs", and it also needed --name, with no default. Without it the failure is: cannot reach the broker at 192.0.2.10:5671 as : username or password not allowed An empty username, and nothing about the cause. The node cannot work its own name out. The broker account it authenticates as is named after it and exists before this machine has been told anything, so the name has to arrive with the rest. It is not a secret and the issuer already knows it. --name stays, as an override for a token issued before the name travelled in one, and says so when it is needed rather than failing at the broker. Also corrects the bundle example, which claimed to stop before the control plane runs and has raised one for some time. A comment about what something does not do is a comment nobody updates. |
||
|
jschoubben
|
ef0d96a1a8 |
Write out what this node says when it joins, for the mesh to read
The enrolment request is a struct in each repository, and this node now reports a third key — the one its secrets are sealed to. That wiring had tests on each side and had never been run across the join, where a renamed field fails silently: enrolment succeeds, the key is absent, and the node looks joined until the first thing sealed to it cannot be opened. So this writes a real one — keys generated the way enrolment generates them, not typed as literals — and the private half of the sealing key beside it, so the other side can prove what it sealed is openable rather than merely present. The mirror of the declaration check that already runs the other way. |
||
|
jschoubben
|
a752fc514b |
A file the mesh can deliver and cannot read
Everything else in a declaration is visible to whatever carried it. The message is signed so it cannot be forged, and signing does not make it unreadable — a password in `content` is a password the broker sees, which is the transitive trust this design refuses everywhere else. So a node generates a third key at enrolment and reports the public half, exactly as it does for its identity and its overlay key. A file may arrive `sealed` instead of `content`; the host opens it with that key and writes the result. The control plane can then store a credential it cannot use, and the broker relays a blob it cannot read. A third key rather than reusing one of the two. The identity key signs and is Ed25519; the overlay key is WireGuard's and is tied to being on the private network, which a machine may not be. A key used for two purposes is one rotation away from breaking the other. Details that are not incidental: - sealed and content together is refused, so "was this the secret or the placeholder" is answerable by looking - a sealed file defaults to 0600 rather than 0644, because the consequence differs; an explicit mode still wins - a node with no sealing key refuses the file rather than skipping it. A machine that quietly omits the one resource carrying a credential looks configured and cannot connect - what is recorded is a digest of what was written, so drift on a credential is still detected without the node keeping the value, and the report that goes back over the broker carries neither The key is made at enrolment rather than on first use. One made later is one the mesh was never told about, so nothing could ever be sealed to it, and the node would look fine and receive nothing. This is why sealing was borrowed from another mesh's mistakes rather than its design: there, credentials sit encrypted in the control plane's database — which guards the database file and nothing else, since the same value is also in each node's environment file in plain text and inside every connection string composed from it. Its own tooling has to search by value rather than by name to find the copies, and says the ones inside composed URLs are usually the only copies in use. |
||
|
jschoubben
|
d81f826089 |
A check that what the control plane emits is what this host accepts
"The host needs no new vocabulary" is the load-bearing claim behind every computed and contributed resource on the other side, and it had never been tested against this parser — only asserted. Skipped unless MESH_EMITTED names a file, so it stays a check somebody runs deliberately rather than a dependency between two repositories. mesh-control plan <node> --json > /tmp/d.json MESH_EMITTED=/tmp/d.json go test ./internal/declaration/ -v Confirmed to fail when the declaration carries an action, which is the thing this parser exists to refuse. |
||
|
jschoubben
|
827ce481f2 |
Somebody editing a managed file is now visible instead of mysterious
Asked how the mesh would know if somebody edited their hosts file. It would not. The file was rewritten within five minutes and the outcome said "updated" -- which is exactly what the mesh changing its own mind looks like. So the change vanished, nothing anywhere said why, and the obvious thing to do is edit it again. The host now records a digest of what it wrote, which is enough to tell the two apart on the next pass: the file matches the declaration unchanged it matches what was last written updated -- the mesh changed its mind it matches neither corrected -- somebody changed it here The machine is put back either way, because holding it to what it was told is the point. What changes is that it says so. A digest rather than the content: the store is read on every reconcile and sits beside the state on disk, and keeping every managed file twice would make it grow with the size of the machine rather than with the number of resources. |
||
|
jschoubben
|
ef400b8c66 | Point the example bundle at the current control plane | ||
|
jschoubben
|
06f393f9aa | Point the example bundle at the current control plane | ||
|
jschoubben
|
1ea4c330df |
A seat is hardware; a graphical session is state
Two questions that had been answered by one capability. graphical-session asks whether a session is running now; seat asks whether one could ever run here. Assignment needs the second -- a headless server can never have a display server, a workstation with nothing installed yet can, and until now those looked identical. The mesh would have assigned xorg to the server and found out at apply time. "seat" rather than "display", and the distinction matters most on a phone. An Android device plainly has a screen and has no seat: nothing there is going to take a DRM device and present an X or Wayland session on it. A capability called display would answer yes and be useless. This one answers no, which is true and useful. Read from what the kernel reports about its connectors, which distinguishes the two ways of not having one: a machine with a graphics card and nothing plugged in is a different thing from a machine with no graphics at all, and somebody deciding where a desktop goes wants to know which they are looking at. Verified against this workstation: it reports card1-DP-1 and card1-DP-2, which are the two monitors actually connected, and ignores the DisplayPort and HDMI that are not -- and the writeback connector reporting "unknown", which counting would have handed a seat to machines that have none. |
||
|
jschoubben
|
4bff67ec69 |
A machine waiting to be enrolled is not a broken one
The launcher already ran `host run`, and `run` on a machine with no identity exited with an error. So a freshly installed host, sitting exactly as intended waiting for somebody to bring it a token, would have counted three failed starts and rolled back its own installation. It waits now, and says what it is waiting for. That is the *hosted* state from the lifecycle: the host is running, it has no identity, and there is nobody to link to. Every machine passes through it. An identity that exists and cannot be read is still a fault rather than a wait. Treating that as "not enrolled yet" would leave a node sitting quietly for ever while the mesh believes it is a member. Also: a node now says it is there once a minute. Nothing but its name, because anything more would be a report, and reports are rare where this is constant -- reading one as the other would make a quiet node look like a stale one. Not published mandatory, unlike a report: losing one is nothing, the next is a minute away, and the mesh reads a gap rather than counting arrivals. Verified in the lab: a node was stopped and the mesh said "out of touch 4m", then it was started and the mesh said "here" again, without anything else being touched. |
||
|
jschoubben
|
980e12a850 |
A node that loses its mesh comes back on its own
Disconnection is an ordinary situation and not a failure, and until now the host treated it as the end: the link dropped and the process returned. A laptop shut for a week would have come back needing somebody to start it again. Now it reconnects, with a backoff that starts at two seconds and slows to two minutes. The two common reasons differ in how long they last -- a broker restarting is back in seconds, a machine that has moved to a network with no route may be hours -- so it starts fast and slows down, and resets once a connection has actually held for thirty seconds. Without that reset, a node that reconnects and immediately drops climbs to the maximum and stays there long after the cause is gone. A wrong certificate is said in full every time rather than folded into a retry count. That does not mean the network is down; it means what answered is not the mesh this node joined, and no waiting fixes it. And it says when it gets back in. It logged every failure and nothing on success, so a log full of "trying again" followed by silence read as still broken when it meant the opposite. The other half: a node now keeps what it was told, not only what it applied. The record of what was applied holds an id, a type and a target -- what removal needs, not what creation needs -- so it could not be re-applied. The declaration is kept whole, signed, and verified again every time it is read back, so the file on disk is trusted for the same reason the message was rather than for being local. A tampered one is refused, and so is one signed by another mesh. With both, the host reconciles against what it was last told every five minutes, connected or not. That is not polling for changes -- changes are pushed -- it is the answer to a machine drifting: a file edited by hand, a container somebody stopped, a service that died. Verified in the lab. The broker was stopped: the node retried at 2s, 4s, 8s, saying why each time, and kept its overlay up throughout. The broker came back and the node rejoined without being touched. A declaration published while a node was away was waiting on the broker and applied the moment it connected, which is the buffer ADR 0006 describes doing its job. |
||
|
jschoubben
|
ba31eef80f | Point the example bundle at the current control plane | ||
|
jschoubben
|
1bc97ed50d |
A service can be declared to reflect a file
Because a running service does not re-read its configuration. Replace the file, find the service running, do nothing -- and the machine keeps behaving as it did while every check passes, because the file is right and the service is up. That is not hypothetical. It is how a third node joining a mesh left the first two carrying a private network that no longer existed, with every part of it reporting success. Declared state rather than a command: the declaration says the running service must reflect these files, and the host works out that it does not. A command to restart would be an action, and the link may not carry one -- the host refused precisely that when I tried it, correctly, which is how this shape was arrived at rather than the other. Scoped to one apply. A change from an earlier one has already been reflected, and restarting for it every time would make a steady machine bounce its services for ever. Also: the node generates its overlay key at enrolment and reports the public half, and the store waits three minutes rather than one for the database -- sixty seconds is not enough for a cold machine running initdb, and it failed that way three times, which is the worst kind of flake because a second run always fixed it. |
||
|
jschoubben
|
732905a6a6 |
A node generates its own key for the private network
Curve25519, which is what WireGuard uses. The private half never leaves the machine and is written to a file of its own, so the interface configuration the mesh composes can point at it without ever carrying it. Separate from the identity keypair on purpose. One signs messages to the mesh and the other encrypts traffic between nodes -- different things verified by different parties at different times, and a key used for two purposes is one rotation away from breaking the other. |
||
|
jschoubben
|
38d7b2d8af | Point the example bundle at the current control plane image | ||
|
jschoubben
|
fa48b5825e |
The bundle and the mesh stop removing each other
04-ISSUES/010. The store now records where each resource came from -- carried, or declared -- and each origin removes only its own. A declaration removes what the mesh previously declared and never what the bundle raised. State written before the field existed reads as carried, because everything a host had applied by then came from its bundle: there was no other way to tell it anything. Guessing the other way would have the first upgrade remove the substrate, which is this fault arriving through the change that fixes it. Verified on the scenario that caused it, and on the property that had to survive it: a later declaration dropping a resource still removes that resource, so removal by omission still means what it meant. Also stops swallowing a publish failure. A node that applied a declaration and could not tell the mesh looked exactly like one that had -- the mesh believing it never answered, the node believing it did, and nothing anywhere saying so. Reports are published mandatory now, so anything the broker cannot route comes back and is said out loud rather than dropped in silence. |
||
|
jschoubben
|
c192572f74 |
Save and Load must agree, and did not
Pushed a failing test in the last commit -- my own gate reported it and I read the count rather than the result. The failure was real and worth having. Adding the membership requirement to Load made Generate produce an identity that Save would write and Load would then refuse. A file that cannot be read back is the worst shape this could take: it is read back on the next start, on a machine nobody is watching, and by then the token that could have fixed it is spent. Save now refuses exactly what Load refuses, and writes nothing when it does. The round-trip test covers the membership too, since that is the half that lets a node come back on its own. |
||
|
jschoubben
|
a488c76b5e |
A node holds its link open, and applies what the mesh signs
The loop the whole thing exists for: told, apply, report. `run` holds one outbound connection open and consumes the node's own queue. Every declaration is verified against the control plane's signing key before a byte of it is read as an instruction -- not once at connect, every time. The transport being pinned is a different question from the instruction being genuine, and pinning only the first would make the second transitive: a compromised broker could forge declarations, and this host applies whatever the link delivers. Malformed and forged are reported differently, because ADR 0004 requires a host to tell "this is not from the mesh I joined" from "this is broken". One means somebody is trying and the other means something needs fixing. A node now keeps what it needs to come back on its own: the broker's address and fingerprint, the signing key it believes, and its own broker password -- which the mesh issues at enrolment to replace the token's secret, so the one-time thing stays one-time and the credential it holds for years is not the one that was pasted into a terminal. Verified in the lab end to end. The node enrolled, held its link, received a signed declaration and applied it -- the file is on the machine with the right contents, and the host's own record lists both resources. That run also found issue 010, which is recorded in novox/hq: the declaration removed every container on the machine, including the control plane that sent it. Correct reconciliation, shared store, and the first thing that happens. |
||
|
jschoubben
|
a4445f5c0a |
A machine joins the mesh it raised
The last step of the first-node path, and the bundle now carries all of it: a container runtime, the store, a database per context, their schemas, the broker with a certificate it generated itself, and the control plane running. Then the machine enrols against the mesh on its own disk. It dials the broker over TLS, refuses anything but the pinned certificate, presents the one-time secret with a public key it generated, and is told the name the mesh has for it. Its specialness lasted two commands, which is what ADR 0004 asked for. The identity is saved only after the mesh says it knows this node. A node holding an identity the mesh never recorded would believe it had joined and be believed by nobody, which is worse than not joining because nothing looks wrong. An already-enrolled machine refuses a valid token rather than quietly acquiring a second identity, and a spent token is refused by the mesh. Both checked. Containers gained a network field. The control plane must reach the store and the broker on the machine it was raised on, before there is any mesh to arrange that; the alternative was publishing ports and guessing an address that works from inside a container, which fails in a worse way. The control plane talks to the broker over loopback in plaintext, deliberately. The TLS on 5671 exists so a node crossing a network can pin a certificate, not for a hop that never leaves the machine. Verified on a sealed lab machine: eleven resources applied from bare, the control plane consuming, a token issued from inside it, and the machine enrolled -- with the recorded public key matching what the host printed, the token marked spent, and the profile stored. |
||
|
jschoubben
|
65d896d96e |
A node makes its own identity, and checks the broker before speaking
The host side of enrolment. It parses a token the control plane issued, dials the broker, refuses anything but the pinned certificate, and generates an Ed25519 keypair whose private half never leaves the machine. Verified against a real LavinMQ serving a real certificate: the pin matched and the node proceeded. Then against a second broker with a different certificate on another port, which was refused -- with an error that says retrying will not help, because it does not mean the network is down, it means the mesh was substituted. InsecureSkipVerify is set and that is the point rather than a weakening. At bootstrap the broker is self-signed and reached at an address, so there is no authority to trace and no name to match. Chain and hostname checks are replaced with something stricter: this exact certificate or nothing, checked in VerifyPeerCertificate, which runs before the handshake completes -- so nothing is sent to the wrong broker. There is a test that counts the bytes an impostor receives, and it is zero. The token format is defined separately here and in the control plane, because this binary requires nothing present and does not import it. They are held together by a test on each side asserting the exact field names, so a rename breaks both immediately rather than at enrolment on a real machine. Two distinctions the identity file has to keep. A machine that never joined has no identity, which is an ordinary state and not a fault. A machine whose identity cannot be read is a different thing entirely, and must not take the same path -- re-enrolling would discard the identity the mesh still believes and need a person with a new token. Fault injection found the second case untested: the corrupt-file test was passing on the parse check, so the read-error path had nothing defending it. It does now. An already-enrolled machine refuses to enrol again rather than quietly acquiring a second identity. What is not built is the link. Enrolment stops after verifying the broker and generating the identity, having saved nothing, so it can be run again unchanged. 132 tests, plus 32 launcher and 9 rollback. |
||
|
jschoubben
|
a740959cb0 |
The bootstrap reaches the broker, and survives a reboot
Five steps now instead of three. A sealed machine goes from bare to a container runtime, a store, the inventory database, that database's schema applied by mesh-control, and LavinMQ running and answering. The database is called inventory rather than mesh. ADR 0008 grants a context only what it exclusively owns and ADR 0006 says the mesh database names a thing that will not exist -- so one database per context, and there is one context. The broker is in the bundle because ADR 0006 now says it must be: the control plane reaches a node only over the link, the link is the broker, so nothing can provision the broker. Two images, which is the cost that record accepts. Verified by reading the system rather than the report: inventory present and mesh absent, the node table with its indexes, the migration row, lavinmqctl answering, 5672 listening. Sealed confirmed both ways -- the internet times out, the lab registry returns 200. Then rebooted, which was the part worth doing rather than assuming. Everything returned: docker from boot: enabled, both containers because this host creates every container --restart unless-stopped, the schema intact in its volume. Three reconciles before and one after all report no change. One thing that reads as a success and was not: the first sealed check said the machine could reach example.com. It was the test that was wrong -- a helper script pasted arguments into a shell line, so a command with quotes was re-split and ran on the workstation. The machine had been sealed the whole time. The helper now requotes each argument. |
||
|
jschoubben
|
e09503acc7 |
A real bundle: a bare machine raises a store and a database
The first three steps of the substrate bootstrap, run on a lab machine confirmed to have no route out. It went from bare to a container runtime installed and enabled, PostgreSQL running from an image pinned by digest, and the control plane's database created inside it -- from the file the host carries, with nothing to ask. Second run changed nothing. `owned` lists all five afterwards, and `mesh` is in the store. It stops before the last two steps because there is no control plane yet: its schema cannot be loaded and its image does not exist. The bundle says so rather than naming something that cannot be applied. Two things fixed on the way. `make host BUNDLE=...` still swapped a file called substrate.lock, which the per-system split had renamed months of decisions ago -- it now takes SYSTEM and replaces that system's bundle. And the .lock files still cited ADR 0060, since the renumbering pass only covered .md, .go, .ts and .sh. One thing learned by it failing first: a directory the host creates is owned by root, and a database inside a container runs as somebody else, so it could not write and the container crash-looped. The store's data is a named volume now, which lets the image set up its own ownership and outlives the container -- which is what you want for the thing holding the mesh's state. Worth noting the failure was caught by the action's verify rather than by the container step. `docker inspect` reported the container running because it was, briefly, between restarts. Running is not working, and the thing that knew the difference was the step that asked the database whether it would answer. |
||
|
jschoubben
|
430e2a1271 |
Fix a comment left odd by the renumbering
Two records merged into one, so a reference to both became 'ADR 0005 and ADR 0005'. |
||
|
jschoubben
|
ee2648188d |
Repoint ADR references after HQ consolidated 65 records to 23
96 comments across the two repos named records that no longer exist. Each now points at the consolidated record that holds its reasoning -- ADR 0034 (a test defends a decision) is 0017, the eight host records are 0005, the four lab records are 0016. Worth noting for next time: these are references from outside HQ, so renumbering there is not free. It cost 38 files here. |
||
|
jschoubben
|
ebba16ce4a |
Per-system bundles, and Android's start problem closed by narrowing it
Two gaps. The bundle's contents are per system even though its mechanism is not, so there are now three: substrate-arch.lock, substrate-alpine.lock and substrate-android.lock. All three are embedded and a host reads only the one it was built for. Arch and Alpine remain placeholders -- the closure for a one-node mesh is still research 011/012's open question, and inventing it here would be worse than an honest placeholder. Android's is not a placeholder. It says a partial host cannot raise a mesh and why: every step of a bootstrap is a package, a container, or an action against one, and those are exactly the shapes it refuses. So a partial host can JOIN a mesh and cannot BE the first node. That belongs where somebody looking for the android bundle will find it. Also separated two things that were being conflated: "this system has no bundle" and "this system was never built". Loading a bundle for debian is not ErrEmpty, and the test asserts they differ. 0062 -- a host may be episodic. There is no way to keep a process running on an ordinary Android device: init needs root, a foreground service can be killed for memory. The answer is not to fight that. It is that being killed IS disconnection, which ADR 0036 already made an ordinary situation -- and everything the design does for a laptop that closes is what an episodic host needs, at a shorter period. An authoritative local store, reconcile on start, last-heard-from reported without an alarm. So the gap closes by requiring less rather than building something. No keep- alive, no Android daemon, no fighting the platform's process management. Two consequences recorded rather than glossed. Last-heard-from is a much weaker signal on an episodic host, so a healthy phone reads as a dead server unless the reader knows which kind it is looking at. And a declaration may take a long time to land, which makes 0058's separation of outstanding from failed load-bearing rather than tidy. Left open deliberately: how an episodic host is actually started, and -- first -- what an Android node is for. Building the start mechanism before deciding that would be building it for nobody. |
||
|
jschoubben
|
02f1fcc865 |
Three hosts: arch, alpine and android
ADR 0060, built. `make hosts` produces mesh-host-arch, mesh-host-alpine and mesh-host-android, each pinned to its system at link time. The claim that "almost all of it is shared" held up. All 36 existing apply tests pass unchanged -- the only edit was naming which system they run against, which was previously implicit. What moved into internal/system is two appliers' worth of code and the probes that go with them. Each system's differences are real and needed re-deriving rather than translating: apk reports absence by EMPTY OUTPUT and exits zero either way, where pacman exits non-zero. Reading apk's exit code the way pacman's is read reports every package as installed. That is the single most dangerous difference between the two and it is invisible until it bites. OpenRC has no LoadState, so "the service does not exist" is read from its prose rather than a field. Same distinction, different evidence -- and this is exactly what an interface spanning both would have had to drop, which is why 0060 rejected one. OpenRC has no is-enabled either. Boot state comes from the runlevel listing: "does it start at boot" becomes "does it appear in rc-update show default". Android is a partial host and that is the point. It implements file, directory and action -- the shapes needing only a filesystem and a way to run something -- and refuses the other three by name, before anything is applied. Its unreachable appliers return ErrUnsupported rather than a zero value, so "unreachable" fails loudly if it stops being true. A host also confirms it is on the machine it was built for, once, at the start. The alpine host on this Arch machine says "this machine is not Alpine" instead of failing later inside a package manager that is not there. And a host built without -X main.builtFor refuses everything, naming the hosts that exist. Two test problems found by injecting faults. One injection did not compile, so the check now reports that separately from a pass. The other passed with the behaviour removed: the missing-service assertion matched "does not exist", which the FALL-THROUGH error also contains because it echoes the raw output. It now asserts the diagnosis, which only the correct branch produces. Verified with the real binaries: android refuses a package naming what it does support; alpine on Arch refuses the machine; arch applies and is idempotent; a system-less build refuses everything. |
||
|
jschoubben
|
5b7b280e3a |
The launcher supervises the host instead of exec'ing it
Jochen: "I thought we did not want to run the host under a systemd/openrc/init loop, but instead had our own host-init program?" -- and that was right. I had moved the give-up logic out of unit files and left RESTART in them, with the launcher exec'ing the host and disappearing. So init still decided when the host came back, which is the arrangement 0061 exists to remove. The launcher now stays and supervises: starts the host as a child, waits, decides. Init is asked for one thing, run this at boot. There is an OpenRC script beside the systemd unit now, four lines each, which is the point -- a second init is transcription rather than a port. The cost of not exec'ing is signals. A supervisor that exits while its child runs leaves the host to be killed rather than to stop, and an apply interrupted that way is the half-configured machine this project is about. So SIGTERM is trapped, passed down, and waited on. Two bugs, both found by the tests rather than by review: A clean exit was counted as a failure. The host exits cleanly to stand aside for a new binary after an upgrade (0057), so a host that upgraded itself three times rolled itself back having worked perfectly every time. The counter now counts CONSECUTIVE FAILURES, incremented after the wait rather than before the start. And when rolling back I reset the counter file but not the variable, so the next failure counted from the old value -- the rolled-back version got one attempt instead of three. Also: the host now clears the counter when it completes a reconcile, at the same moment it records known-good and for the same reason. Without it the count only climbs, and a node up for months rolls itself back on its third ordinary restart -- a healthy machine undone by its own recovery. One test expectation was tightened rather than fixed: "resets the counter after rolling back" asserted exactly 0, which was true only under the old count-before-start semantics. It now asserts the property -- below the limit -- since 1 is correct after a rollback plus one failure. 32 launcher tests, all confirmed to bite. |
||
|
jschoubben
|
f04294c3c1 |
A service can be enabled at boot, and a container uses the runtime the machine has
Two gaps found by testing podman rather than reasoning about it.
The service shape could not say "starts at boot". It ran `systemctl start`, so
`service: docker.service, running` started docker now and it would not come
back after a reboot unless something else had enabled it. A declaration that
reports success and stops being true at the next power cut.
`boot: enabled|disabled` is now a separate field, not a fourth value of
`state`, because the two are orthogonal: a unit can be enabled and stopped (it
returns at boot) or disabled and running (started by hand, gone after one).
Absent means the host asserts nothing, so a machine whose operator enabled
something is not silently disabled by a declaration that never mentioned it.
Boot state is made true BEFORE the unit is started. When an apply fails part
way, enabled-and-stopped comes back at the next boot and running-and-disabled
does not, so the more durable half goes first.
`is-enabled` has the same trap as `is-active` had. Its exit code is non-zero
for nearly everything, and `static` is neither enabled nor disabled -- the unit
has no install section and CANNOT be enabled. Reading it as "disabled" would
have the host try, fail, and blame the wrong thing, which is the same shape as
reading a missing unit as "stopped".
The container applier no longer calls `docker` literally. Verified on this
machine against podman 6.1.0:
docker info --format '{{.ServerVersion}}' -> 29.7.2
podman info --format '{{.ServerVersion}}' -> Error: can't evaluate field
ServerVersion
podman info --format '{{.Version.Version}}' -> 6.1.0
So one probe cannot find both, and a host using docker's would report a machine
running podman as having no container runtime at all. Everything else IS
compatible -- run, rm -f, and docker's own Go template syntax for reading state
and labels all work unchanged on podman, confirmed by running them. That is why
this is a two-entry lookup rather than an interface: only the probe differs.
Detected rather than declared, because adoption keeps what the machine already
has (research 012), which hardcoding one runtime contradicts.
A machine with neither now says so, naming both: "docker: command not found" on
a machine deliberately running podman sends the reader after the wrong thing.
Verified end to end against real docker (container created, running, labelled)
and against an empty PATH (refused, naming both runtimes).
Two injections per behaviour, all confirmed to bite. One injection produced a
build failure that my check read as "no bite" for the third time, so the check
now distinguishes them.
|
||
|
jschoubben
|
057f34f924 |
The init is asked for start and restart; a launcher does the rest
ADR 0061. Recovery was the most systemd-specific part of the host, and it is the part that must work on a machine where nothing else does -- which made unit-file syntax a poor place for it, because syntax cannot be tested and the one time it runs is the one time nobody can afford it wrong. So StartLimitBurst and OnFailure move into a launcher script that init starts instead of the host. The unit drops to start-at-boot and restart-on-exit, which OpenRC, runit, s6 and an Android init.rc can all express. Everything 0059 decided is kept: two watchdogs, roll back once, recovery is local, the rollback shares no code with the host. The counter is the whole mechanism, so it is what the tests are mostly about. Three real problems came out of writing them: A counter file holding "1 2" became "12" -- `tr -d [:space:]` concatenates rather than rejecting -- which is past the limit, so a HEALTHY node rolled itself back. Now it reads the first field and insists on a plain integer. The corrupt-counter test used "not-a-number", which shell arithmetic happens to evaluate to 0, so it passed with the guard removed and proved nothing. Replaced with values that discriminate: "5x" errors under set -e and kills the launcher, and "0x10" is read as HEX 16 -- past the limit, so again a healthy node rolls back. And the test harness itself was wrong. With `set -e` and a bare launcher call, removing a guard killed the script at the first corrupt case and silently skipped everything after -- reporting a full pass over tests that never ran. Every launcher call now records its failure instead of aborting. Same class as the placebo assertion found last time, and the reason to keep injecting faults rather than trusting green. Both scripts run in `make check`. 27 launcher tests, 9 rollback tests, all confirmed to bite. |
||
|
jschoubben
|
f4143806c2 |
Build the rollback mechanism, and test it
ADR 0059's recovery path: the pieces that run when the host will not start. internal/upgrade -- two facts, neither of them the host judging its health. Whether the executable this process started from has been replaced on disk, and which version last completed a reconcile. The first design was wrong and the tests caught it, not review. It asked /proc/self/exe whether it was marked deleted. That is Linux procfs behaviour rather than a fact about files, and it catches only unlink -- a binary swapped by rename onto the same path reads as untouched, which is exactly what a package manager does. Now the identity is captured at start and compared later: no procfs, and neither case missed. known-good is one bare line. The reader is a shell script on a machine where the host is failing to start, so it must not need a parser to be present and working. Written only after a clean apply, which is the whole claim -- not health, because a disconnected node is ordinary and a failing resource is the machine's problem rather than the binary's. packaging/ -- the unit, the rollback unit, and the rollback script. The script shares no code with the host and calls none of it: a binary that cannot start cannot be its own recovery. POSIX sh, nothing that has to be installed. The unit carries Restart=always with a comment saying why on-failure would break every upgrade. Both are tested and both sets of tests were confirmed to bite. Injecting five faults broke exactly the intended tests -- except one, and chasing why it did not found a placebo assertion I had written: `check "exits zero" ... "0" "0"` compares a literal to itself and can never fail. Replaced with the real exit code, after which the injection bites. Also caught: an injection that produced a build failure rather than a test failure, which my grep read as "no failure". Re-run so it compiled, and the test did bite. The script test runs in `make check`, so it is a gate rather than something that was run once. Verified against the real binary: known-good is written beside the store after a clean apply and is NOT written after a failed one. |
||
|
jschoubben
|
9a9937b7e6 |
A struct per resource kind, instead of one struct with every field
Jochen asked why we don't simply have dedicated structs. We should, and the
flat struct was me extending an existing pattern rather than questioning it.
Before: one Resource struct carrying path, content, mode, unit, state, package,
image, name, env, ports, volumes, args, command, verify and in. Because a file
and a container shared it, nothing stopped {"type":"file","image":"postgres"},
so a `uses` map listed which fields each kind was allowed to carry -- a second
place to keep current, and the kind nobody updates is the one that silently
accepts a field the host will never read.
Now: Directory, File, Service, Package, Container and Action are separate
structs behind a Resource interface. File has no Image field, so the mistake is
not detected -- it is unrepresentable. Adding a field to a kind is the whole of
adding it; there is nowhere else that has to agree.
Parsing is two passes: read the envelope and each resource's raw bytes, peek at
"type" to choose the struct, then decode into it. Peeking is lenient on purpose
-- reading strictly there would report an unknown field before knowing which
fields are known.
Unknown fields are found by comparing the JSON keys against the struct's own
json tags rather than by catching the decoder's error. The decoder stops at the
first unknown field, and RefusalError promises every problem at once: a caller
fixing one field at a time learns the next only by running again. Caught by
testing the refactor against a real declaration -- a container carrying both
`unit` and `mode` reported only one of them.
apply.go switches on the concrete type instead of a string, so a new kind that
has no applier is a compile error rather than a runtime default branch.
No behaviour change otherwise. All existing tests pass unmodified except two
that reached for fields the interface no longer exposes.
|
||
|
jschoubben
|
337126603e |
Complete the host's vocabulary: package, container, action
The three shapes the substrate bootstrap needs and the host did not have. Until now tier 1 could not be raised at all -- step 0 is a package, step 1 a container, steps 2 and 3 actions -- so every line of the tier 1 and 2 designs was unbuildable. package -- present, never upgraded, never uninstalled. Removal is "forgotten", not "removed": the host cannot know what else needs the package, uninstalling a container runtime because a declaration changed would stop every container on the node, and the machine may have had it before the mesh saw it. Reporting it removed would claim an effect the host declined to have. container -- identified by a label carrying a digest of the declaration that made it. Comparing every field the runtime reports cannot be done reliably: a runtime normalises, defaults and reorders what it is given, and that is indistinguishable from real drift. There is no in-place update; a container's configuration is fixed at creation, so any change is a replacement, and saying so beats a partial update that leaves the running thing half-declared. This is the one shape the host removes, because it is the one the host created. action -- bundle-only, per ADR 0047. Verify is mandatory and does double duty: it is the idempotency check as well as the read-back. The host does not know what a database is, so "is it already there" is a question only the declaration can ask. `in` runs the action inside a named container, which steps 2 and 3 need. Parse now refuses actions; ParseTrusted permits them. The safe path is the default and the permissive one has to be named. The bundle and a local file handed to a root process use ParseTrusted; the link will use Parse. Also replaced the per-type "fields this type ignores" check with a field-set diff stated as what each type USES. The negative form needs every type revisited whenever a field is added, and the one nobody revisits silently accepts a field it will never read. Images must be pinned by digest (ADR 0046). A bundle naming a tag pins nothing. Verified against a real machine, not only fakes: an action ran and was idempotent on the second apply; an action that exits zero and satisfies nothing fails the apply; a real container was created, labelled, replaced when its declaration changed, exec'd into, and removed; a real package query round- tripped. Each new test was also confirmed to fail on an injected fault -- five injections, each breaking exactly its own test. One existing test changed: a vanished unit is now reported "forgotten" rather than "removed", which is what actually happened. |
||
|
jschoubben
|
08a1263a81 |
Stage 2 — the bundle a host carries
novox/hq ADR 0038: one behaviour, two sources of declaration. This is the source that does not need a mesh — the first node's path. The bundle is embedded in the binary rather than shipped beside it, because "copy it onto a machine and run it is the whole installation" stops being true the moment a second file has to arrive with it. `make host BUNDLE=...` builds a host carrying one; `mesh-host reconcile` applies it; `mesh-host bundle` shows it. A default build carries nothing and REFUSES to reconcile, saying why. A host that applied nothing and reported success would look exactly like one that raised a first node, and the difference would surface later as a mesh that never came up with nothing to point at. Proved on a sealed machine: no route out, no name resolution, one binary copied on, and it configured itself from what it carried. Idempotent on the second run. One bug found by running rather than reasoning, and it is a shape worth naming: `mesh-host bundle` validated the carried bundle through a path that strips comments, while `reconcile` handed the raw bytes to the parser. So the command whose whole job is to check the bundle said yes, and the command that uses it said no. Two paths to one artefact, disagreeing. There is one path now, and a test asserts that what validates is what is applied. What this does NOT prove is stated in the README rather than left implied: the claim under stage 2 is that one host can raise the substrate alone, and the substrate is four container services. There is no container type, because a container needs an image and where images come from is open; what belongs in a substrate is not known, because the closure for a one-node mesh is what research 011 and 012 exist to answer; and the machine used to test this cannot install a container runtime through a sealed network. The mechanism is finished. The claim is not, and shipping a host that claimed a substrate it has never raised would be the fault this whole project is about. 65 tests. |
||
|
jschoubben
|
9d8239afe8 |
Stage 2 — the host applies a declaration
A declaration is JSON, versioned, and an ordered list of resources with stable identities (novox/hq ADR 0043). The vocabulary is directory, file and service, and anything outside it — an unknown version, type or field — refuses the WHOLE declaration. A host that skipped what it did not understand would apply most of what it was sent and report success. It converges rather than executes: applying twice changes nothing the second time, and applying to a drifted machine returns it. A mode is maintained rather than set, because a permission applied at creation is not a permission held — this repository has paid for that once already. It owns a footprint and only that. What it applied and is no longer declared is removed; what it did not create is never touched. Removal runs FIRST, because a resource leaving a declaration while another arrives at the same path is an ordinary rename, and removing afterwards would delete the file just written. The store arrives here rather than at stage 3, as ADR 0043 predicted: nothing can be removed without knowing what was applied. It is written atomically, refuses to start empty when it exists and cannot be read — believing it owns nothing would leave everything behind forever — and is saved even when an apply fails, because what was applied before the failure is on the machine either way. Three faults found by running inside a raised machine rather than by reasoning: A unit that DOES NOT EXIST reads as `inactive` from `systemctl is-active`, exactly as a stopped one does. So declaring a unit stopped reported success for a unit the host cannot manage at all — absence read as satisfaction, which is 04-ISSUES/007 wearing a different hat. LoadState separates them. Removing an orphaned service whose unit has since been uninstalled failed the whole apply, and a host holding such a record could then apply NOTHING, ever, with no way out but editing its state by hand. Removal is now idempotent for the same reason os.RemoveAll is. And the flag parser was wrong in the same way twice: fixing `mesh-host inventory --json` by taking the subcommand off the front left `mesh-host apply decl.json --dry-run` broken identically, because the standard library stops at the first non-flag argument wherever that argument is. Parsed in a loop now. 30 new tests, 55 in total. |
||
|
jschoubben
|
73c010e7ef |
Stage 1 — the host reports what a machine is and can do
Tier 0's first slice, per novox/hq 03-DESIGN/01-to-be/05-the-node-host.md. It applies nothing, connects to nothing, listens on nothing. 2.9 MB, static, no dynamic dependencies: copy it onto a machine and run it is the whole install, which is the property ADR 0041 rests on. A capability is detected, never assumed. Every detector runs something that only succeeds if the thing FUNCTIONS — the daemon is asked for its version, the package database is queried, the firewall is asked to list a ruleset, which needs the privilege as well as the tool. 04-ISSUES/007 is the fault this prevents: a client on disk with its daemon down looks exactly like a working runtime, and a node assigned work on that basis fails when the work arrives. Every verdict carries the reason and the method. A capability reported absent with no reason is the same fault in a new place: something nobody can act on. Two bugs found by running rather than reasoning, both silent: systemctl is-system-running exits non-zero for every state except `running` — including `degraded`, which means units failed and the init is emphatically there. Reading the exit code reported NO service manager on a machine whose init it was. That is 007 in the mirror, and both directions place work wrongly. A verdict now reads what a tool says about itself, not only how it exited. And `mesh-host inventory --json` printed text: the standard library stops parsing at the first non-flag argument, so the flag sat unread and the command exited 0 having ignored what was asked. The parser now takes the subcommand off the front, and a stray or mistyped argument is refused rather than dropped. Detection deliberately does NOT follow ADR 0008. That rule governs applying state, where a failed step means the machine is not what was asked for. A failed probe is a finding — "absent, because the probe failed" — and aborting would replace one legible absence with total ignorance of the rest. 25 tests: structure and logic with a fake runner, and the same detectors against this machine, because a test that fakes the system under detection asserts only that the fake behaves as expected. |