Compare commits

..
Author SHA1 Message Date
jschoubben 7181675c4a A joining machine dials the bus once the hub has answered its tunnel
Issuing the token sends the hub its new peer, and the hub applies it on
its own time; a bus dialled before then timed out naming the bus. The
first tunnel now waits for a handshake with the hub, and says so in the
tunnel's words when there is none (novox/hq ADR 0169).
2026-10-02 18:15:11 +02:00
jschoubben 19e14901c0 The installer lets the first node onto the bus it raised
At genesis the bus's users reach it in no declaration, because the
machine running it has not enrolled. The installer, which raised the
bus from its bundle, places the control plane's composed list beside it
and makes it re-read it: before the machine enrols, for the token's
account, and after, for the node's own (novox/hq issue 146).
2026-10-02 18:02:49 +02:00
jschoubben b9fc3afc14 A machine makes its tunnel key first and joins through the tunnel
nox-mesh-host key makes the tunnel key, or reads the one made, and
prints its public half for the token to be issued for. enrol with a
token that carries a tunnel takes that key, refuses another, writes
mesh0 with the hub as its one peer and starts it, then reaches the bus
over it (novox/hq ADR 0169). Tokens without a tunnel enrol as before.
2026-10-02 18:02:49 +02:00
mesh-admin ca7c4a5915 Merge pull request 'A container may log to the journal (hq ADR 0179)' (#73) from feat/the-intrusion-seat-serves-its-verbs into main 2026-10-02 15:04:02 +00:00
jschoubben b30d9c5b5a A container may log to the journal (hq ADR 0179)
A jail reads a log; a container's output went to a file of the runtime's own under a path that
changes on recreate, so no jail could read a container's service. logging: journald runs the
container with the journal as its driver, named in the spec so moving it recreates it; any other
place is refused.
2026-10-02 17:02:49 +02:00
mesh-admin 5b73e04192 Merge pull request 'A package may be declared absent, and an uninstalled front end is retired for good (hq ADR 0175)' (#71) from feat/the-found-front-end-is-uninstalled into main 2026-10-02 14:29:17 +00:00
jschoubben f57386cdea A package may be declared absent, and an uninstalled front end is retired for good (hq ADR 0175)
absent: true on a package has the host remove it through the machine's own
package manager when it is installed and leave alone a machine that never had
it; read back either way. Undeclaring a package still removes nothing. A
found firewall whose command is gone is recorded as removed, said once, and
asked nothing of.
2026-10-02 16:28:27 +02:00
mesh-admin f92dd3e286 Merge pull request 'Cite hq ADR 0170, not 0169: the firewall seat's record was renumbered' (#70) from fix/adr-0170-cited into main 2026-10-02 12:53:09 +00:00
jschoubben cdbe3ab0a4 Cite hq ADR 0170, not 0169: the firewall seat's record was renumbered after a collision on hq main 2026-10-02 14:52:20 +02:00
mesh-admin a8f1cdb445 Merge pull request 'A machine reports whether its virtualisation daemon runs (hq ADR 0172)' (#69) from jschoubben/the-lab-is-a-module into main 2026-10-02 12:47:55 +00:00
jschoubben ecb3003ba4 A machine reports whether its virtualisation daemon runs
The lab module needs the virtualisation daemon (novox/hq ADR 0172); the
capability is detected by asking the daemon about itself, not by finding
a client on disk.
2026-10-02 14:46:18 +02:00
mesh-admin d3861f82d4 Merge pull request 'A container may declare the capabilities it is granted (hq ADR 0169)' (#68) from feat/the-firewall-seat-serves-its-verbs into main 2026-10-02 11:28:34 +00:00
jschoubben b6dbe0a7b9 A container may declare the capabilities it is granted (hq ADR 0169)
Exactly the names declared reach the runtime, named in the spec so a change
recreates the container; a name that is not a capability's is refused and a
privileged container stays undeclarable. For a seat holder whose runtime
changes the machine's packet filter.
2026-10-02 13:27:34 +02:00
mesh-admin 07bdad9e94 Merge pull request 'The host says what filters the machine, with owners, and keeps the found firewall retired on every converged apply (hq ADR 0168)' (#67) from feat/one-thing-filters-a-converged-machine into main 2026-10-02 09:58:58 +00:00
jschoubben 627ac97d4f The host says what filters the machine, with owners, and keeps the found firewall retired on every converged apply (hq ADR 0168)
Every table and chain that refuses traffic is reported with whose it is:
the mesh's, the found firewall's, the container runtime's own, a ban, or
other — the runtime's user chain is other, which is where both predecessors
kept their rules, in the legacy filter on one machine and invisible to the
mesh. Adoption's threshold does not move; a converged machine's report
grows by its filters and its found firewall's state.

Convergence is a state the host keeps: a found firewall enabled again is
retired again and said; a reconcile that finds it inactive records that it
was found so, never that the mesh did it; a step skipped after a failed
apply is said. A retirement the mesh began and did not finish is finished.

Fixtures are rulesets captured from three machines of the first mesh.
2026-10-02 11:58:16 +02:00
mesh-admin ee2359f29f Merge pull request 'Every physical link faces outside, up or down (hq issue 197)' (#66) from jschoubben/every-physical-link-faces-outside into main 2026-10-02 09:54:06 +00:00
jschoubben cbdbf6b7d3 Every physical link faces outside, up or down
The filter accepts what does not arrive on a link the machine names as
outward, and the host named only links carrying a default route. An
unplugged wired port was left unfiltered for whenever it was plugged in
(novox/hq issue 197). A link backed by a physical device is now named
whether or not it is up.
2026-10-02 11:53:53 +02:00
mesh-admin e12ca3f4dd Merge pull request 'A former target of a kind the host cannot remove is left in place and said, never fatal (hq issue 194)' (#65) from fix/a-former-target-without-a-removal-is-left-and-said into main 2026-10-02 07:36:57 +00:00
jschoubben c47aa5d9b3 A former target of a kind the host cannot remove is left in place and said, never fatal (hq issue 194)
The host delivers its own successor as an archive whose target is a new
directory each version, and since mesh-host 63 the record keeps a resource's
former target for the next apply to remove. An archive has no removal (issue
162), so the first host that replaced itself under that rule refused its own
former version at the first step of every apply, and all four machines applied
nothing from then on. A former target nobody dropped is forgotten and said;
an archive the declaration dropped still refuses.
2026-10-02 02:43:47 +02:00
mesh-admin 3b9692b3cb Merge pull request 'A taken container keeps a found network, a left-out module is kept, and genesis raises the forge as its module declares (hq ADR 0163)' (#64) from feat/a-take-is-a-comparison-the-rest into main 2026-10-02 00:28:17 +00:00
jschoubben fb9c9c3ee8 A taken container keeps a found network, a left-out module is kept, and genesis raises the forge as its module declares (hq ADR 0163)
A container may name networks it also joins once created, for the per-machine
setting that keeps a found network while a neighbour still resolves it there:
joined after the run, part of the spec, refused when it cannot be joined.

A declaration may say which modules the mesh left out because a stored setting
cannot compose with its definition. Absence used to read as removal; a left-out
module's records are kept and said, and its holds are not released.

Genesis raises the bootstrap forge under the gitea module's container name, with
its image digest and its data directory mounted at /data, so the module holds it
by the found rule instead of raising a second forge beside it (issue 090). The
network is the one difference left for a take to say. Before this the forge had
no volume: its repositories were the container's, lost with it.
2026-10-01 23:45:05 +02:00
mesh-admin d53e626366 Merge pull request 'A take is a comparison: the host's facts, former targets, and strays (hq ADR 0163)' (#63) from feat/a-take-is-a-comparison-the-hosts-facts into main 2026-10-01 19:25:56 +00:00
jschoubben 83b3d20e68 A take is a comparison: the host's facts, former targets, and strays (hq ADR 0163)
Every held thing carries what a take compares: for a found container its image and the image's date,
the networks it is on and the other containers on each, its mounts and published ports, beside the
declared image (and its date once pulled), ports and volumes, with the downgrade decided when both
dates are known; for a found file whether the declared content differs and how, as lines lost and
lines new. A resource whose target moved keeps the former target on record as an orphan, so the next
apply removes the container or file the host wrote under the old name (issue 097). Every apply reports
the strays: containers the mesh neither wrote nor holds.
2026-10-01 21:24:37 +02:00
mesh-admin e030aa2387 Merge pull request 'The first user list lets the controller publish assignments and hear its seat's tools (hq #251)' (#62) from fix/first-user-list-matches-the-controller into main 2026-10-01 15:29:55 +00:00
jschoubben c670bef4e1 The first user list lets the controller publish assignments and hear its seat's tools
The controller's own composition (mesh-controller internal/broker, 2026-10-01)
publishes memberships into the assignments stream after each push and
subscribes to its seat's tool subjects; the list the installer carries did
not say so, and a controller on it is refused on the first thing it tries:
"Permissions Violation for Publish to mesh.assignment.novox.builder" (hq #251),
which is why every build asked through the console was lost. The controller's
test that compares the two (TestTheInstallersFirstUserListIsWhatTheControllerWouldCompose,
run with this checkout beside it) named exactly these two subjects, and passes.
2026-10-01 17:29:48 +02:00
mesh-admin 45529bfec2 Merge pull request 'The profile names the network manager that is running, and travels in every report (hq ADR 0161)' (#61) from feat/the-profile-names-the-uplink-and-travels-in-the-report into main 2026-10-01 14:02:16 +00:00
jschoubben b1e9ccff6d The profile names the network manager that is running, and travels in every report (hq ADR 0161)
One capability per manager — uplink-networkmanager, uplink-systemd-networkd, uplink-dhcpcd — from
systemctl is-active, so the uplink seat's holder for a manager this machine does not run is refused
the way any missing capability is, naming it (issue 138). The apply that reports detects the profile
again and sends it, the same shape enrolment sends, so a machine that switched managers reaches the
mesh at its next push.
2026-10-01 15:58:31 +02:00
mesh-admin cbcf0bcc93 Merge pull request 'A node can join the bus the mesh runs on' (#51) from fix/a-node-can-join-the-bus-the-mesh-runs-on into main 2026-10-01 11:18:02 +00:00
jschoubben 6910f07d75 Merge pull request 'Say what a container's host entries now are' (#60) from feat/148-names-are-resolved-not-copied into main 2026-09-30 12:38:15 +00:00
jschoubben 88037b33b7 Say what a container's host entries now are
novox/hq ADR 0148: the mesh's names are no longer among them, only what
the module declared. Comment only; the digest is unchanged.
2026-09-30 14:38:11 +02:00
jschoubben 422ad516d5 Merge pull request 'A declaration carries its order, and a host refuses an older one' (#59) from feat/107-a-declaration-carries-its-order into main 2026-09-30 12:03:10 +00:00
jschoubben 8431ecfb48 A declaration carries its order, and a host refuses an older one
novox/hq 04-ISSUES/107. A declaration's only identity was the digest of
its bytes: a host could say "not the last" and could not say "older". On
the link, nothing refused an older one at all, and the drain picked the
last to arrive — wrong exactly when it mattered, a backlog drained out of
order or a broker that split a burst.

A declaration may now carry a sequence, one higher per send. A host
refuses one lower than what it kept, whole, and says why. The drain keeps
the highest sequence in a batch rather than the last to arrive.

Only when both sides claim an order. Absent reads as zero — "no ordering
claimed", not "first" — so a controller that sends none is still
understood and a host that kept one before it understood them compares
nothing. That is what lets hosts go first and the controller follow, which
is the order 087 says a new field needs.
2026-09-30 14:03:03 +02:00
jschoubben f107d68b2d Merge pull request 'A delivered host knows it is the delivered one' (#58) from fix/163-a-delivered-host-knows-it-is-the-delivered-one into main 2026-09-30 11:46:58 +00:00
jschoubben 1028193c8a A delivered host knows it is the delivered one
novox/hq 04-ISSUES/163. The host asks after every apply whether a newer
host is delivered than the one running, and asked with the link-time
version stamp — which every delivered host carries as "development
build", because the version comes from where the binary sits now (0142).
So a delivered host never matched the newest delivered version, stood
aside on every push for ever, and because standing aside cancels the
report, the mesh never heard from it again.

Measured on two machines: each push produced "host <v> is delivered;
standing aside" for the version already running, then "applied, and could
not tell the mesh: reporting: context canceled". A machine restarting its
host on every push and reporting nothing, reading as healthy.

Asked with the running version now. Half of 0142 was applied to the
report and the known-good record and not here; this is the other half.
2026-09-30 13:46:51 +02:00
jschoubben f576287b51 Merge pull request 'A delivered host reads its version from where it sits' (#57) from fix/161-a-delivered-host-reads-its-version-from-its-path into main 2026-09-30 10:40:29 +00:00
jschoubben 6c6495f6d9 A delivered host reads its version from where it sits
novox/hq ADR 0142, which decided this and was not implemented: "It is
unpacked into a directory named for its version, so it can read its own
version from its path. The stamp goes, and with it the need for a build to
know what it will be called."

The mesh's toolchain stamps no version, on purpose, so a delivered host
called itself "development build" and the mesh could not tell which host
any machine ran — which is the whole of what 087 added. A delivered host
lives at <libexec>/versions/<version>/<binary>, and that directory is the
answer.

A host placed by hand keeps its stamp, which is the honest answer for one
the mesh did not deliver, and is every machine until a delivery reaches
it. A binary sitting anywhere else is not read as a version at all.

The decision is split from the reading so a test can ask about a path
without being that binary.
2026-09-30 12:40:00 +02:00
jschoubben e6d48cf537 Merge pull request 'The mesh delivers the launcher, which is the last link in self-update' (#56) from feat/142-the-mesh-delivers-the-launcher into main 2026-09-30 10:19:07 +00:00
jschoubben b8a766f234 The mesh delivers the launcher, which is the last link in self-update
novox/hq ADR 0141 and 04-ISSUES/142. A version was being delivered to a
machine and nothing started it: the launcher on these machines predates
the versions mechanism and runs the fixed binary path, so the delivery was
correct and inert.

Delivered as a FILE resource, not as part of an archive, and the
difference is the whole reason this is safe. A file is written atomically —
temp file in the same directory, then rename — so the running launcher
keeps the inode it was started from and the next start picks up the new
one. An archive writes in place with truncate, which would cut the file a
running shell is reading halfway through.

The manifest therefore carries a second copy of the script, and a test
refuses any difference between it and packaging/nox-mesh-host-launch.
Proven by drifting one and watching it fail. Two copies of a script is a
bad thing to accept, and the alternative was writing over a running
supervisor.

Together the two resources complete the loop: the version lands, the
running host stands aside because it sees one delivered, and the launcher
that starts next is the one that looks in versions/ and picks the newest
by arrival.
2026-09-30 12:19:00 +02:00
jschoubben 9caea5bc32 Merge pull request 'The delivered binary is named as every machine runs it' (#55) from fix/142-the-delivered-binary-is-named-as-machines-run-it into main 2026-09-30 09:41:52 +00:00
jschoubben df27cee7b7 The delivered binary is named as every machine runs it
nox-mesh-host, not mesh-host. The command directory is cmd/mesh-host and
the launcher looks inside a delivered version for nox-mesh-host — the name
this is installed at and the name in its unit. The first delivery landed
the package's name, reported success, and would have been invisible.
2026-09-30 11:41:45 +02:00
jschoubben d275e64ed3 Merge pull request 'The host declares its own successor, as an archive at a versioned path' (#54) from feat/142-the-host-delivers-its-successor into main 2026-09-30 09:33:19 +00:00
jschoubben 1a628a4d22 The host declares its own successor, as an archive at a versioned path
novox/hq ADR 0141 and 04-ISSUES/142. The host half of the delivery has
been built and tested since 0141 and has never had a version to work on:
versions side by side, the newest runs, the running one stands aside
between reconciles, rollback picks a directory. This is the declaration
that gives it one.

One archive, unpacked to /usr/lib/nox-mesh-host/versions/${version}. The
version resolves to the artifact's digest, so an unchanged build lands at
the path it already had and re-composing a declaration moves nothing.

Not circular: the host applying this is a different version from the one
being written, and neither writes over the other — the kernel refuses to
truncate a running executable, which is the reason the path carries the
version rather than a link pointing at "current".

**The launcher is deliberately not delivered here.** The one on these
machines predates the versions mechanism and runs the fixed binary path,
so a delivered version is inert until it is replaced — and replacing it
from the mesh means writing over a running shell script, which sh reads
incrementally. That wants a designed swap rather than a file resource, and
it is the last piece rather than this one.
2026-09-30 11:33:12 +02:00
jschoubben b5196e974c Merge pull request 'The host is a module, so the mesh can build it' (#53) from feat/142-the-host-is-a-module into main 2026-09-30 07:56:35 +00:00
jschoubben 5162c3b05f The host is a module, so the mesh can build it
novox/hq 04-ISSUES/142 and ADR 0142. Nothing delivered the host because
nothing could compile it, and nothing could compile it partly because the
host was not a thing the mesh builds at all — it had no manifest.

One bundle in Go, for arch, built from cmd/mesh-host. A system is
required for a compiled artifact because a binary is pinned at link time
so a host refuses to touch a machine it was not built for (ADR 0005), and
`arch` is what all four of this mesh's machines report themselves to be.
Another system is another artifact and another build, which is what ADR
0142 means by one per target.

No resources yet. What places a version into a directory named for it
needs an archive resource whose path carries the version, and nothing
interpolates one — the second half of 0141's insight, and the next piece.
2026-09-30 09:56:18 +02:00
jschoubben 98fe8edf35 Merge pull request 'An apply says what it held, not only what it applied' (#52) from fix/125-a-hold-is-a-line-in-the-report into main 2026-09-30 06:47:45 +00:00
jschoubben cbf50185d0 An apply says what it held, not only what it applied
novox/hq 04-ISSUES/125. An adopted node keeps what it found until its
module is taken, which is correct and was recorded only in the node's own
state file. On the edge cut-over the mesh sent 346 resources, the journal
said it applied 330, and nothing anywhere said which sixteen or why —
reading it meant opening state.json by hand, and not reading it took every
public name on the machine down.

The line that reports the apply now carries it, grouped by module and
ordered by name, because the sentence an operator needs is "route-proxy is
assigned and not taken" and the module is the thing `take` acts on. An
apply that held nothing says nothing extra: a line that reports "0 held"
on every converged apply is a line that stops being read.
2026-09-30 08:46:19 +02:00
jschoubben 197258c88c A node can join the bus the mesh runs on
novox/hq 04-ISSUES/146, the layers behind the three already fixed.

A new membership says which bus it is for. Empty meant 'whatever the mesh runs
today' while two buses existed, and became a refusal the moment one did: an
enrolled node came up and reconnected for ever against its own record.

The enrolling client takes its inboxes in the space its user may listen in. A
JetStream publish waits for the stream's acknowledgement on an inbox the client
picks, and its default is one this user may not subscribe to — so the enrolment
failed with a permissions violation on a subject nobody had chosen.

And the enrolment publish carries a message id, so the client's own retry is
discarded by the stream rather than enrolling the machine twice. That one is
not finished: the duplicate survives it, and the issue says where the trail
stops.
2026-09-29 17:36:59 +02:00
jschoubben a3b810f1f0 Merge pull request 'A first node gets as far as its own bus: three faults on the way' (#50) from fix/one-foundation-on-the-bus-the-mesh-runs-on into main 2026-09-29 14:06:33 +00:00
jschoubben 971a6d6d03 A first node gets as far as its own bus: three faults on the way
novox/hq 04-ISSUES/146. Each was right while the mesh ran on the previous
broker, and nothing has raised a foundation since it changed.

The bus's certificate is made by the program that needs it rather than by
openssl inside the broker's image — the bus's image is Alpine with a shell and
no openssl, so the step exited 127 and no mesh could be raised. Self-signed as
before and on purpose; --user 0:0 because the volume is root's and the control
plane's image runs as nobody.

Enrolment no longer opens a raw TLS connection to check the pin: NATS speaks
its own protocol and upgrades afterwards, so the handshake met a plaintext
greeting. The client that presents the token carries the same pinned config
and verifies inside its own handshake, so the secret still leaves only after
the certificate is checked. The raw dial stays as what its tests prove, and is
no longer a path anything takes.

And the token says which bus it is for. Empty meant 'whatever the mesh runs
today' while two buses existed and became a refusal the moment one did.

It now stops at the bus's user list, which is the genesis half of 146.
2026-09-29 15:42:43 +02:00
mesh-admin 04a27caa43 Merge pull request 'A host running as a service says what its apply did' (#49) from fix/a-host-running-as-a-service-says-what-it-did into main 2026-09-29 07:16:01 +00:00
jschoubben 6e90c2692d A host running as a service says what its apply did
The serving path passed nil where the apply writes its detail. Nil is silence, so
everything the apply says — a file held, a container replaced, the found firewall
retired — was visible when a person ran the one-shot command and discarded in the
way the host actually runs, which is always.

Measured: after a machine was converged and its found firewall was not retired,
what the host decided was unrecoverable, because it had said it to nobody. That is
why issue 143 has candidates instead of a cause.

say already reaches stdout and the unit sends that to the journal, so this needed
no new mechanism — only for the argument to be passed. Both paths now reach the
apply through one named helper, so a reader asking where the apply's output goes
finds one answer.

The test asserts the log is never nil and cannot catch the fault it was written
for, which is wiring; that is proved by a deployed host whose journal carries the
detail.
2026-09-29 09:15:53 +02:00
mesh-admin ced54d489f Merge pull request 'A converged machine can speak unasked' (#48) from fix/a-converged-machine-can-speak-unasked into main 2026-09-28 23:13:13 +00:00
jschoubben 94a35a39eb A converged machine can speak unasked
A reconcile is otherwise silent, and the condition deciding when it speaks asked
only what an adopted node reports: what it holds, and the firewall it found. A
converged node has neither, so it could speak only in reply to a declaration —
and the mesh composes no declaration for a node that has not said which links
face outside (ADR 0140). A machine waiting for a push that was waiting for the
machine.

Measured, not predicted: after the control plane learnt to read the links, the
control node recorded its own and the three converged machines sat silent while
their filters were refused.

The condition is now a named predicate, because as an inline expression nothing
could test it — which is why the gap shipped.
2026-09-29 01:13:09 +02:00
mesh-admin ec06369101 Merge pull request 'A machine says which links face outside without being asked' (#47) from fix/a-machine-says-unasked-which-links-face-outside into main 2026-09-28 23:00:28 +00:00
jschoubben bb85af1821 A machine says which links face outside without being asked
The mesh composes no filter for a machine that has not said (ADR 0140), and a
converged machine only speaks unasked when its adoption fingerprint changes. The
links were not in that fingerprint, so a machine that had just learnt to say
could only speak when a declaration arrived — and a declaration cannot be
composed until it has spoken. A machine waiting for a push that is waiting for
the machine.

Found before it bit: every machine in this mesh is in exactly that state right
now, having just been given a host that reports the links to a control plane that
does not yet read them.
2026-09-29 01:00:26 +02:00
mesh-admin 0c3928ad19 Merge pull request 'The host delivers its own successor, and versions live side by side' (#46) from feat/the-host-delivers-its-own-successor into main 2026-09-28 22:29:57 +00:00
jschoubben fbf0fb7d63 The host delivers its own successor, and versions live side by side
The supervision was already right: a clean exit means the host stood aside, and
the launcher's next turn runs what is on disk. Two things made it dead code —
nothing told the running host a successor was waiting, and the rollback resolved
its known-good version through pacman, which no machine here uses and which two
of three operating systems do not have.

Keeping a version rather than a path was the clue. Versions now live in
directories named for them:

- the launcher picks the newest delivered one every time round the loop, or the
  one a rollback pinned, or the host placed by hand when nothing is delivered;
- the running host stands aside between reconciles, never inside one, by exiting
  cleanly — and returns nil so the launcher does not count it as a crash;
- a completed reconcile retires what is older than the predecessor, keeping the
  predecessor because that is what a rollback starts, and never the running one;
- rollback pins the predecessor instead of reinstalling a package: no package
  manager, no cache anyone may clean, same script on every operating system;
- the report says which host version produced it, so 'behind' is answerable.

Newest is when it arrived, never how the name sorts: '1.10' orders before '1.9',
and ordering by name would start an older host and call it an upgrade.

novox/hq ADR 0141. The delivery half — a module carrying the next host — follows;
until then nothing delivers a version and every machine takes the fallback, which
is what it does today.
2026-09-29 00:29:36 +02:00
mesh-admin b005d4ef17 Merge pull request 'A machine says which of its links face outside' (#45) from feat/filter-what-arrives-from-outside into main 2026-09-28 21:37:24 +00:00
jschoubben b0d11c2439 A machine says which of its links face outside
The filter blocks everything passing through the machine and then allows the
machine's own containers back by naming the address ranges they sit on — two
ranges fixed in the control plane and the rest typed after a flip had already
cut a workstation off. A range describes one machine and goes stale in silence.

Read the links carrying a default route instead, from /proc rather than by
asking a program, and report them on every apply. A machine with no route off
itself reports nothing, and the mesh composes no filter for it rather than
writing a rule around a link with no name.

novox/hq ADR 0140. The control plane does not read this yet.
2026-09-28 23:37:06 +02:00
mesh-admin 155689672c Merge pull request 'A container's mesh names are part of what it is' (#44) from fix/a-containers-mesh-names-are-part-of-what-it-is into main 2026-09-28 15:19:39 +00:00
jschoubben e82789a322 A container's mesh names are part of what it is
A container resolves every machine and public name through the entries it is given when it is created,
and nothing re-reads them. The host compared everything about a container except those, so one whose
image and files never changed was left alone holding an overlay address five days out of date — it
restarted 2286 times against a database it could no longer find, and the mesh reported the machine as
doing what it was told (novox/hq 04-ISSUES/135, the same fault as 045 in the field left out).

Sorted, so the digest does not move for a reordering nobody made. The first apply after this recreates
every container that carries mesh names, once.
2026-09-28 17:19:38 +02:00
mesh-admin 99562947f3 Merge pull request 'Genesis lets the control plane state its own facts' (#43) from fix/genesis-lets-the-control-plane-state-its-facts into main 2026-09-28 14:07:22 +00:00
jschoubben c171c64d3a Genesis lets the control plane state its own facts
A mesh raised from nothing must be able to say what it applied and what a machine refused (novox/hq
ADR 0134), and the first user list is what permits it. Kept in step with the controller's own
composition by the test that reads this file.
2026-09-28 16:07:20 +02:00
mesh-admin 0dc5515099 Merge pull request 'A resource's owner is read to the last dot, because a module's name may contain one' (#42) from fix/a-resources-owner-is-read-to-the-last-dot into main 2026-09-28 13:45:01 +00:00
jschoubben 58c0e715c7 A resource's owner is read to the last dot, because a module's name may contain one
novox.be is a module on this mesh. Reading a resource's owner to the first dot made its resources
belong to something called "novox", and a step's gate would then skip whatever else happened to start
that way — silently. A resource's own id never contains a dot, which is what makes the last one the
boundary; the mesh's derived step was changed to add a hyphen rather than a dot for the same reason
(mesh-controller #128).
2026-09-28 15:44:59 +02:00
mesh-admin c5b229f95d Merge pull request 'A step gates its module, not the machine' (#41) from fix/a-step-gates-its-module-not-the-machine into main 2026-09-28 13:38:21 +00:00
jschoubben a9c49724b5 A step gates its module, not the machine
A run-once container that does not complete stops the rest of that module's resources; everything
else on the machine is attempted, as every other shape already is (novox/hq ADR 0136). An action
still gates the machine — genesis is a row of them and they belong to no module. What was not
attempted is reported as skipped, because that and 'nothing to do' are different answers.

Without this, ADR 0135's derived preparation would let one module's unreachable database hold a
machine hostage — the fault 04-ISSUES/011 removed for everything else, and the reason the catalogue
migrates itself at start.
2026-09-28 15:38:19 +02:00
mesh-admin 296ec5ece6 Merge pull request 'Genesis lets the controller ask any module's tool' (#40) from fix/genesis-lets-the-controller-ask into main 2026-09-28 02:21:24 +00:00
60 changed files with 5373 additions and 233 deletions
+153
View File
@@ -0,0 +1,153 @@
package main
import (
"context"
"errors"
"fmt"
"os"
"os/exec"
"path/filepath"
"strings"
"time"
"github.com/novox/mesh-host/internal/apply"
"github.com/novox/mesh-host/internal/identity"
)
// Joining through the tunnel (novox/hq ADR 0169).
//
// **The bus is never open to the internet, so a joining machine reaches it over the tunnel.** It
// makes its tunnel key first and prints the public half; the token is issued for that key, and the
// hub is told the key before the token is shown; the token carries the one peer this machine needs.
// So the tunnel can come up before the mesh has said anything else — the circle ADR 0004 broke by
// carrying the bus's address in the token is broken here by carrying the hub's.
// tunnelConfigPath and tunnelUnit are where the mesh's own declaration puts the private network, so
// the first tunnel is the same interface and unit the mesh takes over, not a second one beside it.
var (
tunnelConfigPath = "/etc/wireguard/mesh0.conf"
tunnelUnit = "wg-quick@mesh0"
// lookPath finds WireGuard's tools; a variable so a test needs none installed.
lookPath = exec.LookPath
)
// keyCommand makes this machine's tunnel key, or reads the one it already made, and prints the
// public half: what the token is issued for. Making it twice would be a token issued for a key the
// machine no longer has, so an existing key is kept.
func keyCommand(opts options) error {
path := identity.OverlayKeyPath(opts.state)
if key, err := identity.LoadOverlayKey(path); err == nil {
fmt.Println(key.Public)
return nil
} else if !errors.Is(err, os.ErrNotExist) {
return err
}
if _, err := os.Stat(identity.Path(opts.state)); err == nil {
return fmt.Errorf("this machine has joined already (%s), and its tunnel key is its own; "+
"there is no key to make", identity.Path(opts.state))
}
key, err := identity.GenerateOverlayKey()
if err != nil {
return err
}
if err := os.MkdirAll(filepath.Dir(path), 0o700); err != nil {
return err
}
if err := os.WriteFile(path, []byte(key.Private+"\n"), 0o600); err != nil {
return fmt.Errorf("cannot write this machine's tunnel key: %w", err)
}
fmt.Println(key.Public)
fmt.Fprintln(os.Stderr, "\nthis machine's tunnel key, made here; the private half stays in "+path+".\n"+
"Issue the token for it — `token issue --new <name> --overlay-key <the line above>` — and enrol with that token.")
return nil
}
// tunnelKeyFor is the key a token through the tunnel was issued for, read from where `key` left it.
// Refused when there is none, or it is another: the hub knows only the key the token names.
func tunnelKeyFor(t *identity.TokenTunnel, state string) (identity.OverlayKey, error) {
path := identity.OverlayKeyPath(state)
key, err := identity.LoadOverlayKey(path)
if errors.Is(err, os.ErrNotExist) {
return identity.OverlayKey{}, fmt.Errorf("this token was issued for a tunnel key, and this " +
"machine has none: run `nox-mesh-host key` here first and issue the token for the key it prints")
}
if err != nil {
return identity.OverlayKey{}, err
}
if key.Public != t.Key {
return identity.OverlayKey{}, fmt.Errorf("this token was issued for the tunnel key %s, and this "+
"machine's is %s — it is another machine's token, or the key was made again; issue a new "+
"token for %s", t.Key, key.Public, key.Public)
}
return key, nil
}
// tunnelConfig is the first tunnel: this machine's address, and the hub as its one peer, reaching the
// whole private network through it. The private key is set from its file, as the mesh's own
// declaration does it, so the file holds no secret.
func tunnelConfig(t *identity.TokenTunnel, keyPath string) string {
return fmt.Sprintf(`# Written by nox-mesh-host enrol: the one peer a joining machine needs (novox/hq ADR 0169).
# The mesh's own declaration replaces this once the machine has joined.
[Interface]
Address = %s
PostUp = wg set %%i private-key %s
[Peer]
PublicKey = %s
Endpoint = %s
AllowedIPs = %s
PersistentKeepalive = 25
`, t.Address, keyPath, t.HubKey, t.HubEndpoint, t.Range)
}
// bringTheTunnelUp writes the first tunnel and starts it, so the bus the token names can be reached.
func bringTheTunnelUp(ctx context.Context, t *identity.TokenTunnel, keyPath string, run apply.Runner) error {
if _, err := lookPath("wg-quick"); err != nil {
return errors.New("joining through the tunnel needs WireGuard's tools on this machine " +
"(wireguard-tools), and wg-quick is not here")
}
if err := os.MkdirAll(filepath.Dir(tunnelConfigPath), 0o700); err != nil {
return err
}
if err := os.WriteFile(tunnelConfigPath, []byte(tunnelConfig(t, keyPath)), 0o600); err != nil {
return fmt.Errorf("cannot write the first tunnel: %w", err)
}
if out, err := run(ctx, "systemctl", "restart", tunnelUnit); err != nil {
return fmt.Errorf("the first tunnel would not start (%s): %v %s", tunnelUnit, err, strings.TrimSpace(out))
}
fmt.Printf("the tunnel to the hub is up: %s, through %s\n", t.Address, t.HubEndpoint)
return waitForTheHub(ctx, run, handshakeWithin)
}
// handshakeWithin is how long the hub has to answer the first tunnel. Issuing the token sent the hub
// this machine as a peer; the hub applies that on its own time, and a bus dialled before it has is a
// timeout that names the bus rather than the tunnel.
var handshakeWithin = 90 * time.Second
// waitForTheHub waits until the tunnel has shaken hands with the hub, so the bus is dialled over a
// tunnel that answers — and says so in the tunnel's own words when it does not.
func waitForTheHub(ctx context.Context, run apply.Runner, within time.Duration) error {
deadline := time.Now().Add(within)
for {
out, err := run(ctx, "wg", "show", "mesh0", "latest-handshakes")
if err == nil {
for _, line := range strings.Split(strings.TrimSpace(out), "\n") {
fields := strings.Fields(line)
if len(fields) == 2 && fields[1] != "0" {
fmt.Println("the hub answered the tunnel")
return nil
}
}
}
if time.Now().After(deadline) {
return fmt.Errorf("the hub has not answered the tunnel in %s: the token may be another machine's, "+
"the hub may not have been sent this machine as a peer, or its tunnel's port is not reachable "+
"from here", within)
}
select {
case <-ctx.Done():
return ctx.Err()
case <-time.After(2 * time.Second):
}
}
}
+139
View File
@@ -0,0 +1,139 @@
package main
import (
"context"
"io"
"os"
"path/filepath"
"strings"
"testing"
"time"
"github.com/novox/mesh-host/internal/identity"
)
// `key` makes the tunnel key once and prints its public half; asked again it prints the same one,
// because a token may already have been issued for it (novox/hq ADR 0169).
func TestKeyMakesTheTunnelKeyOnceAndKeepsIt(t *testing.T) {
dir := t.TempDir()
opts := options{state: filepath.Join(dir, "state.json")}
first := captureStdout(t, func() {
if err := keyCommand(opts); err != nil {
t.Fatal(err)
}
})
second := captureStdout(t, func() {
if err := keyCommand(opts); err != nil {
t.Fatal(err)
}
})
if strings.TrimSpace(first) == "" || strings.TrimSpace(first) != strings.TrimSpace(second) {
t.Fatalf("the key changed between two asks: %q then %q", first, second)
}
info, err := os.Stat(identity.OverlayKeyPath(opts.state))
if err != nil {
t.Fatal(err)
}
if info.Mode().Perm() != 0o600 {
t.Errorf("the private half is readable beyond root: %v", info.Mode().Perm())
}
}
// A token through the tunnel takes the key it was issued for, and says so when this machine has none
// or another.
func TestATokenThroughTheTunnelTakesItsOwnKey(t *testing.T) {
dir := t.TempDir()
state := filepath.Join(dir, "state.json")
tt := &identity.TokenTunnel{Key: "x", Address: "10.42.0.9/32", Range: "10.42.0.0/16", HubKey: "h", HubEndpoint: "198.51.100.1:51820"}
if _, err := tunnelKeyFor(tt, state); err == nil || !strings.Contains(err.Error(), "nox-mesh-host key") {
t.Fatalf("a machine with no key was not told to make one: %v", err)
}
captureStdout(t, func() { _ = keyCommand(options{state: state}) })
if _, err := tunnelKeyFor(tt, state); err == nil || !strings.Contains(err.Error(), "issued for the tunnel key x") {
t.Fatalf("another machine's token was taken: %v", err)
}
mine, _ := identity.LoadOverlayKey(identity.OverlayKeyPath(state))
tt.Key = mine.Public
if got, err := tunnelKeyFor(tt, state); err != nil || got.Public != mine.Public {
t.Fatalf("this machine's own token was refused: %v", err)
}
}
// The first tunnel is the mesh's interface and unit, with the hub as its one peer and no secret in
// the file — the same shape the mesh's declaration replaces it with.
func TestTheFirstTunnelIsTheMeshsInterfaceWithTheHubAsItsPeer(t *testing.T) {
dir := t.TempDir()
tunnelConfigPath = filepath.Join(dir, "wireguard", "mesh0.conf")
lookPath = func(string) (string, error) { return "/usr/bin/wg-quick", nil }
t.Cleanup(func() { tunnelConfigPath = "/etc/wireguard/mesh0.conf" })
var ran []string
run := func(_ context.Context, name string, args ...string) (string, error) {
if name == "wg" {
return "HUBKEY\t1759400000\n", nil
}
ran = append(ran, name+" "+strings.Join(args, " "))
return "", nil
}
tt := &identity.TokenTunnel{Key: "k", Address: "10.42.0.9/32", Range: "10.42.0.0/16", HubKey: "HUBKEY", HubEndpoint: "198.51.100.1:51820"}
captureStdout(t, func() {
if err := bringTheTunnelUp(context.Background(), tt, "/var/lib/mesh-host/overlay.key", run); err != nil {
t.Fatal(err)
}
})
raw, err := os.ReadFile(tunnelConfigPath)
if err != nil {
t.Fatal(err)
}
conf := string(raw)
for _, want := range []string{"Address = 10.42.0.9/32", "PostUp = wg set %i private-key /var/lib/mesh-host/overlay.key",
"PublicKey = HUBKEY", "Endpoint = 198.51.100.1:51820", "AllowedIPs = 10.42.0.0/16", "PersistentKeepalive = 25"} {
if !strings.Contains(conf, want) {
t.Errorf("the first tunnel lacks %q:\n%s", want, conf)
}
}
if strings.Contains(conf, "PrivateKey") {
t.Error("the first tunnel's file holds the private key")
}
if len(ran) != 1 || ran[0] != "systemctl restart wg-quick@mesh0" {
t.Errorf("the tunnel was started as %v", ran)
}
}
// captureStdout is what fn printed to standard output.
func captureStdout(t *testing.T, fn func()) string {
t.Helper()
r, w, err := os.Pipe()
if err != nil {
t.Fatal(err)
}
was := os.Stdout
os.Stdout = w
fn()
os.Stdout = was
w.Close()
out, _ := io.ReadAll(r)
return string(out)
}
// The bus is dialled only once the hub has answered the tunnel, and a hub that never does is said
// as the tunnel's fault rather than the bus's.
func TestTheBusWaitsForTheHubToAnswer(t *testing.T) {
asked := 0
answersOnThird := func(_ context.Context, name string, args ...string) (string, error) {
asked++
if asked < 3 {
return "HUBKEY\t0\n", nil
}
return "HUBKEY\t1759400000\n", nil
}
captureStdout(t, func() {
if err := waitForTheHub(context.Background(), answersOnThird, time.Minute); err != nil {
t.Fatal(err)
}
})
never := func(context.Context, string, ...string) (string, error) { return "HUBKEY\t0\n", nil }
err := waitForTheHub(context.Background(), never, 0)
if err == nil || !strings.Contains(err.Error(), "has not answered the tunnel") {
t.Fatalf("a hub that never answered was not said: %v", err)
}
}
+295 -30
View File
@@ -33,6 +33,7 @@ import (
"github.com/novox/mesh-host/internal/identity"
"github.com/novox/mesh-host/internal/inventory"
"github.com/novox/mesh-host/internal/link"
"github.com/novox/mesh-host/internal/outward"
"github.com/novox/mesh-host/internal/profile"
"github.com/novox/mesh-host/internal/reachable"
"github.com/novox/mesh-host/internal/store"
@@ -50,6 +51,43 @@ var builtFor = ""
var version = "development build"
// runningVersion is this host's version: the directory it was delivered into, or the link-time stamp
// for one placed by hand.
//
// **From where it sits, not from its linker** (novox/hq ADR 0142): "It is unpacked into a directory
// named for its version, so it can read its own version from its path. The stamp goes, and with it the
// need for a build to know what it will be called."
//
// The mesh's toolchain does not stamp a version, on purpose — a build does not know what it will be
// called — so a delivered host read as "development build" and the mesh could not tell which host any
// machine ran (novox/hq 04-ISSUES/161, and 087 for why that matters). The path knows: a delivered host
// lives at `<libexec>/versions/<version>/<binary>`.
//
// A host placed by hand keeps its stamp, which is the honest answer for one the mesh did not deliver.
func runningVersion() string {
self, err := os.Executable()
if err != nil {
return version
}
return versionAt(self, version)
}
// versionAt is runningVersion's decision, with the executable's path and the link-time stamp given —
// so a test can ask it about a path without being that binary.
func versionAt(self, stamped string) string {
// .../versions/<version>/<binary> — the parent is the version, and its parent is the versions
// directory. Checked rather than assumed, so a binary somewhere else does not read a directory
// name as a version.
dir := filepath.Dir(self)
if filepath.Base(filepath.Dir(dir)) != upgrade.VersionsDirName {
return stamped
}
if name := filepath.Base(dir); name != "" && name != "." && name != string(filepath.Separator) {
return name
}
return stamped
}
const usage = `mesh-host — the node host
profile what this machine can be asked to do
@@ -58,6 +96,9 @@ const usage = `mesh-host — the node host
reconcile make this machine match what the mesh last told it — or, before any
mesh has, the bundle this host carries
bundle show what this host carries
key make this machine's tunnel key, or read the one it made, and print the public
half: what its join token is issued for (novox/hq ADR 0169)
enrol --token T join the mesh — through the tunnel when the token was issued for a key
overlay take take over the tunnel found here (novox/hq ADR 0105): its key becomes this
node's overlay key and the mesh is told, signed; --tunnel <iface> when several are up
owned what this host has applied and still owns
@@ -174,6 +215,9 @@ func parseArgs(args []string) (string, options, error) {
func run(ctx context.Context, command string, opts options) error {
jsonOut, timeout := opts.json, opts.timeout
switch command {
case "key":
return keyCommand(opts)
case "profile":
p := profile.Detect(ctx, profile.Default(nil), timeout)
if jsonOut {
@@ -253,7 +297,7 @@ func run(ctx context.Context, command string, opts options) error {
return runLink(ctx, opts)
case "version":
fmt.Println(version)
fmt.Println(runningVersion())
return nil
case "", "help", "-h", "--help":
@@ -419,10 +463,10 @@ func short(digest string) string {
// them again — and everything the mesh declared read as no longer declared and removed. Even the
// very declaration the mesh last sent, applied from a file, would plan to remove the foundation.
// `apply FILE` is for a machine the mesh has not spoken to, and is refused saying so.
func refuseStale(known store.State, kept store.Declared, keptErr error, digest string, from provenance) error {
func refuseStale(known store.State, kept store.Declared, keptErr error, digest string, from provenance, sequence int64) error {
switch from {
case fromDeclared:
return nil
return refuseOlder(kept, keptErr, sequence)
case fromBundle:
if known.Genesis == nil || known.Genesis.Digest == digest {
return nil
@@ -519,7 +563,7 @@ func runApply(ctx context.Context, opts options, d *declaration.Declaration, raw
if err := apply.CheckMode(known, d); err != nil {
return err
}
if err := refuseStale(known, kept, keptErr, digest, from); err != nil {
if err := refuseStale(known, kept, keptErr, digest, from, d.Sequence); err != nil {
return err
}
@@ -593,13 +637,27 @@ func runApply(ctx context.Context, opts options, d *declaration.Declaration, raw
//
// A failure to record is reported and does not fail the apply. The apply worked; what is
// lost is a rollback's ability to come back here, which is worse to hide than to say.
if version != "" {
if err := upgrade.RecordKnownGood(upgrade.KnownGoodPath(opts.state), version); err != nil {
if v := runningVersion(); v != "" {
if err := upgrade.RecordKnownGood(upgrade.KnownGoodPath(opts.state), v); err != nil {
fmt.Fprintf(os.Stderr,
"mesh-host: applied, but could not record %s as known-good: %v\n"+
" a rollback would have nothing to return to.\n", version, err)
}
}
// And retire what is older than this version's predecessor, on the same evidence known-good is
// written on (novox/hq ADR 0141). The predecessor stays, because it is exactly what a rollback
// starts; everything before it has no reader. Never this version, whatever the answer.
//
// A failure is said and does not fail the apply, for the reason above: what is lost is disk, and
// hiding it would make a machine quietly fill up.
if version != "" {
if retired, err := upgrade.Retire(upgrade.VersionsDir(""), version); err != nil {
fmt.Fprintf(os.Stderr, "mesh-host: applied, and could not retire an older host: %v\n", err)
} else if len(retired) > 0 {
fmt.Fprintf(os.Stderr, "mesh-host: retired the host version(s) %s\n",
strings.Join(retired, ", "))
}
}
// And tell the launcher this start worked. Without it the counter only climbs, and a node
// that has been up for months rolls itself back on its third ordinary restart.
if err := upgrade.ClearAttempts(upgrade.AttemptsPath(opts.state)); err != nil {
@@ -691,15 +749,18 @@ func enrol(ctx context.Context, opts options) error {
fmt.Printf(" signing key %s\n",
base64.StdEncoding.EncodeToString(token.Signer)[:16]+"...")
// The check that has to happen before this machine says anything.
conn, err := link.Dial(token.Broker, token.Fingerprint, opts.timeout)
if err != nil {
return err
}
defer conn.Close()
fmt.Println("\nthe broker presented the certificate this token pins")
conn.Close()
// **The pin is checked by the connection that presents this token, not by a dial of our own**
// (novox/hq 04-ISSUES/146). This opened a raw TLS connection to the bus first, which worked
// against the broker the mesh used to run and cannot work against the one it runs now: NATS
// speaks its own protocol before it upgrades to TLS, so an immediate handshake is answered
// with a plaintext line and the enrolment failed with "first record does not look like a TLS
// handshake" — on every node that has tried to join since the bus changed, which is why this
// went unnoticed: none had.
//
// What ADR 0004 requires still holds, and holds better: the client that presents the token
// carries the same pinned configuration, reads the server's greeting, upgrades, and the
// verification runs inside that handshake — so the one-time secret is sent only after the
// certificate has been checked, and nothing of this node's reaches an impostor.
mine, err := identity.Generate(*name)
if err != nil {
@@ -733,7 +794,18 @@ func enrol(ctx context.Context, opts options) error {
fmt.Printf("this node's overlay key is the found tunnel's (%s): %s\n", tun, mine.Overlay.Public)
}
}
if found == nil {
switch {
case found == nil && token.Tunnel != nil:
// Through the tunnel (novox/hq ADR 0169): the key `key` made, which the token names, and the
// tunnel brought up from the token before the bus is dialled — the bus is reached over it.
mine.Overlay, err = tunnelKeyFor(token.Tunnel, opts.state)
if err != nil {
return err
}
if err := bringTheTunnelUp(ctx, token.Tunnel, identity.OverlayKeyPath(opts.state), apply.ExecRunner); err != nil {
return err
}
case found == nil:
mine.Overlay, err = identity.GenerateOverlayKey()
if err != nil {
return err
@@ -763,20 +835,23 @@ func enrol(ctx context.Context, opts options) error {
// control plane cannot decide what a node should run without it, so it travels with the
// request instead of being asked for in a second round trip.
detected := profile.Detect(ctx, profile.Default(nil), opts.timeout)
reported := map[string]any{}
if raw, err := json.Marshal(detected); err == nil {
_ = json.Unmarshal(raw, &reported)
}
reported := profileAsReported(detected)
// Signed with the identity just generated, so the mesh can tell this machine from anyone else
// who knows its public key (novox/hq issue 083).
proof := mine.Sign(link.EnrolProof(token.Secret, mine.Public, mine.Overlay.Public,
sealing.Public, serving.Public))
// The token says where to go and which certificate that address must present. It says nothing
// about which bus is there, and does not need to: every token names the one the mesh runs on
// today until the rollout (novox/hq ADR 0116 step 5), and that is what an empty Transport is.
// about which bus is there, and does not need to: there is one, and this host knows which
// (novox/hq ADR 0131 — the mesh speaks to one seat and the old transport is gone).
//
// **It used to leave this empty** and mean "whatever the mesh runs today", which was true
// while two buses existed and became a refusal the moment one did: an empty transport is not
// the bus's name, so every enrolment ended at "this token is for the \"\" bus"
// (novox/hq 04-ISSUES/146). Nothing caught it because nothing had enrolled since the bus
// changed.
reply, err := link.Enrol(ctx,
link.Approach{Address: token.Broker, Fingerprint: token.Fingerprint},
link.Approach{Address: token.Broker, Fingerprint: token.Fingerprint, Transport: link.OnNATS},
*name, token.Secret,
mine.Public, mine.Overlay.Public, sealing.Public, serving.Public, reported, proof, found,
opts.timeout)
@@ -792,6 +867,13 @@ func enrol(ctx context.Context, opts options) error {
Fingerprint: firstNonEmpty(reply.Fingerprint, token.Fingerprint),
Signer: firstNonEmpty2(reply.Signer, token.Signer),
Password: reply.Password,
// **Which bus this membership is for, said rather than left empty** (novox/hq
// 04-ISSUES/146). The link refuses a membership that names another bus, and an empty name
// is not this one's — so a node enrolled without it came up and reconnected for ever
// against its own record: "this membership is for \"\", and the mesh's bus is nats". The
// reply does not carry it because there is one bus and the host knows which (ADR 0131);
// what was missing was writing that down where the link reads it.
Transport: link.OnNATS,
}
if mine.Membership.Password == "" {
// The mesh did not replace the token's secret, so it is still this node's broker
@@ -978,12 +1060,36 @@ func runLink(ctx context.Context, opts options) error {
sched := apply.NewScheduler(apply.SystemClock(), apply.ExecRunner, say)
go sched.Run(ctx)
// **Standing aside for a successor happens between reconciles and nowhere else** (novox/hq ADR
// 0141). A host that stood aside mid-apply is the half-configured machine this host exists to
// prevent, so the question is asked after an apply has finished and the answer is a clean exit —
// which the launcher already reads as "run whatever is on disk now".
aside, standAside := context.WithCancel(ctx)
defer standAside()
stoodAside := false
applier := func(ctx context.Context, raw, signature []byte) link.Report {
report := applyAndKeep(ctx, opts, raw, &store.Declared{Declaration: raw, Signature: signature}, sched, say)
// **A declaration may carry this machine's membership for another bus.** It arrives as a
// sealed file like any secret, and is read after the rest has applied so the bus it names is
// standing before this machine leaves the one it is on (novox/hq design 28, task 5.2).
adoptDeliveredMembership(identity.Path(opts.state), &mine, say)
// Asked with the version this host is RUNNING, read from where it sits — not the link-time
// stamp, which every delivered host carries as "development build". Asked with the stamp,
// a delivered host never matched the newest delivered version, so it stood aside on every
// push for ever, and standing aside cancels the report, so the mesh never heard from it
// again (novox/hq 04-ISSUES/163).
switch next, waiting, err := upgrade.Successor(upgrade.VersionsDir(""), runningVersion()); {
case err != nil:
// Said, not fatal. A host that cannot read the delivered versions is still running this
// machine correctly; what it has lost is the ability to be replaced.
say(fmt.Sprintf("cannot tell whether a newer host is delivered: %v", err))
case waiting:
say(fmt.Sprintf("host %s is delivered; standing aside so the launcher runs it", next.Version))
stoodAside = true
standAside()
}
return report
}
@@ -1014,7 +1120,7 @@ func runLink(ctx context.Context, opts options) error {
}}
})
return link.HoldRoused(ctx, link.Membership{
held := link.HoldRoused(aside, link.Membership{
Node: mine.Node,
Broker: mine.Membership.Broker,
Fingerprint: mine.Membership.Fingerprint,
@@ -1022,6 +1128,13 @@ func runLink(ctx context.Context, opts options) error {
Transport: mine.Membership.Transport,
Signer: mine.Membership.Signer,
}, applier, say, opts.timeout, rousedBySignal(ctx), outbox)
// **Cleanly**, or the launcher counts standing aside as a crash and rolls the new host back
// before it has run once. The context this returns on was cancelled deliberately, so its error
// is not a fault to report.
if stoodAside {
return nil
}
return held
}
// adoptionWatch remembers what the node last said about what it holds and its firewall, so a
@@ -1036,15 +1149,30 @@ type adoptionWatch struct {
// is what the controller previews a flip from, so a port that opens or closes between deliveries
// must reach it too (novox/hq ADR 0100).
func adoptionFingerprint(r link.Report) string {
parts := []string{"firewall=" + r.Firewall}
// **Which links face outside is part of it, though it is not about adoption** (novox/hq ADR
// 0140). The mesh composes no filter for a machine that has not said, so a machine whose links
// changed — or which has only just learnt to say — has to say so without being asked. Left out,
// it could only speak when a declaration arrived, and a declaration cannot be composed until it
// has spoken: a machine waiting for a push that is waiting for the machine.
parts := []string{"firewall=" + r.Firewall, "outward=" + strings.Join(r.Outward, ",")}
for _, h := range r.Held {
parts = append(parts, "held "+h.ID+"="+h.Changed)
}
// And what filters the machine, with the found firewall's state (novox/hq ADR 0168): a rule the
// operator removes between declarations, or a front end enabled again, is said at the next
// reconcile rather than at the next push.
for _, f := range r.Filters {
parts = append(parts, "filter "+f.Owner+" "+f.Where+" "+f.Refuses)
}
if r.FoundFirewall != nil {
parts = append(parts, fmt.Sprintf("found-firewall %s active=%v retired-by=%s", r.FoundFirewall.Kind,
r.FoundFirewall.Active, r.FoundFirewall.RetiredBy))
}
for _, reach := range r.Reachable {
parts = append(parts, fmt.Sprintf("reach %s %s:%d %s %v %d", reach.Protocol, reach.Address,
reach.Port, reach.By, reach.Published, reach.ContainerPort))
}
sort.Strings(parts[1:])
sort.Strings(parts[2:])
return strings.Join(parts, "\n")
}
@@ -1152,7 +1280,17 @@ func holdTheMachine(ctx context.Context, opts options, mine identity.Identity, s
// A reconcile is otherwise silent. On an adopted node it speaks when what it holds or
// its firewall changed, because that is how a predecessor still writing is caught
// (novox/hq ADR 0100); publish decides whether anything did.
if publish != nil && report.Refused == "" && (len(report.Held) > 0 || report.Firewall != "") {
//
// **And on any node, when it can say which links face outside** (novox/hq ADR 0140). A
// converged node holds nothing and found no firewall, so this gate closed on it and the
// node could speak only in reply to a declaration — while the mesh composes no declaration
// for a node that has not said which links face outside. A machine waiting for a push that
// was waiting for the machine, and measured: three converged machines sat silent while the
// control plane refused to send them a filter.
//
// Offered, not published: whether it is news is still the watch's to decide, so an
// unchanged answer costs one comparison every reconcile and nothing on the bus.
if publish != nil && worthSaying(report) {
publish(report)
}
switch {
@@ -1164,6 +1302,24 @@ func holdTheMachine(ctx context.Context, opts options, mine identity.Identity, s
}
}
// worthSaying is whether a reconcile's report carries anything the mesh needs to hear unasked.
//
// **Named rather than written into the loop**, so the rule can be tested. It was a condition inline
// and it was wrong in a way nothing could catch: it asked only what an adopted node reports, so a
// converged node — which holds nothing and found no firewall — could speak only in reply to a
// declaration, while the mesh composes no declaration for a node that has not said which links face
// outside (novox/hq ADR 0140). Three machines sat silent waiting for a push that was waiting for them.
//
// A refused report says nothing about the machine, so it is not news; the refusal is said on the
// console where a person reads it.
func worthSaying(report link.Report) bool {
if report.Refused != "" {
return false
}
return len(report.Held) > 0 || report.Firewall != "" || len(report.Outward) > 0 ||
len(report.Filters) > 0 || report.FoundFirewall != nil
}
// applyDeclared applies a declaration that has already been proved to come from the mesh.
//
// Signature checking happens before this is called, in the link. By the time anything here runs,
@@ -1173,6 +1329,21 @@ func applyDeclared(ctx context.Context, opts options, raw []byte, sched *apply.S
return applyAndKeep(ctx, opts, raw, nil, sched, say)
}
// announceOr is what the apply writes its detail with, given what the caller has to say things with.
//
// **Never nil.** This argument was nil on the serving path, and nil is silence: everything the apply
// says — a file held, a container replaced, the found firewall retired — was visible when a person ran
// the one-shot command and discarded in the way the host actually runs (novox/hq 04-ISSUES/143).
//
// Named rather than written inline at the call site so both paths reach the apply the same way, and so
// a reader asking "where does the apply's output go" finds one answer.
func announceOr(say link.Announce) func(string) {
if say == nil {
return func(string) {}
}
return say
}
// applying serialises applies within this process.
//
// **Two things apply here: the link and the reconcile loop**, and each reads the node's state,
@@ -1235,8 +1406,18 @@ func applyAndKeep(ctx context.Context, opts options, raw []byte, signed *store.D
// Declared, not carried. A declaration from the mesh removes only what the mesh previously
// declared — never what this machine raised for itself from its bundle (04-ISSUES/010).
// **What the apply says goes to the console, which is the journal when this runs as a service.**
//
// It was nil, and nil is silence. The one-shot path has always passed a real one, so every detail
// the apply produces — a file held, a container replaced, the found firewall retired — was visible
// when a person ran it by hand and discarded in the way the host actually runs. Measured: after a
// machine was converged and its found firewall was not retired, what the host decided was
// unrecoverable, because it had said it to nobody (novox/hq 04-ISSUES/143).
//
// `say` already reaches stdout, and the launcher's unit sends that to the journal, so this needs
// no new mechanism — only for the argument to be passed.
outcome, updated, applyErr := apply.ApplyKeeping(ctx, built, declared, known, store.OriginDeclared,
apply.ExecRunner, nil, sealOpener(opts.state), apply.KeepIn(filepath.Dir(opts.state)))
apply.ExecRunner, announceOr(say), sealOpener(opts.state), apply.KeepIn(filepath.Dir(opts.state)))
// The mode the mesh said, recorded whichever way the apply went: the declaration is kept
// either way, and the node is held to it from the next reconcile (novox/hq ADR 0100).
@@ -1262,12 +1443,45 @@ func applyAndKeep(ctx context.Context, opts options, raw []byte, signed *store.D
sched.Sync(declared, held)
}
report := link.Report{Carried: carriedPorts(updated), Declared: digestOf(raw)}
report := link.Report{Carried: carriedPorts(updated), Declared: digestOf(raw), Host: runningVersion(),
Profile: profileAsReported(profile.Detect(ctx, profile.Default(nil), opts.timeout))}
// Which of this machine's links face outside, for the filter the mesh writes around them
// (novox/hq ADR 0140). Reported whatever the node's mode: a converged node's filter needs it,
// and an adopted one becomes converged without a further round trip. A machine that cannot read
// its own routing table says nothing rather than guessing, and is sent no filter.
if links, err := outward.Links("", ""); err != nil {
fmt.Fprintf(os.Stderr, "mesh-host: applied, and could not read which links face outside: %v\n", err)
} else {
report.Outward = links
}
// What this node found and holds, its firewall, and what is reachable on it — so an adopted
// node never reads as converged (novox/hq ADR 0100).
for _, h := range updated.Held {
report.Held = append(report.Held, link.Held{ID: h.ID, Module: h.Module, Kind: h.Kind,
Target: h.Target, Since: h.Since, Changed: h.Changed, Kept: h.Kept})
Target: h.Target, Since: h.Since, Changed: h.Changed, Kept: h.Kept, Facts: factsAsReported(h.Facts)})
}
// And what runs here that nobody asked for (novox/hq ADR 0163).
if strays, err := apply.Strays(ctx, apply.ExecRunner, updated); err != nil {
fmt.Fprintf(os.Stderr, "mesh-host: applied, and could not list what else runs here: %v\n", err)
} else {
for _, s := range strays {
report.Strays = append(report.Strays, link.Stray{Kind: s.Kind, Name: s.Name, Detail: s.Detail})
}
}
// What filters this machine, with owners, whatever its mode (novox/hq ADR 0168): the mesh says
// truthfully what filters a converged machine, and names what it did not write.
ufwActive := firewall.Active(ctx, apply.ExecRunner)
if filters, err := firewall.Collect(ctx, apply.ExecRunner, ufwActive); err != nil {
fmt.Fprintf(os.Stderr, "mesh-host: applied, and could not read what filters this machine: %v\n", err)
} else {
for _, f := range filters {
report.Filters = append(report.Filters, link.Filter{Where: f.Where, Owner: f.Owner, Refuses: f.Refuses})
}
}
if declared.Adoption == nil && updated.Firewall != nil && updated.Firewall.Kind == string(firewall.UFW) && updated.Firewall.WasActive {
// And, converged, the state of the firewall it was found with and who retired it.
report.FoundFirewall = &link.FoundFirewall{Kind: updated.Firewall.Kind, Active: ufwActive,
RetiredBy: updated.Firewall.RetiredBy}
}
if declared.Adoption != nil {
if updated.Firewall != nil {
@@ -1436,3 +1650,54 @@ func adoptDeliveredMembership(identityPath string, mine *identity.Identity, say
say(fmt.Sprintf("moving to the %s bus at %s — restarting to dial it", next.Transport, next.Broker))
os.Exit(0)
}
// refuseOlder refuses a declaration from the mesh that is older than the one this node holds.
//
// **By sequence, not by arrival** (novox/hq 04-ISSUES/107). What the host kept is the last thing
// the mesh said, signed; a declaration whose sequence is lower was composed before it, whatever
// order they arrived in — a backlog drained after the node was away, or a broker that split a burst.
// Applying it would make the machine into something the mesh had already moved past, which is the
// incident of issue 104 by another door.
//
// Only when both sides claim an order. A declaration with no sequence is one an older controller
// sent, and one kept with no sequence is one this host received before it understood them; in either
// case there is no order to compare, and refusing on a guess would strand the node the moment the
// controller is older than the host. Equal is the same declaration again, which reconciling is for.
func refuseOlder(kept store.Declared, keptErr error, sequence int64) error {
if sequence == 0 || keptErr != nil {
return nil
}
last, err := declaration.ParseTrusted(kept.Declaration)
if err != nil || last.Sequence == 0 {
return nil
}
if sequence < last.Sequence {
return fmt.Errorf("this declaration is older than what the mesh last said to this node: it "+
"is sequence %d, and the one kept here is %d. It arrived late — a backlog, or a broker "+
"that split a burst — and applying it would make this machine into something the mesh has "+
"already moved past. Refused whole; nothing was applied", sequence, last.Sequence)
}
return nil
}
// profileAsReported is the profile as the mesh reads it — the same bytes enrolment sends, so a
// report's profile and an enrolment's are one shape on the controller's side (novox/hq ADR 0161).
func profileAsReported(detected profile.Profile) map[string]any {
reported := map[string]any{}
if raw, err := json.Marshal(detected); err == nil {
_ = json.Unmarshal(raw, &reported)
}
return reported
}
// factsAsReported is a held thing's facts as the mesh reads them: the same bytes the host keeps.
func factsAsReported(f *store.Facts) map[string]any {
if f == nil {
return nil
}
out := map[string]any{}
if raw, err := json.Marshal(f); err == nil {
_ = json.Unmarshal(raw, &out)
}
return out
}
+108
View File
@@ -171,6 +171,22 @@ func TestAReconcileSpeaksOnlyWhenWhatIsHeldChanged(t *testing.T) {
if !w.changed(link.Report{Firewall: "none", Held: rewritten.Held}) {
t.Error("a changed firewall was not said")
}
// What filters the machine is part of it (novox/hq ADR 0168): a predecessor's chain removed by
// hand, or the found firewall enabled again, is said without being asked.
filtered := link.Report{Firewall: "none", Held: rewritten.Held,
Filters: []link.Filter{{Where: "chain HAL-MESH-ONLY (iptables-legacy)", Owner: "other", Refuses: "-j DROP"}}}
if !w.changed(filtered) {
t.Error("a filter appearing was not said")
}
if !w.changed(link.Report{Firewall: "none", Held: rewritten.Held}) {
t.Error("a filter removed was not said")
}
if !w.changed(link.Report{Firewall: "none", Held: rewritten.Held, FoundFirewall: &link.FoundFirewall{Kind: "ufw", Active: true}}) {
t.Error("the found firewall coming back was not said")
}
if !worthSaying(link.Report{Filters: filtered.Filters}) {
t.Error("a report carrying only what filters the machine is not worth saying")
}
}
func TestWhatTheLinkPublishedCountsAsSaid(t *testing.T) {
@@ -501,3 +517,95 @@ func TestReconcileAfterAControllerDeclarationDoesNotReapplyTheBundle(t *testing.
t.Errorf("with nothing said, reconcile did not reach for the carried bundle: %v", err)
}
}
// **A machine says which links face outside without being asked.**
//
// The mesh composes no filter for a machine that has not said (novox/hq ADR 0140), and a machine only
// speaks unasked when this fingerprint changes. Left out of it, a machine that has just learnt to say
// could speak only when a declaration arrived — and a declaration cannot be composed until it has
// spoken. A machine waiting for a push that is waiting for the machine.
func TestANewOutwardLinkIsSaidUnasked(t *testing.T) {
w := &adoptionWatch{}
first := link.Report{Firewall: "none"}
if !w.differs(first) {
t.Fatal("the first report should differ from nothing")
}
w.said(first)
// Only the links changed, and nothing about adoption.
learnt := link.Report{Firewall: "none", Outward: []string{"eth0"}}
if !w.differs(learnt) {
t.Fatal("a machine that has just learnt which links face outside would never say so, " +
"and could then never be sent a filter")
}
w.said(learnt)
if w.differs(link.Report{Firewall: "none", Outward: []string{"eth0"}}) {
t.Fatal("the same links are reported as a change, so the machine would speak on every reconcile")
}
// And a link that changes — a laptop moving from a cable to a radio — is said too, because the
// filter is written around the old one until it is.
if !w.differs(link.Report{Firewall: "none", Outward: []string{"wlan0"}}) {
t.Fatal("a changed outward link is not said, so the filter stays written around the old one")
}
}
// **A converged machine can say which links face outside, unasked.**
//
// A reconcile is otherwise silent, and the condition deciding when it speaks asked only what an
// adopted node reports — what it holds, and the firewall it found. A converged node has neither, so
// it could speak only in reply to a declaration, and the mesh composes no declaration for a node
// that has not said which links face outside (novox/hq ADR 0140). Measured: three converged machines
// sat silent while the control plane refused to send them a filter.
func TestAConvergedMachineSaysItsOutwardLinksUnasked(t *testing.T) {
// A converged node's reconcile: nothing held, no found firewall, and the links it can see.
converged := link.Report{Outward: []string{"eth0"}}
if !worthSaying(converged) {
t.Fatal("a converged machine cannot say which links face outside, so it can never be " +
"sent a filter — a machine waiting for a push that is waiting for the machine")
}
// An adopted node's reasons still hold, because that is how a predecessor still writing is caught.
if !worthSaying(link.Report{Firewall: "ufw"}) {
t.Fatal("an adopted machine no longer says which firewall it found")
}
if !worthSaying(link.Report{Held: []link.Held{{ID: "a-file"}}}) {
t.Fatal("an adopted machine no longer says what it holds")
}
// And a reconcile with nothing to say stays silent, or every machine speaks every five minutes
// about nothing.
if worthSaying(link.Report{}) {
t.Fatal("a reconcile with nothing to say speaks anyway")
}
// A refused report says nothing about the machine; the refusal is for the console.
if worthSaying(link.Report{Outward: []string{"eth0"}, Refused: "not for this node"}) {
t.Fatal("a refused report is offered as news about the machine")
}
}
// **A host running as a service says what its apply did.**
//
// The serving path passed nil where the apply writes its detail, and nil is silence. The one-shot path
// has always passed a real function, so everything the apply says was visible when a person ran it by
// hand and discarded in the way the host actually runs. Measured before this was written: a machine was
// converged, its found firewall was not retired, and what the host decided was unrecoverable because it
// had been said to nobody (novox/hq 04-ISSUES/143).
//
// This asserts only that the apply's log is never nil and that a line reaches what the caller gave.
// **It cannot catch the fault it was written for** — a call site passing nil directly — because that is
// wiring, and wiring is only proved by running the thing. That proof is a deployed host whose journal
// carries the apply's detail, which is how this fix was verified.
func TestTheApplysLogIsNeverNil(t *testing.T) {
if announceOr(nil) == nil {
t.Fatal("a host with nowhere to say things got a nil log, which the apply will call")
}
announceOr(nil)("this goes nowhere and must not panic")
var said []string
announceOr(func(line string) { said = append(said, line) })(" disabled ufw")
if len(said) != 1 || !strings.Contains(said[0], "disabled ufw") {
t.Fatalf("the apply's detail did not reach the caller's announce: %v", said)
}
}
+58
View File
@@ -0,0 +1,58 @@
package main
import (
"encoding/json"
"strings"
"testing"
"github.com/novox/mesh-host/internal/store"
)
// A declaration carries no order, so a host cannot tell an older one from a newer (novox/hq
// 04-ISSUES/107). Its only identity was the digest of its bytes: "not the last" could be said,
// "older" could not.
func keptWith(t *testing.T, sequence int64) store.Declared {
t.Helper()
body, err := json.Marshal(map[string]any{
"declaration": 1, "resources": []any{}, "owns_nothing": true, "sequence": sequence,
})
if err != nil {
t.Fatal(err)
}
return store.Declared{Declaration: body, Signature: []byte("x")}
}
func TestAnOlderDeclarationFromTheMeshIsRefused(t *testing.T) {
err := refuseOlder(keptWith(t, 7), nil, 5)
if err == nil {
t.Fatal("sequence 5 was accepted over a kept 7")
}
if !strings.Contains(err.Error(), "older") || !strings.Contains(err.Error(), "nothing was applied") {
t.Fatalf("the refusal does not say what it is: %v", err)
}
}
func TestANewerOrEqualDeclarationIsNot(t *testing.T) {
if err := refuseOlder(keptWith(t, 7), nil, 8); err != nil {
t.Fatalf("sequence 8 was refused over a kept 7: %v", err)
}
// Equal is the same declaration again, which reconciling is for.
if err := refuseOlder(keptWith(t, 7), nil, 7); err != nil {
t.Fatalf("the same sequence was refused: %v", err)
}
}
func TestNoOrderClaimedMeansNoOrderCompared(t *testing.T) {
// An older controller sends none; a host that received before it understood them kept none.
// Refusing on a guess would strand a node the moment the controller is older than the host.
if err := refuseOlder(keptWith(t, 7), nil, 0); err != nil {
t.Fatalf("a declaration claiming no order was refused: %v", err)
}
if err := refuseOlder(keptWith(t, 0), nil, 3); err != nil {
t.Fatalf("a declaration was refused against a kept one that claimed no order: %v", err)
}
if err := refuseOlder(store.Declared{}, store.ErrNothingDeclared, 3); err != nil {
t.Fatalf("a first declaration was refused: %v", err)
}
}
+48
View File
@@ -0,0 +1,48 @@
package main
import (
"os"
"path/filepath"
"testing"
)
// A component's version comes from where it sits, not from its linker (novox/hq ADR 0142). The mesh's
// toolchain stamps no version — a build does not know what it will be called — so a delivered host
// read as "development build" and the mesh could not tell which host a machine ran (04-ISSUES/161).
func TestADeliveredHostReadsItsVersionFromItsPath(t *testing.T) {
// A delivered host lives at <libexec>/versions/<version>/<binary>.
dir := t.TempDir()
versioned := filepath.Join(dir, "versions", "637f65559d16")
if err := os.MkdirAll(versioned, 0o755); err != nil {
t.Fatal(err)
}
self := filepath.Join(versioned, "nox-mesh-host")
if err := os.WriteFile(self, []byte("#!/bin/sh\n"), 0o755); err != nil {
t.Fatal(err)
}
if got := versionAt(self, "development build"); got != "637f65559d16" {
t.Fatalf("a delivered host read its version as %q", got)
}
}
func TestAHostPlacedByHandKeepsItsStamp(t *testing.T) {
// The honest answer for one the mesh did not deliver — and every machine is in that state until
// a delivery reaches it.
if got := versionAt("/usr/bin/nox-mesh-host", "04a27ca"); got != "04a27ca" {
t.Fatalf("a hand-placed host read its version as %q", got)
}
}
func TestADirectoryThatIsNotAVersionIsNotReadAsOne(t *testing.T) {
// A binary sitting anywhere else must not have its parent directory's name read as a version.
for _, path := range []string{
"/opt/somewhere/nox-mesh-host",
"/usr/lib/nox-mesh-host/launch",
"/home/someone/build/nox-mesh-host",
} {
if got := versionAt(path, "the stamp"); got != "the stamp" {
t.Fatalf("%s read its version as %q", path, got)
}
}
}
+16 -7
View File
@@ -127,12 +127,21 @@
{
"id": "bus-certificate",
"type": "action",
"command": ["docker", "run", "--rm", "--entrypoint", "sh", "-v", "mesh-broker-tls:/tls",
"192.0.2.250:5000/nats@sha256:b83efabe3e7def1e0a4a31ec6e078999bb17c80363f881df35edc70fcb6bb927",
"-c", "test -f /tls/tls.crt || (openssl req -x509 -newkey rsa:2048 -nodes -keyout /tls/tls.key -out /tls/tls.crt -days 3650 -subj '/CN=mesh-broker' -addext 'subjectAltName=DNS:mesh-broker,IP:127.0.0.1' >/dev/null 2>&1 && chmod 644 /tls/tls.crt && chmod 600 /tls/tls.key)"],
"verify": ["docker", "run", "--rm", "--entrypoint", "sh", "-v", "mesh-broker-tls:/tls",
"192.0.2.250:5000/nats@sha256:b83efabe3e7def1e0a4a31ec6e078999bb17c80363f881df35edc70fcb6bb927",
"-c", "test -s /tls/tls.crt && openssl x509 -in /tls/tls.crt -noout"]
// **The mesh makes its own** (novox/hq 04-ISSUES/146). This ran `openssl` inside the
// broker's image while the broker was one that carried it; the bus that replaced it has a
// shell and no openssl, and no other image the bundle names has one either. So the program
// that needs the certificate writes it — already on this machine, since the schema step ran
// it, and asking nothing of the image it writes into. Self-signed on purpose: a host pins
// this server's exact certificate (novox/hq ADR 0004), and at this moment there is no mesh
// to ask an authority of.
// `--user 0:0` because the volume is root's and this image runs as nobody, which is right
// for the long-running control plane and wrong for a one-shot writing into a fresh volume.
"command": ["docker", "run", "--rm", "--user", "0:0", "-v", "mesh-broker-tls:/tls",
"192.0.2.250:5000/mesh-controller@sha256:c67db38439ff0aee242b467486765467bb95801f52175fc5727cc4e437338ace",
"broker", "certificate", "--into", "/tls"],
"verify": ["docker", "run", "--rm", "--user", "0:0", "-v", "mesh-broker-tls:/tls",
"192.0.2.250:5000/mesh-controller@sha256:c67db38439ff0aee242b467486765467bb95801f52175fc5727cc4e437338ace",
"broker", "certificate", "--check", "--into", "/tls"]
},
{
"id": "bus-conf-dir",
@@ -152,7 +161,7 @@
"type": "file",
"path": "/var/lib/mesh-bus-conf/accounts.conf",
"mode": "0600",
"content": "// The first user list, carried by the installer because at genesis there is no mesh to\n// compose one. A bootstrap credential, rotated with the store's and replaced by the\n// controller's own composition from its first start onward.\naccounts {\n MESH {\n jetstream: enabled\n users = [\n { user: \"controller\", password: \"$2a$10$AHqJgOifIVbU41KmATiMhuXFs8xa7Wl2HuN4UVBCXdN2jIQzjqApy\", permissions: {\n publish: { allow: [\"$JS.API.>\", \"$JS.ACK.CONTROL.controller.>\", \"$JS.ACK.EVENTS.controller.>\", \"_INBOX.enrol.>\", \"mesh.control.>\", \"mesh.mod.*.tool.>\", \"mesh.node.>\", \"mesh.seat.mesh-build-machine.accept.>\"] }\n subscribe: { allow: [\"$JS.API.>\", \"_DELIVER.controller\", \"_DELIVER.controller.>\", \"_INBOX.controller.>\", \"mesh.control.>\", \"mesh.mod.mesh-catalog.event.catching-up\", \"mesh.mod.mesh-catalog.event.upgraded\", \"mesh.mod.gitea.event.pull.merged\", \"mesh.seat.mesh-build-machine.event.built\"] }\n allow_responses: { max: 1, ttl: \"1m\" }\n } }\n ]\n }\n}\n"
"content": "// The first user list, carried by the installer because at genesis there is no mesh to\n// compose one. A bootstrap credential, rotated with the store's and replaced by the\n// controller's own composition from its first start onward.\naccounts {\n MESH {\n jetstream: enabled\n users = [\n { user: \"controller\", password: \"$2a$10$AHqJgOifIVbU41KmATiMhuXFs8xa7Wl2HuN4UVBCXdN2jIQzjqApy\", permissions: {\n publish: { allow: [\"$JS.ACK.CONTROL.controller.>\", \"$JS.ACK.EVENTS.controller.>\", \"$JS.API.>\", \"_INBOX.enrol.>\", \"mesh.assignment.>\", \"mesh.control.>\", \"mesh.mod.*.tool.>\", \"mesh.node.>\", \"mesh.seat.mesh-build-machine.accept.>\", \"mesh.seat.mesh-controller.event.applied\", \"mesh.seat.mesh-controller.event.built-before\", \"mesh.seat.mesh-controller.event.refused\"] }\n subscribe: { allow: [\"$JS.API.>\", \"_DELIVER.controller\", \"_DELIVER.controller.>\", \"_INBOX.controller.>\", \"mesh.control.>\", \"mesh.mod.gitea.event.pull.merged\", \"mesh.mod.mesh-catalog.event.catching-up\", \"mesh.mod.mesh-catalog.event.upgraded\", \"mesh.seat.mesh-build-machine.event.built\", \"mesh.seat.mesh-controller.tool.>\"] }\n allow_responses: { max: 1, ttl: \"1m\" }\n } }\n ]\n }\n}\n"
},
{
"id": "broker",
+175 -4
View File
@@ -20,12 +20,14 @@ import (
"os"
"os/exec"
"path/filepath"
"slices"
"sort"
"strconv"
"strings"
"time"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/firewall"
"github.com/novox/mesh-host/internal/store"
"github.com/novox/mesh-host/internal/system"
)
@@ -66,6 +68,10 @@ type Outcome struct {
// Report is what an apply did, in the order it did it.
type Report struct {
Outcomes []Outcome `json:"outcomes"`
// Firewall is what this apply did about the firewall a converged machine was found with, when
// it did or declined anything: retired, retired again, or left in force and why (novox/hq ADR
// 0168). Said rather than an outcome: the plan says the same step the same way.
Firewall string `json:"firewall,omitempty"`
// Tunnel is what this apply says about the tunnel the private network took over, when the
// declaration names one (novox/hq ADR 0105).
Tunnel *TakenTunnel `json:"tunnel,omitempty"`
@@ -198,6 +204,22 @@ func ApplyKeeping(
} else {
action, detail, err = remove(ctx, sys, orphan, run, made)
}
if errors.Is(err, errNoRemoval) && store.IsFormer(orphan.ID) {
// **A former target of a kind the host cannot remove is left in place and forgotten,
// never fatal.** The host's own archive is the case: every version it delivers itself
// has a new target, so the one before is a former target on the first apply of the new
// host — and a removal that refused there stopped every machine applying anything, the
// moment the host that carried former targets (novox/hq ADR 0163, rule 5) first
// replaced itself. What was written stays where it is, said, and the record no longer
// names it; whether an archive gets a removal is issue 162's question, not this apply's.
known.Forget(orphan.ID)
report.Outcomes = append(report.Outcomes, Outcome{
ID: orphan.ID, Type: orphan.Type, Target: orphan.Target,
Action: "forgotten", Detail: "a former target left in place: " + err.Error() + " (novox/hq issue 162)",
})
log(fmt.Sprintf(" forgotten %s (%s): a former target left in place: %v", orphan.ID, orphan.Target, err))
return nil
}
if err != nil {
return &Error{Resource: orphan.ID, Err: err, Done: report}
}
@@ -230,6 +252,18 @@ func ApplyKeeping(
protecting = append(protecting, orphan)
continue
}
// **A module the mesh left out is not a module the mesh removed** (novox/hq ADR 0163, rule
// 6): its resources are absent because a setting stored for it cannot compose, and the
// mesh said so by name. What the host wrote for it stays as it is, recorded, until the
// module is declared again or unassigned.
if module, left := d.LeftOutModuleOf(orphan.ID); left {
report.Outcomes = append(report.Outcomes, Outcome{
ID: orphan.ID, Type: orphan.Type, Target: orphan.Target,
Action: "unchanged", Detail: "kept: " + module + " was left out of this declaration by the mesh, not removed",
})
log(fmt.Sprintf(" kept %s (%s): %s was left out of this declaration by the mesh, not removed", orphan.ID, orphan.Target, module))
continue
}
orphans = append(orphans, orphan)
}
ordered := d.Resources
@@ -270,6 +304,11 @@ func ApplyKeeping(
if declared[h.ID] {
continue
}
if slices.Contains(d.LeftOut, h.Module) {
// Left out, not unassigned (ADR 0163, rule 6): still held for the module, as the
// mesh asked.
continue
}
known.Release(h.ID)
report.Outcomes = append(report.Outcomes, Outcome{ID: h.ID, Type: h.Kind, Target: h.Target,
Action: "forgotten", Detail: "no longer declared; left as found"})
@@ -320,6 +359,9 @@ func ApplyKeeping(
// is a different thing — one is "this machine could not do it", the other is "this was never
// a declaration", and they are fixed in different places.
var failures []*Error
// Modules whose own step did not complete. What follows *within such a module* is not attempted;
// the rest of the machine is (novox/hq ADR 0136).
gated := map[string]bool{}
for i, resource := range ordered {
if !orphansRemoved && i == guardFirst {
// **Only a guard that is up may let the filter go.** Removing the derived filter's
@@ -337,6 +379,17 @@ func ApplyKeeping(
return report, known, err
}
}
// **A module whose step did not complete is skipped from there on** (novox/hq ADR 0136).
// Reported rather than passed over in silence: "not attempted" and "nothing to do" are
// different answers, and only one of them is somebody's to fix.
if module, ours := moduleOf(resource.Identity()); ours && gated[module] {
report.Outcomes = append(report.Outcomes, Outcome{
ID: resource.Identity(), Type: string(resource.Kind()), Target: resource.Target(),
Action: "skipped", Detail: "a step this module declares did not complete",
})
continue
}
// **On an adopted node, what is found is kept until its module is taken** (novox/hq ADR
// 0100, ADR 0103). Before anything is applied: whatever of a module not yet taken is
// present with no record of this host making it — or would reach what is — is held as it
@@ -465,9 +518,26 @@ func ApplyKeeping(
gates = true
}
if gates {
failed.Gated = true
// **A module's step gates that module, not the machine** (novox/hq ADR 0136).
//
// Stopping the whole apply is what this loop's own comment above calls holding a
// machine hostage, and it was already rejected for every other shape (04-ISSUES/011).
// A step exists to make something true before the next thing in *its module* needs it
// — a store seeded before the broker starts, a schema prepared before the version
// that needs it runs — so that is exactly how far the gate reaches. Everything else
// on the machine is independent state and is attempted.
//
// An action still gates the machine: the bootstrap is a row of them, each making the
// next possible, and they belong to no module.
if module, ours := moduleOf(resource.Identity()); ours {
gated[module] = true
log(fmt.Sprintf(" gated %s.*: a step it declares did not complete, so the rest "+
"of it was not attempted", module))
continue
}
failed.Done = report
failed.Others = len(failures) - 1
failed.Gated = true
return report, known, failed
}
continue
@@ -546,16 +616,24 @@ func ApplyKeeping(
}
// A converged node whose found firewall was in force retires it only now, once everything —
// the mesh's derived filter among it — applied cleanly (novox/hq ADR 0100).
// the mesh's derived filter among it — applied cleanly (novox/hq ADR 0100), and on every
// converged apply, not once (ADR 0168). Skipped, it is said: a step that does nothing is never
// silent (issue 143).
if len(failures) == 0 {
if err := retireFirewall(ctx, d, origin, &known, run, log); err != nil {
did, err := retireFirewall(ctx, d, origin, &known, run, log)
if err != nil {
return report, known, &Error{Resource: "the firewall found on this machine", Err: err, Done: report}
}
report.Firewall = did
for _, orphan := range protecting {
if err := removeOrphan(orphan); err != nil {
return report, known, err
}
}
} else if rec := known.Firewall; origin == store.OriginDeclared && d.Adoption == nil && rec != nil &&
rec.Kind == string(firewall.UFW) && rec.WasActive && firewall.Active(ctx, run) {
report.Firewall = fmt.Sprintf("left in force: %d resource(s) failed, and the found firewall is retired only after a clean apply", len(failures))
log(" kept ufw in force: " + report.Firewall)
}
if len(failures) > 0 {
@@ -1291,10 +1369,15 @@ func remove(ctx context.Context, sys system.System, a store.Applied, run Runner,
return "removed", "no longer declared", nil
default:
return "", "", fmt.Errorf("no way to remove a %q", a.Type)
return "", "", fmt.Errorf("%w: a %q", errNoRemoval, a.Type)
}
}
// errNoRemoval is remove's answer for a kind the host has no removal for (novox/hq issue 162): an
// archive, among others. Fatal for an orphan the declaration dropped, so an unassignment nothing can
// undo is never reported as done; not fatal for a former target, which was never dropped by anyone.
var errNoRemoval = errors.New("no way to remove")
// ExecRunner runs a real command, with stdin closed and output captured.
func ExecRunner(ctx context.Context, name string, args ...string) (string, error) {
cmd := exec.CommandContext(ctx, name, args...)
@@ -1324,6 +1407,25 @@ func applyPackage(ctx context.Context, sys system.System, r *declaration.Package
if err != nil {
return out, err
}
if r.Absent {
// Declared absent (novox/hq ADR 0175): removed when it is here, left alone when it is not.
if !installed {
out.Action = "unchanged"
out.Detail = "not installed, as declared"
return out, nil
}
if err := sys.RemovePackage(ctx, run, r.Package); err != nil {
return out, fmt.Errorf("removing %s: %w", r.Package, err)
}
if still, err := sys.PackageInstalled(ctx, run, r.Package); err != nil {
return out, err
} else if still {
return out, fmt.Errorf("%s was removed without error and the package database still has it", r.Package)
}
out.Action = "removed"
out.Detail = "declared absent; its configuration is left where the package manager leaves it"
return out, nil
}
if installed {
out.Action = "unchanged"
out.Detail = "already installed"
@@ -1470,6 +1572,22 @@ func containerSpecReading(r *declaration.Container, declares, reads map[string]s
for _, a := range r.Args {
b.WriteString("arg " + a + "\n")
}
// **A container's declared names are part of what it is** (novox/hq 04-ISSUES/135). What is
// here is what the module declared for itself and nothing else: the mesh's own names are no
// longer written into a container (ADR 0148) — they were once, every container got the whole
// roster at creation and nothing re-read it, so one left alone when the roster moved could not
// reach anything by name for as long as it ran while every check reported it running; and once
// the roster was in this digest so that could be caught, one name moving anywhere replaced
// every container in the mesh (04-ISSUES/151). A container resolves a mesh name through the
// machine's resolver at the moment it asks. What a module declares does not move when the
// roster does, so hashing it costs nothing and catches a manifest that changed.
//
// Sorted, so the digest does not move for a reordering nobody made.
hosts := append([]string(nil), r.Hosts...)
sort.Strings(hosts)
for _, h := range hosts {
b.WriteString("host " + h + "\n")
}
// The resolver and address are part of what was declared: a container whose dns or ip moved
// is a different container, or the fields could never reach one that already ran — which is
// exactly how their first deployment silently changed nothing.
@@ -1479,6 +1597,20 @@ func containerSpecReading(r *declaration.Container, declares, reads map[string]s
if r.IP != "" {
b.WriteString("ip " + r.IP + "\n")
}
// The networks it also joins are part of what it is (ADR 0163, rule 4): kept or let go, the
// container is recreated, and a neighbour's reach changes with it.
for _, n := range r.Networks {
b.WriteString("also-on " + n + "\n")
}
// And the capabilities it was granted (ADR 0170): one gained or dropped is a different
// container, and the runtime cannot change a running one's.
for _, c := range r.Capabilities {
b.WriteString("cap " + c + "\n")
}
// And where it logs (ADR 0179): the runtime cannot move a running container's output.
if r.Logging != "" {
b.WriteString("log " + r.Logging + "\n")
}
// The cadence is part of what was declared, so a changed schedule is a changed spec — the marker
// moves and the install is reported "updated" and re-established. Added only when present, so no
// ordinary container's or run-once step's digest moves for a field it does not set.
@@ -1673,6 +1805,14 @@ func applyContainer(ctx context.Context, r *declaration.Container, run Runner,
if r.Network != "" {
args = append(args, "--network", r.Network)
}
for _, c := range r.Capabilities {
args = append(args, "--cap-add", c)
}
if r.Logging != "" {
// The journal keeps the container's name on every line (CONTAINER_NAME), which is what a
// jail matches on (novox/hq ADR 0179); `docker logs` keeps working against the journal.
args = append(args, "--log-driver", r.Logging)
}
for _, d := range r.Dns {
args = append(args, "--dns", d)
}
@@ -1719,6 +1859,15 @@ func applyContainer(ctx context.Context, r *declaration.Container, run Runner,
if after.Spec != want {
return out, fmt.Errorf("container %s is not the one that was declared after creating it", r.Name)
}
// The found networks a per-machine setting keeps for it (novox/hq ADR 0163, rule 4), joined
// once it runs: a runtime starts a container on one network, and the others are connected.
// Refused, not skipped, when one cannot be joined — a neighbour that was promised to keep
// reaching this container by name would silently not.
for _, n := range r.Networks {
if _, err := run(ctx, cri, "network", "connect", n, r.Name); err != nil {
return out, fmt.Errorf("container %s could not join the kept network %s: %w", r.Name, n, err)
}
}
out.Action = "created"
if existed {
@@ -2247,3 +2396,25 @@ func meshMadeUnits(known store.State) map[string]bool {
}
return made
}
// moduleOf is the module a declared resource belongs to.
//
// The mesh composes a module's resource ids as `<module>.<its own id>`, and **a module's name may
// contain a dot** — `novox.be` is one on this mesh — while a resource's own id never does. So the
// owner is everything before the *last* dot; reading to the first one would make `novox.be.server`
// belong to a module called "novox", and a gate would then skip whatever else happened to start that
// way.
//
// False for what the mesh declares in its own right: the foundation's resources carry no dot at all,
// and the adoption's are named for the mesh rather than for a module. Both belong to no module, and
// their gate is therefore the machine's.
func moduleOf(identity string) (string, bool) {
if strings.HasPrefix(identity, declaration.AdoptionPrefix) {
return "", false
}
at := strings.LastIndex(identity, ".")
if at <= 0 {
return "", false
}
return identity[:at], true
}
+97
View File
@@ -0,0 +1,97 @@
package apply
import (
"context"
"os"
"path/filepath"
"strings"
"testing"
"github.com/novox/mesh-host/internal/store"
)
// A take is a comparison (novox/hq ADR 0163): while a module's container is held, the host reports
// the found image and its age beside the declared one, the networks and who else is on them, the
// mounts and the ports — and says when the declared image is the older.
func TestAHeldContainerCarriesTheFactsATakeCompares(t *testing.T) {
dir, page, m := predecessor(t)
m.containers["hello-web"].image = "web:1.27"
m.containers["hello-web"].imageID = "sha256:found"
m.containers["hello-web"].networks = []string{"predecessor_default"}
m.containers["hello-web"].mounts = []string{"/srv/web:/data"}
m.containers["hello-web"].ports = []string{"80/tcp>0.0.0.0:8080"}
m.images = map[string]string{"sha256:found": "2026-09-17T10:00:00Z", pinned: "2026-08-20T10:00:00Z"}
m.members = map[string][]string{"predecessor_default": {"hello-web", "office", "db"}}
_, state := applyAdopted(t, adopted(t, untakenWeb, webResources(page)), store.State{}, m, dir)
h, ok := state.HeldAt("hello-web.server")
if !ok || h.Facts == nil {
t.Fatalf("a held container carries no facts: %+v", h)
}
f := h.Facts
if f.Image != "web:1.27" || f.ImageCreated != "2026-09-17T10:00:00Z" {
t.Errorf("the found image and its age: %+v", f)
}
if f.DeclaredImage != pinned || f.DeclaredImageCreated != "2026-08-20T10:00:00Z" || !f.Downgrade {
t.Errorf("the declared image, its age, and that it is a downgrade: %+v", f)
}
if got := f.Networks["predecessor_default"]; len(got) != 2 || got[0] != "db" || got[1] != "office" {
t.Errorf("the neighbours on the found network, without the container itself: %v", f.Networks)
}
if len(f.Mounts) != 1 || f.Mounts[0] != "/srv/web:/data" || len(f.Ports) != 1 || f.Ports[0] != "80/tcp>0.0.0.0:8080" {
t.Errorf("mounts and ports as found: %+v", f)
}
// And the held file carries how the declared content differs from what was found.
p, ok := state.HeldAt("hello-web.page")
if !ok || p.Facts == nil || !p.Facts.Differs {
t.Fatalf("a held file that differs from the declared content does not say so: %+v", p)
}
joined := strings.Join(p.Facts.Difference, "\n")
if !strings.Contains(joined, "- the predecessor's page") || !strings.Contains(joined, "+ the mesh's page") {
t.Errorf("the difference does not show what is lost and what is new: %q", joined)
}
_ = os.Remove(filepath.Join(dir, "unused"))
}
// A declared image not yet on the machine leaves its age unknown and the comparison undecided.
func TestAnImageNotYetPulledLeavesTheDowngradeUndecided(t *testing.T) {
dir, page, m := predecessor(t)
m.containers["hello-web"].image = "web:1.27"
m.containers["hello-web"].imageID = "sha256:found"
m.images = map[string]string{"sha256:found": "2026-09-17T10:00:00Z"}
_, state := applyAdopted(t, adopted(t, untakenWeb, webResources(page)), store.State{}, m, dir)
h, _ := state.HeldAt("hello-web.server")
if h.Facts == nil || h.Facts.DeclaredImageCreated != "" || h.Facts.Downgrade {
t.Fatalf("an unknown declared age decided a downgrade: %+v", h.Facts)
}
}
func TestTheDifferenceIsWhatIsLostAndWhatIsNew(t *testing.T) {
differs, lines := differenceOf("a\nprivate scope: local\nb\n", "a\nb\nupstream: public\n")
if !differs || len(lines) != 2 || lines[0] != "- private scope: local" || lines[1] != "+ upstream: public" {
t.Fatalf("got %v %v", differs, lines)
}
if differs, lines := differenceOf("same\n", "same\n"); differs || lines != nil {
t.Fatalf("identical content differs: %v %v", differs, lines)
}
}
// What runs on the machine that the mesh neither wrote nor holds is reported (ADR 0163).
func TestStraysAreWhatRunsHereThatNobodyAsked(t *testing.T) {
m := &machine{containers: map[string]*fakeContainer{
"hello-web": {id: "ours", running: true, image: "web:1"},
"gitea-old": {id: "left-behind", running: true, image: "gitea:1.22"},
"held-thing": {id: "found", running: true, image: "x:1"},
}}
known := store.State{
Resources: []store.Applied{{ID: "hello-web.server", Type: "container", Target: "hello-web"}},
Held: []store.Held{{ID: "other.server", Kind: "container", Target: "held-thing"}},
}
strays, err := Strays(context.Background(), m.run, known)
if err != nil {
t.Fatal(err)
}
if len(strays) != 1 || strays[0].Name != "gitea-old" || !strings.Contains(strays[0].Detail, "gitea:1.22") {
t.Fatalf("strays: %+v", strays)
}
}
+75
View File
@@ -0,0 +1,75 @@
package apply
import (
"context"
"strings"
"testing"
"github.com/novox/mesh-host/internal/store"
)
// The host's own former archive stops nothing (novox/hq issue 194). A new host's first apply finds
// the version before it as a former target of the archive that delivered it; an archive has no
// removal (issue 162), and the refusal stopped every machine applying anything. A former target of
// such a kind is left in place, said, and forgotten. An archive the declaration dropped still fails,
// as 162 has it.
func TestTheHostsOwnFormerArchiveIsLeftInPlaceNotFatal(t *testing.T) {
run := func(_ context.Context, name string, args ...string) (string, error) {
if name == "docker" && args[0] == "info" {
return "29.0.0\n", nil
}
return "", nil
}
dir := t.TempDir()
former := store.FormerID("mesh-host.next", dir+"/versions/old")
known := store.State{Resources: []store.Applied{
{ID: "mesh-host.next", Type: "archive", Target: dir + "/versions/new", Origin: store.OriginDeclared},
{ID: former, Type: "archive", Target: dir + "/versions/old", Origin: store.OriginDeclared},
}}
body, digest := anArchive(t, map[string]string{"nox-mesh-host": "#!/bin/sh\n"})
d := parse(t, `{"declaration":1,"resources":[
{"id":"mesh-host.next","type":"archive","path":"`+dir+`/versions/new","source":"`+serving(t, body)+`","digest":"`+digest+`"},
{"id":"notes.conf","type":"file","path":"`+dir+`/notes.conf","content":"x"}
]}`)
report, state, err := Apply(context.Background(), archHost(t), d, known, store.OriginDeclared, run, nil, nil)
if err != nil {
t.Fatalf("the apply failed: %v", err)
}
if _, still := state.HeldAt(former); still {
t.Fatal("held?")
}
for _, r := range state.Resources {
if r.ID == former {
t.Fatal("the former archive is still on record")
}
}
said := false
for _, o := range report.Outcomes {
if o.ID == former && o.Action == "forgotten" && strings.Contains(o.Detail, "left in place") {
said = true
}
}
if !said {
t.Fatalf("leaving the former archive was not said: %+v", report.Outcomes)
}
applied := false
for _, o := range report.Outcomes {
if o.ID == "notes.conf" && o.Action == "created" {
applied = true
}
}
if !applied {
t.Fatalf("the rest of the declaration was not applied: %+v", report.Outcomes)
}
// An archive the declaration dropped is a different matter: nothing can undo it, and saying
// it was would report an effect the host declined to have (issue 162).
dropped := store.State{Resources: []store.Applied{
{ID: "tool.next", Type: "archive", Target: "/usr/lib/tool/versions/old", Origin: store.OriginDeclared},
}}
only := parse(t, `{"declaration":1,"resources":[{"id":"notes.conf","type":"file","path":"`+dir+`/notes.conf","content":"x"}]}`)
if _, _, err := Apply(context.Background(), archHost(t), only, dropped, store.OriginDeclared, run, nil, nil); err == nil ||
!strings.Contains(err.Error(), "no way to remove") {
t.Fatalf("a dropped archive was passed over: %v", err)
}
}
+125 -4
View File
@@ -8,6 +8,7 @@ import (
"fmt"
"os"
"path/filepath"
"sort"
"strings"
"syscall"
"time"
@@ -433,6 +434,32 @@ type foundContainer struct {
id string
running bool
spec string
// What a take compares (novox/hq ADR 0163): the image and its id, the networks the container
// is on, its mounts and its published ports — empty from a runtime (or a test's fake) that
// answers the short form.
image string
imageID string
networks []string
mounts []string
ports []string
}
// foundFormat is what inspectFound asks the runtime for, tab-separated: the three a hold has
// always needed, then the facts a take compares.
const foundFormat = "{{.Id}}\t{{.State.Running}}\t{{index .Config.Labels \"" + specLabel + "\"}}" +
"\t{{.Config.Image}}\t{{.Image}}" +
"\t{{range $k, $v := .NetworkSettings.Networks}}{{$k}},{{end}}" +
"\t{{range .Mounts}}{{.Source}}:{{.Destination}},{{end}}" +
"\t{{range $p, $b := .NetworkSettings.Ports}}{{$p}}{{range $b}}>{{.HostIp}}:{{.HostPort}}{{end}},{{end}}"
func splitList(s string) []string {
var out []string
for _, part := range strings.Split(s, ",") {
if part = strings.TrimSpace(part); part != "" {
out = append(out, part)
}
}
return out
}
// inspectFound reads a container by name the way a hold needs it: its id, whether it runs, and
@@ -451,8 +478,7 @@ func inspectFound(ctx context.Context, name string, run Runner) (foundContainer,
if err != nil {
return foundContainer{}, false, fmt.Errorf("%w, so nothing can be said about %q", err, name)
}
out, err := run(ctx, cri, "container", "inspect", "--format",
"{{.Id}}\t{{.State.Running}}\t{{index .Config.Labels \""+specLabel+"\"}}", name)
out, err := run(ctx, cri, "container", "inspect", "--format", foundFormat, name)
if err != nil {
if absent(err) {
return foundContainer{}, false, nil
@@ -466,14 +492,100 @@ func inspectFound(ctx context.Context, name string, run Runner) (foundContainer,
name, err)
}
parts := strings.Split(strings.TrimSpace(out), "\t")
for len(parts) < 3 {
for len(parts) < 8 {
parts = append(parts, "")
}
spec := strings.TrimSpace(parts[2])
if spec == "<no value>" {
spec = ""
}
return foundContainer{id: strings.TrimSpace(parts[0]), running: parts[1] == "true", spec: spec}, true, nil
return foundContainer{id: strings.TrimSpace(parts[0]), running: parts[1] == "true", spec: spec,
image: strings.TrimSpace(parts[3]), imageID: strings.TrimSpace(parts[4]),
networks: splitList(parts[5]), mounts: splitList(parts[6]), ports: splitList(parts[7])}, true, nil
}
// factsOf is what a take would compare for a found container (novox/hq ADR 0163): the found
// image and when it was made, the networks and who else is on them, mounts and ports — beside
// what the module declares, and the declared image's date when that image is on the machine.
// Every question the runtime cannot answer leaves its fact empty; a preview says so rather than
// guesses.
func factsOf(ctx context.Context, seen foundContainer, res *declaration.Container, run Runner) *Facts {
cri, err := containerRuntime(ctx, run)
if err != nil {
return nil
}
f := &store.Facts{Image: seen.image, Mounts: seen.mounts, Ports: seen.ports,
DeclaredImage: res.Image, DeclaredPorts: res.Ports, DeclaredVolumes: res.Volumes}
if seen.imageID != "" {
if out, err := run(ctx, cri, "image", "inspect", "--format", "{{.Created}}", seen.imageID); err == nil {
f.ImageCreated = strings.TrimSpace(out)
}
}
if res.Image != "" {
if out, err := run(ctx, cri, "image", "inspect", "--format", "{{.Created}}", res.Image); err == nil {
f.DeclaredImageCreated = strings.TrimSpace(out)
}
}
if found, err := time.Parse(time.RFC3339Nano, f.ImageCreated); err == nil {
if declared, err := time.Parse(time.RFC3339Nano, f.DeclaredImageCreated); err == nil {
f.Downgrade = declared.Before(found)
}
}
for _, network := range seen.networks {
if f.Networks == nil {
f.Networks = map[string][]string{}
}
var members []string
if out, err := run(ctx, cri, "network", "inspect", "--format",
"{{range .Containers}}{{.Name}},{{end}}", network); err == nil {
for _, m := range splitList(out) {
if m != res.Name {
members = append(members, m)
}
}
}
sort.Strings(members)
f.Networks[network] = members
}
return (*Facts)(f)
}
// Facts is store.Facts, named here so hold's callers read as one vocabulary.
type Facts = store.Facts
// differenceOf is how a found file differs from the declared content: the lines only the found
// file has, marked -, then the lines only the declared content has, marked +, in their own order,
// bounded so a report stays a report. Not a diff tool's output: the question a take answers is
// "what would be lost and what would be new", and that is these two lists.
func differenceOf(found, declared string) (bool, []string) {
if found == declared {
return false, nil
}
const bound = 40
count := func(s string) map[string]int {
out := map[string]int{}
for _, line := range strings.Split(s, "\n") {
out[line]++
}
return out
}
inFound, inDeclared := count(found), count(declared)
var out []string
add := func(mark, s string, other map[string]int) {
seen := map[string]int{}
for _, line := range strings.Split(s, "\n") {
seen[line]++
if seen[line] > other[line] && len(out) < bound {
out = append(out, mark+" "+line)
}
}
}
add("-", found, inDeclared)
add("+", declared, inFound)
if len(out) >= bound {
out = append(out, "… and more")
}
return true, out
}
// absent is whether a runtime said the thing is not there, rather than failing to answer. Its own
@@ -540,6 +652,12 @@ func hold(ctx context.Context, sys system.System, r declaration.Resource, module
} else if digestOf(string(content)) != h.Digest {
changed = "rewritten"
}
// What a take would replace it with, and how that differs (novox/hq ADR 0163): a
// file declared whole is compared whole; one written into is not replaced at all.
if res.Into == "" {
differs, lines := differenceOf(string(content), res.Content)
h.Facts = &Facts{Differs: differs, Difference: lines}
}
}
case *declaration.Directory:
info, err := os.Lstat(res.Path)
@@ -628,6 +746,9 @@ func hold(ctx context.Context, sys system.System, r declaration.Resource, module
case h.Running && !seen.running:
changed = "stopped"
}
if exists {
h.Facts = factsOf(ctx, seen, res, run)
}
default:
return out, h, fmt.Errorf("a %s cannot be held", r.Kind())
}
+38 -1
View File
@@ -5,6 +5,7 @@ import (
"errors"
"os"
"path/filepath"
"sort"
"strings"
"testing"
@@ -18,6 +19,10 @@ import (
// label when a host made it. Every command it is asked is written down.
type machine struct {
containers map[string]*fakeContainer
// images is what `image inspect --format {{.Created}}` answers per image or id; members is
// what `network inspect` lists per network (ADR 0163).
images map[string]string
members map[string][]string
asked []string
// wgUp is what `wg show interfaces` answers: the tunnels up on the machine.
wgUp string
@@ -97,6 +102,9 @@ type fakeContainer struct {
id string
running bool
spec string
// What a take compares (ADR 0163), answered in the long inspect form when set.
image, imageID string
networks, mounts, ports []string
}
func (m *machine) run(_ context.Context, name string, args ...string) (string, error) {
@@ -140,9 +148,38 @@ func (m *machine) run(_ context.Context, name string, args ...string) (string, e
running = "true"
}
if strings.HasPrefix(args[3], "{{.Id}}") {
return c.id + "\t" + running + "\t" + c.spec + "\n", nil
line := c.id + "\t" + running + "\t" + c.spec
if c.image != "" {
line += "\t" + c.image + "\t" + c.imageID + "\t" + strings.Join(c.networks, ",") + "," +
"\t" + strings.Join(c.mounts, ",") + "," + "\t" + strings.Join(c.ports, ",") + ","
}
return line + "\n", nil
}
return running + "\t" + c.spec + "\n", nil
case "image":
if len(args) > 1 && args[1] == "inspect" {
if created, ok := m.images[args[len(args)-1]]; ok {
return created + "\n", nil
}
return "", errors.New("no such image")
}
return "", nil
case "network":
if len(args) > 1 && args[1] == "inspect" {
return strings.Join(m.members[args[len(args)-1]], ",") + ",\n", nil
}
return "", nil
case "ps":
var lines []string
for name, c := range m.containers {
state := "exited"
if c.running {
state = "running"
}
lines = append(lines, name+"\t"+c.image+"\t"+state)
}
sort.Strings(lines)
return strings.Join(lines, "\n") + "\n", nil
case "rm":
delete(m.containers, args[len(args)-1])
return "", nil
+214
View File
@@ -0,0 +1,214 @@
package apply
import (
"context"
"errors"
"strings"
"testing"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/store"
)
// A taken container keeps a found network by a per-machine setting (novox/hq ADR 0163, rule 4):
// joined once it runs, part of its spec, and refused when it cannot be joined.
func TestAContainerJoinsTheNetworksItKeeps(t *testing.T) {
var ran []string
connectFails := false
run := func(_ context.Context, name string, args ...string) (string, error) {
if name != "docker" {
return "", errors.New("not installed")
}
ran = append(ran, strings.Join(args, " "))
switch args[0] {
case "info":
return "29.0.0\n", nil
case "container":
if len(ran) > 2 {
return "true\t" + specOfLast, nil
}
return "false\t\n", errors.New("no such container")
case "run":
return "deadbeef\n", nil
case "network":
if connectFails {
return "", errors.New("network predecessor_default not found")
}
}
return "", nil
}
d := parseTrusted(t, `{"declaration":1,"resources":[
{"id":"app","type":"container","name":"app","image":"`+pinned+`",
"networks":["predecessor_default"]}
]}`)
specOfLast = containerSpec(d.Resources[0].(*declaration.Container), inputs{})
alone := *d.Resources[0].(*declaration.Container)
alone.Networks = nil
if specOfLast == containerSpec(&alone, inputs{}) {
t.Fatal("the kept network is not part of the container's spec: kept or let go, the container would be left alone")
}
report, _, err := Apply(context.Background(), archHost(t), d, store.State{}, store.OriginCarried, run, nil, nil)
if err != nil {
t.Fatal(err)
}
joined := false
for i, line := range ran {
if line == "network connect predecessor_default app" {
joined = true
if ran[i-1] != "container inspect --format {{.State.Running}}\t{{index .Config.Labels \""+specLabel+"\"}} app" &&
!strings.HasPrefix(ran[i-1], "container inspect") {
t.Errorf("joined before the container was read back as running: %v", ran)
}
}
}
if !joined || report.Outcomes[0].Action != "created" {
t.Fatalf("the container did not join the kept network: %v\n%+v", ran, report.Outcomes)
}
connectFails, ran = true, nil
if _, _, err := Apply(context.Background(), archHost(t), d, store.State{}, store.OriginCarried, run, nil, nil); err == nil ||
!strings.Contains(err.Error(), "could not join the kept network predecessor_default") {
t.Fatalf("a network that cannot be joined was passed over: %v", err)
}
}
var specOfLast string
// A module the mesh left out of a declaration is not a module the mesh removed (novox/hq ADR 0163,
// rule 6): what the host wrote for it stays, recorded and said; what it holds for it stays held.
// A module simply absent is removed as it always was.
func TestALeftOutModuleIsNeitherRemovedNorForgotten(t *testing.T) {
var removed []string
gone := map[string]bool{}
run := func(_ context.Context, name string, args ...string) (string, error) {
if name != "docker" {
return "", nil
}
switch args[0] {
case "info":
return "29.0.0\n", nil
case "rm":
removed = append(removed, args[len(args)-1])
gone[args[len(args)-1]] = true
case "container":
if gone[args[len(args)-1]] {
return "", errors.New("no such container")
}
return "true\tspec", nil
}
return "", nil
}
known := store.State{
Resources: []store.Applied{
{ID: "web.server", Type: "container", Target: "web", Origin: store.OriginDeclared},
{ID: "old.server", Type: "container", Target: "old", Origin: store.OriginDeclared},
},
Held: []store.Held{{ID: "web.page", Module: "web", Kind: "file", Target: "/srv/web/index.html"}},
}
d := parse(t, `{"declaration":1,"left_out":["web"],"resources":[
{"id":"notes.conf","type":"file","path":"`+t.TempDir()+`/notes.conf","content":"x"}
]}`)
report, state, err := Apply(context.Background(), archHost(t), d, known, store.OriginDeclared, run, nil, nil)
if err != nil {
t.Fatal(err)
}
if len(removed) != 1 || removed[0] != "old" {
t.Fatalf("removed %v; only the module that is absent goes", removed)
}
if _, kept := state.At("container", "web"); !kept {
t.Fatal("the left-out module's record was forgotten")
}
if _, held := state.HeldAt("web.page"); !held {
t.Fatal("the left-out module's hold was released")
}
said := false
for _, o := range report.Outcomes {
if o.ID == "web.server" && o.Action == "unchanged" && strings.Contains(o.Detail, "web was left out of this declaration by the mesh") {
said = true
}
if o.ID == "web.server" && o.Action != "unchanged" {
t.Errorf("the left-out module's container was %s", o.Action)
}
}
if !said {
t.Fatalf("keeping the left-out module's container was not said: %+v", report.Outcomes)
}
}
// A container's capabilities reach the runtime and are part of its spec (novox/hq ADR 0170).
func TestACapabilityReachesTheRuntimeAndTheSpec(t *testing.T) {
var ran []string
run := func(_ context.Context, name string, args ...string) (string, error) {
if name != "docker" {
return "", errors.New("not installed")
}
switch args[0] {
case "info":
return "29.0.0\n", nil
case "container":
return "false\t\n", errors.New("no such container")
case "run":
ran = args
return "deadbeef\n", nil
}
return "", nil
}
d := parseTrusted(t, `{"declaration":1,"resources":[
{"id":"fw","type":"container","name":"fw","image":"`+pinned+`","network":"host","capabilities":["NET_ADMIN"]}
]}`)
_, _, _ = Apply(context.Background(), archHost(t), d, store.State{}, store.OriginCarried, run, nil, nil)
granted := false
for i, a := range ran {
if a == "--cap-add" && i+1 < len(ran) && ran[i+1] == "NET_ADMIN" {
granted = true
}
}
if !granted {
t.Fatalf("the capability was not granted: %v", ran)
}
with := d.Resources[0].(*declaration.Container)
without := *with
without.Capabilities = nil
if containerSpec(with, inputs{}) == containerSpec(&without, inputs{}) {
t.Fatal("a capability is not part of the container's spec")
}
}
// A package may be declared absent (novox/hq ADR 0175): removed when it is installed, read back,
// left alone when it is not.
func TestAPackageDeclaredAbsentIsRemovedWhenPresentAndLeftWhenNot(t *testing.T) {
installed := true
var ran []string
run := func(_ context.Context, name string, args ...string) (string, error) {
ran = append(ran, name+" "+strings.Join(args, " "))
if name != "pacman" {
return "", nil
}
switch args[0] {
case "-Q":
if args[1] == "pacman" || installed {
return args[1] + " 1.0\n", nil
}
return "", errors.New("package not found")
case "-R":
installed = false
}
return "", nil
}
d := parseTrusted(t, `{"declaration":1,"resources":[{"id":"front-end","type":"package","package":"ufw","absent":true}]}`)
report, _, err := Apply(context.Background(), archHost(t), d, store.State{}, store.OriginCarried, run, nil, nil)
if err != nil {
t.Fatal(err)
}
if report.Outcomes[0].Action != "removed" || !strings.Contains(strings.Join(ran, "\n"), "pacman -R --noconfirm ufw") {
t.Fatalf("an installed package declared absent was not removed: %+v\n%v", report.Outcomes[0], ran)
}
ran = nil
report, _, err = Apply(context.Background(), archHost(t), d, store.State{}, store.OriginCarried, run, nil, nil)
if err != nil {
t.Fatal(err)
}
if report.Outcomes[0].Action != "unchanged" || strings.Contains(strings.Join(ran, "\n"), "-R") {
t.Fatalf("a package already absent was touched: %+v\n%v", report.Outcomes[0], ran)
}
}
+43
View File
@@ -0,0 +1,43 @@
package apply
import (
"context"
"strings"
"testing"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/store"
)
// A container declared to log to the journal is run with the journal as its log driver, and the
// place it logs is part of its spec, so moving it recreates the container (novox/hq ADR 0179).
func TestAContainerLoggingToTheJournalIsRunThatWayAndRecreatedWhenMoved(t *testing.T) {
pinned := "postgres@sha256:" + strings.Repeat("a", 64)
var ran []string
run := func(_ context.Context, cmd string, args ...string) (string, error) {
if cmd == "docker" && len(args) > 0 && args[0] == "run" {
ran = args
return "deadbeef\n", nil
}
return "", nil
}
d := parseTrusted(t, `{"declaration":1,"resources":[
{"id":"front","type":"container","name":"front","image":"`+pinned+`","logging":"journald"}
]}`)
_, _, _ = Apply(context.Background(), archHost(t), d, store.State{}, store.OriginCarried, run, nil, nil)
sent := false
for i, a := range ran {
if a == "--log-driver" && i+1 < len(ran) && ran[i+1] == "journald" {
sent = true
}
}
if !sent {
t.Fatalf("the container's output was not sent to the journal: %v", ran)
}
with := d.Resources[0].(*declaration.Container)
without := *with
without.Logging = ""
if containerSpec(with, inputs{}) == containerSpec(&without, inputs{}) {
t.Fatal("where a container logs is not part of its spec, so moving it would not recreate it")
}
}
+54
View File
@@ -0,0 +1,54 @@
package apply
import (
"testing"
"github.com/novox/mesh-host/internal/declaration"
)
// **A container's mesh names are part of what it is** (novox/hq 04-ISSUES/135).
//
// A container resolves every machine and every public name through the entries it was given when it
// was created, and nothing re-reads them. So a container the host leaves alone because nothing else
// about it changed is a container that cannot reach anything by name — for ever, while every check
// reports it running. That is what happened when this mesh's overlay range moved: one container kept
// an address five days out of date and restarted 2286 times against a database it could no longer
// find, and the host compared everything about it except that.
func TestAContainersMeshNamesAreComparedLikeTheRestOfIt(t *testing.T) {
was := &declaration.Container{
Name: "umami", Image: "ghcr.io/example/umami@sha256:" + zeros(64),
Hosts: []string{"novox.internal:10.42.0.1", "umami.novox.be:10.42.0.1"},
}
moved := &declaration.Container{
Name: was.Name, Image: was.Image,
Hosts: []string{"novox.internal:10.10.0.1", "umami.novox.be:10.10.0.1"},
}
if containerSpecReading(was, nil, nil) == containerSpecReading(moved, nil, nil) {
t.Fatal("a container whose mesh names moved compares equal, so it is never recreated")
}
// And the order they arrive in is not a change: the digest must not move for a reordering
// nobody made.
reordered := &declaration.Container{
Name: moved.Name, Image: moved.Image,
Hosts: []string{moved.Hosts[1], moved.Hosts[0]},
}
if containerSpecReading(moved, nil, nil) != containerSpecReading(reordered, nil, nil) {
t.Fatal("the same names in another order read as a different container")
}
// A container the mesh gives no names is unaffected, so nothing is recreated for a field it
// does not set.
plain := &declaration.Container{Name: "plex", Image: was.Image}
if containerSpecReading(plain, nil, nil) == containerSpecReading(was, nil, nil) {
return // different for other reasons, which is fine
}
}
func zeros(n int) string {
out := make([]byte, n)
for i := range out {
out[i] = '0'
}
return string(out)
}
+51 -12
View File
@@ -54,20 +54,53 @@ func foundFirewall(ctx context.Context, d *declaration.Declaration, known *store
return kind, nil
}
// retireFirewall disables the found firewall once a converged declaration has applied cleanly,
// which is when the mesh's derived filter has taken its place. Disabled, never flushed: its
// configuration stays on disk for a return to adopted, and the container runtime's rules are not
// its to take.
// retireFirewall keeps the found firewall retired on a converged machine (novox/hq ADR 0100, ADR
// 0168): disabled, never flushed, its configuration left on disk for a return to adopted, and the
// container runtime's rules not its to take.
//
// **Convergence is a state the host keeps, not a step it takes once.** Every converged apply reads
// whether the front end is in force; enabled again by a package, a boot or a hand, it is retired
// again and said. The record says how it came to be inactive — the mesh disabled it, or a reconcile
// found it so — and the two are never confused: a flip that did not take, followed by a hand that
// did, used to be recorded as the mesh's doing (issue 143).
//
// Only a declaration from the mesh converges a node. A carried bundle never says a node is adopted
// — it cannot — so its silence is not the controller's word that the node was converged, and an
// adopted node re-applying its bundle keeps the firewall it was found with.
//
// Returned is what this apply did about the found firewall, for the report; empty when the machine
// has none or is not converged.
func retireFirewall(ctx context.Context, d *declaration.Declaration, origin string, known *store.State,
run Runner, log func(string)) error {
run Runner, log func(string)) (string, error) {
rec := known.Firewall
if origin != store.OriginDeclared || d.Adoption != nil || rec == nil || rec.Kind != string(firewall.UFW) || !rec.WasActive ||
rec.DisabledByMesh {
return nil
if origin != store.OriginDeclared || d.Adoption != nil || rec == nil || rec.Kind != string(firewall.UFW) || !rec.WasActive {
return "", nil
}
if !firewall.Installed(ctx, run) {
// Uninstalled (novox/hq ADR 0175): retired for good, by the module that replaced it. Said
// once, and nothing is asked of a command that is not there.
if rec.RetiredBy != firewall.RetiredRemoved {
rec.RetiredBy = firewall.RetiredRemoved
log(" the found firewall (ufw) is no longer installed; the mesh's filter is what filters this machine")
return "removed: ufw is no longer installed; the mesh's filter is what filters this machine", nil
}
return "", nil
}
active := firewall.Active(ctx, run)
if !active && !(rec.Forward != nil && !rec.DisabledByMesh) {
// Inactive, and either the mesh's doing already or nobody's recorded here: said as found,
// never as done (issue 143's second fault). A retirement the mesh began and did not finish —
// the forward policy recorded, ufw down, the restore failed — is the one inactive state that
// is still the mesh's to complete, below.
if rec.RetiredBy == "" {
if rec.DisabledByMesh {
rec.RetiredBy = firewall.RetiredByMesh
} else {
rec.RetiredBy = firewall.RetiredFoundSo
log(" ufw is inactive on this converged node, and not by the mesh; recorded as found so")
}
}
return "", nil
}
// **Nothing is retired until what replaces it is in force** (novox/hq ADR 0100). The flip
// loads the mesh's derived filter in ufw's place; disabling ufw before that table is actually
@@ -75,10 +108,10 @@ func retireFirewall(ctx context.Context, d *declaration.Declaration, origin stri
// no filter at all.
loaded, err := firewall.MeshTableLoaded(ctx, run)
if err != nil {
return err
return "", err
}
if !loaded {
return fmt.Errorf("this node is converged and the mesh's own filter (table %s) is not loaded on "+
return "", fmt.Errorf("this node is converged and the mesh's own filter (table %s) is not loaded on "+
"this machine, so ufw was left in force: retiring it would leave the machine filtering "+
"nothing. Assign a filter module to this node, or return it to adopted", firewall.MeshTable)
}
@@ -88,11 +121,17 @@ func retireFirewall(ctx context.Context, d *declaration.Declaration, origin stri
rec.Forward = firewall.ForwardPolicies(ctx, run)
}
if err := firewall.Disable(ctx, run, rec.Forward); err != nil {
return err
return "", err
}
again := rec.DisabledByMesh || rec.RetiredBy != ""
rec.DisabledByMesh = true
rec.RetiredBy = firewall.RetiredByMesh
if again {
log(" disabled ufw again: it had been enabled since the mesh retired it; this node is converged and filtered by the mesh")
return "disabled again: ufw had been enabled since the mesh retired it", nil
}
log(" disabled ufw: this node is converged and filtered by the mesh; ufw's configuration is left on disk")
return nil
return "disabled: this node is converged and filtered by the mesh; ufw's configuration is left on disk", nil
}
// applyOpening makes one opening true through the firewall found here.
+53 -2
View File
@@ -8,6 +8,7 @@ import (
"path/filepath"
"strings"
"testing"
"time"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/store"
@@ -189,14 +190,34 @@ func TestConvergingRetiresTheFoundFirewallAndReturningRestoresIt(t *testing.T) {
}
}
// Converged again: nothing more to retire.
// Converged again: nothing more to retire — the node asks ufw whether it is in force, which is
// what keeps convergence a state rather than a step taken once (novox/hq ADR 0168), and touches
// nothing else.
u.asked = nil
if _, state, err = applyWith(t, converged, state, u.run); err != nil {
t.Fatal(err)
}
if u.index("ufw") >= 0 {
for _, a := range u.asked {
if strings.HasPrefix(a, "ufw") && a != "ufw status" {
t.Errorf("a converged node kept talking to a retired ufw: %v", u.asked)
}
}
if state.Firewall.RetiredBy != "mesh" {
t.Errorf("the record does not say the mesh retired it: %+v", state.Firewall)
}
// Enabled again by a hand: retired again, and said.
u.active = true
u.asked = nil
report, state, err := applyWith(t, converged, state, u.run)
if err != nil {
t.Fatal(err)
}
if u.active || u.index("ufw disable") < 0 {
t.Fatalf("ufw enabled again on a converged node was not retired again: %v", u.asked)
}
if !strings.Contains(report.Firewall, "disabled again") {
t.Errorf("retiring it again was not said: %q", report.Firewall)
}
// Returned to adopted: ufw is enabled before the opening is converged through it.
u.asked = nil
@@ -502,3 +523,33 @@ func TestUfwIsNotRetiredUntilTheMeshsOwnFilterIsLoaded(t *testing.T) {
t.Errorf("ufw was not retired once the mesh's filter was loaded: active %v, %+v", u.active, state.Firewall)
}
}
// A front end that is no longer installed is recorded as removed, said once, and asked nothing of
// (novox/hq ADR 0175).
func TestAnUninstalledFrontEndIsRetiredForGood(t *testing.T) {
dir := t.TempDir()
u := &ufwMachine{installed: false, ruleset: "table inet mesh\n"}
known := store.State{Firewall: &store.FoundFirewall{Kind: "ufw", WasActive: true, DisabledByMesh: true,
RetiredBy: "mesh", FoundAt: time.Now()}}
converged := parse(t, `{"declaration":1,"resources":[`+withConf(dir)+`]}`)
report, state, err := applyWith(t, converged, known, u.run)
if err != nil {
t.Fatal(err)
}
if state.Firewall.RetiredBy != "removed" || !strings.Contains(report.Firewall, "no longer installed") {
t.Fatalf("record %+v, said %q", state.Firewall, report.Firewall)
}
u.asked = nil
report, _, err = applyWith(t, converged, state, u.run)
if err != nil {
t.Fatal(err)
}
if report.Firewall != "" {
t.Errorf("said again: %q", report.Firewall)
}
for _, a := range u.asked {
if strings.HasPrefix(a, "ufw") && a != "ufw status" {
t.Errorf("asked something of a front end that is not there: %v", u.asked)
}
}
}
+82
View File
@@ -316,3 +316,85 @@ func TestAContainerNamingARunOnceStepIsRecreatedWhenItRan(t *testing.T) {
t.Errorf("the recreation did not name the step as its reason: %+v", server)
}
}
func TestAFailedStepGatesItsModuleAndNotTheMachine(t *testing.T) {
// **The blast radius of a step is its module** (novox/hq ADR 0136). A step exists to make
// something true before the next thing in its own module needs it — a store seeded before the
// broker starts, a schema prepared before the version that needs it runs. Stopping the whole
// apply is what this host's own loop calls holding a machine hostage, and it was already
// rejected for every other shape (04-ISSUES/011): a module whose database is briefly
// unreachable must not stop every module declared after it.
var startedNames []string
run := func(ctx context.Context, name string, args ...string) (string, error) {
switch args[0] {
case "info":
return "27.0\n", nil
case "container":
return "false\t\n", errors.New("no such container")
case "run":
startedNames = append(startedNames, nameOf(args))
if nameOf(args) == "catalogue-prepare" {
return "", errors.New("exit status 1") // the schema could not be reached
}
return "deadbeef\n", nil
case "rm":
return "", nil
}
return "", nil
}
d := parseTrusted(t, `{"declaration":1,"resources":[
{"id":"mesh-catalog.runtime-prepare","type":"container","name":"catalogue-prepare","image":"`+pinned+`","run-once":true},
{"id":"mesh-catalog.runtime","type":"container","name":"catalogue","image":"`+pinned+`"},
{"id":"gitea.server","type":"container","name":"forge","image":"`+pinned+`"}
]}`)
report, _, err := Apply(context.Background(), archHost(t), d, store.State{}, store.OriginCarried, run, nil, nil)
if err == nil {
t.Fatal("a failed step was not reported as a failure")
}
started := map[string]bool{}
for _, n := range startedNames {
started[n] = true
}
if started["catalogue"] {
t.Error("the module's own workload ran although its step did not complete")
}
if !started["forge"] {
t.Error("another module was not attempted, so one module's step held the machine hostage")
}
// And the machine's own account says which was not attempted, rather than leaving it to be
// inferred from silence.
var skipped string
for _, o := range report.Outcomes {
if o.Action == "skipped" {
skipped = o.ID
}
}
if skipped != "mesh-catalog.runtime" {
t.Errorf("the report does not say what was not attempted: %q", skipped)
}
}
func TestAResourcesOwnerIsReadToTheLastDot(t *testing.T) {
// **A module's name may contain a dot.** `novox.be` is one on this mesh, so reading a resource's
// owner to the first dot would make its resources belong to something called "novox" — and a gate
// would skip whatever else happened to start that way. A resource's own id never contains one,
// which is what makes the last dot the boundary.
for identity, want := range map[string]string{
"novox.be.server": "novox.be",
"mesh-catalog.runtime-prepare": "mesh-catalog",
"gitea.admin-bootstrap": "gitea",
} {
got, ours := moduleOf(identity)
if !ours || got != want {
t.Errorf("%q belongs to %q (%v), want %q", identity, got, ours, want)
}
}
// What the mesh declares in its own right belongs to no module: the foundation's resources carry
// no dot, and the adoption's are the mesh's.
for _, identity := range []string{"container-runtime", "store-ready", "adoption.guard", ".server"} {
if _, ours := moduleOf(identity); ours {
t.Errorf("%q was read as a module's", identity)
}
}
}
+54
View File
@@ -0,0 +1,54 @@
package apply
import (
"context"
"sort"
"strings"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/store"
)
// Strays is what runs on the machine that the mesh neither wrote nor holds (novox/hq ADR 0163):
// every container the runtime has that no record names and no hold names. The question nothing
// answered on 2026-09-23, when a renamed resource left its old container running for a day; asked
// on every apply now, and reported, so a thing left behind is seen the day it is left.
//
// Containers only, today. A listener nobody declared is harder to attribute to a thing, and the
// machine's own services are not strays; that account is issue 160's.
func Strays(ctx context.Context, run Runner, known store.State) ([]store.Stray, error) {
cri, err := containerRuntime(ctx, run)
if err != nil {
return nil, nil // a machine with no runtime has no containers to stray
}
out, err := run(ctx, cri, "ps", "-a", "--format", "{{.Names}}\t{{.Image}}\t{{.State}}")
if err != nil {
return nil, err
}
ours := map[string]bool{}
for _, r := range known.Resources {
if declaration.Type(r.Type) == declaration.TypeContainer {
ours[r.Target] = true
}
}
for _, h := range known.Held {
if h.Kind == string(declaration.TypeContainer) {
ours[h.Target] = true
}
}
var strays []store.Stray
for _, line := range strings.Split(strings.TrimSpace(out), "\n") {
parts := strings.Split(line, "\t")
name := strings.TrimSpace(parts[0])
if name == "" || ours[name] {
continue
}
detail := ""
if len(parts) > 2 {
detail = strings.TrimSpace(parts[1]) + ", " + strings.TrimSpace(parts[2])
}
strays = append(strays, store.Stray{Kind: string(declaration.TypeContainer), Name: name, Detail: detail})
}
sort.Slice(strays, func(i, j int) bool { return strays[i].Name < strays[j].Name })
return strays, nil
}
+43
View File
@@ -119,6 +119,11 @@ func Enrol(ctx context.Context, o Options, sys system.System, control controlPla
// its ports and range on and not another that came up since.
args = append(args, "--tunnel", o.Tunnel)
}
// The enrolment account the token's secret is the password of exists in the mesh's records
// and nowhere on the bus yet (novox/hq 04-ISSUES/146): placed before the machine presents it.
if err := placeTheBusUsers(ctx, control, say); err != nil {
return out, err
}
joined, err := control.run(joining, o.Host, args...)
cancel()
if err != nil {
@@ -137,6 +142,10 @@ func Enrol(ctx context.Context, o Options, sys system.System, control controlPla
}
out.Joined = true
say(" enrolled as " + o.Node)
// And the node's own account, minted as it enrolled, before its agent connects as it.
if err := placeTheBusUsers(ctx, control, say); err != nil {
return out, err
}
}
// 4. The agent.
@@ -148,6 +157,40 @@ func Enrol(ctx context.Context, o Options, sys system.System, control controlPla
return out, nil
}
// busAccounts and busContainer are where the installer's bundle raises the bus: the file its
// configuration includes, and the container that reads it. The installer raised them, so it is the
// one that knows them (examples/foundation-first-node-nats.lock).
const (
busAccounts = "/var/lib/mesh-bus-conf/accounts.conf"
busContainer = "mesh-broker"
)
// placeTheBusUsers writes the mesh's composed user list beside the bus the installer raised, and makes
// the bus re-read it.
//
// **Genesis's own step** (novox/hq 04-ISSUES/146). Every account on the bus reaches it in the
// declaration of the machine that runs it — which needs that machine to be an enrolled node, and at
// genesis it is not. The control plane composes the list and says it; whoever raised the bus places
// it. That is this installer: it carried the bus in its bundle, so it knows where the bus reads it,
// and the control plane never has to.
func placeTheBusUsers(ctx context.Context, control controlPlane, say func(string)) error {
users, err := control.tell(ctx, "broker", "accounts")
if err != nil {
return fmt.Errorf("the control plane would not say the bus's users, so no machine could "+
"join it: %w", err)
}
if !strings.Contains(users, "accounts") {
return fmt.Errorf("the control plane's account of the bus's users is not one:\n%s", indent(users))
}
script := "umask 077 && cat > " + busAccounts + ".next <<'MESHBUSUSERS'\n" + users + "\nMESHBUSUSERS\n" +
"mv " + busAccounts + ".next " + busAccounts + " && docker kill -s HUP " + busContainer + " >/dev/null"
if out, err := control.run(ctx, "sh", "-c", script); err != nil {
return fmt.Errorf("the bus's users could not be placed at %s: %w\n%s", busAccounts, err, indent(out))
}
say(" bus users placed")
return nil
}
// runTheHost makes sure something on this machine is listening to the mesh, and proves it.
//
// **The installer does not install the service, and says so.** A unit file is a packaging decision
+53
View File
@@ -261,3 +261,56 @@ func TestAListingIsMatchedByNameAndNotBySubstring(t *testing.T) {
t.Error("registry-mirror was not found")
}
}
// **Genesis places the bus's users, before the machine enrols and again after** (novox/hq
// 04-ISSUES/146). The enrolment account exists only in the mesh's records until somebody writes it
// beside the bus; so does the node's own, minted as it enrols. Without the first, the machine is
// refused by the bus it just raised; without the second, its agent is.
func TestAFirstNodeIsLetOntoTheBusItRaised(t *testing.T) {
enrolled := false
runtime := &asked{answer: func(name string, args []string) (string, error) {
joined := strings.Join(args, " ")
switch {
case strings.Contains(joined, "node list"):
if enrolled {
return "anchor here 01J0\n", nil
}
return "", nil
case strings.Contains(joined, "node add"):
return "added anchor\n", nil
case strings.Contains(joined, "token issue"):
return "a token for anchor, good once:\n\n " + strings.Repeat("t", 240) + "\n\n", nil
case strings.Contains(joined, "broker accounts"):
return "accounts {\n MESH { users = [] }\n}\n", nil
case name == "/usr/local/bin/mesh-host":
enrolled = true
return "enrolled as anchor\n", nil
case name == "pgrep", name == "sh":
return "", nil
}
return "", fmt.Errorf("unexpected: %s %v", name, args)
}}
if _, err := Enrol(context.Background(), Options{
Node: "anchor", State: filepath.Join(t.TempDir(), "state.json"), Timeout: time.Second,
Host: "/usr/local/bin/mesh-host", HostInBackground: true,
}, arch(t), controlPlane{container: "temp-mesh-controller", run: runtime.run, timeout: time.Second},
func(string) {}); err != nil {
t.Fatal(err)
}
placed, enrol := []int{}, -1
for i, c := range runtime.commands {
if strings.HasPrefix(c, "sh -c") && strings.Contains(c, "kill -s HUP mesh-broker") &&
strings.Contains(c, "/var/lib/mesh-bus-conf/accounts.conf") {
placed = append(placed, i)
}
if strings.HasPrefix(c, "/usr/local/bin/mesh-host enrol") {
enrol = i
}
}
if enrol < 0 || len(placed) != 2 || placed[0] > enrol || placed[1] < enrol {
t.Fatalf("the bus's users were not placed before the machine enrolled and again after "+
"(placed at %v, enrolled at %d):\n%s", placed, enrol, strings.Join(runtime.commands, "\n"))
}
}
+79
View File
@@ -0,0 +1,79 @@
package bootstrap
import (
"context"
"encoding/json"
"os"
"path/filepath"
"strings"
"testing"
"time"
)
// What genesis raises, it raises as the module that succeeds it declares — name, data directory
// and image — so the module adopts it by the found rule that already exists (novox/hq ADR 0163,
// rule 7; issue 090). The network is the one difference left: the bootstrap forge runs on the
// machine's network to reach the store on its loopback, and a take says so.
func TestGenesisRaisesTheForgeAsTheModuleDeclaresIt(t *testing.T) {
var ran [][]string
run := func(_ context.Context, name string, args ...string) (string, error) {
if name == "docker" && args[0] == "container" {
return "", nil // not raised yet
}
ran = append(ran, append([]string{name}, args...))
return "", nil
}
if err := raiseGiteaServer(context.Background(), run, time.Second, "pw", DefaultPorts(), quietly); err != nil {
t.Fatal(err)
}
var raised []string
for _, r := range ran {
if r[0] == "docker" && r[1] == "run" {
raised = r
}
}
line := strings.Join(raised, " ")
for _, want := range []string{"--name gitea ", "--volume " + giteaDataDir + ":/data", " " + giteaImage} {
if !strings.Contains(line+" ", want) {
t.Errorf("the forge is not raised with %q: %s", want, line)
}
}
if giteaBootstrap != ForgeModule {
t.Errorf("the bootstrap forge is %q and the module names its container %q", giteaBootstrap, ForgeModule)
}
// Against the module's own manifest, where the catalogue is checked out beside this repository.
var manifest []byte
for _, candidate := range []string{"../../../mesh-catalog/modules/gitea/module.json", "../../../../../mesh-catalog/modules/gitea/module.json"} {
if raw, err := os.ReadFile(filepath.Clean(candidate)); err == nil {
manifest = raw
break
}
}
if manifest == nil {
t.Skip("the catalogue is not beside this checkout; the module's pin is not compared")
}
var m struct {
Resources []struct {
ID, Type, Name, Image string
Volumes []string
} `json:"resources"`
}
if err := json.Unmarshal(manifest, &m); err != nil {
t.Fatal(err)
}
for _, r := range m.Resources {
if r.Type != "container" || r.ID != "server" {
continue
}
if r.Name != giteaBootstrap {
t.Errorf("the module names its container %q; genesis raises %q", r.Name, giteaBootstrap)
}
if r.Image != giteaImage {
t.Errorf("the module pins %s; genesis raises %s — the two must move together", r.Image, giteaImage)
}
if len(r.Volumes) != 1 || !strings.HasSuffix(r.Volumes[0], ":/data") {
t.Errorf("the module mounts %v; genesis mounts %s:/data", r.Volumes, giteaDataDir)
}
}
}
+1 -1
View File
@@ -91,7 +91,7 @@ func TestARerunOfGenesisIsNotAMachineInUse(t *testing.T) {
if err := store.Save(o.State, store.State{Resources: []store.Applied{{ID: "store", Type: "container", Target: "mesh-store"}}}); err != nil {
t.Fatal(err)
}
m := inUseRunner{ps: "mesh-gitea-server\t\n", ss: servingSockets}
m := inUseRunner{ps: giteaBootstrap + "\t\n", ss: servingSockets}
if err := RefuseAMachineInUse(context.Background(), o, m.run, quietly); err != nil {
t.Errorf("what an earlier genesis raised was counted as a machine in use: %v", err)
}
+23 -6
View File
@@ -25,11 +25,24 @@ const (
// foundationStore is the foundation's postgres container — the mesh's own memory, raised from the
// bundle. gitea's bootstrap database lives here too, so a mesh runs one postgres (issue 051).
foundationStore = "mesh-store"
// giteaBootstrap is the gitea server raised directly at genesis, before gitea is a module.
giteaBootstrap = "mesh-gitea-server"
// giteaImage is the same upstream image the gitea module runs, pinned identically so the module
// adopts the running server rather than replacing it.
giteaImage = "gitea/gitea@sha256:dfc61e347c8b582df918f4556401bf2cecdfbdb56c5282ae9488dd76fca3e41c"
// giteaBootstrap is the gitea server raised directly at genesis, before gitea is a module —
// under the name the gitea MODULE declares for its container, so the module finds it and holds
// it rather than raising a second forge beside it (novox/hq ADR 0163, rule 7; issue 090).
giteaBootstrap = "gitea"
// giteaImage is the image the gitea module declares for that container, pinned to the same
// digest, so taking the module over is not a downgrade and not an upgrade. **Moves with the
// module's pin**: the two are compared by a take, and a difference is said there — but a
// genesis that raised an older image than the module declares would be taken over as an
// upgrade on first push, which a forge holding the mesh's packages must not have done to it
// unannounced. Checked in TestGenesisRaisesTheForgeAsTheModuleDeclaresIt against the module's
// manifest where the catalogue is beside this checkout.
giteaImage = "gitea/gitea@sha256:87a67ee09d3ae0d1df5fda5dcda3e2a1f9236a45b0a59025d6e00e46adc43bef"
// giteaDataDir is where the module's `data` directory resolves on a machine with the default
// layout (<data root>/<module>/<id>, novox/hq ADR 0112): mounted at /data as the module mounts
// it, so the repositories, attachments and indexes the bootstrap forge accumulates are the
// module's the day it is taken — before this, the forge had no volume and its data was the
// container's, lost with it.
giteaDataDir = "/var/lib/gitea/data"
// packagesOrg is the npm owner: every module consumes `@novox/*` from this gitea org.
packagesOrg = "novox"
// packagesTeam is the org team whose members may read and write the org's packages.
@@ -262,9 +275,13 @@ func raiseGiteaServer(ctx context.Context, run Runner, timeout time.Duration, db
"run", "-d", "--name", giteaBootstrap,
// Host network, like the control plane: it reaches the foundation store on the machine's
// loopback (where the store publishes 5432) and answers on the machine's own 3000, which is
// where mesh-bootstrap and the builder's build containers look for it.
// where mesh-bootstrap and the builder's build containers look for it. The module runs
// bridged and publishes its ports; that is the one difference a take still has to say
// (ADR 0163, rule 7) — the data, the name and the image are the module's already.
"--network", "host",
"--restart", "unless-stopped",
// The module's data directory, so what the forge accumulates is the module's when taken.
"--volume", giteaDataDir + ":/data",
}, env...)
args = append(args, giteaImage)
+104 -1
View File
@@ -870,6 +870,12 @@ type Package struct {
ID string `json:"id"`
Type Type `json:"type"`
Package string `json:"package"`
// Absent declares that the package is NOT installed (novox/hq ADR 0175): the host removes it
// when it is, and leaves a machine that never had it alone. For the one case a module replaces
// software the machine was found with and the operator has decided it does not come back — the
// firewall front end a converged machine's filter module retired. Nothing to undo when the
// declaration drops it: the host does not install what a declaration stopped saying is absent.
Absent bool `json:"absent,omitempty"`
}
func (p *Package) Identity() string { return p.ID }
@@ -940,6 +946,27 @@ type Container struct {
// its siblings can name before any of them can resolve anything.
Dns []string `json:"dns,omitempty"`
// Capabilities are the Linux capabilities this container is granted beyond the runtime's
// default set, by name (novox/hq ADR 0170): a holder's runtime that changes the machine's packet
// filter asks for NET_ADMIN. Exactly these, named in the spec so a change recreates the
// container; a privileged container stays undeclarable.
Capabilities []string `json:"capabilities,omitempty"`
// Logging names where the runtime sends this container's output: "journald" sends it to the
// machine's journal, under the container's name, where what reads the machine's logs — its
// intrusion prevention first of all (novox/hq ADR 0179) — can read it the way it reads the
// machine's own services. Empty keeps the runtime's default, which is a file of the runtime's
// own that nothing but the runtime reads. Part of the spec: a container that logs elsewhere
// is a different container, and the runtime cannot change a running one's driver.
Logging string `json:"logging,omitempty"`
// Networks are networks this container also joins once created, by name — a found network a
// per-machine setting keeps for a taken container (novox/hq ADR 0163, rule 4), so a
// neighbour that resolves it there keeps resolving it until the neighbour is taken too.
// Joined after creation, because a runtime starts a container on one network; part of the
// container's spec, so a network kept or let go recreates it.
Networks []string `json:"networks,omitempty"`
// IP is this container's address on its network, passed to the runtime unchanged.
//
// Only meaningful on a user-defined network, and refused by the runtime elsewhere. Exists for
@@ -1032,6 +1059,26 @@ func (c *Container) validate(where string, _ bool) []string {
"static address anywhere but a user-defined one")
}
}
for _, cap := range c.Capabilities {
if !capabilityName.MatchString(cap) {
problems = append(problems, where+": capabilities names "+strconv.Quote(cap)+", which is not a "+
"capability's name (CAP_NET_ADMIN or NET_ADMIN)")
}
}
if c.Logging != "" && c.Logging != "journald" {
problems = append(problems, where+": logging is "+strconv.Quote(c.Logging)+", and the only place a "+
"container's output can be sent besides the runtime's own file is \"journald\"")
}
for _, n := range c.Networks {
problems = append(problems, (&Network{Name: n}).validate(where+": networks", false)...)
if n == c.Network {
problems = append(problems, where+": networks names "+n+", which is already the container's network")
}
}
if len(c.Networks) > 0 && (c.RunOnce || c.Schedule != "") {
problems = append(problems, where+": networks is for a container that keeps running; a step "+
"runs and exits, and joins nothing afterwards")
}
return append(problems, checkImage(where, c.Image)...)
}
@@ -1137,6 +1184,41 @@ type Declaration struct {
// converged node — which is every node the mesh raised before adoption existed, and so the
// only form an older controller ever sends (novox/hq ADR 0100).
Adoption *Adoption
// Sequence orders this declaration against every other the mesh has sent this node: each
// send is one higher than the last, assigned under the control plane's hold on the node
// (novox/hq 04-ISSUES/107). Zero is a declaration that carries no order — every one an older
// controller sent, and the bundle genesis applies — and a host makes no ordering claim about
// one of those.
//
// **The one property a declaration needs that its signature does not give it.** A signature
// says the mesh sent this; it cannot say the mesh sent it AFTER the one the host is holding.
// Before this, "older" was inferred from arrival within a batch and a 750ms window, and a
// backlog longer than the batch, or a slow broker, applied a declaration the mesh had already
// superseded.
Sequence int64
// LeftOut names the modules of this machine's set the mesh left out of this declaration,
// because a setting stored for one cannot compose with its definition (novox/hq ADR 0163,
// rule 6). A machine is told everything or nothing about what it IS told; this is what it is
// not told, said. The host keeps what it holds for a left-out module and touches none of
// what it wrote for it — its resources are absent from the declaration, and absence would
// otherwise read as removal.
LeftOut []string
}
// LeftOutModuleOf says which left-out module a recorded resource belongs to, if any: its id is the
// module's name, a dot, and the module's own id for it. A module's name may contain a dot, so the
// longest left-out name that prefixes the id wins; a false match keeps a thing an apply would
// otherwise remove, which is the conservative mistake.
func (d *Declaration) LeftOutModuleOf(id string) (string, bool) {
best := ""
for _, m := range d.LeftOut {
if strings.HasPrefix(id, m+".") && len(m) > len(best) {
best = m
}
}
return best, best != ""
}
// Adoption is a node's mode, as the controller records it: the node is adopted, and these are
@@ -1157,6 +1239,10 @@ type Adoption struct {
Untaken map[string][]string `json:"untaken,omitempty"`
}
// capabilityName is what a Linux capability is called: upper case, underscores, an optional CAP_
// prefix. The runtime accepts either spelling.
var capabilityName = regexp.MustCompile(`^(CAP_)?[A-Z][A-Z0-9_]*$`)
// AdoptionPrefix is the id prefix of what the mesh itself declares because a node is adopted —
// its openings and its guard. Nothing under it belongs to a module, so none of it is ever held.
const AdoptionPrefix = "adoption."
@@ -1274,6 +1360,11 @@ type envelope struct {
// a bug quietly strip a machine.
OwnsNothing bool `json:"owns_nothing,omitempty"`
Resources []json.RawMessage `json:"resources"`
// Sequence is optional on the wire, so a controller that does not send one is still
// understood: absent reads as zero, which is "no ordering claimed" rather than "first".
Sequence int64 `json:"sequence,omitempty"`
// LeftOut is optional on the wire too, and absent when nothing was left out (ADR 0163).
LeftOut []string `json:"left_out,omitempty"`
}
func parse(raw []byte, allowActions bool) (*Declaration, error) {
@@ -1290,8 +1381,20 @@ func parse(raw []byte, allowActions bool) (*Declaration, error) {
env.Version, Version)}}
}
d := &Declaration{Version: env.Version, For: env.For, Adoption: env.Adoption}
d := &Declaration{Version: env.Version, For: env.For, Adoption: env.Adoption, Sequence: env.Sequence,
LeftOut: env.LeftOut}
var problems []string
if len(env.LeftOut) > 0 && allowActions {
// The bundle is carried with the binary and leaves nothing out: which module a setting
// stopped composing for is the mesh's record (ADR 0163).
problems = append(problems, "a carried bundle says modules were left out, and only the "+
"mesh can say that")
}
for _, m := range env.LeftOut {
if strings.TrimSpace(m) == "" {
problems = append(problems, "left_out names a module with no name")
}
}
if len(env.Resources) == 0 && !env.OwnsNothing {
problems = append(problems, "no resources. An empty declaration is a mistake, not a "+
+81
View File
@@ -464,3 +464,84 @@ func TestAnExplicitlyEmptyDeclarationIsAccepted(t *testing.T) {
t.Fatalf("an unmarked empty declaration must still be refused; got %v", err)
}
}
// A container's kept networks are names, not its own network, and not for a step (novox/hq ADR
// 0163, rule 4); and the mesh may say which modules it left out, which a carried bundle may not.
func TestKeptNetworksAndLeftOutModulesAreReadStrictly(t *testing.T) {
pinnedImage := "postgres@sha256:" + strings.Repeat("a", 64)
d, err := Parse([]byte(`{"declaration":1,"left_out":["web"],"resources":[
{"id":"app","type":"container","name":"app","image":"` + pinnedImage + `","networks":["predecessor_default"]}
]}`))
if err != nil {
t.Fatal(err)
}
if got := d.Resources[0].(*Container).Networks; len(got) != 1 || got[0] != "predecessor_default" {
t.Fatalf("the kept network was not read: %v", got)
}
if m, left := d.LeftOutModuleOf("web.server"); !left || m != "web" {
t.Fatalf("web.server is not web's: %q %v", m, left)
}
if _, left := d.LeftOutModuleOf("webapp.server"); left {
t.Fatal("webapp.server was taken for web's")
}
for name, raw := range map[string]string{
"a bad network name": `{"declaration":1,"resources":[
{"id":"app","type":"container","name":"app","image":"` + pinnedImage + `","networks":["a/b"]}]}`,
"its own network": `{"declaration":1,"resources":[
{"id":"app","type":"container","name":"app","image":"` + pinnedImage + `","network":"own","networks":["own"]}]}`,
"a step": `{"declaration":1,"resources":[
{"id":"app","type":"container","name":"app","image":"` + pinnedImage + `","run-once":true,"networks":["x"]}]}`,
"a nameless module": `{"declaration":1,"left_out":[""],"resources":[
{"id":"app","type":"container","name":"app","image":"` + pinnedImage + `"}]}`,
} {
if _, err := Parse([]byte(raw)); err == nil {
t.Errorf("%s was accepted", name)
}
}
if _, err := ParseTrusted([]byte(`{"declaration":1,"left_out":["web"],"resources":[
{"id":"app","type":"container","name":"app","image":"` + pinnedImage + `"}]}`)); err == nil ||
!strings.Contains(err.Error(), "only the mesh can say that") {
t.Fatalf("a carried bundle leaving modules out was accepted: %v", err)
}
}
// A container may ask for a capability by name, and nothing else (novox/hq ADR 0170).
func TestACapabilityIsNamedOrRefused(t *testing.T) {
image := "postgres@sha256:" + strings.Repeat("a", 64)
d, err := Parse([]byte(`{"declaration":1,"resources":[
{"id":"fw","type":"container","name":"fw","image":"` + image + `","network":"host","capabilities":["NET_ADMIN","CAP_NET_RAW"]}
]}`))
if err != nil {
t.Fatal(err)
}
if got := d.Resources[0].(*Container).Capabilities; len(got) != 2 || got[0] != "NET_ADMIN" {
t.Fatalf("capabilities read as %v", got)
}
for _, bad := range []string{`"net_admin"`, `"ALL;rm -rf /"`, `"privileged"`} {
if _, err := Parse([]byte(`{"declaration":1,"resources":[
{"id":"fw","type":"container","name":"fw","image":"` + image + `","capabilities":[` + bad + `]}]}`)); err == nil {
t.Errorf("%s was accepted as a capability", bad)
}
}
}
// A container may send its output to the machine's journal, and nowhere else but the runtime's own
// file (novox/hq ADR 0179): what reads the machine's logs then reads the container's too.
func TestAContainerMayLogToTheJournalAndNowhereElse(t *testing.T) {
image := "postgres@sha256:" + strings.Repeat("a", 64)
d, err := Parse([]byte(`{"declaration":1,"resources":[
{"id":"front","type":"container","name":"front","image":"` + image + `","logging":"journald"}
]}`))
if err != nil {
t.Fatal(err)
}
if got := d.Resources[0].(*Container).Logging; got != "journald" {
t.Fatalf("logging read as %q", got)
}
for _, bad := range []string{`"syslog"`, `"none"`, `"json-file"`} {
if _, err := Parse([]byte(`{"declaration":1,"resources":[
{"id":"front","type":"container","name":"front","image":"` + image + `","logging":` + bad + `}]}`)); err == nil {
t.Errorf("%s was accepted as a place to log", bad)
}
}
}
+335
View File
@@ -0,0 +1,335 @@
package firewall
import (
"context"
"fmt"
"regexp"
"sort"
"strings"
)
// What filters a machine, said with an owner (novox/hq ADR 0168).
//
// "The firewall found" names one front end, and a machine carries rules from several sources: the
// front end's own, the container runtime's plumbing, a ban list, the mesh's own tables, and whatever
// a predecessor installed directly — on both machines of the first mesh, in the user chain the
// runtime leaves for an administrator, where the mesh's reader of rules counted it as the runtime's.
// So the host reports every table and chain that refuses traffic, each with whose it is, and the
// mesh says truthfully what filters a converged machine. It removes none of it.
// Owners of a refusal.
const (
// OwnerMesh is the mesh's own tables: the derived filter and the guard.
OwnerMesh = "mesh"
// OwnerFoundFirewall is the front end found on the machine — ufw's chains.
OwnerFoundFirewall = "found-firewall"
// OwnerRuntime is the container runtime's own plumbing: its chains, the forward policy it sets
// when it turns forwarding on, its guard against reaching a container's address from off its
// bridge. Not the user chain it leaves for an administrator.
OwnerRuntime = "runtime"
// OwnerBan is a refusal that names the sources it refuses, in a chain that accepts nothing — a
// ban list, which is not a firewall.
OwnerBan = "ban"
// OwnerOther is everything else: rules the mesh did not write and cannot attribute. Where a
// predecessor's rules live.
OwnerOther = "other"
)
// A Filter is one place on the machine that refuses traffic: a chain of a table, or a chain of the
// legacy filter, with its owner and what it refuses in one line.
type Filter struct {
// Where names the chain: "table ip filter, chain DOCKER-USER", or "chain HAL-MESH-ONLY
// (iptables-legacy)".
Where string `json:"where"`
// Owner is one of the owners above.
Owner string `json:"owner"`
// Refuses is the first refusing line, counters stripped, and how many more there are.
Refuses string `json:"refuses"`
table, chain string
}
// userChain is the chain the container runtime creates empty and leaves for an administrator's
// rules, consulted before its own forwarding. Nothing in it is the runtime's.
const userChain = "DOCKER-USER"
// Filters classifies every refusing chain of an `nft list ruleset` and of the legacy filter's `-S`
// listings (by tool: iptables-legacy, ip6tables-legacy), in the order they appear.
func Filters(ruleset string, legacy map[string]string, ufwActive bool) []Filter {
var out []Filter
r := parseNft(ruleset)
refusing := map[string][]nftRule{} // by "table\x00chain"
for _, rule := range r.refusals {
k := rule.table + "\x00" + rule.chain
refusing[k] = append(refusing[k], rule)
}
for _, k := range r.chainOrder {
c := r.chains[k]
table, chain, _ := strings.Cut(k, "\x00")
rules := refusing[k]
if !c.dropping && len(rules) == 0 {
continue
}
f := Filter{table: table, chain: chain, Where: "table " + table + ", chain " + chain}
switch {
case table == MeshTable || table == "inet mesh_guard":
f.Owner = OwnerMesh
case strings.HasPrefix(chain, "ufw"):
f.Owner = OwnerFoundFirewall
if !ufwActive {
// Left behind by a retired front end, and still refusing: not ufw's any more in
// any sense that matters, since nothing maintains it.
f.Owner = OwnerOther
}
case chain == userChain:
f.Owner = OwnerOther
case c.dropping && (r.managed[table] || iptablesTable(table)) && runtimes(table, chain, c.policyLine):
f.Owner = OwnerRuntime
case len(rules) > 0 && (r.managed[table] || iptablesTable(table)) && allRuntimes(table, chain, rules):
f.Owner = OwnerRuntime
case len(rules) > 0 && allBans(r, rules):
f.Owner = OwnerBan
case c.dropping && !iptablesTable(table) && !r.managed[table] && len(rules) == 0:
// A table of its own whose base chain drops by policy: a firewall nobody declared.
f.Owner = OwnerOther
default:
f.Owner = OwnerOther
}
if ufwActive && (r.managed[table] || iptablesTable(table)) && f.Owner == OwnerOther && len(rules) == 0 && c.dropping {
// A base chain ufw set to drop while it is in force is ufw's.
f.Owner = OwnerFoundFirewall
}
f.Refuses = refusesLine(c, rules)
out = append(out, f)
}
tools := make([]string, 0, len(legacy))
for tool := range legacy {
tools = append(tools, tool)
}
sort.Strings(tools)
for _, tool := range tools {
out = append(out, legacyFilters(legacy[tool], tool, ufwActive)...)
}
return out
}
// allRuntimes is whether every refusal in a chain is the runtime's own.
func allRuntimes(table, chain string, rules []nftRule) bool {
for _, rule := range rules {
if !runtimes(table, chain, rule.line) {
return false
}
}
return true
}
// allBans is whether every refusal in a chain only bans the sources it names.
func allBans(r *nftRuleset, rules []nftRule) bool {
for _, rule := range rules {
if !r.onlyBans(rule) {
return false
}
}
return true
}
var counters = regexp.MustCompile(`\s*counter packets \d+ bytes \d+`)
// refusesLine is one line a person reads: the policy when the chain drops by policy, else the first
// refusing rule with its counters stripped, and how many more there are.
func refusesLine(c *nftChain, rules []nftRule) string {
var parts []string
if c.dropping {
parts = append(parts, "policy drop")
}
if len(rules) > 0 {
line := strings.TrimSpace(counters.ReplaceAllString(rules[0].line, ""))
if len(rules) > 1 {
line += fmt.Sprintf(" (and %d more)", len(rules)-1)
}
parts = append(parts, line)
}
return strings.Join(parts, "; ")
}
// legacyFilters classifies the chains of an `iptables-legacy -S` listing that refuse.
func legacyFilters(rules, tool string, ufwActive bool) []Filter {
policy := map[string]string{}
accepting := map[string]bool{}
jumpedFrom := map[string][]string{}
for _, line := range strings.Split(rules, "\n") {
fields := strings.Fields(line)
if len(fields) < 3 {
continue
}
switch fields[0] {
case "-P":
policy[fields[1]] = fields[2]
case "-A":
for i, f := range fields {
if (f == "-j" || f == "-g") && i+1 < len(fields) {
switch fields[i+1] {
case "ACCEPT":
accepting[fields[1]] = true
case "DROP", "REJECT", "RETURN", "LOG":
default:
jumpedFrom[fields[i+1]] = append(jumpedFrom[fields[i+1]], fields[1])
}
}
}
}
}
var entered func(chain string, seen map[string]bool) bool
entered = func(chain string, seen map[string]bool) bool {
if seen[chain] || accepting[chain] || len(jumpedFrom[chain]) == 0 {
return false
}
seen[chain] = true
for _, from := range jumpedFrom[chain] {
if p, builtIn := policy[from]; builtIn {
if p != "ACCEPT" {
return false
}
continue
}
if !entered(from, seen) {
return false
}
}
return true
}
ban := func(chain, line string) bool {
return bansSources(line) && entered(chain, map[string]bool{})
}
type seen struct {
owner string
lines []string
}
chains := map[string]*seen{}
var order []string
note := func(chain, owner, line string) {
s := chains[chain]
if s == nil {
s = &seen{owner: owner}
chains[chain] = s
order = append(order, chain)
}
if owner == OwnerOther || s.owner == "" {
s.owner = owner
}
s.lines = append(s.lines, line)
}
for _, line := range strings.Split(rules, "\n") {
fields := strings.Fields(line)
if len(fields) < 3 {
continue
}
chain := fields[1]
switch fields[0] {
case "-P":
if fields[2] != "DROP" {
continue
}
owner := OwnerOther
if chain == "FORWARD" {
owner = OwnerRuntime
}
if ufwActive {
owner = OwnerFoundFirewall
}
note(chain, owner, "policy DROP")
case "-A":
refuses := false
for i, f := range fields {
if f == "-j" && i+1 < len(fields) && (fields[i+1] == "DROP" || fields[i+1] == "REJECT") {
refuses = true
}
}
if !refuses {
continue
}
owner := OwnerOther
switch {
case strings.HasPrefix(chain, "ufw"):
owner = OwnerFoundFirewall
if !ufwActive {
owner = OwnerOther
}
case chain != userChain && strings.HasPrefix(chain, "DOCKER"):
owner = OwnerRuntime
case ban(chain, line):
owner = OwnerBan
}
note(chain, owner, strings.TrimSpace(line))
}
}
var out []Filter
for _, chain := range order {
s := chains[chain]
refuses := s.lines[0]
if len(s.lines) > 1 {
refuses += fmt.Sprintf(" (and %d more)", len(s.lines)-1)
}
out = append(out, Filter{Where: "chain " + chain + " (" + tool + ")", Owner: s.owner, Refuses: refuses})
}
return out
}
// Collect reads what filters this machine now: its nftables ruleset and, where the legacy tools
// exist, their listings. A machine without nft is read through iptables, as Detect reads it.
func Collect(ctx context.Context, run Runner, ufwActive bool) ([]Filter, error) {
ruleset := ""
noNft := false
out, err := run(ctx, "nft", "list", "ruleset")
switch {
case err == nil:
ruleset = out
case missing(err):
noNft = true
default:
return nil, fmt.Errorf("cannot read this machine's packet filter: %w", err)
}
legacy := map[string]string{}
tools := []string{"iptables-legacy", "ip6tables-legacy"}
if noNft {
tools = append(tools, "iptables", "ip6tables")
}
for _, tool := range tools {
if out, err := run(ctx, tool, "-S"); err == nil && strings.TrimSpace(out) != "" {
legacy[tool] = out
}
}
return Filters(ruleset, legacy, ufwActive), nil
}
// Alone is whether a machine is filtered by the mesh alone: nothing in the list but the mesh's
// own tables, the runtime's plumbing and bans (novox/hq ADR 0168).
func Alone(filters []Filter) bool {
for _, f := range filters {
if f.Owner == OwnerOther || f.Owner == OwnerFoundFirewall {
return false
}
}
return true
}
// Active says whether ufw is in force on this machine now. A machine without ufw is not.
func Active(ctx context.Context, run Runner) bool {
out, err := run(ctx, "ufw", "status")
return err == nil && statusActive(out)
}
// Installed says whether ufw is on this machine at all: a command that is not there is a front end
// that was uninstalled (novox/hq ADR 0175), not one that is silent.
func Installed(ctx context.Context, run Runner) bool {
_, err := run(ctx, "ufw", "status")
return !missing(err)
}
// Retirements of a found firewall, as the host records them.
const (
RetiredByMesh = "mesh"
RetiredFoundSo = "found-inactive"
// RetiredRemoved is a front end uninstalled by the module that replaced it (ADR 0175).
RetiredRemoved = "removed"
)
+130
View File
@@ -0,0 +1,130 @@
package firewall
import (
"os"
"strings"
"testing"
)
func fixture(t *testing.T, name string) string {
t.Helper()
raw, err := os.ReadFile("testdata/" + name)
if err != nil {
t.Fatal(err)
}
return string(raw)
}
func ownerOf(filters []Filter, where string) string {
for _, f := range filters {
if f.Where == where {
return f.Owner
}
}
return "(not reported)"
}
// Every refusing table and chain is classified with an owner (novox/hq ADR 0168), over rulesets
// captured from three machines of the first mesh. The control node: a ban list reached through the
// runtime's user chain is a ban; a refusal left in that chain, and a chain a retired front end left
// behind, are *other*; the runtime's own and the mesh's own are theirs.
func TestTheControlNodesRefusalsAreClassified(t *testing.T) {
got := Filters(fixture(t, "control-node.nft"), nil, false)
for where, want := range map[string]string{
"table ip filter, chain f2b-recidive": OwnerBan,
"table ip filter, chain DOCKER": OwnerRuntime,
"table ip raw, chain PREROUTING": OwnerRuntime,
"table inet mesh, chain input": OwnerMesh,
"table inet mesh, chain forward": OwnerMesh,
"table ip6 filter, chain DOCKER-USER": OwnerOther,
"table ip6 filter, chain ufw6-docker-logging-deny": OwnerOther,
} {
if o := ownerOf(got, where); o != want {
t.Errorf("%s: %s, want %s", where, o, want)
}
}
if Alone(got) {
t.Error("a machine with a refusal in the runtime's user chain reads as filtered by the mesh alone")
}
// What refuses adoption does not move (rule 4): the user chain's refusals are reported, not
// refused. The chain a retired front end left behind, still dropping, is what it always was
// to Detect — a refusal nobody speaks for, in one table.
if refusing := Refusing(fixture(t, "control-node.nft"), false); len(refusing) != 1 || refusing[0] != "table ip6 filter" {
t.Errorf("adoption's threshold moved: %v", refusing)
}
// The counters are stripped from what a person reads.
for _, f := range got {
if strings.Contains(f.Refuses, "counter packets") {
t.Errorf("counters in the line: %s", f.Refuses)
}
}
}
// The laptop: the runtime's forward policy and bridge guards, a virtualisation host and an endpoint
// agent that refuse nothing, and the mesh — filtered by the mesh alone.
func TestTheLaptopIsFilteredByTheMeshAlone(t *testing.T) {
got := Filters(fixture(t, "laptop.nft"), nil, false)
for where, want := range map[string]string{
"table ip filter, chain FORWARD": OwnerRuntime,
"table ip filter, chain DOCKER": OwnerRuntime,
"table ip raw, chain PREROUTING": OwnerRuntime,
"table inet mesh, chain input": OwnerMesh,
} {
if o := ownerOf(got, where); o != want {
t.Errorf("%s: %s, want %s", where, o, want)
}
}
for _, f := range got {
if strings.Contains(f.Where, "incus") || strings.Contains(f.Where, "fct_") {
t.Errorf("a table that refuses nothing is reported: %+v", f)
}
}
if !Alone(got) {
t.Errorf("the laptop is not read as filtered by the mesh alone: %+v", got)
}
}
// The home server: its rules are in the legacy filter, where a predecessor's chain still drops what
// arrives on the outward link for the forwarded path — invisible to the mesh until now (issue 144).
func TestThePredecessorsChainInTheLegacyFilterIsOther(t *testing.T) {
mesh := "table inet mesh {\n\tchain forward {\n\t\ttype filter hook forward priority filter; policy drop;\n\t}\n}\n"
got := Filters(mesh, map[string]string{"iptables-legacy": fixture(t, "home-server-legacy-S.txt")}, false)
for where, want := range map[string]string{
"table inet mesh, chain forward": OwnerMesh,
"chain FORWARD (iptables-legacy)": OwnerRuntime,
"chain DOCKER (iptables-legacy)": OwnerRuntime,
"chain HAL-MESH-ONLY (iptables-legacy)": OwnerOther,
} {
if o := ownerOf(got, where); o != want {
t.Errorf("%s: %s, want %s", where, o, want)
}
}
var other Filter
for _, f := range got {
if f.Owner == OwnerOther {
other = f
}
}
if !strings.Contains(other.Refuses, "-j DROP") {
t.Errorf("what the predecessor's chain refuses is not said: %+v", other)
}
if Alone(got) {
t.Error("a machine with a predecessor's chain reads as filtered by the mesh alone")
}
}
// With the front end in force, its chains are its own; retired, a chain it left behind that still
// refuses is nobody's and said so.
func TestAFrontEndsChainsAreItsWhileItIsInForce(t *testing.T) {
ruleset := dockerOnly(t) + ufwChains
for _, f := range Filters(ruleset, nil, true) {
if strings.Contains(f.Where, "ufw") && f.Owner != OwnerFoundFirewall {
t.Errorf("active: %+v", f)
}
}
for _, f := range Filters(ruleset, nil, false) {
if strings.Contains(f.Where, "ufw") && f.Owner != OwnerOther {
t.Errorf("retired: %+v", f)
}
}
}
+92 -71
View File
@@ -128,42 +128,82 @@ func statusActive(out string) bool {
// mesh needs, so a refusal that names the sources it refuses, in a table or a chain that accepts
// nothing and is entered only from chains whose policy accepts, is not counted.
func Refusing(ruleset string, ufwActive bool) []string {
type rule struct{ table, chain, line string }
type chainOf struct {
var refusing []string
for _, f := range Filters(ruleset, nil, ufwActive) {
if f.Owner != OwnerOther || f.chain == userChain {
// A refusal in the runtime's user chain is reported as *other* and does not refuse
// adoption (novox/hq ADR 0168, rule 4): both predecessors kept their rules there.
continue
}
name := "table " + f.table
if len(refusing) == 0 || refusing[len(refusing)-1] != name {
if !contains(refusing, name) {
refusing = append(refusing, name)
}
}
}
return refusing
}
func contains(list []string, s string) bool {
for _, x := range list {
if x == s {
return true
}
}
return false
}
// nftRule is one line of a ruleset that refuses, with where it is.
type nftRule struct{ table, chain, line string }
// nftChain is what a parse knows about one chain.
type nftChain struct {
base, dropping, accepts bool
policyLine string
jumpedFrom []string
}
chains := map[string]*chainOf{} // by "table\x00chain"
tableAccepts := map[string]bool{}
var tables []string
var refusals []rule
managed := map[string]bool{}
var table, chain string
get := func(t, c string) *chainOf {
}
// nftRuleset is `nft list ruleset`, read: its tables in order, its chains, every refusing line,
// and which tables iptables-nft manages.
type nftRuleset struct {
tables []string
chains map[string]*nftChain // by "table\x00chain"
chainOrder []string
tableAccepts map[string]bool
refusals []nftRule
managed map[string]bool
}
func (r *nftRuleset) get(t, c string) *nftChain {
k := t + "\x00" + c
if chains[k] == nil {
chains[k] = &chainOf{}
}
return chains[k]
if r.chains[k] == nil {
r.chains[k] = &nftChain{}
r.chainOrder = append(r.chainOrder, k)
}
return r.chains[k]
}
func parseNft(ruleset string) *nftRuleset {
r := &nftRuleset{chains: map[string]*nftChain{}, tableAccepts: map[string]bool{}, managed: map[string]bool{}}
var table, chain string
for _, raw := range strings.Split(ruleset, "\n") {
line := strings.TrimSpace(raw)
switch {
case strings.HasPrefix(line, "# Warning: table ") && strings.Contains(line, "managed by iptables-nft"):
name := strings.TrimPrefix(line, "# Warning: table ")
name, _, _ = strings.Cut(name, " is managed")
managed[name] = true
r.managed[name] = true
continue
case strings.HasPrefix(line, "table "):
table = strings.TrimSuffix(strings.TrimSpace(strings.TrimPrefix(line, "table ")), "{")
table = strings.TrimSpace(table)
tables = append(tables, table)
r.tables = append(r.tables, table)
chain = ""
continue
case strings.HasPrefix(line, "chain "):
chain = strings.TrimSpace(strings.TrimSuffix(strings.TrimPrefix(line, "chain "), "{"))
get(table, chain)
r.get(table, chain)
continue
case strings.HasPrefix(line, "set ") || strings.HasPrefix(line, "map ") ||
strings.HasPrefix(line, "flowtable "):
@@ -172,7 +212,7 @@ func Refusing(ruleset string, ufwActive bool) []string {
case line == "" || line == "}" || strings.HasPrefix(line, "#") || chain == "":
continue
}
c := get(table, chain)
c := r.get(table, chain)
if strings.HasPrefix(line, "type ") {
c.base = true
c.policyLine = line
@@ -183,85 +223,66 @@ func Refusing(ruleset string, ufwActive bool) []string {
if i := strings.Index(line, verb); i >= 0 {
target := strings.Fields(line[i+len(verb):])
if len(target) > 0 {
get(table, target[0]).jumpedFrom = append(get(table, target[0]).jumpedFrom, chain)
r.get(table, target[0]).jumpedFrom = append(r.get(table, target[0]).jumpedFrom, chain)
}
}
}
if accepts(line) {
c.accepts = true
tableAccepts[table] = true
r.tableAccepts[table] = true
}
if verdictRefuses(line) {
refusals = append(refusals, rule{table, chain, line})
r.refusals = append(r.refusals, nftRule{table, chain, line})
}
}
return r
}
skipped := func(table string) bool {
if table == "inet mesh" || table == "inet mesh_guard" {
return true
}
return (managed[table] || iptablesTable(table)) && ufwActive
}
// onlyBans is whether a refusal only refuses the sources it names: in a table that accepts
// nothing and whose base chains all accept by default, or in a chain that accepts nothing and
// is entered only from base chains that accept by default.
onlyBans := func(r rule) bool {
if !bansSources(r.line) {
// onlyBans is whether a refusal only refuses the sources it names: in a table that accepts nothing
// and whose base chains all accept by default, or in a chain that accepts nothing and is entered
// only from base chains that accept by default.
func (r *nftRuleset) onlyBans(rule nftRule) bool {
if !bansSources(rule.line) {
return false
}
allAccepting := true
for k, c := range chains {
if strings.HasPrefix(k, r.table+"\x00") && c.base && !strings.Contains(c.policyLine, "policy accept") {
for k, c := range r.chains {
if strings.HasPrefix(k, rule.table+"\x00") && c.base && !strings.Contains(c.policyLine, "policy accept") {
allAccepting = false
}
}
if !tableAccepts[r.table] && allAccepting {
if !r.tableAccepts[rule.table] && allAccepting {
return true
}
c := get(r.table, r.chain)
return r.enteredAccepting(rule.table, rule.chain, map[string]bool{})
}
// enteredAccepting is whether a chain accepts nothing and is entered only through chains that
// accept by default — base chains whose policy accepts, or chains that are themselves entered that
// way and accept nothing. A ban list jumped to from the runtime's user chain, which the forward
// chain enters with an accepting policy, is still a ban list.
func (r *nftRuleset) enteredAccepting(table, chain string, seen map[string]bool) bool {
if seen[chain] {
return false
}
seen[chain] = true
c := r.get(table, chain)
if c.base || c.accepts || len(c.jumpedFrom) == 0 {
return false
}
for _, from := range c.jumpedFrom {
caller := get(r.table, from)
if !caller.base || !strings.Contains(caller.policyLine, "policy accept") {
caller := r.get(table, from)
if caller.base {
if !strings.Contains(caller.policyLine, "policy accept") {
return false
}
continue
}
if caller.accepts || !r.enteredAccepting(table, from, seen) {
return false
}
}
return true
}
counted := map[string]bool{}
for k, c := range chains {
t, name, _ := strings.Cut(k, "\x00")
if skipped(t) || !c.dropping {
continue
}
if (managed[t] || iptablesTable(t)) && runtimes(t, name, c.policyLine) {
continue
}
counted[t] = true
}
for _, r := range refusals {
if skipped(r.table) || counted[r.table] {
continue
}
if (managed[r.table] || iptablesTable(r.table)) && runtimes(r.table, r.chain, r.line) {
continue
}
if onlyBans(r) {
continue
}
counted[r.table] = true
}
var refusing []string
for _, t := range tables {
if counted[t] {
counted[t] = false
refusing = append(refusing, "table "+t)
}
}
return refusing
}
// iptablesTable is whether a table is one iptables-nft writes. Named rather than read from the
+588
View File
@@ -0,0 +1,588 @@
# Warning: table ip filter is managed by iptables-nft, do not touch!
table ip filter {
chain INPUT {
type filter hook input priority filter; policy accept;
ip protocol tcp counter packets 945757787 bytes 1737008792038 jump f2b-sshd
ip protocol tcp counter packets 945756610 bytes 1737008898620 jump f2b-recidive
counter packets 2862213204 bytes 3144751431654 jump ufw-before-logging-input
counter packets 2862213204 bytes 3144751431654 jump ufw-before-input
counter packets 989333889 bytes 1776344988272 jump ufw-after-input
counter packets 989303248 bytes 1776343408920 jump ufw-after-logging-input
counter packets 989303248 bytes 1776343408920 jump ufw-reject-input
counter packets 989303248 bytes 1776343408920 jump ufw-track-input
}
chain FORWARD {
type filter hook forward priority filter; policy accept;
oifname "mesh0" counter packets 1613103 bytes 2577614868 accept
iifname "mesh0" counter packets 995195 bytes 84526284 accept
counter packets 20454697 bytes 11504107676 jump DOCKER-USER
counter packets 20442192 bytes 11503368404 jump DOCKER-FORWARD
counter packets 12438285 bytes 10907281833 jump ufw-before-logging-forward
counter packets 12438285 bytes 10907281833 jump ufw-before-forward
counter packets 384 bytes 39643 jump ufw-after-forward
counter packets 384 bytes 39643 jump ufw-after-logging-forward
counter packets 384 bytes 39643 jump ufw-reject-forward
counter packets 384 bytes 39643 jump ufw-track-forward
}
chain OUTPUT {
type filter hook output priority filter; policy accept;
counter packets 3195070897 bytes 3951725261199 jump ufw-before-logging-output
counter packets 3195070897 bytes 3951725261199 jump ufw-before-output
counter packets 945745931 bytes 1778546547406 jump ufw-after-output
counter packets 945745931 bytes 1778546547406 jump ufw-after-logging-output
counter packets 945745931 bytes 1778546547406 jump ufw-reject-output
counter packets 945745931 bytes 1778546547406 jump ufw-track-output
}
chain DOCKER-FORWARD {
counter packets 20442192 bytes 11503368404 jump DOCKER-CT
counter packets 8079126 bytes 1230017985 jump DOCKER-INTERNAL
counter packets 8079126 bytes 1230017985 jump DOCKER-BRIDGE
iifname "br-cadedce55fe9" counter packets 0 bytes 0 accept
iifname "br-dd007c7e67bc" counter packets 0 bytes 0 accept
iifname "br-a5fbc29c2c2a" counter packets 0 bytes 0 accept
iifname "br-6eb1e7f7f847" counter packets 0 bytes 0 accept
iifname "br-8ce143481a5b" counter packets 14700 bytes 2493600 accept
iifname "br-84e7d0cfeada" counter packets 0 bytes 0 accept
iifname "br-f8b083119d99" counter packets 264 bytes 57438 accept
iifname "br-0d1490cc67c9" counter packets 732468 bytes 351624109 accept
iifname "br-3b338a381229" counter packets 137 bytes 11876 accept
iifname "br-3008d408e73a" counter packets 25380 bytes 1564417 accept
iifname "br-ca07a9577a7f" counter packets 0 bytes 0 accept
iifname "docker0" counter packets 6695791 bytes 795364128 accept
iifname "br-a63fa64a9e18" counter packets 0 bytes 0 accept
iifname "br-1ccb887b3344" counter packets 237174 bytes 36804979 accept
iifname "br-9fd22324ec08" counter packets 0 bytes 0 accept
iifname "br-73641cceafc3" counter packets 36 bytes 6614 accept
iifname "br-77eb8a9e2ba1" counter packets 0 bytes 0 accept
iifname "br-e99ce5248c84" counter packets 0 bytes 0 accept
iifname "br-e5d78502832d" counter packets 0 bytes 0 accept
iifname "br-2e4a76a7cd2e" counter packets 20339 bytes 1799711 accept
iifname "br-72fd626a8ff7" counter packets 0 bytes 0 accept
}
chain DOCKER-USER {
ip protocol tcp counter packets 3054221 bytes 3329963005 jump f2b-sshd
ip protocol tcp counter packets 3054221 bytes 3329963005 jump f2b-recidive
counter packets 1421378091 bytes 2045004412819 return
}
chain ufw-before-logging-input {
}
chain ufw-before-logging-output {
}
chain ufw-before-logging-forward {
}
chain ufw-before-input {
}
chain ufw-before-output {
}
chain ufw-before-forward {
}
chain ufw-after-input {
}
chain ufw-after-output {
}
chain ufw-after-forward {
}
chain ufw-after-logging-input {
}
chain ufw-after-logging-output {
}
chain ufw-after-logging-forward {
}
chain ufw-reject-input {
}
chain ufw-reject-output {
}
chain ufw-reject-forward {
}
chain ufw-track-input {
}
chain ufw-track-output {
}
chain ufw-track-forward {
}
chain DOCKER {
ip daddr 172.17.0.7 iifname != "docker0" oifname "docker0" tcp dport 5432 counter packets 8 bytes 480 accept
ip daddr 172.19.0.2 iifname != "br-72fd626a8ff7" oifname "br-72fd626a8ff7" tcp dport 8080 counter packets 0 bytes 0 accept
ip daddr 172.17.0.6 iifname != "docker0" oifname "docker0" tcp dport 9443 counter packets 0 bytes 0 accept
ip daddr 172.17.0.6 iifname != "docker0" oifname "docker0" tcp dport 9000 counter packets 0 bytes 0 accept
ip daddr 192.168.176.2 iifname != "br-f8b083119d99" oifname "br-f8b083119d99" tcp dport 9001 counter packets 0 bytes 0 accept
ip daddr 192.168.176.2 iifname != "br-f8b083119d99" oifname "br-f8b083119d99" tcp dport 9000 counter packets 47769 bytes 2866140 accept
ip daddr 172.20.0.2 iifname != "br-6eb1e7f7f847" oifname "br-6eb1e7f7f847" tcp dport 8080 counter packets 0 bytes 0 accept
ip daddr 172.27.0.2 iifname != "br-3008d408e73a" oifname "br-3008d408e73a" tcp dport 3000 counter packets 0 bytes 0 accept
ip daddr 192.168.48.5 iifname != "br-77eb8a9e2ba1" oifname "br-77eb8a9e2ba1" tcp dport 80 counter packets 0 bytes 0 accept
ip daddr 192.168.48.4 iifname != "br-77eb8a9e2ba1" oifname "br-77eb8a9e2ba1" tcp dport 80 counter packets 0 bytes 0 accept
ip daddr 192.168.48.3 iifname != "br-77eb8a9e2ba1" oifname "br-77eb8a9e2ba1" tcp dport 80 counter packets 0 bytes 0 accept
ip daddr 192.168.48.2 iifname != "br-77eb8a9e2ba1" oifname "br-77eb8a9e2ba1" tcp dport 9000 counter packets 0 bytes 0 accept
ip daddr 172.18.0.2 iifname != "br-2e4a76a7cd2e" oifname "br-2e4a76a7cd2e" tcp dport 80 counter packets 0 bytes 0 accept
ip daddr 172.17.0.5 iifname != "docker0" oifname "docker0" tcp dport 80 counter packets 0 bytes 0 accept
ip daddr 172.17.0.3 iifname != "docker0" oifname "docker0" tcp dport 8222 counter packets 0 bytes 0 accept
ip daddr 172.17.0.3 iifname != "docker0" oifname "docker0" tcp dport 4222 counter packets 97 bytes 5744 accept
ip daddr 172.28.0.2 iifname != "br-8ce143481a5b" oifname "br-8ce143481a5b" tcp dport 1433 counter packets 0 bytes 0 accept
ip daddr 192.168.80.2 iifname != "br-e99ce5248c84" oifname "br-e99ce5248c84" tcp dport 8080 counter packets 0 bytes 0 accept
ip daddr 192.168.112.3 iifname != "br-e5d78502832d" oifname "br-e5d78502832d" tcp dport 9000 counter packets 0 bytes 0 accept
ip daddr 192.168.112.2 iifname != "br-e5d78502832d" oifname "br-e5d78502832d" tcp dport 80 counter packets 0 bytes 0 accept
ip daddr 192.168.128.2 iifname != "br-73641cceafc3" oifname "br-73641cceafc3" tcp dport 27017 counter packets 14 bytes 840 accept
ip daddr 192.168.208.2 iifname != "br-9fd22324ec08" oifname "br-9fd22324ec08" tcp dport 35621 counter packets 0 bytes 0 accept
ip daddr 192.168.203.13 iifname != "br-1ccb887b3344" oifname "br-1ccb887b3344" tcp dport 4243 counter packets 0 bytes 0 accept
ip daddr 192.168.203.12 iifname != "br-1ccb887b3344" oifname "br-1ccb887b3344" tcp dport 995 counter packets 194 bytes 11000 accept
ip daddr 192.168.203.12 iifname != "br-1ccb887b3344" oifname "br-1ccb887b3344" tcp dport 993 counter packets 188 bytes 9394 accept
ip daddr 192.168.203.12 iifname != "br-1ccb887b3344" oifname "br-1ccb887b3344" tcp dport 587 counter packets 444 bytes 23312 accept
ip daddr 192.168.203.12 iifname != "br-1ccb887b3344" oifname "br-1ccb887b3344" tcp dport 465 counter packets 104 bytes 5852 accept
ip daddr 192.168.203.12 iifname != "br-1ccb887b3344" oifname "br-1ccb887b3344" tcp dport 443 counter packets 0 bytes 0 accept
ip daddr 192.168.203.12 iifname != "br-1ccb887b3344" oifname "br-1ccb887b3344" tcp dport 143 counter packets 443 bytes 25280 accept
ip daddr 192.168.203.12 iifname != "br-1ccb887b3344" oifname "br-1ccb887b3344" tcp dport 110 counter packets 192 bytes 9561 accept
ip daddr 192.168.203.12 iifname != "br-1ccb887b3344" oifname "br-1ccb887b3344" tcp dport 80 counter packets 0 bytes 0 accept
ip daddr 192.168.203.12 iifname != "br-1ccb887b3344" oifname "br-1ccb887b3344" tcp dport 25 counter packets 430 bytes 22919 accept
ip daddr 172.17.0.2 iifname != "docker0" oifname "docker0" tcp dport 3000 counter packets 0 bytes 0 accept
ip daddr 172.17.0.2 iifname != "docker0" oifname "docker0" tcp dport 22 counter packets 1578 bytes 93884 accept
ip daddr 172.17.0.4 iifname != "docker0" oifname "docker0" tcp dport 5000 counter packets 25 bytes 1492 accept
iifname != "br-cadedce55fe9" oifname "br-cadedce55fe9" counter packets 0 bytes 0 drop
iifname != "br-dd007c7e67bc" oifname "br-dd007c7e67bc" counter packets 0 bytes 0 drop
iifname != "br-a5fbc29c2c2a" oifname "br-a5fbc29c2c2a" counter packets 0 bytes 0 drop
iifname != "br-6eb1e7f7f847" oifname "br-6eb1e7f7f847" counter packets 0 bytes 0 drop
iifname != "br-8ce143481a5b" oifname "br-8ce143481a5b" counter packets 0 bytes 0 drop
iifname != "br-84e7d0cfeada" oifname "br-84e7d0cfeada" counter packets 0 bytes 0 drop
iifname != "br-f8b083119d99" oifname "br-f8b083119d99" counter packets 0 bytes 0 drop
iifname != "br-0d1490cc67c9" oifname "br-0d1490cc67c9" counter packets 0 bytes 0 drop
iifname != "br-3b338a381229" oifname "br-3b338a381229" counter packets 0 bytes 0 drop
iifname != "br-3008d408e73a" oifname "br-3008d408e73a" counter packets 0 bytes 0 drop
iifname != "br-ca07a9577a7f" oifname "br-ca07a9577a7f" counter packets 0 bytes 0 drop
iifname != "docker0" oifname "docker0" counter packets 0 bytes 0 drop
iifname != "br-a63fa64a9e18" oifname "br-a63fa64a9e18" counter packets 0 bytes 0 drop
iifname != "br-1ccb887b3344" oifname "br-1ccb887b3344" counter packets 0 bytes 0 drop
iifname != "br-9fd22324ec08" oifname "br-9fd22324ec08" counter packets 0 bytes 0 drop
iifname != "br-73641cceafc3" oifname "br-73641cceafc3" counter packets 0 bytes 0 drop
iifname != "br-77eb8a9e2ba1" oifname "br-77eb8a9e2ba1" counter packets 0 bytes 0 drop
iifname != "br-e99ce5248c84" oifname "br-e99ce5248c84" counter packets 0 bytes 0 drop
iifname != "br-e5d78502832d" oifname "br-e5d78502832d" counter packets 0 bytes 0 drop
iifname != "br-2e4a76a7cd2e" oifname "br-2e4a76a7cd2e" counter packets 0 bytes 0 drop
iifname != "br-72fd626a8ff7" oifname "br-72fd626a8ff7" counter packets 0 bytes 0 drop
}
chain DOCKER-BRIDGE {
oifname "br-cadedce55fe9" counter packets 0 bytes 0 jump DOCKER
oifname "br-dd007c7e67bc" counter packets 0 bytes 0 jump DOCKER
oifname "br-a5fbc29c2c2a" counter packets 0 bytes 0 jump DOCKER
oifname "br-6eb1e7f7f847" counter packets 799 bytes 47940 jump DOCKER
oifname "br-8ce143481a5b" counter packets 0 bytes 0 jump DOCKER
oifname "br-84e7d0cfeada" counter packets 0 bytes 0 jump DOCKER
oifname "br-f8b083119d99" counter packets 98911 bytes 5934660 jump DOCKER
oifname "br-0d1490cc67c9" counter packets 69740 bytes 4118476 jump DOCKER
oifname "br-3b338a381229" counter packets 32 bytes 1920 jump DOCKER
oifname "br-3008d408e73a" counter packets 458 bytes 27480 jump DOCKER
oifname "br-ca07a9577a7f" counter packets 0 bytes 0 jump DOCKER
oifname "docker0" counter packets 87073 bytes 5223529 jump DOCKER
oifname "br-a63fa64a9e18" counter packets 1353 bytes 81180 jump DOCKER
oifname "br-1ccb887b3344" counter packets 7662 bytes 419862 jump DOCKER
oifname "br-9fd22324ec08" counter packets 173 bytes 10380 jump DOCKER
oifname "br-73641cceafc3" counter packets 162 bytes 9720 jump DOCKER
oifname "br-77eb8a9e2ba1" counter packets 94 bytes 5640 jump DOCKER
oifname "br-e99ce5248c84" counter packets 8 bytes 480 jump DOCKER
oifname "br-e5d78502832d" counter packets 26 bytes 1560 jump DOCKER
oifname "br-2e4a76a7cd2e" counter packets 7 bytes 420 jump DOCKER
oifname "br-72fd626a8ff7" counter packets 0 bytes 0 jump DOCKER
}
chain DOCKER-CT {
oifname "br-cadedce55fe9" xt match "conntrack" counter packets 0 bytes 0 accept
oifname "br-dd007c7e67bc" xt match "conntrack" counter packets 0 bytes 0 accept
oifname "br-a5fbc29c2c2a" xt match "conntrack" counter packets 0 bytes 0 accept
oifname "br-6eb1e7f7f847" xt match "conntrack" counter packets 38458 bytes 6234236 accept
oifname "br-8ce143481a5b" xt match "conntrack" counter packets 60403 bytes 20478794 accept
oifname "br-84e7d0cfeada" xt match "conntrack" counter packets 0 bytes 0 accept
oifname "br-f8b083119d99" xt match "conntrack" counter packets 1008024 bytes 206364436 accept
oifname "br-0d1490cc67c9" xt match "conntrack" counter packets 871134 bytes 1416174426 accept
oifname "br-3b338a381229" xt match "conntrack" counter packets 4649 bytes 2311375 accept
oifname "br-3008d408e73a" xt match "conntrack" counter packets 13415 bytes 1974731 accept
oifname "br-ca07a9577a7f" xt match "conntrack" counter packets 0 bytes 0 accept
oifname "docker0" xt match "conntrack" counter packets 8909862 bytes 7892163419 accept
oifname "br-a63fa64a9e18" xt match "conntrack" counter packets 16688 bytes 6822829 accept
oifname "br-1ccb887b3344" xt match "conntrack" counter packets 461453 bytes 141913887 accept
oifname "br-9fd22324ec08" xt match "conntrack" counter packets 1677 bytes 427538 accept
oifname "br-73641cceafc3" xt match "conntrack" counter packets 417619 bytes 35343624 accept
oifname "br-77eb8a9e2ba1" xt match "conntrack" counter packets 125437 bytes 94155193 accept
oifname "br-e99ce5248c84" xt match "conntrack" counter packets 91 bytes 19173 accept
oifname "br-e5d78502832d" xt match "conntrack" counter packets 128653 bytes 40539569 accept
oifname "br-2e4a76a7cd2e" xt match "conntrack" counter packets 20257 bytes 158631592 accept
oifname "br-72fd626a8ff7" xt match "conntrack" counter packets 0 bytes 0 accept
}
chain DOCKER-INTERNAL {
}
chain f2b-recidive {
ip saddr 2.57.122.209 counter packets 0 bytes 0 xt target "REJECT"
ip saddr 2.57.122.76 counter packets 127 bytes 7600 xt target "REJECT"
ip saddr 195.178.110.228 counter packets 17 bytes 1000 xt target "REJECT"
ip saddr 2.57.122.74 counter packets 11 bytes 620 xt target "REJECT"
ip saddr 195.178.110.26 counter packets 56 bytes 3360 xt target "REJECT"
ip saddr 92.118.39.77 counter packets 2 bytes 80 xt target "REJECT"
ip saddr 92.118.39.71 counter packets 1 bytes 40 xt target "REJECT"
ip saddr 45.148.10.240 counter packets 0 bytes 0 xt target "REJECT"
ip saddr 195.178.110.30 counter packets 8 bytes 320 xt target "REJECT"
counter packets 948810608 bytes 1740338848565 return
}
chain f2b-sshd {
counter packets 948810709 bytes 1740338655735 return
}
}
# Warning: table ip6 filter is managed by iptables-nft, do not touch!
table ip6 filter {
chain INPUT {
type filter hook input priority filter; policy accept;
counter packets 5426360 bytes 34419159588 jump ufw6-before-logging-input
counter packets 5426360 bytes 34419159588 jump ufw6-before-input
counter packets 367982 bytes 3415126642 jump ufw6-after-input
counter packets 367982 bytes 3415126642 jump ufw6-after-logging-input
counter packets 367982 bytes 3415126642 jump ufw6-reject-input
counter packets 367982 bytes 3415126642 jump ufw6-track-input
}
chain FORWARD {
type filter hook forward priority filter; policy accept;
counter packets 0 bytes 0 jump DOCKER-USER
counter packets 0 bytes 0 jump DOCKER-FORWARD
counter packets 0 bytes 0 jump ufw6-before-logging-forward
counter packets 0 bytes 0 jump ufw6-before-forward
counter packets 0 bytes 0 jump ufw6-after-forward
counter packets 0 bytes 0 jump ufw6-after-logging-forward
counter packets 0 bytes 0 jump ufw6-reject-forward
counter packets 0 bytes 0 jump ufw6-track-forward
}
chain OUTPUT {
type filter hook output priority filter; policy accept;
counter packets 6004354 bytes 1866587952 jump ufw6-before-logging-output
counter packets 6004354 bytes 1866587952 jump ufw6-before-output
counter packets 2241898 bytes 639173314 jump ufw6-after-output
counter packets 2241898 bytes 639173314 jump ufw6-after-logging-output
counter packets 2241898 bytes 639173314 jump ufw6-reject-output
counter packets 2241898 bytes 639173314 jump ufw6-track-output
}
chain DOCKER-FORWARD {
counter packets 0 bytes 0 jump DOCKER-CT
counter packets 0 bytes 0 jump DOCKER-INTERNAL
counter packets 0 bytes 0 jump DOCKER-BRIDGE
}
chain DOCKER-USER {
counter packets 0 bytes 0 jump ufw6-user-forward
xt match "conntrack" counter packets 0 bytes 0 return
xt match "conntrack" counter packets 0 bytes 0 drop
iifname "docker0" oifname "docker0" counter packets 0 bytes 0 accept
ip6 saddr fd00::/8 counter packets 0 bytes 0 return
ip6 daddr fd00::/8 xt match "conntrack" counter packets 0 bytes 0 jump ufw6-docker-logging-deny
counter packets 0 bytes 0 return
}
chain ufw6-before-logging-input {
}
chain ufw6-before-logging-output {
}
chain ufw6-before-logging-forward {
}
chain ufw6-before-input {
}
chain ufw6-before-output {
}
chain ufw6-before-forward {
}
chain ufw6-after-input {
}
chain ufw6-after-output {
}
chain ufw6-after-forward {
}
chain ufw6-after-logging-input {
}
chain ufw6-after-logging-output {
}
chain ufw6-after-logging-forward {
}
chain ufw6-reject-input {
}
chain ufw6-reject-output {
}
chain ufw6-reject-forward {
}
chain ufw6-track-input {
}
chain ufw6-track-output {
}
chain ufw6-track-forward {
}
chain ufw6-user-forward {
}
chain ufw6-docker-logging-deny {
limit rate 3/minute burst 10 packets counter packets 0 bytes 0 xt target "LOG"
counter packets 0 bytes 0 drop
}
chain DOCKER {
}
chain DOCKER-BRIDGE {
}
chain DOCKER-CT {
}
chain DOCKER-INTERNAL {
}
}
# Warning: table ip nat is managed by iptables-nft, do not touch!
table ip nat {
chain PREROUTING {
type nat hook prerouting priority dstnat; policy accept;
xt match "addrtype" counter packets 10757093 bytes 647829087 jump DOCKER
}
chain OUTPUT {
type nat hook output priority dstnat; policy accept;
ip daddr != 127.0.0.0/8 xt match "addrtype" counter packets 128611 bytes 7705178 jump DOCKER
}
chain POSTROUTING {
type nat hook postrouting priority srcnat; policy accept;
ip saddr 172.19.0.0/16 oifname != "br-72fd626a8ff7" counter packets 0 bytes 0 xt target "MASQUERADE"
ip saddr 172.18.0.0/16 oifname != "br-2e4a76a7cd2e" counter packets 825 bytes 49500 xt target "MASQUERADE"
ip saddr 192.168.112.0/20 oifname != "br-e5d78502832d" counter packets 126 bytes 7560 xt target "MASQUERADE"
ip saddr 192.168.80.0/20 oifname != "br-e99ce5248c84" counter packets 0 bytes 0 xt target "MASQUERADE"
ip saddr 192.168.48.0/20 oifname != "br-77eb8a9e2ba1" counter packets 99 bytes 5940 xt target "MASQUERADE"
ip saddr 192.168.128.0/20 oifname != "br-73641cceafc3" counter packets 536 bytes 32160 xt target "MASQUERADE"
ip saddr 192.168.208.0/20 oifname != "br-9fd22324ec08" counter packets 2 bytes 120 xt target "MASQUERADE"
ip saddr 192.168.203.0/24 oifname != "br-1ccb887b3344" counter packets 37248 bytes 2854476 xt target "MASQUERADE"
ip saddr 192.168.64.0/20 oifname != "br-a63fa64a9e18" counter packets 353 bytes 21180 xt target "MASQUERADE"
ip saddr 172.17.0.0/16 oifname != "docker0" counter packets 99933 bytes 6001436 xt target "MASQUERADE"
ip saddr 172.21.0.0/16 oifname != "br-84e7d0cfeada" counter packets 2 bytes 128 xt target "MASQUERADE"
ip saddr 192.168.176.0/20 oifname != "br-f8b083119d99" counter packets 209 bytes 12644 xt target "MASQUERADE"
ip saddr 172.20.0.0/16 oifname != "br-6eb1e7f7f847" counter packets 699 bytes 42516 xt target "MASQUERADE"
ip saddr 172.28.0.0/16 oifname != "br-8ce143481a5b" counter packets 1934 bytes 116040 xt target "MASQUERADE"
ip saddr 172.27.0.0/16 oifname != "br-3008d408e73a" counter packets 2904 bytes 174240 xt target "MASQUERADE"
ip saddr 172.25.0.0/16 oifname != "br-cadedce55fe9" counter packets 0 bytes 0 xt target "MASQUERADE"
ip saddr 172.24.0.0/16 oifname != "br-3b338a381229" counter packets 385 bytes 23164 xt target "MASQUERADE"
ip saddr 192.168.224.0/20 oifname != "br-ca07a9577a7f" counter packets 0 bytes 0 xt target "MASQUERADE"
ip saddr 192.168.0.0/20 oifname != "br-a5fbc29c2c2a" counter packets 10 bytes 600 xt target "MASQUERADE"
ip saddr 172.31.0.0/16 oifname != "br-dd007c7e67bc" counter packets 0 bytes 0 xt target "MASQUERADE"
ip saddr 192.168.240.0/20 oifname != "br-0d1490cc67c9" counter packets 587045 bytes 35224163 xt target "MASQUERADE"
}
chain DOCKER {
iifname != "docker0" tcp dport 5100 counter packets 8247 bytes 494812 xt target "DNAT"
iifname != "docker0" tcp dport 222 counter packets 2304 bytes 137444 xt target "DNAT"
iifname != "docker0" tcp dport 20000 counter packets 1532 bytes 91584 xt target "DNAT"
iifname != "br-1ccb887b3344" tcp dport 25 counter packets 433 bytes 23099 xt target "DNAT"
iifname != "br-1ccb887b3344" tcp dport 7080 counter packets 35 bytes 1864 xt target "DNAT"
iifname != "br-1ccb887b3344" tcp dport 110 counter packets 195 bytes 9741 xt target "DNAT"
iifname != "br-1ccb887b3344" tcp dport 143 counter packets 448 bytes 25580 xt target "DNAT"
iifname != "br-1ccb887b3344" tcp dport 7443 counter packets 58 bytes 2868 xt target "DNAT"
iifname != "br-1ccb887b3344" tcp dport 465 counter packets 107 bytes 6032 xt target "DNAT"
iifname != "br-1ccb887b3344" tcp dport 587 counter packets 447 bytes 23492 xt target "DNAT"
iifname != "br-1ccb887b3344" tcp dport 993 counter packets 201 bytes 10174 xt target "DNAT"
iifname != "br-1ccb887b3344" tcp dport 995 counter packets 197 bytes 11180 xt target "DNAT"
iifname != "br-1ccb887b3344" tcp dport 20004 counter packets 5 bytes 300 xt target "DNAT"
iifname != "br-9fd22324ec08" tcp dport 20005 counter packets 5 bytes 300 xt target "DNAT"
iifname != "br-73641cceafc3" tcp dport 20006 counter packets 19 bytes 1140 xt target "DNAT"
iifname != "br-e5d78502832d" tcp dport 20007 counter packets 5 bytes 284 xt target "DNAT"
iifname != "br-e5d78502832d" tcp dport 20008 counter packets 4 bytes 240 xt target "DNAT"
iifname != "br-e99ce5248c84" tcp dport 1842 counter packets 12 bytes 720 xt target "DNAT"
iifname != "br-8ce143481a5b" tcp dport 4848 counter packets 40 bytes 1960 xt target "DNAT"
iifname != "docker0" tcp dport 4222 counter packets 3846 bytes 231012 xt target "DNAT"
ip daddr 127.0.0.1 iifname != "docker0" tcp dport 8222 counter packets 0 bytes 0 xt target "DNAT"
iifname != "docker0" tcp dport 20003 counter packets 18942 bytes 1136512 xt target "DNAT"
iifname != "br-2e4a76a7cd2e" tcp dport 9070 counter packets 195 bytes 11676 xt target "DNAT"
iifname != "br-77eb8a9e2ba1" tcp dport 9102 counter packets 13 bytes 772 xt target "DNAT"
iifname != "br-77eb8a9e2ba1" tcp dport 8102 counter packets 17 bytes 944 xt target "DNAT"
iifname != "br-77eb8a9e2ba1" tcp dport 8104 counter packets 16 bytes 916 xt target "DNAT"
iifname != "br-77eb8a9e2ba1" tcp dport 8103 counter packets 13 bytes 756 xt target "DNAT"
iifname != "br-3008d408e73a" tcp dport 1212 counter packets 189 bytes 11188 xt target "DNAT"
iifname != "br-6eb1e7f7f847" tcp dport 20009 counter packets 138 bytes 8280 xt target "DNAT"
iifname != "br-f8b083119d99" tcp dport 20001 counter packets 47780 bytes 2866736 xt target "DNAT"
iifname != "br-f8b083119d99" tcp dport 20002 counter packets 7 bytes 404 xt target "DNAT"
iifname != "docker0" tcp dport 20010 counter packets 74 bytes 4424 xt target "DNAT"
iifname != "docker0" tcp dport 20011 counter packets 4 bytes 240 xt target "DNAT"
iifname != "br-72fd626a8ff7" tcp dport 20012 counter packets 237 bytes 14220 xt target "DNAT"
iifname != "docker0" tcp dport 6852 counter packets 16489 bytes 989324 xt target "DNAT"
}
}
# Warning: table ip6 nat is managed by iptables-nft, do not touch!
table ip6 nat {
chain PREROUTING {
type nat hook prerouting priority dstnat; policy accept;
xt match "addrtype" counter packets 399 bytes 22104 jump DOCKER
}
chain OUTPUT {
type nat hook output priority dstnat; policy accept;
ip6 daddr != ::1 xt match "addrtype" counter packets 0 bytes 0 jump DOCKER
}
chain DOCKER {
}
}
table ip raw {
chain PREROUTING {
type filter hook prerouting priority raw; policy accept;
ip daddr 127.0.0.1 iifname != "lo" tcp dport 8222 counter packets 0 bytes 0 drop
}
}
table ip mangle {
chain FORWARD {
type filter hook forward priority mangle; policy accept;
}
}
table inet mesh {
chain input {
type filter hook input priority filter; policy drop;
ct state established,related accept
ct state invalid drop
iif "lo" accept
iifname != { "mesh0", "enp9s0" } accept
icmp type echo-request accept
icmpv6 type { echo-request, nd-router-advert, nd-neighbor-solicit, nd-neighbor-advert } accept
iifname != { "mesh0", "enp9s0" } udp dport { 53, 67 } accept
iifname != { "mesh0", "enp9s0" } tcp dport 53 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } tcp dport 22 accept
tcp dport 22 accept
tcp dport 4222 accept
tcp dport 22 accept
tcp dport 25 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } tcp dport 53 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } udp dport 53 accept
tcp dport 80 accept
tcp dport 110 accept
tcp dport 143 accept
tcp dport 222 accept
tcp dport 443 accept
tcp dport 465 accept
tcp dport 587 accept
tcp dport 993 accept
tcp dport 995 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } tcp dport 1212 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } tcp dport 1842 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } tcp dport 4222 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } tcp dport 4848 accept
tcp dport 5100 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } tcp dport 6852 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } tcp dport 7080 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } tcp dport 7443 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } tcp dport 8102 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } tcp dport 8103 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } tcp dport 8104 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } tcp dport 9000 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } tcp dport 9070 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } tcp dport 9102 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } tcp dport 20000 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } tcp dport 20001 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } tcp dport 20002 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } tcp dport 20003 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } tcp dport 20004 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } tcp dport 20005 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } tcp dport 20006 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } tcp dport 20007 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } tcp dport 20008 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } tcp dport 20009 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } tcp dport 20010 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } tcp dport 20011 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } tcp dport 20012 accept
udp dport 51820 accept
}
chain output {
type filter hook output priority filter; policy accept;
}
chain forward {
type filter hook forward priority filter; policy drop;
ct state established,related accept
ct state invalid drop
iifname != { "mesh0", "enp9s0" } accept
iifname "mesh0" oifname "mesh0" accept
ct original proto-dst 22 accept
ct original proto-dst 25 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } ct original proto-dst 53 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } ct original proto-dst 53 accept
ct original proto-dst 80 accept
ct original proto-dst 110 accept
ct original proto-dst 143 accept
ct original proto-dst 222 accept
ct original proto-dst 443 accept
ct original proto-dst 465 accept
ct original proto-dst 587 accept
ct original proto-dst 993 accept
ct original proto-dst 995 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } ct original proto-dst 1212 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } ct original proto-dst 1842 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } ct original proto-dst 4222 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } ct original proto-dst 4848 accept
ct original proto-dst 5100 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } ct original proto-dst 6852 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } ct original proto-dst 7080 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } ct original proto-dst 7443 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } ct original proto-dst 8102 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } ct original proto-dst 8103 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } ct original proto-dst 8104 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } ct original proto-dst 9000 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } ct original proto-dst 9070 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } ct original proto-dst 9102 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } ct original proto-dst 20000 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } ct original proto-dst 20001 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } ct original proto-dst 20002 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } ct original proto-dst 20003 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } ct original proto-dst 20004 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } ct original proto-dst 20005 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } ct original proto-dst 20006 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } ct original proto-dst 20007 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } ct original proto-dst 20008 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } ct original proto-dst 20009 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } ct original proto-dst 20010 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } ct original proto-dst 20011 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } ct original proto-dst 20012 accept
ct original proto-dst 51820 accept
ct original proto-dst 4222 accept
}
}
+149
View File
@@ -0,0 +1,149 @@
-P INPUT ACCEPT
-P FORWARD DROP
-P OUTPUT ACCEPT
-N DOCKER
-N DOCKER-BRIDGE
-N DOCKER-CT
-N DOCKER-FORWARD
-N DOCKER-INTERNAL
-N DOCKER-USER
-N HAL-MESH-ONLY
-N ufw-after-forward
-N ufw-after-input
-N ufw-after-logging-forward
-N ufw-after-logging-input
-N ufw-after-logging-output
-N ufw-after-output
-N ufw-before-forward
-N ufw-before-input
-N ufw-before-logging-forward
-N ufw-before-logging-input
-N ufw-before-logging-output
-N ufw-before-output
-N ufw-reject-forward
-N ufw-reject-input
-N ufw-reject-output
-N ufw-track-forward
-N ufw-track-input
-N ufw-track-output
-A INPUT -j ufw-before-logging-input
-A INPUT -j ufw-before-input
-A INPUT -j ufw-after-input
-A INPUT -j ufw-after-logging-input
-A INPUT -j ufw-reject-input
-A INPUT -j ufw-track-input
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-FORWARD
-A FORWARD -j ufw-before-logging-forward
-A FORWARD -j ufw-before-forward
-A FORWARD -j ufw-after-forward
-A FORWARD -j ufw-after-logging-forward
-A FORWARD -j ufw-reject-forward
-A FORWARD -j ufw-track-forward
-A OUTPUT -j ufw-before-logging-output
-A OUTPUT -j ufw-before-output
-A OUTPUT -j ufw-after-output
-A OUTPUT -j ufw-after-logging-output
-A OUTPUT -j ufw-reject-output
-A OUTPUT -j ufw-track-output
-A DOCKER -d 172.17.0.18/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 8686 -j ACCEPT
-A DOCKER -d 172.17.0.14/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 8989 -j ACCEPT
-A DOCKER -d 172.17.0.15/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 7878 -j ACCEPT
-A DOCKER -d 172.17.0.5/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 9117 -j ACCEPT
-A DOCKER -d 172.17.0.13/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 6789 -j ACCEPT
-A DOCKER -d 172.19.0.2/32 ! -i br-32062158f584 -o br-32062158f584 -p tcp -m tcp --dport 8080 -j ACCEPT
-A DOCKER -d 172.17.0.2/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 5432 -j ACCEPT
-A DOCKER -d 172.27.0.2/32 ! -i br-0910a98c6158 -o br-0910a98c6158 -p tcp -m tcp --dport 5678 -j ACCEPT
-A DOCKER -d 172.17.0.21/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 3579 -j ACCEPT
-A DOCKER -d 172.17.0.19/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 8181 -j ACCEPT
-A DOCKER -d 172.17.0.17/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 8787 -j ACCEPT
-A DOCKER -d 172.17.0.16/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 6767 -j ACCEPT
-A DOCKER -d 172.17.0.12/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 3000 -j ACCEPT
-A DOCKER -d 172.17.0.11/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 80 -j ACCEPT
-A DOCKER -d 172.17.0.10/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 9443 -j ACCEPT
-A DOCKER -d 172.17.0.10/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 9000 -j ACCEPT
-A DOCKER -d 172.17.0.9/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 3000 -j ACCEPT
-A DOCKER -d 172.17.0.7/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 1880 -j ACCEPT
-A DOCKER -d 172.28.0.2/32 ! -i br-b11461b5b028 -o br-b11461b5b028 -p tcp -m tcp --dport 80 -j ACCEPT
-A DOCKER -d 172.23.0.14/32 ! -i br-66ffa5c1cba5 -o br-66ffa5c1cba5 -p tcp -m tcp --dport 6543 -j ACCEPT
-A DOCKER -d 172.23.0.14/32 ! -i br-66ffa5c1cba5 -o br-66ffa5c1cba5 -p tcp -m tcp --dport 5432 -j ACCEPT
-A DOCKER -d 172.23.0.5/32 ! -i br-66ffa5c1cba5 -o br-66ffa5c1cba5 -p tcp -m tcp --dport 8000 -j ACCEPT
-A DOCKER -d 172.26.0.3/32 ! -i br-b0fec361ccaa -o br-b0fec361ccaa -p tcp -m tcp --dport 6167 -j ACCEPT
-A DOCKER -d 172.26.0.2/32 ! -i br-b0fec361ccaa -o br-b0fec361ccaa -p tcp -m tcp --dport 80 -j ACCEPT
-A DOCKER -d 172.17.0.8/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 8000 -j ACCEPT
-A DOCKER -d 172.17.0.6/32 ! -i docker0 -o docker0 -p udp -m udp --dport 10001 -j ACCEPT
-A DOCKER -d 172.17.0.6/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 8880 -j ACCEPT
-A DOCKER -d 172.17.0.6/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 8843 -j ACCEPT
-A DOCKER -d 172.17.0.6/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 8443 -j ACCEPT
-A DOCKER -d 172.17.0.6/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 8080 -j ACCEPT
-A DOCKER -d 172.17.0.6/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 6789 -j ACCEPT
-A DOCKER -d 172.17.0.6/32 ! -i docker0 -o docker0 -p udp -m udp --dport 5514 -j ACCEPT
-A DOCKER -d 172.17.0.6/32 ! -i docker0 -o docker0 -p udp -m udp --dport 3478 -j ACCEPT
-A DOCKER -d 172.17.0.6/32 ! -i docker0 -o docker0 -p udp -m udp --dport 1900 -j ACCEPT
-A DOCKER -d 172.25.0.3/32 ! -i br-b98821f7dc38 -o br-b98821f7dc38 -p tcp -m tcp --dport 8000 -j ACCEPT
-A DOCKER -d 172.18.0.3/32 ! -i br-442a0bfc65f8 -o br-442a0bfc65f8 -p tcp -m tcp --dport 1433 -j ACCEPT
-A DOCKER -d 172.20.0.3/32 ! -i br-afa37ac8b33d -o br-afa37ac8b33d -p tcp -m tcp --dport 8081 -j ACCEPT
-A DOCKER -d 172.20.0.3/32 ! -i br-afa37ac8b33d -o br-afa37ac8b33d -p tcp -m tcp --dport 1883 -j ACCEPT
-A DOCKER -d 172.17.0.4/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 8086 -j ACCEPT
-A DOCKER -d 172.21.0.2/32 ! -i br-df15d8e19ec7 -o br-df15d8e19ec7 -p tcp -m tcp --dport 6379 -j ACCEPT
-A DOCKER -d 172.30.0.3/32 ! -i br-521eab9a3a5e -o br-521eab9a3a5e -p tcp -m tcp --dport 8283 -j ACCEPT
-A DOCKER -d 172.17.0.3/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 3000 -j ACCEPT
-A DOCKER ! -i br-32062158f584 -o br-32062158f584 -j DROP
-A DOCKER ! -i docker0 -o docker0 -j DROP
-A DOCKER ! -i br-521eab9a3a5e -o br-521eab9a3a5e -j DROP
-A DOCKER ! -i br-df15d8e19ec7 -o br-df15d8e19ec7 -j DROP
-A DOCKER ! -i br-afa37ac8b33d -o br-afa37ac8b33d -j DROP
-A DOCKER ! -i br-442a0bfc65f8 -o br-442a0bfc65f8 -j DROP
-A DOCKER ! -i br-b98821f7dc38 -o br-b98821f7dc38 -j DROP
-A DOCKER ! -i br-b0fec361ccaa -o br-b0fec361ccaa -j DROP
-A DOCKER ! -i br-66ffa5c1cba5 -o br-66ffa5c1cba5 -j DROP
-A DOCKER ! -i br-b11461b5b028 -o br-b11461b5b028 -j DROP
-A DOCKER ! -i br-2df4e541b877 -o br-2df4e541b877 -j DROP
-A DOCKER ! -i br-0910a98c6158 -o br-0910a98c6158 -j DROP
-A DOCKER-BRIDGE -o br-32062158f584 -j DOCKER
-A DOCKER-BRIDGE -o docker0 -j DOCKER
-A DOCKER-BRIDGE -o br-521eab9a3a5e -j DOCKER
-A DOCKER-BRIDGE -o br-df15d8e19ec7 -j DOCKER
-A DOCKER-BRIDGE -o br-afa37ac8b33d -j DOCKER
-A DOCKER-BRIDGE -o br-442a0bfc65f8 -j DOCKER
-A DOCKER-BRIDGE -o br-b98821f7dc38 -j DOCKER
-A DOCKER-BRIDGE -o br-b0fec361ccaa -j DOCKER
-A DOCKER-BRIDGE -o br-66ffa5c1cba5 -j DOCKER
-A DOCKER-BRIDGE -o br-b11461b5b028 -j DOCKER
-A DOCKER-BRIDGE -o br-2df4e541b877 -j DOCKER
-A DOCKER-BRIDGE -o br-0910a98c6158 -j DOCKER
-A DOCKER-CT -o br-32062158f584 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A DOCKER-CT -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A DOCKER-CT -o br-521eab9a3a5e -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A DOCKER-CT -o br-df15d8e19ec7 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A DOCKER-CT -o br-afa37ac8b33d -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A DOCKER-CT -o br-442a0bfc65f8 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A DOCKER-CT -o br-b98821f7dc38 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A DOCKER-CT -o br-b0fec361ccaa -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A DOCKER-CT -o br-66ffa5c1cba5 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A DOCKER-CT -o br-b11461b5b028 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A DOCKER-CT -o br-2df4e541b877 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A DOCKER-CT -o br-0910a98c6158 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A DOCKER-FORWARD -j DOCKER-CT
-A DOCKER-FORWARD -j DOCKER-INTERNAL
-A DOCKER-FORWARD -j DOCKER-BRIDGE
-A DOCKER-FORWARD -i br-32062158f584 -j ACCEPT
-A DOCKER-FORWARD -i docker0 -j ACCEPT
-A DOCKER-FORWARD -i br-521eab9a3a5e -j ACCEPT
-A DOCKER-FORWARD -i br-df15d8e19ec7 -j ACCEPT
-A DOCKER-FORWARD -i br-afa37ac8b33d -j ACCEPT
-A DOCKER-FORWARD -i br-442a0bfc65f8 -j ACCEPT
-A DOCKER-FORWARD -i br-b98821f7dc38 -j ACCEPT
-A DOCKER-FORWARD -i br-b0fec361ccaa -j ACCEPT
-A DOCKER-FORWARD -i br-66ffa5c1cba5 -j ACCEPT
-A DOCKER-FORWARD -i br-b11461b5b028 -j ACCEPT
-A DOCKER-FORWARD -i br-2df4e541b877 -j ACCEPT
-A DOCKER-FORWARD -i br-0910a98c6158 -j ACCEPT
-A DOCKER-USER -i enp6s0 -p tcp -m conntrack --ctstate NEW -j HAL-MESH-ONLY
-A HAL-MESH-ONLY -m conntrack --ctorigdstport 6881 -j RETURN
-A HAL-MESH-ONLY -m conntrack --ctorigdstport 80 -j RETURN
-A HAL-MESH-ONLY -m conntrack --ctorigdstport 443 -j RETURN
-A HAL-MESH-ONLY -s 10.0.0.0/8 -j RETURN
-A HAL-MESH-ONLY -s 172.16.0.0/12 -j RETURN
-A HAL-MESH-ONLY -s 192.168.0.0/16 -j RETURN
-A HAL-MESH-ONLY -m comment --comment "HAL: not public -> mesh only" -j DROP
+327
View File
@@ -0,0 +1,327 @@
table ip mangle {
chain FORWARD {
type filter hook forward priority mangle; policy accept;
}
}
# Warning: table ip nat is managed by iptables-nft, do not touch!
table ip nat {
chain PREROUTING {
type nat hook prerouting priority dstnat; policy accept;
xt match "addrtype" counter packets 12072 bytes 4564241 jump DOCKER
}
chain OUTPUT {
type nat hook output priority dstnat; policy accept;
ip daddr != 127.0.0.0/8 xt match "addrtype" counter packets 1854 bytes 111240 jump DOCKER
}
chain POSTROUTING {
type nat hook postrouting priority srcnat; policy accept;
ip saddr 172.17.0.0/16 oifname != "docker0" counter packets 903 bytes 61577 xt target "MASQUERADE"
ip saddr 172.21.0.0/16 oifname != "br-86a5d6b30e2b" counter packets 344 bytes 27744 xt target "MASQUERADE"
ip saddr 172.25.0.0/16 oifname != "br-61495e14a004" counter packets 374 bytes 33016 xt target "MASQUERADE"
ip saddr 172.30.0.0/16 oifname != "br-5107796ee9b4" counter packets 352 bytes 28224 xt target "MASQUERADE"
ip saddr 172.18.0.0/16 oifname != "br-cfd337ac4e58" counter packets 339 bytes 27444 xt target "MASQUERADE"
ip saddr 172.19.0.0/16 oifname != "br-8f0c6ee01425" counter packets 351 bytes 28164 xt target "MASQUERADE"
ip saddr 172.22.0.0/16 oifname != "br-75ac3c36e87f" counter packets 333 bytes 27084 xt target "MASQUERADE"
ip saddr 172.20.0.0/16 oifname != "br-0529801521bc" counter packets 343 bytes 27404 xt target "MASQUERADE"
}
chain DOCKER {
iifname != "br-61495e14a004" tcp dport 5680 counter packets 2 bytes 120 xt target "DNAT"
ip daddr 127.0.0.1 iifname != "br-61495e14a004" tcp dport 15673 counter packets 0 bytes 0 xt target "DNAT"
ip daddr 127.0.0.1 iifname != "docker0" tcp dport 55432 counter packets 0 bytes 0 xt target "DNAT"
}
}
# Warning: table ip filter is managed by iptables-nft, do not touch!
table ip filter {
chain DOCKER-FORWARD {
counter packets 702328 bytes 1801910999 jump DOCKER-CT
counter packets 337315 bytes 23928864 jump DOCKER-INTERNAL
counter packets 337315 bytes 23928864 jump DOCKER-BRIDGE
iifname "br-75ac3c36e87f" counter packets 0 bytes 0 accept
iifname "br-86a5d6b30e2b" counter packets 0 bytes 0 accept
iifname "br-8f0c6ee01425" counter packets 0 bytes 0 accept
iifname "br-cfd337ac4e58" counter packets 0 bytes 0 accept
iifname "br-0529801521bc" counter packets 0 bytes 0 accept
iifname "br-5107796ee9b4" counter packets 0 bytes 0 accept
iifname "br-61495e14a004" counter packets 0 bytes 0 accept
iifname "docker0" counter packets 337315 bytes 23928864 accept
}
chain FORWARD {
type filter hook forward priority filter; policy drop;
counter packets 702328 bytes 1801910999 jump DOCKER-USER
counter packets 702328 bytes 1801910999 jump DOCKER-FORWARD
}
chain DOCKER-USER {
ip protocol tcp counter packets 702510 bytes 1801965868 jump f2b-sshd
oifname "incusbr0" counter packets 0 bytes 0 accept
iifname "incusbr0" counter packets 0 bytes 0 accept
}
chain f2b-sshd {
counter packets 10423854 bytes 13891318049 return
}
chain INPUT {
type filter hook input priority filter; policy accept;
ip protocol tcp counter packets 9721344 bytes 12089352181 jump f2b-sshd
}
chain DOCKER {
ip daddr 172.17.0.2 iifname != "docker0" oifname "docker0" tcp dport 5432 counter packets 0 bytes 0 accept
ip daddr 172.25.0.2 iifname != "br-61495e14a004" oifname "br-61495e14a004" tcp dport 15672 counter packets 0 bytes 0 accept
ip daddr 172.25.0.2 iifname != "br-61495e14a004" oifname "br-61495e14a004" tcp dport 5672 counter packets 0 bytes 0 accept
iifname != "br-75ac3c36e87f" oifname "br-75ac3c36e87f" counter packets 0 bytes 0 drop
iifname != "br-86a5d6b30e2b" oifname "br-86a5d6b30e2b" counter packets 0 bytes 0 drop
iifname != "br-8f0c6ee01425" oifname "br-8f0c6ee01425" counter packets 0 bytes 0 drop
iifname != "br-cfd337ac4e58" oifname "br-cfd337ac4e58" counter packets 0 bytes 0 drop
iifname != "br-0529801521bc" oifname "br-0529801521bc" counter packets 0 bytes 0 drop
iifname != "br-5107796ee9b4" oifname "br-5107796ee9b4" counter packets 0 bytes 0 drop
iifname != "br-61495e14a004" oifname "br-61495e14a004" counter packets 0 bytes 0 drop
iifname != "docker0" oifname "docker0" counter packets 0 bytes 0 drop
}
chain DOCKER-BRIDGE {
oifname "br-75ac3c36e87f" counter packets 0 bytes 0 jump DOCKER
oifname "br-86a5d6b30e2b" counter packets 0 bytes 0 jump DOCKER
oifname "br-8f0c6ee01425" counter packets 0 bytes 0 jump DOCKER
oifname "br-cfd337ac4e58" counter packets 0 bytes 0 jump DOCKER
oifname "br-0529801521bc" counter packets 0 bytes 0 jump DOCKER
oifname "br-5107796ee9b4" counter packets 0 bytes 0 jump DOCKER
oifname "br-61495e14a004" counter packets 0 bytes 0 jump DOCKER
oifname "docker0" counter packets 0 bytes 0 jump DOCKER
}
chain DOCKER-CT {
oifname "br-75ac3c36e87f" xt match "conntrack" counter packets 0 bytes 0 accept
oifname "br-86a5d6b30e2b" xt match "conntrack" counter packets 0 bytes 0 accept
oifname "br-8f0c6ee01425" xt match "conntrack" counter packets 0 bytes 0 accept
oifname "br-cfd337ac4e58" xt match "conntrack" counter packets 0 bytes 0 accept
oifname "br-0529801521bc" xt match "conntrack" counter packets 0 bytes 0 accept
oifname "br-5107796ee9b4" xt match "conntrack" counter packets 0 bytes 0 accept
oifname "br-61495e14a004" xt match "conntrack" counter packets 0 bytes 0 accept
oifname "docker0" xt match "conntrack" counter packets 365013 bytes 1777982135 accept
}
chain DOCKER-INTERNAL {
}
}
# Warning: table ip6 nat is managed by iptables-nft, do not touch!
table ip6 nat {
chain PREROUTING {
type nat hook prerouting priority dstnat; policy accept;
xt match "addrtype" counter packets 363 bytes 67927 jump DOCKER
}
chain OUTPUT {
type nat hook output priority dstnat; policy accept;
ip6 daddr != ::1 xt match "addrtype" counter packets 0 bytes 0 jump DOCKER
}
chain DOCKER {
}
}
table ip6 filter {
chain DOCKER-FORWARD {
counter packets 0 bytes 0 jump DOCKER-CT
counter packets 0 bytes 0 jump DOCKER-INTERNAL
counter packets 0 bytes 0 jump DOCKER-BRIDGE
}
chain FORWARD {
type filter hook forward priority filter; policy accept;
counter packets 0 bytes 0 jump DOCKER-USER
counter packets 0 bytes 0 jump DOCKER-FORWARD
}
chain DOCKER-USER {
}
chain DOCKER {
}
chain DOCKER-BRIDGE {
}
chain DOCKER-CT {
}
chain DOCKER-INTERNAL {
}
}
table ip raw {
chain PREROUTING {
type filter hook prerouting priority raw; policy accept;
ip daddr 172.25.0.2 iifname != "br-61495e14a004" counter packets 0 bytes 0 drop
ip daddr 127.0.0.1 iifname != "lo" tcp dport 15673 counter packets 0 bytes 0 drop
ip daddr 172.17.0.2 iifname != "docker0" counter packets 0 bytes 0 drop
ip daddr 127.0.0.1 iifname != "lo" tcp dport 55432 counter packets 0 bytes 0 drop
}
}
table inet incus {
set bridges {
type ifname
elements = { "incusbr0" }
}
chain pstrt.incusbr0 {
type nat hook postrouting priority srcnat; policy accept;
ip saddr 10.7.169.0/24 oifname @bridges accept
ip saddr 10.7.169.0/24 ip daddr != 10.7.169.0/24 masquerade
ip6 saddr fd42:cbc4:e123:f6::/64 oifname @bridges accept
ip6 saddr fd42:cbc4:e123:f6::/64 ip6 daddr != fd42:cbc4:e123:f6::/64 masquerade
}
chain fwd.incusbr0 {
type filter hook forward priority filter; policy accept;
ip version 4 oifname "incusbr0" accept
ip version 4 iifname "incusbr0" accept
ip6 version 6 oifname "incusbr0" accept
ip6 version 6 iifname "incusbr0" accept
}
chain in.incusbr0 {
type filter hook input priority filter; policy accept;
iifname "incusbr0" tcp dport 53 accept
iifname "incusbr0" udp dport 53 accept
iifname "incusbr0" icmp type { destination-unreachable, time-exceeded, parameter-problem } accept
iifname "incusbr0" udp dport 67 accept
iifname "incusbr0" ip protocol udp udp checksum set 0
iifname "incusbr0" icmpv6 type { destination-unreachable, packet-too-big, time-exceeded, parameter-problem, nd-router-solicit, nd-neighbor-solicit, nd-neighbor-advert, mld2-listener-report } accept
iifname "incusbr0" udp dport 547 accept
}
chain out.incusbr0 {
type filter hook output priority filter; policy accept;
oifname "incusbr0" tcp sport 53 accept
oifname "incusbr0" udp sport 53 accept
oifname "incusbr0" icmp type { destination-unreachable, time-exceeded, parameter-problem } accept
oifname "incusbr0" udp sport 67 accept
oifname "incusbr0" ip protocol udp udp checksum set 0
oifname "incusbr0" icmpv6 type { destination-unreachable, packet-too-big, time-exceeded, parameter-problem, echo-request, nd-router-advert, nd-neighbor-solicit, nd-neighbor-advert, mld2-listener-report } accept
oifname "incusbr0" udp sport 547 accept
}
}
table ip fct_filter {
chain OUTPUT {
type filter hook output priority filter; policy accept;
}
chain FCT-QUARANTINE-EMS {
}
chain FCT-QUARANTINE-FAZ {
}
chain FORWARD {
type filter hook forward priority filter; policy accept;
}
chain FCT-WEBFILTER-QUIC-CHAIN {
}
chain INPUT {
type filter hook input priority filter; policy accept;
}
chain FCT-QUARANTINE {
}
chain FCT-DNS-QUIC-FILTER {
}
chain FCT-VPN-CHAIN {
}
}
table ip fct_nat {
chain OUTPUT {
type nat hook output priority dstnat; policy accept;
}
chain FCT-DNS-UDP-CHAIN-STAGE-2 {
}
chain FCT-DNS-UDP-CHAIN-STAGE-1 {
}
chain FCT-TCP-CHAIN {
}
chain FCT-DNS-DOH-CHAIN-STAGE-1 {
}
chain FCT-WEBFILTER-CHAIN {
}
chain FCT-DNS-DOH-CHAIN-STAGE-2 {
}
}
table ip6 fct_filter {
chain FCT-QUARANTINE {
}
chain INPUT {
type filter hook input priority filter; policy accept;
}
chain FORWARD {
type filter hook forward priority filter; policy accept;
}
chain OUTPUT {
type filter hook output priority filter; policy accept;
}
}
table ip fct_mangle {
chain PREROUTING {
type filter hook prerouting priority mangle; policy accept;
}
chain FCT-UDP-STAGE-1 {
}
chain OUTPUT {
type route hook output priority mangle; policy accept;
}
chain FCT-UDP-STAGE-2 {
}
chain FCT-UDP-OUTPUT {
}
}
table inet mesh {
chain input {
type filter hook input priority filter; policy drop;
ct state established,related accept
ct state invalid drop
iif "lo" accept
iifname != { "mesh0", "wlp3s0" } accept
icmp type echo-request accept
icmpv6 type { echo-request, nd-router-advert, nd-neighbor-solicit, nd-neighbor-advert } accept
iifname != { "mesh0", "wlp3s0" } udp dport { 53, 67 } accept
iifname != { "mesh0", "wlp3s0" } tcp dport 53 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } tcp dport 22 accept
tcp dport 22 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } tcp dport 53 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } udp dport 53 accept
}
chain output {
type filter hook output priority filter; policy accept;
}
chain forward {
type filter hook forward priority filter; policy drop;
ct state established,related accept
ct state invalid drop
iifname != { "mesh0", "wlp3s0" } accept
iifname "mesh0" oifname "mesh0" accept
ct original proto-dst 22 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } ct original proto-dst 53 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } ct original proto-dst 53 accept
}
}
+23
View File
@@ -409,3 +409,26 @@ func TestATokenSaysWhatTheMeshCallsThisMachine(t *testing.T) {
t.Fatalf("the name did not survive the token: %q", token.Node)
}
}
// A token through the tunnel carries the one peer, in the field names the control plane writes
// (novox/hq ADR 0169), and an incomplete tunnel is refused naming what is missing.
func TestATokenThroughTheTunnelParsesAndAPartOneIsRefused(t *testing.T) {
whole := map[string]any{"v": 1, "node": "n", "broker": "10.42.0.1:4222", "fingerprint": "sha256:x",
"signer": make([]byte, 32), "secret": "s",
"tunnel": map[string]any{"key": "k", "address": "10.42.0.9/32", "range": "10.42.0.0/16",
"hub_key": "h", "hub_endpoint": "198.51.100.1:51820"}}
raw, _ := json.Marshal(whole)
got, err := ParseToken(base64.RawURLEncoding.EncodeToString(raw))
if err != nil {
t.Fatal(err)
}
if got.Tunnel == nil || got.Tunnel.HubEndpoint != "198.51.100.1:51820" || got.Tunnel.Range != "10.42.0.0/16" {
t.Fatalf("the tunnel was not read: %+v", got.Tunnel)
}
whole["tunnel"] = map[string]any{"key": "k"}
raw, _ = json.Marshal(whole)
if _, err := ParseToken(base64.RawURLEncoding.EncodeToString(raw)); err == nil ||
!strings.Contains(err.Error(), "the hub's tunnel key") {
t.Fatalf("a token with half a tunnel was taken: %v", err)
}
}
+15
View File
@@ -5,6 +5,8 @@ import (
"crypto/rand"
"encoding/base64"
"fmt"
"os"
"strings"
)
// The node's key on the private network, which is a different key from the one that says who it
@@ -64,6 +66,19 @@ func OverlayKeyFrom(privateBase64 string) (OverlayKey, error) {
}, nil
}
// LoadOverlayKey reads the key `key` made and left in its file (novox/hq ADR 0169).
func LoadOverlayKey(path string) (OverlayKey, error) {
raw, err := os.ReadFile(path)
if err != nil {
return OverlayKey{}, err
}
key, err := OverlayKeyFrom(strings.TrimSpace(string(raw)))
if err != nil {
return OverlayKey{}, fmt.Errorf("%s does not hold a tunnel key: %w", path, err)
}
return key, nil
}
// OverlayKeyPath is where the private half lives: a file of its own, referenced by the interface
// configuration rather than embedded in it.
//
+26
View File
@@ -35,6 +35,21 @@ type Token struct {
// firewall found here before enrolling, because an adopted node keeps that firewall in force.
// Absent for a converged node.
Adopted bool `json:"adopted,omitempty"`
// Tunnel is this machine's first tunnel, when the token was issued for the key it made with
// `key` (novox/hq ADR 0169): its own address and the hub to reach. It brings the tunnel up from
// this alone and reaches the bus over it, so the bus never has to face the internet.
Tunnel *TokenTunnel `json:"tunnel,omitempty"`
}
// TokenTunnel is the joining machine's side of its first tunnel. Field names are the wire format
// the control plane writes.
type TokenTunnel struct {
Key string `json:"key"`
Address string `json:"address"`
Range string `json:"range"`
HubKey string `json:"hub_key"`
HubEndpoint string `json:"hub_endpoint"`
}
// ParseToken reads a token a person pasted.
@@ -70,6 +85,17 @@ func ParseToken(encoded string) (Token, error) {
if strings.TrimSpace(t.Secret) == "" {
missing = append(missing, "the one-time secret")
}
if tt := t.Tunnel; tt != nil {
for _, part := range []struct{ value, says string }{
{tt.Key, "the tunnel key it was issued for"}, {tt.Address, "this machine's address"},
{tt.Range, "the private network's range"}, {tt.HubKey, "the hub's tunnel key"},
{tt.HubEndpoint, "where the hub's tunnel is dialled"},
} {
if strings.TrimSpace(part.value) == "" {
missing = append(missing, part.says)
}
}
}
if len(missing) > 0 {
// Refused whole rather than used partially. A token missing the fingerprint would have
// this node connect to whatever answers at that address, and one missing the signing key
+21 -1
View File
@@ -3,6 +3,7 @@ package link
import (
"context"
"crypto/rand"
"crypto/sha256"
"encoding/hex"
"errors"
"fmt"
@@ -63,8 +64,15 @@ func presentNats(_ context.Context, to Approach, node, secret string,
// subscribe its own inbox and nothing else (design 25 §6). The secret is its password, the same
// string the request claims, so the server proves somebody holds the token and the request
// proves the same thing to the controller without it having to ask the server who connected.
// **Its own inbox space, because that is the only one it may listen in** (novox/hq
// 04-ISSUES/146). A JetStream publish waits for the stream's acknowledgement on an inbox the
// client picks, and the client's default is `_INBOX.<random>` — which this user may not
// subscribe to, so the enrolment failed with a permissions violation on a subject nobody had
// chosen. The permission is `_INBOX.enrol.<node>.>` (design 25 §6), so the client is told to
// pick its inboxes there; the reply address below is in the same space for the same reason.
conn, err := nats.Connect(natsURL(to.Address),
nats.Secure(config),
nats.CustomInboxPrefix("_INBOX.enrol."+node),
nats.UserInfo("enrol."+node, secret),
nats.Name("mesh-host/enrol/"+node),
nats.Timeout(timeout),
@@ -124,7 +132,19 @@ func (a *natsAsking) Ask(ctx context.Context, request []byte, wait time.Duration
defer cancel()
// Into the stream and awaited: an enrolment the bus never accepted must fail here rather than be
// assumed, because the node has nothing else to go on.
if _, err := a.js.Publish(EnrolSubject, addressed, nats.Context(publish)); err != nil {
//
// **Once, however many times it is sent** (novox/hq 04-ISSUES/146). The client re-publishes when
// an acknowledgement is slow, and the mesh enrolled the machine on each copy — minting a second
// credential, which replaced the first, which is the one the node had already been given. The
// machine then reconnected for ever as a user whose password the mesh had rotated out from under
// it, and the controller's log said "enrolled anchor" twice in the same second.
//
// The id is the message: the same bytes carry the same id, so the stream discards the client's
// own retry, and a genuine second attempt — which carries a new reply address — is a different
// message and is let through.
sum := sha256.Sum256(addressed)
if _, err := a.js.Publish(EnrolSubject, addressed,
nats.MsgId(hex.EncodeToString(sum[:])), nats.Context(publish)); err != nil {
return nil, fmt.Errorf("cannot ask the mesh to enrol this node: %w", err)
}
+46
View File
@@ -0,0 +1,46 @@
package link
import (
"strings"
"testing"
)
// The count that did not add up was the only symptom sixteen held resources had, and reading it meant
// opening the node's state file by hand (novox/hq 04-ISSUES/125). The line that says what an apply did
// says what it did not, too.
func TestTheApplyLineSaysWhatItHeldAndForWhichModule(t *testing.T) {
got := heldNote([]Held{
{ID: "ca", Module: "route-proxy", Kind: "directory"},
{ID: "certs", Module: "route-proxy", Kind: "directory"},
{ID: "server", Module: "route-proxy", Kind: "container"},
{ID: "mail", Module: "mailu", Kind: "container"},
})
if !strings.Contains(got, "4 held") {
t.Fatalf("the count of what was held is not in the line: %q", got)
}
// The module is the thing an operator can act on: `take` takes a module.
if !strings.Contains(got, "route-proxy: 3") || !strings.Contains(got, "mailu: 1") {
t.Fatalf("the line does not break the holds down by module: %q", got)
}
// Ordered, so two machines holding the same things read the same and a diff of two reports is
// about what changed.
if strings.Index(got, "mailu") > strings.Index(got, "route-proxy") {
t.Fatalf("modules are not in a stated order: %q", got)
}
// It says why, because "held" alone reads as a failure and this is correct behaviour.
if !strings.Contains(got, "taken") {
t.Fatalf("the line does not say a hold ends when the module is taken: %q", got)
}
}
func TestAnApplyThatHeldNothingSaysNothingExtra(t *testing.T) {
// A converged machine holds nothing, which is most applies. Reporting "0 held" on every one of
// them is how a line stops being read.
if got := heldNote(nil); got != "" {
t.Fatalf("an apply with no holds added %q to its line", got)
}
if got := heldNote([]Held{}); got != "" {
t.Fatalf("an apply with no holds added %q to its line", got)
}
}
+66
View File
@@ -110,6 +110,47 @@ type Report struct {
// and the mesh's up in its place, and where the found configuration's original was kept.
Tunnel *CarriedTunnel `json:"tunnel,omitempty"`
// Filters is what filters this machine now, every table and chain that refuses traffic with its
// owner — the mesh's, the found firewall's, the container runtime's own, a ban list, or other
// (novox/hq ADR 0168). Every node reports it, adopted or converged, so the mesh can say
// truthfully what filters a converged machine and name what it did not write.
Filters []Filter `json:"filters,omitempty"`
// FoundFirewall is the state of the firewall a converged machine was found with: whether it is
// in force now, and how it came to be inactive — the mesh disabled it, or a reconcile found it so
// (ADR 0168). Nil on a machine found with none, and on an adopted one, where Firewall says it.
FoundFirewall *FoundFirewall `json:"found_firewall,omitempty"`
// Strays is what runs on the machine that the mesh neither wrote nor holds (novox/hq ADR
// 0163): containers nobody declared and nobody holds, the ones a cutover leaves behind.
Strays []Stray `json:"strays,omitempty"`
// Profile is what this machine can do, detected again by the apply that reports (novox/hq
// ADR 0161) — the same shape enrolment sends — so a capability gained or lost since enrolment,
// a network manager switched, reaches the mesh at the next push rather than never.
Profile map[string]any `json:"profile,omitempty"`
// Host is the version of the host that produced this report (novox/hq ADR 0141).
//
// Without it nothing can say a machine is behind, so "every machine current with its source"
// could not include the host — the one component the mesh did not deliver. It is a fact the
// machine states about itself, like the firewall it found and the links that face outside.
Host string `json:"host,omitempty"`
// Outward is the links on this machine that face outside it — the ones carrying a default
// route (novox/hq ADR 0140). Every node reports it, adopted or converged, because a converged
// node's filter is written around it.
//
// **It replaces a list of addresses.** The filter used to block everything passing through the
// machine and then allow the machine's own containers back by naming the ranges they sit on.
// A range describes one machine and goes stale in silence; the link carrying the default route
// is read afresh on every report and does not change when a module is added or removed.
//
// Empty means this machine has no route off itself. The mesh then composes no filter for it and
// leaves the one it has, rather than writing a rule around a link with no name — a rule set
// that does not load is a machine filtering nothing while its unit reports success.
Outward []string `json:"outward,omitempty"`
// Rekey is this node taking a found tunnel's key as its overlay key after enrolment (novox/hq
// ADR 0105). Not an account of the machine: a report carrying one says nothing else.
Rekey *Rekey `json:"rekey,omitempty"`
@@ -179,6 +220,16 @@ type Held struct {
Changed string `json:"changed,omitempty"`
// Kept is where a file's original was kept.
Kept string `json:"kept,omitempty"`
// Facts is the found thing beside what the module declares — what a take compares (novox/hq
// ADR 0163). The same shape the host keeps; the controller reads it as data.
Facts map[string]any `json:"facts,omitempty"`
}
// A Stray is a container the mesh neither wrote nor holds (ADR 0163).
type Stray struct {
Kind string `json:"kind"`
Name string `json:"name"`
Detail string `json:"detail,omitempty"`
}
// Reach is one thing reachable on the machine: a listening socket, or a published container port.
@@ -193,3 +244,18 @@ type Reach struct {
Published bool `json:"published,omitempty"`
ContainerPort int `json:"container-port,omitempty"`
}
// A Filter is one place on the machine that refuses traffic, with its owner (novox/hq ADR 0168):
// the same shape the host's firewall package reads, carried as data.
type Filter struct {
Where string `json:"where"`
Owner string `json:"owner"`
Refuses string `json:"refuses"`
}
// FoundFirewall is the state of a converged machine's found firewall (ADR 0168).
type FoundFirewall struct {
Kind string `json:"kind"`
Active bool `json:"active"`
RetiredBy string `json:"retired_by,omitempty"`
}
+50
View File
@@ -0,0 +1,50 @@
package link
import (
"encoding/json"
"testing"
"time"
)
// The drain picked the last to arrive. A backlog longer than the batch, or a broker that split a
// burst, delivered a superseded declaration last (novox/hq 04-ISSUES/107).
func sequenced(t *testing.T, n int64) *said {
t.Helper()
inner, err := json.Marshal(map[string]any{"declaration": 1, "resources": []any{}, "sequence": n})
if err != nil {
t.Fatal(err)
}
body, err := json.Marshal(Signed{Declaration: inner, Signature: []byte("s")})
if err != nil {
t.Fatal(err)
}
return &said{body: body}
}
func TestTheDrainKeepsTheHighestSequenceNotTheLastToArrive(t *testing.T) {
waiting := make(chan Declaration, 8)
waiting <- sequenced(t, 9)
waiting <- sequenced(t, 4) // arrived last, composed earlier
latest, aside := newest(waiting, sequenced(t, 8), 30*time.Millisecond)
if got := sequenceOf(latest.Body()); got != 9 {
t.Fatalf("the drain kept sequence %d, and 9 was waiting", got)
}
if len(aside) != 2 {
t.Fatalf("%d set aside, wanted 2 (the 8 and the late 4)", len(aside))
}
}
func TestWithoutSequencesTheLastToArriveStillWins(t *testing.T) {
// The behaviour this had before, kept for a controller that sends no order.
apply, aside := newest(arriving("two", "three"), &said{body: []byte("one")}, 30*time.Millisecond)
if string(apply.Body()) != "three" || len(aside) != 2 {
t.Fatalf("applied %q with %d set aside", apply.Body(), len(aside))
}
}
func TestAnUnreadableBodyClaimsNoOrder(t *testing.T) {
if got := sequenceOf([]byte("not json")); got != 0 {
t.Fatalf("garbage claimed sequence %d", got)
}
}
+14 -3
View File
@@ -73,8 +73,19 @@ func PinnedConfig(pin string) (*tls.Config, error) {
}, nil
}
// Dial opens a TLS connection to the broker, refusing anything but the pinned certificate.
func Dial(address, pin string, timeout time.Duration) (*tls.Conn, error) {
// dialPinned completes a TLS handshake against an address, refusing anything but the pinned
// certificate.
//
// **Not how the bus is reached, and it used to be** (novox/hq 04-ISSUES/146). Enrolment opened one
// of these before it said anything, which was right while the broker answered TLS immediately and
// wrong the moment the mesh moved to a bus that speaks its own protocol first. The pin itself was
// never the problem — PinnedConfig is what the NATS client is given, and the verification runs
// inside the handshake that client performs.
//
// It stays here because this is where the pin is proven: the tests beside it run a real TLS server
// and assert that a wrong certificate is refused before a byte of application data is sent. What it
// must not become again is something a caller uses to reach the bus.
func dialPinned(address, pin string, timeout time.Duration) (*tls.Conn, error) {
config, err := PinnedConfig(pin)
if err != nil {
return nil, err
@@ -86,7 +97,7 @@ func Dial(address, pin string, timeout time.Duration) (*tls.Conn, error) {
if errors.Is(err, ErrWrongCertificate) {
return nil, err
}
return nil, fmt.Errorf("cannot reach the broker at %s: %w", address, err)
return nil, fmt.Errorf("cannot reach %s: %w", address, err)
}
return conn, nil
}
+4 -4
View File
@@ -63,7 +63,7 @@ func server(t *testing.T) (address string, fingerprint string) {
func TestTheRightBrokerIsAccepted(t *testing.T) {
address, pin := server(t)
conn, err := Dial(address, pin, 5*time.Second)
conn, err := dialPinned(address, pin, 5*time.Second)
if err != nil {
t.Fatalf("the broker its token describes was refused: %v", err)
}
@@ -76,7 +76,7 @@ func TestADifferentBrokerIsRefused(t *testing.T) {
address, _ := server(t)
_, other := server(t)
_, err := Dial(address, other, 5*time.Second)
_, err := dialPinned(address, other, 5*time.Second)
if err == nil {
t.Fatal("a broker presenting a different certificate was accepted")
}
@@ -130,7 +130,7 @@ func TestNothingIsSentToTheWrongBroker(t *testing.T) {
// A pin for a certificate this server does not have.
_, elsewhere := server(t)
if _, err := Dial(listener.Addr().String(), elsewhere, 5*time.Second); err == nil {
if _, err := dialPinned(listener.Addr().String(), elsewhere, 5*time.Second); err == nil {
t.Fatal("the impostor was accepted")
}
if n := <-received; n > 0 {
@@ -160,7 +160,7 @@ func TestAnUnreachableBrokerIsAnOrdinaryFailure(t *testing.T) {
address := listener.Addr().String()
listener.Close()
_, err = Dial(address, pin, 2*time.Second)
_, err = dialPinned(address, pin, 2*time.Second)
if err == nil {
t.Fatal("dialling a closed port succeeded")
}
+67 -2
View File
@@ -6,6 +6,8 @@ import (
"encoding/json"
"errors"
"fmt"
"sort"
"strings"
"time"
)
@@ -250,9 +252,11 @@ func Run(ctx context.Context, m Membership, apply Applier, say Announce, timeout
case report.Refused != "":
say("refused a declaration: " + report.Refused)
case len(report.Failed) > 0:
say(fmt.Sprintf("applied %d and failed: %v", len(report.Applied), report.Failed))
say(fmt.Sprintf("applied %d and failed: %v%s",
len(report.Applied), report.Failed, heldNote(report.Held)))
default:
say(fmt.Sprintf("applied %d resource(s)", len(report.Applied)))
say(fmt.Sprintf("applied %d resource(s)%s",
len(report.Applied), heldNote(report.Held)))
}
publishReport(ctx, link, m, report, say, timeout)
// Settled after the report is published. A node that dies between applying and
@@ -296,6 +300,16 @@ func newest(arriving <-chan Declaration, first Declaration, window time.Duration
if !ok {
return latest, superseded
}
// **By sequence when both carry one, by arrival when either does not** (novox/hq
// 04-ISSUES/107). Arrival is what this window had to go on, and it is wrong exactly
// when it matters — a backlog drained out of order. A declaration that says where it
// stands is believed over when it turned up; one that does not is the older
// controller's, and arrival is all there is.
if sequenceOf(next.Body()) < sequenceOf(latest.Body()) &&
sequenceOf(next.Body()) > 0 && sequenceOf(latest.Body()) > 0 {
superseded = append(superseded, next)
continue
}
superseded = append(superseded, latest)
latest = next
case <-time.After(window):
@@ -304,6 +318,24 @@ func newest(arriving <-chan Declaration, first Declaration, window time.Duration
}
}
// sequenceOf is the order a signed declaration claims, or zero when it claims none or cannot be
// read. Read from the envelope alone; the signature is verified later, when the winner is applied,
// and a forged message that lied about its sequence would only set aside real ones — which are
// reported as set aside, and the next push sends the current one again.
func sequenceOf(body []byte) int64 {
var signed Signed
if err := json.Unmarshal(body, &signed); err != nil {
return 0
}
var d struct {
Sequence int64 `json:"sequence"`
}
if err := json.Unmarshal(signed.Declaration, &d); err != nil {
return 0
}
return d.Sequence
}
// declaredIn is the id a signed declaration carries, for a report about one that was not applied.
// Empty if the message is not one — a forged or garbled message is refused by handleBody when its
// turn comes; here it is only named.
@@ -392,3 +424,36 @@ func publishAlive(ctx context.Context, bus Bus, m Membership, say Announce,
say("could not tell the mesh this node is here: " + err.Error())
}
}
// heldNote is what this apply did NOT do, for the line that says what it did.
//
// **A count that does not add up is the only symptom a held resource had** (novox/hq 04-ISSUES/125).
// An adopted node keeps what it found until its module is taken (ADR 0100), and that is correct — but
// it was recorded only in the node's own state file. On the edge cut-over the mesh sent 346 resources,
// the journal said it applied 330, and nothing anywhere said which sixteen or why. Reading it took
// opening state.json by hand; not reading it took every public name on the machine down, because the
// operator had four green surfaces and a discrepancy nobody could interpret.
//
// So the line that reports the apply carries it. Grouped by module and ordered by name, because the
// sentence an operator needs is "route-proxy is assigned and not taken", and the module is the thing
// they can act on — `take` is the verb, and it takes a module.
func heldNote(held []Held) string {
if len(held) == 0 {
return ""
}
byModule := map[string]int{}
for _, h := range held {
byModule[h.Module]++
}
names := make([]string, 0, len(byModule))
for name := range byModule {
names = append(names, name)
}
sort.Strings(names)
parts := make([]string, 0, len(names))
for _, name := range names {
parts = append(parts, fmt.Sprintf("%s: %d", name, byModule[name]))
}
return fmt.Sprintf(", %d held until their module is taken (%s)",
len(held), strings.Join(parts, ", "))
}
+185
View File
@@ -0,0 +1,185 @@
// Package outward reads which of this machine's links face outside it (novox/hq ADR 0140).
//
// The filter the mesh derives constrains traffic arriving from outside the machine and says nothing
// about traffic that did not. To write that rule the mesh has to know which links "outside" arrives
// on, and that is a thing only the machine can say — so it says it, once per report, the way it
// already reports the kind of firewall it found and the tunnel it carried.
//
// **It replaces a list of addresses.** The filter used to allow the machine's own containers back
// through by naming the address ranges they sit on: two ranges fixed in the control plane's source
// and the rest typed by an operator. A range describes one machine and goes stale silently
// (novox/hq 04-ISSUES/137 and /141). A link that carries the default route is a fact the machine
// reads afresh every time, and it does not change when a module is added or removed.
//
// It reads the kernel's routing tables directly rather than asking a program. A module naming a
// program the machine does not have is how the mesh already reported success while doing nothing
// (novox/hq 04-ISSUES/136), and every machine has /proc.
package outward
import (
"bufio"
"fmt"
"os"
"path/filepath"
"sort"
"strings"
)
// ProcNet is where the kernel publishes its routing tables. A parameter so a test can hold a
// routing table without one.
const ProcNet = "/proc/net"
// SysClassNet is where the kernel lists the machine's network interfaces, one directory each. A
// parameter for the same reason.
const SysClassNet = "/sys/class/net"
// Links are the interfaces carrying a default route, for both address families, and every interface
// backed by a physical device, sorted and without repeats.
//
// **A physical link faces outside whether or not it is up** (novox/hq issue 197). The filter accepts
// whatever did not arrive on a link named here, so a link left out of this list is not filtered at
// all. A cable unplugged when the machine last reported carries no default route, and was left out:
// plugged in, everything arriving on it was accepted until the next report and the next push — and a
// second physical link that never carries the default route was never filtered. A physical device is
// read from the kernel's own list, where it has a `device` entry; a bridge, a veth, the tunnel and the
// loopback have none, and stay what they are, this machine's own.
//
// A machine may have more than one: a laptop with a cable and a radio has two, and both face
// outside. A machine with none — no route off itself — returns nothing, and the mesh refuses to
// compose a filter for it rather than writing a rule around a link with no name, which would be a
// rule set that does not load and a machine filtering nothing while its unit reports success.
func Links(procNet, sysClassNet string) ([]string, error) {
if procNet == "" {
procNet = ProcNet
}
if sysClassNet == "" {
sysClassNet = SysClassNet
}
seen := map[string]bool{}
four, err := defaultsV4(filepath.Join(procNet, "route"))
if err != nil {
return nil, err
}
six, err := defaultsV6(filepath.Join(procNet, "ipv6_route"))
if err != nil {
return nil, err
}
devices, err := physical(sysClassNet)
if err != nil {
return nil, err
}
for _, name := range append(append(four, six...), devices...) {
if name != "" && name != "lo" {
seen[name] = true
}
}
out := make([]string, 0, len(seen))
for name := range seen {
out = append(out, name)
}
sort.Strings(out)
return out, nil
}
// physical is every interface the kernel lists with a device behind it. A list that is not there is
// not an error — a machine without sysfs mounted reports what its routing table says, as before.
func physical(sysClassNet string) ([]string, error) {
entries, err := os.ReadDir(sysClassNet)
if os.IsNotExist(err) {
return nil, nil
}
if err != nil {
return nil, err
}
var out []string
for _, e := range entries {
if _, err := os.Stat(filepath.Join(sysClassNet, e.Name(), "device")); err == nil {
out = append(out, e.Name())
}
}
return out, nil
}
// defaultsV4 reads /proc/net/route, whose columns are
//
// Iface Destination Gateway Flags RefCnt Use Metric Mask ...
//
// with addresses in hexadecimal. A default route is destination zero with mask zero — the mask
// matters, because a route to the zero address with a real mask is not a default route.
func defaultsV4(path string) ([]string, error) {
lines, err := rows(path)
if err != nil {
return nil, err
}
var out []string
for _, fields := range lines {
if len(fields) < 8 {
continue
}
if isZeroHex(fields[1]) && isZeroHex(fields[7]) {
out = append(out, fields[0])
}
}
return out, nil
}
// defaultsV6 reads /proc/net/ipv6_route, whose columns are
//
// dest destprefix src srcprefix nexthop metric refcnt use flags iface
//
// A default route is the zero destination with a zero prefix length.
func defaultsV6(path string) ([]string, error) {
lines, err := rows(path)
if err != nil {
return nil, err
}
var out []string
for _, fields := range lines {
if len(fields) < 10 {
continue
}
if isZeroHex(fields[0]) && isZeroHex(fields[1]) {
out = append(out, fields[9])
}
}
return out, nil
}
// rows reads a routing table into fields per line, skipping a header and blank lines. A table that
// is not there is not an error: a machine without the second address family has no file for it,
// and that is not a machine that cannot be filtered.
func rows(path string) ([][]string, error) {
file, err := os.Open(path)
if os.IsNotExist(err) {
return nil, nil
}
if err != nil {
return nil, fmt.Errorf("cannot read the routing table at %s: %w", path, err)
}
defer file.Close()
var out [][]string
scanner := bufio.NewScanner(file)
for scanner.Scan() {
line := strings.TrimSpace(scanner.Text())
if line == "" || strings.HasPrefix(line, "Iface") {
continue
}
out = append(out, strings.Fields(line))
}
if err := scanner.Err(); err != nil {
return nil, fmt.Errorf("cannot read the routing table at %s: %w", path, err)
}
return out, nil
}
// isZeroHex is whether a hexadecimal field is all zeroes, whatever its width — the v4 table writes
// eight digits and the v6 table thirty-two, and a prefix length is two.
func isZeroHex(field string) bool {
if field == "" {
return false
}
return strings.Trim(strings.ToLower(field), "0") == ""
}
+156
View File
@@ -0,0 +1,156 @@
package outward
import (
"os"
"path/filepath"
"reflect"
"testing"
)
// A routing table as the kernel writes it: a default route, a route to the zero address that is not
// one, and a route on the loopback. Only the default route's link faces outside.
const routeV4 = `Iface Destination Gateway Flags RefCnt Use Metric Mask MTU Window IRTT
enp9s0 00000000 01FEA8C0 0003 0 0 100 00000000 0 0 0
docker0 000011AC 00000000 0001 0 0 0 0000FFFF 0 0 0
enp9s0 00000000 00000000 0001 0 0 100 00FFFFFF 0 0 0
lo 00000000 00000000 0003 0 0 0 00000000 0 0 0
`
const routeV6 = `00000000000000000000000000000000 00 00000000000000000000000000000000 00 fe800000000000000000000000000001 00000400 00000001 00000000 00000003 wlan0
fd0000000000000000000000000000000 40 00000000000000000000000000000000 00 00000000000000000000000000000000 00000100 00000000 00000000 00000001 enp9s0
`
func write(t *testing.T, dir, name, body string) {
t.Helper()
if err := os.WriteFile(filepath.Join(dir, name), []byte(body), 0o644); err != nil {
t.Fatal(err)
}
}
func TestLinksAreTheOnesCarryingADefaultRoute(t *testing.T) {
dir := t.TempDir()
write(t, dir, "route", routeV4)
write(t, dir, "ipv6_route", routeV6)
got, err := Links(dir, t.TempDir())
if err != nil {
t.Fatal(err)
}
// The cable from the v4 table and the radio from the v6 one. Not docker0, whose route is not a
// default; not the loopback, which faces nothing; and not the v6 route with a real prefix.
want := []string{"enp9s0", "wlan0"}
if !reflect.DeepEqual(got, want) {
t.Fatalf("outward links are %v, want %v", got, want)
}
}
// A route to the zero address with a real mask is not a default route. Trusting the destination
// alone would name every link with such a route as facing outside, and a filter that treats an
// internal bridge as outward constrains this machine's own guests — the fault ADR 0140 removes.
func TestAZeroDestinationWithAMaskIsNotADefaultRoute(t *testing.T) {
dir := t.TempDir()
write(t, dir, "route", `Iface Destination Gateway Flags RefCnt Use Metric Mask MTU Window IRTT
br-abc 00000000 00000000 0001 0 0 0 00FFFFFF 0 0 0
`)
got, err := Links(dir, t.TempDir())
if err != nil {
t.Fatal(err)
}
if len(got) != 0 {
t.Fatalf("outward links are %v, want none", got)
}
}
// A machine with no route off itself says so, rather than guessing. The mesh refuses to compose a
// filter for it; a rule written around a link with no name does not load, and a rule set that does
// not load is a machine filtering nothing while its unit reports success.
func TestNoDefaultRouteIsNoLinks(t *testing.T) {
dir := t.TempDir()
write(t, dir, "route", "Iface\tDestination\tGateway \tFlags\tRefCnt\tUse\tMetric\tMask\t\tMTU\tWindow\tIRTT\n")
got, err := Links(dir, t.TempDir())
if err != nil {
t.Fatal(err)
}
if len(got) != 0 {
t.Fatalf("outward links are %v, want none", got)
}
}
// A machine without the second address family has no file for it. That is not a machine that cannot
// be filtered, so a missing table is read as no routes rather than as a failure.
func TestAMissingTableIsNotAFailure(t *testing.T) {
dir := t.TempDir()
write(t, dir, "route", routeV4)
got, err := Links(dir, t.TempDir())
if err != nil {
t.Fatalf("a missing v6 table should not fail: %v", err)
}
if !reflect.DeepEqual(got, []string{"enp9s0"}) {
t.Fatalf("outward links are %v, want [enp9s0]", got)
}
}
// The same link carrying a default route in both families is reported once.
func TestALinkIsReportedOnce(t *testing.T) {
dir := t.TempDir()
write(t, dir, "route", routeV4)
write(t, dir, "ipv6_route",
"00000000000000000000000000000000 00 00000000000000000000000000000000 00 "+
"fe800000000000000000000000000001 00000400 00000001 00000000 00000003 enp9s0\n")
got, err := Links(dir, t.TempDir())
if err != nil {
t.Fatal(err)
}
if !reflect.DeepEqual(got, []string{"enp9s0"}) {
t.Fatalf("outward links are %v, want [enp9s0]", got)
}
}
// Against this machine's own routing table, so the parse is held to what the kernel actually writes
// and not only to a fixture written to agree with it.
func TestAgainstThisMachinesOwnTable(t *testing.T) {
got, err := Links("", "")
if err != nil {
t.Fatal(err)
}
if len(got) == 0 {
t.Skip("this machine has no default route")
}
t.Logf("this machine's outward links: %v", got)
}
// sysNet is a /sys/class/net: each name a directory, with a `device` entry when a device backs it.
func sysNet(t *testing.T, physical []string, virtual []string) string {
t.Helper()
dir := t.TempDir()
for _, name := range physical {
if err := os.MkdirAll(filepath.Join(dir, name, "device"), 0o755); err != nil {
t.Fatal(err)
}
}
for _, name := range virtual {
if err := os.MkdirAll(filepath.Join(dir, name), 0o755); err != nil {
t.Fatal(err)
}
}
return dir
}
// **A physical link faces outside whether or not it carries the default route** (novox/hq issue
// 197). A machine on its radio with its cable unplugged reported only the radio, and the filter then
// accepted everything arriving on the cable the moment it was plugged in. Bridges, veths, the tunnel
// and the loopback have no device behind them and stay this machine's own.
func TestEveryPhysicalLinkFacesOutsideUpOrDown(t *testing.T) {
proc := t.TempDir()
write(t, proc, "route", `Iface Destination Gateway Flags RefCnt Use Metric Mask MTU Window IRTT
wlp5s0 00000000 01FEA8C0 0003 0 0 600 00000000 0 0 0
`)
sys := sysNet(t, []string{"wlp5s0", "enp6s0"}, []string{"lo", "docker0", "br-0123456789ab", "veth1", "mesh0"})
got, err := Links(proc, sys)
if err != nil {
t.Fatal(err)
}
if want := []string{"enp6s0", "wlp5s0"}; !reflect.DeepEqual(got, want) {
t.Fatalf("outward links are %v, want %v", got, want)
}
}
+31
View File
@@ -15,11 +15,22 @@ const (
CapServiceManager = "service-manager"
CapFirewall = "firewall"
CapOverlay = "overlay"
// CapVirtualisation is a running virtualisation daemon: what the lab raises its machines on
// (novox/hq ADR 0172), and what grants a module the daemon's socket.
CapVirtualisation = "virtualisation"
CapGraphicalSession = "graphical-session"
// CapSeat is hardware: somewhere a display server COULD run. CapGraphicalSession above is
// state: whether one IS running. Assignment needs the first.
CapSeat = "seat"
CapPrivileged = "privileged"
// The network manager this machine runs, one capability per dialect (novox/hq ADR 0161): the
// uplink seat's holder declares its own, so the holder for a manager the machine does not run
// is refused the way any missing capability is, naming it. Active, not installed — a machine
// may have two of these on disk and runs one.
CapUplinkNetworkManager = "uplink-networkmanager"
CapUplinkSystemdNetworkd = "uplink-systemd-networkd"
CapUplinkDhcpcd = "uplink-dhcpcd"
)
// commandCapability is the shape most detectors take: run something, and treat a working
@@ -197,11 +208,31 @@ func Default(runner Runner) []Detector {
why: "lists the ruleset — needs the tool AND the privilege to use it",
runner: runner,
},
commandCapability{
name: CapVirtualisation, command: "incus", args: []string{"info"},
why: "asks the virtualisation daemon about itself — a running daemon, not an installed client",
runner: runner,
},
commandCapability{
name: CapOverlay, command: "wg", args: []string{"show", "interfaces"},
why: "asks the kernel for interfaces — needs the module, not just the tool",
runner: runner,
},
commandCapability{
name: CapUplinkNetworkManager, command: "systemctl", args: []string{"is-active", "NetworkManager.service"},
why: "asks the init whether NetworkManager is running — the dialect the uplink seat's holder must speak",
runner: runner,
},
commandCapability{
name: CapUplinkSystemdNetworkd, command: "systemctl", args: []string{"is-active", "systemd-networkd.service"},
why: "asks the init whether systemd-networkd is running — the dialect the uplink seat's holder must speak",
runner: runner,
},
commandCapability{
name: CapUplinkDhcpcd, command: "systemctl", args: []string{"is-active", "dhcpcd.service"},
why: "asks the init whether dhcpcd is running — the dialect the uplink seat's holder must speak",
runner: runner,
},
}
}
+39
View File
@@ -0,0 +1,39 @@
package profile
import (
"context"
"errors"
"testing"
)
// The uplink seat's holder must be the dialect the machine runs (novox/hq ADR 0161): the profile
// names the network manager found active, one capability per manager, and nothing for one that is
// merely installed.
func TestTheProfileNamesTheNetworkManagerThatIsRunning(t *testing.T) {
runner := func(_ context.Context, name string, args ...string) (string, error) {
if name == "systemctl" && len(args) == 2 && args[0] == "is-active" {
if args[1] == "NetworkManager.service" {
return "active\n", nil
}
return "inactive\n", errors.New("exit status 3")
}
return "", errors.New("not here")
}
var have []Detector
for _, d := range Default(Runner(runner)) {
switch d.Name() {
case CapUplinkNetworkManager, CapUplinkSystemdNetworkd, CapUplinkDhcpcd:
have = append(have, d)
}
}
if len(have) != 3 {
t.Fatalf("expected a detector per manager, found %d", len(have))
}
for _, d := range have {
v := d.Detect(context.Background())
want := d.Name() == CapUplinkNetworkManager
if v.Present != want {
t.Errorf("%s: present=%v, want %v (%s)", d.Name(), v.Present, want, v.Detail)
}
}
}
+29
View File
@@ -0,0 +1,29 @@
package store
import "testing"
// A resource whose target moves leaves what the host wrote under the old target on record as a
// former one, undeclared by construction, so the next apply removes it (novox/hq issue 097, ADR 0163).
func TestARecordWhoseTargetMovedKeepsTheFormerTargetToRemove(t *testing.T) {
s := State{}
s.Record(Applied{ID: "gitea.server", Type: "container", Target: "mesh-gitea", Origin: OriginDeclared})
s.Record(Applied{ID: "gitea.server", Type: "container", Target: "gitea", Origin: OriginDeclared})
if len(s.Resources) != 2 {
t.Fatalf("a moved target produced %d record(s): %+v", len(s.Resources), s.Resources)
}
orphans := s.Orphans(map[string]bool{"gitea.server": true}, OriginDeclared)
if len(orphans) != 1 || orphans[0].Target != "mesh-gitea" || !IsFormer(orphans[0].ID) {
t.Fatalf("the former target is not an orphan to remove: %+v", orphans)
}
s.Forget(orphans[0].ID)
if len(s.Resources) != 1 || s.Resources[0].Target != "gitea" {
t.Fatalf("forgetting the former target touched the current one: %+v", s.Resources)
}
// The same target again is not a move; a carried record is not the host's to remove.
s.Record(Applied{ID: "gitea.server", Type: "container", Target: "gitea", Origin: OriginDeclared})
s.Record(Applied{ID: "bundle", Type: "file", Target: "/a"})
s.Record(Applied{ID: "bundle", Type: "file", Target: "/b"})
if len(s.Resources) != 2 {
t.Fatalf("an unmoved or carried record grew the list: %+v", s.Resources)
}
}
+63 -1
View File
@@ -18,6 +18,7 @@ import (
"os"
"path/filepath"
"sort"
"strings"
"time"
)
@@ -231,8 +232,13 @@ type FoundFirewall struct {
// retires, and returning it to adopted restores.
WasActive bool `json:"was_active,omitempty"`
// DisabledByMesh is set when converging retired it, so returning to adopted enables it again
// and nothing else ever does.
// and nothing else ever does. It means exactly that (novox/hq ADR 0168): a reconcile that finds
// the firewall already inactive records RetiredBy and never this.
DisabledByMesh bool `json:"disabled_by_mesh,omitempty"`
// RetiredBy says how the found firewall came to be inactive on a converged machine: "mesh" when
// the mesh disabled it, "found-inactive" when a reconcile found it so and nothing of the mesh's
// had done it. Empty while it is in force or the machine is adopted.
RetiredBy string `json:"retired_by,omitempty"`
// Forward is each family's forward policy as it was before the mesh disabled the firewall,
// by the tool that sets it — recorded before, so a retirement retried puts back what the
// machine had.
@@ -274,6 +280,41 @@ type Held struct {
// reverted: that is how a predecessor still writing is caught.
Changed string `json:"changed,omitempty"`
ChangedAt time.Time `json:"changed_at,omitempty"`
// Facts is what a take would compare: the found thing beside what the module declares
// (novox/hq ADR 0163). Read fresh on every apply while held, so the controller's preview
// speaks of the machine as it is.
Facts *Facts `json:"facts,omitempty"`
}
// Facts is a held thing beside what its module declares — what a take compares (ADR 0163).
type Facts struct {
// A found container: the image it runs and when that image was made; the networks it is on
// and the other containers on each; what it mounts; what it publishes.
Image string `json:"image,omitempty"`
ImageCreated string `json:"image_created,omitempty"`
Networks map[string][]string `json:"networks,omitempty"`
Mounts []string `json:"mounts,omitempty"`
Ports []string `json:"ports,omitempty"`
// What the module declares for it, and the declared image's creation date when the image
// is on the machine already.
DeclaredImage string `json:"declared_image,omitempty"`
DeclaredImageCreated string `json:"declared_image_created,omitempty"`
DeclaredPorts []string `json:"declared_ports,omitempty"`
DeclaredVolumes []string `json:"declared_volumes,omitempty"`
// Downgrade is true when both creation dates are known and the declared image is the older.
Downgrade bool `json:"downgrade,omitempty"`
// A found file: whether the declared content differs from what was found, and how, as lines
// only in the found file (-) and lines only in the declared one (+), bounded.
Differs bool `json:"differs,omitempty"`
Difference []string `json:"difference,omitempty"`
}
// A Stray is something running on the machine that the mesh neither wrote nor holds
// (novox/hq ADR 0163): the answer to "what is here that nobody asked for".
type Stray struct {
Kind string `json:"kind"`
Name string `json:"name"`
Detail string `json:"detail,omitempty"`
}
// Recorded reports whether this host has a record, of any origin, of putting something of this
@@ -462,6 +503,20 @@ func Save(path string, s State) error {
func (s *State) Record(a Applied) {
for i, existing := range s.Resources {
if existing.ID == a.ID {
// A resource whose target moved leaves what the host wrote under the old target
// behind — a container under the old name, a file at the old path. Rewriting the
// record would erase the only trace of it (novox/hq issue 097, ADR 0163), so the old
// target stays on record as a former one, undeclared by construction, until the next
// apply removes it the way it removes anything the host wrote and no longer declares.
// What was found is held, never recorded here, and so never removed by this.
if originOf(existing) == OriginDeclared && existing.Target != "" && a.Target != "" &&
existing.Target != a.Target && existing.Type == a.Type {
former := existing
former.ID = FormerID(existing.ID, existing.Target)
s.Resources[i] = a
s.Resources = append(s.Resources, former)
return
}
s.Resources[i] = a
return
}
@@ -469,6 +524,13 @@ func (s *State) Record(a Applied) {
s.Resources = append(s.Resources, a)
}
// FormerID names the record of a resource's former target: the resource's id and the target it
// had, so the record is distinct from the current one and is never what a declaration names.
func FormerID(id, target string) string { return id + "@former:" + target }
// IsFormer says whether a record names a former target.
func IsFormer(id string) bool { return strings.Contains(id, "@former:") }
// Forget drops a resource from what the node owns.
func (s *State) Forget(id string) {
kept := s.Resources[:0]
+5
View File
@@ -46,6 +46,11 @@ func (a alpine) PackageInstalled(ctx context.Context, run Runner, name string) (
return strings.TrimSpace(out) != "", nil
}
func (alpine) RemovePackage(ctx context.Context, run Runner, name string) error {
_, err := run(ctx, "apk", "del", name)
return err
}
func (alpine) InstallPackage(ctx context.Context, run Runner, name string) error {
_, err := run(ctx, "apk", "add", "--no-cache", name)
return err
+4
View File
@@ -65,6 +65,10 @@ func (a android) InstallPackage(context.Context, Runner, string) error {
return fmt.Errorf("%w: package", ErrUnsupported)
}
func (a android) RemovePackage(context.Context, Runner, string) error {
return fmt.Errorf("%w: package", ErrUnsupported)
}
func (a android) ServiceState(context.Context, Runner, string) (string, error) {
return "", fmt.Errorf("%w: service (init is not reachable without root)", ErrUnsupported)
}
+8
View File
@@ -39,6 +39,14 @@ func (a arch) PackageInstalled(ctx context.Context, run Runner, name string) (bo
return true, nil
}
// RemovePackage removes one package and nothing it depends on: `-R`, not `-Rs`, because what else
// relied on a dependency is not this declaration's to know. pacman keeps a configuration file the
// operator changed as `.pacsave`, which is what "never flushed" comes to once the front end is gone.
func (arch) RemovePackage(ctx context.Context, run Runner, name string) error {
_, err := run(ctx, "pacman", "-R", "--noconfirm", name)
return err
}
func (arch) InstallPackage(ctx context.Context, run Runner, name string) error {
out, err := run(ctx, "pacman", "-S", "--noconfirm", "--needed", name)
if err == nil {
+3
View File
@@ -51,6 +51,9 @@ type System interface {
PackageInstalled(ctx context.Context, run Runner, name string) (bool, error)
InstallPackage(ctx context.Context, run Runner, name string) error
// RemovePackage uninstalls one package, leaving its dependencies and anything the operator
// changed in its configuration where the package manager leaves them (novox/hq ADR 0175).
RemovePackage(ctx context.Context, run Runner, name string) error
// ServiceState is "running" or "stopped". A unit that does not exist is an error, never
// "stopped" — reporting absence as satisfaction is the fault this host exists to prevent.
+167
View File
@@ -16,7 +16,9 @@ import (
"fmt"
"os"
"path/filepath"
"sort"
"strings"
"time"
)
// Files the launcher reads and this binary writes. Next to the store, because they are node
@@ -157,3 +159,168 @@ func ReadKnownGood(path string) (string, error) {
}
return strings.TrimSpace(string(raw)), nil
}
// Where delivered versions live, and what the binary inside one is called.
//
// **A directory named for its version, never a link and never a write over what is running**
// (novox/hq ADR 0141). Two facts follow from that one choice: the kernel refuses to truncate a
// running executable, so the path a delivery writes must not be the path being executed; and a
// rollback needs the previous version still present, which a single path cannot offer.
//
// The mesh creates no links (novox/hq ADR 0012), so nothing points at "current". The version is in
// the path, which is why nothing has to be told what is running.
const (
// DefaultLibexec is where the host's own files live. Fixed rather than derived from where the
// running executable sits: the first host to understand any of this was copied to a machine by
// hand, and one that looked for its successor beside itself would never find a delivered version
// — which is every machine in this mesh on the day this ships.
DefaultLibexec = "/usr/lib/nox-mesh-host"
VersionsDirName = "versions"
BinaryName = "nox-mesh-host"
// PinnedName is the version a rollback chose, which the launcher runs instead of the newest.
// Without it the launcher would start the newest again and the rollback would flap.
PinnedName = "rollback-pinned"
)
// VersionsDir is where delivered versions live, given where the host's libexec is. An empty libexec
// means the default, and the environment overrides it so a test needs no root.
func VersionsDir(libexec string) string {
if libexec == "" {
libexec = os.Getenv("MESH_HOST_LIBEXEC")
}
if libexec == "" {
libexec = DefaultLibexec
}
return filepath.Join(libexec, VersionsDirName)
}
// PinnedPath is where a rollback records the version it chose.
func PinnedPath(statePath string) string {
return filepath.Join(filepath.Dir(statePath), PinnedName)
}
// Delivered is one version present on the machine.
type Delivered struct {
// Version is the directory's name, which is the version.
Version string
// Binary is the executable inside it.
Binary string
// At is when it arrived, which is how "newest" is decided.
At time.Time
}
// Versions are the versions delivered to this machine, newest first.
//
// **Newest by when it arrived, not by its name.** A version string comes from what the source was
// tagged or described as, and those do not sort: "1.10" before "1.9", a commit hash before either.
// Ordering by name would run an older host and call it an upgrade. When it arrived is a fact the
// filesystem keeps and the delivery sets.
//
// A directory with no executable in it is not a version. A delivery that was interrupted leaves one,
// and running the newest would then mean running nothing.
func Versions(dir string) ([]Delivered, error) {
entries, err := os.ReadDir(dir)
if errors.Is(err, os.ErrNotExist) {
return nil, nil
}
if err != nil {
return nil, fmt.Errorf("cannot read the delivered versions at %s: %w", dir, err)
}
var out []Delivered
for _, entry := range entries {
if !entry.IsDir() {
continue
}
binary := filepath.Join(dir, entry.Name(), BinaryName)
info, err := os.Stat(binary)
if err != nil || info.IsDir() {
continue
}
at := info.ModTime()
if d, err := entry.Info(); err == nil && d.ModTime().After(at) {
at = d.ModTime()
}
out = append(out, Delivered{Version: entry.Name(), Binary: binary, At: at})
}
// Newest first, and by name when two arrived in the same instant so the answer is never
// arbitrary — a test that passes half the time is worse than one that fails.
sort.Slice(out, func(a, b int) bool {
if out[a].At.Equal(out[b].At) {
return out[a].Version > out[b].Version
}
return out[a].At.After(out[b].At)
})
return out, nil
}
// Successor is the version this machine should be running instead of the given one, if any.
//
// Empty when the running version is the newest, which is the ordinary answer. The host asks this
// between reconciles and nowhere else: standing aside mid-apply is the half-configured machine the
// host exists to prevent (novox/hq ADR 0141).
func Successor(dir, running string) (Delivered, bool, error) {
delivered, err := Versions(dir)
if err != nil {
return Delivered{}, false, err
}
if len(delivered) == 0 {
return Delivered{}, false, nil
}
newest := delivered[0]
// A machine whose running version is not among the delivered ones is the machine every mesh has
// one of: the host was put there by hand before any of this existed. Treating that as "stand
// aside" is correct — what was delivered is what the mesh asked for.
if newest.Version == running {
return Delivered{}, false, nil
}
return newest, true, nil
}
// Retire removes delivered versions older than the running one's predecessor.
//
// The running version and the one before it are kept, and nothing else: the predecessor is exactly
// what a rollback starts, and every version before that is weight with no reader. Called after a
// reconcile completes, which is the same evidence known-good is written on — retiring on any weaker
// signal would delete the thing a failing host is about to need.
//
// Never the running version, whatever it is asked. A host that deleted its own image would survive
// until it stopped and then be unstartable, and the launcher's rollback reads a version, not a
// process.
func Retire(dir, running string) ([]string, error) {
delivered, err := Versions(dir)
if err != nil {
return nil, err
}
keep := map[string]bool{running: true}
for i, d := range delivered {
if d.Version != running {
continue
}
// Its predecessor is the next one down the list, which is the next oldest.
if i+1 < len(delivered) {
keep[delivered[i+1].Version] = true
}
break
}
// A running version that was never delivered has no predecessor among these, so the newest
// delivered one is what a rollback would reach for. Keep it.
if len(keep) == 1 && len(delivered) > 0 {
keep[delivered[0].Version] = true
}
var removed []string
for _, d := range delivered {
if keep[d.Version] {
continue
}
if err := os.RemoveAll(filepath.Join(dir, d.Version)); err != nil {
return removed, fmt.Errorf("cannot retire the host version %s: %w", d.Version, err)
}
removed = append(removed, d.Version)
}
sort.Strings(removed)
return removed, nil
}
+180
View File
@@ -0,0 +1,180 @@
package upgrade
import (
"os"
"path/filepath"
"reflect"
"sort"
"testing"
"time"
)
// deliver writes a version as a delivery would: a directory named for it with the binary inside.
// at fixes when it arrived, because "newest" is when it arrived and a test must not race the clock.
func deliver(t *testing.T, dir, version string, at time.Time) string {
t.Helper()
into := filepath.Join(dir, version)
if err := os.MkdirAll(into, 0o755); err != nil {
t.Fatal(err)
}
binary := filepath.Join(into, BinaryName)
if err := os.WriteFile(binary, []byte("#!/bin/sh\nexit 0\n"), 0o755); err != nil {
t.Fatal(err)
}
if err := os.Chtimes(binary, at, at); err != nil {
t.Fatal(err)
}
if err := os.Chtimes(into, at, at); err != nil {
t.Fatal(err)
}
return binary
}
// **Newest is when it arrived, not how its name sorts.**
//
// A version string is whatever the source was tagged or described as, and those do not sort: "1.10"
// orders before "1.9", and a commit hash orders before either. Ordering by name would start an older
// host and call that an upgrade.
func TestNewestIsWhenItArrivedAndNotHowItSorts(t *testing.T) {
dir := t.TempDir()
base := time.Now().Add(-time.Hour)
deliver(t, dir, "1.10", base) // sorts LAST by name, arrived first
deliver(t, dir, "1.9", base.Add(time.Minute)) // sorts first by name, arrived last
got, err := Versions(dir)
if err != nil {
t.Fatal(err)
}
if len(got) != 2 || got[0].Version != "1.9" {
t.Fatalf("newest is %+v, want the one that arrived last (1.9)", got)
}
}
// A delivery that was interrupted leaves a directory with no executable in it. Running "the newest"
// would then mean running nothing, so it is not a version.
func TestADirectoryWithNoBinaryIsNotAVersion(t *testing.T) {
dir := t.TempDir()
if err := os.MkdirAll(filepath.Join(dir, "half-delivered"), 0o755); err != nil {
t.Fatal(err)
}
deliver(t, dir, "good", time.Now().Add(-time.Hour))
got, err := Versions(dir)
if err != nil {
t.Fatal(err)
}
if len(got) != 1 || got[0].Version != "good" {
t.Fatalf("versions are %+v, want only the one with a binary", got)
}
}
// Nothing delivered is not a fault. A machine whose host was placed by hand has no versions
// directory at all, and that must read as "no successor" rather than as an error that stops a
// reconcile.
func TestNoVersionsDirectoryIsNotAnError(t *testing.T) {
got, err := Versions(filepath.Join(t.TempDir(), "absent"))
if err != nil {
t.Fatalf("an absent versions directory should not be an error: %v", err)
}
if len(got) != 0 {
t.Fatalf("versions are %+v, want none", got)
}
}
func TestTheNewestVersionIsTheSuccessorAndTheRunningOneIsNot(t *testing.T) {
dir := t.TempDir()
base := time.Now().Add(-time.Hour)
deliver(t, dir, "one", base)
deliver(t, dir, "two", base.Add(time.Minute))
next, yes, err := Successor(dir, "one")
if err != nil {
t.Fatal(err)
}
if !yes || next.Version != "two" {
t.Fatalf("successor is %+v (%v), want two", next, yes)
}
if _, yes, err := Successor(dir, "two"); err != nil || yes {
t.Fatalf("the newest version is its own successor (%v, %v)", yes, err)
}
}
// A host put there by hand, before any of this existed, is not among the delivered versions. What the
// mesh delivered is what it asked for, so that is a successor — otherwise the first delivery to such a
// machine would be ignored for ever, which is every machine in this mesh today.
func TestAHostThatWasNeverDeliveredHasASuccessor(t *testing.T) {
dir := t.TempDir()
deliver(t, dir, "delivered", time.Now().Add(-time.Hour))
next, yes, err := Successor(dir, "copied-by-hand")
if err != nil {
t.Fatal(err)
}
if !yes || next.Version != "delivered" {
t.Fatalf("successor is %+v (%v), want the delivered one", next, yes)
}
}
// The running version and its predecessor are kept, and nothing else. The predecessor is exactly what
// a rollback starts; everything older has no reader.
func TestRetireKeepsTheRunningVersionAndItsPredecessor(t *testing.T) {
dir := t.TempDir()
base := time.Now().Add(-4 * time.Hour)
for i, v := range []string{"one", "two", "three", "four"} {
deliver(t, dir, v, base.Add(time.Duration(i)*time.Hour))
}
removed, err := Retire(dir, "four")
if err != nil {
t.Fatal(err)
}
sort.Strings(removed)
if !reflect.DeepEqual(removed, []string{"one", "two"}) {
t.Fatalf("retired %v, want one and two — three is the predecessor a rollback needs", removed)
}
for _, kept := range []string{"three", "four"} {
if _, err := os.Stat(filepath.Join(dir, kept, BinaryName)); err != nil {
t.Fatalf("%s was retired and a rollback now has nowhere to go: %v", kept, err)
}
}
}
// **Never the running version, whatever it is asked.** A host that deleted its own image would run
// until it stopped and then be unstartable, and the launcher's rollback reads a version rather than a
// process.
func TestRetireNeverRemovesTheRunningVersion(t *testing.T) {
dir := t.TempDir()
base := time.Now().Add(-2 * time.Hour)
deliver(t, dir, "older", base)
deliver(t, dir, "newer", base.Add(time.Hour))
// Asked while running the OLDER one, which is what a machine looks like between a delivery and
// the moment it stands aside.
if _, err := Retire(dir, "older"); err != nil {
t.Fatal(err)
}
if _, err := os.Stat(filepath.Join(dir, "older", BinaryName)); err != nil {
t.Fatalf("the running version was retired: %v", err)
}
}
// A machine running a hand-placed host keeps the newest delivered version, because that is what a
// rollback would reach for. Retiring it would leave the machine with no way back at all.
func TestRetireKeepsTheNewestWhenTheRunningVersionWasNeverDelivered(t *testing.T) {
dir := t.TempDir()
base := time.Now().Add(-3 * time.Hour)
deliver(t, dir, "old", base)
deliver(t, dir, "new", base.Add(time.Hour))
removed, err := Retire(dir, "copied-by-hand")
if err != nil {
t.Fatal(err)
}
if !reflect.DeepEqual(removed, []string{"old"}) {
t.Fatalf("retired %v, want only old — new is the rollback target", removed)
}
if _, err := os.Stat(filepath.Join(dir, "new", BinaryName)); err != nil {
t.Fatalf("the only delivered version was retired: %v", err)
}
}
+32
View File
File diff suppressed because one or more lines are too long
+65
View File
@@ -187,5 +187,70 @@ sleep 1
check "a crash is counted" "unlike a clean exit, which is not" "$(count)" "1"
kill -TERM "$LP" 2>/dev/null; sleep 1; pkill -f "$MESH_HOST_BIN" 2>/dev/null || true
# --- which version it runs (novox/hq ADR 0141) ------------------------------------------------
#
# Versions live side by side in directories named for them. The launcher picks one every time round
# the loop, never once: standing aside for a successor is a clean exit, and the next turn has to run
# what is on disk NOW — resolved once, the same binary would restart for ever and no upgrade would
# ever take.
# deliver a version as the mesh would, recording which one ran so a test can assert the choice.
deliver() {
mkdir -p "$MESH_HOST_LIBEXEC/versions/$1"
cat > "$MESH_HOST_LIBEXEC/versions/$1/nox-mesh-host" <<STUB
#!/bin/sh
echo "$1" >> "\$MESH_HOST_STATE_DIR/which.ran"
exit "\${STUB_HOST_EXIT:-1}"
STUB
chmod +x "$MESH_HOST_LIBEXEC/versions/$1/nox-mesh-host"
# When it arrived is what "newest" means, so it is set rather than left to the clock.
touch -d "$2" "$MESH_HOST_LIBEXEC/versions/$1/nox-mesh-host" "$MESH_HOST_LIBEXEC/versions/$1"
}
which_ran() { cat "$MESH_HOST_STATE_DIR/which.ran" 2>/dev/null || echo NONE; }
# Newest is when it arrived, not how its name sorts: "1.10" orders before "1.9" by name, so ordering
# by name would run an older host and call it an upgrade.
setup
deliver 1.10 "2 hours ago"
deliver 1.9 "1 hour ago"
"$LAUNCH" >/dev/null 2>&1 || true
check "runs the newest delivered version" "newest is when it arrived, not how the name sorts" \
"$(which_ran)" "1.9"
# A pin from a rollback beats the newest, or the launcher would start the failing binary again and
# the rollback would flap.
setup
deliver 1.9 "2 hours ago"
deliver 2.0 "1 hour ago"
echo 1.9 > "$MESH_HOST_STATE_DIR/rollback-pinned"
"$LAUNCH" >/dev/null 2>&1 || true
check "a pinned version beats the newest" "otherwise a rollback starts the binary it just rejected" \
"$(which_ran)" "1.9"
# A pin naming a version that is not there is ignored rather than fatal: the machine choosing for
# itself is better than a machine that starts nothing.
setup
deliver 2.0 "1 hour ago"
echo 1.9 > "$MESH_HOST_STATE_DIR/rollback-pinned"
"$LAUNCH" >/dev/null 2>&1 || true
check "an undeliverable pin is ignored" "a machine that starts nothing is worse than one that chooses" \
"$(which_ran)" "2.0"
# An interrupted delivery leaves a directory with no binary in it. Treating it as the newest would
# mean running nothing.
setup
deliver 1.9 "2 hours ago"
mkdir -p "$MESH_HOST_LIBEXEC/versions/2.0-half"
touch -d "1 minute ago" "$MESH_HOST_LIBEXEC/versions/2.0-half"
"$LAUNCH" >/dev/null 2>&1 || true
check "skips a version with no binary" "a directory is not a version; the binary is" \
"$(which_ran)" "1.9"
# Nothing delivered: the host placed by hand, which is how the first one always arrives. Without this
# the change would strand every machine in the mesh on the day it ships.
setup
"$LAUNCH" >/dev/null 2>&1 || true
check "falls back to the host placed by hand" "every first host arrives this way" "$(started)" "yes"
printf '\nlaunch: %d passed, %d failed\n' "$PASS" "$FAIL"
[ "$FAIL" -eq 0 ]
+48
View File
@@ -0,0 +1,48 @@
package packaging_test
import (
"encoding/json"
"os"
"testing"
)
// The mesh delivers the launcher, so the manifest carries a copy of it (novox/hq 04-ISSUES/142).
//
// **Two copies of one script is a drift waiting to happen**, and the only reason to accept it is that
// a file resource is written atomically — temp file, then rename — while an archive writes in place
// with truncate. The running launcher keeps the inode it was started from and the next start picks up
// the new one; unpacking an archive over it would truncate the file a running shell is reading.
//
// So: two copies, and this is the check that they are the same one.
func TestTheManifestCarriesTheLauncherExactly(t *testing.T) {
onDisk, err := os.ReadFile("nox-mesh-host-launch")
if err != nil {
t.Fatal(err)
}
raw, err := os.ReadFile("../module.json")
if err != nil {
t.Fatal(err)
}
var manifest struct {
Resources []struct {
ID string `json:"id"`
Content string `json:"content"`
} `json:"resources"`
}
if err := json.Unmarshal(raw, &manifest); err != nil {
t.Fatal(err)
}
for _, r := range manifest.Resources {
if r.ID != "launcher" {
continue
}
if r.Content != string(onDisk) {
t.Fatal("the launcher the mesh would deliver is not the launcher in this repository. " +
"Copy packaging/nox-mesh-host-launch into module.json's `launcher` resource — the " +
"machines run what the manifest says, and this file is what gets reviewed")
}
return
}
t.Fatal("module.json declares no `launcher` resource, so nothing delivers the launcher and a " +
"delivered host version is never started")
}
+44 -1
View File
@@ -16,16 +16,51 @@ set -u
STATE_DIR="${MESH_HOST_STATE_DIR:-/var/lib/mesh-host}"
LIBEXEC="${MESH_HOST_LIBEXEC:-/usr/lib/nox-mesh-host}"
HOST="${MESH_HOST_BIN:-/usr/bin/nox-mesh-host}"
# The host that was placed by hand, used only when nothing has been delivered. The first host on a
# machine always arrives this way; every one after it is delivered (novox/hq ADR 0141).
FALLBACK="${MESH_HOST_BIN:-/usr/bin/nox-mesh-host}"
VERSIONS="$LIBEXEC/versions"
BINARY="nox-mesh-host"
LIMIT="${MESH_HOST_START_LIMIT:-3}"
BACKOFF="${MESH_HOST_BACKOFF:-5}"
ONCE="${MESH_HOST_RUN_ONCE:-}" # tests run one iteration; nothing else sets this
ATTEMPTS="$STATE_DIR/start-attempts"
HALTED="$STATE_DIR/halted"
PINNED="$STATE_DIR/rollback-pinned"
say() { echo "nox-mesh-host-launch: $*" >&2; }
# Which host to run: the version a rollback pinned, or the most recently delivered one, or the one
# placed by hand when nothing has been delivered (novox/hq ADR 0141).
#
# **Asked every time round the loop, not once.** Standing aside for a successor is a clean exit, and
# the next turn has to run what is on disk NOW — resolving this once would restart the same binary
# for ever and the upgrade would never take.
#
# Newest by when it arrived, never by how its name sorts: a version string is whatever the source was
# described as, and those do not sort — "1.10" orders before "1.9". Ordering by name would start an
# older host and call it an upgrade.
pick_host() {
if [ -s "$PINNED" ]; then
pinned="$(tr -d '[:space:]' < "$PINNED" 2>/dev/null || true)"
if [ -n "$pinned" ] && [ -x "$VERSIONS/$pinned/$BINARY" ]; then
echo "$VERSIONS/$pinned/$BINARY"
return 0
fi
say "the pinned version '$pinned' is not delivered; ignoring the pin"
fi
# A directory with no executable in it is not a version: an interrupted delivery leaves one, and
# running "the newest" would then mean running nothing.
for candidate in $(ls -1t "$VERSIONS" 2>/dev/null || true); do
if [ -x "$VERSIONS/$candidate/$BINARY" ]; then
echo "$VERSIONS/$candidate/$BINARY"
return 0
fi
done
echo "$FALLBACK"
}
child=
stopping=
@@ -95,6 +130,14 @@ while :; do
fi
fi
HOST="$(pick_host)"
if [ ! -x "$HOST" ]; then
say "no host to run: nothing delivered under $VERSIONS and $FALLBACK is not executable."
printf 'no host binary\n' > "$HALTED"
exit 0
fi
say "running $HOST"
"$HOST" run &
child=$!
status=0
+29 -19
View File
@@ -1,20 +1,29 @@
#!/bin/sh
# Put the host back on the last version that worked.
#
# novox/hq ADR 0005. This runs when nox-mesh-host will not start, so it shares no code with it
# and calls none of it: a binary that cannot start cannot be its own recovery. POSIX sh, no
# novox/hq ADR 0005 and ADR 0141. This runs when nox-mesh-host will not start, so it shares no code
# with it and calls none of it: a binary that cannot start cannot be its own recovery. POSIX sh, no
# bashisms, nothing that has to be installed.
#
# It is deliberately dull. Everything it does is one of: read a file, run the package manager,
# ask the service manager to try again.
# It is deliberately dull. Everything it does is one of: read a file, look at a directory, write a
# file.
#
# **It used to reinstall a package.** It read the known-good version and asked one operating system's
# package manager for it, out of that package manager's cache. Two things were wrong with that. No
# machine in this mesh had the host installed as a package, so the recovery could not run on any of
# them; and the host is built per operating system (ADR 0005), so a recovery written in one package
# manager's terms could not run on two of the three. Versions now live side by side in directories
# named for them, so going back is choosing a directory — which is the same on every machine.
set -eu
STATE_DIR="${MESH_HOST_STATE_DIR:-/var/lib/mesh-host}"
PKG_CACHE="${MESH_HOST_PKG_CACHE:-/var/cache/pacman/pkg}"
PACKAGE="${MESH_HOST_PACKAGE:-nox-mesh-host}"
LIBEXEC="${MESH_HOST_LIBEXEC:-/usr/lib/nox-mesh-host}"
VERSIONS="$LIBEXEC/versions"
BINARY="nox-mesh-host"
KNOWN_GOOD="$STATE_DIR/known-good"
ATTEMPTED="$STATE_DIR/rollback-attempted"
PINNED="$STATE_DIR/rollback-pinned"
say() { echo "nox-mesh-host-rollback: $*" >&2; }
@@ -43,23 +52,24 @@ if [ -z "$VERSION" ]; then
exit 0
fi
PKG="$(ls "$PKG_CACHE"/"$PACKAGE"-"$VERSION"-*.pkg.tar.* 2>/dev/null | head -n 1 || true)"
if [ -z "$PKG" ]; then
say "known-good is $VERSION and no package for it is in $PKG_CACHE."
say "the cache was cleaned, or that version was never installed from here."
# The version that last worked may be the one that was placed by hand, which is not delivered and has
# no directory. Nothing to choose, and saying so is better than pinning a version that is not there —
# the launcher would ignore the pin and start the newest again, which is the binary that is failing.
if [ ! -x "$VERSIONS/$VERSION/$BINARY" ]; then
say "known-good is $VERSION and no such version is delivered under $VERSIONS."
say "it was retired, or that host was placed by hand and never delivered."
say "cannot roll back. this node needs a person."
exit 1
fi
say "rolling back to $VERSION ($PKG)"
say "rolling back to $VERSION ($VERSIONS/$VERSION/$BINARY)"
printf '%s\n' "$VERSION" > "$ATTEMPTED"
if ! pacman -U --noconfirm "$PKG"; then
say "the package manager refused to install $PKG."
exit 1
fi
# The pin is what stops the launcher starting the newest again. Written last, so a failure above
# leaves the machine choosing for itself rather than pinned to something this script did not verify.
printf '%s\n' "$VERSION" > "$PINNED"
# Deliberately does NOT start anything. The launcher called this and will exec the host next,
# so starting it here would run two. novox/hq ADR 0005 moved that responsibility; this script
# installs a version and says so, and nothing else.
say "rolled back to $VERSION. the launcher will start it."
# Deliberately does NOT start anything. The launcher called this and will run the host next, so
# starting it here would run two. novox/hq ADR 0005 moved that responsibility; this script chooses a
# version and says so, and nothing else.
say "pinned $VERSION. the launcher will start it."
+58 -51
View File
@@ -1,9 +1,15 @@
#!/bin/sh
# Tests for nox-mesh-host-rollback.
#
# It runs on a machine where the host will not start, which is the one moment nobody can afford
# it to be wrong — and the one moment it is hardest to debug. So it is tested here, against a
# real filesystem, with a stub package manager that records what it was asked to do.
# It runs on a machine where the host will not start, which is the one moment nobody can afford it to
# be wrong — and the one moment it is hardest to debug. So it is tested here, against a real
# filesystem holding real delivered versions.
#
# **These used to stub a package manager.** The script reinstalled the known-good version with
# `pacman -U` out of the package cache, which no machine in this mesh used and which two of the three
# operating systems the host is built for do not have (novox/hq ADR 0141). Going back is now choosing
# a directory, so there is nothing to stub: the thing under test is the filesystem, and a fake would
# only assert that the fake behaves as expected (novox/hq ADR 0017).
set -eu
cd "$(dirname "$0")"
SCRIPT="$PWD/nox-mesh-host-rollback"
@@ -12,26 +18,15 @@ PASS=0; FAIL=0
setup() {
WORK="$(mktemp -d)"
export MESH_HOST_STATE_DIR="$WORK/state"
export MESH_HOST_PKG_CACHE="$WORK/cache"
export MESH_HOST_PACKAGE="nox-mesh-host"
mkdir -p "$MESH_HOST_STATE_DIR" "$MESH_HOST_PKG_CACHE" "$WORK/bin"
export MESH_HOST_LIBEXEC="$WORK/libexec"
mkdir -p "$MESH_HOST_STATE_DIR" "$MESH_HOST_LIBEXEC/versions"
}
# Stubs on PATH. Not mocks of the script's own logic — the boundary is real commands, and
# these record the calls so a test can assert what the script asked the machine to do.
cat > "$WORK/bin/pacman" <<'STUB'
#!/bin/sh
echo "$@" >> "$MESH_HOST_STATE_DIR/pacman.calls"
[ -n "${STUB_PACMAN_FAILS:-}" ] && exit 1
exit 0
STUB
cat > "$WORK/bin/systemctl" <<'STUB'
#!/bin/sh
echo "$@" >> "$MESH_HOST_STATE_DIR/systemctl.calls"
exit 0
STUB
chmod +x "$WORK/bin/pacman" "$WORK/bin/systemctl"
PATH="$WORK/bin:$PATH"; export PATH
unset STUB_PACMAN_FAILS || true
# deliver a version the way the mesh would: a directory named for it, with the binary inside.
deliver() {
mkdir -p "$MESH_HOST_LIBEXEC/versions/$1"
printf '#!/bin/sh\nexit 0\n' > "$MESH_HOST_LIBEXEC/versions/$1/nox-mesh-host"
chmod +x "$MESH_HOST_LIBEXEC/versions/$1/nox-mesh-host"
}
check() { # name, condition-description, actual, expected
@@ -41,32 +36,38 @@ check() { # name, condition-description, actual, expected
# --- a normal rollback ---------------------------------------------------------------------
setup
echo "1.4.2" > "$MESH_HOST_STATE_DIR/known-good"
touch "$MESH_HOST_PKG_CACHE/nox-mesh-host-1.4.2-1-x86_64.pkg.tar.zst"
"$SCRIPT" >/dev/null 2>&1
check "installs the known-good version" "pacman is asked to install the cached package" \
"$(grep -c 'nox-mesh-host-1.4.2' "$MESH_HOST_STATE_DIR/pacman.calls" 2>/dev/null || echo 0)" "1"
# It installs and stops. The launcher execs the host next, and starting it here would run two
# (novox/hq ADR 0005).
check "does not start anything itself" "the launcher owns starting" \
"$([ -f "$MESH_HOST_STATE_DIR/systemctl.calls" ] && echo started || echo not-started)" "not-started"
deliver 1.4.2
deliver 1.5.0
echo 1.4.2 > "$MESH_HOST_STATE_DIR/known-good"
RC=0; "$SCRIPT" >/dev/null 2>&1 || RC=$?
check "pins the known-good version" "the launcher reads the pin and runs that version instead of the newest" \
"$(cat "$MESH_HOST_STATE_DIR/rollback-pinned" 2>/dev/null || echo MISSING)" "1.4.2"
check "records that it rolled back" "the attempted marker holds the version" \
"$(cat "$MESH_HOST_STATE_DIR/rollback-attempted" 2>/dev/null || echo MISSING)" "1.4.2"
check "succeeds" "a rollback that found its version is not a failure" "$RC" "0"
# It chooses and stops. The launcher runs the host next, and starting it here would run two
# (novox/hq ADR 0005).
check "does not start anything itself" "the launcher owns starting" \
"$(ls "$MESH_HOST_STATE_DIR" | grep -c started || true)" "0"
# The version it rolled back FROM is left alone: it is the newest, and retiring it is the running
# host's job after a reconcile it completes, never a recovery's.
check "leaves the failing version on disk" "a recovery deletes nothing" \
"$([ -x "$MESH_HOST_LIBEXEC/versions/1.5.0/nox-mesh-host" ] && echo present || echo gone)" "present"
# --- it rolls back only once ---------------------------------------------------------------
setup
echo "1.4.2" > "$MESH_HOST_STATE_DIR/known-good"
echo "1.4.2" > "$MESH_HOST_STATE_DIR/rollback-attempted"
touch "$MESH_HOST_PKG_CACHE/nox-mesh-host-1.4.2-1-x86_64.pkg.tar.zst"
"$SCRIPT" >/dev/null 2>&1
deliver 1.4.2
echo 1.4.2 > "$MESH_HOST_STATE_DIR/known-good"
echo 1.4.2 > "$MESH_HOST_STATE_DIR/rollback-attempted"
"$SCRIPT" >/dev/null 2>&1 || true
check "does not roll back twice" "a second failure is the machine, not the binary" \
"$([ -f "$MESH_HOST_STATE_DIR/pacman.calls" ] && echo called || echo not-called)" "not-called"
"$([ -e "$MESH_HOST_STATE_DIR/rollback-pinned" ] && echo pinned || echo untouched)" "untouched"
# --- nothing to roll back to ---------------------------------------------------------------
setup
set +e; "$SCRIPT" >/dev/null 2>&1; RC=$?; set -e
RC=0; "$SCRIPT" >/dev/null 2>&1 || RC=$?
check "no known-good: does nothing" "a host that never reconciled has no version to return to" \
"$([ -f "$MESH_HOST_STATE_DIR/pacman.calls" ] && echo called || echo not-called)" "not-called"
"$([ -e "$MESH_HOST_STATE_DIR/rollback-attempted" ] && echo attempted || echo untouched)" "untouched"
# The exit code is asserted from a real run, not from a literal. An earlier version of this
# compared "0" to "0" and could not fail — which hid an injected fault that made the script die
# here instead of returning cleanly.
@@ -74,23 +75,29 @@ check "no known-good: exits zero" "an installation failure is not a rollback fai
setup
printf ' \n' > "$MESH_HOST_STATE_DIR/known-good"
"$SCRIPT" >/dev/null 2>&1
check "blank known-good: refuses to guess" "installing nothing and reporting success is the fault this prevents" \
"$([ -f "$MESH_HOST_STATE_DIR/pacman.calls" ] && echo called || echo not-called)" "not-called"
"$SCRIPT" >/dev/null 2>&1 || true
check "blank known-good: refuses to guess" "pinning nothing and reporting success is the fault this prevents" \
"$([ -e "$MESH_HOST_STATE_DIR/rollback-pinned" ] && echo pinned || echo untouched)" "untouched"
# --- the cache was cleaned ------------------------------------------------------------------
# --- the known-good version is not delivered -------------------------------------------------
# It was retired, or that host was placed on the machine by hand and never delivered — which is how
# every first host arrives. Pinning it anyway would have the launcher ignore the pin and start the
# newest again, which is the binary that is failing.
setup
echo "1.4.2" > "$MESH_HOST_STATE_DIR/known-good"
set +e; "$SCRIPT" >/dev/null 2>&1; RC=$?; set -e
check "missing package: fails loudly" "cannot roll back, and says so rather than reporting success" "$RC" "1"
deliver 1.5.0
echo 1.4.2 > "$MESH_HOST_STATE_DIR/known-good"
RC=0; "$SCRIPT" >/dev/null 2>&1 || RC=$?
check "version not delivered: fails loudly" "cannot roll back, and says so rather than reporting success" "$RC" "1"
check "version not delivered: pins nothing" "a pin the launcher would ignore is worse than none" \
"$([ -e "$MESH_HOST_STATE_DIR/rollback-pinned" ] && echo pinned || echo untouched)" "untouched"
# --- the package manager refuses -------------------------------------------------------------
# --- a version directory with no binary in it ------------------------------------------------
# An interrupted delivery leaves one. Pinning it would start nothing.
setup
echo "1.4.2" > "$MESH_HOST_STATE_DIR/known-good"
touch "$MESH_HOST_PKG_CACHE/nox-mesh-host-1.4.2-1-x86_64.pkg.tar.zst"
STUB_PACMAN_FAILS=1 ; export STUB_PACMAN_FAILS
set +e; "$SCRIPT" >/dev/null 2>&1; RC=$?; set -e
check "pacman fails: exits non-zero" "a failed rollback is a failure the launcher must see" "$RC" "1"
mkdir -p "$MESH_HOST_LIBEXEC/versions/1.4.2"
echo 1.4.2 > "$MESH_HOST_STATE_DIR/known-good"
RC=0; "$SCRIPT" >/dev/null 2>&1 || RC=$?
check "half-delivered version: fails loudly" "a directory is not a version; the binary is" "$RC" "1"
printf '\nrollback: %d passed, %d failed\n' "$PASS" "$FAIL"
[ "$FAIL" -eq 0 ]