Author SHA1 Message Date
jochen a95c2b6ea9 A unit may be user-scoped: applied through the account's own manager (hq ADR 0177)
A workstation's per-user daemons — a window manager's reload watcher, an
audio mask, a memory guard — are units in the operator account's own service
manager, and until now had no form the mesh could send (to-be 29). The
`service` shape gains `scope` ("system", the default, or "user") and `user`
(the account, named ${machine:account} by a module); a user-scoped unit
without an account, or a system unit naming one, is refused at parse.

The host reaches the account's manager as `systemctl --user --machine=<account>@`
from its own process: no environment to forge, no user to switch to. Done on
the runner rather than per system, since every system's reading of a unit
already goes through systemctl. Apply, reflect-only and removal all go through
the same manager, and the applied record carries scope and user so removal
gives the unit back to the manager it came from. It answers only while that
manager runs — a login, or lingering enabled for the account; declaring
lingering is a follow-up.

Tests: a user-scoped unit is started and enabled in the account's manager and
recorded with its scope; a system unit never sees --user; the validation of
scope and user.
2026-10-02 16:48:14 +02:00
mesh-admin 5b73e04192 Merge pull request 'A package may be declared absent, and an uninstalled front end is retired for good (hq ADR 0175)' (#71) from feat/the-found-front-end-is-uninstalled into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-02 14:29:17 +00:00
jschoubben f57386cdea A package may be declared absent, and an uninstalled front end is retired for good (hq ADR 0175)
absent: true on a package has the host remove it through the machine's own
package manager when it is installed and leave alone a machine that never had
it; read back either way. Undeclaring a package still removes nothing. A
found firewall whose command is gone is recorded as removed, said once, and
asked nothing of.
2026-10-02 16:28:27 +02:00
mesh-admin f92dd3e286 Merge pull request 'Cite hq ADR 0170, not 0169: the firewall seat's record was renumbered' (#70) from fix/adr-0170-cited into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-02 12:53:09 +00:00
jschoubben cdbe3ab0a4 Cite hq ADR 0170, not 0169: the firewall seat's record was renumbered after a collision on hq main 2026-10-02 14:52:20 +02:00
mesh-admin a8f1cdb445 Merge pull request 'A machine reports whether its virtualisation daemon runs (hq ADR 0172)' (#69) from jschoubben/the-lab-is-a-module into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-02 12:47:55 +00:00
jschoubben ecb3003ba4 A machine reports whether its virtualisation daemon runs
The lab module needs the virtualisation daemon (novox/hq ADR 0172); the
capability is detected by asking the daemon about itself, not by finding
a client on disk.
2026-10-02 14:46:18 +02:00
mesh-admin d3861f82d4 Merge pull request 'A container may declare the capabilities it is granted (hq ADR 0169)' (#68) from feat/the-firewall-seat-serves-its-verbs into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-02 11:28:34 +00:00
jschoubben b6dbe0a7b9 A container may declare the capabilities it is granted (hq ADR 0169)
Exactly the names declared reach the runtime, named in the spec so a change
recreates the container; a name that is not a capability's is refused and a
privileged container stays undeclarable. For a seat holder whose runtime
changes the machine's packet filter.
2026-10-02 13:27:34 +02:00
mesh-admin 07bdad9e94 Merge pull request 'The host says what filters the machine, with owners, and keeps the found firewall retired on every converged apply (hq ADR 0168)' (#67) from feat/one-thing-filters-a-converged-machine into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-02 09:58:58 +00:00
jschoubben 627ac97d4f The host says what filters the machine, with owners, and keeps the found firewall retired on every converged apply (hq ADR 0168)
Every table and chain that refuses traffic is reported with whose it is:
the mesh's, the found firewall's, the container runtime's own, a ban, or
other — the runtime's user chain is other, which is where both predecessors
kept their rules, in the legacy filter on one machine and invisible to the
mesh. Adoption's threshold does not move; a converged machine's report
grows by its filters and its found firewall's state.

Convergence is a state the host keeps: a found firewall enabled again is
retired again and said; a reconcile that finds it inactive records that it
was found so, never that the mesh did it; a step skipped after a failed
apply is said. A retirement the mesh began and did not finish is finished.

Fixtures are rulesets captured from three machines of the first mesh.
2026-10-02 11:58:16 +02:00
mesh-admin ee2359f29f Merge pull request 'Every physical link faces outside, up or down (hq issue 197)' (#66) from jschoubben/every-physical-link-faces-outside into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-02 09:54:06 +00:00
jschoubben cbdbf6b7d3 Every physical link faces outside, up or down
The filter accepts what does not arrive on a link the machine names as
outward, and the host named only links carrying a default route. An
unplugged wired port was left unfiltered for whenever it was plugged in
(novox/hq issue 197). A link backed by a physical device is now named
whether or not it is up.
2026-10-02 11:53:53 +02:00
mesh-admin e12ca3f4dd Merge pull request 'A former target of a kind the host cannot remove is left in place and said, never fatal (hq issue 194)' (#65) from fix/a-former-target-without-a-removal-is-left-and-said into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-02 07:36:57 +00:00
jschoubben c47aa5d9b3 A former target of a kind the host cannot remove is left in place and said, never fatal (hq issue 194)
The host delivers its own successor as an archive whose target is a new
directory each version, and since mesh-host 63 the record keeps a resource's
former target for the next apply to remove. An archive has no removal (issue
162), so the first host that replaced itself under that rule refused its own
former version at the first step of every apply, and all four machines applied
nothing from then on. A former target nobody dropped is forgotten and said;
an archive the declaration dropped still refuses.
2026-10-02 02:43:47 +02:00
mesh-admin 3b9692b3cb Merge pull request 'A taken container keeps a found network, a left-out module is kept, and genesis raises the forge as its module declares (hq ADR 0163)' (#64) from feat/a-take-is-a-comparison-the-rest into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-02 00:28:17 +00:00
jschoubben fb9c9c3ee8 A taken container keeps a found network, a left-out module is kept, and genesis raises the forge as its module declares (hq ADR 0163)
A container may name networks it also joins once created, for the per-machine
setting that keeps a found network while a neighbour still resolves it there:
joined after the run, part of the spec, refused when it cannot be joined.

A declaration may say which modules the mesh left out because a stored setting
cannot compose with its definition. Absence used to read as removal; a left-out
module's records are kept and said, and its holds are not released.

Genesis raises the bootstrap forge under the gitea module's container name, with
its image digest and its data directory mounted at /data, so the module holds it
by the found rule instead of raising a second forge beside it (issue 090). The
network is the one difference left for a take to say. Before this the forge had
no volume: its repositories were the container's, lost with it.
2026-10-01 23:45:05 +02:00
mesh-admin d53e626366 Merge pull request 'A take is a comparison: the host's facts, former targets, and strays (hq ADR 0163)' (#63) from feat/a-take-is-a-comparison-the-hosts-facts into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-01 19:25:56 +00:00
jschoubben 83b3d20e68 A take is a comparison: the host's facts, former targets, and strays (hq ADR 0163)
Every held thing carries what a take compares: for a found container its image and the image's date,
the networks it is on and the other containers on each, its mounts and published ports, beside the
declared image (and its date once pulled), ports and volumes, with the downgrade decided when both
dates are known; for a found file whether the declared content differs and how, as lines lost and
lines new. A resource whose target moved keeps the former target on record as an orphan, so the next
apply removes the container or file the host wrote under the old name (issue 097). Every apply reports
the strays: containers the mesh neither wrote nor holds.
2026-10-01 21:24:37 +02:00
mesh-admin e030aa2387 Merge pull request 'The first user list lets the controller publish assignments and hear its seat's tools (hq #251)' (#62) from fix/first-user-list-matches-the-controller into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-01 15:29:55 +00:00
jschoubben c670bef4e1 The first user list lets the controller publish assignments and hear its seat's tools
The controller's own composition (mesh-controller internal/broker, 2026-10-01)
publishes memberships into the assignments stream after each push and
subscribes to its seat's tool subjects; the list the installer carries did
not say so, and a controller on it is refused on the first thing it tries:
"Permissions Violation for Publish to mesh.assignment.novox.builder" (hq #251),
which is why every build asked through the console was lost. The controller's
test that compares the two (TestTheInstallersFirstUserListIsWhatTheControllerWouldCompose,
run with this checkout beside it) named exactly these two subjects, and passes.
2026-10-01 17:29:48 +02:00
mesh-admin 45529bfec2 Merge pull request 'The profile names the network manager that is running, and travels in every report (hq ADR 0161)' (#61) from feat/the-profile-names-the-uplink-and-travels-in-the-report into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-01 14:02:16 +00:00
jschoubben b1e9ccff6d The profile names the network manager that is running, and travels in every report (hq ADR 0161)
One capability per manager — uplink-networkmanager, uplink-systemd-networkd, uplink-dhcpcd — from
systemctl is-active, so the uplink seat's holder for a manager this machine does not run is refused
the way any missing capability is, naming it (issue 138). The apply that reports detects the profile
again and sends it, the same shape enrolment sends, so a machine that switched managers reaches the
mesh at its next push.
2026-10-01 15:58:31 +02:00
mesh-admin cbcf0bcc93 Merge pull request 'A node can join the bus the mesh runs on' (#51) from fix/a-node-can-join-the-bus-the-mesh-runs-on into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-10-01 11:18:02 +00:00
jschoubben 6910f07d75 Merge pull request 'Say what a container's host entries now are' (#60) from feat/148-names-are-resolved-not-copied into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-09-30 12:38:15 +00:00
jschoubben 88037b33b7 Say what a container's host entries now are
novox/hq ADR 0148: the mesh's names are no longer among them, only what
the module declared. Comment only; the digest is unchanged.
2026-09-30 14:38:11 +02:00
jschoubben 422ad516d5 Merge pull request 'A declaration carries its order, and a host refuses an older one' (#59) from feat/107-a-declaration-carries-its-order into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-09-30 12:03:10 +00:00
jschoubben 8431ecfb48 A declaration carries its order, and a host refuses an older one
novox/hq 04-ISSUES/107. A declaration's only identity was the digest of
its bytes: a host could say "not the last" and could not say "older". On
the link, nothing refused an older one at all, and the drain picked the
last to arrive — wrong exactly when it mattered, a backlog drained out of
order or a broker that split a burst.

A declaration may now carry a sequence, one higher per send. A host
refuses one lower than what it kept, whole, and says why. The drain keeps
the highest sequence in a batch rather than the last to arrive.

Only when both sides claim an order. Absent reads as zero — "no ordering
claimed", not "first" — so a controller that sends none is still
understood and a host that kept one before it understood them compares
nothing. That is what lets hosts go first and the controller follow, which
is the order 087 says a new field needs.
2026-09-30 14:03:03 +02:00
jschoubben f107d68b2d Merge pull request 'A delivered host knows it is the delivered one' (#58) from fix/163-a-delivered-host-knows-it-is-the-delivered-one into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-09-30 11:46:58 +00:00
jschoubben 1028193c8a A delivered host knows it is the delivered one
novox/hq 04-ISSUES/163. The host asks after every apply whether a newer
host is delivered than the one running, and asked with the link-time
version stamp — which every delivered host carries as "development
build", because the version comes from where the binary sits now (0142).
So a delivered host never matched the newest delivered version, stood
aside on every push for ever, and because standing aside cancels the
report, the mesh never heard from it again.

Measured on two machines: each push produced "host <v> is delivered;
standing aside" for the version already running, then "applied, and could
not tell the mesh: reporting: context canceled". A machine restarting its
host on every push and reporting nothing, reading as healthy.

Asked with the running version now. Half of 0142 was applied to the
report and the known-good record and not here; this is the other half.
2026-09-30 13:46:51 +02:00
jschoubben f576287b51 Merge pull request 'A delivered host reads its version from where it sits' (#57) from fix/161-a-delivered-host-reads-its-version-from-its-path into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-09-30 10:40:29 +00:00
jschoubben 6c6495f6d9 A delivered host reads its version from where it sits
novox/hq ADR 0142, which decided this and was not implemented: "It is
unpacked into a directory named for its version, so it can read its own
version from its path. The stamp goes, and with it the need for a build to
know what it will be called."

The mesh's toolchain stamps no version, on purpose, so a delivered host
called itself "development build" and the mesh could not tell which host
any machine ran — which is the whole of what 087 added. A delivered host
lives at <libexec>/versions/<version>/<binary>, and that directory is the
answer.

A host placed by hand keeps its stamp, which is the honest answer for one
the mesh did not deliver, and is every machine until a delivery reaches
it. A binary sitting anywhere else is not read as a version at all.

The decision is split from the reading so a test can ask about a path
without being that binary.
2026-09-30 12:40:00 +02:00
jschoubben e6d48cf537 Merge pull request 'The mesh delivers the launcher, which is the last link in self-update' (#56) from feat/142-the-mesh-delivers-the-launcher into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-09-30 10:19:07 +00:00
jschoubben b8a766f234 The mesh delivers the launcher, which is the last link in self-update
novox/hq ADR 0141 and 04-ISSUES/142. A version was being delivered to a
machine and nothing started it: the launcher on these machines predates
the versions mechanism and runs the fixed binary path, so the delivery was
correct and inert.

Delivered as a FILE resource, not as part of an archive, and the
difference is the whole reason this is safe. A file is written atomically —
temp file in the same directory, then rename — so the running launcher
keeps the inode it was started from and the next start picks up the new
one. An archive writes in place with truncate, which would cut the file a
running shell is reading halfway through.

The manifest therefore carries a second copy of the script, and a test
refuses any difference between it and packaging/nox-mesh-host-launch.
Proven by drifting one and watching it fail. Two copies of a script is a
bad thing to accept, and the alternative was writing over a running
supervisor.

Together the two resources complete the loop: the version lands, the
running host stands aside because it sees one delivered, and the launcher
that starts next is the one that looks in versions/ and picks the newest
by arrival.
2026-09-30 12:19:00 +02:00
jschoubben 9caea5bc32 Merge pull request 'The delivered binary is named as every machine runs it' (#55) from fix/142-the-delivered-binary-is-named-as-machines-run-it into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-09-30 09:41:52 +00:00
jschoubben df27cee7b7 The delivered binary is named as every machine runs it
nox-mesh-host, not mesh-host. The command directory is cmd/mesh-host and
the launcher looks inside a delivered version for nox-mesh-host — the name
this is installed at and the name in its unit. The first delivery landed
the package's name, reported success, and would have been invisible.
2026-09-30 11:41:45 +02:00
jschoubben d275e64ed3 Merge pull request 'The host declares its own successor, as an archive at a versioned path' (#54) from feat/142-the-host-delivers-its-successor into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-09-30 09:33:19 +00:00
jschoubben 1a628a4d22 The host declares its own successor, as an archive at a versioned path
novox/hq ADR 0141 and 04-ISSUES/142. The host half of the delivery has
been built and tested since 0141 and has never had a version to work on:
versions side by side, the newest runs, the running one stands aside
between reconciles, rollback picks a directory. This is the declaration
that gives it one.

One archive, unpacked to /usr/lib/nox-mesh-host/versions/${version}. The
version resolves to the artifact's digest, so an unchanged build lands at
the path it already had and re-composing a declaration moves nothing.

Not circular: the host applying this is a different version from the one
being written, and neither writes over the other — the kernel refuses to
truncate a running executable, which is the reason the path carries the
version rather than a link pointing at "current".

**The launcher is deliberately not delivered here.** The one on these
machines predates the versions mechanism and runs the fixed binary path,
so a delivered version is inert until it is replaced — and replacing it
from the mesh means writing over a running shell script, which sh reads
incrementally. That wants a designed swap rather than a file resource, and
it is the last piece rather than this one.
2026-09-30 11:33:12 +02:00
jschoubben b5196e974c Merge pull request 'The host is a module, so the mesh can build it' (#53) from feat/142-the-host-is-a-module into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-09-30 07:56:35 +00:00
jschoubben 5162c3b05f The host is a module, so the mesh can build it
novox/hq 04-ISSUES/142 and ADR 0142. Nothing delivered the host because
nothing could compile it, and nothing could compile it partly because the
host was not a thing the mesh builds at all — it had no manifest.

One bundle in Go, for arch, built from cmd/mesh-host. A system is
required for a compiled artifact because a binary is pinned at link time
so a host refuses to touch a machine it was not built for (ADR 0005), and
`arch` is what all four of this mesh's machines report themselves to be.
Another system is another artifact and another build, which is what ADR
0142 means by one per target.

No resources yet. What places a version into a directory named for it
needs an archive resource whose path carries the version, and nothing
interpolates one — the second half of 0141's insight, and the next piece.
2026-09-30 09:56:18 +02:00
jschoubben 98fe8edf35 Merge pull request 'An apply says what it held, not only what it applied' (#52) from fix/125-a-hold-is-a-line-in-the-report into main
mesh/delivery held for a person: merged without a passing check: only a person decides that it goes on
2026-09-30 06:47:45 +00:00
jschoubben cbf50185d0 An apply says what it held, not only what it applied
novox/hq 04-ISSUES/125. An adopted node keeps what it found until its
module is taken, which is correct and was recorded only in the node's own
state file. On the edge cut-over the mesh sent 346 resources, the journal
said it applied 330, and nothing anywhere said which sixteen or why —
reading it meant opening state.json by hand, and not reading it took every
public name on the machine down.

The line that reports the apply now carries it, grouped by module and
ordered by name, because the sentence an operator needs is "route-proxy is
assigned and not taken" and the module is the thing `take` acts on. An
apply that held nothing says nothing extra: a line that reports "0 held"
on every converged apply is a line that stops being read.
2026-09-30 08:46:19 +02:00
jschoubben 197258c88c A node can join the bus the mesh runs on
novox/hq 04-ISSUES/146, the layers behind the three already fixed.

A new membership says which bus it is for. Empty meant 'whatever the mesh runs
today' while two buses existed, and became a refusal the moment one did: an
enrolled node came up and reconnected for ever against its own record.

The enrolling client takes its inboxes in the space its user may listen in. A
JetStream publish waits for the stream's acknowledgement on an inbox the client
picks, and its default is one this user may not subscribe to — so the enrolment
failed with a permissions violation on a subject nobody had chosen.

And the enrolment publish carries a message id, so the client's own retry is
discarded by the stream rather than enrolling the machine twice. That one is
not finished: the duplicate survives it, and the issue says where the trail
stops.
2026-09-29 17:36:59 +02:00
jschoubben a3b810f1f0 Merge pull request 'A first node gets as far as its own bus: three faults on the way' (#50) from fix/one-foundation-on-the-bus-the-mesh-runs-on into main 2026-09-29 14:06:33 +00:00
jschoubben 971a6d6d03 A first node gets as far as its own bus: three faults on the way
novox/hq 04-ISSUES/146. Each was right while the mesh ran on the previous
broker, and nothing has raised a foundation since it changed.

The bus's certificate is made by the program that needs it rather than by
openssl inside the broker's image — the bus's image is Alpine with a shell and
no openssl, so the step exited 127 and no mesh could be raised. Self-signed as
before and on purpose; --user 0:0 because the volume is root's and the control
plane's image runs as nobody.

Enrolment no longer opens a raw TLS connection to check the pin: NATS speaks
its own protocol and upgrades afterwards, so the handshake met a plaintext
greeting. The client that presents the token carries the same pinned config
and verifies inside its own handshake, so the secret still leaves only after
the certificate is checked. The raw dial stays as what its tests prove, and is
no longer a path anything takes.

And the token says which bus it is for. Empty meant 'whatever the mesh runs
today' while two buses existed and became a refusal the moment one did.

It now stops at the bus's user list, which is the genesis half of 146.
2026-09-29 15:42:43 +02:00
mesh-admin 04a27caa43 Merge pull request 'A host running as a service says what its apply did' (#49) from fix/a-host-running-as-a-service-says-what-it-did into main 2026-09-29 07:16:01 +00:00
jschoubben 6e90c2692d A host running as a service says what its apply did
The serving path passed nil where the apply writes its detail. Nil is silence, so
everything the apply says — a file held, a container replaced, the found firewall
retired — was visible when a person ran the one-shot command and discarded in the
way the host actually runs, which is always.

Measured: after a machine was converged and its found firewall was not retired,
what the host decided was unrecoverable, because it had said it to nobody. That is
why issue 143 has candidates instead of a cause.

say already reaches stdout and the unit sends that to the journal, so this needed
no new mechanism — only for the argument to be passed. Both paths now reach the
apply through one named helper, so a reader asking where the apply's output goes
finds one answer.

The test asserts the log is never nil and cannot catch the fault it was written
for, which is wiring; that is proved by a deployed host whose journal carries the
detail.
2026-09-29 09:15:53 +02:00
mesh-admin ced54d489f Merge pull request 'A converged machine can speak unasked' (#48) from fix/a-converged-machine-can-speak-unasked into main 2026-09-28 23:13:13 +00:00
jschoubben 94a35a39eb A converged machine can speak unasked
A reconcile is otherwise silent, and the condition deciding when it speaks asked
only what an adopted node reports: what it holds, and the firewall it found. A
converged node has neither, so it could speak only in reply to a declaration —
and the mesh composes no declaration for a node that has not said which links
face outside (ADR 0140). A machine waiting for a push that was waiting for the
machine.

Measured, not predicted: after the control plane learnt to read the links, the
control node recorded its own and the three converged machines sat silent while
their filters were refused.

The condition is now a named predicate, because as an inline expression nothing
could test it — which is why the gap shipped.
2026-09-29 01:13:09 +02:00
mesh-admin ec06369101 Merge pull request 'A machine says which links face outside without being asked' (#47) from fix/a-machine-says-unasked-which-links-face-outside into main 2026-09-28 23:00:28 +00:00
jschoubben bb85af1821 A machine says which links face outside without being asked
The mesh composes no filter for a machine that has not said (ADR 0140), and a
converged machine only speaks unasked when its adoption fingerprint changes. The
links were not in that fingerprint, so a machine that had just learnt to say
could only speak when a declaration arrived — and a declaration cannot be
composed until it has spoken. A machine waiting for a push that is waiting for
the machine.

Found before it bit: every machine in this mesh is in exactly that state right
now, having just been given a host that reports the links to a control plane that
does not yet read them.
2026-09-29 01:00:26 +02:00
mesh-admin 0c3928ad19 Merge pull request 'The host delivers its own successor, and versions live side by side' (#46) from feat/the-host-delivers-its-own-successor into main 2026-09-28 22:29:57 +00:00
jschoubben fbf0fb7d63 The host delivers its own successor, and versions live side by side
The supervision was already right: a clean exit means the host stood aside, and
the launcher's next turn runs what is on disk. Two things made it dead code —
nothing told the running host a successor was waiting, and the rollback resolved
its known-good version through pacman, which no machine here uses and which two
of three operating systems do not have.

Keeping a version rather than a path was the clue. Versions now live in
directories named for them:

- the launcher picks the newest delivered one every time round the loop, or the
  one a rollback pinned, or the host placed by hand when nothing is delivered;
- the running host stands aside between reconciles, never inside one, by exiting
  cleanly — and returns nil so the launcher does not count it as a crash;
- a completed reconcile retires what is older than the predecessor, keeping the
  predecessor because that is what a rollback starts, and never the running one;
- rollback pins the predecessor instead of reinstalling a package: no package
  manager, no cache anyone may clean, same script on every operating system;
- the report says which host version produced it, so 'behind' is answerable.

Newest is when it arrived, never how the name sorts: '1.10' orders before '1.9',
and ordering by name would start an older host and call it an upgrade.

novox/hq ADR 0141. The delivery half — a module carrying the next host — follows;
until then nothing delivers a version and every machine takes the fallback, which
is what it does today.
2026-09-29 00:29:36 +02:00
mesh-admin b005d4ef17 Merge pull request 'A machine says which of its links face outside' (#45) from feat/filter-what-arrives-from-outside into main 2026-09-28 21:37:24 +00:00
jschoubben b0d11c2439 A machine says which of its links face outside
The filter blocks everything passing through the machine and then allows the
machine's own containers back by naming the address ranges they sit on — two
ranges fixed in the control plane and the rest typed after a flip had already
cut a workstation off. A range describes one machine and goes stale in silence.

Read the links carrying a default route instead, from /proc rather than by
asking a program, and report them on every apply. A machine with no route off
itself reports nothing, and the mesh composes no filter for it rather than
writing a rule around a link with no name.

novox/hq ADR 0140. The control plane does not read this yet.
2026-09-28 23:37:06 +02:00
mesh-admin 155689672c Merge pull request 'A container's mesh names are part of what it is' (#44) from fix/a-containers-mesh-names-are-part-of-what-it-is into main 2026-09-28 15:19:39 +00:00
jschoubben e82789a322 A container's mesh names are part of what it is
A container resolves every machine and public name through the entries it is given when it is created,
and nothing re-reads them. The host compared everything about a container except those, so one whose
image and files never changed was left alone holding an overlay address five days out of date — it
restarted 2286 times against a database it could no longer find, and the mesh reported the machine as
doing what it was told (novox/hq 04-ISSUES/135, the same fault as 045 in the field left out).

Sorted, so the digest does not move for a reordering nobody made. The first apply after this recreates
every container that carries mesh names, once.
2026-09-28 17:19:38 +02:00
mesh-admin 99562947f3 Merge pull request 'Genesis lets the control plane state its own facts' (#43) from fix/genesis-lets-the-control-plane-state-its-facts into main 2026-09-28 14:07:22 +00:00
jschoubben c171c64d3a Genesis lets the control plane state its own facts
A mesh raised from nothing must be able to say what it applied and what a machine refused (novox/hq
ADR 0134), and the first user list is what permits it. Kept in step with the controller's own
composition by the test that reads this file.
2026-09-28 16:07:20 +02:00
mesh-admin 0dc5515099 Merge pull request 'A resource's owner is read to the last dot, because a module's name may contain one' (#42) from fix/a-resources-owner-is-read-to-the-last-dot into main 2026-09-28 13:45:01 +00:00
jschoubben 58c0e715c7 A resource's owner is read to the last dot, because a module's name may contain one
novox.be is a module on this mesh. Reading a resource's owner to the first dot made its resources
belong to something called "novox", and a step's gate would then skip whatever else happened to start
that way — silently. A resource's own id never contains a dot, which is what makes the last one the
boundary; the mesh's derived step was changed to add a hyphen rather than a dot for the same reason
(mesh-controller #128).
2026-09-28 15:44:59 +02:00
mesh-admin c5b229f95d Merge pull request 'A step gates its module, not the machine' (#41) from fix/a-step-gates-its-module-not-the-machine into main 2026-09-28 13:38:21 +00:00
jschoubben a9c49724b5 A step gates its module, not the machine
A run-once container that does not complete stops the rest of that module's resources; everything
else on the machine is attempted, as every other shape already is (novox/hq ADR 0136). An action
still gates the machine — genesis is a row of them and they belong to no module. What was not
attempted is reported as skipped, because that and 'nothing to do' are different answers.

Without this, ADR 0135's derived preparation would let one module's unreachable database hold a
machine hostage — the fault 04-ISSUES/011 removed for everything else, and the reason the catalogue
migrates itself at start.
2026-09-28 15:38:19 +02:00
mesh-admin 296ec5ece6 Merge pull request 'Genesis lets the controller ask any module's tool' (#40) from fix/genesis-lets-the-controller-ask into main 2026-09-28 02:21:24 +00:00
jschoubben 45b9a507a1 Genesis lets the controller ask any module's tool
The controller is the way in for tool calls (novox/hq ADR 0095); the first user list must say so,
or a mesh raised from nothing refuses its own first ask. Kept in step with the controller's
composition by the test that reads this file.
2026-09-28 04:21:22 +02:00
mesh-admin 5c410aaf54 Merge pull request 'One bus: the AMQP transport is gone from the host' (#39) from feat/one-bus into main 2026-09-28 01:54:23 +00:00
jschoubben 2478b5f127 One bus: the AMQP transport is gone from the host
The mesh runs on the seat's bus alone (novox/hq ADR 0131, design 28 task 5.5). The host's old
dialling and enrolment paths are deleted with the switch that chose between them; a membership or
a token naming another bus is refused before anything is sent, rather than dialled on a transport
that no longer exists.
2026-09-28 03:54:22 +02:00
mesh-admin c82e933268 Merge pull request 'Genesis: the first user list lets the controller hear the forge's merges' (#38) from fix/genesis-follows-the-forge into main 2026-09-28 01:13:18 +00:00
jschoubben e212da6bd2 Genesis: the first user list lets the controller hear the forge's merges 2026-09-28 03:13:16 +02:00
mesh-admin 673912ccdc Merge pull request 'A host adopts a delivered membership at start, not only after a declaration' (#37) from feat/a-host-adopts-a-membership-at-start into main 2026-09-28 00:40:27 +00:00
jschoubben 68a193d6f0 A host adopts a delivered membership at start, not only after a declaration
The rescue path: an operator writes the membership file by hand on a machine no
bus can reach — rotated while it still held the old password — and restarts the
host. Same file, same check as the declaration path.
2026-09-28 02:21:56 +02:00
jschoubben 4fba884d47 Merge pull request 'Genesis: the first user list lets the controller hear its consumers, and the account has JetStream' (#36) from fix/genesis-accounts-hear-and-have-jetstream into main 2026-09-27 23:47:41 +00:00
jschoubben 15f0dabf32 Genesis: the first user list lets the controller hear its consumers, and the account has JetStream
Two things the controller's own composition now derives and the installer's
carried list did not: the delivery subjects of the controller's consumers, and
JetStream enabled on the account — without which the first bound consumer is
refused. Found live on a mesh moved rather than raised; a fresh genesis would
have met both at first start.
2026-09-28 01:46:14 +02:00
jschoubben 4ef41ad5a0 Merge pull request 'A host logs in as node.<name> and hears answers on its own inbox' (#35) from fix/a-host-uses-its-own-inbox into main 2026-09-27 23:18:30 +00:00
jschoubben d749de989c A host hears the server's answers on its own inbox
The mesh grants a machine exactly its own inbox prefix; the client used the
default one and was refused a subscription to it.
2026-09-28 01:16:23 +02:00
jschoubben 14e64844df A host logs in to the new bus as node.<name>
The mesh names a machine's bus user node.<name>; the host dialled with the bare
name and every machine was refused the first time it reached the server:
"authentication error - User". Same string as the composed user list, or nothing
connects.
2026-09-28 01:13:52 +02:00
jschoubben d82fc9a121 Merge pull request 'A machine moves to the bus its declaration tells it to' (#34) from feat/a-machine-moves-to-the-bus-it-is-told into main 2026-09-27 22:09:08 +00:00
jschoubben 9fd3762e93 A machine moves to the bus its declaration tells it to
Until now a membership — bus address, fingerprint, password — was written once,
at enrolment, and nothing ever rewrote it, so a machine already enrolled could not
be moved to another bus at all (novox/hq design 28, task 5.2). The mesh now
delivers a membership for the new bus as a sealed file in the declaration, like
any secret; the host reads it after the declaration has applied, saves it as its
identity, and exits cleanly so the service manager restarts it dialling the bus it
names. The same path a machine takes after a reboot — so nothing new has to be
right for it to work. A membership carries its transport; empty means the bus the
mesh ran on before, so nothing written earlier reads as unset.
2026-09-28 00:08:44 +02:00
jschoubben 587b8aa220 Merge pull request 'A host reaches either bus, and a genesis template that raises the mesh on the new one' (#33) from feat/nats-genesis into main 2026-09-27 17:36:48 +00:00
jschoubben 48e2ff2de5 Merge remote-tracking branch 'origin/main' into feat/nats-genesis 2026-09-27 18:38:45 +02:00
jschoubben 6a3435629e A foundation template that raises the mesh on the bus being built
The same twelve steps, with the difference that matters: the mesh composes its own user
list and at genesis there is none, so this carries the first one — the controller's
account at a bootstrap password, rotated with the store's and replaced by the
controller's own composition from its first start.

The server's settings and the user list are separate files in one directory. Separate
because the settings belong to whoever raises the server and the users belong to the
mesh; in one directory of necessity, because an include path resolves relative to the
including file's own directory, so an absolute one sends the server looking underneath
that directory and it refuses to start.

No `verify` in the TLS block. That makes the server demand a client certificate and
nothing in the mesh presents one — a host pins this server's exact certificate and
authenticates with a password.

The controller's permissions here are checked against what the controller derives, by a
test in its own repository reading this file. They are two statements of one fact, and a
template that granted less than the controller needs would produce a mesh that comes up
and is refused on its first act.
2026-09-27 16:39:32 +02:00
jschoubben 9072f60a30 Enrolment behind a seam, with both transports
The last of the host's link that still named a transport. `Asking` is one
enrolment conversation — a connection made with the token, a question asked, and
an answer waited for — and it is its own seam rather than part of `Link` because
almost nothing about it is the same: the credential is a one-time secret, there
is no declaration to hear, and a node that fails here is not in the mesh at all,
where a node that fails in `Link` has merely lost touch with one it belongs to.

`Enrol`'s thirteen arguments became an `Approach` — where, which certificate,
which bus — and the request it already had. The token says nothing about which
bus, and does not need to: every token names the one the mesh runs on today until
the rollout.

**The reply address is the whole of what changes on the new bus**, and it is
forced rather than preferred. Verified against a running server, both halves: the
answer reaches the node at the address its request carried in the payload, and
the transport's own reply field held something else entirely by the time the
consumer saw it — the consumer's ack address, exactly as design 25 §2 says. The
test asserts the field is *not* the node's inbox, so a future server that stopped
claiming it would fail this rather than let the reason quietly become folklore.

The inbox is under `_INBOX.enrol.<node>.`, which is exactly what the enrolling
user may subscribe and no wider, with a random tail per attempt: a reply left
over from an attempt that timed out is not the answer to this question, which is
what the correlation id does on the other transport. Subscribed before anything
is published, because a node that published first could miss an answer to a
question nobody was listening for.
2026-09-27 01:31:46 +02:00
jschoubben 6e208f7b3e The host's inbound behind a seam, with both transports
The outbound half went behind `Bus` and a node's two statements stopped naming a
transport. This is the other half, and where the transport reached furthest: the
run loop selected on a channel of the client library's own delivery type, so
every part of holding a node in its mesh knew which bus it was on.

`Link` is dialling, hearing and saying in one interface, because dialling is
where the transport is chosen and choosing it twice is how one half of a node
ends up on a different bus from the other. `Declaration` has one way of being
done rather than two: a declaration set aside for a newer one is settled exactly
as an applied one is, on both buses, and the difference is a fact the report
carries.

Four things this settled.

**The host declares nothing on the new bus.** On the bus the mesh has it declares
its own queue, because a queue that is not there means a node that hears
nothing. Here it binds to a consumer the mesh made when the node enrolled, and a
missing one is said as the mesh's to answer rather than quietly created with
whatever this client happens to default to.

**The pin is easier here than in the tool runtime, not harder.** The Go client
takes a *tls.Config, so the same PinnedConfig with the same VerifyPeerCertificate
does the work — the subject-alternative-name constraint recorded against the
runtime's client is that client's, because it takes PEM strings with no verify
hook. A host checks the fingerprint and nothing else.

**Binding needs the subject as well as the consumer.** An empty subject is
refused rather than taken to mean "whatever that consumer delivers", which the
server said plainly and only when asked.

**Reconnection stays the caller's.** Hold already decides when to try again and
how long to wait; a client reconnecting underneath it would make that reasoning
a duplicate of the library's.

The drain keeps its live half and loses its catch-up half, as it said it would:
verified that three declarations pushed to an absent node leave one on the
stream, and it is the newest.

One test-harness lesson worth the comment it got: delete-then-add is not a reset.
A test that did that inherited the previous test's messages, and the symptom was
a declaration counted as delivered twice — which reads as a redelivery bug in the
code under test rather than as a dirty stream.
2026-09-27 01:25:01 +02:00
jschoubben 54f6eb661a Merge pull request 'A taken tunnel's found configuration is retired once the take is proven (hq ADR 0119)' (#32) from feat/a-taken-tunnels-predecessor-is-retired into main 2026-09-26 22:59:51 +00:00
jschoubben 646be4fdf4 Merge pull request 'Undeclaring gives a unit back the state it was found in, and removes a process the mesh made (hq ADR 0118)' (#31) from feat/undeclaring-leaves-the-machines-units into main 2026-09-26 22:59:30 +00:00
jschoubben e688b3b816 Merge pull request 'A file written into a marked block, so a shared hosts file keeps every line that is not the mesh's (hq 128)' (#30) from feat/a-file-written-into-a-marked-block into main 2026-09-26 22:59:21 +00:00
jochen 50776b8613 review: a retired configuration that comes back is retired again on its first original, and only wg-quick's own file is ever removed (hq ADR 0119)
Put back by hand, it was found afresh and its copy became the hold's original, so a machine that
kept restoring it kept growing copies and lost which one was first. The first original now stays
the record's, content that differs is kept once beside it, and the note says a rollback means
unassigning the private network. Nothing is removed unless it is <wireguard dir>/<iface>.conf,
not a path the mesh writes, and not a link, which would leave the key-bearing target behind.
2026-09-27 00:55:50 +02:00
jochen b462f461c6 A taken tunnel's found configuration is retired once the take is proven (hq ADR 0119)
Kept on disk it was the take's fallback; once the mesh's interface is up in its place and a peer
has handshaken with it, it is an unmaintained way back onto the network, held for ever. It is now
removed from where its unit reads it, its kept original verified first and left as it is, and the
hold ends. Until proven — no handshake, or wg not answering — it is kept and the report says why.
The retirement is recorded apart from holds, so later applies, an undeclare, and a reassignment
find it retired rather than missing, and nothing writes it back.
2026-09-27 00:47:57 +02:00
jochen 23a4436499 review: a unit whose file the mesh wrote is stopped when undeclared, and what was found survives a failed first apply (hq ADR 0118)
Records written before Found existed left the adoption guard and the converge filter loaded on
undeclare, then deleted their unit files from under them; a unit whose own file the mesh created
is now the mesh's, whatever its record says. Found is kept apart the moment it is read, so a
first apply that enabled and then failed is not read back as the machine's; boot is found the
first time the mesh sets it; a service once stateless, or moved to another unit, is found afresh
(the old unit given back). The unit is read after the reload that loads a file written in the
same apply, and removal reports what it actually did.
2026-09-27 00:41:02 +02:00
jochen 08c0f40ff0 review: refuse a process named ".", ".." or with a leading dash, so removing one cannot delete the mesh's own directory (hq ADR 0118)
removeProcess deletes filepath.Join(daemonRoot, name) whole; a process named ".." made that
/var/lib/mesh. The declaration and the removal now hold the name to one rule.
2026-09-27 00:41:02 +02:00
jochen 3112c881e4 Undeclaring gives a unit back the state it was found in, and removes a process the mesh made (hq ADR 0118, issue 130)
A service undeclared used to be stopped: unassigning the private network stopped the container
runtime, unassigning sshd would stop ssh, an uplink module would take the machine offline. The
host now records the unit's state when it first applies it and restores that on undeclare —
found running stays running; started by the mesh (the converge filter) is stopped again; nothing
is started on the way out; a pre-existing record leaves the unit alone.

An undeclared process had no removal at all and failed every apply on its node; its unit, timer
and bundle are now removed.
2026-09-27 00:21:25 +02:00
jochen 06aaac0820 A service may omit its state, so unassigning an uplink module never stops the machine's network manager (hq ADR 0117) 2026-09-27 00:11:58 +02:00
jochen fdc768c476 review: rebuild a file the mesh once wrote whole, keep links, give back a missing line end, and release a hold only after the write (hq issue 128) 2026-09-27 00:09:34 +02:00
jschoubben 25449a31c3 The host's outbound behind a seam, with both transports
Step 3.5's first half, mirroring the controller's. A host says exactly two
things unprompted, and the difference between them is the whole interface:
a report must arrive, and a heartbeat must not be insisted on. So a report
goes through JetStream — it is the message the store-window guarantee is
about — and a heartbeat stays on core, because a heartbeat in a stream is
the mesh's least valuable message competing for retention with its most
valuable.

The host still imports nothing of the mesh's own (ADR 0005): this is its
own interface over its own libraries. It agrees with the controller because
a fixture holds both to one envelope, which is the only agreement that
survives two repositories.

Also recorded, where the next person reads it rather than in a plan: the
"newest wins" window narrows at the rollout and does not disappear. Last-
per-subject makes the catch-up half the stream's, and sequence orders them
definitively — but three pushes to a connected node are still three
deliveries. Saying which half goes is worth more than "can probably be
removed", which is how a load-bearing window gets deleted in a hurry.
2026-09-27 00:01:42 +02:00
jochen 1cb895346d Write into a marked block of a text file instead of over it, so a shared hosts file keeps every line that is not the mesh's (hq issue 128) 2026-09-26 23:51:36 +02:00
jochen 3ae999497f Format control_test.go so the gate's fmt step passes on main 2026-09-26 23:51:36 +02:00
jschoubben 32d7234637 Merge pull request 'A host accepts an explicitly-empty declaration (hq 127)' (#29) from feat/an-explicit-empty-declaration into main 2026-09-26 21:39:56 +00:00
jschoubben 2722e7b36e A host accepts an explicitly-empty declaration (hq 127)
The empty-resources guard refused every empty body as a likely mistake,
with no way to say emptiness was meant — so the control plane could
never tell a node to drop its last resource. The envelope gains
owns_nothing: with it, an empty declaration is applied (the node drops
what the mesh owned); without it, empty is still refused, so a
truncated or mis-composed body cannot silently strip a machine. One
test, both directions.
2026-09-26 23:39:32 +02:00
jschoubben 808e93a477 Merge pull request 'A found tunnel carries its MTU to the mesh' (#28) from feat/a-taken-tunnel-carries-its-mtu into main 2026-09-26 20:40:09 +00:00
jschoubben 8e2a1e762d A found tunnel carries its MTU to the mesh
The host parses MTU from the found [Interface] and reports it, so the
mesh's interface can come up with the same MTU when it takes the tunnel
over. A path tuned to 1380 regresses to the 1420 default otherwise —
invisible to ping, fatal to TLS handshakes and transfers over that path
(novox/hq: the mesh had no MTU concept). Zero when the config named
none, and the mesh writes no MTU line then.
2026-09-26 22:39:54 +02:00
jschoubben 5dbc8e5c3c Merge pull request 'the spec names the resolver and the address' (#27) from fix/the-spec-names-the-resolver-and-address into main 2026-09-25 21:51:42 +00:00
jschoubben 260bf0b752 the spec names the resolver and the address
dns and ip were declared, validated, handed to the runtime — and part of
no comparison, so their first deployment compared every container equal
and changed nothing, silently. The same shape as 04-ISSUES/045: a field
that is not in the spec is a field that can never reach a container that
already runs.
2026-09-25 23:51:30 +02:00
jschoubben 93caf26aed Merge pull request 'a container may name its resolvers and its own address' (#26) from feat/a-container-may-name-its-resolver-and-its-address into main 2026-09-25 21:48:44 +00:00
jschoubben 3ac765db65 a container may name its resolvers and its own address
Mailu's 2024.06 admin refuses to serve behind a resolver that does not
validate DNSSEC, and the runtime's own forwarder (127.0.0.11) validates
nothing — so a module shipping its own validating resolver had a
resolver nothing could be pointed at. Found live, blocking a cutover:
the admin sat unhealthy, submission answered 454, and the declaration
language had no words for the fix.

Two fields on a container, both handed to the runtime verbatim: dns —
the resolvers it asks — and ip, its static address on its user-defined
network, which exists for exactly one shape: a container others must
reach before name resolution works, the resolver itself being the case
that forced it. Both take only addresses and are refused on arrival
otherwise — a name here would reach the runtime verbatim and be refused
at create, after the old container was already gone.
2026-09-25 23:48:31 +02:00
jschoubben 7e245dae92 Merge pull request 'inspect by kind, not the ambiguous bare form — a same-named network stops a container from ever being found' (#25) from fix/inspect-by-kind-not-the-ambiguous-bare-form into main 2026-09-24 18:29:44 +00:00
jschoubben 5dd439df50 inspect by kind, not the ambiguous bare form — a same-named network stops a container from ever being found
docker inspect <name> resolves across every object kind, not just
containers. A module regularly names a network the same as the
container that joins it (keycloak does this today, ordinarily) — so
when the container does not exist yet but the same-named network
already does, the bare form answers with the network's JSON instead
of reporting the container absent, and the template these callers use
(.State.Running) fails to execute against it entirely.

Live on novox tonight: minio's LB container, named the same as its
network ("minio"), could never be created — every apply crashed on
"the container runtime could not say whether minio is here", stuck
since first push, because the check itself never got a clean answer.

Fixed at every call site asking a container's state by name
(containerState, inspectFound, NamesFree, raiseGiteaServer,
containerRunning) by scoping to `docker container inspect`, matching
the type-scoped form this codebase already uses correctly for
networks, volumes and images elsewhere. Also scoped the one image
inspect that was still bare (publish.go), for the same reason.

mesh-host runs as a host-level service (nox-mesh-host.service), not a
Docker module — merging this does not redeploy it. The live novox
failure persists until the service itself is rebuilt and updated.
2026-09-24 19:50:16 +02:00
jschoubben f68139c9d1 Merge pull request 'Take over the found tunnel: its key, its port, its peers; stop it, never flush (hq ADR 0105)' (#24) from feat/adopt-the-tunnel into main 2026-09-23 22:38:36 +00:00
jschoubben fc593b9dfd Stop nothing the mesh cannot replace, give the tunnel back on failure, and take it over after enrolment
Review of the ADR 0105 build (hq ADR 0105). The takeover stopped the found
unit and then found out whether the mesh's interface would do; a start that
failed left the machine with no tunnel at all.

Now nothing is stopped until the declared interface listens on the found port
at the found address and the key file it names holds the found key — the
refusal names the remedy — and a mesh interface that fails to start after the
takeover has the found unit started again, with the account saying so. The
account has three states (not taken, taken, down) and is given on every
takeover, failure included. An interface raised by hand is looked at again
for a moment and then refused naming `wg-quick down`. A found unit started
again by hand beside the mesh's is said, not stopped: on the hub it cannot
hold the port, and on a spoke two interfaces with one key would fight.

`mesh-host overlay take --tunnel <iface>` is the path for a node that
enrolled before the mesh knew to take a tunnel over: the found key becomes its
overlay key — identity, sealing and serving keys untouched, so nothing sealed
to the node is remade — and the mesh is told with a rekey signed by the
identity key, over the key left, the key taken and the tunnel. Told first,
written second, so a run again puts right whichever half did not happen.
2026-09-24 00:02:08 +02:00
jschoubben 83306b2dba Merge pull request 'Recreate a container when the content of a file it reads at creation changes (hq issue 103)' (#22) from fix/recreate-on-content-change into main 2026-09-23 21:44:21 +00:00
jschoubben 4840e21405 Say in the plan which file a container would be recreated for
The plan says what an apply would change from the declaration and the
record, before the machine is touched. The apply now recreates a container
when the content of a file it reads at creation changed, and the plan said
"check" for every recorded container — true, but a preview that hides the
one step somebody asked about.

So a recorded container whose record of what it read differs from what this
apply will hand it — a plain file declared here, by its declared content;
otherwise what this host last wrote at that path — is planned as an update
naming the file, the same comparison applyContainer makes. What the record
cannot settle stays a check: a file neither declared nor recorded is read
from the machine by the apply, not by the plan; and a container with no
record of what it read was labelled before the host kept that record and is
accepted as it is.

novox/hq 04-ISSUES/103, 104
2026-09-23 23:42:41 +02:00
jschoubben 982b84310e Look at what a container mounts directly, accept a pre-upgrade label, and write the genesis secret without a newline
Review of the first cut found four things.

A directory mounted into a container is no longer looked inside, not even
for the files this host wrote there. The controller records every
provider's received and contributions file as a plain file under a mounted
directory, so folding those in would have recreated the route proxy — which
re-reads its routes live, by design — on every route change, and killed
every provisioner sidecar, which polls what it receives, mid-reconcile on
every grant. Whether a service reads a file under its directory once or
watches it is the service's; restart-on is how a module says "once", and it
stays the opt-in. Env-files and files mounted directly remain by content.

Genesis wrote the superuser secret as `value\n`; `secret accept` strips the
line ending by design, so the postgres module declared `value` — and with
a mounted file's content in the spec, phase three would have recreated the
store it meant to adopt in place, with the temporary control plane
connected to it. Genesis now writes the value alone. readCredentialFile
tolerated both endings already. Pinned with the bytes the genesis code
path writes, then the module's declaration of the same container: it must
reconcile.

A container carrying a label from before the host folded in what it reads
is accepted rather than recreated, when that label matches the spec as it
used to be computed: what it reads is recorded then, a change is caught
from that record from the next apply on, and the label is renewed at the
next genuine recreate. Recreating them all would have been a restart storm
across the mesh in declaration order, the store first. The trade-off is
stated in the code: a container already stale at upgrade time is not
caught, and could not have been either way.

The record of what a container read is looked up by its name when its
declared id has none — the bundle's `store` becomes `postgres.server` for
the same container — so a change on the day it is adopted still names the
file. The by-target lookup takes the most recently applied record, since
the bundle's record for the same target is never removed by the mesh's.

novox/hq 04-ISSUES/103
2026-09-23 23:40:27 +02:00
jschoubben c60228e719 Recreate a container when the content of a file it reads at creation changes
The host decided whether a container was still the one declared by a digest
of its declaration, and the declaration names an env-file's path and a
mount's path — never what is in them. So when the store was given a new
port, the host rewrote the forge's and the analytics service's environment
files, correctly, and left both containers running with the old port in
their environment: a container reads its env-file when it is CREATED, and
`docker restart` hands it the same environment again. Both looked healthy
until they answered 502.

What a running container takes in at creation is now part of its spec, by
content: every env-file, a file bind-mounted into it, and every file this
host wrote at or under a directory bind-mounted into it — the secrets,
bindings and configs under a module's state directories. The digest is the
one the store already records for a file the host wrote (`wrote`), read
from the state as it stands when the container is reached, so a file
rewritten earlier in the same apply is already the new one; a file the host
has no record of — an env-file a predecessor left, the superuser secret
genesis writes before any declaration names it — is read from disk, which
is what keeps adopting a running store in place a reconcile and not a
recreate.

Deliberately not part of it: what else is in a bind-mounted directory,
which is the service's own data and changes while it runs; a named volume;
a seed created once, which digests as the seed the host wrote and not as
what has grown in it; and a step — a run-once or scheduled container reads
its files when it runs and runs fresh each time. On an adopted node a held
container is held before any of this is looked at.

The host records what each container was created reading, per file, so
the recreate can say which file changed — "recreated: <file> changed" in
the report and, now with its detail, in the log. A container made before
this record existed is recreated once and says so.

novox/hq 04-ISSUES/103
2026-09-23 23:40:27 +02:00
jschoubben 977df39e0d Merge pull request 'Refuse a declaration for the other mode, or older than the mesh's last, and preview before applying (hq issue 104)' (#23) from fix/reconcile-refuses-stale into main 2026-09-23 21:38:33 +00:00
jschoubben f08a8ea3f7 Refuse every file once the mesh has spoken, plan the cutover as one, and let the kept declaration repair the mode
Review of the fix for hq issue 104 found three faults in it. A file applied
on an enrolled node — the mesh's own last declaration included — is applied
as the bundle is, so its resources are recorded as the machine's own and
what the mesh declared reads as undeclared: the plan removed the foundation.
`apply FILE` is for a machine the mesh has not spoken to, and is now refused
saying so whenever declared.json exists. The plan looked at what is held
before what the declaration says is taken, so the one cutover ADR 0100 says
must be previewed read as a hold; it now decides in holdOnAdopted's order,
models a step run inside a held container, and a test holds the plan's
sequence to the apply's outcomes. Genesis wrote the mode on every run, so a
re-run after `converge` left the state saying adopted while the kept,
signed declaration said converged, and the reconcile loop refused every five
minutes with no delivery coming to end it: genesis now writes the mode only
when none is recorded, and where the state and the verified kept declaration
disagree, the kept declaration wins and the repair is said.

Also: a file lock beside the state, taken by the link service, the host's
own commands and the installer alike, so a `reconcile` run by hand no
longer races the loop's save — chosen over refusing while a named service is
active, which would miss a `mesh-host run` started by hand; `--json
--dry-run` emits {plan} like an apply emits {plan, report}; the README's
duplicate flag line; and the bundle refusal is about the digest, not a claim
the carried bytes can never match what genesis applied.
2026-09-23 23:35:49 +02:00
jschoubben 7283924a35 Take over the found tunnel: its key, its port, its peers; stop it, never flush
On an adopted machine the private network takes the predecessor's tunnel
over in place (hq ADR 0105). Genesis finds the one interface up besides the
mesh's own, settles the hub's port and the mesh's range on it, and skips
ADR 0100's non-overlap check for a range that is now the tunnel's; a
--hub-port or --overlay-range that disagrees is refused naming the tunnel's.

At enrolment the found interface's private key becomes this node's overlay
key — the one credential the mesh takes rather than mints — stored where a
generated one is stored, never printed and never sent; the tunnel (port,
address, range, peers) travels with the keys so the mesh composes from it
before the first declaration.

The interface's service may say what it takes over. Before the mesh's unit
starts, the found configuration is kept like any held file and the found
unit is stopped and disabled; nothing is flushed, and an interface still up
after its unit stopped refuses the takeover rather than half-working. The
report says what was carried: interface, port, range, peer count, taken or
not, and where the original was kept.
2026-09-23 23:26:35 +02:00
jschoubben 27c4b765b2 Refuse a declaration for the other mode, or older than the mesh's last, and say what an apply would change first
An operator ran `mesh-host reconcile` on an adopted control-node with twelve
modules assigned. It applied the bundle the host carries — the genesis
declaration, foundation only, converged: recreated the store, failed on the
broker's held port, wrote the converged base filter and started its service,
and stopped at the first failing action. The filter closed the machine for
forty-five minutes. The host reported the node adopted in every report, the
declaration said converged, and nothing compared the two; nothing was printed
before acting (hq issue 104).

The host now records the node's mode — from every declaration the mesh sends,
and at genesis from what the operator said — and refuses, at the point of
application, a declaration that says the other mode, naming both and the act
that changes it. Only a declaration the link delivers, signed, changes the
mode: that is how `converge` and `adopt` arrive, so the flip still works and
nothing else can do it. Genesis marks the bundle consumed, with the digest of
what it applied, so `reconcile` holds a node the mesh has spoken to against
what the mesh last said and never the bundle, and refuses the carried bytes
when they are not what genesis applied. A file is refused when it is not what
the mesh last said: a declaration carries no sequence and no issued-at, so the
host cannot tell older from newer, and says so. Both commands print what they
would change — a hold, a removal, an action named as one — before touching
anything, and --dry-run is that list and nothing more.
2026-09-23 23:15:28 +02:00
jschoubben 9176aea6c4 Merge pull request 'The packages port given at genesis is a node setting, not manifest text (hq issue 085)' (#21) from feat/packages-port into main 2026-09-22 21:59:46 +02:00
jschoubben 744beb6c51 A re-run does not replace the registered forge with whatever checkout it was given
Registering a module is an overwrite. Recording the forge's port ran `module add`
every time, so a genesis re-run pointed at an older catalogue would replace the
manifest of a forge that is built and assigned — with a push a few lines later.
Registering is only here so a settings row has a module row to hang on, and that
row is already there on a mesh that knows the forge. So: ask first, and skip.

Two comments narrowed to what is true. What follows the node's setting is what
the mesh derives from a module's ports — its container mapping, its filter rule,
its opening and what it serves. The forge's own address in its runtime's
environment (hq 088) and its route contribution's port do not, and are already
wrong for any port the mesh assigned. And a settings layer is the module's, not
one resource's: a second mergeable file on the builder would be given `serves`
too.

novox/hq 04-ISSUES/085
2026-09-22 21:56:42 +02:00
jschoubben c4ce57997e The packages port given at genesis is a module's setting, like every other
Every foundation port given at genesis became a per-node setting of the module
that binds it, except the package registry's: that one was fixed by rewriting
the builder's manifest when the installer registered it. Registering the builder
again from the catalogue undid it, and the forge's own module, when it took the
bootstrap forge over, came up on the catalogue's port — which on a machine where
a predecessor holds 3000 points the builder at the predecessor's forge.

So the rewrite is gone, and the port is recorded twice as a setting, both from
the one input:

- the forge's module is registered at genesis — not assigned, nothing of it runs
  — so the controller has something to hold `{"ports": {"3000": <given>}}`
  against. Assigning the forge later raises it on the port this machine was
  given, and its container, its filter rule, its opening, what it serves and
  what consumers are told all read it from there.
- the builder is given `{"serves": {"port": <given>}}`, which merges into the
  binding it carries in place of one nothing can resolve yet.

A genesis on the catalogue's port records nothing and registers nothing, so it
does exactly what it did before.

novox/hq 04-ISSUES/085, ADR 0100
2026-09-22 21:40:02 +02:00
jschoubben 0db1fbdb3b Merge pull request 'Adoption mode: a node in use is adopted before it is converged (hq ADR 0100–0103)' (#20) from feat/adoption-mode into main 2026-09-22 21:01:46 +02:00
jschoubben 6dce63b534 Say plainly where the raised package registry runs, and why an action outside a container still runs 2026-09-22 20:01:26 +02:00
jschoubben c03a31cec5 Count only a record of making something at a path as the mesh's, not an access record (hq ADR 0103) 2026-09-22 20:01:14 +02:00
jschoubben 01e8affa89 Count an unasked report as said only once the broker has taken it (hq ADR 0100) 2026-09-22 20:00:59 +02:00
jschoubben b4c21b4f67 Read a machine's iptables rules when it has no nft, instead of calling it unfiltered (hq ADR 0100) 2026-09-22 19:59:22 +02:00
jschoubben c7ff0b9026 Refuse an endpoint whose port is not the one the private network's hub binds (hq ADR 0100) 2026-09-22 19:58:37 +02:00
jschoubben 04665d36c8 Exempt only the daemons a fresh machine was measured to run from counting as in use (hq ADR 0101) 2026-09-22 19:57:52 +02:00
jschoubben 7beb752be0 Take a key or list member the host may have written itself as the mesh's, so undeclaring gives the file back (hq ADR 0102) 2026-09-22 19:57:15 +02:00
jschoubben 0bd22e50f2 Apply one declaration at a time, so the link and the reconcile do not lose each other's record 2026-09-22 19:56:05 +02:00
jschoubben 9839006d48 Retire the found firewall only once the mesh's own filter is loaded on the machine (hq ADR 0100) 2026-09-22 19:54:43 +02:00
jschoubben 5f126021b7 Keep the originals the carried bundle writes over beside the node's state (hq ADR 0100) 2026-09-22 19:53:50 +02:00
jschoubben eb2f4fcf53 Keep an original by its path and its content, so a second original at one path is not discarded (hq ADR 0100) 2026-09-22 19:53:15 +02:00
jschoubben 232ed74940 gofmt the ufw rule's direction field 2026-09-22 19:52:40 +02:00
jschoubben 40e8ea9fda Hold a unit an administrator installed whatever its state, and a packaged unit only when the machine uses it (hq ADR 0103) 2026-09-22 19:52:37 +02:00
jschoubben d3f2595968 Read a ufw rule's direction: an outgoing rule answers no opening, and incoming is the default ufw merges on (hq ADR 0103) 2026-09-22 19:51:35 +02:00
jschoubben dc861fb4a8 Do not arm a scheduled step of a module held as found on an adopted node (hq ADR 0103) 2026-09-22 19:48:05 +02:00
jschoubben da008460ac Fail a resource when the container runtime cannot answer, instead of reading silence as nothing there (hq ADR 0100) 2026-09-22 19:46:56 +02:00
jschoubben 0ea646b384 Keep the derived filter in force when a guard resource failed on the way back to adopted (hq ADR 0103) 2026-09-22 19:45:33 +02:00
jschoubben 60bb3d895c Count a unit as found only when the machine runs it or starts it at boot, so a packaged unit nothing ran is not held (hq ADR 0103, found by the adoption bed) 2026-09-22 18:47:48 +02:00
jschoubben bde5e3461c Keep the original of any file the host writes over without a record of it, on every node, and name where (hq ADR 0100) 2026-09-22 18:35:03 +02:00
jschoubben 272e1a65ea Reload the guard for a changed table and restart it only for a changed unit, as the controller declares it (hq ADR 0103) 2026-09-22 18:33:57 +02:00
jschoubben b4f3eaf11b Release a whole-file hold once the file is declared written into (hq ADR 0102) 2026-09-22 18:33:45 +02:00
jschoubben bd3fd17ad8 Do not count the machine's own plumbing mounted into a container as found data (hq ADR 0103) 2026-09-22 18:33:29 +02:00
jschoubben a4e4632077 gofmt the store's firewall record 2026-09-22 18:33:13 +02:00
jschoubben 444ad8f3cf Record the forward policies before disabling ufw, so a retried retirement restores them (hq ADR 0100) 2026-09-22 18:33:09 +02:00
jschoubben aef4993d10 Hold an archive, a process's unit and a user found on an adopted node for an untaken module (hq ADR 0103) 2026-09-22 18:32:20 +02:00
jschoubben 491e04fb8f Load the guard before removing the derived filter when a node returns to adopted, and defer the adoption's orphans only on the flip (hq ADR 0103) 2026-09-22 18:30:13 +02:00
jschoubben b531c47486 Refuse an opening ufw would merge into a found rule that does other than a plain allow, and read log types in either place (hq ADR 0103) 2026-09-22 18:29:06 +02:00
jschoubben facf6af46a Add the mesh's members to a list found in a file written into, and take back only those (hq ADR 0102) 2026-09-22 18:27:51 +02:00
jschoubben 48a8f4cf9d Point the builder's package binding at the port given with --packages-port (hq ADR 0100) 2026-09-22 18:16:00 +02:00
jschoubben 4a095df2f9 Let go of a hold whose resource is no longer declared, touching nothing on disk (hq ADR 0100) 2026-09-22 18:14:37 +02:00
jschoubben 824cb60cbb Hold a found directory, a found service's unit, a container that would mount found data, and a step run in a held container on an adopted node (hq ADR 0103) 2026-09-22 18:14:37 +02:00
jschoubben 35ecf68393 Accept --registry as a host alone again, completing it with the registry's port (hq ADR 0100) 2026-09-22 18:08:50 +02:00
jschoubben d59d232656 Let a re-run of genesis find the package registry it raised itself under its own name (hq ADR 0100) 2026-09-22 18:08:20 +02:00
jschoubben 078e84c681 Read an adopted or converged node's mode from its state on a re-run of genesis, and refuse a flag that disagrees (hq ADR 0103) 2026-09-22 18:08:02 +02:00
jschoubben 52e139d96f Add no opening a found ufw rule already answers, since ufw takes rules differing only in comment for one, as captured on a lab machine (hq ADR 0103) 2026-09-22 18:06:47 +02:00
jschoubben da65f84c45 Read fail2ban's bans as no firewall, and an iptables-nft reject as a refusal, from rulesets captured on a lab machine (hq ADR 0100) 2026-09-22 18:04:48 +02:00
jschoubben 8e2f75454d Put back the forward policy ufw disable opens when the found firewall is retired, as measured on a lab machine (hq ADR 0100) 2026-09-22 18:03:30 +02:00
jschoubben 588ab71d14 Remove an adopted node's guard and openings last on the flip, and keep them if anything failed (hq ADR 0103) 2026-09-22 18:02:46 +02:00
jschoubben 9033e3da98 Retire the found firewall only on a converged declaration from the mesh, never on a carried apply (hq ADR 0100) 2026-09-22 18:01:49 +02:00
jschoubben c989b57439 Guard the broker's plaintext port too at an adopted genesis: the filter admits it from the private network only (hq ADR 0103) 2026-09-22 18:01:14 +02:00
jschoubben 14b3ffbd40 Guard only packets addressed to this machine, and load the guard before the network and stop it only at shutdown (hq ADR 0103) 2026-09-22 18:00:54 +02:00
jschoubben 1e0c6a3516 Publish a reconcile report when what is reachable changed, so the controller's converge preview stays fresh (hq ADR 0100) 2026-09-22 18:00:03 +02:00
jschoubben 0f126d137c Write into a file the machine shares instead of over it, and reload a service that re-reads its configuration instead of restarting it (hq ADR 0102) 2026-09-22 17:48:57 +02:00
jschoubben 6eabed63eb Reload the service manager's units before restarting a service whose files changed, and start the guard before the network as the controller declares it (hq ADR 0100) 2026-09-22 17:38:17 +02:00
jschoubben 3e0e6e6b7e Delete a forwarded opening the way ufw accepts it, and read a fresh machine's resolver as not in use — both measured on a lab machine (hq ADR 0100) 2026-09-22 17:37:01 +02:00
jschoubben 5e3dd3f59c Raise a machine in use adopted: keep its firewall, load no dropping table, guard the mesh's own ports, and take only the mesh's own modules (hq ADR 0100) 2026-09-22 17:32:44 +02:00
jschoubben 4811f176fd Refuse a converged genesis on a machine in use, naming every container and listener counted (hq ADR 0100) 2026-09-22 17:29:32 +02:00
jschoubben 3964d9da0a Take the foundation's ports as genesis inputs, check them free, and hand them to the controller as the node's settings (hq ADR 0100) 2026-09-22 17:28:36 +02:00
jschoubben 770f589401 Report what an adopted node holds, its firewall and what is reachable, and speak unasked when that changes (hq ADR 0100) 2026-09-22 17:22:31 +02:00
jschoubben 3c90d155b3 Converge openings through the firewall an adopted node was found with, and retire it only when the node converges (hq ADR 0100) 2026-09-22 17:19:49 +02:00
jschoubben 3a613113be Keep what an adopted node was found holding until its module is taken, and report it held (hq ADR 0100) 2026-09-22 17:14:04 +02:00
jschoubben fcc447c216 Read a node's adoption from every declaration, so the host knows which modules are untaken (hq ADR 0100) 2026-09-22 17:11:26 +02:00
jschoubben 4fc0330937 Merge pull request 'An enrolling node asks again while the mesh cannot answer, and proves it holds its key (issue 083)' (#19) from multiple-fixes into main 2026-09-22 14:42:56 +02:00
jschoubben 406a5559b0 An enrolling node signs its request with the identity it just generated, so the mesh can tell it from anyone who knows its public key (novox/hq issue 083) 2026-09-22 14:33:31 +02:00
jschoubben eaebae7b36 Review of 083: the give-up message says to wait out the mesh's hold before asking again; the installer no longer says the token is spent when it may not be 2026-09-22 14:23:20 +02:00
jschoubben c8bbdb2da5 An enrolling node asks again, with the same request, while the mesh says it cannot answer — for as long as the mesh holds the token for it (novox/hq issue 083) 2026-09-22 14:07:25 +02:00
jschoubben 0ed8914c07 Merge pull request 'A run-once step names what it reads and runs again when it changed (ADR 0099)' (#18) from multiple-fixes into main 2026-09-21 23:58:46 +02:00
jschoubben 8da78de210 A run-once step names what it reads and runs again when it changed (novox/hq ADR 0099, issue 077) 2026-09-21 23:25:06 +02:00
jschoubben e8b6d9ac31 Merge pull request 'Genesis registers the control plane with the manifest its build produced (hq issue 072)' (#17) from feat/one-controller-manifest into main 2026-09-21 19:23:17 +02:00
jschoubben ffe7dbd348 Step 9 without a build is refused, and a test says so 2026-09-21 19:21:04 +02:00
jschoubben 8afbe57814 The pinning line shows the built image's id, not one digit of it 2026-09-21 15:27:13 +02:00
jschoubben 5223169226 Genesis registers the control plane with the manifest its build produced
The control plane's manifest existed twice: at the root of its repository, read
whenever the mesh rebuilds it from source, and as a copy in the catalogue, read by
genesis. Nothing kept them equal, and the first rebuild replaced the mesh's record
with the repository's shape while every later push was refused (novox/hq
04-ISSUES/072). The builder's one-shot result already carries the manifest it built,
artifact resolved to the image; step 3 keeps it and step 9 registers it, re-pinning
the built image's bare id to the reference the registry assigned. The catalogue is
still read for the registry's and the builder's manifests and for phase two.
2026-09-21 15:17:47 +02:00
jschoubben 4661025eb7 Merge pull request 'The host applies the newest declaration, a file may be created once, the foundation filters first (issues 031, 035, 054)' (#16) from feat/migration-blockers into main 2026-09-21 13:48:51 +02:00
jschoubben c32eada62b The base filter opens the bus and the registry in the input chain too
A container on the machine dialling a port the machine publishes reaches it
through the runtime's proxy — input, not forward — and the builder could not
reach the broker. The derived ruleset opens the mesh's own ports in both
chains; the base one now does the same.
2026-09-21 12:28:57 +02:00
jschoubben b72b71a989 The host applies the newest declaration, a file may be created once, the foundation filters first
031: a window of unacknowledged declarations is drained to the newest; the
rest are set aside and reported as superseded. 035: a file resource may say
create-once — written when absent, kept untouched when present (ADR 0087).
054: the bundle installs nftables and loads a base ruleset before the store
and broker, in the table the filter module later replaces (ADR 0088).
2026-09-21 12:11:52 +02:00
jschoubben d7eea1ab66 Merge pull request 'The installer delivers any MESH_…_FILE own secret, not only the store's' (#15) from feat/secret-not-in-environment into main 2026-09-21 11:59:15 +02:00
jschoubben c5c42376c1 The installer delivers any MESH_…_FILE own secret, not only the store's (ADR 0086) 2026-09-21 10:10:33 +02:00
155 changed files with 24394 additions and 649 deletions
+13 -4
View File
@@ -55,12 +55,13 @@ connects to nothing and listens on nothing — what it applies comes from a file
mesh-host profile what this machine can be asked to do
mesh-host inventory what this machine is, and what it holds
mesh-host apply FILE make this machine match a declaration from a file
mesh-host reconcile make this machine match the declaration this host carries
mesh-host reconcile make this machine match what the mesh last told it — or, before
any mesh has, the bundle this host carries
mesh-host bundle show what this host carries
mesh-host owned what this host has applied and still owns
--json machine-readable
--state where this node keeps what it knows
--dry-run read and check the declaration, change nothing
--dry-run say what applying would change, and change nothing
```
```
@@ -114,8 +115,16 @@ A host built for a machine carries its declaration **inside the binary**:
make host BUNDLE=path/to/foundation.lock
```
`mesh-host reconcile` then applies it. That is the first node's path — no mesh present, nothing
fetched, nothing else copied onto the machine. `copy it and run it` stops being true the moment
`mesh-host reconcile` then applies it, on a machine the mesh has told nothing yet. That is the
first node's path — no mesh present, nothing fetched, nothing else copied onto the machine. Once
the mesh has spoken, `reconcile` holds the machine to what it last said and never to the bundle,
which genesis consumed; a bundle or a file is refused when it says the other mode than the node
is in; and `apply FILE` is refused altogether once the mesh has spoken — a file is applied as the
bundle is, its resources recorded as the machine's own, so on an enrolled node it would plan to
remove the foundation. `apply FILE` is for a machine the mesh has not spoken to. (hq's to-be
node lifecycle describes `apply repair.json` as a rescue on an enrolled node; that line is being
amended in hq, and no rescue path exists here yet.) Both commands say what they would change
before changing anything, and `--dry-run` is that alone. `copy it and run it` stops being true the moment
a second file has to arrive with it, which is why the bundle is embedded rather than beside it.
**A default build carries nothing and refuses to reconcile**, saying so. A host that applied
+89 -4
View File
@@ -18,6 +18,7 @@ package main
import (
"context"
"encoding/json"
"errors"
"flag"
"fmt"
"io"
@@ -25,6 +26,7 @@ import (
"net/http"
"os"
"os/signal"
"strconv"
"syscall"
"time"
@@ -96,8 +98,8 @@ const usage = `mesh-bootstrap — make a bare machine into a mesh
everything this mesh will ever run
--source-path the module's directory inside that repository, if not its root
--catalog a checkout of the mesh's catalogue, holding the registry's, the
control plane's and the builder's manifests. Without it this stops
after step 6
control plane's, the builder's and — when a packages port is given
— the forge's manifests. Without it this stops after step 6
--node the name this machine is known by (default: its hostname)
--registry where this mesh keeps its own images (default ` + defaultRegistry + `)
every node pulls the control plane from this, so on a mesh of more
@@ -126,6 +128,23 @@ const usage = `mesh-bootstrap — make a bare machine into a mesh
--packet-filter which packet filter to run (nftables)
--extras catalogue modules beyond the floor, comma-separated
The foundation's ports are this machine's, each checked free before anything is
raised and kept as the node's setting for the module that binds it:
--store-port 5432 --bus-port 5671 --amqp-port 5672 --management-port 15672
--registry-port 5000 (follows --registry, and must agree with it)
--packages-port 3000 --hub-port 51820/udp
--overlay-range the private network's range (default 10.42.0.0/16); refused
if it overlaps an interface or route the machine already has
--adopted raise a machine in use as an adopted node: what it runs and its
firewall stay as they are, the foundation's filter is not loaded and
the mesh guards its own ports instead, and each module is taken on it
one at a time. Without it, a machine in use is refused
--tunnel adopted: the interface of the tunnel the private network takes over
(its key, port, range and peers); found by itself when one is up, and
needed only when several are. --hub-port and --overlay-range then
follow the tunnel
The installer carries a builder, not a control plane. What raises a mesh is therefore
the same thing that will maintain it, and the control plane a mesh ends up running is
one it built itself, from a repository and a commit it can name and build again.
@@ -176,7 +195,9 @@ func parseArgs(args []string) (string, bootstrap.Options, bool, error) {
HostService: defaultService,
// Longer than the host's 10s: these probes reach a container runtime that may be busy
// pulling, and a probe that times out on a working machine is a false refusal.
Timeout: 30 * time.Second,
Ports: bootstrap.DefaultPorts(),
OverlayRange: bootstrap.DefaultOverlayRange,
Timeout: 30 * time.Second,
// A socket-activated runtime queued behind the network, and a control plane running its
// first `initdb`-shaped wait, are both minutes rather than seconds.
Wait: 3 * time.Minute,
@@ -210,9 +231,50 @@ func parseArgs(args []string) (string, bootstrap.Options, bool, error) {
return "", opts, false, fmt.Errorf(
"unexpected argument %q — try `mesh-bootstrap help`", positionals[0])
}
if err := registryAgrees(set, &opts); err != nil {
return "", opts, false, err
}
return command, opts, jsonOut, nil
}
// registryAgrees makes --registry and --registry-port say one port (novox/hq ADR 0100): the
// registry is raised on the port the node gives it, and every node pulls from the address given.
// Either may be said alone and the other follows; said both ways, they must agree.
func registryAgrees(set *flag.FlagSet, opts *bootstrap.Options) error {
said := map[string]bool{}
set.Visit(func(f *flag.Flag) { said[f.Name] = true })
host, portText, err := net.SplitHostPort(opts.Registry)
var missing *net.AddrError
if errors.As(err, &missing) && missing.Err == "missing port in address" {
// A host alone, as --registry took before its port became the node's: the registry's
// port — the one given, or the catalogue's — completes it.
port := opts.Ports.Registry
if port == 0 {
port = bootstrap.DefaultPorts().Registry
}
opts.Ports.Registry = port
opts.Registry = net.JoinHostPort(strings.Trim(opts.Registry, "[]"), strconv.Itoa(port))
return nil
}
if err != nil {
return fmt.Errorf("--registry %q is not host:port: %w", opts.Registry, err)
}
port, err := strconv.Atoi(portText)
if err != nil {
return fmt.Errorf("--registry %q does not end in a port", opts.Registry)
}
switch {
case said["registry-port"] && said["registry"] && port != opts.Ports.Registry:
return fmt.Errorf("--registry %s and --registry-port %d name two ports for one registry",
opts.Registry, opts.Ports.Registry)
case said["registry-port"]:
opts.Registry = net.JoinHostPort(host, strconv.Itoa(opts.Ports.Registry))
case said["registry"]:
opts.Ports.Registry = port
}
return nil
}
func newFlagSet(opts *bootstrap.Options, jsonOut *bool) *flag.FlagSet {
set := flag.NewFlagSet("mesh-bootstrap", flag.ContinueOnError)
set.SetOutput(os.Stderr)
@@ -221,7 +283,7 @@ func newFlagSet(opts *bootstrap.Options, jsonOut *bool) *flag.FlagSet {
set.StringVar(&opts.Out, "out", opts.Out, "where the produced bundle is written")
set.StringVar(&opts.State, "state", opts.State, "where this node records what it has applied")
set.StringVar(&opts.Catalogue, "catalog", opts.Catalogue,
"a checkout of the mesh's catalogue; without it this stops after the foundation")
"a checkout of the mesh's catalogue, for the registry's, the builder's and the forge's manifests and what phase two installs; without it this stops after the foundation")
set.StringVar(&opts.Source.Repository, "source", opts.Source.Repository,
"the repository the control plane is built from, on a mesh that already exists")
set.StringVar(&opts.Source.Ref, "source-ref", opts.Source.Ref,
@@ -255,6 +317,29 @@ func newFlagSet(opts *bootstrap.Options, jsonOut *bool) *flag.FlagSet {
set.StringVar(&opts.SDKSource.Ref, "sdk-ref", opts.SDKSource.Ref,
"what of it to build (default main)")
set.StringVar(&opts.Site, "site", "main", "where this machine sits, for the private network")
// The foundation's ports are this node's (novox/hq ADR 0100): each is checked free before
// anything is raised, and becomes the node's setting for the module that binds it.
for _, p := range []struct {
name, what string
into *int
}{
{"store-port", "the store", &opts.Ports.Store},
{"bus-port", "the bus (amqps)", &opts.Ports.Bus},
{"amqp-port", "the broker's AMQP", &opts.Ports.AMQP},
{"management-port", "the broker's management, on loopback", &opts.Ports.Management},
{"registry-port", "the registry", &opts.Ports.Registry},
{"packages-port", "the package registry", &opts.Ports.Packages},
{"hub-port", "the private network's hub (udp)", &opts.Ports.Hub},
} {
set.IntVar(p.into, p.name, *p.into, "the machine's port for "+p.what)
}
set.BoolVar(&opts.Adopted, "adopted", false,
"raise this machine adopted: keep what it runs and its firewall until each module is taken")
set.StringVar(&opts.OverlayRange, "overlay-range", opts.OverlayRange,
"the private network's address range; must not overlap a tunnel the machine already runs")
set.StringVar(&opts.Tunnel, "tunnel", "",
"adopted: the found tunnel's interface the private network takes over; found by itself when one is up")
if opts.Answers == nil {
opts.Answers = map[string]string{}
}
+40
View File
@@ -164,3 +164,43 @@ func TestTheNodeNameCanBeSaid(t *testing.T) {
t.Errorf("--catalog parsed as %q", opts.Catalogue)
}
}
// Defends novox/hq ADR 0100: the foundation's ports are inputs to genesis, and the registry's port
// and the address nodes pull from say one port.
func TestTheFoundationsPortsAreGiven(t *testing.T) {
_, opts, _, err := parseArgs([]string{"--store-port", "5433", "--hub-port", "51821", "--overlay-range", "10.77.0.0/16"})
if err != nil {
t.Fatal(err)
}
if opts.Ports.Store != 5433 || opts.Ports.Hub != 51821 || opts.Ports.Bus != 5671 || opts.OverlayRange != "10.77.0.0/16" {
t.Errorf("ports read as %+v, range %s", opts.Ports, opts.OverlayRange)
}
}
func TestTheRegistrysPortAndAddressAgree(t *testing.T) {
_, opts, _, err := parseArgs([]string{"--registry-port", "5100"})
if err != nil || opts.Registry != "127.0.0.1:5100" {
t.Errorf("--registry-port alone: %s %v", opts.Registry, err)
}
_, opts, _, err = parseArgs([]string{"--registry", "192.0.2.10:5100"})
if err != nil || opts.Ports.Registry != 5100 {
t.Errorf("--registry alone: %d %v", opts.Ports.Registry, err)
}
if _, _, _, err := parseArgs([]string{"--registry", "192.0.2.10:5000", "--registry-port", "5100"}); err == nil {
t.Error("two ports for one registry were accepted")
}
}
func TestARegistryGivenAsAHostAloneTakesTheRegistrysPort(t *testing.T) {
_, opts, _, err := parseArgs([]string{"--registry", "192.0.2.10"})
if err != nil || opts.Registry != "192.0.2.10:5000" || opts.Ports.Registry != 5000 {
t.Errorf("--registry host alone: %s %d %v", opts.Registry, opts.Ports.Registry, err)
}
_, opts, _, err = parseArgs([]string{"--registry", "192.0.2.10", "--registry-port", "5100"})
if err != nil || opts.Registry != "192.0.2.10:5100" {
t.Errorf("--registry host with --registry-port: %s %v", opts.Registry, err)
}
if _, _, _, err := parseArgs([]string{"--registry", "192.0.2.10:notaport"}); err == nil {
t.Error("a registry with a port that is not a number was accepted")
}
}
+895 -66
View File
File diff suppressed because it is too large Load Diff
+475 -1
View File
@@ -1,9 +1,24 @@
package main
import (
"github.com/novox/mesh-host/internal/store"
"bytes"
"context"
"crypto/ed25519"
"encoding/json"
"errors"
"os"
"path/filepath"
"strings"
"testing"
"time"
"github.com/novox/mesh-host/internal/apply"
"github.com/novox/mesh-host/internal/bundle"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/identity"
"github.com/novox/mesh-host/internal/link"
"github.com/novox/mesh-host/internal/store"
"github.com/novox/mesh-host/internal/system"
)
// Argument handling gets tests because it already failed silently once: `mesh-host inventory
@@ -135,3 +150,462 @@ func TestAFlagAfterAPositionalIsRead(t *testing.T) {
}
}
}
// Defends novox/hq ADR 0100: a reconcile on an adopted node speaks unasked only when what it holds
// or its firewall changed — which is how a predecessor still writing is caught, without a report
// every five minutes saying nothing new.
func TestAReconcileSpeaksOnlyWhenWhatIsHeldChanged(t *testing.T) {
w := &adoptionWatch{}
held := link.Report{Firewall: "ufw", Held: []link.Held{{ID: "hello-web.page"}, {ID: "hello-web.server"}}}
if !w.changed(held) {
t.Fatal("the first report of a hold was not said")
}
again := link.Report{Firewall: "ufw", Held: []link.Held{{ID: "hello-web.server"}, {ID: "hello-web.page"}}}
if w.changed(again) {
t.Error("the same holds in another order were said again")
}
rewritten := link.Report{Firewall: "ufw", Held: []link.Held{{ID: "hello-web.page", Changed: "rewritten"}, {ID: "hello-web.server"}}}
if !w.changed(rewritten) {
t.Error("a held file rewritten by something else was not said")
}
if !w.changed(link.Report{Firewall: "none", Held: rewritten.Held}) {
t.Error("a changed firewall was not said")
}
// What filters the machine is part of it (novox/hq ADR 0168): a predecessor's chain removed by
// hand, or the found firewall enabled again, is said without being asked.
filtered := link.Report{Firewall: "none", Held: rewritten.Held,
Filters: []link.Filter{{Where: "chain HAL-MESH-ONLY (iptables-legacy)", Owner: "other", Refuses: "-j DROP"}}}
if !w.changed(filtered) {
t.Error("a filter appearing was not said")
}
if !w.changed(link.Report{Firewall: "none", Held: rewritten.Held}) {
t.Error("a filter removed was not said")
}
if !w.changed(link.Report{Firewall: "none", Held: rewritten.Held, FoundFirewall: &link.FoundFirewall{Kind: "ufw", Active: true}}) {
t.Error("the found firewall coming back was not said")
}
if !worthSaying(link.Report{Filters: filtered.Filters}) {
t.Error("a report carrying only what filters the machine is not worth saying")
}
}
func TestWhatTheLinkPublishedCountsAsSaid(t *testing.T) {
w := &adoptionWatch{}
report := link.Report{Firewall: "ufw", Held: []link.Held{{ID: "a"}}}
applier := w.noting(func(context.Context, []byte, []byte) link.Report { return report })
applier(context.Background(), nil, nil)
if w.changed(report) {
t.Error("a reconcile repeated what the link had just published")
}
}
func TestAReconcileSpeaksWhenWhatIsReachableChanged(t *testing.T) {
// The controller previews a flip from what the node last said is reachable; a port that opened
// since must reach it without waiting for the next delivery (novox/hq ADR 0100).
w := &adoptionWatch{}
before := link.Report{Firewall: "ufw", Reachable: []link.Reach{
{Protocol: "tcp", Address: "0.0.0.0", Port: 22, By: "sshd"}}}
if !w.changed(before) {
t.Fatal("the first report was not said")
}
reordered := link.Report{Firewall: "ufw", Reachable: []link.Reach{
{Protocol: "tcp", Address: "0.0.0.0", Port: 22, By: "sshd"}}}
if w.changed(reordered) {
t.Error("the same reachable set was said again")
}
opened := link.Report{Firewall: "ufw", Reachable: append(before.Reachable,
link.Reach{Protocol: "tcp", Address: "0.0.0.0", Port: 8080, By: "hello-web", Published: true, ContainerPort: 80})}
if !w.changed(opened) {
t.Error("a newly published port was not said")
}
}
// Defends the node's own record: the link and the reconcile loop both apply, and each reads the
// state, acts, and writes it back — so they must not run at the same time, or the last save loses
// what the other recorded.
func TestOnlyOneApplyRunsAtATime(t *testing.T) {
// A host is built for one system at link time, and a test binary has no link time: this asks
// the machine it runs on, and stands aside where the answer is no.
built, err := system.For("arch")
if err != nil || built.Confirm(context.Background(), apply.ExecRunner) != nil {
t.Skip("this machine is not one these tests can apply on")
}
was := builtFor
builtFor = "arch"
t.Cleanup(func() { builtFor = was })
dir := t.TempDir()
opts := options{state: filepath.Join(dir, "state.json")}
raw := []byte(`{"declaration":1,"resources":[{"id":"a","type":"file","path":"` +
filepath.Join(dir, "a.conf") + `","content":"x\n"}]}`)
// Whatever else is applying — the link, while this is the reconcile — this waits for it.
applying.Lock()
done := make(chan link.Report, 1)
go func() { done <- applyAndKeep(context.Background(), opts, raw, nil, nil, nil) }()
select {
case report := <-done:
applying.Unlock()
t.Fatalf("an apply ran while another held the node: %+v", report)
case <-time.After(50 * time.Millisecond):
}
if _, err := os.Stat(filepath.Join(dir, "a.conf")); !errors.Is(err, os.ErrNotExist) {
applying.Unlock()
t.Fatal("the waiting apply had already touched the machine")
}
applying.Unlock()
select {
case report := <-done:
if report.Refused != "" {
t.Fatalf("refused: %s", report.Refused)
}
case <-time.After(10 * time.Second):
t.Fatal("the apply never ran once the node was free")
}
known, loadErr := store.Load(opts.state)
if loadErr != nil || len(known.Resources) != 1 {
t.Errorf("the apply recorded %d resource(s): %v", len(known.Resources), loadErr)
}
}
// Defends novox/hq ADR 0100: a change is counted as said only once the mesh has been told. Queued
// and lost — the link down when the reconcile spoke — it must be said again.
func TestAChangeThatNeverReachedTheMeshIsSaidAgain(t *testing.T) {
w := &adoptionWatch{}
held := link.Report{Firewall: "ufw", Held: []link.Held{{ID: "hello-web.page", Changed: "rewritten"}}}
if !w.differs(held) {
t.Fatal("the first report of a change was not new")
}
// The link was down: nothing published it, so nothing says it was said.
if !w.differs(held) {
t.Error("a change that never reached the mesh was counted as said")
}
w.said(held)
if w.differs(held) {
t.Error("a change the mesh was told was said again")
}
}
// Defends novox/hq issue 104: a declaration for the other mode than this node is in is refused at
// the point of application, whichever command delivered it, naming both — an adopted control-node
// once applied its converged genesis bundle and closed itself for forty-five minutes.
func stateWithMode(t *testing.T, mode string) options {
t.Helper()
dir := t.TempDir()
opts := options{state: filepath.Join(dir, "state.json"), out: &bytes.Buffer{}}
if err := store.Save(opts.state, store.State{Mode: mode}); err != nil {
t.Fatal(err)
}
return opts
}
func TestAConvergedDeclarationIsRefusedOnAnAdoptedNode(t *testing.T) {
opts := stateWithMode(t, store.ModeAdopted)
path := filepath.Join(filepath.Dir(opts.state), "filter.conf")
raw := []byte(`{"declaration":1,"resources":[{"id":"filter","type":"file","path":"` + path +
`","content":"table inet filter { chain input { policy drop; } }\n"}]}`)
d, err := declaration.ParseFileTrusted(raw)
if err != nil {
t.Fatal(err)
}
for _, from := range []provenance{fromFile, fromBundle} {
err := runApply(context.Background(), opts, d, raw, from)
if err == nil {
t.Fatalf("a converged declaration was applied to an adopted node (from %d)", from)
}
want := "this node is adopted; the declaration says converged"
if !strings.Contains(err.Error(), want) || !strings.Contains(err.Error(), "`converge`") {
t.Errorf("the refusal does not name both modes and the act that changes it: %v", err)
}
}
if _, err := os.Stat(path); !errors.Is(err, os.ErrNotExist) {
t.Error("the refused declaration touched the machine")
}
}
func TestTheKeptDeclarationRepairsARecordThatDisagrees(t *testing.T) {
// What a node kept is signed by the mesh and verified on load; the state's note of the mode is
// the host's own. A genesis re-run after `converge`, or a state saved when the kept declaration
// could not be, leaves them apart — and a loop that refused every five minutes would hold the
// node off what the mesh said until a delivery that comes only when something changes. The
// kept declaration wins, and the repair is said.
opts := stateWithMode(t, store.ModeConverged)
raw := []byte(`{"declaration":1,"adoption":{"taken":[]},"resources":[{"id":"a","type":"file","path":"` +
filepath.Join(filepath.Dir(opts.state), "a.conf") + `","content":"x\n"}]}`)
var said []string
report := applyAndKeep(context.Background(), opts, raw, nil, nil, func(line string) { said = append(said, line) })
if strings.Contains(report.Refused, "the declaration says") {
t.Fatalf("what the node kept was refused against its own note: %s", report.Refused)
}
if len(said) != 1 || !strings.Contains(said[0], "record said converged") ||
!strings.Contains(said[0], "says adopted") || !strings.Contains(said[0], "repaired") {
t.Errorf("the repair was not said: %q", said)
}
}
func TestTheMeshItselfMayChangeTheMode(t *testing.T) {
// The flip is a declaration: `converge` on the controller records the mode and sends the
// first converged declaration. Delivered by the link, signed, it is not held to the record —
// it becomes it. (With no system linked in, the apply is refused later for that; what this
// checks is that the refusal is not the mode's.)
opts := stateWithMode(t, store.ModeAdopted)
raw := []byte(`{"declaration":1,"resources":[{"id":"a","type":"file","path":"` +
filepath.Join(filepath.Dir(opts.state), "a.conf") + `","content":"x\n"}]}`)
report := applyAndKeep(context.Background(), opts, raw, &store.Declared{Declaration: raw}, nil, nil)
if strings.Contains(report.Refused, "the declaration says") {
t.Errorf("the mesh's own flip was refused for its mode: %s", report.Refused)
}
}
// Defends novox/hq issue 104: a declaration older than what the mesh last said is refused, naming
// both — and `apply FILE` is refused altogether once the mesh has spoken, the last declaration
// itself included: from a file it is applied as the bundle is, which would record the mesh's
// resources as this machine's own and remove the foundation as undeclared.
func TestAnOlderDeclarationIsRefused(t *testing.T) {
opts := stateWithMode(t, "")
genesis := []byte(`{"declaration":1,"resources":[{"id":"g","type":"file","path":"/tmp/g","content":"genesis\n"}]}`)
since := []byte(`{"declaration":1,"resources":[{"id":"g","type":"file","path":"/tmp/g","content":"since\n"}]}`)
other := []byte(`{"declaration":1,"resources":[{"id":"g","type":"file","path":"/tmp/g","content":"other\n"}]}`)
if err := store.Save(opts.state, store.State{Genesis: &store.Genesis{Digest: apply.DigestOf(genesis),
At: time.Now(), Rewritten: true}}); err != nil {
t.Fatal(err)
}
if err := store.SaveDeclared(store.DeclaredPath(opts.state), store.Declared{Declaration: since,
Signature: []byte("unverified here")}); err != nil {
t.Fatal(err)
}
parsed := func(raw []byte) *declaration.Declaration {
d, err := declaration.ParseFileTrusted(raw)
if err != nil {
t.Fatal(err)
}
return d
}
err := runApply(context.Background(), opts, parsed(genesis), genesis, fromFile)
if err == nil {
t.Fatal("the bundle genesis consumed was applied over what the mesh said since")
}
for _, want := range []string{"older", short(apply.DigestOf(genesis)), short(apply.DigestOf(since))} {
if !strings.Contains(err.Error(), want) {
t.Errorf("the refusal does not say %q: %v", want, err)
}
}
err = runApply(context.Background(), opts, parsed(other), other, fromFile)
if err == nil {
t.Fatal("a declaration that is not what the mesh last said was applied")
}
if !strings.Contains(err.Error(), "for a machine the mesh has not spoken to") ||
!strings.Contains(err.Error(), short(apply.DigestOf(since))) {
t.Errorf("the refusal does not say what a file is for and what was last said: %v", err)
}
// The very declaration the mesh last sent, from a file: still a file.
err = runApply(context.Background(), opts, parsed(since), since, fromFile)
if err == nil || !strings.Contains(err.Error(), "applied as the bundle is") {
t.Errorf("the last declaration, from a file, was not refused as a file: %v", err)
}
if known, _ := store.Load(opts.state); len(known.Resources) != 0 {
t.Errorf("a refused file recorded %d resource(s)", len(known.Resources))
}
// A bundle other than the one genesis consumed: what genesis applied was rewritten for this
// machine, so the carried bytes never are.
err = runApply(context.Background(), opts, parsed(other), other, fromBundle)
if err == nil || !strings.Contains(err.Error(), "consumed") ||
!strings.Contains(err.Error(), short(apply.DigestOf(genesis))) {
t.Errorf("a bundle other than the consumed one was not refused naming it: %v", err)
}
}
// Defends novox/hq issue 104: what an apply would change is said before anything is, and
// `--dry-run` is that and nothing else — an action listed as the action it is.
func TestADryRunChangesNothingAndListsTheActions(t *testing.T) {
opts := stateWithMode(t, "")
opts.dryRun = true
out := &bytes.Buffer{}
opts.out = out
path := filepath.Join(filepath.Dir(opts.state), "a.conf")
raw := []byte(`{"declaration":1,"resources":[
{"id":"a","type":"file","path":"` + path + `","content":"x\n"},
{"id":"init","type":"action","command":["createdb","mesh"],"verify":["psql","-c","select 1"]}]}`)
d, err := declaration.ParseFileTrusted(raw)
if err != nil {
t.Fatal(err)
}
if err := runApply(context.Background(), opts, d, raw, fromFile); err != nil {
t.Fatalf("a dry run failed: %v", err)
}
if _, err := os.Stat(path); !errors.Is(err, os.ErrNotExist) {
t.Error("a dry run wrote the file")
}
for _, want := range []string{"would change", "create file a", "run action init",
"`createdb mesh`", "--dry-run: nothing applied"} {
if !strings.Contains(out.String(), want) {
t.Errorf("the preview does not say %q:\n%s", want, out.String())
}
}
if strings.Contains(out.String(), "applying:") {
t.Errorf("a dry run went on to apply:\n%s", out.String())
}
// Machine-readable, the same shape as an apply's: the plan, and no report.
out.Reset()
opts.json = true
if err := runApply(context.Background(), opts, d, raw, fromFile); err != nil {
t.Fatal(err)
}
var shape struct {
Plan []apply.Step `json:"plan"`
Report *json.RawMessage `json:"report"`
}
if err := json.Unmarshal(out.Bytes(), &shape); err != nil || len(shape.Plan) != 2 || shape.Report != nil {
t.Errorf("a json dry run is not {plan} alone: %v\n%s", err, out.String())
}
}
// Defends novox/hq issue 104: once the mesh has told this node anything, `reconcile` holds it to
// that — never to the bundle the host carries, which genesis consumed.
func TestReconcileAfterAControllerDeclarationDoesNotReapplyTheBundle(t *testing.T) {
was := builtFor
builtFor = "arch"
t.Cleanup(func() { builtFor = was })
opts := stateWithMode(t, store.ModeAdopted)
controllerPublic, controllerPrivate, err := ed25519.GenerateKey(nil)
if err != nil {
t.Fatal(err)
}
said := []byte(`{"declaration":1,"adoption":{"taken":[]},"resources":[{"id":"a","type":"file","path":"/tmp/a","content":"x\n"}]}`)
if err := store.SaveDeclared(store.DeclaredPath(opts.state), store.Declared{Declaration: said,
Signature: ed25519.Sign(controllerPrivate, said)}); err != nil {
t.Fatal(err)
}
// Told, and unable to prove by whom: refused, and the bundle is not applied in its place.
_, _, _, err = reconcileSource(opts)
if err == nil || !strings.Contains(err.Error(), "not applied in its place") {
t.Errorf("a node that cannot prove what it was told fell back to something: %v", err)
}
mine, err := identity.Generate("workstation")
if err != nil {
t.Fatal(err)
}
mine.Membership = identity.Membership{Broker: "198.51.100.10:5671", Fingerprint: "sha256:0",
Signer: controllerPublic, Password: "issued"}
if err := identity.Save(identity.Path(opts.state), mine); err != nil {
t.Fatal(err)
}
d, raw, from, err := reconcileSource(opts)
if err != nil {
t.Fatal(err)
}
if from != fromDeclared || !bytes.Equal(raw, said) || d.Adoption == nil {
t.Errorf("reconcile chose %d with %d bytes, not what the mesh last said", from, len(raw))
}
// And before the mesh has said anything: the bundle, as it always was. A test binary carries
// only the placeholder, and asking for it is what proves the path.
if err := os.Remove(store.DeclaredPath(opts.state)); err != nil {
t.Fatal(err)
}
_, _, _, err = reconcileSource(opts)
if !errors.Is(err, bundle.ErrEmpty) {
t.Errorf("with nothing said, reconcile did not reach for the carried bundle: %v", err)
}
}
// **A machine says which links face outside without being asked.**
//
// The mesh composes no filter for a machine that has not said (novox/hq ADR 0140), and a machine only
// speaks unasked when this fingerprint changes. Left out of it, a machine that has just learnt to say
// could speak only when a declaration arrived — and a declaration cannot be composed until it has
// spoken. A machine waiting for a push that is waiting for the machine.
func TestANewOutwardLinkIsSaidUnasked(t *testing.T) {
w := &adoptionWatch{}
first := link.Report{Firewall: "none"}
if !w.differs(first) {
t.Fatal("the first report should differ from nothing")
}
w.said(first)
// Only the links changed, and nothing about adoption.
learnt := link.Report{Firewall: "none", Outward: []string{"eth0"}}
if !w.differs(learnt) {
t.Fatal("a machine that has just learnt which links face outside would never say so, " +
"and could then never be sent a filter")
}
w.said(learnt)
if w.differs(link.Report{Firewall: "none", Outward: []string{"eth0"}}) {
t.Fatal("the same links are reported as a change, so the machine would speak on every reconcile")
}
// And a link that changes — a laptop moving from a cable to a radio — is said too, because the
// filter is written around the old one until it is.
if !w.differs(link.Report{Firewall: "none", Outward: []string{"wlan0"}}) {
t.Fatal("a changed outward link is not said, so the filter stays written around the old one")
}
}
// **A converged machine can say which links face outside, unasked.**
//
// A reconcile is otherwise silent, and the condition deciding when it speaks asked only what an
// adopted node reports — what it holds, and the firewall it found. A converged node has neither, so
// it could speak only in reply to a declaration, and the mesh composes no declaration for a node
// that has not said which links face outside (novox/hq ADR 0140). Measured: three converged machines
// sat silent while the control plane refused to send them a filter.
func TestAConvergedMachineSaysItsOutwardLinksUnasked(t *testing.T) {
// A converged node's reconcile: nothing held, no found firewall, and the links it can see.
converged := link.Report{Outward: []string{"eth0"}}
if !worthSaying(converged) {
t.Fatal("a converged machine cannot say which links face outside, so it can never be " +
"sent a filter — a machine waiting for a push that is waiting for the machine")
}
// An adopted node's reasons still hold, because that is how a predecessor still writing is caught.
if !worthSaying(link.Report{Firewall: "ufw"}) {
t.Fatal("an adopted machine no longer says which firewall it found")
}
if !worthSaying(link.Report{Held: []link.Held{{ID: "a-file"}}}) {
t.Fatal("an adopted machine no longer says what it holds")
}
// And a reconcile with nothing to say stays silent, or every machine speaks every five minutes
// about nothing.
if worthSaying(link.Report{}) {
t.Fatal("a reconcile with nothing to say speaks anyway")
}
// A refused report says nothing about the machine; the refusal is for the console.
if worthSaying(link.Report{Outward: []string{"eth0"}, Refused: "not for this node"}) {
t.Fatal("a refused report is offered as news about the machine")
}
}
// **A host running as a service says what its apply did.**
//
// The serving path passed nil where the apply writes its detail, and nil is silence. The one-shot path
// has always passed a real function, so everything the apply says was visible when a person ran it by
// hand and discarded in the way the host actually runs. Measured before this was written: a machine was
// converged, its found firewall was not retired, and what the host decided was unrecoverable because it
// had been said to nobody (novox/hq 04-ISSUES/143).
//
// This asserts only that the apply's log is never nil and that a line reaches what the caller gave.
// **It cannot catch the fault it was written for** — a call site passing nil directly — because that is
// wiring, and wiring is only proved by running the thing. That proof is a deployed host whose journal
// carries the apply's detail, which is how this fix was verified.
func TestTheApplysLogIsNeverNil(t *testing.T) {
if announceOr(nil) == nil {
t.Fatal("a host with nowhere to say things got a nil log, which the apply will call")
}
announceOr(nil)("this goes nowhere and must not panic")
var said []string
announceOr(func(line string) { said = append(said, line) })(" disabled ufw")
if len(said) != 1 || !strings.Contains(said[0], "disabled ufw") {
t.Fatalf("the apply's detail did not reach the caller's announce: %v", said)
}
}
+58
View File
@@ -0,0 +1,58 @@
package main
import (
"encoding/json"
"strings"
"testing"
"github.com/novox/mesh-host/internal/store"
)
// A declaration carries no order, so a host cannot tell an older one from a newer (novox/hq
// 04-ISSUES/107). Its only identity was the digest of its bytes: "not the last" could be said,
// "older" could not.
func keptWith(t *testing.T, sequence int64) store.Declared {
t.Helper()
body, err := json.Marshal(map[string]any{
"declaration": 1, "resources": []any{}, "owns_nothing": true, "sequence": sequence,
})
if err != nil {
t.Fatal(err)
}
return store.Declared{Declaration: body, Signature: []byte("x")}
}
func TestAnOlderDeclarationFromTheMeshIsRefused(t *testing.T) {
err := refuseOlder(keptWith(t, 7), nil, 5)
if err == nil {
t.Fatal("sequence 5 was accepted over a kept 7")
}
if !strings.Contains(err.Error(), "older") || !strings.Contains(err.Error(), "nothing was applied") {
t.Fatalf("the refusal does not say what it is: %v", err)
}
}
func TestANewerOrEqualDeclarationIsNot(t *testing.T) {
if err := refuseOlder(keptWith(t, 7), nil, 8); err != nil {
t.Fatalf("sequence 8 was refused over a kept 7: %v", err)
}
// Equal is the same declaration again, which reconciling is for.
if err := refuseOlder(keptWith(t, 7), nil, 7); err != nil {
t.Fatalf("the same sequence was refused: %v", err)
}
}
func TestNoOrderClaimedMeansNoOrderCompared(t *testing.T) {
// An older controller sends none; a host that received before it understood them kept none.
// Refusing on a guess would strand a node the moment the controller is older than the host.
if err := refuseOlder(keptWith(t, 7), nil, 0); err != nil {
t.Fatalf("a declaration claiming no order was refused: %v", err)
}
if err := refuseOlder(keptWith(t, 0), nil, 3); err != nil {
t.Fatalf("a declaration was refused against a kept one that claimed no order: %v", err)
}
if err := refuseOlder(store.Declared{}, store.ErrNothingDeclared, 3); err != nil {
t.Fatalf("a first declaration was refused: %v", err)
}
}
+94
View File
@@ -0,0 +1,94 @@
package main
import (
"crypto/ed25519"
"strings"
"testing"
"github.com/novox/mesh-host/internal/identity"
"github.com/novox/mesh-host/internal/link"
"github.com/novox/mesh-host/internal/tunnel"
)
// novox/hq ADR 0105: `overlay take` moves this node's overlay key onto the found tunnel's and
// nothing else — identity, sealing and serving keys stay as they were — and tells the mesh with a
// proof signed by the identity key, over the previous key, the new one and the tunnel.
func anEnrolledNode(t *testing.T) identity.Identity {
t.Helper()
mine, err := identity.Generate("anchor")
if err != nil {
t.Fatal(err)
}
mine.Overlay, err = identity.GenerateOverlayKey()
if err != nil {
t.Fatal(err)
}
mine.Membership = identity.Membership{Broker: "198.51.100.1:5671", Fingerprint: "sha256:aa",
Signer: make([]byte, ed25519.PublicKeySize), Password: "p"}
return mine
}
func aFoundTunnel(t *testing.T) tunnel.Found {
t.Helper()
private, err := identity.GenerateOverlayKey()
if err != nil {
t.Fatal(err)
}
found, err := tunnel.Parse([]byte("[Interface]\nPrivateKey = " + private.Private + "\nListenPort = 51900\n" +
"Address = 192.0.2.1/24\n[Peer]\nPublicKey = PEER-A=\nAllowedIPs = 192.0.2.2/32\n"))
if err != nil {
t.Fatal(err)
}
found.Interface, found.Unit, found.Config = "wg0", "wg-quick@wg0", "/etc/wireguard/wg0.conf"
return found
}
func TestTakingATunnelMovesOnlyTheOverlayKeyAndSignsForIt(t *testing.T) {
mine := anEnrolledNode(t)
found := aFoundTunnel(t)
before := mine.Overlay.Public
taken, rekey, err := rekeyOnto(mine, found)
if err != nil {
t.Fatal(err)
}
if taken.Overlay.Public != found.PublicKey || taken.Overlay.Private != found.PrivateKey() {
t.Fatal("the overlay key is not the tunnel's")
}
if string(taken.Public) != string(mine.Public) || string(taken.Private) != string(mine.Private) ||
taken.Node != mine.Node || taken.Membership.Password != mine.Membership.Password ||
taken.Membership.Broker != mine.Membership.Broker {
t.Fatal("something other than the overlay key moved")
}
if taken.OverlayBefore != before {
t.Errorf("the key before the take was not kept: %q", taken.OverlayBefore)
}
if rekey.Previous != before || rekey.OverlayKey != found.PublicKey || rekey.Tunnel == nil ||
rekey.Tunnel.PublicKey != found.PublicKey || len(rekey.Tunnel.Peers) != 1 {
t.Fatalf("the rekey does not say what moved: %+v", rekey)
}
if !ed25519.Verify(ed25519.PublicKey(mine.Public),
link.RekeyProof("anchor", before, found.PublicKey, rekey.Tunnel), rekey.Proof) {
t.Fatal("the rekey is not signed by this node's identity key over what it says")
}
if ed25519.Verify(ed25519.PublicKey(mine.Public),
link.RekeyProof("laptop", before, found.PublicKey, rekey.Tunnel), rekey.Proof) {
t.Fatal("the proof is not bound to the node")
}
for _, said := range []string{rekey.Previous, rekey.OverlayKey, rekey.Tunnel.Interface} {
if strings.Contains(said, found.PrivateKey()) {
t.Fatal("the private key travels")
}
}
// Run again after the take — the mesh not yet told, or told and refused — the previous key it
// names is still the one before the take, so the mesh can tell a repeat from a replay.
again, second, err := rekeyOnto(taken, found)
if err != nil {
t.Fatal(err)
}
if second.Previous != before || again.OverlayBefore != before || again.Overlay.Public != found.PublicKey {
t.Fatalf("a take run again does not name the key before the first: %+v", second)
}
}
+48
View File
@@ -0,0 +1,48 @@
package main
import (
"os"
"path/filepath"
"testing"
)
// A component's version comes from where it sits, not from its linker (novox/hq ADR 0142). The mesh's
// toolchain stamps no version — a build does not know what it will be called — so a delivered host
// read as "development build" and the mesh could not tell which host a machine ran (04-ISSUES/161).
func TestADeliveredHostReadsItsVersionFromItsPath(t *testing.T) {
// A delivered host lives at <libexec>/versions/<version>/<binary>.
dir := t.TempDir()
versioned := filepath.Join(dir, "versions", "637f65559d16")
if err := os.MkdirAll(versioned, 0o755); err != nil {
t.Fatal(err)
}
self := filepath.Join(versioned, "nox-mesh-host")
if err := os.WriteFile(self, []byte("#!/bin/sh\n"), 0o755); err != nil {
t.Fatal(err)
}
if got := versionAt(self, "development build"); got != "637f65559d16" {
t.Fatalf("a delivered host read its version as %q", got)
}
}
func TestAHostPlacedByHandKeepsItsStamp(t *testing.T) {
// The honest answer for one the mesh did not deliver — and every machine is in that state until
// a delivery reaches it.
if got := versionAt("/usr/bin/nox-mesh-host", "04a27ca"); got != "04a27ca" {
t.Fatalf("a hand-placed host read its version as %q", got)
}
}
func TestADirectoryThatIsNotAVersionIsNotReadAsOne(t *testing.T) {
// A binary sitting anywhere else must not have its parent directory's name read as a version.
for _, path := range []string{
"/opt/somewhere/nox-mesh-host",
"/usr/lib/nox-mesh-host/launch",
"/home/someone/build/nox-mesh-host",
} {
if got := versionAt(path, "the stamp"); got != "the stamp" {
t.Fatalf("%s read its version as %q", path, got)
}
}
}
+200
View File
@@ -0,0 +1,200 @@
// foundation-first-node-nats.lock — what a machine must be before a mesh exists, on the bus being
// built (novox/hq ADR 0106, design 25).
//
// The same twelve steps as foundation-first-node.lock, with one difference that matters: **the mesh
// writes its own user list, and at genesis there is no mesh yet to write it.** So this carries the
// first one — the controller's own account, at a well-known bootstrap password, exactly as the store
// is reached at `postgres:bootstrap` and the old bus at `guest:guest`. It is rotated with those, and
// from the controller's first composition onward the file is the controller's to write.
//
// The accounts file is its own file beside the server's configuration, because the server's own
// settings belong to whoever raises it and the users belong to the mesh (design 25 §4). Both live in
// one directory, of necessity: an include path is resolved relative to the including file's own
// directory, so a server given an absolute one looks for it underneath that directory and refuses to
// start.
//
// No `verify` on the TLS block, deliberately — that setting makes the server demand a *client*
// certificate, and nothing in the mesh presents one: a host pins this server's exact certificate and
// authenticates with a password (ADR 0004, design 25 §4).
{
"declaration": 1,
"resources": [
{
"id": "container-runtime",
"type": "package",
"package": "docker"
},
{
"id": "container-runtime-running",
"type": "service",
"unit": "docker.service",
"state": "running",
"boot": "enabled"
},
// **A filter before anything listens** (novox/hq issue 054, ADR 0088). The store and the
// broker are adopted as modules later and so bind to every interface from the moment they
// start; the packet filter that governs who may reach them is a module too, installed a
// dozen steps later. Between the two, a control-node facing the network had its store and
// its bus open to anyone who could reach the machine. So the foundation carries a filter of
// its own — the same table the filter module will replace wholesale once it can derive one:
// drop by default, keep loopback, replies, ssh and the mesh's own ports (the bus a node
// enrols over, the registry a node pulls from), and let the container runtime's own
// networks through the forward chain so containers keep working. A published container port
// is forwarded, never input (issue 047), which is why the forward chain is where the store's
// and broker's ports are refused from outside — and a container on this machine dialling a
// port this machine publishes reaches it through the runtime's proxy, which IS input, which
// is why the bus and the registry are opened in both chains, exactly as the derived ruleset
// does.
{
"id": "base-filter-package",
"type": "package",
"package": "nftables"
},
{
"id": "base-filter",
"type": "file",
"path": "/etc/nftables.conf",
"mode": "0644",
"content": "#!/usr/sbin/nft -f\n# the foundation's own filter, until the mesh derives one (novox/hq issue 054)\ntable inet mesh {}\ndelete table inet mesh\n\ntable inet mesh {\n\tchain input {\n\t\ttype filter hook input priority filter; policy drop;\n\t\tct state established,related accept\n\t\tct state invalid drop\n\t\tiif lo accept\n\t\ticmp type echo-request accept\n\t\ticmpv6 type { echo-request, nd-neighbor-solicit, nd-neighbor-advert, nd-router-advert } accept\n\t\t# ssh, from anywhere — never closed\n\t\ttcp dport 22 accept\n\t\t# the mesh's own, from anywhere: the bus a node enrols over and a container on this machine reaches through the proxy, the registry a node pulls from\n\t\ttcp dport 5671 accept\n\t\ttcp dport 5000 accept\n\t}\n\tchain output {\n\t\ttype filter hook output priority filter; policy accept;\n\t}\n\tchain forward {\n\t\ttype filter hook forward priority filter; policy drop;\n\t\tct state established,related accept\n\t\tct state invalid drop\n\t\t# the container runtime's bridge networks, and the networks its compose files are given\n\t\tip saddr 172.16.0.0/12 accept\n\t\tip saddr 192.168.128.0/17 accept\n\t\t# the mesh's own: the bus a node enrols over, the registry a node pulls from\n\t\tct original proto-dst 5671 accept\n\t\tct original proto-dst 5000 accept\n\t}\n}\n"
},
{
"id": "base-filter-loaded",
"type": "service",
"unit": "nftables.service",
"state": "running",
"boot": "enabled",
"restart-on": ["base-filter"]
},
{
"id": "store",
"type": "container",
"name": "mesh-store",
"image": "192.0.2.250:5000/postgres@sha256:7abf537131b66ed5af448d90653abf1679b0c7e9a1f07efdd4c3108a401b259a",
"env": {
"POSTGRES_PASSWORD": "bootstrap",
"PGDATA": "/var/lib/postgresql/data/pgdata"
},
"ports": ["5432:5432"],
"volumes": ["mesh-store-data:/var/lib/postgresql/data"]
},
// Over TCP, not the socket. While the store initialises it runs a temporary server on the
// socket ONLY, then stops it and starts the real one — so a socket check passes, the action
// exits happy, and the verify a moment later lands in the gap and fails. The action and its
// verify must ask the same question, or the action can succeed into a state verify rejects.
{
"id": "store-ready",
"type": "action",
"in": "mesh-store",
"command": ["sh", "-c", "for i in $(seq 1 180); do pg_isready -h 127.0.0.1 -U postgres >/dev/null 2>&1 && exit 0; sleep 1; done; echo 'the store did not answer within 180s; its own last words follow'; pg_isready -h 127.0.0.1 -U postgres; tail -n 20 /var/lib/postgresql/data/log/*.log 2>/dev/null; exit 1"],
"verify": ["pg_isready", "-h", "127.0.0.1", "-U", "postgres"]
},
{
"id": "inventory-database",
"type": "action",
"in": "mesh-store",
"command": ["sh", "-c", "psql -U postgres -c 'CREATE DATABASE inventory'"],
"verify": ["sh", "-c", "psql -U postgres -lqt | cut -d'|' -f1 | grep -qw inventory"]
},
{
"id": "identity-database",
"type": "action",
"in": "mesh-store",
"command": ["sh", "-c", "psql -U postgres -c 'CREATE DATABASE identity'"],
"verify": ["sh", "-c", "psql -U postgres -lqt | cut -d'|' -f1 | grep -qw identity"]
},
// Each context owns its own database (novox/hq ADR 0008). A third one is a third database,
// created the same way and named the same way — which is the whole of adding a context to the
// bootstrap, and is why the count is not something the foundation has an opinion about.
{
"id": "licences-database",
"type": "action",
"in": "mesh-store",
"command": ["sh", "-c", "psql -U postgres -c 'CREATE DATABASE licences'"],
"verify": ["sh", "-c", "psql -U postgres -lqt | cut -d'|' -f1 | grep -qw licences"]
},
{
"id": "context-schemas",
"type": "action",
"command": ["docker", "run", "--rm", "--network", "container:mesh-store",
"-e", "MESH_STORE_INVENTORY=postgres://postgres:bootstrap@127.0.0.1:5432/inventory?sslmode=disable",
"-e", "MESH_STORE_IDENTITY=postgres://postgres:bootstrap@127.0.0.1:5432/identity?sslmode=disable",
"-e", "MESH_STORE_LICENCES=postgres://postgres:bootstrap@127.0.0.1:5432/licences?sslmode=disable",
"192.0.2.250:5000/mesh-controller@sha256:c67db38439ff0aee242b467486765467bb95801f52175fc5727cc4e437338ace",
"migrate"],
"verify": ["sh", "-c", "docker exec mesh-store psql -U postgres -d inventory -tAc \"select to_regclass('public.node')\" | grep -qx node && docker exec mesh-store psql -U postgres -d identity -tAc \"select to_regclass('public.signing_key')\" | grep -qx signing_key && docker exec mesh-store psql -U postgres -d licences -tAc \"select to_regclass('public.licence')\" | grep -qx licence"]
},
{
"id": "bus-certificate",
"type": "action",
// **The mesh makes its own** (novox/hq 04-ISSUES/146). This ran `openssl` inside the
// broker's image while the broker was one that carried it; the bus that replaced it has a
// shell and no openssl, and no other image the bundle names has one either. So the program
// that needs the certificate writes it — already on this machine, since the schema step ran
// it, and asking nothing of the image it writes into. Self-signed on purpose: a host pins
// this server's exact certificate (novox/hq ADR 0004), and at this moment there is no mesh
// to ask an authority of.
// `--user 0:0` because the volume is root's and this image runs as nobody, which is right
// for the long-running control plane and wrong for a one-shot writing into a fresh volume.
"command": ["docker", "run", "--rm", "--user", "0:0", "-v", "mesh-broker-tls:/tls",
"192.0.2.250:5000/mesh-controller@sha256:c67db38439ff0aee242b467486765467bb95801f52175fc5727cc4e437338ace",
"broker", "certificate", "--into", "/tls"],
"verify": ["docker", "run", "--rm", "--user", "0:0", "-v", "mesh-broker-tls:/tls",
"192.0.2.250:5000/mesh-controller@sha256:c67db38439ff0aee242b467486765467bb95801f52175fc5727cc4e437338ace",
"broker", "certificate", "--check", "--into", "/tls"]
},
{
"id": "bus-conf-dir",
"type": "directory",
"path": "/var/lib/mesh-bus-conf",
"mode": "0700"
},
{
"id": "bus-conf",
"type": "file",
"path": "/var/lib/mesh-bus-conf/nats.conf",
"mode": "0644",
"content": "port: 4222\nhttp: 127.0.0.1:8222\n\ntls {\n cert_file: \"/tls/tls.crt\"\n key_file: \"/tls/tls.key\"\n}\n\njetstream {\n store_dir: \"/data\"\n}\n\ninclude accounts.conf\n"
},
{
"id": "bus-accounts",
"type": "file",
"path": "/var/lib/mesh-bus-conf/accounts.conf",
"mode": "0600",
"content": "// The first user list, carried by the installer because at genesis there is no mesh to\n// compose one. A bootstrap credential, rotated with the store's and replaced by the\n// controller's own composition from its first start onward.\naccounts {\n MESH {\n jetstream: enabled\n users = [\n { user: \"controller\", password: \"$2a$10$AHqJgOifIVbU41KmATiMhuXFs8xa7Wl2HuN4UVBCXdN2jIQzjqApy\", permissions: {\n publish: { allow: [\"$JS.ACK.CONTROL.controller.>\", \"$JS.ACK.EVENTS.controller.>\", \"$JS.API.>\", \"_INBOX.enrol.>\", \"mesh.assignment.>\", \"mesh.control.>\", \"mesh.mod.*.tool.>\", \"mesh.node.>\", \"mesh.seat.mesh-build-machine.accept.>\", \"mesh.seat.mesh-controller.event.applied\", \"mesh.seat.mesh-controller.event.built-before\", \"mesh.seat.mesh-controller.event.refused\"] }\n subscribe: { allow: [\"$JS.API.>\", \"_DELIVER.controller\", \"_DELIVER.controller.>\", \"_INBOX.controller.>\", \"mesh.control.>\", \"mesh.mod.gitea.event.pull.merged\", \"mesh.mod.mesh-catalog.event.catching-up\", \"mesh.mod.mesh-catalog.event.upgraded\", \"mesh.seat.mesh-build-machine.event.built\", \"mesh.seat.mesh-controller.tool.>\"] }\n allow_responses: { max: 1, ttl: \"1m\" }\n } }\n ]\n }\n}\n"
},
{
"id": "broker",
"type": "container",
"name": "mesh-broker",
"image": "192.0.2.250:5000/nats@sha256:b83efabe3e7def1e0a4a31ec6e078999bb17c80363f881df35edc70fcb6bb927",
"ports": ["5671:4222", "127.0.0.1:8222:8222"],
"volumes": ["mesh-broker-data:/data", "mesh-broker-tls:/tls:ro", "/var/lib/mesh-bus-conf:/etc/nats:ro"],
"args": ["-c", "/etc/nats/nats.conf", "-js"]
},
{
"id": "broker-ready",
"type": "action",
"command": ["sh", "-c", "for i in $(seq 1 60); do docker run --rm --network host --entrypoint sh 192.0.2.250:5000/nats@sha256:b83efabe3e7def1e0a4a31ec6e078999bb17c80363f881df35edc70fcb6bb927 -c 'nc -z 127.0.0.1 5671' >/dev/null 2>&1 && exit 0; sleep 1; done; exit 1"],
"verify": ["sh", "-c", "docker run --rm --network host --entrypoint sh 192.0.2.250:5000/nats@sha256:b83efabe3e7def1e0a4a31ec6e078999bb17c80363f881df35edc70fcb6bb927 -c 'nc -z 127.0.0.1 5671'"]
},
{
"id": "control-plane",
"type": "container",
"name": "mesh-controller",
"image": "192.0.2.250:5000/mesh-controller@sha256:c67db38439ff0aee242b467486765467bb95801f52175fc5727cc4e437338ace",
"network": "host",
"args": ["serve"],
"volumes": ["mesh-broker-tls:/broker-tls:ro"],
"env": {
"MESH_STORE_INVENTORY": "postgres://postgres:bootstrap@127.0.0.1:5432/inventory?sslmode=disable",
"MESH_STORE_IDENTITY": "postgres://postgres:bootstrap@127.0.0.1:5432/identity?sslmode=disable",
"MESH_STORE_LICENCES": "postgres://postgres:bootstrap@127.0.0.1:5432/licences?sslmode=disable",
"MESH_BUS_NATS": "nats://controller:bootstrap@127.0.0.1:5671",
"MESH_BROKER_ADDRESS": "192.0.2.10:5671",
"MESH_BROKER_CERTIFICATE": "/broker-tls/tls.crt"
}
}
]
}
+34
View File
@@ -46,6 +46,40 @@
"state": "running",
"boot": "enabled"
},
// **A filter before anything listens** (novox/hq issue 054, ADR 0088). The store and the
// broker are adopted as modules later and so bind to every interface from the moment they
// start; the packet filter that governs who may reach them is a module too, installed a
// dozen steps later. Between the two, a control-node facing the network had its store and
// its bus open to anyone who could reach the machine. So the foundation carries a filter of
// its own — the same table the filter module will replace wholesale once it can derive one:
// drop by default, keep loopback, replies, ssh and the mesh's own ports (the bus a node
// enrols over, the registry a node pulls from), and let the container runtime's own
// networks through the forward chain so containers keep working. A published container port
// is forwarded, never input (issue 047), which is why the forward chain is where the store's
// and broker's ports are refused from outside — and a container on this machine dialling a
// port this machine publishes reaches it through the runtime's proxy, which IS input, which
// is why the bus and the registry are opened in both chains, exactly as the derived ruleset
// does.
{
"id": "base-filter-package",
"type": "package",
"package": "nftables"
},
{
"id": "base-filter",
"type": "file",
"path": "/etc/nftables.conf",
"mode": "0644",
"content": "#!/usr/sbin/nft -f\n# the foundation's own filter, until the mesh derives one (novox/hq issue 054)\ntable inet mesh {}\ndelete table inet mesh\n\ntable inet mesh {\n\tchain input {\n\t\ttype filter hook input priority filter; policy drop;\n\t\tct state established,related accept\n\t\tct state invalid drop\n\t\tiif lo accept\n\t\ticmp type echo-request accept\n\t\ticmpv6 type { echo-request, nd-neighbor-solicit, nd-neighbor-advert, nd-router-advert } accept\n\t\t# ssh, from anywhere — never closed\n\t\ttcp dport 22 accept\n\t\t# the mesh's own, from anywhere: the bus a node enrols over and a container on this machine reaches through the proxy, the registry a node pulls from\n\t\ttcp dport 5671 accept\n\t\ttcp dport 5000 accept\n\t}\n\tchain output {\n\t\ttype filter hook output priority filter; policy accept;\n\t}\n\tchain forward {\n\t\ttype filter hook forward priority filter; policy drop;\n\t\tct state established,related accept\n\t\tct state invalid drop\n\t\t# the container runtime's bridge networks, and the networks its compose files are given\n\t\tip saddr 172.16.0.0/12 accept\n\t\tip saddr 192.168.128.0/17 accept\n\t\t# the mesh's own: the bus a node enrols over, the registry a node pulls from\n\t\tct original proto-dst 5671 accept\n\t\tct original proto-dst 5000 accept\n\t}\n}\n"
},
{
"id": "base-filter-loaded",
"type": "service",
"unit": "nftables.service",
"state": "running",
"boot": "enabled",
"restart-on": ["base-filter"]
},
{
"id": "store",
"type": "container",
+10 -4
View File
@@ -1,9 +1,15 @@
module github.com/novox/mesh-host
go 1.25.0
go 1.26.0
require (
github.com/rabbitmq/amqp091-go v1.14.0 // indirect
golang.org/x/crypto v0.55.0 // indirect
golang.org/x/sys v0.47.0 // indirect
github.com/nats-io/nats.go v1.54.0
golang.org/x/crypto v0.57.0
)
require (
github.com/klauspost/compress v1.20.0 // indirect
github.com/nats-io/nkeys v0.4.16 // indirect
github.com/nats-io/nuid v1.0.1 // indirect
golang.org/x/sys v0.48.0 // indirect
)
+12 -6
View File
@@ -1,6 +1,12 @@
github.com/rabbitmq/amqp091-go v1.14.0 h1:RSaT7aOKt/OrkVUyswPDW29lnRz9psuGmfZFBmLqLek=
github.com/rabbitmq/amqp091-go v1.14.0/go.mod h1:Hy4jKW5kQART1u+JkDTF9YYOQUHXqMuhrgxOEeS7G4o=
golang.org/x/crypto v0.55.0 h1:+KWHjbgOaAQ66dh/YlkZKHlz9ZUlq61AFirAR9ntP8M=
golang.org/x/crypto v0.55.0/go.mod h1:uq0V9dE/fzQuJtbnL+2EhWOE63vo164FY8xqEnV9xis=
golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs=
golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
github.com/klauspost/compress v1.20.0 h1:a3C1ke2ohxFymNlb2HWAHjDeKCI90scRskErZkR0ezA=
github.com/klauspost/compress v1.20.0/go.mod h1:LUdAzn7YLVvxLpc7y3V1m40wESHTgc1422pwwBSKYuI=
github.com/nats-io/nats.go v1.54.0 h1:vsXoOxjHp/GmPUN+EcI7uOf/uB+iAP+kEsAFNQN0yzA=
github.com/nats-io/nats.go v1.54.0/go.mod h1:y+DZoD1oBOYfZTU681eTUiUjI0vbqYGixNVFHcjHJ0k=
github.com/nats-io/nkeys v0.4.16 h1:rd5oAuLOb8mnAycB0xleuEBNS1pVVnN0fv/FF34Eypg=
github.com/nats-io/nkeys v0.4.16/go.mod h1:llLgWoI0o4z/Q57q2R1kHfmocyhGV6VG/U18Glg1Afs=
github.com/nats-io/nuid v1.0.1 h1:5iA8DT8V7q8WK2EScv2padNa/rTESc1KdnPw4TC2paw=
github.com/nats-io/nuid v1.0.1/go.mod h1:19wcPz3Ph3q0Jbyiqsd0kePYG7A95tJPxeL+1OSON2c=
golang.org/x/crypto v0.57.0 h1:3ZVCjf8Ggz7zneR/EHRVx68Ctf+2pmIMP2UFhh9cC6M=
golang.org/x/crypto v0.57.0/go.mod h1:Fdz0i5U6CoizGwLda9DttjSk6qlZo25zYNtR+ycvuZA=
golang.org/x/sys v0.48.0 h1:bbX/i/6MgT9BVLM9RT1thmxL04yeTAhbEz4SyadbXoo=
golang.org/x/sys v0.48.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
+1086 -47
View File
File diff suppressed because it is too large Load Diff
+159 -32
View File
@@ -348,33 +348,111 @@ func TestAnUnknownServiceStateIsRefusedNotGuessed(t *testing.T) {
}
}
func TestADroppedServiceIsStoppedNotDeleted(t *testing.T) {
// The host did not install the unit and does not own the unit file — only the state it put
// the unit into.
var commands []string
func TestADroppedServiceIsGivenBackTheStateItWasFoundIn(t *testing.T) {
// novox/hq ADR 0118: undeclaring removes what the mesh made, gives back what it changed, and
// leaves what was the machine's. A service resource never installs a unit — so undeclaring it
// undoes what the mesh did to the unit, and nothing more. Stopping every undeclared unit is
// how unassigning the private network stopped the container runtime (novox/hq issue 130).
cases := []struct {
name string
found *store.FoundUnit
action string
stop bool
disable bool
}{
{"recorded before the host kept what it found", nil, "forgotten", false, false},
{"running before the mesh", &store.FoundUnit{State: "running"}, "forgotten", false, false},
{"running and enabled before the mesh", &store.FoundUnit{State: "running", Boot: "enabled"}, "forgotten", false, false},
{"started by the mesh", &store.FoundUnit{State: "stopped"}, "restored", true, false},
{"started and enabled by the mesh", &store.FoundUnit{State: "stopped", Boot: "disabled"}, "restored", true, true},
{"enabled by the mesh, running before it", &store.FoundUnit{State: "running", Boot: "disabled"}, "restored", false, true},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
var commands []string
run := func(ctx context.Context, name string, args ...string) (string, error) {
commands = append(commands, strings.Join(args, " "))
if args[0] == "show" {
return "LoadState=loaded\nActiveState=active\n", nil
}
return "", nil
}
state := store.State{Resources: []store.Applied{
{ID: "s", Type: "service", Target: "unit.service", Found: c.found},
}}
d := parse(t, `{"declaration":1,"resources":[
{"id":"other","type":"file","path":"`+filepath.Join(t.TempDir(), "a")+`","content":"a\n"}
]}`)
report, after, err := Apply(context.Background(), archHost(t), d, state, store.OriginCarried, run, nil, nil)
if err != nil {
t.Fatal(err)
}
joined := strings.Join(commands, "; ")
if stopped := strings.Contains(joined, "stop unit.service"); stopped != c.stop {
t.Errorf("stopped %v, want %v: %s", stopped, c.stop, joined)
}
if disabled := strings.Contains(joined, "disable unit.service"); disabled != c.disable {
t.Errorf("disabled %v, want %v: %s", disabled, c.disable, joined)
}
if strings.Contains(joined, "start unit.service") || strings.Contains(joined, "mask") {
t.Errorf("the host started or masked a unit on its way out: %s", joined)
}
if o := outcomeOf(report, "s"); o.Action != c.action {
t.Errorf("outcome %+v, want %s", o, c.action)
}
if _, still := after.Find("s"); still {
t.Error("the host still believes it owns the undeclared service")
}
})
}
}
func TestAServiceRecordsTheStateItWasFoundInOnceAndCarriesIt(t *testing.T) {
// Read the first time the host applies the unit — before it starts or enables anything —
// and never again: by the next apply, the unit's state is the mesh's doing.
active := "inactive"
enabled := "disabled"
run := func(ctx context.Context, name string, args ...string) (string, error) {
commands = append(commands, strings.Join(args, " "))
if args[0] == "show" {
return "LoadState=loaded\nActiveState=active\n", nil
switch args[0] {
case "show":
return "LoadState=loaded\nActiveState=" + active + "\n", nil
case "is-enabled":
return enabled, nil
case "start":
active = "active"
case "enable":
enabled = "enabled"
}
return "", nil
}
state := store.State{Resources: []store.Applied{
{ID: "s", Type: "service", Target: "gone.service"},
}}
d := parse(t, `{"declaration":1,"resources":[
{"id":"other","type":"file","path":"`+filepath.Join(t.TempDir(), "a")+`","content":"a\n"}
{"id":"s","type":"service","unit":"filter.service","state":"running","boot":"enabled"}
]}`)
if _, _, err := Apply(context.Background(), archHost(t), d, state, store.OriginCarried, run, nil, nil); err != nil {
_, first, err := Apply(context.Background(), archHost(t), d, store.State{}, store.OriginCarried, run, nil, nil)
if err != nil {
t.Fatal(err)
}
joined := strings.Join(commands, "; ")
if !strings.Contains(joined, "stop gone.service") {
t.Errorf("the dropped service was not stopped: %s", joined)
rec, _ := first.Find("s")
if rec.Found == nil || rec.Found.State != "stopped" || rec.Found.Boot != "disabled" {
t.Fatalf("first apply recorded %+v, want stopped and disabled", rec.Found)
}
if strings.Contains(joined, "disable") || strings.Contains(joined, "mask") {
t.Errorf("the host did more than stop a unit it does not own: %s", joined)
_, second, err := Apply(context.Background(), archHost(t), d, first, store.OriginCarried, run, nil, nil)
if err != nil {
t.Fatal(err)
}
rec, _ = second.Find("s")
if rec.Found == nil || rec.Found.State != "stopped" {
t.Errorf("a later apply replaced what was found with what the mesh made: %+v", rec.Found)
}
// A record from before the host kept what it found is not given one later.
old := store.State{Resources: []store.Applied{{ID: "s", Type: "service", Target: "filter.service"}}}
_, third, err := Apply(context.Background(), archHost(t), d, old, store.OriginCarried, run, nil, nil)
if err != nil {
t.Fatal(err)
}
if rec, _ = third.Find("s"); rec.Found != nil {
t.Errorf("a record that predates the field was given one after the mesh had acted: %+v", rec.Found)
}
}
@@ -635,7 +713,7 @@ func TestAContainerThatExitsImmediatelyFailsTheApply(t *testing.T) {
switch {
case args[0] == "info":
return "27.0\n", nil
case args[0] == "inspect":
case args[0] == "container":
return "false\t" + "", nil // exists, not running
case args[0] == "run":
return "deadbeef\n", nil
@@ -666,14 +744,14 @@ func TestAContainerWhoseDeclarationChangedIsReplaced(t *testing.T) {
d := parseTrusted(t, `{"declaration":1,"resources":[
{"id":"store","type":"container","name":"store","image":"`+pinned+`","env":{"PGDATA":"/data"}}
]}`)
want := containerSpec(d.Resources[0].(*declaration.Container), nil)
want := containerSpec(d.Resources[0].(*declaration.Container), inputs{})
var removed, created bool
run := func(ctx context.Context, name string, args ...string) (string, error) {
switch args[0] {
case "info":
return "27.0\n", nil
case "inspect":
case "container":
if created {
return "true\t" + want, nil
}
@@ -704,14 +782,14 @@ func TestAContainerThatMatchesIsLeftAlone(t *testing.T) {
d := parseTrusted(t, `{"declaration":1,"resources":[
{"id":"store","type":"container","name":"store","image":"`+pinned+`","env":{"PGDATA":"/data"}}
]}`)
spec := containerSpec(d.Resources[0].(*declaration.Container), nil)
spec := containerSpec(d.Resources[0].(*declaration.Container), inputs{})
var touched bool
run := func(ctx context.Context, name string, args ...string) (string, error) {
switch args[0] {
case "info":
return "27.0\n", nil
case "inspect":
case "container":
return "true\t" + spec, nil
}
touched = true
@@ -748,15 +826,15 @@ func TestAContainerIsRecreatedWhenARestartOnResourceChanged(t *testing.T) {
// what a container reads is part of what it is, so the old content yields a different spec.
was := map[string]string{"config": declaredDigest(&declaration.File{Content: "{\"token\":\"old\"}\n"})}
now := map[string]string{"config": declaredDigest(d.Resources[0].(*declaration.File))}
stale := containerSpec(d.Resources[1].(*declaration.Container), was)
fresh := containerSpec(d.Resources[1].(*declaration.Container), now)
stale := containerSpec(d.Resources[1].(*declaration.Container), inputs{declares: was})
fresh := containerSpec(d.Resources[1].(*declaration.Container), inputs{declares: now})
var removed, created bool
run := func(ctx context.Context, name string, args ...string) (string, error) {
switch args[0] {
case "info":
return "27.0\n", nil
case "inspect":
case "container":
// Already there and running, created against the file as it was. After the host
// recreates it, the runtime holds the one it just made — as a real one would.
if created {
@@ -1000,7 +1078,7 @@ func TestAContainerUsesTheRuntimeTheMachineHas(t *testing.T) {
switch args[0] {
case "info":
return "6.1.0\n", nil
case "inspect":
case "container":
return "false\t\n", errors.New("no such container")
case "run":
return "deadbeef\n", nil
@@ -1114,6 +1192,18 @@ func TestAServiceIsRestartedWhenWhatItReflectsChanges(t *testing.T) {
if !stopped || !started {
t.Errorf("the file changed and the service was not restarted; commands were %v", commands)
}
reloaded, stop := -1, -1
for i, c := range commands {
if strings.Contains(c, "daemon-reload") && reloaded < 0 {
reloaded = i
}
if strings.Contains(c, "stop thing.service") && stop < 0 {
stop = i
}
}
if reloaded < 0 || reloaded > stop {
t.Errorf("the service was restarted without the unit files being read again first; commands were %v", commands)
}
}
}
@@ -1314,7 +1404,7 @@ func TestAContainerIsGivenTheMeshsNames(t *testing.T) {
switch args[0] {
case "info":
return "29.0.0\n", nil
case "inspect":
case "container":
return "false\t\n", errors.New("no such container")
case "run":
ran = args
@@ -1349,7 +1439,7 @@ func TestAContainerGivenNoNamesIsRunAsBefore(t *testing.T) {
switch args[0] {
case "info":
return "29.0.0\n", nil
case "inspect":
case "container":
return "false\t\n", errors.New("no such container")
case "run":
ran = args
@@ -1526,15 +1616,15 @@ func TestAContainerStaleFromAnEarlierApplyIsReplaced(t *testing.T) {
// The container was created when the file said something else.
was := map[string]string{"env": declaredDigest(&declaration.File{Content: "PASSWORD=old\n"})}
now := map[string]string{"env": declaredDigest(d.Resources[0].(*declaration.File))}
stale := containerSpec(d.Resources[1].(*declaration.Container), was)
fresh := containerSpec(d.Resources[1].(*declaration.Container), now)
stale := containerSpec(d.Resources[1].(*declaration.Container), inputs{declares: was})
fresh := containerSpec(d.Resources[1].(*declaration.Container), inputs{declares: now})
var removed, created bool
run := func(ctx context.Context, name string, args ...string) (string, error) {
switch args[0] {
case "info":
return "27.0\n", nil
case "inspect":
case "container":
if created {
return "true\t" + fresh, nil
}
@@ -1558,3 +1648,40 @@ func TestAContainerStaleFromAnEarlierApplyIsReplaced(t *testing.T) {
"(removed=%v created=%v)", removed, created)
}
}
// A seed is written once. What grows in it afterwards is somebody else's work the mesh asked for,
// and a reconcile leaves it alone — content, mode and owner — and says so (novox/hq issue 035).
func TestASeedIsCreatedOnceAndWhatGrowsInItIsKept(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "acl.conf")
d := parse(t, fmt.Sprintf(`{"declaration":1,"resources":[
{"id":"acl","type":"file","path":%q,"content":"user default on\n","mode":"0600","create-once":true}
]}`, path))
report, state, err := Apply(context.Background(), archHost(t), d, store.State{},
store.OriginCarried, noServices, nil, nil)
if err != nil {
t.Fatal(err)
}
if got := report.Outcomes[0].Action; got != "created" {
t.Fatalf("first apply: %q", got)
}
// The program persists into it.
if err := os.WriteFile(path, []byte("user default on\nuser app-one on >secret\n"), 0o600); err != nil {
t.Fatal(err)
}
report, _, err = Apply(context.Background(), archHost(t), d, state,
store.OriginCarried, noServices, nil, nil)
if err != nil {
t.Fatal(err)
}
if got := report.Outcomes[0].Action; got != "kept" {
t.Fatalf("second apply: %q — a seed was reconciled", got)
}
grown, _ := os.ReadFile(path)
if string(grown) != "user default on\nuser app-one on >secret\n" {
t.Fatalf("what grew in the seed was wiped: %q", grown)
}
}
+433
View File
@@ -0,0 +1,433 @@
package apply
import (
"errors"
"fmt"
"net"
"os"
"path/filepath"
"strings"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/store"
)
// A file written into a marked block, never over (novox/hq issue 128, ADR 0102).
//
// **The file is the machine's; the mesh owns lines in it.** The machine's hosts file is the case
// that needed it. The mesh wrote it whole — its own header, localhost, the machine's name and every
// name in the mesh — and on a workstation that file is shared: the distribution's lines, a local
// development tool's own marked blocks rewritten whenever its projects change, the operator's
// hand-added names. Written whole, all of those went at the next change to the mesh's names, with
// no failure anywhere: the tool believed it had written its block, and the mesh believed it owned
// the file. It is ADR 0102's failure exactly, in a file ADR 0102's JSON verb cannot speak.
//
// So the host finds the lines between `# BEGIN mesh <id>` and `# END mesh <id>`, rewrites those and
// nothing else, and records what they held before. Every line outside the markers is kept byte for
// byte — including another tool's `# BEGIN …` blocks, which are that tool's. Undeclared, the region
// is given back what it held, or taken out with its markers when it held nothing, and a file the
// mesh created goes only if nothing but whitespace is left.
// applyBlock writes a file's declared lines into its region of the file already at its path.
//
// **A link stays a link.** Where the path is a symbolic link — a hosts file some distributions keep
// elsewhere and link into /etc — the file read, written and renamed over is the one it points to,
// so the link and whatever manages it are left as they were. A file written whole, or into JSON,
// still replaces a link with a file; that is unchanged here.
func applyBlock(r *declaration.File, previous store.Applied) (Outcome, error) {
out := begin(r)
opening, closing := declaration.BlockMarkers(r.ID)
want := blockBody(r.Content)
real, err := realPath(r.Path)
if err != nil {
return out, err
}
raw, err := os.ReadFile(real)
existed := err == nil
if err != nil && !errors.Is(err, os.ErrNotExist) {
return out, err
}
// What the file is, taken once with what it holds: its mode and owner are the machine's and
// go back onto what is written. A file read and then not there to stat is a failure, never a
// file with no owner.
var info os.FileInfo
if existed {
if info, err = os.Stat(real); err != nil {
return out, fmt.Errorf("read %s and cannot see it: %w", r.Path, err)
}
}
existing := string(raw)
rec := store.Into{Format: declaration.IntoBlock}
var note string
rebuilt := false
// **A file the mesh once wrote whole** (novox/hq issue 128). The resource keeps its id when its
// module moves from writing the file whole to writing into it, and the file on the machine is
// then the mesh's own old write — its header, its loopback lines, its names. Adding the region
// after that would leave the old names above the new ones, and a resolver takes the first
// line that answers: the region would be shadowed by what it replaced. So the file is rebuilt:
// the original the mesh kept before its first write, with the region in it; or, where the mesh
// made the file itself, the loopback lines every machine needs, kept as the machine's, with the
// region beside them. Changed since the mesh wrote it, the file is somebody's again and is
// written into as it stands, and the outcome says so.
if existed && previous.Into == nil && previous.Wrote != "" {
if digestOf(existing) == previous.Wrote {
if previous.Kept != "" {
original, err := os.ReadFile(previous.Kept)
if err != nil {
return out, fmt.Errorf("%s was written whole by the mesh over an original kept at %s, "+
"which cannot be read to give it back: %w; it was left as it is", r.Path, previous.Kept, err)
}
existing = string(original)
note = "the mesh's old whole file replaced by the original kept at " + previous.Kept + ", with the region in it"
} else {
existing = loopbackOf(existing)
rec.Created = true
note = "the mesh's old whole file replaced by its loopback lines and the region"
}
// Not what was read: the whole of it was the mesh's, and the file is written afresh.
rebuilt = true
} else {
note = "a file the mesh once wrote whole, changed since; its old lines were kept"
}
}
lines := linesOf(existing)
at, found, err := regionIn(lines, opening, closing)
if err != nil {
// Refused, never guessed at: markers the host cannot pair are markers it cannot write
// between without risking lines that are not the mesh's.
return out, fmt.Errorf("%s: %w; it was left as it is", r.Path, err)
}
// A record of a block is carried; anything else — no record, a file once written whole, one
// once written into as JSON — is a file the host is seeing for the first time as a block.
recorded := previous.Into != nil && previous.Into.Format == declaration.IntoBlock
if recorded && existed {
rec.Created = previous.Into.Created
rec.Region = previous.Into.Region
rec.Separated = previous.Into.Separated
rec.At = previous.Into.At
rec.Ended = previous.Into.Ended
} else if !rebuilt {
// A file gone since the last apply is made again, and made by the mesh: what it held
// before went with it, so there is nothing to give back but the file's absence.
rec.Created = !existed
if found {
// **What the host may have written itself is not the machine's** — the same reasoning
// as a key in a JSON file (novox/hq ADR 0102). With no record, a region already holding
// exactly the declared lines cannot be told from one this host wrote a moment ago and
// died before saving; remembered as the machine's, it would be put back on undeclare
// for ever. So it is the mesh's, and undeclaring takes it out.
if held := at.body(lines); held != want {
rec.Region = &held
}
}
}
// Drift: the machine no longer holds, between the mesh's markers, what this host last put
// there. Judged only against a record of a block: a digest of a whole file says nothing about
// a region of it.
drifted := recorded && previous.Wrote != "" && existed &&
(!found || digestOf(at.body(lines)) != previous.Wrote)
var next string
switch {
case !existed:
next = regionOf(opening, closing, want)
case found:
// Where it is, whatever At says: the region is never moved, because moving it moves the
// machine's lines around it.
next = strings.Join(lines[:at.begin+1], "") + want + strings.Join(lines[at.end:], "")
case r.At == declaration.AtStart:
// Above everything, and one blank line between the region and the machine's first line
// unless there is one already — a line in some files means what the lines above it say.
rec.At, rec.Separated, rec.Ended = declaration.AtStart, false, false
next = regionOf(opening, closing, want)
if existing != "" && !strings.HasPrefix(existing, "\n") {
next += "\n"
rec.Separated = true
}
next += existing
default:
// At the end, apart from whatever is there: the file's last line is ended if it was not,
// and one blank line separates the region from the machine's lines unless there is one.
rec.At, rec.Separated, rec.Ended = "", false, false
next = existing
if next != "" && !strings.HasSuffix(next, "\n") {
next += "\n"
rec.Ended = true
}
if next != "" && next != "\n" && !strings.HasSuffix(next, "\n\n") {
next += "\n"
rec.Separated = true
}
next += regionOf(opening, closing, want)
}
// What was not the mesh's is what it was. By construction — and checked, because a slip in
// splicing lines is exactly the fault this mode exists to prevent, and it must never be written.
if found {
after := linesOf(next)
if where, ok, err := regionIn(after, opening, closing); err != nil || !ok || outside(after, where) != outside(lines, at) {
return out, fmt.Errorf("%s: writing the region would change lines outside it; it was left as it is", r.Path)
}
}
same := existed && next == string(raw)
if !same {
mode := os.FileMode(0o644)
if info != nil {
mode = info.Mode().Perm() // the machine's file keeps the machine's mode
} else if mode, err = modeOf(r.Mode, mode); err != nil {
return out, err
}
if err := os.MkdirAll(filepath.Dir(real), 0o755); err != nil {
return out, err
}
if err := writeAtomically(real, []byte(next), mode); err != nil {
return out, err
}
if info != nil {
// The write is a new file renamed over the old, so it belongs to whoever wrote it. The
// machine's file keeps the machine's owner, as it keeps its mode.
if err := keepOwner(real, info); err != nil {
return out, err
}
} else if err := own(real, r.Owner); err != nil {
return out, err
}
}
// Read back: the region holds what was declared. Only the region — another tool writing its
// own lines in the moment after the rename is not a failed write. What remains is the moment
// between reading the file and renaming over it: a line another tool writes there is lost, and
// found again at its next write. Nothing short of a lock every writer honours closes that, and
// the other writers of a hosts file honour none.
written, err := os.ReadFile(real)
if err != nil {
return out, fmt.Errorf("wrote into %s and cannot read it back: %w", r.Path, err)
}
back := linesOf(string(written))
if where, ok, err := regionIn(back, opening, closing); err != nil || !ok || where.body(back) != want {
return out, fmt.Errorf("%s does not hold the mesh's region after writing into it", r.Path)
}
out.into = &rec
out.wrote = digestOf(want)
switch {
case note != "" && !same:
out.Action = "updated"
out.Detail = note
case !existed:
out.Action = "created"
out.Detail = "written into; the file was not there"
case same:
out.Action = "unchanged"
case drifted:
out.Action = "corrected"
out.Detail = "the mesh's region had been changed on the machine; every line outside it was kept"
case !found:
out.Action = "updated"
where := "end"
if rec.At == declaration.AtStart {
where = "start"
}
out.Detail = "the mesh's region added at the " + where + "; every other line kept as it was"
default:
out.Action = "updated"
out.Detail = "the mesh's region rewritten; every line outside it kept as it was"
}
return out, nil
}
// removeBlock gives back what a file written into a block held before the mesh's region.
func removeBlock(a store.Applied) (string, string, error) {
real, err := realPath(a.Target)
if err != nil {
return "", "", err
}
raw, err := os.ReadFile(real)
if errors.Is(err, os.ErrNotExist) {
return "forgotten", "no longer there", nil
}
if err != nil {
return "", "", err
}
info, err := os.Stat(real)
if err != nil {
return "", "", fmt.Errorf("read %s and cannot see it: %w", a.Target, err)
}
opening, closing := declaration.BlockMarkers(a.ID)
lines := linesOf(string(raw))
at, found, err := regionIn(lines, opening, closing)
if err != nil {
return "kept", err.Error() + ", so nothing was taken out of it; remove the mesh's region by hand", nil
}
next, action, detail := string(raw), "forgotten", "the mesh's region was no longer in it"
switch {
case found && a.Into.Region != nil:
next = strings.Join(lines[:at.begin+1], "") + *a.Into.Region + strings.Join(lines[at.end:], "")
action, detail = "restored", "no longer declared; the region was given back what it held"
case found:
from, to := at.begin, at.end+1
// The blank line the host added beside the region, when a blank line still stands there.
// Whether it is the same one the host added cannot be known from the file; a blank line
// is the one line whose going changes nothing any program reads, so it is taken. A line
// that is not blank is never taken, whoever put it there.
if a.Into.Separated {
if a.Into.At == declaration.AtStart {
if to < len(lines) && lines[to] == "\n" {
to++
}
} else if from > 0 && lines[from-1] == "\n" {
from--
}
}
next = strings.Join(lines[:from], "") + strings.Join(lines[to:], "")
// And the line end the host gave the machine's last line, if that line is still last.
if a.Into.Ended && strings.Join(lines[to:], "") == "" {
next = strings.TrimSuffix(next, "\n")
}
action, detail = "restored", "no longer declared; the mesh's region was taken out and every other line kept"
}
if a.Into.Created && strings.TrimSpace(next) == "" && real == a.Target {
if err := os.Remove(real); err != nil {
return "", "", err
}
return "removed", "no longer declared; the mesh had created it and nothing else was in it", nil
}
if next == string(raw) {
return action, detail, nil
}
if err := writeAtomically(real, []byte(next), info.Mode().Perm()); err != nil {
return "", "", err
}
if err := keepOwner(real, info); err != nil {
return "", "", err
}
return action, detail, nil
}
// realPath is the file a path names, through any links; a path that is not there yet is itself.
// A link to nothing is refused: writing through it would replace the link with a file.
func realPath(path string) (string, error) {
real, err := filepath.EvalSymlinks(path)
if err == nil {
return real, nil
}
if _, lerr := os.Lstat(path); errors.Is(lerr, os.ErrNotExist) {
return path, nil
}
return "", fmt.Errorf("%s is a link the host cannot follow to a file: %w; it was left as it is", path, err)
}
// loopbackOf is the lines of a file that answer for the machine itself — localhost, its own name on
// 127.0.1.1, ::1 — and nothing else: what the mesh's old whole hosts file carried that the machine
// needs, without the mesh's header or its names.
func loopbackOf(text string) string {
var b strings.Builder
for _, line := range linesOf(text) {
fields := strings.Fields(line)
if len(fields) < 2 {
continue
}
if ip := net.ParseIP(fields[0]); ip != nil && ip.IsLoopback() {
b.WriteString(strings.TrimSuffix(line, "\n") + "\n")
}
}
return b.String()
}
// outside is every line of a file but the mesh's region, markers included, as one string.
func outside(lines []string, at region) string {
end := at.end + 1
if end > len(lines) {
end = len(lines)
}
return strings.Join(lines[:at.begin], "") + "\x00" + strings.Join(lines[end:], "")
}
// blockBody is the declared lines as they stand in the region: ending in exactly one line end, or
// nothing at all when there are no lines.
func blockBody(content string) string {
trimmed := strings.TrimRight(content, "\n")
if trimmed == "" {
return ""
}
return trimmed + "\n"
}
func regionOf(begin, end, body string) string {
return begin + "\n" + body + end + "\n"
}
// linesOf splits text into lines that keep their line ends, so joining them again gives back
// exactly the bytes that were read — a last line without one included.
func linesOf(text string) []string {
return strings.SplitAfter(text, "\n")
}
// region is where the mesh's markers stand, as indices into the lines of a file.
type region struct{ begin, end int }
// body is what stands between the markers.
func (r region) body(lines []string) string {
return strings.Join(lines[r.begin+1:r.end], "")
}
// regionIn finds the mesh's markers for one resource. A line is a marker only if it is exactly the
// marker, so another tool's block and another resource's region are never it. Markers that do not
// form one pair — a begin with no end, an end before its begin, either twice — are an error rather
// than a best guess, because a guess is how the host would rewrite lines that are not its own.
func regionIn(lines []string, begin, end string) (region, bool, error) {
at := region{begin: -1, end: -1}
for i, line := range lines {
switch strings.TrimSuffix(line, "\n") {
case begin:
if at.begin >= 0 {
return at, false, fmt.Errorf("%q is in it more than once", begin)
}
at.begin = i
case end:
if at.end >= 0 {
return at, false, fmt.Errorf("%q is in it more than once", end)
}
at.end = i
}
}
switch {
case at.begin < 0 && at.end < 0:
return at, false, nil
case at.begin < 0:
return at, false, fmt.Errorf("%q is in it with no %q before it", end, begin)
case at.end < 0:
return at, false, fmt.Errorf("%q is in it with no %q after it", begin, end)
case at.end < at.begin:
return at, false, fmt.Errorf("%q stands before %q", end, begin)
}
return at, true, nil
}
// keepOwner gives a file rewritten through a new one back to whoever owned what it replaced.
// Changed only where it differs, so a host that is not root can still write a file it owns.
func keepOwner(path string, was os.FileInfo) error {
uid, gid, ok := ownerOf(was)
if !ok {
return nil
}
now, err := os.Stat(path)
if err != nil {
return err
}
if u, g, ok := ownerOf(now); ok && u == uid && g == gid {
return nil
}
if err := os.Chown(path, uid, gid); err != nil {
return fmt.Errorf("cannot give %s back to its owner %d:%d: %w", path, uid, gid, err)
}
return nil
}
+618
View File
@@ -0,0 +1,618 @@
package apply
import (
"context"
"encoding/json"
"fmt"
"os"
"path/filepath"
"strings"
"testing"
"github.com/novox/mesh-host/internal/store"
)
// Defends novox/hq issue 128 and ADR 0102: a text file the mesh shares with software it did not
// install is written into a marked block, never over — every line outside the mesh's markers is
// the machine's and is kept byte for byte, and undeclaring gives the file back.
const namesID = "mesh-wireguard.fact-node-names"
func blockDecl(t *testing.T, path, content string, extra ...string) string {
t.Helper()
more := ""
for _, e := range extra {
more += "," + e
}
return fmt.Sprintf(`{"declaration":1,"resources":[
{"id":%q,"type":"file","path":%q,"into":"block","content":%q%s}
]}`, namesID, path, content, more)
}
func applyBlockDecl(t *testing.T, raw string, known store.State) (Report, store.State) {
t.Helper()
report, state, err := Apply(context.Background(), archHost(t), parse(t, raw), known, store.OriginDeclared, nil, nil, nil)
if err != nil {
t.Fatal(err)
}
return report, state
}
func undeclare(t *testing.T, known store.State) (Report, store.State) {
t.Helper()
report, state, err := Apply(context.Background(), archHost(t), somethingElse(t), known, store.OriginDeclared, nil, nil, nil)
if err != nil {
t.Fatal(err)
}
return report, state
}
func readText(t *testing.T, path string) string {
t.Helper()
raw, err := os.ReadFile(path)
if err != nil {
t.Fatal(err)
}
return string(raw)
}
func marked(id, body string) string {
return "# BEGIN mesh " + id + "\n" + body + "# END mesh " + id + "\n"
}
// A workstation's hosts file, the way issue 128 found it: the distribution's lines, a development
// tool's own marked blocks, and the operator's hand-added names.
const workstationHosts = "127.0.0.1\tlocalhost\n" +
"127.0.1.1\tg14.localdomain g14\n" +
"\n" +
"# BEGIN devtool project-a\n" +
"127.0.0.1 a.test api.a.test\n" +
"# END devtool project-a\n" +
"# BEGIN devtool project-b\n" +
"127.0.0.1 b.test\n" +
"# END devtool project-b\n" +
"192.168.1.20 printer # the operator's\n"
const meshNames = "10.42.0.1 ace\n10.42.0.2 novox\n"
func TestABlockKeepsEveryLineOutsideItsMarkers(t *testing.T) {
path := filepath.Join(t.TempDir(), "hosts")
if err := os.WriteFile(path, []byte(workstationHosts), 0o640); err != nil {
t.Fatal(err)
}
report, state := applyBlockDecl(t, blockDecl(t, path, meshNames), store.State{})
want := workstationHosts + "\n" + marked(namesID, meshNames)
if got := readText(t, path); got != want {
t.Fatalf("the file after writing into it:\n%q\nwant\n%q", got, want)
}
if got := report.Outcomes[0].Action; got != "updated" {
t.Errorf("adding the region was %q", got)
}
if info, _ := os.Stat(path); info.Mode().Perm() != 0o640 {
t.Errorf("the machine's file mode was changed to %o", info.Mode().Perm())
}
rec, _ := state.Find(namesID)
if rec.Into == nil || rec.Into.Format != "block" || rec.Into.Region != nil || rec.Into.Created {
t.Fatalf("recorded as %+v", rec.Into)
}
// Again, with nothing changed: nothing written.
report, state = applyBlockDecl(t, blockDecl(t, path, meshNames), state)
if got := report.Outcomes[0].Action; got != "unchanged" {
t.Errorf("a second apply was %q", got)
}
// The development tool rewrites its block, and the operator adds a line after the mesh's
// region; the mesh's names change. Only the region moves.
edited := strings.Replace(readText(t, path), "127.0.0.1 b.test\n", "127.0.0.1 b.test c.test\n", 1) +
"10.0.0.5 nas # added after\n"
_ = os.WriteFile(path, []byte(edited), 0o640)
changed := meshNames + "10.42.0.3 shanks\n"
report, state = applyBlockDecl(t, blockDecl(t, path, changed), state)
want = strings.Replace(edited, marked(namesID, meshNames), marked(namesID, changed), 1)
if got := readText(t, path); got != want {
t.Fatalf("rewriting the region moved something else:\n%q\nwant\n%q", got, want)
}
if got := report.Outcomes[0].Action; got != "updated" {
t.Errorf("rewriting the region was %q", got)
}
// Undeclared: the region, its markers and the blank line the host put before it go; every
// other line is where it was.
report, _ = undeclare(t, state)
want = strings.Replace(edited, "\n"+marked(namesID, meshNames), "", 1)
if got := readText(t, path); got != want {
t.Fatalf("undeclaring left:\n%q\nwant\n%q", got, want)
}
if got := report.Outcomes[0].Action; got != "restored" {
t.Errorf("undeclaring was %q", got)
}
}
func TestUndeclaringABlockAddedAtTheEndGivesTheFileBackExactly(t *testing.T) {
for name, original := range map[string]string{
"ending in a line": "127.0.0.1 localhost\n",
"ending in a blank line": "127.0.0.1 localhost\n\n",
"empty": "",
} {
t.Run(name, func(t *testing.T) {
path := filepath.Join(t.TempDir(), "hosts")
_ = os.WriteFile(path, []byte(original), 0o644)
_, state := applyBlockDecl(t, blockDecl(t, path, meshNames), store.State{})
undeclare(t, state)
if got := readText(t, path); got != original {
t.Errorf("undeclaring left %q, the machine had %q", got, original)
}
})
}
}
func TestABlockAddedAtTheEndIsSetApartFromTheMachinesLines(t *testing.T) {
for name, c := range map[string]struct{ before, after string }{
"no line end": {"127.0.0.1 localhost", "127.0.0.1 localhost\n\n" + marked(namesID, meshNames)},
"a line end": {"127.0.0.1 localhost\n", "127.0.0.1 localhost\n\n" + marked(namesID, meshNames)},
"a blank line already": {"127.0.0.1 localhost\n\n", "127.0.0.1 localhost\n\n" + marked(namesID, meshNames)},
"empty": {"", marked(namesID, meshNames)},
"only a blank line": {"\n", "\n" + marked(namesID, meshNames)},
"content without an end": {"x\n\n", "x\n\n" + marked(namesID, meshNames)},
} {
t.Run(name, func(t *testing.T) {
path := filepath.Join(t.TempDir(), "hosts")
_ = os.WriteFile(path, []byte(c.before), 0o644)
applyBlockDecl(t, blockDecl(t, path, meshNames), store.State{})
if got := readText(t, path); got != c.after {
t.Errorf("got %q, want %q", got, c.after)
}
})
}
}
func TestTheRegionEndsInExactlyOneLineEnd(t *testing.T) {
for content, body := range map[string]string{
"10.42.0.1 ace": "10.42.0.1 ace\n",
"10.42.0.1 ace\n": "10.42.0.1 ace\n",
"10.42.0.1 ace\n\n\n": "10.42.0.1 ace\n",
"": "",
"\n\n": "",
} {
path := filepath.Join(t.TempDir(), "hosts")
_, state := applyBlockDecl(t, blockDecl(t, path, content), store.State{})
if got := readText(t, path); got != marked(namesID, body) {
t.Errorf("content %q was written as %q", content, got)
}
// And the same content again is not a change.
report, _ := applyBlockDecl(t, blockDecl(t, path, content), state)
if got := report.Outcomes[0].Action; got != "unchanged" {
t.Errorf("content %q applied twice was %q", content, got)
}
}
}
func TestARegionAlreadyThereIsRewrittenInPlaceAndGivenBack(t *testing.T) {
path := filepath.Join(t.TempDir(), "hosts")
before := "127.0.0.1 localhost\n"
after := "# BEGIN devtool x\n127.0.0.1 x.test\n# END devtool x\n192.168.1.20 printer\n"
found := "10.42.0.9 old-name\n"
original := before + marked(namesID, found) + after
_ = os.WriteFile(path, []byte(original), 0o644)
report, state := applyBlockDecl(t, blockDecl(t, path, meshNames), store.State{})
if got := readText(t, path); got != before+marked(namesID, meshNames)+after {
t.Fatalf("the region was not rewritten in place: %q", got)
}
if got := report.Outcomes[0].Action; got != "updated" {
t.Errorf("rewriting a found region was %q", got)
}
rec, _ := state.Find(namesID)
if rec.Into.Region == nil || *rec.Into.Region != found {
t.Fatalf("what the region held before was recorded as %v", rec.Into.Region)
}
// Undeclared: what the region held goes back, where it was.
report, _ = undeclare(t, state)
if got := readText(t, path); got != original {
t.Errorf("undeclaring left %q, the machine had %q", got, original)
}
if got := report.Outcomes[0].Action; got != "restored" {
t.Errorf("undeclaring was %q", got)
}
}
func TestARegionWithNoRecordHoldingExactlyTheDeclaredLinesIsTheMeshs(t *testing.T) {
// A host that wrote the region and died before saving its state: what is between the markers
// is exactly what the mesh declares, and remembered as the machine's it would never go.
path := filepath.Join(t.TempDir(), "hosts")
original := "127.0.0.1 localhost\n"
_ = os.WriteFile(path, []byte(original+"\n"+marked(namesID, meshNames)), 0o644)
_, state := applyBlockDecl(t, blockDecl(t, path, meshNames), store.State{})
if rec, _ := state.Find(namesID); rec.Into.Region != nil {
t.Fatalf("the mesh's own lines were recorded as the machine's: %q", *rec.Into.Region)
}
undeclare(t, state)
if got := readText(t, path); !strings.HasPrefix(got, original) || strings.Contains(got, "BEGIN mesh") {
t.Errorf("undeclaring left the mesh's region behind: %q", got)
}
}
func TestADriftedRegionIsCorrected(t *testing.T) {
path := filepath.Join(t.TempDir(), "hosts")
_ = os.WriteFile(path, []byte(workstationHosts), 0o644)
_, state := applyBlockDecl(t, blockDecl(t, path, meshNames), store.State{})
written := readText(t, path)
_ = os.WriteFile(path, []byte(strings.Replace(written, "10.42.0.2 novox\n", "10.42.0.2 novox\n6.6.6.6 evil\n", 1)), 0o644)
report, _ := applyBlockDecl(t, blockDecl(t, path, meshNames), state)
if got := report.Outcomes[0].Action; got != "corrected" {
t.Errorf("a region edited on the machine was %q", got)
}
if got := readText(t, path); got != written {
t.Errorf("the region was not put back: %q", got)
}
// The region taken out by hand is drift too, and it is put back.
_ = os.WriteFile(path, []byte(workstationHosts), 0o644)
report, _ = applyBlockDecl(t, blockDecl(t, path, meshNames), state)
if got := report.Outcomes[0].Action; got != "corrected" {
t.Errorf("a region removed on the machine was %q", got)
}
if got := readText(t, path); got != written {
t.Errorf("the region was not put back: %q", got)
}
}
func TestTwoRegionsInOneFileAreEachTheirOwn(t *testing.T) {
path := filepath.Join(t.TempDir(), "hosts")
_ = os.WriteFile(path, []byte(workstationHosts), 0o644)
raw := fmt.Sprintf(`{"declaration":1,"resources":[
{"id":"a.names","type":"file","path":%q,"into":"block","content":"10.42.0.1 ace\n"},
{"id":"b.names","type":"file","path":%q,"into":"block","content":"10.43.0.1 lab\n"}
]}`, path, path)
_, state := applyBlockDecl(t, raw, store.State{})
want := workstationHosts + "\n" + marked("a.names", "10.42.0.1 ace\n") + "\n" + marked("b.names", "10.43.0.1 lab\n")
if got := readText(t, path); got != want {
t.Fatalf("two regions:\n%q\nwant\n%q", got, want)
}
report, state := applyBlockDecl(t, raw, state)
for _, o := range report.Outcomes {
if o.Action != "unchanged" {
t.Errorf("%s applied twice was %q", o.ID, o.Action)
}
}
// One undeclared: only its region goes.
only := fmt.Sprintf(`{"declaration":1,"resources":[
{"id":"b.names","type":"file","path":%q,"into":"block","content":"10.43.0.1 lab\n"}
]}`, path)
applyBlockDecl(t, only, state)
want = workstationHosts + "\n" + marked("b.names", "10.43.0.1 lab\n")
if got := readText(t, path); got != want {
t.Errorf("undeclaring one region:\n%q\nwant\n%q", got, want)
}
}
func TestABlockInAFileThatWasNotThereIsCreatedAndRemovedWithIt(t *testing.T) {
path := filepath.Join(t.TempDir(), "conf.d", "mesh.conf")
report, state := applyBlockDecl(t, blockDecl(t, path, meshNames, `"mode":"0600"`), store.State{})
if got := readText(t, path); got != marked(namesID, meshNames) {
t.Fatalf("a created file holds %q", got)
}
if info, _ := os.Stat(path); info.Mode().Perm() != 0o600 {
t.Errorf("a created file is mode %o, declared 0600", info.Mode().Perm())
}
if got := report.Outcomes[0].Action; got != "created" {
t.Errorf("creating was %q", got)
}
if rec, _ := state.Find(namesID); !rec.Into.Created {
t.Error("the mesh creating the file was not recorded")
}
report, _ = undeclare(t, state)
if _, err := os.Stat(path); !os.IsNotExist(err) {
t.Errorf("a file the mesh created, holding only its region, was left behind")
}
if got := report.Outcomes[0].Action; got != "removed" {
t.Errorf("undeclaring was %q", got)
}
// Somebody else wrote into it meanwhile: it is no longer only the mesh's, and it stays.
_, state = applyBlockDecl(t, blockDecl(t, path, meshNames), store.State{})
_ = os.WriteFile(path, []byte(readText(t, path)+"their = line\n"), 0o600)
undeclare(t, state)
if got := readText(t, path); got != "their = line\n" {
t.Errorf("undeclaring a created file somebody wrote into left %q", got)
}
}
func TestMarkersThatDoNotPairAreRefusedAndLeftAlone(t *testing.T) {
for name, text := range map[string]string{
"a begin with no end": "a\n# BEGIN mesh " + namesID + "\nb\n",
"an end with no begin": "a\n# END mesh " + namesID + "\n",
"an end before a begin": "# END mesh " + namesID + "\n# BEGIN mesh " + namesID + "\n",
"a begin twice": marked(namesID, "x\n") + "# BEGIN mesh " + namesID + "\n",
} {
t.Run(name, func(t *testing.T) {
path := filepath.Join(t.TempDir(), "hosts")
_ = os.WriteFile(path, []byte(text), 0o644)
if _, _, err := Apply(context.Background(), archHost(t), parse(t, blockDecl(t, path, meshNames)),
store.State{}, store.OriginDeclared, nil, nil, nil); err == nil {
t.Fatal("markers that do not pair were written between")
}
if got := readText(t, path); got != text {
t.Errorf("the file was changed: %q", got)
}
})
}
}
func TestAMarkerOfAnotherIDIsNotThisRegion(t *testing.T) {
// The id is part of the marker: a region whose id merely starts with this one is not it.
path := filepath.Join(t.TempDir(), "hosts")
other := marked(namesID+"-extra", "10.9.9.9 other\n")
_ = os.WriteFile(path, []byte(other), 0o644)
_, state := applyBlockDecl(t, blockDecl(t, path, meshNames), store.State{})
if got := readText(t, path); got != other+"\n"+marked(namesID, meshNames) {
t.Fatalf("got %q", got)
}
undeclare(t, state)
if got := readText(t, path); got != other {
t.Errorf("undeclaring touched the other region: %q", got)
}
}
func TestABlockAtTheStartStandsAboveEverything(t *testing.T) {
// dhcpcd scopes every line after `interface X` to that interface, so the mesh's global options
// go above all of it.
dhcpcd := "hostname\nduid\n\ninterface enp6s0\nstatic ip_address=192.168.1.5/24\n"
opts := "nohook resolv.conf\ndenyinterfaces mesh0\n"
for name, c := range map[string]struct{ before, after string }{
"a file with content": {dhcpcd, marked(namesID, opts) + "\n" + dhcpcd},
"an empty file": {"", marked(namesID, opts)},
"a file opening blank": {"\n" + dhcpcd, marked(namesID, opts) + "\n" + dhcpcd},
"a last line with no end": {"interface enp6s0", marked(namesID, opts) + "\ninterface enp6s0"},
} {
t.Run(name, func(t *testing.T) {
path := filepath.Join(t.TempDir(), "dhcpcd.conf")
_ = os.WriteFile(path, []byte(c.before), 0o644)
report, state := applyBlockDecl(t, blockDecl(t, path, opts, `"at":"start"`), store.State{})
if got := readText(t, path); got != c.after {
t.Fatalf("got %q, want %q", got, c.after)
}
if got := report.Outcomes[0].Action; got != "updated" {
t.Errorf("adding the region was %q", got)
}
report, state = applyBlockDecl(t, blockDecl(t, path, opts, `"at":"start"`), state)
if got := report.Outcomes[0].Action; got != "unchanged" {
t.Errorf("a second apply was %q", got)
}
undeclare(t, state)
if got := readText(t, path); got != c.before {
t.Errorf("undeclaring left %q, the machine had %q", got, c.before)
}
})
}
t.Run("a file that was not there", func(t *testing.T) {
path := filepath.Join(t.TempDir(), "dhcpcd.conf")
applyBlockDecl(t, blockDecl(t, path, opts, `"at":"start"`), store.State{})
if got := readText(t, path); got != marked(namesID, opts) {
t.Errorf("got %q", got)
}
})
}
func TestARegionAlreadyThereIsNotMovedWhereverAtSaysItGoes(t *testing.T) {
for _, at := range []string{`"at":"start"`, `"at":"end"`} {
path := filepath.Join(t.TempDir(), "dhcpcd.conf")
original := "hostname\n" + marked(namesID, "old\n") + "interface enp6s0\n"
_ = os.WriteFile(path, []byte(original), 0o644)
applyBlockDecl(t, blockDecl(t, path, "nohook resolv.conf\n", at), store.State{})
want := "hostname\n" + marked(namesID, "nohook resolv.conf\n") + "interface enp6s0\n"
if got := readText(t, path); got != want {
t.Errorf("%s: a found region was moved: %q", at, got)
}
}
}
func TestAFileWrittenIntoABlockIsNeverHeldOnAnAdoptedNode(t *testing.T) {
path := filepath.Join(t.TempDir(), "hosts")
_ = os.WriteFile(path, []byte(workstationHosts), 0o644)
resource := fmt.Sprintf(`{"id":%q,"type":"file","path":%q,"into":"block","content":%q}`, namesID, path, meshNames)
d := adopted(t, `{"taken":[],"untaken":{"mesh-wireguard":["`+namesID+`"]}}`, resource)
report, state := applyAdopted(t, d, store.State{}, &machine{}, t.TempDir())
if got := outcomeOf(report, namesID).Action; got == "held" {
t.Fatal("a file written into a block was held, though it replaces nothing that was found")
}
if len(state.Held) != 0 {
t.Errorf("something was held: %+v", state.Held)
}
if got := readText(t, path); got != workstationHosts+"\n"+marked(namesID, meshNames) {
t.Errorf("the adopted node's file was not written into: %q", got)
}
// Held from when it was declared whole, the hold does not keep the region out.
path2 := filepath.Join(t.TempDir(), "hosts")
_ = os.WriteFile(path2, []byte(workstationHosts), 0o644)
known := store.State{Held: []store.Held{{ID: namesID, Module: "mesh-wireguard", Kind: "file", Target: path2}}}
resource2 := fmt.Sprintf(`{"id":%q,"type":"file","path":%q,"into":"block","content":%q}`, namesID, path2, meshNames)
d2 := adopted(t, `{"taken":[],"untaken":{"mesh-wireguard":["`+namesID+`"]}}`, resource2)
report, state = applyAdopted(t, d2, known, &machine{}, t.TempDir())
if got := outcomeOf(report, namesID).Action; got != "updated" {
t.Errorf("the file was %q, not written into", got)
}
if len(state.Held) != 0 {
t.Errorf("the old hold outlived the block declaration: %+v", state.Held)
}
// And the preview says the same: written into, not held.
for _, s := range Plan(d2, known, store.OriginDeclared) {
if s.ID == namesID && s.Verb == "hold" {
t.Errorf("the preview holds a file written into a block: %+v", s)
}
}
}
func TestTheRecordOfABlockSurvivesTheStateFile(t *testing.T) {
// What undeclaring needs is in the state a host saves, not only in memory.
path := filepath.Join(t.TempDir(), "hosts")
original := "a\n" + marked(namesID, "old\n")
_ = os.WriteFile(path, []byte(original), 0o644)
_, state := applyBlockDecl(t, blockDecl(t, path, meshNames, `"at":"start"`), store.State{})
raw, err := json.Marshal(state)
if err != nil {
t.Fatal(err)
}
var back store.State
if err := json.Unmarshal(raw, &back); err != nil {
t.Fatal(err)
}
undeclare(t, back)
if got := readText(t, path); got != original {
t.Errorf("undeclaring from a saved state left %q", got)
}
}
// The mesh's old whole hosts file, as the controller composed it before issue 128.
const oldWholeHosts = "# Generated by the mesh. Do not edit — this file is replaced whenever a machine\n" +
"# joins or leaves, and an edit would survive until then and vanish.\n\n" +
"127.0.0.1\tlocalhost\n" +
"::1\t\tlocalhost ip6-localhost ip6-loopback\n" +
"127.0.1.1\tg14\n" +
"\n" +
"10.42.0.1\tace.internal\tace\n" +
"10.42.0.9\tg14.internal\tg14\t# this machine\n"
func wholeDecl(path, content string) string {
return fmt.Sprintf(`{"declaration":1,"resources":[
{"id":%q,"type":"file","path":%q,"content":%q}
]}`, namesID, path, content)
}
func applyKeepingIn(t *testing.T, raw string, known store.State, keepDir string) (Report, store.State) {
t.Helper()
report, state, err := ApplyKeeping(context.Background(), archHost(t), parse(t, raw), known,
store.OriginDeclared, (&machine{}).run, nil, nil, KeepIn(keepDir))
if err != nil {
t.Fatalf("apply failed: %v", err)
}
return report, state
}
func TestAFileTheMeshWroteWholeAndMadeItselfKeepsOnlyItsLoopbackLines(t *testing.T) {
// Written whole into a file that was not there, then declared as a block under the same id:
// the old names must not stay above the region, where a resolver would answer from them first.
path := filepath.Join(t.TempDir(), "hosts")
_, state := applyKeepingIn(t, wholeDecl(path, oldWholeHosts), store.State{}, t.TempDir())
report, state := applyKeepingIn(t, blockDecl(t, path, meshNames), state, t.TempDir())
floor := "127.0.0.1\tlocalhost\n::1\t\tlocalhost ip6-localhost ip6-loopback\n127.0.1.1\tg14\n"
if got := readText(t, path); got != floor+"\n"+marked(namesID, meshNames) {
t.Fatalf("the old whole file became:\n%q", got)
}
o := outcomeOf(report, namesID)
if o.Action != "updated" || !strings.Contains(o.Detail, "loopback lines") {
t.Errorf("the rebuild was reported as %q: %s", o.Action, o.Detail)
}
if rec, _ := state.Find(namesID); !rec.Into.Created {
t.Error("a file the mesh made itself was not recorded as the mesh's")
}
report, _ = applyKeepingIn(t, blockDecl(t, path, meshNames), state, t.TempDir())
if got := outcomeOf(report, namesID).Action; got != "unchanged" {
t.Errorf("applied again, the rebuilt file was %q", got)
}
undeclare(t, state)
if got := readText(t, path); got != floor {
t.Errorf("undeclared, the file holds %q", got)
}
}
func TestAFileTheMeshWroteWholeOverAnOriginalGetsTheOriginalBack(t *testing.T) {
path := filepath.Join(t.TempDir(), "hosts")
_ = os.WriteFile(path, []byte(workstationHosts), 0o644)
keep := t.TempDir()
_, state := applyKeepingIn(t, wholeDecl(path, oldWholeHosts), store.State{}, keep)
if rec, _ := state.Find(namesID); rec.Kept == "" {
t.Fatal("where the original was kept was not recorded")
}
report, state := applyKeepingIn(t, blockDecl(t, path, meshNames), state, keep)
if got := readText(t, path); got != workstationHosts+"\n"+marked(namesID, meshNames) {
t.Fatalf("the old whole file became:\n%q", got)
}
if d := outcomeOf(report, namesID).Detail; !strings.Contains(d, "original kept at") {
t.Errorf("the rebuild was reported as: %s", d)
}
undeclare(t, state)
if got := readText(t, path); got != workstationHosts {
t.Errorf("undeclared, the machine did not get its original back: %q", got)
}
}
func TestAFileTheMeshWroteWholeAndSomebodyChangedIsWrittenIntoAsItStands(t *testing.T) {
path := filepath.Join(t.TempDir(), "hosts")
_, state := applyKeepingIn(t, wholeDecl(path, oldWholeHosts), store.State{}, t.TempDir())
edited := oldWholeHosts + "192.168.1.20 printer\n"
_ = os.WriteFile(path, []byte(edited), 0o644)
report, _ := applyKeepingIn(t, blockDecl(t, path, meshNames), state, t.TempDir())
if got := readText(t, path); got != edited+"\n"+marked(namesID, meshNames) {
t.Fatalf("an edited whole file became:\n%q", got)
}
if d := outcomeOf(report, namesID).Detail; !strings.Contains(d, "changed since; its old lines were kept") {
t.Errorf("the outcome does not say so: %s", d)
}
}
func TestALinkedFileStaysALink(t *testing.T) {
dir := t.TempDir()
real := filepath.Join(dir, "static", "hosts")
_ = os.MkdirAll(filepath.Dir(real), 0o755)
_ = os.WriteFile(real, []byte(workstationHosts), 0o644)
link := filepath.Join(dir, "hosts")
if err := os.Symlink(real, link); err != nil {
t.Fatal(err)
}
_, state := applyBlockDecl(t, blockDecl(t, link, meshNames), store.State{})
if info, err := os.Lstat(link); err != nil || info.Mode()&os.ModeSymlink == 0 {
t.Fatalf("the link was replaced by a file")
}
if got := readText(t, real); got != workstationHosts+"\n"+marked(namesID, meshNames) {
t.Errorf("the file the link names holds %q", got)
}
undeclare(t, state)
if info, err := os.Lstat(link); err != nil || info.Mode()&os.ModeSymlink == 0 {
t.Fatalf("undeclaring replaced the link with a file")
}
if got := readText(t, real); got != workstationHosts {
t.Errorf("undeclared, the file the link names holds %q", got)
}
}
func TestALastLineWithNoEndIsGivenBackWithNone(t *testing.T) {
path := filepath.Join(t.TempDir(), "hosts")
_ = os.WriteFile(path, []byte("x"), 0o644)
_, state := applyBlockDecl(t, blockDecl(t, path, meshNames), store.State{})
if got := readText(t, path); got != "x\n\n"+marked(namesID, meshNames) {
t.Fatalf("got %q", got)
}
undeclare(t, state)
if got := readText(t, path); got != "x" {
t.Errorf("undeclaring left %q, the machine had %q", got, "x")
}
}
func TestAFailedBlockWriteKeepsItsHold(t *testing.T) {
// Held from when it was declared whole, then declared as a block into a file whose markers do
// not pair: the write is refused, and the hold — with where its original is — stays.
path := filepath.Join(t.TempDir(), "hosts")
broken := "a\n# BEGIN mesh " + namesID + "\n"
_ = os.WriteFile(path, []byte(broken), 0o644)
known := store.State{Held: []store.Held{{ID: namesID, Module: "mesh-wireguard", Kind: "file",
Target: path, Kept: "/var/lib/mesh/kept/hosts"}}}
resource := fmt.Sprintf(`{"id":%q,"type":"file","path":%q,"into":"block","content":%q}`, namesID, path, meshNames)
d := adopted(t, `{"taken":[],"untaken":{"mesh-wireguard":["`+namesID+`"]}}`, resource)
_, state, err := ApplyKeeping(context.Background(), archHost(t), d, known,
store.OriginDeclared, (&machine{}).run, nil, nil, KeepIn(t.TempDir()))
if err == nil {
t.Fatal("a write into unpaired markers was not refused")
}
h, held := state.HeldAt(namesID)
if !held || h.Kept != "/var/lib/mesh/kept/hosts" {
t.Errorf("a failed write released the hold: %+v", state.Held)
}
}
+97
View File
@@ -0,0 +1,97 @@
package apply
import (
"context"
"os"
"path/filepath"
"strings"
"testing"
"github.com/novox/mesh-host/internal/store"
)
// A take is a comparison (novox/hq ADR 0163): while a module's container is held, the host reports
// the found image and its age beside the declared one, the networks and who else is on them, the
// mounts and the ports — and says when the declared image is the older.
func TestAHeldContainerCarriesTheFactsATakeCompares(t *testing.T) {
dir, page, m := predecessor(t)
m.containers["hello-web"].image = "web:1.27"
m.containers["hello-web"].imageID = "sha256:found"
m.containers["hello-web"].networks = []string{"predecessor_default"}
m.containers["hello-web"].mounts = []string{"/srv/web:/data"}
m.containers["hello-web"].ports = []string{"80/tcp>0.0.0.0:8080"}
m.images = map[string]string{"sha256:found": "2026-09-17T10:00:00Z", pinned: "2026-08-20T10:00:00Z"}
m.members = map[string][]string{"predecessor_default": {"hello-web", "office", "db"}}
_, state := applyAdopted(t, adopted(t, untakenWeb, webResources(page)), store.State{}, m, dir)
h, ok := state.HeldAt("hello-web.server")
if !ok || h.Facts == nil {
t.Fatalf("a held container carries no facts: %+v", h)
}
f := h.Facts
if f.Image != "web:1.27" || f.ImageCreated != "2026-09-17T10:00:00Z" {
t.Errorf("the found image and its age: %+v", f)
}
if f.DeclaredImage != pinned || f.DeclaredImageCreated != "2026-08-20T10:00:00Z" || !f.Downgrade {
t.Errorf("the declared image, its age, and that it is a downgrade: %+v", f)
}
if got := f.Networks["predecessor_default"]; len(got) != 2 || got[0] != "db" || got[1] != "office" {
t.Errorf("the neighbours on the found network, without the container itself: %v", f.Networks)
}
if len(f.Mounts) != 1 || f.Mounts[0] != "/srv/web:/data" || len(f.Ports) != 1 || f.Ports[0] != "80/tcp>0.0.0.0:8080" {
t.Errorf("mounts and ports as found: %+v", f)
}
// And the held file carries how the declared content differs from what was found.
p, ok := state.HeldAt("hello-web.page")
if !ok || p.Facts == nil || !p.Facts.Differs {
t.Fatalf("a held file that differs from the declared content does not say so: %+v", p)
}
joined := strings.Join(p.Facts.Difference, "\n")
if !strings.Contains(joined, "- the predecessor's page") || !strings.Contains(joined, "+ the mesh's page") {
t.Errorf("the difference does not show what is lost and what is new: %q", joined)
}
_ = os.Remove(filepath.Join(dir, "unused"))
}
// A declared image not yet on the machine leaves its age unknown and the comparison undecided.
func TestAnImageNotYetPulledLeavesTheDowngradeUndecided(t *testing.T) {
dir, page, m := predecessor(t)
m.containers["hello-web"].image = "web:1.27"
m.containers["hello-web"].imageID = "sha256:found"
m.images = map[string]string{"sha256:found": "2026-09-17T10:00:00Z"}
_, state := applyAdopted(t, adopted(t, untakenWeb, webResources(page)), store.State{}, m, dir)
h, _ := state.HeldAt("hello-web.server")
if h.Facts == nil || h.Facts.DeclaredImageCreated != "" || h.Facts.Downgrade {
t.Fatalf("an unknown declared age decided a downgrade: %+v", h.Facts)
}
}
func TestTheDifferenceIsWhatIsLostAndWhatIsNew(t *testing.T) {
differs, lines := differenceOf("a\nprivate scope: local\nb\n", "a\nb\nupstream: public\n")
if !differs || len(lines) != 2 || lines[0] != "- private scope: local" || lines[1] != "+ upstream: public" {
t.Fatalf("got %v %v", differs, lines)
}
if differs, lines := differenceOf("same\n", "same\n"); differs || lines != nil {
t.Fatalf("identical content differs: %v %v", differs, lines)
}
}
// What runs on the machine that the mesh neither wrote nor holds is reported (ADR 0163).
func TestStraysAreWhatRunsHereThatNobodyAsked(t *testing.T) {
m := &machine{containers: map[string]*fakeContainer{
"hello-web": {id: "ours", running: true, image: "web:1"},
"gitea-old": {id: "left-behind", running: true, image: "gitea:1.22"},
"held-thing": {id: "found", running: true, image: "x:1"},
}}
known := store.State{
Resources: []store.Applied{{ID: "hello-web.server", Type: "container", Target: "hello-web"}},
Held: []store.Held{{ID: "other.server", Kind: "container", Target: "held-thing"}},
}
strays, err := Strays(context.Background(), m.run, known)
if err != nil {
t.Fatal(err)
}
if len(strays) != 1 || strays[0].Name != "gitea-old" || !strings.Contains(strays[0].Detail, "gitea:1.22") {
t.Fatalf("strays: %+v", strays)
}
}
+75
View File
@@ -0,0 +1,75 @@
package apply
import (
"context"
"strings"
"testing"
"github.com/novox/mesh-host/internal/store"
)
// The host's own former archive stops nothing (novox/hq issue 194). A new host's first apply finds
// the version before it as a former target of the archive that delivered it; an archive has no
// removal (issue 162), and the refusal stopped every machine applying anything. A former target of
// such a kind is left in place, said, and forgotten. An archive the declaration dropped still fails,
// as 162 has it.
func TestTheHostsOwnFormerArchiveIsLeftInPlaceNotFatal(t *testing.T) {
run := func(_ context.Context, name string, args ...string) (string, error) {
if name == "docker" && args[0] == "info" {
return "29.0.0\n", nil
}
return "", nil
}
dir := t.TempDir()
former := store.FormerID("mesh-host.next", dir+"/versions/old")
known := store.State{Resources: []store.Applied{
{ID: "mesh-host.next", Type: "archive", Target: dir + "/versions/new", Origin: store.OriginDeclared},
{ID: former, Type: "archive", Target: dir + "/versions/old", Origin: store.OriginDeclared},
}}
body, digest := anArchive(t, map[string]string{"nox-mesh-host": "#!/bin/sh\n"})
d := parse(t, `{"declaration":1,"resources":[
{"id":"mesh-host.next","type":"archive","path":"`+dir+`/versions/new","source":"`+serving(t, body)+`","digest":"`+digest+`"},
{"id":"notes.conf","type":"file","path":"`+dir+`/notes.conf","content":"x"}
]}`)
report, state, err := Apply(context.Background(), archHost(t), d, known, store.OriginDeclared, run, nil, nil)
if err != nil {
t.Fatalf("the apply failed: %v", err)
}
if _, still := state.HeldAt(former); still {
t.Fatal("held?")
}
for _, r := range state.Resources {
if r.ID == former {
t.Fatal("the former archive is still on record")
}
}
said := false
for _, o := range report.Outcomes {
if o.ID == former && o.Action == "forgotten" && strings.Contains(o.Detail, "left in place") {
said = true
}
}
if !said {
t.Fatalf("leaving the former archive was not said: %+v", report.Outcomes)
}
applied := false
for _, o := range report.Outcomes {
if o.ID == "notes.conf" && o.Action == "created" {
applied = true
}
}
if !applied {
t.Fatalf("the rest of the declaration was not applied: %+v", report.Outcomes)
}
// An archive the declaration dropped is a different matter: nothing can undo it, and saying
// it was would report an effect the host declined to have (issue 162).
dropped := store.State{Resources: []store.Applied{
{ID: "tool.next", Type: "archive", Target: "/usr/lib/tool/versions/old", Origin: store.OriginDeclared},
}}
only := parse(t, `{"declaration":1,"resources":[{"id":"notes.conf","type":"file","path":"`+dir+`/notes.conf","content":"x"}]}`)
if _, _, err := Apply(context.Background(), archHost(t), only, dropped, store.OriginDeclared, run, nil, nil); err == nil ||
!strings.Contains(err.Error(), "no way to remove") {
t.Fatalf("a dropped archive was passed over: %v", err)
}
}
+387
View File
@@ -0,0 +1,387 @@
package apply
import (
"context"
"encoding/json"
"os"
"path/filepath"
"strings"
"testing"
"github.com/novox/mesh-host/internal/store"
)
// Defends novox/hq ADR 0118: undeclaring removes what the mesh made, gives back what it changed,
// and leaves what was the machine's — which needs what was found kept exactly, and a unit the mesh
// made known for the mesh's whatever its record says.
func unitsMachine(units map[string]*fakeUnit) *machine {
return &machine{containers: map[string]*fakeContainer{}, units: units}
}
// nothingButA is a declaration of one unrelated file, so everything recorded is undeclared.
func nothingButA(t *testing.T) string {
return `{"declaration":1,"resources":[
{"id":"other","type":"file","path":"` + filepath.Join(t.TempDir(), "a") + `","content":"a\n"}]}`
}
// copyOf is a state as a later load of it would be: sharing nothing with the one it came from.
func copyOf(t *testing.T, s store.State) store.State {
t.Helper()
raw, err := json.Marshal(s)
if err != nil {
t.Fatal(err)
}
var out store.State
if err := json.Unmarshal(raw, &out); err != nil {
t.Fatal(err)
}
return out
}
func applyOn(t *testing.T, raw string, known store.State, m *machine) (Report, store.State, error) {
t.Helper()
return Apply(context.Background(), archHost(t), parse(t, raw), known, store.OriginDeclared, m.run, nil, nil)
}
func TestAUnitWhoseFileTheMeshWroteIsStoppedWhateverItsRecordSays(t *testing.T) {
// The adoption guard's unit file is the mesh's own file resource, written where there was none.
// Its service recorded before the host kept what it found has no Found, and forgetting it left
// the guard's table loaded on a converged node and its unit file deleted from under it.
units := t.TempDir()
was := unitDir
unitDir = units
t.Cleanup(func() { unitDir = was })
unitFile := filepath.Join(units, "mesh-guard.service")
if err := os.WriteFile(unitFile, []byte("[Unit]\n"), 0o644); err != nil {
t.Fatal(err)
}
m := unitsMachine(map[string]*fakeUnit{"mesh-guard.service": {active: "active", enabled: "enabled"}})
fileThereAtStop := false
run := func(ctx context.Context, name string, args ...string) (string, error) {
if name == "systemctl" && args[0] == "stop" {
_, err := os.Stat(unitFile)
fileThereAtStop = err == nil
}
return m.run(ctx, name, args...)
}
// As a host before this change recorded them: the unit file first, then the service it
// starts, and no Found on the service.
known := store.State{Resources: []store.Applied{
{ID: "adoption.guard-unit", Type: "file", Target: unitFile, Origin: store.OriginDeclared},
{ID: "adoption.guard-running", Type: "service", Target: "mesh-guard.service", Origin: store.OriginDeclared},
}}
report, after, err := applyWith(t, parse(t, nothingButA(t)), known, run)
if err != nil {
t.Fatal(err)
}
if u := m.units["mesh-guard.service"]; u.active != "inactive" || u.enabled != "disabled" {
t.Errorf("the mesh's own unit was left %s and %s: %v", u.active, u.enabled, m.asked)
}
if !fileThereAtStop {
t.Error("the unit was stopped after its file was deleted, or not at all")
}
if o := outcomeOf(report, "adoption.guard-running"); o.Action != "removed" || !strings.Contains(o.Detail, "unit file") {
t.Errorf("outcome %+v", o)
}
if _, err := os.Stat(unitFile); !os.IsNotExist(err) {
t.Error("the unit file outlived its record")
}
if len(after.Resources) != 1 {
t.Errorf("still recorded: %v", after.IDs())
}
// A unit file the host wrote OVER — its original kept — is the machine's unit, and a record
// with no Found leaves it as it is.
if err := os.WriteFile(unitFile, []byte("[Unit]\n"), 0o644); err != nil {
t.Fatal(err)
}
m = unitsMachine(map[string]*fakeUnit{"mesh-guard.service": {active: "active", enabled: "enabled"}})
known.Resources[0].Kept = filepath.Join(t.TempDir(), "original")
if err := os.WriteFile(known.Resources[0].Kept, []byte("[Unit]\n"), 0o600); err != nil {
t.Fatal(err)
}
if _, _, err := applyWith(t, parse(t, nothingButA(t)), known, m.run); err != nil {
t.Fatal(err)
}
if m.did("systemctl stop") || m.did("systemctl disable") {
t.Errorf("a unit whose file the mesh only wrote over was stopped: %v", m.asked)
}
}
func TestOnlyAUnitsOwnFileTheMeshCreatedMakesItTheMeshs(t *testing.T) {
known := store.State{Resources: []store.Applied{
{ID: "a", Type: "file", Target: "/etc/systemd/system/made.service"},
{ID: "b", Type: "file", Target: "/run/systemd/system/runtime.service"},
{ID: "c", Type: "file", Target: "/etc/systemd/system/kept.service", Kept: "/var/lib/mesh-host/kept/x"},
{ID: "d", Type: "file", Target: "/etc/systemd/system/into.service", Into: &store.Into{Format: "block"}},
{ID: "e", Type: "file", Target: "/etc/systemd/system/docker.service.d/mesh.conf"},
{ID: "f", Type: "file", Target: "/etc/mesh/elsewhere.service"},
{ID: "g", Type: "directory", Target: "/etc/systemd/system/dir.service"},
}}
made := meshMadeUnits(known)
for unit, want := range map[string]bool{"made.service": true, "runtime.service": true, "kept.service": false,
"into.service": false, "mesh.conf": false, "docker.service.d": false, "elsewhere.service": false, "dir.service": false} {
if made[unit] != want {
t.Errorf("%s: made %v, want %v", unit, made[unit], want)
}
}
}
func TestWhatWasFoundOutlivesAFirstApplyThatFailed(t *testing.T) {
// Enabled, then it would not start: no record. The next apply must not read the enable as
// the machine's — or undeclaring leaves enabled a unit the mesh enabled.
m := unitsMachine(map[string]*fakeUnit{"filter.service": {active: "inactive", enabled: "disabled", wontStart: true}})
declared := `{"declaration":1,"resources":[
{"id":"s","type":"service","unit":"filter.service","state":"running","boot":"enabled"}]}`
_, first, err := applyOn(t, declared, store.State{}, m)
if err == nil || !m.did("systemctl enable filter.service") {
t.Fatalf("the fixture did not enable and then fail: %v, %v", err, m.asked)
}
if _, ok := first.Find("s"); ok {
t.Fatal("a failed apply was recorded")
}
if p := first.FoundFirst["s"]; p.State != "stopped" || p.Boot != "disabled" || p.Unit != "filter.service" {
t.Fatalf("what was found was not kept through the failure: %+v", first.FoundFirst)
}
failed := copyOf(t, first)
m.units["filter.service"].wontStart = false
_, second, err := applyOn(t, declared, first, m)
if err != nil {
t.Fatal(err)
}
if rec, _ := second.Find("s"); rec.Found == nil || rec.Found.State != "stopped" || rec.Found.Boot != "disabled" {
t.Errorf("the record carries the mesh's own effect as found: %+v", rec.Found)
}
if second.FoundFirst != nil {
t.Errorf("kept apart after the record carried it: %+v", second.FoundFirst)
}
if _, _, err := applyOn(t, nothingButA(t), second, m); err != nil {
t.Fatal(err)
}
if u := m.units["filter.service"]; u.active != "inactive" || u.enabled != "disabled" {
t.Errorf("undeclared, the unit was left %s and %s", u.active, u.enabled)
}
// Undeclared before it was ever recorded, what was found goes with it — only for its origin.
if _, kept, _ := Apply(context.Background(), archHost(t), parse(t, nothingButA(t)), copyOf(t, failed),
store.OriginCarried, m.run, nil, nil); kept.FoundFirst["s"].State == "" {
t.Error("a carried apply dropped what the mesh's declaration found")
}
if _, dropped, err := applyOn(t, nothingButA(t), copyOf(t, failed), m); err != nil || dropped.FoundFirst != nil {
t.Errorf("kept for a service no longer declared: %+v, %v", dropped.FoundFirst, err)
}
}
func TestBootIsFoundTheFirstTimeTheMeshSetsIt(t *testing.T) {
// The declaration said nothing about boot at first, so the mesh never touched it: what is
// there when a declaration first does is still the machine's.
m := unitsMachine(map[string]*fakeUnit{"web.service": {active: "active", enabled: "disabled"}})
known := store.State{Resources: []store.Applied{{ID: "s", Type: "service", Target: "web.service",
Origin: store.OriginDeclared, Found: &store.FoundUnit{Unit: "web.service", State: "running"}}}}
_, after, err := applyOn(t, `{"declaration":1,"resources":[
{"id":"s","type":"service","unit":"web.service","state":"running","boot":"enabled"}]}`, known, m)
if err != nil {
t.Fatal(err)
}
rec, _ := after.Find("s")
if rec.Found == nil || rec.Found.State != "running" || rec.Found.Boot != "disabled" {
t.Fatalf("found %+v, want running and disabled", rec.Found)
}
report, _, err := applyOn(t, nothingButA(t), after, m)
if err != nil {
t.Fatal(err)
}
if u := m.units["web.service"]; u.active != "active" || u.enabled != "disabled" {
t.Errorf("undeclared, the unit is %s and %s; want running, and disabled again", u.active, u.enabled)
}
if o := outcomeOf(report, "s"); o.Action != "restored" || o.Detail != "disabled at boot, as the host found it" {
t.Errorf("outcome %+v", o)
}
}
func TestAServiceOnceDeclaredWithNoStateIsFoundWhenFirstGivenOne(t *testing.T) {
// Declared with no state (novox/hq ADR 0117), the mesh never started or stopped it — so what
// is there when a declaration first gives it one is what the machine had.
m := unitsMachine(map[string]*fakeUnit{"net.service": {active: "inactive", enabled: "disabled"}})
known := store.State{Resources: []store.Applied{{ID: "s", Type: "service", Target: "net.service",
Origin: store.OriginDeclared, Stateless: true}}}
_, after, err := applyOn(t, `{"declaration":1,"resources":[
{"id":"s","type":"service","unit":"net.service","state":"running"}]}`, known, m)
if err != nil {
t.Fatal(err)
}
if rec, _ := after.Find("s"); rec.Found == nil || rec.Found.State != "stopped" {
t.Fatalf("found %+v, want stopped", rec.Found)
}
if _, _, err := applyOn(t, nothingButA(t), after, m); err != nil {
t.Fatal(err)
}
if m.units["net.service"].active != "inactive" {
t.Error("the unit the mesh started outlived its declaration")
}
}
func TestAUnitWrittenInTheSameApplyIsLoadedBeforeItIsRead(t *testing.T) {
// Read before the service manager is told about its new file, the unit is not there to find.
dir := t.TempDir()
m := unitsMachine(map[string]*fakeUnit{"fresh.service": {active: "inactive", enabled: "disabled"}})
_, _, err := applyOn(t, `{"declaration":1,"resources":[
{"id":"unit","type":"file","path":"`+filepath.Join(dir, "fresh.service")+`","content":"[Unit]\n"},
{"id":"s","type":"service","unit":"fresh.service","state":"running","restart-on":["unit"]}]}`,
store.State{}, m)
if err != nil {
t.Fatal(err)
}
reload, show := -1, -1
for i, a := range m.asked {
if a == "systemctl daemon-reload" && reload < 0 {
reload = i
}
if strings.HasPrefix(a, "systemctl show fresh.service") && show < 0 {
show = i
}
}
if reload < 0 || show < 0 || reload > show {
t.Errorf("the unit was read before its file was loaded: %v", m.asked)
}
}
func TestAServiceMovedToAnotherUnitGivesTheOldOneBackAndFindsTheNewOne(t *testing.T) {
m := unitsMachine(map[string]*fakeUnit{
"old.service": {active: "active", enabled: "enabled"},
"new.service": {active: "inactive", enabled: "disabled"},
})
known := store.State{Resources: []store.Applied{{ID: "s", Type: "service", Target: "old.service",
Origin: store.OriginDeclared, Found: &store.FoundUnit{Unit: "old.service", State: "stopped"}}}}
report, after, err := applyOn(t, `{"declaration":1,"resources":[
{"id":"s","type":"service","unit":"new.service","state":"running"}]}`, known, m)
if err != nil {
t.Fatal(err)
}
if m.units["old.service"].active != "inactive" {
t.Error("the unit the mesh started is still running though nothing declares it")
}
if m.units["new.service"].active != "active" {
t.Error("the unit now declared was not started")
}
rec, _ := after.Find("s")
if rec.Found == nil || rec.Found.Unit != "new.service" || rec.Found.State != "stopped" {
t.Errorf("what was found about the old unit was carried to the new one: %+v", rec.Found)
}
if o := outcomeOf(report, "s"); !strings.Contains(o.Detail, "old.service stopped, as the host found it") {
t.Errorf("giving the old unit back went unsaid: %+v", o)
}
}
func TestARemovalSaysWhatItDid(t *testing.T) {
cases := []struct {
name string
found *store.FoundUnit
unit *fakeUnit
action, detail string
}{
{"found stopped and still stopped", &store.FoundUnit{State: "stopped"},
&fakeUnit{active: "inactive", enabled: "disabled"}, "forgotten", "already as the host found it (stopped)"},
{"started by the mesh", &store.FoundUnit{State: "stopped"},
&fakeUnit{active: "active", enabled: "disabled"}, "restored", "stopped, as the host found it"},
{"started and enabled by the mesh", &store.FoundUnit{State: "stopped", Boot: "disabled"},
&fakeUnit{active: "active", enabled: "enabled"}, "restored", "stopped, disabled at boot, as the host found it"},
{"found running, stopped by the mesh", &store.FoundUnit{State: "running"},
&fakeUnit{active: "inactive", enabled: "disabled"}, "forgotten",
"left as it is; the mesh stopped it and does not start anything on the way out"},
{"found running and enabled, disabled by the mesh", &store.FoundUnit{State: "running", Boot: "enabled"},
&fakeUnit{active: "active", enabled: "disabled"}, "forgotten",
"left as it is; the mesh disabled it at boot and does not start anything on the way out"},
{"found running, still running", &store.FoundUnit{State: "running"},
&fakeUnit{active: "active", enabled: "enabled"}, "forgotten", "it was running before the mesh; left as it is"},
// Found says to stop it, so the machine is asked about it — and it is gone.
{"started by the mesh, since uninstalled", &store.FoundUnit{State: "stopped"},
nil, "forgotten", "the unit no longer exists"},
{"recorded before the host kept what it found", nil,
&fakeUnit{active: "active", enabled: "enabled"}, "forgotten", "recorded before the host kept what it found; left as it is"},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
units := map[string]*fakeUnit{}
if c.unit != nil {
units["unit.service"] = c.unit
}
m := unitsMachine(units)
known := store.State{Resources: []store.Applied{{ID: "s", Type: "service", Target: "unit.service",
Origin: store.OriginDeclared, Found: c.found}}}
report, after, err := applyOn(t, nothingButA(t), known, m)
if err != nil {
t.Fatal(err)
}
if o := outcomeOf(report, "s"); o.Action != c.action || o.Detail != c.detail {
t.Errorf("said %s: %s; want %s: %s", o.Action, o.Detail, c.action, c.detail)
}
if c.unit == nil && !m.did("systemctl show unit.service") {
t.Errorf("the machine was never asked whether the unit is there: %v", m.asked)
}
if m.did("systemctl start") || m.did("systemctl enable") {
t.Errorf("something was started on the way out: %v", m.asked)
}
if _, still := after.Find("s"); still {
t.Error("still recorded")
}
})
}
}
func TestAUnitTheMeshStartedIsStoppedWhenUndeclaredAndOneFoundRunningIsNot(t *testing.T) {
// End to end: found by the first apply, carried, given back.
m := unitsMachine(map[string]*fakeUnit{
"filter.service": {active: "inactive", enabled: "disabled"},
"runtime.service": {active: "active", enabled: "enabled"},
})
_, state, err := applyOn(t, `{"declaration":1,"resources":[
{"id":"filter","type":"service","unit":"filter.service","state":"running","boot":"enabled"},
{"id":"runtime","type":"service","unit":"runtime.service","state":"running","boot":"enabled"}]}`,
store.State{}, m)
if err != nil {
t.Fatal(err)
}
if m.units["filter.service"].active != "active" {
t.Fatal("the fixture did not start the filter")
}
if _, _, err := applyOn(t, nothingButA(t), state, m); err != nil {
t.Fatal(err)
}
if u := m.units["filter.service"]; u.active != "inactive" || u.enabled != "disabled" {
t.Errorf("the filter the mesh started and enabled is %s and %s", u.active, u.enabled)
}
if u := m.units["runtime.service"]; u.active != "active" || u.enabled != "enabled" {
t.Errorf("the runtime that was running before the mesh is %s and %s", u.active, u.enabled)
}
}
func TestAUnitHeldAndThenTakenIsFoundAsThePredecessorLeftIt(t *testing.T) {
// Held while its module was untaken, nothing was applied and nothing found; taken, the first
// apply finds the predecessor's unit — stopped, but started at boot — before starting it.
dir := t.TempDir()
m := unitsMachine(map[string]*fakeUnit{"hello.service": {active: "inactive", enabled: "enabled"}})
service := `{"id":"hello-web.unit","type":"service","unit":"hello.service","state":"running","boot":"enabled"}`
_, held := applyAdopted(t, adopted(t, untaken("hello-web.unit"), service), store.State{}, m, dir)
if _, ok := held.HeldAt("hello-web.unit"); !ok {
t.Fatal("the fixture's unit was not held")
}
_, taken := applyAdopted(t, adopted(t, `{"taken":["hello-web"]}`, service), held, m, dir)
rec, _ := taken.Find("hello-web.unit")
if rec.Found == nil || rec.Found.State != "stopped" || rec.Found.Boot != "enabled" {
t.Fatalf("found %+v, want the predecessor's stopped and enabled", rec.Found)
}
if m.units["hello.service"].active != "active" {
t.Fatal("the taken unit was not started")
}
if _, _, err := applyOn(t, nothingButA(t), taken, m); err != nil {
t.Fatal(err)
}
if u := m.units["hello.service"]; u.active != "inactive" || u.enabled != "enabled" {
t.Errorf("given back as %s and %s; the predecessor left it stopped and enabled", u.active, u.enabled)
}
}
+780
View File
@@ -0,0 +1,780 @@
package apply
import (
"context"
"crypto/sha256"
"encoding/hex"
"errors"
"fmt"
"os"
"path/filepath"
"sort"
"strings"
"syscall"
"time"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/store"
"github.com/novox/mesh-host/internal/system"
)
// Keep records the original of a file found on an adopted node, before anything else happens to
// it, and says where (novox/hq ADR 0100). It never overwrites an original it already kept: the
// first copy is the one that was there before the mesh.
type Keep func(path string, content []byte, mode os.FileMode) (string, error)
// KeepIn keeps originals under dir/kept, each named for the path it came from AND for what was in
// it, readable by root alone — a predecessor's configuration may carry its credentials.
//
// **By content as well as path, because a path has more than one original.** A file held, let go
// when its module was unassigned, rewritten by the predecessor and found again is a second
// original; named by path alone the second copy was silently discarded while the report said it
// was kept (novox/hq ADR 0100). The same content at the same path is kept once.
func KeepIn(dir string) Keep {
return func(path string, content []byte, _ os.FileMode) (string, error) {
where := sha256.Sum256([]byte(path))
what := sha256.Sum256(content)
kept := filepath.Join(dir, "kept", hex.EncodeToString(where[:])[:12]+"-"+
hex.EncodeToString(what[:])[:12]+"-"+filepath.Base(path))
if _, err := os.Lstat(kept); err == nil {
return kept, nil
}
if err := os.MkdirAll(filepath.Dir(kept), 0o700); err != nil {
return "", err
}
if err := writeAtomically(kept, content, 0o600); err != nil {
return "", err
}
back, err := os.ReadFile(kept)
if err != nil || string(back) != string(content) {
return "", fmt.Errorf("kept the original of %s at %s and cannot read it back", path, kept)
}
return kept, nil
}
}
// foundBefore is what an adopted apply finds on the machine before it changes anything: each
// directory, service unit and container mount source of an untaken module that is present with no
// record (novox/hq ADR 0103). Looked at first, because the apply itself makes such things — a
// file's parent directory, a unit file a module writes, a package that brings its unit — and what
// the mesh made in this apply was not found.
type foundBefore struct {
is map[string]bool
// trouble is what could not be asked about, by the same key, so the resource that would need
// the answer fails rather than proceeding as if the machine had nothing there.
trouble map[string]string
}
func (f foundBefore) has(key string) bool { return f.is[key] }
// why is the reason a key could not be settled, and empty when there was none.
func (f foundBefore) why(key string) string { return f.trouble[key] }
func lookBefore(ctx context.Context, sys system.System, d *declaration.Declaration, known store.State,
run Runner) foundBefore {
seen := foundBefore{is: map[string]bool{}, trouble: map[string]string{}}
if d.Adoption == nil {
return seen
}
cri, asked := "", false
for _, r := range d.Resources {
if _, untaken := d.Adoption.UntakenModuleOf(r.Identity()); !untaken {
continue
}
if _, held := known.HeldAt(r.Identity()); held {
continue
}
switch res := r.(type) {
case *declaration.Directory:
if present(res.Path) && !recordedPath(known, res.Path) {
seen.is["path:"+res.Path] = true
}
case *declaration.Archive:
// Unpacking over it, and re-owning it recursively, would change the predecessor's
// files.
if present(res.Path) && !recordedPath(known, res.Path) {
seen.is["path:"+res.Path] = true
}
case *declaration.Process:
// Its unit would be written over and restarted.
if !known.Recorded(string(declaration.TypeProcess), res.Name) &&
present(filepath.Join(unitDir, res.Name+".service")) {
seen.is["unit-file:"+res.Name] = true
}
case *declaration.User:
// Its shell and groups would be changed.
if !known.Recorded(string(declaration.TypeUser), res.Name) {
if _, exists, err := system.LookUpUser(ctx, run, res.Name); err == nil && exists {
seen.is["user:"+res.Name] = true
}
}
case *declaration.Service:
if res.Stateless() || known.Recorded(string(declaration.TypeService), res.Unit) {
continue
}
// **Found is a unit somebody put on this machine, or one the machine uses.**
//
// Where it comes from first: a unit the service manager loads from outside /usr —
// /etc/systemd/system or /run/systemd/system — was installed by an administrator, so
// it is a predecessor's whatever state it is in, and one deliberately stopped and
// disabled must stay that way (novox/hq ADR 0103).
//
// A unit a package ships, under /usr, is not held by its mere presence: the private
// network's own wg-quick@mesh0 is an instance of a template the tunnel package ships,
// nothing had ever run it, and holding it kept the private network from ever coming up
// (found by the adoption bed). Such a unit is held only if the machine actually uses
// it — running, or started at boot.
state, err := sys.ServiceState(ctx, run, res.Unit)
if err != nil {
continue
}
if from, ok := sys.(unitFiles); ok {
if path, err := from.ServiceUnitFile(ctx, run, res.Unit); err == nil && installedByHand(path) {
seen.is["unit:"+res.Unit] = true
continue
}
}
boot, _ := sys.ServiceBoot(ctx, run, res.Unit)
if state == "running" || boot == "enabled" {
seen.is["unit:"+res.Unit] = true
}
case *declaration.Container:
if known.Recorded(string(declaration.TypeContainer), res.Name) {
continue
}
for _, v := range res.Volumes {
src := mountSource(v)
switch {
case src == "":
case strings.HasPrefix(src, "/"):
if !systemPath(src) && present(src) && !recordedPath(known, src) {
seen.is["path:"+src] = true
}
default:
if !asked {
cri, _ = containerRuntime(ctx, run)
asked = true
}
if cri == "" {
continue
}
if _, err := run(ctx, cri, "volume", "inspect", src); err == nil {
seen.is["volume:"+src] = true
} else if !absent(err) {
seen.trouble["volume:"+src] = fmt.Sprintf(
"the container runtime could not say whether the volume %s is here: %v", src, err)
}
}
}
}
}
return seen
}
// unitFiles is a service manager that can say where it loads a unit from.
type unitFiles interface {
ServiceUnitFile(ctx context.Context, run Runner, unit string) (string, error)
}
// installedByHand is whether a unit file is one somebody put on this machine rather than one a
// package ships: anywhere but /usr, where distributions keep what they install.
func installedByHand(path string) bool {
if path == "" {
return false
}
return !strings.HasPrefix(filepath.Clean(path), "/usr/")
}
func present(path string) bool {
_, err := os.Lstat(path)
return err == nil
}
// recordedPath is whether this host has a record of MAKING something at a path — a directory it
// created, a file it wrote, an archive it unpacked. An access record is not one of those: it says
// the mesh set permissions on a path it does not own, which is exactly what it does to a path
// somebody else's software made, so a path it only has access for is still found (novox/hq ADR 0103).
func recordedPath(known store.State, path string) bool {
for _, kind := range []declaration.Type{declaration.TypeDirectory, declaration.TypeFile,
declaration.TypeArchive} {
if known.Recorded(string(kind), path) {
return true
}
}
return false
}
// systemPath is whether a bind-mount source is the machine's own plumbing — the runtime's socket,
// the kernel's filesystems, the devices, the clock — which every machine has and no predecessor's
// data lives in. Mounting it shares nothing that was found.
func systemPath(src string) bool {
clean := filepath.Clean(src)
for _, exact := range []string{"/etc/localtime", "/etc/timezone", "/etc/hosts", "/etc/resolv.conf",
"/etc/machine-id", "/etc/passwd", "/etc/group"} {
if clean == exact {
return true
}
}
for _, under := range []string{"/run", "/var/run", "/sys", "/proc", "/dev", "/usr/share/zoneinfo",
"/etc/ssl", "/etc/ca-certificates", "/etc/pki", "/lib/modules", "/usr/lib/modules"} {
if clean == under || strings.HasPrefix(clean, under+"/") {
return true
}
}
return false
}
// mountSource is what a volume mapping mounts: a path on the machine, or a named volume. Empty for
// an anonymous volume, which mounts nothing that could already be there.
func mountSource(mapping string) string {
src, _, ok := strings.Cut(mapping, ":")
if !ok {
return ""
}
return src
}
// runsIn is the container a resource runs inside, if any: an action's `in`, or a run-once step
// sharing a container's namespace. An action with no `in` runs on the machine itself and is not
// held for a container: it reaches nothing a predecessor holds by running there, and holding every
// action of an untaken module would stop a module preparing itself before its cutover.
func runsIn(r declaration.Resource) string {
switch res := r.(type) {
case *declaration.Action:
return res.In
case *declaration.Container:
if res.RunOnce {
if name, ok := strings.CutPrefix(res.Network, "container:"); ok {
return name
}
}
}
return ""
}
// heldContainer is what is held under a container's name.
func heldContainer(known store.State, name string) (store.Held, bool) {
for _, h := range known.Held {
if h.Kind == string(declaration.TypeContainer) && h.Target == name {
return h, true
}
}
return store.Held{}, false
}
// replacesNothing is a resource that takes nothing found on the machine from it, so on an adopted
// node it is never held and never previewed as replacing what was found: a file written into
// (novox/hq ADR 0102), and a service whose unit's lifecycle is the machine's (novox/hq ADR 0117).
func replacesNothing(r declaration.Resource) bool {
switch res := r.(type) {
case *declaration.File:
return res.Into != ""
case *declaration.Service:
return res.Stateless()
}
return false
}
// holdOnAdopted decides whether a resource of an adopted node is held rather than applied, and
// holds it (novox/hq ADR 0100, ADR 0103). For a module not yet taken, what is present with no
// record is kept as it is: a file or a container under its name, a directory, a service's unit,
// and a container that would mount a path or a volume found there. An action or a run-once step
// run inside a held container is held with it. Once held, a resource stays held — changed or gone
// — until its module is taken, and it is never recorded as applied, so never removed as an orphan.
//
// Held is false for a resource to apply as usual. News is whether the hold is new or changed,
// which is what is worth a line in the log.
func holdOnAdopted(ctx context.Context, sys system.System, r declaration.Resource, d *declaration.Declaration,
known *store.State, before foundBefore, run Runner, keep Keep, changed map[string]bool,
now time.Time) (held, news bool, out Outcome, err error) {
was, already := known.HeldAt(r.Identity())
if in := runsIn(r); in != "" {
if container, isHeld := heldContainer(*known, in); isHeld {
module, untaken := d.Adoption.UntakenModuleOf(r.Identity())
if !untaken {
module = container.Module
}
h := was
if !already {
h = store.Held{ID: r.Identity(), Kind: string(r.Kind()), Target: r.Target(), Since: now}
}
h.Module, h.Why = module, "runs in "+in
known.RecordHeld(h)
out = begin(r)
out.Action = "held"
out.Detail = fmt.Sprintf("runs in %s, which is held as found; not run until %s is taken", in, module)
return true, !already, out, nil
}
}
// A file written into replaces nothing that was found, so it is never held (novox/hq ADR
// 0102) — and a hold from when it was declared whole must not keep the mesh's keys out. That
// hold is released by the apply once the write has worked, not here: a write that fails keeps
// it, and with it where the original was kept. A service whose lifecycle is the machine's
// replaces nothing either (novox/hq ADR 0117).
if replacesNothing(r) {
return false, false, out, nil
}
module, untaken := d.Adoption.UntakenModuleOf(r.Identity())
if !untaken {
return false, false, out, nil
}
why := was.Why
isFound := already
if !already {
switch res := r.(type) {
case *declaration.File:
if res.Into == "" {
if isFound, err = found(ctx, r, run, *known); err != nil {
return false, false, begin(r), err
}
}
case *declaration.Container:
if known.Recorded(string(declaration.TypeContainer), res.Name) {
break
}
_, exists, err := inspectFound(ctx, res.Name, run)
if err != nil {
return false, false, begin(r), err
}
if exists {
if isFound, err = found(ctx, r, run, *known); err != nil {
return false, false, begin(r), err
}
break
}
// Not there under its name, and still it would share what was found: created, it
// would mount the predecessor's data beside the predecessor's own container.
for _, v := range res.Volumes {
src := mountSource(v)
key := "volume:" + src
if strings.HasPrefix(src, "/") {
key = "path:" + src
}
if src == "" {
continue
}
if trouble := before.why(key); trouble != "" {
return false, false, begin(r), fmt.Errorf(
"%s, so it is not safe to create a container that would mount it", trouble)
}
if before.has(key) {
isFound, why = true, "would mount "+src+", found on the machine"
break
}
}
case *declaration.Directory:
isFound = before.has("path:" + res.Path)
case *declaration.Archive:
isFound = before.has("path:" + res.Path)
case *declaration.Process:
isFound = before.has("unit-file:" + res.Name)
case *declaration.User:
isFound = before.has("user:" + res.Name)
case *declaration.Service:
isFound = before.has("unit:" + res.Unit)
}
}
if !isFound {
return false, false, out, nil
}
out, h, err := hold(ctx, sys, r, module, was, already, why, run, keep, now)
if err != nil {
return true, false, out, err
}
// A held service is not started, stopped, enabled or restarted — but a reload stops nothing,
// so one the module names still happens (novox/hq ADR 0102, ADR 0103).
if svc, ok := r.(*declaration.Service); ok && svc.State == "running" {
if which := restartedBy(svc.ReloadOn, changed); len(which) > 0 {
if state, err := sys.ServiceState(ctx, run, svc.Unit); err == nil && state == "running" {
reloader, can := sys.(serviceReloader)
if !can {
return true, false, out, fmt.Errorf("%s must be reloaded for %s and this machine's "+
"service manager cannot reload a unit", svc.Unit, strings.Join(which, ", "))
}
if err := reloader.ReloadService(ctx, run, svc.Unit); err != nil {
return true, false, out, fmt.Errorf("reloading the held %s: %w", svc.Unit, err)
}
out.Detail += "; reloaded for " + strings.Join(which, ", ") + ", which stops nothing"
}
}
}
known.RecordHeld(h)
return true, !already || h.Changed != was.Changed, out, nil
}
// found is whether a declared file or container is present on the machine with no record of this
// host making it (novox/hq ADR 0100). A container carrying the host's own spec label was made by
// a host, whatever this store says, so it is never found.
func found(ctx context.Context, r declaration.Resource, run Runner, known store.State) (bool, error) {
if known.Recorded(string(r.Kind()), r.Target()) {
return false, nil
}
switch res := r.(type) {
case *declaration.File:
_, err := os.Lstat(res.Path)
if errors.Is(err, os.ErrNotExist) {
return false, nil
}
return err == nil, err
case *declaration.Container:
seen, exists, err := inspectFound(ctx, res.Name, run)
if err != nil || !exists {
return false, err
}
return seen.spec == "", nil
}
return false, nil
}
type foundContainer struct {
id string
running bool
spec string
// What a take compares (novox/hq ADR 0163): the image and its id, the networks the container
// is on, its mounts and its published ports — empty from a runtime (or a test's fake) that
// answers the short form.
image string
imageID string
networks []string
mounts []string
ports []string
}
// foundFormat is what inspectFound asks the runtime for, tab-separated: the three a hold has
// always needed, then the facts a take compares.
const foundFormat = "{{.Id}}\t{{.State.Running}}\t{{index .Config.Labels \"" + specLabel + "\"}}" +
"\t{{.Config.Image}}\t{{.Image}}" +
"\t{{range $k, $v := .NetworkSettings.Networks}}{{$k}},{{end}}" +
"\t{{range .Mounts}}{{.Source}}:{{.Destination}},{{end}}" +
"\t{{range $p, $b := .NetworkSettings.Ports}}{{$p}}{{range $b}}>{{.HostIp}}:{{.HostPort}}{{end}},{{end}}"
func splitList(s string) []string {
var out []string
for _, part := range strings.Split(s, ",") {
if part = strings.TrimSpace(part); part != "" {
out = append(out, part)
}
}
return out
}
// inspectFound reads a container by name the way a hold needs it: its id, whether it runs, and
// whether a host made it.
//
// **`container inspect`, not the bare form.** A name is not unique across object kinds — a
// module regularly names a network the same as the container that joins it (`keycloak` names
// both, and it is ordinary). The bare form resolves across every kind and returns whichever it
// finds, so a container that does not exist yet but a same-named network does answers with the
// network's JSON — no `.State` field at all — and the template below fails to execute rather
// than failing to find anything. That reads as "the runtime could not say", which this function's
// caller correctly refuses to build on (novox/hq ADR 0100) — but there was something to say, a
// question of kind, not of ambiguity that should have stopped anything.
func inspectFound(ctx context.Context, name string, run Runner) (foundContainer, bool, error) {
cri, err := containerRuntime(ctx, run)
if err != nil {
return foundContainer{}, false, fmt.Errorf("%w, so nothing can be said about %q", err, name)
}
out, err := run(ctx, cri, "container", "inspect", "--format", foundFormat, name)
if err != nil {
if absent(err) {
return foundContainer{}, false, nil
}
// **A runtime that could not answer is not a machine with nothing there.** Read as
// absence, a daemon that is down or a permission denied would let the mesh create its own
// container over a predecessor's — the one thing an adopted node must never do
// (novox/hq ADR 0100).
return foundContainer{}, false, fmt.Errorf(
"the container runtime could not say whether %s is here, so it is not safe to make one: %w",
name, err)
}
parts := strings.Split(strings.TrimSpace(out), "\t")
for len(parts) < 8 {
parts = append(parts, "")
}
spec := strings.TrimSpace(parts[2])
if spec == "<no value>" {
spec = ""
}
return foundContainer{id: strings.TrimSpace(parts[0]), running: parts[1] == "true", spec: spec,
image: strings.TrimSpace(parts[3]), imageID: strings.TrimSpace(parts[4]),
networks: splitList(parts[5]), mounts: splitList(parts[6]), ports: splitList(parts[7])}, true, nil
}
// factsOf is what a take would compare for a found container (novox/hq ADR 0163): the found
// image and when it was made, the networks and who else is on them, mounts and ports — beside
// what the module declares, and the declared image's date when that image is on the machine.
// Every question the runtime cannot answer leaves its fact empty; a preview says so rather than
// guesses.
func factsOf(ctx context.Context, seen foundContainer, res *declaration.Container, run Runner) *Facts {
cri, err := containerRuntime(ctx, run)
if err != nil {
return nil
}
f := &store.Facts{Image: seen.image, Mounts: seen.mounts, Ports: seen.ports,
DeclaredImage: res.Image, DeclaredPorts: res.Ports, DeclaredVolumes: res.Volumes}
if seen.imageID != "" {
if out, err := run(ctx, cri, "image", "inspect", "--format", "{{.Created}}", seen.imageID); err == nil {
f.ImageCreated = strings.TrimSpace(out)
}
}
if res.Image != "" {
if out, err := run(ctx, cri, "image", "inspect", "--format", "{{.Created}}", res.Image); err == nil {
f.DeclaredImageCreated = strings.TrimSpace(out)
}
}
if found, err := time.Parse(time.RFC3339Nano, f.ImageCreated); err == nil {
if declared, err := time.Parse(time.RFC3339Nano, f.DeclaredImageCreated); err == nil {
f.Downgrade = declared.Before(found)
}
}
for _, network := range seen.networks {
if f.Networks == nil {
f.Networks = map[string][]string{}
}
var members []string
if out, err := run(ctx, cri, "network", "inspect", "--format",
"{{range .Containers}}{{.Name}},{{end}}", network); err == nil {
for _, m := range splitList(out) {
if m != res.Name {
members = append(members, m)
}
}
}
sort.Strings(members)
f.Networks[network] = members
}
return (*Facts)(f)
}
// Facts is store.Facts, named here so hold's callers read as one vocabulary.
type Facts = store.Facts
// differenceOf is how a found file differs from the declared content: the lines only the found
// file has, marked -, then the lines only the declared content has, marked +, in their own order,
// bounded so a report stays a report. Not a diff tool's output: the question a take answers is
// "what would be lost and what would be new", and that is these two lists.
func differenceOf(found, declared string) (bool, []string) {
if found == declared {
return false, nil
}
const bound = 40
count := func(s string) map[string]int {
out := map[string]int{}
for _, line := range strings.Split(s, "\n") {
out[line]++
}
return out
}
inFound, inDeclared := count(found), count(declared)
var out []string
add := func(mark, s string, other map[string]int) {
seen := map[string]int{}
for _, line := range strings.Split(s, "\n") {
seen[line]++
if seen[line] > other[line] && len(out) < bound {
out = append(out, mark+" "+line)
}
}
}
add("-", found, inDeclared)
add("+", declared, inFound)
if len(out) >= bound {
out = append(out, "… and more")
}
return true, out
}
// absent is whether a runtime said the thing is not there, rather than failing to answer. Its own
// words: docker and podman both say "No such object", "No such container" or "No such volume".
func absent(err error) bool {
said := strings.ToLower(err.Error())
for _, missing := range []string{"no such object", "no such container", "no such volume",
"no such image"} {
if strings.Contains(said, missing) {
return true
}
}
return false
}
// hold keeps a found file or container as it is, and reports it — the first time by recording
// what was found, every time after by comparing against that. Nothing is reverted, restarted or
// created: a held target that disappears stays held and gone until its module is taken.
func hold(ctx context.Context, sys system.System, r declaration.Resource, module string, was store.Held,
already bool, why string, run Runner, keep Keep, now time.Time) (Outcome, store.Held, error) {
out := begin(r)
h := was
if !already {
h = store.Held{ID: r.Identity(), Module: module, Kind: string(r.Kind()),
Target: r.Target(), Since: now, Why: why}
}
h.Module = module
detail := "found on the machine; kept until " + module + " is taken"
var changed string
switch res := r.(type) {
case *declaration.File:
info, err := os.Lstat(res.Path)
switch {
case errors.Is(err, os.ErrNotExist):
if !already {
return out, h, fmt.Errorf("%s was found and is gone before it could be kept", res.Path)
}
changed = "gone"
case err != nil:
return out, h, err
default:
content, err := os.ReadFile(res.Path)
if err != nil {
return out, h, fmt.Errorf("%s was found and cannot be read to keep it: %w", res.Path, err)
}
if !already {
// The original first, before anything is recorded: a hold with no kept copy
// would be a promise the host cannot keep.
if keep == nil {
return out, h, fmt.Errorf(
"%s was found on this adopted node and this host has nowhere to keep its original", res.Path)
}
kept, err := keep(res.Path, content, info.Mode().Perm())
if err != nil {
return out, h, fmt.Errorf("keeping the original of %s: %w", res.Path, err)
}
h.Kept = kept
h.Digest = digestOf(string(content))
h.Mode = fmt.Sprintf("%04o", info.Mode().Perm())
if st, ok := info.Sys().(*syscall.Stat_t); ok {
h.Owner = fmt.Sprintf("%d:%d", st.Uid, st.Gid)
}
} else if digestOf(string(content)) != h.Digest {
changed = "rewritten"
}
// What a take would replace it with, and how that differs (novox/hq ADR 0163): a
// file declared whole is compared whole; one written into is not replaced at all.
if res.Into == "" {
differs, lines := differenceOf(string(content), res.Content)
h.Facts = &Facts{Differs: differs, Difference: lines}
}
}
case *declaration.Directory:
info, err := os.Lstat(res.Path)
switch {
case errors.Is(err, os.ErrNotExist):
if !already {
return out, h, fmt.Errorf("%s was found and is gone before it could be held", res.Path)
}
changed = "gone"
case err != nil:
return out, h, err
case !already:
// Its mode and owner as found, which the mesh leaves: a database refuses to start
// on a data directory whose mode changed.
h.Mode = fmt.Sprintf("%04o", info.Mode().Perm())
if st, ok := info.Sys().(*syscall.Stat_t); ok {
h.Owner = fmt.Sprintf("%d:%d", st.Uid, st.Gid)
}
}
detail = "found on the machine; its mode, owner and contents kept until " + module + " is taken"
case *declaration.Archive:
if _, err := os.Lstat(res.Path); errors.Is(err, os.ErrNotExist) {
if !already {
return out, h, fmt.Errorf("%s was found and is gone before it could be held", res.Path)
}
changed = "gone"
} else if err != nil {
return out, h, err
}
detail = "something is already at " + res.Path + "; nothing unpacked over it or re-owned until " +
module + " is taken"
case *declaration.Process:
unit := filepath.Join(unitDir, res.Name+".service")
if !present(unit) {
if !already {
return out, h, fmt.Errorf("%s was found and is gone before it could be held", unit)
}
changed = "gone"
}
detail = "its unit " + unit + " was found on the machine; not written over or restarted until " +
module + " is taken"
case *declaration.User:
_, exists, err := system.LookUpUser(ctx, run, res.Name)
switch {
case err != nil:
return out, h, err
case !exists && !already:
return out, h, fmt.Errorf("the user %s was found and is gone before it could be held", res.Name)
case !exists:
changed = "gone"
}
detail = "the user was found on the machine; its shell and groups are kept until " + module + " is taken"
case *declaration.Service:
state, err := sys.ServiceState(ctx, run, res.Unit)
switch {
case err != nil && !already:
return out, h, fmt.Errorf("the unit %s was found and cannot be read to hold it: %w", res.Unit, err)
case err != nil:
changed = "gone"
case !already:
h.Running = state == "running"
case h.Running && state != "running":
changed = "stopped"
}
detail = "its unit was found on the machine; its state and whether it starts at boot are " +
"kept until " + module + " is taken"
case *declaration.Container:
if h.Why != "" && h.Container == "" {
// Held for what it would mount, never created: there is nothing of it to compare.
detail = "not created: it " + h.Why + "; kept until " + module + " is taken"
break
}
seen, exists, err := inspectFound(ctx, res.Name, run)
if err != nil {
return out, h, err
}
switch {
case !exists && !already:
return out, h, fmt.Errorf("container %s was found and is gone before it could be held", res.Name)
case !already:
h.Container, h.Running = seen.id, seen.running
case !exists:
changed = "gone"
case seen.id != h.Container:
changed = "replaced"
case h.Running && !seen.running:
changed = "stopped"
}
if exists {
h.Facts = factsOf(ctx, seen, res, run)
}
default:
return out, h, fmt.Errorf("a %s cannot be held", r.Kind())
}
if changed != h.Changed {
h.Changed = changed
h.ChangedAt = now
if changed == "" {
h.ChangedAt = time.Time{}
}
}
out.Action = "held"
out.Detail = detail
if h.Changed != "" {
out.Detail += "; " + h.Changed + " by something other than the mesh since it was found, and not reverted"
}
return out, h, nil
}
// takenDetail is what an outcome says when a module's cutover replaced what was held for it.
func takenDetail(h store.Held) string {
if h.Kind == string(declaration.TypeAction) || (h.Why != "" && h.Container == "") {
return "taken: no longer held (" + h.Why + ")"
}
if h.Kept != "" {
return "taken: replaced what was found; original kept at " + h.Kept
}
return "taken: replaced what was found"
}
File diff suppressed because it is too large Load Diff
+66
View File
@@ -0,0 +1,66 @@
package apply
import (
"context"
"errors"
"testing"
)
// A name is not unique across object kinds: a module regularly names a network the same as the
// container that joins it (keycloak does this today, ordinarily). `docker inspect <name>`, unlike
// `docker container inspect <name>`, resolves across every kind — so when the container does not
// exist yet but a same-named network does, the bare form answers with the network's JSON instead
// of reporting the container absent. containerState and inspectFound must ask by kind, or a
// same-named network makes them unable to tell "not here yet" from "the runtime is broken"
// (novox/hq ADR 0100's refusal, tripped by nothing wrong).
//
// dockerLikeByKind is Docker's real behaviour, not the bug: `container inspect` only ever
// answers from the container namespace. A fake that also answered the bare, unscoped form would
// not catch a regression back to it — this one refuses to, on purpose.
func dockerLikeByKind(containers map[string]bool) func(context.Context, string, ...string) (string, error) {
return func(_ context.Context, name string, args ...string) (string, error) {
if name != "docker" {
return "", errors.New("unexpected program: " + name)
}
if len(args) > 0 && args[0] == "info" {
// containerRuntime's probe, answered so inspectFound gets past it to the check under
// test.
return "27.0\n", nil
}
if len(args) < 2 || args[0] != "container" || args[1] != "inspect" {
return "", errors.New("unexpected command: only `docker container inspect` is modelled here")
}
target := args[len(args)-1]
if !containers[target] {
return "", errors.New("Error: No such container: " + target)
}
return "false\t\n", nil
}
}
func TestContainerStateAsksTheContainerNamespaceNotTheBareForm(t *testing.T) {
// "minio" exists only as a network in this scenario — never in `containers` — matching the
// live failure this guards: a module's network and its container share a name, and the
// container does not exist yet.
run := dockerLikeByKind(map[string]bool{"keycloak": true})
if _, err := containerState(context.Background(), "minio", run); err == nil {
t.Fatal("a container that does not exist should report absent, not be mistaken for found")
}
if state, err := containerState(context.Background(), "keycloak", run); err != nil {
t.Fatalf("a container that does exist should be found: %v", err)
} else if state.Running {
t.Errorf("the fake said not running; containerState disagreed: %+v", state)
}
}
func TestInspectFoundAsksTheContainerNamespaceNotTheBareForm(t *testing.T) {
run := dockerLikeByKind(map[string]bool{"keycloak": true})
if _, exists, err := inspectFound(context.Background(), "minio", run); err != nil || exists {
t.Fatalf("a container that does not exist should be reported absent cleanly, not refused: exists=%v err=%v", exists, err)
}
if _, exists, err := inspectFound(context.Background(), "keycloak", run); err != nil || !exists {
t.Fatalf("a container that does exist should be found: exists=%v err=%v", exists, err)
}
}
+388
View File
@@ -0,0 +1,388 @@
package apply
import (
"bytes"
"encoding/json"
"errors"
"fmt"
"os"
"path/filepath"
"slices"
"sort"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/store"
)
// A file written into, never over (novox/hq ADR 0102).
//
// **The file is the machine's; the mesh owns keys in it.** The container runtime's configuration
// is the case that needed it: the mesh states one fact there — its registry is trusted over the
// private network — and writing the file whole replaced everything the machine had set, down to
// where the runtime keeps its data. So the host reads what is there, sets only the declared keys,
// keeps every other key as it found it, and records what each of its keys held before. Undeclared,
// each key goes back, and a file the mesh created goes only if nothing but its keys is left.
// applyInto writes a file's declared keys into the object already at its path.
func applyInto(r *declaration.File, previous store.Applied) (Outcome, error) {
out := begin(r)
if r.Into != declaration.IntoJSON {
return out, fmt.Errorf("%s: into %q is not a format this host writes into", r.Path, r.Into)
}
var declared map[string]json.RawMessage
if err := json.Unmarshal([]byte(r.Content), &declared); err != nil {
return out, fmt.Errorf("%s: the keys to write are not a JSON object: %w", r.Path, err)
}
existing, err := os.ReadFile(r.Path)
existed := err == nil
if err != nil && !errors.Is(err, os.ErrNotExist) {
return out, err
}
object := map[string]json.RawMessage{}
if existed && len(bytes.TrimSpace(existing)) > 0 {
if err := json.Unmarshal(existing, &object); err != nil || object == nil {
// Refused, never replaced: a file the host cannot read as an object is a file it
// cannot write into without losing whatever it is.
return out, fmt.Errorf("%s is not a JSON object, so the mesh cannot write its keys into it "+
"without replacing what is there; it was left as it is", r.Path)
}
}
rec := store.Into{Format: declaration.IntoJSON, Before: map[string]json.RawMessage{},
Added: map[string][]json.RawMessage{}}
if previous.Into != nil {
rec.Created = previous.Into.Created
for k, v := range previous.Into.Before {
rec.Before[k] = v
}
rec.Absent = slices.Clone(previous.Into.Absent)
for k, v := range previous.Into.Added {
rec.Added[k] = slices.Clone(v)
}
} else {
rec.Created = !existed
}
tracked := func(k string) bool {
_, before := rec.Before[k]
_, added := rec.Added[k]
return before || added || slices.Contains(rec.Absent, k)
}
// Drift: the machine no longer holds what this host last set in its keys.
drifted := previous.Wrote != "" && existed && digestOf(viewOf(object, rec, keysTracked(rec))) != previous.Wrote
// Keys the mesh set before and no longer declares go back to what they held.
for _, k := range keysTracked(rec) {
if _, still := declared[k]; still {
continue
}
giveBack(object, &rec, k)
}
// Declared keys: remember what each held the first time, then set it. A list is the
// machine's too — a predecessor's own trusted registries, say — so the mesh adds its members
// to it rather than replacing it, and remembers exactly which it added.
for _, k := range keysIn(declared) {
_, scalar := rec.Before[k]
if isList(declared[k]) && !scalar {
current, had := object[k]
if had && !isList(current) {
return out, fmt.Errorf("%s: the mesh adds to the list %q, and the machine holds something "+
"other than a list there; it was left as it is", r.Path, k)
}
if !tracked(k) && !had {
rec.Absent = append(rec.Absent, k)
}
merged, added, err := addMembers(current, declared[k], rec.Added[k], previous.Into == nil)
if err != nil {
return out, fmt.Errorf("%s: %q: %w", r.Path, k, err)
}
rec.Added[k] = added
object[k] = merged
continue
}
if !tracked(k) {
v, had := object[k]
// **What the host may have written itself is not the machine's.** With no record of
// this file — the first apply, or a host that wrote and died before saving its state —
// a key already holding exactly what the mesh declares cannot be told from one the
// mesh set a moment ago. Remembered as the machine's, it would never be given back:
// undeclaring would leave the mesh's own value behind for ever. So it is the mesh's,
// and undeclaring takes it out (novox/hq ADR 0102).
if had && !(previous.Into == nil && canonical(v) == canonical(declared[k])) {
rec.Before[k] = v
} else {
rec.Absent = append(rec.Absent, k)
}
}
object[k] = declared[k]
}
want, err := render(object)
if err != nil {
return out, err
}
same := existed && canonical(existing) == canonical(want)
if !same {
mode := os.FileMode(0o644)
if info, err := os.Stat(r.Path); err == nil {
mode = info.Mode().Perm() // the machine's file keeps the machine's mode
} else if r.Mode != "" {
if m, err := modeOf(r.Mode, mode); err == nil {
mode = m
}
}
if err := os.MkdirAll(filepath.Dir(r.Path), 0o755); err != nil {
return out, err
}
if err := writeAtomically(r.Path, want, mode); err != nil {
return out, err
}
}
// Read back: every declared key holds what was declared.
written, err := os.ReadFile(r.Path)
if err != nil {
return out, fmt.Errorf("wrote into %s and cannot read it back: %w", r.Path, err)
}
var check map[string]json.RawMessage
if err := json.Unmarshal(written, &check); err != nil {
return out, fmt.Errorf("%s is not a JSON object after writing into it: %w", r.Path, err)
}
for k, v := range declared {
if _, list := rec.Added[k]; list {
members, _ := membersOf(v)
have, err := membersOf(check[k])
if err != nil {
return out, fmt.Errorf("%s does not hold a list at %q after writing into it", r.Path, k)
}
for _, m := range members {
if !hasMember(have, m) {
return out, fmt.Errorf("%s does not hold the declared %s in %q after writing into it", r.Path, m, k)
}
}
continue
}
if canonical(check[k]) != canonical(v) {
return out, fmt.Errorf("%s does not hold the declared %q after writing into it", r.Path, k)
}
}
if len(rec.Before) == 0 {
rec.Before = nil
}
if len(rec.Added) == 0 {
rec.Added = nil
}
out.into = &rec
out.wrote = digestOf(viewOf(check, rec, keysIn(declared)))
switch {
case !existed:
out.Action = "created"
out.Detail = "written into; the file was not there"
case same:
out.Action = "unchanged"
case drifted:
out.Action = "corrected"
out.Detail = "the mesh's keys had been changed on the machine; the rest of the file was kept"
default:
out.Action = "updated"
out.Detail = "the mesh's keys written in; every other key kept as it was"
}
return out, nil
}
// removeInto gives back what a file written into held before the mesh's keys.
func removeInto(a store.Applied) (string, string, error) {
existing, err := os.ReadFile(a.Target)
if errors.Is(err, os.ErrNotExist) {
return "forgotten", "no longer there", nil
}
if err != nil {
return "", "", err
}
object := map[string]json.RawMessage{}
if len(bytes.TrimSpace(existing)) > 0 {
if err := json.Unmarshal(existing, &object); err != nil || object == nil {
return "kept", "no longer a JSON object, so the mesh's keys were left in it; " +
"remove them by hand", nil
}
}
rec := *a.Into
for _, k := range keysTracked(rec) {
giveBack(object, &rec, k)
}
if a.Into.Created && len(object) == 0 {
if err := os.Remove(a.Target); err != nil {
return "", "", err
}
return "removed", "no longer declared; the mesh had created it and nothing else was in it", nil
}
want, err := render(object)
if err != nil {
return "", "", err
}
info, err := os.Stat(a.Target)
if err != nil {
return "", "", err
}
if err := writeAtomically(a.Target, want, info.Mode().Perm()); err != nil {
return "", "", err
}
return "restored", "no longer declared; the mesh's keys were given back what they held", nil
}
func giveBack(object map[string]json.RawMessage, rec *store.Into, k string) {
if added, list := rec.Added[k]; list {
// Only the members the mesh added go; the list and everything else in it stay, unless
// the mesh made the key and nothing is left in it.
wasAbsent := slices.Contains(rec.Absent, k)
if current, had := object[k]; had && isList(current) {
have, _ := membersOf(current)
have = slices.DeleteFunc(have, func(m json.RawMessage) bool { return hasMember(added, m) })
if len(have) == 0 && wasAbsent {
delete(object, k)
} else {
object[k] = listOf(have)
}
}
delete(rec.Added, k)
rec.Absent = slices.DeleteFunc(rec.Absent, func(a string) bool { return a == k })
return
}
if v, had := rec.Before[k]; had {
object[k] = v
delete(rec.Before, k)
return
}
delete(object, k)
rec.Absent = slices.DeleteFunc(rec.Absent, func(a string) bool { return a == k })
}
func keysTracked(rec store.Into) []string {
var keys []string
for k := range rec.Before {
keys = append(keys, k)
}
for k := range rec.Added {
keys = append(keys, k)
}
keys = append(keys, rec.Absent...)
sort.Strings(keys)
return slices.Compact(keys)
}
// viewOf is what the mesh holds itself to in a file written into: each scalar key's value, and for
// a list only whether each member the mesh added is still there — what the machine keeps beside
// them is not the mesh's to judge.
func viewOf(object map[string]json.RawMessage, rec store.Into, keys []string) string {
var b bytes.Buffer
for _, k := range keys {
if added, list := rec.Added[k]; list {
have, _ := membersOf(object[k])
b.WriteString(k + " holds")
for _, m := range added {
fmt.Fprintf(&b, " %s=%v", canonical(m), hasMember(have, m))
}
b.WriteString("\n")
continue
}
b.WriteString(k + "=" + canonical(object[k]) + "\n")
}
return b.String()
}
func isList(raw json.RawMessage) bool {
t := bytes.TrimSpace(raw)
return len(t) > 0 && t[0] == '['
}
func membersOf(raw json.RawMessage) ([]json.RawMessage, error) {
if len(bytes.TrimSpace(raw)) == 0 {
return nil, nil
}
var members []json.RawMessage
if err := json.Unmarshal(raw, &members); err != nil {
return nil, err
}
return members, nil
}
func hasMember(list []json.RawMessage, m json.RawMessage) bool {
for _, have := range list {
if canonical(have) == canonical(m) {
return true
}
}
return false
}
func listOf(members []json.RawMessage) json.RawMessage {
if members == nil {
members = []json.RawMessage{}
}
raw, _ := json.Marshal(members)
return raw
}
// addMembers adds the declared members to the machine's list, dropping only members the mesh
// added before and no longer declares. It returns the list and exactly which members the mesh
// added — a declared member the machine already had is the machine's, and is never recorded.
// unrecorded says there is no record of this file yet, in which case a declared member already in
// the list may be one the host itself wrote before it could save its state, and is taken as the
// mesh's.
func addMembers(current, declared json.RawMessage, addedBefore []json.RawMessage,
unrecorded bool) (json.RawMessage, []json.RawMessage, error) {
have, err := membersOf(current)
if err != nil {
return nil, nil, err
}
want, err := membersOf(declared)
if err != nil {
return nil, nil, err
}
added := []json.RawMessage{}
for _, a := range addedBefore {
if hasMember(want, a) {
added = append(added, a)
continue
}
have = slices.DeleteFunc(have, func(m json.RawMessage) bool { return canonical(m) == canonical(a) })
}
for _, m := range want {
if !hasMember(have, m) {
have = append(have, m)
} else if !unrecorded {
continue // the machine's own, and never the mesh's to take out
}
if !hasMember(added, m) {
added = append(added, m)
}
}
return listOf(have), added, nil
}
func keysIn(m map[string]json.RawMessage) []string {
keys := make([]string, 0, len(m))
for k := range m {
keys = append(keys, k)
}
sort.Strings(keys)
return keys
}
// canonical is a JSON value compacted, so formatting is not mistaken for a change.
func canonical(raw []byte) string {
var b bytes.Buffer
if err := json.Compact(&b, raw); err != nil {
return string(raw)
}
return b.String()
}
func render(object map[string]json.RawMessage) ([]byte, error) {
b, err := json.MarshalIndent(object, "", " ")
if err != nil {
return nil, err
}
return append(b, '\n'), nil
}
+342
View File
@@ -0,0 +1,342 @@
package apply
import (
"context"
"encoding/json"
"fmt"
"os"
"path/filepath"
"strings"
"testing"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/store"
)
// Defends novox/hq ADR 0102: a file the mesh shares with software it did not install is written
// into, never over, and a service that re-reads its configuration is reloaded, not restarted.
func intoDecl(t *testing.T, path, keys string) string {
t.Helper()
return fmt.Sprintf(`{"declaration":1,"resources":[
{"id":"networking.registry-trust","type":"file","path":%q,"into":"json","content":%q}
]}`, path, keys)
}
func readObject(t *testing.T, path string) map[string]any {
t.Helper()
raw, err := os.ReadFile(path)
if err != nil {
t.Fatal(err)
}
var o map[string]any
if err := json.Unmarshal(raw, &o); err != nil {
t.Fatalf("%s is not a JSON object: %v\n%s", path, err, raw)
}
return o
}
// The machine's own runtime settings, the way a predecessor leaves them.
const machinesOwn = `{"data-root":"/srv/docker","log-opts":{"max-size":"10m"},"insecure-registries":["192.0.2.7:5000"]}`
func TestWritingIntoKeepsEveryKeyTheMachineHad(t *testing.T) {
path := filepath.Join(t.TempDir(), "daemon.json")
if err := os.WriteFile(path, []byte(machinesOwn), 0o600); err != nil {
t.Fatal(err)
}
d := parse(t, intoDecl(t, path, `{"insecure-registries":["10.42.0.1:5000"]}`))
report, state, err := Apply(context.Background(), archHost(t), d, store.State{}, store.OriginDeclared, nil, nil, nil)
if err != nil {
t.Fatal(err)
}
o := readObject(t, path)
if o["data-root"] != "/srv/docker" {
t.Errorf("the machine's data directory was not kept: %v", o)
}
if fmt.Sprint(o["log-opts"]) != "map[max-size:10m]" {
t.Errorf("the machine's logging settings were not kept: %v", o)
}
if fmt.Sprint(o["insecure-registries"]) != "[192.0.2.7:5000 10.42.0.1:5000]" {
t.Errorf("the mesh's member was not added beside the machine's own: %v", o)
}
if info, _ := os.Stat(path); info.Mode().Perm() != 0o600 {
t.Errorf("the machine's file mode was changed to %o", info.Mode().Perm())
}
if got := report.Outcomes[0].Action; got != "updated" {
t.Errorf("writing into was reported as %q", got)
}
// Again, with nothing changed: nothing to do.
report, state, err = Apply(context.Background(), archHost(t), d, state, store.OriginDeclared, nil, nil, nil)
if err != nil {
t.Fatal(err)
}
if got := report.Outcomes[0].Action; got != "unchanged" {
t.Errorf("a second apply was %q", got)
}
// Undeclared: the key goes back to what the machine had, and the file stays.
empty := somethingElse(t)
report, _, err = Apply(context.Background(), archHost(t), empty, state, store.OriginDeclared, nil, nil, nil)
if err != nil {
t.Fatal(err)
}
o = readObject(t, path)
if fmt.Sprint(o["insecure-registries"]) != "[192.0.2.7:5000]" || o["data-root"] != "/srv/docker" {
t.Errorf("undeclaring did not give the machine back what it had: %v", o)
}
if got := report.Outcomes[0].Action; got != "restored" {
t.Errorf("undeclaring was reported as %q", got)
}
}
func TestAFileWrittenIntoThatWasNotThereIsRemovedWhenOnlyTheMeshsKeysAreLeft(t *testing.T) {
path := filepath.Join(t.TempDir(), "daemon.json")
d := parse(t, intoDecl(t, path, `{"insecure-registries":["10.42.0.1:5000"]}`))
report, state, err := Apply(context.Background(), archHost(t), d, store.State{}, store.OriginDeclared, nil, nil, nil)
if err != nil {
t.Fatal(err)
}
if got := report.Outcomes[0].Action; got != "created" {
t.Errorf("writing into a file that was not there was %q", got)
}
// Somebody else adds a key of their own: the file is no longer only the mesh's.
o := readObject(t, path)
o["debug"] = true
raw, _ := json.Marshal(o)
_ = os.WriteFile(path, raw, 0o644)
empty := somethingElse(t)
if _, _, err := Apply(context.Background(), archHost(t), empty, state, store.OriginDeclared, nil, nil, nil); err != nil {
t.Fatal(err)
}
o = readObject(t, path)
if _, still := o["insecure-registries"]; still || o["debug"] != true {
t.Errorf("undeclaring should remove the mesh's key and keep the other: %v", o)
}
// Without the other key, the file the mesh created goes.
path2 := filepath.Join(t.TempDir(), "daemon.json")
d2 := parse(t, intoDecl(t, path2, `{"insecure-registries":["10.42.0.1:5000"]}`))
_, state2, err := Apply(context.Background(), archHost(t), d2, store.State{}, store.OriginDeclared, nil, nil, nil)
if err != nil {
t.Fatal(err)
}
if _, _, err := Apply(context.Background(), archHost(t), empty, state2, store.OriginDeclared, nil, nil, nil); err != nil {
t.Fatal(err)
}
if _, err := os.Stat(path2); !os.IsNotExist(err) {
t.Errorf("a file the mesh created, holding only its keys, was left behind")
}
}
func TestAKeyNoLongerDeclaredGoesBackAndANewOneIsRemembered(t *testing.T) {
path := filepath.Join(t.TempDir(), "daemon.json")
_ = os.WriteFile(path, []byte(machinesOwn), 0o644)
first := parse(t, intoDecl(t, path, `{"insecure-registries":["10.42.0.1:5000"]}`))
_, state, err := Apply(context.Background(), archHost(t), first, store.State{}, store.OriginDeclared, nil, nil, nil)
if err != nil {
t.Fatal(err)
}
second := parse(t, intoDecl(t, path, `{"registry-mirrors":["http://10.42.0.1:5000"]}`))
if _, _, err := Apply(context.Background(), archHost(t), second, state, store.OriginDeclared, nil, nil, nil); err != nil {
t.Fatal(err)
}
o := readObject(t, path)
if fmt.Sprint(o["insecure-registries"]) != "[192.0.2.7:5000]" {
t.Errorf("a key the mesh stopped declaring was not given back: %v", o)
}
if fmt.Sprint(o["registry-mirrors"]) != "[http://10.42.0.1:5000]" {
t.Errorf("the newly declared key was not written: %v", o)
}
}
func TestAFileThatIsNotAnObjectIsRefusedAndLeftAlone(t *testing.T) {
path := filepath.Join(t.TempDir(), "daemon.json")
_ = os.WriteFile(path, []byte("# not json at all\n"), 0o644)
d := parse(t, intoDecl(t, path, `{"insecure-registries":["10.42.0.1:5000"]}`))
if _, _, err := Apply(context.Background(), archHost(t), d, store.State{}, store.OriginDeclared, nil, nil, nil); err == nil {
t.Fatal("writing into a file that is not a JSON object was not refused")
}
raw, _ := os.ReadFile(path)
if string(raw) != "# not json at all\n" {
t.Errorf("a file the mesh could not write into was changed: %q", raw)
}
}
func TestAFileWrittenIntoIsNeverHeldOnAnAdoptedNode(t *testing.T) {
path := filepath.Join(t.TempDir(), "daemon.json")
_ = os.WriteFile(path, []byte(machinesOwn), 0o644)
d := adopted(t, `{"taken":[],"untaken":{"networking":["networking.registry-trust"]}}`,
fmt.Sprintf(`{"id":"networking.registry-trust","type":"file","path":%q,"into":"json","content":%q}`,
path, `{"insecure-registries":["10.42.0.1:5000"]}`))
m := &machine{}
report, state := applyAdopted(t, d, store.State{}, m, t.TempDir())
if got := outcomeOf(report, "networking.registry-trust").Action; got == "held" {
t.Fatal("a file written into was held, though it replaces nothing that was found")
}
if len(state.Held) != 0 {
t.Errorf("something was held: %+v", state.Held)
}
o := readObject(t, path)
if o["data-root"] != "/srv/docker" || fmt.Sprint(o["insecure-registries"]) != "[192.0.2.7:5000 10.42.0.1:5000]" {
t.Errorf("the adopted node's file was not written into: %v", o)
}
}
func TestAServiceIsReloadedNotRestartedForWhatItReloadsOn(t *testing.T) {
path := filepath.Join(t.TempDir(), "daemon.json")
d := parse(t, fmt.Sprintf(`{"declaration":1,"resources":[
{"id":"trust","type":"file","path":%q,"into":"json","content":%q},
{"id":"runtime","type":"service","unit":"docker.service","state":"running","reload-on":["trust"]}
]}`, path, `{"insecure-registries":["10.42.0.1:5000"]}`))
var commands []string
if _, _, err := Apply(context.Background(), archHost(t), d, store.State{}, store.OriginDeclared,
recordingServices(&commands), nil, nil); err != nil {
t.Fatal(err)
}
joined := strings.Join(commands, "\n")
if !strings.Contains(joined, "systemctl reload docker.service") {
t.Errorf("the runtime was not reloaded; commands were %v", commands)
}
if strings.Contains(joined, "stop docker.service") || strings.Contains(joined, "restart docker.service") {
t.Errorf("the runtime was stopped, which stops every container on the machine; commands were %v", commands)
}
}
// somethingElse is a declaration that no longer holds the file: only an unrelated directory.
func somethingElse(t *testing.T) *declaration.Declaration {
t.Helper()
return parse(t, fmt.Sprintf(`{"declaration":1,"resources":[
{"id":"other","type":"directory","path":%q}
]}`, filepath.Join(t.TempDir(), "other")))
}
func TestAListIsAddedToNeverReplaced(t *testing.T) {
// The predecessor's own trusted registries are kept; the mesh adds its own and, undeclared,
// takes back only what it added (novox/hq ADR 0102).
path := filepath.Join(t.TempDir(), "daemon.json")
_ = os.WriteFile(path, []byte(`{"insecure-registries":["192.0.2.7:5000","10.42.0.9:5000"]}`), 0o644)
// First the mesh's own member alone, so there is a record of this file.
first := parse(t, intoDecl(t, path, `{"insecure-registries":["10.42.0.1:5000"]}`))
_, state, err := Apply(context.Background(), archHost(t), first, store.State{}, store.OriginDeclared, nil, nil, nil)
if err != nil {
t.Fatal(err)
}
// Now 10.42.0.9 is declared too, and was already the machine's: it is never the mesh's to remove.
d := parse(t, intoDecl(t, path, `{"insecure-registries":["10.42.0.1:5000","10.42.0.9:5000"]}`))
_, state, err = Apply(context.Background(), archHost(t), d, state, store.OriginDeclared, nil, nil, nil)
if err != nil {
t.Fatal(err)
}
if got := fmt.Sprint(readObject(t, path)["insecure-registries"]); got != "[192.0.2.7:5000 10.42.0.9:5000 10.42.0.1:5000]" {
t.Fatalf("the list after writing into it: %s", got)
}
rec, _ := state.Find("networking.registry-trust")
if added := rec.Into.Added["insecure-registries"]; len(added) != 1 || canonical(added[0]) != `"10.42.0.1:5000"` {
t.Errorf("recorded as added: %s", added)
}
// The predecessor adds a member of its own: not the mesh's drift.
o := readObject(t, path)
o["insecure-registries"] = append(o["insecure-registries"].([]any), "198.51.100.3:5000")
raw, _ := json.Marshal(o)
_ = os.WriteFile(path, raw, 0o644)
report, state, err := Apply(context.Background(), archHost(t), d, state, store.OriginDeclared, nil, nil, nil)
if err != nil {
t.Fatal(err)
}
if got := report.Outcomes[0].Action; got != "unchanged" {
t.Errorf("a member the machine added was taken for drift: %q", got)
}
// Somebody takes the mesh's member out: that is drift, and it is put back.
o = readObject(t, path)
o["insecure-registries"] = []any{"192.0.2.7:5000", "10.42.0.9:5000", "198.51.100.3:5000"}
raw, _ = json.Marshal(o)
_ = os.WriteFile(path, raw, 0o644)
report, state, err = Apply(context.Background(), archHost(t), d, state, store.OriginDeclared, nil, nil, nil)
if err != nil {
t.Fatal(err)
}
if got := report.Outcomes[0].Action; got != "corrected" {
t.Errorf("the mesh's member removed by hand was %q", got)
}
// Undeclared: only the member the mesh added goes.
if _, _, err := Apply(context.Background(), archHost(t), somethingElse(t), state, store.OriginDeclared, nil, nil, nil); err != nil {
t.Fatal(err)
}
if got := fmt.Sprint(readObject(t, path)["insecure-registries"]); got != "[192.0.2.7:5000 10.42.0.9:5000 198.51.100.3:5000]" {
t.Errorf("undeclaring took more than the mesh added: %s", got)
}
}
func TestAListTheMeshCreatedGoesWhenEmptied(t *testing.T) {
path := filepath.Join(t.TempDir(), "daemon.json")
_ = os.WriteFile(path, []byte(`{"data-root":"/srv/docker"}`), 0o644)
d := parse(t, intoDecl(t, path, `{"insecure-registries":["10.42.0.1:5000"]}`))
_, state, err := Apply(context.Background(), archHost(t), d, store.State{}, store.OriginDeclared, nil, nil, nil)
if err != nil {
t.Fatal(err)
}
if _, _, err := Apply(context.Background(), archHost(t), somethingElse(t), state, store.OriginDeclared, nil, nil, nil); err != nil {
t.Fatal(err)
}
if o := readObject(t, path); fmt.Sprint(o) != "map[data-root:/srv/docker]" {
t.Errorf("the key the mesh created was not removed: %v", o)
}
}
func TestAHoldFromAWholeFileDoesNotKeepOutAnIntoWrite(t *testing.T) {
// Declared whole before, the runtime's file was held; declared into now, it is written into.
path := filepath.Join(t.TempDir(), "daemon.json")
_ = os.WriteFile(path, []byte(machinesOwn), 0o644)
known := store.State{Held: []store.Held{{ID: "networking.registry-trust", Module: "networking",
Kind: "file", Target: path}}}
d := adopted(t, `{"taken":[],"untaken":{"networking":["networking.registry-trust"]}}`,
fmt.Sprintf(`{"id":"networking.registry-trust","type":"file","path":%q,"into":"json","content":%q}`,
path, `{"insecure-registries":["10.42.0.1:5000"]}`))
report, state := applyAdopted(t, d, known, &machine{}, t.TempDir())
if got := outcomeOf(report, "networking.registry-trust").Action; got != "updated" {
t.Errorf("the file was %q, not written into", got)
}
if len(state.Held) != 0 {
t.Errorf("the old hold outlived the into declaration: %+v", state.Held)
}
if fmt.Sprint(readObject(t, path)["insecure-registries"]) != "[192.0.2.7:5000 10.42.0.1:5000]" {
t.Errorf("the mesh's member was not written in: %v", readObject(t, path))
}
}
func TestAWriteWithNoRecordOfItIsTheMeshsOwn(t *testing.T) {
// A host that wrote into the file and died before saving its state comes back with no record
// of it. What is there is then exactly what the mesh declares — and remembered as the
// machine's it would never be given back (novox/hq ADR 0102).
path := filepath.Join(t.TempDir(), "daemon.json")
_ = os.WriteFile(path, []byte(`{"data-root":"/srv/docker","insecure-registries":["192.0.2.7:5000"]}`), 0o644)
d := parse(t, intoDecl(t, path, `{"live-restore":true,"insecure-registries":["10.42.0.1:5000"]}`))
if _, _, err := Apply(context.Background(), archHost(t), d, store.State{}, store.OriginDeclared, nil, nil, nil); err != nil {
t.Fatal(err)
}
// The crash: the state was never saved, so the next apply knows nothing of this file.
_, state, err := Apply(context.Background(), archHost(t), d, store.State{}, store.OriginDeclared, nil, nil, nil)
if err != nil {
t.Fatal(err)
}
rec, _ := state.Find("networking.registry-trust")
if _, asTheMachines := rec.Into.Before["live-restore"]; asTheMachines {
t.Error("the mesh's own key was remembered as the machine's")
}
if _, _, err := Apply(context.Background(), archHost(t), somethingElse(t), state, store.OriginDeclared, nil, nil, nil); err != nil {
t.Fatal(err)
}
o := readObject(t, path)
if _, still := o["live-restore"]; still {
t.Errorf("undeclaring left the mesh's key behind: %v", o)
}
if fmt.Sprint(o["insecure-registries"]) != "[192.0.2.7:5000]" || o["data-root"] != "/srv/docker" {
t.Errorf("the machine did not get its file back: %v", o)
}
}
+214
View File
@@ -0,0 +1,214 @@
package apply
import (
"context"
"errors"
"strings"
"testing"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/store"
)
// A taken container keeps a found network by a per-machine setting (novox/hq ADR 0163, rule 4):
// joined once it runs, part of its spec, and refused when it cannot be joined.
func TestAContainerJoinsTheNetworksItKeeps(t *testing.T) {
var ran []string
connectFails := false
run := func(_ context.Context, name string, args ...string) (string, error) {
if name != "docker" {
return "", errors.New("not installed")
}
ran = append(ran, strings.Join(args, " "))
switch args[0] {
case "info":
return "29.0.0\n", nil
case "container":
if len(ran) > 2 {
return "true\t" + specOfLast, nil
}
return "false\t\n", errors.New("no such container")
case "run":
return "deadbeef\n", nil
case "network":
if connectFails {
return "", errors.New("network predecessor_default not found")
}
}
return "", nil
}
d := parseTrusted(t, `{"declaration":1,"resources":[
{"id":"app","type":"container","name":"app","image":"`+pinned+`",
"networks":["predecessor_default"]}
]}`)
specOfLast = containerSpec(d.Resources[0].(*declaration.Container), inputs{})
alone := *d.Resources[0].(*declaration.Container)
alone.Networks = nil
if specOfLast == containerSpec(&alone, inputs{}) {
t.Fatal("the kept network is not part of the container's spec: kept or let go, the container would be left alone")
}
report, _, err := Apply(context.Background(), archHost(t), d, store.State{}, store.OriginCarried, run, nil, nil)
if err != nil {
t.Fatal(err)
}
joined := false
for i, line := range ran {
if line == "network connect predecessor_default app" {
joined = true
if ran[i-1] != "container inspect --format {{.State.Running}}\t{{index .Config.Labels \""+specLabel+"\"}} app" &&
!strings.HasPrefix(ran[i-1], "container inspect") {
t.Errorf("joined before the container was read back as running: %v", ran)
}
}
}
if !joined || report.Outcomes[0].Action != "created" {
t.Fatalf("the container did not join the kept network: %v\n%+v", ran, report.Outcomes)
}
connectFails, ran = true, nil
if _, _, err := Apply(context.Background(), archHost(t), d, store.State{}, store.OriginCarried, run, nil, nil); err == nil ||
!strings.Contains(err.Error(), "could not join the kept network predecessor_default") {
t.Fatalf("a network that cannot be joined was passed over: %v", err)
}
}
var specOfLast string
// A module the mesh left out of a declaration is not a module the mesh removed (novox/hq ADR 0163,
// rule 6): what the host wrote for it stays, recorded and said; what it holds for it stays held.
// A module simply absent is removed as it always was.
func TestALeftOutModuleIsNeitherRemovedNorForgotten(t *testing.T) {
var removed []string
gone := map[string]bool{}
run := func(_ context.Context, name string, args ...string) (string, error) {
if name != "docker" {
return "", nil
}
switch args[0] {
case "info":
return "29.0.0\n", nil
case "rm":
removed = append(removed, args[len(args)-1])
gone[args[len(args)-1]] = true
case "container":
if gone[args[len(args)-1]] {
return "", errors.New("no such container")
}
return "true\tspec", nil
}
return "", nil
}
known := store.State{
Resources: []store.Applied{
{ID: "web.server", Type: "container", Target: "web", Origin: store.OriginDeclared},
{ID: "old.server", Type: "container", Target: "old", Origin: store.OriginDeclared},
},
Held: []store.Held{{ID: "web.page", Module: "web", Kind: "file", Target: "/srv/web/index.html"}},
}
d := parse(t, `{"declaration":1,"left_out":["web"],"resources":[
{"id":"notes.conf","type":"file","path":"`+t.TempDir()+`/notes.conf","content":"x"}
]}`)
report, state, err := Apply(context.Background(), archHost(t), d, known, store.OriginDeclared, run, nil, nil)
if err != nil {
t.Fatal(err)
}
if len(removed) != 1 || removed[0] != "old" {
t.Fatalf("removed %v; only the module that is absent goes", removed)
}
if _, kept := state.At("container", "web"); !kept {
t.Fatal("the left-out module's record was forgotten")
}
if _, held := state.HeldAt("web.page"); !held {
t.Fatal("the left-out module's hold was released")
}
said := false
for _, o := range report.Outcomes {
if o.ID == "web.server" && o.Action == "unchanged" && strings.Contains(o.Detail, "web was left out of this declaration by the mesh") {
said = true
}
if o.ID == "web.server" && o.Action != "unchanged" {
t.Errorf("the left-out module's container was %s", o.Action)
}
}
if !said {
t.Fatalf("keeping the left-out module's container was not said: %+v", report.Outcomes)
}
}
// A container's capabilities reach the runtime and are part of its spec (novox/hq ADR 0170).
func TestACapabilityReachesTheRuntimeAndTheSpec(t *testing.T) {
var ran []string
run := func(_ context.Context, name string, args ...string) (string, error) {
if name != "docker" {
return "", errors.New("not installed")
}
switch args[0] {
case "info":
return "29.0.0\n", nil
case "container":
return "false\t\n", errors.New("no such container")
case "run":
ran = args
return "deadbeef\n", nil
}
return "", nil
}
d := parseTrusted(t, `{"declaration":1,"resources":[
{"id":"fw","type":"container","name":"fw","image":"`+pinned+`","network":"host","capabilities":["NET_ADMIN"]}
]}`)
_, _, _ = Apply(context.Background(), archHost(t), d, store.State{}, store.OriginCarried, run, nil, nil)
granted := false
for i, a := range ran {
if a == "--cap-add" && i+1 < len(ran) && ran[i+1] == "NET_ADMIN" {
granted = true
}
}
if !granted {
t.Fatalf("the capability was not granted: %v", ran)
}
with := d.Resources[0].(*declaration.Container)
without := *with
without.Capabilities = nil
if containerSpec(with, inputs{}) == containerSpec(&without, inputs{}) {
t.Fatal("a capability is not part of the container's spec")
}
}
// A package may be declared absent (novox/hq ADR 0175): removed when it is installed, read back,
// left alone when it is not.
func TestAPackageDeclaredAbsentIsRemovedWhenPresentAndLeftWhenNot(t *testing.T) {
installed := true
var ran []string
run := func(_ context.Context, name string, args ...string) (string, error) {
ran = append(ran, name+" "+strings.Join(args, " "))
if name != "pacman" {
return "", nil
}
switch args[0] {
case "-Q":
if args[1] == "pacman" || installed {
return args[1] + " 1.0\n", nil
}
return "", errors.New("package not found")
case "-R":
installed = false
}
return "", nil
}
d := parseTrusted(t, `{"declaration":1,"resources":[{"id":"front-end","type":"package","package":"ufw","absent":true}]}`)
report, _, err := Apply(context.Background(), archHost(t), d, store.State{}, store.OriginCarried, run, nil, nil)
if err != nil {
t.Fatal(err)
}
if report.Outcomes[0].Action != "removed" || !strings.Contains(strings.Join(ran, "\n"), "pacman -R --noconfirm ufw") {
t.Fatalf("an installed package declared absent was not removed: %+v\n%v", report.Outcomes[0], ran)
}
ran = nil
report, _, err = Apply(context.Background(), archHost(t), d, store.State{}, store.OriginCarried, run, nil, nil)
if err != nil {
t.Fatal(err)
}
if report.Outcomes[0].Action != "unchanged" || strings.Contains(strings.Join(ran, "\n"), "-R") {
t.Fatalf("a package already absent was touched: %+v\n%v", report.Outcomes[0], ran)
}
}
+54
View File
@@ -0,0 +1,54 @@
package apply
import (
"testing"
"github.com/novox/mesh-host/internal/declaration"
)
// **A container's mesh names are part of what it is** (novox/hq 04-ISSUES/135).
//
// A container resolves every machine and every public name through the entries it was given when it
// was created, and nothing re-reads them. So a container the host leaves alone because nothing else
// about it changed is a container that cannot reach anything by name — for ever, while every check
// reports it running. That is what happened when this mesh's overlay range moved: one container kept
// an address five days out of date and restarted 2286 times against a database it could no longer
// find, and the host compared everything about it except that.
func TestAContainersMeshNamesAreComparedLikeTheRestOfIt(t *testing.T) {
was := &declaration.Container{
Name: "umami", Image: "ghcr.io/example/umami@sha256:" + zeros(64),
Hosts: []string{"novox.internal:10.42.0.1", "umami.novox.be:10.42.0.1"},
}
moved := &declaration.Container{
Name: was.Name, Image: was.Image,
Hosts: []string{"novox.internal:10.10.0.1", "umami.novox.be:10.10.0.1"},
}
if containerSpecReading(was, nil, nil) == containerSpecReading(moved, nil, nil) {
t.Fatal("a container whose mesh names moved compares equal, so it is never recreated")
}
// And the order they arrive in is not a change: the digest must not move for a reordering
// nobody made.
reordered := &declaration.Container{
Name: moved.Name, Image: moved.Image,
Hosts: []string{moved.Hosts[1], moved.Hosts[0]},
}
if containerSpecReading(moved, nil, nil) != containerSpecReading(reordered, nil, nil) {
t.Fatal("the same names in another order read as a different container")
}
// A container the mesh gives no names is unaffected, so nothing is recreated for a field it
// does not set.
plain := &declaration.Container{Name: "plex", Image: was.Image}
if containerSpecReading(plain, nil, nil) == containerSpecReading(was, nil, nil) {
return // different for other reasons, which is fine
}
}
func zeros(n int) string {
out := make([]byte, n)
for i := range out {
out[i] = '0'
}
return string(out)
}
+55
View File
@@ -0,0 +1,55 @@
package apply
import (
"fmt"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/store"
)
// A node is adopted or converged, and a declaration says which it is for (novox/hq ADR 0100).
//
// **The two are not interchangeable, and the host knows which it is.** A converged declaration
// carries the mesh's drop-by-default filter and retires the firewall the node was found with; on
// an adopted node that closes the machine to everything the predecessor still serves — which is
// what happened when an operator ran `reconcile` on an adopted control-node and it applied the
// converged genesis bundle (novox/hq issue 104). The host reported "adopted" in every report and
// compared nothing. Now it compares, at the point of application, whichever command delivered
// the declaration.
//
// Only the mesh changes a node's mode, and it does so by sending a declaration: the flip arrives
// over the link as the first converged declaration after adopted ones (`converge` on the
// controller), and the way back as the first adopted one (`adopt`). So a declaration the link
// delivers, signed and verified, is the mode's authority and is not checked against the record —
// it becomes the record. Everything else — the carried bundle, a file, the kept declaration a
// disconnected node re-applies — is held to the mode already recorded.
// ModeOf says which mode a declaration is for.
func ModeOf(d *declaration.Declaration) string {
if d.Adoption != nil {
return store.ModeAdopted
}
return store.ModeConverged
}
// CheckMode refuses a declaration whose mode is not the one this node has recorded. A node with
// no recorded mode — a machine nothing has said a mode to yet — takes either.
func CheckMode(known store.State, d *declaration.Declaration) error {
if known.Mode == "" || known.Mode == ModeOf(d) {
return nil
}
act := "`converge`"
if ModeOf(d) == store.ModeAdopted {
act = "`adopt`"
}
return fmt.Errorf("this node is %s; the declaration says %s — %s declaration is not applied "+
"to %s node; %s on the controller is the act that changes it, and it sends the "+
"declaration that does", known.Mode, ModeOf(d), article(ModeOf(d)), article(known.Mode), act)
}
func article(mode string) string {
if mode == store.ModeAdopted {
return "an adopted"
}
return "a converged"
}
+32
View File
@@ -0,0 +1,32 @@
package apply
import (
"strings"
"testing"
"github.com/novox/mesh-host/internal/store"
)
// Defends novox/hq issue 104: a declaration is refused for the other mode than the node is in,
// naming both and the act that changes it; a node no mode has been said to takes either.
func TestADeclarationForTheOtherModeIsRefused(t *testing.T) {
converged := parse(t, `{"declaration":1,"resources":[{"id":"a","type":"file","path":"/tmp/a","content":"x\n"}]}`)
adopted := parse(t, `{"declaration":1,"adoption":{"taken":[]},"resources":[{"id":"a","type":"file","path":"/tmp/a","content":"x\n"}]}`)
err := CheckMode(store.State{Mode: store.ModeAdopted}, converged)
if err == nil || !strings.Contains(err.Error(), "this node is adopted; the declaration says converged") ||
!strings.Contains(err.Error(), "`converge`") {
t.Errorf("a converged declaration on an adopted node: %v", err)
}
err = CheckMode(store.State{Mode: store.ModeConverged}, adopted)
if err == nil || !strings.Contains(err.Error(), "this node is converged; the declaration says adopted") ||
!strings.Contains(err.Error(), "`adopt`") {
t.Errorf("an adopted declaration on a converged node: %v", err)
}
if err := CheckMode(store.State{Mode: store.ModeAdopted}, adopted); err != nil {
t.Errorf("the node's own mode was refused: %v", err)
}
if err := CheckMode(store.State{}, converged); err != nil {
t.Errorf("a node in no mode yet refused a declaration: %v", err)
}
}
+177
View File
@@ -0,0 +1,177 @@
package apply
import (
"context"
"fmt"
"time"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/firewall"
"github.com/novox/mesh-host/internal/store"
)
// foundFirewall settles, before anything else in an apply, which firewall this node has — and on
// an adopted node that the mesh had converged, puts it back in force first (novox/hq ADR 0100).
//
// Only an adopted node asks. It is detected on every apply rather than remembered, so a firewall
// switched on after adoption is spoken to from the next reconcile; what is remembered is what was
// found first, and whether the mesh retired it. An unsupported firewall refuses the whole
// declaration: the mesh could neither open what it needs through it nor say what it would close.
func foundFirewall(ctx context.Context, d *declaration.Declaration, known *store.State, run Runner,
log func(string)) (firewall.Kind, error) {
if d.Adoption == nil {
return "", nil
}
rec := known.Firewall
if rec != nil && rec.DisabledByMesh && rec.Kind == string(firewall.UFW) {
// Returned to adopted: the found firewall is enabled again before the openings are
// converged through it, and the derived filter is gone with this declaration.
if err := firewall.Enable(ctx, run); err != nil {
return "", err
}
rec.DisabledByMesh = false
rec.Forward = nil
log(" enabled ufw again: this node is adopted, and the firewall found on it is in force")
}
kind, name, err := firewall.Detect(ctx, run)
if err != nil {
return "", err
}
if kind == firewall.Unsupported {
return "", fmt.Errorf(
"this machine is filtered by %s, and no host speaks that firewall yet. An adopted node "+
"keeps the firewall it was found with, so the mesh could neither open what it needs "+
"through it nor say what it would close; this declaration is refused whole", name)
}
if rec == nil {
rec = &store.FoundFirewall{Kind: string(kind), WasActive: kind == firewall.UFW,
FoundAt: time.Now().UTC()}
} else {
rec.Kind = string(kind)
rec.WasActive = rec.WasActive || kind == firewall.UFW
}
known.Firewall = rec
return kind, nil
}
// retireFirewall keeps the found firewall retired on a converged machine (novox/hq ADR 0100, ADR
// 0168): disabled, never flushed, its configuration left on disk for a return to adopted, and the
// container runtime's rules not its to take.
//
// **Convergence is a state the host keeps, not a step it takes once.** Every converged apply reads
// whether the front end is in force; enabled again by a package, a boot or a hand, it is retired
// again and said. The record says how it came to be inactive — the mesh disabled it, or a reconcile
// found it so — and the two are never confused: a flip that did not take, followed by a hand that
// did, used to be recorded as the mesh's doing (issue 143).
//
// Only a declaration from the mesh converges a node. A carried bundle never says a node is adopted
// — it cannot — so its silence is not the controller's word that the node was converged, and an
// adopted node re-applying its bundle keeps the firewall it was found with.
//
// Returned is what this apply did about the found firewall, for the report; empty when the machine
// has none or is not converged.
func retireFirewall(ctx context.Context, d *declaration.Declaration, origin string, known *store.State,
run Runner, log func(string)) (string, error) {
rec := known.Firewall
if origin != store.OriginDeclared || d.Adoption != nil || rec == nil || rec.Kind != string(firewall.UFW) || !rec.WasActive {
return "", nil
}
if !firewall.Installed(ctx, run) {
// Uninstalled (novox/hq ADR 0175): retired for good, by the module that replaced it. Said
// once, and nothing is asked of a command that is not there.
if rec.RetiredBy != firewall.RetiredRemoved {
rec.RetiredBy = firewall.RetiredRemoved
log(" the found firewall (ufw) is no longer installed; the mesh's filter is what filters this machine")
return "removed: ufw is no longer installed; the mesh's filter is what filters this machine", nil
}
return "", nil
}
active := firewall.Active(ctx, run)
if !active && !(rec.Forward != nil && !rec.DisabledByMesh) {
// Inactive, and either the mesh's doing already or nobody's recorded here: said as found,
// never as done (issue 143's second fault). A retirement the mesh began and did not finish —
// the forward policy recorded, ufw down, the restore failed — is the one inactive state that
// is still the mesh's to complete, below.
if rec.RetiredBy == "" {
if rec.DisabledByMesh {
rec.RetiredBy = firewall.RetiredByMesh
} else {
rec.RetiredBy = firewall.RetiredFoundSo
log(" ufw is inactive on this converged node, and not by the mesh; recorded as found so")
}
}
return "", nil
}
// **Nothing is retired until what replaces it is in force** (novox/hq ADR 0100). The flip
// loads the mesh's derived filter in ufw's place; disabling ufw before that table is actually
// loaded — a filter module not assigned, or a unit that did not load — leaves the machine with
// no filter at all.
loaded, err := firewall.MeshTableLoaded(ctx, run)
if err != nil {
return "", err
}
if !loaded {
return "", fmt.Errorf("this node is converged and the mesh's own filter (table %s) is not loaded on "+
"this machine, so ufw was left in force: retiring it would leave the machine filtering "+
"nothing. Assign a filter module to this node, or return it to adopted", firewall.MeshTable)
}
if rec.Forward == nil {
// Recorded before ufw is touched: disabling it opens the forward policy, and a retry
// must know what it was (novox/hq ADR 0100).
rec.Forward = firewall.ForwardPolicies(ctx, run)
}
if err := firewall.Disable(ctx, run, rec.Forward); err != nil {
return "", err
}
again := rec.DisabledByMesh || rec.RetiredBy != ""
rec.DisabledByMesh = true
rec.RetiredBy = firewall.RetiredByMesh
if again {
log(" disabled ufw again: it had been enabled since the mesh retired it; this node is converged and filtered by the mesh")
return "disabled again: ufw had been enabled since the mesh retired it", nil
}
log(" disabled ufw: this node is converged and filtered by the mesh; ufw's configuration is left on disk")
return "disabled: this node is converged and filtered by the mesh; ufw's configuration is left on disk", nil
}
// applyOpening makes one opening true through the firewall found here.
func applyOpening(ctx context.Context, o *declaration.Opening, run Runner, kind firewall.Kind) (Outcome, error) {
out := begin(o)
switch kind {
case firewall.None:
out.Action = "unchanged"
out.Detail = "no firewall found; nothing filters this port"
return out, nil
case firewall.UFW:
done, err := firewall.Converge(ctx, run, o)
if err != nil {
return out, err
}
out.Action = done.Action
out.Detail = "through ufw, marked " + firewall.Mark(o)
if done.SatisfiedBy != "" {
// ufw would take a rule differing only in its comment for the same one, so the
// mesh's is not added beside it (novox/hq ADR 0103).
out.Detail = "satisfied by a rule found in ufw (" + done.SatisfiedBy +
"); the mesh added nothing and will remove nothing"
}
return out, nil
}
return out, fmt.Errorf("no firewall is known for this node, so %s cannot be opened", o.Target())
}
// removeOpening deletes the rules the mesh marked for an opening no longer declared, and nothing
// the machine had before.
func removeOpening(ctx context.Context, a store.Applied, run Runner, rec *store.FoundFirewall) (string, string, error) {
if rec == nil || rec.Kind != string(firewall.UFW) {
return "forgotten", "no firewall held a rule for it", nil
}
n, err := firewall.Remove(ctx, run, a.ID)
if err != nil {
return "", "", err
}
if n == 0 {
return "forgotten", "ufw held no rule marked for it", nil
}
return "removed", fmt.Sprintf("%d ufw rule(s) marked as the mesh's deleted", n), nil
}
+555
View File
@@ -0,0 +1,555 @@
package apply
import (
"context"
"errors"
"os"
"os/exec"
"path/filepath"
"strings"
"testing"
"time"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/store"
)
// Defends novox/hq ADR 0100: the firewall found on an adopted node stays in force; converging the
// node retires it by disabling it, and returning the node to adopted enables it again.
type ufwMachine struct {
installed, active bool
rules []string
ruleset string
asked []string
// forward is iptables' forward policy when set; empty is a machine without iptables. failP
// is how many -P calls fail before one succeeds.
forward string
failP int
}
func (u *ufwMachine) iptables(args []string) (string, error) {
if len(args) == 3 && args[0] == "-P" {
if u.failP > 0 {
u.failP--
return "", errors.New("iptables: resource temporarily unavailable")
}
u.forward = args[2]
return "", nil
}
return "-P FORWARD " + u.forward + "\n-A FORWARD -j DOCKER-USER\n", nil
}
func (u *ufwMachine) run(_ context.Context, name string, args ...string) (string, error) {
u.asked = append(u.asked, name+" "+strings.Join(args, " "))
switch name {
case "nft":
return u.ruleset, nil
case "iptables":
if u.forward == "" {
return "", &exec.Error{Name: name, Err: exec.ErrNotFound}
}
return u.iptables(args)
case "ufw":
if !u.installed {
return "", &exec.Error{Name: name, Err: exec.ErrNotFound}
}
default:
return "", &exec.Error{Name: name, Err: exec.ErrNotFound}
}
switch args[0] {
case "status":
if u.active {
return "Status: active\n", nil
}
return "Status: inactive\n", nil
case "show":
out := "Added user rules (see 'ufw status' for running firewall):\n"
for _, r := range u.rules {
out += "ufw " + r + "\n"
}
return out, nil
case "--force":
u.active = true
return "", nil
case "disable":
u.active = false
if u.forward != "" {
u.forward = "ACCEPT" // as measured: ufw disable opens the forward policy
}
return "", nil
case "delete":
want := strings.Join(args[1:], " ")
for i, r := range u.rules {
if strings.ReplaceAll(r, "'", "") == want {
u.rules = append(u.rules[:i], u.rules[i+1:]...)
return "", nil
}
}
return "", errors.New("Could not delete non-existent rule")
default:
// Printed back the way it was given, with the comment quoted as ufw does.
line := strings.Join(args[:len(args)-1], " ") + " '" + args[len(args)-1] + "'"
u.rules = append(u.rules, line)
return "", nil
}
}
func (u *ufwMachine) index(prefix string) int {
for i, a := range u.asked {
if strings.HasPrefix(a, prefix) {
return i
}
}
return -1
}
const busOpening = `{"id":"adoption.opening-tcp-5671-incoming","type":"opening","port":5671,"protocol":"tcp","from":"everywhere","path":"incoming"}`
func withConf(dir string) string {
return `{"id":"x.conf","type":"file","path":"` + filepath.Join(dir, "x.conf") + `","content":"x\n"}`
}
func applyWith(t *testing.T, d *declaration.Declaration, known store.State, run Runner) (Report, store.State, error) {
t.Helper()
return ApplyKeeping(context.Background(), archHost(t), d, known, store.OriginDeclared, run, nil, nil,
KeepIn(t.TempDir()))
}
func TestAnOpeningOnAMachineWithNoFirewallChangesNothing(t *testing.T) {
dir := t.TempDir()
u := &ufwMachine{}
report, state, err := applyWith(t, adopted(t, `{"taken":[]}`, busOpening+","+withConf(dir)), store.State{}, u.run)
if err != nil {
t.Fatal(err)
}
o := outcomeOf(report, "adoption.opening-tcp-5671-incoming")
if o.Action != "unchanged" || !strings.Contains(o.Detail, "nothing filters this port") {
t.Errorf("an opening with no firewall: %+v", o)
}
if state.Firewall == nil || state.Firewall.Kind != "none" {
t.Errorf("the firewall found was not recorded: %+v", state.Firewall)
}
}
func TestAnUnsupportedFirewallRefusesTheWholeDeclaration(t *testing.T) {
dir := t.TempDir()
u := &ufwMachine{ruleset: "table inet filter {\n\tchain input {\n\t\ttype filter hook input priority filter; policy drop;\n\t}\n}\n"}
_, state, err := applyWith(t, adopted(t, `{"taken":[]}`, busOpening+","+withConf(dir)), store.State{}, u.run)
if err == nil || !strings.Contains(err.Error(), "no host speaks that firewall") {
t.Fatalf("an unsupported firewall was not refused: %v", err)
}
if _, statErr := os.Stat(filepath.Join(dir, "x.conf")); !errors.Is(statErr, os.ErrNotExist) {
t.Error("part of a refused declaration was applied")
}
if state.Firewall != nil {
t.Errorf("an unsupported firewall was recorded: %+v", state.Firewall)
}
}
func TestConvergingRetiresTheFoundFirewallAndReturningRestoresIt(t *testing.T) {
dir := t.TempDir()
u := &ufwMachine{installed: true, active: true, rules: []string{"allow 22/tcp"}}
// Adopted: the opening goes through ufw.
_, state, err := applyWith(t, adopted(t, `{"taken":[]}`, busOpening+","+withConf(dir)), store.State{}, u.run)
if err != nil {
t.Fatal(err)
}
if len(u.rules) != 2 || !u.active {
t.Fatalf("adopted: rules %v, active %v", u.rules, u.active)
}
if state.Firewall == nil || state.Firewall.Kind != "ufw" || !state.Firewall.WasActive {
t.Fatalf("adopted: firewall recorded as %+v", state.Firewall)
}
// Converged: the derived filter is loaded, the opening's rule goes, and only then is ufw
// disabled — never reset.
u.asked = nil
u.ruleset = "table inet mesh\n"
converged := parse(t, `{"declaration":1,"resources":[`+withConf(dir)+`]}`)
_, state, err = applyWith(t, converged, state, u.run)
if err != nil {
t.Fatal(err)
}
if u.active || !state.Firewall.DisabledByMesh {
t.Fatalf("converged: ufw still active (%v) or not recorded as retired (%+v)", u.active, state.Firewall)
}
if len(u.rules) != 1 || u.rules[0] != "allow 22/tcp" {
t.Errorf("converged: the operator's rules were touched, or the mesh's left: %v", u.rules)
}
// What protected the adopted node goes last: after the derived filter applied and ufw was
// retired (novox/hq ADR 0103).
if del, dis := u.index("ufw delete"), u.index("ufw disable"); dis < 0 || del < dis {
t.Errorf("converged: the opening was removed before ufw was retired: %v", u.asked)
}
for _, a := range u.asked {
if strings.Contains(a, "reset") {
t.Errorf("converged: ufw was reset: %s", a)
}
}
// Converged again: nothing more to retire — the node asks ufw whether it is in force, which is
// what keeps convergence a state rather than a step taken once (novox/hq ADR 0168), and touches
// nothing else.
u.asked = nil
if _, state, err = applyWith(t, converged, state, u.run); err != nil {
t.Fatal(err)
}
for _, a := range u.asked {
if strings.HasPrefix(a, "ufw") && a != "ufw status" {
t.Errorf("a converged node kept talking to a retired ufw: %v", u.asked)
}
}
if state.Firewall.RetiredBy != "mesh" {
t.Errorf("the record does not say the mesh retired it: %+v", state.Firewall)
}
// Enabled again by a hand: retired again, and said.
u.active = true
u.asked = nil
report, state, err := applyWith(t, converged, state, u.run)
if err != nil {
t.Fatal(err)
}
if u.active || u.index("ufw disable") < 0 {
t.Fatalf("ufw enabled again on a converged node was not retired again: %v", u.asked)
}
if !strings.Contains(report.Firewall, "disabled again") {
t.Errorf("retiring it again was not said: %q", report.Firewall)
}
// Returned to adopted: ufw is enabled before the opening is converged through it.
u.asked = nil
_, state, err = applyWith(t, adopted(t, `{"taken":[]}`, busOpening+","+withConf(dir)), state, u.run)
if err != nil {
t.Fatal(err)
}
if !u.active || state.Firewall.DisabledByMesh {
t.Fatalf("returned: ufw active %v, record %+v", u.active, state.Firewall)
}
if en, add := u.index("ufw --force enable"), u.index("ufw allow"); en < 0 || add < en {
t.Errorf("returned: ufw was not enabled before the opening was added: %v", u.asked)
}
if len(u.rules) != 2 {
t.Errorf("returned: the opening was not converged again: %v", u.rules)
}
}
func TestAConvergedNodeThatWasNeverAdoptedNeverAsksAboutAFirewall(t *testing.T) {
dir := t.TempDir()
u := &ufwMachine{installed: true, active: true}
if _, _, err := applyWith(t, parse(t, `{"declaration":1,"resources":[`+withConf(dir)+`]}`), store.State{}, u.run); err != nil {
t.Fatal(err)
}
if len(u.asked) != 0 {
t.Errorf("a converged apply asked the machine about its firewall: %v", u.asked)
}
}
func TestAnOpeningOnAConvergedNodeIsRefused(t *testing.T) {
if _, err := declaration.Parse([]byte(`{"declaration":1,"resources":[` + busOpening + `]}`)); err == nil {
t.Error("an opening was accepted on a node the declaration does not say is adopted")
}
}
func TestACarriedApplyOnAnAdoptedNodeLeavesItsFirewallInForce(t *testing.T) {
// The bundle, re-applied by the installer or the one-shot CLI, never says a node is adopted.
// That is not the controller converging it, so ufw must stay enabled (novox/hq ADR 0100).
dir := t.TempDir()
u := &ufwMachine{installed: true, active: true, rules: []string{"allow 22/tcp"}}
_, state, err := applyWith(t, adopted(t, `{"taken":[]}`, busOpening+","+withConf(dir)), store.State{}, u.run)
if err != nil {
t.Fatal(err)
}
u.asked = nil
carried := parse(t, `{"declaration":1,"resources":[`+withConf(filepath.Join(dir, "bundle"))+`]}`)
_, state, err = ApplyKeeping(context.Background(), archHost(t), carried, state, store.OriginCarried,
u.run, nil, nil, nil)
if err != nil {
t.Fatal(err)
}
if !u.active || state.Firewall.DisabledByMesh || u.index("ufw disable") >= 0 {
t.Fatalf("a carried apply retired the found firewall: active %v, record %+v, asked %v",
u.active, state.Firewall, u.asked)
}
}
func TestAFlipThatFailsKeepsTheGuardAndTheOpenings(t *testing.T) {
// Converging a node removes its openings and its guard only once everything else applied and
// the found firewall is retired. A flip that fails part-way keeps them, so the store is never
// left unguarded behind a filter that did not load (novox/hq ADR 0103).
dir := t.TempDir()
guard := filepath.Join(dir, "guard.nft")
guardFile := `{"id":"adoption.guard","type":"file","path":"` + guard + `","content":"table inet mesh_guard {}\n"}`
u := &ufwMachine{installed: true, active: true, rules: []string{"allow 22/tcp"}}
_, state, err := applyWith(t, adopted(t, `{"taken":[]}`, busOpening+","+guardFile+","+withConf(dir)),
store.State{}, u.run)
if err != nil {
t.Fatal(err)
}
// The derived filter cannot be written: its path is under a file.
blocked := filepath.Join(dir, "not-a-directory")
if err := os.WriteFile(blocked, []byte("x"), 0o644); err != nil {
t.Fatal(err)
}
filter := `{"id":"nftables.config","type":"file","path":"` + filepath.Join(blocked, "nftables.conf") + `","content":"table inet mesh {}\n"}`
converged := parse(t, `{"declaration":1,"resources":[`+withConf(dir)+`,`+filter+`]}`)
u.asked = nil
u.ruleset = "table inet mesh\n" // what the filter's unit loads once the file is written
_, state, err = applyWith(t, converged, state, u.run)
if err == nil {
t.Fatal("the failing flip reported success")
}
if !u.active || u.index("ufw disable") >= 0 {
t.Errorf("the found firewall was retired by a flip that failed: %v", u.asked)
}
if len(u.rules) != 2 || u.index("ufw delete") >= 0 {
t.Errorf("the opening was removed by a flip that failed: %v", u.rules)
}
if _, statErr := os.Stat(guard); statErr != nil {
t.Errorf("the guard was removed by a flip that failed: %v", statErr)
}
for _, id := range []string{"adoption.guard", "adoption.opening-tcp-5671-incoming"} {
if _, ok := state.Find(id); !ok {
t.Errorf("%s was forgotten, so the next flip would never remove it", id)
}
}
// Fixed, the next flip completes: filter, retire, and only then the guard and the openings.
if err := os.Remove(blocked); err != nil {
t.Fatal(err)
}
u.asked = nil
_, state, err = applyWith(t, converged, state, u.run)
if err != nil {
t.Fatal(err)
}
if u.active || len(u.rules) != 1 {
t.Errorf("the completed flip left ufw active %v, rules %v", u.active, u.rules)
}
if _, statErr := os.Stat(guard); !errors.Is(statErr, os.ErrNotExist) {
t.Errorf("the guard outlived the completed flip: %v", statErr)
}
if _, ok := state.Find("adoption.guard"); ok {
t.Error("the guard is still recorded after the completed flip")
}
}
func TestAnOpeningAFoundRuleAnswersIsReportedSatisfied(t *testing.T) {
// novox/hq ADR 0103: the mesh adds nothing beside a rule ufw would take for the same one.
dir := t.TempDir()
u := &ufwMachine{installed: true, active: true, rules: []string{"allow 22/tcp", "allow 5671/tcp"}}
report, _, err := applyWith(t, adopted(t, `{"taken":[]}`, busOpening+","+withConf(dir)), store.State{}, u.run)
if err != nil {
t.Fatal(err)
}
o := outcomeOf(report, "adoption.opening-tcp-5671-incoming")
if o.Action != "unchanged" || !strings.Contains(o.Detail, "satisfied by a rule found in ufw (allow 5671/tcp)") {
t.Errorf("the opening was not reported satisfied: %+v", o)
}
if len(u.rules) != 2 || u.index("ufw allow") >= 0 {
t.Errorf("a rule was added beside the found one: %v", u.rules)
}
}
// Defends novox/hq ADR 0103: returned to adopted, the guard is up before the derived filter's
// orphans go, and stays up if removing them fails.
func TestReturningToAdoptedLoadsTheGuardBeforeRemovingTheFilter(t *testing.T) {
dir := t.TempDir()
guard := filepath.Join(dir, "guard.nft")
guardFile := `{"id":"adoption.guard","type":"file","path":"` + guard + `","content":"table inet mesh_guard {}\n"}`
// The filter's unit file is the mesh's own, written where there was none — which is what makes
// its unit the mesh's to stop, with or without a record of what was found (novox/hq ADR 0118).
units := t.TempDir()
was := unitDir
unitDir = units
t.Cleanup(func() { unitDir = was })
filterUnit := filepath.Join(units, "mesh-filter.service")
for _, stopFails := range []bool{false, true} {
_ = os.Remove(guard)
if err := os.WriteFile(filterUnit, []byte("[Unit]\n"), 0o644); err != nil {
t.Fatal(err)
}
guardUpAtStop := false
run := func(_ context.Context, name string, args ...string) (string, error) {
if name != "systemctl" {
return "", &exec.Error{Name: name, Err: exec.ErrNotFound}
}
switch args[0] {
case "show":
return "LoadState=loaded\nActiveState=active\nType=oneshot\nRemainAfterExit=yes\n", nil
case "stop":
_, err := os.Stat(guard)
guardUpAtStop = err == nil
if stopFails {
return "", errors.New("the filter would not stop")
}
}
return "", nil
}
// As a host recorded them before it kept what it found: no Found on the service.
converged := store.State{Resources: []store.Applied{
{ID: "nftables.unit", Type: "file", Target: filterUnit, Origin: store.OriginDeclared},
{ID: "nftables.load", Type: "service", Target: "mesh-filter.service", Origin: store.OriginDeclared}}}
_, state, err := applyWith(t, adopted(t, `{"taken":[]}`, withConf(dir)+","+guardFile), converged, run)
if !guardUpAtStop {
t.Errorf("stop fails %v: the derived filter was stopped before the guard was written", stopFails)
}
if stopFails {
if err == nil {
t.Error("a failed removal was not reported")
}
if _, statErr := os.Stat(guard); statErr != nil {
t.Error("the guard is not up after the filter's removal failed")
}
if _, ok := state.Find("adoption.guard"); !ok {
t.Error("the guard applied before the failure was not recorded")
}
if _, still := state.Find("nftables.load"); !still {
t.Error("the filter that would not stop was forgotten, so nothing would stop it later")
}
} else if err != nil {
t.Fatal(err)
}
}
}
func TestAStaleOpeningOnAnAdoptedNodeIsRemovedAsAnyOrphan(t *testing.T) {
// Only the flip defers the adoption's own orphans; an adopted node drops a stale opening at
// once, before what replaces it is applied.
dir := t.TempDir()
u := &ufwMachine{installed: true, active: true, rules: []string{"allow 22/tcp"}}
_, state, err := applyWith(t, adopted(t, `{"taken":[]}`, busOpening+","+withConf(dir)), store.State{}, u.run)
if err != nil {
t.Fatal(err)
}
u.asked = nil
other := `{"id":"adoption.opening-tcp-5000-incoming","type":"opening","port":5000,"protocol":"tcp","from":"everywhere","path":"incoming"}`
if _, _, err = applyWith(t, adopted(t, `{"taken":[]}`, other+","+withConf(dir)), state, u.run); err != nil {
t.Fatal(err)
}
if del, add := u.index("ufw delete"), u.index("ufw allow"); del < 0 || add < 0 || del > add {
t.Errorf("the stale opening was not removed before the new one was added: %v", u.asked)
}
}
func TestARetiredFirewallRetriedStillPutsBackTheForwardPolicy(t *testing.T) {
// The forward policy is recorded before ufw is disabled, so a retirement that failed after
// the disable restores what the machine had, not what the disable left (novox/hq ADR 0100).
dir := t.TempDir()
u := &ufwMachine{installed: true, active: true, forward: "DROP", failP: 1, ruleset: "table inet mesh\n"}
_, state, err := applyWith(t, adopted(t, `{"taken":[]}`, withConf(dir)), store.State{}, u.run)
if err != nil {
t.Fatal(err)
}
converged := parse(t, `{"declaration":1,"resources":[`+withConf(dir)+`]}`)
if _, state, err = applyWith(t, converged, state, u.run); err == nil {
t.Fatal("the failed restore was not reported")
}
if u.active || u.forward != "ACCEPT" || state.Firewall.Forward["iptables"] != "DROP" {
t.Fatalf("after the failed attempt: active %v, forward %s, recorded %+v", u.active, u.forward, state.Firewall)
}
if _, state, err = applyWith(t, converged, state, u.run); err != nil {
t.Fatal(err)
}
if u.forward != "DROP" || !state.Firewall.DisabledByMesh {
t.Errorf("the retry did not put the forward policy back: %s, %+v", u.forward, state.Firewall)
}
}
func TestAGuardThatFailsLeavesTheDerivedFilterInForce(t *testing.T) {
// Returning to adopted: if the guard cannot be raised, the filter it replaces must not be
// stopped — its stop deletes the mesh's table, and the node would have neither (novox/hq ADR 0103).
dir := t.TempDir()
blocked := filepath.Join(dir, "not-a-directory")
if err := os.WriteFile(blocked, []byte("x"), 0o644); err != nil {
t.Fatal(err)
}
guardFile := `{"id":"adoption.guard","type":"file","path":"` + filepath.Join(blocked, "guard.nft") +
`","content":"table inet mesh_guard {}\n"}`
stopped := false
run := func(_ context.Context, name string, args ...string) (string, error) {
if name != "systemctl" {
return "", &exec.Error{Name: name, Err: exec.ErrNotFound}
}
if args[0] == "show" {
return "LoadState=loaded\nActiveState=active\nType=oneshot\nRemainAfterExit=yes\n", nil
}
if args[0] == "stop" {
stopped = true
}
return "", nil
}
converged := store.State{Resources: []store.Applied{
{ID: "nftables.load", Type: "service", Target: "mesh-filter.service", Origin: store.OriginDeclared}}}
_, state, err := applyWith(t, adopted(t, `{"taken":[]}`, withConf(dir)+","+guardFile), converged, run)
if err == nil {
t.Fatal("a guard that could not be written reported success")
}
if stopped {
t.Error("the derived filter was stopped though the guard is not up")
}
if _, gone := state.Find("nftables.load"); !gone {
t.Error("the filter was forgotten, so nothing would ever stop it")
}
}
func TestUfwIsNotRetiredUntilTheMeshsOwnFilterIsLoaded(t *testing.T) {
// The flip retires the found firewall because the mesh's derived filter takes its place. If
// that table is not loaded, retiring would leave the machine filtering nothing (novox/hq ADR 0100).
dir := t.TempDir()
u := &ufwMachine{installed: true, active: true, rules: []string{"allow 22/tcp"}}
_, state, err := applyWith(t, adopted(t, `{"taken":[]}`, busOpening+","+withConf(dir)), store.State{}, u.run)
if err != nil {
t.Fatal(err)
}
converged := parse(t, `{"declaration":1,"resources":[`+withConf(dir)+`]}`)
_, state, err = applyWith(t, converged, state, u.run)
if err == nil || !strings.Contains(err.Error(), "table inet mesh") {
t.Fatalf("ufw was retired with nothing in its place: %v", err)
}
if !u.active || state.Firewall.DisabledByMesh {
t.Errorf("ufw was disabled: active %v, %+v", u.active, state.Firewall)
}
// Once the table is loaded, the same declaration retires it.
u.ruleset = "table inet mesh\n"
if _, state, err = applyWith(t, converged, state, u.run); err != nil {
t.Fatal(err)
}
if u.active || !state.Firewall.DisabledByMesh {
t.Errorf("ufw was not retired once the mesh's filter was loaded: active %v, %+v", u.active, state.Firewall)
}
}
// A front end that is no longer installed is recorded as removed, said once, and asked nothing of
// (novox/hq ADR 0175).
func TestAnUninstalledFrontEndIsRetiredForGood(t *testing.T) {
dir := t.TempDir()
u := &ufwMachine{installed: false, ruleset: "table inet mesh\n"}
known := store.State{Firewall: &store.FoundFirewall{Kind: "ufw", WasActive: true, DisabledByMesh: true,
RetiredBy: "mesh", FoundAt: time.Now()}}
converged := parse(t, `{"declaration":1,"resources":[`+withConf(dir)+`]}`)
report, state, err := applyWith(t, converged, known, u.run)
if err != nil {
t.Fatal(err)
}
if state.Firewall.RetiredBy != "removed" || !strings.Contains(report.Firewall, "no longer installed") {
t.Fatalf("record %+v, said %q", state.Firewall, report.Firewall)
}
u.asked = nil
report, _, err = applyWith(t, converged, state, u.run)
if err != nil {
t.Fatal(err)
}
if report.Firewall != "" {
t.Errorf("said again: %q", report.Firewall)
}
for _, a := range u.asked {
if strings.HasPrefix(a, "ufw") && a != "ufw status" {
t.Errorf("asked something of a front end that is not there: %v", u.asked)
}
}
}
+337
View File
@@ -0,0 +1,337 @@
package apply
import (
"fmt"
"strings"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/firewall"
"github.com/novox/mesh-host/internal/store"
)
// A declaration is said before it is done.
//
// An apply that prints what it did after it did it is a report; what an operator reaching for
// `reconcile` under pressure needs is a preview — the base filter that closed an adopted
// control-node for forty-five minutes was listed nowhere until it was on disk (novox/hq issue
// 104). Converging already previews on the controller; the host's own commands now do too, and
// `--dry-run` is the preview and nothing else.
// Step is one thing an apply would do to this machine.
type Step struct {
// Verb is create · update · check · hold · run · remove · forget · restore · disable · enable.
Verb string `json:"verb"`
Type string `json:"type,omitempty"`
ID string `json:"id,omitempty"`
Target string `json:"target,omitempty"`
Why string `json:"why,omitempty"`
}
func (s Step) String() string {
line := fmt.Sprintf("%-8s %-10s %s", s.Verb, s.Type, s.ID)
if s.Target != "" && s.Target != s.ID {
line += " (" + s.Target + ")"
}
if s.Why != "" {
line += " — " + s.Why
}
return line
}
// Plan says what applying a declaration would change, in the order ApplyKeeping would do it,
// before anything on the machine is touched.
//
// **Read from the declaration and the node's own record, not from the machine.** What the
// record cannot settle — whether a file recorded here has since drifted, whether a container
// runs the spec it was made from — is said as a check, because that is what the apply does: it
// reads the machine and corrects it. What the record does settle is said as it is: a resource
// with no record is created; a plain file whose declared content differs from what this host
// last wrote is updated; a hold is kept; an action is run — an action is a command, and a
// preview that folded it into "check" would hide the one kind of step that is not read back
// from state.
func Plan(d *declaration.Declaration, known store.State, origin string) []Step {
var steps []Step
declared := map[string]bool{}
for _, r := range d.Resources {
declared[r.Identity()] = true
}
// The found tunnel's configuration is held under an id of its own, declared for as long as the
// service taking it over is — as ApplyKeeping counts it, or a plan would forget a hold the
// apply keeps (novox/hq ADR 0105).
if svc := takesOver(d); svc != nil {
declared[takeOverID(svc)] = true
}
rec := known.Firewall
ufw := rec != nil && rec.Kind == string(firewall.UFW)
if d.Adoption != nil && ufw && rec.DisabledByMesh {
steps = append(steps, Step{Verb: "enable", Type: "firewall", ID: "ufw",
Why: "this node is adopted again, so the firewall found on it is put back in force"})
}
// What is held and no longer declared is let go of on paper only (ApplyKeeping does this
// before the resources); the file or container itself is left as found.
if origin == store.OriginDeclared {
for _, h := range known.Held {
if declared[h.ID] {
continue
}
steps = append(steps, Step{Verb: "forget", Type: h.Kind, ID: h.ID, Target: h.Target,
Why: "held for " + h.Module + " and no longer declared; left as found"})
}
}
var protecting, orphans []Step
made := meshMadeUnits(known)
for _, orphan := range known.Orphans(declared, origin) {
step := Step{Verb: "remove", Type: orphan.Type, ID: orphan.ID, Target: orphan.Target,
Why: "recorded here and no longer declared"}
switch {
case orphan.Stateless:
step.Verb, step.Why = "forget", "no longer declared; its unit's state was never the mesh's and is left as it is"
case orphan.Type == string(declaration.TypeService):
// What removal will do, said before it does it (novox/hq ADR 0118), in removeService's
// words. "restore" only where it may stop or disable something — the record cannot say
// whether the unit is still as the mesh left it, so "may" is as far as a preview goes —
// and a unit whose file the mesh wrote is named as the mesh's, since that one is
// stopped whatever was found.
f := orphan.Found
switch {
case made[orphan.Target]:
step.Verb, step.Why = "remove", "no longer declared; the mesh wrote its unit file, so it is "+
"stopped and disabled at boot before that file goes"
case f == nil:
step.Verb, step.Why = "forget", "no longer declared; recorded before the host kept what it "+
"found, so it is left as it is"
case f.State == "stopped" || f.Boot == "disabled":
var back []string
if f.State == "stopped" {
back = append(back, "stopped")
}
if f.Boot == "disabled" {
back = append(back, "disabled at boot")
}
step.Verb, step.Why = "restore", "no longer declared; the host found it "+
strings.Join(back, " and ")+", and it goes back to that if the mesh changed it"
default:
step.Verb, step.Why = "forget", "no longer declared; it was running before the mesh and is "+
"left as it is — nothing is started or stopped on the way out"
}
}
if d.Adoption == nil && strings.HasPrefix(orphan.ID, declaration.AdoptionPrefix) {
step.Why = "what protected this node while adopted; removed last, once everything else applied"
protecting = append(protecting, step)
continue
}
orphans = append(orphans, step)
}
// The guard goes up before anything is removed on an adopted node; on a converged one the
// removals go first (novox/hq ADR 0103).
var guard, rest []declaration.Resource
for _, r := range d.Resources {
if d.Adoption != nil && strings.HasPrefix(r.Identity(), guardPrefix) {
guard = append(guard, r)
continue
}
rest = append(rest, r)
}
for _, r := range guard {
steps = append(steps, planned(r, d, known))
}
steps = append(steps, orphans...)
for _, r := range rest {
step := planned(r, d, known)
if svc, ok := r.(*declaration.Service); ok && svc.TakesOver != nil && d.Adoption != nil && step.Verb != "hold" {
// The take comes before the service that replaces the tunnel, as it does in the apply.
steps = append(steps, plannedTake(svc, known))
}
steps = append(steps, step)
}
// Only a declaration from the mesh converges a node; a bundle or a file never retires the
// firewall found here, and neither says so in a plan.
if d.Adoption == nil && origin == store.OriginDeclared && ufw && rec.WasActive && !rec.DisabledByMesh {
steps = append(steps, Step{Verb: "disable", Type: "firewall", ID: "ufw",
Why: "this node converges: retired once the mesh's own filter is loaded, never before; " +
"its configuration stays on disk"})
}
steps = append(steps, protecting...)
return steps
}
// planned is what one declared resource would come to.
//
// **In the order holdOnAdopted decides it**, because the one cutover ADR 0100 says must be
// previewed is the one a plan gets backwards if it looks at the record first: a resource this
// node holds for a module the declaration now says is taken is not held any longer — it is
// applied, and what was found is replaced. The declaration's word on which modules are untaken
// comes first; the record of what is held only says what that replacement replaces.
func planned(r declaration.Resource, d *declaration.Declaration, known store.State) Step {
step := Step{Type: string(r.Kind()), ID: r.Identity(), Target: r.Target()}
if d.Adoption != nil {
// Something run inside a held container is held with it, while that container's module
// is untaken; once the module is taken the container is replaced before this runs.
if in := runsIn(r); in != "" {
if container, isHeld := heldContainer(known, in); isHeld {
if _, untaken := d.Adoption.Untaken[container.Module]; untaken {
step.Verb = "hold"
step.Why = "runs in " + in + ", which is held as found; not run until " + container.Module + " is taken"
return step
}
}
}
// A file written into, or a service whose lifecycle is the machine's, replaces nothing that
// was found, so it is never held (ADR 0102, ADR 0117).
into := replacesNothing(r)
h, held := known.HeldAt(r.Identity())
module, untaken := d.Adoption.UntakenModuleOf(r.Identity())
switch {
case into:
case untaken && held:
step.Verb, step.Why = "hold", "found on this machine and kept as it is until "+module+" is taken"
return step
case untaken:
step.Verb = "create"
step.Why = "unless it is found on this machine — then held as it is until " + module + " is taken"
return step
case held:
// The cutover: the module is taken, and what was held for it is replaced.
step.Verb = "create"
if _, recorded := known.Find(r.Identity()); recorded {
step.Verb = "update"
}
step.Why = h.Module + " is taken: replaces what was found and held"
if h.Kept != "" {
step.Why += "; the original stays at " + h.Kept
}
return step
}
}
if a, ok := r.(*declaration.Action); ok {
// Named as what it is. Its verify decides whether it runs, and that is read from the
// machine, not the record.
step.Verb = "run"
step.Target = ""
step.Why = fmt.Sprintf("an action: `%s`, unless its verify `%s` already passes; if it fails, "+
"nothing after it is attempted", strings.Join(a.Command, " "), strings.Join(a.Verify, " "))
if a.In != "" {
step.Why = "in " + a.In + ", " + step.Why
}
return step
}
if svc, ok := r.(*declaration.Service); ok && svc.Stateless() {
// Nothing is created: the unit and whether it runs are the machine's (novox/hq ADR 0117).
step.Verb, step.Why = "check", "its lifecycle is the machine's; reloaded or restarted only if "+
"running when what it reflects changes"
return step
}
was, recorded := known.Find(r.Identity())
if !recorded {
step.Verb, step.Why = "create", "no record of it on this node"
return step
}
if want := wouldWrite(r); want != "" && was.Wrote != "" && want != was.Wrote {
step.Verb, step.Why = "update", "the declaration changed since this host applied it"
return step
}
if c, ok := r.(*declaration.Container); ok {
if changed := readsChanged(c, d, known, was.Reads); len(changed) > 0 {
step.Verb = "update"
step.Why = "recreated: " + strings.Join(changed, ", ") + " changed since it was created"
return step
}
}
step.Verb, step.Why = "check", "recorded here; corrected if this machine drifted from it"
return step
}
// plannedTake is what the take of a found tunnel would do to its configuration (novox/hq ADR 0105,
// ADR 0119): kept as found while the take is not proven, and retired — removed from where its unit
// reads it, its original staying kept — by the first apply that finds the mesh's interface up in
// its place with a peer handshaken. Whether that is this apply is read from the machine, which a
// plan does not do, so it says when rather than whether. One the mesh retired already is said as
// retired: nothing brings it back.
func plannedTake(svc *declaration.Service, known store.State) Step {
t := svc.TakesOver
step := Step{Verb: "hold", Type: string(declaration.TypeFile), ID: takeOverID(svc), Target: t.Config}
if r, ok := known.RetiredAt(t.Config); ok {
step.Verb = "check"
step.Why = "retired once the take of " + t.Interface + " was proven; its original stays at " + r.Kept +
" and the mesh never brings it back"
return step
}
step.Why = "the configuration of the tunnel " + t.Interface + ", kept as found while " + svc.Unit +
" takes it over (" + t.Unit + " stopped and disabled, never flushed); retired — removed from " +
t.Config + ", its original staying kept — once the take is proven by a peer handshaking on " +
strings.TrimPrefix(svc.Unit, "wg-quick@")
if h, ok := known.HeldAt(takeOverID(svc)); ok && h.Kept != "" {
step.Why += "; the original is at " + h.Kept
}
return step
}
// readsChanged is which of the files a container was created reading the apply will hand it
// changed — the same comparison applyContainer makes (novox/hq 04-ISSUES/103), settled from the
// declaration and the record alone.
//
// A file's digest is what this apply will record for it: a plain file declared here, by its
// declared content; otherwise what this host last wrote there, under any id. A file neither
// declares nor records — an env-file a predecessor left — is read by the apply from the machine,
// which a plan does not do, so it stays a check. A container with no record of what it read was
// labelled before the host kept that record and is accepted as it is, so it is a check too.
func readsChanged(c *declaration.Container, d *declaration.Declaration, known store.State,
wasReading map[string]string) []string {
if len(wasReading) == 0 {
return nil
}
willWrite := map[string]string{}
for _, r := range d.Resources {
if f, ok := r.(*declaration.File); ok {
if want := wouldWrite(f); want != "" {
willWrite[f.Path] = want
}
}
}
// Only what it still reads: a file it was created reading and no longer names is a changed
// declaration, not a changed file.
stillReads := map[string]bool{}
for _, path := range c.EnvFile {
stillReads[path] = true
}
for _, v := range c.Volumes {
if src := mountSource(v); strings.HasPrefix(src, "/") {
stillReads[src] = true
}
}
var changed []string
for _, path := range sortedKeys(wasReading) {
if !stillReads[path] {
continue
}
now, settled := willWrite[path]
if !settled {
if f, recorded := known.At(string(declaration.TypeFile), path); recorded {
now, settled = f.Wrote, true
}
}
if settled && now != wasReading[path] {
changed = append(changed, path)
}
}
return changed
}
// wouldWrite is the digest a plain file would be recorded under, or empty where only the apply
// can know: a sealed file, one with secrets in it, one written into, one carrying bytes.
func wouldWrite(r declaration.Resource) string {
f, ok := r.(*declaration.File)
if !ok || f.Into != "" || f.Bytes != "" || f.Secret() || len(f.Secrets) > 0 {
return ""
}
return digestOf(f.Content)
}
+302
View File
@@ -0,0 +1,302 @@
package apply
import (
"context"
"os"
"path/filepath"
"strings"
"testing"
"time"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/store"
)
// Defends novox/hq issue 104: what an apply would change is said before anything is, from the
// declaration and the node's record — and an action is named as the action it is.
func trusted(t *testing.T, raw string) *declaration.Declaration {
t.Helper()
d, err := declaration.ParseFileTrusted([]byte(raw))
if err != nil {
t.Fatalf("fixture is not a valid declaration: %v", err)
}
return d
}
func verbs(steps []Step) string {
var out []string
for _, s := range steps {
out = append(out, s.Verb+" "+s.ID)
}
return strings.Join(out, ", ")
}
func TestAPlanNamesAnActionAsAnAction(t *testing.T) {
d := trusted(t, `{"declaration":1,"resources":[
{"id":"init","type":"action","command":["createdb","mesh"],"verify":["psql","-c","select 1"]}]}`)
steps := Plan(d, store.State{}, store.OriginCarried)
if len(steps) != 1 || steps[0].Verb != "run" {
t.Fatalf("an action was planned as %s", verbs(steps))
}
if !strings.Contains(steps[0].Why, "createdb mesh") || !strings.Contains(steps[0].Why, "nothing after it") {
t.Errorf("the plan does not say what the action runs and what failing it means: %q", steps[0].Why)
}
}
func TestAPlanSaysWhatIsRecordedAndWhatIsNot(t *testing.T) {
d := parse(t, `{"declaration":1,"resources":[
{"id":"new","type":"file","path":"/tmp/new","content":"a\n"},
{"id":"same","type":"file","path":"/tmp/same","content":"b\n"},
{"id":"moved","type":"file","path":"/tmp/moved","content":"c\n"},
{"id":"sealed","type":"file","path":"/tmp/sealed","sealed":"AAAA","mode":"0600"}]}`)
known := store.State{}
known.Record(store.Applied{ID: "same", Type: "file", Target: "/tmp/same", Wrote: digestOf("b\n")})
known.Record(store.Applied{ID: "moved", Type: "file", Target: "/tmp/moved", Wrote: digestOf("old\n")})
known.Record(store.Applied{ID: "sealed", Type: "file", Target: "/tmp/sealed", Wrote: digestOf("secret")})
known.Record(store.Applied{ID: "gone", Type: "file", Target: "/tmp/gone"})
got := verbs(Plan(d, known, store.OriginCarried))
want := "remove gone, create new, check same, update moved, check sealed"
if got != want {
t.Errorf("planned %q, want %q", got, want)
}
}
func TestAPlanSaysTheFirewallAConvergingNodeRetires(t *testing.T) {
d := parse(t, `{"declaration":1,"resources":[{"id":"a","type":"file","path":"/tmp/a","content":"x\n"}]}`)
known := store.State{Firewall: &store.FoundFirewall{Kind: "ufw", WasActive: true, FoundAt: time.Now()}}
known.Record(store.Applied{ID: "adoption.guard.table", Type: "file", Target: "/etc/guard", Origin: store.OriginDeclared})
got := verbs(Plan(d, known, store.OriginDeclared))
// The firewall goes last but for what protected the node, which goes after it.
if got != "create a, disable ufw, remove adoption.guard.table" {
t.Errorf("a converging node planned %q", got)
}
// A file or the bundle never retires the firewall found here, and says nothing about it.
if got := verbs(Plan(d, known, store.OriginCarried)); strings.Contains(got, "ufw") {
t.Errorf("a carried declaration planned to touch the firewall: %q", got)
}
}
func TestAPlanHoldsWhatAnAdoptedNodeFound(t *testing.T) {
d := parse(t, `{"declaration":1,"adoption":{"taken":[],"untaken":{"hello-web":["hello-web.page","hello-web.server"]}},
"resources":[
{"id":"hello-web.page","type":"file","path":"/srv/index.html","content":"x\n"},
{"id":"hello-web.server","type":"container","name":"hello-web","image":"example/web@sha256:0000000000000000000000000000000000000000000000000000000000000000"}]}`)
known := store.State{}
known.RecordHeld(store.Held{ID: "hello-web.page", Module: "hello-web", Kind: "file", Target: "/srv/index.html"})
steps := Plan(d, known, store.OriginDeclared)
if len(steps) != 2 || steps[0].Verb != "hold" || steps[1].Verb != "create" {
t.Fatalf("an adopted node planned %s", verbs(steps))
}
if !strings.Contains(steps[1].Why, "held as it is until hello-web is taken") {
t.Errorf("the plan does not say an untaken module's resource is held if found: %q", steps[1].Why)
}
}
// both is a machine with a container runtime and ufw on it at once.
type both struct {
m *machine
u *ufwMachine
}
func (b *both) run(ctx context.Context, name string, args ...string) (string, error) {
switch name {
case "nft", "ufw", "iptables", "ip6tables", "firewall-cmd":
return b.u.run(ctx, name, args...)
}
return b.m.run(ctx, name, args...)
}
func ids(steps []Step) []string {
var out []string
for _, s := range steps {
if s.Type == "firewall" {
continue // said, not an outcome
}
out = append(out, s.ID)
}
return out
}
func outcomeIDs(r Report) []string {
var out []string
for _, o := range r.Outcomes {
out = append(out, o.ID)
}
return out
}
func write(t *testing.T, path, content string) {
t.Helper()
if err := os.WriteFile(path, []byte(content), 0o644); err != nil {
t.Fatal(err)
}
}
// Defends novox/hq issue 104: the plan is the apply, said first — the same resources in the same
// order, and the one cutover ADR 0100 says must be previewed said as a cutover, not a hold.
func TestThePlanIsTheApplyInOrder(t *testing.T) {
dir := t.TempDir()
guard, page, keep, old := filepath.Join(dir, "guard.nft"), filepath.Join(dir, "index.html"),
filepath.Join(dir, "keep.html"), filepath.Join(dir, "old.conf")
for _, p := range []string{page, keep, old} {
write(t, p, "the predecessor's\n")
}
// Returning to adopted, with a guard to raise, an orphan, a hold that stays, a hold whose
// module is now taken, and a hold no longer declared.
back := adopted(t, `{"taken":["hello-web"],"untaken":{"keep":["keep.page"]}}`,
`{"id":"adoption.guard.table","type":"file","path":"`+guard+`","content":"table inet mesh-guard {}\n"},
{"id":"hello-web.page","type":"file","path":"`+page+`","content":"the mesh's page\n"},
{"id":"keep.page","type":"file","path":"`+keep+`","content":"the mesh's keep\n"}`)
known := store.State{Firewall: &store.FoundFirewall{Kind: "ufw", WasActive: true, DisabledByMesh: true, FoundAt: time.Now()}}
known.Record(store.Applied{ID: "old.conf", Type: "file", Target: old, Origin: store.OriginDeclared})
for id, module := range map[string]string{"hello-web.page": "hello-web", "keep.page": "keep", "gone.page": "gone"} {
known.RecordHeld(store.Held{ID: id, Module: module, Kind: "file", Target: filepath.Join(dir, id), Since: time.Now()})
}
fake := &both{m: &machine{containers: map[string]*fakeContainer{}}, u: &ufwMachine{installed: true}}
plan := Plan(back, known, store.OriginDeclared)
report, state, err := ApplyKeeping(context.Background(), archHost(t), back, known, store.OriginDeclared,
fake.run, nil, nil, KeepIn(dir))
if err != nil {
t.Fatal(err)
}
if got, want := verbs(plan), "enable ufw, forget gone.page, create adoption.guard.table, remove old.conf, "+
"create hello-web.page, hold keep.page"; got != want {
t.Errorf("planned %q, want %q", got, want)
}
if got, want := strings.Join(ids(plan), " "), strings.Join(outcomeIDs(report), " "); got != want {
t.Errorf("the plan said %q and the apply did %q", got, want)
}
taken := outcomeOf(report, "hello-web.page")
if !strings.Contains(taken.Detail, "taken") || !strings.Contains(plan[4].Why, "hello-web is taken: replaces what was found") {
t.Errorf("the cutover was applied as %q and planned as %q", taken.Detail, plan[4].Why)
}
if !fake.u.active || state.Firewall.DisabledByMesh {
t.Error("returning to adopted did not enable ufw again")
}
// Converged, from the mesh: an orphan, a new file, ufw retired, and what protected the node
// removed last.
flip := parse(t, `{"declaration":1,"resources":[{"id":"a","type":"file","path":"`+filepath.Join(dir, "a.conf")+`","content":"a\n"}]}`)
write(t, old, "again\n")
known = store.State{Firewall: &store.FoundFirewall{Kind: "ufw", WasActive: true, FoundAt: time.Now()}}
known.Record(store.Applied{ID: "adoption.guard.table", Type: "file", Target: guard, Origin: store.OriginDeclared})
known.Record(store.Applied{ID: "old.conf", Type: "file", Target: old, Origin: store.OriginDeclared})
fake = &both{m: &machine{containers: map[string]*fakeContainer{}}, u: &ufwMachine{installed: true, active: true,
ruleset: "table inet mesh {\n}\n"}}
plan = Plan(flip, known, store.OriginDeclared)
report, state, err = ApplyKeeping(context.Background(), archHost(t), flip, known, store.OriginDeclared,
fake.run, nil, nil, KeepIn(dir))
if err != nil {
t.Fatal(err)
}
if got, want := verbs(plan), "remove old.conf, create a, disable ufw, remove adoption.guard.table"; got != want {
t.Errorf("planned %q, want %q", got, want)
}
if got, want := strings.Join(ids(plan), " "), strings.Join(outcomeIDs(report), " "); got != want {
t.Errorf("the plan said %q and the apply did %q", got, want)
}
if fake.u.active || !state.Firewall.DisabledByMesh {
t.Error("converging did not retire ufw")
}
}
func TestAResourceRunInAHeldContainerIsPlannedAsItIsApplied(t *testing.T) {
// A run-once step sharing a container's namespace runs in it, as an action's `in` does.
img := "example/x@sha256:0000000000000000000000000000000000000000000000000000000000000000"
d := parse(t, `{"declaration":1,"adoption":{"taken":["web"],"untaken":{"db":["db.server"]}},"resources":[
{"id":"web.server","type":"container","name":"web","image":"`+img+`"},
{"id":"db.server","type":"container","name":"db","image":"`+img+`"},
{"id":"db.init","type":"container","name":"db-init","image":"`+img+`","run-once":true,"network":"container:db"},
{"id":"web.warm","type":"container","name":"web-warm","image":"`+img+`","run-once":true,"network":"container:web"}]}`)
known := store.State{}
known.RecordHeld(store.Held{ID: "db.server", Module: "db", Kind: "container", Target: "db", Container: "predecessor"})
known.RecordHeld(store.Held{ID: "web.server", Module: "web", Kind: "container", Target: "web", Container: "predecessor"})
got := verbs(Plan(d, known, store.OriginDeclared))
// db is untaken, so what runs in it waits; web is taken, so its held container is replaced (the
// cutover) and what runs in it runs.
if got != "create web.server, hold db.server, hold db.init, create web.warm" {
t.Errorf("planned %q", got)
}
}
func TestAPlanSaysAContainerIsRecreatedWhenAFileItReadsChanged(t *testing.T) {
// The apply recreates a container when the content of a file it reads at creation changed
// (novox/hq 04-ISSUES/103); the plan says so from the record alone — what the container was
// created reading, against what this apply will write. And a container recorded before the
// host kept that record is accepted, so it is a check, not an update.
dir := t.TempDir()
env := filepath.Join(dir, "forge.env")
declare := func(port string) *declaration.Declaration {
return trusted(t, `{"declaration":1,"resources":[
{"id":"forge.env","type":"file","path":"`+env+`","content":"DATABASE_PORT=`+port+`\n"},
{"id":"forge.server","type":"container","name":"forge","image":"`+pinned+`","env-file":["`+env+`"]}]}`)
}
created := digestOf("DATABASE_PORT=5432\n")
known := store.State{}
known.Record(store.Applied{ID: "forge.env", Type: "file", Target: env, Wrote: created})
known.Record(store.Applied{ID: "forge.server", Type: "container", Target: "forge",
Reads: map[string]string{env: created}})
if got := verbs(Plan(declare("5432"), known, store.OriginCarried)); got != "check forge.env, check forge.server" {
t.Errorf("nothing changed and the plan says %q", got)
}
steps := Plan(declare("5433"), known, store.OriginCarried)
if got := verbs(steps); got != "update forge.env, update forge.server" {
t.Fatalf("the env-file changes and the plan says %q", got)
}
if !strings.Contains(steps[1].Why, env+" changed") {
t.Errorf("the plan does not say which file: %+v", steps[1])
}
// No record of what it read: labelled by an earlier host, accepted as it is.
known.Record(store.Applied{ID: "forge.server", Type: "container", Target: "forge"})
if got := verbs(Plan(declare("5433"), known, store.OriginCarried)); got != "update forge.env, check forge.server" {
t.Errorf("a container with no record of what it read is planned as %q", got)
}
}
func TestAPlanSaysWhichUnitsAnUndeclareGivesBackAndWhichItLeaves(t *testing.T) {
// novox/hq ADR 0118, said before it is done: "restore" only where removal may stop or disable
// something, and a unit whose file the mesh wrote named as the mesh's.
known := store.State{}
known.Record(store.Applied{ID: "guard-unit", Type: "file", Target: "/etc/systemd/system/mesh-guard.service"})
known.Record(store.Applied{ID: "guard", Type: "service", Target: "mesh-guard.service"})
known.Record(store.Applied{ID: "filter", Type: "service", Target: "filter.service",
Found: &store.FoundUnit{State: "stopped"}})
known.Record(store.Applied{ID: "boot", Type: "service", Target: "boot.service",
Found: &store.FoundUnit{State: "running", Boot: "disabled"}})
known.Record(store.Applied{ID: "runtime", Type: "service", Target: "docker.service",
Found: &store.FoundUnit{State: "running", Boot: "enabled"}})
known.Record(store.Applied{ID: "old", Type: "service", Target: "sshd.service"})
known.Record(store.Applied{ID: "nm", Type: "service", Target: "NetworkManager.service", Stateless: true})
steps := Plan(parse(t, nothingButA(t)), known, store.OriginCarried)
got := verbs(steps)
want := "forget nm, forget old, forget runtime, restore boot, restore filter, remove guard, remove guard-unit, create other"
if got != want {
t.Fatalf("planned %s\nwant %s", got, want)
}
for _, s := range steps {
switch s.ID {
case "guard":
if !strings.Contains(s.Why, "unit file") {
t.Errorf("the mesh's own unit was not named as the mesh's: %q", s.Why)
}
case "filter":
if !strings.Contains(s.Why, "found it stopped") {
t.Errorf("what the unit goes back to went unsaid: %q", s.Why)
}
case "old":
if !strings.Contains(s.Why, "left as it is") {
t.Errorf("a unit with nothing found was not said to be left: %q", s.Why)
}
}
}
}
+55 -3
View File
@@ -30,10 +30,11 @@ import (
// Under the mesh's own directory rather than somewhere a distribution owns: these are files the
// mesh puts there and replaces, and putting them where a package manager also writes is how two
// owners end up disagreeing about one path.
const daemonRoot = "/var/lib/mesh/daemons"
var daemonRoot = "/var/lib/mesh/daemons"
// unitDir is where the mesh writes the units it owns.
const unitDir = "/etc/systemd/system"
// unitDir is where the mesh writes the units it owns. A variable only so a test can point it at a
// directory of its own.
var unitDir = "/etc/systemd/system"
func applyProcess(ctx context.Context, r *declaration.Process, run Runner,
changed map[string]bool, previous store.Applied) (Outcome, error) {
@@ -289,3 +290,54 @@ func unitValue(key, value string) string {
).Replace(value)
return `"` + key + "=" + escaped + `"`
}
// removeProcess takes away a process the mesh no longer declares: its timer and unit stopped and
// disabled, their files removed, the supervisor told, and the unpacked bundle deleted.
//
// **All of it is the host's**, which is why all of it goes (novox/hq ADR 0118). Before this there
// was no way to remove a process at all, and one left undeclared failed every apply on its node
// until someone edited the host's state by hand — unassigning any module that ran its own code
// stranded the machine.
//
// Idempotent, like every removal: a unit already gone is not an error, and a record whose files
// have all vanished is forgotten rather than reported as removed.
func removeProcess(ctx context.Context, a store.Applied, run Runner) (string, string, error) {
name := a.Target
if problem := declaration.ProcessNameProblem(name); problem != "" {
// The name is a unit name and a directory under the mesh's own, and what goes is that
// directory, whole. One that could climb out of either — ".." is the mesh's own directory's
// parent — is refused rather than acted on, whatever wrote it into the record.
return "", "", fmt.Errorf("a process recorded under %q cannot be removed by name: %s", name, problem)
}
found := false
for _, unit := range []string{name + ".timer", name + ".service"} {
path := filepath.Join(unitDir, unit)
if _, err := os.Stat(path); err != nil {
continue
}
found = true
// The timer first, so a scheduled run cannot start the service between the two.
if _, err := run(ctx, "systemctl", "disable", "--now", unit); err != nil {
return "", "", fmt.Errorf("stopping %s: %w", unit, err)
}
if err := os.Remove(path); err != nil && !os.IsNotExist(err) {
return "", "", err
}
}
if found {
if _, err := run(ctx, "systemctl", "daemon-reload"); err != nil {
return "", "", err
}
}
bundle := filepath.Join(daemonRoot, name)
if _, err := os.Stat(bundle); err == nil {
found = true
if err := os.RemoveAll(bundle); err != nil {
return "", "", err
}
}
if !found {
return "forgotten", "no longer there", nil
}
return "removed", "stopped; its unit and its bundle removed — the mesh's own code", nil
}
+93
View File
@@ -1,10 +1,14 @@
package apply
import (
"context"
"os"
"path/filepath"
"strings"
"testing"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/store"
)
func aProcess() *declaration.Process {
@@ -163,3 +167,92 @@ func TestAnEnvironmentValueMeansWhatTheDeclarationSaid(t *testing.T) {
t.Fatalf("a quote was not escaped, so the value ends early: %s", line)
}
}
func TestAnUndeclaredProcessIsRemovedWithItsUnitAndBundle(t *testing.T) {
// novox/hq ADR 0118: a process's unit and bundle are the host's own, so they go with the
// declaration. Before, there was no way to remove a process at all, and one left undeclared
// failed every apply on its node.
units, bundles := t.TempDir(), t.TempDir()
wasUnits, wasBundles := unitDir, daemonRoot
unitDir, daemonRoot = units, bundles
t.Cleanup(func() { unitDir, daemonRoot = wasUnits, wasBundles })
for _, f := range []string{"mesh-job.service", "mesh-job.timer"} {
if err := os.WriteFile(filepath.Join(units, f), []byte("[Unit]\n"), 0o644); err != nil {
t.Fatal(err)
}
}
if err := os.MkdirAll(filepath.Join(bundles, "mesh-job", "bin"), 0o755); err != nil {
t.Fatal(err)
}
var commands []string
run := func(ctx context.Context, name string, args ...string) (string, error) {
commands = append(commands, strings.Join(args, " "))
return "", nil
}
known := store.State{Resources: []store.Applied{{ID: "p", Type: "process", Target: "mesh-job"}}}
d := parse(t, `{"declaration":1,"resources":[
{"id":"f","type":"file","path":"`+filepath.Join(t.TempDir(), "a")+`","content":"a\n"}
]}`)
report, after, err := Apply(context.Background(), archHost(t), d, known, store.OriginCarried, run, nil, nil)
if err != nil {
t.Fatalf("an undeclared process failed the apply: %v", err)
}
joined := strings.Join(commands, "; ")
timer, service := strings.Index(joined, "disable --now mesh-job.timer"), strings.Index(joined, "disable --now mesh-job.service")
if timer < 0 || service < 0 || timer > service {
t.Errorf("the timer and the unit were not stopped, timer first: %s", joined)
}
if !strings.Contains(joined, "daemon-reload") {
t.Errorf("the service manager was not told its units changed: %s", joined)
}
for _, gone := range []string{filepath.Join(units, "mesh-job.service"), filepath.Join(units, "mesh-job.timer"), filepath.Join(bundles, "mesh-job")} {
if _, err := os.Stat(gone); !os.IsNotExist(err) {
t.Errorf("%s is still there", gone)
}
}
if o := outcomeOf(report, "p"); o.Action != "removed" {
t.Errorf("outcome %+v, want removed", o)
}
if _, still := after.Find("p"); still {
t.Error("the host still believes it owns the process")
}
// Again, with everything already gone: forgotten, not an error.
_, _, err = Apply(context.Background(), archHost(t), d, known, store.OriginCarried, run, nil, nil)
if err != nil {
t.Errorf("removing a process that is already gone failed: %v", err)
}
}
func TestAProcessRecordedUnderAPathlikeNameIsRefusedNotRemoved(t *testing.T) {
// filepath.Join(daemonRoot, "..") is the mesh's own directory, and removal deletes what that
// names, whole. A record is what some host wrote, perhaps under looser rules than today's, so
// the removal holds the name to the declaration's rule again (novox/hq ADR 0118).
root := t.TempDir()
bundles := filepath.Join(root, "daemons")
wasUnits, wasBundles := unitDir, daemonRoot
unitDir, daemonRoot = t.TempDir(), bundles
t.Cleanup(func() { unitDir, daemonRoot = wasUnits, wasBundles })
precious := filepath.Join(root, "state.json")
if err := os.MkdirAll(filepath.Join(bundles, "other"), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(precious, []byte("{}"), 0o600); err != nil {
t.Fatal(err)
}
for _, name := range []string{"..", ".", "", "-x"} {
known := store.State{Resources: []store.Applied{{ID: "p", Type: "process", Target: name}}}
d := parse(t, `{"declaration":1,"resources":[
{"id":"f","type":"file","path":"`+filepath.Join(t.TempDir(), "a")+`","content":"a\n"}
]}`)
if _, _, err := Apply(context.Background(), archHost(t), d, known, store.OriginCarried, noServices, nil, nil); err == nil {
t.Errorf("a process recorded as %q was removed by name", name)
}
for _, still := range []string{precious, filepath.Join(bundles, "other")} {
if _, err := os.Stat(still); err != nil {
t.Fatalf("removing a process recorded as %q took %s with it", name, still)
}
}
}
}
+316
View File
@@ -0,0 +1,316 @@
package apply
import (
"context"
"os"
"path/filepath"
"strings"
"testing"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/store"
)
// Defends novox/hq 04-ISSUES/103: a container is recreated when the CONTENT of a file it reads at
// creation changes, not only when its path does. A container takes its env-file and its mounted
// files in once, when it is created; `docker restart` hands it the same environment again, so
// only a recreate carries a rewritten file into the process.
//
// The runtime here is the `machine` fake: it keeps the spec label the host gave a container and
// hands it back on inspect, so the comparison under test is the one the host really makes,
// against what it really wrote — not against a spec a test imagined.
func applyCarried(t *testing.T, d *declaration.Declaration, known store.State, m *machine,
log func(string)) (Report, store.State) {
t.Helper()
report, state, err := Apply(context.Background(), archHost(t), d, known, store.OriginCarried, m.run, log, nil)
if err != nil {
t.Fatalf("apply failed: %v", err)
}
return report, state
}
func TestAContainerIsRecreatedWhenItsEnvFileChanged(t *testing.T) {
// The night of the issue: the store was given a new port, the host rewrote the forge's
// environment file with it — and left the forge running with the old one.
dir := t.TempDir()
env := filepath.Join(dir, "forge.env")
declare := func(port string) *declaration.Declaration {
return parseTrusted(t, `{"declaration":1,"resources":[
{"id":"forge.env","type":"file","path":"`+env+`","content":"DATABASE_PORT=`+port+`\n","mode":"0600"},
{"id":"forge.server","type":"container","name":"forge","image":"`+pinned+`","env-file":["`+env+`"]}
]}`)
}
m := &machine{containers: map[string]*fakeContainer{}}
var logged []string
log := func(line string) { logged = append(logged, line) }
report, state := applyCarried(t, declare("5432"), store.State{}, m, log)
if o := outcomeOf(report, "forge.server"); o.Action != "created" {
t.Fatalf("the container was not created: %+v", report.Outcomes)
}
// The store moved. The file is rewritten in this apply, before the container is reached, and
// the container must follow it in the same pass.
m.asked, logged = nil, nil
report, state = applyCarried(t, declare("5433"), state, m, log)
if !m.removed("forge") || !m.did("docker run") {
t.Fatalf("the container kept running with the old environment after its env-file changed: %v", m.asked)
}
o := outcomeOf(report, "forge.server")
if o.Action != "updated" || o.Detail != "recreated: "+env+" changed" {
t.Errorf("the recreate did not say which file changed: %+v", o)
}
var said bool
for _, line := range logged {
if strings.Contains(line, "updated forge.server") && strings.Contains(line, "recreated: "+env+" changed") {
said = true
}
}
if !said {
t.Errorf("the log did not say which file made the container recreate: %q", logged)
}
// And with nothing moved, it is left alone: content is part of the identity, not a tripwire.
m.asked = nil
report, _ = applyCarried(t, declare("5433"), state, m, log)
if m.did("docker rm") || m.did("docker run") || report.Changed() {
t.Errorf("a container whose env-file did not change was recreated: %v %+v", m.asked, report.Outcomes)
}
}
func TestAnEnvFileTheHostDidNotWriteIsStillReadForWhatItHolds(t *testing.T) {
// The host has no record of this file — a predecessor left it, or something else on the
// machine maintains it — and the container still reads it once. Its content is read from the
// disk, so a change is a recreate exactly as for a file the host wrote.
dir := t.TempDir()
env := filepath.Join(dir, "app.env")
if err := os.WriteFile(env, []byte("TOKEN=old\n"), 0o600); err != nil {
t.Fatal(err)
}
d := parseTrusted(t, `{"declaration":1,"resources":[
{"id":"app.server","type":"container","name":"app","image":"`+pinned+`","env-file":["`+env+`"]}
]}`)
m := &machine{containers: map[string]*fakeContainer{}}
_, state := applyCarried(t, d, store.State{}, m, nil)
if err := os.WriteFile(env, []byte("TOKEN=new\n"), 0o600); err != nil {
t.Fatal(err)
}
m.asked = nil
report, _ := applyCarried(t, d, state, m, nil)
if !m.removed("app") || !m.did("docker run") {
t.Fatalf("a container reading an env-file the host did not write was not recreated when it changed: %v", m.asked)
}
if o := outcomeOf(report, "app.server"); o.Detail != "recreated: "+env+" changed" {
t.Errorf("the recreate did not name the file: %+v", o)
}
}
func TestAContainerIsRecreatedWhenAMountedSecretChanged(t *testing.T) {
// A rotated credential has the same shape as a moved port: the host writes the file the
// container mounts, and the process holds the value it was created with.
dir := t.TempDir()
secret := filepath.Join(dir, "db.secret")
declare := func(value string) *declaration.Declaration {
return parseTrusted(t, `{"declaration":1,"resources":[
{"id":"app.secret","type":"file","path":"`+secret+`","content":"`+value+`","mode":"0600"},
{"id":"app.server","type":"container","name":"app","image":"`+pinned+`",
"volumes":["`+secret+`:/run/secrets/db:ro"]}
]}`)
}
m := &machine{containers: map[string]*fakeContainer{}}
_, state := applyCarried(t, declare("hunter2"), store.State{}, m, nil)
m.asked = nil
report, _ := applyCarried(t, declare("correct-horse-battery-staple"), state, m, nil)
if !m.removed("app") || !m.did("docker run") {
t.Fatalf("the container kept the secret it was created with after the mounted file changed: %v", m.asked)
}
if o := outcomeOf(report, "app.server"); o.Action != "updated" || o.Detail != "recreated: "+secret+" changed" {
t.Errorf("the recreate did not say which file changed: %+v", o)
}
}
func TestAMountedDirectoryIsNotLookedInside(t *testing.T) {
// A bind-mounted directory is not part of what a container is — not the data the service
// grows in it, and not the files the host itself writes there either. Whether a service reads
// a file under its directory once at start or watches it live is the service's business: the
// route proxy re-reads its routes live, a provisioner sidecar polls what it receives every few
// seconds, and recreating either for a file the host rewrote would kill them for nothing. A
// module whose container does read such a file once says so with restart-on, which stays the
// opt-in.
dir := t.TempDir()
state := filepath.Join(dir, "state")
config := filepath.Join(state, "config.toml")
declare := func(level, restartOn string) *declaration.Declaration {
return parseTrusted(t, `{"declaration":1,"resources":[
{"id":"app.state","type":"directory","path":"`+state+`"},
{"id":"app.config","type":"file","path":"`+config+`","content":"level = \"`+level+`\"\n"},
{"id":"app.server","type":"container","name":"app","image":"`+pinned+`",
"volumes":["`+state+`:/var/lib/app"]`+restartOn+`}
]}`)
}
m := &machine{containers: map[string]*fakeContainer{}}
_, known := applyCarried(t, declare("info", ""), store.State{}, m, nil)
// The service grows its data in the directory it was given.
if err := os.WriteFile(filepath.Join(state, "app.db"), []byte("rows"), 0o600); err != nil {
t.Fatal(err)
}
if err := os.MkdirAll(filepath.Join(state, "cache"), 0o700); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(state, "cache", "index"), []byte("entries"), 0o600); err != nil {
t.Fatal(err)
}
m.asked = nil
report, known := applyCarried(t, declare("info", ""), known, m, nil)
if m.did("docker rm") || m.did("docker run") || report.Changed() {
t.Errorf("a container was recreated for data its service wrote in a mounted directory: %v %+v",
m.asked, report.Outcomes)
}
// The host rewrites its own file under the same directory: still not a reason. The container
// did not name it.
m.asked = nil
report, known = applyCarried(t, declare("debug", ""), known, m, nil)
if m.did("docker rm") || m.did("docker run") {
t.Errorf("a container was recreated for a file under a mounted directory it did not name: %v", m.asked)
}
if o := outcomeOf(report, "app.config"); o.Action != "updated" {
t.Fatalf("the config was not rewritten: %+v", o)
}
// Naming it is what makes it a reason, as before this change.
m.asked = nil
report, _ = applyCarried(t, declare("trace", `,"restart-on":["app.config"]`), known, m, nil)
if !m.removed("app") || !m.did("docker run") {
t.Fatalf("a container naming a rewritten file under its mount was not recreated: %v", m.asked)
}
if o := outcomeOf(report, "app.server"); !strings.Contains(o.Detail, "app.config") {
t.Errorf("the recreate did not name why: %+v", o)
}
}
func TestAContainerLabelledBeforeTheHostReadItsFilesIsAcceptedNotRecreated(t *testing.T) {
// The first apply after the host upgrades finds every container carrying a label computed
// without the file lines. Recreating them all would be a restart storm across the mesh in
// declaration order, the store first. A label that matches the spec as it used to be computed
// is accepted: what the container reads is recorded now, and from then on a change is caught.
dir := t.TempDir()
env := filepath.Join(dir, "forge.env")
declare := func(port string) *declaration.Declaration {
return parseTrusted(t, `{"declaration":1,"resources":[
{"id":"forge.env","type":"file","path":"`+env+`","content":"DATABASE_PORT=`+port+`\n","mode":"0600"},
{"id":"forge.server","type":"container","name":"forge","image":"`+pinned+`","env-file":["`+env+`"]}
]}`)
}
// The machine as the previous host left it: the file written and recorded, the container up
// under the label that host computed — the spec with nothing about the file's content.
if err := os.WriteFile(env, []byte("DATABASE_PORT=5432\n"), 0o600); err != nil {
t.Fatal(err)
}
d := declare("5432")
legacy := containerSpecReading(d.Resources[1].(*declaration.Container), nil, nil)
known := store.State{}
known.Record(store.Applied{ID: "forge.env", Type: "file", Origin: store.OriginCarried, Target: env,
Wrote: digestOf("DATABASE_PORT=5432\n")})
known.Record(store.Applied{ID: "forge.server", Type: "container", Origin: store.OriginCarried, Target: "forge"})
m := &machine{containers: map[string]*fakeContainer{"forge": {id: "made-by-host", running: true, spec: legacy}}}
report, known := applyCarried(t, d, known, m, nil)
if m.did("docker rm") || m.did("docker run") || report.Changed() {
t.Fatalf("a container labelled by the previous host was recreated on upgrade: %v %+v", m.asked, report.Outcomes)
}
if got, _ := known.Find("forge.server"); got.Reads[env] != digestOf("DATABASE_PORT=5432\n") {
t.Fatalf("what the accepted container reads was not recorded: %+v", got)
}
// Accepted stays accepted: the next pass with nothing moved is quiet too.
m.asked = nil
report, known = applyCarried(t, d, known, m, nil)
if m.did("docker rm") || m.did("docker run") || report.Changed() {
t.Fatalf("an accepted container was recreated on the pass after: %v", m.asked)
}
// And a change to the file is caught from the record, and the label is renewed.
m.asked = nil
report, _ = applyCarried(t, declare("5433"), known, m, nil)
if !m.removed("forge") || !m.did("docker run") {
t.Fatalf("an accepted container was not recreated when its env-file changed: %v", m.asked)
}
if o := outcomeOf(report, "forge.server"); o.Detail != "recreated: "+env+" changed" {
t.Errorf("the recreate did not name the file: %+v", o)
}
if m.containers["forge"].spec == legacy {
t.Error("the recreated container still carries the legacy label")
}
}
func TestAContainerAdoptedUnderANewIdStillSaysWhichFileChanged(t *testing.T) {
// The bundle's `store` becomes the postgres module's `postgres.server`: the same container by
// name, under a new id with no record of its own. A file change on that day is a real change,
// and the record of what it read is under the old id — by name, it is found.
dir := t.TempDir()
env := filepath.Join(dir, "store.env")
m := &machine{containers: map[string]*fakeContainer{}}
raised := parseTrusted(t, `{"declaration":1,"resources":[
{"id":"env","type":"file","path":"`+env+`","content":"PORT=5432\n","mode":"0600"},
{"id":"store","type":"container","name":"mesh-store","image":"`+pinned+`","env-file":["`+env+`"]}
]}`)
_, known := applyCarried(t, raised, store.State{}, m, nil)
adopted := parseTrusted(t, `{"declaration":1,"resources":[
{"id":"postgres.env","type":"file","path":"`+env+`","content":"PORT=5433\n","mode":"0600"},
{"id":"postgres.server","type":"container","name":"mesh-store","image":"`+pinned+`","env-file":["`+env+`"]}
]}`)
m.asked = nil
report, _, err := Apply(context.Background(), archHost(t), adopted, known, store.OriginDeclared, m.run, nil, nil)
if err != nil {
t.Fatal(err)
}
if o := outcomeOf(report, "postgres.server"); o.Action != "updated" || o.Detail != "recreated: "+env+" changed" {
t.Errorf("a container adopted under a new id did not say which file changed: %+v", o)
}
}
func TestAHeldContainerIsNotRecreatedByAChangedHeldFile(t *testing.T) {
// On an adopted node the predecessor's container and the file it reads are both held as
// found (novox/hq ADR 0100). The predecessor rewriting its own file is reported on the file
// — and is nothing to recreate the container for: it is not the host's to recreate.
dir := t.TempDir()
env := filepath.Join(dir, "hello.env")
if err := os.WriteFile(env, []byte("PORT=5432\n"), 0o600); err != nil {
t.Fatal(err)
}
m := &machine{containers: map[string]*fakeContainer{
"hello-web": {id: "predecessor-id", running: true},
}}
d := adopted(t, untaken("hello-web.env", "hello-web.server"),
`{"id":"hello-web.env","type":"file","path":"`+env+`","content":"PORT=5433\n","mode":"0600"},
{"id":"hello-web.server","type":"container","name":"hello-web","image":"`+pinned+`","env-file":["`+env+`"]}`)
report, state := applyAdopted(t, d, store.State{}, m, dir)
if outcomeOf(report, "hello-web.env").Action != "held" || outcomeOf(report, "hello-web.server").Action != "held" {
t.Fatalf("the predecessor's file and container were not held: %+v", report.Outcomes)
}
if err := os.WriteFile(env, []byte("PORT=5434\n"), 0o600); err != nil {
t.Fatal(err)
}
m.asked = nil
report, state = applyAdopted(t, d, state, m, dir)
for _, a := range m.asked {
if strings.HasPrefix(a, "docker run") || strings.HasPrefix(a, "docker rm") {
t.Fatalf("a held container was acted on because a held file changed: %s", a)
}
}
if o := outcomeOf(report, "hello-web.server"); o.Action != "held" {
t.Errorf("the container is no longer held: %+v", o)
}
if h, _ := state.HeldAt("hello-web.env"); h.Changed != "rewritten" {
t.Errorf("the predecessor's rewrite was not reported on the file: %+v", h)
}
if h, _ := state.HeldAt("hello-web.server"); h.Changed != "" {
t.Errorf("a file change was charged to the container: %+v", h)
}
}
+182 -15
View File
@@ -3,6 +3,7 @@ package apply
import (
"context"
"errors"
"path/filepath"
"strings"
"testing"
@@ -68,7 +69,7 @@ func TestARunOnceStepIsRunToCompletionNotLeftRunning(t *testing.T) {
if !ok {
t.Fatal("a completed run-once step was not recorded")
}
if applied.Wrote != containerSpec(d.Resources[0].(*declaration.Container), nil) {
if applied.Wrote != containerSpec(d.Resources[0].(*declaration.Container), inputs{}) {
t.Errorf("the run-once record is not the declaration's digest: %q", applied.Wrote)
}
}
@@ -112,7 +113,7 @@ func TestAFailedRunOnceStepGatesWhatFollows(t *testing.T) {
switch args[0] {
case "info":
return "27.0\n", nil
case "inspect":
case "container":
return "false\t\n", errors.New("no such container")
case "run":
startedNames = append(startedNames, nameOf(args))
@@ -154,7 +155,7 @@ func TestARunOnceStepAlreadyCompletedIsNotReRun(t *testing.T) {
d := parseTrusted(t, `{"declaration":1,"resources":[
{"id":"seed","type":"container","name":"seed","image":"`+pinned+`","run-once":true}
]}`)
want := containerSpec(d.Resources[0].(*declaration.Container), nil)
want := containerSpec(d.Resources[0].(*declaration.Container), inputs{})
var ran bool
run := func(ctx context.Context, name string, args ...string) (string, error) {
@@ -216,18 +217,184 @@ func TestARunOnceStepReRunsWhenItsDeclarationChanged(t *testing.T) {
}
}
func TestARunOnceContainerCannotAlsoDeclareRestartOn(t *testing.T) {
// restart-on brings a running container back when a file it read changed; a run-once step does
// not stay running. The two lifecycles contradict, so the parser refuses the pair rather than
// silently resolving to one.
_, err := declaration.Parse([]byte(`{"declaration":1,"resources":[
{"id":"f","type":"file","path":"/tmp/x","content":"y"},
{"id":"seed","type":"container","name":"seed","image":"` + pinned + `","run-once":true,"restart-on":["f"]}
]}`))
if err == nil {
t.Fatal("a run-once container that also declared restart-on was accepted")
func TestARunOnceStepRunsAgainWhenWhatItReadsChanged(t *testing.T) {
// A step that fetches a fact from a provider names the binding file it reads. What a
// container reads is part of its digest, so when the provider moved and the mesh rewrote the
// file, the step's marker no longer matches and it runs again (novox/hq ADR 0099) — the same
// rule that recreates a running container, read as "again" for a step.
dir := t.TempDir()
env := filepath.Join(dir, "acme.env")
d := parseTrusted(t, `{"declaration":1,"resources":[
{"id":"env","type":"file","path":"`+env+`","content":"ACME_ROOTS=https://10.0.0.2/roots.pem\n"},
{"id":"trust","type":"container","name":"trust","image":"`+pinned+`","run-once":true,"restart-on":["env"]}
]}`)
// The record from when the provider was elsewhere: the step's digest against the old file.
was := map[string]string{"env": declaredDigest(&declaration.File{Content: "ACME_ROOTS=https://10.0.0.1/roots.pem\n"})}
known := store.State{}
known.Record(store.Applied{ID: "trust", Type: "container", Origin: store.OriginCarried, Target: "trust",
Wrote: containerSpec(d.Resources[1].(*declaration.Container), inputs{declares: was})})
var ran bool
run := func(ctx context.Context, name string, args ...string) (string, error) {
switch args[0] {
case "info":
return "27.0\n", nil
case "run":
ran = true
return "", nil
}
return "", nil
}
if !strings.Contains(err.Error(), "run-once") {
t.Errorf("refused for the wrong reason: %v", err)
report, _, err := Apply(context.Background(), archHost(t), d, known, store.OriginCarried, run, nil, nil)
if err != nil {
t.Fatalf("apply failed: %v", err)
}
if !ran {
t.Fatal("a run-once step whose named file changed was not run again")
}
if report.Outcomes[1].Action != "created" {
t.Errorf("the re-run step was not reported as having run: %+v", report.Outcomes[1])
}
// And with the file unchanged, the step stays done: "again" is when something changed.
ran = false
now := map[string]string{"env": declaredDigest(d.Resources[0].(*declaration.File))}
settled := store.State{}
settled.Record(store.Applied{ID: "env", Type: "file", Origin: store.OriginCarried, Target: env, Wrote: now["env"]})
settled.Record(store.Applied{ID: "trust", Type: "container", Origin: store.OriginCarried, Target: "trust",
Wrote: containerSpec(d.Resources[1].(*declaration.Container), inputs{declares: now})})
if _, _, err := Apply(context.Background(), archHost(t), d, settled, store.OriginCarried, run, nil, nil); err != nil {
t.Fatalf("re-apply failed: %v", err)
}
if ran {
t.Error("a run-once step whose named file did not change was run again")
}
}
func TestAContainerNamingARunOnceStepIsRecreatedWhenItRan(t *testing.T) {
// The service that consumes what a step made names the step: when the step ran this pass —
// fetched a new root — the running container holds the old one, and its spec did not move,
// so restart-on is what brings it back with the new one (novox/hq ADR 0099).
d := parseTrusted(t, `{"declaration":1,"resources":[
{"id":"trust","type":"container","name":"trust","image":"`+pinned+`","run-once":true},
{"id":"server","type":"container","name":"server","image":"`+pinned+`","restart-on":["trust"]}
]}`)
declares := map[string]string{"trust": declaredDigest(d.Resources[0].(*declaration.Container))}
spec := containerSpec(d.Resources[1].(*declaration.Container), inputs{declares: declares})
var removed, created bool
run := func(ctx context.Context, name string, args ...string) (string, error) {
switch args[0] {
case "info":
return "27.0\n", nil
case "container":
// The server is up, made from exactly this spec — nothing but the step's run says
// it must be replaced.
return "true\t" + spec, nil
case "rm":
if len(args) > 0 && args[len(args)-1] == "server" {
removed = true
}
return "", nil
case "run":
if args[len(args)-1] != "trust" && strings.Contains(strings.Join(args, " "), "--name server") {
created = true
}
return "deadbeef\n", nil
}
return "", nil
}
report, _, err := Apply(context.Background(), archHost(t), d, store.State{}, store.OriginCarried, run, nil, nil)
if err != nil {
t.Fatalf("apply failed: %v", err)
}
if !removed || !created {
t.Fatalf("the container naming the step was not recreated after the step ran (removed=%v created=%v)", removed, created)
}
server := report.Outcomes[len(report.Outcomes)-1]
if server.Action != "updated" || !strings.Contains(server.Detail, "trust") {
t.Errorf("the recreation did not name the step as its reason: %+v", server)
}
}
func TestAFailedStepGatesItsModuleAndNotTheMachine(t *testing.T) {
// **The blast radius of a step is its module** (novox/hq ADR 0136). A step exists to make
// something true before the next thing in its own module needs it — a store seeded before the
// broker starts, a schema prepared before the version that needs it runs. Stopping the whole
// apply is what this host's own loop calls holding a machine hostage, and it was already
// rejected for every other shape (04-ISSUES/011): a module whose database is briefly
// unreachable must not stop every module declared after it.
var startedNames []string
run := func(ctx context.Context, name string, args ...string) (string, error) {
switch args[0] {
case "info":
return "27.0\n", nil
case "container":
return "false\t\n", errors.New("no such container")
case "run":
startedNames = append(startedNames, nameOf(args))
if nameOf(args) == "catalogue-prepare" {
return "", errors.New("exit status 1") // the schema could not be reached
}
return "deadbeef\n", nil
case "rm":
return "", nil
}
return "", nil
}
d := parseTrusted(t, `{"declaration":1,"resources":[
{"id":"mesh-catalog.runtime-prepare","type":"container","name":"catalogue-prepare","image":"`+pinned+`","run-once":true},
{"id":"mesh-catalog.runtime","type":"container","name":"catalogue","image":"`+pinned+`"},
{"id":"gitea.server","type":"container","name":"forge","image":"`+pinned+`"}
]}`)
report, _, err := Apply(context.Background(), archHost(t), d, store.State{}, store.OriginCarried, run, nil, nil)
if err == nil {
t.Fatal("a failed step was not reported as a failure")
}
started := map[string]bool{}
for _, n := range startedNames {
started[n] = true
}
if started["catalogue"] {
t.Error("the module's own workload ran although its step did not complete")
}
if !started["forge"] {
t.Error("another module was not attempted, so one module's step held the machine hostage")
}
// And the machine's own account says which was not attempted, rather than leaving it to be
// inferred from silence.
var skipped string
for _, o := range report.Outcomes {
if o.Action == "skipped" {
skipped = o.ID
}
}
if skipped != "mesh-catalog.runtime" {
t.Errorf("the report does not say what was not attempted: %q", skipped)
}
}
func TestAResourcesOwnerIsReadToTheLastDot(t *testing.T) {
// **A module's name may contain a dot.** `novox.be` is one on this mesh, so reading a resource's
// owner to the first dot would make its resources belong to something called "novox" — and a gate
// would skip whatever else happened to start that way. A resource's own id never contains one,
// which is what makes the last dot the boundary.
for identity, want := range map[string]string{
"novox.be.server": "novox.be",
"mesh-catalog.runtime-prepare": "mesh-catalog",
"gitea.admin-bootstrap": "gitea",
} {
got, ours := moduleOf(identity)
if !ours || got != want {
t.Errorf("%q belongs to %q (%v), want %q", identity, got, ours, want)
}
}
// What the mesh declares in its own right belongs to no module: the foundation's resources carry
// no dot, and the adoption's are the mesh's.
for _, identity := range []string{"container-runtime", "store-ready", "adoption.guard", ".server"} {
if _, ours := moduleOf(identity); ours {
t.Errorf("%q was read as a module's", identity)
}
}
}
+14 -2
View File
@@ -86,7 +86,11 @@ func NewScheduler(clock Clock, run Runner, log func(string)) *Scheduler {
// A job whose declaration is unchanged keeps its place in the cadence — its next due time and
// whether a run is in flight — so an ordinary reconcile every few minutes does not keep resetting
// the clock out from under a schedule and prevent it ever firing.
func (s *Scheduler) Sync(d *declaration.Declaration) {
// held is the ids this node holds as found — what an adopted node keeps until its module is taken
// (novox/hq ADR 0100). A step of a module not yet taken is not armed: run on its cadence it would
// work on the predecessor's data, under the predecessor's service, which is the one thing an
// adopted node must not do. Nil on a converged node, where nothing is held.
func (s *Scheduler) Sync(d *declaration.Declaration, held map[string]bool) {
s.mu.Lock()
defer s.mu.Unlock()
@@ -96,6 +100,14 @@ func (s *Scheduler) Sync(d *declaration.Declaration) {
if !ok || c.Schedule == "" {
continue
}
if module, untaken := d.Adoption.UntakenModuleOf(c.Identity()); untaken || held[c.Identity()] {
if module == "" {
module = "its module"
}
s.log(fmt.Sprintf("scheduled step %s: not armed while %s is held as found on this node",
c.Identity(), module))
continue
}
cron, err := declaration.ParseCron(c.Schedule)
if err != nil {
// The declaration parser already refused a malformed cron before this runs, so a
@@ -110,7 +122,7 @@ func (s *Scheduler) Sync(d *declaration.Declaration) {
// Nothing to read: a scheduled container may not declare restart-on — it runs to completion
// on its cadence rather than staying running to be restarted — so its identity cannot
// depend on another resource's content and there is nothing to pass.
spec := containerSpec(c, nil)
spec := containerSpec(c, inputs{})
if existing := s.jobs[c.Identity()]; existing != nil && existing.spec == spec {
// Unchanged: keep where it is in its cadence, refresh the declaration pointer only.
existing.container = c
+51 -6
View File
@@ -208,7 +208,7 @@ func TestAScheduledStepDoesNotGateWhatFollows(t *testing.T) {
switch args[0] {
case "info":
return "27.0\n", nil
case "inspect":
case "container":
// The container name is the last argument to `docker inspect --format ... <name>`.
target := args[len(args)-1]
if spec, up := specs[target]; up {
@@ -244,7 +244,7 @@ func TestAScheduledStepRunsWhenDueAndNotBefore(t *testing.T) {
d := &declaration.Declaration{Version: 1, Resources: []declaration.Resource{
scheduledContainer(t, "* * * * *"), // every minute; next due 12:01:00
}}
s.Sync(d)
s.Sync(d, nil)
// Not yet due: 12:00:45 is before 12:01:00, so nothing runs.
s.Advance(context.Background(), time.Date(2026, 9, 7, 12, 0, 45, 0, time.UTC))
@@ -306,7 +306,7 @@ func TestAFailedRunIsRecordedAndDoesNotFailAnything(t *testing.T) {
s := NewScheduler(clock, run, log)
s.Sync(&declaration.Declaration{Version: 1, Resources: []declaration.Resource{
scheduledContainer(t, "* * * * *"),
}})
}}, nil)
// Fire a run that exits non-zero. Advance returns nothing — there is no error to fail an apply,
// because the run happens outside any apply and outside the store.
@@ -371,7 +371,7 @@ func TestASlowRunSkipsTheNextDueRunRatherThanStacking(t *testing.T) {
s := NewScheduler(clock, run, log)
s.Sync(&declaration.Declaration{Version: 1, Resources: []declaration.Resource{
scheduledContainer(t, "* * * * *"), // every minute
}})
}}, nil)
// First occurrence: 12:01 is due — starts a run that blocks in the runner.
s.Advance(context.Background(), time.Date(2026, 9, 7, 12, 1, 5, 0, time.UTC))
@@ -414,7 +414,7 @@ func TestSyncForgetsAScheduleTheDeclarationNoLongerNames(t *testing.T) {
s.Sync(&declaration.Declaration{Version: 1, Resources: []declaration.Resource{
scheduledContainer(t, "* * * * *"),
}})
}}, nil)
s.mu.Lock()
have := len(s.jobs)
s.mu.Unlock()
@@ -427,10 +427,55 @@ func TestSyncForgetsAScheduleTheDeclarationNoLongerNames(t *testing.T) {
parseTrusted(t, `{"declaration":1,"resources":[
{"id":"web","type":"container","name":"web","image":"`+pinned+`"}
]}`).Resources[0],
}})
}}, nil)
s.Advance(context.Background(), time.Date(2026, 9, 7, 12, 5, 5, 0, time.UTC))
s.Wait()
if n := rec.fireCount(); n != 0 {
t.Errorf("a schedule the declaration no longer names still fired (%d run(s))", n)
}
}
// Defends novox/hq ADR 0103: a scheduled step of a module not yet taken is not armed — run on its
// cadence it would work on the predecessor's data.
func TestAScheduledStepOfAnUntakenModuleIsNotArmed(t *testing.T) {
clock := &fixedClock{now: time.Date(2026, 9, 7, 12, 0, 30, 0, time.UTC)}
rec := &recordingRun{}
var said []string
s := NewScheduler(clock, rec.run, func(line string) { said = append(said, line) })
d := &declaration.Declaration{Version: 1,
Adoption: &declaration.Adoption{Untaken: map[string][]string{"backups": {"sync"}}},
Resources: []declaration.Resource{
scheduledContainer(t, "* * * * *"),
}}
s.Sync(d, nil)
s.Advance(context.Background(), time.Date(2026, 9, 7, 12, 1, 5, 0, time.UTC))
s.Wait()
if rec.fireCount() != 0 {
t.Errorf("a held module's scheduled step ran %d time(s)", rec.fireCount())
}
if len(said) == 0 || !strings.Contains(said[0], "held as found") {
t.Errorf("nothing said why the step was not armed: %v", said)
}
// Held by id — a step whose own container was found on the machine.
s2 := NewScheduler(clock, rec.run, nil)
s2.Sync(&declaration.Declaration{Version: 1, Resources: []declaration.Resource{
scheduledContainer(t, "* * * * *"),
}}, map[string]bool{"sync": true})
s2.Advance(context.Background(), time.Date(2026, 9, 7, 12, 1, 5, 0, time.UTC))
s2.Wait()
if rec.fireCount() != 0 {
t.Errorf("a held scheduled step ran %d time(s)", rec.fireCount())
}
// Taken: the same step is armed and runs.
taken := &declaration.Declaration{Version: 1, Adoption: &declaration.Adoption{Taken: []string{"backups"}},
Resources: []declaration.Resource{scheduledContainer(t, "* * * * *")}}
s.Sync(taken, nil)
s.Advance(context.Background(), time.Date(2026, 9, 7, 12, 2, 5, 0, time.UTC))
s.Wait()
if rec.fireCount() == 0 {
t.Error("a taken module's scheduled step never ran")
}
}
+191
View File
@@ -0,0 +1,191 @@
package apply
import (
"context"
"fmt"
"path/filepath"
"strings"
"testing"
"github.com/novox/mesh-host/internal/store"
)
// Defends novox/hq ADR 0117: a service that omits its state leaves the unit's lifecycle to the
// machine. The mesh reflects its triggers on a unit already running and does nothing else to it —
// never starts, stops, enables or disables it, and forgets it when undeclared.
// unitIn is a service manager whose one unit is active or not, recording what it is asked.
func unitIn(active bool, commands *[]string) Runner {
return func(_ context.Context, name string, args ...string) (string, error) {
line := name + " " + strings.Join(args, " ")
*commands = append(*commands, line)
switch {
case strings.Contains(line, "is-enabled"):
return "enabled", nil
case strings.Contains(line, "show") && strings.Contains(line, "ActiveState"):
if active {
return "LoadState=loaded\nActiveState=active\nSubState=running", nil
}
return "LoadState=loaded\nActiveState=inactive\nSubState=dead", nil
}
return "", nil
}
}
func statelessDecl(path, content, triggers string) string {
return fmt.Sprintf(`{"declaration":1,"resources":[
{"id":"uplink.conf","type":"file","path":%q,"into":"block","content":%q},
{"id":"uplink.manager","type":"service","unit":"NetworkManager.service",%s}
]}`, path, content, triggers)
}
// touched is whether any command would change the unit's lifecycle.
func touched(commands []string) []string {
var changing []string
for _, c := range commands {
for _, verb := range []string{" start ", " stop ", " restart ", " enable ", " disable ", " reload "} {
if strings.Contains(c+" ", verb) {
changing = append(changing, c)
}
}
}
return changing
}
func TestAStatelessServiceRunningIsReloadedForItsTrigger(t *testing.T) {
path := filepath.Join(t.TempDir(), "mesh.conf")
var commands []string
report, _, err := Apply(context.Background(), archHost(t),
parse(t, statelessDecl(path, "[main]\ndns=none\n", `"reload-on":["uplink.conf"]`)),
store.State{}, store.OriginDeclared, unitIn(true, &commands), nil, nil)
if err != nil {
t.Fatal(err)
}
joined := strings.Join(commands, "\n")
if !strings.Contains(joined, "systemctl reload NetworkManager.service") {
t.Errorf("the running manager was not reloaded; commands were %v", commands)
}
for _, c := range touched(commands) {
if !strings.Contains(c, "reload") {
t.Errorf("the manager's lifecycle was touched: %s", c)
}
}
if o := outcomeOf(report, "uplink.manager"); o.Action != "updated" || !strings.Contains(o.Detail, "reloaded for uplink.conf") {
t.Errorf("reported as %q: %s", o.Action, o.Detail)
}
}
func TestAStatelessServiceNotRunningIsLeftSo(t *testing.T) {
path := filepath.Join(t.TempDir(), "mesh.conf")
for _, triggers := range []string{`"reload-on":["uplink.conf"]`, `"restart-on":["uplink.conf"]`} {
var commands []string
report, _, err := Apply(context.Background(), archHost(t),
parse(t, statelessDecl(path, fmt.Sprintf("# %s\n", triggers), triggers)),
store.State{}, store.OriginDeclared, unitIn(false, &commands), nil, nil)
if err != nil {
t.Fatal(err)
}
if changing := touched(commands); len(changing) > 0 {
t.Errorf("%s: an inactive unit was acted on: %v", triggers, changing)
}
if o := outcomeOf(report, "uplink.manager"); o.Action != "unchanged" ||
o.Detail != "not running; the change applies at its next start" {
t.Errorf("%s: reported as %q: %s", triggers, o.Action, o.Detail)
}
}
}
func TestAStatelessServiceIsRestartedOnlyForItsRestartTrigger(t *testing.T) {
path := filepath.Join(t.TempDir(), "mesh.conf")
var commands []string
d := parse(t, statelessDecl(path, "x\n", `"restart-on":["uplink.conf"]`))
_, state, err := Apply(context.Background(), archHost(t), d, store.State{}, store.OriginDeclared,
unitIn(true, &commands), nil, nil)
if err != nil {
t.Fatal(err)
}
if !strings.Contains(strings.Join(commands, "\n"), "stop NetworkManager.service") {
t.Errorf("not restarted for its restart trigger; commands were %v", commands)
}
// Nothing it reflects changed: nothing is asked of the unit at all.
commands = nil
report, _, err := Apply(context.Background(), archHost(t), d, state, store.OriginDeclared,
unitIn(true, &commands), nil, nil)
if err != nil {
t.Fatal(err)
}
if changing := touched(commands); len(changing) > 0 {
t.Errorf("with nothing changed the unit was acted on: %v", changing)
}
if got := outcomeOf(report, "uplink.manager").Action; got != "unchanged" {
t.Errorf("with nothing changed it was %q", got)
}
}
func TestAStatelessServiceUndeclaredIsForgottenNotStopped(t *testing.T) {
path := filepath.Join(t.TempDir(), "mesh.conf")
var commands []string
_, state, err := Apply(context.Background(), archHost(t),
parse(t, statelessDecl(path, "x\n", `"reload-on":["uplink.conf"]`)),
store.State{}, store.OriginDeclared, unitIn(true, &commands), nil, nil)
if err != nil {
t.Fatal(err)
}
if rec, _ := state.Find("uplink.manager"); !rec.Stateless {
t.Fatal("the record does not say the service was stateless")
}
if steps := Plan(somethingElse(t), state, store.OriginDeclared); !hasStep(steps, "forget", "uplink.manager") {
t.Errorf("the preview does not forget it: %v", steps)
}
commands = nil
report, _, err := Apply(context.Background(), archHost(t), somethingElse(t), state, store.OriginDeclared,
unitIn(true, &commands), nil, nil)
if err != nil {
t.Fatal(err)
}
if changing := touched(commands); len(changing) > 0 {
t.Errorf("undeclaring acted on the unit: %v", changing)
}
o := outcomeOf(report, "uplink.manager")
if o.Action != "forgotten" || o.Detail != "its state was never the mesh's" {
t.Errorf("undeclaring was %q: %s", o.Action, o.Detail)
}
}
func TestAStatelessServiceIsNeverHeldOnAnAdoptedNode(t *testing.T) {
path := filepath.Join(t.TempDir(), "mesh.conf")
resources := fmt.Sprintf(`{"id":"uplink.conf","type":"file","path":%q,"into":"block","content":"x\n"},
{"id":"uplink.manager","type":"service","unit":"NetworkManager.service","reload-on":["uplink.conf"]}`, path)
d := adopted(t, `{"taken":[],"untaken":{"uplink":["uplink.conf","uplink.manager"]}}`, resources)
for _, s := range Plan(d, store.State{}, store.OriginDeclared) {
if s.ID == "uplink.manager" && (s.Verb == "hold" || s.Verb == "create" || strings.Contains(s.Why, "replaces")) {
t.Errorf("the preview holds or replaces a stateless service: %+v", s)
}
}
var commands []string
report, state, err := ApplyKeeping(context.Background(), archHost(t), d, store.State{},
store.OriginDeclared, unitIn(true, &commands), nil, nil, KeepIn(t.TempDir()))
if err != nil {
t.Fatal(err)
}
if got := outcomeOf(report, "uplink.manager").Action; got == "held" {
t.Fatal("a stateless service was held, though it replaces nothing that was found")
}
if len(state.Held) != 0 {
t.Errorf("something was held: %+v", state.Held)
}
for _, c := range touched(commands) {
if !strings.Contains(c, "reload") {
t.Errorf("the adopted node's manager lifecycle was touched: %s", c)
}
}
}
func hasStep(steps []Step, verb, id string) bool {
for _, s := range steps {
if s.Verb == verb && s.ID == id {
return true
}
}
return false
}
+54
View File
@@ -0,0 +1,54 @@
package apply
import (
"context"
"sort"
"strings"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/store"
)
// Strays is what runs on the machine that the mesh neither wrote nor holds (novox/hq ADR 0163):
// every container the runtime has that no record names and no hold names. The question nothing
// answered on 2026-09-23, when a renamed resource left its old container running for a day; asked
// on every apply now, and reported, so a thing left behind is seen the day it is left.
//
// Containers only, today. A listener nobody declared is harder to attribute to a thing, and the
// machine's own services are not strays; that account is issue 160's.
func Strays(ctx context.Context, run Runner, known store.State) ([]store.Stray, error) {
cri, err := containerRuntime(ctx, run)
if err != nil {
return nil, nil // a machine with no runtime has no containers to stray
}
out, err := run(ctx, cri, "ps", "-a", "--format", "{{.Names}}\t{{.Image}}\t{{.State}}")
if err != nil {
return nil, err
}
ours := map[string]bool{}
for _, r := range known.Resources {
if declaration.Type(r.Type) == declaration.TypeContainer {
ours[r.Target] = true
}
}
for _, h := range known.Held {
if h.Kind == string(declaration.TypeContainer) {
ours[h.Target] = true
}
}
var strays []store.Stray
for _, line := range strings.Split(strings.TrimSpace(out), "\n") {
parts := strings.Split(line, "\t")
name := strings.TrimSpace(parts[0])
if name == "" || ours[name] {
continue
}
detail := ""
if len(parts) > 2 {
detail = strings.TrimSpace(parts[1]) + ", " + strings.TrimSpace(parts[2])
}
strays = append(strays, store.Stray{Kind: string(declaration.TypeContainer), Name: name, Detail: detail})
}
sort.Slice(strays, func(i, j int) bool { return strays[i].Name < strays[j].Name })
return strays, nil
}
+579
View File
@@ -0,0 +1,579 @@
package apply
import (
"context"
"errors"
"fmt"
"os"
"path/filepath"
"strings"
"time"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/store"
"github.com/novox/mesh-host/internal/system"
"github.com/novox/mesh-host/internal/tunnel"
)
// The private network takes over the tunnel it found (novox/hq ADR 0105).
//
// The controller says so on the interface's service: `takes-over` names the found interface, the
// unit that raised it and its configuration file. Before the mesh's unit is started, the host keeps
// that file like any held file — the original recorded before anything else happens to it — and
// stops and disables the found unit. Never a flush: `wg set … peer … remove` is never run, the
// file is never written, and the found interface goes down the way its own unit takes it down.
// Then the mesh's interface comes up, with the found key the node took at enrolment, on the found
// port, with the found peers in its list — and a peer of the tunnel cannot tell it changed hands.
//
// Every apply, not once: a found unit somebody starts again would take the port back from the
// mesh's interface, so it is stopped again and said so. That is the one place an adopted node
// undoes something done by hand, and it is because the tunnel is the mesh's now.
//
// **Until the take is proven, and then the found configuration is retired** (novox/hq ADR 0119).
// Keeping it on disk was the caution the take needed: if the mesh's interface does not come up,
// the found unit is started again and the peers never notice. That caution is spent once the
// tunnel is taken — the found unit down and disabled, the mesh's interface up — and a peer has
// handshaken with the mesh's interface. From then on a configuration nothing maintains, one
// command away from raising a second way onto the network, is not a rollback path but a door
// nobody watches. So it is removed from where its unit reads it; its original, kept before
// anything happened to it (ADR 0100), stays kept; and the hold on it ends. A take never proven
// keeps it, and says so — a broken take is visible, not silently retired.
// TakenTunnel is what an apply says about a tunnel it took over, for the node's report.
type TakenTunnel struct {
Interface string
Port int
Range string
Peers int
// State is "not-taken" (the found interface still up, the mesh's not), "taken" (the found one
// down and disabled, the mesh's up with its key) or "down" (the found one down and the mesh's
// not up: the peers reach nothing). Note is what this apply did about it — and, for a taken
// tunnel, whether the take is proven and its found configuration retired (novox/hq ADR 0119).
// Kept is where the found configuration's original is, retired or not.
State string
Note string
Kept string
}
// The states, as the link says them.
const (
NotTaken = "not-taken"
Taken = "taken"
TunnelDown = "down"
)
// takeoverRecheck is how often, and takeoverRechecks how many times, a found interface still up
// after its unit stopped is looked at again before the takeover is refused: `wg-quick down` by a
// person takes a moment. Variables so a test need not wait.
var (
takeoverRecheck = 2 * time.Second
takeoverRechecks = 3
)
// takeOverID is the held record's id for the found configuration: the service's own with a suffix,
// so it is declared for as long as the service is and never mistaken for the service itself.
func takeOverID(svc *declaration.Service) string { return svc.ID + ".takes-over" }
// takeOver keeps the found tunnel's configuration and stops its unit, ahead of the service that
// replaces it. Returned is the hold's outcome, and what was found for the report.
//
// **Nothing is stopped until the mesh's interface is known to be able to replace it** (the record's
// option 2 is exactly this going wrong): the declared configuration must listen on the found port
// at the found address, and the key file it points at must hold the found key. Only then is the
// found unit stopped — and `stopped` says whether this apply did, so a mesh interface that then
// fails to start can have the found unit started again.
func takeOver(ctx context.Context, sys system.System, svc *declaration.Service, d *declaration.Declaration,
known *store.State, run Runner, keep Keep, now time.Time) (out Outcome, facts TakenTunnel, stopped bool, err error) {
t := svc.TakesOver
id := takeOverID(svc)
module, _ := d.Adoption.UntakenModuleOf(svc.ID)
if module == "" {
module = "the private network"
}
facts = TakenTunnel{Interface: t.Interface, State: NotTaken}
// 0. What the found configuration says, before anything: the checks below are against it.
found, ferr := readFoundTunnel(t.Config)
// 1. The configuration, kept like any held file. A synthetic file resource stands for it, so
// the same code keeps its original, digests it and notices it changing.
//
// **Unless it was retired** (novox/hq ADR 0119): the take was proven and the mesh removed
// it, so there is nothing to hold and nothing missing — only where its original is, which
// the retirement recorded. A hold still standing is let go: that is what retiring it meant.
//
// **One that comes back is held again on its FIRST original** — the one kept before anything
// happened to it (ADR 0100), never whatever was put back — and retired again by the first
// apply that finds the take still proven, keeping what came back only if it differs from
// what is already kept. Put back by hand while the private network is assigned, it is not a
// rollback: that means unassigning the private network first, and the note says so.
file := &declaration.File{ID: id, Type: declaration.TypeFile, Path: t.Config}
var held store.Held
retired, wasRetired := known.RetiredAt(t.Config)
cameBack := wasRetired && present(t.Config)
switch {
case wasRetired && !cameBack:
known.Release(id)
held = store.Held{Kept: retired.Kept}
out = begin(file)
out.Action = "unchanged"
facts.Note = "the found configuration " + t.Config + " was retired once the take was proven; " +
"its original is kept at " + retired.Kept + " and the mesh never brings it back"
default:
was, already := known.HeldAt(id)
why := "the configuration of the tunnel " + t.Interface + ", taken over by " + svc.Unit
if cameBack && !already {
digest := retired.Digest
if digest == "" {
if raw, err := os.ReadFile(retired.Kept); err == nil {
digest = digestOf(string(raw))
}
}
was = store.Held{ID: id, Module: module, Kind: string(declaration.TypeFile), Target: t.Config,
Since: now, Why: why, Kept: retired.Kept, Digest: digest}
already = true
}
out, held, err = hold(ctx, sys, file, module, was, already, why, run, keep, now)
if err != nil {
return begin(file), facts, false, fmt.Errorf("keeping the found tunnel's configuration: %w", err)
}
known.RecordHeld(held)
if cameBack {
facts.Note = "the found configuration " + t.Config + " came back after it was retired; while the " +
"private network is assigned the mesh retires it again, so rolling back to the found tunnel " +
"means unassigning the private network first"
}
}
facts.Kept = held.Kept
// What the file says, for the report: from the machine, or from the kept original when the
// machine's copy is gone. The private key stays in the file; nothing here keeps it.
unread := ""
if ferr != nil && held.Kept != "" {
found, ferr = readFoundTunnel(held.Kept)
}
if ferr == nil {
facts.Port, facts.Range, facts.Peers = found.Port, found.Range, len(found.Peers)
} else {
unread = ferr.Error()
}
// 2. Where things stand: the found unit, and the mesh's.
foundState, unitErr := sys.ServiceState(ctx, run, t.Unit)
meshState, _ := sys.ServiceState(ctx, run, svc.Unit)
if foundState == "running" && meshState == "running" {
// Both up. On the hub this cannot last — the found unit cannot bind the port the mesh's
// holds — and on a spoke two interfaces with one key flap between them. Not stopped again
// by the mesh: what is found on an adopted node is reported, and the first takeover was
// the one act (the PR note says why). Said, so a person sees it.
facts.Note = t.Unit + " is running again beside the mesh's interface; not stopped by the mesh — " +
"`systemctl stop " + t.Unit + "` on the machine"
}
// 3. Before the found unit is stopped: can the mesh's interface replace it? Its declared
// configuration must listen on the found port at the found address, and the key file it
// points at must hold the found key, or the peers would be dropped the moment it came up.
if foundState == "running" && meshState != "running" {
if ferr != nil {
return out, facts, false, fmt.Errorf("the found tunnel's configuration at %s cannot be read as a "+
"tunnel's (%v), so nothing says what the mesh's interface must match; %s is left running",
t.Config, ferr, t.Unit)
}
if err := replaces(d, svc, found); err != nil {
return out, facts, false, fmt.Errorf("%w; %s is left running", err, t.Unit)
}
}
// 4. The found unit: stopped if it runs and the mesh's does not, disabled if it starts at
// boot. A unit that is not there is not an error — the interface may have been raised
// another way, which the check below catches — and neither is one already down.
var did []string
switch {
case unitErr != nil:
did = append(did, t.Unit+" is not a unit here")
case foundState == "running" && meshState != "running":
if err := sys.SetServiceState(ctx, run, t.Unit, "stopped"); err != nil {
return out, facts, false, fmt.Errorf("stopping the found %s: %w", t.Unit, err)
}
after, err := sys.ServiceState(ctx, run, t.Unit)
if err != nil {
return out, facts, true, err
}
if after != "stopped" {
return out, facts, true, fmt.Errorf("%s was asked to stop and is %s", t.Unit, after)
}
stopped = true
did = append(did, "stopped "+t.Unit)
}
if unitErr == nil {
if boot, err := sys.ServiceBoot(ctx, run, t.Unit); err == nil && boot == "enabled" {
if err := sys.SetServiceBoot(ctx, run, t.Unit, "disabled"); err != nil {
return out, facts, stopped, fmt.Errorf("disabling the found %s at boot: %w", t.Unit, err)
}
did = append(did, "disabled it at boot")
}
}
// 5. The interface is gone. If it is still up, something other than its unit raised it —
// the predecessor brings its up by hand — and the mesh's interface cannot take its port
// and address while it is. Looked at again for a moment, since a person taking it down
// takes a moment; then refused, naming what to do.
if meshState != "running" {
for try := 0; ; try++ {
if !interfaceUp(ctx, run, t.Interface) {
break
}
if try >= takeoverRechecks {
return out, facts, stopped, fmt.Errorf("%s is still up although its unit %s is not running: it was "+
"raised by hand, not by its unit, and the mesh's interface cannot take its port and "+
"address while it is. On the machine: `wg-quick down %s` — the next reconcile takes it "+
"over. Nothing was flushed", t.Interface, t.Unit, t.Interface)
}
select {
case <-ctx.Done():
return out, facts, stopped, ctx.Err()
case <-time.After(takeoverRecheck):
}
}
}
out.Detail = "the tunnel " + t.Interface + "'s configuration, kept as found"
switch {
case cameBack:
out.Detail = "the tunnel " + t.Interface + "'s configuration, back after it was retired; held until " +
"it is retired again"
case wasRetired:
out.Detail = "the tunnel " + t.Interface + "'s configuration, retired once the take was proven"
}
if held.Kept != "" {
out.Detail += " (original at " + held.Kept + ")"
}
if len(did) > 0 {
out.Detail += "; " + strings.Join(did, ", ") + " — never flushed"
}
if held.Changed != "" {
out.Detail += "; " + held.Changed + " by something other than the mesh since it was found"
}
if unread != "" {
// Said, not swallowed: the report would otherwise say a tunnel with no port and no
// peers was carried, which reads as a tunnel that was not one.
out.Detail += "; what it says could not be read as a tunnel's: " + unread
}
return out, facts, stopped, nil
}
// readFoundTunnel is the found configuration as a tunnel.
func readFoundTunnel(path string) (tunnel.Found, error) {
raw, err := os.ReadFile(path)
if err != nil {
return tunnel.Found{}, err
}
return tunnel.Parse(raw)
}
// interfaceUp is whether a WireGuard interface is up on the machine.
func interfaceUp(ctx context.Context, run Runner, iface string) bool {
up, err := run(ctx, "wg", "show", "interfaces")
if err != nil {
return false
}
for _, name := range strings.Fields(up) {
if name == iface {
return true
}
}
return false
}
// replaces holds the mesh's declared interface configuration against the found tunnel it is to
// replace: same port, same address, and a key file holding the found key. The configuration is
// the file the service restarts on; its `PostUp = wg set %i private-key <path>` names the key.
func replaces(d *declaration.Declaration, svc *declaration.Service, found tunnel.Found) error {
var conf *declaration.File
for _, r := range d.Resources {
f, ok := r.(*declaration.File)
if !ok {
continue
}
for _, id := range svc.RestartOn {
if f.ID == id {
conf = f
}
}
}
if conf == nil {
return fmt.Errorf("%s takes over %s and restarts on no declared file, so the interface it would "+
"raise cannot be checked against the found one", svc.Unit, found.Interface)
}
port, address, keyPath := "", "", ""
for _, line := range strings.Split(conf.Content, "\n") {
key, value, ok := strings.Cut(strings.TrimSpace(line), "=")
if !ok {
continue
}
key, value = strings.ToLower(strings.TrimSpace(key)), strings.TrimSpace(value)
switch key {
case "listenport":
port = value
case "address":
address = strings.TrimSpace(strings.Split(value, ",")[0])
case "postup":
if _, after, ok := strings.Cut(value, "private-key "); ok {
keyPath = strings.Fields(after)[0]
}
}
}
var wrong []string
if port != fmt.Sprint(found.Port) {
wrong = append(wrong, fmt.Sprintf("it listens on port %q and the tunnel on %d", port, found.Port))
}
if host(address) != host(found.Address) {
wrong = append(wrong, fmt.Sprintf("its address is %q and the tunnel's %s", address, found.Address))
}
switch raw, err := os.ReadFile(keyPath); {
case keyPath == "":
wrong = append(wrong, "it names no key file")
case err != nil:
wrong = append(wrong, fmt.Sprintf("its key file %s cannot be read (%v)", keyPath, err))
default:
public, perr := tunnel.PublicKeyOf(strings.TrimSpace(string(raw)))
if perr != nil || public != found.PublicKey {
wrong = append(wrong, fmt.Sprintf("the key at %s is not the tunnel's — `mesh-host overlay take "+
"--tunnel %s` on this machine takes it, then push again", keyPath, found.Interface))
}
}
if len(wrong) > 0 {
return fmt.Errorf("the mesh's interface would not replace the tunnel on %s: %s — the peers would be "+
"dropped the moment it came up. Re-place the hub on the tunnel's address and port and push again",
found.Interface, strings.Join(wrong, "; "))
}
return nil
}
// host is an address without its prefix length.
func host(address string) string {
if i := strings.Index(address, "/"); i >= 0 {
return address[:i]
}
return address
}
// tunnelState is where the tunnel stands, read from the machine: the found unit or interface up
// and the mesh's not is not taken; the mesh's up and the found one down is taken; neither up is
// down — the peers reach nothing.
func tunnelState(ctx context.Context, sys system.System, foundUnit, meshUnit string, run Runner) string {
foundState, _ := sys.ServiceState(ctx, run, foundUnit)
meshState, _ := sys.ServiceState(ctx, run, meshUnit)
foundUp := foundState == "running" || interfaceUp(ctx, run, strings.TrimPrefix(foundUnit, "wg-quick@"))
switch {
case meshState == "running" && !foundUp:
return Taken
case meshState == "running":
// Both up: not a takeover that holds, and said as not taken so nobody reads it as one.
return NotTaken
case foundUp:
return NotTaken
default:
return TunnelDown
}
}
// restoreFound starts the found unit again after the mesh's interface failed to replace it, so the
// machine has the tunnel it had rather than none, and says so in the account.
func restoreFound(ctx context.Context, sys system.System, unit string, run Runner, facts *TakenTunnel) {
if err := sys.SetServiceState(ctx, run, unit, "running"); err != nil {
facts.State = TunnelDown
facts.Note += "; " + unit + " could not be started again (" + err.Error() + ") — on the machine: systemctl start " + unit
return
}
if state, err := sys.ServiceState(ctx, run, unit); err != nil || state != "running" {
facts.State = TunnelDown
facts.Note += "; " + unit + " was started again and is not running — on the machine: systemctl start " + unit
return
}
facts.State = NotTaken
facts.Note += "; " + unit + " was started again, so the machine has the tunnel it had"
}
// wireguardDir is where a found tunnel's configuration may be retired from: wg-quick's own, and
// nowhere else. A variable so a test can hand in a directory.
var wireguardDir = tunnel.ConfigDir
// retireFound removes the found tunnel's configuration from where its unit reads it, once the take
// is proven, and ends the hold on it (novox/hq ADR 0119). Asked after the mesh's service applied
// and the tunnel reads as taken; retired says whether this apply retired it, and out is then what
// replaces the take's outcome for the configuration.
//
// **Proven is taken and a handshake.** Taken alone — the found unit down and disabled, the mesh's
// interface up — says the mesh's interface exists, not that any peer reaches it: an interface up
// with the wrong key is taken and carries nothing. A peer that has completed a handshake with it
// has checked its key, so that is the proof, asked of the kernel through `wg`. Any handshake counts,
// however old: a change to the mesh's configuration restarts its unit, which recreates the
// interface and resets its counters, so a time that is there at all was made by this interface.
// Anything short of one — no peer yet, every time zero, `wg` missing or failing — keeps the file,
// and the account says which: a take that never proves itself is visible rather than silently
// retired.
//
// **Only what the take names, and only wg-quick's own file.** Nothing is removed unless the path
// is exactly `<wireguard dir>/<found interface>.conf`, is not a path the mesh itself writes, and is
// a file rather than a link: removing a link would leave the key-bearing file it points at where it
// is, a retirement in name only, so that one is said and left to a person.
//
// **The original must still be kept.** It is the record of what the predecessor was and a
// person's only way back (ADR 0100); a kept copy that has gone missing is said, and the file is
// not removed, since removing it then would lose the only copy. What is on disk now, if it differs
// from the first original and from what was kept at the last retirement, is kept too before it
// goes — by content, so the first original is never overwritten and a file that keeps coming back
// the same keeps nothing more.
//
// The found unit is left disabled; without its configuration it cannot raise the interface, so
// every later apply's check of it finds nothing to do. Nothing here ever writes the file back.
func retireFound(ctx context.Context, svc *declaration.Service, d *declaration.Declaration, known *store.State,
run Runner, keep Keep, facts *TakenTunnel, now time.Time) (out Outcome, retired bool) {
t := svc.TakesOver
id := takeOverID(svc)
if facts.State != Taken {
return out, false
}
held, isHeld := known.HeldAt(id)
if !isHeld {
// Retired already (takeOver let any hold go and said so), or never held: nothing to do.
return out, false
}
say := func(note string) {
if facts.Note != "" {
facts.Note += "; "
}
facts.Note += note
}
notRetired := func(why string) (Outcome, bool) {
say("the found configuration " + t.Config + " is not retired: " + why)
return Outcome{}, false
}
mesh := strings.TrimPrefix(svc.Unit, "wg-quick@")
peers, err := tunnel.Handshaken(ctx, tunnel.Runner(run), mesh)
if err != nil {
say("taken, not yet proven: " + err.Error() + "; the found configuration " + t.Config + " is kept")
return out, false
}
if peers == 0 {
say("taken, not yet proven: no peer has handshaken on " + mesh + "; the found configuration " +
t.Config + " is kept")
return out, false
}
proven := fmt.Sprintf("proven: %d peer(s) handshaken on %s", peers, mesh)
say(proven)
// What may be removed at all.
if want := filepath.Join(wireguardDir, t.Interface+".conf"); t.Config != want {
return notRetired("only " + want + ", the found interface's own wg-quick configuration, is ever " +
"retired by the mesh, and the take names " + t.Config)
}
if known.Recorded(string(declaration.TypeFile), t.Config) || declaresFile(d, t.Config) {
return notRetired("it is a path the mesh itself writes")
}
if info, err := os.Lstat(t.Config); err == nil && info.Mode()&os.ModeSymlink != 0 {
target, _ := os.Readlink(t.Config)
return notRetired("it is a link to " + target + "; removing the link would leave the key-bearing file " +
"it points at, so it must be retired by hand — both are kept")
}
// The kept original, read back — not just named in a record.
if held.Kept == "" {
return notRetired("no original of it was kept, so removing it would leave no record of what the " +
"predecessor was")
}
original, err := os.ReadFile(held.Kept)
if err != nil || (held.Digest != "" && digestOf(string(original)) != held.Digest) {
why := "is missing"
if err == nil {
why = "no longer holds what was found"
} else if !errors.Is(err, os.ErrNotExist) {
why = "cannot be read (" + err.Error() + ")"
}
return notRetired("its kept original " + held.Kept + " " + why + ", so removing it would lose the only copy")
}
before, cameBack := known.RetiredAt(t.Config)
record := store.Retired{ID: id, Path: t.Config, Kept: held.Kept, Digest: digestOf(string(original)), At: now}
if cameBack {
// The first original stays the record's, and so does what the last retirement kept.
record.Extra, record.ExtraDigest, record.Again = before.Extra, before.ExtraDigest, before.Again+1
}
newCopy := ""
gone := !present(t.Config)
if !gone {
current, err := os.ReadFile(t.Config)
if err != nil {
return notRetired("it cannot be read (" + err.Error() + ")")
}
if sum := digestOf(string(current)); sum != record.Digest && sum != record.ExtraDigest {
if keep == nil {
return notRetired("it holds something other than its kept original and this host has nowhere " +
"to keep it")
}
where, err := keep(t.Config, current, 0o600)
if err != nil {
return notRetired("keeping what it holds now failed (" + err.Error() + ")")
}
record.Extra, record.ExtraDigest, newCopy = where, sum, where
}
if err := os.Remove(t.Config); err != nil && !errors.Is(err, os.ErrNotExist) {
return notRetired("removing it failed (" + err.Error() + ")")
}
if present(t.Config) {
return notRetired("it is still there after it was removed")
}
}
known.RecordRetired(record)
known.Release(id)
facts.Kept = held.Kept
copied := ""
if newCopy != "" {
copied = "; what it held, which differed from the original, is kept at " + newCopy
}
out = Outcome{ID: id, Type: string(declaration.TypeFile), Target: t.Config, Action: "removed"}
switch {
case cameBack:
say("the found configuration came back and was retired again — its original still kept at " +
held.Kept + copied + "; rolling back to the found tunnel means unassigning the private network first")
out.Detail = "the found configuration came back and was retired again; original kept at " + held.Kept + copied
default:
say("the found configuration " + t.Config + " is retired — its original kept at " + held.Kept + copied +
", " + t.Unit + " left disabled, and the mesh never brings it back")
out.Detail = "retired: the take of " + t.Interface + " is " + proven + "; original kept at " + held.Kept + copied
}
if gone {
// Already gone — removed by something other than the mesh, or by an apply whose record was
// never saved. Nothing removed here; the hold ends all the same.
out.Action = "unchanged"
out.Detail = "retired: the take of " + t.Interface + " is " + proven + " and " + t.Config +
" was already gone; original kept at " + held.Kept
}
return out, true
}
// declaresFile is whether a declaration writes a file at a path.
func declaresFile(d *declaration.Declaration, path string) bool {
for _, r := range d.Resources {
if f, ok := r.(*declaration.File); ok && filepath.Clean(f.Path) == filepath.Clean(path) {
return true
}
}
return false
}
// takesOver is the service in a declaration that takes over a tunnel, if any: one per node, since
// a machine has one private network.
func takesOver(d *declaration.Declaration) *declaration.Service {
for _, r := range d.Resources {
if svc, ok := r.(*declaration.Service); ok && svc.TakesOver != nil {
return svc
}
}
return nil
}
// errNotAdopted is a takeover on a declaration that does not say the node is adopted, which the
// parser refuses already; kept as a second line of defence at the point of acting.
var errNotAdopted = errors.New("a tunnel is taken over on an adopted node only")
+715
View File
@@ -0,0 +1,715 @@
package apply
import (
"crypto/ecdh"
"crypto/rand"
"encoding/base64"
"errors"
"os"
"path/filepath"
"strings"
"testing"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/store"
)
// novox/hq ADR 0105: the host raises the mesh's interface with the found key and peers, stops the
// found interface without flushing it, and keeps its configuration — and stops nothing until the
// mesh's interface is known to be able to replace it.
// foundKey is the predecessor's private key, a real one made once per run: the key is what the
// takeover must never print or copy, so it had better be one.
var foundKey = func() string {
k, err := ecdh.X25519().GenerateKey(rand.Reader)
if err != nil {
panic(err)
}
return base64.StdEncoding.EncodeToString(k.Bytes())
}()
var foundConf = "[Interface]\nPrivateKey = " + foundKey + "\n" +
"ListenPort = 51900\nAddress = 192.0.2.1/24\n\n[Peer]\nPublicKey = PEER-A=\nAllowedIPs = 192.0.2.2/32\n" +
"\n[Peer]\nPublicKey = PEER-B=\nAllowedIPs = 192.0.2.3/32\n"
// aTakeover is the private network's declaration for an adopted hub whose interface takes over
// the found tunnel: the mesh's configuration — on the found port and address, its key set from the
// node's own key file, the found peers in its list — and the interface's service naming what it
// replaces. Port and address are parameters so a test can declare a wrong one.
func aTakeover(t *testing.T, config, mesh, keyFile, port, address string) *declaration.Declaration {
t.Helper()
return adopted(t,
`{"taken":[],"untaken":{"mesh-wireguard":["mesh-wireguard.overlay-config","mesh-wireguard.overlay-up"]}}`,
`{"id":"mesh-wireguard.overlay-config","type":"file","path":"`+mesh+`","mode":"0600",
"content":"[Interface]\nAddress = `+address+`/32\nListenPort = `+port+`\nPostUp = wg set %i private-key `+keyFile+`\n\n[Peer]\nPublicKey = PEER-A=\nAllowedIPs = 192.0.2.2/32\n"},
{"id":"mesh-wireguard.overlay-up","type":"service","unit":"wg-quick@mesh0","state":"running","boot":"enabled",
"restart-on":["mesh-wireguard.overlay-config"],
"takes-over":{"interface":"wg0","unit":"wg-quick@wg0","config":"`+config+`"}}`)
}
// aHubInUse is a machine with the predecessor's tunnel up and the mesh's not yet: the found
// configuration on disk, and the node's key file holding the found key, as enrolment left it.
func aHubInUse(t *testing.T) (dir, config, mesh, keyFile string, m *machine) {
t.Helper()
dir = t.TempDir()
config = filepath.Join(dir, "wg0.conf")
mesh = filepath.Join(dir, "mesh0.conf")
keyFile = filepath.Join(dir, "overlay.key")
if err := os.WriteFile(config, []byte(foundConf), 0o600); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(keyFile, []byte(foundKey+"\n"), 0o600); err != nil {
t.Fatal(err)
}
m = &machine{containers: map[string]*fakeContainer{}, units: map[string]*fakeUnit{
"wg-quick@wg0": {active: "active", enabled: "enabled"},
"wg-quick@mesh0": {active: "inactive", enabled: "disabled", fragment: "/usr/lib/systemd/system/wg-quick@.service"},
}}
takeoverRecheck = 0
// The found configuration lives in this test's own wireguard directory (novox/hq ADR 0119).
was := wireguardDir
wireguardDir = dir
t.Cleanup(func() { wireguardDir = was })
return dir, config, mesh, keyFile, m
}
func TestTheFoundTunnelIsStoppedNeverFlushedAndItsConfigurationKept(t *testing.T) {
dir, config, mesh, keyFile, m := aHubInUse(t)
report, state := applyAdopted(t, aTakeover(t, config, mesh, keyFile, "51900", "192.0.2.1"), store.State{}, m, dir)
// The found interface: its unit stopped and disabled, and nothing else done to it.
if u := m.units["wg-quick@wg0"]; u.active != "inactive" || u.enabled != "disabled" {
t.Fatalf("the found unit was not stopped and disabled: %+v", u)
}
for _, asked := range m.asked {
// Only ever asked about: which interfaces are up, and whether a peer has handshaken with
// the mesh's own (novox/hq ADR 0119).
if strings.HasPrefix(asked, "wg ") && !strings.HasPrefix(asked, "wg show interfaces") &&
asked != "wg show mesh0 latest-handshakes" {
t.Errorf("the found interface was touched with %q; it is stopped, never flushed", asked)
}
if strings.HasPrefix(asked, "wg-quick") || strings.Contains(asked, "peer remove") {
t.Errorf("the found interface was flushed: %q", asked)
}
}
// Its configuration: on disk as it was, its original kept, held for the module.
if got, _ := os.ReadFile(config); string(got) != foundConf {
t.Fatalf("the found configuration was changed:\n%s", got)
}
held, ok := state.HeldAt("mesh-wireguard.overlay-up.takes-over")
if !ok || held.Kind != "file" || held.Target != config || held.Kept == "" || held.Module != "mesh-wireguard" {
t.Fatalf("the found configuration is not held: %+v", held)
}
if kept, _ := os.ReadFile(held.Kept); string(kept) != foundConf {
t.Fatalf("the original was not kept as found: %q", kept)
}
// The mesh's interface: up, enabled, with the found peers in the file the mesh wrote.
if u := m.units["wg-quick@mesh0"]; u.active != "active" || u.enabled != "enabled" {
t.Fatalf("the mesh's interface was not raised: %+v", u)
}
if got, _ := os.ReadFile(mesh); !strings.Contains(string(got), "PEER-A=") || strings.Contains(string(got), "PrivateKey") {
t.Fatalf("the mesh's configuration does not carry the found peer, or carries a key:\n%s", got)
}
// And the report says so, with what was found — port, range, peers — and never the key.
if report.Tunnel == nil || report.Tunnel.State != Taken || report.Tunnel.Port != 51900 ||
report.Tunnel.Range != "192.0.2.0/24" || report.Tunnel.Peers != 2 || report.Tunnel.Kept != held.Kept {
t.Fatalf("the report does not say what was carried: %+v", report.Tunnel)
}
for _, o := range report.Outcomes {
if strings.Contains(o.Detail, foundKey) {
t.Errorf("the found key was printed in an outcome: %+v", o)
}
}
if strings.Contains(report.Tunnel.Note, foundKey) {
t.Error("the found key was printed in the account")
}
if o := outcomeOf(report, "mesh-wireguard.overlay-up.takes-over"); o.Action != "held" ||
!strings.Contains(o.Detail, "stopped wg-quick@wg0") || !strings.Contains(o.Detail, "never flushed") {
t.Errorf("the takeover was not reported as a hold that stopped the found unit: %+v", o)
}
if _, recorded := state.Find("mesh-wireguard.overlay-up.takes-over"); recorded {
t.Error("the found configuration was recorded as applied, so it would be removed as an orphan")
}
}
func TestNothingIsStoppedUntilTheMeshsInterfaceCanReplaceTheFoundOne(t *testing.T) {
dir, config, mesh, keyFile, m := aHubInUse(t)
otherKey := filepath.Join(dir, "other.key")
k, _ := ecdh.X25519().GenerateKey(rand.Reader)
if err := os.WriteFile(otherKey, []byte(base64.StdEncoding.EncodeToString(k.Bytes())+"\n"), 0o600); err != nil {
t.Fatal(err)
}
cases := map[string]*declaration.Declaration{
"another port": aTakeover(t, config, mesh, keyFile, "51821", "192.0.2.1"),
"another address": aTakeover(t, config, mesh, keyFile, "51900", "10.42.0.1"),
"another key": aTakeover(t, config, mesh, otherKey, "51900", "192.0.2.1"),
"no key file": aTakeover(t, config, mesh, filepath.Join(dir, "missing.key"), "51900", "192.0.2.1"),
}
for name, d := range cases {
m.asked = nil
report, state, err := ApplyKeeping(t.Context(), archHost(t), d, store.State{},
store.OriginDeclared, m.run, nil, nil, KeepIn(dir))
if err == nil || !strings.Contains(err.Error(), "would not replace the tunnel") {
t.Fatalf("%s: the takeover was not refused: %v", name, err)
}
if name == "another key" && !strings.Contains(err.Error(), "overlay take") {
t.Errorf("%s: the refusal does not name the remedy: %v", name, err)
}
if m.units["wg-quick@wg0"].active != "active" || m.did("systemctl stop wg-quick@wg0") {
t.Fatalf("%s: the found unit was stopped although the mesh's interface could not replace it", name)
}
if m.units["wg-quick@mesh0"].active == "active" {
t.Fatalf("%s: the mesh's interface was started on top of the found one", name)
}
if report.Tunnel == nil || report.Tunnel.State != NotTaken || !strings.Contains(report.Tunnel.Note, "would not replace") {
t.Fatalf("%s: the account does not say the tunnel is not taken and why: %+v", name, report.Tunnel)
}
if _, held := state.HeldAt("mesh-wireguard.overlay-up.takes-over"); !held {
t.Errorf("%s: the found configuration was not kept before the refusal", name)
}
}
}
func TestAMeshInterfaceThatFailsToStartGivesTheFoundOneBack(t *testing.T) {
dir, config, mesh, keyFile, m := aHubInUse(t)
m.units["wg-quick@mesh0"].wontStart = true
report, _, err := ApplyKeeping(t.Context(), archHost(t), aTakeover(t, config, mesh, keyFile, "51900", "192.0.2.1"),
store.State{}, store.OriginDeclared, m.run, nil, nil, KeepIn(dir))
if err == nil {
t.Fatal("a mesh interface that did not come up was reported as applied")
}
if !m.did("systemctl stop wg-quick@wg0") || !m.did("systemctl start wg-quick@wg0") {
t.Fatalf("the found unit was not stopped and then started again: %v", m.asked)
}
if m.units["wg-quick@wg0"].active != "active" {
t.Fatal("the machine was left with no tunnel at all")
}
if report.Tunnel == nil || report.Tunnel.State != NotTaken ||
!strings.Contains(report.Tunnel.Note, "did not come up") || !strings.Contains(report.Tunnel.Note, "started again") {
t.Fatalf("the account does not say the mesh's interface failed and the found one was given back: %+v", report.Tunnel)
}
}
func TestATakeoverIsSteadyAndAFoundUnitUpAgainIsSaidNotStopped(t *testing.T) {
dir, config, mesh, keyFile, m := aHubInUse(t)
d := aTakeover(t, config, mesh, keyFile, "51900", "192.0.2.1")
_, state := applyAdopted(t, d, store.State{}, m, dir)
m.asked = nil
report, again := applyAdopted(t, d, state, m, dir)
if report.Changed() {
t.Errorf("a second apply moved the machine: %+v", report.Outcomes)
}
if _, still := again.HeldAt("mesh-wireguard.overlay-up.takes-over"); !still {
t.Error("the hold on the found configuration was forgotten while the service still declares it")
}
if m.did("systemctl stop wg-quick@wg0") {
t.Error("a found unit already down was stopped again")
}
if report.Tunnel == nil || report.Tunnel.State != Taken {
t.Errorf("a steady takeover does not read as taken: %+v", report.Tunnel)
}
// Somebody starts the found unit again beside the mesh's interface. Not stopped by the mesh —
// on the hub it cannot hold the port, on a spoke stopping it would be a fight — but said.
m.units["wg-quick@wg0"].active = "active"
m.asked = nil
report, _ = applyAdopted(t, d, again, m, dir)
if m.did("systemctl stop wg-quick@wg0") {
t.Error("a found unit started again by hand was stopped by the mesh")
}
if report.Tunnel == nil || report.Tunnel.State != NotTaken || !strings.Contains(report.Tunnel.Note, "running again beside") {
t.Errorf("the account does not say the found unit is up again: %+v", report.Tunnel)
}
}
func TestAFoundInterfaceRaisedByHandIsRefusedNamingTheRemedy(t *testing.T) {
dir, config, mesh, keyFile, m := aHubInUse(t)
// The unit is not running, yet the interface is up: the predecessor raised it by hand.
m.units["wg-quick@wg0"].active = "inactive"
m.wgUp = "wg0 mesh0\n"
report, state, err := ApplyKeeping(t.Context(), archHost(t), aTakeover(t, config, mesh, keyFile, "51900", "192.0.2.1"),
store.State{}, store.OriginDeclared, m.run, nil, nil, KeepIn(dir))
if err == nil || !strings.Contains(err.Error(), "wg-quick down wg0") || !strings.Contains(err.Error(), "Nothing was flushed") {
t.Fatalf("an interface raised by hand was not refused naming the remedy: %v", err)
}
if m.units["wg-quick@mesh0"].active == "active" {
t.Error("the mesh's interface was started on a port the found one still holds")
}
// Looked at more than once before giving up: a person taking it down takes a moment.
shows := 0
for _, a := range m.asked {
if a == "wg show interfaces" {
shows++
}
}
if shows < takeoverRechecks+1 {
t.Errorf("the interface was looked at %d time(s) before the refusal; a person needs a moment", shows)
}
if report.Tunnel == nil || report.Tunnel.State != NotTaken {
t.Errorf("the account does not say the tunnel is not taken: %+v", report.Tunnel)
}
if _, held := state.HeldAt("mesh-wireguard.overlay-up.takes-over"); !held {
t.Error("the found configuration was not kept before the refusal")
}
}
func TestATakeoverIsRefusedOnAConvergedDeclaration(t *testing.T) {
_, err := declaration.Parse([]byte(`{"declaration":1,"resources":[
{"id":"up","type":"service","unit":"wg-quick@mesh0","state":"running",
"takes-over":{"interface":"wg0","unit":"wg-quick@wg0","config":"/etc/wireguard/wg0.conf"}}]}`))
if err == nil || !strings.Contains(err.Error(), "adopted") {
t.Fatalf("a takeover on a converged node was accepted: %v", err)
}
}
// novox/hq ADR 0119: once the take is proven — taken, and a peer handshaken on the mesh's
// interface — the found configuration is removed from where its unit reads it, its original stays
// kept and the hold on it ends. Never before, and never brought back.
const takesOverID = "mesh-wireguard.overlay-up.takes-over"
// handshaken is `wg show mesh0 latest-handshakes` with one of the two peers through.
const handshaken = "PEER-A=\t1790000000\nPEER-B=\t0\n"
func TestAProvenTakeRetiresTheFoundConfiguration(t *testing.T) {
dir, config, mesh, keyFile, m := aHubInUse(t)
m.handshakes = handshaken
report, state := applyAdopted(t, aTakeover(t, config, mesh, keyFile, "51900", "192.0.2.1"), store.State{}, m, dir)
if _, err := os.Lstat(config); !os.IsNotExist(err) {
t.Fatalf("a proven take left the found configuration where its unit reads it: %v", err)
}
retired, ok := state.RetiredAt(config)
if !ok || retired.Kept == "" || retired.ID != takesOverID {
t.Fatalf("the retirement was not recorded: %+v", state.Retired)
}
if kept, _ := os.ReadFile(retired.Kept); string(kept) != foundConf {
t.Fatalf("the kept original did not survive the retirement: %q", kept)
}
if _, held := state.HeldAt(takesOverID); held {
t.Error("the hold on the found configuration did not end with its retirement")
}
if u := m.units["wg-quick@wg0"]; u.active != "inactive" || u.enabled != "disabled" {
t.Errorf("the found unit is not left down and disabled: %+v", u)
}
if report.Tunnel == nil || report.Tunnel.State != Taken || report.Tunnel.Kept != retired.Kept ||
!strings.Contains(report.Tunnel.Note, "proven: 1 peer(s) handshaken on mesh0") ||
!strings.Contains(report.Tunnel.Note, "is retired") {
t.Fatalf("the account does not say the take is proven and the configuration retired: %+v", report.Tunnel)
}
if o := outcomeOf(report, takesOverID); o.Action != "removed" || !strings.Contains(o.Detail, "retired") {
t.Errorf("the retirement is not what the apply says it did to the file: %+v", o)
}
// And the account still carries what was found, read from the kept original.
if report.Tunnel.Port != 51900 || report.Tunnel.Peers != 2 {
t.Errorf("the account lost what the tunnel was: %+v", report.Tunnel)
}
}
func TestATakeNotProvenKeepsTheFoundConfigurationAndSaysSo(t *testing.T) {
cases := map[string]struct {
handshakes string
fail error
says string
}{
"no peer at all": {"", nil, "no peer has handshaken on mesh0"},
"every handshake at zero": {"PEER-A=\t0\nPEER-B=\t0\n", nil, "no peer has handshaken on mesh0"},
"wg is not there": {"", errors.New(`exec: "wg": executable file not found in $PATH`), "executable file not found"},
"the answer is nonsense": {"unable to access interface\n", nil, "not a peer and a time"},
}
for name, c := range cases {
dir, config, mesh, keyFile, m := aHubInUse(t)
m.handshakes, m.handshakesFail = c.handshakes, c.fail
d := aTakeover(t, config, mesh, keyFile, "51900", "192.0.2.1")
report, state := applyAdopted(t, d, store.State{}, m, dir)
if got, _ := os.ReadFile(config); string(got) != foundConf {
t.Fatalf("%s: a take not proven lost the found configuration", name)
}
if _, held := state.HeldAt(takesOverID); !held {
t.Errorf("%s: the hold ended although the take is not proven", name)
}
if _, retired := state.RetiredAt(config); retired {
t.Errorf("%s: recorded as retired", name)
}
if report.Tunnel == nil || report.Tunnel.State != Taken ||
!strings.Contains(report.Tunnel.Note, "taken, not yet proven") ||
!strings.Contains(report.Tunnel.Note, c.says) || !strings.Contains(report.Tunnel.Note, "is kept") {
t.Errorf("%s: the account does not say the take is not proven and why: %+v", name, report.Tunnel)
}
// A later apply that finds a peer through retires it: the take itself need not be the one.
m.handshakes, m.handshakesFail = handshaken, nil
_, state = applyAdopted(t, d, state, m, dir)
if _, err := os.Lstat(config); !os.IsNotExist(err) {
t.Errorf("%s: the apply after the take was proven kept the found configuration", name)
}
if _, retired := state.RetiredAt(config); !retired {
t.Errorf("%s: the later retirement was not recorded", name)
}
}
}
func TestAFoundConfigurationWhoseKeptOriginalIsMissingIsNotRetired(t *testing.T) {
dir, config, mesh, keyFile, m := aHubInUse(t)
d := aTakeover(t, config, mesh, keyFile, "51900", "192.0.2.1")
_, state := applyAdopted(t, d, store.State{}, m, dir)
held, _ := state.HeldAt(takesOverID)
if err := os.Remove(held.Kept); err != nil {
t.Fatal(err)
}
m.handshakes = handshaken
report, state := applyAdopted(t, d, state, m, dir)
if got, _ := os.ReadFile(config); string(got) != foundConf {
t.Fatal("the found configuration was removed with no kept original left of it")
}
if _, still := state.HeldAt(takesOverID); !still {
t.Error("the hold ended although nothing was retired")
}
if _, retired := state.RetiredAt(config); retired {
t.Error("recorded as retired")
}
if report.Tunnel == nil || !strings.Contains(report.Tunnel.Note, "is not retired") ||
!strings.Contains(report.Tunnel.Note, held.Kept+" is missing") {
t.Errorf("the account does not say the kept original is missing: %+v", report.Tunnel)
}
}
func TestAFoundConfigurationRewrittenSinceItWasFoundIsKeptAgainBeforeItGoes(t *testing.T) {
dir, config, mesh, keyFile, m := aHubInUse(t)
d := aTakeover(t, config, mesh, keyFile, "51900", "192.0.2.1")
_, state := applyAdopted(t, d, store.State{}, m, dir)
rewritten := foundConf + "\n[Peer]\nPublicKey = PEER-C=\nAllowedIPs = 192.0.2.4/32\n"
if err := os.WriteFile(config, []byte(rewritten), 0o600); err != nil {
t.Fatal(err)
}
m.handshakes = handshaken
_, state = applyAdopted(t, d, state, m, dir)
if _, err := os.Lstat(config); !os.IsNotExist(err) {
t.Fatal("a proven take kept a rewritten configuration")
}
retired, _ := state.RetiredAt(config)
if first, _ := os.ReadFile(retired.Kept); string(first) != foundConf {
t.Errorf("the first original was overwritten: %q", first)
}
kept, _ := filepath.Glob(filepath.Join(dir, "kept", "*-wg0.conf"))
var found bool
for _, k := range kept {
if got, _ := os.ReadFile(k); string(got) == rewritten {
found = true
}
}
if !found {
t.Errorf("what the file held when it was retired was not kept: %v", kept)
}
}
func TestARetiredTakeIsSteadyAndItsFoundUnitFindsNothingToDo(t *testing.T) {
dir, config, mesh, keyFile, m := aHubInUse(t)
m.handshakes = handshaken
d := aTakeover(t, config, mesh, keyFile, "51900", "192.0.2.1")
_, state := applyAdopted(t, d, store.State{}, m, dir)
retired, _ := state.RetiredAt(config)
// wg-quick@wg0 with no configuration: inactive, and disabled — the check of it every apply
// makes must find nothing to do and fail on nothing.
m.asked = nil
report, again := applyAdopted(t, d, state, m, dir)
if report.Changed() {
t.Errorf("an apply after the retirement moved the machine: %+v", report.Outcomes)
}
if m.did("systemctl stop wg-quick@wg0") || m.did("systemctl start wg-quick@wg0") {
t.Errorf("the retired tunnel's unit was acted on: %v", m.asked)
}
if _, err := os.Lstat(config); !os.IsNotExist(err) {
t.Error("the found configuration came back")
}
if _, held := again.HeldAt(takesOverID); held {
t.Error("a retired configuration is held again")
}
if r, ok := again.RetiredAt(config); !ok || r != retired {
t.Errorf("the retirement was not kept as it was: %+v", again.Retired)
}
if o := outcomeOf(report, takesOverID); o.Action != "unchanged" || !strings.Contains(o.Detail, "retired") {
t.Errorf("the retired configuration is not said as retired: %+v", o)
}
if report.Tunnel == nil || report.Tunnel.State != Taken || report.Tunnel.Kept != retired.Kept ||
!strings.Contains(report.Tunnel.Note, "retired") || report.Tunnel.Port != 51900 {
t.Errorf("the account of a retired take does not say so: %+v", report.Tunnel)
}
// Enabled at boot again by a person: disabled again, as any take does, and still no error.
m.units["wg-quick@wg0"].enabled = "enabled"
_, _ = applyAdopted(t, d, again, m, dir)
if m.units["wg-quick@wg0"].enabled != "disabled" {
t.Error("the found unit enabled again by hand was left to start at boot")
}
}
func TestUndeclaringThePrivateNetworkAfterRetirementBringsNothingBack(t *testing.T) {
dir, config, mesh, keyFile, m := aHubInUse(t)
m.handshakes = handshaken
d := aTakeover(t, config, mesh, keyFile, "51900", "192.0.2.1")
_, state := applyAdopted(t, d, store.State{}, m, dir)
// The private network unassigned: only something else is declared.
other := adopted(t, `{"taken":[],"untaken":{}}`,
`{"id":"other.file","type":"file","path":"`+filepath.Join(dir, "other.conf")+`","content":"x\n"}`)
m.asked = nil
_, after := applyAdopted(t, other, state, m, dir)
if _, err := os.Lstat(config); !os.IsNotExist(err) {
t.Fatal("undeclaring the private network brought the found configuration back")
}
if m.did("systemctl start wg-quick@wg0") || m.did("systemctl enable wg-quick@wg0") {
t.Errorf("undeclaring the private network started the found tunnel: %v", m.asked)
}
if _, ok := after.RetiredAt(config); !ok {
t.Error("the retirement was forgotten with the private network")
}
// Assigned again, it finds the configuration retired rather than missing, and raises the
// mesh's interface.
report, _ := applyAdopted(t, d, after, m, dir)
if report.Tunnel == nil || report.Tunnel.State != Taken {
t.Errorf("the private network assigned again did not take the tunnel: %+v", report.Tunnel)
}
if _, err := os.Lstat(config); !os.IsNotExist(err) {
t.Error("assigning the private network again brought the found configuration back")
}
}
func TestAPlanSaysTheFoundConfigurationIsRetiredWhenTheTakeIsProven(t *testing.T) {
dir, config, mesh, keyFile, m := aHubInUse(t)
d := aTakeover(t, config, mesh, keyFile, "51900", "192.0.2.1")
plan := Plan(d, store.State{}, store.OriginDeclared)
report, state := applyAdopted(t, d, store.State{}, m, dir)
if got, want := strings.Join(ids(plan), " "), strings.Join(outcomeIDs(report), " "); got != want {
t.Errorf("the plan said %q and the apply did %q", got, want)
}
var take Step
for _, s := range plan {
if s.ID == takesOverID {
take = s
}
}
if take.Verb != "hold" || take.Target != config || !strings.Contains(take.Why, "retired — removed from "+config) ||
!strings.Contains(take.Why, "handshaking on mesh0") {
t.Errorf("the plan does not say the found configuration is retired once proven: %+v", take)
}
// Held and still declared: never planned as forgotten.
if strings.Contains(verbs(Plan(d, state, store.OriginDeclared)), "forget "+takesOverID) {
t.Error("the plan forgets a hold the apply keeps")
}
m.handshakes = handshaken
_, state = applyAdopted(t, d, state, m, dir)
for _, s := range Plan(d, state, store.OriginDeclared) {
if s.ID == takesOverID && (s.Verb != "check" || !strings.Contains(s.Why, "retired once the take")) {
t.Errorf("a retired configuration is planned as %+v", s)
}
}
}
// keptCopies is every copy kept of the found configuration.
func keptCopies(t *testing.T, dir string) []string {
t.Helper()
kept, err := filepath.Glob(filepath.Join(dir, "kept", "*-wg0.conf"))
if err != nil {
t.Fatal(err)
}
return kept
}
func TestAConfigurationPutBackIsRetiredAgainOnItsFirstOriginalAndSettles(t *testing.T) {
dir, config, mesh, keyFile, m := aHubInUse(t)
m.handshakes = handshaken
d := aTakeover(t, config, mesh, keyFile, "51900", "192.0.2.1")
_, state := applyAdopted(t, d, store.State{}, m, dir)
first, _ := state.RetiredAt(config)
// Put back by hand with the original, while no peer is through yet: held on the first
// original, and the account says what a rollback takes.
write(t, config, foundConf)
m.handshakes = ""
report, state := applyAdopted(t, d, state, m, dir)
if held, ok := state.HeldAt(takesOverID); !ok || held.Kept != first.Kept {
t.Fatalf("what came back is not held on the first original: %+v", held)
}
if report.Tunnel == nil || !strings.Contains(report.Tunnel.Note, "came back after it was retired") ||
!strings.Contains(report.Tunnel.Note, "unassigning the private network first") {
t.Errorf("the account does not say a rollback means unassigning the private network: %+v", report.Tunnel)
}
// Proven: retired again, nothing more kept, said once.
m.handshakes = handshaken
report, state = applyAdopted(t, d, state, m, dir)
again, _ := state.RetiredAt(config)
if _, err := os.Lstat(config); !os.IsNotExist(err) || again.Kept != first.Kept || again.Extra != "" {
t.Fatalf("put back as it was, it was not retired again on the first original: %+v", again)
}
if o := outcomeOf(report, takesOverID); o.Action != "removed" ||
!strings.HasPrefix(o.Detail, "the found configuration came back and was retired again") ||
strings.Contains(o.Detail, "differed") {
t.Errorf("the second retirement is not said as one: %+v", o)
}
if !strings.Contains(report.Tunnel.Note, "unassigning the private network first") {
t.Errorf("the account does not say what a rollback takes: %q", report.Tunnel.Note)
}
if n := len(keptCopies(t, dir)); n != 1 {
t.Errorf("%d copies kept of one content", n)
}
if report, _ := applyAdopted(t, d, state, m, dir); report.Changed() {
t.Errorf("a steady machine moved after the second retirement: %+v", report.Outcomes)
}
// Put back with something else: that is kept beside the first original, which stays the record's.
other := strings.Replace(foundConf, "PEER-B=", "PEER-Z=", 1)
write(t, config, other)
report, state = applyAdopted(t, d, state, m, dir)
third, _ := state.RetiredAt(config)
if third.Kept != first.Kept || third.Extra == "" || third.Extra == first.Kept {
t.Fatalf("other content was not kept apart from the first original: %+v", third)
}
if got, _ := os.ReadFile(third.Extra); string(got) != other {
t.Errorf("the extra copy does not hold what was put back: %q", got)
}
if o := outcomeOf(report, takesOverID); !strings.Contains(o.Detail, third.Extra) ||
!strings.Contains(report.Tunnel.Note, third.Extra) {
t.Errorf("where the extra copy is was not said: %+v / %q", o, report.Tunnel.Note)
}
// And the same other content again: nothing more kept, the record as it was.
write(t, config, other)
report, state = applyAdopted(t, d, state, m, dir)
fourth, _ := state.RetiredAt(config)
if fourth.Kept != first.Kept || fourth.Extra != third.Extra || len(keptCopies(t, dir)) != 2 {
t.Errorf("the same content put back again grew the copies: %+v, %v", fourth, keptCopies(t, dir))
}
if o := outcomeOf(report, takesOverID); strings.Contains(o.Detail, "differed") {
t.Errorf("a copy already kept was said as new: %+v", o)
}
if report, _ := applyAdopted(t, d, state, m, dir); report.Changed() {
t.Errorf("a steady machine moved: %+v", report.Outcomes)
}
}
func TestOnlyWgQuicksOwnConfigurationIsRetired(t *testing.T) {
// Not under the wireguard directory.
dir, config, mesh, keyFile, m := aHubInUse(t)
wireguardDir = filepath.Join(dir, "elsewhere")
m.handshakes = handshaken
report, state := applyAdopted(t, aTakeover(t, config, mesh, keyFile, "51900", "192.0.2.1"), store.State{}, m, dir)
if got, _ := os.ReadFile(config); string(got) != foundConf {
t.Fatal("a configuration outside wg-quick's directory was removed")
}
if _, held := state.HeldAt(takesOverID); !held || !strings.Contains(report.Tunnel.Note, "is not retired: only ") {
t.Errorf("the refusal is not said, or the hold ended: %+v", report.Tunnel)
}
// A path the mesh itself writes.
dir, config, mesh, keyFile, m = aHubInUse(t)
m.handshakes = handshaken
known := store.State{}
known.Record(store.Applied{ID: "bundle.wg0", Type: "file", Target: config, Origin: store.OriginCarried})
report, _ = applyAdopted(t, aTakeover(t, config, mesh, keyFile, "51900", "192.0.2.1"), known, m, dir)
if got, _ := os.ReadFile(config); string(got) != foundConf {
t.Fatal("a path the mesh writes was retired")
}
if !strings.Contains(report.Tunnel.Note, "a path the mesh itself writes") {
t.Errorf("the refusal is not said: %+v", report.Tunnel)
}
}
func TestAFoundConfigurationThatIsALinkIsKeptAndLeftToAPerson(t *testing.T) {
dir, config, mesh, keyFile, m := aHubInUse(t)
target := filepath.Join(dir, "predecessor", "hub.conf")
if err := os.MkdirAll(filepath.Dir(target), 0o700); err != nil {
t.Fatal(err)
}
write(t, target, foundConf)
if err := os.Remove(config); err != nil {
t.Fatal(err)
}
if err := os.Symlink(target, config); err != nil {
t.Fatal(err)
}
m.handshakes = handshaken
report, state := applyAdopted(t, aTakeover(t, config, mesh, keyFile, "51900", "192.0.2.1"), store.State{}, m, dir)
if _, err := os.Lstat(config); err != nil {
t.Fatal("the link was removed, leaving the key-bearing file it points at")
}
if got, _ := os.ReadFile(target); string(got) != foundConf {
t.Fatal("the file the link points at was touched")
}
held, ok := state.HeldAt(takesOverID)
if !ok {
t.Fatal("the hold ended")
}
if kept, _ := os.ReadFile(held.Kept); string(kept) != foundConf {
t.Errorf("what was kept is not what the link points at: %q", kept)
}
if !strings.Contains(report.Tunnel.Note, "is a link to "+target) || !strings.Contains(report.Tunnel.Note, "by hand") {
t.Errorf("the account does not say the link must be retired by hand: %q", report.Tunnel.Note)
}
}
func TestARetirementWhoseRecordWasNeverSavedIsRecordedByTheNextApply(t *testing.T) {
dir, config, mesh, keyFile, m := aHubInUse(t)
d := aTakeover(t, config, mesh, keyFile, "51900", "192.0.2.1")
_, state := applyAdopted(t, d, store.State{}, m, dir)
held, _ := state.HeldAt(takesOverID)
// An apply removed the file and stopped before its state was saved.
if err := os.Remove(config); err != nil {
t.Fatal(err)
}
m.handshakes = handshaken
report, state := applyAdopted(t, d, state, m, dir)
if r, ok := state.RetiredAt(config); !ok || r.Kept != held.Kept {
t.Fatalf("the retirement was not recorded: %+v", state.Retired)
}
if _, still := state.HeldAt(takesOverID); still {
t.Error("the hold did not end")
}
if o := outcomeOf(report, takesOverID); o.Action != "unchanged" || !strings.Contains(o.Detail, "already gone") {
t.Errorf("a file already gone is not said as such: %+v", o)
}
}
func TestARemovalThatFailsKeepsTheFileAndTheHold(t *testing.T) {
if os.Geteuid() == 0 {
t.Skip("root removes from a directory it may not write to")
}
dir, _, mesh, keyFile, m := aHubInUse(t)
wg := filepath.Join(dir, "wireguard")
if err := os.MkdirAll(wg, 0o700); err != nil {
t.Fatal(err)
}
config := filepath.Join(wg, "wg0.conf")
write(t, config, foundConf)
wireguardDir = wg
d := aTakeover(t, config, mesh, keyFile, "51900", "192.0.2.1")
_, state := applyAdopted(t, d, store.State{}, m, dir)
if err := os.Chmod(wg, 0o500); err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = os.Chmod(wg, 0o700) })
m.handshakes = handshaken
report, state := applyAdopted(t, d, state, m, dir)
if got, _ := os.ReadFile(config); string(got) != foundConf {
t.Fatal("the found configuration is gone although it could not be removed")
}
if _, held := state.HeldAt(takesOverID); !held {
t.Error("the hold ended although nothing was retired")
}
if _, retired := state.RetiredAt(config); retired {
t.Error("recorded as retired")
}
if !strings.Contains(report.Tunnel.Note, "removing it failed") || !strings.Contains(report.Tunnel.Note, "permission denied") {
t.Errorf("the failed removal is not said: %q", report.Tunnel.Note)
}
}
+99
View File
@@ -0,0 +1,99 @@
package apply
import (
"context"
"fmt"
"strings"
"testing"
"github.com/novox/mesh-host/internal/store"
)
// Defends novox/hq ADR 0177: a unit in the operator account's own service manager is applied
// through that manager — `systemctl --user --machine=<account>@` — and never as a system unit of
// the same name; its record remembers the scope so removal goes the same way.
// accountManager is a user's service manager whose one unit starts when asked, recording the
// commands and refusing any that reach it outside the account's scope.
func accountManager(account string, commands *[]string) Runner {
active, enabled := false, false
return func(_ context.Context, name string, args ...string) (string, error) {
line := name + " " + strings.Join(args, " ")
*commands = append(*commands, line)
if name == "systemctl" && !strings.HasPrefix(line, "systemctl --user --machine="+account+"@ ") {
return "", fmt.Errorf("a system-scope command reached the account's manager: %s", line)
}
switch {
case strings.Contains(line, " start "):
active = true
return "", nil
case strings.Contains(line, " stop "):
active = false
return "", nil
case strings.Contains(line, " enable "):
enabled = true
return "", nil
case strings.Contains(line, " disable "):
enabled = false
return "", nil
case strings.Contains(line, "is-enabled"):
if enabled {
return "enabled", nil
}
return "disabled", nil
case strings.Contains(line, "show") && strings.Contains(line, "ActiveState"):
if active {
return "LoadState=loaded\nActiveState=active\nSubState=running", nil
}
return "LoadState=loaded\nActiveState=inactive\nSubState=dead", nil
}
return "", nil
}
}
func TestAUserScopedUnitIsAppliedThroughTheAccountsManager(t *testing.T) {
var commands []string
decl := `{"declaration":1,"resources":[
{"id":"i3.watcher","type":"service","unit":"i3-reload-watcher.service","state":"running","boot":"enabled","scope":"user","user":"ops"}
]}`
report, known, err := Apply(context.Background(), archHost(t), parse(t, decl),
store.State{}, store.OriginDeclared, accountManager("ops", &commands), nil, nil)
if err != nil {
t.Fatalf("apply: %v\n%s", err, strings.Join(commands, "\n"))
}
if !report.Changed() {
t.Fatal("a unit that was stopped and is now running changed nothing")
}
var started, enabled bool
for _, c := range commands {
if c == "systemctl --user --machine=ops@ start i3-reload-watcher.service" {
started = true
}
if c == "systemctl --user --machine=ops@ enable i3-reload-watcher.service" {
enabled = true
}
}
if !started || !enabled {
t.Fatalf("the unit was not started and enabled in the account's manager:\n%s", strings.Join(commands, "\n"))
}
recorded, ok := known.At("service", "i3-reload-watcher.service")
if !ok || recorded.Scope != "user" || recorded.User != "ops" {
t.Fatalf("the record does not say whose manager the unit is in: %+v", recorded)
}
}
func TestASystemUnitIsUntouchedByTheScope(t *testing.T) {
var commands []string
decl := `{"declaration":1,"resources":[
{"id":"x.daemon","type":"service","unit":"sshd.service","state":"running","boot":"enabled"}
]}`
if _, _, err := Apply(context.Background(), archHost(t), parse(t, decl),
store.State{}, store.OriginDeclared, unitIn(true, &commands), nil, nil); err != nil {
t.Fatal(err)
}
for _, c := range commands {
if strings.Contains(c, "--user") || strings.Contains(c, "--machine") {
t.Fatalf("a system unit was addressed to an account's manager: %s", c)
}
}
}
+49 -1
View File
@@ -376,7 +376,7 @@ func TestAContainerIsGivenItsEnvironmentFiles(t *testing.T) {
var ran []string
run := func(_ context.Context, name string, args ...string) (string, error) {
ran = append(ran, name+" "+strings.Join(args, " "))
if len(args) > 0 && args[0] == "inspect" {
if len(args) > 0 && args[0] == "container" {
return "", fmt.Errorf("no such container")
}
return "", nil
@@ -403,3 +403,51 @@ func TestAContainerIsGivenItsEnvironmentFiles(t *testing.T) {
}
}
}
// A container may name its resolvers and its own address — the shape a module shipping its own
// validating DNS needs: the resolver pinned where its siblings can find it, the siblings pointed
// at it. Both flags take addresses, so both reach the runtime verbatim.
func TestAContainerIsGivenItsResolverAndItsAddress(t *testing.T) {
var ran []string
run := func(_ context.Context, name string, args ...string) (string, error) {
ran = append(ran, name+" "+strings.Join(args, " "))
if len(args) > 0 && args[0] == "container" {
return "", fmt.Errorf("no such container")
}
return "", nil
}
d := declare(t, `{"id":"imap","type":"container","name":"mailu-imap",`+
`"image":"dovecot@sha256:0000000000000000000000000000000000000000000000000000000000000000",`+
`"network":"mailu","dns":["192.168.203.254"],"ip":"192.168.203.7"}`)
_, _, _ = Apply(context.Background(), archHost(t), d, store.State{},
store.OriginDeclared, run, nil, nil)
var started string
for _, line := range ran {
if strings.Contains(line, "run ") {
started = line
}
}
for _, want := range []string{"--dns 192.168.203.254", "--ip 192.168.203.7"} {
if !strings.Contains(started, want) {
t.Errorf("the container was started without %q:\n%s", want, started)
}
}
}
// The resolver and address are part of the spec — a container whose dns or ip moved is a
// different container, or the fields can never reach one that already runs. That is not
// hypothetical: their first deployment compared equal and changed nothing.
func TestAChangedResolverOrAddressIsAChangedContainer(t *testing.T) {
base := &declaration.Container{ID: "c", Name: "x", Image: "a@sha256:00"}
withDns := &declaration.Container{ID: "c", Name: "x", Image: "a@sha256:00", Dns: []string{"192.168.203.254"}}
withIP := &declaration.Container{ID: "c", Name: "x", Image: "a@sha256:00", IP: "192.168.203.254"}
plain := containerSpecReading(base, nil, nil)
if containerSpecReading(withDns, nil, nil) == plain {
t.Error("adding a resolver did not change the spec, so it can never reach a running container")
}
if containerSpecReading(withIP, nil, nil) == plain {
t.Error("adding an address did not change the spec, so it can never reach a running container")
}
}
+130
View File
@@ -0,0 +1,130 @@
package bootstrap
import (
"context"
"errors"
"os"
"path/filepath"
"strings"
"testing"
"github.com/novox/mesh-host/internal/apply"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/store"
)
// Defends phase three's premise (phase3.go): the store genesis raised is adopted by the postgres
// module IN PLACE — same name, same image, same spec — so the applier reconciles it and never
// recreates the mesh's memory with the temporary control plane connected to it.
//
// The host folds a mounted file's content into the container's spec (novox/hq 04-ISSUES/103), so
// this now depends on a byte: the superuser file genesis writes and mounts must be the same bytes
// the module later declares. The module's value is what `secret accept` took — the operator's
// file with its line ending removed and nothing else (mesh-control, asSupplied). Reproduced before
// it was fixed: genesis wrote `value\n`, the module wrote `value`, and the store was recreated
// during install.
// labelled is a runtime that keeps the spec label the host gives a container and hands it back.
type labelled struct {
spec map[string]string
created []string
removed []string
}
func (l *labelled) run(_ context.Context, _ string, args ...string) (string, error) {
switch args[0] {
case "info":
return "27.0\n", nil
case "container":
spec, ok := l.spec[args[len(args)-1]]
if !ok {
return "", errors.New("no such container")
}
return "true\t" + spec + "\n", nil
case "rm":
l.removed = append(l.removed, args[len(args)-1])
delete(l.spec, args[len(args)-1])
case "run":
var name, spec string
for i, a := range args {
if a == "--name" {
name = args[i+1]
}
if a == "--label" && strings.HasPrefix(args[i+1], "mesh-host.spec=") {
spec = strings.TrimPrefix(args[i+1], "mesh-host.spec=")
}
}
l.spec[name] = spec
l.created = append(l.created, name)
return "made\n", nil
}
return "", nil
}
func TestTheStoreGenesisRaisedIsAdoptedInPlaceNotRecreated(t *testing.T) {
dir := t.TempDir()
secret := filepath.Join(dir, "superuser.secret")
// The bytes genesis really writes — the code path, not a fixture that agrees with it.
if _, made, err := keptOrMade(secret, false); err != nil || !made {
t.Fatalf("genesis did not make the superuser secret: made=%v err=%v", made, err)
}
onDisk, err := os.ReadFile(secret)
if err != nil {
t.Fatal(err)
}
image := "docker.io/library/postgres@sha256:" + strings.Repeat("ab", 32)
mounts := `"volumes":["mesh-store-data:/var/lib/postgresql/data","` + secret + `:` + storeSuperuserMount + `:ro"]`
// The foundation's store, as the produced bundle raises it (RewriteRoot): the file mounted,
// declared by nothing — genesis wrote it before there was a declaration to name it.
raise, err := declaration.ParseTrusted([]byte(`{"declaration":1,"resources":[
{"id":"` + StoreID + `","type":"container","name":"mesh-store","image":"` + image + `",
"env":{"POSTGRES_PASSWORD_FILE":"` + storeSuperuserMount + `"},` + mounts + `}
]}`))
if err != nil {
t.Fatal(err)
}
runtime := &labelled{spec: map[string]string{}}
_, known, err := apply.Apply(context.Background(), arch(t), raise, store.State{}, store.OriginCarried,
runtime.run, nil, nil)
if err != nil {
t.Fatalf("raising the foundation's store: %v", err)
}
if len(runtime.created) != 1 {
t.Fatalf("the store was not raised once: %v", runtime.created)
}
// The postgres module's declaration of the same store: the superuser file as `secret accept`
// took it in — its line ending removed and nothing else — then the same container.
accepted := strings.TrimRight(string(onDisk), "\r\n")
adopt, err := declaration.ParseTrusted([]byte(`{"declaration":1,"resources":[
{"id":"postgres.superuser","type":"file","path":"` + secret + `","content":"` + accepted + `","mode":"0600"},
{"id":"postgres.server","type":"container","name":"mesh-store","image":"` + image + `",
"env":{"POSTGRES_PASSWORD_FILE":"` + storeSuperuserMount + `"},` + mounts + `}
]}`))
if err != nil {
t.Fatal(err)
}
runtime.created, runtime.removed = nil, nil
report, _, err := apply.Apply(context.Background(), arch(t), adopt, known, store.OriginDeclared,
runtime.run, nil, nil)
if err != nil {
t.Fatalf("adopting the store: %v", err)
}
if len(runtime.removed) > 0 || len(runtime.created) > 0 {
t.Fatalf("the module's declaration recreated the store genesis raised (removed %v, created %v): "+
"the file genesis mounted and the file the module declares are not the same bytes",
runtime.removed, runtime.created)
}
for _, o := range report.Outcomes {
if o.ID == "postgres.server" && o.Action != "unchanged" {
t.Errorf("the store was not adopted in place: %+v", o)
}
if o.ID == "postgres.superuser" && o.Action != "unchanged" {
t.Errorf("the module rewrote the superuser file genesis wrote: %+v", o)
}
}
}
+202
View File
@@ -0,0 +1,202 @@
package bootstrap
import (
"bytes"
"context"
"encoding/json"
"fmt"
"sort"
"strconv"
"strings"
"github.com/novox/mesh-host/internal/declaration"
)
// What an adopted genesis changes about the foundation (novox/hq ADR 0100).
//
// **The firewall found on the machine stays in force.** The foundation's own filter drops by
// default, and every base chain at a hook runs; an accept ends only its own chain and a drop in any
// is final — so loading it would close whatever the machine serves. On an adopted machine it is
// not loaded. Its duty, the store never reachable from outside, passes to the mesh's guard: a table
// of the mesh's own that only refuses, and only the foundation's own ports, which genesis has just
// checked free — so it cannot close anything the machine serves.
// The guard, as the controller declares it: the same ids, paths and text, so the first push
// finds it already there and takes it over unchanged.
const (
guardID = declaration.AdoptionPrefix + "guard"
guardUnitID = declaration.AdoptionPrefix + "guard-unit"
guardRunningID = declaration.AdoptionPrefix + "guard-running"
guardPath = "/etc/mesh/guard.nft"
guardUnit = "mesh-guard.service"
guardUnitPath = "/etc/systemd/system/" + guardUnit
)
// AsGuard renders the mesh's refusal-only table for the given machine ports. It passes everything
// by default; it refuses the ports except from the machine itself — its loopback and the container
// runtime's own networks — and from the private network, known by the interface a packet arrives
// on and never by its source address; at prerouting, ahead of the runtime's destination
// translation, in the inet family so both address families. It matches only packets addressed to
// this machine: what the machine routes for others is never its business (novox/hq ADR 0103).
//
// Character for character the controller's (mesh-controller internal/catalogue AsGuard); a test
// on each side holds its copy to the same golden text.
func AsGuard(ports []int) string {
sorted := append([]int{}, ports...)
sort.Ints(sorted)
listed := make([]string, len(sorted))
for i, p := range sorted {
listed[i] = strconv.Itoa(p)
}
var b strings.Builder
b.WriteString("table inet mesh_guard {}\n")
b.WriteString("delete table inet mesh_guard\n")
b.WriteString("table inet mesh_guard {\n")
b.WriteString("\tchain prerouting {\n")
b.WriteString("\t\ttype filter hook prerouting priority raw; policy accept;\n")
fmt.Fprintf(&b, "\t\tfib daddr type local iifname != \"lo\" iifname != \"docker0\" iifname != \"br-*\" "+
"iifname != \"mesh0\" tcp dport { %s } drop\n", strings.Join(listed, ", "))
b.WriteString("\t}\n")
b.WriteString("}\n")
return b.String()
}
// guardUnitText is the unit that loads the guard. Stopping it deletes only its own table — never a
// flush, which would take the container runtime's rules and the found firewall with it.
func guardUnitText() string {
return "[Unit]\n" +
"Description=The mesh's guard: refuses its own ports from outside (novox/hq ADR 0100)\n" +
"DefaultDependencies=no\n" +
"Wants=network-pre.target\n" +
"Before=network-pre.target shutdown.target\n" +
"Conflicts=shutdown.target\n" +
"\n" +
"[Service]\n" +
"Type=oneshot\n" +
"RemainAfterExit=yes\n" +
"ExecStart=nft -f " + guardPath + "\n" +
"ExecReload=nft -f " + guardPath + "\n" +
"ExecStop=nft delete table inet mesh_guard\n" +
"\n" +
"[Install]\n" +
"WantedBy=multi-user.target\n"
}
// guardResources are the guard as three resources of kinds the host already has.
func guardResources(ports []int) []map[string]any {
return []map[string]any{
{"id": guardID, "type": "file", "path": guardPath, "content": AsGuard(ports), "mode": "0644"},
{"id": guardUnitID, "type": "file", "path": guardUnitPath, "content": guardUnitText(), "mode": "0644"},
// A changed table is reloaded — the unit's ExecReload loads it in one transaction, so the
// ports are never unguarded — and only a changed unit restarts it. As the controller
// declares it, so the first push finds nothing different.
{"id": guardRunningID, "type": "service", "unit": guardUnit, "state": "running",
"boot": "enabled", "reload-on": []any{guardID}, "restart-on": []any{guardUnitID}},
}
}
// guardAfter is where the guard goes: once the container runtime runs, before anything publishes
// a port.
const guardAfter = "container-runtime-running"
// AdoptedRewrite says what RewriteAdopted did.
type AdoptedRewrite struct {
Removed []string
Guarded []int
}
// RewriteAdopted makes the produced bundle one for an adopted machine: the foundation's own filter
// taken out, and the mesh's guard put in its place, guarding on this node the store's port, the
// broker's management port and the broker's plaintext port. The last is published on every
// interface and the foundation's filter admits it from the private network only, so a found
// firewall that filters only incoming traffic would leave it reachable from anywhere (novox/hq ADR
// 0103). Every one is a port of a module genesis takes. The nftables package stays: the guard is loaded with it, and
// installing a package loads no table. Openings are not the bundle's — the first push declares
// them, once there is a controller to derive them.
func RewriteAdopted(r *Rewritten, p FoundationPorts) (AdoptedRewrite, error) {
var out AdoptedRewrite
p = p.orDefaults()
bundle := r.Bundle
var err error
for _, id := range []string{"base-filter-loaded", "base-filter"} {
if !r.declares(id) {
continue
}
if bundle, err = removeResource(bundle, id); err != nil {
return out, err
}
out.Removed = append(out.Removed, id)
}
out.Guarded = []int{p.Store, p.Management, p.AMQP}
var text bytes.Buffer
text.WriteString(",\n // The mesh's guard (novox/hq ADR 0100): this machine is adopted, so its own firewall\n" +
" // stays in force and the foundation's filter is not loaded. The guard only refuses: the\n" +
" // store's port and the broker's management and plaintext ports, except from the machine and\n" +
" // the private network.")
for _, res := range guardResources(out.Guarded) {
var one bytes.Buffer
enc := json.NewEncoder(&one)
enc.SetEscapeHTML(false)
if err := enc.Encode(res); err != nil {
return out, err
}
text.WriteString("\n ")
text.Write(bytes.TrimSpace(one.Bytes()))
text.WriteString(",")
}
insert := bytes.TrimSuffix(text.Bytes(), []byte(","))
_, _, to, err := resourceAt(bundle, guardAfter)
if err != nil {
return out, fmt.Errorf("the guard goes after %q, and %w", guardAfter, err)
}
rest := bundle[to:]
joined := make([]byte, 0, len(bundle)+len(insert))
joined = append(joined, bundle[:to]...)
joined = append(joined, insert...)
// What followed the resource — its own comma, or the end of the list — now follows the guard.
if trimmed := bytes.TrimLeft(rest, " \t\r\n"); len(trimmed) > 0 && trimmed[0] != ',' && trimmed[0] != ']' {
return out, fmt.Errorf("the bundle does not separate %q from what follows it the way a list does", guardAfter)
}
joined = append(joined, rest...)
parsed, err := declaration.ParseFileTrusted(joined)
if err != nil {
return out, fmt.Errorf("the bundle stopped being a declaration once it was made an adopted one, which is this installer's fault: %w", err)
}
r.Bundle, r.Declaration, r.Resources = joined, parsed, len(parsed.Resources)
return out, nil
}
// declares is whether the produced bundle names a resource.
func (r Rewritten) declares(id string) bool {
for _, res := range r.Declaration.Resources {
if res.Identity() == id {
return true
}
}
return false
}
// genesisTakes are the modules an adopted genesis takes as it installs them: the foundation's and
// the mesh's own, whose names genesis checked free, so taking them replaces nothing a predecessor
// ran. The private network is not among them — it rewrites the machine's hosts file and the
// container runtime's configuration whole — and neither is anything the operator installs later.
var genesisTakes = map[string]bool{
RegistryModule: true, ControlPlaneModule: true, BuilderModule: true,
"postgres": true, "lavinmq": true, "mesh-vault": true, "mesh-catalog": true,
}
// takeIfAdopted takes one of genesis's own modules on an adopted node, once it is assigned and
// before the push that raises it.
func takeIfAdopted(ctx context.Context, o Options, control controlPlane, module string, say func(string)) error {
if !o.Adopted || !genesisTakes[module] {
return nil
}
if _, err := control.tell(ctx, "take", o.Node, module); err != nil {
return err
}
say(" taken " + module + " on " + o.Node + " — the mesh's own, its name checked free")
return nil
}
+233
View File
@@ -0,0 +1,233 @@
package bootstrap
import (
"context"
"errors"
"fmt"
"strings"
"testing"
"time"
"github.com/novox/mesh-host/internal/declaration"
)
// Defends novox/hq ADR 0100: an adopted genesis loads no table that drops by default or holds an
// accept; the mesh guards its own ports in a table that only refuses; and genesis takes the mesh's
// own modules as it installs them, and nothing else.
// The same golden text the controller's test holds its AsGuard to.
const goldenGuard = `table inet mesh_guard {}
delete table inet mesh_guard
table inet mesh_guard {
chain prerouting {
type filter hook prerouting priority raw; policy accept;
fib daddr type local iifname != "lo" iifname != "docker0" iifname != "br-*" iifname != "mesh0" tcp dport { 5432, 15672 } drop
}
}
`
func TestTheGuardIsExactlyThisTable(t *testing.T) {
if got := AsGuard([]int{15672, 5432}); got != goldenGuard {
t.Fatalf("the guard changed:\n%s", got)
}
}
// The same golden unit the controller's test holds its guard unit to. It is loaded before the
// network is up, so it carries no default dependencies, and it is stopped only at shutdown.
const goldenGuardUnit = `[Unit]
Description=The mesh's guard: refuses its own ports from outside (novox/hq ADR 0100)
DefaultDependencies=no
Wants=network-pre.target
Before=network-pre.target shutdown.target
Conflicts=shutdown.target
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=nft -f /etc/mesh/guard.nft
ExecReload=nft -f /etc/mesh/guard.nft
ExecStop=nft delete table inet mesh_guard
[Install]
WantedBy=multi-user.target
`
func TestTheGuardUnitIsExactlyThisUnit(t *testing.T) {
if got := guardUnitText(); got != goldenGuardUnit {
t.Fatalf("the guard's unit changed:\n%s", got)
}
}
func TestTheGuardRefusesOnlyWhatIsAddressedToThisMachine(t *testing.T) {
// A machine that routes for others — a predecessor's private-network hub — must not have a
// packet for another machine's database port refused (novox/hq ADR 0103).
for _, line := range strings.Split(AsGuard([]int{5432}), "\n") {
if strings.Contains(line, " drop") && !strings.HasPrefix(strings.TrimSpace(line), "fib daddr type local ") {
t.Errorf("a refusal matches packets not addressed to this machine: %q", line)
}
}
}
func TestAnAdoptedBundleLoadsNoDroppingTableAndExactlyTheGuard(t *testing.T) {
r := producedBundle(t)
p := FoundationPorts{Store: 5433, Management: 15673, AMQP: 5773}
if _, err := RewritePorts(&r, p, ""); err != nil {
t.Fatal(err)
}
got, err := RewriteAdopted(&r, p)
if err != nil {
t.Fatal(err)
}
if strings.Join(got.Removed, ",") != "base-filter-loaded,base-filter" {
t.Errorf("removed %v", got.Removed)
}
at := map[string]int{}
var guards []*declaration.File
for i, res := range r.Declaration.Resources {
at[res.Identity()] = i
if f, ok := res.(*declaration.File); ok {
if strings.Contains(f.Content, "policy drop") || strings.Contains(f.Content, " accept\n") &&
!strings.Contains(f.Content, "policy accept") {
t.Errorf("%s loads a table that drops or accepts: %q", f.ID, f.Content)
}
if f.Path == guardPath {
guards = append(guards, f)
}
}
if s, ok := res.(*declaration.Service); ok && s.Unit == "nftables.service" {
t.Errorf("the foundation's filter is still loaded by %s", s.ID)
}
}
if len(guards) != 1 {
t.Fatalf("%d guard table(s)", len(guards))
}
// The store's, the broker's plaintext and its management port: each one the filter admits
// from the private network only (novox/hq ADR 0103).
if !strings.Contains(guards[0].Content, "tcp dport { 5433, 5773, 15673 } drop") {
t.Errorf("the guard does not refuse this node's ports: %s", guards[0].Content)
}
if strings.Count(guards[0].Content, "accept") != 1 || !strings.Contains(guards[0].Content, "policy accept") {
t.Errorf("the guard holds an accept of its own: %s", guards[0].Content)
}
for _, id := range []string{guardID, guardUnitID, guardRunningID} {
if _, ok := at[id]; !ok {
t.Errorf("the bundle has no %s", id)
}
}
if !(at["container-runtime-running"] < at[guardID] && at[guardRunningID] < at["store"]) {
t.Errorf("the guard is not between the runtime and the store: %v", at)
}
if _, kept := at["base-filter-package"]; !kept {
t.Error("nft, which loads the guard, is no longer installed")
}
unit := r.Declaration.Resources[at[guardRunningID]].(*declaration.Service)
// A changed table is reloaded, never restarted: a restart deletes the table before loading
// it again, leaving the ports unguarded in between.
if unit.Unit != guardUnit || unit.State != "running" || strings.Join(unit.RestartOn, ",") != guardUnitID ||
strings.Join(unit.ReloadOn, ",") != guardID {
t.Errorf("the guard's service: %+v", unit)
}
stop := r.Declaration.Resources[at[guardUnitID]].(*declaration.File).Content
if !strings.Contains(stop, "ExecStop=nft delete table inet mesh_guard") || strings.Contains(stop, "flush") {
t.Errorf("stopping the guard does not delete only its own table: %s", stop)
}
}
func TestAConvergedBundleIsNotMadeAnAdoptedOne(t *testing.T) {
// The converged genesis keeps the foundation's filter, byte for byte (novox/hq ADR 0088).
r := producedBundle(t)
for _, res := range r.Declaration.Resources {
if strings.HasPrefix(res.Identity(), declaration.AdoptionPrefix) {
t.Errorf("a converged bundle carries %s", res.Identity())
}
}
if !r.declares("base-filter-loaded") {
t.Error("a converged bundle lost its filter")
}
}
func TestAnAdoptedGenesisTakesTheMeshsOwnModulesBeforePushingThem(t *testing.T) {
t.Setenv("TMPDIR", t.TempDir())
for _, c := range []struct {
module string
takes bool
}{{RegistryModule, true}, {ControlPlaneModule, true}, {BuilderModule, true}, {"gitea", false}} {
rec := &controlRecorder{settings: map[string]string{}}
control := controlPlane{container: "temp-mesh-controller", run: rec.run, timeout: time.Second}
o := Options{Node: "anchor", Adopted: true, Wait: time.Second}
if _, err := installModule(context.Background(), o, control, c.module, []byte(`{}`), quietly); err != nil {
t.Fatal(err)
}
assign, take, push := rec.index("assign anchor "+c.module), rec.index("take anchor "+c.module), rec.index("push anchor")
if !c.takes {
if take >= 0 {
t.Errorf("%s was taken at genesis", c.module)
}
continue
}
if !(assign >= 0 && assign < take && take < push) {
t.Errorf("%s: assign %d, take %d, push %d: %v", c.module, assign, take, push, rec.told)
}
}
}
func TestAConvergedGenesisTakesNothing(t *testing.T) {
t.Setenv("TMPDIR", t.TempDir())
rec := &controlRecorder{settings: map[string]string{}}
control := controlPlane{container: "temp-mesh-controller", run: rec.run, timeout: time.Second}
if _, err := installModule(context.Background(), Options{Node: "anchor", Wait: time.Second}, control,
RegistryModule, []byte(`{}`), quietly); err != nil {
t.Fatal(err)
}
if rec.index("take") >= 0 {
t.Errorf("a converged genesis took a module: %v", rec.told)
}
}
func TestAnAdoptedGenesisOpensTheRegistryFromAnywhere(t *testing.T) {
t.Setenv("TMPDIR", t.TempDir())
rec := &controlRecorder{settings: map[string]string{}}
control := controlPlane{container: "temp-mesh-controller", run: rec.run, timeout: time.Second}
o := Options{Node: "anchor", Adopted: true, Ports: FoundationPorts{Registry: 5100}, Wait: time.Second}
if _, err := installModule(context.Background(), o, control, RegistryModule, []byte(`{}`), quietly); err != nil {
t.Fatal(err)
}
if got := rec.settings["distribution-settings.json"]; got != `{"expose":{"5000":"anywhere"},"ports":{"5000":5100}}` {
t.Errorf("the registry was told %s", got)
}
}
func TestAnAdoptedGenesisChoosesTheFilterAndLoadsNone(t *testing.T) {
rec := &controlRecorder{settings: map[string]string{}}
control := controlPlane{container: "mesh-controller", run: rec.run, timeout: time.Second}
o := Options{Node: "anchor", Adopted: true, Answers: map[string]string{"packet-filter": "nftables"}}
filter, err := ChooseAndInstallFilter(context.Background(), o, control, quietly)
if err != nil || filter != "nftables" {
t.Fatalf("%q %v", filter, err)
}
if len(rec.told) != 0 {
t.Errorf("an adopted genesis installed a filter: %v", rec.told)
}
}
func TestAnAdoptedNodeIsRecordedAdopted(t *testing.T) {
stop := errors.New("stop here")
runtime := &asked{answer: func(name string, args []string) (string, error) {
joined := strings.Join(args, " ")
switch {
case strings.Contains(joined, "node list"):
return "", nil
case strings.Contains(joined, "node add"):
return "", nil
}
return "", fmt.Errorf("%w: %s %v", stop, name, args)
}}
_, _ = Enrol(context.Background(), Options{
Node: "anchor", Adopted: true, State: t.TempDir() + "/state.json", Timeout: time.Second,
Host: "/usr/local/bin/mesh-host", HostInBackground: true,
}, arch(t), controlPlane{container: "temp-mesh-controller", run: runtime.run, timeout: time.Second},
func(string) {})
if !runtime.ran("node add anchor --adopted") {
t.Errorf("the node was not added adopted: %v", runtime.commands)
}
}
+38 -3
View File
@@ -4,7 +4,9 @@ import (
"context"
"errors"
"fmt"
"path/filepath"
"strings"
"time"
"github.com/novox/mesh-host/internal/apply"
"github.com/novox/mesh-host/internal/declaration"
@@ -32,8 +34,16 @@ type Runner = apply.Runner
// What it does not do is the host's own lifecycle bookkeeping — recording a known-good version,
// clearing the launcher's start counter. Those are facts about a running `mesh-host`, and this is
// not one.
//
// What it does record is that the bundle was consumed, and in which mode the operator raised
// the machine. `raw` is the exact bytes of what is applied — the bundle as rewritten for this
// machine — and its digest is what `mesh-host reconcile` later holds the carried bundle against,
// by digest: a host built from the carried template does not match it, since genesis rewrote the
// ports, root credentials and adoption; a host built from the lock genesis wrote out does.
// Applying the unrewritten template on a raised node recreated the store and loaded the converged
// filter on an adopted one (novox/hq issue 104).
func ApplyBundle(ctx context.Context, o Options, sys system.System, d *declaration.Declaration,
run Runner, say func(string)) (apply.Report, error) {
raw []byte, run Runner, say func(string)) (apply.Report, error) {
// Refuse a shape this host cannot apply before anything is applied, exactly as `mesh-host`
// does: finding out half way through is the half-configured machine tier 0 exists to prevent.
@@ -41,13 +51,38 @@ func ApplyBundle(ctx context.Context, o Options, sys system.System, d *declarati
return apply.Report{}, err
}
// One apply at a time on this machine: a host already running here applies too.
unlock, err := store.Lock(o.State, func() { say(" waiting another apply holds this node's state") })
if err != nil {
return apply.Report{}, err
}
defer unlock()
known, err := store.Load(o.State)
if err != nil {
return apply.Report{}, err
}
report, updated, applyErr := apply.Apply(ctx, sys, d, known, store.OriginCarried, run,
func(line string) { say(" " + strings.TrimPrefix(line, " ")) }, refuseSealed)
// The bundle writes over whatever the machine has at the paths the foundation needs — a
// distribution's own /etc/nftables.conf among them — so it keeps the original of each file it
// has no record of, beside the node's state, exactly as a declaration from the mesh does
// (novox/hq ADR 0100).
report, updated, applyErr := apply.ApplyKeeping(ctx, sys, d, known, store.OriginCarried, run,
func(line string) { say(" " + strings.TrimPrefix(line, " ")) }, refuseSealed,
apply.KeepIn(filepath.Dir(o.State)))
// The bundle is consumed, whichever way the apply went: what is on the machine came from these
// bytes, and the carried ones must not be applied over it. The mode is the operator's word at
// genesis, and only at genesis: the controller records the same and says it in every
// declaration from then on (novox/hq ADR 0100), so a node the mesh has spoken to — this
// installer re-run on it, or finishing a pivot — keeps the mode it has, which may since have
// been flipped.
updated.Genesis = &store.Genesis{Digest: apply.DigestOf(raw), At: time.Now().UTC(), Rewritten: true}
if updated.Mode == "" {
updated.Mode = store.ModeConverged
if o.Adopted {
updated.Mode = store.ModeAdopted
}
}
// Saved whichever way it went, for the reason `mesh-host` gives: what was applied before a
// failure is on the machine either way, and a host that did not record it would believe it
+91
View File
@@ -3,8 +3,16 @@ package bootstrap
import (
"context"
"errors"
"fmt"
"os"
"path/filepath"
"strings"
"testing"
"github.com/novox/mesh-host/internal/apply"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/store"
"github.com/novox/mesh-host/internal/system"
)
// `mesh-host` is built for one operating system and pins it at link time. An installer run by hand
@@ -89,3 +97,86 @@ func TestASealedFileInAFoundationIsRefusedWithAReason(t *testing.T) {
t.Errorf("the refusal does not say why there is no key: %v", err)
}
}
// Defends novox/hq ADR 0100: the carried bundle keeps the original of a file it writes over that
// the host has no record of — the distribution's own ruleset, say.
func TestTheBundleKeepsTheOriginalOfWhatItWritesOver(t *testing.T) {
dir := t.TempDir()
conf := filepath.Join(dir, "nftables.conf")
if err := os.WriteFile(conf, []byte("# the distribution's own\n"), 0o644); err != nil {
t.Fatal(err)
}
d, err := declaration.ParseFileTrusted([]byte(`{"declaration":1,"resources":[
{"id":"base-filter","type":"file","path":"` + conf + `","content":"table inet mesh {}\n"}]}`))
if err != nil {
t.Fatal(err)
}
sys, err := system.For("arch")
if err != nil {
t.Fatal(err)
}
o := Options{State: filepath.Join(dir, "state.json")}
report, err := ApplyBundle(context.Background(), o, sys, d, nil, nil, quietly)
if err != nil {
t.Fatal(err)
}
detail := report.Outcomes[0].Detail
at := strings.Index(detail, "kept at ")
if at < 0 {
t.Fatalf("the bundle wrote over a file it had no record of and kept nothing: %q", detail)
}
if got, err := os.ReadFile(detail[at+len("kept at "):]); err != nil ||
string(got) != "# the distribution's own\n" {
t.Errorf("the kept original is %q (%v)", got, err)
}
}
// Defends novox/hq issue 104: genesis consumes the bundle, recording the digest of what it applied
// and the mode the operator raised the machine in, so the host's own `reconcile` never applies the
// carried bytes over it.
func TestGenesisConsumesTheBundleAndRecordsTheMode(t *testing.T) {
dir := t.TempDir()
raw := []byte(`{"declaration":1,"resources":[
{"id":"a","type":"file","path":"` + filepath.Join(dir, "a.conf") + `","content":"x\n"}]}`)
d, err := declaration.ParseFileTrusted(raw)
if err != nil {
t.Fatal(err)
}
sys, err := system.For("arch")
if err != nil {
t.Fatal(err)
}
for _, adopted := range []bool{false, true} {
o := Options{State: filepath.Join(dir, fmt.Sprintf("state-%v.json", adopted)), Adopted: adopted}
if _, err := ApplyBundle(context.Background(), o, sys, d, raw, nil, quietly); err != nil {
t.Fatal(err)
}
known, err := store.Load(o.State)
if err != nil {
t.Fatal(err)
}
if known.Genesis == nil || known.Genesis.Digest != apply.DigestOf(raw) || !known.Genesis.Rewritten {
t.Errorf("adopted=%v: genesis did not record the bundle it consumed: %+v", adopted, known.Genesis)
}
want := store.ModeConverged
if adopted {
want = store.ModeAdopted
}
if known.Mode != want {
t.Errorf("adopted=%v: genesis recorded the mode as %q, want %q", adopted, known.Mode, want)
}
}
// Re-run on a node the mesh has spoken to since — and converged — genesis leaves the mode
// alone: it is the operator's word at genesis, and the controller's from then on.
o := Options{State: filepath.Join(dir, "state-flipped.json"), Adopted: true}
if err := store.Save(o.State, store.State{Mode: store.ModeConverged}); err != nil {
t.Fatal(err)
}
if _, err := ApplyBundle(context.Background(), o, sys, d, raw, nil, quietly); err != nil {
t.Fatal(err)
}
if known, _ := store.Load(o.State); known.Mode != store.ModeConverged {
t.Errorf("a genesis re-run set the mode back to %q over the mesh's converged", known.Mode)
}
}
+109 -7
View File
@@ -34,6 +34,9 @@ import (
"fmt"
"strings"
"time"
"github.com/novox/mesh-host/internal/firewall"
"github.com/novox/mesh-host/internal/tunnel"
)
// Step names one stage. A failure says which one, because "the bootstrap failed" is a sentence
@@ -137,9 +140,11 @@ type Options struct {
Node string
// Catalogue is a checkout of the mesh's catalogue repository, which is where the registry's and
// the control plane's manifests are read from. Empty stops the installer after the foundation:
// there is no pivot without manifests, and pretending otherwise would leave a machine that
// looks installed and cannot upgrade itself.
// the builder's manifests are read from, and everything phase two installs. Not the control
// plane's: that one comes out of the build at step 3, from the root of its own repository
// (novox/hq ADR 0069). Empty stops the installer after the foundation: there is no pivot
// without manifests, and pretending otherwise would leave a machine that looks installed and
// cannot upgrade itself.
Catalogue string
// Source is where the control plane is built from — a repository on a mesh that already
@@ -183,6 +188,25 @@ type Options struct {
Prompt func(Choice) (string, error)
// Extras are catalogue modules beyond the floor, asked for by name.
Extras []string
// Ports are the ports the foundation binds on this machine (novox/hq ADR 0100). Inputs to
// genesis, each checked free before anything is raised, and then the node's settings for the
// foundation's modules — so adopting the foundation as modules leaves it where it was raised.
// Zero means the catalogue's defaults.
Ports FoundationPorts
// OverlayRange is the private network's address range, checked against every interface and
// route the machine already has. Empty means the mesh's default.
OverlayRange string
// Adopted raises this machine as an adopted node (novox/hq ADR 0100): what is on it is kept
// until each module is taken, its firewall stays in force, and the mesh guards its own ports
// in a table that only refuses. Without it, a machine in use is refused.
Adopted bool
// Tunnel names the found tunnel's interface an adopted hub takes over (novox/hq ADR 0105), when
// more than one is up and the machine cannot say which. Empty finds the one that is up. Once
// found, the tunnel's port is the hub's and its range the private network's; --hub-port and
// --overlay-range may agree with it or be left unsaid.
Tunnel string
}
// pivots reports whether this run goes past the foundation.
@@ -285,6 +309,17 @@ type Result struct {
// Stopped names why a run went no further. Empty on a run that pivoted.
Stopped string `json:"stopped,omitempty"`
// Adopted, the firewall found, and the ports the foundation was raised on (novox/hq ADR 0100).
Adopted bool `json:"adopted,omitempty"`
Firewall string `json:"firewall,omitempty"`
Ports FoundationPorts `json:"ports"`
// Filter is the packet filter chosen for when the node converges; an adopted genesis loads
// none, and the flip assigns this one.
Filter string `json:"filter-on-converge,omitempty"`
// Tunnel is the found tunnel an adopted genesis takes over (novox/hq ADR 0105): what was read
// from it, never its key.
Tunnel *tunnel.Found `json:"tunnel,omitempty"`
}
// Run performs the bootstrap, saying what it is doing as it goes.
@@ -337,7 +372,12 @@ func Run(ctx context.Context, o Options, d Deps, say func(string)) (Result, erro
if say == nil {
say = func(string) {}
}
result := Result{DryRun: o.DryRun}
result := Result{DryRun: o.DryRun, Adopted: o.Adopted}
o.Ports = o.Ports.orDefaults()
result.Ports = o.Ports
if err := o.Ports.Check(); err != nil {
return result, failed(StepPreflight, err)
}
// ---- 1. preflight -------------------------------------------------------------------
say("preflight — what has to be true before anything is changed")
@@ -348,6 +388,40 @@ func Run(ctx context.Context, o Options, d Deps, say func(string)) (Result, erro
// Which half of the host applies things here. Asked of the machine and proved, because
// `mesh-host` pins this at link time and an installer run by hand has no link time.
// An adopted machine keeps the firewall it was found with, so the mesh must speak it; one no
// host speaks is refused here, before anything changes (novox/hq ADR 0100).
if o.Adopted {
kind, name, err := firewall.Detect(ctx, d.Run)
if err != nil {
return result, failed(StepPreflight, err)
}
if kind == firewall.Unsupported {
return result, failed(StepPreflight, fmt.Errorf(
"this machine is filtered by %s, and no host speaks that firewall yet. An adopted "+
"machine keeps its firewall in force, so the mesh could neither open what it needs "+
"through it nor say what it would close. Nothing was changed", name))
}
result.Firewall = string(kind)
say(" adopted what is on this machine is kept; its firewall (" + string(kind) + ") stays in force")
// The tunnel the predecessor left, which the private network takes over (novox/hq ADR
// 0105): its port is the hub's and its range is the mesh's from here on, so both are
// settled before the ports are checked free and the bundle rewritten.
found, err := TakeTheTunnel(&o, d.Run)
if err != nil {
return result, failed(StepPreflight, err)
}
if found != nil {
result.Tunnel = found
result.Ports = o.Ports
say(fmt.Sprintf(" tunnel %s — the private network takes it over: its port %d is "+
"the hub's, its range %s the mesh's, and its %d peer(s) are carried until they enrol",
found.Interface, found.Port, found.Range, len(found.Peers)))
} else {
say(" tunnel none up on this machine; the private network is raised on its own port and range")
}
}
sys, err := WorkOutSystem(ctx, d.Run, o.System)
if err != nil {
return result, failed(StepPreflight, err)
@@ -397,12 +471,32 @@ func Run(ctx context.Context, o Options, d Deps, say func(string)) (Result, erro
if err := RefuseExistingServers(ctx, d.Run, creds); err != nil {
return result, failed(StepBundle, err)
}
// The foundation's ports, its private network's range and its containers' names are checked
// free before anything is raised (novox/hq ADR 0100), each refusal naming what holds it.
if err := CheckTheMachine(ctx, o, d.Run, rewritten.Declaration, say); err != nil {
return result, failed(StepBundle, err)
}
// From here on nothing this installer says contains the values it just made.
say = Masking(say, creds)
root, err := RewriteRoot(&rewritten, creds)
if err != nil {
return result, failed(StepBundle, err)
}
moved, err := RewritePorts(&rewritten, o.Ports, o.OverlayRange)
if err != nil {
return result, failed(StepBundle, err)
}
if moved.Places > 0 {
say(fmt.Sprintf(" ports %d place(s) rewritten to this node's foundation ports", moved.Places))
}
if o.Adopted {
adopted, err := RewriteAdopted(&rewritten, o.Ports)
if err != nil {
return result, failed(StepBundle, err)
}
say(fmt.Sprintf(" adopted bundle the foundation's filter is not loaded (%s); the mesh's guard refuses %v from outside",
strings.Join(adopted.Removed, ", "), adopted.Guarded))
}
for _, c := range []struct {
what, path string
made bool
@@ -498,7 +592,7 @@ func Run(ctx context.Context, o Options, d Deps, say func(string)) (Result, erro
// ---- 4. apply -----------------------------------------------------------------------
say("apply — raising the foundation")
report, err := ApplyBundle(ctx, o, sys, rewritten.Declaration, d.Run, say)
report, err := ApplyBundle(ctx, o, sys, rewritten.Declaration, rewritten.Bundle, d.Run, say)
result.Applied, result.Changed = len(report.Outcomes), report.Changed()
if err != nil {
return result, failed(StepApply, err)
@@ -585,7 +679,7 @@ func Run(ctx context.Context, o Options, d Deps, say func(string)) (Result, erro
// ---- 9. control plane -----------------------------------------------------------------
say("control plane — installed as an ordinary module, pinned to that digest")
permanent, err := InstallControlPlane(ctx, o, d, temporary, rewritten.Declaration,
published.Reference, say)
built, published.Reference, say)
result.Permanent, result.PermanentAnswered = permanent.Container, permanent.Answered
result.StoresDelivered = permanent.Delivered
if err != nil {
@@ -677,7 +771,9 @@ func Run(ctx context.Context, o Options, d Deps, say func(string)) (Result, erro
// ---- 17. filter -----------------------------------------------------------------------
say("filter — required, so the question is which, not whether")
if err := ChooseAndInstallFilter(ctx, o, permanentControl, say); err != nil {
filter, err := ChooseAndInstallFilter(ctx, o, permanentControl, say)
result.Filter = filter
if err != nil {
return result, failed(StepFilter, err)
}
@@ -695,6 +791,12 @@ func Run(ctx context.Context, o Options, d Deps, say func(string)) (Result, erro
return result, failed(StepExport, err)
}
if o.Adopted {
say("\nthis machine is a mesh of one adopted node: it builds its own software, holds its graph " +
"and sits on its private network, and what it ran before is kept as it was, behind the firewall " +
"it was found with. Take each module on it once its data has moved; converge it when done.")
return result, nil
}
say("\nthis machine is a mesh of one node: it builds its own software, holds its graph, " +
"sits on its private network, and filters what modules declared.")
say("what remains is somebody else's: adding nodes, and assigning what they should run.")
+28 -4
View File
@@ -1,6 +1,7 @@
package bootstrap
import (
"bytes"
"context"
"encoding/json"
"errors"
@@ -57,13 +58,19 @@ type Built struct {
// Image is the artifact, named by the digest of its own configuration — the identity a machine
// can use with nothing serving it, and the same one the installer used for a carried image.
Image string
// Manifest is the module as the mesh should hold it: the manifest at the root of the repository
// that was built, its artifact resolved to Image. It is the control plane's ONE manifest
// (novox/hq ADR 0069) — the installer used to read a second copy out of the catalogue, and the
// two drifted apart the first time somebody edited one (novox/hq 04-ISSUES/072).
Manifest []byte
}
// builderOutput is the part of the builder's one-shot result this needs.
type builderOutput struct {
Module string `json:"module"`
Commit string `json:"commit"`
Made []struct {
Module string `json:"module"`
Commit string `json:"commit"`
Manifest json.RawMessage `json:"manifest"`
Made []struct {
Name string `json:"name"`
Kind string `json:"kind"`
Reference string `json:"reference"`
@@ -142,8 +149,25 @@ func BuildControlPlane(ctx context.Context, run Runner, builderTag string, sourc
result.Module, len(images))
}
// The manifest is what the mesh will hold the control plane as, so a result without one is
// a build the installer cannot finish — refused here, beside the builder that said it, rather
// than at step 9 with a message about a missing file.
manifest := bytes.TrimSpace(result.Manifest)
if len(manifest) == 0 || bytes.Equal(manifest, []byte("null")) {
return Built{}, fmt.Errorf(
"%s built, and the builder reported no manifest for it. The installer registers the "+
"control plane with the manifest the build produced — the one at the root of its "+
"repository, its artifact resolved — and has no other copy to use", result.Module)
}
if !bytes.Contains(manifest, []byte(`"`+images[0]+`"`)) {
return Built{}, fmt.Errorf(
"%s built %s, and the manifest the builder reported does not name that image, so "+
"the installer cannot tell which of its resources runs the control plane",
result.Module, images[0])
}
say(fmt.Sprintf(" built %s from %s", result.Module, shortRef(result.Commit)))
return Built{Module: result.Module, Commit: result.Commit, Image: images[0]}, nil
return Built{Module: result.Module, Commit: result.Commit, Image: images[0], Manifest: manifest}, nil
}
func shortRef(ref string) string {
+54
View File
@@ -1,6 +1,7 @@
package bootstrap
import (
"context"
"strings"
"testing"
)
@@ -39,3 +40,56 @@ func TestARepositoryAndACommitIsEnough(t *testing.T) {
t.Fatalf("a repository and a commit were refused: %v", err)
}
}
// **The build hands over the manifest, and the installer registers that one.** The control plane
// used to have two manifests — one at the root of its repository, which the mesh reads whenever
// it rebuilds the control plane from source, and a copy in the catalogue, which genesis read — and
// nothing kept them equal (novox/hq 04-ISSUES/072). The builder already reports the manifest it
// built, artifact resolved to the image; genesis takes it from there and reads no second copy.
func TestTheBuildHandsOverTheManifestTheMeshWillHold(t *testing.T) {
manifest := `{"module":"mesh-controller","version":"1","resources":[` +
`{"id":"server","type":"container","name":"mesh-controller","image":"` + builtImage + `"}]}`
runtime := &asked{answer: func(string, []string) (string, error) {
return `{"module":"mesh-controller","commit":"a1b2c3d4","manifest":` + manifest +
`,"made":[{"name":"server","kind":"image","reference":"` + builtImage + `"}]}` + "\n", nil
}}
built, err := BuildControlPlane(context.Background(), runtime.run, "mesh-builder:test",
Source{Repository: "https://example.invalid/mesh-controller.git", Ref: "a1b2c3d4"}, false, func(string) {})
if err != nil {
t.Fatal(err)
}
if built.Image != builtImage {
t.Errorf("the built image is %q", built.Image)
}
if string(built.Manifest) != manifest {
t.Errorf("the manifest handed over is not the one the builder reported:\n%s", built.Manifest)
}
}
// A result without a manifest is a build the installer cannot finish, and it is refused beside the
// builder that said it rather than at step 9 with a message about a missing file.
func TestABuildReportingNoManifestIsRefused(t *testing.T) {
runtime := &asked{answer: func(string, []string) (string, error) {
return `{"module":"mesh-controller","commit":"a1b2c3d4",` +
`"made":[{"name":"server","kind":"image","reference":"` + builtImage + `"}]}`, nil
}}
_, err := BuildControlPlane(context.Background(), runtime.run, "mesh-builder:test",
Source{Repository: "https://example.invalid/mesh-controller.git", Ref: "a1b2c3d4"}, false, func(string) {})
if err == nil || !strings.Contains(err.Error(), "no manifest") {
t.Fatalf("a build reporting no manifest was accepted, or refused for another reason: %v", err)
}
}
// And a manifest that does not name the image the build produced describes some other build.
func TestABuildWhoseManifestNamesAnotherImageIsRefused(t *testing.T) {
runtime := &asked{answer: func(string, []string) (string, error) {
return `{"module":"mesh-controller","commit":"a1b2c3d4","manifest":{"module":"mesh-controller",` +
`"resources":[{"id":"server","type":"container","image":"sha256:` + strings.Repeat("9", 64) + `"}]},` +
`"made":[{"name":"server","kind":"image","reference":"` + builtImage + `"}]}`, nil
}}
_, err := BuildControlPlane(context.Background(), runtime.run, "mesh-builder:test",
Source{Repository: "https://example.invalid/mesh-controller.git", Ref: "a1b2c3d4"}, false, func(string) {})
if err == nil || !strings.Contains(err.Error(), "does not name that image") {
t.Fatalf("a manifest naming another image was accepted, or refused for another reason: %v", err)
}
}
+1 -1
View File
@@ -53,7 +53,7 @@ func InstallBuilder(ctx context.Context, o Options, d Deps, control controlPlane
"This is the manifest that makes the builder an ordinary module. Without it the mesh "+
"has the image and no way to run it, so nothing can be built here", err)
}
pinned, places, err := pinImage(manifest, published.Reference, BuilderModule)
pinned, places, err := pinPlaceholder(manifest, published.Reference, BuilderModule)
if err != nil {
return out, err
}
+169
View File
@@ -0,0 +1,169 @@
package bootstrap
import (
"context"
"os"
"path/filepath"
"strings"
"testing"
"time"
)
// Defends novox/hq 04-ISSUES/085: the port given for the package registry at genesis is a setting
// of a module and not text in a manifest, so registering that manifest again does not put the
// catalogue's number back.
// theBuildersBinding is the resource the builder carries in place of a binding nothing can resolve
// yet, exactly as the catalogue's manifest has it — settable, with the port as its only default.
const theBuildersBinding = `{
"module": "builder",
"resources": [
{
"id": "package-binding",
"type": "file",
"path": "/var/lib/mesh/builder/package-registry.json",
"mode": "0600",
"merge": "json",
"protected": ["provision", "from", "at", "as"],
"content": "{\"provision\": \"package-registry\", \"from\": \"gitea\", \"at\": \"127.0.0.1\", \"as\": \"mesh-builder\", \"serves\": {\"scheme\": \"http\", \"port\": 3000, \"npm-path\": \"/api/packages/novox/npm/\"}}\n"
}
]
}`
func recording(t *testing.T) (*controlRecorder, controlPlane) {
t.Helper()
t.Setenv("TMPDIR", t.TempDir())
c := &controlRecorder{settings: map[string]string{}}
return c, controlPlane{container: "temp-mesh-controller", run: c.run, timeout: time.Second}
}
func TestTheBuildersPackageRegistryPortIsASettingAndNotTheManifest(t *testing.T) {
c, control := recording(t)
o := Options{Node: "anchor", Ports: FoundationPorts{Packages: 3100}, Wait: time.Second}
if _, err := installModule(context.Background(), o, control, BuilderModule,
[]byte(theBuildersBinding), quietly); err != nil {
t.Fatal(err)
}
if got := c.settings["builder-settings.json"]; got != `{"serves":{"port":3100}}` {
t.Errorf("the builder was told %q about the package registry's port", got)
}
// The manifest reaches the mesh as the catalogue wrote it. A port rewritten into it here is a
// port the next registration from the catalogue silently takes back.
if got := c.settings["builder-module.json"]; got != theBuildersBinding {
t.Errorf("the installer changed the builder's manifest:\n%s", got)
}
set, push := c.index("settings set builder"), c.index("push anchor")
if set < 0 || push < 0 || set > push {
t.Fatalf("the port was not set before the push that raises the builder: %v", c.told)
}
}
func TestTheBuilderOnTheDefaultPackagesPortIsToldNothing(t *testing.T) {
c, control := recording(t)
o := Options{Node: "anchor", Wait: time.Second}
if _, err := installModule(context.Background(), o, control, BuilderModule,
[]byte(theBuildersBinding), quietly); err != nil {
t.Fatal(err)
}
if c.index("settings") >= 0 {
t.Errorf("a genesis on the catalogue's packages port set a setting: %v", c.told)
}
}
// aCatalogueWith writes a checkout holding one module's manifest, which is all the installer reads
// a catalogue for.
func aCatalogueWith(t *testing.T, module, manifest string) string {
t.Helper()
dir := t.TempDir()
at := filepath.Join(dir, catalogueDir, module)
if err := os.MkdirAll(at, 0o700); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(at, "module.json"), []byte(manifest), 0o600); err != nil {
t.Fatal(err)
}
return dir
}
const theForgesManifest = `{"module": "gitea", "version": "1"}`
func TestThePackagesPortIsTheForgeModulesSettingOnThisNode(t *testing.T) {
c, control := recording(t)
o := Options{Node: "anchor", Catalogue: aCatalogueWith(t, ForgeModule, theForgesManifest),
Ports: FoundationPorts{Packages: 3100}, Wait: time.Second}
if err := recordTheForgesPort(context.Background(), o, control, quietly); err != nil {
t.Fatal(err)
}
if got := c.settings["gitea-settings.json"]; got != `{"ports":{"3000":3100}}` {
t.Errorf("the forge module was told %q about its port", got)
}
// Registered so there is something to hold the setting against, and never assigned: the forge
// module cannot run until the base it stands on has been built.
add, set := c.index("module add"), c.index("settings set gitea")
if add < 0 || set < 0 || add > set {
t.Fatalf("the forge's setting was recorded against a module the mesh does not know: %v", c.told)
}
if c.index("assign") >= 0 {
t.Errorf("genesis assigned the forge module: %v", c.told)
}
if !strings.Contains(c.told[set], "--node anchor") {
t.Errorf("the forge's port was set for the whole mesh, not this machine: %s", c.told[set])
}
}
// knowingControl is a control plane that already has the forge in its catalogue, as a mesh that
// has been running has.
type knowingControl struct{ controlRecorder }
func (k *knowingControl) run(ctx context.Context, name string, args ...string) (string, error) {
out, err := k.controlRecorder.run(ctx, name, args...)
if name == "docker" && args[0] == "exec" && len(args) > 4 &&
args[3] == "module" && args[4] == "list" {
return ForgeModule + " 1 assigned to anchor\n", nil
}
return out, err
}
func TestARerunDoesNotReplaceAForgeTheMeshAlreadyHas(t *testing.T) {
t.Setenv("TMPDIR", t.TempDir())
k := &knowingControl{controlRecorder{settings: map[string]string{}}}
control := controlPlane{container: "mesh-controller", run: k.run, timeout: time.Second}
// An older checkout than the mesh is running, which is what makes an overwrite here damage.
o := Options{Node: "anchor", Catalogue: aCatalogueWith(t, ForgeModule, `{"module": "gitea"}`),
Ports: FoundationPorts{Packages: 3100}, Wait: time.Second}
if err := recordTheForgesPort(context.Background(), o, control, quietly); err != nil {
t.Fatal(err)
}
// Registering is an overwrite, and the manifest of a forge that is built and assigned is not
// this installer's to replace with whatever checkout it was pointed at.
if k.index("module add") >= 0 {
t.Errorf("a re-run replaced the registered forge's manifest: %v", k.told)
}
// The port is still recorded: a settings row needs the module row, and it is already there.
if got := k.settings["gitea-settings.json"]; got != `{"ports":{"3000":3100}}` {
t.Errorf("the forge's port was not recorded on a mesh that knows it: %q", got)
}
}
func TestAGenesisOnTheCataloguesPackagesPortRegistersNoForge(t *testing.T) {
c, control := recording(t)
o := Options{Node: "anchor", Catalogue: aCatalogueWith(t, ForgeModule, theForgesManifest),
Wait: time.Second}
if err := recordTheForgesPort(context.Background(), o, control, quietly); err != nil {
t.Fatal(err)
}
if len(c.told) != 0 {
t.Errorf("a genesis on the defaults told the mesh something new: %v", c.told)
}
}
func TestAForgeManifestTheCatalogueDoesNotHaveIsNamed(t *testing.T) {
_, control := recording(t)
o := Options{Node: "anchor", Catalogue: t.TempDir(),
Ports: FoundationPorts{Packages: 3100}, Wait: time.Second}
err := recordTheForgesPort(context.Background(), o, control, quietly)
if err == nil || !strings.Contains(err.Error(), ForgeModule) ||
!strings.Contains(err.Error(), "package registry") {
t.Errorf("a missing forge manifest was not explained: %v", err)
}
}
+56 -56
View File
@@ -53,26 +53,33 @@ type Permanent struct {
// exactly the path for a value the mesh must carry and could not have invented — and they are read
// out of the bundle this installer produced rather than reconstructed, because the bundle is what
// created them and a second opinion about what a DSN should say is a second chance to be wrong.
//
// **The manifest is the one the build produced** — the manifest at the root of the control plane's
// own repository, its artifact resolved to the image built at step 3 (novox/hq ADR 0069). The
// installer used to read a second copy out of the catalogue and pin its placeholder; the copies
// drifted, and the first rebuild from source replaced the mesh's record with the repository's shape
// while every later push was refused on its behalf (novox/hq 04-ISSUES/072). There is one manifest
// now, and the only thing this step changes in it is the image's name: the id the machine built it
// under becomes the reference the registry assigned at step 8.
func InstallControlPlane(ctx context.Context, o Options, d Deps, control controlPlane,
foundation *declaration.Declaration, image string, say func(string)) (Permanent, error) {
foundation *declaration.Declaration, built Built, image string, say func(string)) (Permanent, error) {
out := Permanent{Image: image}
manifest, err := readManifest(o.Catalogue, ControlPlaneModule)
if err != nil {
if len(built.Manifest) == 0 {
return out, fmt.Errorf(
"%w\n"+
"This is the manifest that makes the control plane an ordinary module. Without it "+
"the machine keeps the temporary control plane the foundation raised, which works "+
"and cannot be upgraded — so the install stops here rather than pretending to "+
"have pivoted", err)
"the control plane was not built, so there is no manifest to register it with. " +
"This is the manifest that makes the control plane an ordinary module. Without it " +
"the machine keeps the temporary control plane the foundation raised, which works " +
"and cannot be upgraded — so the install stops here rather than pretending to " +
"have pivoted")
}
pinned, places, err := pinImage(manifest, image, ControlPlaneModule)
pinned, places, err := pinImage(built.Manifest, built.Image, image, ControlPlaneModule)
if err != nil {
return out, err
}
say(fmt.Sprintf(" pinned %s, in %d place(s)", image, places))
say(fmt.Sprintf(" pinned %s → %s, in %d place(s)", shortImage(built.Image), image, places))
container, _, err := containerIn(pinned, controlPlaneResourceIn(pinned))
if err != nil {
@@ -118,52 +125,34 @@ func InstallControlPlane(ctx context.Context, o Options, d Deps, control control
return out, nil
}
// pinImage replaces the catalogue's placeholder digest with what the registry assigned.
// pinImage replaces the image the build named with the reference the registry assigned.
//
// **Textual, and every place it appears.** A manifest may name its image in more than one resource
// — the catalogue's converted modules routinely carry a runtime container beside the application's
// — and the same reasoning as the bundle rewrite applies: replacing one and not the others leaves
// something pointing at an image nothing serves, and it fails half way through an apply rather
// than here.
// — a migrate step beside the server, a runtime beside the application — and the same reasoning
// as the bundle rewrite applies: replacing one and not the others leaves something pointing at an
// image nothing serves, and it fails half way through an apply rather than here.
//
// It refuses a manifest with no placeholder in it. That is not pedantry: a manifest already
// carrying a real digest is one somebody pinned by hand, and quietly registering it would install a
// control plane that is not the image this machine just published — which is the one thing this
// step exists to guarantee.
func pinImage(manifest []byte, reference, module string) ([]byte, int, error) {
places := bytes.Count(manifest, []byte(placeholderDigest))
// The built image is named by the digest of its own configuration, `sha256:…` with no registry in
// front, which only the machine that built it can resolve. The whole JSON string moves to the
// registry's `<registry>/<repository>@sha256:…`, so every machine the module is later pushed to
// pulls it from the mesh's own store.
//
// It refuses a manifest that does not name the built image. That is not pedantry: a manifest
// naming some other image is one that describes some other build, and quietly registering it would
// install a control plane that is not the image this machine just published — which is the one
// thing this step exists to guarantee.
func pinImage(manifest []byte, built, reference, module string) ([]byte, int, error) {
from := []byte(`"` + built + `"`)
places := bytes.Count(manifest, from)
if places == 0 {
return nil, 0, fmt.Errorf(
"the %s module's manifest carries no placeholder digest (%s), so there is nothing to "+
"pin to the image this machine just published.\n"+
"A manifest already naming a digest was pinned by somebody else, to some other "+
"build. Registering it would install a module that is not the one this "+
"installer carried and pushed", module, placeholderDigest)
"the %s module's manifest does not name the image this machine built (%s), so there "+
"is nothing to pin to the image it just published.\n"+
"A manifest naming some other image describes some other build. Registering it "+
"would install a module that is not the one this installer built and pushed",
module, built)
}
// The reference the registry gave back is `<registry>/<repository>@sha256:…`, and what the
// manifest holds is `<something>@sha256:0…0`. Replacing only the digest would leave the
// manifest's own repository name in front of it — which may be `mesh-controller` with no
// registry, and a runtime would then pull it from the internet. The whole reference moves.
var out bytes.Buffer
rest := manifest
for {
at := bytes.Index(rest, []byte(placeholderDigest))
if at < 0 {
out.Write(rest)
break
}
// Back up over the repository this digest belongs to, which runs to the opening quote.
start := bytes.LastIndexByte(rest[:at], '"')
if start < 0 {
return nil, 0, fmt.Errorf(
"the %s module's manifest has a placeholder digest that is not inside a JSON "+
"string, so the installer cannot tell what image it belongs to", module)
}
out.Write(rest[:start+1])
out.WriteString(reference)
rest = rest[at+len(placeholderDigest):]
}
pinned := out.Bytes()
pinned := bytes.ReplaceAll(manifest, from, []byte(`"`+reference+`"`))
// Read back. A substitution on text can catch more than it was aimed at, and the manifest is
// about to be handed to the mesh as the description of what it runs.
@@ -172,17 +161,16 @@ func pinImage(manifest []byte, reference, module string) ([]byte, int, error) {
return nil, 0, fmt.Errorf(
"pinning the %s module's image broke its manifest: %w", module, err)
}
if bytes.Contains(pinned, []byte(placeholderDigest)) {
if bytes.Contains(pinned, from) {
return nil, 0, fmt.Errorf(
"the %s module's manifest still carries a placeholder digest after pinning",
module)
"the %s module's manifest still names the built image after pinning", module)
}
return pinned, places, nil
}
// controlPlaneResourceIn is the id of the resource that runs the control plane.
//
// The manifest is written by the catalogue and the installer does not get to name its resources.
// The manifest is the control plane's own and the installer does not get to name its resources.
// What it can do is find the one container whose image is the one just pinned — and when a manifest
// declares exactly one container, that is the answer without any searching at all.
func controlPlaneResourceIn(manifest []byte) string {
@@ -334,8 +322,10 @@ func secretsByVariableIn(manifest []byte) (map[string]string, error) {
case "container":
inside := mountedFrom(r.Volumes)
for key, path := range r.Env {
if !strings.HasPrefix(key, storeVariablePrefix) ||
!strings.HasSuffix(key, storeFileSuffix) {
// Any `MESH_…_FILE` naming an own-secret's file, not only the store's: the broker
// settings took the same shape once a secret stopped travelling in an env-file
// (novox/hq ADR 0086, issue 041).
if !strings.HasPrefix(key, "MESH_") || !strings.HasSuffix(key, storeFileSuffix) {
continue
}
on := path
@@ -408,3 +398,13 @@ func sortedKeys(m map[string]string) []string {
sort.Strings(keys)
return keys
}
// shortImage is an image id as a person reads one: the first twelve hex digits, without the
// algorithm in front — `shortRef` would keep the prefix and show one digit of the digest.
func shortImage(id string) string {
digest := strings.TrimPrefix(id, "sha256:")
if len(digest) > 12 {
return digest[:12]
}
return digest
}
+88 -36
View File
@@ -11,13 +11,15 @@ import (
// that could go wrong quietly: pinning it to the wrong image, and delivering it store connections
// the mesh invented rather than the ones the foundation actually made.
// theControlPlaneModule is the catalogue's manifest, trimmed to what this installer reads.
// theControlPlaneModule is the manifest the build produces at step 3: the control plane's own,
// from the root of its repository, its artifact resolved to the image the machine built — named by
// the digest of its own configuration, with no registry in front (novox/hq ADR 0069).
//
// A fixture rather than the file itself, unlike the foundation example the rewrite tests use: the
// catalogue is a different repository on a different branch, and a test that read it would pass or
// fail according to what somebody else had checked out. What it must stay faithful to is the
// SHAPE — the placeholder digest, the own-secret per context, the mount from the machine's path to
// the container's, and the environment file that fills what is not a path.
// control plane is a different repository on a different branch, and a test that read it would
// pass or fail according to what somebody else had checked out. What it must stay faithful to is
// the SHAPE — the built image's bare id, the own-secret per context, the mount from the machine's
// path to the container's, and the environment file that fills what is not a path.
const theControlPlaneModule = `{
"module": "mesh-controller",
"version": "1",
@@ -37,7 +39,7 @@ const theControlPlaneModule = `{
"mode": "0600",
"content": "MESH_BROKER_AMQP=${secret:broker}\nMESH_BROKER_MANAGEMENT=${secret:broker-management}\nMESH_BROKER_ADDRESS=${machine:at}:5671\n"},
{"id": "server", "type": "container", "name": "mesh-controller",
"image": "mesh-controller@` + placeholderDigest + `",
"image": "` + builtImage + `",
"network": "host", "args": ["serve"],
"env-file": ["/var/lib/mesh/mesh-controller/broker.env"],
"env": {
@@ -58,57 +60,62 @@ const theControlPlaneModule = `{
const pushedReference = "127.0.0.1:5000/mesh-controller@sha256:" +
"eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee"
// **The whole reference moves, not only the digest.** The manifest's placeholder names a
// repository too, and replacing sixty-four zeros inside it would leave `mesh-controller@sha256:…`
// with no registry in front — which a runtime would go to the internet for, and this mesh's
// control plane exists in no public registry by design.
// builtImage is what step 3 built, as the machine that built it names it.
const builtImage = "sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"
// theBuild is what step 3 hands step 9.
func theBuild(manifest string) Built {
return Built{Module: "mesh-controller", Commit: "a1b2c3d4", Image: builtImage, Manifest: []byte(manifest)}
}
// **The whole reference moves.** The build names its image by the bare digest of its configuration,
// which only the machine that built it can resolve; the mesh's record must name what the registry
// assigned, `<registry>/<repository>@sha256:…`, or every other machine the module is pushed to
// would go looking for an image nothing serves.
func TestTheControlPlaneIsPinnedToWhatThisMeshsRegistryAssigned(t *testing.T) {
pinned, places, err := pinImage([]byte(theControlPlaneModule), pushedReference, "mesh-controller")
pinned, places, err := pinImage([]byte(theControlPlaneModule), builtImage, pushedReference, "mesh-controller")
if err != nil {
t.Fatal(err)
}
if places != 1 {
t.Errorf("the placeholder was found in %d place(s)", places)
t.Errorf("the built image was found in %d place(s)", places)
}
if !strings.Contains(string(pinned), `"image": "`+pushedReference+`"`) {
t.Errorf("the manifest does not name the pushed image:\n%s", pinned)
}
if strings.Contains(string(pinned), `"mesh-controller@sha256:`) {
t.Errorf("the digest was replaced and the manifest's own repository name was left in "+
"front of it, so nothing says which registry serves it:\n%s", pinned)
if strings.Contains(string(pinned), builtImage) {
t.Errorf("the built image's bare id survived, which no other machine can resolve:\n%s", pinned)
}
}
// A manifest already naming a real digest was pinned by somebody else, to some other build.
// Registering it would install a control plane that is not the image this machine just published,
// which is the one thing this step exists to guarantee.
func TestAManifestAlreadyPinnedByHandIsRefused(t *testing.T) {
already := strings.Replace(theControlPlaneModule, placeholderDigest,
"sha256:"+strings.Repeat("9", 64), 1)
if _, _, err := pinImage([]byte(already), pushedReference, "mesh-controller"); err == nil {
t.Fatal("a manifest already pinned to some other image was accepted")
// A manifest naming some other image describes some other build. Registering it would install a
// control plane that is not the image this machine just published, which is the one thing this
// step exists to guarantee.
func TestAManifestNamingAnotherBuildIsRefused(t *testing.T) {
other := strings.Replace(theControlPlaneModule, builtImage, "sha256:"+strings.Repeat("9", 64), 1)
if _, _, err := pinImage([]byte(other), builtImage, pushedReference, "mesh-controller"); err == nil {
t.Fatal("a manifest naming some other image was accepted")
}
}
// Every placeholder moves. A manifest naming its image in a second resource — a runtime container
// beside the application's, which the catalogue's converted modules routinely carry — would
// otherwise be left half pinned, and fail inside an apply rather than here.
// Every place moves. A manifest naming its image in a second resource — a migrate step beside the
// server — would otherwise be left half pinned, and fail inside an apply rather than here.
func TestEveryPlaceTheManifestNamesTheImageIsPinned(t *testing.T) {
twice := strings.Replace(theControlPlaneModule,
`{"id": "mesh-state", "type": "directory", "path": "/var/lib/mesh/mesh-controller", "mode": "0700"},`,
`{"id": "mesh-state", "type": "directory", "path": "/var/lib/mesh/mesh-controller", "mode": "0700"},
{"id": "migrate", "type": "container", "name": "mesh-controller-migrate", "run-once": true,
"image": "mesh-controller@`+placeholderDigest+`", "args": ["migrate"]},`, 1)
"image": "`+builtImage+`", "args": ["migrate"]},`, 1)
pinned, places, err := pinImage([]byte(twice), pushedReference, "mesh-controller")
pinned, places, err := pinImage([]byte(twice), builtImage, pushedReference, "mesh-controller")
if err != nil {
t.Fatal(err)
}
if places != 2 {
t.Errorf("the placeholder was found in %d place(s), and the manifest names it twice", places)
t.Errorf("the built image was found in %d place(s), and the manifest names it twice", places)
}
if strings.Contains(string(pinned), placeholderDigest) {
t.Error("a placeholder survived the pinning")
if strings.Contains(string(pinned), builtImage) {
t.Error("the built image's id survived the pinning")
}
}
@@ -230,7 +237,7 @@ func TestAManifestWantingNoStoresIsRefusedWithTheShapeItShouldHave(t *testing.T)
bare := `{"module":"mesh-controller","version":"1","resources":[
{"id":"container","type":"container","name":"mesh-controller",
"image":"mesh-controller@` + placeholderDigest + `"}]}`
"image":"` + builtImage + `"}]}`
_, err = deliverStores(context.Background(), Options{Node: "anchor"}, control,
[]byte(bare), rewritten.Declaration, func(string) {})
@@ -247,7 +254,7 @@ func TestAManifestWantingNoStoresIsRefusedWithTheShapeItShouldHave(t *testing.T)
// a reply proves the sealed connections it was given are the ones the foundation made.
func TestThePermanentControlPlaneIsAskedTheSameQuestion(t *testing.T) {
runtime := &asked{answer: aMeshThatAgrees(map[string]string{
"module list": "",
"module list": "",
"exec mesh-controller /mesh-controller": "1 node, 0 waiting\n",
})}
control := controlPlane{container: "temp-mesh-controller", run: runtime.run, timeout: time.Second}
@@ -256,9 +263,11 @@ func TestThePermanentControlPlaneIsAskedTheSameQuestion(t *testing.T) {
t.Fatal(err)
}
out, err := InstallControlPlane(context.Background(),
installing(t, catalogueWith(t, ControlPlaneModule, theControlPlaneModule)),
Deps{Run: runtime.run}, control, rewritten.Declaration, pushedReference, func(string) {})
// No catalogue: the control plane's manifest is the one the build produced (novox/hq
// 04-ISSUES/072), and step 9 reads nothing from the catalogue any more.
out, err := InstallControlPlane(context.Background(), installing(t, t.TempDir()),
Deps{Run: runtime.run}, control, rewritten.Declaration, theBuild(theControlPlaneModule),
pushedReference, func(string) {})
if err != nil {
t.Fatal(err)
}
@@ -273,3 +282,46 @@ func TestThePermanentControlPlaneIsAskedTheSameQuestion(t *testing.T) {
t.Errorf("the module was never registered: %v", runtime.commands)
}
}
// The broker settings take the file shape too (novox/hq ADR 0086): a `MESH_BROKER_…_FILE` pointing
// at a mounted own-secret is delivered as that secret, exactly as a store connection is.
func TestABrokerSettingReadFromAFileIsDeliveredToo(t *testing.T) {
manifest := `{
"module": "mesh-controller", "version": "1",
"own-secrets": {"broker": "/var/lib/mesh/mesh-controller/broker", "inventory": "/var/lib/mesh/mesh-controller/inventory"},
"resources": [{
"id": "server", "type": "container", "name": "mesh-controller", "image": "x@sha256:0",
"volumes": ["/var/lib/mesh/mesh-controller/broker:/run/secrets/broker:ro",
"/var/lib/mesh/mesh-controller/inventory:/run/secrets/inventory:ro"],
"env": {"MESH_BROKER_AMQP_FILE": "/run/secrets/broker", "MESH_STORE_INVENTORY_FILE": "/run/secrets/inventory",
"MESH_BROKER_CERTIFICATE": "/broker-tls/tls.crt"}
}]}`
wanted, err := secretsByVariableIn([]byte(manifest))
if err != nil {
t.Fatal(err)
}
if wanted["MESH_BROKER_AMQP"] != "broker" || wanted["MESH_STORE_INVENTORY"] != "inventory" {
t.Fatalf("wanted %v", wanted)
}
if _, has := wanted["MESH_BROKER_CERTIFICATE"]; has {
t.Fatal("a plain path variable was taken for a secret")
}
}
// Step 9 with nothing built is a caller's fault, and it stops rather than pretending to pivot.
func TestTheControlPlaneCannotBeInstalledWithoutABuild(t *testing.T) {
runtime := &asked{answer: aMeshThatAgrees(map[string]string{})}
control := controlPlane{container: "temp-mesh-controller", run: runtime.run, timeout: time.Second}
rewritten, err := Rewrite(theRealBundle(t), held)
if err != nil {
t.Fatal(err)
}
_, err = InstallControlPlane(context.Background(), installing(t, t.TempDir()),
Deps{Run: runtime.run}, control, rewritten.Declaration, Built{}, pushedReference, func(string) {})
if err == nil || !strings.Contains(err.Error(), "was not built") {
t.Fatalf("a missing build was not refused: %v", err)
}
if runtime.ran("module add") {
t.Error("something was registered without a manifest")
}
}
+16 -3
View File
@@ -72,7 +72,13 @@ func Enrol(ctx context.Context, o Options, sys system.System, control controlPla
if mentions(nodes, o.Node) {
say(" already a node " + o.Node)
} else {
if _, err := control.tell(ctx, "node", "add", o.Node); err != nil {
add := []string{"node", "add", o.Node}
if o.Adopted {
// The controller records the node's mode; an adopted one keeps what it was found with
// until each module is taken (novox/hq ADR 0100).
add = append(add, "--adopted")
}
if _, err := control.tell(ctx, add...); err != nil {
return out, err
}
out.Added = true
@@ -106,12 +112,19 @@ func Enrol(ctx context.Context, o Options, sys system.System, control controlPla
return out, err
}
joining, cancel := context.WithTimeout(ctx, o.Wait)
joined, err := control.run(joining, o.Host, "enrol", "--token", token, "--state", o.State)
args := []string{"enrol", "--token", token, "--state", o.State}
if o.Tunnel != "" {
// The found tunnel's key becomes this node's overlay key, and the tunnel travels with
// the enrolment (novox/hq ADR 0105). Named, so the host takes the one genesis settled
// its ports and range on and not another that came up since.
args = append(args, "--tunnel", o.Tunnel)
}
joined, err := control.run(joining, o.Host, args...)
cancel()
if err != nil {
return out, fmt.Errorf(
"%s would not enrol this machine: %w\n%s\n"+
"The token is one-time and has now been spent; running this again issues "+
"The token is one-time and may have been spent; running this again issues "+
"another, so a re-run is safe", o.Host, err, indent(strings.TrimSpace(joined)))
}
if !strings.Contains(joined, "enrolled as "+o.Node) {
+79
View File
@@ -0,0 +1,79 @@
package bootstrap
import (
"context"
"encoding/json"
"os"
"path/filepath"
"strings"
"testing"
"time"
)
// What genesis raises, it raises as the module that succeeds it declares — name, data directory
// and image — so the module adopts it by the found rule that already exists (novox/hq ADR 0163,
// rule 7; issue 090). The network is the one difference left: the bootstrap forge runs on the
// machine's network to reach the store on its loopback, and a take says so.
func TestGenesisRaisesTheForgeAsTheModuleDeclaresIt(t *testing.T) {
var ran [][]string
run := func(_ context.Context, name string, args ...string) (string, error) {
if name == "docker" && args[0] == "container" {
return "", nil // not raised yet
}
ran = append(ran, append([]string{name}, args...))
return "", nil
}
if err := raiseGiteaServer(context.Background(), run, time.Second, "pw", DefaultPorts(), quietly); err != nil {
t.Fatal(err)
}
var raised []string
for _, r := range ran {
if r[0] == "docker" && r[1] == "run" {
raised = r
}
}
line := strings.Join(raised, " ")
for _, want := range []string{"--name gitea ", "--volume " + giteaDataDir + ":/data", " " + giteaImage} {
if !strings.Contains(line+" ", want) {
t.Errorf("the forge is not raised with %q: %s", want, line)
}
}
if giteaBootstrap != ForgeModule {
t.Errorf("the bootstrap forge is %q and the module names its container %q", giteaBootstrap, ForgeModule)
}
// Against the module's own manifest, where the catalogue is checked out beside this repository.
var manifest []byte
for _, candidate := range []string{"../../../mesh-catalog/modules/gitea/module.json", "../../../../../mesh-catalog/modules/gitea/module.json"} {
if raw, err := os.ReadFile(filepath.Clean(candidate)); err == nil {
manifest = raw
break
}
}
if manifest == nil {
t.Skip("the catalogue is not beside this checkout; the module's pin is not compared")
}
var m struct {
Resources []struct {
ID, Type, Name, Image string
Volumes []string
} `json:"resources"`
}
if err := json.Unmarshal(manifest, &m); err != nil {
t.Fatal(err)
}
for _, r := range m.Resources {
if r.Type != "container" || r.ID != "server" {
continue
}
if r.Name != giteaBootstrap {
t.Errorf("the module names its container %q; genesis raises %q", r.Name, giteaBootstrap)
}
if r.Image != giteaImage {
t.Errorf("the module pins %s; genesis raises %s — the two must move together", r.Image, giteaImage)
}
if len(r.Volumes) != 1 || !strings.HasSuffix(r.Volumes[0], ":/data") {
t.Errorf("the module mounts %v; genesis mounts %s:/data", r.Volumes, giteaDataDir)
}
}
}
+146
View File
@@ -0,0 +1,146 @@
package bootstrap
import (
"context"
"fmt"
"net"
"strings"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/reachable"
"github.com/novox/mesh-host/internal/store"
)
// quiet are the processes every fresh machine runs that serve nobody: name resolution (whose
// link-local resolver listens on TCP as well as UDP, on every address) and the network manager's
// address configuration. ss names a process by its first fifteen characters, so both spellings are
// here.
//
// **Only what the measurement found** (novox/hq ADR 0101): these two hold every listener on a
// freshly installed lab machine (testdata/fresh-machine-listeners.txt) and nothing else does. A
// daemon joins this list with a measurement of a fresh machine that holds it, never by guess — a
// time client or an address-configuration client listening on a machine that does not run one as
// standard is something somebody installed, and that is a machine in use.
var quiet = map[string]bool{
"systemd-resolved": true, "systemd-resolve": true,
"systemd-networkd": true, "systemd-network": true,
}
// InUse says what makes this machine a machine in use (novox/hq ADR 0100): every running container
// no host made, and every socket listening on an address other than loopback that is neither ssh's
// nor held by what every fresh machine runs. ours names
// what the mesh itself runs, which a re-run of genesis finds and does not count.
func InUse(ctx context.Context, run Runner, ours func(name string) bool) ([]string, []reachable.Reach, error) {
var containers []string
out, err := run(ctx, "docker", "ps", "--format", "{{.Names}}\t{{.Label \"mesh-host.spec\"}}")
if err != nil {
return nil, nil, fmt.Errorf("cannot ask the container runtime what is running here: %w", err)
}
for _, line := range strings.Split(out, "\n") {
name, label, _ := strings.Cut(strings.TrimSpace(line), "\t")
label = strings.TrimSpace(label)
if name == "" || (label != "" && label != "<no value>") || ours(name) {
continue
}
containers = append(containers, name)
}
listening, err := run(ctx, "ss", "-Hltunp")
if err != nil {
return nil, nil, fmt.Errorf("cannot read what listens on this machine: %w", err)
}
var listeners []reachable.Reach
for _, r := range reachable.Sockets(listening) {
if counts(r) && !ours(r.By) {
listeners = append(listeners, r)
}
}
return containers, listeners, nil
}
func counts(r reachable.Reach) bool {
if ip := net.ParseIP(r.Address); ip != nil && ip.IsLoopback() {
return false
}
switch r.Protocol {
case "tcp":
return r.By != "sshd" && !(r.By == "" && r.Port == 22) && !quiet[r.By]
case "udp":
return !quiet[r.By]
}
return false
}
// RefuseAMachineInUse is the check a converged genesis makes before changing anything: a machine
// in use is refused, naming every container and listener counted, because raising the foundation's
// filter there would close what it serves — a forgotten --adopted must not close a working machine.
// An adopted genesis is told what it found, and goes on.
func RefuseAMachineInUse(ctx context.Context, o Options, run Runner, say func(string)) error {
known, err := store.Load(o.State)
if err != nil {
return err
}
if len(known.Resources) > 0 {
// **The machine says how it was raised** (novox/hq ADR 0103). A re-run must not change
// the node's mode by a flag forgotten or added: without --adopted the bundle would load
// the foundation's dropping filter over the found firewall, and with it on a converged
// machine the filter the node relies on would be removed as no longer carried.
switch adopted := RecordsAdoption(known); {
case adopted && !o.Adopted:
return fmt.Errorf("this machine was raised adopted, and genesis was run again without --adopted. " +
"Run it again the way it was raised: pass --adopted. Returning it to converged is the " +
"controller's act (converge), never genesis's; nothing was changed")
case !adopted && o.Adopted:
return fmt.Errorf("this machine was raised converged, and genesis was run again with --adopted, " +
"which would remove the foundation's filter it relies on. Run it again without --adopted; " +
"returning a node to adopted is the controller's act (adopt); nothing was changed")
}
// What genesis raised on an earlier run is the mesh's, and it is what the machine now
// serves; the question was answered the first time.
say(" in use not asked: this machine carries what an earlier genesis raised")
return nil
}
containers, listeners, err := InUse(ctx, run, func(string) bool { return false })
if err != nil {
return err
}
if len(containers) == 0 && len(listeners) == 0 {
say(" in use no: no container runs and nothing listens beyond ssh")
return nil
}
var named []string
for _, c := range containers {
named = append(named, "container "+c)
}
for _, l := range listeners {
by := l.By
if by == "" {
by = "an unnamed process"
}
named = append(named, fmt.Sprintf("%s %s:%d by %s", l.Protocol, l.Address, l.Port, by))
}
if o.Adopted {
say(fmt.Sprintf(" in use yes, and adopted: %d thing(s) found are kept", len(named)))
return nil
}
return fmt.Errorf("this machine is in use, and a converged genesis would close what it serves:\n - %s\n"+
"If it is meant to join the mesh keeping what it runs, pass --adopted: its firewall stays in "+
"force and every module is taken on it one at a time. Nothing was changed",
strings.Join(named, "\n - "))
}
// RecordsAdoption is whether this machine's state says it is an adopted node: it holds something
// of the mesh's that only an adopted node has — the guard or an opening, under the adoption prefix
// — or something it found and holds. A node the controller converged has neither any more; the
// record of the firewall it found outlives the flip, so it is not read as the mode.
func RecordsAdoption(known store.State) bool {
if len(known.Held) > 0 {
return true
}
for _, r := range known.Resources {
if strings.HasPrefix(r.ID, declaration.AdoptionPrefix) {
return true
}
}
return false
}
+179
View File
@@ -0,0 +1,179 @@
package bootstrap
import (
"context"
"os"
"path/filepath"
"strings"
"testing"
"github.com/novox/mesh-host/internal/store"
)
// Defends novox/hq ADR 0100: a converged genesis refuses a machine in use, naming every container
// and listener it counted.
// Lines as `ss -Hltunp` prints them. The ssh, samba, loopback and proxy lines are captured from a
// real machine; the resolver and network-manager lines are written in the same shape. What a fresh
// machine actually runs is measured in testdata/fresh-machine-listeners.txt.
const inUseSockets = `tcp LISTEN 0 128 0.0.0.0:22 0.0.0.0:* users:(("sshd",pid=1188536,fd=6))
tcp LISTEN 0 128 [::]:22 [::]:* users:(("sshd",pid=1188536,fd=7))
tcp LISTEN 0 32 127.0.0.1:53 0.0.0.0:* users:(("dnsmasq",pid=1189392,fd=7))
tcp LISTEN 0 4096 127.0.0.1:5432 0.0.0.0:* users:(("docker-proxy",pid=1854543,fd=7))
udp UNCONN 0 0 0.0.0.0:5355 0.0.0.0:* users:(("systemd-resolve",pid=301,fd=11))
udp UNCONN 0 0 192.0.2.10%eth0:68 0.0.0.0:* users:(("systemd-network",pid=280,fd=19))
`
const servingSockets = `tcp LISTEN 0 50 0.0.0.0:445 0.0.0.0:* users:(("smbd",pid=1248,fd=29))
tcp LISTEN 0 4096 0.0.0.0:8080 0.0.0.0:* users:(("docker-proxy",pid=1920035,fd=7))
udp UNCONN 0 0 0.0.0.0:123 0.0.0.0:* users:(("ntpd",pid=1070791,fd=17))
`
type inUseRunner struct{ ps, ss string }
func (m inUseRunner) run(_ context.Context, name string, args ...string) (string, error) {
if name == "docker" {
return m.ps, nil
}
return m.ss, nil
}
func TestAFreshMachineIsNotInUse(t *testing.T) {
containers, listeners, err := InUse(context.Background(), inUseRunner{ss: inUseSockets}.run,
func(string) bool { return false })
if err != nil {
t.Fatal(err)
}
if len(containers) != 0 || len(listeners) != 0 {
t.Errorf("ssh, loopback and the daemons a fresh machine runs were counted: %v %v", containers, listeners)
}
}
func TestAMachineServingIsInUse(t *testing.T) {
m := inUseRunner{ps: "hello-web\t\nmesh-store\tabc123\n", ss: inUseSockets + servingSockets}
containers, listeners, err := InUse(context.Background(), m.run, func(string) bool { return false })
if err != nil {
t.Fatal(err)
}
if len(containers) != 1 || containers[0] != "hello-web" {
t.Errorf("containers counted: %v (one a host made is not a predecessor's)", containers)
}
var by []string
for _, l := range listeners {
by = append(by, l.By)
}
if strings.Join(by, " ") != "smbd docker-proxy ntpd" {
t.Errorf("listeners counted: %v", listeners)
}
}
func TestAConvergedGenesisRefusesAMachineInUseNamingEverything(t *testing.T) {
o := Options{State: filepath.Join(t.TempDir(), "state.json")}
m := inUseRunner{ps: "hello-web\t\n", ss: inUseSockets + servingSockets}
err := RefuseAMachineInUse(context.Background(), o, m.run, quietly)
if err == nil {
t.Fatal("a machine in use was not refused")
}
for _, want := range []string{"container hello-web", "tcp 0.0.0.0:445 by smbd", "tcp 0.0.0.0:8080 by docker-proxy",
"udp 0.0.0.0:123 by ntpd", "--adopted"} {
if !strings.Contains(err.Error(), want) {
t.Errorf("the refusal does not name %q: %v", want, err)
}
}
o.Adopted = true
if err := RefuseAMachineInUse(context.Background(), o, m.run, quietly); err != nil {
t.Errorf("an adopted genesis was refused a machine in use: %v", err)
}
}
func TestARerunOfGenesisIsNotAMachineInUse(t *testing.T) {
o := Options{State: filepath.Join(t.TempDir(), "state.json")}
if err := store.Save(o.State, store.State{Resources: []store.Applied{{ID: "store", Type: "container", Target: "mesh-store"}}}); err != nil {
t.Fatal(err)
}
m := inUseRunner{ps: giteaBootstrap + "\t\n", ss: servingSockets}
if err := RefuseAMachineInUse(context.Background(), o, m.run, quietly); err != nil {
t.Errorf("what an earlier genesis raised was counted as a machine in use: %v", err)
}
}
func TestAFreshlyInstalledMachineAsMeasuredIsNotInUse(t *testing.T) {
// Captured with `ss -Hltunp` on a freshly installed lab machine: its resolver listens on TCP on
// every address, which the record's words alone would count.
raw, err := os.ReadFile("testdata/fresh-machine-listeners.txt")
if err != nil {
t.Fatal(err)
}
run := func(ctx context.Context, name string, args ...string) (string, error) {
if name == "ss" {
return string(raw), nil
}
return "", nil
}
containers, listeners, err := InUse(context.Background(), run, func(string) bool { return false })
if err != nil {
t.Fatal(err)
}
if len(containers) != 0 || len(listeners) != 0 {
t.Errorf("a fresh machine read as in use: containers %v, listeners %v", containers, listeners)
}
}
// Defends novox/hq ADR 0103: a machine raised adopted stays adopted if genesis is run again. The
// installer reads the mode from what the machine records, and refuses a flag that disagrees.
func TestARerunWithoutTheFlagOnAnAdoptedMachineIsRefused(t *testing.T) {
o := Options{State: filepath.Join(t.TempDir(), "state.json")}
adoptedState := store.State{Resources: []store.Applied{
{ID: "store", Type: "container", Target: "mesh-store", Origin: store.OriginCarried},
{ID: "adoption.guard", Type: "file", Target: "/etc/mesh/guard.nft", Origin: store.OriginCarried},
}}
if err := store.Save(o.State, adoptedState); err != nil {
t.Fatal(err)
}
m := inUseRunner{ss: inUseSockets}
err := RefuseAMachineInUse(context.Background(), o, m.run, quietly)
if err == nil || !strings.Contains(err.Error(), "pass --adopted") {
t.Fatalf("a re-run without --adopted on an adopted machine was not refused: %v", err)
}
o.Adopted = true
if err := RefuseAMachineInUse(context.Background(), o, m.run, quietly); err != nil {
t.Errorf("a re-run with --adopted on an adopted machine was refused: %v", err)
}
}
func TestARerunWithTheFlagOnAConvergedMachineIsRefused(t *testing.T) {
o := Options{State: filepath.Join(t.TempDir(), "state.json"), Adopted: true}
converged := store.State{Resources: []store.Applied{
{ID: "store", Type: "container", Target: "mesh-store", Origin: store.OriginCarried},
{ID: "base-filter", Type: "file", Target: "/etc/nftables.conf", Origin: store.OriginCarried},
},
// Converged by the controller from adopted: the firewall it found is still recorded.
Firewall: &store.FoundFirewall{Kind: "ufw", WasActive: true, DisabledByMesh: true}}
if err := store.Save(o.State, converged); err != nil {
t.Fatal(err)
}
err := RefuseAMachineInUse(context.Background(), o, inUseRunner{ss: inUseSockets}.run, quietly)
if err == nil || !strings.Contains(err.Error(), "without --adopted") {
t.Fatalf("a re-run with --adopted on a converged machine was not refused: %v", err)
}
o.Adopted = false
if err := RefuseAMachineInUse(context.Background(), o, inUseRunner{ss: inUseSockets}.run, quietly); err != nil {
t.Errorf("a converged re-run of a converged machine was refused: %v", err)
}
}
func TestOnlyTheDaemonsTheMeasurementFoundAreQuiet(t *testing.T) {
// novox/hq ADR 0101: the exempt daemons are the ones a fresh machine was measured to run —
// the resolver and the network manager. A time client or a DHCP client listening beyond
// loopback is something somebody put there, and that is a machine in use.
sockets := `udp UNCONN 0 0 0.0.0.0:123 0.0.0.0:* users:(("systemd-timesyn",pid=260,fd=9))
udp UNCONN 0 0 0.0.0.0:68 0.0.0.0:* users:(("dhcpcd",pid=270,fd=9))
`
_, listeners, err := InUse(context.Background(), inUseRunner{ss: sockets}.run, func(string) bool { return false })
if err != nil {
t.Fatal(err)
}
if len(listeners) != 2 {
t.Errorf("counted %d listener(s), want the time client and the DHCP client: %+v", len(listeners), listeners)
}
}
+80
View File
@@ -1,7 +1,9 @@
package bootstrap
import (
"bytes"
"context"
"encoding/json"
"fmt"
"os"
"path/filepath"
@@ -122,6 +124,9 @@ func registerAndAssign(ctx context.Context, o Options, control controlPlane, mod
// the only place the reason appears.
say(indent(refusal))
}
if err := prepareModule(ctx, o, control, module, say); err != nil {
return out, err
}
return out, nil
}
@@ -143,3 +148,78 @@ func pushNode(ctx context.Context, o Options, control controlPlane, say func(str
say(" pushed " + o.Node)
return strings.TrimSpace(said), nil
}
// pinPlaceholder replaces the catalogue's placeholder digest with what the registry assigned — the
// builder's manifest, which the catalogue still holds (the control plane's comes out of its own
// build, see pinImage).
//
// **Textual, and every place it appears.** A manifest may name its image in more than one resource
// — the catalogue's converted modules routinely carry a runtime container beside the application's
// — and the same reasoning as the bundle rewrite applies: replacing one and not the others leaves
// something pointing at an image nothing serves, and it fails half way through an apply rather
// than here.
//
// It refuses a manifest with no placeholder in it. That is not pedantry: a manifest already
// carrying a real digest is one somebody pinned by hand, and quietly registering it would install a
// control plane that is not the image this machine just published — which is the one thing this
// step exists to guarantee.
func pinPlaceholder(manifest []byte, reference, module string) ([]byte, int, error) {
places := bytes.Count(manifest, []byte(placeholderDigest))
if places == 0 {
return nil, 0, fmt.Errorf(
"the %s module's manifest carries no placeholder digest (%s), so there is nothing to "+
"pin to the image this machine just published.\n"+
"A manifest already naming a digest was pinned by somebody else, to some other "+
"build. Registering it would install a module that is not the one this "+
"installer carried and pushed", module, placeholderDigest)
}
// The reference the registry gave back is `<registry>/<repository>@sha256:…`, and what the
// manifest holds is `<something>@sha256:0…0`. Replacing only the digest would leave the
// manifest's own repository name in front of it — which may be `mesh-controller` with no
// registry, and a runtime would then pull it from the internet. The whole reference moves.
var out bytes.Buffer
rest := manifest
for {
at := bytes.Index(rest, []byte(placeholderDigest))
if at < 0 {
out.Write(rest)
break
}
// Back up over the repository this digest belongs to, which runs to the opening quote.
start := bytes.LastIndexByte(rest[:at], '"')
if start < 0 {
return nil, 0, fmt.Errorf(
"the %s module's manifest has a placeholder digest that is not inside a JSON "+
"string, so the installer cannot tell what image it belongs to", module)
}
out.Write(rest[:start+1])
out.WriteString(reference)
rest = rest[at+len(placeholderDigest):]
}
pinned := out.Bytes()
// Read back. A substitution on text can catch more than it was aimed at, and the manifest is
// about to be handed to the mesh as the description of what it runs.
var checked map[string]any
if err := json.Unmarshal(pinned, &checked); err != nil {
return nil, 0, fmt.Errorf(
"pinning the %s module's image broke its manifest: %w", module, err)
}
if bytes.Contains(pinned, []byte(placeholderDigest)) {
return nil, 0, fmt.Errorf(
"the %s module's manifest still carries a placeholder digest after pinning",
module)
}
return pinned, places, nil
}
// prepareModule is what genesis tells the controller about a module on this node once it is
// assigned and before it is pushed: the ports this node gave it, and on an adopted node that the
// module is taken (novox/hq ADR 0100).
func prepareModule(ctx context.Context, o Options, control controlPlane, module string,
say func(string)) error {
if err := setFoundationSettings(ctx, o, control, module, say); err != nil {
return err
}
return takeIfAdopted(ctx, o, control, module, say)
}
+45 -6
View File
@@ -3,8 +3,10 @@ package bootstrap
import (
"context"
"encoding/json"
"errors"
"fmt"
"net"
"strconv"
"strings"
"time"
)
@@ -87,6 +89,9 @@ func InstallFromCatalogue(ctx context.Context, o Options, control controlPlane,
if _, err := control.tell(ctx, "assign", o.Node, module); err != nil {
return err
}
if err := prepareModule(ctx, o, control, module, say); err != nil {
return err
}
if _, err := pushNode(ctx, o, control, say); err != nil {
return err
}
@@ -120,7 +125,7 @@ func PlaceOnTheNetwork(ctx context.Context, o Options, control controlPlane,
Name: "endpoint",
Question: "Where do other machines reach this one for the private network? " +
"(host:port; the host other machines dial)",
Default: derivedEndpoint(brokerAddress),
Default: derivedEndpoint(brokerAddress, o.Ports.orDefaults().Hub),
}, o.Answers["endpoint"], o.Prompt, say)
if err != nil {
return err
@@ -129,6 +134,10 @@ func PlaceOnTheNetwork(ctx context.Context, o Options, control controlPlane,
return fmt.Errorf("the private network needs an endpoint other machines can dial, and " +
"nothing said one: pass --endpoint, or --broker-address so one can be derived")
}
endpoint, err = endpointAgrees(endpoint, o.Ports.orDefaults().Hub)
if err != nil {
return err
}
if _, err := control.tell(ctx, "assign", o.Node, module); err != nil {
return err
@@ -146,16 +155,22 @@ func PlaceOnTheNetwork(ctx context.Context, o Options, control controlPlane,
// ChooseAndInstallFilter picks the packet filter — required, so the question is which, not
// whether — and installs it.
func ChooseAndInstallFilter(ctx context.Context, o Options, control controlPlane, say func(string)) error {
func ChooseAndInstallFilter(ctx context.Context, o Options, control controlPlane, say func(string)) (string, error) {
filter, err := decide(Choice{
Name: "packet-filter",
Question: "Which packet filter should this machine run?",
Options: []string{"nftables"},
}, o.Answers["packet-filter"], o.Prompt, say)
if err != nil {
return err
return "", err
}
return InstallFromCatalogue(ctx, o, control, filter, say)
if o.Adopted {
// The firewall found here stays in force until the node converges; the filter is
// chosen now and assigned by the flip (novox/hq ADR 0100).
say(" not installed " + filter + " — this machine is adopted; converging it assigns " + filter)
return filter, nil
}
return filter, InstallFromCatalogue(ctx, o, control, filter, say)
}
// InstallExtras installs what was asked for beyond the floor.
@@ -200,14 +215,38 @@ func builds(manifest []byte) bool {
return m.Build != nil && len(m.Build.Artifacts) > 0
}
// endpointAgrees holds the endpoint other machines dial to the port this node gave the private
// network's hub (novox/hq ADR 0100): the hub binds what --hub-port says, so an endpoint naming
// another port is an address nothing answers on. A host alone takes the hub's port.
func endpointAgrees(endpoint string, hub int) (string, error) {
_, portText, err := net.SplitHostPort(endpoint)
var missing *net.AddrError
if errors.As(err, &missing) && missing.Err == "missing port in address" {
return net.JoinHostPort(strings.Trim(endpoint, "[]"), strconv.Itoa(hub)), nil
}
if err != nil {
return "", fmt.Errorf("--endpoint %q is not host:port: %w", endpoint, err)
}
port, err := strconv.Atoi(portText)
if err != nil {
return "", fmt.Errorf("--endpoint %q does not end in a port", endpoint)
}
if port != hub {
return "", fmt.Errorf("--endpoint %s names port %d and the private network's hub binds %d "+
"(--hub-port): other machines would dial a port nothing answers on. Give one port for the "+
"hub; nothing was changed", endpoint, port, hub)
}
return endpoint, nil
}
// derivedEndpoint is the default place other machines dial for the private network: the same host
// they already dial for the broker, on WireGuard's ordinary port. One fact, not two.
func derivedEndpoint(brokerAddress string) string {
func derivedEndpoint(brokerAddress string, hub int) string {
host, _, err := net.SplitHostPort(brokerAddress)
if err != nil || host == "" {
return ""
}
return net.JoinHostPort(host, "51820")
return net.JoinHostPort(host, strconv.Itoa(hub))
}
func refOr(ref string) string {
+24 -3
View File
@@ -1,14 +1,17 @@
package bootstrap
import "testing"
import (
"strings"
"testing"
)
// The endpoint other machines dial defaults to the host they already dial — the broker's — on
// WireGuard's port. One fact, not two that drift.
func TestTheEndpointDerivesFromTheBrokerAddress(t *testing.T) {
if got := derivedEndpoint("192.0.2.10:5671"); got != "192.0.2.10:51820" {
if got := derivedEndpoint("192.0.2.10:5671", 51820); got != "192.0.2.10:51820" {
t.Fatalf("derived %q", got)
}
if got := derivedEndpoint(""); got != "" {
if got := derivedEndpoint("", 51820); got != "" {
t.Fatalf("an endpoint was invented from nothing: %q", got)
}
}
@@ -23,3 +26,21 @@ func TestOnlyAManifestWithArtifactsBuilds(t *testing.T) {
t.Fatal("a manifest with nothing to build was built anyway")
}
}
// Defends novox/hq ADR 0100: the hub's port is the node's, and the endpoint other machines dial
// must name it — an endpoint on another port is an address nothing answers on.
func TestTheEndpointAgreesWithTheHubsPort(t *testing.T) {
if got, err := endpointAgrees("192.0.2.10:51820", 51820); err != nil || got != "192.0.2.10:51820" {
t.Errorf("an endpoint on the hub's port: %q %v", got, err)
}
if got, err := endpointAgrees("192.0.2.10", 51821); err != nil || got != "192.0.2.10:51821" {
t.Errorf("a host alone did not take the hub's port: %q %v", got, err)
}
_, err := endpointAgrees("192.0.2.10:51820", 51821)
if err == nil || !strings.Contains(err.Error(), "--hub-port") {
t.Errorf("two ports for one hub were accepted: %v", err)
}
if _, err := endpointAgrees("192.0.2.10:not-a-port", 51820); err == nil {
t.Error("an endpoint whose port is not a number was accepted")
}
}
+3
View File
@@ -122,6 +122,9 @@ func installProvider(ctx context.Context, o Options, control controlPlane, modul
if _, err := control.tell(ctx, "assign", o.Node, module); err != nil {
return err
}
if err := prepareModule(ctx, o, control, module, say); err != nil {
return err
}
if beforePush != nil {
if err := beforePush(); err != nil {
return err
+116 -18
View File
@@ -25,11 +25,24 @@ const (
// foundationStore is the foundation's postgres container — the mesh's own memory, raised from the
// bundle. gitea's bootstrap database lives here too, so a mesh runs one postgres (issue 051).
foundationStore = "mesh-store"
// giteaBootstrap is the gitea server raised directly at genesis, before gitea is a module.
giteaBootstrap = "mesh-gitea-server"
// giteaImage is the same upstream image the gitea module runs, pinned identically so the module
// adopts the running server rather than replacing it.
giteaImage = "gitea/gitea@sha256:dfc61e347c8b582df918f4556401bf2cecdfbdb56c5282ae9488dd76fca3e41c"
// giteaBootstrap is the gitea server raised directly at genesis, before gitea is a module —
// under the name the gitea MODULE declares for its container, so the module finds it and holds
// it rather than raising a second forge beside it (novox/hq ADR 0163, rule 7; issue 090).
giteaBootstrap = "gitea"
// giteaImage is the image the gitea module declares for that container, pinned to the same
// digest, so taking the module over is not a downgrade and not an upgrade. **Moves with the
// module's pin**: the two are compared by a take, and a difference is said there — but a
// genesis that raised an older image than the module declares would be taken over as an
// upgrade on first push, which a forge holding the mesh's packages must not have done to it
// unannounced. Checked in TestGenesisRaisesTheForgeAsTheModuleDeclaresIt against the module's
// manifest where the catalogue is beside this checkout.
giteaImage = "gitea/gitea@sha256:87a67ee09d3ae0d1df5fda5dcda3e2a1f9236a45b0a59025d6e00e46adc43bef"
// giteaDataDir is where the module's `data` directory resolves on a machine with the default
// layout (<data root>/<module>/<id>, novox/hq ADR 0112): mounted at /data as the module mounts
// it, so the repositories, attachments and indexes the bootstrap forge accumulates are the
// module's the day it is taken — before this, the forge had no volume and its data was the
// container's, lost with it.
giteaDataDir = "/var/lib/gitea/data"
// packagesOrg is the npm owner: every module consumes `@novox/*` from this gitea org.
packagesOrg = "novox"
// packagesTeam is the org team whose members may read and write the org's packages.
@@ -42,10 +55,26 @@ const (
// giteaDBRole/giteaDBName is gitea's own database in the foundation store.
giteaDBRole = "mesh_gitea"
giteaDBName = "mesh_gitea"
// giteaPort is where the raised server answers on the machine.
giteaPort = 3000
// defaultGiteaPort is where the raised server answers on the machine unless the node gave the
// package registry another port (novox/hq ADR 0100).
defaultGiteaPort = 3000
)
// ForgeModule is the module that owns the package registry — the one the bootstrap forge above is
// a stand-in for, and which takes it over once the base exists.
//
// Named here because the port given for the package registry is that module's setting on this node
// and not this installer's private number (novox/hq 04-ISSUES/085). What follows that setting is
// what the mesh derives from a module's ports: the forge's container mapping, its rule in the
// filter, its opening on an adopted node, and what it says it serves — so a consumer the mesh binds
// is told where the machine actually put the registry.
//
// What does NOT follow it, and is not this change's to fix: the address of itself the forge's
// runtime is given, which the catalogue writes as a literal (novox/hq 04-ISSUES/088), and the port
// in its route contribution. Both are already wrong for any machine port the mesh assigned, with a
// given port or without one.
const ForgeModule = "gitea"
// RaisePackageRegistry puts a working npm registry in front of the base build. It is idempotent:
// every step tolerates having been done, because genesis is safe to run again.
func RaisePackageRegistry(ctx context.Context, o Options, d Deps, control controlPlane,
@@ -60,17 +89,18 @@ func RaisePackageRegistry(ctx context.Context, o Options, d Deps, control contro
}
say(" seeding gitea's database in the foundation store")
ports := o.Ports.orDefaults()
if err := seedGiteaDatabase(ctx, run, o.Timeout, dbPassword, say); err != nil {
return err
}
say(" raising the gitea server on that database")
if err := raiseGiteaServer(ctx, run, o.Timeout, dbPassword, say); err != nil {
if err := raiseGiteaServer(ctx, run, o.Timeout, dbPassword, ports, say); err != nil {
return err
}
say(" waiting for gitea to answer")
base := fmt.Sprintf("http://127.0.0.1:%d", giteaPort)
base := fmt.Sprintf("http://127.0.0.1:%d", ports.Packages)
if err := waitForGitea(ctx, d, o, base, say); err != nil {
return err
}
@@ -98,6 +128,11 @@ func RaisePackageRegistry(ctx context.Context, o Options, d Deps, control contro
return err
}
say(" recording the port it was given as the forge module's own")
if err := recordTheForgesPort(ctx, o, control, say); err != nil {
return err
}
say(" delivering the builder its registry credential")
if err := deliverBuilderNpm(ctx, o, control, builderPassword, say); err != nil {
return err
@@ -106,6 +141,59 @@ func RaisePackageRegistry(ctx context.Context, o Options, d Deps, control contro
return nil
}
// recordTheForgesPort makes the port given for the package registry a setting of the module that
// serves it, on this node (novox/hq 04-ISSUES/085, ADR 0100).
//
// **The forge is the one foundation port that was not a setting.** The store's, the bus's, the
// broker's management port and the registry's each become a `ports` setting of the module that
// binds them, set where that module is registered and assigned; the forge is raised by hand here,
// long before its module can be built, so there was no registration to hang it on and the number
// lived in rewritten manifest text instead. So the manifest is registered here — not assigned, and
// nothing of it runs — for the one thing registering buys: a settings row needs the module row to
// exist. Whenever somebody later assigns the forge on this node, it comes up on the port this
// machine was given rather than on the catalogue's.
//
// **Registered only when the mesh does not already know it, which is the opposite of every other
// `module add` here.** The rest of the installer registers every run on purpose: the manifest is
// what changes between runs, and skipping it would pin the mesh to a previous image. This one
// changes nothing about the forge and wants nothing of the checkout's manifest — and registering
// is an overwrite, so a re-run of genesis pointed at an older catalogue would replace the manifest
// of a forge that is built and assigned, with a push a few lines later. The port is the only thing
// this is here to record, and the setting does not need the manifest to be this checkout's.
//
// A node given the catalogue's own port records nothing and registers nothing, so a genesis on the
// defaults does exactly what it did before.
func recordTheForgesPort(ctx context.Context, o Options, control controlPlane,
say func(string)) error {
if o.Ports.orDefaults().Packages == DefaultPorts().Packages {
return nil
}
known, err := control.tell(ctx, "module", "list")
if err != nil {
return err
}
if mentions(known, ForgeModule) {
say(" known " + ForgeModule + " — left as the mesh has it; only its port is set here")
} else {
manifest, err := readManifest(o.Catalogue, ForgeModule)
if err != nil {
return fmt.Errorf("%w\n"+
"This is the module that owns the forge genesis just raised. Without it the port this "+
"machine was given for the package registry is nobody's setting, and taking the forge "+
"over would put it back on the catalogue's port", err)
}
remote := "/" + ForgeModule + "-module.json"
if err := control.carrying(ctx, ForgeModule+"-module.json", manifest, remote); err != nil {
return err
}
if _, err := control.tell(ctx, "module", "add", remote); err != nil {
return err
}
say(" registered " + ForgeModule + " — registered, not assigned: nothing of it runs yet")
}
return setFoundationSettings(ctx, o, control, ForgeModule, say)
}
// seedGiteaDatabase creates gitea's role and database inside the foundation postgres, the same way
// the foundation creates its own — psql run through the store container (the map's Route B). The role
// is created before the database because the database is owned by it. Both are tolerant of already
@@ -146,17 +234,18 @@ func seedGiteaDatabase(ctx context.Context, run Runner, timeout time.Duration, p
return nil
}
// raiseGiteaServer starts the gitea server container against the foundation store. It joins the
// store's network namespace so `127.0.0.1:5432` reaches postgres, and publishes its own port on the
// machine so the builder and this installer can reach it. Started if absent, left alone if present.
// raiseGiteaServer starts the gitea server container against the foundation store. It runs on the
// machine's own network, so `127.0.0.1` reaches the store where it publishes its port, and it binds
// its own port there for the builder and this installer. Started if absent, left alone if present.
func raiseGiteaServer(ctx context.Context, run Runner, timeout time.Duration, dbPassword string,
say func(string)) error {
ports FoundationPorts, say func(string)) error {
asking, cancel := context.WithTimeout(ctx, timeout)
defer cancel()
// Already there: a re-run does not raise a second one. `docker start` is a no-op on a running
// container and revives a stopped one.
if out, _ := run(asking, "docker", "inspect", "--format", "{{.Id}}", giteaBootstrap); strings.TrimSpace(out) != "" {
// container and revives a stopped one. `container inspect`, not the bare form: a name is not
// unique across object kinds, and `.Id` resolves on a network or volume too.
if out, _ := run(asking, "docker", "container", "inspect", "--format", "{{.Id}}", giteaBootstrap); strings.TrimSpace(out) != "" {
_, _ = run(asking, "docker", "start", giteaBootstrap)
return nil
}
@@ -164,7 +253,7 @@ func raiseGiteaServer(ctx context.Context, run Runner, timeout time.Duration, db
env := []string{
"-e", "GITEA__database__DB_TYPE=postgres",
// The store is reached on the shared network namespace's loopback.
"-e", "GITEA__database__HOST=127.0.0.1:5432",
"-e", fmt.Sprintf("GITEA__database__HOST=127.0.0.1:%d", ports.Store),
"-e", "GITEA__database__NAME=" + giteaDBName,
"-e", "GITEA__database__USER=" + giteaDBRole,
"-e", "GITEA__database__PASSWD=" + dbPassword,
@@ -174,16 +263,25 @@ func raiseGiteaServer(ctx context.Context, run Runner, timeout time.Duration, db
// package metadata hands npm a tarball URL built from ROOT_URL, and a client only sends its
// stored credential to the host it was stored for. A default ROOT_URL of localhost is a
// different host than the binding's 127.0.0.1, so the credential would not be sent.
"-e", fmt.Sprintf("GITEA__server__ROOT_URL=http://127.0.0.1:%d/", giteaPort),
"-e", fmt.Sprintf("GITEA__server__ROOT_URL=http://127.0.0.1:%d/", ports.Packages),
"-e", "USER_UID=1000", "-e", "USER_GID=1000",
}
if ports.Packages != defaultGiteaPort {
// On the machine's network the server binds its own port, so a port given for it is
// the one it is told to listen on.
env = append(env, "-e", fmt.Sprintf("GITEA__server__HTTP_PORT=%d", ports.Packages))
}
args := append([]string{
"run", "-d", "--name", giteaBootstrap,
// Host network, like the control plane: it reaches the foundation store on the machine's
// loopback (where the store publishes 5432) and answers on the machine's own 3000, which is
// where mesh-bootstrap and the builder's build containers look for it.
// where mesh-bootstrap and the builder's build containers look for it. The module runs
// bridged and publishes its ports; that is the one difference a take still has to say
// (ADR 0163, rule 7) — the data, the name and the image are the module's already.
"--network", "host",
"--restart", "unless-stopped",
// The module's data directory, so what the forge accumulates is the module's when taken.
"--volume", giteaDataDir + ":/data",
}, env...)
args = append(args, giteaImage)
+534
View File
@@ -0,0 +1,534 @@
package bootstrap
import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"net"
"sort"
"strconv"
"strings"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/reachable"
"github.com/novox/mesh-host/internal/store"
"github.com/novox/mesh-host/internal/tunnel"
)
// FoundationPorts are the machine's ports the foundation binds (novox/hq ADR 0100).
//
// **The node's, not the catalogue's.** A machine in use may already hold one — a predecessor's
// registry on 5000, its broker's management port — and a port fixed in the bundle and the manifests
// surfaces as a container that fails to bind, and one changed at genesis would be changed back when
// the foundation is adopted as modules. So each is an input here, checked free, rewritten into the
// bundle, and handed to the controller as that node's setting for the module that binds it.
type FoundationPorts struct {
Store int `json:"store"`
Bus int `json:"bus"`
AMQP int `json:"amqp"`
Management int `json:"management"`
Registry int `json:"registry"`
Packages int `json:"packages"`
Hub int `json:"hub"`
}
// DefaultPorts are the catalogue's numbers.
func DefaultPorts() FoundationPorts {
return FoundationPorts{Store: 5432, Bus: 5671, AMQP: 5672, Management: 15672, Registry: 5000,
Packages: 3000, Hub: 51820}
}
// DefaultOverlayRange is the controller's default private-network range.
const DefaultOverlayRange = "10.42.0.0/16"
// orDefaults fills every port left unsaid.
func (p FoundationPorts) orDefaults() FoundationPorts {
d := DefaultPorts()
for _, f := range []struct{ got, def *int }{
{&p.Store, &d.Store}, {&p.Bus, &d.Bus}, {&p.AMQP, &d.AMQP}, {&p.Management, &d.Management},
{&p.Registry, &d.Registry}, {&p.Packages, &d.Packages}, {&p.Hub, &d.Hub},
} {
if *f.got == 0 {
*f.got = *f.def
}
}
return p
}
// named is each port with what it is and its protocol, in a fixed order.
func (p FoundationPorts) named() []namedPort {
return []namedPort{
{"the store", "tcp", p.Store}, {"the bus", "tcp", p.Bus}, {"the broker's AMQP", "tcp", p.AMQP},
{"the broker's management", "tcp", p.Management}, {"the registry", "tcp", p.Registry},
{"the package registry", "tcp", p.Packages}, {"the private network's hub", "udp", p.Hub},
}
}
type namedPort struct {
what, protocol string
port int
}
// Check refuses a port out of range, or one port given for two things.
func (p FoundationPorts) Check() error {
seen := map[string]string{}
for _, n := range p.named() {
if n.port < 1 || n.port > 65535 {
return fmt.Errorf("%s's port is %d, and a port is 1-65535", n.what, n.port)
}
key := n.protocol + "/" + strconv.Itoa(n.port)
if other, twice := seen[key]; twice {
return fmt.Errorf("%s and %s were both given %s", other, n.what, key)
}
seen[key] = n.what
}
return nil
}
// moduleSettings is what each foundation module is told about its ports on this node: the port it
// declares, to the machine's port it is given. Only what differs from the catalogue — a converged
// genesis on the defaults sets nothing, and so changes nothing it did before.
func (p FoundationPorts) moduleSettings() map[string]map[string]int {
d := DefaultPorts()
out := map[string]map[string]int{}
add := func(module string, declared, given int) {
if given == declared {
return
}
if out[module] == nil {
out[module] = map[string]int{}
}
out[module][strconv.Itoa(declared)] = given
}
add("postgres", d.Store, p.Store)
add("lavinmq", d.Bus, p.Bus)
add("lavinmq", d.AMQP, p.AMQP)
add("lavinmq", d.Management, p.Management)
add(RegistryModule, d.Registry, p.Registry)
add(ForgeModule, d.Packages, p.Packages)
return out
}
// PortsSetting is the controller's settings key for a module's given ports.
const PortsSetting = "ports"
// ServesSetting is the settings key that carries what a provider serves, inside the one binding a
// module carries rather than resolves — the builder's package binding (novox/hq 04-ISSUES/085).
//
// **A port given at genesis has to be a setting wherever it is read, or something puts it back.**
// The builder reaches the package registry through a binding written in its own manifest, because
// at genesis no module yet provides `package-registry` and so there is nothing for the controller
// to resolve it from. That binding used to be rewritten, as text, when the installer registered the
// builder — which a later registration from the catalogue undid silently. Set as a node's setting
// instead, it is held by the controller rather than by the manifest, so re-registering the builder
// leaves it where it was.
//
// **A settings layer is the module's, not one resource's.** The controller lays it over every file
// of that module which merges as JSON, so `serves` lands in the package binding only because that
// binding is the builder's one mergeable file. A second mergeable file added to the builder would
// be given a `serves` key too, meaning nothing to whatever reads it. Worth knowing before adding
// one.
const ServesSetting = "serves"
// setFoundationSettings tells the controller the ports this node gave a foundation module — and,
// on an adopted node, that the registry is reached from anywhere, as a node pulls from it before it
// has a private-network address (novox/hq ADR 0100). Done after the module is registered and before
// the push that raises it, so the first declaration already names the node's ports.
func setFoundationSettings(ctx context.Context, o Options, control controlPlane, module string,
say func(string)) error {
values := map[string]any{}
if ports := o.Ports.orDefaults().moduleSettings()[module]; len(ports) > 0 {
values[PortsSetting] = ports
}
if o.Adopted && module == RegistryModule {
values["expose"] = map[string]string{strconv.Itoa(DefaultPorts().Registry): "anywhere"}
}
if packages := o.Ports.orDefaults().Packages; module == BuilderModule &&
packages != DefaultPorts().Packages {
// The one binding nothing resolves: the builder dials the forge by a number it carries.
// The rest of the binding — the provision, the forge it is from, the account it presents —
// is the manifest's and stays there.
values[ServesSetting] = map[string]int{"port": packages}
}
if len(values) == 0 {
return nil
}
raw, err := json.Marshal(values)
if err != nil {
return err
}
remote := "/" + module + "-settings.json"
if err := control.carrying(ctx, module+"-settings.json", raw, remote); err != nil {
return err
}
if _, err := control.tell(ctx, "settings", "set", module, remote, "--node", o.Node); err != nil {
return err
}
say(" settings " + module + " on " + o.Node + ": " + string(raw))
return nil
}
// PortsRewrite says what RewritePorts changed.
type PortsRewrite struct {
Places int
}
// RewritePorts puts the node's foundation ports into the produced bundle, in place of the
// template's, byte for byte like every other rewrite — so the file keeps its comments and a person
// can read what was applied. A port left at its default is not touched, so a genesis on the
// defaults produces exactly the bundle it did before.
//
// Only the machine's side moves: the outer port of each mapping, the addresses the control plane
// dials on the machine's loopback, and the address nodes are told to dial. What a container listens
// on inside itself, and what an action reaches inside the store's own network, stay as they are.
func RewritePorts(r *Rewritten, p FoundationPorts, overlayRange string) (PortsRewrite, error) {
var out PortsRewrite
p = p.orDefaults()
d := DefaultPorts()
bundle := r.Bundle
var err error
replace := func(from, to, what string) {
if err != nil || from == to {
return
}
bundle, err = replaceOnce(bundle, from, to, what)
out.Places++
}
if p.Store != d.Store {
replace(`"ports": ["5432:5432"]`, fmt.Sprintf(`"ports": ["%d:5432"]`, p.Store), "the store's published port")
}
if p.Bus != d.Bus || p.AMQP != d.AMQP || p.Management != d.Management {
replace(`"ports": ["5671:5671", "5672:5672", "127.0.0.1:15672:15672"]`,
fmt.Sprintf(`"ports": ["%d:5671", "%d:5672", "127.0.0.1:%d:15672"]`, p.Bus, p.AMQP, p.Management),
"the broker's published ports")
}
if err != nil {
return out, err
}
// The control plane runs on the machine's network and dials the store and the broker on its
// loopback, so its connection strings name the machine's ports. The schema step reaches the
// store inside the store's own network and keeps the container's port — so these are found by
// the control plane's environment, not by searching for the text.
control, cerr := controlPlaneIn(r.Declaration)
if cerr != nil {
return out, cerr
}
for _, key := range sortedKeys(control.Env) {
value := control.Env[key]
now := value
now = strings.ReplaceAll(now, "@127.0.0.1:5432/", fmt.Sprintf("@127.0.0.1:%d/", p.Store))
now = strings.ReplaceAll(now, "@127.0.0.1:5672/", fmt.Sprintf("@127.0.0.1:%d/", p.AMQP))
if strings.HasSuffix(now, "@127.0.0.1:15672") {
now = strings.TrimSuffix(now, "15672") + strconv.Itoa(p.Management)
}
if key == brokerAddressVar {
if host, port, splitErr := net.SplitHostPort(value); splitErr == nil && port == "5671" {
now = net.JoinHostPort(host, strconv.Itoa(p.Bus))
}
}
if now != value {
replace(`"`+key+`": "`+value+`"`, `"`+key+`": "`+now+`"`, "the control plane's "+key)
}
}
if err != nil {
return out, err
}
// The foundation's own filter, where the template carries one: it admits the bus and the
// registry from anywhere, on whatever port they are.
for _, f := range []struct{ def, now int }{{d.Bus, p.Bus}, {d.Registry, p.Registry}} {
if f.def == f.now {
continue
}
for _, form := range []string{"tcp dport %d accept", "ct original proto-dst %d accept"} {
from, to := fmt.Sprintf(form, f.def), fmt.Sprintf(form, f.now)
if n := bytes.Count(bundle, []byte(from)); n > 0 {
bundle = bytes.ReplaceAll(bundle, []byte(from), []byte(to))
out.Places += n
}
}
}
// The private network's range, when it is not the default, is the controller's to know.
if overlayRange != "" && overlayRange != DefaultOverlayRange {
replace(`"`+brokerAddressVar+`": `,
`"MESH_OVERLAY_CIDR": "`+overlayRange+`",
"`+brokerAddressVar+`": `, "where the control plane is told the private network's range")
if err != nil {
return out, err
}
}
if out.Places == 0 {
return out, nil
}
parsed, perr := declaration.ParseFileTrusted(bundle)
if perr != nil {
return out, fmt.Errorf("the bundle stopped being a declaration after its ports were rewritten, which is this installer's fault: %w", perr)
}
r.Bundle, r.Declaration, r.Resources = bundle, parsed, len(parsed.Resources)
if c, cerr := controlPlaneIn(parsed); cerr == nil {
r.BrokerAddress = c.Env[brokerAddressVar]
}
return out, nil
}
// PortsFree refuses a foundation port something else already holds, naming what holds it. What the
// mesh itself raised on an earlier run of genesis is not counted: ours says which holders are.
func PortsFree(ctx context.Context, run Runner, p FoundationPorts, ours func(reachable.Reach) bool) error {
out, err := run(ctx, "ss", "-Hltunp")
if err != nil {
return fmt.Errorf("cannot read which ports this machine holds, so the foundation's cannot be checked free: %w", err)
}
sockets := reachable.Sockets(out)
var published []reachable.Reach
if ps, err := run(ctx, "docker", "ps", "--format", "{{.Names}}\t{{.Ports}}"); err == nil {
published = reachable.Published(ps)
}
held := reachable.Merge(sockets, published)
var problems []string
for _, n := range p.orDefaults().named() {
var by []string
for _, r := range held {
if r.Protocol != n.protocol || r.Port != n.port || ours(r) {
continue
}
holder := r.By
if holder == "" {
holder = "something ss does not name"
}
if r.Published {
holder = "the container " + r.By
}
if !contains(by, holder) {
by = append(by, holder)
}
}
if len(by) > 0 {
problems = append(problems, fmt.Sprintf("%s's port %s/%d is held by %s",
n.what, n.protocol, n.port, strings.Join(by, ", ")))
}
}
if len(problems) > 0 {
return fmt.Errorf("the foundation's ports must be free before anything is raised:\n - %s\n"+
"Give it another with the matching flag (--store-port, --bus-port, --amqp-port, "+
"--management-port, --registry-port, --packages-port, --hub-port); nothing was changed",
strings.Join(problems, "\n - "))
}
return nil
}
// TakeTheTunnel finds the tunnel an adopted machine's private network takes over (novox/hq ADR
// 0105) and settles the options on it: the hub's port is the tunnel's, the mesh's range is the
// tunnel's, and the interface is named for the enrolment that takes its key. Nil when no tunnel is
// up, which is an ordinary machine. A --hub-port or --overlay-range that disagrees with the
// tunnel is refused: the peers dial the tunnel's port and live in its range, and a mesh raised
// beside them on other numbers is the two-tunnel shape the record rejects.
func TakeTheTunnel(o *Options, run Runner) (*tunnel.Found, error) {
found, err := tunnel.Find(context.Background(), tunnel.Runner(run), o.Tunnel)
if errors.Is(err, tunnel.ErrNone) {
return nil, nil
}
if err != nil {
return nil, fmt.Errorf("%w. An adopted hub takes over the tunnel it finds; nothing was changed", err)
}
if o.Ports.Hub != 0 && o.Ports.Hub != DefaultPorts().Hub && o.Ports.Hub != found.Port {
return nil, fmt.Errorf("--hub-port %d disagrees with the tunnel %s, which listens on %d: the "+
"private network takes over that tunnel on its own port, so leave --hub-port unsaid or "+
"say %d", o.Ports.Hub, found.Interface, found.Port, found.Port)
}
if o.OverlayRange != "" && o.OverlayRange != DefaultOverlayRange && o.OverlayRange != found.Range {
return nil, fmt.Errorf("--overlay-range %s disagrees with the tunnel %s, whose range is %s: the "+
"private network takes over that tunnel with its range, so leave --overlay-range unsaid "+
"or say %s", o.OverlayRange, found.Interface, found.Range, found.Range)
}
o.Tunnel = found.Interface
o.Ports.Hub = found.Port
o.OverlayRange = found.Range
return &found, nil
}
// OverlayClear refuses a private-network range that overlaps an address or a route the machine
// already has — a predecessor's tunnel still running — naming the interface. The mesh's own
// interface is not counted.
func OverlayClear(ctx context.Context, run Runner, overlayRange string) error {
if overlayRange == "" {
overlayRange = DefaultOverlayRange
}
_, mine, err := net.ParseCIDR(overlayRange)
if err != nil {
return fmt.Errorf("the private network's range %q is not a range: %w", overlayRange, err)
}
var clashes []string
if out, err := run(ctx, "ip", "-o", "addr", "show"); err == nil {
for _, line := range strings.Split(out, "\n") {
f := strings.Fields(line)
// 3: wg0 inet 10.42.0.1/24 scope global wg0
if len(f) < 4 || (f[2] != "inet" && f[2] != "inet6") {
continue
}
iface := strings.TrimSuffix(f[1], ":")
if clash(mine, f[3]) && iface != meshInterface {
clashes = append(clashes, fmt.Sprintf("%s holds %s", iface, f[3]))
}
}
} else {
return fmt.Errorf("cannot read this machine's addresses to check the private network's range: %w", err)
}
if out, err := run(ctx, "ip", "-o", "route", "show"); err == nil {
for _, line := range strings.Split(out, "\n") {
f := strings.Fields(line)
// 10.42.0.0/16 dev wg0 proto kernel scope link src 10.42.0.1
if len(f) < 3 || f[0] == "default" {
continue
}
iface := ""
for i := range f {
if f[i] == "dev" && i+1 < len(f) {
iface = f[i+1]
}
}
if iface != meshInterface && clash(mine, f[0]) {
clashes = append(clashes, fmt.Sprintf("%s routes %s", iface, f[0]))
}
}
}
if len(clashes) > 0 {
return fmt.Errorf("the private network's range %s overlaps what this machine already has: %s.\n"+
"A tunnel a predecessor still runs would take the mesh's traffic. Give another range with "+
"--overlay-range; nothing was changed", overlayRange, strings.Join(clashes, "; "))
}
return nil
}
// meshInterface is the private network's own interface, which a re-run finds holding its range.
const meshInterface = "mesh0"
func clash(mine *net.IPNet, other string) bool {
if !strings.Contains(other, "/") {
if ip := net.ParseIP(other); ip != nil {
return mine.Contains(ip)
}
return false
}
ip, theirs, err := net.ParseCIDR(other)
if err != nil {
return false
}
return mine.Contains(theirs.IP) || theirs.Contains(mine.IP) || mine.Contains(ip)
}
// NamesFree refuses a foundation or bundle container name that a container already has, when no
// host made that container and this node has no record of it — a predecessor's container under the
// mesh's name, which raising the foundation would replace.
func NamesFree(ctx context.Context, run Runner, names []string, known store.State) error {
var taken []string
sorted := append([]string{}, names...)
sort.Strings(sorted)
for _, name := range sorted {
// `container inspect`: a name is not unique across object kinds, and the bare form can
// resolve to a same-named network or volume instead of reporting the container absent.
out, err := run(ctx, "docker", "container", "inspect", "--format",
"{{index .Config.Labels \"mesh-host.spec\"}}", name)
if err != nil {
continue // no such container
}
label := strings.TrimSpace(out)
if label != "" && label != "<no value>" {
continue
}
if known.Recorded(string(declaration.TypeContainer), name) {
continue
}
if name == giteaBootstrap && len(known.Resources) > 0 {
// Genesis raises the package registry itself, by hand and before the host records
// anything of it, so on a re-run it is found under its own name with no label and no
// record. A machine that carries what an earlier genesis raised made it.
continue
}
taken = append(taken, name)
}
if len(taken) > 0 {
return fmt.Errorf("this machine already runs a container under the name the foundation uses, "+
"and nothing of the mesh's made it: %s.\nRaising the foundation would replace it. Rename or "+
"stop it first; nothing was changed", strings.Join(taken, ", "))
}
return nil
}
// CheckTheMachine is every check genesis makes before raising anything that this machine does not
// already hold what the foundation needs: its ports, its private network's range, its containers'
// names (novox/hq ADR 0100). A re-run of genesis finds the foundation it raised and does not count
// it.
func CheckTheMachine(ctx context.Context, o Options, run Runner, bundle *declaration.Declaration,
say func(string)) error {
known, err := store.Load(o.State)
if err != nil {
return err
}
rerun := len(known.Resources) > 0
names := foundationNames(bundle)
mine := map[string]bool{}
for _, n := range names {
mine[n] = true
}
p := o.Ports.orDefaults()
ours := func(r reachable.Reach) bool {
switch {
case mine[r.By]:
return true
case !rerun:
return false
case r.By == "gitea" && r.Port == p.Packages:
// The package registry runs on the machine's network, so ss names its process.
return true
case r.By == "" && r.Protocol == "udp" && r.Port == p.Hub:
// The private network's hub is a kernel interface and has no process.
return true
}
return false
}
if o.Tunnel != "" {
// The hub's port is the found tunnel's, held by that tunnel until the mesh's interface
// takes it over (novox/hq ADR 0105): held by design, not by something else.
inner := ours
ours = func(r reachable.Reach) bool {
return inner(r) || (r.Protocol == "udp" && r.Port == p.Hub)
}
}
if err := PortsFree(ctx, run, p, ours); err != nil {
return err
}
say(fmt.Sprintf(" ports free store %d, bus %d, amqp %d, management %d, registry %d, packages %d, hub %d/udp",
p.Store, p.Bus, p.AMQP, p.Management, p.Registry, p.Packages, p.Hub))
if o.Tunnel != "" {
// One tunnel and one range: the mesh's range IS the found tunnel's, so the rule that the
// two must not overlap applies only where a found tunnel is left running beside the mesh's
// (ADR 0100, narrowed by ADR 0105).
say(fmt.Sprintf(" range %s is the tunnel %s's, taken over; not checked against it",
o.OverlayRange, o.Tunnel))
} else if err := OverlayClear(ctx, run, o.OverlayRange); err != nil {
return err
}
if err := NamesFree(ctx, run, names, known); err != nil {
return err
}
return nil
}
// foundationNames are the containers the foundation and genesis raise under fixed names.
func foundationNames(bundle *declaration.Declaration) []string {
names := []string{ControlPlaneModule, giteaBootstrap, "mesh-registry"}
for _, n := range containerNames(bundle) {
if !contains(names, n) {
names = append(names, n)
}
}
sort.Strings(names)
return names
}
+376
View File
@@ -0,0 +1,376 @@
package bootstrap
import (
"context"
"crypto/ecdh"
"crypto/rand"
"encoding/base64"
"errors"
"os"
"path/filepath"
"strings"
"testing"
"time"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/reachable"
"github.com/novox/mesh-host/internal/store"
"github.com/novox/mesh-host/internal/tunnel"
)
// Defends novox/hq ADR 0100: the foundation's ports are the node's — inputs to genesis, checked free,
// rewritten into the bundle, and handed to the controller as the node's settings.
func producedBundle(t *testing.T) Rewritten {
t.Helper()
template, err := os.ReadFile("../../examples/foundation-first-node.lock")
if err != nil {
t.Skip("no example bundle beside this checkout")
}
r, err := Rewrite(template, "sha256:"+strings.Repeat("ab", 32))
if err != nil {
t.Fatal(err)
}
if _, err := RewriteRoot(&r, RootCredentials{Store: "s", Broker: "b"}); err != nil {
t.Fatal(err)
}
return r
}
func containerNamed(d *declaration.Declaration, name string) *declaration.Container {
for _, r := range d.Resources {
if c, ok := r.(*declaration.Container); ok && c.Name == name {
return c
}
}
return nil
}
func TestTheDefaultPortsLeaveTheBundleAsItWas(t *testing.T) {
r := producedBundle(t)
before := string(r.Bundle)
got, err := RewritePorts(&r, DefaultPorts(), DefaultOverlayRange)
if err != nil {
t.Fatal(err)
}
if got.Places != 0 || string(r.Bundle) != before {
t.Errorf("the default ports rewrote %d place(s)", got.Places)
}
}
func TestGivenPortsMoveOnlyTheMachinesSide(t *testing.T) {
r := producedBundle(t)
p := FoundationPorts{Store: 5433, Bus: 5771, AMQP: 5772, Management: 15673, Registry: 5100}
got, err := RewritePorts(&r, p, "10.77.0.0/16")
if err != nil {
t.Fatal(err)
}
if got.Places == 0 {
t.Fatal("nothing was rewritten")
}
storeC := containerNamed(r.Declaration, "mesh-store")
if len(storeC.Ports) != 1 || storeC.Ports[0] != "5433:5432" {
t.Errorf("the store publishes %v", storeC.Ports)
}
broker := containerNamed(r.Declaration, "mesh-broker")
if strings.Join(broker.Ports, " ") != "5771:5671 5772:5672 127.0.0.1:15673:15672" {
t.Errorf("the broker publishes %v", broker.Ports)
}
control, err := controlPlaneIn(r.Declaration)
if err != nil {
t.Fatal(err)
}
for key, value := range control.Env {
if strings.HasPrefix(key, "MESH_STORE_") && !strings.Contains(value, "@127.0.0.1:5433/") {
t.Errorf("%s still dials %s", key, value)
}
}
if !strings.Contains(control.Env["MESH_BROKER_AMQP"], "@127.0.0.1:5772/") ||
!strings.HasSuffix(control.Env["MESH_BROKER_MANAGEMENT"], "@127.0.0.1:15673") {
t.Errorf("the broker is dialled at %s and %s", control.Env["MESH_BROKER_AMQP"], control.Env["MESH_BROKER_MANAGEMENT"])
}
if control.Env["MESH_BROKER_ADDRESS"] != "192.0.2.10:5771" || r.BrokerAddress != "192.0.2.10:5771" {
t.Errorf("nodes are told to dial %s (%s)", control.Env["MESH_BROKER_ADDRESS"], r.BrokerAddress)
}
if control.Env["MESH_OVERLAY_CIDR"] != "10.77.0.0/16" {
t.Errorf("the control plane is told the range %q", control.Env["MESH_OVERLAY_CIDR"])
}
// The schema step reaches the store inside its own network, on the container's port.
text := string(r.Bundle)
if !strings.Contains(text, `MESH_STORE_INVENTORY=postgres://postgres:s@127.0.0.1:5432/inventory`) {
t.Error("the schema step's connection, inside the store's network, was moved off the container's port")
}
for _, want := range []string{"tcp dport 5771 accept", "ct original proto-dst 5771 accept",
"tcp dport 5100 accept", "ct original proto-dst 5100 accept"} {
if !strings.Contains(text, want) {
t.Errorf("the base filter does not say %q", want)
}
}
if strings.Contains(text, "dport 5671 accept") || strings.Contains(text, "dport 5000 accept") {
t.Error("the base filter still admits a default port")
}
}
func TestATemplateThatDoesNotSayItsPortsAsExpectedIsRefused(t *testing.T) {
r := producedBundle(t)
r.Bundle = []byte(strings.Replace(string(r.Bundle), `"ports": ["5432:5432"]`, `"ports": [ "5432:5432" ]`, 1))
if _, err := RewritePorts(&r, FoundationPorts{Store: 5433}, ""); err == nil {
t.Error("a store port the installer could not find was silently left")
}
}
func TestTwoThingsOnOnePortAreRefused(t *testing.T) {
p := DefaultPorts()
p.Registry = p.Store
if err := p.Check(); err == nil {
t.Error("the registry and the store were both given one port")
}
p = DefaultPorts()
p.Hub = 5432 // udp, beside the store's tcp: two different ports
if err := p.Check(); err != nil {
t.Errorf("a udp port beside a tcp one of the same number was refused: %v", err)
}
}
// machineRunner answers ss, docker ps, docker inspect and ip from fixtures.
type machineRunner struct {
ss, ps, addrs, routes, wg string
unlabelled map[string]bool
labelled map[string]bool
}
func (m machineRunner) run(_ context.Context, name string, args ...string) (string, error) {
switch {
case name == "ss":
return m.ss, nil
case name == "docker" && args[0] == "ps":
return m.ps, nil
case name == "docker" && args[0] == "container":
n := args[len(args)-1]
if m.labelled[n] {
return "abc\n", nil
}
if m.unlabelled[n] {
return "\n", nil
}
return "", errors.New("no such container")
case name == "ip" && args[1] == "addr":
return m.addrs, nil
case name == "ip" && args[1] == "route":
return m.routes, nil
case name == "wg":
return m.wg, nil
}
return "", nil
}
func noneOurs(reachable.Reach) bool { return false }
func TestABusyPortIsRefusedNamingItsHolder(t *testing.T) {
m := machineRunner{
ss: "tcp LISTEN 0 4096 0.0.0.0:5000 0.0.0.0:* users:((\"docker-proxy\",pid=1,fd=7))\n" +
"tcp LISTEN 0 4096 127.0.0.1:15672 0.0.0.0:* users:((\"beam.smp\",pid=2,fd=7))\n",
ps: "predecessor-registry\t0.0.0.0:5000->5000/tcp\n",
}
err := PortsFree(context.Background(), m.run, DefaultPorts(), noneOurs)
if err == nil {
t.Fatal("held ports were not refused")
}
for _, want := range []string{"predecessor-registry", "beam.smp", "tcp/5000", "tcp/15672", "--registry-port"} {
if !strings.Contains(err.Error(), want) {
t.Errorf("the refusal does not say %q: %v", want, err)
}
}
p := DefaultPorts()
p.Registry, p.Management = 5100, 15673
if err := PortsFree(context.Background(), m.run, p, noneOurs); err != nil {
t.Errorf("other ports given and still refused: %v", err)
}
}
func TestTheFoundationsOwnContainersAreNotCountedOnARerun(t *testing.T) {
m := machineRunner{
ss: "tcp LISTEN 0 4096 0.0.0.0:5432 0.0.0.0:* users:((\"docker-proxy\",pid=1,fd=7))\n",
ps: "mesh-store\t0.0.0.0:5432->5432/tcp\n",
}
ours := func(r reachable.Reach) bool { return r.By == "mesh-store" }
if err := PortsFree(context.Background(), m.run, DefaultPorts(), ours); err != nil {
t.Errorf("the foundation's own store was counted as holding its port: %v", err)
}
}
func TestAnOverlappingTunnelIsRefusedNamingItsInterface(t *testing.T) {
m := machineRunner{
addrs: "1: lo inet 127.0.0.1/8 scope host lo\n5: wg0 inet 10.42.3.1/24 scope global wg0\n7: mesh0 inet 10.42.0.1/16 scope global mesh0\n",
routes: "default via 192.0.2.1 dev eth0\n10.42.3.0/24 dev wg0 proto kernel scope link src 10.42.3.1\n",
}
err := OverlayClear(context.Background(), m.run, "")
if err == nil || !strings.Contains(err.Error(), "wg0") || strings.Contains(err.Error(), "mesh0") {
t.Fatalf("the overlap was not named by its interface alone: %v", err)
}
if err := OverlayClear(context.Background(), m.run, "10.77.0.0/16"); err != nil {
t.Errorf("a clear range was refused: %v", err)
}
}
func TestAPredecessorsContainerUnderTheMeshsNameIsRefused(t *testing.T) {
m := machineRunner{unlabelled: map[string]bool{"mesh-registry": true}, labelled: map[string]bool{"mesh-store": true}}
err := NamesFree(context.Background(), m.run, []string{"mesh-store", "mesh-registry", "mesh-broker"}, store.State{})
if err == nil || !strings.Contains(err.Error(), "mesh-registry") || strings.Contains(err.Error(), "mesh-store") {
t.Fatalf("names: %v", err)
}
known := store.State{Resources: []store.Applied{{ID: "x", Type: "container", Target: "mesh-registry"}}}
if err := NamesFree(context.Background(), m.run, []string{"mesh-registry"}, known); err != nil {
t.Errorf("a container this node has a record of was refused: %v", err)
}
}
// controlRecorder is a control plane that answers everything and writes down what it was told,
// with the content of every file carried to it, by the name it was carried under.
type controlRecorder struct {
told []string
settings map[string]string
}
func (c *controlRecorder) run(_ context.Context, name string, args ...string) (string, error) {
if name == "docker" && args[0] == "cp" {
raw, _ := os.ReadFile(args[1])
c.settings[filepath.Base(args[2])] = string(raw)
return "", nil
}
if name == "docker" && args[0] == "exec" {
c.told = append(c.told, strings.Join(args[3:], " "))
}
return "", nil
}
func (c *controlRecorder) index(prefix string) int {
for i, t := range c.told {
if strings.HasPrefix(t, prefix) {
return i
}
}
return -1
}
func TestTheNodesPortsAreSetBeforeTheModuleIsPushed(t *testing.T) {
t.Setenv("TMPDIR", t.TempDir())
c := &controlRecorder{settings: map[string]string{}}
control := controlPlane{container: "temp-mesh-controller", run: c.run, timeout: time.Second}
o := Options{Node: "anchor", Ports: FoundationPorts{Registry: 5100}, Wait: time.Second}
if _, err := installModule(context.Background(), o, control, RegistryModule, []byte(`{}`), quietly); err != nil {
t.Fatal(err)
}
set, push := c.index("settings set distribution"), c.index("push anchor")
if set < 0 || push < 0 || set > push {
t.Fatalf("settings were not set before the push: %v", c.told)
}
if add := c.index("module add"); add > set {
t.Errorf("settings were set before the module existed: %v", c.told)
}
if !strings.Contains(c.told[set], "--node anchor") {
t.Errorf("the settings are not the node's: %s", c.told[set])
}
if got := c.settings["distribution-settings.json"]; got != `{"ports":{"5000":5100}}` {
t.Errorf("the registry was told %s", got)
}
}
func TestAGenesisOnTheDefaultsSetsNoSettings(t *testing.T) {
t.Setenv("TMPDIR", t.TempDir())
c := &controlRecorder{settings: map[string]string{}}
control := controlPlane{container: "temp-mesh-controller", run: c.run, timeout: time.Second}
o := Options{Node: "anchor", Wait: time.Second}
if _, err := installModule(context.Background(), o, control, RegistryModule, []byte(`{}`), quietly); err != nil {
t.Fatal(err)
}
if c.index("settings") >= 0 {
t.Errorf("a converged genesis on the default ports set settings: %v", c.told)
}
}
func TestARerunOfGenesisFindsItsOwnPackageRegistry(t *testing.T) {
// Raised by genesis itself with no label and no record, so a re-run finds it unlabelled.
m := machineRunner{unlabelled: map[string]bool{giteaBootstrap: true}}
rerun := store.State{Resources: []store.Applied{{ID: "store", Type: "container", Target: "mesh-store"}}}
if err := NamesFree(context.Background(), m.run, []string{giteaBootstrap, "mesh-store"}, rerun); err != nil {
t.Errorf("a re-run refused the package registry genesis raised: %v", err)
}
// On a machine genesis never ran on, a container under that name is a predecessor's.
if err := NamesFree(context.Background(), m.run, []string{giteaBootstrap}, store.State{}); err == nil {
t.Error("a container under the package registry's name on a fresh machine was not refused")
}
}
// novox/hq ADR 0105: an adopted genesis takes over the tunnel it finds — its port is the hub's,
// its range the mesh's, and neither is refused for being held by it.
func TestAnAdoptedGenesisSettlesOnTheTunnelItFinds(t *testing.T) {
private, err := aFoundKey()
if err != nil {
t.Fatal(err)
}
conf := "[Interface]\nPrivateKey = " + private + "\nListenPort = 51900\nAddress = 192.0.2.1/24\n" +
"[Peer]\nPublicKey = PEER=\nAllowedIPs = 192.0.2.2/32\n"
tunnel.ReadFile = func(path string) ([]byte, error) {
if path == tunnel.ConfigDir+"/wg0.conf" {
return []byte(conf), nil
}
return nil, errors.New("no such file")
}
t.Cleanup(func() { tunnel.ReadFile = os.ReadFile })
m := machineRunner{
wg: "wg0\n",
ss: "udp UNCONN 0 0 0.0.0.0:51900 0.0.0.0:*\n",
addrs: "5: wg0 inet 192.0.2.1/24 scope global wg0\n",
routes: "192.0.2.0/24 dev wg0 proto kernel scope link src 192.0.2.1\n",
}
o := Options{Adopted: true, Ports: DefaultPorts(), OverlayRange: DefaultOverlayRange, State: filepath.Join(t.TempDir(), "state.json")}
found, err := TakeTheTunnel(&o, m.run)
if err != nil || found == nil {
t.Fatalf("the tunnel was not found and taken: %+v %v", found, err)
}
if o.Tunnel != "wg0" || o.Ports.Hub != 51900 || o.OverlayRange != "192.0.2.0/24" {
t.Fatalf("genesis did not settle on the tunnel's port and range: %+v", o)
}
// Its port is held by the tunnel and its range overlaps the tunnel's — by design, not refused.
if err := CheckTheMachine(context.Background(), o, m.run, producedBundle(t).Declaration, func(string) {}); err != nil {
t.Fatalf("the machine was refused for the tunnel it takes over: %v", err)
}
// Whereas the same machine not taking it over is refused on both counts (ADR 0100).
plain := o
plain.Tunnel = ""
if err := CheckTheMachine(context.Background(), plain, m.run, producedBundle(t).Declaration, func(string) {}); err == nil ||
!strings.Contains(err.Error(), "51900") {
t.Fatalf("a tunnel not taken over stopped being refused for holding the hub's port: %v", err)
}
plain.Ports.Hub = 51821
if err := CheckTheMachine(context.Background(), plain, m.run, producedBundle(t).Declaration, func(string) {}); err == nil ||
!strings.Contains(err.Error(), "wg0") {
t.Fatalf("a tunnel not taken over stopped being refused for overlapping the range: %v", err)
}
// Numbers that disagree with the tunnel are refused, naming the tunnel's.
for name, given := range map[string]Options{
"--hub-port": {Adopted: true, Ports: FoundationPorts{Hub: 51821}, OverlayRange: DefaultOverlayRange},
"--overlay-range": {Adopted: true, Ports: DefaultPorts(), OverlayRange: "10.77.0.0/16"},
} {
if _, err := TakeTheTunnel(&given, m.run); err == nil || !strings.Contains(err.Error(), name) {
t.Errorf("a %s disagreeing with the tunnel was accepted: %v", name, err)
}
}
// And no tunnel up is an ordinary machine.
m.wg = "mesh0\n"
none := Options{Adopted: true, Ports: DefaultPorts(), OverlayRange: DefaultOverlayRange}
if found, err := TakeTheTunnel(&none, m.run); err != nil || found != nil || none.Ports.Hub != DefaultPorts().Hub {
t.Errorf("a machine with no tunnel was not left as it was: %+v %v", found, err)
}
}
func aFoundKey() (string, error) {
k, err := ecdh.X25519().GenerateKey(rand.Reader)
if err != nil {
return "", err
}
return base64.StdEncoding.EncodeToString(k.Bytes()), nil
}
+5
View File
@@ -105,6 +105,11 @@ func Preflight(ctx context.Context, o Options, d Deps, say func(string)) ([]byte
if err := waitForRuntime(ctx, d.Run, o.Timeout, o.Wait, say); err != nil {
return nil, err
}
// A converged genesis refuses a machine in use (novox/hq ADR 0100) — asked once the runtime
// answers, so what it runs can be counted, and before anything changes.
if err := RefuseAMachineInUse(ctx, o, d.Run, say); err != nil {
return nil, err
}
// 4. Can this machine reach what the bundle's images come from?
//
+1 -1
View File
@@ -150,7 +150,7 @@ func digestOf(ctx context.Context, o Options, d Deps, remote string) (string, er
// The registry has it. What digest, according to the runtime that pushed it.
reading, cancel := context.WithTimeout(ctx, o.Timeout)
out, err := d.Run(reading, "docker", "inspect", "--format", "{{json .RepoDigests}}",
out, err := d.Run(reading, "docker", "image", "inspect", "--format", "{{json .RepoDigests}}",
remote+":"+genesisTag)
cancel()
if err != nil {
+4 -4
View File
@@ -50,7 +50,7 @@ func TestAnImageNoRegistryHasEverHeldIsPushed(t *testing.T) {
case "push":
pushed = true
return "", nil
case "inspect":
case "image":
return `["127.0.0.1:5000/mesh-controller@sha256:` + strings.Repeat("a", 64) + `"]`, nil
}
return "", fmt.Errorf("unexpected: %v", args)
@@ -80,7 +80,7 @@ func TestAnImageTheRegistryAlreadyServesIsNotPushedAgain(t *testing.T) {
return http.StatusOK, `{"name":"mesh-controller","tags":["genesis"]}`, nil
},
func(_ string, args []string) (string, error) {
if args[0] == "inspect" {
if args[0] == "image" {
return `["127.0.0.1:5000/mesh-controller@sha256:` + strings.Repeat("b", 64) + `"]`, nil
}
return "", fmt.Errorf("unexpected: %v", args)
@@ -111,7 +111,7 @@ func TestTheDigestComesFromThisMeshsOwnRegistry(t *testing.T) {
return http.StatusOK, `{"tags":["genesis"]}`, nil
},
func(_ string, args []string) (string, error) {
if args[0] == "inspect" {
if args[0] == "image" {
return `["` + elsewhere + `","` + ours + `"]`, nil
}
return "", fmt.Errorf("unexpected: %v", args)
@@ -166,7 +166,7 @@ func TestATagIsNotAPin(t *testing.T) {
return http.StatusOK, `{"tags":["genesis"]}`, nil
},
func(_ string, args []string) (string, error) {
if args[0] == "inspect" {
if args[0] == "image" {
return `["127.0.0.1:5000/mesh-controller:genesis"]`, nil
}
return "", nil
+1 -1
View File
@@ -62,7 +62,7 @@ func aMeshThatAgrees(answers map[string]string) func(string, []string) (string,
return "", fmt.Errorf("unexpected program %q", name)
case args[0] == "cp":
return "", nil
case args[0] == "inspect":
case args[0] == "container":
return "true running\n", nil
case args[0] == "exec":
return "", nil
+19 -8
View File
@@ -89,7 +89,7 @@ func RetireTheTemporaryControlPlane(ctx context.Context, o Options, sys system.S
// same state file. What makes this a removal rather than a no-op is that the state file
// records the container as something this installer applied, and the declaration no longer
// asks for it.
report, err := ApplyBundle(ctx, o, sys, without, run, say)
report, err := ApplyBundle(ctx, o, sys, without, bundle, run, say)
if err != nil {
return out, fmt.Errorf(
"%w\n\nThe permanent control plane is running and the temporary one is still here. "+
@@ -132,18 +132,29 @@ func RetireTheTemporaryControlPlane(ctx context.Context, o Options, sys system.S
// where the comment explaining it lives. A comment that outlives the thing it describes is worse
// than no comment: it is the file telling somebody the machine has a control plane it does not.
func removeResource(bundle []byte, id string) ([]byte, error) {
previous, from, to, err := resourceAt(bundle, id)
if err != nil {
return nil, err
}
return cut(bundle, previous, from, to), nil
}
// resourceAt finds one resource's object in a bundle's text by its id: where the one before it
// ended, and where it starts and ends — comments and strings skipped, so an id quoted in a comment
// or a command is never mistaken for the resource.
func resourceAt(bundle []byte, id string) (previous, from, to int, err error) {
array := indexOutsideStrings(bundle, `"resources"`)
if array < 0 {
return nil, fmt.Errorf("this bundle has no resources array, so there is nothing to take out of it")
return 0, 0, 0, fmt.Errorf("this bundle has no resources array, so there is nothing to take out of it")
}
open := indexOutsideStrings(bundle[array:], "[")
if open < 0 {
return nil, fmt.Errorf("this bundle's resources are not a list")
return 0, 0, 0, fmt.Errorf("this bundle's resources are not a list")
}
open += array
depth, from := 0, -1
previous := open
depth := 0
from, previous = -1, open
inString, escaped, inLine, inBlock := false, false, false, false
for i := open + 1; i < len(bundle); i++ {
c := bundle[i]
@@ -181,16 +192,16 @@ func removeResource(bundle []byte, id string) ([]byte, error) {
break
}
if isResource(bundle[from:i+1], id) {
return cut(bundle, previous, from, i+1), nil
return previous, from, i + 1, nil
}
previous = i + 1
from = -1
case c == ']' && depth == 0:
return nil, fmt.Errorf(
return 0, 0, 0, fmt.Errorf(
"this bundle declares no %q, so there is nothing to take out of it", id)
}
}
return nil, fmt.Errorf("this bundle's resources list does not end")
return 0, 0, 0, fmt.Errorf("this bundle's resources list does not end")
}
// isResource reports whether one resource's text is the one wanted.
+9 -1
View File
@@ -102,8 +102,16 @@ func keptOrMade(path string, dryRun bool) (value string, made bool, err error) {
}
// Written whole and renamed into place, at 0600, owned by whoever runs the installer — root,
// which is also who the host runs as when it later writes the sealed copy here.
//
// **The value alone, no line ending.** The module that adopts the store declares this same
// file, and what it declares is the value as `secret accept` took it — its line ending gone,
// by design. The host folds a mounted file's content into the container's spec (novox/hq
// 04-ISSUES/103), so a genesis that wrote `value\n` here would raise a store whose label
// digests one byte more than the module's file, and phase three would RECREATE the store it
// meant to adopt in place, with the temporary control plane connected to it. readCredentialFile
// tolerates either ending, so a file an earlier genesis wrote still reads.
tmp := path + ".genesis"
if err := os.WriteFile(tmp, []byte(value+"\n"), 0o600); err != nil {
if err := os.WriteFile(tmp, []byte(value), 0o600); err != nil {
return "", false, err
}
if err := os.Rename(tmp, path); err != nil {
+47
View File
@@ -165,3 +165,50 @@ func TestTheBrokerAdminMarkerHasALineEnding(t *testing.T) {
}
}
}
// The foundation filters before anything listens: nftables is in place before the store, and its
// rules refuse the store's and broker's client ports from outside while keeping ssh, the bus a node
// enrols over and the registry a node pulls from (novox/hq issue 054).
func TestTheFoundationFiltersBeforeAnythingListens(t *testing.T) {
template, err := os.ReadFile("../../examples/foundation-first-node.lock")
if err != nil {
t.Skip("no example bundle beside this checkout")
}
r, err := Rewrite(template, "sha256:"+strings.Repeat("ab", 32))
if err != nil {
t.Fatal(err)
}
if _, err := RewriteRoot(&r, RootCredentials{Store: "s", Broker: "b"}); err != nil {
t.Fatal(err)
}
var filterAt, loadedAt, storeAt, brokerAt = -1, -1, -1, -1
var rules string
for i, res := range r.Declaration.Resources {
switch res.Identity() {
case "base-filter":
filterAt = i
rules = res.(*declaration.File).Content
case "base-filter-loaded":
loadedAt = i
case "store":
storeAt = i
case "broker":
brokerAt = i
}
}
if !(filterAt >= 0 && filterAt < loadedAt && loadedAt < storeAt && storeAt < brokerAt) {
t.Fatalf("order: filter %d loaded %d store %d broker %d", filterAt, loadedAt, storeAt, brokerAt)
}
for _, want := range []string{"policy drop", "tcp dport 22 accept", "ct original proto-dst 5671 accept",
"ct original proto-dst 5000 accept", "tcp dport 5671 accept", "tcp dport 5000 accept",
"ip saddr 172.16.0.0/12 accept", "table inet mesh"} {
if !strings.Contains(rules, want) {
t.Errorf("the base filter lacks %q", want)
}
}
for _, mustNot := range []string{"5432", "5672"} {
if strings.Contains(rules, mustNot) {
t.Errorf("the base filter names %s, which must be reached from the machine and its containers only", mustNot)
}
}
}
+12
View File
@@ -0,0 +1,12 @@
udp UNCONN 0 0 0.0.0.0:5353 0.0.0.0:* users:(("systemd-resolve",pid=262,fd=17))
udp UNCONN 0 0 0.0.0.0:5355 0.0.0.0:* users:(("systemd-resolve",pid=262,fd=13))
udp UNCONN 0 0 127.0.0.54:53 0.0.0.0:* users:(("systemd-resolve",pid=262,fd=24))
udp UNCONN 0 0 127.0.0.53%lo:53 0.0.0.0:* users:(("systemd-resolve",pid=262,fd=22))
udp UNCONN 0 0 [::]:5353 [::]:* users:(("systemd-resolve",pid=262,fd=18))
udp UNCONN 0 0 [::]:5355 [::]:* users:(("systemd-resolve",pid=262,fd=15))
udp UNCONN 0 0 [fe80::1266:6aff:fe24:628d]%enp5s0:546 [::]:* users:(("systemd-network",pid=272,fd=36))
tcp LISTEN 0 4096 127.0.0.1:39473 0.0.0.0:* users:(("containerd",pid=394,fd=14))
tcp LISTEN 0 4096 0.0.0.0:5355 0.0.0.0:* users:(("systemd-resolve",pid=262,fd=14))
tcp LISTEN 0 4096 127.0.0.53%lo:53 0.0.0.0:* users:(("systemd-resolve",pid=262,fd=23))
tcp LISTEN 0 4096 127.0.0.54:53 0.0.0.0:* users:(("systemd-resolve",pid=262,fd=25))
tcp LISTEN 0 4096 [::]:5355 [::]:* users:(("systemd-resolve",pid=262,fd=16))
+3 -2
View File
@@ -129,8 +129,9 @@ func containerRunning(ctx context.Context, run Runner, probe time.Duration, name
defer cancel()
// Both facts in one answer, so a container that is not running is reported with what it IS
// rather than with the absence of what it should be.
out, err := run(asking, "docker", "inspect", "--format", "{{.State.Running}} {{.State.Status}}", name)
// rather than with the absence of what it should be. `container inspect`, not the bare form:
// a same-named network or volume would otherwise answer in the container's place.
out, err := run(asking, "docker", "container", "inspect", "--format", "{{.State.Running}} {{.State.Status}}", name)
if err != nil {
return containerState{}, fmt.Errorf(
"the container %q is not there at all, and the apply reported it applied: %w", name, err)
+7 -7
View File
@@ -30,7 +30,7 @@ func TestAContainerThatIsUpIsNotAControlPlaneThatReplies(t *testing.T) {
runtime := &asked{answer: func(_ string, args []string) (string, error) {
switch args[0] {
case "inspect":
case "container":
return "true running\n", nil
case "exec":
// Up, and saying nothing. The program inside is not answering.
@@ -59,7 +59,7 @@ func TestAControlPlaneThatSaysNothingHasNotAnswered(t *testing.T) {
defer func() { answerEvery = previous }()
runtime := &asked{answer: func(_ string, args []string) (string, error) {
if args[0] == "inspect" {
if args[0] == "container" {
return "true running\n", nil
}
return " \n", nil
@@ -74,7 +74,7 @@ func TestAControlPlaneThatSaysNothingHasNotAnswered(t *testing.T) {
// The foundation answering is the whole point, and what it said is reported rather than asserted.
func TestAFoundationThatIsUpAndAnsweringIsAccepted(t *testing.T) {
runtime := &asked{answer: func(_ string, args []string) (string, error) {
if args[0] == "inspect" {
if args[0] == "container" {
return "true running\n", nil
}
return "1 node, 0 waiting\n", nil
@@ -112,7 +112,7 @@ func TestAControlPlaneThatIsStillStartingIsWaitedFor(t *testing.T) {
attempts := 0
runtime := &asked{answer: func(_ string, args []string) (string, error) {
if args[0] == "inspect" {
if args[0] == "container" {
return "true running\n", nil
}
attempts++
@@ -132,10 +132,10 @@ func TestAControlPlaneThatIsStillStartingIsWaitedFor(t *testing.T) {
// than being told only that something is not what it should be.
func TestAContainerThatExitedIsNamedWithItsState(t *testing.T) {
runtime := &asked{answer: func(_ string, args []string) (string, error) {
if args[0] == "inspect" && args[len(args)-1] == "mesh-broker" {
if args[0] == "container" && args[len(args)-1] == "mesh-broker" {
return "false exited\n", nil
}
if args[0] == "inspect" {
if args[0] == "container" {
return "true running\n", nil
}
return "", fmt.Errorf("unexpected command: %v", args)
@@ -155,7 +155,7 @@ func TestAContainerThatExitedIsNamedWithItsState(t *testing.T) {
// `FROM scratch` and has no shell for a command line to be interpreted by.
func TestTheControlPlaneIsAskedByRunningItsOwnBinary(t *testing.T) {
runtime := &asked{answer: func(_ string, args []string) (string, error) {
if args[0] == "inspect" {
if args[0] == "container" {
return "true running\n", nil
}
return "1 node\n", nil
+132
View File
@@ -0,0 +1,132 @@
package declaration
import (
"strings"
"testing"
)
// Defends novox/hq ADR 0100: every declaration says whether the node is adopted and which of its
// modules are taken, and the host refuses one it cannot read that from unambiguously.
const adoptedResources = `"resources":[
{"id":"hello-web.page","type":"file","path":"/var/lib/hello-web/index.html","content":"a\n"},
{"id":"hello-web.server","type":"container","name":"hello-web","image":"sha256:` + sixtyFour + `"},
{"id":"hello-web.data","type":"directory","path":"/var/lib/hello-web"}
]`
const sixtyFour = "0000000000000000000000000000000000000000000000000000000000000000"
func TestAnAdoptionIsReadWithTheDeclaration(t *testing.T) {
d, err := Parse([]byte(`{"adoption":{"taken":["postgres"],"untaken":{"hello-web":["hello-web.page","hello-web.server"]}},
"declaration":1,` + adoptedResources + `}`))
if err != nil {
t.Fatal(err)
}
if d.Adoption == nil {
t.Fatal("the adoption was dropped")
}
if len(d.Adoption.Taken) != 1 || d.Adoption.Taken[0] != "postgres" {
t.Errorf("taken read as %v", d.Adoption.Taken)
}
if module, ok := d.Adoption.UntakenModuleOf("hello-web.server"); !ok || module != "hello-web" {
t.Errorf("the container's untaken module read as %q, %v", module, ok)
}
if _, ok := d.Adoption.UntakenModuleOf("hello-web.data"); ok {
t.Error("a resource the adoption does not name was said to be untaken")
}
}
func TestADeclarationWithNoAdoptionIsConverged(t *testing.T) {
d, err := Parse([]byte(`{"declaration":1,` + adoptedResources + `}`))
if err != nil {
t.Fatal(err)
}
if d.Adoption != nil {
t.Errorf("a declaration saying nothing about adoption read as adopted: %+v", d.Adoption)
}
if _, ok := d.Adoption.UntakenModuleOf("hello-web.page"); ok {
t.Error("a converged node has an untaken module")
}
}
func TestAnAdoptionNamingAnUnknownIDIsRefused(t *testing.T) {
refusal := refusalFor(t, `{"adoption":{"taken":[],"untaken":{"hello-web":["hello-web.missing"]}},
"declaration":1,`+adoptedResources+`}`)
if !strings.Contains(strings.Join(refusal.Problems, "\n"), "hello-web.missing") {
t.Errorf("the unknown id was not named: %v", refusal.Problems)
}
}
func TestAnAdoptionMayNameAResourceOfAnyKind(t *testing.T) {
// A directory, a service or an action can reach what was found as surely as a file can, so
// the controller lists every resource of an untaken module (novox/hq ADR 0103).
d, err := Parse([]byte(`{"adoption":{"taken":[],"untaken":{"hello-web":["hello-web.data"]}},
"declaration":1,` + adoptedResources + `}`))
if err != nil {
t.Fatalf("a directory of an untaken module was refused: %v", err)
}
if module, ok := d.Adoption.UntakenModuleOf("hello-web.data"); !ok || module != "hello-web" {
t.Errorf("the directory is not its module's: %q %v", module, ok)
}
}
func TestAnIDUnderTwoModulesIsRefused(t *testing.T) {
refusal := refusalFor(t, `{"adoption":{"taken":[],"untaken":{"a":["hello-web.page"],"b":["hello-web.page"]}},
"declaration":1,`+adoptedResources+`}`)
if !strings.Contains(strings.Join(refusal.Problems, "\n"), "both") {
t.Errorf("an id under two modules was accepted: %v", refusal.Problems)
}
}
func TestAModuleBothTakenAndUntakenIsRefused(t *testing.T) {
refusal := refusalFor(t, `{"adoption":{"taken":["hello-web"],"untaken":{"hello-web":["hello-web.page"]}},
"declaration":1,`+adoptedResources+`}`)
if !strings.Contains(strings.Join(refusal.Problems, "\n"), "both taken and untaken") {
t.Errorf("a module both taken and untaken was accepted: %v", refusal.Problems)
}
}
func TestTheMeshsOwnResourcesAreNeverUntaken(t *testing.T) {
refusal := refusalFor(t, `{"adoption":{"taken":[],"untaken":{"x":["adoption.guard"]}},
"declaration":1,"resources":[
{"id":"adoption.guard","type":"file","path":"/etc/mesh/guard.nft","content":"x"}]}`)
if !strings.Contains(strings.Join(refusal.Problems, "\n"), "belongs to no module") {
t.Errorf("an adoption. id was accepted as untaken: %v", refusal.Problems)
}
}
func TestAnAdoptionWithAnUnknownFieldIsRefused(t *testing.T) {
refusalFor(t, `{"adoption":{"taken":[],"held":["x"]},"declaration":1,`+adoptedResources+`}`)
}
func TestACarriedBundleCannotSayTheNodeIsAdopted(t *testing.T) {
_, err := ParseTrusted([]byte(`{"adoption":{"taken":[]},"declaration":1,` + adoptedResources + `}`))
if err == nil || !strings.Contains(err.Error(), "only the mesh can say") {
t.Fatalf("a bundle claiming adoption was not refused: %v", err)
}
}
// novox/hq ADR 0105: a service may take over a found tunnel, said whole and on an adopted node.
func TestTakingOverATunnelIsSaidWholeAndForARunningService(t *testing.T) {
adoptedWith := func(service string) error {
_, err := Parse([]byte(`{"declaration":1,"adoption":{"taken":[]},"resources":[` + service + `]}`))
return err
}
good := `{"id":"up","type":"service","unit":"wg-quick@mesh0","state":"running",
"takes-over":{"interface":"wg0","unit":"wg-quick@wg0","config":"/etc/wireguard/wg0.conf"}}`
if err := adoptedWith(good); err != nil {
t.Fatalf("a whole takeover on an adopted node was refused: %v", err)
}
for name, bad := range map[string]string{
"its own unit": `{"id":"up","type":"service","unit":"wg-quick@wg0","state":"running",
"takes-over":{"interface":"wg0","unit":"wg-quick@wg0","config":"/etc/wireguard/wg0.conf"}}`,
"no config": `{"id":"up","type":"service","unit":"wg-quick@mesh0","state":"running",
"takes-over":{"interface":"wg0","unit":"wg-quick@wg0"}}`,
"a stopped service": `{"id":"up","type":"service","unit":"wg-quick@mesh0","state":"stopped",
"takes-over":{"interface":"wg0","unit":"wg-quick@wg0","config":"/etc/wireguard/wg0.conf"}}`,
} {
if err := adoptedWith(bad); err == nil {
t.Errorf("a takeover naming %s was accepted", name)
}
}
}
+52
View File
@@ -0,0 +1,52 @@
package declaration
import (
"strings"
"testing"
)
// Defends novox/hq issue 128: a file written into a block carries only its lines, in content,
// never a line the host keeps as its own marker, and says where a new region goes only as a block.
func TestAFileWrittenIntoABlockIsRefusedUnlessItIsOnlyItsLines(t *testing.T) {
for name, c := range map[string]struct{ resource, refusal string }{
"a begin marker in content": {`{"id":"f","type":"file","path":"/etc/hosts","into":"block","content":"a\n# BEGIN mesh f\nb\n"}`, "# BEGIN mesh f"},
"an end marker in content": {`{"id":"f","type":"file","path":"/etc/hosts","into":"block","content":"a\n# END mesh other\n"}`, "# END mesh other"},
"a marker with a CR": {`{"id":"f","type":"file","path":"/etc/hosts","into":"block","content":"# BEGIN mesh x\r\n"}`, "marker"},
"sealed": {`{"id":"f","type":"file","path":"/etc/hosts","into":"block","sealed":"abc"}`, "not sealed"},
"bytes": {`{"id":"f","type":"file","path":"/etc/hosts","into":"block","bytes":"YQ=="}`, "not sealed, bytes"},
"secrets": {`{"id":"f","type":"file","path":"/etc/hosts","into":"block","content":"${secret:s}","secrets":{"s":"abc"}}`, "secrets"},
"create-once": {`{"id":"f","type":"file","path":"/etc/hosts","into":"block","content":"a","create-once":true}`, "create-once"},
"at on a whole file": {`{"id":"f","type":"file","path":"/etc/hosts","content":"a","at":"start"}`, `at "start"`},
"at on a JSON file": {`{"id":"f","type":"file","path":"/etc/x.json","into":"json","content":"{}","at":"end"}`, `at "end"`},
"at somewhere else": {`{"id":"f","type":"file","path":"/etc/hosts","into":"block","content":"a","at":"middle"}`, `"start" or "end"`},
"an id ending in a space": {`{"id":"f ","type":"file","path":"/etc/hosts","into":"block","content":"a"}`, "whitespace"},
"an unknown format": {`{"id":"f","type":"file","path":"/etc/hosts","into":"lines","content":"a"}`, `"json" or "block"`},
} {
_, err := Parse([]byte(`{"declaration":1,"resources":[` + c.resource + `]}`))
if err == nil || !strings.Contains(err.Error(), c.refusal) {
t.Errorf("%s: want a refusal naming %q, got %v", name, c.refusal, err)
}
}
}
func TestAFileWrittenIntoABlockIsRead(t *testing.T) {
d, err := Parse([]byte(`{"declaration":1,"resources":[
{"id":"mesh-wireguard.fact-node-names","type":"file","path":"/etc/hosts","into":"block","content":"10.42.0.1 ace\n# BEGIN devtool x\n"},
{"id":"dhcpcd.options","type":"file","path":"/etc/dhcpcd.conf","into":"block","content":"nohook resolv.conf\n","at":"start"},
{"id":"empty","type":"file","path":"/etc/x","into":"block","content":"","at":"end"}
]}`))
if err != nil {
t.Fatal(err)
}
if f := d.Resources[0].(*File); f.Into != IntoBlock || f.At != "" {
t.Errorf("read as into %q at %q", f.Into, f.At)
}
if f := d.Resources[1].(*File); f.At != AtStart {
t.Errorf("at was read as %q", f.At)
}
if begin, end := BlockMarkers("mesh-wireguard.fact-node-names"); begin != "# BEGIN mesh mesh-wireguard.fact-node-names" ||
end != "# END mesh mesh-wireguard.fact-node-names" {
t.Errorf("the markers are %q and %q", begin, end)
}
}
+587 -32
View File
@@ -11,10 +11,12 @@ import (
"encoding/json"
"fmt"
"io"
"net"
"reflect"
"regexp"
"slices"
"sort"
"strconv"
"strings"
)
@@ -78,6 +80,12 @@ const (
// cadence. Tools, hooks and event consumers are not separate modes: they are loaded by a tool
// host, which is itself a process that stays up.
TypeProcess Type = "process"
// TypeOpening is a port the mesh needs reachable on an adopted node, converged through the
// firewall found there in that firewall's own terms (novox/hq ADR 0100). A state, not a
// command: the host adds the rule it marks as the mesh's when it is missing, and removes only
// what it marked — which is what lets it travel over the link.
TypeOpening Type = "opening"
)
// Resource is one thing that should be true of the machine.
@@ -135,6 +143,48 @@ type File struct {
Content string `json:"content"`
Mode string `json:"mode,omitempty"`
// CreateOnce says the content is a seed: written when the file is absent, and left alone —
// content, mode and owner — whenever it is present.
//
// **Two intentions had one vocabulary** (novox/hq issue 035, ADR 0087). "This file has this
// content, for ever" is what an ordinary file says, and the host holds the machine to it. A
// module that needs a file to exist before a program first starts — an access list the
// program then persists into, a bootstrap configuration it rewrites — needs the other thing,
// and with only the first available, every reconcile restored the seed behind the running
// program and erased what had grown in it, reporting success. What grows in a seeded file
// is somebody else's work the mesh asked for; the mesh removes nothing it did not create
// (ADR 0030), and it does not overwrite that either.
CreateOnce bool `json:"create-once,omitempty"`
// Into says the file is shared with software the mesh did not install, and the content is
// the mesh's part of it: written into what is there, never over it (novox/hq ADR 0102). Only
// "json" is spoken — the content is a JSON object whose keys the host sets in the file's
// object, keeping every other key as it found it and recording what each of its keys held
// before, so undeclaring the file gives those back.
//
// "block" is the same idea for a file that is not structured (novox/hq issue 128): the
// content is the mesh's lines, and the host owns only the region between `# BEGIN mesh <id>`
// and `# END mesh <id>`, keeping every line outside it byte for byte. The machine's hosts file
// is the case that needed it — on a workstation the distribution, a local development tool and
// the operator all write into it, and the mesh writing it whole took their lines away at the
// next change to the mesh's names, silently. Marked blocks are the shape the other tools in
// that file already use, and `#` is the comment character of every file this serves.
//
// Written into, in either format, the file's mode and owner are the machine's: a declared mode
// and owner apply only to a file the host creates, and a file that was there keeps its own.
Into string `json:"into,omitempty"`
// At is where a file written into a block has its region added when the file does not hold
// one yet: "end", the default, or "start". A region already there stays where it is, whatever
// this says — moving it would move the lines around it, and those are the machine's.
//
// **Some files give a line its meaning by what stands above it.** dhcpcd's configuration scopes
// every line after `interface X` to that interface, and a real one ends with exactly that — an
// interface and its static address. A region added at the end would make the mesh's global
// options (`nohook resolv.conf`, `denyinterfaces mesh0`) options of one interface, and dhcpcd
// would read them without complaint. At the start, nothing stands above them.
At string `json:"at,omitempty"`
// Sealed is content encrypted to this node's sealing key, for a file the mesh must deliver
// without being able to read.
//
@@ -205,6 +255,58 @@ func (f *File) validate(where string, _ bool) []string {
if f.Path == "" {
problems = append(problems, where+": a file needs a path")
}
switch f.Into {
case "":
case IntoJSON:
var object map[string]json.RawMessage
if err := json.Unmarshal([]byte(f.Content), &object); err != nil || object == nil {
problems = append(problems, where+
": a file written into JSON carries a JSON object of the keys it sets")
}
case IntoBlock:
// The markers are how the host finds its region again. A marker in the content would be
// a second region, or the end of this one, the next time the file is read — and the host
// would then rewrite, or on undeclare take out, lines that were never the mesh's.
for _, line := range strings.Split(f.Content, "\n") {
if strings.HasPrefix(line, BlockBegin) || strings.HasPrefix(line, BlockEnd) {
problems = append(problems, fmt.Sprintf(
"%s: a file written into a block carries the mesh's lines, and %q is a marker the "+
"host keeps for itself", where, strings.TrimRight(line, "\r")))
break
}
}
// The id is written into the markers, so it has to stay on one line — and whitespace at
// either end of it is whitespace the host would have to match exactly in a line some
// editor may trim.
if strings.ContainsAny(f.ID, "\r\n") {
problems = append(problems, where+
": a file written into a block names its region by its id, and this id spans lines")
} else if strings.TrimSpace(f.ID) != f.ID {
problems = append(problems, where+
": a file written into a block names its region by its id, and this id begins or ends in whitespace")
}
default:
problems = append(problems, fmt.Sprintf(
"%s: into %q; a file is written into \"json\" or \"block\", or omits it to be written whole",
where, f.Into))
}
switch {
case f.At == "":
case f.Into != IntoBlock:
problems = append(problems, fmt.Sprintf(
"%s: at %q; only a file written into a block has a place its region is added", where, f.At))
case f.At != AtStart && f.At != AtEnd:
problems = append(problems, fmt.Sprintf(
"%s: at %q; a block is added at \"start\" or \"end\", or omits it to be added at the end",
where, f.At))
}
if f.Into == IntoJSON || f.Into == IntoBlock {
if f.Sealed != "" || f.Bytes != "" || len(f.Secrets) > 0 || f.CreateOnce {
problems = append(problems, where+
": a file written into says only its part, in content — not sealed, bytes, "+
"secrets or create-once")
}
}
var said []string
for name, value := range map[string]string{
"content": f.Content, "sealed": f.Sealed, "bytes": f.Bytes,
@@ -473,16 +575,34 @@ func (d *Process) Identity() string { return d.ID }
func (d *Process) Kind() Type { return TypeProcess }
func (d *Process) Target() string { return d.Name }
// ProcessNameProblem says what is wrong with a process name, or nothing.
//
// The name becomes a unit name, a file under the unit directory and a directory under the mesh's
// own — the one removing the process deletes, whole (novox/hq ADR 0118). So a name that is not one
// plain path element is refused: with a separator it writes somewhere nobody meant, and "." or
// ".." IS the mesh's directory or its parent — removing a process named ".." would delete every
// bundle the mesh has, and more. One with a leading dash is read by the service manager as an
// option, not a unit. Exported because the removal checks the recorded name again: a record is
// what the host wrote, and a host of an older version wrote it under looser rules.
func ProcessNameProblem(name string) string {
switch {
case name == "":
return "a process needs a name, which is what its unit is called"
case strings.ContainsAny(name, "/ \t"):
return "a process name becomes a unit name, so it cannot contain a path separator or a space"
case name == "." || name == "..":
return fmt.Sprintf("a process name becomes a directory under the mesh's own, and %q would be "+
"that directory or its parent", name)
case strings.HasPrefix(name, "-"):
return "a process name cannot begin with a dash: the service manager would read it as an option"
}
return ""
}
func (d *Process) validate(where string, _ bool) []string {
var problems []string
if d.Name == "" {
problems = append(problems, where+": a process needs a name, which is what its unit is called")
}
if strings.ContainsAny(d.Name, "/ \t") {
// It becomes a unit name and a file on disk. A name with a separator in it would write
// somewhere nobody meant.
problems = append(problems, where+": a process name becomes a unit name, so it cannot "+
"contain a path separator or a space")
if problem := ProcessNameProblem(d.Name); problem != "" {
problems = append(problems, where+": "+problem)
}
if d.Source == "" {
problems = append(problems, where+": a process needs somewhere to fetch its bundle from")
@@ -543,10 +663,20 @@ func (d *Process) validate(where string, _ bool) []string {
// (it will come back at boot), or disabled and running (started by hand, gone after a reboot).
// Folding them into one field would make the second expressible only by accident.
type Service struct {
ID string `json:"id"`
Type Type `json:"type"`
Unit string `json:"unit"`
State string `json:"state"`
ID string `json:"id"`
Type Type `json:"type"`
Unit string `json:"unit"`
// State is "running" or "stopped" — or absent, and then **the unit's lifecycle is the
// machine's; the mesh only reflects its triggers** (novox/hq ADR 0117). The uplink modules
// declare the machine's own network manager this way: the mesh writes into its configuration
// and needs it to read that again, and nothing more. Stated, the host would start the manager
// on a machine that uses another one — two managers fighting over the same links — and, when
// the module was unassigned, stop it: the machine's network, the channel the mesh itself
// arrives on, gone at the moment of a routine change. So a service without a state is never
// started, stopped, enabled or disabled, is reloaded or restarted only when a trigger changed
// and it is already running, and undeclared is simply forgotten. It says nothing unless it
// names a trigger, and it may not say boot or takes-over, which are both lifecycle.
State string `json:"state,omitempty"`
// Boot is "enabled" or "disabled" — whether the unit starts at boot. Optional: absent means
// the host asserts nothing about it and leaves whatever is there.
//
@@ -554,6 +684,16 @@ type Service struct {
// declaration that reports success and stops being true at the next power cut.
Boot string `json:"boot,omitempty"`
// Scope is whose service manager the unit belongs to: "system" (absent means system), or
// "user" — the operator account's own manager (novox/hq ADR 0177). A workstation's per-user
// daemons — a window manager's reload watcher, an audio mask, a memory guard — are units in
// that manager, and until this they had no form the mesh could send. A user-scoped unit names
// its User; the host talks to that account's manager and never starts a system unit by the
// same name.
Scope string `json:"scope,omitempty"`
// User is the account whose manager a user-scoped unit lives in. Required with scope "user",
// refused otherwise; a module names it ${machine:account} and never the person.
User string `json:"user,omitempty"`
// RestartOn names resources whose change means this service must be restarted.
//
// Because a running service does not re-read its configuration. Replace the file, find the
@@ -567,26 +707,190 @@ type Service struct {
// would be an action, and the link may not carry one (novox/hq ADR 0005) — so this is not a
// way around that rule, it is the shape the rule leaves.
RestartOn []string `json:"restart-on,omitempty"`
// ReloadOn names resources whose change means this service must be reloaded — for a service
// that re-reads its configuration when told to, where a restart would stop what it runs: the
// container runtime, whose restart stops every container on the machine (novox/hq ADR 0102).
// A change that is also in RestartOn restarts it, which covers a reload.
ReloadOn []string `json:"reload-on,omitempty"`
// TakesOver names the found tunnel this service replaces (novox/hq ADR 0105): before this unit
// is started, the named unit is stopped and disabled — never flushed — and its configuration
// file is kept like any held file — until the take is proven by a peer's handshake, and then
// retired, its original staying kept (novox/hq ADR 0119). Only on an adopted node, and only
// said by the controller, which knows the found tunnel's key is this node's own: without that,
// starting this unit on the found one's port would drop every peer's packets.
TakesOver *TakeOver `json:"takes-over,omitempty"`
}
// TakeOver is a found tunnel a service replaces: its interface, the unit that raised it, and its
// configuration file.
type TakeOver struct {
Interface string `json:"interface"`
Unit string `json:"unit"`
Config string `json:"config"`
}
// Stateless reports whether the unit's lifecycle is the machine's, and the mesh only reflects the
// service's triggers (novox/hq ADR 0117).
func (s *Service) Stateless() bool { return s.State == "" }
func (s *Service) Identity() string { return s.ID }
func (s *Service) Kind() Type { return TypeService }
func (s *Service) Target() string { return s.Unit }
// ScopeSystem and ScopeUser are the two managers a unit may belong to (novox/hq ADR 0177).
const (
ScopeSystem = "system"
ScopeUser = "user"
)
// UserScoped is whether this unit lives in an account's own service manager.
func (s *Service) UserScoped() bool { return s.Scope == ScopeUser }
func (s *Service) validate(where string, _ bool) []string {
var problems []string
if s.Unit == "" {
problems = append(problems, where+": a service needs a unit")
}
if s.State != "running" && s.State != "stopped" {
switch s.Scope {
case "", ScopeSystem:
if s.User != "" {
problems = append(problems, where+": a system unit names no user; only a user-scoped unit does")
}
case ScopeUser:
if s.User == "" {
problems = append(problems, where+": a user-scoped unit names the account whose manager it lives in")
}
default:
problems = append(problems, fmt.Sprintf(
"%s: state %q; a service is \"running\" or \"stopped\"", where, s.State))
"%s: scope %q; a unit is in the \"system\" manager or the operator account's \"user\" one", where, s.Scope))
}
switch {
case s.State == "running" || s.State == "stopped":
case s.State != "":
problems = append(problems, fmt.Sprintf(
"%s: state %q; a service is \"running\" or \"stopped\", or omits state to leave the "+
"unit's lifecycle to the machine", where, s.State))
case s.Boot != "" || s.TakesOver != nil:
problems = append(problems, where+": a service that omits state leaves the unit's lifecycle "+
"to the machine, and boot and takes-over are both its lifecycle")
case len(s.RestartOn) == 0 && len(s.ReloadOn) == 0:
problems = append(problems, where+": a service that omits state leaves the unit's lifecycle "+
"to the machine, and names no restart-on or reload-on — it declares nothing")
}
if s.Boot != "" && s.Boot != "enabled" && s.Boot != "disabled" {
problems = append(problems, fmt.Sprintf(
"%s: boot %q; a service is \"enabled\" or \"disabled\" at boot, or omits it to "+
"leave the machine's own setting alone", where, s.Boot))
}
if t := s.TakesOver; t != nil {
switch {
case t.Unit == "" || t.Config == "" || t.Interface == "":
problems = append(problems, where+": takes-over names the found tunnel's interface, unit "+
"and config, and this leaves one out")
case t.Unit == s.Unit:
problems = append(problems, fmt.Sprintf("%s: takes-over names %s, which is this service's own unit",
where, t.Unit))
case s.State != "running" && s.State != "":
problems = append(problems, where+": a service that takes over a tunnel is running — stopping "+
"the found one for a service that will not run would leave the peers with nothing")
}
}
return problems
}
// What a file is written into (novox/hq ADR 0102): a JSON object whose keys the mesh sets, or a
// text file in which the mesh owns one marked block of lines (novox/hq issue 128).
const (
IntoJSON = "json"
IntoBlock = "block"
)
// The lines that delimit the mesh's region in a file written into a block, each followed by the
// resource's id. Exact lines, never patterns: another tool's `# BEGIN …` block in the same file is
// that tool's, and a marker that merely resembled the mesh's must not be taken for it.
const (
BlockBegin = "# BEGIN mesh "
BlockEnd = "# END mesh "
)
// Where a file written into a block has its region added, when it has none yet.
const (
AtStart = "start"
AtEnd = "end"
)
// BlockMarkers are the two lines, without their line ends, that delimit a resource's region.
func BlockMarkers(id string) (begin, end string) {
return BlockBegin + id, BlockEnd + id
}
// Opening is a port reachable on an adopted node, from where, and on which path.
//
// **From** is everywhere or mesh — the private network, by its interface. **Path** is incoming,
// for something listening on the machine, or forwarded, for a published container port: the found
// firewall sees a published port after the runtime has translated it, so a forwarded opening names
// the container's own port in To as well as the machine's in Port.
type Opening struct {
ID string `json:"id"`
Type Type `json:"type"`
Port int `json:"port"`
Protocol string `json:"protocol"`
From string `json:"from"`
Path string `json:"path"`
To int `json:"to,omitempty"`
}
// Where an opening admits from, and the path it is on.
const (
FromEverywhere = "everywhere"
FromMesh = "mesh"
PathIncoming = "incoming"
PathForwarded = "forwarded"
)
func (o *Opening) Identity() string { return o.ID }
func (o *Opening) Kind() Type { return TypeOpening }
func (o *Opening) Target() string {
if o.Path == PathForwarded {
return fmt.Sprintf("%s/%d forwarded to %d from %s", o.Protocol, o.Port, o.To, o.From)
}
return fmt.Sprintf("%s/%d %s from %s", o.Protocol, o.Port, o.Path, o.From)
}
func (o *Opening) validate(where string, _ bool) []string {
var problems []string
if o.Port < 1 || o.Port > 65535 {
problems = append(problems, fmt.Sprintf("%s: an opening's port is 1-65535, not %d", where, o.Port))
}
if o.Protocol != "tcp" && o.Protocol != "udp" {
problems = append(problems, fmt.Sprintf("%s: an opening is tcp or udp, not %q", where, o.Protocol))
}
if o.From != FromEverywhere && o.From != FromMesh {
problems = append(problems, fmt.Sprintf(
"%s: an opening is from %q or %q, not %q", where, FromEverywhere, FromMesh, o.From))
}
switch o.Path {
case PathIncoming:
if o.To != 0 {
problems = append(problems, where+
": an incoming opening names no container port; only a forwarded one does")
}
case PathForwarded:
if o.To < 1 || o.To > 65535 {
problems = append(problems, where+
": a forwarded opening names the container's port it reaches, as to, 1-65535")
}
default:
problems = append(problems, fmt.Sprintf(
"%s: an opening's path is %q or %q, not %q", where, PathIncoming, PathForwarded, o.Path))
}
if !strings.HasPrefix(o.ID, AdoptionPrefix) {
problems = append(problems, fmt.Sprintf(
"%s: an opening is the mesh's own, so its id starts %q", where, AdoptionPrefix))
}
return problems
}
@@ -598,6 +902,12 @@ type Package struct {
ID string `json:"id"`
Type Type `json:"type"`
Package string `json:"package"`
// Absent declares that the package is NOT installed (novox/hq ADR 0175): the host removes it
// when it is, and leaves a machine that never had it alone. For the one case a module replaces
// software the machine was found with and the operator has decided it does not come back — the
// firewall front end a converged machine's filter module retired. Nothing to undo when the
// declaration drops it: the host does not install what a declaration stopped saying is absent.
Absent bool `json:"absent,omitempty"`
}
func (p *Package) Identity() string { return p.ID }
@@ -658,14 +968,50 @@ type Container struct {
// worse failure mode.
Network string `json:"network,omitempty"`
// Dns is the resolvers this container asks, passed to the runtime unchanged.
//
// **Because some software refuses to run behind the runtime's forwarding resolver.** A mail
// server's admin demands a DNSSEC-validating resolver, and the runtime's own (127.0.0.11)
// forwards to whatever the machine has — so a module that ships its own validating resolver
// must be able to point its other containers at it. Addresses, not names: the runtime's flag
// takes only addresses, which is also why IP below exists — the resolver has to be somewhere
// its siblings can name before any of them can resolve anything.
Dns []string `json:"dns,omitempty"`
// Capabilities are the Linux capabilities this container is granted beyond the runtime's
// default set, by name (novox/hq ADR 0170): a holder's runtime that changes the machine's packet
// filter asks for NET_ADMIN. Exactly these, named in the spec so a change recreates the
// container; a privileged container stays undeclarable.
Capabilities []string `json:"capabilities,omitempty"`
// Networks are networks this container also joins once created, by name — a found network a
// per-machine setting keeps for a taken container (novox/hq ADR 0163, rule 4), so a
// neighbour that resolves it there keeps resolving it until the neighbour is taken too.
// Joined after creation, because a runtime starts a container on one network; part of the
// container's spec, so a network kept or let go recreates it.
Networks []string `json:"networks,omitempty"`
// IP is this container's address on its network, passed to the runtime unchanged.
//
// Only meaningful on a user-defined network, and refused by the runtime elsewhere. Exists for
// exactly one shape: a container others must reach *before* name resolution works — a
// module's own DNS resolver being the case that forced it (see Dns).
IP string `json:"ip,omitempty"`
// RestartOn names resources whose change means this container must be recreated — the same
// field a service has, for the same reason (novox/hq 04-ISSUES/009). A container reads a
// mounted file once at start; a changed file leaves the running process holding the old value,
// while every check passes because the file on disk is right. The container's spec — image,
// env, volumes — does not include a mounted file's *content*, so a settings change that
// re-renders that file is invisible to the ordinary spec diff. This closes that: the host
// recreates the container when one of these resources changed this pass, even if the spec
// matches.
// while every check passes because the file on disk is right. The host recreates the container
// when one of these resources changed this pass, even if the spec matches.
//
// What a running container reads at creation — its env-files, and a file mounted into it
// directly — is part of its spec by content since novox/hq 04-ISSUES/103, and needs no naming
// here. A directory mounted into it is NOT looked inside, not even for files the host wrote
// there: whether a service reads such a file once or watches it live is the service's, and
// RestartOn is how a module says "once, at start" — a config the host renders under the
// module's state directory, a step whose result it consumes. On a run-once step it means *run
// again*: a step that fetches a fact from a provider names the binding it reads, and is run
// again when the provider moved (novox/hq ADR 0099).
RestartOn []string `json:"restart-on,omitempty"`
// RunOnce marks a container the host runs to completion rather than leaves running: a step,
@@ -700,13 +1046,9 @@ func (c *Container) validate(where string, _ bool) []string {
if c.Name == "" {
problems = append(problems, where+": a container needs a name")
}
// restart-on brings a *running* container back when a file it read changed; a run-once step
// does not stay running to be brought back. Declaring both asks for two contradictory
// lifecycles at once, so it is refused rather than silently resolved to one of them.
if c.RunOnce && len(c.RestartOn) > 0 {
problems = append(problems, where+": a run-once container cannot also declare restart-on; "+
"it runs to completion rather than staying running to be restarted")
}
// A run-once step may name what it reads under restart-on. For a step the word means *run
// again*: what a container reads is part of its digest, so a step whose named resource changed
// is a different step and runs again (novox/hq ADR 0099). Not refused.
// A container runs once and gates, on a cadence, or stays up — never two of these
// (novox/hq ADR 0053). run-once and schedule are the two "runs to completion" lifecycles and
// contradict each other, and a scheduled step does not stay running to be brought back by
@@ -724,6 +1066,39 @@ func (c *Container) validate(where string, _ bool) []string {
problems = append(problems, where+": "+err.Error())
}
}
// The runtime's flags take addresses, and a name here would be handed to it verbatim and
// refused at create — after the old container was already removed. Refused on arrival instead.
for _, d := range c.Dns {
if net.ParseIP(d) == nil {
problems = append(problems, where+": dns "+strconv.Quote(d)+" is not an address; "+
"the runtime's resolver flag takes only addresses")
}
}
if c.IP != "" {
if net.ParseIP(c.IP) == nil {
problems = append(problems, where+": ip "+strconv.Quote(c.IP)+" is not an address")
}
if c.Network == "" {
problems = append(problems, where+": an ip needs a network; the runtime refuses a "+
"static address anywhere but a user-defined one")
}
}
for _, cap := range c.Capabilities {
if !capabilityName.MatchString(cap) {
problems = append(problems, where+": capabilities names "+strconv.Quote(cap)+", which is not a "+
"capability's name (CAP_NET_ADMIN or NET_ADMIN)")
}
}
for _, n := range c.Networks {
problems = append(problems, (&Network{Name: n}).validate(where+": networks", false)...)
if n == c.Network {
problems = append(problems, where+": networks names "+n+", which is already the container's network")
}
}
if len(c.Networks) > 0 && (c.RunOnce || c.Schedule != "") {
problems = append(problems, where+": networks is for a container that keeps running; a step "+
"runs and exits, and joins nothing afterwards")
}
return append(problems, checkImage(where, c.Image)...)
}
@@ -800,6 +1175,8 @@ func newOf(t Type) Resource {
return &Access{}
case TypeProcess:
return &Process{}
case TypeOpening:
return &Opening{}
}
return nil
}
@@ -808,7 +1185,7 @@ func newOf(t Type) Resource {
func Vocabulary() []Type {
return []Type{
TypeAccess, TypeAction, TypeArchive, TypeContainer, TypeDirectory, TypeFile,
TypeNetwork, TypePackage, TypeProcess, TypeService, TypeUser,
TypeNetwork, TypeOpening, TypePackage, TypeProcess, TypeService, TypeUser,
}
}
@@ -822,6 +1199,138 @@ type Declaration struct {
// Resources, in the order they are applied. The host does not sort them: ordering is a
// decision, and deciding is not what the host does (novox/hq ADR 0005).
Resources []Resource
// Adoption says this node is adopted, and which of its modules have been taken. Nil is a
// converged node — which is every node the mesh raised before adoption existed, and so the
// only form an older controller ever sends (novox/hq ADR 0100).
Adoption *Adoption
// Sequence orders this declaration against every other the mesh has sent this node: each
// send is one higher than the last, assigned under the control plane's hold on the node
// (novox/hq 04-ISSUES/107). Zero is a declaration that carries no order — every one an older
// controller sent, and the bundle genesis applies — and a host makes no ordering claim about
// one of those.
//
// **The one property a declaration needs that its signature does not give it.** A signature
// says the mesh sent this; it cannot say the mesh sent it AFTER the one the host is holding.
// Before this, "older" was inferred from arrival within a batch and a 750ms window, and a
// backlog longer than the batch, or a slow broker, applied a declaration the mesh had already
// superseded.
Sequence int64
// LeftOut names the modules of this machine's set the mesh left out of this declaration,
// because a setting stored for one cannot compose with its definition (novox/hq ADR 0163,
// rule 6). A machine is told everything or nothing about what it IS told; this is what it is
// not told, said. The host keeps what it holds for a left-out module and touches none of
// what it wrote for it — its resources are absent from the declaration, and absence would
// otherwise read as removal.
LeftOut []string
}
// LeftOutModuleOf says which left-out module a recorded resource belongs to, if any: its id is the
// module's name, a dot, and the module's own id for it. A module's name may contain a dot, so the
// longest left-out name that prefixes the id wins; a false match keeps a thing an apply would
// otherwise remove, which is the conservative mistake.
func (d *Declaration) LeftOutModuleOf(id string) (string, bool) {
best := ""
for _, m := range d.LeftOut {
if strings.HasPrefix(id, m+".") && len(m) > len(best) {
best = m
}
}
return best, best != ""
}
// Adoption is a node's mode, as the controller records it: the node is adopted, and these are
// the modules taken on it so far (novox/hq ADR 0100).
//
// **Authoritative, and only ever stated by the controller.** A host does not work out whether it
// is adopted; it is told, in every declaration, so a host restarted from the declaration it kept
// is in the same mode it was in before.
//
// Untaken names, per module assigned here and not yet taken, the ids of its resources. Any kind
// may be listed: a file, a directory, a service's unit or a container can already be on the
// machine, and an action run inside a held container reaches what was found (novox/hq ADR 0103);
// the host decides per kind what can be held. The host cannot split a resource id into its
// module, because module names may contain dots, so the controller says which ids belong to which
// module rather than leaving the host to guess.
type Adoption struct {
Taken []string `json:"taken"`
Untaken map[string][]string `json:"untaken,omitempty"`
}
// capabilityName is what a Linux capability is called: upper case, underscores, an optional CAP_
// prefix. The runtime accepts either spelling.
var capabilityName = regexp.MustCompile(`^(CAP_)?[A-Z][A-Z0-9_]*$`)
// AdoptionPrefix is the id prefix of what the mesh itself declares because a node is adopted —
// its openings and its guard. Nothing under it belongs to a module, so none of it is ever held.
const AdoptionPrefix = "adoption."
// UntakenModuleOf says which untaken module declares a resource, if any.
func (a *Adoption) UntakenModuleOf(id string) (string, bool) {
if a == nil {
return "", false
}
for module, ids := range a.Untaken {
if slices.Contains(ids, id) {
return module, true
}
}
return "", false
}
// checkAdoption holds what an adoption says against the resources beside it. Every problem is a
// refusal: a host that misread which module is untaken would replace a predecessor's service the
// operator never took.
func checkAdoption(a *Adoption, resources []Resource, allowActions bool) []string {
if a == nil {
return nil
}
if allowActions {
// The bundle is carried with the binary and raises a foundation before any mesh exists.
// Whether a node is adopted is the controller's record, and a bundle that claimed it would
// be the host deciding its own mode (novox/hq ADR 0100).
return []string{"a carried bundle says the node is adopted, and only the mesh can say " +
"that: a node's mode is the controller's record, sent in every declaration"}
}
kinds := map[string]Type{}
for _, r := range resources {
kinds[r.Identity()] = r.Kind()
}
var problems []string
for _, module := range a.Taken {
if _, both := a.Untaken[module]; both {
problems = append(problems, fmt.Sprintf(
"adoption: the module %q is said to be both taken and untaken", module))
}
}
owner := map[string]string{}
modules := make([]string, 0, len(a.Untaken))
for module := range a.Untaken {
modules = append(modules, module)
}
sort.Strings(modules)
for _, module := range modules {
for _, id := range a.Untaken[module] {
if strings.HasPrefix(id, AdoptionPrefix) {
problems = append(problems, fmt.Sprintf(
"adoption: %q is the mesh's own and belongs to no module, so it cannot be untaken", id))
continue
}
if first, twice := owner[id]; twice {
problems = append(problems, fmt.Sprintf(
"adoption: %q is said to belong to both %q and %q", id, first, module))
continue
}
owner[id] = module
if _, declared := kinds[id]; !declared {
problems = append(problems, fmt.Sprintf(
"adoption: %q of the untaken module %q is not in this declaration", id, module))
}
}
}
return problems
}
// RefusalError refuses a whole declaration, naming every problem at once.
@@ -860,9 +1369,22 @@ func ParseTrusted(raw []byte) (*Declaration, error) { return parse(raw, true) }
// first pass takes the envelope and each resource's bytes; the second decodes each one into
// the struct for its kind, strictly.
type envelope struct {
Version int `json:"declaration"`
For string `json:"for,omitempty"`
Resources []json.RawMessage `json:"resources"`
Version int `json:"declaration"`
For string `json:"for,omitempty"`
Adoption *Adoption `json:"adoption,omitempty"`
// OwnsNothing is the control plane saying, explicitly, that this node's declaration is empty
// on purpose — it owns nothing the mesh put there (novox/hq issue 127). Without it an empty
// resources list is refused as a likely mistake; with it the node applies the empty
// declaration and drops what it last held. The two are distinguished because a truncated or
// mis-composed body arrives as empty too, and a host that could not tell them apart would let
// a bug quietly strip a machine.
OwnsNothing bool `json:"owns_nothing,omitempty"`
Resources []json.RawMessage `json:"resources"`
// Sequence is optional on the wire, so a controller that does not send one is still
// understood: absent reads as zero, which is "no ordering claimed" rather than "first".
Sequence int64 `json:"sequence,omitempty"`
// LeftOut is optional on the wire too, and absent when nothing was left out (ADR 0163).
LeftOut []string `json:"left_out,omitempty"`
}
func parse(raw []byte, allowActions bool) (*Declaration, error) {
@@ -879,10 +1401,22 @@ func parse(raw []byte, allowActions bool) (*Declaration, error) {
env.Version, Version)}}
}
d := &Declaration{Version: env.Version, For: env.For}
d := &Declaration{Version: env.Version, For: env.For, Adoption: env.Adoption, Sequence: env.Sequence,
LeftOut: env.LeftOut}
var problems []string
if len(env.LeftOut) > 0 && allowActions {
// The bundle is carried with the binary and leaves nothing out: which module a setting
// stopped composing for is the mesh's record (ADR 0163).
problems = append(problems, "a carried bundle says modules were left out, and only the "+
"mesh can say that")
}
for _, m := range env.LeftOut {
if strings.TrimSpace(m) == "" {
problems = append(problems, "left_out names a module with no name")
}
}
if len(env.Resources) == 0 {
if len(env.Resources) == 0 && !env.OwnsNothing {
problems = append(problems, "no resources. An empty declaration is a mistake, not a "+
"machine with nothing on it — say so with an explicit empty list if that is meant")
}
@@ -942,6 +1476,27 @@ func parse(raw []byte, allowActions bool) (*Declaration, error) {
problems = append(problems, resource.validate(where, allowActions)...)
d.Resources = append(d.Resources, resource)
}
problems = append(problems, checkAdoption(env.Adoption, d.Resources, allowActions)...)
if env.Adoption == nil {
for _, r := range d.Resources {
if r.Kind() == TypeOpening {
// On a converged node the mesh's own filter admits what is declared, and the
// found firewall is retired; an opening there would be a rule in a firewall the
// mesh has disabled (novox/hq ADR 0100).
problems = append(problems, fmt.Sprintf(
"resource %q: an opening is for an adopted node, and this declaration does not "+
"say the node is adopted", r.Identity()))
}
if svc, ok := r.(*Service); ok && svc.TakesOver != nil {
// A tunnel is taken over on an adopted node, where what is found is kept: on a
// converged one there is nothing found to take over, and stopping a unit the
// mesh did not declare would be the host deciding (novox/hq ADR 0105).
problems = append(problems, fmt.Sprintf(
"resource %q: taking over a tunnel is for an adopted node, and this declaration "+
"does not say the node is adopted", r.Identity()))
}
}
}
if len(problems) > 0 {
return nil, &RefusalError{Problems: problems}
+105 -4
View File
@@ -266,7 +266,7 @@ func TestAFieldTheNewTypesDoNotUseIsRefused(t *testing.T) {
}
}
func TestTheVocabularyIsTheElevenShapesTheMeshNeeds(t *testing.T) {
func TestTheVocabularyIsTheTwelveShapesTheMeshNeeds(t *testing.T) {
// Six of them the bootstrap uses (novox/hq 07-the-foundation.md), and removing one is a
// failing test rather than a discovery during a first-node install.
//
@@ -282,7 +282,7 @@ func TestTheVocabularyIsTheElevenShapesTheMeshNeeds(t *testing.T) {
}
for _, want := range []Type{
TypeDirectory, TypeFile, TypeService, TypePackage, TypeContainer, TypeAction,
TypeUser, TypeArchive, TypeNetwork, TypeAccess, TypeProcess,
TypeUser, TypeArchive, TypeNetwork, TypeAccess, TypeProcess, TypeOpening,
} {
if !speaks[want] {
t.Errorf("the host no longer speaks %q", want)
@@ -309,8 +309,12 @@ func TestTheVocabularyIsTheElevenShapesTheMeshNeeds(t *testing.T) {
// It is a full-host shape rather than a portable one: it needs a process supervisor to install
// into. It does NOT need a container runtime, which is the point — only software that
// genuinely needs isolation asks for a container.
if len(speaks) != 11 {
t.Errorf("the vocabulary is %d shapes rather than 11; every addition widens what a compromised "+
//
// `opening` is the twelfth, and novox/hq ADR 0100 is its decision: on an adopted node the
// firewall found there stays in force, and what the mesh needs reachable is converged through
// it as a state the host marks as the mesh's — never a command, which the link may not carry.
if len(speaks) != 12 {
t.Errorf("the vocabulary is %d shapes rather than 12; every addition widens what a compromised "+
"control plane can express, so a change here is a decision: %s",
len(speaks), vocabulary())
}
@@ -423,3 +427,100 @@ func TestASecretTheContentNeverUsesIsRefused(t *testing.T) {
t.Fatal("a secret the content never mentions was accepted")
}
}
// The runtime's resolver and address flags take only addresses; a name would be refused at
// create, after the old container was already gone. Refused on arrival instead — and an address
// without a user-defined network is refused for the same reason.
func TestAContainersResolverAndAddressAreAddressesOrRefused(t *testing.T) {
refused := func(body string) []string {
_, err := Parse([]byte(`{"declaration":1,"resources":[` + body + `]}`))
if err == nil {
return nil
}
return []string{err.Error()}
}
base := `"id":"c","type":"container","name":"x",` +
`"image":"a@sha256:0000000000000000000000000000000000000000000000000000000000000000"`
if p := refused(`{` + base + `,"network":"m","dns":["resolver.local"]}`); len(p) == 0 {
t.Error("a resolver named by name was accepted; the runtime takes only addresses")
}
if p := refused(`{` + base + `,"ip":"192.168.203.7"}`); len(p) == 0 {
t.Error("a static address with no network was accepted; the runtime refuses it")
}
if p := refused(`{` + base + `,"network":"m","dns":["192.168.203.254"],"ip":"192.168.203.7"}`); len(p) != 0 {
t.Errorf("a well-formed resolver and address were refused: %v", p)
}
}
func TestAnExplicitlyEmptyDeclarationIsAccepted(t *testing.T) {
// A deliberately-empty declaration (novox/hq issue 127) says owns_nothing, and is applied so
// the node drops what it last held — distinct from an accidental empty body, which is refused.
if _, err := Parse([]byte(`{"declaration":1,"owns_nothing":true,"resources":[]}`)); err != nil {
t.Fatalf("an explicitly-empty declaration must be accepted: %v", err)
}
// Without the marker, an empty declaration is still refused as a likely mistake.
_, err := Parse([]byte(`{"declaration":1,"resources":[]}`))
if err == nil || !strings.Contains(err.Error(), "no resources") {
t.Fatalf("an unmarked empty declaration must still be refused; got %v", err)
}
}
// A container's kept networks are names, not its own network, and not for a step (novox/hq ADR
// 0163, rule 4); and the mesh may say which modules it left out, which a carried bundle may not.
func TestKeptNetworksAndLeftOutModulesAreReadStrictly(t *testing.T) {
pinnedImage := "postgres@sha256:" + strings.Repeat("a", 64)
d, err := Parse([]byte(`{"declaration":1,"left_out":["web"],"resources":[
{"id":"app","type":"container","name":"app","image":"` + pinnedImage + `","networks":["predecessor_default"]}
]}`))
if err != nil {
t.Fatal(err)
}
if got := d.Resources[0].(*Container).Networks; len(got) != 1 || got[0] != "predecessor_default" {
t.Fatalf("the kept network was not read: %v", got)
}
if m, left := d.LeftOutModuleOf("web.server"); !left || m != "web" {
t.Fatalf("web.server is not web's: %q %v", m, left)
}
if _, left := d.LeftOutModuleOf("webapp.server"); left {
t.Fatal("webapp.server was taken for web's")
}
for name, raw := range map[string]string{
"a bad network name": `{"declaration":1,"resources":[
{"id":"app","type":"container","name":"app","image":"` + pinnedImage + `","networks":["a/b"]}]}`,
"its own network": `{"declaration":1,"resources":[
{"id":"app","type":"container","name":"app","image":"` + pinnedImage + `","network":"own","networks":["own"]}]}`,
"a step": `{"declaration":1,"resources":[
{"id":"app","type":"container","name":"app","image":"` + pinnedImage + `","run-once":true,"networks":["x"]}]}`,
"a nameless module": `{"declaration":1,"left_out":[""],"resources":[
{"id":"app","type":"container","name":"app","image":"` + pinnedImage + `"}]}`,
} {
if _, err := Parse([]byte(raw)); err == nil {
t.Errorf("%s was accepted", name)
}
}
if _, err := ParseTrusted([]byte(`{"declaration":1,"left_out":["web"],"resources":[
{"id":"app","type":"container","name":"app","image":"` + pinnedImage + `"}]}`)); err == nil ||
!strings.Contains(err.Error(), "only the mesh can say that") {
t.Fatalf("a carried bundle leaving modules out was accepted: %v", err)
}
}
// A container may ask for a capability by name, and nothing else (novox/hq ADR 0170).
func TestACapabilityIsNamedOrRefused(t *testing.T) {
image := "postgres@sha256:" + strings.Repeat("a", 64)
d, err := Parse([]byte(`{"declaration":1,"resources":[
{"id":"fw","type":"container","name":"fw","image":"` + image + `","network":"host","capabilities":["NET_ADMIN","CAP_NET_RAW"]}
]}`))
if err != nil {
t.Fatal(err)
}
if got := d.Resources[0].(*Container).Capabilities; len(got) != 2 || got[0] != "NET_ADMIN" {
t.Fatalf("capabilities read as %v", got)
}
for _, bad := range []string{`"net_admin"`, `"ALL;rm -rf /"`, `"privileged"`} {
if _, err := Parse([]byte(`{"declaration":1,"resources":[
{"id":"fw","type":"container","name":"fw","image":"` + image + `","capabilities":[` + bad + `]}]}`)); err == nil {
t.Errorf("%s was accepted as a capability", bad)
}
}
}
+35
View File
@@ -0,0 +1,35 @@
package declaration
import (
"strings"
"testing"
)
// Defends novox/hq ADR 0102: a file written into carries only a JSON object of its keys, in a
// format the host speaks, and a service may name what it is reloaded on.
func TestAFileWrittenIntoIsRefusedUnlessItIsAnObjectOfKeys(t *testing.T) {
for name, c := range map[string]struct{ resource, refusal string }{
"another format": {`{"id":"f","type":"file","path":"/etc/x","into":"toml","content":"a = 1"}`, `into "toml"`},
"not an object": {`{"id":"f","type":"file","path":"/etc/x","into":"json","content":"[1,2]"}`, "JSON object"},
"with create-once": {`{"id":"f","type":"file","path":"/etc/x","into":"json","content":"{}","create-once":true}`, "create-once"},
} {
_, err := Parse([]byte(`{"declaration":1,"resources":[` + c.resource + `]}`))
if err == nil || !strings.Contains(err.Error(), c.refusal) {
t.Errorf("%s: want a refusal naming %q, got %v", name, c.refusal, err)
}
}
d, err := Parse([]byte(`{"declaration":1,"resources":[
{"id":"f","type":"file","path":"/etc/x","into":"json","content":"{\"k\":1}"},
{"id":"s","type":"service","unit":"docker.service","state":"running","reload-on":["f"]}
]}`))
if err != nil {
t.Fatal(err)
}
if f := d.Resources[0].(*File); f.Into != IntoJSON {
t.Errorf("into was read as %q", f.Into)
}
if s := d.Resources[1].(*Service); len(s.ReloadOn) != 1 || s.ReloadOn[0] != "f" {
t.Errorf("reload-on was read as %v", s.ReloadOn)
}
}
+4 -1
View File
@@ -55,7 +55,10 @@ func TestAProcessMustSayWhatToRun(t *testing.T) {
// Its name becomes a unit name and a path, so a separator in it would write somewhere nobody meant.
func TestAProcesssNameCannotEscapeItsUnit(t *testing.T) {
for _, bad := range []string{"", "../escape", "two words", "a/b"} {
// "." and ".." are one path element each, and the mesh's own bundle directory and its parent:
// removing a process named ".." would delete every bundle the mesh has, and more (novox/hq ADR
// 0118). A leading dash is an option to the service manager, not a unit.
for _, bad := range []string{"", "../escape", "two words", "a/b", ".", "..", "-", "--now"} {
d := aProcess()
d.Name = bad
if problems := d.validate("a process", false); len(problems) == 0 {
+30
View File
@@ -0,0 +1,30 @@
package declaration
import (
"strings"
"testing"
)
// Defends novox/hq ADR 0117: a service may leave its unit's lifecycle to the machine, and then
// says nothing but its triggers.
func TestAServiceWithoutAStateSaysOnlyItsTriggers(t *testing.T) {
for name, c := range map[string]struct{ resource, refusal string }{
"no trigger": {`{"id":"s","type":"service","unit":"NetworkManager.service"}`, "declares nothing"},
"with boot": {`{"id":"s","type":"service","unit":"NetworkManager.service","boot":"enabled","reload-on":["f"]}`, "boot and takes-over"},
"with takes-over": {`{"id":"s","type":"service","unit":"a.service","reload-on":["f"],"takes-over":{"interface":"wg0","unit":"b.service","config":"/etc/x"}}`, "boot and takes-over"},
"an unknown state": {`{"id":"s","type":"service","unit":"a.service","state":"paused"}`, "omits state to leave the unit's lifecycle to the machine"},
} {
_, err := Parse([]byte(`{"declaration":1,"resources":[{"id":"f","type":"file","path":"/etc/x","content":"x"},` + c.resource + `]}`))
if err == nil || !strings.Contains(err.Error(), c.refusal) {
t.Errorf("%s: want a refusal naming %q, got %v", name, c.refusal, err)
}
}
d, err := Parse([]byte(`{"declaration":1,"resources":[{"id":"f","type":"file","path":"/etc/x","content":"x"},
{"id":"s","type":"service","unit":"NetworkManager.service","restart-on":["f"]}]}`))
if err != nil {
t.Fatal(err)
}
if s := d.Resources[1].(*Service); !s.Stateless() {
t.Error("a service without a state was not read as stateless")
}
}
+30
View File
@@ -0,0 +1,30 @@
package declaration
import (
"strings"
"testing"
)
// novox/hq ADR 0177: a unit is in the system manager or an account's own; a user-scoped one names
// the account, a system one may not, and any other word is refused.
func TestAUserScopedUnitNamesItsAccountAndASystemOneMayNot(t *testing.T) {
cases := []struct{ scope, user, wants string }{
{"user", "ops", ""},
{"", "", ""},
{"system", "", ""},
{"user", "", "names the account"},
{"", "ops", "names no user"},
{"session", "ops", "scope \"session\""},
}
for _, c := range cases {
s := &Service{ID: "m.u", Type: TypeService, Unit: "u.service", State: "running", Scope: c.scope, User: c.user}
problems := s.validate("m.u", false)
got := strings.Join(problems, "; ")
if c.wants == "" && len(problems) != 0 {
t.Fatalf("scope %q user %q refused: %s", c.scope, c.user, got)
}
if c.wants != "" && !strings.Contains(got, c.wants) {
t.Fatalf("scope %q user %q: wanted a refusal saying %q, got %q", c.scope, c.user, c.wants, got)
}
}
}
+335
View File
@@ -0,0 +1,335 @@
package firewall
import (
"context"
"fmt"
"regexp"
"sort"
"strings"
)
// What filters a machine, said with an owner (novox/hq ADR 0168).
//
// "The firewall found" names one front end, and a machine carries rules from several sources: the
// front end's own, the container runtime's plumbing, a ban list, the mesh's own tables, and whatever
// a predecessor installed directly — on both machines of the first mesh, in the user chain the
// runtime leaves for an administrator, where the mesh's reader of rules counted it as the runtime's.
// So the host reports every table and chain that refuses traffic, each with whose it is, and the
// mesh says truthfully what filters a converged machine. It removes none of it.
// Owners of a refusal.
const (
// OwnerMesh is the mesh's own tables: the derived filter and the guard.
OwnerMesh = "mesh"
// OwnerFoundFirewall is the front end found on the machine — ufw's chains.
OwnerFoundFirewall = "found-firewall"
// OwnerRuntime is the container runtime's own plumbing: its chains, the forward policy it sets
// when it turns forwarding on, its guard against reaching a container's address from off its
// bridge. Not the user chain it leaves for an administrator.
OwnerRuntime = "runtime"
// OwnerBan is a refusal that names the sources it refuses, in a chain that accepts nothing — a
// ban list, which is not a firewall.
OwnerBan = "ban"
// OwnerOther is everything else: rules the mesh did not write and cannot attribute. Where a
// predecessor's rules live.
OwnerOther = "other"
)
// A Filter is one place on the machine that refuses traffic: a chain of a table, or a chain of the
// legacy filter, with its owner and what it refuses in one line.
type Filter struct {
// Where names the chain: "table ip filter, chain DOCKER-USER", or "chain HAL-MESH-ONLY
// (iptables-legacy)".
Where string `json:"where"`
// Owner is one of the owners above.
Owner string `json:"owner"`
// Refuses is the first refusing line, counters stripped, and how many more there are.
Refuses string `json:"refuses"`
table, chain string
}
// userChain is the chain the container runtime creates empty and leaves for an administrator's
// rules, consulted before its own forwarding. Nothing in it is the runtime's.
const userChain = "DOCKER-USER"
// Filters classifies every refusing chain of an `nft list ruleset` and of the legacy filter's `-S`
// listings (by tool: iptables-legacy, ip6tables-legacy), in the order they appear.
func Filters(ruleset string, legacy map[string]string, ufwActive bool) []Filter {
var out []Filter
r := parseNft(ruleset)
refusing := map[string][]nftRule{} // by "table\x00chain"
for _, rule := range r.refusals {
k := rule.table + "\x00" + rule.chain
refusing[k] = append(refusing[k], rule)
}
for _, k := range r.chainOrder {
c := r.chains[k]
table, chain, _ := strings.Cut(k, "\x00")
rules := refusing[k]
if !c.dropping && len(rules) == 0 {
continue
}
f := Filter{table: table, chain: chain, Where: "table " + table + ", chain " + chain}
switch {
case table == MeshTable || table == "inet mesh_guard":
f.Owner = OwnerMesh
case strings.HasPrefix(chain, "ufw"):
f.Owner = OwnerFoundFirewall
if !ufwActive {
// Left behind by a retired front end, and still refusing: not ufw's any more in
// any sense that matters, since nothing maintains it.
f.Owner = OwnerOther
}
case chain == userChain:
f.Owner = OwnerOther
case c.dropping && (r.managed[table] || iptablesTable(table)) && runtimes(table, chain, c.policyLine):
f.Owner = OwnerRuntime
case len(rules) > 0 && (r.managed[table] || iptablesTable(table)) && allRuntimes(table, chain, rules):
f.Owner = OwnerRuntime
case len(rules) > 0 && allBans(r, rules):
f.Owner = OwnerBan
case c.dropping && !iptablesTable(table) && !r.managed[table] && len(rules) == 0:
// A table of its own whose base chain drops by policy: a firewall nobody declared.
f.Owner = OwnerOther
default:
f.Owner = OwnerOther
}
if ufwActive && (r.managed[table] || iptablesTable(table)) && f.Owner == OwnerOther && len(rules) == 0 && c.dropping {
// A base chain ufw set to drop while it is in force is ufw's.
f.Owner = OwnerFoundFirewall
}
f.Refuses = refusesLine(c, rules)
out = append(out, f)
}
tools := make([]string, 0, len(legacy))
for tool := range legacy {
tools = append(tools, tool)
}
sort.Strings(tools)
for _, tool := range tools {
out = append(out, legacyFilters(legacy[tool], tool, ufwActive)...)
}
return out
}
// allRuntimes is whether every refusal in a chain is the runtime's own.
func allRuntimes(table, chain string, rules []nftRule) bool {
for _, rule := range rules {
if !runtimes(table, chain, rule.line) {
return false
}
}
return true
}
// allBans is whether every refusal in a chain only bans the sources it names.
func allBans(r *nftRuleset, rules []nftRule) bool {
for _, rule := range rules {
if !r.onlyBans(rule) {
return false
}
}
return true
}
var counters = regexp.MustCompile(`\s*counter packets \d+ bytes \d+`)
// refusesLine is one line a person reads: the policy when the chain drops by policy, else the first
// refusing rule with its counters stripped, and how many more there are.
func refusesLine(c *nftChain, rules []nftRule) string {
var parts []string
if c.dropping {
parts = append(parts, "policy drop")
}
if len(rules) > 0 {
line := strings.TrimSpace(counters.ReplaceAllString(rules[0].line, ""))
if len(rules) > 1 {
line += fmt.Sprintf(" (and %d more)", len(rules)-1)
}
parts = append(parts, line)
}
return strings.Join(parts, "; ")
}
// legacyFilters classifies the chains of an `iptables-legacy -S` listing that refuse.
func legacyFilters(rules, tool string, ufwActive bool) []Filter {
policy := map[string]string{}
accepting := map[string]bool{}
jumpedFrom := map[string][]string{}
for _, line := range strings.Split(rules, "\n") {
fields := strings.Fields(line)
if len(fields) < 3 {
continue
}
switch fields[0] {
case "-P":
policy[fields[1]] = fields[2]
case "-A":
for i, f := range fields {
if (f == "-j" || f == "-g") && i+1 < len(fields) {
switch fields[i+1] {
case "ACCEPT":
accepting[fields[1]] = true
case "DROP", "REJECT", "RETURN", "LOG":
default:
jumpedFrom[fields[i+1]] = append(jumpedFrom[fields[i+1]], fields[1])
}
}
}
}
}
var entered func(chain string, seen map[string]bool) bool
entered = func(chain string, seen map[string]bool) bool {
if seen[chain] || accepting[chain] || len(jumpedFrom[chain]) == 0 {
return false
}
seen[chain] = true
for _, from := range jumpedFrom[chain] {
if p, builtIn := policy[from]; builtIn {
if p != "ACCEPT" {
return false
}
continue
}
if !entered(from, seen) {
return false
}
}
return true
}
ban := func(chain, line string) bool {
return bansSources(line) && entered(chain, map[string]bool{})
}
type seen struct {
owner string
lines []string
}
chains := map[string]*seen{}
var order []string
note := func(chain, owner, line string) {
s := chains[chain]
if s == nil {
s = &seen{owner: owner}
chains[chain] = s
order = append(order, chain)
}
if owner == OwnerOther || s.owner == "" {
s.owner = owner
}
s.lines = append(s.lines, line)
}
for _, line := range strings.Split(rules, "\n") {
fields := strings.Fields(line)
if len(fields) < 3 {
continue
}
chain := fields[1]
switch fields[0] {
case "-P":
if fields[2] != "DROP" {
continue
}
owner := OwnerOther
if chain == "FORWARD" {
owner = OwnerRuntime
}
if ufwActive {
owner = OwnerFoundFirewall
}
note(chain, owner, "policy DROP")
case "-A":
refuses := false
for i, f := range fields {
if f == "-j" && i+1 < len(fields) && (fields[i+1] == "DROP" || fields[i+1] == "REJECT") {
refuses = true
}
}
if !refuses {
continue
}
owner := OwnerOther
switch {
case strings.HasPrefix(chain, "ufw"):
owner = OwnerFoundFirewall
if !ufwActive {
owner = OwnerOther
}
case chain != userChain && strings.HasPrefix(chain, "DOCKER"):
owner = OwnerRuntime
case ban(chain, line):
owner = OwnerBan
}
note(chain, owner, strings.TrimSpace(line))
}
}
var out []Filter
for _, chain := range order {
s := chains[chain]
refuses := s.lines[0]
if len(s.lines) > 1 {
refuses += fmt.Sprintf(" (and %d more)", len(s.lines)-1)
}
out = append(out, Filter{Where: "chain " + chain + " (" + tool + ")", Owner: s.owner, Refuses: refuses})
}
return out
}
// Collect reads what filters this machine now: its nftables ruleset and, where the legacy tools
// exist, their listings. A machine without nft is read through iptables, as Detect reads it.
func Collect(ctx context.Context, run Runner, ufwActive bool) ([]Filter, error) {
ruleset := ""
noNft := false
out, err := run(ctx, "nft", "list", "ruleset")
switch {
case err == nil:
ruleset = out
case missing(err):
noNft = true
default:
return nil, fmt.Errorf("cannot read this machine's packet filter: %w", err)
}
legacy := map[string]string{}
tools := []string{"iptables-legacy", "ip6tables-legacy"}
if noNft {
tools = append(tools, "iptables", "ip6tables")
}
for _, tool := range tools {
if out, err := run(ctx, tool, "-S"); err == nil && strings.TrimSpace(out) != "" {
legacy[tool] = out
}
}
return Filters(ruleset, legacy, ufwActive), nil
}
// Alone is whether a machine is filtered by the mesh alone: nothing in the list but the mesh's
// own tables, the runtime's plumbing and bans (novox/hq ADR 0168).
func Alone(filters []Filter) bool {
for _, f := range filters {
if f.Owner == OwnerOther || f.Owner == OwnerFoundFirewall {
return false
}
}
return true
}
// Active says whether ufw is in force on this machine now. A machine without ufw is not.
func Active(ctx context.Context, run Runner) bool {
out, err := run(ctx, "ufw", "status")
return err == nil && statusActive(out)
}
// Installed says whether ufw is on this machine at all: a command that is not there is a front end
// that was uninstalled (novox/hq ADR 0175), not one that is silent.
func Installed(ctx context.Context, run Runner) bool {
_, err := run(ctx, "ufw", "status")
return !missing(err)
}
// Retirements of a found firewall, as the host records them.
const (
RetiredByMesh = "mesh"
RetiredFoundSo = "found-inactive"
// RetiredRemoved is a front end uninstalled by the module that replaced it (ADR 0175).
RetiredRemoved = "removed"
)
+130
View File
@@ -0,0 +1,130 @@
package firewall
import (
"os"
"strings"
"testing"
)
func fixture(t *testing.T, name string) string {
t.Helper()
raw, err := os.ReadFile("testdata/" + name)
if err != nil {
t.Fatal(err)
}
return string(raw)
}
func ownerOf(filters []Filter, where string) string {
for _, f := range filters {
if f.Where == where {
return f.Owner
}
}
return "(not reported)"
}
// Every refusing table and chain is classified with an owner (novox/hq ADR 0168), over rulesets
// captured from three machines of the first mesh. The control node: a ban list reached through the
// runtime's user chain is a ban; a refusal left in that chain, and a chain a retired front end left
// behind, are *other*; the runtime's own and the mesh's own are theirs.
func TestTheControlNodesRefusalsAreClassified(t *testing.T) {
got := Filters(fixture(t, "control-node.nft"), nil, false)
for where, want := range map[string]string{
"table ip filter, chain f2b-recidive": OwnerBan,
"table ip filter, chain DOCKER": OwnerRuntime,
"table ip raw, chain PREROUTING": OwnerRuntime,
"table inet mesh, chain input": OwnerMesh,
"table inet mesh, chain forward": OwnerMesh,
"table ip6 filter, chain DOCKER-USER": OwnerOther,
"table ip6 filter, chain ufw6-docker-logging-deny": OwnerOther,
} {
if o := ownerOf(got, where); o != want {
t.Errorf("%s: %s, want %s", where, o, want)
}
}
if Alone(got) {
t.Error("a machine with a refusal in the runtime's user chain reads as filtered by the mesh alone")
}
// What refuses adoption does not move (rule 4): the user chain's refusals are reported, not
// refused. The chain a retired front end left behind, still dropping, is what it always was
// to Detect — a refusal nobody speaks for, in one table.
if refusing := Refusing(fixture(t, "control-node.nft"), false); len(refusing) != 1 || refusing[0] != "table ip6 filter" {
t.Errorf("adoption's threshold moved: %v", refusing)
}
// The counters are stripped from what a person reads.
for _, f := range got {
if strings.Contains(f.Refuses, "counter packets") {
t.Errorf("counters in the line: %s", f.Refuses)
}
}
}
// The laptop: the runtime's forward policy and bridge guards, a virtualisation host and an endpoint
// agent that refuse nothing, and the mesh — filtered by the mesh alone.
func TestTheLaptopIsFilteredByTheMeshAlone(t *testing.T) {
got := Filters(fixture(t, "laptop.nft"), nil, false)
for where, want := range map[string]string{
"table ip filter, chain FORWARD": OwnerRuntime,
"table ip filter, chain DOCKER": OwnerRuntime,
"table ip raw, chain PREROUTING": OwnerRuntime,
"table inet mesh, chain input": OwnerMesh,
} {
if o := ownerOf(got, where); o != want {
t.Errorf("%s: %s, want %s", where, o, want)
}
}
for _, f := range got {
if strings.Contains(f.Where, "incus") || strings.Contains(f.Where, "fct_") {
t.Errorf("a table that refuses nothing is reported: %+v", f)
}
}
if !Alone(got) {
t.Errorf("the laptop is not read as filtered by the mesh alone: %+v", got)
}
}
// The home server: its rules are in the legacy filter, where a predecessor's chain still drops what
// arrives on the outward link for the forwarded path — invisible to the mesh until now (issue 144).
func TestThePredecessorsChainInTheLegacyFilterIsOther(t *testing.T) {
mesh := "table inet mesh {\n\tchain forward {\n\t\ttype filter hook forward priority filter; policy drop;\n\t}\n}\n"
got := Filters(mesh, map[string]string{"iptables-legacy": fixture(t, "home-server-legacy-S.txt")}, false)
for where, want := range map[string]string{
"table inet mesh, chain forward": OwnerMesh,
"chain FORWARD (iptables-legacy)": OwnerRuntime,
"chain DOCKER (iptables-legacy)": OwnerRuntime,
"chain HAL-MESH-ONLY (iptables-legacy)": OwnerOther,
} {
if o := ownerOf(got, where); o != want {
t.Errorf("%s: %s, want %s", where, o, want)
}
}
var other Filter
for _, f := range got {
if f.Owner == OwnerOther {
other = f
}
}
if !strings.Contains(other.Refuses, "-j DROP") {
t.Errorf("what the predecessor's chain refuses is not said: %+v", other)
}
if Alone(got) {
t.Error("a machine with a predecessor's chain reads as filtered by the mesh alone")
}
}
// With the front end in force, its chains are its own; retired, a chain it left behind that still
// refuses is nobody's and said so.
func TestAFrontEndsChainsAreItsWhileItIsInForce(t *testing.T) {
ruleset := dockerOnly(t) + ufwChains
for _, f := range Filters(ruleset, nil, true) {
if strings.Contains(f.Where, "ufw") && f.Owner != OwnerFoundFirewall {
t.Errorf("active: %+v", f)
}
}
for _, f := range Filters(ruleset, nil, false) {
if strings.Contains(f.Where, "ufw") && f.Owner != OwnerOther {
t.Errorf("retired: %+v", f)
}
}
}
+950
View File
@@ -0,0 +1,950 @@
// Package firewall speaks the firewall found on an adopted node, in that firewall's own terms
// (novox/hq ADR 0100).
//
// **The found firewall stays in force.** On an adopted node the mesh loads nothing that drops by
// default or holds an accept; what it needs reachable it converges as openings through what it
// found, marks each rule as its own, and removes only what it marked. It never resets or flushes:
// the rules the machine already had are the operator's, and they are what keeps it serving.
package firewall
import (
"context"
"crypto/sha256"
"encoding/hex"
"errors"
"fmt"
"os/exec"
"regexp"
"strconv"
"strings"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/system"
)
// Runner executes a command.
type Runner = system.Runner
// Kind is what firewall a machine has, as far as the mesh is concerned.
type Kind string
const (
// UFW is an active ufw — the one kind found on the machines measured, and the one spoken.
UFW Kind = "ufw"
// None is a machine where nothing refuses anything, which needs no openings.
None Kind = "none"
// Unsupported is a firewall no host speaks yet. A machine with one is refused adoption: the
// mesh could neither open what it needs nor know what it would be closing.
Unsupported Kind = "unsupported"
)
// deletion is the arguments that delete a rule as `ufw show added` printed it. A route rule is
// deleted with `route delete …`: ufw refuses `delete route …` as invalid syntax. And ufw answers
// success when asked to delete a rule it does not hold, so every deletion is read back.
func deletion(rule string) []string {
w := words(rule)
if len(w) > 0 && w[0] == "route" {
return append([]string{"route", "delete"}, w[1:]...)
}
return append([]string{"delete"}, w...)
}
// MeshInterface is the private network's interface, the way an opening from the mesh is known.
// It must be the controller's overlay interface name.
const MeshInterface = "mesh0"
// Detect says which firewall this machine has. For Unsupported the string names it.
func Detect(ctx context.Context, run Runner) (Kind, string, error) {
if out, err := run(ctx, "firewall-cmd", "--state"); err == nil && strings.TrimSpace(out) == "running" {
return Unsupported, "firewalld", nil
}
ufwActive := false
if out, err := run(ctx, "ufw", "status"); err == nil {
ufwActive = statusActive(out)
}
noNft := false
out, err := run(ctx, "nft", "list", "ruleset")
switch {
case err == nil:
if refusing := Refusing(out, ufwActive); len(refusing) > 0 {
return Unsupported, "nftables rules that refuse traffic, in " + strings.Join(refusing, ", "), nil
}
case missing(err):
// **No nft on this machine does not mean no rules.** iptables-nft writes tables nft would
// have shown, and a machine whose only tool is iptables answers about them through that.
// Read as "nothing filters here", a machine with an iptables firewall would be adopted
// with no openings and nothing would reach the mesh (novox/hq ADR 0100).
noNft = true
default:
return "", "", fmt.Errorf("cannot read this machine's packet filter to know what it has: %w", err)
}
if !ufwActive {
// iptables with the legacy backend is invisible to nft; and where nft is not installed,
// the iptables command is the only way to see anything at all.
tools := []string{"iptables-legacy", "ip6tables-legacy"}
if noNft {
tools = append(tools, "iptables", "ip6tables")
}
for _, legacy := range tools {
out, err := run(ctx, legacy, "-S")
if err != nil {
continue
}
if refusing := RefusingLegacy(out); len(refusing) > 0 {
return Unsupported, legacy + " rules that refuse traffic, in " + strings.Join(refusing, ", "), nil
}
}
}
if ufwActive {
return UFW, "ufw", nil
}
return None, "", nil
}
func missing(err error) bool {
return errors.Is(err, exec.ErrNotFound)
}
func statusActive(out string) bool {
for _, line := range strings.Split(out, "\n") {
if strings.HasPrefix(strings.TrimSpace(line), "Status:") {
return strings.TrimSpace(strings.TrimPrefix(strings.TrimSpace(line), "Status:")) == "active"
}
}
return false
}
// Refusing names the tables of an `nft list ruleset` holding something that refuses traffic — a
// drop or reject, or a base chain whose policy drops — and that is neither the mesh's own nor the
// container runtime's. With ufw active, the tables iptables-nft manages are ufw's and the runtime's
// and are not counted.
//
// **A ban is not a firewall.** fail2ban refuses the sources it banned and passes everything else;
// captured on a lab machine with both of its backends (testdata/fail2ban-nftables.nft,
// testdata/fail2ban-iptables.nft). The mesh opens nothing through it and it closes nothing the
// mesh needs, so a refusal that names the sources it refuses, in a table or a chain that accepts
// nothing and is entered only from chains whose policy accepts, is not counted.
func Refusing(ruleset string, ufwActive bool) []string {
var refusing []string
for _, f := range Filters(ruleset, nil, ufwActive) {
if f.Owner != OwnerOther || f.chain == userChain {
// A refusal in the runtime's user chain is reported as *other* and does not refuse
// adoption (novox/hq ADR 0168, rule 4): both predecessors kept their rules there.
continue
}
name := "table " + f.table
if len(refusing) == 0 || refusing[len(refusing)-1] != name {
if !contains(refusing, name) {
refusing = append(refusing, name)
}
}
}
return refusing
}
func contains(list []string, s string) bool {
for _, x := range list {
if x == s {
return true
}
}
return false
}
// nftRule is one line of a ruleset that refuses, with where it is.
type nftRule struct{ table, chain, line string }
// nftChain is what a parse knows about one chain.
type nftChain struct {
base, dropping, accepts bool
policyLine string
jumpedFrom []string
}
// nftRuleset is `nft list ruleset`, read: its tables in order, its chains, every refusing line,
// and which tables iptables-nft manages.
type nftRuleset struct {
tables []string
chains map[string]*nftChain // by "table\x00chain"
chainOrder []string
tableAccepts map[string]bool
refusals []nftRule
managed map[string]bool
}
func (r *nftRuleset) get(t, c string) *nftChain {
k := t + "\x00" + c
if r.chains[k] == nil {
r.chains[k] = &nftChain{}
r.chainOrder = append(r.chainOrder, k)
}
return r.chains[k]
}
func parseNft(ruleset string) *nftRuleset {
r := &nftRuleset{chains: map[string]*nftChain{}, tableAccepts: map[string]bool{}, managed: map[string]bool{}}
var table, chain string
for _, raw := range strings.Split(ruleset, "\n") {
line := strings.TrimSpace(raw)
switch {
case strings.HasPrefix(line, "# Warning: table ") && strings.Contains(line, "managed by iptables-nft"):
name := strings.TrimPrefix(line, "# Warning: table ")
name, _, _ = strings.Cut(name, " is managed")
r.managed[name] = true
continue
case strings.HasPrefix(line, "table "):
table = strings.TrimSuffix(strings.TrimSpace(strings.TrimPrefix(line, "table ")), "{")
table = strings.TrimSpace(table)
r.tables = append(r.tables, table)
chain = ""
continue
case strings.HasPrefix(line, "chain "):
chain = strings.TrimSpace(strings.TrimSuffix(strings.TrimPrefix(line, "chain "), "{"))
r.get(table, chain)
continue
case strings.HasPrefix(line, "set ") || strings.HasPrefix(line, "map ") ||
strings.HasPrefix(line, "flowtable "):
chain = ""
continue
case line == "" || line == "}" || strings.HasPrefix(line, "#") || chain == "":
continue
}
c := r.get(table, chain)
if strings.HasPrefix(line, "type ") {
c.base = true
c.policyLine = line
c.dropping = strings.Contains(line, "policy drop")
continue
}
for _, verb := range []string{"jump ", "goto "} {
if i := strings.Index(line, verb); i >= 0 {
target := strings.Fields(line[i+len(verb):])
if len(target) > 0 {
r.get(table, target[0]).jumpedFrom = append(r.get(table, target[0]).jumpedFrom, chain)
}
}
}
if accepts(line) {
c.accepts = true
r.tableAccepts[table] = true
}
if verdictRefuses(line) {
r.refusals = append(r.refusals, nftRule{table, chain, line})
}
}
return r
}
// onlyBans is whether a refusal only refuses the sources it names: in a table that accepts nothing
// and whose base chains all accept by default, or in a chain that accepts nothing and is entered
// only from base chains that accept by default.
func (r *nftRuleset) onlyBans(rule nftRule) bool {
if !bansSources(rule.line) {
return false
}
allAccepting := true
for k, c := range r.chains {
if strings.HasPrefix(k, rule.table+"\x00") && c.base && !strings.Contains(c.policyLine, "policy accept") {
allAccepting = false
}
}
if !r.tableAccepts[rule.table] && allAccepting {
return true
}
return r.enteredAccepting(rule.table, rule.chain, map[string]bool{})
}
// enteredAccepting is whether a chain accepts nothing and is entered only through chains that
// accept by default — base chains whose policy accepts, or chains that are themselves entered that
// way and accept nothing. A ban list jumped to from the runtime's user chain, which the forward
// chain enters with an accepting policy, is still a ban list.
func (r *nftRuleset) enteredAccepting(table, chain string, seen map[string]bool) bool {
if seen[chain] {
return false
}
seen[chain] = true
c := r.get(table, chain)
if c.base || c.accepts || len(c.jumpedFrom) == 0 {
return false
}
for _, from := range c.jumpedFrom {
caller := r.get(table, from)
if caller.base {
if !strings.Contains(caller.policyLine, "policy accept") {
return false
}
continue
}
if caller.accepts || !r.enteredAccepting(table, from, seen) {
return false
}
}
return true
}
// iptablesTable is whether a table is one iptables-nft writes. Named rather than read from the
// warning nft prints above it, because nft does not print that for every such table: a captured
// ruleset carried it on ip filter and not on ip raw, where the runtime keeps its drops.
func iptablesTable(table string) bool {
family, name, _ := strings.Cut(table, " ")
if family != "ip" && family != "ip6" {
return false
}
switch name {
case "filter", "nat", "raw", "mangle", "security":
return true
}
return false
}
// runtimes is whether a refusal in an iptables-nft table is the container runtime's own: in its
// DOCKER chains, its forward policy, or its guard against reaching a container's address directly
// from outside its bridge, in the raw table.
func runtimes(table, chain, line string) bool {
_, name, _ := strings.Cut(table, " ")
switch {
case strings.HasPrefix(chain, "DOCKER"):
return true
case name == "filter" && chain == "FORWARD" && strings.HasPrefix(line, "type "):
return true
case name == "raw" && chain == "PREROUTING":
return strings.Contains(line, "daddr") && strings.Contains(line, "iifname !=")
}
return false
}
// iptables-nft prints a REJECT target it cannot translate as `xt target "REJECT"`, measured in
// testdata/fail2ban-iptables.nft; a refusal written that way is a refusal too.
var verdict = regexp.MustCompile(`(^|\s)(drop|reject)(\s|$)|xt target "(DROP|REJECT)"`)
func verdictRefuses(line string) bool {
return verdict.MatchString(line)
}
var acceptVerdict = regexp.MustCompile(`(^|\s)accept(\s|;|$)|xt target "ACCEPT"`)
func accepts(line string) bool {
return acceptVerdict.MatchString(line)
}
// bansSources is whether a refusal names the sources it refuses — a set or an address — rather
// than refusing everyone but some.
func bansSources(line string) bool {
f := strings.Fields(line)
for i, w := range f {
if (w == "saddr" || w == "-s") && i+1 < len(f) && f[i+1] != "!=" && !strings.HasPrefix(f[i+1], "!") {
return i == 0 || f[i-1] != "!"
}
}
return false
}
// RefusingLegacy names the chains of an `iptables-legacy -S` that refuse traffic outside the
// container runtime's own. A ban — a refusal of the sources it names, in a chain that accepts
// nothing and is entered only from built-in chains whose policy accepts — is not counted, as in
// Refusing (testdata/fail2ban-iptables-S.txt).
func RefusingLegacy(rules string) []string {
policy := map[string]string{}
accepting := map[string]bool{}
jumpedFrom := map[string][]string{}
for _, line := range strings.Split(rules, "\n") {
fields := strings.Fields(line)
if len(fields) < 3 {
continue
}
switch fields[0] {
case "-P":
policy[fields[1]] = fields[2]
case "-A":
for i, f := range fields {
if (f == "-j" || f == "-g") && i+1 < len(fields) {
switch fields[i+1] {
case "ACCEPT":
accepting[fields[1]] = true
case "DROP", "REJECT", "RETURN", "LOG":
default:
jumpedFrom[fields[i+1]] = append(jumpedFrom[fields[i+1]], fields[1])
}
}
}
}
}
ban := func(chain, line string) bool {
if !bansSources(line) || accepting[chain] || len(jumpedFrom[chain]) == 0 {
return false
}
for _, from := range jumpedFrom[chain] {
if policy[from] != "ACCEPT" {
return false
}
}
return true
}
var refusing []string
seen := map[string]bool{}
for _, line := range strings.Split(rules, "\n") {
fields := strings.Fields(line)
if len(fields) < 3 {
continue
}
chain := fields[1]
refuses := false
switch fields[0] {
case "-P":
refuses = fields[2] == "DROP" && chain != "FORWARD"
case "-A":
for i, f := range fields {
if f == "-j" && i+1 < len(fields) && (fields[i+1] == "DROP" || fields[i+1] == "REJECT") {
refuses = !strings.HasPrefix(chain, "DOCKER") && !ban(chain, line)
}
}
}
if refuses && !seen[chain] {
seen[chain] = true
refusing = append(refusing, "chain "+chain)
}
}
return refusing
}
// --- ufw ---------------------------------------------------------------------------------------
// Mark is the comment every rule the mesh adds carries: whose it is, which opening, and a digest
// of the rule itself, so a rule the opening no longer describes is recognised as stale without the
// host having to know how ufw prints a rule back.
func Mark(o *declaration.Opening) string {
sum := sha256.Sum256([]byte(strings.Join(Rule(o), " ")))
return marker(o.ID) + " " + hex.EncodeToString(sum[:])[:8]
}
func marker(id string) string { return "mesh-host " + id }
// markedFor is whether a comment is the mesh's, for this opening.
func markedFor(comment, id string) bool {
return comment == marker(id) || strings.HasPrefix(comment, marker(id)+" ")
}
// Rule is the ufw rule an opening becomes, without its comment.
//
// incoming from everywhere allow proto tcp to any port P
// incoming from the mesh allow in on mesh0 proto tcp to any port P
// forwarded route allow [in on mesh0] proto tcp to any port <container port>
//
// A forwarded opening names the container's port because ufw's route rules are matched after the
// runtime's destination translation.
func Rule(o *declaration.Opening) []string {
var rule []string
port := o.Port
if o.Path == declaration.PathForwarded {
rule = append(rule, "route")
port = o.To
}
rule = append(rule, "allow")
if o.From == declaration.FromMesh {
rule = append(rule, "in", "on", MeshInterface)
}
return append(rule, "proto", o.Protocol, "to", "any", "port", strconv.Itoa(port))
}
var commentOf = regexp.MustCompile(`comment '([^']*)'`)
// added is every rule `ufw show added` lists, each without its leading "ufw".
func added(ctx context.Context, run Runner) ([]string, error) {
out, err := run(ctx, "ufw", "show", "added")
if err != nil {
return nil, fmt.Errorf("reading ufw's rules: %w", err)
}
var rules []string
for _, line := range strings.Split(out, "\n") {
line = strings.TrimSpace(line)
if strings.HasPrefix(line, "ufw ") {
rules = append(rules, strings.TrimPrefix(line, "ufw "))
}
}
return rules, nil
}
func comment(rule string) string {
m := commentOf.FindStringSubmatch(rule)
if m == nil {
return ""
}
return m[1]
}
// words splits a rule as ufw printed it into arguments, keeping a quoted comment whole.
func words(rule string) []string {
var out []string
var cur strings.Builder
quoted, any := false, false
for _, r := range rule {
switch {
case r == '\'':
quoted = !quoted
any = true
case r == ' ' && !quoted:
if any {
out = append(out, cur.String())
cur.Reset()
any = false
}
default:
cur.WriteRune(r)
any = true
}
}
if any {
out = append(out, cur.String())
}
return out
}
// A ufw rule, as `ufw show added` prints it or as it is given, reduced to what ufw compares.
//
// **ufw treats two rules that differ only in their comment as one rule.** Measured on a lab
// machine (testdata/ufw-comment-only.txt): adding `route allow proto tcp to any port 8080 comment
// 'mesh-host …'` beside an operator's `route allow 8080/tcp` answers "Rule updated", and the
// operator's rule now carries the mesh's mark — so removing the opening later would delete the
// operator's rule. The same holds for an incoming rule and for one with a comment of its own.
type ufwRule struct {
route bool
action, in, out string
// dir is which way the rule matches: "" (ufw's default, incoming and forwarded), "in" or
// "out". A rule on the outgoing path admits nothing that arrives.
dir string
log string
from, fromPort, to string
port, proto, app string
comment string
}
// parseRule reads a rule in either of ufw's forms — the short `allow 22/tcp` and the long `allow
// in on mesh0 to any port 5432 proto tcp` — into the fields ufw compares. Not ok for anything it
// does not recognise, which is then never taken to answer an opening.
func parseRule(rule string) (ufwRule, bool) {
r := ufwRule{from: "any", to: "any", comment: comment(rule)}
// A log type may stand after the action or after the direction; either way it is a property
// of the rule, not of where it matches. The word after `comment` is the comment, whatever it
// says.
var w []string
all := words(rule)
for i := 0; i < len(all); i++ {
switch {
case all[i] == "comment" && i+1 < len(all):
w = append(w, all[i], all[i+1])
i++
case all[i] == "log" || all[i] == "log-all":
r.log = all[i]
default:
w = append(w, all[i])
}
}
i := 0
if i < len(w) && w[i] == "route" {
r.route = true
i++
}
if i >= len(w) {
return r, false
}
switch w[i] {
case "allow", "deny", "reject", "limit":
r.action = w[i]
default:
return r, false
}
i++
for i < len(w) && (w[i] == "in" || w[i] == "out") {
dir := w[i]
i++
iface := ""
if i+1 < len(w) && w[i] == "on" {
iface = w[i+1]
i += 2
}
r.dir = dir
if dir == "in" {
r.in = iface
} else {
r.out = iface
}
}
if i < len(w) && w[i] != "from" && w[i] != "to" && w[i] != "proto" && w[i] != "comment" &&
w[i] != "app" && w[i] != "log" && w[i] != "log-all" {
// The short form: a port with its protocol, a bare port, or an application's name.
port, proto, hasProto := strings.Cut(w[i], "/")
if isPorts(port) {
r.port = port
if hasProto {
r.proto = proto
}
} else {
r.app = w[i]
}
i++
}
for ; i < len(w); i++ {
next := func() string {
if i+1 < len(w) {
i++
return w[i]
}
return ""
}
switch w[i] {
case "from":
r.from = next()
if i+1 < len(w) && w[i+1] == "port" {
i++
r.fromPort = next()
}
case "to":
r.to = next()
if i+1 < len(w) && w[i+1] == "port" {
i++
r.port = next()
}
case "port":
r.port = next()
case "proto":
r.proto = next()
case "app":
r.app = next()
case "comment":
next()
case "log", "log-all":
default:
return r, false
}
}
if r.proto == "any" {
r.proto = ""
}
// Incoming is ufw's default direction, and it prints `allow in 9005/tcp` back as
// `allow 9005/tcp` and merges the two — captured in testdata/ufw-direction.txt. An outgoing
// rule is its own rule and stays one.
if r.dir == "in" && r.in == "" {
r.dir = ""
}
return r, true
}
func isPorts(s string) bool {
if s == "" {
return false
}
for _, c := range s {
if (c < '0' || c > '9') && c != ':' && c != ',' {
return false
}
}
return true
}
// sameAs is whether ufw would take two rules for one: everything equal but the comment, the
// action and the log type. Adding one beside the other updates it in place — its comment, and its
// action or log type — rather than adding a second.
func (r ufwRule) sameAs(o ufwRule) bool {
r.comment, o.comment = "", ""
r.action, o.action = "", ""
r.log, o.log = "", ""
return r == o
}
// admits is whether a rule already lets through what an opening says: the same path, allowed from
// any source to any address of this machine, on the opening's port and protocol — or on any
// protocol — and on any interface, or the private network's for an opening from it.
func (r ufwRule) admits(o *declaration.Opening) bool {
want, ok := parseRule(strings.Join(Rule(o), " "))
if !ok || r.route != want.route || r.action != "allow" || r.app != "" ||
r.from != "any" || r.fromPort != "" || r.to != "any" {
return false
}
// A rule on the outgoing path lets this machine reach others; it admits nothing that arrives,
// so it never answers an opening.
if r.dir == "out" || r.out != "" {
return false
}
if r.proto != "" && r.proto != want.proto {
return false
}
if r.in != "" && r.in != want.in {
return false
}
return portsInclude(r.port, want.port)
}
// portsInclude is whether a ufw port list — 80, 80,443, or 8000:8100 — names a port.
func portsInclude(list, port string) bool {
p, err := strconv.Atoi(port)
if err != nil {
return false
}
for _, part := range strings.Split(list, ",") {
lo, hi, isRange := strings.Cut(part, ":")
a, err := strconv.Atoi(lo)
if err != nil {
continue
}
b := a
if isRange {
if b, err = strconv.Atoi(hi); err != nil {
continue
}
}
if a <= p && p <= b {
return true
}
}
return false
}
// Converged is what converging an opening did. SatisfiedBy names the rule already there that
// answers the opening, when one does; the mesh then adds nothing, and so will remove nothing.
type Converged struct {
Action string
SatisfiedBy string
}
// Converge makes one opening true in ufw: its marked rule present, and any rule marked for it that
// no longer describes it deleted. Nothing unmarked is touched. The outcome is created, updated or
// unchanged, read back from ufw rather than assumed.
//
// **An opening a rule already answers is not added** (novox/hq ADR 0103). If ufw holds a rule not
// marked for this opening that already admits what it says — the operator's, or one the mesh
// added for another opening — the opening is satisfied by it: adding the mesh's would take that
// rule over if it differs only in its comment, and removing the opening would then delete it.
func Converge(ctx context.Context, run Runner, o *declaration.Opening) (Converged, error) {
rules, err := added(ctx, run)
if err != nil {
return Converged{}, err
}
mark := Mark(o)
present := false
var stale []string
satisfiedBy := ""
want, _ := parseRule(strings.Join(Rule(o), " "))
for _, rule := range rules {
c := comment(rule)
switch {
case c == mark:
present = true
case markedFor(c, o.ID):
stale = append(stale, rule)
default:
parsed, ok := parseRule(rule)
if !ok {
continue
}
// ufw would take the mesh's rule for this one and rewrite its action or log type:
// an operator's refusal, or a limit, would silently become an allow — and removing the
// opening would then delete it. A plain allow answers the opening; anything else is a
// conflict the operator decides (novox/hq ADR 0103).
if parsed.sameAs(want) && (parsed.action != "allow" || parsed.log != "") {
return Converged{}, fmt.Errorf("ufw holds %q, which ufw takes for the same rule as the "+
"mesh's opening for %s, differing in what it does; adding the opening would change "+
"it, so nothing was added. Change or remove that rule, or have the mesh stop "+
"declaring the opening", rule, o.Target())
}
if satisfiedBy == "" && parsed.admits(o) {
satisfiedBy = rule
}
}
}
if present && len(stale) == 0 {
return Converged{Action: "unchanged"}, nil
}
for _, rule := range stale {
if _, err := run(ctx, "ufw", deletion(rule)...); err != nil {
return Converged{}, fmt.Errorf("deleting the mesh's stale ufw rule %q: %w", rule, err)
}
}
if !present && satisfiedBy != "" {
after, err := added(ctx, run)
if err != nil {
return Converged{}, err
}
for _, rule := range after {
if markedFor(comment(rule), o.ID) {
return Converged{}, fmt.Errorf("ufw still lists a stale rule marked for %s after deleting it", o.ID)
}
}
action := "unchanged"
if len(stale) > 0 {
action = "updated"
}
return Converged{Action: action, SatisfiedBy: satisfiedBy}, nil
}
if !present {
args := append(Rule(o), "comment", mark)
if _, err := run(ctx, "ufw", args...); err != nil {
return Converged{}, fmt.Errorf("adding the ufw rule for %s: %w", o.Target(), err)
}
}
after, err := added(ctx, run)
if err != nil {
return Converged{}, err
}
found, leftover := false, 0
for _, rule := range after {
c := comment(rule)
if c == mark {
found = true
} else if markedFor(c, o.ID) {
leftover++
}
}
if !found {
return Converged{}, fmt.Errorf("ufw was asked for %s and does not list it afterwards", o.Target())
}
if leftover > 0 {
return Converged{}, fmt.Errorf("ufw still lists %d stale rule(s) marked for %s after deleting them", leftover, o.ID)
}
if len(stale) > 0 {
return Converged{Action: "updated"}, nil
}
return Converged{Action: "created"}, nil
}
// Remove deletes the rules marked for one opening, and nothing else.
func Remove(ctx context.Context, run Runner, id string) (int, error) {
rules, err := added(ctx, run)
if err != nil {
return 0, err
}
removed := 0
for _, rule := range rules {
if !markedFor(comment(rule), id) {
continue
}
if _, err := run(ctx, "ufw", deletion(rule)...); err != nil {
return removed, fmt.Errorf("deleting the mesh's ufw rule %q: %w", rule, err)
}
removed++
}
after, err := added(ctx, run)
if err != nil {
return removed, err
}
for _, rule := range after {
if markedFor(comment(rule), id) {
return removed, fmt.Errorf("ufw still lists a rule marked for %s after deleting it", id)
}
}
return removed, nil
}
// Enable turns ufw back on, as found, and reads back that it is.
func Enable(ctx context.Context, run Runner) error {
if _, err := run(ctx, "ufw", "--force", "enable"); err != nil {
return fmt.Errorf("enabling ufw again: %w", err)
}
return expectActive(ctx, run, true)
}
// Disable retires ufw without flushing it: its configuration stays on disk, and the container
// runtime's rules are not its to remove.
//
// **Nor is the forward policy ufw's to open.** Measured on a lab machine running the container
// runtime with a published port (testdata/ufw-disable-iptables-before.txt and -after.txt):
// `ufw disable` sets every built-in chain's policy to accept, the forward chain's among them. The
// runtime had set that one to drop when it turned forwarding on, and it does not set it again while
// forwarding stays on — not even on a restart. Left so, a retired ufw turns the machine into a
// router for anyone who can reach it. So each family's forward policy is read before, and one that
// was drop is put back and read back. before is ForwardPolicies as read before the first attempt.
func Disable(ctx context.Context, run Runner, before map[string]string) error {
if _, err := run(ctx, "ufw", "disable"); err != nil {
return fmt.Errorf("disabling ufw: %w", err)
}
if err := expectActive(ctx, run, false); err != nil {
return err
}
for _, tool := range []string{"iptables", "ip6tables"} {
if before[tool] != "DROP" {
continue
}
if now, ok := forwardPolicy(ctx, run, tool); ok && now == "DROP" {
continue
}
if _, err := run(ctx, tool, "-P", "FORWARD", "DROP"); err != nil {
return fmt.Errorf("ufw is disabled, and %s's forward policy, which was drop, could not be put back: %w",
tool, err)
}
if now, ok := forwardPolicy(ctx, run, tool); !ok || now != "DROP" {
return fmt.Errorf("ufw is disabled, and %s's forward policy was put back to drop and reads %q",
tool, now)
}
}
return nil
}
// MeshTable is the derived filter's table, the thing that must be in force before the firewall
// found on a machine is retired.
const MeshTable = "inet mesh"
// MeshTableLoaded asks the machine whether the mesh's own filter is loaded. Read from the machine
// rather than assumed from the declaration: a table declared and not loaded is exactly the case
// where disabling the found firewall would leave the machine with nothing.
func MeshTableLoaded(ctx context.Context, run Runner) (bool, error) {
out, err := run(ctx, "nft", "list", "tables")
if err != nil {
if missing(err) {
return false, nil
}
return false, fmt.Errorf("cannot read which tables this machine has loaded: %w", err)
}
for _, line := range strings.Split(out, "\n") {
rest, ok := strings.CutPrefix(strings.TrimSpace(line), "table "+MeshTable)
if ok && (rest == "" || strings.HasPrefix(rest, " ") || strings.HasPrefix(rest, "{")) {
return true, nil
}
}
return false, nil
}
// ForwardPolicies reads each family's forward policy, by the tool that sets it. Read before ufw is
// disabled and kept by the caller, so a retirement that fails half-way is retried with what the
// machine had — not with what the half-done disable left.
func ForwardPolicies(ctx context.Context, run Runner) map[string]string {
out := map[string]string{}
for _, tool := range []string{"iptables", "ip6tables"} {
if policy, ok := forwardPolicy(ctx, run, tool); ok {
out[tool] = policy
}
}
return out
}
// forwardPolicy reads the forward chain's policy the way iptables prints it: "-P FORWARD DROP".
// Not ok when the tool is absent or says nothing readable.
func forwardPolicy(ctx context.Context, run Runner, tool string) (string, bool) {
out, err := run(ctx, tool, "-S", "FORWARD")
if err != nil {
return "", false
}
for _, line := range strings.Split(out, "\n") {
f := strings.Fields(line)
if len(f) == 3 && f[0] == "-P" && f[1] == "FORWARD" {
return f[2], true
}
}
return "", false
}
func expectActive(ctx context.Context, run Runner, want bool) error {
out, err := run(ctx, "ufw", "status")
if err != nil {
return fmt.Errorf("reading ufw's status back: %w", err)
}
if statusActive(out) != want {
state := "inactive"
if want {
state = "active"
}
return fmt.Errorf("ufw was asked to be %s and says: %s", state, strings.TrimSpace(out))
}
return nil
}
+789
View File
@@ -0,0 +1,789 @@
package firewall
import (
"context"
"errors"
"fmt"
"os"
"os/exec"
"strings"
"testing"
"github.com/novox/mesh-host/internal/declaration"
)
// Defends novox/hq ADR 0100: the firewall found on an adopted node stays in force, the mesh opens
// what it needs through it in its own terms, and removes only what it marked.
func dockerOnly(t *testing.T) string {
t.Helper()
// Captured from a real machine running the container runtime and nothing else that filters:
// its nat, filter and raw tables as iptables-nft writes them.
raw, err := os.ReadFile("testdata/docker-only.nft")
if err != nil {
t.Fatal(err)
}
return string(raw)
}
const aDroppingTable = `
table inet filter {
chain input {
type filter hook input priority filter; policy drop;
ct state established,related accept
tcp dport 22 accept
}
}
`
const ufwChains = `
# Warning: table ip filter is managed by iptables-nft, do not touch!
table ip filter {
chain INPUT {
type filter hook input priority filter; policy drop;
counter packets 0 bytes 0 jump ufw-before-input
}
chain ufw-user-input {
tcp dport 22 counter packets 0 bytes 0 accept
}
chain ufw-reject-input {
counter packets 0 bytes 0 reject
}
}
`
const theMeshsOwn = `
table inet mesh {
chain input {
type filter hook input priority filter; policy drop;
iif lo accept
}
}
table inet mesh_guard {
chain prerouting {
type filter hook prerouting priority raw; policy accept;
iifname != "lo" tcp dport { 5432, 15672 } drop
}
}
`
func TestTheContainerRuntimesOwnRulesAreNotAFirewall(t *testing.T) {
if got := Refusing(dockerOnly(t), false); len(got) != 0 {
t.Errorf("the runtime's own rules read as a firewall: %v", got)
}
}
func TestTheMeshsOwnTablesAreNotAFirewall(t *testing.T) {
if got := Refusing(dockerOnly(t)+theMeshsOwn, false); len(got) != 0 {
t.Errorf("the mesh's own tables read as a found firewall: %v", got)
}
}
func TestATableThatDropsIsAFirewall(t *testing.T) {
got := Refusing(dockerOnly(t)+aDroppingTable, false)
if len(got) != 1 || got[0] != "table inet filter" {
t.Errorf("a dropping table was not named: %v", got)
}
}
func TestUfwsOwnChainsAreUfwsWhenItIsActive(t *testing.T) {
if got := Refusing(dockerOnly(t)+ufwChains, true); len(got) != 0 {
t.Errorf("ufw's own chains read as a second firewall: %v", got)
}
if got := Refusing(dockerOnly(t)+ufwChains, false); len(got) == 0 {
t.Error("iptables rules that refuse, with ufw not active, were not counted")
}
}
func TestLegacyIptablesThatRefusesIsAFirewall(t *testing.T) {
docker := "-P INPUT ACCEPT\n-P FORWARD DROP\n-P OUTPUT ACCEPT\n-N DOCKER\n-A DOCKER -i docker0 -j DROP\n"
if got := RefusingLegacy(docker); len(got) != 0 {
t.Errorf("the runtime's legacy rules read as a firewall: %v", got)
}
if got := RefusingLegacy(docker + "-A INPUT -p tcp --dport 25 -j REJECT\n"); len(got) != 1 {
t.Errorf("a legacy reject was not counted: %v", got)
}
}
// fakeUFW is ufw as far as the host can see it: a status, and user rules it prints back in its
// own canonical form — deliberately not the order the host wrote them in.
type fakeUFW struct {
active bool
installed bool
rules []string
ruleset string
firewalld bool
asked []string
// iptablesActive and iptablesInactive are what `iptables -S` prints with ufw active and
// after it is disabled; empty is a machine without iptables. forward is a policy set since.
iptablesActive, iptablesInactive string
forward string
// noNft is a machine with no nft binary; iptablesRules is what `iptables -S` prints there.
noNft bool
iptablesRules string
}
// iptables answers `iptables -S FORWARD` from the captured output for ufw's state, and records
// a forward policy set with -P.
func (f *fakeUFW) iptables(name string, args []string) (string, error) {
if f.iptablesRules != "" && len(args) == 1 && args[0] == "-S" {
if name == "ip6tables" {
return "", nil
}
return f.iptablesRules, nil
}
if f.iptablesActive == "" {
return "", &exec.Error{Name: name, Err: exec.ErrNotFound}
}
if name == "ip6tables" {
return "", &exec.Error{Name: name, Err: exec.ErrNotFound}
}
if len(args) == 3 && args[0] == "-P" && args[1] == "FORWARD" {
f.forward = args[2]
return "", nil
}
captured := f.iptablesInactive
if f.active {
captured = f.iptablesActive
}
var out []string
for _, line := range strings.Split(captured, "\n") {
fields := strings.Fields(line)
if len(fields) >= 2 && fields[1] == "FORWARD" {
if fields[0] == "-P" && f.forward != "" && !f.active {
line = "-P FORWARD " + f.forward
}
out = append(out, line)
}
}
return strings.Join(out, "\n") + "\n", nil
}
// canonical is a rule the way ufw prints it back, as captured (testdata/ufw-comment-only.txt): the
// short form `allow 5671/tcp` for a rule on no interface, the long form `allow in on mesh0 to any
// port 5432 proto tcp` for one on an interface; the comment last.
func canonical(args []string) (rule, commentText string) {
var route, in, port, proto string
for i := 0; i < len(args); i++ {
switch args[i] {
case "route":
route = "route "
case "in":
in = args[i+2]
i += 2
case "port":
port = args[i+1]
i++
case "proto":
proto = args[i+1]
i++
case "comment":
commentText = args[i+1]
i++
}
}
if in != "" {
return route + "allow in on " + in + " to any port " + port + " proto " + proto, commentText
}
return route + "allow " + port + "/" + proto, commentText
}
func (f *fakeUFW) run(_ context.Context, name string, args ...string) (string, error) {
f.asked = append(f.asked, name+" "+strings.Join(args, " "))
switch name {
case "firewall-cmd":
if f.firewalld {
return "running\n", nil
}
return "", &exec.Error{Name: name, Err: exec.ErrNotFound}
case "nft":
if f.noNft {
return "", &exec.Error{Name: name, Err: exec.ErrNotFound}
}
return f.ruleset, nil
case "iptables-legacy", "ip6tables-legacy":
return "", &exec.Error{Name: name, Err: exec.ErrNotFound}
case "iptables", "ip6tables":
return f.iptables(name, args)
case "ufw":
default:
return "", fmt.Errorf("unexpected %s", name)
}
if !f.installed {
return "", &exec.Error{Name: name, Err: exec.ErrNotFound}
}
switch {
case args[0] == "status":
if f.active {
return "Status: active\n\nTo Action From\n", nil
}
return "Status: inactive\n", nil
case args[0] == "show":
out := "Added user rules (see 'ufw status' for running firewall):\n"
for _, r := range f.rules {
out += "ufw " + r + "\n"
}
return out, nil
case args[0] == "--force" && args[1] == "enable":
f.active = true
return "Firewall is active and enabled on system startup\n", nil
case args[0] == "disable":
f.active = false
f.forward = ""
return "Firewall stopped and disabled on system startup\n", nil
case args[0] == "delete" && len(args) > 1 && args[1] == "route":
// As the real ufw answers it (captured in testdata/ufw-delete.txt): a route rule is
// deleted with `route delete`, never `delete route`.
return "", errors.New("ERROR: Invalid syntax")
case args[0] == "delete", args[0] == "route" && len(args) > 1 && args[1] == "delete":
rest := args[1:]
if args[0] == "route" {
rest = append([]string{"route"}, args[2:]...)
}
for i, r := range f.rules {
if strings.Join(words(r), "\x00") == strings.Join(rest, "\x00") {
f.rules = append(f.rules[:i], f.rules[i+1:]...)
return "Rule deleted\n", nil
}
}
return "", errors.New("Could not delete non-existent rule")
default:
rule, note := canonical(args)
line := rule
if note != "" {
line += " comment '" + note + "'"
}
// As the real ufw does (testdata/ufw-comment-only.txt): a rule differing from one it holds
// only in its comment is the same rule, and its comment is replaced.
for i, r := range f.rules {
if bare, _, _ := strings.Cut(r, " comment '"); bare == rule {
f.rules[i] = line
return "Rule updated\nRule updated (v6)\n", nil
}
}
f.rules = append(f.rules, line)
return "Rule added\nRule added (v6)\n", nil
}
}
func (f *fakeUFW) added() int {
n := 0
for _, a := range f.asked {
if strings.HasPrefix(a, "ufw allow") || strings.HasPrefix(a, "ufw route") {
n++
}
}
return n
}
func opening(id string, port int, from, path string, to int) *declaration.Opening {
return &declaration.Opening{ID: id, Type: declaration.TypeOpening, Port: port, Protocol: "tcp",
From: from, Path: path, To: to}
}
func TestAnOpeningBecomesTheUfwRuleForItsPathAndOrigin(t *testing.T) {
for _, c := range []struct {
o *declaration.Opening
want string
}{
{opening("adoption.a", 5671, "everywhere", "incoming", 0), "allow proto tcp to any port 5671"},
{opening("adoption.b", 5432, "mesh", "incoming", 0), "allow in on mesh0 proto tcp to any port 5432"},
{opening("adoption.c", 20001, "everywhere", "forwarded", 8080), "route allow proto tcp to any port 8080"},
{opening("adoption.d", 20001, "mesh", "forwarded", 8080), "route allow in on mesh0 proto tcp to any port 8080"},
} {
if got := strings.Join(Rule(c.o), " "); got != c.want {
t.Errorf("%s: %q, want %q", c.o.ID, got, c.want)
}
}
}
func TestAnOpeningIsAddedOnceAndMarkedAsTheMeshs(t *testing.T) {
f := &fakeUFW{installed: true, active: true, rules: []string{"allow 22/tcp"}}
o := opening("adoption.opening-tcp-5671-incoming", 5671, "everywhere", "incoming", 0)
action, err := Converge(context.Background(), f.run, o)
if err != nil || action.Action != "created" {
t.Fatalf("first converge: %q %v", action, err)
}
if !strings.Contains(f.rules[1], "comment 'mesh-host adoption.opening-tcp-5671-incoming ") {
t.Errorf("the rule is not marked as the mesh's: %v", f.rules)
}
action, err = Converge(context.Background(), f.run, o)
if err != nil || action.Action != "unchanged" {
t.Fatalf("second converge: %q %v", action, err)
}
if f.added() != 1 {
t.Errorf("re-converging added again: %v", f.asked)
}
}
func TestAnOpeningLostToAReloadIsAddedAgain(t *testing.T) {
f := &fakeUFW{installed: true, active: true}
o := opening("adoption.x", 5671, "everywhere", "incoming", 0)
if _, err := Converge(context.Background(), f.run, o); err != nil {
t.Fatal(err)
}
f.rules = nil // what a reload that lost the rule leaves
action, err := Converge(context.Background(), f.run, o)
if err != nil || action.Action != "created" || len(f.rules) != 1 {
t.Fatalf("a lost opening was not put back: %q %v %v", action, err, f.rules)
}
}
func TestAChangedOpeningReplacesOnlyItsOwnRule(t *testing.T) {
f := &fakeUFW{installed: true, active: true, rules: []string{"allow 22/tcp", "allow 8080/tcp comment 'someone else'"}}
if _, err := Converge(context.Background(), f.run, opening("adoption.x", 5671, "everywhere", "incoming", 0)); err != nil {
t.Fatal(err)
}
action, err := Converge(context.Background(), f.run, opening("adoption.x", 5671, "mesh", "incoming", 0))
if err != nil || action.Action != "updated" {
t.Fatalf("%q %v", action, err)
}
if len(f.rules) != 3 || f.rules[0] != "allow 22/tcp" || f.rules[1] != "allow 8080/tcp comment 'someone else'" ||
!strings.Contains(f.rules[2], "in on mesh0") {
t.Errorf("rules afterwards: %v", f.rules)
}
}
func TestRemovingAnOpeningRemovesOnlyWhatWasMarkedForIt(t *testing.T) {
f := &fakeUFW{installed: true, active: true, rules: []string{"allow 22/tcp", "allow 8080/tcp"}}
for _, o := range []*declaration.Opening{
opening("adoption.a", 5671, "everywhere", "incoming", 0),
opening("adoption.ab", 5000, "everywhere", "incoming", 0),
} {
if _, err := Converge(context.Background(), f.run, o); err != nil {
t.Fatal(err)
}
}
n, err := Remove(context.Background(), f.run, "adoption.a")
if err != nil || n != 1 {
t.Fatalf("removed %d: %v", n, err)
}
if len(f.rules) != 3 || f.rules[0] != "allow 22/tcp" || f.rules[1] != "allow 8080/tcp" ||
!strings.Contains(f.rules[2], "adoption.ab") {
t.Errorf("more than the marked rule went: %v", f.rules)
}
}
func TestEnableAndDisableReadBack(t *testing.T) {
f := &fakeUFW{installed: true, active: true}
if err := Disable(context.Background(), f.run, nil); err != nil || f.active {
t.Fatalf("disable: %v", err)
}
if err := Enable(context.Background(), f.run); err != nil || !f.active {
t.Fatalf("enable: %v", err)
}
for _, a := range f.asked {
if strings.Contains(a, "reset") || strings.Contains(a, "flush") {
t.Errorf("the found firewall was reset: %s", a)
}
}
}
func TestDetectingTheFoundFirewall(t *testing.T) {
for _, c := range []struct {
name string
f *fakeUFW
want Kind
}{
{"nothing but the runtime", &fakeUFW{ruleset: dockerOnly(t)}, None},
{"ufw active", &fakeUFW{installed: true, active: true, ruleset: dockerOnly(t) + ufwChains}, UFW},
{"ufw installed and inactive", &fakeUFW{installed: true, ruleset: dockerOnly(t)}, None},
{"firewalld", &fakeUFW{firewalld: true, ruleset: dockerOnly(t)}, Unsupported},
{"an nftables table of its own", &fakeUFW{ruleset: dockerOnly(t) + aDroppingTable}, Unsupported},
{"ufw beside an nftables table", &fakeUFW{installed: true, active: true, ruleset: dockerOnly(t) + ufwChains + aDroppingTable}, Unsupported},
} {
got, name, err := Detect(context.Background(), c.f.run)
if err != nil {
t.Fatalf("%s: %v", c.name, err)
}
if got != c.want {
t.Errorf("%s: detected %s (%s), want %s", c.name, got, name, c.want)
}
if got == Unsupported && name == "" {
t.Errorf("%s: an unsupported firewall was not named", c.name)
}
}
}
// The fixtures below were captured from a real ufw 0.36.2 on a lab machine, not written by hand:
// ufw prints a rule back in its own shorter form, so the mark in the comment is the only thing the
// host relies on.
func TestTheMarksAreReadFromWhatUfwReallyPrints(t *testing.T) {
raw, err := os.ReadFile("testdata/ufw-show-added.txt")
if err != nil {
t.Fatal(err)
}
run := func(ctx context.Context, name string, args ...string) (string, error) { return string(raw), nil }
rules, err := added(context.Background(), run)
if err != nil {
t.Fatal(err)
}
if len(rules) != 7 {
t.Fatalf("read %d rules, want 7: %q", len(rules), rules)
}
marked := 0
for _, r := range rules {
if strings.HasPrefix(comment(r), "mesh-host ") {
marked++
}
}
if marked != 5 {
t.Errorf("read %d marked rules, want 5", marked)
}
if !markedFor(comment(rules[5]), "adoption.opening-tcp-8443-forwarded") {
t.Errorf("the forwarded rule from the mesh lost its mark: %q", rules[5])
}
}
func TestEveryRealRuleIsDeletedInTheFormUfwAccepts(t *testing.T) {
raw, err := os.ReadFile("testdata/ufw-show-added.txt")
if err != nil {
t.Fatal(err)
}
run := func(ctx context.Context, name string, args ...string) (string, error) { return string(raw), nil }
rules, _ := added(context.Background(), run)
// Each of these was run on the lab machine and answered "Rule deleted" (testdata/ufw-delete.txt).
want := map[string]string{
"allow 5671/tcp comment 'mesh-host adoption.opening-tcp-5671-incoming 1a2b3c4d'": "delete allow 5671/tcp comment|mesh-host adoption.opening-tcp-5671-incoming 1a2b3c4d",
"allow in on mesh0 to any port 5432 proto tcp comment 'mesh-host adoption.opening-tcp-5432-incoming deadbeef'": "delete allow in on mesh0 to any port 5432 proto tcp comment|mesh-host adoption.opening-tcp-5432-incoming deadbeef",
"route allow 80/tcp comment 'mesh-host adoption.opening-tcp-8081-forwarded 0badf00d'": "route delete allow 80/tcp comment|mesh-host adoption.opening-tcp-8081-forwarded 0badf00d",
"route allow in on mesh0 to any port 443 proto tcp comment 'mesh-host adoption.opening-tcp-8443-forwarded cafe0001'": "route delete allow in on mesh0 to any port 443 proto tcp comment|mesh-host adoption.opening-tcp-8443-forwarded cafe0001",
}
seen := 0
for _, r := range rules {
w, ok := want[r]
if !ok {
continue
}
seen++
d := deletion(r)
got := strings.Join(d[:len(d)-1], " ") + "|" + d[len(d)-1]
if got != w {
t.Errorf("deleting %q\n got %s\n want %s", r, got, w)
}
}
if seen != len(want) {
t.Errorf("matched %d of %d captured rules", seen, len(want))
}
}
func TestARealUfwRulesetIsUfw(t *testing.T) {
raw, err := os.ReadFile("testdata/ufw-active.nft")
if err != nil {
t.Fatal(err)
}
status, err := os.ReadFile("testdata/ufw-status-active.txt")
if err != nil {
t.Fatal(err)
}
if !statusActive(string(status)) {
t.Fatal("the captured status does not read as active")
}
if refusing := Refusing(string(raw), true); len(refusing) > 0 {
t.Errorf("a machine with ufw active and nothing else read as refusing in %v", refusing)
}
if refusing := Refusing(string(raw), false); len(refusing) == 0 {
t.Error("ufw's drop chains, with ufw not known to be active, read as refusing nothing")
}
}
func TestRetiringUfwKeepsTheMachineFromRoutingForOthers(t *testing.T) {
// Captured on a lab machine running the container runtime with a published port: ufw active,
// then `ufw disable`. Disabling set the forward policy the runtime had set to drop to accept.
before, err := os.ReadFile("testdata/ufw-disable-iptables-before.txt")
if err != nil {
t.Fatal(err)
}
after, err := os.ReadFile("testdata/ufw-disable-iptables-after.txt")
if err != nil {
t.Fatal(err)
}
if !strings.Contains(string(before), "-P FORWARD DROP") || !strings.Contains(string(after), "-P FORWARD ACCEPT") {
t.Fatal("the captures no longer show ufw disable opening the forward policy")
}
f := &fakeUFW{installed: true, active: true, iptablesActive: string(before), iptablesInactive: string(after)}
if err := Disable(context.Background(), f.run, ForwardPolicies(context.Background(), f.run)); err != nil {
t.Fatal(err)
}
if f.active {
t.Fatal("ufw is still active")
}
if f.forward != "DROP" {
t.Errorf("the forward policy was left open after ufw was retired: %v", f.asked)
}
for _, a := range f.asked {
if strings.Contains(a, "-F") || strings.Contains(a, "flush") || strings.Contains(a, "reset") {
t.Errorf("retiring ufw flushed something: %s", a)
}
}
}
func TestRetiringUfwOnAMachineWithoutIptablesStillRetiresIt(t *testing.T) {
f := &fakeUFW{installed: true, active: true}
if err := Disable(context.Background(), f.run, nil); err != nil || f.active {
t.Fatalf("disable: %v, active %v", err, f.active)
}
}
// Captured on a lab machine with fail2ban banning one documentation address in its sshd jail,
// once with its nftables backend and once with its iptables backend (iptables-nft), ufw inactive.
func captured(t *testing.T, name string) string {
t.Helper()
raw, err := os.ReadFile("testdata/" + name)
if err != nil {
t.Fatal(err)
}
return string(raw)
}
func TestFail2bansBansAreNotAFirewall(t *testing.T) {
for _, name := range []string{"fail2ban-nftables.nft", "fail2ban-iptables.nft"} {
ruleset := captured(t, name)
if !strings.Contains(ruleset, "192.0.2.55") {
t.Fatalf("%s holds no ban", name)
}
if got := Refusing(ruleset, false); len(got) != 0 {
t.Errorf("%s: fail2ban's bans read as a firewall: %v", name, got)
}
kind, what, err := Detect(context.Background(), (&fakeUFW{ruleset: ruleset}).run)
if err != nil || kind != None {
t.Errorf("%s: a machine with only fail2ban detected as %s (%s) %v", name, kind, what, err)
}
}
if got := RefusingLegacy(captured(t, "fail2ban-iptables-S.txt")); len(got) != 0 {
t.Errorf("fail2ban's iptables bans read as a firewall: %v", got)
}
}
func TestARefusalOfEveryoneButSomeIsStillAFirewall(t *testing.T) {
// A ban names the sources it refuses. A table that refuses every source but some, or every
// port but some, closes what the mesh would open, whatever its policy says.
for name, table := range map[string]string{
"all but a range": "table inet own {\n\tchain input {\n\t\ttype filter hook input priority filter; policy accept;\n\t\tip saddr != 10.0.0.0/8 drop\n\t}\n}\n",
"all but ssh": "table inet own {\n\tchain input {\n\t\ttype filter hook input priority filter; policy accept;\n\t\ttcp dport != 22 drop\n\t}\n}\n",
"iptables reject": "# Warning: table ip filter is managed by iptables-nft, do not touch!\ntable ip filter {\n\tchain INPUT {\n\t\ttype filter hook input priority filter; policy accept;\n\t\tcounter packets 0 bytes 0 xt target \"REJECT\"\n\t}\n}\n",
"ban beside a dropping policy": "table inet own {\n\tchain input {\n\t\ttype filter hook input priority filter; policy drop;\n\t\tip saddr 192.0.2.9 drop\n\t}\n}\n",
} {
if got := Refusing(dockerOnly(t)+table, false); len(got) == 0 {
t.Errorf("%s: not counted as a firewall", name)
}
}
legacy := "-P INPUT ACCEPT\n-N own\n-A INPUT -j own\n-A own ! -s 10.0.0.0/8 -j DROP\n"
if got := RefusingLegacy(legacy); len(got) == 0 {
t.Error("a legacy refusal of all but a range was not counted")
}
}
// Defends novox/hq ADR 0103: an opening a found rule already answers is not added, because ufw
// takes two rules differing only in their comment for one (testdata/ufw-comment-only.txt).
func TestUfwTakesTheMeshsRuleAndTheOperatorsForOne(t *testing.T) {
// The capture: each mesh rule answered "Rule updated" beside the operator's equivalent.
raw := captured(t, "ufw-comment-only.txt")
if strings.Count(raw, "Rule updated\n") != 3 {
t.Fatalf("the capture no longer shows ufw updating an equivalent rule:\n%s", raw)
}
for _, c := range []struct {
operators string
o *declaration.Opening
}{
{"route allow 8080/tcp", opening("adoption.opening-tcp-8080-forwarded", 20001, "everywhere", "forwarded", 8080)},
{"allow 5671/tcp", opening("adoption.opening-tcp-5671-incoming", 5671, "everywhere", "incoming", 0)},
{"allow in on mesh0 to any port 5432 proto tcp comment 'operator note'", opening("adoption.opening-tcp-5432-incoming", 5432, "mesh", "incoming", 0)},
} {
theirs, ok := parseRule(c.operators)
mine, ok2 := parseRule(strings.Join(Rule(c.o), " ") + " comment '" + Mark(c.o) + "'")
if !ok || !ok2 || !theirs.sameAs(mine) {
t.Errorf("%q and the mesh's %v are one rule to ufw, and read as two", c.operators, Rule(c.o))
}
if !theirs.admits(c.o) {
t.Errorf("%q does not read as answering %s", c.operators, c.o.Target())
}
}
}
func TestEveryCapturedRuleFormIsRead(t *testing.T) {
want := map[string]string{
"allow 22/tcp": "tcp 22 in= from=any", "allow 9200": " 9200 in= from=any",
"allow from 192.0.2.0/24 to any port 9300 proto tcp": "tcp 9300 in= from=192.0.2.0/24",
"allow in on eth0 to any port 9301 proto tcp": "tcp 9301 in=eth0 from=any",
"allow 9500:9510/tcp": "tcp 9500:9510 in= from=any",
"allow 80,443/tcp": "tcp 80,443 in= from=any",
"allow in on mesh0 to any port 5432 proto tcp": "tcp 5432 in=mesh0 from=any",
"route allow 8080/tcp": "tcp 8080 in= from=any",
"allow 9900/tcp": "tcp 9900 in= from=any",
"allow out 5671/tcp": "tcp 5671 in= from=any",
"deny out 5672/tcp": "tcp 5672 in= from=any",
"allow out on eth0 to any port 5673 proto tcp": "tcp 5673 in= from=any",
"allow log 9001/tcp": "tcp 9001 in= from=any",
"route allow log 8084/tcp": "tcp 8084 in= from=any",
"allow in on mesh0 log-all to any port 9002 proto tcp": "tcp 9002 in=mesh0 from=any",
}
rules, err := added(context.Background(), func(context.Context, string, ...string) (string, error) {
return captured(t, "ufw-forms.txt"), nil
})
if err != nil || len(rules) != 21 {
t.Fatalf("read %d rules: %v", len(rules), err)
}
for _, rule := range rules {
r, ok := parseRule(rule)
if !ok {
t.Errorf("a rule ufw printed was not read: %q", rule)
continue
}
if w, listed := want[rule]; listed {
if got := r.proto + " " + r.port + " in=" + r.in + " from=" + r.from; got != w {
t.Errorf("%q read as %q, want %q", rule, got, w)
}
}
}
}
func TestAnOpeningAFoundRuleAnswersIsNotAddedAndItsRemovalLeavesTheRule(t *testing.T) {
for _, c := range []struct {
name, operators string
o *declaration.Opening
}{
{"forwarded, the same rule", "route allow 8080/tcp", opening("adoption.fwd", 20001, "everywhere", "forwarded", 8080)},
{"incoming, the same rule", "allow 5671/tcp", opening("adoption.bus", 5671, "everywhere", "incoming", 0)},
{"with a comment of its own", "allow in on mesh0 to any port 5432 proto tcp comment 'operator note'", opening("adoption.store", 5432, "mesh", "incoming", 0)},
{"broader: from anywhere", "allow 5432/tcp", opening("adoption.store", 5432, "mesh", "incoming", 0)},
{"broader: any protocol, a range", "allow 5000:5100", opening("adoption.registry", 5000, "everywhere", "incoming", 0)},
} {
f := &fakeUFW{installed: true, active: true, rules: []string{"allow 22/tcp", c.operators}}
done, err := Converge(context.Background(), f.run, c.o)
if err != nil {
t.Fatalf("%s: %v", c.name, err)
}
if done.SatisfiedBy != c.operators || done.Action != "unchanged" || f.added() != 0 {
t.Errorf("%s: %+v, asked %v", c.name, done, f.asked)
}
if n, err := Remove(context.Background(), f.run, c.o.ID); err != nil || n != 0 {
t.Errorf("%s: removing the opening removed %d: %v", c.name, n, err)
}
if len(f.rules) != 2 || f.rules[1] != c.operators {
t.Errorf("%s: the operator's rule did not survive: %v", c.name, f.rules)
}
}
}
func TestARuleThatDoesNotAnswerTheOpeningLeavesItToBeAdded(t *testing.T) {
for _, operators := range []string{
"allow from 192.0.2.0/24 to any port 5671 proto tcp", // narrower: from one range
"allow in on eth0 to any port 5671 proto tcp", // narrower: one interface
"allow 5671/udp", // another protocol
"route allow 5671/tcp", // another path
"allow to 192.0.2.1 port 5671 proto tcp", // one address
} {
f := &fakeUFW{installed: true, active: true, rules: []string{operators}}
done, err := Converge(context.Background(), f.run, opening("adoption.bus", 5671, "everywhere", "incoming", 0))
if err != nil || done.Action != "created" || done.SatisfiedBy != "" {
t.Errorf("%q: %+v %v", operators, done, err)
}
}
}
func TestAnOpeningWhoseFoundRuleIsGoneIsAddedAgain(t *testing.T) {
f := &fakeUFW{installed: true, active: true, rules: []string{"allow 5671/tcp"}}
o := opening("adoption.bus", 5671, "everywhere", "incoming", 0)
if done, err := Converge(context.Background(), f.run, o); err != nil || done.SatisfiedBy == "" {
t.Fatalf("%+v %v", done, err)
}
f.rules = nil // the operator deleted theirs
if done, err := Converge(context.Background(), f.run, o); err != nil || done.Action != "created" {
t.Fatalf("the opening was not added once nothing answered it: %+v %v", done, err)
}
}
func TestARuleUfwWouldMergeThatDoesOtherThanAllowRefusesTheOpening(t *testing.T) {
// ufw takes two rules differing only in action or log type for one, and adding the mesh's
// would turn the operator's refusal into an allow (novox/hq ADR 0103).
for _, operators := range []string{
"deny 5671/tcp",
"reject 5671/tcp",
"limit 5671/tcp",
"allow log 5671/tcp",
"allow log-all proto tcp to any port 5671",
"deny in log to any port 5671 proto tcp comment 'operator note'",
} {
f := &fakeUFW{installed: true, active: true, rules: []string{operators}}
_, err := Converge(context.Background(), f.run, opening("adoption.bus", 5671, "everywhere", "incoming", 0))
if err == nil || !strings.Contains(err.Error(), operators) {
t.Errorf("%q: the conflict was not refused naming the rule: %v", operators, err)
}
if f.added() != 0 || len(f.rules) != 1 || f.rules[0] != operators {
t.Errorf("%q: something was added or changed: %v %v", operators, f.asked, f.rules)
}
}
}
func TestALogTypeIsReadInEitherPlace(t *testing.T) {
for rule, want := range map[string]string{
"allow log 22/tcp": "allow log 22 tcp in=",
"allow in log-all on mesh0 to any port 5432 proto tcp": "allow log-all 5432 tcp in=mesh0",
"route deny log in on mesh0 to any port 80 proto tcp": "deny log 80 tcp in=mesh0",
"allow 22/tcp comment 'log'": "allow 22 tcp in=",
} {
r, ok := parseRule(rule)
if got := r.action + " " + r.log + " " + r.port + " " + r.proto + " in=" + r.in; !ok || got != want {
t.Errorf("%q read as %q (%v), want %q", rule, got, ok, want)
}
}
}
func TestAnOutgoingRuleNeverAnswersAnOpening(t *testing.T) {
// `ufw allow out 5671/tcp` lets this machine reach others; nothing arrives through it, and
// ufw keeps it as a rule of its own — captured in testdata/ufw-direction.txt.
raw := captured(t, "ufw-direction.txt")
if !strings.Contains(raw, "ufw allow out 9007/tcp\nufw allow 9007/tcp") {
t.Fatalf("the capture no longer shows an outgoing rule standing beside an incoming one:\n%s", raw)
}
for _, operators := range []string{"allow out 5671/tcp", "allow out on eth0 to any port 5671 proto tcp",
"deny out 5671/tcp"} {
f := &fakeUFW{installed: true, active: true, rules: []string{operators}}
done, err := Converge(context.Background(), f.run, opening("adoption.bus", 5671, "everywhere", "incoming", 0))
if err != nil {
t.Errorf("%q: an outgoing rule was taken for a conflict: %v", operators, err)
continue
}
if done.Action != "created" || done.SatisfiedBy != "" {
t.Errorf("%q: an outgoing rule answered an incoming opening: %+v", operators, done)
}
}
}
func TestIncomingIsUfwsDefaultDirection(t *testing.T) {
// Captured: `deny in 9006/tcp` and `allow 9006/tcp` are one rule to ufw, so the mesh must read
// them as one too, or it would take an operator's refusal over.
raw := captured(t, "ufw-direction.txt")
if !strings.Contains(raw, "ufw allow 9005/tcp") || strings.Contains(raw, "ufw deny 9006/tcp") {
t.Fatalf("the capture no longer shows `in` as the default direction:\n%s", raw)
}
f := &fakeUFW{installed: true, active: true, rules: []string{"deny in to any port 5671 proto tcp"}}
if _, err := Converge(context.Background(), f.run, opening("adoption.bus", 5671, "everywhere", "incoming", 0)); err == nil {
t.Error("an incoming refusal ufw would merge was not refused")
}
}
func TestAMachineWithIptablesRulesAndNoNftIsNotReadAsUnfiltered(t *testing.T) {
// nft is not installed, and iptables-nft holds a firewall of somebody's. Read as "nothing
// filters here" the mesh would adopt it, open nothing, and be unreachable (novox/hq ADR 0100).
rules := "-P INPUT DROP\n-P FORWARD DROP\n-P OUTPUT ACCEPT\n-A INPUT -p tcp -m tcp --dport 22 -j ACCEPT\n"
f := &fakeUFW{noNft: true, iptablesRules: rules}
kind, what, err := Detect(context.Background(), f.run)
if err != nil {
t.Fatal(err)
}
if kind != Unsupported {
t.Errorf("a machine filtered by iptables with no nft read as %s (%s)", kind, what)
}
// And a machine with nothing but the runtime's own rules and no nft is still unfiltered.
docker := "-P INPUT ACCEPT\n-P FORWARD DROP\n-P OUTPUT ACCEPT\n-N DOCKER\n-A DOCKER -i docker0 -j DROP\n"
if kind, _, err := Detect(context.Background(), (&fakeUFW{noNft: true, iptablesRules: docker}).run); err != nil || kind != None {
t.Errorf("a machine with only the runtime's rules read as %s: %v", kind, err)
}
}
+588
View File
@@ -0,0 +1,588 @@
# Warning: table ip filter is managed by iptables-nft, do not touch!
table ip filter {
chain INPUT {
type filter hook input priority filter; policy accept;
ip protocol tcp counter packets 945757787 bytes 1737008792038 jump f2b-sshd
ip protocol tcp counter packets 945756610 bytes 1737008898620 jump f2b-recidive
counter packets 2862213204 bytes 3144751431654 jump ufw-before-logging-input
counter packets 2862213204 bytes 3144751431654 jump ufw-before-input
counter packets 989333889 bytes 1776344988272 jump ufw-after-input
counter packets 989303248 bytes 1776343408920 jump ufw-after-logging-input
counter packets 989303248 bytes 1776343408920 jump ufw-reject-input
counter packets 989303248 bytes 1776343408920 jump ufw-track-input
}
chain FORWARD {
type filter hook forward priority filter; policy accept;
oifname "mesh0" counter packets 1613103 bytes 2577614868 accept
iifname "mesh0" counter packets 995195 bytes 84526284 accept
counter packets 20454697 bytes 11504107676 jump DOCKER-USER
counter packets 20442192 bytes 11503368404 jump DOCKER-FORWARD
counter packets 12438285 bytes 10907281833 jump ufw-before-logging-forward
counter packets 12438285 bytes 10907281833 jump ufw-before-forward
counter packets 384 bytes 39643 jump ufw-after-forward
counter packets 384 bytes 39643 jump ufw-after-logging-forward
counter packets 384 bytes 39643 jump ufw-reject-forward
counter packets 384 bytes 39643 jump ufw-track-forward
}
chain OUTPUT {
type filter hook output priority filter; policy accept;
counter packets 3195070897 bytes 3951725261199 jump ufw-before-logging-output
counter packets 3195070897 bytes 3951725261199 jump ufw-before-output
counter packets 945745931 bytes 1778546547406 jump ufw-after-output
counter packets 945745931 bytes 1778546547406 jump ufw-after-logging-output
counter packets 945745931 bytes 1778546547406 jump ufw-reject-output
counter packets 945745931 bytes 1778546547406 jump ufw-track-output
}
chain DOCKER-FORWARD {
counter packets 20442192 bytes 11503368404 jump DOCKER-CT
counter packets 8079126 bytes 1230017985 jump DOCKER-INTERNAL
counter packets 8079126 bytes 1230017985 jump DOCKER-BRIDGE
iifname "br-cadedce55fe9" counter packets 0 bytes 0 accept
iifname "br-dd007c7e67bc" counter packets 0 bytes 0 accept
iifname "br-a5fbc29c2c2a" counter packets 0 bytes 0 accept
iifname "br-6eb1e7f7f847" counter packets 0 bytes 0 accept
iifname "br-8ce143481a5b" counter packets 14700 bytes 2493600 accept
iifname "br-84e7d0cfeada" counter packets 0 bytes 0 accept
iifname "br-f8b083119d99" counter packets 264 bytes 57438 accept
iifname "br-0d1490cc67c9" counter packets 732468 bytes 351624109 accept
iifname "br-3b338a381229" counter packets 137 bytes 11876 accept
iifname "br-3008d408e73a" counter packets 25380 bytes 1564417 accept
iifname "br-ca07a9577a7f" counter packets 0 bytes 0 accept
iifname "docker0" counter packets 6695791 bytes 795364128 accept
iifname "br-a63fa64a9e18" counter packets 0 bytes 0 accept
iifname "br-1ccb887b3344" counter packets 237174 bytes 36804979 accept
iifname "br-9fd22324ec08" counter packets 0 bytes 0 accept
iifname "br-73641cceafc3" counter packets 36 bytes 6614 accept
iifname "br-77eb8a9e2ba1" counter packets 0 bytes 0 accept
iifname "br-e99ce5248c84" counter packets 0 bytes 0 accept
iifname "br-e5d78502832d" counter packets 0 bytes 0 accept
iifname "br-2e4a76a7cd2e" counter packets 20339 bytes 1799711 accept
iifname "br-72fd626a8ff7" counter packets 0 bytes 0 accept
}
chain DOCKER-USER {
ip protocol tcp counter packets 3054221 bytes 3329963005 jump f2b-sshd
ip protocol tcp counter packets 3054221 bytes 3329963005 jump f2b-recidive
counter packets 1421378091 bytes 2045004412819 return
}
chain ufw-before-logging-input {
}
chain ufw-before-logging-output {
}
chain ufw-before-logging-forward {
}
chain ufw-before-input {
}
chain ufw-before-output {
}
chain ufw-before-forward {
}
chain ufw-after-input {
}
chain ufw-after-output {
}
chain ufw-after-forward {
}
chain ufw-after-logging-input {
}
chain ufw-after-logging-output {
}
chain ufw-after-logging-forward {
}
chain ufw-reject-input {
}
chain ufw-reject-output {
}
chain ufw-reject-forward {
}
chain ufw-track-input {
}
chain ufw-track-output {
}
chain ufw-track-forward {
}
chain DOCKER {
ip daddr 172.17.0.7 iifname != "docker0" oifname "docker0" tcp dport 5432 counter packets 8 bytes 480 accept
ip daddr 172.19.0.2 iifname != "br-72fd626a8ff7" oifname "br-72fd626a8ff7" tcp dport 8080 counter packets 0 bytes 0 accept
ip daddr 172.17.0.6 iifname != "docker0" oifname "docker0" tcp dport 9443 counter packets 0 bytes 0 accept
ip daddr 172.17.0.6 iifname != "docker0" oifname "docker0" tcp dport 9000 counter packets 0 bytes 0 accept
ip daddr 192.168.176.2 iifname != "br-f8b083119d99" oifname "br-f8b083119d99" tcp dport 9001 counter packets 0 bytes 0 accept
ip daddr 192.168.176.2 iifname != "br-f8b083119d99" oifname "br-f8b083119d99" tcp dport 9000 counter packets 47769 bytes 2866140 accept
ip daddr 172.20.0.2 iifname != "br-6eb1e7f7f847" oifname "br-6eb1e7f7f847" tcp dport 8080 counter packets 0 bytes 0 accept
ip daddr 172.27.0.2 iifname != "br-3008d408e73a" oifname "br-3008d408e73a" tcp dport 3000 counter packets 0 bytes 0 accept
ip daddr 192.168.48.5 iifname != "br-77eb8a9e2ba1" oifname "br-77eb8a9e2ba1" tcp dport 80 counter packets 0 bytes 0 accept
ip daddr 192.168.48.4 iifname != "br-77eb8a9e2ba1" oifname "br-77eb8a9e2ba1" tcp dport 80 counter packets 0 bytes 0 accept
ip daddr 192.168.48.3 iifname != "br-77eb8a9e2ba1" oifname "br-77eb8a9e2ba1" tcp dport 80 counter packets 0 bytes 0 accept
ip daddr 192.168.48.2 iifname != "br-77eb8a9e2ba1" oifname "br-77eb8a9e2ba1" tcp dport 9000 counter packets 0 bytes 0 accept
ip daddr 172.18.0.2 iifname != "br-2e4a76a7cd2e" oifname "br-2e4a76a7cd2e" tcp dport 80 counter packets 0 bytes 0 accept
ip daddr 172.17.0.5 iifname != "docker0" oifname "docker0" tcp dport 80 counter packets 0 bytes 0 accept
ip daddr 172.17.0.3 iifname != "docker0" oifname "docker0" tcp dport 8222 counter packets 0 bytes 0 accept
ip daddr 172.17.0.3 iifname != "docker0" oifname "docker0" tcp dport 4222 counter packets 97 bytes 5744 accept
ip daddr 172.28.0.2 iifname != "br-8ce143481a5b" oifname "br-8ce143481a5b" tcp dport 1433 counter packets 0 bytes 0 accept
ip daddr 192.168.80.2 iifname != "br-e99ce5248c84" oifname "br-e99ce5248c84" tcp dport 8080 counter packets 0 bytes 0 accept
ip daddr 192.168.112.3 iifname != "br-e5d78502832d" oifname "br-e5d78502832d" tcp dport 9000 counter packets 0 bytes 0 accept
ip daddr 192.168.112.2 iifname != "br-e5d78502832d" oifname "br-e5d78502832d" tcp dport 80 counter packets 0 bytes 0 accept
ip daddr 192.168.128.2 iifname != "br-73641cceafc3" oifname "br-73641cceafc3" tcp dport 27017 counter packets 14 bytes 840 accept
ip daddr 192.168.208.2 iifname != "br-9fd22324ec08" oifname "br-9fd22324ec08" tcp dport 35621 counter packets 0 bytes 0 accept
ip daddr 192.168.203.13 iifname != "br-1ccb887b3344" oifname "br-1ccb887b3344" tcp dport 4243 counter packets 0 bytes 0 accept
ip daddr 192.168.203.12 iifname != "br-1ccb887b3344" oifname "br-1ccb887b3344" tcp dport 995 counter packets 194 bytes 11000 accept
ip daddr 192.168.203.12 iifname != "br-1ccb887b3344" oifname "br-1ccb887b3344" tcp dport 993 counter packets 188 bytes 9394 accept
ip daddr 192.168.203.12 iifname != "br-1ccb887b3344" oifname "br-1ccb887b3344" tcp dport 587 counter packets 444 bytes 23312 accept
ip daddr 192.168.203.12 iifname != "br-1ccb887b3344" oifname "br-1ccb887b3344" tcp dport 465 counter packets 104 bytes 5852 accept
ip daddr 192.168.203.12 iifname != "br-1ccb887b3344" oifname "br-1ccb887b3344" tcp dport 443 counter packets 0 bytes 0 accept
ip daddr 192.168.203.12 iifname != "br-1ccb887b3344" oifname "br-1ccb887b3344" tcp dport 143 counter packets 443 bytes 25280 accept
ip daddr 192.168.203.12 iifname != "br-1ccb887b3344" oifname "br-1ccb887b3344" tcp dport 110 counter packets 192 bytes 9561 accept
ip daddr 192.168.203.12 iifname != "br-1ccb887b3344" oifname "br-1ccb887b3344" tcp dport 80 counter packets 0 bytes 0 accept
ip daddr 192.168.203.12 iifname != "br-1ccb887b3344" oifname "br-1ccb887b3344" tcp dport 25 counter packets 430 bytes 22919 accept
ip daddr 172.17.0.2 iifname != "docker0" oifname "docker0" tcp dport 3000 counter packets 0 bytes 0 accept
ip daddr 172.17.0.2 iifname != "docker0" oifname "docker0" tcp dport 22 counter packets 1578 bytes 93884 accept
ip daddr 172.17.0.4 iifname != "docker0" oifname "docker0" tcp dport 5000 counter packets 25 bytes 1492 accept
iifname != "br-cadedce55fe9" oifname "br-cadedce55fe9" counter packets 0 bytes 0 drop
iifname != "br-dd007c7e67bc" oifname "br-dd007c7e67bc" counter packets 0 bytes 0 drop
iifname != "br-a5fbc29c2c2a" oifname "br-a5fbc29c2c2a" counter packets 0 bytes 0 drop
iifname != "br-6eb1e7f7f847" oifname "br-6eb1e7f7f847" counter packets 0 bytes 0 drop
iifname != "br-8ce143481a5b" oifname "br-8ce143481a5b" counter packets 0 bytes 0 drop
iifname != "br-84e7d0cfeada" oifname "br-84e7d0cfeada" counter packets 0 bytes 0 drop
iifname != "br-f8b083119d99" oifname "br-f8b083119d99" counter packets 0 bytes 0 drop
iifname != "br-0d1490cc67c9" oifname "br-0d1490cc67c9" counter packets 0 bytes 0 drop
iifname != "br-3b338a381229" oifname "br-3b338a381229" counter packets 0 bytes 0 drop
iifname != "br-3008d408e73a" oifname "br-3008d408e73a" counter packets 0 bytes 0 drop
iifname != "br-ca07a9577a7f" oifname "br-ca07a9577a7f" counter packets 0 bytes 0 drop
iifname != "docker0" oifname "docker0" counter packets 0 bytes 0 drop
iifname != "br-a63fa64a9e18" oifname "br-a63fa64a9e18" counter packets 0 bytes 0 drop
iifname != "br-1ccb887b3344" oifname "br-1ccb887b3344" counter packets 0 bytes 0 drop
iifname != "br-9fd22324ec08" oifname "br-9fd22324ec08" counter packets 0 bytes 0 drop
iifname != "br-73641cceafc3" oifname "br-73641cceafc3" counter packets 0 bytes 0 drop
iifname != "br-77eb8a9e2ba1" oifname "br-77eb8a9e2ba1" counter packets 0 bytes 0 drop
iifname != "br-e99ce5248c84" oifname "br-e99ce5248c84" counter packets 0 bytes 0 drop
iifname != "br-e5d78502832d" oifname "br-e5d78502832d" counter packets 0 bytes 0 drop
iifname != "br-2e4a76a7cd2e" oifname "br-2e4a76a7cd2e" counter packets 0 bytes 0 drop
iifname != "br-72fd626a8ff7" oifname "br-72fd626a8ff7" counter packets 0 bytes 0 drop
}
chain DOCKER-BRIDGE {
oifname "br-cadedce55fe9" counter packets 0 bytes 0 jump DOCKER
oifname "br-dd007c7e67bc" counter packets 0 bytes 0 jump DOCKER
oifname "br-a5fbc29c2c2a" counter packets 0 bytes 0 jump DOCKER
oifname "br-6eb1e7f7f847" counter packets 799 bytes 47940 jump DOCKER
oifname "br-8ce143481a5b" counter packets 0 bytes 0 jump DOCKER
oifname "br-84e7d0cfeada" counter packets 0 bytes 0 jump DOCKER
oifname "br-f8b083119d99" counter packets 98911 bytes 5934660 jump DOCKER
oifname "br-0d1490cc67c9" counter packets 69740 bytes 4118476 jump DOCKER
oifname "br-3b338a381229" counter packets 32 bytes 1920 jump DOCKER
oifname "br-3008d408e73a" counter packets 458 bytes 27480 jump DOCKER
oifname "br-ca07a9577a7f" counter packets 0 bytes 0 jump DOCKER
oifname "docker0" counter packets 87073 bytes 5223529 jump DOCKER
oifname "br-a63fa64a9e18" counter packets 1353 bytes 81180 jump DOCKER
oifname "br-1ccb887b3344" counter packets 7662 bytes 419862 jump DOCKER
oifname "br-9fd22324ec08" counter packets 173 bytes 10380 jump DOCKER
oifname "br-73641cceafc3" counter packets 162 bytes 9720 jump DOCKER
oifname "br-77eb8a9e2ba1" counter packets 94 bytes 5640 jump DOCKER
oifname "br-e99ce5248c84" counter packets 8 bytes 480 jump DOCKER
oifname "br-e5d78502832d" counter packets 26 bytes 1560 jump DOCKER
oifname "br-2e4a76a7cd2e" counter packets 7 bytes 420 jump DOCKER
oifname "br-72fd626a8ff7" counter packets 0 bytes 0 jump DOCKER
}
chain DOCKER-CT {
oifname "br-cadedce55fe9" xt match "conntrack" counter packets 0 bytes 0 accept
oifname "br-dd007c7e67bc" xt match "conntrack" counter packets 0 bytes 0 accept
oifname "br-a5fbc29c2c2a" xt match "conntrack" counter packets 0 bytes 0 accept
oifname "br-6eb1e7f7f847" xt match "conntrack" counter packets 38458 bytes 6234236 accept
oifname "br-8ce143481a5b" xt match "conntrack" counter packets 60403 bytes 20478794 accept
oifname "br-84e7d0cfeada" xt match "conntrack" counter packets 0 bytes 0 accept
oifname "br-f8b083119d99" xt match "conntrack" counter packets 1008024 bytes 206364436 accept
oifname "br-0d1490cc67c9" xt match "conntrack" counter packets 871134 bytes 1416174426 accept
oifname "br-3b338a381229" xt match "conntrack" counter packets 4649 bytes 2311375 accept
oifname "br-3008d408e73a" xt match "conntrack" counter packets 13415 bytes 1974731 accept
oifname "br-ca07a9577a7f" xt match "conntrack" counter packets 0 bytes 0 accept
oifname "docker0" xt match "conntrack" counter packets 8909862 bytes 7892163419 accept
oifname "br-a63fa64a9e18" xt match "conntrack" counter packets 16688 bytes 6822829 accept
oifname "br-1ccb887b3344" xt match "conntrack" counter packets 461453 bytes 141913887 accept
oifname "br-9fd22324ec08" xt match "conntrack" counter packets 1677 bytes 427538 accept
oifname "br-73641cceafc3" xt match "conntrack" counter packets 417619 bytes 35343624 accept
oifname "br-77eb8a9e2ba1" xt match "conntrack" counter packets 125437 bytes 94155193 accept
oifname "br-e99ce5248c84" xt match "conntrack" counter packets 91 bytes 19173 accept
oifname "br-e5d78502832d" xt match "conntrack" counter packets 128653 bytes 40539569 accept
oifname "br-2e4a76a7cd2e" xt match "conntrack" counter packets 20257 bytes 158631592 accept
oifname "br-72fd626a8ff7" xt match "conntrack" counter packets 0 bytes 0 accept
}
chain DOCKER-INTERNAL {
}
chain f2b-recidive {
ip saddr 2.57.122.209 counter packets 0 bytes 0 xt target "REJECT"
ip saddr 2.57.122.76 counter packets 127 bytes 7600 xt target "REJECT"
ip saddr 195.178.110.228 counter packets 17 bytes 1000 xt target "REJECT"
ip saddr 2.57.122.74 counter packets 11 bytes 620 xt target "REJECT"
ip saddr 195.178.110.26 counter packets 56 bytes 3360 xt target "REJECT"
ip saddr 92.118.39.77 counter packets 2 bytes 80 xt target "REJECT"
ip saddr 92.118.39.71 counter packets 1 bytes 40 xt target "REJECT"
ip saddr 45.148.10.240 counter packets 0 bytes 0 xt target "REJECT"
ip saddr 195.178.110.30 counter packets 8 bytes 320 xt target "REJECT"
counter packets 948810608 bytes 1740338848565 return
}
chain f2b-sshd {
counter packets 948810709 bytes 1740338655735 return
}
}
# Warning: table ip6 filter is managed by iptables-nft, do not touch!
table ip6 filter {
chain INPUT {
type filter hook input priority filter; policy accept;
counter packets 5426360 bytes 34419159588 jump ufw6-before-logging-input
counter packets 5426360 bytes 34419159588 jump ufw6-before-input
counter packets 367982 bytes 3415126642 jump ufw6-after-input
counter packets 367982 bytes 3415126642 jump ufw6-after-logging-input
counter packets 367982 bytes 3415126642 jump ufw6-reject-input
counter packets 367982 bytes 3415126642 jump ufw6-track-input
}
chain FORWARD {
type filter hook forward priority filter; policy accept;
counter packets 0 bytes 0 jump DOCKER-USER
counter packets 0 bytes 0 jump DOCKER-FORWARD
counter packets 0 bytes 0 jump ufw6-before-logging-forward
counter packets 0 bytes 0 jump ufw6-before-forward
counter packets 0 bytes 0 jump ufw6-after-forward
counter packets 0 bytes 0 jump ufw6-after-logging-forward
counter packets 0 bytes 0 jump ufw6-reject-forward
counter packets 0 bytes 0 jump ufw6-track-forward
}
chain OUTPUT {
type filter hook output priority filter; policy accept;
counter packets 6004354 bytes 1866587952 jump ufw6-before-logging-output
counter packets 6004354 bytes 1866587952 jump ufw6-before-output
counter packets 2241898 bytes 639173314 jump ufw6-after-output
counter packets 2241898 bytes 639173314 jump ufw6-after-logging-output
counter packets 2241898 bytes 639173314 jump ufw6-reject-output
counter packets 2241898 bytes 639173314 jump ufw6-track-output
}
chain DOCKER-FORWARD {
counter packets 0 bytes 0 jump DOCKER-CT
counter packets 0 bytes 0 jump DOCKER-INTERNAL
counter packets 0 bytes 0 jump DOCKER-BRIDGE
}
chain DOCKER-USER {
counter packets 0 bytes 0 jump ufw6-user-forward
xt match "conntrack" counter packets 0 bytes 0 return
xt match "conntrack" counter packets 0 bytes 0 drop
iifname "docker0" oifname "docker0" counter packets 0 bytes 0 accept
ip6 saddr fd00::/8 counter packets 0 bytes 0 return
ip6 daddr fd00::/8 xt match "conntrack" counter packets 0 bytes 0 jump ufw6-docker-logging-deny
counter packets 0 bytes 0 return
}
chain ufw6-before-logging-input {
}
chain ufw6-before-logging-output {
}
chain ufw6-before-logging-forward {
}
chain ufw6-before-input {
}
chain ufw6-before-output {
}
chain ufw6-before-forward {
}
chain ufw6-after-input {
}
chain ufw6-after-output {
}
chain ufw6-after-forward {
}
chain ufw6-after-logging-input {
}
chain ufw6-after-logging-output {
}
chain ufw6-after-logging-forward {
}
chain ufw6-reject-input {
}
chain ufw6-reject-output {
}
chain ufw6-reject-forward {
}
chain ufw6-track-input {
}
chain ufw6-track-output {
}
chain ufw6-track-forward {
}
chain ufw6-user-forward {
}
chain ufw6-docker-logging-deny {
limit rate 3/minute burst 10 packets counter packets 0 bytes 0 xt target "LOG"
counter packets 0 bytes 0 drop
}
chain DOCKER {
}
chain DOCKER-BRIDGE {
}
chain DOCKER-CT {
}
chain DOCKER-INTERNAL {
}
}
# Warning: table ip nat is managed by iptables-nft, do not touch!
table ip nat {
chain PREROUTING {
type nat hook prerouting priority dstnat; policy accept;
xt match "addrtype" counter packets 10757093 bytes 647829087 jump DOCKER
}
chain OUTPUT {
type nat hook output priority dstnat; policy accept;
ip daddr != 127.0.0.0/8 xt match "addrtype" counter packets 128611 bytes 7705178 jump DOCKER
}
chain POSTROUTING {
type nat hook postrouting priority srcnat; policy accept;
ip saddr 172.19.0.0/16 oifname != "br-72fd626a8ff7" counter packets 0 bytes 0 xt target "MASQUERADE"
ip saddr 172.18.0.0/16 oifname != "br-2e4a76a7cd2e" counter packets 825 bytes 49500 xt target "MASQUERADE"
ip saddr 192.168.112.0/20 oifname != "br-e5d78502832d" counter packets 126 bytes 7560 xt target "MASQUERADE"
ip saddr 192.168.80.0/20 oifname != "br-e99ce5248c84" counter packets 0 bytes 0 xt target "MASQUERADE"
ip saddr 192.168.48.0/20 oifname != "br-77eb8a9e2ba1" counter packets 99 bytes 5940 xt target "MASQUERADE"
ip saddr 192.168.128.0/20 oifname != "br-73641cceafc3" counter packets 536 bytes 32160 xt target "MASQUERADE"
ip saddr 192.168.208.0/20 oifname != "br-9fd22324ec08" counter packets 2 bytes 120 xt target "MASQUERADE"
ip saddr 192.168.203.0/24 oifname != "br-1ccb887b3344" counter packets 37248 bytes 2854476 xt target "MASQUERADE"
ip saddr 192.168.64.0/20 oifname != "br-a63fa64a9e18" counter packets 353 bytes 21180 xt target "MASQUERADE"
ip saddr 172.17.0.0/16 oifname != "docker0" counter packets 99933 bytes 6001436 xt target "MASQUERADE"
ip saddr 172.21.0.0/16 oifname != "br-84e7d0cfeada" counter packets 2 bytes 128 xt target "MASQUERADE"
ip saddr 192.168.176.0/20 oifname != "br-f8b083119d99" counter packets 209 bytes 12644 xt target "MASQUERADE"
ip saddr 172.20.0.0/16 oifname != "br-6eb1e7f7f847" counter packets 699 bytes 42516 xt target "MASQUERADE"
ip saddr 172.28.0.0/16 oifname != "br-8ce143481a5b" counter packets 1934 bytes 116040 xt target "MASQUERADE"
ip saddr 172.27.0.0/16 oifname != "br-3008d408e73a" counter packets 2904 bytes 174240 xt target "MASQUERADE"
ip saddr 172.25.0.0/16 oifname != "br-cadedce55fe9" counter packets 0 bytes 0 xt target "MASQUERADE"
ip saddr 172.24.0.0/16 oifname != "br-3b338a381229" counter packets 385 bytes 23164 xt target "MASQUERADE"
ip saddr 192.168.224.0/20 oifname != "br-ca07a9577a7f" counter packets 0 bytes 0 xt target "MASQUERADE"
ip saddr 192.168.0.0/20 oifname != "br-a5fbc29c2c2a" counter packets 10 bytes 600 xt target "MASQUERADE"
ip saddr 172.31.0.0/16 oifname != "br-dd007c7e67bc" counter packets 0 bytes 0 xt target "MASQUERADE"
ip saddr 192.168.240.0/20 oifname != "br-0d1490cc67c9" counter packets 587045 bytes 35224163 xt target "MASQUERADE"
}
chain DOCKER {
iifname != "docker0" tcp dport 5100 counter packets 8247 bytes 494812 xt target "DNAT"
iifname != "docker0" tcp dport 222 counter packets 2304 bytes 137444 xt target "DNAT"
iifname != "docker0" tcp dport 20000 counter packets 1532 bytes 91584 xt target "DNAT"
iifname != "br-1ccb887b3344" tcp dport 25 counter packets 433 bytes 23099 xt target "DNAT"
iifname != "br-1ccb887b3344" tcp dport 7080 counter packets 35 bytes 1864 xt target "DNAT"
iifname != "br-1ccb887b3344" tcp dport 110 counter packets 195 bytes 9741 xt target "DNAT"
iifname != "br-1ccb887b3344" tcp dport 143 counter packets 448 bytes 25580 xt target "DNAT"
iifname != "br-1ccb887b3344" tcp dport 7443 counter packets 58 bytes 2868 xt target "DNAT"
iifname != "br-1ccb887b3344" tcp dport 465 counter packets 107 bytes 6032 xt target "DNAT"
iifname != "br-1ccb887b3344" tcp dport 587 counter packets 447 bytes 23492 xt target "DNAT"
iifname != "br-1ccb887b3344" tcp dport 993 counter packets 201 bytes 10174 xt target "DNAT"
iifname != "br-1ccb887b3344" tcp dport 995 counter packets 197 bytes 11180 xt target "DNAT"
iifname != "br-1ccb887b3344" tcp dport 20004 counter packets 5 bytes 300 xt target "DNAT"
iifname != "br-9fd22324ec08" tcp dport 20005 counter packets 5 bytes 300 xt target "DNAT"
iifname != "br-73641cceafc3" tcp dport 20006 counter packets 19 bytes 1140 xt target "DNAT"
iifname != "br-e5d78502832d" tcp dport 20007 counter packets 5 bytes 284 xt target "DNAT"
iifname != "br-e5d78502832d" tcp dport 20008 counter packets 4 bytes 240 xt target "DNAT"
iifname != "br-e99ce5248c84" tcp dport 1842 counter packets 12 bytes 720 xt target "DNAT"
iifname != "br-8ce143481a5b" tcp dport 4848 counter packets 40 bytes 1960 xt target "DNAT"
iifname != "docker0" tcp dport 4222 counter packets 3846 bytes 231012 xt target "DNAT"
ip daddr 127.0.0.1 iifname != "docker0" tcp dport 8222 counter packets 0 bytes 0 xt target "DNAT"
iifname != "docker0" tcp dport 20003 counter packets 18942 bytes 1136512 xt target "DNAT"
iifname != "br-2e4a76a7cd2e" tcp dport 9070 counter packets 195 bytes 11676 xt target "DNAT"
iifname != "br-77eb8a9e2ba1" tcp dport 9102 counter packets 13 bytes 772 xt target "DNAT"
iifname != "br-77eb8a9e2ba1" tcp dport 8102 counter packets 17 bytes 944 xt target "DNAT"
iifname != "br-77eb8a9e2ba1" tcp dport 8104 counter packets 16 bytes 916 xt target "DNAT"
iifname != "br-77eb8a9e2ba1" tcp dport 8103 counter packets 13 bytes 756 xt target "DNAT"
iifname != "br-3008d408e73a" tcp dport 1212 counter packets 189 bytes 11188 xt target "DNAT"
iifname != "br-6eb1e7f7f847" tcp dport 20009 counter packets 138 bytes 8280 xt target "DNAT"
iifname != "br-f8b083119d99" tcp dport 20001 counter packets 47780 bytes 2866736 xt target "DNAT"
iifname != "br-f8b083119d99" tcp dport 20002 counter packets 7 bytes 404 xt target "DNAT"
iifname != "docker0" tcp dport 20010 counter packets 74 bytes 4424 xt target "DNAT"
iifname != "docker0" tcp dport 20011 counter packets 4 bytes 240 xt target "DNAT"
iifname != "br-72fd626a8ff7" tcp dport 20012 counter packets 237 bytes 14220 xt target "DNAT"
iifname != "docker0" tcp dport 6852 counter packets 16489 bytes 989324 xt target "DNAT"
}
}
# Warning: table ip6 nat is managed by iptables-nft, do not touch!
table ip6 nat {
chain PREROUTING {
type nat hook prerouting priority dstnat; policy accept;
xt match "addrtype" counter packets 399 bytes 22104 jump DOCKER
}
chain OUTPUT {
type nat hook output priority dstnat; policy accept;
ip6 daddr != ::1 xt match "addrtype" counter packets 0 bytes 0 jump DOCKER
}
chain DOCKER {
}
}
table ip raw {
chain PREROUTING {
type filter hook prerouting priority raw; policy accept;
ip daddr 127.0.0.1 iifname != "lo" tcp dport 8222 counter packets 0 bytes 0 drop
}
}
table ip mangle {
chain FORWARD {
type filter hook forward priority mangle; policy accept;
}
}
table inet mesh {
chain input {
type filter hook input priority filter; policy drop;
ct state established,related accept
ct state invalid drop
iif "lo" accept
iifname != { "mesh0", "enp9s0" } accept
icmp type echo-request accept
icmpv6 type { echo-request, nd-router-advert, nd-neighbor-solicit, nd-neighbor-advert } accept
iifname != { "mesh0", "enp9s0" } udp dport { 53, 67 } accept
iifname != { "mesh0", "enp9s0" } tcp dport 53 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } tcp dport 22 accept
tcp dport 22 accept
tcp dport 4222 accept
tcp dport 22 accept
tcp dport 25 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } tcp dport 53 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } udp dport 53 accept
tcp dport 80 accept
tcp dport 110 accept
tcp dport 143 accept
tcp dport 222 accept
tcp dport 443 accept
tcp dport 465 accept
tcp dport 587 accept
tcp dport 993 accept
tcp dport 995 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } tcp dport 1212 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } tcp dport 1842 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } tcp dport 4222 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } tcp dport 4848 accept
tcp dport 5100 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } tcp dport 6852 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } tcp dport 7080 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } tcp dport 7443 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } tcp dport 8102 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } tcp dport 8103 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } tcp dport 8104 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } tcp dport 9000 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } tcp dport 9070 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } tcp dport 9102 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } tcp dport 20000 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } tcp dport 20001 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } tcp dport 20002 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } tcp dport 20003 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } tcp dport 20004 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } tcp dport 20005 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } tcp dport 20006 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } tcp dport 20007 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } tcp dport 20008 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } tcp dport 20009 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } tcp dport 20010 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } tcp dport 20011 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } tcp dport 20012 accept
udp dport 51820 accept
}
chain output {
type filter hook output priority filter; policy accept;
}
chain forward {
type filter hook forward priority filter; policy drop;
ct state established,related accept
ct state invalid drop
iifname != { "mesh0", "enp9s0" } accept
iifname "mesh0" oifname "mesh0" accept
ct original proto-dst 22 accept
ct original proto-dst 25 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } ct original proto-dst 53 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } ct original proto-dst 53 accept
ct original proto-dst 80 accept
ct original proto-dst 110 accept
ct original proto-dst 143 accept
ct original proto-dst 222 accept
ct original proto-dst 443 accept
ct original proto-dst 465 accept
ct original proto-dst 587 accept
ct original proto-dst 993 accept
ct original proto-dst 995 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } ct original proto-dst 1212 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } ct original proto-dst 1842 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } ct original proto-dst 4222 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } ct original proto-dst 4848 accept
ct original proto-dst 5100 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } ct original proto-dst 6852 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } ct original proto-dst 7080 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } ct original proto-dst 7443 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } ct original proto-dst 8102 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } ct original proto-dst 8103 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } ct original proto-dst 8104 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } ct original proto-dst 9000 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } ct original proto-dst 9070 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } ct original proto-dst 9102 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } ct original proto-dst 20000 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } ct original proto-dst 20001 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } ct original proto-dst 20002 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } ct original proto-dst 20003 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } ct original proto-dst 20004 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } ct original proto-dst 20005 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } ct original proto-dst 20006 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } ct original proto-dst 20007 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } ct original proto-dst 20008 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } ct original proto-dst 20009 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } ct original proto-dst 20010 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } ct original proto-dst 20011 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } ct original proto-dst 20012 accept
ct original proto-dst 51820 accept
ct original proto-dst 4222 accept
}
}
+297
View File
@@ -0,0 +1,297 @@
# Warning: table ip nat is managed by iptables-nft, do not touch!
table ip nat {
chain DOCKER {
iifname != "br-c70303d221ee" tcp dport 5680 counter packets 0 bytes 0 xt target "DNAT"
ip daddr 127.0.0.1 iifname != "br-c70303d221ee" tcp dport 15673 counter packets 0 bytes 0 xt target "DNAT"
iifname != "br-3636e05760a9" tcp dport 59000 counter packets 0 bytes 0 xt target "DNAT"
iifname != "br-3636e05760a9" tcp dport 59001 counter packets 0 bytes 0 xt target "DNAT"
iifname != "br-3636e05760a9" tcp dport 55672 counter packets 0 bytes 0 xt target "DNAT"
iifname != "br-3636e05760a9" tcp dport 55673 counter packets 0 bytes 0 xt target "DNAT"
iifname != "br-3636e05760a9" tcp dport 55432 counter packets 0 bytes 0 xt target "DNAT"
ip daddr 127.0.0.1 iifname != "docker0" tcp dport 57732 counter packets 0 bytes 0 xt target "DNAT"
ip daddr 127.0.0.1 iifname != "docker0" tcp dport 57733 counter packets 0 bytes 0 xt target "DNAT"
iifname != "docker0" tcp dport 5314 counter packets 0 bytes 0 xt target "DNAT"
iifname != "br-613eb68ef5fb" tcp dport 4848 counter packets 0 bytes 0 xt target "DNAT"
iifname != "br-b3240c822bce" tcp dport 5679 counter packets 0 bytes 0 xt target "DNAT"
ip daddr 127.0.0.1 iifname != "br-b3240c822bce" tcp dport 15672 counter packets 0 bytes 0 xt target "DNAT"
iifname != "br-4b504efd6080" tcp dport 9000 counter packets 0 bytes 0 xt target "DNAT"
iifname != "br-4b504efd6080" tcp dport 9001 counter packets 0 bytes 0 xt target "DNAT"
ip daddr 127.0.0.1 iifname != "br-ef6df03f71a0" tcp dport 5432 counter packets 0 bytes 0 xt target "DNAT"
ip daddr 127.0.0.1 iifname != "br-ef6df03f71a0" tcp dport 8081 counter packets 0 bytes 0 xt target "DNAT"
ip daddr 127.0.0.1 iifname != "br-ec480f77ac34" tcp dport 6379 counter packets 0 bytes 0 xt target "DNAT"
ip daddr 127.0.0.1 iifname != "br-af4c9c2aa60a" tcp dport 8001 counter packets 0 bytes 0 xt target "DNAT"
ip daddr 127.0.0.1 iifname != "br-669fda75f1ac" tcp dport 28080 counter packets 0 bytes 0 xt target "DNAT"
ip daddr 127.0.0.1 iifname != "br-af4c9c2aa60a" tcp dport 6789 counter packets 0 bytes 0 xt target "DNAT"
iifname != "br-af4c9c2aa60a" tcp dport 8770 counter packets 0 bytes 0 xt target "DNAT"
iifname != "br-af4c9c2aa60a" tcp dport 1212 counter packets 0 bytes 0 xt target "DNAT"
iifname != "br-af4c9c2aa60a" tcp dport 80 counter packets 0 bytes 0 xt target "DNAT"
iifname != "br-af4c9c2aa60a" tcp dport 443 counter packets 0 bytes 0 xt target "DNAT"
ip daddr 127.0.0.1 iifname != "docker0" tcp dport 55541 counter packets 0 bytes 0 xt target "DNAT"
}
chain PREROUTING {
type nat hook prerouting priority dstnat; policy accept;
xt match "addrtype" counter packets 437947 bytes 71410534 jump DOCKER
}
chain OUTPUT {
type nat hook output priority dstnat; policy accept;
ip daddr != 127.0.0.0/8 xt match "addrtype" counter packets 5761 bytes 423317 jump DOCKER
}
chain POSTROUTING {
type nat hook postrouting priority srcnat; policy accept;
ip saddr 172.22.0.0/16 oifname != "br-65f6dc782562" counter packets 124 bytes 16328 xt target "MASQUERADE"
ip saddr 172.28.0.0/16 oifname != "br-669fda75f1ac" counter packets 127 bytes 16928 xt target "MASQUERADE"
ip saddr 172.31.0.0/16 oifname != "br-ec480f77ac34" counter packets 127 bytes 16928 xt target "MASQUERADE"
ip saddr 192.168.48.0/20 oifname != "br-ef6df03f71a0" counter packets 127 bytes 16928 xt target "MASQUERADE"
ip saddr 172.25.0.0/16 oifname != "br-4b504efd6080" counter packets 129 bytes 17052 xt target "MASQUERADE"
ip saddr 192.168.32.0/20 oifname != "br-613eb68ef5fb" counter packets 1124 bytes 76748 xt target "MASQUERADE"
ip saddr 172.17.0.0/16 oifname != "docker0" counter packets 1833 bytes 150990 xt target "MASQUERADE"
ip saddr 172.18.0.0/16 oifname != "br-07a5e2f2c42f" counter packets 504 bytes 65112 xt target "MASQUERADE"
ip saddr 172.27.0.0/16 oifname != "br-160f55da427c" counter packets 508 bytes 65784 xt target "MASQUERADE"
ip saddr 192.168.16.0/20 oifname != "br-dba077b9b543" counter packets 503 bytes 64784 xt target "MASQUERADE"
ip saddr 192.168.64.0/20 oifname != "br-d44fef8fd602" counter packets 1282 bytes 111308 xt target "MASQUERADE"
ip saddr 172.30.0.0/16 oifname != "br-af4c9c2aa60a" counter packets 2503 bytes 190324 xt target "MASQUERADE"
ip saddr 172.21.0.0/16 oifname != "br-9d6c95e8d80c" counter packets 1289 bytes 112644 xt target "MASQUERADE"
ip saddr 172.23.0.0/16 oifname != "br-679db9b21e00" counter packets 506 bytes 65384 xt target "MASQUERADE"
ip saddr 172.24.0.0/16 oifname != "br-40094534a5ee" counter packets 508 bytes 65784 xt target "MASQUERADE"
ip saddr 172.26.0.0/16 oifname != "br-3dcb6ef83ea1" counter packets 508 bytes 65784 xt target "MASQUERADE"
ip saddr 172.20.0.0/16 oifname != "br-3a760a74f4f6" counter packets 1153 bytes 104344 xt target "MASQUERADE"
ip saddr 192.168.80.0/20 oifname != "br-3636e05760a9" counter packets 546 bytes 70540 xt target "MASQUERADE"
ip saddr 172.29.0.0/16 oifname != "br-b3240c822bce" counter packets 551 bytes 71492 xt target "MASQUERADE"
ip saddr 172.19.0.0/16 oifname != "br-c70303d221ee" counter packets 1136 bytes 106592 xt target "MASQUERADE"
}
}
# Warning: table ip filter is managed by iptables-nft, do not touch!
table ip filter {
chain DOCKER {
ip daddr 172.17.0.5 iifname != "docker0" oifname "docker0" tcp dport 5432 counter packets 0 bytes 0 accept
ip daddr 172.30.0.2 iifname != "br-af4c9c2aa60a" oifname "br-af4c9c2aa60a" tcp dport 443 counter packets 0 bytes 0 accept
ip daddr 172.30.0.2 iifname != "br-af4c9c2aa60a" oifname "br-af4c9c2aa60a" tcp dport 80 counter packets 0 bytes 0 accept
ip daddr 172.30.0.10 iifname != "br-af4c9c2aa60a" oifname "br-af4c9c2aa60a" tcp dport 3000 counter packets 0 bytes 0 accept
ip daddr 172.30.0.5 iifname != "br-af4c9c2aa60a" oifname "br-af4c9c2aa60a" tcp dport 8000 counter packets 0 bytes 0 accept
ip daddr 172.30.0.3 iifname != "br-af4c9c2aa60a" oifname "br-af4c9c2aa60a" tcp dport 6789 counter packets 0 bytes 0 accept
ip daddr 172.28.0.3 iifname != "br-669fda75f1ac" oifname "br-669fda75f1ac" tcp dport 8080 counter packets 0 bytes 0 accept
ip daddr 172.30.0.4 iifname != "br-af4c9c2aa60a" oifname "br-af4c9c2aa60a" tcp dport 5540 counter packets 0 bytes 0 accept
ip daddr 172.31.0.2 iifname != "br-ec480f77ac34" oifname "br-ec480f77ac34" tcp dport 6379 counter packets 0 bytes 0 accept
ip daddr 192.168.48.3 iifname != "br-ef6df03f71a0" oifname "br-ef6df03f71a0" tcp dport 8081 counter packets 0 bytes 0 accept
ip daddr 192.168.48.2 iifname != "br-ef6df03f71a0" oifname "br-ef6df03f71a0" tcp dport 5432 counter packets 0 bytes 0 accept
ip daddr 172.25.0.2 iifname != "br-4b504efd6080" oifname "br-4b504efd6080" tcp dport 9001 counter packets 0 bytes 0 accept
ip daddr 172.25.0.2 iifname != "br-4b504efd6080" oifname "br-4b504efd6080" tcp dport 9000 counter packets 0 bytes 0 accept
ip daddr 172.29.0.2 iifname != "br-b3240c822bce" oifname "br-b3240c822bce" tcp dport 15672 counter packets 0 bytes 0 accept
ip daddr 172.29.0.2 iifname != "br-b3240c822bce" oifname "br-b3240c822bce" tcp dport 5672 counter packets 0 bytes 0 accept
ip daddr 192.168.32.2 iifname != "br-613eb68ef5fb" oifname "br-613eb68ef5fb" tcp dport 1433 counter packets 0 bytes 0 accept
ip daddr 172.17.0.3 iifname != "docker0" oifname "docker0" tcp dport 5000 counter packets 0 bytes 0 accept
ip daddr 172.17.0.2 iifname != "docker0" oifname "docker0" tcp dport 15672 counter packets 0 bytes 0 accept
ip daddr 172.17.0.2 iifname != "docker0" oifname "docker0" tcp dport 5672 counter packets 0 bytes 0 accept
ip daddr 192.168.80.4 iifname != "br-3636e05760a9" oifname "br-3636e05760a9" tcp dport 5432 counter packets 0 bytes 0 accept
ip daddr 192.168.80.3 iifname != "br-3636e05760a9" oifname "br-3636e05760a9" tcp dport 15672 counter packets 0 bytes 0 accept
ip daddr 192.168.80.3 iifname != "br-3636e05760a9" oifname "br-3636e05760a9" tcp dport 5672 counter packets 0 bytes 0 accept
ip daddr 192.168.80.2 iifname != "br-3636e05760a9" oifname "br-3636e05760a9" tcp dport 9001 counter packets 0 bytes 0 accept
ip daddr 192.168.80.2 iifname != "br-3636e05760a9" oifname "br-3636e05760a9" tcp dport 9000 counter packets 0 bytes 0 accept
ip daddr 172.19.0.2 iifname != "br-c70303d221ee" oifname "br-c70303d221ee" tcp dport 15672 counter packets 0 bytes 0 accept
ip daddr 172.19.0.2 iifname != "br-c70303d221ee" oifname "br-c70303d221ee" tcp dport 5672 counter packets 0 bytes 0 accept
iifname != "br-c70303d221ee" oifname "br-c70303d221ee" counter packets 0 bytes 0 drop
iifname != "br-b3240c822bce" oifname "br-b3240c822bce" counter packets 0 bytes 0 drop
iifname != "br-3636e05760a9" oifname "br-3636e05760a9" counter packets 0 bytes 0 drop
iifname != "br-3a760a74f4f6" oifname "br-3a760a74f4f6" counter packets 0 bytes 0 drop
iifname != "br-3dcb6ef83ea1" oifname "br-3dcb6ef83ea1" counter packets 0 bytes 0 drop
iifname != "br-40094534a5ee" oifname "br-40094534a5ee" counter packets 0 bytes 0 drop
iifname != "br-679db9b21e00" oifname "br-679db9b21e00" counter packets 0 bytes 0 drop
iifname != "br-9d6c95e8d80c" oifname "br-9d6c95e8d80c" counter packets 0 bytes 0 drop
iifname != "br-af4c9c2aa60a" oifname "br-af4c9c2aa60a" counter packets 0 bytes 0 drop
iifname != "br-d44fef8fd602" oifname "br-d44fef8fd602" counter packets 0 bytes 0 drop
iifname != "br-dba077b9b543" oifname "br-dba077b9b543" counter packets 0 bytes 0 drop
iifname != "br-160f55da427c" oifname "br-160f55da427c" counter packets 0 bytes 0 drop
iifname != "br-07a5e2f2c42f" oifname "br-07a5e2f2c42f" counter packets 0 bytes 0 drop
iifname != "docker0" oifname "docker0" counter packets 0 bytes 0 drop
iifname != "br-613eb68ef5fb" oifname "br-613eb68ef5fb" counter packets 0 bytes 0 drop
iifname != "br-4b504efd6080" oifname "br-4b504efd6080" counter packets 0 bytes 0 drop
iifname != "br-ef6df03f71a0" oifname "br-ef6df03f71a0" counter packets 0 bytes 0 drop
iifname != "br-ec480f77ac34" oifname "br-ec480f77ac34" counter packets 0 bytes 0 drop
iifname != "br-669fda75f1ac" oifname "br-669fda75f1ac" counter packets 0 bytes 0 drop
iifname != "br-65f6dc782562" oifname "br-65f6dc782562" counter packets 0 bytes 0 drop
}
chain DOCKER-FORWARD {
counter packets 6530319 bytes 11196484299 jump DOCKER-CT
counter packets 3312487 bytes 5001091084 jump DOCKER-INTERNAL
counter packets 3312487 bytes 5001091084 jump DOCKER-BRIDGE
iifname "br-c70303d221ee" counter packets 0 bytes 0 accept
iifname "br-b3240c822bce" counter packets 0 bytes 0 accept
iifname "br-3636e05760a9" counter packets 43 bytes 9355 accept
iifname "br-3a760a74f4f6" counter packets 0 bytes 0 accept
iifname "br-3dcb6ef83ea1" counter packets 0 bytes 0 accept
iifname "br-40094534a5ee" counter packets 0 bytes 0 accept
iifname "br-679db9b21e00" counter packets 0 bytes 0 accept
iifname "br-9d6c95e8d80c" counter packets 0 bytes 0 accept
iifname "br-af4c9c2aa60a" counter packets 2761311 bytes 4959915711 accept
iifname "br-d44fef8fd602" counter packets 0 bytes 0 accept
iifname "br-dba077b9b543" counter packets 0 bytes 0 accept
iifname "br-160f55da427c" counter packets 0 bytes 0 accept
iifname "br-07a5e2f2c42f" counter packets 0 bytes 0 accept
iifname "docker0" counter packets 460394 bytes 25754554 accept
iifname "br-613eb68ef5fb" counter packets 10805 bytes 1792862 accept
iifname "br-4b504efd6080" counter packets 33 bytes 2892 accept
iifname "br-ef6df03f71a0" counter packets 0 bytes 0 accept
iifname "br-ec480f77ac34" counter packets 0 bytes 0 accept
iifname "br-669fda75f1ac" counter packets 0 bytes 0 accept
iifname "br-65f6dc782562" counter packets 0 bytes 0 accept
}
chain DOCKER-BRIDGE {
oifname "br-c70303d221ee" counter packets 0 bytes 0 jump DOCKER
oifname "br-b3240c822bce" counter packets 0 bytes 0 jump DOCKER
oifname "br-3636e05760a9" counter packets 0 bytes 0 jump DOCKER
oifname "br-3a760a74f4f6" counter packets 0 bytes 0 jump DOCKER
oifname "br-3dcb6ef83ea1" counter packets 0 bytes 0 jump DOCKER
oifname "br-40094534a5ee" counter packets 0 bytes 0 jump DOCKER
oifname "br-679db9b21e00" counter packets 0 bytes 0 jump DOCKER
oifname "br-9d6c95e8d80c" counter packets 0 bytes 0 jump DOCKER
oifname "br-af4c9c2aa60a" counter packets 118 bytes 8400 jump DOCKER
oifname "br-d44fef8fd602" counter packets 0 bytes 0 jump DOCKER
oifname "br-dba077b9b543" counter packets 0 bytes 0 jump DOCKER
oifname "br-160f55da427c" counter packets 0 bytes 0 jump DOCKER
oifname "br-07a5e2f2c42f" counter packets 0 bytes 0 jump DOCKER
oifname "docker0" counter packets 0 bytes 0 jump DOCKER
oifname "br-613eb68ef5fb" counter packets 0 bytes 0 jump DOCKER
oifname "br-4b504efd6080" counter packets 0 bytes 0 jump DOCKER
oifname "br-ef6df03f71a0" counter packets 0 bytes 0 jump DOCKER
oifname "br-ec480f77ac34" counter packets 0 bytes 0 jump DOCKER
oifname "br-669fda75f1ac" counter packets 0 bytes 0 jump DOCKER
oifname "br-65f6dc782562" counter packets 0 bytes 0 jump DOCKER
}
chain DOCKER-CT {
oifname "br-c70303d221ee" xt match "conntrack" counter packets 0 bytes 0 accept
oifname "br-b3240c822bce" xt match "conntrack" counter packets 0 bytes 0 accept
oifname "br-3636e05760a9" xt match "conntrack" counter packets 35 bytes 23113 accept
oifname "br-3a760a74f4f6" xt match "conntrack" counter packets 0 bytes 0 accept
oifname "br-3dcb6ef83ea1" xt match "conntrack" counter packets 0 bytes 0 accept
oifname "br-40094534a5ee" xt match "conntrack" counter packets 0 bytes 0 accept
oifname "br-679db9b21e00" xt match "conntrack" counter packets 0 bytes 0 accept
oifname "br-9d6c95e8d80c" xt match "conntrack" counter packets 0 bytes 0 accept
oifname "br-af4c9c2aa60a" xt match "conntrack" counter packets 2488066 bytes 1053784742 accept
oifname "br-d44fef8fd602" xt match "conntrack" counter packets 0 bytes 0 accept
oifname "br-dba077b9b543" xt match "conntrack" counter packets 0 bytes 0 accept
oifname "br-160f55da427c" xt match "conntrack" counter packets 0 bytes 0 accept
oifname "br-07a5e2f2c42f" xt match "conntrack" counter packets 0 bytes 0 accept
oifname "docker0" xt match "conntrack" counter packets 666252 bytes 5079577802 accept
oifname "br-613eb68ef5fb" xt match "conntrack" counter packets 7926 bytes 7636543 accept
oifname "br-4b504efd6080" xt match "conntrack" counter packets 29 bytes 10870 accept
oifname "br-ef6df03f71a0" xt match "conntrack" counter packets 0 bytes 0 accept
oifname "br-ec480f77ac34" xt match "conntrack" counter packets 0 bytes 0 accept
oifname "br-669fda75f1ac" xt match "conntrack" counter packets 0 bytes 0 accept
oifname "br-65f6dc782562" xt match "conntrack" counter packets 0 bytes 0 accept
}
chain DOCKER-INTERNAL {
}
chain FORWARD {
type filter hook forward priority filter; policy drop;
counter packets 33747166 bytes 176750349038 jump DOCKER-USER
counter packets 6530319 bytes 11196484299 jump DOCKER-FORWARD
}
chain DOCKER-USER {
oifname "mlab*" counter packets 15657582 bytes 162801189460 accept
iifname "mlab*" counter packets 10958315 bytes 687322055 accept
oifname "incusbr0" counter packets 388768 bytes 2053271282 accept
iifname "incusbr0" counter packets 212182 bytes 12081942 accept
}
}
# Warning: table ip6 nat is managed by iptables-nft, do not touch!
table ip6 nat {
chain DOCKER {
}
chain PREROUTING {
type nat hook prerouting priority dstnat; policy accept;
xt match "addrtype" counter packets 532 bytes 113834 jump DOCKER
}
chain OUTPUT {
type nat hook output priority dstnat; policy accept;
ip6 daddr != ::1 xt match "addrtype" counter packets 0 bytes 0 jump DOCKER
}
}
table ip6 filter {
chain DOCKER {
}
chain DOCKER-FORWARD {
counter packets 0 bytes 0 jump DOCKER-CT
counter packets 0 bytes 0 jump DOCKER-INTERNAL
counter packets 0 bytes 0 jump DOCKER-BRIDGE
}
chain DOCKER-BRIDGE {
}
chain DOCKER-CT {
}
chain DOCKER-INTERNAL {
}
chain FORWARD {
type filter hook forward priority filter; policy accept;
counter packets 0 bytes 0 jump DOCKER-USER
counter packets 0 bytes 0 jump DOCKER-FORWARD
}
chain DOCKER-USER {
}
}
table ip raw {
chain PREROUTING {
type filter hook prerouting priority raw; policy accept;
ip daddr 172.19.0.2 iifname != "br-c70303d221ee" counter packets 0 bytes 0 drop
ip daddr 192.168.80.2 iifname != "br-3636e05760a9" counter packets 0 bytes 0 drop
ip daddr 192.168.80.3 iifname != "br-3636e05760a9" counter packets 0 bytes 0 drop
ip daddr 127.0.0.1 iifname != "lo" tcp dport 15673 counter packets 0 bytes 0 drop
ip daddr 192.168.80.4 iifname != "br-3636e05760a9" counter packets 0 bytes 0 drop
ip daddr 172.17.0.2 iifname != "docker0" counter packets 0 bytes 0 drop
ip daddr 127.0.0.1 iifname != "lo" tcp dport 57732 counter packets 0 bytes 0 drop
ip daddr 127.0.0.1 iifname != "lo" tcp dport 57733 counter packets 0 bytes 0 drop
ip daddr 172.17.0.3 iifname != "docker0" counter packets 0 bytes 0 drop
ip daddr 172.17.0.4 iifname != "docker0" counter packets 0 bytes 0 drop
ip daddr 192.168.32.2 iifname != "br-613eb68ef5fb" counter packets 0 bytes 0 drop
ip daddr 172.30.0.7 iifname != "br-af4c9c2aa60a" counter packets 0 bytes 0 drop
ip daddr 172.29.0.2 iifname != "br-b3240c822bce" counter packets 0 bytes 0 drop
ip daddr 127.0.0.1 iifname != "lo" tcp dport 15672 counter packets 0 bytes 0 drop
ip daddr 172.25.0.2 iifname != "br-4b504efd6080" counter packets 0 bytes 0 drop
ip daddr 172.30.0.9 iifname != "br-af4c9c2aa60a" counter packets 0 bytes 0 drop
ip daddr 192.168.48.2 iifname != "br-ef6df03f71a0" counter packets 0 bytes 0 drop
ip daddr 127.0.0.1 iifname != "lo" tcp dport 5432 counter packets 0 bytes 0 drop
ip daddr 192.168.48.3 iifname != "br-ef6df03f71a0" counter packets 0 bytes 0 drop
ip daddr 127.0.0.1 iifname != "lo" tcp dport 8081 counter packets 0 bytes 0 drop
ip daddr 172.30.0.8 iifname != "br-af4c9c2aa60a" counter packets 0 bytes 0 drop
ip daddr 172.31.0.2 iifname != "br-ec480f77ac34" counter packets 0 bytes 0 drop
ip daddr 127.0.0.1 iifname != "lo" tcp dport 6379 counter packets 0 bytes 0 drop
ip daddr 172.30.0.4 iifname != "br-af4c9c2aa60a" counter packets 0 bytes 0 drop
ip daddr 127.0.0.1 iifname != "lo" tcp dport 8001 counter packets 0 bytes 0 drop
ip daddr 172.31.0.3 iifname != "br-ec480f77ac34" counter packets 0 bytes 0 drop
ip daddr 172.28.0.2 iifname != "br-669fda75f1ac" counter packets 0 bytes 0 drop
ip daddr 172.30.0.6 iifname != "br-af4c9c2aa60a" counter packets 0 bytes 0 drop
ip daddr 172.28.0.3 iifname != "br-669fda75f1ac" counter packets 0 bytes 0 drop
ip daddr 127.0.0.1 iifname != "lo" tcp dport 28080 counter packets 0 bytes 0 drop
ip daddr 172.30.0.3 iifname != "br-af4c9c2aa60a" counter packets 0 bytes 0 drop
ip daddr 127.0.0.1 iifname != "lo" tcp dport 6789 counter packets 0 bytes 0 drop
ip daddr 172.30.0.5 iifname != "br-af4c9c2aa60a" counter packets 0 bytes 0 drop
ip daddr 172.22.0.2 iifname != "br-65f6dc782562" counter packets 0 bytes 0 drop
ip daddr 172.30.0.10 iifname != "br-af4c9c2aa60a" counter packets 0 bytes 0 drop
ip daddr 172.22.0.3 iifname != "br-65f6dc782562" counter packets 0 bytes 0 drop
ip daddr 172.30.0.2 iifname != "br-af4c9c2aa60a" counter packets 0 bytes 0 drop
ip daddr 172.17.0.5 iifname != "docker0" counter packets 0 bytes 0 drop
ip daddr 127.0.0.1 iifname != "lo" tcp dport 55541 counter packets 0 bytes 0 drop
}
}
table ip mangle {
chain FORWARD {
type filter hook forward priority mangle; policy accept;
tcp flags & (syn | rst) == syn counter packets 13760 bytes 825476 xt target "TCPMSS"
}
}
+22
View File
@@ -0,0 +1,22 @@
-P INPUT ACCEPT
-P FORWARD DROP
-P OUTPUT ACCEPT
-N DOCKER
-N DOCKER-BRIDGE
-N DOCKER-CT
-N DOCKER-FORWARD
-N DOCKER-INTERNAL
-N DOCKER-USER
-N f2b-sshd
-A INPUT -p tcp -m multiport --dports 22 -j f2b-sshd
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-FORWARD
-A DOCKER ! -i docker0 -o docker0 -j DROP
-A DOCKER-BRIDGE -o docker0 -j DOCKER
-A DOCKER-CT -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A DOCKER-FORWARD -j DOCKER-CT
-A DOCKER-FORWARD -j DOCKER-INTERNAL
-A DOCKER-FORWARD -j DOCKER-BRIDGE
-A DOCKER-FORWARD -i docker0 -j ACCEPT
-A f2b-sshd -s 192.0.2.55/32 -j REJECT --reject-with icmp-port-unreachable
-A f2b-sshd -j RETURN
+103
View File
@@ -0,0 +1,103 @@
table ip nat {
chain DOCKER {
}
chain PREROUTING {
type nat hook prerouting priority dstnat; policy accept;
xt match "addrtype" counter packets 0 bytes 0 jump DOCKER
}
chain OUTPUT {
type nat hook output priority dstnat; policy accept;
ip daddr != 127.0.0.0/8 xt match "addrtype" counter packets 0 bytes 0 jump DOCKER
}
chain POSTROUTING {
type nat hook postrouting priority srcnat; policy accept;
ip saddr 172.17.0.0/16 oifname != "docker0" counter packets 0 bytes 0 xt target "MASQUERADE"
}
}
table ip filter {
chain DOCKER {
iifname != "docker0" oifname "docker0" counter packets 0 bytes 0 drop
}
chain DOCKER-FORWARD {
counter packets 0 bytes 0 jump DOCKER-CT
counter packets 0 bytes 0 jump DOCKER-INTERNAL
counter packets 0 bytes 0 jump DOCKER-BRIDGE
iifname "docker0" counter packets 0 bytes 0 accept
}
chain DOCKER-BRIDGE {
oifname "docker0" counter packets 0 bytes 0 jump DOCKER
}
chain DOCKER-CT {
oifname "docker0" xt match "conntrack" counter packets 0 bytes 0 accept
}
chain DOCKER-INTERNAL {
}
chain FORWARD {
type filter hook forward priority filter; policy drop;
counter packets 0 bytes 0 jump DOCKER-USER
counter packets 0 bytes 0 jump DOCKER-FORWARD
}
chain DOCKER-USER {
}
chain f2b-sshd {
ip saddr 192.0.2.55 counter packets 0 bytes 0 xt target "REJECT"
counter packets 0 bytes 0 return
}
chain INPUT {
type filter hook input priority filter; policy accept;
ip protocol tcp xt match "multiport" counter packets 0 bytes 0 jump f2b-sshd
}
}
table ip6 nat {
chain DOCKER {
}
chain PREROUTING {
type nat hook prerouting priority dstnat; policy accept;
xt match "addrtype" counter packets 0 bytes 0 jump DOCKER
}
chain OUTPUT {
type nat hook output priority dstnat; policy accept;
ip6 daddr != ::1 xt match "addrtype" counter packets 0 bytes 0 jump DOCKER
}
}
table ip6 filter {
chain DOCKER {
}
chain DOCKER-FORWARD {
counter packets 0 bytes 0 jump DOCKER-CT
counter packets 0 bytes 0 jump DOCKER-INTERNAL
counter packets 0 bytes 0 jump DOCKER-BRIDGE
}
chain DOCKER-BRIDGE {
}
chain DOCKER-CT {
}
chain DOCKER-INTERNAL {
}
chain FORWARD {
type filter hook forward priority filter; policy accept;
counter packets 0 bytes 0 jump DOCKER-USER
counter packets 0 bytes 0 jump DOCKER-FORWARD
}
chain DOCKER-USER {
}
}
+105
View File
@@ -0,0 +1,105 @@
table ip nat {
chain DOCKER {
}
chain PREROUTING {
type nat hook prerouting priority dstnat; policy accept;
xt match "addrtype" counter packets 0 bytes 0 jump DOCKER
}
chain OUTPUT {
type nat hook output priority dstnat; policy accept;
ip daddr != 127.0.0.0/8 xt match "addrtype" counter packets 0 bytes 0 jump DOCKER
}
chain POSTROUTING {
type nat hook postrouting priority srcnat; policy accept;
ip saddr 172.17.0.0/16 oifname != "docker0" counter packets 0 bytes 0 xt target "MASQUERADE"
}
}
table ip filter {
chain DOCKER {
iifname != "docker0" oifname "docker0" counter packets 0 bytes 0 drop
}
chain DOCKER-FORWARD {
counter packets 0 bytes 0 jump DOCKER-CT
counter packets 0 bytes 0 jump DOCKER-INTERNAL
counter packets 0 bytes 0 jump DOCKER-BRIDGE
iifname "docker0" counter packets 0 bytes 0 accept
}
chain DOCKER-BRIDGE {
oifname "docker0" counter packets 0 bytes 0 jump DOCKER
}
chain DOCKER-CT {
oifname "docker0" xt match "conntrack" counter packets 0 bytes 0 accept
}
chain DOCKER-INTERNAL {
}
chain FORWARD {
type filter hook forward priority filter; policy drop;
counter packets 0 bytes 0 jump DOCKER-USER
counter packets 0 bytes 0 jump DOCKER-FORWARD
}
chain DOCKER-USER {
}
}
table ip6 nat {
chain DOCKER {
}
chain PREROUTING {
type nat hook prerouting priority dstnat; policy accept;
xt match "addrtype" counter packets 0 bytes 0 jump DOCKER
}
chain OUTPUT {
type nat hook output priority dstnat; policy accept;
ip6 daddr != ::1 xt match "addrtype" counter packets 0 bytes 0 jump DOCKER
}
}
table ip6 filter {
chain DOCKER {
}
chain DOCKER-FORWARD {
counter packets 0 bytes 0 jump DOCKER-CT
counter packets 0 bytes 0 jump DOCKER-INTERNAL
counter packets 0 bytes 0 jump DOCKER-BRIDGE
}
chain DOCKER-BRIDGE {
}
chain DOCKER-CT {
}
chain DOCKER-INTERNAL {
}
chain FORWARD {
type filter hook forward priority filter; policy accept;
counter packets 0 bytes 0 jump DOCKER-USER
counter packets 0 bytes 0 jump DOCKER-FORWARD
}
chain DOCKER-USER {
}
}
table inet f2b-table {
set addr-set-sshd {
type ipv4_addr
flags interval
elements = { 192.0.2.55 }
}
chain f2b-chain {
type filter hook input priority filter - 1; policy accept;
tcp dport 22 ip saddr @addr-set-sshd reject with icmp port-unreachable
}
}
+149
View File
@@ -0,0 +1,149 @@
-P INPUT ACCEPT
-P FORWARD DROP
-P OUTPUT ACCEPT
-N DOCKER
-N DOCKER-BRIDGE
-N DOCKER-CT
-N DOCKER-FORWARD
-N DOCKER-INTERNAL
-N DOCKER-USER
-N HAL-MESH-ONLY
-N ufw-after-forward
-N ufw-after-input
-N ufw-after-logging-forward
-N ufw-after-logging-input
-N ufw-after-logging-output
-N ufw-after-output
-N ufw-before-forward
-N ufw-before-input
-N ufw-before-logging-forward
-N ufw-before-logging-input
-N ufw-before-logging-output
-N ufw-before-output
-N ufw-reject-forward
-N ufw-reject-input
-N ufw-reject-output
-N ufw-track-forward
-N ufw-track-input
-N ufw-track-output
-A INPUT -j ufw-before-logging-input
-A INPUT -j ufw-before-input
-A INPUT -j ufw-after-input
-A INPUT -j ufw-after-logging-input
-A INPUT -j ufw-reject-input
-A INPUT -j ufw-track-input
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-FORWARD
-A FORWARD -j ufw-before-logging-forward
-A FORWARD -j ufw-before-forward
-A FORWARD -j ufw-after-forward
-A FORWARD -j ufw-after-logging-forward
-A FORWARD -j ufw-reject-forward
-A FORWARD -j ufw-track-forward
-A OUTPUT -j ufw-before-logging-output
-A OUTPUT -j ufw-before-output
-A OUTPUT -j ufw-after-output
-A OUTPUT -j ufw-after-logging-output
-A OUTPUT -j ufw-reject-output
-A OUTPUT -j ufw-track-output
-A DOCKER -d 172.17.0.18/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 8686 -j ACCEPT
-A DOCKER -d 172.17.0.14/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 8989 -j ACCEPT
-A DOCKER -d 172.17.0.15/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 7878 -j ACCEPT
-A DOCKER -d 172.17.0.5/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 9117 -j ACCEPT
-A DOCKER -d 172.17.0.13/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 6789 -j ACCEPT
-A DOCKER -d 172.19.0.2/32 ! -i br-32062158f584 -o br-32062158f584 -p tcp -m tcp --dport 8080 -j ACCEPT
-A DOCKER -d 172.17.0.2/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 5432 -j ACCEPT
-A DOCKER -d 172.27.0.2/32 ! -i br-0910a98c6158 -o br-0910a98c6158 -p tcp -m tcp --dport 5678 -j ACCEPT
-A DOCKER -d 172.17.0.21/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 3579 -j ACCEPT
-A DOCKER -d 172.17.0.19/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 8181 -j ACCEPT
-A DOCKER -d 172.17.0.17/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 8787 -j ACCEPT
-A DOCKER -d 172.17.0.16/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 6767 -j ACCEPT
-A DOCKER -d 172.17.0.12/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 3000 -j ACCEPT
-A DOCKER -d 172.17.0.11/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 80 -j ACCEPT
-A DOCKER -d 172.17.0.10/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 9443 -j ACCEPT
-A DOCKER -d 172.17.0.10/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 9000 -j ACCEPT
-A DOCKER -d 172.17.0.9/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 3000 -j ACCEPT
-A DOCKER -d 172.17.0.7/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 1880 -j ACCEPT
-A DOCKER -d 172.28.0.2/32 ! -i br-b11461b5b028 -o br-b11461b5b028 -p tcp -m tcp --dport 80 -j ACCEPT
-A DOCKER -d 172.23.0.14/32 ! -i br-66ffa5c1cba5 -o br-66ffa5c1cba5 -p tcp -m tcp --dport 6543 -j ACCEPT
-A DOCKER -d 172.23.0.14/32 ! -i br-66ffa5c1cba5 -o br-66ffa5c1cba5 -p tcp -m tcp --dport 5432 -j ACCEPT
-A DOCKER -d 172.23.0.5/32 ! -i br-66ffa5c1cba5 -o br-66ffa5c1cba5 -p tcp -m tcp --dport 8000 -j ACCEPT
-A DOCKER -d 172.26.0.3/32 ! -i br-b0fec361ccaa -o br-b0fec361ccaa -p tcp -m tcp --dport 6167 -j ACCEPT
-A DOCKER -d 172.26.0.2/32 ! -i br-b0fec361ccaa -o br-b0fec361ccaa -p tcp -m tcp --dport 80 -j ACCEPT
-A DOCKER -d 172.17.0.8/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 8000 -j ACCEPT
-A DOCKER -d 172.17.0.6/32 ! -i docker0 -o docker0 -p udp -m udp --dport 10001 -j ACCEPT
-A DOCKER -d 172.17.0.6/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 8880 -j ACCEPT
-A DOCKER -d 172.17.0.6/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 8843 -j ACCEPT
-A DOCKER -d 172.17.0.6/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 8443 -j ACCEPT
-A DOCKER -d 172.17.0.6/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 8080 -j ACCEPT
-A DOCKER -d 172.17.0.6/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 6789 -j ACCEPT
-A DOCKER -d 172.17.0.6/32 ! -i docker0 -o docker0 -p udp -m udp --dport 5514 -j ACCEPT
-A DOCKER -d 172.17.0.6/32 ! -i docker0 -o docker0 -p udp -m udp --dport 3478 -j ACCEPT
-A DOCKER -d 172.17.0.6/32 ! -i docker0 -o docker0 -p udp -m udp --dport 1900 -j ACCEPT
-A DOCKER -d 172.25.0.3/32 ! -i br-b98821f7dc38 -o br-b98821f7dc38 -p tcp -m tcp --dport 8000 -j ACCEPT
-A DOCKER -d 172.18.0.3/32 ! -i br-442a0bfc65f8 -o br-442a0bfc65f8 -p tcp -m tcp --dport 1433 -j ACCEPT
-A DOCKER -d 172.20.0.3/32 ! -i br-afa37ac8b33d -o br-afa37ac8b33d -p tcp -m tcp --dport 8081 -j ACCEPT
-A DOCKER -d 172.20.0.3/32 ! -i br-afa37ac8b33d -o br-afa37ac8b33d -p tcp -m tcp --dport 1883 -j ACCEPT
-A DOCKER -d 172.17.0.4/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 8086 -j ACCEPT
-A DOCKER -d 172.21.0.2/32 ! -i br-df15d8e19ec7 -o br-df15d8e19ec7 -p tcp -m tcp --dport 6379 -j ACCEPT
-A DOCKER -d 172.30.0.3/32 ! -i br-521eab9a3a5e -o br-521eab9a3a5e -p tcp -m tcp --dport 8283 -j ACCEPT
-A DOCKER -d 172.17.0.3/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 3000 -j ACCEPT
-A DOCKER ! -i br-32062158f584 -o br-32062158f584 -j DROP
-A DOCKER ! -i docker0 -o docker0 -j DROP
-A DOCKER ! -i br-521eab9a3a5e -o br-521eab9a3a5e -j DROP
-A DOCKER ! -i br-df15d8e19ec7 -o br-df15d8e19ec7 -j DROP
-A DOCKER ! -i br-afa37ac8b33d -o br-afa37ac8b33d -j DROP
-A DOCKER ! -i br-442a0bfc65f8 -o br-442a0bfc65f8 -j DROP
-A DOCKER ! -i br-b98821f7dc38 -o br-b98821f7dc38 -j DROP
-A DOCKER ! -i br-b0fec361ccaa -o br-b0fec361ccaa -j DROP
-A DOCKER ! -i br-66ffa5c1cba5 -o br-66ffa5c1cba5 -j DROP
-A DOCKER ! -i br-b11461b5b028 -o br-b11461b5b028 -j DROP
-A DOCKER ! -i br-2df4e541b877 -o br-2df4e541b877 -j DROP
-A DOCKER ! -i br-0910a98c6158 -o br-0910a98c6158 -j DROP
-A DOCKER-BRIDGE -o br-32062158f584 -j DOCKER
-A DOCKER-BRIDGE -o docker0 -j DOCKER
-A DOCKER-BRIDGE -o br-521eab9a3a5e -j DOCKER
-A DOCKER-BRIDGE -o br-df15d8e19ec7 -j DOCKER
-A DOCKER-BRIDGE -o br-afa37ac8b33d -j DOCKER
-A DOCKER-BRIDGE -o br-442a0bfc65f8 -j DOCKER
-A DOCKER-BRIDGE -o br-b98821f7dc38 -j DOCKER
-A DOCKER-BRIDGE -o br-b0fec361ccaa -j DOCKER
-A DOCKER-BRIDGE -o br-66ffa5c1cba5 -j DOCKER
-A DOCKER-BRIDGE -o br-b11461b5b028 -j DOCKER
-A DOCKER-BRIDGE -o br-2df4e541b877 -j DOCKER
-A DOCKER-BRIDGE -o br-0910a98c6158 -j DOCKER
-A DOCKER-CT -o br-32062158f584 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A DOCKER-CT -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A DOCKER-CT -o br-521eab9a3a5e -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A DOCKER-CT -o br-df15d8e19ec7 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A DOCKER-CT -o br-afa37ac8b33d -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A DOCKER-CT -o br-442a0bfc65f8 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A DOCKER-CT -o br-b98821f7dc38 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A DOCKER-CT -o br-b0fec361ccaa -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A DOCKER-CT -o br-66ffa5c1cba5 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A DOCKER-CT -o br-b11461b5b028 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A DOCKER-CT -o br-2df4e541b877 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A DOCKER-CT -o br-0910a98c6158 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A DOCKER-FORWARD -j DOCKER-CT
-A DOCKER-FORWARD -j DOCKER-INTERNAL
-A DOCKER-FORWARD -j DOCKER-BRIDGE
-A DOCKER-FORWARD -i br-32062158f584 -j ACCEPT
-A DOCKER-FORWARD -i docker0 -j ACCEPT
-A DOCKER-FORWARD -i br-521eab9a3a5e -j ACCEPT
-A DOCKER-FORWARD -i br-df15d8e19ec7 -j ACCEPT
-A DOCKER-FORWARD -i br-afa37ac8b33d -j ACCEPT
-A DOCKER-FORWARD -i br-442a0bfc65f8 -j ACCEPT
-A DOCKER-FORWARD -i br-b98821f7dc38 -j ACCEPT
-A DOCKER-FORWARD -i br-b0fec361ccaa -j ACCEPT
-A DOCKER-FORWARD -i br-66ffa5c1cba5 -j ACCEPT
-A DOCKER-FORWARD -i br-b11461b5b028 -j ACCEPT
-A DOCKER-FORWARD -i br-2df4e541b877 -j ACCEPT
-A DOCKER-FORWARD -i br-0910a98c6158 -j ACCEPT
-A DOCKER-USER -i enp6s0 -p tcp -m conntrack --ctstate NEW -j HAL-MESH-ONLY
-A HAL-MESH-ONLY -m conntrack --ctorigdstport 6881 -j RETURN
-A HAL-MESH-ONLY -m conntrack --ctorigdstport 80 -j RETURN
-A HAL-MESH-ONLY -m conntrack --ctorigdstport 443 -j RETURN
-A HAL-MESH-ONLY -s 10.0.0.0/8 -j RETURN
-A HAL-MESH-ONLY -s 172.16.0.0/12 -j RETURN
-A HAL-MESH-ONLY -s 192.168.0.0/16 -j RETURN
-A HAL-MESH-ONLY -m comment --comment "HAL: not public -> mesh only" -j DROP
+327
View File
@@ -0,0 +1,327 @@
table ip mangle {
chain FORWARD {
type filter hook forward priority mangle; policy accept;
}
}
# Warning: table ip nat is managed by iptables-nft, do not touch!
table ip nat {
chain PREROUTING {
type nat hook prerouting priority dstnat; policy accept;
xt match "addrtype" counter packets 12072 bytes 4564241 jump DOCKER
}
chain OUTPUT {
type nat hook output priority dstnat; policy accept;
ip daddr != 127.0.0.0/8 xt match "addrtype" counter packets 1854 bytes 111240 jump DOCKER
}
chain POSTROUTING {
type nat hook postrouting priority srcnat; policy accept;
ip saddr 172.17.0.0/16 oifname != "docker0" counter packets 903 bytes 61577 xt target "MASQUERADE"
ip saddr 172.21.0.0/16 oifname != "br-86a5d6b30e2b" counter packets 344 bytes 27744 xt target "MASQUERADE"
ip saddr 172.25.0.0/16 oifname != "br-61495e14a004" counter packets 374 bytes 33016 xt target "MASQUERADE"
ip saddr 172.30.0.0/16 oifname != "br-5107796ee9b4" counter packets 352 bytes 28224 xt target "MASQUERADE"
ip saddr 172.18.0.0/16 oifname != "br-cfd337ac4e58" counter packets 339 bytes 27444 xt target "MASQUERADE"
ip saddr 172.19.0.0/16 oifname != "br-8f0c6ee01425" counter packets 351 bytes 28164 xt target "MASQUERADE"
ip saddr 172.22.0.0/16 oifname != "br-75ac3c36e87f" counter packets 333 bytes 27084 xt target "MASQUERADE"
ip saddr 172.20.0.0/16 oifname != "br-0529801521bc" counter packets 343 bytes 27404 xt target "MASQUERADE"
}
chain DOCKER {
iifname != "br-61495e14a004" tcp dport 5680 counter packets 2 bytes 120 xt target "DNAT"
ip daddr 127.0.0.1 iifname != "br-61495e14a004" tcp dport 15673 counter packets 0 bytes 0 xt target "DNAT"
ip daddr 127.0.0.1 iifname != "docker0" tcp dport 55432 counter packets 0 bytes 0 xt target "DNAT"
}
}
# Warning: table ip filter is managed by iptables-nft, do not touch!
table ip filter {
chain DOCKER-FORWARD {
counter packets 702328 bytes 1801910999 jump DOCKER-CT
counter packets 337315 bytes 23928864 jump DOCKER-INTERNAL
counter packets 337315 bytes 23928864 jump DOCKER-BRIDGE
iifname "br-75ac3c36e87f" counter packets 0 bytes 0 accept
iifname "br-86a5d6b30e2b" counter packets 0 bytes 0 accept
iifname "br-8f0c6ee01425" counter packets 0 bytes 0 accept
iifname "br-cfd337ac4e58" counter packets 0 bytes 0 accept
iifname "br-0529801521bc" counter packets 0 bytes 0 accept
iifname "br-5107796ee9b4" counter packets 0 bytes 0 accept
iifname "br-61495e14a004" counter packets 0 bytes 0 accept
iifname "docker0" counter packets 337315 bytes 23928864 accept
}
chain FORWARD {
type filter hook forward priority filter; policy drop;
counter packets 702328 bytes 1801910999 jump DOCKER-USER
counter packets 702328 bytes 1801910999 jump DOCKER-FORWARD
}
chain DOCKER-USER {
ip protocol tcp counter packets 702510 bytes 1801965868 jump f2b-sshd
oifname "incusbr0" counter packets 0 bytes 0 accept
iifname "incusbr0" counter packets 0 bytes 0 accept
}
chain f2b-sshd {
counter packets 10423854 bytes 13891318049 return
}
chain INPUT {
type filter hook input priority filter; policy accept;
ip protocol tcp counter packets 9721344 bytes 12089352181 jump f2b-sshd
}
chain DOCKER {
ip daddr 172.17.0.2 iifname != "docker0" oifname "docker0" tcp dport 5432 counter packets 0 bytes 0 accept
ip daddr 172.25.0.2 iifname != "br-61495e14a004" oifname "br-61495e14a004" tcp dport 15672 counter packets 0 bytes 0 accept
ip daddr 172.25.0.2 iifname != "br-61495e14a004" oifname "br-61495e14a004" tcp dport 5672 counter packets 0 bytes 0 accept
iifname != "br-75ac3c36e87f" oifname "br-75ac3c36e87f" counter packets 0 bytes 0 drop
iifname != "br-86a5d6b30e2b" oifname "br-86a5d6b30e2b" counter packets 0 bytes 0 drop
iifname != "br-8f0c6ee01425" oifname "br-8f0c6ee01425" counter packets 0 bytes 0 drop
iifname != "br-cfd337ac4e58" oifname "br-cfd337ac4e58" counter packets 0 bytes 0 drop
iifname != "br-0529801521bc" oifname "br-0529801521bc" counter packets 0 bytes 0 drop
iifname != "br-5107796ee9b4" oifname "br-5107796ee9b4" counter packets 0 bytes 0 drop
iifname != "br-61495e14a004" oifname "br-61495e14a004" counter packets 0 bytes 0 drop
iifname != "docker0" oifname "docker0" counter packets 0 bytes 0 drop
}
chain DOCKER-BRIDGE {
oifname "br-75ac3c36e87f" counter packets 0 bytes 0 jump DOCKER
oifname "br-86a5d6b30e2b" counter packets 0 bytes 0 jump DOCKER
oifname "br-8f0c6ee01425" counter packets 0 bytes 0 jump DOCKER
oifname "br-cfd337ac4e58" counter packets 0 bytes 0 jump DOCKER
oifname "br-0529801521bc" counter packets 0 bytes 0 jump DOCKER
oifname "br-5107796ee9b4" counter packets 0 bytes 0 jump DOCKER
oifname "br-61495e14a004" counter packets 0 bytes 0 jump DOCKER
oifname "docker0" counter packets 0 bytes 0 jump DOCKER
}
chain DOCKER-CT {
oifname "br-75ac3c36e87f" xt match "conntrack" counter packets 0 bytes 0 accept
oifname "br-86a5d6b30e2b" xt match "conntrack" counter packets 0 bytes 0 accept
oifname "br-8f0c6ee01425" xt match "conntrack" counter packets 0 bytes 0 accept
oifname "br-cfd337ac4e58" xt match "conntrack" counter packets 0 bytes 0 accept
oifname "br-0529801521bc" xt match "conntrack" counter packets 0 bytes 0 accept
oifname "br-5107796ee9b4" xt match "conntrack" counter packets 0 bytes 0 accept
oifname "br-61495e14a004" xt match "conntrack" counter packets 0 bytes 0 accept
oifname "docker0" xt match "conntrack" counter packets 365013 bytes 1777982135 accept
}
chain DOCKER-INTERNAL {
}
}
# Warning: table ip6 nat is managed by iptables-nft, do not touch!
table ip6 nat {
chain PREROUTING {
type nat hook prerouting priority dstnat; policy accept;
xt match "addrtype" counter packets 363 bytes 67927 jump DOCKER
}
chain OUTPUT {
type nat hook output priority dstnat; policy accept;
ip6 daddr != ::1 xt match "addrtype" counter packets 0 bytes 0 jump DOCKER
}
chain DOCKER {
}
}
table ip6 filter {
chain DOCKER-FORWARD {
counter packets 0 bytes 0 jump DOCKER-CT
counter packets 0 bytes 0 jump DOCKER-INTERNAL
counter packets 0 bytes 0 jump DOCKER-BRIDGE
}
chain FORWARD {
type filter hook forward priority filter; policy accept;
counter packets 0 bytes 0 jump DOCKER-USER
counter packets 0 bytes 0 jump DOCKER-FORWARD
}
chain DOCKER-USER {
}
chain DOCKER {
}
chain DOCKER-BRIDGE {
}
chain DOCKER-CT {
}
chain DOCKER-INTERNAL {
}
}
table ip raw {
chain PREROUTING {
type filter hook prerouting priority raw; policy accept;
ip daddr 172.25.0.2 iifname != "br-61495e14a004" counter packets 0 bytes 0 drop
ip daddr 127.0.0.1 iifname != "lo" tcp dport 15673 counter packets 0 bytes 0 drop
ip daddr 172.17.0.2 iifname != "docker0" counter packets 0 bytes 0 drop
ip daddr 127.0.0.1 iifname != "lo" tcp dport 55432 counter packets 0 bytes 0 drop
}
}
table inet incus {
set bridges {
type ifname
elements = { "incusbr0" }
}
chain pstrt.incusbr0 {
type nat hook postrouting priority srcnat; policy accept;
ip saddr 10.7.169.0/24 oifname @bridges accept
ip saddr 10.7.169.0/24 ip daddr != 10.7.169.0/24 masquerade
ip6 saddr fd42:cbc4:e123:f6::/64 oifname @bridges accept
ip6 saddr fd42:cbc4:e123:f6::/64 ip6 daddr != fd42:cbc4:e123:f6::/64 masquerade
}
chain fwd.incusbr0 {
type filter hook forward priority filter; policy accept;
ip version 4 oifname "incusbr0" accept
ip version 4 iifname "incusbr0" accept
ip6 version 6 oifname "incusbr0" accept
ip6 version 6 iifname "incusbr0" accept
}
chain in.incusbr0 {
type filter hook input priority filter; policy accept;
iifname "incusbr0" tcp dport 53 accept
iifname "incusbr0" udp dport 53 accept
iifname "incusbr0" icmp type { destination-unreachable, time-exceeded, parameter-problem } accept
iifname "incusbr0" udp dport 67 accept
iifname "incusbr0" ip protocol udp udp checksum set 0
iifname "incusbr0" icmpv6 type { destination-unreachable, packet-too-big, time-exceeded, parameter-problem, nd-router-solicit, nd-neighbor-solicit, nd-neighbor-advert, mld2-listener-report } accept
iifname "incusbr0" udp dport 547 accept
}
chain out.incusbr0 {
type filter hook output priority filter; policy accept;
oifname "incusbr0" tcp sport 53 accept
oifname "incusbr0" udp sport 53 accept
oifname "incusbr0" icmp type { destination-unreachable, time-exceeded, parameter-problem } accept
oifname "incusbr0" udp sport 67 accept
oifname "incusbr0" ip protocol udp udp checksum set 0
oifname "incusbr0" icmpv6 type { destination-unreachable, packet-too-big, time-exceeded, parameter-problem, echo-request, nd-router-advert, nd-neighbor-solicit, nd-neighbor-advert, mld2-listener-report } accept
oifname "incusbr0" udp sport 547 accept
}
}
table ip fct_filter {
chain OUTPUT {
type filter hook output priority filter; policy accept;
}
chain FCT-QUARANTINE-EMS {
}
chain FCT-QUARANTINE-FAZ {
}
chain FORWARD {
type filter hook forward priority filter; policy accept;
}
chain FCT-WEBFILTER-QUIC-CHAIN {
}
chain INPUT {
type filter hook input priority filter; policy accept;
}
chain FCT-QUARANTINE {
}
chain FCT-DNS-QUIC-FILTER {
}
chain FCT-VPN-CHAIN {
}
}
table ip fct_nat {
chain OUTPUT {
type nat hook output priority dstnat; policy accept;
}
chain FCT-DNS-UDP-CHAIN-STAGE-2 {
}
chain FCT-DNS-UDP-CHAIN-STAGE-1 {
}
chain FCT-TCP-CHAIN {
}
chain FCT-DNS-DOH-CHAIN-STAGE-1 {
}
chain FCT-WEBFILTER-CHAIN {
}
chain FCT-DNS-DOH-CHAIN-STAGE-2 {
}
}
table ip6 fct_filter {
chain FCT-QUARANTINE {
}
chain INPUT {
type filter hook input priority filter; policy accept;
}
chain FORWARD {
type filter hook forward priority filter; policy accept;
}
chain OUTPUT {
type filter hook output priority filter; policy accept;
}
}
table ip fct_mangle {
chain PREROUTING {
type filter hook prerouting priority mangle; policy accept;
}
chain FCT-UDP-STAGE-1 {
}
chain OUTPUT {
type route hook output priority mangle; policy accept;
}
chain FCT-UDP-STAGE-2 {
}
chain FCT-UDP-OUTPUT {
}
}
table inet mesh {
chain input {
type filter hook input priority filter; policy drop;
ct state established,related accept
ct state invalid drop
iif "lo" accept
iifname != { "mesh0", "wlp3s0" } accept
icmp type echo-request accept
icmpv6 type { echo-request, nd-router-advert, nd-neighbor-solicit, nd-neighbor-advert } accept
iifname != { "mesh0", "wlp3s0" } udp dport { 53, 67 } accept
iifname != { "mesh0", "wlp3s0" } tcp dport 53 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } tcp dport 22 accept
tcp dport 22 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } tcp dport 53 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } udp dport 53 accept
}
chain output {
type filter hook output priority filter; policy accept;
}
chain forward {
type filter hook forward priority filter; policy drop;
ct state established,related accept
ct state invalid drop
iifname != { "mesh0", "wlp3s0" } accept
iifname "mesh0" oifname "mesh0" accept
ct original proto-dst 22 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } ct original proto-dst 53 accept
ip saddr { 10.10.0.1, 10.10.0.2, 10.10.0.3, 10.10.0.4 } ct original proto-dst 53 accept
}
}
+478
View File
@@ -0,0 +1,478 @@
table ip nat {
chain DOCKER {
}
chain PREROUTING {
type nat hook prerouting priority dstnat; policy accept;
xt match "addrtype" counter packets 2 bytes 1160 jump DOCKER
}
chain OUTPUT {
type nat hook output priority dstnat; policy accept;
ip daddr != 127.0.0.0/8 xt match "addrtype" counter packets 0 bytes 0 jump DOCKER
}
chain POSTROUTING {
type nat hook postrouting priority srcnat; policy accept;
ip saddr 172.17.0.0/16 oifname != "docker0" counter packets 0 bytes 0 xt target "MASQUERADE"
}
}
table ip filter {
chain DOCKER {
iifname != "docker0" oifname "docker0" counter packets 0 bytes 0 drop
}
chain DOCKER-FORWARD {
counter packets 0 bytes 0 jump DOCKER-CT
counter packets 0 bytes 0 jump DOCKER-INTERNAL
counter packets 0 bytes 0 jump DOCKER-BRIDGE
iifname "docker0" counter packets 0 bytes 0 accept
}
chain DOCKER-BRIDGE {
oifname "docker0" counter packets 0 bytes 0 jump DOCKER
}
chain DOCKER-CT {
oifname "docker0" xt match "conntrack" counter packets 0 bytes 0 accept
}
chain DOCKER-INTERNAL {
}
chain FORWARD {
type filter hook forward priority filter; policy drop;
counter packets 0 bytes 0 jump DOCKER-USER
counter packets 0 bytes 0 jump DOCKER-FORWARD
counter packets 0 bytes 0 jump ufw-before-logging-forward
counter packets 0 bytes 0 jump ufw-before-forward
counter packets 0 bytes 0 jump ufw-after-forward
counter packets 0 bytes 0 jump ufw-after-logging-forward
counter packets 0 bytes 0 jump ufw-reject-forward
counter packets 0 bytes 0 jump ufw-track-forward
}
chain DOCKER-USER {
}
chain ufw-before-logging-input {
}
chain ufw-before-logging-output {
}
chain ufw-before-logging-forward {
}
chain ufw-before-input {
iifname "lo" counter packets 0 bytes 0 accept
xt match "conntrack" counter packets 0 bytes 0 accept
xt match "conntrack" counter packets 0 bytes 0 jump ufw-logging-deny
xt match "conntrack" counter packets 0 bytes 0 drop
ip protocol icmp xt match "icmp" counter packets 0 bytes 0 accept
ip protocol icmp xt match "icmp" counter packets 0 bytes 0 accept
ip protocol icmp xt match "icmp" counter packets 0 bytes 0 accept
ip protocol icmp xt match "icmp" counter packets 0 bytes 0 accept
udp sport 67 udp dport 68 counter packets 0 bytes 0 accept
counter packets 0 bytes 0 jump ufw-not-local
ip daddr 224.0.0.251 udp dport 5353 counter packets 0 bytes 0 accept
ip daddr 239.255.255.250 udp dport 1900 counter packets 0 bytes 0 accept
counter packets 0 bytes 0 jump ufw-user-input
}
chain ufw-before-output {
oifname "lo" counter packets 0 bytes 0 accept
xt match "conntrack" counter packets 0 bytes 0 accept
counter packets 0 bytes 0 jump ufw-user-output
}
chain ufw-before-forward {
xt match "conntrack" counter packets 0 bytes 0 accept
ip protocol icmp xt match "icmp" counter packets 0 bytes 0 accept
ip protocol icmp xt match "icmp" counter packets 0 bytes 0 accept
ip protocol icmp xt match "icmp" counter packets 0 bytes 0 accept
ip protocol icmp xt match "icmp" counter packets 0 bytes 0 accept
counter packets 0 bytes 0 jump ufw-user-forward
}
chain ufw-after-input {
udp dport 137 counter packets 0 bytes 0 jump ufw-skip-to-policy-input
udp dport 138 counter packets 0 bytes 0 jump ufw-skip-to-policy-input
tcp dport 139 counter packets 0 bytes 0 jump ufw-skip-to-policy-input
tcp dport 445 counter packets 0 bytes 0 jump ufw-skip-to-policy-input
udp dport 67 counter packets 0 bytes 0 jump ufw-skip-to-policy-input
udp dport 68 counter packets 0 bytes 0 jump ufw-skip-to-policy-input
xt match "addrtype" counter packets 0 bytes 0 jump ufw-skip-to-policy-input
}
chain ufw-after-output {
}
chain ufw-after-forward {
}
chain ufw-after-logging-input {
limit rate 3/minute burst 10 packets counter packets 0 bytes 0 xt target "LOG"
}
chain ufw-after-logging-output {
}
chain ufw-after-logging-forward {
limit rate 3/minute burst 10 packets counter packets 0 bytes 0 xt target "LOG"
}
chain ufw-reject-input {
}
chain ufw-reject-output {
}
chain ufw-reject-forward {
}
chain ufw-track-input {
}
chain ufw-track-output {
ip protocol tcp xt match "conntrack" counter packets 0 bytes 0 accept
ip protocol udp xt match "conntrack" counter packets 0 bytes 0 accept
}
chain ufw-track-forward {
}
chain INPUT {
type filter hook input priority filter; policy drop;
counter packets 1 bytes 76 jump ufw-before-logging-input
counter packets 1 bytes 76 jump ufw-before-input
counter packets 0 bytes 0 jump ufw-after-input
counter packets 0 bytes 0 jump ufw-after-logging-input
counter packets 0 bytes 0 jump ufw-reject-input
counter packets 0 bytes 0 jump ufw-track-input
}
chain OUTPUT {
type filter hook output priority filter; policy accept;
counter packets 1 bytes 76 jump ufw-before-logging-output
counter packets 1 bytes 76 jump ufw-before-output
counter packets 1 bytes 76 jump ufw-after-output
counter packets 1 bytes 76 jump ufw-after-logging-output
counter packets 1 bytes 76 jump ufw-reject-output
counter packets 1 bytes 76 jump ufw-track-output
}
chain ufw-logging-deny {
xt match "conntrack" limit rate 3/minute burst 10 packets counter packets 0 bytes 0 return
limit rate 3/minute burst 10 packets counter packets 0 bytes 0 xt target "LOG"
}
chain ufw-logging-allow {
limit rate 3/minute burst 10 packets counter packets 0 bytes 0 xt target "LOG"
}
chain ufw-skip-to-policy-input {
counter packets 0 bytes 0 drop
}
chain ufw-skip-to-policy-output {
counter packets 0 bytes 0 accept
}
chain ufw-skip-to-policy-forward {
counter packets 0 bytes 0 drop
}
chain ufw-not-local {
xt match "addrtype" counter packets 0 bytes 0 return
xt match "addrtype" counter packets 0 bytes 0 return
xt match "addrtype" counter packets 0 bytes 0 return
limit rate 3/minute burst 10 packets counter packets 0 bytes 0 jump ufw-logging-deny
counter packets 0 bytes 0 drop
}
chain ufw-user-input {
tcp dport 22 counter packets 0 bytes 0 accept
tcp dport 8080 counter packets 0 bytes 0 accept
udp dport 51820 counter packets 0 bytes 0 accept
}
chain ufw-user-output {
}
chain ufw-user-forward {
tcp dport 80 counter packets 0 bytes 0 accept
iifname "mesh0" tcp dport 443 counter packets 0 bytes 0 accept
}
chain ufw-user-logging-input {
}
chain ufw-user-logging-output {
}
chain ufw-user-logging-forward {
}
chain ufw-user-limit {
limit rate 3/minute burst 5 packets counter packets 0 bytes 0 xt target "LOG"
counter packets 0 bytes 0 xt target "REJECT"
}
chain ufw-user-limit-accept {
counter packets 0 bytes 0 accept
}
}
table ip6 nat {
chain DOCKER {
}
chain PREROUTING {
type nat hook prerouting priority dstnat; policy accept;
xt match "addrtype" counter packets 0 bytes 0 jump DOCKER
}
chain OUTPUT {
type nat hook output priority dstnat; policy accept;
ip6 daddr != ::1 xt match "addrtype" counter packets 0 bytes 0 jump DOCKER
}
}
table ip6 filter {
chain DOCKER {
}
chain DOCKER-FORWARD {
counter packets 0 bytes 0 jump DOCKER-CT
counter packets 0 bytes 0 jump DOCKER-INTERNAL
counter packets 0 bytes 0 jump DOCKER-BRIDGE
}
chain DOCKER-BRIDGE {
}
chain DOCKER-CT {
}
chain DOCKER-INTERNAL {
}
chain FORWARD {
type filter hook forward priority filter; policy drop;
counter packets 0 bytes 0 jump DOCKER-USER
counter packets 0 bytes 0 jump DOCKER-FORWARD
counter packets 0 bytes 0 jump ufw6-before-logging-forward
counter packets 0 bytes 0 jump ufw6-before-forward
counter packets 0 bytes 0 jump ufw6-after-forward
counter packets 0 bytes 0 jump ufw6-after-logging-forward
counter packets 0 bytes 0 jump ufw6-reject-forward
counter packets 0 bytes 0 jump ufw6-track-forward
}
chain DOCKER-USER {
}
chain ufw6-before-logging-input {
}
chain ufw6-before-logging-output {
}
chain ufw6-before-logging-forward {
}
chain ufw6-before-input {
iifname "lo" counter packets 0 bytes 0 accept
xt match "rt" counter packets 0 bytes 0 drop
xt match "conntrack" counter packets 0 bytes 0 accept
meta l4proto ipv6-icmp xt match "icmp6" counter packets 0 bytes 0 accept
xt match "conntrack" counter packets 0 bytes 0 jump ufw6-logging-deny
xt match "conntrack" counter packets 0 bytes 0 drop
meta l4proto ipv6-icmp xt match "icmp6" counter packets 0 bytes 0 accept
meta l4proto ipv6-icmp xt match "icmp6" counter packets 0 bytes 0 accept
meta l4proto ipv6-icmp xt match "icmp6" counter packets 0 bytes 0 accept
meta l4proto ipv6-icmp xt match "icmp6" counter packets 0 bytes 0 accept
meta l4proto ipv6-icmp xt match "icmp6" counter packets 0 bytes 0 accept
meta l4proto ipv6-icmp xt match "icmp6" xt match "hl" counter packets 0 bytes 0 accept
meta l4proto ipv6-icmp xt match "icmp6" xt match "hl" counter packets 0 bytes 0 accept
meta l4proto ipv6-icmp xt match "icmp6" xt match "hl" counter packets 0 bytes 0 accept
meta l4proto ipv6-icmp xt match "icmp6" xt match "hl" counter packets 0 bytes 0 accept
meta l4proto ipv6-icmp xt match "icmp6" xt match "hl" counter packets 0 bytes 0 accept
meta l4proto ipv6-icmp xt match "icmp6" xt match "hl" counter packets 0 bytes 0 accept
ip6 saddr fe80::/10 meta l4proto ipv6-icmp xt match "icmp6" counter packets 0 bytes 0 accept
ip6 saddr fe80::/10 meta l4proto ipv6-icmp xt match "icmp6" counter packets 0 bytes 0 accept
ip6 saddr fe80::/10 meta l4proto ipv6-icmp xt match "icmp6" counter packets 0 bytes 0 accept
ip6 saddr fe80::/10 meta l4proto ipv6-icmp xt match "icmp6" counter packets 0 bytes 0 accept
meta l4proto ipv6-icmp xt match "icmp6" xt match "hl" counter packets 0 bytes 0 accept
meta l4proto ipv6-icmp xt match "icmp6" xt match "hl" counter packets 0 bytes 0 accept
ip6 saddr fe80::/10 meta l4proto ipv6-icmp xt match "icmp6" xt match "hl" counter packets 0 bytes 0 accept
ip6 saddr fe80::/10 meta l4proto ipv6-icmp xt match "icmp6" xt match "hl" counter packets 0 bytes 0 accept
ip6 saddr fe80::/10 meta l4proto ipv6-icmp xt match "icmp6" xt match "hl" counter packets 0 bytes 0 accept
meta l4proto ipv6-icmp xt match "icmp6" counter packets 0 bytes 0 accept
meta l4proto ipv6-icmp xt match "icmp6" counter packets 0 bytes 0 accept
meta l4proto ipv6-icmp xt match "icmp6" counter packets 0 bytes 0 accept
meta l4proto ipv6-icmp xt match "icmp6" counter packets 0 bytes 0 accept
ip6 saddr fe80::/10 ip6 daddr fe80::/10 udp sport 547 udp dport 546 counter packets 0 bytes 0 accept
ip6 daddr ff02::fb udp dport 5353 counter packets 0 bytes 0 accept
ip6 daddr ff02::f udp dport 1900 counter packets 0 bytes 0 accept
counter packets 0 bytes 0 jump ufw6-user-input
}
chain ufw6-before-output {
oifname "lo" counter packets 0 bytes 0 accept
xt match "rt" counter packets 0 bytes 0 drop
xt match "conntrack" counter packets 0 bytes 0 accept
meta l4proto ipv6-icmp xt match "icmp6" counter packets 0 bytes 0 accept
meta l4proto ipv6-icmp xt match "icmp6" counter packets 0 bytes 0 accept
meta l4proto ipv6-icmp xt match "icmp6" counter packets 0 bytes 0 accept
meta l4proto ipv6-icmp xt match "icmp6" counter packets 0 bytes 0 accept
meta l4proto ipv6-icmp xt match "icmp6" counter packets 0 bytes 0 accept
meta l4proto ipv6-icmp xt match "icmp6" counter packets 0 bytes 0 accept
meta l4proto ipv6-icmp xt match "icmp6" xt match "hl" counter packets 0 bytes 0 accept
meta l4proto ipv6-icmp xt match "icmp6" xt match "hl" counter packets 0 bytes 0 accept
meta l4proto ipv6-icmp xt match "icmp6" xt match "hl" counter packets 0 bytes 0 accept
meta l4proto ipv6-icmp xt match "icmp6" xt match "hl" counter packets 0 bytes 0 accept
meta l4proto ipv6-icmp xt match "icmp6" xt match "hl" counter packets 0 bytes 0 accept
meta l4proto ipv6-icmp xt match "icmp6" xt match "hl" counter packets 0 bytes 0 accept
ip6 saddr fe80::/10 meta l4proto ipv6-icmp xt match "icmp6" counter packets 0 bytes 0 accept
ip6 saddr fe80::/10 meta l4proto ipv6-icmp xt match "icmp6" counter packets 0 bytes 0 accept
ip6 saddr fe80::/10 meta l4proto ipv6-icmp xt match "icmp6" counter packets 0 bytes 0 accept
ip6 saddr fe80::/10 meta l4proto ipv6-icmp xt match "icmp6" counter packets 0 bytes 0 accept
meta l4proto ipv6-icmp xt match "icmp6" xt match "hl" counter packets 0 bytes 0 accept
meta l4proto ipv6-icmp xt match "icmp6" xt match "hl" counter packets 0 bytes 0 accept
ip6 saddr fe80::/10 meta l4proto ipv6-icmp xt match "icmp6" xt match "hl" counter packets 0 bytes 0 accept
ip6 saddr fe80::/10 meta l4proto ipv6-icmp xt match "icmp6" xt match "hl" counter packets 0 bytes 0 accept
ip6 saddr fe80::/10 meta l4proto ipv6-icmp xt match "icmp6" xt match "hl" counter packets 0 bytes 0 accept
counter packets 0 bytes 0 jump ufw6-user-output
}
chain ufw6-before-forward {
xt match "rt" counter packets 0 bytes 0 drop
xt match "conntrack" counter packets 0 bytes 0 accept
meta l4proto ipv6-icmp xt match "icmp6" counter packets 0 bytes 0 accept
meta l4proto ipv6-icmp xt match "icmp6" counter packets 0 bytes 0 accept
meta l4proto ipv6-icmp xt match "icmp6" counter packets 0 bytes 0 accept
meta l4proto ipv6-icmp xt match "icmp6" counter packets 0 bytes 0 accept
meta l4proto ipv6-icmp xt match "icmp6" counter packets 0 bytes 0 accept
meta l4proto ipv6-icmp xt match "icmp6" counter packets 0 bytes 0 accept
counter packets 0 bytes 0 jump ufw6-user-forward
}
chain ufw6-after-input {
udp dport 137 counter packets 0 bytes 0 jump ufw6-skip-to-policy-input
udp dport 138 counter packets 0 bytes 0 jump ufw6-skip-to-policy-input
tcp dport 139 counter packets 0 bytes 0 jump ufw6-skip-to-policy-input
tcp dport 445 counter packets 0 bytes 0 jump ufw6-skip-to-policy-input
udp dport 546 counter packets 0 bytes 0 jump ufw6-skip-to-policy-input
udp dport 547 counter packets 0 bytes 0 jump ufw6-skip-to-policy-input
}
chain ufw6-after-output {
}
chain ufw6-after-forward {
}
chain ufw6-after-logging-input {
limit rate 3/minute burst 10 packets counter packets 0 bytes 0 xt target "LOG"
}
chain ufw6-after-logging-output {
}
chain ufw6-after-logging-forward {
limit rate 3/minute burst 10 packets counter packets 0 bytes 0 xt target "LOG"
}
chain ufw6-reject-input {
}
chain ufw6-reject-output {
}
chain ufw6-reject-forward {
}
chain ufw6-track-input {
}
chain ufw6-track-output {
meta l4proto tcp xt match "conntrack" counter packets 0 bytes 0 accept
meta l4proto udp xt match "conntrack" counter packets 0 bytes 0 accept
}
chain ufw6-track-forward {
}
chain INPUT {
type filter hook input priority filter; policy drop;
counter packets 1 bytes 128 jump ufw6-before-logging-input
counter packets 1 bytes 128 jump ufw6-before-input
counter packets 0 bytes 0 jump ufw6-after-input
counter packets 0 bytes 0 jump ufw6-after-logging-input
counter packets 0 bytes 0 jump ufw6-reject-input
counter packets 0 bytes 0 jump ufw6-track-input
}
chain OUTPUT {
type filter hook output priority filter; policy accept;
counter packets 4 bytes 304 jump ufw6-before-logging-output
counter packets 4 bytes 304 jump ufw6-before-output
counter packets 0 bytes 0 jump ufw6-after-output
counter packets 0 bytes 0 jump ufw6-after-logging-output
counter packets 0 bytes 0 jump ufw6-reject-output
counter packets 0 bytes 0 jump ufw6-track-output
}
chain ufw6-logging-deny {
xt match "conntrack" limit rate 3/minute burst 10 packets counter packets 0 bytes 0 return
limit rate 3/minute burst 10 packets counter packets 0 bytes 0 xt target "LOG"
}
chain ufw6-logging-allow {
limit rate 3/minute burst 10 packets counter packets 0 bytes 0 xt target "LOG"
}
chain ufw6-skip-to-policy-input {
counter packets 0 bytes 0 drop
}
chain ufw6-skip-to-policy-output {
counter packets 0 bytes 0 accept
}
chain ufw6-skip-to-policy-forward {
counter packets 0 bytes 0 drop
}
chain ufw6-user-input {
tcp dport 22 counter packets 0 bytes 0 accept
tcp dport 8080 counter packets 0 bytes 0 accept
udp dport 51820 counter packets 0 bytes 0 accept
}
chain ufw6-user-output {
}
chain ufw6-user-forward {
tcp dport 80 counter packets 0 bytes 0 accept
iifname "mesh0" tcp dport 443 counter packets 0 bytes 0 accept
}
chain ufw6-user-logging-input {
}
chain ufw6-user-logging-output {
}
chain ufw6-user-logging-forward {
}
chain ufw6-user-limit {
limit rate 3/minute burst 5 packets counter packets 0 bytes 0 xt target "LOG"
counter packets 0 bytes 0 xt target "REJECT"
}
chain ufw6-user-limit-accept {
counter packets 0 bytes 0 accept
}
}
+37
View File
@@ -0,0 +1,37 @@
$ ufw allow 22/tcp
Rules updated
Rules updated (v6)
$ ufw --force enable
Firewall is active and enabled on system startup
$ ufw route allow 8080/tcp
Rule added
Rule added (v6)
$ ufw route allow proto tcp to any port 8080 comment 'mesh-host adoption.opening-tcp-8080-forwarded 1a2b3c4d'
Rule updated
Rule updated (v6)
$ ufw allow 5671/tcp
Rule added
Rule added (v6)
$ ufw allow proto tcp to any port 5671 comment 'mesh-host adoption.opening-tcp-5671-incoming 1a2b3c4d'
Rule updated
Rule updated (v6)
$ ufw allow in on mesh0 to any port 5432 proto tcp comment 'operator note'
Rule added
Rule added (v6)
$ ufw allow in on mesh0 proto tcp to any port 5432 comment 'mesh-host adoption.opening-tcp-5432-incoming 1a2b3c4d'
Rule updated
Rule updated (v6)
$ ufw show added
Added user rules (see 'ufw status' for running firewall):
ufw allow 22/tcp
ufw route allow 8080/tcp comment 'mesh-host adoption.opening-tcp-8080-forwarded 1a2b3c4d'
ufw allow 5671/tcp comment 'mesh-host adoption.opening-tcp-5671-incoming 1a2b3c4d'
ufw allow in on mesh0 to any port 5432 proto tcp comment 'mesh-host adoption.opening-tcp-5432-incoming 1a2b3c4d'
$ ufw route delete allow 8080/tcp comment 'mesh-host adoption.opening-tcp-8080-forwarded 1a2b3c4d'
Rule deleted
Rule deleted (v6)
$ ufw show added
Added user rules (see 'ufw status' for running firewall):
ufw allow 22/tcp
ufw allow 5671/tcp comment 'mesh-host adoption.opening-tcp-5671-incoming 1a2b3c4d'
ufw allow in on mesh0 to any port 5432 proto tcp comment 'mesh-host adoption.opening-tcp-5432-incoming 1a2b3c4d'
+40
View File
@@ -0,0 +1,40 @@
Rule deleted
Rule deleted (v6)
rc=0
Rule deleted
Rule deleted (v6)
rc=0
ERROR: Invalid syntax
rc=1
===ADDED2
Added user rules (see 'ufw status' for running firewall):
ufw allow 22/tcp
ufw allow 8080/tcp
ufw route allow 80/tcp comment 'mesh-host adoption.opening-tcp-8081-forwarded 0badf00d'
ufw route allow in on mesh0 to any port 443 proto tcp comment 'mesh-host adoption.opening-tcp-8443-forwarded cafe0001'
ufw allow 51820/udp comment 'mesh-host adoption.opening-udp-51820-incoming 11112222'
Firewall reloaded
reload rc=0
===AFTERRELOAD
3
Firewall stopped and disabled on system startup
===DISABLED
Status: inactive
/etc/ufw/user.rules
3
Firewall is active and enabled on system startup
Status: active
Rule deleted
Rule deleted (v6)
rc=0
Rule deleted
Rule deleted (v6)
rc=0
Could not delete non-existent rule
Could not delete non-existent rule (v6)
wrongcomment rc=0
Added user rules (see 'ufw status' for running firewall):
ufw allow 22/tcp
ufw allow 8080/tcp
ufw allow 51820/udp comment 'mesh-host adoption.opening-udp-51820-incoming 11112222'
Status: active
+21
View File
@@ -0,0 +1,21 @@
$ ufw allow in 9005/tcp
Rules updated
Rules updated (v6)
$ ufw deny in 9006/tcp
Rules updated
Rules updated (v6)
$ ufw allow 9006/tcp
Rules updated
Rules updated (v6)
$ ufw allow out 9007/tcp
Rules updated
Rules updated (v6)
$ ufw allow 9007/tcp
Rules updated
Rules updated (v6)
$ ufw show added
Added user rules (see 'ufw status' for running firewall):
ufw allow 9005/tcp
ufw allow 9006/tcp
ufw allow out 9007/tcp
ufw allow 9007/tcp

Some files were not shown because too many files have changed in this diff Show More