mesh-catalog/modules/docker

docker

The container runtime as a module (novox/hq to-be 42 phase 1, item 8; research 027/01–02; ADR 0166, ADR 0207). It claims the node seat node-container-runtime. That seat carries no verbs yet: its verbs, and the host creating containers through its holder, wait on ADR 0166's acceptance. Until then the tools below are the module's own.

What it declares

resource	what	the host's rule
package	docker	installed if absent; never uninstalled when the module goes
buildx	docker-buildx	the same. Only the build machine has it today; docker build needs it for BuildKit everywhere
socket	docker.socket running, enabled at boot	given back as found when the module goes (ADR 0118)
prune-service, prune-timer	/etc/systemd/system/docker-prune.{service,timer}, written whole	removed with the module
prune	docker-prune.timer running, enabled at boot; restarted when either file changes	stopped and disabled with the module (the mesh made the unit)
daemon	live-restore and the mesh's registry in insecure-registries written into /etc/docker/daemon.json, beside other keys	each key given back as found when the module goes; only the member this module added leaves the list
runtime	docker.service running, enabled at boot; reloaded, never restarted, when daemon changes	given back as found (ADR 0118)

The weekly prune takes dangling images and build cache unused for a week, and nothing else. It takes no volume, no container and no image a container uses, so it never touches a container the mesh holds. It runs at idle priority, at a random point in the hour after the weekly mark. A run missed while the machine was off happens at the next boot.

Capabilities: package-manager, service-manager, privileged. It does not declare container-runtime: under ADR 0165, which is still proposed, that word means a running daemon, and the module that installs the daemon cannot require it.

The runtime's own file and service (issue 190, hq ADR 0196, ADR 0222)

This module writes two keys into /etc/docker/daemon.json (into: json, ADR 0102): live-restore and insecure-registries. dnsmasq used to write live-restore, beside dns; it no longer writes either. Under ADR 0196 a container copies its machine's resolvers, so no module writes dns. The controller's private network wrote insecure-registries; under ADR 0222 the controller writes nothing into this file, and this module states the registry itself (the order below).

• daemon: {"live-restore": true, "insecure-registries": ["${seat:mesh-artifact-store:reach}"]}, merged into the file beside the keys others write.
• ${seat:mesh-artifact-store:reach} is where this machine reaches the mesh's artifact store (host:port), filled in by the controller: no binding, no credential, the same address the mesh composes into every image it built. Trusting it in the clear is ADR 0082's decision: every path to it is inside the private network's encryption. While no machine on the network holds the store the answer is empty, and the controller drops the empty member, so the list gets nothing.
• insecure-registries is a list, and the host adds to it rather than replacing it: a machine's own trusted registries stay, and undeclaring takes out only the member this module added.
• runtime: docker.service running, enabled at boot, and reloaded, never restarted, when daemon changes. A restart stops every container. A reload turns live-restore on and takes the trusted registries, and with live-restore on a later restart keeps every container running.

In the apply that moves the key, the host first gives back dnsmasq's resources, then applies this module's: live-restore is set again in the same apply, and the daemon is reloaded once.

dns is removed from the file then, but the daemon reads it only at its next start, and a running container keeps the resolvers it was created with. Each container pinned to a machine's own resolver is restarted before that machine's dnsmasq goes (ADR 0194, step 4).

The order it lands in. The controller that fills ${seat:…:reach} is deployed first: one that does not know the placeholder would send it through unfilled. Then this module. Then the controller stops generating the private network's registry-trust and registry-trust-reload and refuses a generated resource that collides with a module's (issue 190, steps 2 and 5). In the apply that moves the member, the host removes the private network's record first (the member leaves the list) and then applies this module's (it is added back, recorded as this module's); the daemon is reloaded once, for daemon. The address is the same one, so the runtime's trust does not change.

Still elsewhere:

• Nobody writes log rotation. One machine has log-driver and log-opts by hand; they are left as they are until a size is chosen for every machine.

What it does not declare yet, and why

One thing this module should own is still declared elsewhere. The controller refuses two modules on one node that declare the same path, unit, name or package (checkResources, mesh-controller internal/catalogue/resolve.go), so declaring it here would make the module unassignable everywhere. The refusals were checked against the controller's own check:

zsh and docker both declare the name "${machine:account}"

The operator account's membership of the docker group

The right shape is the host's user shape. Its groups are additive: the host runs usermod --append and never takes a group away.

{"id": "group", "type": "user", "name": "${machine:account}", "groups": ["docker"]}

zsh already declares a user resource for the same account (its login shell). The controller compares name across modules, so the two collide.

The change proposed (mesh-controller, checkResources): judge a user resource by the fields it sets, not by its name:

• shell and home stay single-owner;
• groups may be declared by any number of modules, because the host only adds them.

Then this module declares the resource above, and no module has to carry another's group.

Today the operator account is in the group on every machine, by hand. Nothing is lost while it waits.

The bootstrap's runtime

On the machine the mesh was first installed on, the foundation bundle declared package docker (container-runtime) and docker.service running and enabled (container-runtime-running). ADR 0207 §5 exempts them.

• The host records them under their bare ids, with origin carried. A mesh declaration's orphan pass never sees them (mesh-host store.go).
• So docker.package here is a second record of the same package. The apply says "already installed", and neither record ever uninstalls it.
• docker.runtime is likewise a second record of docker.service. Its found state is running, because genesis started it, so undeclaring this module leaves the daemon running.

Tools

The tools run as the operator account. If the daemon's socket refuses that account, a call is asked again through sudo -n (a process keeps the groups it started with). Every call has a 20 s bound. A failure is an error naming how it failed, never an empty answer.

Every container on the machine is in scope. A container the mesh holds carries the host's label mesh-host.id (its value names the assignment), and every answer says mesh_held.

tool		what
docker_list	r	every container: image, state, health, restarts, ports, mounts, compose project, mesh_held; filter by owner, state or name
docker_inspect	r	one container whole, environment values left out (names kept)
docker_logs	r	the last lines of both streams, merged in order, with timestamps (default 200, at most 2000); a secret the container printed is shown as [redacted: <name>]
docker_secrets_in_logs	r	which containers printed a secret they were given, by name, never by value (below)
docker_stats	r	CPU, memory, I/O and process count per running container, heaviest first
docker_start / docker_stop / docker_restart	a	one container. On a mesh-held one, the answer says the host restores its declared state at its next apply
docker_top	r	the processes inside one container
docker_images	r	images, largest first, with the containers using each; dangling, unused or used
docker_prune	a	dangling images and build cache, and stopped containers the mesh does not hold if containers is true. A dry run unless dry_run is false. Never a volume
docker_disk_usage	r	docker system df -v: total, active and reclaimable per kind, with the largest of each
docker_networks	r	networks, subnets, and the containers on each
docker_volumes	r	volumes, who mounts each, whether the mesh holds one of them, anonymous or not, and sizes if asked
docker_events	r	the runtime's events over a window ending now (default 60 min, at most 24 h), without exec noise
docker_daemon_config	r	daemon.json as on disk, docker info's essentials, and keys the daemon has not taken yet
docker_unlabelled	r	the containers the mesh does not hold: the cleanup list
docker_problems	r	unhealthy, restarting, dead, killed for memory, failed, or restarted five times or more
docker_ports	r	every published port, and the containers on the host's network

Secrets in a container's own log (hq issue 268)

Software prints what it is given: a server announcing its password as it starts, a startup script echoing the database URI it connects with. The container's log is then a copy of the secret, held by whoever reads it — this bundle's docker_logs among them. docker_secrets_in_logs reads the last lines of each container's log (the mesh's by default, 5000 lines each, at most 50000) and compares them with:

• the values of the container's environment whose names say they are secrets (PASSWORD, SECRET, TOKEN, API_KEY, …; not a path, a URL, a number or a switch), as given and URL-encoded;
• the password inside any URI its environment holds;
• the shape scheme://user:password@, anywhere in a line, whatever the source — a password a program already masked (***) is not one.

A finding names the container, the module, the assignment and the secret's variable, with how many lines carry it and the first and last time. It never carries the value or the line. A secret delivered only as a mounted file, never in the environment, is not known here — the bundle runs as the operator account, which cannot read the host's 0600 files — and is caught only inside a URI.

docker_logs redacts the same values before it answers, because what it answers is read by agents and kept in their transcripts. Its answer says how many it redacted, and points here.

A finding is a secret to rotate once the program stops printing it; recreating the container drops its old log (the runtime's file goes with the container).

Tests

go test ./...

The tests run against a fake runner and cover:

• escalation through sudo -n on a refused socket, and never as root;
• each failure named by its cause;
• a name or id never read as an option;
• mesh-held marking;
• the environment left out of inspect;
• the restore note on a mesh-held act;
• prune being a dry run by default and never reaching a volume, a mesh container or --volumes;
• the log merge;
• a printed secret found by name and never answered by value, in the scan and in docker_logs;
• size parsing;
• what the daemon has not yet taken;
• event filtering;
• volume ownership;
• that the tools served are exactly the manifest's tools.