novox/hq

Files

T

jschoubben d009c3efef Issue 129: three machines of four trust the mesh, not one

Extended after the first was proven. Every converged machine now holds
the anchor and verifies an internal name with a plain client; before,
the two unassigned ones answered 'unable to get local issuer
certificate' and held no entry for the mesh.

ace is excluded on purpose: it is adopted, so a module assigned there is
held rather than run, which is right and is not trust.

Both the resolution and 0147's insight said one machine of four, which
was true for about twenty minutes.

2026-09-30 02:30:23 +02:00

5.0 KiB

Raw Blame History

129 — resolved: the workstation trusts the mesh, and stops when told to

Done and measured on the machine, 2026-09-30.

What was done

Three steps, not the one the plan expected — the module was merged and had never been registered (see the diagnosis):

Registered ca-trust from the catalogue. No artifact to build: it declares a directory, a script, a unit and a service, and no image.
Assigned it to the workstation the issue was opened on, and pushed.
Verified with a plain client, then unassigned and pushed again to exercise removal, then assigned and pushed once more.

The host's own account of arriving:

created ca-trust.state  (/var/lib/ca-trust)
created ca-trust.anchor (/var/lib/ca-trust/anchor)
created ca-trust.unit   (/etc/systemd/system/mesh-ca-trust.service)
updated ca-trust.trust  (mesh-ca-trust.service): boot disabled to enabled, stopped to running

It works, by the check the report asked for

The report's own reproduction, with no flags and nothing installed by hand:

$ curl -sS -o /dev/null -w '%{http_code}' https://git.novox.internal/
200
$ openssl s_client -connect git.novox.internal:443 -servername git.novox.internal
issuer=O=Mesh Internal CA, CN=Mesh Internal CA Intermediate CA
Verify return code: 0 (ok)
$ trust list | grep -A2 Mesh
    label: Mesh Internal CA Root CA
    trust: anchor
    category: authority

Four internal names, all verifying: git 200, umami 200, keycloak 302, drive 302. Before this, every one of them was curl: (60) … unable to get local issuer certificate (20).

And the consequence the report named specifically: git over HTTPS to the mesh's forge, which it said had forced the working clone URL to be ssh-only.

$ git ls-remote https://git.novox.internal/novox/hq.git HEAD
76fbe323ea3401fcdadbf500c61bd3fa5a0a8603	HEAD

Removal is symmetric, which nothing had ever shown

ADR 0147 says the module anchors the authority and takes it away again. That half had never run. Unassigning and pushing:

removed ca-trust.trust (mesh-ca-trust.service)
removed ca-trust.unit  (/etc/systemd/system/mesh-ca-trust.service)
removed ca-trust.anchor (/var/lib/ca-trust/anchor)
removed ca-trust.state  (/var/lib/ca-trust)

Then: the anchor file gone, trust list naming no authority of the mesh's, and the plain client back to unable to get local issuer certificate (20). A machine that leaves the mesh stops trusting it, as the record claims.

The order is what makes it work, and is worth saying. The service is removed first, so systemd runs the unit's ExecStop — which is what deletes the certificate and refreshes the bundles — while the script it calls still exists. Had the script or the state directory gone first, stopping the unit would have had nothing to run, and the anchor would have been left behind with nothing declaring it. Nothing in the module says this; it is the host's removal order that makes the module's symmetry real.

What this leaves

Three machines of four (extended the same day, after the first was proven). ca-trust is now assigned to every converged machine, and each verifies with a plain client:
```
novox    mesh CA in trust store: 1    https://git.<node>.internal/ -> 200
g14      mesh CA in trust store: 1    https://git.<node>.internal/ -> 200
shanks   mesh CA in trust store: 1    https://git.<node>.internal/ -> 200
```
Before, each of the two that had not been assigned it answered curl: (60) … unable to get local issuer certificate (20) and held no entry for the mesh.

ace is deliberately not among them. It is the adopted machine, still carrying the predecessor's resolver and filter, and it is not converged until the network and module-assignment work is settled. Assigning a module to it would hold rather than run (ADR 0100), which is correct and is not the same as trusting anything.
It arrives per assignment, which is a shape worth questioning. The report's reasoning — that being on the private network is what makes a machine one that speaks to the mesh's names — argues for a fact carried to every machine on the network, the way the roster and the registry trust are. ADR 0147 chose a module assigned per machine, and this record does not reopen it; the cost is that a machine joining the mesh trusts nothing until somebody remembers a second command.
service-manager reports degraded on this machine and the module ran anyway. Worth knowing that the capability gate passes on a degraded service manager, since a module whose whole delivery is a unit is the kind that would be worst served by one.
The predecessor's authority is still in the trust store, beside the mesh's now rather than instead of it. The report notes that it cannot be retired while anything on the machine speaks TLS to a mesh name; that is no longer true here, and retiring it is its own piece of work.

5.0 KiB Raw Blame History