jschoubben
e41eed0852
The certificate is genuine, from Mesh Internal CA, and nothing on the workstation trusts it — verbatim the error the report gives. The public name on the same proxy verifies cleanly, which puts the fault exactly where the report puts it. What is in the way is not an assignment. `ca-trust` is merged in the catalogue and has never been registered with the mesh — 39 of 76 manifests are — so there is no module to assign. It dry-runs clean and needs no artifact built. Two findings from reproducing it, both their own issues: 157 — every routed name is published with an `.internal` alias that nothing serves. The hosts file says keycloak.novox.be.internal; the proxy serves keycloak.novox.internal and refuses the other by name. The first three names I tried came from the hosts file and failed with a TLS alert rather than a verification error, which pointed at a regression that had not happened. 158 — the proxy re-logs all 52 routes every two seconds, 31 times a minute. The one line that explained 157 sat between two of them. Also recorded, because it was nearly filed as a defect and is not one: step-ca publishes roots as /roots.pem, which is PEM, so ca-trust's fetch and its refuse-a-non-certificate guard are both right. Its other endpoint /roots returns JSON that contains the text the guard greps for, so the guard is sound only because of which path is published.
3.9 KiB
129 — diagnosis
2026-09-30, from the workstation the issue was opened on.
Still live, and reproduced exactly
The certificate is genuine, the authority is the mesh's, and nothing on the machine trusts it:
$ openssl s_client -connect keycloak.novox.internal:443 -servername keycloak.novox.internal
subject=CN=keycloak.novox.internal
issuer=O=Mesh Internal CA, CN=Mesh Internal CA Intermediate CA
Verify return code: 20 (unable to get local issuer certificate)
$ curl https://keycloak.novox.internal/
curl: (60) SSL certificate OpenSSL verify result: unable to get local issuer certificate (20)
trust list holds no entry for the mesh. The anchors present are two mkcert development roots and
the predecessor's lab root — the report's account of the trust store is unchanged.
The public name on the same proxy verifies cleanly (CN=keycloak.novox.be, Let's Encrypt, return code
0), which places the fault exactly where the report puts it: not in the proxy, not in the authority,
and not in the certificate.
The name matters, and the report's "every HTTPS name the mesh serves internally" is too broad. The
served internal name is <label>.<node>.internal. The hosts file also carries
<label>.<public-domain>.internal, which nothing serves and which fails differently — that is
issue 157, found while
reproducing this, and it cost the first several minutes of this diagnosis.
The authority serves what the module needs
step-ca is up and healthy, and publishes roots: /roots.pem for both acme-ca and
internal-acme-ca. That endpoint returns PEM:
$ curl -sk https://127.0.0.1:9000/roots.pem
-----BEGIN CERTIFICATE-----
MIIBvzCCAWWgAwIBAgIQYa2CkdJk16JyG/dVy2qoEzAKBggqhkjOPQQDAjA+…
So ${bound:internal-acme-ca:roots} in the ca-trust module composes to a URL that returns a
certificate, and the module's own check — refuse a body that is not one — is checking the right thing.
Worth recording because it was nearly filed as a defect: step-ca also serves /roots, which returns
{"crts":["-----BEGIN CERTIFICATE-----\n…"]}. That body contains the literal text the module greps
for, so had the module used /roots it would have installed JSON into the anchors directory and
reported success. It does not use it. The guard is sound only because the published path is the PEM
one, which is worth knowing before anybody changes either.
What is actually in the way
The module is not registered. The report and the work plan both say it exists and is merged, which
it does — mesh-catalog modules/ca-trust, on main. But the mesh has never been told about it:
$ mesh-controller module list | grep -iE 'ca-trust|step-ca'
step-ca 1 built 67f5f4cf on novox
39 of the catalogue's 76 manifests are registered. ca-trust is one of the 37 that are not, so it
cannot be assigned to anything — "assign it to one machine" has no module to name.
A dry run confirms it registers cleanly and needs no artifact built: it declares a directory, a script, a unit and a service, and no image.
$ mesh-controller build <catalogue> --path modules/ca-trust --dry-run
… the manifest, parsed and validated
So the remaining work is three steps, not one
- Register it — build it from the catalogue, which pins nothing because it has no artifacts.
- Assign it to a machine. The workstation this was observed on is the honest first choice: it is where a person meets the fault, and it is where the check can be made with a plain client.
- Verify
curl https://<label>.<node>.internal/with no flags, andtrust listnaming the mesh.
Then removal, which the module declares and nothing has exercised: unassigning must take the anchor away and refresh the bundles (ADR 0147), and that is the half most likely to be wrong, because it is the half nobody reaches by accident.