whole-mesh-novox goes green: the store superuser, the 060 artifact shape, and the missing CA #37

2026-09-20T14:46:43Z

jschoubben commented

2026-09-20 14:46:43 +00:00

The first P0 whole-mesh bed proven: the whole novox service set resolves, installs, and converges on one node in one push, behind the foundation. It had never resolved, then never converged; fixed, in order:

The store superuser is delivered via secret accept before the push. The postgres module raises mesh-store with a fixed POSTGRES_PASSWORD=bootstrap, but module add minted a random superuser own-secret that didn't match — so the provisioner couldn't log in and created no consumer roles, and every DB consumer (gitea, keycloak, nextcloud, umami, mailu) failed. This was the real cause behind what looked like per-module gaps; keycloak and umami converge the moment it's delivered (ADR 0078, hq phase3 deliverSuperuser).
loadManifest maps a runtime container's 060 artifact to the stocked mesh-runtime-<module> image (keyed on the module name), so the push is no longer refused by built(), and drops the build section.
step-ca added to the set — the web modules hard-require route, route-proxy provides it but requires acme-ca, and nothing provided that, so the whole web stack never resolved.
mesh() retries through the controller recreating itself during the 057 cascade (No such exec instance), so a real success isn't read as a failed push.
Stale identities: registry→distribution, firewall→nftables, and the postgres server container is mesh-store (adopted), not postgres.
invoicing dropped (private-registry images the lab can't pull).

Green run: mesh-lab 67b2d38, mesh-host 1f483fc, mesh-controller 630eed8 — CORE (postgres, redis, mongodb, keycloak, gitea, nextcloud, umami, portainer, verdaccio, distribution, step-ca, route-proxy) all converge.

Remaining KNOWN_GAPS are genuine catalog/upstream/resource gaps, reported and non-gating: minio (stale Docker Hub digest), mssql (Error 945, memory), mailu (config env), photos (alpine placeholder), nftables (service).

The first P0 whole-mesh bed proven: the whole novox service set resolves, installs, and converges on one node in one push, behind the foundation. It had never resolved, then never converged; fixed, in order: - **The store superuser is delivered via `secret accept` before the push.** The postgres module raises `mesh-store` with a fixed `POSTGRES_PASSWORD=bootstrap`, but `module add` minted a *random* superuser own-secret that didn't match — so the provisioner couldn't log in and created **no** consumer roles, and every DB consumer (gitea, keycloak, nextcloud, umami, mailu) failed. This was the real cause behind what looked like per-module gaps; keycloak and umami converge the moment it's delivered (ADR 0078, hq phase3 deliverSuperuser). - **`loadManifest` maps a runtime container's 060 `artifact` to the stocked `mesh-runtime-<module>` image** (keyed on the module name), so the push is no longer refused by `built()`, and drops the build section. - **`step-ca` added to the set** — the web modules hard-require `route`, route-proxy provides it but requires `acme-ca`, and nothing provided that, so the whole web stack never resolved. - **`mesh()` retries through the controller recreating itself during the 057 cascade** (`No such exec instance`), so a real success isn't read as a failed push. - **Stale identities**: registry→distribution, firewall→nftables, and the postgres server container is `mesh-store` (adopted), not `postgres`. - **invoicing dropped** (private-registry images the lab can't pull). Green run: mesh-lab 67b2d38, mesh-host 1f483fc, mesh-controller 630eed8 — CORE (postgres, redis, mongodb, keycloak, gitea, nextcloud, umami, portainer, verdaccio, distribution, step-ca, route-proxy) all converge. Remaining KNOWN_GAPS are genuine catalog/upstream/resource gaps, reported and non-gating: minio (stale Docker Hub digest), mssql (Error 945, memory), mailu (config env), photos (alpine placeholder), nftables (service).

jschoubben added 2 commits 2026-09-20 14:46:43 +00:00

whole-mesh-novox goes green: the store superuser, the artifact shape, and the missing CA f94b5d4ab5

The bed had never resolved, then never converged. Fixed, in order:
- loadManifest maps a runtime container's 060 `artifact` to the stocked
  mesh-runtime-<module> image (keyed on the module name), so the push is
  no longer refused by built() — and drops the build section.
- Stale identities renamed: registry->distribution, firewall->nftables.
- step-ca added to the set: the web modules hard-require `route`,
  route-proxy provides it but requires `acme-ca`, and nothing provided
  that — so the whole web stack never resolved. step-ca is the missing CA.
- THE STORE SUPERUSER is delivered via `secret accept` before the push.
  postgres raises mesh-store with POSTGRES_PASSWORD=bootstrap, but
  `module add` minted a random superuser own-secret that did not match,
  so the provisioner could not log in and created NO consumer roles —
  every DB consumer (gitea/keycloak/nextcloud/umami/mailu) failed. This
  was the real cause behind what looked like per-module gaps; keycloak
  and umami converge once it is delivered (ADR 0078, hq phase3).
- mesh() retries through the controller recreating itself during the
  057 cascade (No such exec instance), so a real success is not read as
  a failed push.
- invoicing dropped (private-registry images the lab cannot pull).

Remaining KNOWN_GAPS are genuine catalog/upstream/resource gaps: minio
(stale Docker Hub digest), mssql (Error 945, memory), mailu (config
env), photos (alpine placeholder), nftables (service).

whole-mesh-novox: the postgres server container is mesh-store, not postgres 67b2d38022

The CORE convergence wait hung on a container named 'postgres' that
never exists — the postgres module's server is the adopted-store
container 'mesh-store' (like lavinmq's mesh-broker). Everything else
converged; this was the last phantom-name blocker.

jschoubben closed this pull request

2026-09-20 20:26:15 +00:00

jschoubben deleted branch bed/whole-mesh-novox-artifact-shape

2026-09-20 20:26:15 +00:00

Pull request closed

This pull request cannot be reopened because the branch was deleted.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: novox/mesh-lab#37