whole-mesh rehearsal beds: the real node sets converge on one substrate #18

2026-09-08T18:06:13Z

jschoubben commented

2026-09-08 18:06:13 +00:00

The whole-mesh rehearsal — standing up the real node→module sets in incus and converging them, which had never been run. Four beds (each loads the committed module.json from mesh-catalog, no hand-written manifests):

whole-mesh-novox — novox's 17 converted services on one node behind the substrate; green.
whole-mesh-ace — ace's 24 services incl. the ADR-0051 media shared-library; green.
whole-mesh-full — both server sets converge together on one substrate (37 per-node broker accounts, isolated overlay/identity/DNS, no cross-node interference); green.

The whole-mesh-full bed also proves the dry-run fixes end to end: fail2ban is now hostable (the firewall capability fix), and each of the 7 credential modules is delivered a secret via the real secret accept operator path and its sidecar advances past its "no credential" crash. Tolerated, documented lab gaps: fail2ban's package can't install on the offline RFC-5737 segment, and the credential sidecars fail app-auth with fake tokens (both expected, not gated). SUITE_EXIT=0.

The whole-mesh rehearsal — standing up the real node→module sets in incus and converging them, which had never been run. Four beds (each loads the committed module.json from mesh-catalog, no hand-written manifests): - `whole-mesh-novox` — novox's 17 converted services on one node behind the substrate; green. - `whole-mesh-ace` — ace's 24 services incl. the ADR-0051 media shared-library; green. - `whole-mesh-full` — both server sets converge together on one substrate (37 per-node broker accounts, isolated overlay/identity/DNS, no cross-node interference); green. The `whole-mesh-full` bed also proves the dry-run fixes end to end: fail2ban is now hostable (the `firewall` capability fix), and each of the 7 credential modules is delivered a secret via the real `secret accept` operator path and its sidecar advances past its "no credential" crash. Tolerated, documented lab gaps: fail2ban's package can't install on the offline RFC-5737 segment, and the credential sidecars fail app-auth with fake tokens (both expected, not gated). SUITE_EXIT=0.

jschoubben added 4 commits 2026-09-08 18:06:13 +00:00

Add whole-mesh novox dry-run bed (stage 1 of whole-mesh rehearsal) 581fd6da77

Install the real novox server's converted service set together on one node
behind the substrate — the whole-catalogue install this rebuild never ran.
The bed loads each committed module.json from mesh-catalog (no hand-written
manifests), rewrites image refs to the scenario registry's digests, and
remaps the co-located host-port collisions (nextcloud/invoicing/route-proxy
:80, minio/invoicing :9000, gitea/umami :3000).

Proven green: the whole set of 17 modules RESOLVES and applies (191
resources); the CORE 13 converge whole — all five providers (postgres,
redis, minio, mongodb, mssql) plus keycloak, gitea, nextcloud and invoicing
reaching their providers and staying up, plus portainer, verdaccio, registry
and route-proxy.

Reported as escalated gaps (do not gate green): fail2ban (declares
capability intrusion-prevention that no host detector provides, and an
unappliable assignment blocks whole-node resolution), umami/photos/mailu
(catalog manifests do not wire the runtime/app env the images need; photos'
server image is an alpine placeholder), and firewall (nftables.service is a
oneshot that exits, but the module declares state running so mesh-host marks
it failed).

Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF

Add whole-mesh ace dry-run bed (stage 2 of whole-mesh rehearsal) 90651e1e73

Install the real ace server's converted service set (24 modules) together on
one node behind the substrate — sibling of the whole-mesh-novox bed, the
media/home-automation half. Loads each committed module.json from
mesh-catalog, rewrites image refs to the scenario registry's digests, remaps
the co-located host-port collisions (qbittorrent/searxng/unifi :8080,
nzbget/unifi :6789), and pre-creates the ADR-0051 operator-owned media
library dirs under /services/media so the media stack's `accesses` resolve.

Proven green: the whole 24-module set RESOLVES and applies (214 resources,
node applied+current) — the ADR-0051 shared-dir `accesses` mechanism works
cleanly across eight co-accessing media modules. The CORE 17 converge whole:
postgres/redis/mssql, sonarr/radarr/lidarr/jackett/tautulli/bookshelf,
mosquitto/influxdb/grafana/baserow/nodered/searxng/unifi/portainer.

Reported as escalated gaps (do not gate green): six tool-runtime sidecars
crash-loop because the committed manifest does not wire the app credential
they need (plex MESH_PLEX_TOKEN, bazarr MESH_BAZARR_API_KEY, nzbget
MESH_NZBGET_URL/PASSWORD, qbittorrent MESH_QBITTORRENT_URL/PASSWORD, ombi
MESH_OMBI_API_KEY, home-assistant MESH_HOMEASSISTANT_TOKEN) — the umami/photos
class from novox; each server is up, only the sidecar is down. sonarr/radarr/
lidarr/jackett/tautulli self-configure from the app's config file and their
runtimes come up. letta's app has a first-boot postgres migration race
(pgvector the deeper blocker, per two-node-db).

Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF

Add whole-mesh full three-node bed (stage 3: both server sets, one substrate) fcdcd338c0

Combine the novox (17-module) and ace (24-module) sets on ONE substrate and
prove both node-plans converge together. anchor runs the substrate only;
novox and ace each run their own self-contained set (own postgres/redis), so
nothing crosses a node boundary except enrolment and the shared broker/store.
The four modules both nodes run (postgres, redis, mssql, portainer) are added
once and assigned to each node, each getting its own per-node broker account.
An overlay is placed across all three nodes.

Proven green: both nodes converge together on the one substrate. ace reaches
applied+current with all 17 of its CORE up (and letta too this run); novox
reaches all 13 CORE up with its only failed resource the known firewall.load
oneshot gap. The two node-plans share one broker without collision — distinct
novox-<mod> and ace-<mod> accounts for the modules both run. No new cross-node
bug (overlay/DNS/identity/port) surfaced; ports are per-VM and the sets are
node-self-contained. Tolerates the same nine credential-sidecar gaps and
firewall's nftables.service oneshot documented in the per-server beds.

Resource envelope: 3 VMs (anchor 4GiB, novox 16GiB, ace 18GiB) + registry
scenery, ~79 union images (~35GB) stocked to one registry VM and pulled
concurrently by both nodes; fit within 125GiB host RAM and the 180GiB lab pool.

Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF

whole-mesh-full: prove the dry-run fixes (fail2ban hostable, credential own-secrets) 591f2a641c

Re-runs the capstone from main after the dry-run fixes merged.

fail2ban: added to the novox set. The capability fix (intrusion-prevention ->
firewall) makes it HOSTABLE — it is now assigned, not refused — which is the
gate. Its service reaching active is a host concern the offline lab cannot meet
(the VM ships nftables but not fail2ban, and the isolated segment has no route to
the package mirror, so pacman cannot fetch it), so fail2ban joins GAPS_NOVOX: its
failed package resource is tolerated like firewall's oneshot nftables.service.

7 credential sidecars: before the push, a FAKE app credential is delivered for
each (plex/bazarr/ombi/home-assistant/nzbget/qbittorrent on ace, umami on novox)
through the real operator path — `secret accept <node> <module> <name> --from`.
The bed asserts each sidecar advances PAST its old "no credential" crash (it reads
the delivered value); app-auth failure against the real app with a bogus value is
expected and not gated.

Result: SUITE_EXIT=0. Both node-plans converge on one substrate (novox 13/13
core, ace 17/17 core), fail2ban hostable, all 7 sidecars past their crash.

Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF

jschoubben merged commit a5f53f524a into main

2026-09-08 18:06:19 +00:00

jschoubben deleted branch feat/whole-mesh

2026-09-08 18:06:19 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: novox/mesh-lab#18