The four sidecar beds (grafana, plex, sonarr, redis) proved a sidecar comes up and
serves tools with the module's server cut away; the catalogue beds and the whole-mesh
beds prove the modules whole. The minio and postgres grant beds proved a grant
mechanism with a second store beside the foundation's; the grant bed and the vault bed
prove it against the catalogue. Ten conversions become six deletions (novox/hq
04-ISSUES/074).
A module never built on this workstation had no node_modules, tsc failed on the
first import behind /dev/null, and the suite reported a build that said nothing.
Two files with two values, two holders in the vault's ledger — the identity with the
local name after it — and one rotate moves both (novox/hq ADR 0094).
A module's name is its tool namespace and its broker scope, so a fixture running
the module's runtime cannot carry another name (novox/hq ADR 0093). The grant and
backend-network beds now read the catalogue's redis and install mesh-vault, which
provides the secret it requires; the redis-node scenario stocks the vault's runtime.
The remaining declared copies say why they stand (novox/hq 04-ISSUES/074).
A per-module bed stocks mesh-runtime-<module>:development from the workstation's
image store, built by hand by a script the suite never called; six were two weeks
older than the manifests they served. For the beds named, every runtime their
scenarios stock is compared against the module's source and the tool runtime and
SDK it is built on, and rebuilt where older, missing or uncommitted; a failed
build stops the suite (novox/hq 04-ISSUES/075).
The control plane's manifest comes out of the build the installer runs (novox/hq ADR
0069, 04-ISSUES/072); the catalogue no longer holds a copy, and a bed that demanded
one would stop a genesis that is about to succeed.
Review findings: a sibling-path fallback read a catalogue the receipt never claimed;
a manifest literal naming its version first slipped the fence; the shared loader
thirteen beds install through had no test short of a lab run.
The control plane's manifest is leaving the catalogue (ADR 0069, issue 072); a marker
that named it would make every catalogue-reading bed skip the day it goes.
adopted-store-cross-node, two-node-db and the three whole-mesh beds each carried a
private loader; they drifted. The ace loader never resolved a runtime artifact, so a
module the mesh builds travelled unresolved; whole-mesh-full still asked for
'registry' and 'firewall', which the catalogue names distribution and nftables, and
swallowed the miss as NOT ASSIGNED. One loader now (novox/hq 04-ISSUES/073).
catalogueModule() in the harness reads a module's manifest from the catalogue and
rewrites only what the lab must: the build section goes, each artifact becomes the
image the machine holds, images are pinned, and a bed may declare a host-port remap
or a lab-local address. confluence, gitlab, openai-consumer, audit-logger, ollama,
local-model-consumer, model-usage, mosquitto, anthropic-manager and
anthropic-consumer now install the catalogue's manifest. A unit test refuses any
inline copy naming a catalogue module unless the bed is declared with its reason;
the declared list is the debt (novox/hq 04-ISSUES/073).
A bed installs a catalogue module by reading its manifest from MESH_LAB_CATALOG at
run time, so a receipt naming no catalogue commit cannot say whether a catalogue
change was ever proven. Claimed under either spelling of the variable; not built,
because a manifest is read, not compiled (novox/hq 04-ISSUES/073).
The base ruleset (ADR 0088) admits ssh, the bus and the registry and nothing else until the mesh
derives one, and the mesh derives one only where the filter module is assigned — which genesis
does and these beds did not. Without it the hub's WireGuard port stayed closed, no joined node's
tunnel formed, and every module dialling the anchor by its overlay name timed out fetching the
broker's certificate; the model-usage bed showed it as a login that failed for a role never made.
The vault bed grows into a create-once file and pushes again; the genesis bed
probes the machine from the workstation for the whole install and asserts the
store's port never answers while the bus's does.
Applying the packet filter restarts the container runtime a few seconds
after the installer's last push returns; a command racing that window dies
with 'No such exec instance'. Wait for the node to report applied and
current, and retry that error like the recreate it is.
V5: the template's password is refused by the store, the operator key and the
export sit beside the bundle at 0600, the vault keeps the export, and a person
with the key recovers the superuser off the mesh and opens the store with it.
V2 dials the broker with the administrator password genesis made.
Design 13's three logins, for a secret that had no owner before (novox/hq
ADR 0085): the delivered password authenticates against the real redis, the
one `rotate secret` delivers authenticates, and the one rotated away is
refused. Plus the owner's half: the vault's ledger names the holder and the
fingerprint, notices the rotation, and answers over the mesh by fingerprint,
never by value. Runs the catalogue's own manifests.
The CORE convergence wait hung on a container named 'postgres' that
never exists — the postgres module's server is the adopted-store
container 'mesh-store' (like lavinmq's mesh-broker). Everything else
converged; this was the last phantom-name blocker.
The bed had never resolved, then never converged. Fixed, in order:
- loadManifest maps a runtime container's 060 `artifact` to the stocked
mesh-runtime-<module> image (keyed on the module name), so the push is
no longer refused by built() — and drops the build section.
- Stale identities renamed: registry->distribution, firewall->nftables.
- step-ca added to the set: the web modules hard-require `route`,
route-proxy provides it but requires `acme-ca`, and nothing provided
that — so the whole web stack never resolved. step-ca is the missing CA.
- THE STORE SUPERUSER is delivered via `secret accept` before the push.
postgres raises mesh-store with POSTGRES_PASSWORD=bootstrap, but
`module add` minted a random superuser own-secret that did not match,
so the provisioner could not log in and created NO consumer roles —
every DB consumer (gitea/keycloak/nextcloud/umami/mailu) failed. This
was the real cause behind what looked like per-module gaps; keycloak
and umami converge once it is delivered (ADR 0078, hq phase3).
- mesh() retries through the controller recreating itself during the
057 cascade (No such exec instance), so a real success is not read as
a failed push.
- invoicing dropped (private-registry images the lab cannot pull).
Remaining KNOWN_GAPS are genuine catalog/upstream/resource gaps: minio
(stale Docker Hub digest), mssql (Error 945, memory), mailu (config
env), photos (alpine placeholder), nftables (service).
- The RestartCount==0 assertion cannot discriminate the 058 fix: the
consumer is built after its broker is already up, so patient and
exit-on-unreachable code both connect first-try; and restart-on
recreates reset the count. Downgraded to an honest liveness check and
the comment now points at the mesh-tools unit test as the real proof.
- The trust-failure dump ran base64 -d over declared.json, which is JSON
(not base64), so it always reported 'no trust' — removed; the adjacent
python check that decodes the inner declaration field is kept.
- The 063 comment claimed the vhost is re-listed after the restart; the
code only runs a TCP probe. Comment corrected to what the code proves,
and the docker-proxy-vs-DNAT coupling is noted.
A broker that is reachable only until its conntrack entry drops passes
every test written before it restarts. The bed now restarts mesh-broker
after adoption and asserts the joined node can still reach 5671 — the
forward rule, not a surviving entry, carrying the connection.
The provider's workaround re-push is gone: pushing the consumer's node
must cascade to the provider (issue 057), and the vhost assertion is
what says so. The joined consumer must also show zero container-runtime
restarts (issue 058): a runtime whose broker is not up yet waits for it
in-process, so overlay-after-container ordering produces no churn.
Run 11 failed with node2's daemon.json never written and nothing to say
whether the declaration lacked the trust or never applied. The dump now
answers that, and the push output is printed so a compose that refused
is visible in the run log.
The networking module delivers the registry trust, so the bed pushes both machines after
assigning it and waits for each runtime to actually hold the trust (file present AND the
daemon reloaded) before the first build pushes to anchor.internal:5000.
https://claude.ai/code/session_01D6qtiYU3P9jk3pnAXyAFyx
ready() asked a gone instance for its snapshots before judge could say 'no longer
standing'; a stale warm.json from any earlier scenario made every warm run fail in
milliseconds. The question is now asked only of an instance that still exists.
https://claude.ai/code/session_01D6qtiYU3P9jk3pnAXyAFyx
MESH_LAB_WARM=1 snapshots the post-genesis, both-nodes-enrolled state and restores it in
seconds on later runs — refused, not silently rebuilt, when the commits have moved
(src/warm.ts, the mechanism mesh.test.ts already uses and this session had ignored).
Fresh stays the default.
https://claude.ai/code/session_01D6qtiYU3P9jk3pnAXyAFyx
jschoubben