Merge pull request 'Multiple fixes: status vocabulary aligned, 065/026/070/007 resolved, 066/069/049/046 located, 064 diagnosed' (#64) from feat/multiple-fixes into main

This commit was merged in pull request #64.
This commit is contained in:
2026-09-21 19:23:34 +02:00
48 changed files with 440 additions and 53 deletions
+5 -5
View File
@@ -12,8 +12,8 @@ What is enforced:
known `status:`. A TO-BE doc names at least one decision (`decisions:`) -- no known `status:`. A TO-BE doc names at least one decision (`decisions:`) -- no
design without a decision -- and once `in-progress` or `implemented` it names design without a decision -- and once `in-progress` or `implemented` it names
its owning code (`code:`) -- no development without a design that says where. its owning code (`code:`) -- no development without a design that says where.
issues a known `status:`; once `located` or `fixed`, `located-in:` names the owner; issues a known `status:`; once `located`, `located-in:` names the owner;
once `fixed` or `resolved`, `fixed-by:` says what fixed it (prose counts -- once `resolved`, `fixed-by:` says what fixed it (prose counts --
"nothing, the capability existed" is an answer). "nothing, the capability existed" is an answer).
research a known `status:`; a `graduated` overview says what it `became:`, and every research a known `status:`; a `graduated` overview says what it `became:`, and every
target it names exists. target it names exists.
@@ -37,7 +37,7 @@ import sys
ROOT = os.path.normpath(os.path.join(os.path.dirname(__file__), "..", "..")) ROOT = os.path.normpath(os.path.join(os.path.dirname(__file__), "..", ".."))
DESIGN_STATUSES = {"proposed", "designed", "in-progress", "implemented", "abandoned"} DESIGN_STATUSES = {"proposed", "designed", "in-progress", "implemented", "abandoned"}
ISSUE_STATUSES = {"open", "diagnosing", "located", "fixed", "resolved"} ISSUE_STATUSES = {"open", "diagnosing", "located", "resolved", "wontfix"}
RESEARCH_STATUSES = {"active", "graduated", "abandoned"} RESEARCH_STATUSES = {"active", "graduated", "abandoned"}
@@ -117,9 +117,9 @@ def main():
status = front.get("status") status = front.get("status")
if status not in ISSUE_STATUSES: if status not in ISSUE_STATUSES:
bad(path, "status %r is not one of %s" % (status, sorted(ISSUE_STATUSES))) bad(path, "status %r is not one of %s" % (status, sorted(ISSUE_STATUSES)))
if status in ("located", "fixed") and not listy(front, "located-in"): if status in ("located", "resolved") and not listy(front, "located-in"):
bad(path, "status %s but located-in is empty" % status) bad(path, "status %s but located-in is empty" % status)
if status in ("fixed", "resolved") and not listy(front, "fixed-by"): if status == "resolved" and not listy(front, "fixed-by"):
bad(path, "status %s but fixed-by says nothing" % status) bad(path, "status %s but fixed-by says nothing" % status)
# ---- research ---------------------------------------------------------------------- # ---- research ----------------------------------------------------------------------
@@ -0,0 +1,65 @@
---
topic: the mesh
status: accepted
date: 2026-09-21
deciders: jochen
reconstructed: false
extends: 02-DECISIONS/0010-delivery.md
---
# 90. A failure that repeats is said to be stuck
## Context
Delivery is a comparison, not a one-shot ([ADR 0010](0010-delivery.md)): a node re-applies the
declaration it holds on a steady interval and reports each time. That is right for a failure that
goes away by itself — the overlay not up yet, a registry briefly unreachable — and it makes a
failure that will never go away look exactly the same. A resource nothing can ever apply is
attempted, fails, is reported, and is attempted again every few minutes, indefinitely; the mesh
keeps one report per machine, replaced, so each attempt arrives as "failed" at a fresh time.
Nothing distinguished "failed once, will succeed when its dependency arrives" from "failed
identically for ever", and nothing escalated the second
([issue 065](../04-ISSUES/065-a-permanently-failing-resource-is-retried-for-ever-with-no-escalation/00-report.md)).
## Considered Options
1. **The host gives up** after some number of attempts. Rejected: the host does not know whether
a failure is permanent — that a registry has not answered three times is not evidence it never
will — and a host that stops trying is a node that must be pushed to again by hand.
2. **A duration** since the failure was first seen. Rejected as the signal: a laptop shut for a
week has had one attempt, and a week is not evidence of anything.
3. **The controller counts identical reports**, and says when there are enough of them. Adopted.
## Decision
The mesh keeps, beside each machine's last report, when the current failure was first reported
and how many reports in a row have said it — the same outcome, the same refusal, the same failed
resources by id. Not by the host's words: an error carrying a duration or a counter would read as
new on every report, and the resource looping on it is exactly what this is for. A report that
says something different starts the count again; a clean apply clears it. **Three identical reports in a row make a machine stuck**: `status` says
so beside the failure, with the count and the time it began, and the machine-readable status
carries the same three facts. The host keeps retrying; being stuck is a statement about the
mesh's knowledge, not an instruction to the machine.
The controller counts rather than the host, because only it sees every node: one stuck machine
and a mesh-wide fault are different situations, and the host cannot tell them apart.
## Consequences
A resource that will never apply is visible from `status` after three reconcile intervals, to
anyone who looks, without being asked for. What got harder: nothing on the machine changes — a
gating failure that stops what follows still stops it, and this only makes the wait visible.
Whether a stuck machine should also be raised as an event, and what a gating failure should do,
stay open in the issue's own questions.
## How it is checked
An inventory test records the same failure three times and asserts the count and the unchanged
start; the same resource failing in other words, and asserts the count went on; a different
failure, and asserts it restarted; a clean apply, and asserts both cleared. The status command's own test asserts a stuck machine is said to be one.
## References
- [issue 065](../04-ISSUES/065-a-permanently-failing-resource-is-retried-for-ever-with-no-escalation/00-report.md)
- [ADR 0010](0010-delivery.md)
- [`03-DESIGN/01-to-be/10-delivery.md`](../03-DESIGN/01-to-be/10-delivery.md)
@@ -0,0 +1,68 @@
---
topic: what runs on it
status: accepted
date: 2026-09-21
deciders: jochen
reconstructed: false
extends: 02-DECISIONS/0051-shared-data-is-the-operators.md
---
# 91. A mount is declared, and there are three things it can be
## Context
A container's bind mount whose source does not exist is created by the container runtime, as
root, with whatever mode it picks. So `owner` and `mode` — which exist so a module can say who
its data belongs to — never reach the directories that hold data, and the rule that keeps a
directory when a module goes away ([ADR 0030](0030-data-outlives-the-mesh-that-declared-it.md))
does not cover them, because the mesh has never heard of them
([issue 026](../04-ISSUES/026-the-data-directories-are-not-declared/00-report.md)). Fourteen
such mounts were declared by hand; a check that every mount is declared was then written and
withdrawn, because it refused the builder: the builder mounts the container runtime's socket,
which is not its data, already exists, and belongs to the machine. Declaring it as the module's
own directory would be a lie the host would act on.
## Considered Options
1. **Declare the socket as a directory anyway.** Rejected: the host would create, own and
protect a path that is the machine's.
2. **A new manifest field** naming machine paths a module may mount. Rejected: the manifest
already says the module needs the container runtime, and a second field would say the same
thing in paths.
3. **Three declarations, one for each kind of path a mount can be.** Adopted.
## Decision
A container may not mount a path the module never declared, and a path is declared in one of
three ways, which are the three things a path can be:
- **the module's own** — a directory or file resource, or where a secret, a grant or a
contribution lands. Created and owned by the mesh for this module, kept when the module goes;
- **the operator's** — an `accesses` entry ([ADR 0051](0051-shared-data-is-the-operators.md)):
pre-existing, shared, granted for use, never owned;
- **the machine's** — a facility a declared capability grants. `container-runtime` grants its
socket. The path exists, the machine owns it, and the capability is the declaration.
A mount under a declared directory is declared. The check runs where the manifest is parsed,
and names the path and the three remedies.
## Consequences
Every directory that holds a module's data is one the mesh created with the module's owner and
mode, and one ADR 0030 protects. What got harder: a manifest borrowed from a compose file no
longer passes on the strength of its volume lines; each must say what kind of path it mounts.
The table of what a capability grants is small and in the catalogue's parser; a new capability
that grants a path adds a row.
## How it is checked
Manifest tests refuse an undeclared mount, accept one under a declared directory, accept one
the module accesses, and accept the runtime's socket with the capability and refuse it without.
A test parses every manifest in the catalogue beside the checkout and fails on any that breaks
the rule, so the catalogue cannot drift back.
## References
- [issue 026](../04-ISSUES/026-the-data-directories-are-not-declared/00-report.md)
- [ADR 0030](0030-data-outlives-the-mesh-that-declared-it.md), [ADR 0051](0051-shared-data-is-the-operators.md)
- [`03-DESIGN/01-to-be/18-building-a-module.md`](../03-DESIGN/01-to-be/18-building-a-module.md)
@@ -0,0 +1,61 @@
---
topic: the tiers
status: accepted
date: 2026-09-21
deciders: jochen
reconstructed: false
extends: 02-DECISIONS/0085-a-secret-is-a-provision.md
---
# 92. An operator delivers a pair credential, and the mesh never replaces it
## Context
[ADR 0085](0085-a-secret-is-a-provision.md) names three species of secret and gives the vault
two of them: a module's own secret, which the mesh mints, and an operator-delivered secret — a
credential for something outside the mesh, which only a person can supply. Under 0085 a secret
from the vault is a pair credential between the consumer and the vault. The controller's one
command that takes a value from a person wrote only a module's own secret, sealed to one node.
Nothing could put a value into a pair, so the third species had no entry
([issue 070](../04-ISSUES/070-an-operator-cannot-deliver-a-pair-credential/00-report.md)), and
an operator's credential could be held only as an own secret — un-audited, un-rotatable, the
gap 0085 opened to close.
## Considered Options
1. **A new verb on the pair.** Rejected: `secret accept` already means "a value a person
supplied, sealed on the way in, plaintext discarded"; a second verb would mean the same.
2. **`secret accept` grows a provider end.** Adopted.
## Decision
`secret accept <node> <module> <name> --provider <node>` seals the supplied value to the
consumer's node, to the provider's node, and to the operator's key when the mesh has one, and
records the pair as `accepted`. Every pair credential now says where it came from: `made` or
`accepted`.
An accepted pair is never replaced by a made one. When a sealing key at either end changes, the
mesh cannot re-seal a value it does not hold, so the read is refused and names the remedy —
accept it again. `rotate` refuses an accepted pair for the same reason: the mesh cannot make its
replacement, and deleting it would have the next read mint one, delivered and reported as
applied while failing to authenticate somewhere else entirely. Rotating an accepted credential
is accepting a new value.
## Consequences
A credential for something outside the mesh lives in the vault's ledger with the others, sealed
to both ends and recoverable by the operator. What got harder: a mesh whose node keys change
cannot heal an accepted pair by itself; a person is asked. That is the honest shape — the value
was never the mesh's to make.
## How it is checked
An inventory test accepts a value into a pair, reads it back twice unchanged with origin
`accepted`, asserts `rotate` refuses it naming the remedy while a made pair still rotates;
another changes a node's key and asserts the read is refused, then accepts again and reads.
## References
- [issue 070](../04-ISSUES/070-an-operator-cannot-deliver-a-pair-credential/00-report.md), [issue 069](../04-ISSUES/069-one-secret-provision-yields-one-value/00-report.md)
- [ADR 0085](0085-a-secret-is-a-provision.md)
- [`03-DESIGN/01-to-be/24-the-secrets-vault.md`](../03-DESIGN/01-to-be/24-the-secrets-vault.md)
+3
View File
@@ -87,6 +87,7 @@ python3 00-META/checks/index.py fail if stale
- **0077** — [The parts are named controller, foundation, node — not control plane, substrate, master](0077-the-controller-and-the-foundation.md) - **0077** — [The parts are named controller, foundation, node — not control plane, substrate, master](0077-the-controller-and-the-foundation.md)
- **0083** — [One push leaves the mesh consistent](0083-one-push-leaves-the-mesh-consistent.md) - **0083** — [One push leaves the mesh consistent](0083-one-push-leaves-the-mesh-consistent.md)
- **0088** — [The foundation filters before anything listens](0088-the-foundation-filters-before-anything-listens.md) - **0088** — [The foundation filters before anything listens](0088-the-foundation-filters-before-anything-listens.md)
- **0090** — [A failure that repeats is said to be stuck](0090-a-failure-that-repeats-is-said-to-be-stuck.md)
### Its tiers, from the bottom up ### Its tiers, from the bottom up
@@ -111,6 +112,7 @@ python3 00-META/checks/index.py fail if stale
- **0075** — [An artifact store is a provision; a package registry is a different one](0075-two-stores-and-which-provides-what.md) - **0075** — [An artifact store is a provision; a package registry is a different one](0075-two-stores-and-which-provides-what.md)
- **0078** — [The store and the broker are ordinary modules](0078-the-store-and-broker-are-modules.md) - **0078** — [The store and the broker are ordinary modules](0078-the-store-and-broker-are-modules.md)
- **0079** — [The foundation seats are named after their servers](0079-the-foundation-seats-are-named-after-their-servers.md) - **0079** — [The foundation seats are named after their servers](0079-the-foundation-seats-are-named-after-their-servers.md)
- **0092** — [An operator delivers a pair credential, and the mesh never replaces it](0092-an-operator-delivers-a-pair-credential.md)
### What runs on them, and how it gets there ### What runs on them, and how it gets there
@@ -140,6 +142,7 @@ python3 00-META/checks/index.py fail if stale
- **0084** — [Which provider serves a consumer, when the mesh runs more than one](0084-which-provider-serves-a-consumer.md) - **0084** — [Which provider serves a consumer, when the mesh runs more than one](0084-which-provider-serves-a-consumer.md)
- **0085** — [A secret is a provision, and the vault is the module that provides it](0085-a-secret-is-a-provision.md) - **0085** — [A secret is a provision, and the vault is the module that provides it](0085-a-secret-is-a-provision.md)
- **0087** — [A seeded file is created once, and what grows in it is not the mesh's](0087-a-seeded-file-is-created-once.md) - **0087** — [A seeded file is created once, and what grows in it is not the mesh's](0087-a-seeded-file-is-created-once.md)
- **0091** — [A mount is declared, and there are three things it can be](0091-a-mount-is-declared-three-ways.md)
### How it is built ### How it is built
+11 -1
View File
@@ -5,8 +5,9 @@ code:
- mesh-controller internal/builder - mesh-controller internal/builder
- mesh-controller cmd/mesh-controller (build, build --behind, push, status) - mesh-controller cmd/mesh-controller (build, build --behind, push, status)
- mesh-controller internal/inventory/builds.go - mesh-controller internal/inventory/builds.go
updated: 2026-08-31 updated: 2026-09-21
decisions: decisions:
- 02-DECISIONS/0090-a-failure-that-repeats-is-said-to-be-stuck.md
- 02-DECISIONS/0082-the-registry-is-reached-by-name-and-trusted-by-the-overlay.md - 02-DECISIONS/0082-the-registry-is-reached-by-name-and-trusted-by-the-overlay.md
- 02-DECISIONS/0010-delivery.md - 02-DECISIONS/0010-delivery.md
- 02-DECISIONS/0009-modules-and-the-graph.md - 02-DECISIONS/0009-modules-and-the-graph.md
@@ -216,3 +217,12 @@ the remedy is the same push:
**`status` says it and `push --behind` acts on it**, and both because the alternative is a flag that **`status` says it and `push --behind` acts on it**, and both because the alternative is a flag that
knows something the person reading the status does not. knows something the person reading the status does not.
**A failure that repeats is said to be stuck**
([ADR 0090](../../02-DECISIONS/0090-a-failure-that-repeats-is-said-to-be-stuck.md)). A machine
re-applies on its interval and reports each time, so a resource nothing can ever apply arrives as
the same failure over and over, at a fresh time each time. The mesh keeps, beside the last report,
when the current failure began and how many reports in a row have said it — the same resources by
id, whatever the words; three make the machine stuck, and `status` says so beside the failure. The host keeps trying — stuck is what the mesh
knows, not what the machine is told. *How it is checked:* an inventory test counts three identical
reports, a different one, and a clean apply; the status test asserts the word appears.
@@ -7,6 +7,7 @@ code:
- mesh-catalog modules/builder - mesh-catalog modules/builder
updated: 2026-09-21 updated: 2026-09-21
decisions: decisions:
- 02-DECISIONS/0091-a-mount-is-declared-three-ways.md
- 02-DECISIONS/0087-a-seeded-file-is-created-once.md - 02-DECISIONS/0087-a-seeded-file-is-created-once.md
- 02-DECISIONS/0040-what-a-module-is.md - 02-DECISIONS/0040-what-a-module-is.md
- 02-DECISIONS/0039-what-the-sdk-holds-and-refuses.md - 02-DECISIONS/0039-what-the-sdk-holds-and-refuses.md
@@ -182,6 +183,16 @@ disagrees with it.
| `computed` | marks a module the controller generates rather than an author writing | | `computed` | marks a module the controller generates rather than an author writing |
| `build.artifacts` | what it produces | | `build.artifacts` | what it produces |
**A container mounts only what the manifest declares**
([ADR 0091](../../02-DECISIONS/0091-a-mount-is-declared-three-ways.md)). A bind mount the module
never declared is created by the container runtime as root, so the module's owner and mode never
reach its data and the rule that keeps data when a module goes away does not cover it. A path is
declared in one of three ways, for the three things a path can be: the module's own (a directory
or file resource, or where a secret, grant or contribution lands), the operator's (an `accesses`
entry), or the machine's (a facility a declared capability grants — `container-runtime` grants its
socket). *How it is checked:* the parser refuses an undeclared mount naming the path and the three
remedies, and a test parses every manifest in the catalogue beside the checkout.
### What it builds ### What it builds
| kind | is | | kind | is |
+7 -1
View File
@@ -4,6 +4,7 @@ status: implemented
code: [mesh-catalog, mesh-controller, mesh-host] code: [mesh-catalog, mesh-controller, mesh-host]
updated: 2026-09-21 updated: 2026-09-21
decisions: decisions:
- 02-DECISIONS/0092-an-operator-delivers-a-pair-credential.md
- 02-DECISIONS/0085-a-secret-is-a-provision.md - 02-DECISIONS/0085-a-secret-is-a-provision.md
- 02-DECISIONS/0031-the-control-plane-authenticates-nobody.md - 02-DECISIONS/0031-the-control-plane-authenticates-nobody.md
- 02-DECISIONS/0048-a-provider-creates-the-credential-the-mesh-minted.md - 02-DECISIONS/0048-a-provider-creates-the-credential-the-mesh-minted.md
@@ -45,7 +46,12 @@ whose job it is to give it one."*
A module that needs a secret for its own use requires a `secret` provision, exactly as it requires A module that needs a secret for its own use requires a `secret` provision, exactly as it requires
a database from the store. The vault generates the value — or takes custody of one an operator a database from the store. The vault generates the value — or takes custody of one an operator
delivered — and the credential belongs to the consumer↔vault pair. Because it is an ordinary pair delivered, through `secret accept … --provider`, which seals it to both ends and records the pair
as accepted ([ADR 0092](../../02-DECISIONS/0092-an-operator-delivers-a-pair-credential.md)) — and
the credential belongs to the consumer↔vault pair. An accepted pair is the one exception to what
follows: the mesh cannot make its replacement, so it is neither remade when a key changes nor
rotated; both are refused aloud, and accepting a new value is the rotation. *How it is checked:*
an inventory test accepts, reads back unchanged, and asserts the two refusals name the remedy. Because it is an ordinary pair
credential, **everything already built for pair credentials applies to it unchanged**: it rotates credential, **everything already built for pair credentials applies to it unchanged**: it rotates
with the one command that discards a credential and delivers both ends together, it is one secret with the one command that discards a credential and delivers both ends together, it is one secret
per holder so rotating one touches nothing else, and *who holds this* is a query rather than an per holder so rotating one touches nothing else, and *who holds this* is a query rather than an
@@ -1,11 +1,12 @@
--- ---
status: diagnosing status: resolved
opened: 2026-08-23 opened: 2026-08-23
located-in: [hal] located-in: [hal (the instance), mesh-host internal/bootstrap (preflight asks the daemon), mesh-controller internal/catalogue (capabilities)]
fixed-by: fixed-by:
- "the instance only: PR #962 — incus hook. Merged, ran on one node in 6s of a 60s budget; pipeline #6832 green in 48s. The class remains open." - "the instance only: PR #962 — incus hook. Merged, ran on one node in 6s of a 60s budget; pipeline #6832 green in 48s. The class remains open."
- "partly, on the mesh: mesh-host 73c010e, 9d8239a — preflight and the profile detectors ask the daemon, not the package (installed-but-broken reports absent). The class question the report asks of hal is not answered" - "partly, on the mesh: mesh-host 73c010e, 9d8239a — preflight and the profile detectors ask the daemon, not the package (installed-but-broken reports absent)"
amended-design: - "the class, on the mesh: a manifest declares capabilities, an action declares verify (ADR 0005), and a module's own assertions are the lab's verdict (design 01). The old arrangement is replaced module by module, so its count is never taken"
amended-design: 03-DESIGN/01-to-be/01-end-to-end-testing.md
--- ---
# 007 — An installed package is not an available capability # 007 — An installed package is not an available capability
@@ -0,0 +1,20 @@
# Diagnosis — 2026-09-21
1. The instance was fixed in the arrangement being replaced, by a hook, and verified by hand; the
report's remaining question was about the class: does a module declare a capability — the
outcome — separately from the package that provides it, and how many other packages sit in the
same state.
2. In the mesh the class is answered twice over, by design rather than by hook. A module's
manifest declares `capabilities` — what the machine must be able to do — and the host's
preflight and profile detectors ask the daemon, not the package, so an installed-but-broken
runtime reports absent. And an action a declaration runs carries `verify`, which the host
treats as the read-back and the idempotency check both: "is it there" is asked of the outcome,
never inferred from the step ([ADR 0005](../../02-DECISIONS/0005-the-node-host.md)). The lab
design goes further and makes a module's own assertions the verdict on every delivery
([design 01](../../03-DESIGN/01-to-be/01-end-to-end-testing.md)).
3. The count of other packages in that state in the old arrangement was never taken, and will not
be: the old arrangement is being replaced module by module, and each module that crosses over
declares what it needs and is proven in the lab.
**Located in:** the arrangement being replaced, for the instance; the mesh, for the class, where
it is answered by `capabilities` on the manifest, `verify` on an action, and the lab's verdict.
@@ -1,5 +1,5 @@
--- ---
status: fixed status: resolved
opened: 2026-08-28 opened: 2026-08-28
located-in: [mesh-lab, mesh-host] located-in: [mesh-lab, mesh-host]
fixed-by: fixed-by:
@@ -1,5 +1,5 @@
--- ---
status: fixed status: resolved
opened: 2026-08-29 opened: 2026-08-29
located-in: [mesh-host, mesh-control] located-in: [mesh-host, mesh-control]
fixed-by: fixed-by:
@@ -1,5 +1,5 @@
--- ---
status: fixed status: resolved
opened: 2026-08-30 opened: 2026-08-30
located-in: [mesh-host] located-in: [mesh-host]
fixed-by: mesh-host — apply attempts every resource and reports every failure fixed-by: mesh-host — apply attempts every resource and reports every failure
@@ -1,5 +1,5 @@
--- ---
status: fixed status: resolved
opened: 2026-09-01 opened: 2026-09-01
located-in: [mesh-control] located-in: [mesh-control]
fixed-by: mesh-control df62bb5 fixed-by: mesh-control df62bb5
@@ -1,5 +1,5 @@
--- ---
status: fixed status: resolved
opened: 2026-09-01 opened: 2026-09-01
located-in: [mesh-control] located-in: [mesh-control]
fixed-by: mesh-control 0af3ea1 fixed-by: mesh-control 0af3ea1
@@ -1,5 +1,5 @@
--- ---
status: fixed status: resolved
opened: 2026-09-01 opened: 2026-09-01
located-in: [mesh-control] located-in: [mesh-control]
fixed-by: mesh-control 122680b fixed-by: mesh-control 122680b
@@ -1,5 +1,5 @@
--- ---
status: fixed status: resolved
opened: 2026-09-01 opened: 2026-09-01
located-in: [mesh-lab] located-in: [mesh-lab]
fixed-by: mesh-lab 3503ad9 fixed-by: mesh-lab 3503ad9
@@ -1,9 +1,9 @@
--- ---
status: located status: resolved
opened: 2026-09-01 opened: 2026-09-01
located-in: [mesh-control] located-in: [mesh-control]
fixed-by: partly — mesh-controller f5b03e1 declares the data directories; the gate refusing a container mount the module never declared (53eb000) was withdrawn in 83c6a2f and nothing replaces it fixed-by: mesh-controller f5b03e1 (the fourteen declared); ADR 0091 and mesh-controller feat/multiple-fixes (the check, back, with the three declarations — the socket by capability, the operator's by accesses)
amended-design: amended-design: 03-DESIGN/01-to-be/18-building-a-module.md
--- ---
# 026 — The data directories are mounted and never declared # 026 — The data directories are mounted and never declared
@@ -0,0 +1,14 @@
# Diagnosis — 2026-09-21
1. The catalogue was read again for mounts no resource declares: twenty-four remained, in two
kinds only. Nine media modules mount the operator's library, and every one of those paths is
already in the module's `accesses` — the vocabulary [ADR 0051](../../02-DECISIONS/0051-shared-data-is-the-operators.md)
gave exactly this. Four modules mount the container runtime's socket, and every one of them
declares the `container-runtime` capability.
2. So the field the report said would have to be invented already exists twice over, and the
check that was withdrawn needed only to read both: an access is a declared path, and a
capability declares the facility it grants.
**Located in:** the catalogue's parser. The check is back, refuses an undeclared mount naming the
path and the three remedies, and a test parses the whole catalogue. Decided in
[ADR 0091](../../02-DECISIONS/0091-a-mount-is-declared-three-ways.md).
@@ -1,5 +1,5 @@
--- ---
status: fixed status: resolved
opened: 2026-09-01 opened: 2026-09-01
located-in: [mesh-control, mesh-host] located-in: [mesh-control, mesh-host]
fixed-by: mesh-control 1f5b70a, 41f7c51; mesh-host b91342a fixed-by: mesh-control 1f5b70a, 41f7c51; mesh-host b91342a
@@ -1,5 +1,5 @@
--- ---
status: fixed status: resolved
opened: 2026-09-01 opened: 2026-09-01
located-in: [mesh-control] located-in: [mesh-control]
fixed-by: mesh-control be62f49; mesh-lab f85dbb0 fixed-by: mesh-control be62f49; mesh-lab f85dbb0
@@ -1,5 +1,5 @@
--- ---
status: fixed status: resolved
opened: 2026-09-01 opened: 2026-09-01
located-in: [mesh-control] located-in: [mesh-control]
fixed-by: mesh-control 38d4e77 fixed-by: mesh-control 38d4e77
@@ -1,7 +1,7 @@
--- ---
status: resolved status: resolved
opened: 2026-09-04 opened: 2026-09-04
located-in: [] located-in: [mesh-host internal/apply (restart-on), mesh-catalog]
fixed-by: mesh-host aa441ba (restart-on); mesh-catalog 550393b (runtime config as a mergeable file + restart-on); proven by mesh-lab runtime-restart-on-config.test.ts fixed-by: mesh-host aa441ba (restart-on); mesh-catalog 550393b (runtime config as a mergeable file + restart-on); proven by mesh-lab runtime-restart-on-config.test.ts
amended-design: amended-design:
--- ---
@@ -1,7 +1,7 @@
--- ---
status: resolved status: resolved
opened: 2026-09-02 opened: 2026-09-02
located-in: [] located-in: [mesh-host internal/apply (file create-once), mesh-host examples]
fixed-by: ADR 0087; mesh-host #16 (a file resource may say create-once; kept, never corrected); proven by the apply tests and the vault bed fixed-by: ADR 0087; mesh-host #16 (a file resource may say create-once; kept, never corrected); proven by the apply tests and the vault bed
amended-design: 03-DESIGN/01-to-be/18-building-a-module.md amended-design: 03-DESIGN/01-to-be/18-building-a-module.md
--- ---
@@ -1,7 +1,7 @@
--- ---
status: resolved status: resolved
opened: 2026-09-10 opened: 2026-09-10
located-in: [] located-in: [mesh-host cmd/mesh-bootstrap, mesh-lab test/integration/genesis.ts]
fixed-by: mesh-host cmd/mesh-bootstrap (ADR 0067); mesh-lab genesis.ts is the one description of genesis both beds call. Still open: a running mesh cannot say it was raised by the installer fixed-by: mesh-host cmd/mesh-bootstrap (ADR 0067); mesh-lab genesis.ts is the one description of genesis both beds call. Still open: a running mesh cannot say it was raised by the installer
amended-design: amended-design:
--- ---
@@ -2,7 +2,7 @@
status: resolved status: resolved
opened: 2026-09-12 opened: 2026-09-12
resolved: 2026-09-12 resolved: 2026-09-12
located-in: [] located-in: [mesh-controller (the verb existed)]
fixed-by: nothing — the capability already existed and the wrong verb was used fixed-by: nothing — the capability already existed and the wrong verb was used
amended-design: amended-design:
--- ---
@@ -1,7 +1,7 @@
--- ---
status: resolved status: resolved
opened: 2026-09-13 opened: 2026-09-13
located-in: [] located-in: [mesh-host internal/apply (container spec digests)]
fixed-by: mesh-host e7f94e0 (restart-on digests folded into the container spec, a standing comparison); unit-tested in apply_test.go, no lab assertion yet fixed-by: mesh-host e7f94e0 (restart-on digests folded into the container spec, a standing comparison); unit-tested in apply_test.go, no lab assertion yet
amended-design: amended-design:
--- ---
@@ -1,7 +1,7 @@
--- ---
status: open status: located
opened: 2026-09-14 opened: 2026-09-14
located-in: [] located-in: [mesh-controller internal/builder (upstream artifacts)]
fixed-by: fixed-by:
amended-design: amended-design:
--- ---
@@ -0,0 +1,17 @@
# Diagnosis — 2026-09-21
1. What was established stands: the failure is in going through a machine's image store, whose
newer store keeps an index and refuses to push a single platform out of it. Every variant of
pull-then-push was tried and failed the same way.
2. The first open question answers the other two. A copy between registries never needs a
platform, because it moves what is there — the index and every manifest it names, or one
manifest if the module says so. The registry API is enough for it: read the index, read each
manifest, mount or upload each blob by digest, put the manifests and then the index under the
module's repository. No image store is involved and the runtime's behaviour stops mattering.
3. Whether the mesh mirrors an index or a platform is then a choice the copy can offer rather
than a limitation; today every machine on one mesh is the same architecture, and that
assumption is now written down here rather than nowhere.
**Located in:** the builder's upstream-artifact step. Not fixed here: a registry-to-registry copy
is a few hundred lines against the registry API and is proven only against a real registry
serving a real index, which is a lab run of its own.
@@ -1,7 +1,7 @@
--- ---
status: resolved status: resolved
opened: 2026-09-14 opened: 2026-09-14
located-in: [] located-in: [mesh-controller (the derived filter)]
fixed-by: mesh-controller c8d8211 (forward chain drops by default, never closes ssh), 25e42b3; proven by one-node-mesh.test.ts (a machine filters exactly what its modules declared) fixed-by: mesh-controller c8d8211 (forward chain drops by default, never closes ssh), 25e42b3; proven by one-node-mesh.test.ts (a machine filters exactly what its modules declared)
amended-design: amended-design:
--- ---
@@ -1,7 +1,7 @@
--- ---
status: open status: located
opened: 2026-09-14 opened: 2026-09-14
located-in: [] located-in: [mesh-controller cmd/mesh-controller, mesh-tools (the request contract)]
fixed-by: fixed-by:
amended-design: amended-design:
--- ---
@@ -0,0 +1,18 @@
# Diagnosis — 2026-09-21
1. The refusal is the scope working as designed: a module's broker account covers what it emits
and consumes and nothing else. A tool call needs a reply queue the caller creates and a publish
to the serving module's request queue, and no declared scope grants either.
2. Of the three callers the report names, two already have an account with the right shape. The
controller holds an admin connection to the broker, and an agent acting for an operator runs
through the controller. Another module is the only caller that would need something minted.
3. The honest first step is therefore the third option in the report: the controller is the way
in. A `mesh-controller ask <module> <tool>` needs no new account, routes every question
through one process — which is where an audit of who asked what belongs anyway — and settles
what a module declares about being asked by declaring nothing: serving a tool is being
askable through the controller. A module-to-module call, if one is ever wanted, is a grant
like any other, and a later decision.
**Located in:** the controller (a command speaking the tool request/reply over its own connection)
and the tool runtime's request contract. Not fixed here: it is a new command against a protocol
the runtime owns, and needs a lab run against a tools-only bed to be proven.
@@ -1,7 +1,7 @@
--- ---
status: resolved status: resolved
opened: 2026-09-14 opened: 2026-09-14
located-in: [] located-in: [mesh-controller (the catalogue), mesh-catalog]
fixed-by: mesh-controller 3ae7b88, 2b82872; mesh-catalog d2dce34; proven by one-node-mesh.test.ts (the catalogue holds every module the control plane built) fixed-by: mesh-controller 3ae7b88, 2b82872; mesh-catalog d2dce34; proven by one-node-mesh.test.ts (the catalogue holds every module the control plane built)
amended-design: amended-design:
--- ---
@@ -1,5 +1,5 @@
--- ---
status: fixed status: resolved
opened: 2026-09-14 opened: 2026-09-14
located-in: [mesh-host, mesh-catalog] located-in: [mesh-host, mesh-catalog]
fixed-by: mesh-host 56124c3; mesh-catalog 5e4dc37; mesh-lab 440e265 fixed-by: mesh-host 56124c3; mesh-catalog 5e4dc37; mesh-lab 440e265
@@ -1,7 +1,7 @@
--- ---
status: resolved status: resolved
opened: 2026-09-14 opened: 2026-09-14
located-in: [] located-in: [mesh-controller (the derived filter)]
fixed-by: mesh-controller dda001d (the firewall opens the port the mesh itself runs on), 25e42b3; TestTheBrokersPortIsOpenedThoughNoModuleDeclaresIt fixed-by: mesh-controller dda001d (the firewall opens the port the mesh itself runs on), 25e42b3; TestTheBrokersPortIsOpenedThoughNoModuleDeclaresIt
amended-design: amended-design:
--- ---
@@ -1,7 +1,7 @@
--- ---
status: resolved status: resolved
opened: 2026-09-15 opened: 2026-09-15
located-in: [] located-in: [mesh-tools, mesh-host, mesh-sdk]
fixed-by: mesh-tools b618057 (the SDK resolved by version from the registry, one pin); mesh-host 7986306; mesh-controller 4b9bc50. Caveat: the base image still builds with npm install and no lock, so the build is not reproducible fixed-by: mesh-tools b618057 (the SDK resolved by version from the registry, one pin); mesh-host 7986306; mesh-controller 4b9bc50. Caveat: the base image still builds with npm install and no lock, so the build is not reproducible
amended-design: amended-design:
--- ---
@@ -1,7 +1,7 @@
--- ---
status: resolved status: resolved
opened: 2026-09-16 opened: 2026-09-16
located-in: [] located-in: [mesh-host examples/foundation-first-node.lock, mesh-host internal/bundle]
fixed-by: ADR 0088; mesh-host #16 (the foundation bundle installs nftables and loads a base ruleset before the store); proven by the bundle test and the genesis bed probing from outside during the install fixed-by: ADR 0088; mesh-host #16 (the foundation bundle installs nftables and loads a base ruleset before the store); proven by the bundle test and the genesis bed probing from outside during the install
amended-design: 03-DESIGN/01-to-be/07-the-foundation.md amended-design: 03-DESIGN/01-to-be/07-the-foundation.md
--- ---
@@ -0,0 +1,20 @@
# Diagnosis — 2026-09-21
1. The package half was read against what the mesh now runs. The mesh's package registry has
proxied the public one for every package since early September, before this issue was opened;
and the builder's `.npmrc` names the mesh's registry for the mesh's own scope only, so a public
package is asked of the public registry directly. Either the failing build ran before the
proxy, or its Dockerfile set the registry itself, or the build machine could not reach the
public registry at that moment. The report does not say which, and the 404 cannot be placed
without running the build again. Not fixed; not reproduced either.
2. The image half has a precedent. The lab hit the same refusal pulling a vendor tool and found
the cause was the public hub denying anonymous pulls, not the mesh's redirect; the fix was to
pin the same image from another registry. A `COPY --from` of a public image is a build input
the manifest does not declare, so the mesh cannot pre-fetch or pin it the way it pins bases —
which is the second open question, and the one a decision has to settle.
3. Ruled out: that the build environment has no internet. Package installs from the system's
repositories succeed in the same Dockerfiles.
**Located in:** the builder (what it tells npm and the runtime) and the catalogue (three modules
whose Dockerfiles fetch what no manifest names). Still open: a build must be re-run to place the
404, and a decision is needed on whether a vendor image is declared as a build input.
@@ -1,9 +1,9 @@
--- ---
status: open status: resolved
opened: 2026-09-20 opened: 2026-09-20
located-in: [] located-in: [mesh-controller internal/inventory (node_report), mesh-controller cmd/mesh-controller (status)]
fixed-by: fixed-by: ADR 0090; mesh-controller feat/multiple-fixes (the report counts identical failures; status says stuck after three)
amended-design: amended-design: 03-DESIGN/01-to-be/10-delivery.md
--- ---
# 065 — A permanently failing resource is retried for ever with no escalation # 065 — A permanently failing resource is retried for ever with no escalation
@@ -0,0 +1,17 @@
# Diagnosis — 2026-09-21
1. The controller keeps one report per machine, replaced on every report, by design: the question
is the machine's current state and a history would bury it. A host reports after every apply
and applies on its reconcile interval, so a permanent failure is a row that says "failed" at a
fresh time every few minutes, indistinguishable from a failure that just happened.
2. The four open questions, answered in turn. Escalate: yes, as a word in `status`, which is
where "what is wrong" is already read. The signal: consecutive identical reports, not a
duration — a machine shut for a week has had one attempt. Where: the controller, which sees
every node and can tell one stuck machine from a mesh-wide fault; the host does not know
whether a failure is permanent and must keep trying. A gating failure: unchanged by this, and
left open in the report.
**Located in:** the controller's node report and `status`. The fix keeps, beside the last report,
when the current failure began and how many reports in a row have said it; three make the machine
stuck, said in `status` and in its JSON. Decided in
[ADR 0090](../../02-DECISIONS/0090-a-failure-that-repeats-is-said-to-be-stuck.md).
@@ -1,7 +1,7 @@
--- ---
status: open status: located
opened: 2026-09-20 opened: 2026-09-20
located-in: [] located-in: [mesh-host internal/apply, mesh-controller internal/inventory (node_report)]
fixed-by: fixed-by:
amended-design: amended-design:
--- ---
@@ -0,0 +1,20 @@
# Diagnosis — 2026-09-21
1. The mixed state is already a first-class, visible one. The controller keeps three outcomes for
a machine's last report — applied, failed, refused — and defines `failed` as "some of it: the
machine is in a state nobody declared", with the failed resources and how many did apply beside
it. `status` lists such a machine as not doing what it was told. So the second open question
is answered as it stands: "partway through a change" is distinct from "converged" and from
"refused", and has been since the report was kept.
2. What is not visible is duration, and that is [issue 065](../065-a-permanently-failing-resource-is-retried-for-ever-with-no-escalation/00-report.md),
resolved alongside: a machine that stays in the mixed state now reads as stuck.
3. The first and third questions — pairings whose half-state is harmful, and whether `restart-on`
is the seed of a grouping — are a design decision the record does not yet contain. `restart-on`
couples a service to files within one apply but does not withhold either when the other fails.
No incident has produced a harmful pair; the report was written from reading the loop. A
grouping primitive without a case that needs it would be a rule enforced against nothing.
**Located in:** mesh-host `internal/apply` (the loop) and the controller's report. Left open for
the grouping question alone; it closes when a coupled pair that must not be half-applied is
found in a module, and the declaration gains a way to say so — or when enough modules have run
that the absence is evidence. The visibility half is done.
@@ -1,7 +1,7 @@
--- ---
status: resolved status: resolved
opened: 2026-09-20 opened: 2026-09-20
located-in: [] located-in: [mesh-controller, hq 03-DESIGN/01-to-be/23-choosing-a-provider.md]
fixed-by: graduated — hq fc4ab37 (ADR 0084, to-be design 23) fixed-by: graduated — hq fc4ab37 (ADR 0084, to-be design 23)
amended-design: 03-DESIGN/01-to-be/23-choosing-a-provider.md amended-design: 03-DESIGN/01-to-be/23-choosing-a-provider.md
--- ---
@@ -1,7 +1,7 @@
--- ---
status: resolved status: resolved
opened: 2026-09-20 opened: 2026-09-20
located-in: [] located-in: [hq 03-DESIGN/01-to-be/24-the-secrets-vault.md]
fixed-by: graduated — hq fc4ab37 (ADR 0085, to-be design 24) fixed-by: graduated — hq fc4ab37 (ADR 0085, to-be design 24)
amended-design: 03-DESIGN/01-to-be/24-the-secrets-vault.md amended-design: 03-DESIGN/01-to-be/24-the-secrets-vault.md
--- ---
@@ -1,7 +1,7 @@
--- ---
status: open status: located
opened: 2026-09-20 opened: 2026-09-20
located-in: [] located-in: [mesh-controller internal/catalogue (requires/secrets), mesh-controller internal/inventory (secret key)]
fixed-by: fixed-by:
amended-design: amended-design:
--- ---
@@ -0,0 +1,23 @@
# Diagnosis — 2026-09-21
1. The constraint is real and where the report put it: a module requires a provision name once,
a pair credential is keyed on (name, consumer node, consumer module, provider), and the login
the vault derives is keyed on the (node, module) pair. Two secrets for one module and one vault
collide on every key.
2. The third question was answered by reading the catalogue as it is today: nine modules hold two
or more own secrets besides their broker account. Two hold a relay user beside a relay
password — the user is a name, not a secret, and could travel as configuration. The other
seven hold genuinely independent values with independent lifetimes: an internal token beside
an admin password; a source, an admin and a relay password; an admin password beside an API
token; a root certificate, its key, that key's password and an intermediate's. None of those
derives from another. So "one value per module, derivation the module's business" is not an
answer for most of them.
3. So the answer is neither option as the report framed them. A module keeps one `secret`
provision per independent value, and needs a way to require the same provision name more than
once under distinct local names — the way `secrets:` already maps a provision to a path. That
is a manifest-vocabulary decision, and it moves the pair key: the credential must be keyed on
the local name, not the provision name, or the second pair overwrites the first.
**Located in:** the manifest's `requires`/`secrets` vocabulary (the catalogue parser) and the pair
credential's key (the controller's secret store). Not fixed here: the key change touches every
existing pair and belongs in a feature of its own, with a lab run against the vault bed.
@@ -1,9 +1,9 @@
--- ---
status: open status: resolved
opened: 2026-09-20 opened: 2026-09-20
located-in: [mesh-controller] located-in: [mesh-controller internal/inventory (secret), mesh-controller cmd/mesh-controller (secret accept)]
fixed-by: fixed-by: ADR 0092; mesh-controller feat/multiple-fixes (secret accept --provider; origin on the pair; remake and rotate refused)
amended-design: amended-design: 03-DESIGN/01-to-be/24-the-secrets-vault.md
--- ---
# An operator cannot deliver a pair credential, so the vault's third species has no entry # An operator cannot deliver a pair credential, so the vault's third species has no entry
@@ -0,0 +1,13 @@
# Diagnosis — 2026-09-21
1. The three open questions, answered. It is `secret accept` growing a provider end, not a new
verb: the verb already means a value a person supplied, sealed on the way in. An accepted pair
refuses `rotate` — the mesh cannot make the replacement — and accepting a new value is the
rotation. The origin becomes a fact of every pair credential, `made` or `accepted`, so the
vault's ledger can say which a person supplied.
2. One consequence the report did not name: a pair credential is remade whenever either end's
sealing key changes, and an accepted one cannot be — the mesh does not hold the value. The
read is refused aloud with the remedy rather than quietly replaced by a minted one.
**Located in:** the controller's pair-credential store and the `secret accept` command. Decided
in [ADR 0092](../../02-DECISIONS/0092-an-operator-delivers-a-pair-credential.md).