ADR 0098; issue 076 opened and resolved; ADR 0097's base refusal live; ADR 0096 proven against the public hub; 074 down to one bed #69

Merged
jschoubben merged 5 commits from multiple-fixes into main 2026-09-21 20:58:27 +00:00
11 changed files with 234 additions and 8 deletions
@@ -54,7 +54,10 @@ A test raises a fake upstream registry serving an index over two platforms behin
challenge, and a fake mesh registry that records what arrives: every blob of both platforms challenge, and a fake mesh registry that records what arrives: every blob of both platforms
arrives once, two manifests and the index are put under their digests, the reference returned arrives once, two manifests and the index are put under their digests, the reference returned
pins the index under the module's repository, and a second copy uploads nothing. A reference pins the index under the module's repository, and a second copy uploads nothing. A reference
test reads names the way a runtime does. test reads names the way a runtime does. *Proven against the real thing the same day:* the genesis
bed built the tool runtime through the mesh's builder with its node base copied out of the public
hub into the mesh's registry by this code — after one finding the fake could not give: the builder
ran on the default bridge, where loopback is not the machine, and now runs on the host network.
## References ## References
@@ -36,10 +36,11 @@ image the manifest did not declare is refused before the build, naming the image
its own stages, declared arguments and `scratch` are not fetches. An unpinned vendor image is its own stages, declared arguments and `scratch` are not fetches. An unpinned vendor image is
refused: a tag is what somebody else can move. refused: a tag is what somebody else can move.
A recipe whose `FROM` names an undeclared base is **said, not yet refused**: the mesh's own images A recipe whose `FROM` names an undeclared base was at first said, not refused: the mesh's own
— the control plane's, the builder's, the tool runtime's — start from a public base and declare images — the control plane's, the tool runtime's, the route proxy's — started from a public base
none, and refusing those refuses genesis. They declare their bases next; until then every build and declared none, and refusing those refuses genesis. *Amended the same day:* those three declare
names the undeclared base and the remedy. their bases now, and an undeclared `FROM` is refused like an undeclared copy. The builder's own
image and the examples are built by `make`, not by the mesh, and take arguments with defaults.
The package half of the issue is not decided here: the mesh's package registry already proxies The package half of the issue is not decided here: the mesh's package registry already proxies
the public one, and the failure the report saw has to be run again to be placed. the public one, and the failure the report saw has to be run again to be placed.
@@ -0,0 +1,61 @@
---
topic: the tiers
status: accepted
date: 2026-09-21
deciders: jochen
reconstructed: false
extends: 02-DECISIONS/0085-a-secret-is-a-provision.md
---
# 98. A fact a provider makes at first start is fetched from it, not carried in its manifest
## Context
The catalogue's certificate authority declared its root certificate, its root key and that key's
password as its own secrets, and told the container to initialise from them. The mesh mints an
own secret nobody delivers as random bytes, and random bytes are not a certificate: issued, the
authority could not start; only an operator hand-making its root could raise it
([issue 076](../04-ISSUES/076-a-served-fact-made-at-first-start-cannot-be-served/00-report.md)).
The authority can make its own root at first start. What it could not do then was tell the mesh
what that root is: a consumer was given `${bound:acme-ca:root}` from the provider's `serves`,
which is written in the manifest before anything runs.
## Considered Options
1. **A secret the module makes**, with the mesh taking custody once the file exists. Rejected
for now: a node would have to send a value up to the mesh, which no channel does today, and
a root key is the one thing the mesh has no reason to hold.
2. **A served fact the provider contributes at run time.** Rejected for now: the same new
channel, for a fact that is not secret at all.
3. **The consumer fetches it from the provider**, over the mesh network, through a gate before
the thing that needs it starts. Adopted.
## Decision
A provider's `serves` names where a fact made at first start can be fetched — the authority
serves its root at a path beside its ACME directory — and a consumer fetches it in a `run-once`
step declared before the resource that needs it, from the provider's bound address. The mesh
network is where the fetch happens, which is what makes fetching without a prior trust
acceptable: it is the network the mesh itself authenticates. The mesh mints only what it can
make: the authority's password. The root key stays where it was made.
## Consequences
The catalogue's authority starts, and the proxy that requires it trusts what it fetched. What
got harder: a consumer of such a fact carries one more resource, the gate that fetches it, and
a fact that changes after first start is refetched only when the declaration changes.
## How it is checked
The route-forwarding bed installs the authority, the proxy and a consumer from the catalogue and
asserts a routed name is served through the proxy. The proxy refuses to start on a bundle that is
not a certificate, so the name being served proves the gate fetched one; the gate itself refuses
a body that is not a certificate. That the proxy obtains a certificate from this authority through
that root is the certificate bed's proof, against the same authority with the same proxy. The
catalogue-wide manifest test parses both manifests.
## References
- [issue 076](../04-ISSUES/076-a-served-fact-made-at-first-start-cannot-be-served/00-report.md)
- [ADR 0053](0053-a-step-that-runs-on-a-schedule.md), [ADR 0085](0085-a-secret-is-a-provision.md)
- [`03-DESIGN/01-to-be/08-connectivity.md`](../03-DESIGN/01-to-be/08-connectivity.md)
+1
View File
@@ -115,6 +115,7 @@ python3 00-META/checks/index.py fail if stale
- **0092** — [An operator delivers a pair credential, and the mesh never replaces it](0092-an-operator-delivers-a-pair-credential.md) - **0092** — [An operator delivers a pair credential, and the mesh never replaces it](0092-an-operator-delivers-a-pair-credential.md)
- **0094** — [A module may hold several secrets from one provider, each a pair of its own](0094-a-module-may-hold-several-secrets-from-one-provider.md) - **0094** — [A module may hold several secrets from one provider, each a pair of its own](0094-a-module-may-hold-several-secrets-from-one-provider.md)
- **0095** — [The control plane is the way to ask a module](0095-the-control-plane-is-the-way-to-ask-a-module.md) - **0095** — [The control plane is the way to ask a module](0095-the-control-plane-is-the-way-to-ask-a-module.md)
- **0098** — [A fact a provider makes at first start is fetched from it, not carried in its manifest](0098-a-fact-a-provider-makes-at-first-start-is-fetched-from-it.md)
### What runs on them, and how it gets there ### What runs on them, and how it gets there
+14 -1
View File
@@ -7,8 +7,9 @@ code:
- mesh-controller internal/identity/authority.go - mesh-controller internal/identity/authority.go
- mesh-host internal/identity/serving.go - mesh-host internal/identity/serving.go
- mesh-host internal/apply (the service that reflects a rule set) - mesh-host internal/apply (the service that reflects a rule set)
updated: 2026-09-09 updated: 2026-09-21
decisions: decisions:
- 02-DECISIONS/0098-a-fact-a-provider-makes-at-first-start-is-fetched-from-it.md
- 02-DECISIONS/0005-the-node-host.md - 02-DECISIONS/0005-the-node-host.md
- 02-DECISIONS/0004-a-node-and-how-it-joins.md - 02-DECISIONS/0004-a-node-and-how-it-joins.md
- 02-DECISIONS/0007-connectivity.md - 02-DECISIONS/0007-connectivity.md
@@ -557,6 +558,18 @@ fingerprint in its token ([ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-j
so nothing needs the CA before membership. It certifies internal names afterwards, and that is so nothing needs the CA before membership. It certifies internal names afterwards, and that is
all it does. all it does.
**The internal authority makes its own root at first start, and a consumer fetches it**
([ADR 0098](../../02-DECISIONS/0098-a-fact-a-provider-makes-at-first-start-is-fetched-from-it.md)).
The mesh mints the authority's password and nothing else of its: a root certificate and its key
are things only the authority can make, and a served fact written in a manifest cannot carry what
does not exist until the authority has run. So the authority serves its root at a path beside its
ACME directory, and the proxy that requires it fetches that root over the mesh network in a
run-once step before it starts. The step is run once per declaration: a root that changes
after first start is fetched again only when the declaration changes
([issue 077](../../04-ISSUES/077-a-fact-fetched-at-first-start-is-fetched-once/00-report.md)).
*How it is checked:* the route-forwarding bed installs the authority, the proxy and a consumer
from the catalogue and asserts the routed name is served.
### What was built ### What was built
*2026-08-31.* *2026-08-31.*
+2 -2
View File
@@ -221,8 +221,8 @@ and nothing uploaded on a second copy.
entry is a module's artifact or an image published elsewhere, pinned by digest, read from one entry is a module's artifact or an image published elsewhere, pinned by digest, read from one
build argument; the image is copied into the mesh's registry before the build and the recipe is build argument; the image is copied into the mesh's registry before the build and the recipe is
handed the copy. A recipe whose `COPY --from` names a registry image the manifest did not declare handed the copy. A recipe whose `COPY --from` names a registry image the manifest did not declare
is refused before the build, naming it and the remedy; an undeclared `FROM` is said, not yet is refused before the build, naming it and the remedy, and so is an undeclared `FROM`: the mesh's
refused, because the mesh's own images start from a public base and declare none. *How it is own images declare the bases they start from. *How it is
checked:* builder tests on a declared and an unpinned vendor image, and a recipe test on what checked:* builder tests on a declared and an unpinned vendor image, and a recipe test on what
counts as a copy and what as a base. counts as a copy and what as a base.
@@ -27,3 +27,9 @@ sidecar beds proved a sidecar alone, which the module beds prove whole; the mini
grant beds proved a grant with a second store beside the foundation's, which the grant bed and grant beds proved a grant with a second store beside the foundation's, which the grant bed and
the vault bed prove against the catalogue. Retired, with their scenarios. Two remain declared: the vault bed prove against the catalogue. Retired, with their scenarios. Two remain declared:
route-forwarding, which needs the certificate authority beside the proxy, and the large mesh test. route-forwarding, which needs the certificate authority beside the proxy, and the large mesh test.
*Route-forwarding's conversion is blocked:* the catalogue's authority cannot be raised as written
([issue 076](../076-a-served-fact-made-at-first-start-cannot-be-served/00-report.md)).
*Later the same day.* Route-forwarding converted, with the authority beside the proxy
(ADR 0098). One bed remains declared: the large mesh test, with its three fixtures.
@@ -0,0 +1,45 @@
---
status: resolved
opened: 2026-09-21
located-in: [mesh-catalog modules/step-ca, mesh-catalog modules/route-proxy]
fixed-by: ADR 0098; mesh-catalog multiple-fixes (the authority makes its own root and serves it; the proxy fetches it through a gate); proven by the route-forwarding bed
amended-design: 03-DESIGN/01-to-be/08-connectivity.md
---
# A served fact made at first start cannot be served, so the catalogue's authority cannot start
## Symptom, as observed
The catalogue's certificate authority module declares its root certificate, its root key and
that key's password as its own secrets, and writes each into a file the container is told to
initialise from. An own secret nobody delivers is minted by the mesh as random bytes, and random
bytes are not a certificate: issued that way, the authority cannot initialise. It could be
raised by an operator making a root with openssl and delivering all three through `secret
accept` — the whole-mesh bed did exactly that, and has not run since it was converted — but a
module that only starts once a person has hand-made its key material is not a module a mesh
can raise. Found while converting the route-forwarding bed to the catalogue's proxy, which
requires the authority beside it.
The authority can make its own root at first start — the certificate bed raises it that way and
it issues within a second. What it cannot do then is tell the mesh what that root is: a
consumer of `acme-ca` is given `${bound:acme-ca:root}` from the provider's `serves`, which is
written in the manifest before anything runs.
## Why it matters beyond this instance
- **Two kinds of secret the vocabulary does not distinguish.** A value the mesh may invent (a
password) and a value only the module can produce (a key pair, a certificate) are both
"own secrets", and the mesh invents both.
- **A served fact that exists only after first start** has no way into a binding. Anything a
module generates and its consumers must trust — a root, a public key, a fingerprint — is in
the same position.
- Every consumer of `acme-ca`, which today is the route proxy, is blocked with it.
## What would close it
Either a module may say a secret is *made by the module* — the mesh reserves the name, the
module writes the value once, the mesh takes custody of it and delivers it where it is bound —
or a served fact may be *contributed at run time* by the provider's runtime rather than written
in its manifest. The first is the smaller change and covers the root certificate; the second is
what a fingerprint or a public key wants. Decided, then the authority raised in the lab beside
the proxy, which is the route-forwarding bed's conversion.
@@ -0,0 +1,29 @@
# Diagnosis — 2026-09-21
1. The certificate bed already raised the same authority image with no root supplied, and it made
its own root and issued within a second. The manifest's three minted "secrets" were not needed
by the authority; they were needed by the consumer, which was handed the root as a served fact.
2. Of the three ways to get a fact made at first start to a consumer, two need a channel from a
node up to the mesh that does not exist. The third needs nothing new: the provider serves the
fact at a path, and the consumer fetches it over the mesh network in a gate before it starts.
**Located in:** the two manifests. Decided in
[ADR 0098](../../02-DECISIONS/0098-a-fact-a-provider-makes-at-first-start-is-fetched-from-it.md).
**Proven** the same day: the route-forwarding bed raises the authority, the proxy and a consumer
from the catalogue on one node and serves a public name through the proxy. Two things the run
taught, both about the bed rather than the decision:
- The authority certifies itself for the machine's private-network address, which is what a
consumer on any node dials. A machine raised from the foundation bundle has no such address
until it is placed on the overlay, so a bed must place it first — as a hub of one, the way a
real first node is.
- With the overlay's networking and the three modules in **one** push, the proxy's fetch of the
roots timed out at the private-network address; with the overlay converged first and the
modules pushed after, it passes. The order between modules is not the cause: the controller
applies a node's providers before its consumers. The overlay interface and the filter that
admits it were not there yet, and the gate, as first written, tried once with no timeout — a
fetch that hangs holds the node's whole apply. The gate now retries with a timeout and refuses
a body that is not a certificate. A consumer whose first start dials a provider still assumes
the provider's network exists; a fact fetched once per declaration is
[issue 077](../077-a-fact-fetched-at-first-start-is-fetched-once/00-report.md).
@@ -0,0 +1,35 @@
---
status: open
opened: 2026-09-21
located-in: [mesh-host internal/apply (run-once marker), mesh-catalog modules/route-proxy]
---
# 077 — A fact fetched at first start is fetched once per declaration
## Symptom
A consumer fetches a fact its provider made at first start through a run-once step
([ADR 0098](../../02-DECISIONS/0098-a-fact-a-provider-makes-at-first-start-is-fetched-from-it.md)):
the route proxy fetches the certificate authority's root before it starts. The host runs a
run-once step once per declaration digest. When the authority is re-initialised — its state
wiped, or the module moved to another node, where it makes a new root — the proxy's declaration
is unchanged, so the step does not run again. The proxy keeps the old root, refuses the new
authority's certificates, and its own healing path, keyed on the root it holds, never fires.
Observed by reading the apply loop and the proxy, not from an incident. No bed re-keys an
authority.
## Why it matters beyond the instance
Any fact a provider makes at first start has the same shape: the consumer's declaration does not
change when the provider's fact does. A run-once step cannot say "again when the provider
changed", and a restart trigger is not allowed on a run-once step (ADR 0053), so there is no
declarative remedy today.
## What would close it
Either the run-once marker includes something of the provider's — the provider's declaration
digest, or an epoch the mesh raises when a provider is re-issued or moved — or the gate is not
run-once but a validator that runs before every start of the service and is cheap when nothing
changed. Decided, then proven by a bed that re-keys the authority and watches the proxy trust the
new root.
@@ -0,0 +1,32 @@
---
status: open
opened: 2026-09-21
located-in: [mesh-controller internal/inventory (secrets), mesh-controller cmd (secret accept)]
---
# 078 — A delivered secret is accepted under any name
## Symptom
`secret accept <node> <module> <name>` stores a value for a module under a name it does not
check against the module's manifest. A name the manifest no longer declares — an own secret that
became a requirement kept in the vault, or a name that never existed — is stored silently. The
row is dead: nothing reads it, the vault mints a value instead, and the operator believes they
delivered a secret the module is not using.
Found by review, not by a run: the whole-mesh bed delivered four such names after their modules
moved to the several-secrets vocabulary ([ADR 0094](../../02-DECISIONS/0094-a-module-may-hold-several-secrets-from-one-provider.md)),
and nothing said so.
## Why it matters beyond the instance
A silent acceptance is the shape of failure the mesh is built to refuse: an operator's action
that changes nothing and reports success. It hides every stale delivery, in beds and in operation
alike.
## What would close it
Acceptance is refused for a name the module's current manifest does not declare as an own
secret, with the names it does declare in the refusal. A unit test delivers under an undeclared
name and expects the refusal; the whole-mesh bed then fails loudly if a delivery goes stale
again.