A secret comes into being seven ways today: provider credentials, own secrets (54 modules), broker
accounts through a command that is easy to forget, a vault that only records what the controller
mints (6 modules), operator values, licences, and root secrets. The vault was built to end own secrets
and did not; the old path was never retired.
0113 is rewritten as a waterfall. The vault makes every secret and nothing else does. A provider that
needs a secret for a consumer requires it from the vault, declared once in its provision's contract
and expanded per consumer by resolution; the vault delivers it to both holders, each sealed to its own
node, so a provider's code is unchanged. Own secrets, broker passwords, operator values and licence
credentials take the same path. Genesis is not an exception: it raises the vault first and asks it,
so there is one way a secret is made from the first one on. The vault can sit at the bottom because it
requires nothing but a broker account.
One shared mint function in the SDK was considered and rejected: generation becomes uniform but custody
stays spread over every provider's machine, and each SDK language needs its own implementation.
Rotation is asked of the vault and is provider-first: the value goes to the holder that accepts it,
which confirms, before the holder that presents it gets it, so the lockout window shrinks to the
consumer's own restart, and an unconfirmed provider holds the rotation rather than half-doing it. The
host derives which processes to restart or recreate from the requirement a definition reads, so no
definition declares restart-on for a secret. A rotation shows unconfirmed until each consumer restarted
and passed its health check. Issue 103 becomes a prerequisite.
The file is renamed to match what it now decides. 0112 follows.
0113 — the plaintext claim was false under its own mechanism: handing a provider's answer to the
controller puts every secret on the broker and in the controller in the clear. The provider now seals
each secret field itself, to the consumer node's public key the mesh hands it, and the controller
carries sealed fields it cannot open. That is stricter than today, where the controller holds every
minted credential in the clear. Option 3 (plaintext to the controller) is recorded and rejected. The
foundation exception now covers root-secret rotation (0085) and forms like the broker admin's hash, so
no phase claims to remove the broker's bootstrap step. To-be 24 and 13 are named among what it amends.
27 — resolution is consistent with 0110: co-location and the only provider apply only where no seat
delivers the provision, so an unheld seat is refused even with one provider. The secret-field rule now
matches 0086 exactly (a declared env-file, never a container environment value). The seat placeholder
is the controller's, and the one module reading it moves to a host port. Contracts are held by the
controller and written down in phase 1, so they can be checked; every rule has a check. An operator's
secret is still the operator's, with the vault as custodian. Which seats a module holds is listed as
not settled.
0110 — the unheld-seat-with-one-provider case and the one-answer-for-everyone rule have checks; the
claim about moved manifests is corrected. 26 — the table governs and the code catches up, not the
reverse; scope and capacity agree with the glossary; moving a seat is described as it really is today.
0112 — aligned with 27, and lists 0049 and 26 among what it changes.
Issue 118 is renumbered 119: another branch took 118 first. 'Control-plane' is gone from 0110 and 0111.
The design pass. Everything a module needs is a requirement: a name, a contract, and one of four kinds
of provider — a module, the node's host, the mesh, the operator. Installing a module resolves every
requirement or refuses, naming everything missing at once. It retires six mechanisms that grew
separately: provisions through bindings, settings, assigned ports, machine facts, minted secrets and
literals in the definition.
ADR 0113, proposed: a provider makes what it provides, and the mesh carries it back sealed to the
consumer's node. It is the return path ADR 0048 left "to a separate decision", now needed three ways:
data provisions with nothing to answer with, contracts needing a value the controller cannot make, and
a vault that generates nothing. Who a consumer is stays the mesh's (ADR 0049). Genesis is the one
exception. On acceptance it supersedes 0048 and amends 0085.
ADR 0112 is revised from three sources to that single concept.
ADR 0110 is amended for two points raised in review. The vault gets the mesh-vault seat (issue 106).
A seat's holder outranks co-location for a provision it delivers. Writing that down exposed an
inconsistency: mesh-store delivering postgres-database would have sent every database consumer to the
control-node, against to-be 23's node-local stores. So a seat delivers a provision only where the mesh
has one answer for everyone — artifact store, npm registry, git, vault — and mesh-store and mesh-broker
deliver nothing. 23 and 26 follow.
'Control plane' becomes 'controller' in the records written today.
- Secrets follow ADR 0085 as amended: a module's own secret is a provision the controller mints and
the vault records. The previous commit had that backwards. Whether the vault should generate
instead is recorded as an open question, not decided.
- A directory's contract is owner and mode only. The persistence flag was the keep flag ADR 0030
refused; a directory is kept while it holds anything, and disposable data is a named volume (0107).
- An operator's shared data stays an access (ADR 0051), which rejected an operator-owned directory.
Only where its path is written moves to the assignment.
- The records it changes on acceptance are named: 0051, 0091, 0046 (settings keyed by instance),
0084 (a provider is a node and an instance), and the glossary, which gains its new words only
when the record is accepted.
- How it is checked covers every stated rule. Container-side paths are no longer flagged by the
host-path rule, and code fallbacks are covered.
- Provisions are what other modules provide. A seat's occupant is not listed as one, and the vault
is not described as selectable per assignment.
- 'Control plane' becomes 'controller'. The provider count is ten of eleven, not eleven of twelve.
The first draft listed minted secrets under what the mesh generates. ADR 0085 made a module's own
secret — a password, an internal token, an external key it was handed — a secret provision answered
by the vault, like a database by the store. What the mesh still mints is the delivery credential for
each provision a module takes (ADR 0048), the vault's own included.
Issue 118 records what a review of where module code reads its files found: 789 host-path strings
in 70 of the catalogue's 71 definitions, every one a decision the definition makes about a machine.
Mounts are checked (ADR 0091); the same paths retyped as values are not. It records what that has
already allowed — a DNS provider that would provision nobody silently, a contributions file that
names credentials by host path and so forces every provider to mount at the identical path, an SDK
loop that treats an unwritten contributions file as empty without a word, defaults in code that
disagree with their own manifests — and that no module can be assigned to one node twice, because
every identity is keyed by the module's name.
ADR 0112, proposed for review, answers it the way ADR 0038 answered ports: a definition names
variables, and installing it resolves every one or refuses, from three sources — the assignment's
own configuration, provisions the mesh resolves against a contract, and what the mesh generates or
knows. A directory becomes a provision: the module requires one by name with its owner, mode and
persistence, and where it lands is the assignment's. The mesh's own files stop carrying host paths.
An assignment gets an identity of its own, so a module may run twice on one node.
Checking copies for agreement was rejected as checking something that should not exist; rewriting
paths per assignment was rejected as inferring which strings are paths by their shape. Syntax, a
node's default layout, and when a second instance becomes possible are left to the design.
The enumeration behind the first set read manifests in two repositories and missed a claim made
in the control plane's own code: the private-network module it ships claims the-private-network at
node scope. A closed set without it would refuse the control plane's own module. Thirteen claims in
use, naming twelve seats.
Seats have been doing two jobs and neither is written down. The mechanism ADR 0009 introduced is
enforced — a second holder is refused — but any well-formed name becomes a seat by being claimed,
and nothing can say which seats a mesh has or who holds them: holdings are assembled while planning
and discarded. The enumeration done while preparing this missed the control plane's own manifest,
because core modules' manifests live in its repository rather than the catalogue.
0110 closes the set. Each seat has a name, a scope, what occupying it delivers, and the record that
made it one; a claim outside the set is refused. A seat is held by a module assignment, and what the
mesh knows about the holder is what it knows about that assignment — nothing is stored beside it. A
seat may deliver a provision, and then its holder answers for it among several providers: pin, then
the holder, then the only provider, then refused. That keeps 0009's "refused, never guessed": the
seat is the choice made once, mesh-wide, instead of a pin per consumer node. The first set is the
eleven seats already claimed plus 0109's npm-package-registry, so nothing in use is refused.
Two concepts — seats for exclusion, a new word for consumable singulars — was rejected: both mean
"this mesh's one X", and the overview a person wants is one list.
0111 gives the mesh a git seat and makes a build source one of two explicit forms: a repository on
the seat's holder, recorded by its path and cloned from wherever the holder runs at build time; or
an external URL, recorded and cloned exactly as given. Recognising self-hosted sources by matching
URLs against the forge's address was rejected — it fails in the one case it exists for, after the
forge moves. Credentials for private repositories are left undecided and said so.
Design: new to-be 26 (the seats); 23 gains the seat step in resolution; 18's source entry names
the two forms; the glossary's seat and provision entries say where they meet. 0109 is carried from
its own branch so every link here resolves.
The gap is closed in the proxy: policy applies, the four capabilities exist, the table is keyed
by host and path with a total ordering, and the two failure modes that rot quietly are held by
tests — a declaration carrying a credential refused rather than served, an unreadable secret
failing closed.
Resolved rather than left open because the issue reports a gap in the proxy and that gap is
gone. But the record says plainly what it does not yet allow: an operator still cannot move the
affected routes, because that needs the mesh side — a manifest able to declare these values and
the controller minting the secret auth names. Until both exist the capability is reachable only
by writing the routes file by hand. That is the ordinary build-out of a contract this issue's
decision created, and it belongs to to-be 08 rather than here.
The open questions are marked answered and kept rather than deleted, pointing at ADR 0108 —
what was rejected and why is the half worth having, and a section still saying "the fix should
not be written before these are answered" after the fix was written reads as though nobody
looked.
One finding kept in the record: priority was read with the reader for ports, which caps at
65535, and the one real rule this reproduces is declared at 100000. It parsed to zero, so
refusal and path scoping would have shipped looking complete and doing nothing on the only case
that motivated them. A validator borrowed from a neighbouring field is a silent default.
Issue 116 found the mesh's proxy applies nothing to a request — host lookup, forward. Against
what the replaced ingress actually relies on, four capabilities are missing: authentication
(three dependents, each gating an admin surface with no login of its own), refusal scoped to a
path (one, a live incident mitigation), path-scoped routing with priority, and redirect.
Policy goes on the route rather than beside it. A proxy-side settings layer keyed by route name
would keep the grant literally clean, but then "what protects this route" is answered from two
files nothing keeps in step — and a route's protection is part of what a route is.
The set is closed at those four, so a fifth is an amendment and each addition is earned by a
dependent that exists. An open middleware surface was rejected: it recreates what is being
replaced, and narrowing one later is far harder than widening a closed one.
Where policy needs a credential the declaration names a secret and never carries the value,
which keeps the existing secret machinery the only thing holding credentials. Inlining a hash
was rejected as the first credential in a declaration — a precedent easier to set than withdraw.
This re-keys the routing table by host and path with priority, which follows from the decision
rather than being a separate one: two of the four need one host routed more than one way. Equal
priorities must resolve identically every time or the proxy stops being reproducible.
The record says how it is checked, including the negative case that rots quietly — a
declaration carrying a credential value rather than a reference must be refused, so the
rejected option cannot return by accident.
08-connectivity §3 names the record and gains the subsection; issue 116 gains amended-design.
Two things the report got wrong, and one it could not have found the way it looked.
Disclosure first: it carried a real hostname and an absolute node path, in a public
repository. Both are gone; the ingress, the modules and the routes are named by role, as the
rest of 04-ISSUES does.
The count was low. Basic authentication has three dependents in the catalogue, not one — the
key-value store's browser UI, a database web UI, and the ingress's own dashboard. All three
are credential-less admin surfaces whose only gate is a middleware the mesh's proxy lacks.
The earlier version read only the node's dynamic configuration directory, which cannot see
what modules declare as container labels; counting needs both sources, and the report now
says so.
Two gaps were missing entirely. Redirect rules: two live routes canonicalise a www name onto
its apex, they exist only on the node and not in the catalogue, and they fail silently rather
than erroring. And path-scoped routing with priority, which is the one that reorders the
issue: the table maps host to exactly one target, so a host cannot be routed two ways, and
the refusal rule matches a path on a host already routed elsewhere. Authentication and a
source filter would not make it expressible. Path scoping is a prerequisite, not a sibling.
Also corrected: the refusal rule was described as an address-scoped deny. It is an allow-list
holding a single documentation-range address — deny-everyone — so reading it as address-scoped
points at the wrong fix. And its severity was understated: its own header records it as
incident response closing an abused write primitive, which is not "a real exposure" but a live
mitigation.
The open questions now say plainly that they are design questions and the fix should not be
written before they are answered, and one is added: whether a declaration may carry a
credential at all.
The previous version ranked candidates on whether they preserved single sign-on to the
object store's console. That is not a requirement: a "user" of the store is normally an
application, so the requirement is per-application keys scoped to buckets — which the mesh
already mints. And the console login it ranked on never worked; the module's own hook comment
records "policy claim missing", a failing login written up as progress.
It also omitted the incumbent's own maintained fork, which changes the question from "which
product replaces it" into two decisions: repoint, or migrate — and if migrating, to which.
Repointing costs an image reference; migrating costs a data copy, two handler rewrites and a
maintenance window. Repointing does not foreclose migrating, which is the argument for taking
it first.
On the corrected requirement Garage ranks first — its per-key-per-bucket model is the
requirement verbatim, its admin API matches how the mesh provisions, and the highest-risk
consumer is first-party documented against it. Its remaining gap (no versioning, no
server-side encryption, partial lifecycle) is unmeasured against the buckets and is the one
thing that could still disqualify it.
Measured and folded in: 230 GiB logical, 82,496 objects, 468 GiB raw at 2.03x, eight drive
directories on one filesystem on one machine. That last fact decides more than any feature —
the erasure coding is not buying independent-drive redundancy, so the redundancy model is
close to irrelevant and only storage overhead remains, which at this volume is a rounding
error against the headroom.
Both errors are recorded at the end of 01 rather than quietly fixed. A configured feature is
not an observed one; and when a dependency dies, "who took it over" precedes "what replaces
it" — searching for alternatives by construction returns things that are not the incumbent.
The diagnosis carried a table headed "claims that could not be substantiated", denying a
module.json, a digest pin, and an all-zeros runtime digest. All three exist. The table is
withdrawn in full and replaced with what is actually true, plus the two claims that remain
genuinely unverified rather than disproven.
The cause: one repository was searched and absence in it was written up as absence. The
catalogue of the mesh being built is a separate repository, not checked out where the search
ran, and all four claims were about that repository. Compounding it, the predecessor's
object-store module and the one being cut over to were treated as one thing — they are
different files in different repositories, one pinning a tag with no sidecar, the other a
digest with two container resources.
Also corrected in the report: located-in named the wrong repository; the "pins a tag" passage
described the predecessor; the open question about pinning by digest is struck, because this
module already does and it made no difference — a deleted digest resolves to nothing either
way. The section on why nothing broke is now scoped explicitly to the predecessor's
machinery.
The lesson kept in the record: "zero occurrences anywhere in the tree" is only as strong as
the tree searched, and a diagnosis must say which tree. A confident rebuttal of a correct
report is worse than no diagnosis — it sends the next person to the wrong place with a
written record behind them.
A thought raised mid-task needed remembering but not working on, and there was no
mechanism for that — so it was recorded by hand. This is that, made repeatable.
Records to Claude's persistent memory rather than the repository, deliberately. A parked
thought has no number, owner or status: giving it one asserts triage that deferring says
has not happened. A shared "deferred" document would be a central status file, which
AGENTS.md forbids. And a repository write means a branch, a commit and an MR — the drift
the skill exists to prevent.
Wraps no playbook, because deferring precedes the development cycle rather than being part
of it. It does say which playbook a thought would need if it graduates, and that recording
"undetermined" is the honest answer when the evidence does not say.
The stop condition is the substance: at most two lines, then return to what was in
progress. No plan, no triage question, nothing opened.
SeaweedFS was scoped as primary because it looked like the only candidate preserving
OIDC console login. Measured: its admin UI is Apache-2.0 but its identity-provider
integration is not — console SSO sits behind the per-TB commercial licence, alongside
point-in-time recovery and automatic EC repair. The free build gives OIDC on the S3 API
via STS and a console authenticated by local username and password.
So the answer to the gating question is that no candidate preserves the current feature
set for free, which this effort had written down as a possible outcome. Reopened across
three candidates with the requirement-by-requirement evidence in 01.
Two corrections to what the overview recorded. RustFS is not a binary-level drop-in
retaining existing data: API and on-disk compatibility are separate paths and the on-disk
one is preview-scoped. And it carries an open defect in the credential path the bucket
provision depends on, which gates it specifically.
Nothing graduates before two measurements named in 01: whether an authenticating proxy
is an acceptable answer to console SSO, and which S3 endpoints consumers actually call —
the latter because Garage does not implement the full span and cannot be ranked until
that is counted.
Review of my own text found an unattributed claim — "the mesh's claim that a node
can be rebuilt from its declarations" — which is not a stated principle anywhere.
Replaced with the design position that genuinely covers it: to-be 07 chooses
references over payload because "reproducibility comes from pinning the identity of
a thing rather than carrying its bytes". This incident is that choice's failure mode
when the identity stops resolving, which is a sharper point than the one I made.
Scope stated honestly: the passage is about the foundation bundle and this module is
not in it, but pin-identity-fetch-bytes is how every module gets third-party images.
Also names the tension the mirroring question actually carries — mirroring is a move
away from references-over-payload, so it is a decision, not a fix.
The symptom arrived diagnosed as "the registry disabled anonymous pulls for the
whole vendor namespace". It did not hold: sibling repositories in that namespace
pull normally, the "$disabled" token field appears on every repository including
working ones and describes signing rather than access, and "actions": [] with a
401 is byte-identical to what an invented repository name returns. Both registries'
own APIs establish deletion instead.
Recorded because the correction is the expensive part to rediscover, and because
the instance was harmless while the standing condition is not: no node that does
not already hold the images can ever provision the module again, and nothing
detects that until one tries.
Research 015 scopes the replacement. It is not a redesign — the foundation design
already commits to S3 the protocol rather than the product, and the object store
is an ordinary module, so this instantiates an existing principle. The live OIDC
wiring is the requirement that gates the choice, and it is checked first.
jochen