From 0bc4b7774fe3a2c928de5ed1cddbe60ed282d094 Mon Sep 17 00:00:00 2001 From: jochen Date: Mon, 31 Aug 2026 02:56:50 +0200 Subject: [PATCH] Record what four more pieces of the mesh became Rotation and the provisioner contract; model access as a provision answered by a record, with ADR 0024's other two gaps left as gaps; exposure, which closes the open question about revoking a route; and the delivery loop, which closes the gap ADR 0010 left when it replaced a pipeline with a comparison. --- 03-DESIGN/01-to-be/08-connectivity.md | 38 +++++++- 03-DESIGN/01-to-be/12-a-module-repository.md | 30 ++++++ .../13-credentials-and-their-rotation.md | 95 ++++++++++++++++++ 03-DESIGN/01-to-be/14-model-access.md | 97 +++++++++++++++++++ 4 files changed, 258 insertions(+), 2 deletions(-) create mode 100644 03-DESIGN/01-to-be/13-credentials-and-their-rotation.md create mode 100644 03-DESIGN/01-to-be/14-model-access.md diff --git a/03-DESIGN/01-to-be/08-connectivity.md b/03-DESIGN/01-to-be/08-connectivity.md index 5404819..9cb4d9c 100644 --- a/03-DESIGN/01-to-be/08-connectivity.md +++ b/03-DESIGN/01-to-be/08-connectivity.md @@ -3,6 +3,7 @@ layer: to-be status: designed code: - mesh-control internal/catalogue/filtering.go + - mesh-control examples/route-proxy - mesh-control internal/identity/authority.go - mesh-host internal/identity/serving.go - mesh-host internal/apply (the service that reflects a rule set) @@ -263,6 +264,38 @@ name rather than supplying nothing and receiving credentials. **A workload on an unreachable node is proxied by a reachable one, across the overlay.** Which is the case is a mesh-level fact, which is the fourth reason exposure is control-plane work. +### What was built + +*2026-08-31.* Nothing new in the vocabulary, which was the claim and is now the fact: a route is a +provision, a proxy provides it, and a module that must be reachable requires it. The consumer +contributes the name it wants and the port it listens on; the proxy receives every consumer that +asked; the consumer is told what the provider serves, which is how it knows its own name. + +**One field was missing, and it is the one anything reaching back needs.** A contribution now +carries **where the mesh says that machine is**. A database is reached *by* its consumer, so the +mesh never had to tell a provider where anybody was; a proxy is the other direction — it is told +to send traffic to a consumer and has to open a connection. Without it every provider implementing +a provision would have to know how the mesh names machines, which is a convention leaking into +every module. + +**Exposure and filtering are different questions and a module answers both.** A workload says what +it listens on and who may reach it; separately, it says it wants a route. A module that asked for a +route and not for the port is unreachable by the proxy it just asked for — which the lab +demonstrates, because the machine is already filtering by the time this runs. + +**Withdrawal, which was open above.** The file the proxy is given is the whole truth about who has +a route, so a proxy replaces its table rather than merging. Merging would keep serving a name whose +module was unassigned — and *a stale public name pointing at nothing fails more visibly than a +stale grant* is the reason it must not survive, not a reason to tolerate it. + +**A name a proxy does not serve is refused by saying which it does.** A route that was withdrawn +and a name that never existed are different things, and a bare 404 makes an operator go and read +the mesh to tell them apart. + +*Checked in the lab by a request to the name reaching the workload across the private network and +returning the workload's own answer, then by unassigning the module and requiring the same request +to stop working.* + ## 4 — Filtering **Derived from what is assigned here, and from the overlay's shape** — a node's open ports are a @@ -428,8 +461,9 @@ The list is worth having in one place, because it is most of the argument: - **Renumbering the overlay.** Made *possible* by declaring the hub rather than inferring it from an address, but no procedure exists, and a graph delivered node by node has an ordering problem while it is half-applied. -- **Revoking a route** when a module is unassigned ([ADR 0007](../../02-DECISIONS/0007-connectivity.md)). - A stale public name pointing at nothing fails more visibly than a stale grant. +- ~~**Revoking a route** when a module is unassigned.~~ **Resolved** 2026-08-31 — see §3. The file + a proxy is given is the whole truth about who has a route, so a route does not outlive the module + that asked for it. - **IPv6.** [ADR 0007](../../02-DECISIONS/0007-connectivity.md) makes it expressible; nothing here says the overlay or the resolver handle it. - **Reporting declared-versus-observed.** ADR 0007 makes the disagreement detectable and does not diff --git a/03-DESIGN/01-to-be/12-a-module-repository.md b/03-DESIGN/01-to-be/12-a-module-repository.md index 2b4d799..e75ea79 100644 --- a/03-DESIGN/01-to-be/12-a-module-repository.md +++ b/03-DESIGN/01-to-be/12-a-module-repository.md @@ -151,6 +151,36 @@ the arrangement working: a build machine shares the runtime it was given rather it sits on. **A module is cloned from the forge over a URL**, and "build this directory" is a convenience for a builder somebody started by hand. +### And the loop is closed + +*2026-08-31.* [ADR 0010](../../02-DECISIONS/0010-delivery.md) replaced a pipeline with a comparison +and named the risk: **losing the question "did my change go out?"**. The mesh could already answer +which modules were behind their source — and then a person read that list and retyped each +repository, which is a person being the loop, and the loop is the thing the pipeline was doing +before it was taken away. + +`build --behind` is the other half, and it is the mirror of `push --behind`: the mesh knows what is +stale, so it builds it. The two forms are deliberately not combined — naming a repository and +asking which need building are different requests, and guessing which was meant would sometimes +build something nobody named. + +**One failing does not stop the others**, for the same reason one broken module no longer blocks a +machine's whole declaration: a mesh where one bad repository holds back nine good ones is a mesh +where nobody dares add the tenth. + +**Each is built from its own recorded ref**, not from the commit the mesh happened to notice. +Pinning to that would quietly turn a tracked branch into a pin — a change of meaning nobody asked +for, arrived at by an implementation detail. + +**Building is not delivering, and the two stay separate.** A machine keeps running what it has +until it is told otherwise; the mesh changing its mind is not a machine acting on it, and +collapsing the two is how a mesh comes to report success for something that has not happened. + +*Checked end to end: a commit, a build, a catalogue entry, and a machine that ends up running what +the source says — with both halves that make the answer trustworthy. It is still running the old +one until it is pushed, and it stops being reported as behind once it has caught up, because a +status that says "behind" for ever is one nobody reads.* + ## What is kept **Every result, including the failures.** A failed build that leaves no trace is indistinguishable diff --git a/03-DESIGN/01-to-be/13-credentials-and-their-rotation.md b/03-DESIGN/01-to-be/13-credentials-and-their-rotation.md new file mode 100644 index 0000000..d368074 --- /dev/null +++ b/03-DESIGN/01-to-be/13-credentials-and-their-rotation.md @@ -0,0 +1,95 @@ +--- +layer: to-be +status: designed +code: + - mesh-control internal/inventory/secrets.go + - mesh-control cmd/mesh-control/rotate.go + - mesh-control examples/postgres-provisioner +updated: 2026-08-31 +decisions: + - 02-DECISIONS/0001-mesh-brokers-nodes-host-agents-think.md + - 02-DECISIONS/0009-modules-and-the-graph.md +--- + +# 13 — Credentials, and moving them + +*Written 2026-08-31, when rotation was built. The delivery half was already proven; this is the +half [ADR 0001](../../02-DECISIONS/0001-mesh-brokers-nodes-host-agents-think.md) records as +unowned, and it was **measurably false** in the system being replaced.* + +## What went wrong before, precisely + +`provision_ensure`, documented as *NEVER rotates an existing secret*, minted a new password on +every adoption and updated **only the provider's row**. Consumers on three nodes held dead +credentials for two days. Two rows for one provision were written 216 ms apart, so at most one +could have matched the live role. The mesh reported success throughout. + +Three separate faults, and it is worth naming them apart because they have different fixes: + +| | | +|---|---| +| **one credential, many holders** | rotating it was necessarily a fan-out, and nothing enumerated who held it | +| **the record moved and the consumers did not** | the change and the delivery were different acts, and only the first happened | +| **nothing said so** | the mesh could not tell a rotated credential from a working one, so nobody looked | + +## What replaces it + +**Every pair has its own credential.** A provision between one consumer and one provider is one +password, made once and kept. So rotating a machine's credential touches one role and leaves every +other consumer alone — and *who holds this* is a query rather than an assumption. That alone +removes the first fault: there is no shared secret to fan out. + +**The change and the delivery are one command.** `rotate` discards the credential and sends both +ends, and it does the sending itself. Leaving that to whoever remembers is the second fault +exactly, and the interval in which it goes wrong is unbounded — two days, in the recorded case. + +**It is all-or-nothing.** If any affected machine cannot be resolved, nothing is sent and the old +credential keeps working. A mesh that has not rotated is far better than one that has +half-rotated, and the difference is that the first is obvious. + +**The window is stated rather than hidden.** A role's password changes on the provider and the file +changes on the consumer, and those cannot be simultaneous. So there is an interval in which a +consumer cannot authenticate, and the honest thing is to make it as short as the broker allows and +to say it exists. `status` names who is still behind. + +## The provider makes it true, and the mesh cannot + +The mesh generated the password, sealed it to the machine that must accept it, and **discarded the +plaintext** — so it cannot tell a database to start accepting it. Something on that machine reads +what the host wrote and makes it true. + +That something is part of the module, not part of the control plane. **The control plane decides +and never touches a machine; a provisioner runs on the machine and touches it.** Two files, because +the mesh could not compose a document containing a value it does not have: + +| | | +|---|---| +| a manifest | every consumer, what it asked for, and where its credential is | +| one file per consumer | that consumer's password, alone | + +**Both, or neither works.** A provisioner given the passwords and not the manifest finds a +directory of unexplained secrets and reports that nothing has been granted — which is true, and +reads exactly like a credential that was never delivered. That has now happened once, here. + +**The password a provisioner uses is itself a file the mesh wrote.** Passing it through the +environment needs a person in the middle of the one path that exists so there is not one, and puts +a superuser password where `docker inspect` prints it. + +## How it is checked + +Not by comparing two files. **Two ends holding a matching string proves they agree, not that either +is right** — the recorded fault produced two ends that agreed with each other and not with the +database. + +So the check is three logins against a real PostgreSQL, from the consumer's own machine, over the +private network: + +1. the delivered credential authenticates +2. after rotation, the new one authenticates +3. **the one that was rotated away does not** + +The third is what makes it a rotation rather than an addition. Without it the check passes against +a provider that added a password and removed nothing. + +**Not over loopback.** `pg_hba` trusts anything there, so every password looks correct — a +deliberately wrong one returned a row for an afternoon before that was noticed. diff --git a/03-DESIGN/01-to-be/14-model-access.md b/03-DESIGN/01-to-be/14-model-access.md new file mode 100644 index 0000000..8499bc1 --- /dev/null +++ b/03-DESIGN/01-to-be/14-model-access.md @@ -0,0 +1,97 @@ +--- +layer: to-be +status: designed +code: + - mesh-control internal/licences + - mesh-control cmd/mesh-control/licence.go +updated: 2026-08-31 +decisions: + - 02-DECISIONS/0024-model-access-is-a-provision.md + - 02-DECISIONS/0009-modules-and-the-graph.md +--- + +# 14 — Model access + +*[ADR 0024](../../02-DECISIONS/0024-model-access-is-a-provision.md) decided it and listed four +things the mesh did not have. Written 2026-08-31, when two of them were built. **The other two are +still gaps and are still written as gaps** — the record's own warning is that pretending otherwise +is how a plan becomes a surprise.* + +## What was built + +**A licence is a record, and the first provision no machine answers.** Everything else the mesh +brokers is answered by something running on a node. A hosted model is on nobody's machine and is +reached over the public internet, so the rule that refuses two ends sharing no private network — +correct everywhere else — must not apply to it. A machine on no private network at all can hold a +licence, and that is not a special case to remember: it falls out of the answer not being a +machine. + +**The name is the operator's.** *The personal account*, *the organisation's account*. Those are +names a person uses, and the mesh uses them too, because the whole point is saying **which one** a +consumer uses — and an anonymous credential hanging off a provider cannot be said. Many to many, +so deliberately **not a claim**: two machines sharing an account is the ordinary case rather than +a collision. + +**One provision name for all of them.** A module requires `model-access`, never `anthropic`. A +module that named a provider could not be moved onto a model the mesh runs itself without editing +it — and moving it is the point. + +**A model in a machine's own set answers it locally**, and no record is consulted. That is what +makes *the mesh's own model* an ordinary answer rather than a parallel arrangement. + +### Accept: taking a value the mesh did not make + +Every other credential here the mesh generated, sealed to both ends and discarded. An API key +arrives from a person, and the missing verb was *accept*: **take a value, seal it to each holder, +discard the plaintext.** A mesh that kept operator-supplied keys readably is the arrangement this +project measured and rejected. + +**It seals to the holders that exist at that moment**, and this has a consequence that must be +said out loud rather than discovered: + +> A consumer put on a licence *after* the key was supplied has no key, and **the mesh cannot make +> one** — it discarded the only copy. + +So that state is reported at every point a person could meet it: when the consumer is put on the +licence, in `licence list`, and — decisively — **the declaration is refused** rather than written +without the file. A machine that resolves cleanly and receives nothing fails later, somewhere that +names neither the licence nor the mesh. + +**A key is read from a file or standard input, never an argument.** A key on a command line is a +key in shell history and in every process listing taken while it ran. It is never echoed back: +what is stored is unreadable by whoever holds it, the control plane included, and printing it +would put the one copy that matters on a terminal. + +## Refusing is felt, and that is the design working + +ADR 0024 predicted it: *a mesh holding three ways to reach a model refuses every consumer that has +not said which — which is correct and is a great deal of saying-which the first time.* + +It is correct, and correct is not the same as usable. So the refusal names **the candidates and +the exact command**. The difference between a mesh that refuses helpfully and one that merely +refuses is whether anybody can act on it without going and reading something else. + +## Still gaps + +Unchanged from [ADR 0024](../../02-DECISIONS/0024-model-access-is-a-provision.md), and deliberately +not half-built: + +**A consumer that is not a machine.** *This worker uses that licence* is a binding to an agent, not +to a node. What is delivered still lands on a machine; what is **chosen** is chosen per agent, and +the provisions model has no consumer identity other than a node. What exists today is per module +per machine, which is a step toward it and is not it. + +**Switching is a reaction, not a declaration.** A licence that hits its limit and must be swapped is +a response to something observed. Expressing it as a declaration would make the declaration mean +*whatever is working right now*, which is not a thing anybody declared. It belongs with +observability, changing a binding — and the binding is then declared as usual. **Saying this +plainly is what stops the declaration language growing a conditional**, and nothing built here +grew one. + +## How it is checked + +In the lab, on real machines, in the order a person would meet it: a consumer is refused with both +candidates named; put on one and still refused because no key exists; the key is given on standard +input and not echoed; the public half arrives saying it came from a record rather than a machine; +the key arrives readable only by that machine — and it is **nowhere in the control plane's own +database**, nor in anything that crossed the broker.