Review before merge: the multi-holder boundary, the window's open race, the sweep's bounds
ADR 0201 gains the boundary found reading it back: a consumer keeping several holders of a deriving provider is refused, because the two ends have no way to agree. ADR 0189 gains two consequences — the sweep is bounded because it runs inside a build, and an apply arriving mid-window reopens it. That last one is issue 224, recorded rather than fixed: the host's rule for a stopped container is to replace it, and while-stopped is the first thing that makes a stopped container intentional. Both candidate fixes are decisions with their own cost. Nothing is worse than it was; the store has never collected.
This commit is contained in:
@@ -107,6 +107,16 @@ unreferenced.
|
|||||||
- The store is briefly unavailable each night, for as long as collection takes. Everything that
|
- The store is briefly unavailable each night, for as long as collection takes. Everything that
|
||||||
pulls from it retries; nothing in the mesh treats a momentary store as a failure
|
pulls from it retries; nothing in the mesh treats a momentary store as a failure
|
||||||
([ADR 0185](0185-a-control-plane-behind-its-seats-row-serves-what-it-can.md)).
|
([ADR 0185](0185-a-control-plane-behind-its-seats-row-serves-what-it-can.md)).
|
||||||
|
- **An apply arriving during the window reopens it**, because the host's rule for a container it
|
||||||
|
finds stopped is to replace it, and `while-stopped` is the first thing that makes a stopped
|
||||||
|
container intentional. Found by reading this before it merged, recorded as
|
||||||
|
[issue 224](../04-ISSUES/224-an-apply-reopens-a-maintenance-window-by-recreating-what-it-held-still/00-report.md)
|
||||||
|
rather than fixed here: the two candidate fixes — the window takes the apply lock, or the apply
|
||||||
|
learns which containers are held — are each a decision with its own cost, and neither belongs
|
||||||
|
inside this record. Nothing is worse than it was; the store has never collected at all.
|
||||||
|
- **The sweep is bounded**: at most two hundred artifacts and sixty seconds per build, stopping at
|
||||||
|
the first refusal, because it runs inside somebody's build. What is left over is offered again
|
||||||
|
next time. The store stops growing from the first sweep; it does not empty in one.
|
||||||
|
|
||||||
## How this is checked
|
## How this is checked
|
||||||
|
|
||||||
|
|||||||
@@ -94,6 +94,15 @@ a literal in a consumer's definition is not merely redundant — it is the one t
|
|||||||
disagree with what the provider will actually create. The three object-store consumers lose their
|
disagree with what the provider will actually create. The three object-store consumers lose their
|
||||||
hand-written bucket names in this change.
|
hand-written bucket names in this change.
|
||||||
|
|
||||||
|
**5. A consumer that keeps several holders of one provision may not be served a derived value.**
|
||||||
|
Each holder gets its own login, `…_<local>` ([ADR 0094](0094-a-module-may-hold-several-secrets-from-one-provider.md)),
|
||||||
|
and a provider derives from the login — so it would make one resource per holder, while the
|
||||||
|
consumer's side has one binding and one `${bound:<provision>:<key>}`, both derived from the
|
||||||
|
un-suffixed identity. That is this record's own failure one case to the side, and just as quiet:
|
||||||
|
the consumer would authenticate and be refused on every object. Refused at resolution, naming
|
||||||
|
both ends. Lifting it means giving the consumer's side a local dimension, which is a decision and
|
||||||
|
not an omission.
|
||||||
|
|
||||||
## Consequences
|
## Consequences
|
||||||
|
|
||||||
- One more thing a definition may say, and one less thing a module may be wrong about. The
|
- One more thing a definition may say, and one less thing a module may be wrong about. The
|
||||||
@@ -105,6 +114,10 @@ hand-written bucket names in this change.
|
|||||||
- A provider that already serves consumers keeps serving them: the derived value equals what the
|
- A provider that already serves consumers keeps serving them: the derived value equals what the
|
||||||
code derived, so no bucket, database or login changes name. This is a change of **who says it**,
|
code derived, so no bucket, database or login changes name. This is a change of **who says it**,
|
||||||
not of **what is said**.
|
not of **what is said**.
|
||||||
|
- A refusal here fails **that machine's push**, naming the definition, and nothing else. That is
|
||||||
|
deliberate and is the opposite of a module quietly left out: a definition that transcribes
|
||||||
|
somebody else's rule is wrong everywhere, not just here, and the loud failure is in front of
|
||||||
|
whoever can fix it.
|
||||||
- The mesh now holds a rule in another system's alphabet — one rule, `dns`, stated once. A second
|
- The mesh now holds a rule in another system's alphabet — one rule, `dns`, stated once. A second
|
||||||
alphabet is a decision, not an addition: the cost of each is that the mesh must be right about
|
alphabet is a decision, not an addition: the cost of each is that the mesh must be right about
|
||||||
somebody else's naming, and that cost is only worth paying where the mesh already mints the name.
|
somebody else's naming, and that cost is only worth paying where the mesh already mints the name.
|
||||||
@@ -118,6 +131,8 @@ hand-written bucket names in this change.
|
|||||||
consumer — one test asserting the three agree, because agreeing is the whole point.
|
consumer — one test asserting the three agree, because agreeing is the whole point.
|
||||||
- Two consumers of one provider on one machine get two different derived values, and neither gets
|
- Two consumers of one provider on one machine get two different derived values, and neither gets
|
||||||
the other's.
|
the other's.
|
||||||
|
- A consumer with several holders of a deriving provider is refused, with both ends named — the
|
||||||
|
test asserts the refusal, not merely that something failed.
|
||||||
- A catalogue-wide test refuses a consumer definition that writes a literal where its provider
|
- A catalogue-wide test refuses a consumer definition that writes a literal where its provider
|
||||||
derives: the provider's `serves` names the key, so the catalogue can say which definitions
|
derives: the provider's `serves` names the key, so the catalogue can say which definitions
|
||||||
transcribe one.
|
transcribe one.
|
||||||
|
|||||||
+64
@@ -0,0 +1,64 @@
|
|||||||
|
---
|
||||||
|
status: open
|
||||||
|
opened: 2026-10-04
|
||||||
|
located-in: [mesh-host internal/apply/apply.go, mesh-host internal/apply/schedule.go]
|
||||||
|
fixed-by:
|
||||||
|
amended-design:
|
||||||
|
---
|
||||||
|
|
||||||
|
# 224 — An apply arriving during a maintenance window reopens it, by recreating the container the window is holding still
|
||||||
|
|
||||||
|
## What was observed
|
||||||
|
|
||||||
|
Reviewing [ADR 0189](../../02-DECISIONS/0189-the-store-keeps-what-the-records-name.md)'s
|
||||||
|
`while-stopped` before merging it, 2026-10-04. Found by reading, not by running.
|
||||||
|
|
||||||
|
A scheduled step may hold its module's containers still while it runs. The host stops them, runs
|
||||||
|
the step, starts them again. Nothing tells the **apply** that a window is open, and the apply's
|
||||||
|
rule for a container it finds stopped is to replace it:
|
||||||
|
|
||||||
|
```
|
||||||
|
case existed && (before.Spec == want || legacy) && before.Running && len(reasons) == 0:
|
||||||
|
out.Action = "unchanged"
|
||||||
|
case existed:
|
||||||
|
rm -f
|
||||||
|
```
|
||||||
|
|
||||||
|
`before.Running` is false for a container a window is holding, so the second branch takes it:
|
||||||
|
the container is removed and recreated, **running**, in the middle of the step that required it
|
||||||
|
to be still.
|
||||||
|
|
||||||
|
## Why it matters
|
||||||
|
|
||||||
|
For the store, which is what the field was built for, the chain is: a push lands at 03:30 → the
|
||||||
|
apply recreates the registry → the registry accepts an upload from a build running at the same
|
||||||
|
time → `garbage-collect`, already past its mark phase, sweeps the blob that upload just wrote.
|
||||||
|
The image is then in the store with a layer missing, and the build that made it reported success.
|
||||||
|
|
||||||
|
Two things have to coincide, so it is not likely. It is also not rare enough to leave unsaid: the
|
||||||
|
mesh pushes on every merge, at any hour, and a collection over a store this size is minutes rather
|
||||||
|
than seconds.
|
||||||
|
|
||||||
|
**The general shape is the one that matters.** `while-stopped` is the first thing in the mesh that
|
||||||
|
makes a container's stopped state *intentional*. Everything else in the host reads "stopped" as
|
||||||
|
"broken, fix it", which is right everywhere else and wrong here. Any future use of the field
|
||||||
|
inherits this.
|
||||||
|
|
||||||
|
## What this is not
|
||||||
|
|
||||||
|
Not a regression. The store has never collected anything, so nothing is worse than it was; this
|
||||||
|
is a hole in something new rather than something that broke.
|
||||||
|
|
||||||
|
## Open questions
|
||||||
|
|
||||||
|
- **Should a window take the apply lock?** The daemon already serialises applies with `applying`
|
||||||
|
and, across processes, with `store.Lock`. A window that held it would make the race impossible.
|
||||||
|
The cost is that a push arriving mid-window waits for minutes, and a push that waits is what
|
||||||
|
[issue 185](../185-a-refused-membership-publish-stops-the-controller/00-report.md)'s
|
||||||
|
family of outages looked like from outside.
|
||||||
|
- **Or should the apply learn that a container is held?** Narrower: the scheduler says which
|
||||||
|
containers a window currently holds, and `applyContainer` reports those unchanged instead of
|
||||||
|
recreating them. Nothing blocks, and the apply tells the truth for the minutes it matters —
|
||||||
|
at the cost of a second source for "is this container meant to be running".
|
||||||
|
- Either way: should the *report* say a window is open, so a machine that looks half-stopped at
|
||||||
|
03:31 reads as working rather than broken?
|
||||||
Reference in New Issue
Block a user