Compare commits
27
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
63586dcd50 | ||
|
|
c613cf0b82 | ||
|
|
903340516c | ||
|
|
793e98d45a | ||
|
|
261bee1cab | ||
|
|
eec84794e3 | ||
|
|
3e84e4cb60 | ||
|
|
32107428c4 | ||
|
|
ec9bab5ee8 | ||
|
|
69c6ff73e6 | ||
|
|
8273faa4f0 | ||
|
|
de5c89a211 | ||
|
|
ba49876e48 | ||
|
|
7e158c739e | ||
|
|
7dd83d4435 | ||
|
|
a04f1b90a0 | ||
|
|
c08b621491 | ||
|
|
b63d27ded1 | ||
|
|
1e51c062f6 | ||
|
|
1ebbf9007a | ||
|
|
1714132a08 | ||
|
|
9f66452fff | ||
|
|
80b02dc9a0 | ||
|
|
fbd00a16ca | ||
|
|
17b64270b0 | ||
|
|
3acc1ec219 | ||
|
|
582be67f05 |
+13
-5
@@ -389,6 +389,10 @@ becomes a message). **Decided by** ADR 0227, 0231, 0240, 0252.
|
||||
its **budget**, its back-off and its brake. A healer may not withdraw, delete or recreate data.
|
||||
- **drill** — something broken on purpose to see the mesh raise and clear its condition; a drill is
|
||||
never counted as a repair.
|
||||
- **rehearsal** — a question the operator starts at the controller's terminal (`rehearse`) and answers on a
|
||||
channel, to see an answer travel the whole way: its approval performs nothing and is recorded as the
|
||||
operator's decision ([ADR 0259](../02-DECISIONS/0259-an-ask-is-a-message-on-the-bus-any-channel-answers-it-and-the-asker-acts-on-a-warrant.md)).
|
||||
Not a drill: nothing is broken.
|
||||
- **hand-act** — a repair a person made by hand, recorded so the mesh knows it happened.
|
||||
- **witness** — what rolls a core component back when its new build does not become healthy: never the
|
||||
component itself (to-be 45 §8).
|
||||
@@ -443,10 +447,10 @@ it should not. **Recorded by** the controller. **Upstream of** Provisioning and
|
||||
## Operator and conversation — the person the mesh works for
|
||||
|
||||
**Purpose.** To let the mesh and its operator talk: tell, ask, answer, and act only on an answer it can
|
||||
trust. **Recorded by** the router and the controller (authorising asks).
|
||||
trust. **Recorded by** the router (asks, warrants and the operator's identities).
|
||||
**Downstream of** Health and repair, Change and delivery, and Identity and access. An authorising
|
||||
answer acts in whichever domain asked, through that domain's own verb; the conversation owns the
|
||||
asking, never the act. **Decided by** ADR 0152, 0175, 0234, 0272.
|
||||
asking, never the act. **Decided by** ADR 0152, 0175, 0234, 0259, 0272.
|
||||
**Uses:** operator, person, proof (Identity and access), release (Change and delivery).
|
||||
|
||||
- **mesh MCP server** — the endpoint on a node's loopback through which every agent and person on that
|
||||
@@ -476,10 +480,14 @@ asking, never the act. **Decided by** ADR 0152, 0175, 0234, 0272.
|
||||
the condition beside its **summary**, which keeps the plans, commits and verbs for whoever looks closer
|
||||
([ADR 0253](../02-DECISIONS/0253-a-condition-says-itself-to-the-operator-in-plain-words-beside-its-summary.md)).
|
||||
- **ask** — a request for the operator's input, of a declared kind (yes-no, one-of, text, number,
|
||||
date, acknowledge). An **authorising ask** is one whose answer performs an action; the controller
|
||||
holds it and checks its proofs (ADR 0234).
|
||||
date, acknowledge). An **authorising ask** is one whose chosen option its asker performs. The router
|
||||
holds every ask, checks each answer's proofs and issues a **warrant**; the asker acts on it with its
|
||||
own grants (ADR 0234, ADR 0259).
|
||||
- **warrant** — the router's record that a person chose one option of one ask: the ask, the option,
|
||||
who chose (the operator, as a channel kind's identity), through which channel, the proofs and when,
|
||||
published for that asker alone. The only thing an asker acts on (ADR 0259).
|
||||
- **operator message / input** — what arrives on `intake`, once the router has checked the sender
|
||||
against the controller's list of the operator's identities: a **trusted** one is an operator message,
|
||||
against its own list of the operator's identities: a **trusted** one is an operator message,
|
||||
addressed to an agent by `@name` or thread or else to the **responder** (`@mesh`, the router's own
|
||||
participant, which answers from read verbs); anything else is **untrusted input** — data, never
|
||||
instructions.
|
||||
|
||||
@@ -289,6 +289,12 @@ consumers are later work.
|
||||
|
||||
### 8. Asks that authorise
|
||||
|
||||
> **Who holds the ask changed — 2026-10-08, by [ADR 0259](0259-an-ask-is-a-message-on-the-bus-any-channel-answers-it-and-the-asker-acts-on-a-warrant.md).**
|
||||
> What stands: the tiers, the proofs, the desk never authorising, and every tier completable on the away
|
||||
> channel. What moved: the router, not the controller, holds authorising asks, the operator's identities
|
||||
> and the factors, and judges each answer; the asker performs the option itself on a warrant. The
|
||||
> controller's verbs `authorise request`, `authorise answer` and `authorisations` below are replaced there.
|
||||
|
||||
**An ask whose answer performs an action is held by the controller.** The router carries it like any
|
||||
other ask; only the controller performs.
|
||||
|
||||
|
||||
+589
@@ -0,0 +1,589 @@
|
||||
---
|
||||
topic: the mesh
|
||||
status: accepted
|
||||
date: 2026-10-09
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
supersedes-in-part:
|
||||
- 0234-the-mesh-holds-a-conversation-with-its-operator.md
|
||||
extends: 02-DECISIONS/0234-the-mesh-holds-a-conversation-with-its-operator.md
|
||||
---
|
||||
|
||||
# 259. An ask is a message on the bus, any channel answers it, and the asker acts on a warrant
|
||||
|
||||
## Context
|
||||
|
||||
On 2026-10-08 the operator chose Telegram as the next channel for the operator's decisions. "Needs you:
|
||||
release it, or stop it" must be answerable from the phone, with real authorisation, and without an agent
|
||||
session.
|
||||
|
||||
What the mesh had on its main branches that day:
|
||||
|
||||
- **The desk performs only an acknowledgement**
|
||||
([ADR 0258](0258-a-desk-click-performs-only-an-acknowledgement-and-an-answer-is-a-one-time-ticket.md)).
|
||||
Release, stop, start and restart are said in words: "from the mesh MCP server; this notification cannot do
|
||||
it". The messenger (the `operator-channel` seat's holder) calls `conditions silence` itself when the
|
||||
operator chooses Silence on the desk.
|
||||
- **The messenger holds the Telegram client in its own process**, says conditions there as words without
|
||||
buttons, and keeps the bot token as its own secret `telegram-token`. Its status said on 2026-10-08 that the
|
||||
token held is not a bot token: no bot has been given yet.
|
||||
- **[ADR 0234](0234-the-mesh-holds-a-conversation-with-its-operator.md) §8 gives authorising asks to the
|
||||
controller**: it holds the ask, keeps the operator's identities and factors, checks the proofs and
|
||||
performs the action itself. None of it is built ([to-be 46](../03-DESIGN/01-to-be/46-the-conversation-with-the-operator.md)
|
||||
phase 5).
|
||||
- **Any node's tool runner may call any tool on any node** (ADR 0175). A tool call therefore proves
|
||||
nothing about who made it: the laptop's runner carries the agents' calls. What the bus *does* prove is the
|
||||
publisher of a module's events and a seat's events: only the principal granted a subject may publish it,
|
||||
which is why a condition's events need no signature (design 29 §2).
|
||||
|
||||
The operator's directions, 2026-10-08, on the first proposal (a Telegram path through the controller):
|
||||
|
||||
- "I don't think we should mangle Telegram into our controller. Now we're using Telegram, but tomorrow we
|
||||
could be using Signal. And today the controller needs to receive a human decision, but tomorrow a
|
||||
completely different module might require human approval."
|
||||
- **Two generic boundaries.** Asking a person is a mesh-wide service any module uses; the asking module
|
||||
performs the action itself, with its own grants, and records who decided. Channels are pluggable: a
|
||||
channel renders the question, receives the answer and verifies the sender by its own means. Nothing
|
||||
outside the channel module knows which service it talks to.
|
||||
- **The controller only asks and acts.**
|
||||
- **The flow is native to the bus**: a request is a message, pending requests survive a restart, each
|
||||
answer is recorded once, the asker reads its request's state from a bucket, and the bus's permissions
|
||||
decide who may publish what. Events carry no secret; request and reply may. No HTTP between modules.
|
||||
|
||||
The operator approved this record on 2026-10-08, with the word *warrant*, and decided two points it had
|
||||
left open:
|
||||
|
||||
- **The `telegram` module runs on the control node under a service account of its own, not the
|
||||
operator's.** Claude Code runs as the operator on every machine; the service account keeps an agent away
|
||||
from the bot token and the channel's state.
|
||||
- **A newly linked identity waits one hour before it can approve**; acknowledge works at once. Every link
|
||||
and unlink is announced on every channel. This holds until a code from an authenticator app exists.
|
||||
|
||||
The operator accepted this record as revised on 2026-10-09 — every part marked "added" or "revised" on that day,
|
||||
after the security reviews and the confirmation review — and asked that the mesh be able to ask them a question,
|
||||
and ask for an approval, from then on.
|
||||
|
||||
### Against GENESIS
|
||||
|
||||
- **Core value — sovereignty.** Telegram stays an outside dependency taken deliberately (ADR 0234), and
|
||||
is now one module that can be removed without touching anything that asks.
|
||||
- **Core value — failure must be loud.** An ask that no channel can carry is refused in words to its asker
|
||||
at once, never queued in silence. An answer that is refused is said on the channel it came from.
|
||||
- **Effect — "when something genuinely needs a decision, you are asked, with the context".** Any module
|
||||
can now ask, not only the controller.
|
||||
|
||||
## Considered Options
|
||||
|
||||
### Where an ask is held
|
||||
|
||||
1. **The controller holds authorising asks** (ADR 0234 §8). Rejected by the operator: the next module
|
||||
that needs a person (a backup module, the delivery owner) would ask through the controller, and the
|
||||
controller would perform actions in other domains.
|
||||
2. **A new seat for decisions, beside `operator-channel`.** Rejected: an ask needs the router's routing,
|
||||
the work context and the content rule; a second seat would ask the router for each of them, and two
|
||||
holders would keep two lists of who the operator is.
|
||||
3. **The `operator-channel` seat's holder, the router, holds every ask, authorising or not, and judges
|
||||
the answers.** Chosen. ADR 0234 already gives it ordinary asks; this adds the authorising ones.
|
||||
|
||||
### Who performs the action
|
||||
|
||||
1. **The decision holder calls the asker's verb.** Rejected: the router would act in every domain with a
|
||||
grant to every verb, which is what ADR 0258 took away from the desk.
|
||||
2. **The asker performs on hearing "yes" from anyone.** Rejected in ADR 0234: a relayed "yes" is not an
|
||||
answer.
|
||||
3. **The asker performs on a warrant**: the router's record that the operator chose an option, published
|
||||
on a subject only the router may publish, for that asker alone. Chosen. The bus proves the warrant's
|
||||
source as it proves a condition's. A signature was weighed and rejected: it needs a key in every asker
|
||||
and proves nothing the server's permissions do not.
|
||||
|
||||
### How an ask, a warrant and an answer travel
|
||||
|
||||
1. **Tool calls** (an `ask` verb, an `answer` verb). Rejected: any tool runner may call any tool, so the
|
||||
router could not tell the controller's ask from an agent's, nor a channel's answer from anybody's.
|
||||
2. **HTTP between modules.** Rejected by the operator's direction, and it would carry its own
|
||||
authentication beside the bus's.
|
||||
3. **Subjects whose last token names the publisher**, granted to that publisher alone, on streams that
|
||||
survive a restart. Chosen. The mesh already grants a machine its own token on a node seat's events
|
||||
(ADR 0219); this is the same rule for a caller's module and a channel's kind.
|
||||
|
||||
### Who is the operator on a channel
|
||||
|
||||
1. **A setting naming the operator's account on the service.** Rejected: any principal granted
|
||||
`mesh-controller.settings` may write a setting, agents included, so the operator's identity would be
|
||||
anyone's to set.
|
||||
2. **The channel module's own allow-list.** Rejected in ADR 0234: the channel would decide who the
|
||||
operator is.
|
||||
3. **The router's list of the operator's identities, per channel kind, written only by linking.**
|
||||
Chosen. A channel module passes the sender as the service authenticated it, as an opaque string; the
|
||||
router alone compares it with its list.
|
||||
|
||||
### How a channel takes its answers (Telegram)
|
||||
|
||||
1. **A webhook.** The control node is the stably reachable node, so it could take one. Rejected all the
|
||||
same: it needs a public route, a name and a certificate into the control node, guarded only by a
|
||||
secret header; a stolen token could point it elsewhere.
|
||||
2. **Long polling**, as ADR 0234 chose. Chosen: outbound only, unaffected by the control node's address,
|
||||
answered as fast as a webhook, and a stolen token can read updates (visible as a conflict) but not
|
||||
inject one.
|
||||
|
||||
### Where the desk's buttons live
|
||||
|
||||
1. **Inside the router**, as its desktop adapter (to-be 46 §11). Rejected: the router would know one
|
||||
channel, and the operator asked that the desk be just another channel.
|
||||
2. **The dunst module** claims the channel. Rejected: dunst holds a node seat on each machine, and a
|
||||
channel kind has one holder for the mesh.
|
||||
3. **A small module of its own, kind `desktop`**, sending through each machine's `node-notifier` and
|
||||
reading dunst's `action-chosen`. Chosen.
|
||||
|
||||
## Decision
|
||||
|
||||
### 1. The words
|
||||
|
||||
- An **ask** keeps its glossary sense; an **authorising ask** is one whose chosen option the asker
|
||||
performs. Each option carries its **assurance level**: `acknowledge`, `approve` or `destroy` (ADR 0234's
|
||||
tiers, the glossary's word).
|
||||
- A **warrant** (new) is the router's record that a person chose one option of one ask: the ask, the
|
||||
option, who chose (the operator, as a channel kind's identity), through which channel, the proofs, and
|
||||
when. It is the only thing an asker acts on.
|
||||
- **The router** keeps its sense (the `operator-channel` seat's holder). It now holds every ask and
|
||||
issues warrants. It never performs an option.
|
||||
|
||||
### 2. The boundaries
|
||||
|
||||
| Party | Does | Never |
|
||||
|---|---|---|
|
||||
| **an asker** (any module; first the controller) | publishes an ask under its own name; reads its asks' state; acts on its own warrants with its own grants; records the warrant in its own record of the act | performs without a warrant; reads another asker's asks |
|
||||
| **the router** (`operator-channel` holder, the `messenger` module) | keeps asks, tokens and the operator's identities; routes an ask to the channels whose capabilities meet each option's level; judges each answer; issues one warrant per ask | calls an asker's verb; knows a service's API |
|
||||
| **a channel** (a module claiming a kind on `channel` and `intake`: `telegram`, `desktop`) | shows what the router sends, with the router's tokens on its buttons; passes back the token chosen and the sender as its service authenticated it | judges whether the sender is the operator; holds another channel's secret |
|
||||
|
||||
### 3. The subjects, streams and buckets
|
||||
|
||||
Every name below follows the mesh's existing shapes (`mesh.seat.<seat>.<accept|event|tool>.…`, a module's
|
||||
state bucket, the EVENTS stream). `<asker>` is the asking module's name; `<kind>` a channel's kind.
|
||||
|
||||
| What | Subject or bucket | Kept in | Publisher (granted) | Reader (granted) |
|
||||
|---|---|---|---|---|
|
||||
| an ask | `mesh.seat.operator-channel.accept.ask.<asker>` | the seat's work queue `SEAT_OPERATOR_CHANNEL` (kept 7 days, removed when the router has recorded it) | the asker, its own name only | the router |
|
||||
| an ask cancelled by its asker | `mesh.seat.operator-channel.accept.cancel.<asker>` | the same | the asker, its own name only | the router |
|
||||
| the asks and their state | the router's bucket `asks`, key `<asker>.<id>` | its own state, closed records kept 30 days | the router | the router; each asker reads its own prefix only (direct get) |
|
||||
| answer tickets | in each ask's record in `asks`: one per option and channel kind, used once, dead when the ask closes | its own state | the router | the router |
|
||||
| the operator's identities | the router's bucket `identities`, key `<kind>.<identity>` | its own state | the router | the router |
|
||||
| a warrant, or an ask's end without one | `mesh.seat.operator-channel.event.decided.<asker>`, message id `<asker>.<id>` | EVENTS (7 days), deduplicated by the message id | the router only | the asker, its own name only, through its durable consumer |
|
||||
| show, edit or say on a channel | `mesh.seat.channel.accept.<show\|edit\|send>.<kind>` | the seat's work queue `SEAT_CHANNEL`, one worker per kind | the router | the holder of that kind only |
|
||||
| an answer chosen, a link asked, a message that failed, the channel's standing | `mesh.seat.intake.event.<choice\|link\|failed\|standing>.<kind>` | EVENTS | the holder of that kind only | the router |
|
||||
| a code typed (a link code; later a TOTP code) | `mesh.seat.intake.proof.code.<kind>`, core request and reply | nothing: never on a stream | the holder of that kind only | the router, which answers |
|
||||
|
||||
**The grants, composed by the controller as it composes every bus account:**
|
||||
|
||||
- **A caller-named verb.** A seat may mark an accepted verb or an emitted event as named by its caller
|
||||
(`ask.*`, `cancel.*`, `decided.*`). A module that uses the seat is granted to publish the accept, and to
|
||||
subscribe the event, with its own module name as the last token, and no other. A tool runner carrying
|
||||
modules is granted the same for each module it carries. The controller asks as `mesh-controller`.
|
||||
- **A kinded bench** (ADR 0234 §2, built here for `channel` and `intake`). A claim carries `kind`; two
|
||||
claims of one kind are refused. A holder subscribes its own kind's accepts, publishes its own kind's
|
||||
events and proofs, and nothing of another kind. Each kind has its own worker on the seat's queue, so a
|
||||
channel that is offline keeps its work and holds up no other.
|
||||
- **A seat's `proof` verbs** are core request and reply on `mesh.seat.<seat>.proof.<verb>.<kind>`: no
|
||||
stream covers them, so a code is never persisted (the mesh's rule: events carry no secret). A holder
|
||||
asks with its own kind; the modules that watch the seat answer.
|
||||
- **The seat declares all of it**: `kinded`, the verbs named `by-caller`, its `proofs`, and the holder's
|
||||
`records` its users read under their own name. Only `channel` and `intake` may be kinded.
|
||||
- **A claim of a kinded bench carries `kind` and `capabilities`** (`channel-capabilities/1`, ADR 0234 §2).
|
||||
The controller names every kind, its holder and its capabilities in the membership of each module that
|
||||
uses or watches the bench: that is where the router reads them, never from a channel.
|
||||
- **A record readable by its asker.** The router's `asks` bucket is read by an asker under its own prefix
|
||||
only: direct get on `$KV.<bucket>.<asker>.>`.
|
||||
- **No `answer` tool.** A tool runner may call any tool, so an answer that arrived as a tool call could
|
||||
come from anyone. An answer arrives only as an intake event of a kind.
|
||||
|
||||
### 4. An ask
|
||||
|
||||
An ask carries: an `id` the asker chose, unique to it; the **question** in plain words, held to the plain
|
||||
rule and the content rule (ADR 0253, ADR 0234 §6); its **options**, at most four, each a label, what it
|
||||
does in plain words, its assurance level and an option id; **who may answer** (the role `operator`, the
|
||||
only role today); an **expiry**; **what happens on expiry**, in words the operator is shown ("the delivery
|
||||
stays held"); and optionally a subject it is about (a condition's key), so a newer ask about the same
|
||||
thing replaces the older.
|
||||
|
||||
- **Refused at once, in words, on the decided subject**: an ask whose words break a rule, one none of whose
|
||||
options any ready channel can carry, or a fourth open ask from one asker. An option no channel can carry
|
||||
now (`destroy` until a TOTP factor exists; `approve` while no account is linked) is not offered, and the
|
||||
words say where it can still be given.
|
||||
- **An authorising ask never defaults.** At its expiry the router ends it with `expired`; the asker does
|
||||
what it said it would.
|
||||
- **An approve option expires within 24 hours; a destroy option within 10 minutes of being shown**
|
||||
(ADR 0234).
|
||||
|
||||
### 5. The router judges an answer
|
||||
|
||||
On an intake `choice` of kind K carrying token T and sender S, the router refuses at the first failure and
|
||||
says the refusal on K's channel:
|
||||
|
||||
1. T is known, unused, unexpired, and was issued for kind K;
|
||||
2. the ask is open, and the option T names belongs to it;
|
||||
3. S is on the router's list of the operator's identities for K;
|
||||
4. K's declared capabilities, **read from the controller's record of its claim and never from the
|
||||
channel**, meet the option's level: `acknowledge` needs `choice` and `exact-render`; `approve` needs
|
||||
`verified-sender` as well (P1); `destroy` needs two proofs, one of them a code (P2); and for `approve`
|
||||
and above, **the controller's `root-free` verb answers, at that moment, that no agent can become root
|
||||
without a person** on the router's machine and on K's channel's machine (§8 rule 2, added on 2026-10-09:
|
||||
an absent condition is no pass);
|
||||
5. the ask is closed with a compare-and-set on its record, so a second answer loses.
|
||||
|
||||
Then it publishes the warrant on `decided.<asker>`, and sends every channel that showed the ask an edit
|
||||
naming the outcome and removing its buttons.
|
||||
|
||||
**The desk declares no `verified-sender`**, so it answers only `acknowledge` options, as ADR 0258 decided.
|
||||
Telegram declares it while placed where no agent runs as the operator, so it answers `approve`.
|
||||
|
||||
### 6. The warrant, and the asker's act
|
||||
|
||||
The warrant carries the ask's id and asker, the option, `by` (the operator, as a kind's identity: "the
|
||||
operator, via telegram, user id verified"), the channel module, the proofs present, the assurance level
|
||||
met, and when. It carries no secret.
|
||||
|
||||
- **The asker acts once per ask id**, keeping that in its own record, and only on a warrant from its own
|
||||
`decided` subject. A warrant that says no time it was given, or for an ask that says no expiry, is refused:
|
||||
neither can be shown to be in time (the confirmation review of 2026-10-09).
|
||||
- **A silence never takes an approval back** (the confirmation review of 2026-10-09, M1). The controller
|
||||
asks a condition's acknowledgements (Silence) apart from its approvals (Release, Stop, Restart), and choosing
|
||||
Silence silences the condition; the first build then took the approval ask back as no longer wanted, so an
|
||||
acknowledgement, which a desk click may give, ended an approval. An approval asked about a condition that is
|
||||
silenced since stays open, unchanged, until it is answered on a channel that proves who answered, or expires.
|
||||
- **A warrant authorises one act, as the person was shown it** (added on 2026-10-09, before the code was
|
||||
merged). Every option above `acknowledge` carries the digest of exactly what the asker performs when it is
|
||||
chosen — the verb, the machine and every argument (`binds`); an ask without it is refused at the router.
|
||||
The router puts the digest of the ask as it took it in the warrant. The asker acts only when that digest
|
||||
is the one of the ask it keeps, and only when the act it is about to perform is the one the option bound:
|
||||
an ask published again under the same id, or a record of the act changed after the ask, is refused and
|
||||
nothing is done. **Both digests are over named fields in a canonical encoding** (each value written as its
|
||||
length, a colon and its bytes, in a fixed order, versioned), never over a language's encoding of a
|
||||
structure, so any language reproduces them and a field added to an ask changes no digest until it is
|
||||
added to the encoding on purpose (the review of 2026-10-09).
|
||||
- **The router publishes each warrant under a message id nobody can guess** (a random one per publish), on
|
||||
the asker's `decided` subject, which only the router may publish; the asker deduplicates on its own record
|
||||
(a compare-and-set claim), never on the bus's duplicate window. A guessed id published first by another
|
||||
principal into the same stream would otherwise have the real warrant dropped as a duplicate (the review of
|
||||
2026-10-09).
|
||||
- **Its record of the act names the warrant**: a hand-act's `by`, `via`, `ask` and `proofs` (ADR 0234 §8),
|
||||
a delivery's transition, a silence's why. The hand-act log counts it as a person's decision, not a repair.
|
||||
- **The controller only asks and acts.** For a condition that needs the operator it asks with the
|
||||
condition's actions as options, and on the warrant it calls the verb it named, with its own grant. It
|
||||
holds no channel, no identity and no factor.
|
||||
- **A warrant is not yet the only way to these acts.** The acts a warrant authorises today —
|
||||
`mesh-delivery.release` and `stop`, the controller's `plans` go and stop, `node-service-manager.restart`,
|
||||
and `conditions silence` — are still verbs an agent may call directly with its own grant. Until
|
||||
[research 039](../01-RESEARCH/039-a-tool-that-needs-a-persons-approval/00-overview.md) (a tool that needs a
|
||||
person's approval) withholds them from every caller without a warrant, a warrant proves that the operator
|
||||
chose an act, not that only the operator could have caused it. Said here so no reader takes the warrant for
|
||||
that second guarantee.
|
||||
|
||||
### 7. Who the operator is on a channel: linking
|
||||
|
||||
- **An answer from an account that is not the operator's** is refused, and the operator is told on the linked
|
||||
account which account tried (by name and number); the other account is told nothing (added on 2026-10-09:
|
||||
a question forwarded, or a phone in other hands, is something the operator must hear of).
|
||||
- **The router's list is the only list.** It is changed only by linking and unlinking, never by a setting.
|
||||
- **Linking:** the operator sends the bot `/start`. The channel publishes `link.<kind>` with the sender
|
||||
and the name the service shows. The router shows a six-digit code on the desk, with that name, valid
|
||||
for ten minutes. The operator sends the code to the bot; the channel passes it on the `proof` subject;
|
||||
the router links the identity when it matches, and says so on every channel.
|
||||
- **A code is shown to a person or not at all** (added on 2026-10-09, after the review). The router issues a
|
||||
code only while the desk channel says it is ready, and tells the sender the desk's reason when it is not;
|
||||
a code is never queued for later. The desk is ready only while a named desk machine's notifier said within
|
||||
five minutes that it is not paused (the notifier's `paused` event), and the code goes only to such a
|
||||
machine; shown nowhere, it is said back to the router as failed. Every message carrying a code is marked secret: no
|
||||
channel and no router record keeps a copy, the desk channel stores none and hands it to the node's
|
||||
notifier as transient, and the notifier keeps it out of its history, where an agent could read it later.
|
||||
- **The desk channel runs as an account of its own**, like the router and Telegram (§8 rule 1): the codes
|
||||
pass through it. Registration holds every holder of a `private` kind to it, as it holds a `verified-sender`
|
||||
holder (the confirmation review of 2026-10-09).
|
||||
- **One identity per kind.** A second link of a kind is refused until the first is unlinked.
|
||||
- **Unlinking is an approve ask, answered on the account that is linked** (revised after the security
|
||||
review of 2026-10-08). It is never a plain tool: whoever unlinks the operator's account could link one of
|
||||
their own in its place. The notice of an unlink, and of any link for an hour after it, is sent to the
|
||||
account being replaced by name, before anything changes, and to every other channel.
|
||||
- **A link within a day of an unlink waits for the replaced account's approval**, asked there. When that
|
||||
account is lost with its phone, the operator acts as root at the router's machine (break-glass, as ADR 0234
|
||||
§8 does for a lost factor): a regular file `unlink.<kind>` or `allow-link.<kind>`, owned by root, in the
|
||||
router's own state directory (mode 0700, owned by the router's account). The router acts on it within a
|
||||
minute, deletes it, ends every open ask of its own, and announces on every channel what root did. A file of
|
||||
any other owner, or not a regular file, is refused and said: an agent without root cannot make one.
|
||||
- **Linking is bounded**: at most three links asked per account an hour and ten per kind, counted in the
|
||||
router's own state so a restart forgets nothing (a count that cannot be read or written refuses the link); the code's notice on the desk names the account by
|
||||
the name and the number the service gives, and only a channel declaring `private` (the desk) may show a
|
||||
code. **A residual risk of the bound, accepted** (the confirmation review of 2026-10-09): anyone may make
|
||||
throwaway accounts and send the bot `/start` ten times in an hour, so the operator's own link is refused
|
||||
for the rest of that hour. It denies a link, never makes one, and every refused link is said; the operator
|
||||
links when the hour has passed. A bound per sender alone would let the same throwaway accounts flood the
|
||||
desk with codes, which is worse.
|
||||
- **The ask to unlink or relink binds the account**, not only the kind (added on 2026-10-09): a warrant for
|
||||
unlinking one account never unlinks the one linked after it. A link re-checks that its kind has no account
|
||||
when it is made, and an account just unlinked cannot link again at once from a code shown before.
|
||||
- **An account still in its first hour can be unlinked without its own approval** (added on 2026-10-09, after
|
||||
the review): it cannot approve yet, so its consent protects nothing. When another account sends `/start`
|
||||
while one is in its first hour, the router asks at the desk, as an acknowledgement bound to that account,
|
||||
whether to unlink it, and tells the sender so. If the hour has passed when the answer arrives, the unlink
|
||||
is refused. After the hour, unlinking needs the linked account's approval, or break-glass.
|
||||
- **An unlink at the desk in the first hour bars nothing** (the confirmation review of 2026-10-09, finding
|
||||
N1-router). A desk click is anyone's at the desk machine — a program there can forge one — and the first
|
||||
build barred the unlinked account for a day and made the next link wait for it. Another account's `/start`
|
||||
offered the desk the unlink, a forged click took it, and the operator's own account stayed barred, again
|
||||
and again, so the operator could never approve. Now such an unlink lets any account, the operator's too,
|
||||
link again at once with a code shown on the desk. **The day's bar is set only by an unlink the account
|
||||
itself approved, or root's**; root's `allow-link` at the router's machine lifts any bar. What is said on
|
||||
every channel names which it was.
|
||||
- **A new link answers `acknowledge` at once and `approve` only after one hour.** The hour runs from the
|
||||
moment the link is made, is kept in the identity's record, and is said in the announcement ("it can
|
||||
approve from 15:40"). An approve answer inside the hour is refused and said on the channel. Decided by
|
||||
the operator on 2026-10-08, until an authenticator-app code exists.
|
||||
- **Once a TOTP factor exists**, linking and unlinking are destroy asks (ADR 0234 §8).
|
||||
- **A residual risk is accepted until then:** on the X11 desk an agent running as the operator can read the
|
||||
code and send it from an account of its own. It is said loudly on every channel when it happens; during
|
||||
the hour the account cannot approve, and the operator can unlink it from the desk without its approval (the
|
||||
rule above — corrected on 2026-10-09: before that rule, the hour gave no way to unlink an account that
|
||||
would not consent); and the factor closes it.
|
||||
|
||||
### 8. Who may say what on the bus: the trusted parties' own accounts
|
||||
|
||||
*Revised after the security review of 2026-10-08; the revision accepted by the operator the same day.* The first version of
|
||||
this section rested on the tool runner running as root with a credential only root reads. **On the control
|
||||
node it runs as the operator's account, its bus credential belongs to that account, and that account may
|
||||
become root without a password.** Every agent runs as that account. So an agent there could read the runner's
|
||||
credential and publish anything any carried module may: a warrant, an answer from Telegram, a link. The
|
||||
service account of the first version kept the bot token from nobody.
|
||||
|
||||
The revision rests on three rules.
|
||||
|
||||
**1. The trusted parties hold bus accounts of their own, never a runner's.**
|
||||
|
||||
- The **router** (`operator-channel`'s holder) and **every channel that declares `verified-sender`** run as
|
||||
their own process, as their own service account (`messenger`, `telegram`), with a bus account of their
|
||||
own: an own secret `broker`, owned by that account, mode 0600. The manifest says so with `runs-as`; the
|
||||
controller composes for such a module a runtime of its own — the machine's runtime program, run as that
|
||||
account, serving that module alone on that credential — and the machine's runtime never carries it.
|
||||
- **The controller refuses to compose** into any runner's user, or into any principal whose process runs as
|
||||
the node's operator account:
|
||||
- publishing on a by-caller event of `operator-channel` (a warrant);
|
||||
- writing the router's state (asks, identities);
|
||||
- publishing on, or answering, a kinded bench's subjects for a kind declaring `verified-sender`.
|
||||
A manifest asking for it is refused at registration, and a runner carrying such a module is refused at
|
||||
composition, naming the module.
|
||||
- **Proofs are answered by the holder of `operator-channel` alone**, not by any module watching the bench;
|
||||
and **a kind's accepts are published by that holder alone**, not by every user of the bench.
|
||||
- **The tool runner hands a bundle no bus variable** of its own: the credential and the bus's address are
|
||||
taken out of every bundle's environment.
|
||||
|
||||
**2. `verified-sender` is honoured only where the controller can show who holds it.**
|
||||
|
||||
The controller passes a kind's `verified-sender` to the router only when its holder runs as its own
|
||||
account, with its own bus account, on a machine where **no account an agent runs as can become root without
|
||||
a person**. Otherwise it strips the capability from the membership, and the router answers that kind only
|
||||
as far as acknowledgements, saying why. The same check holds the router: while it fails on the router's
|
||||
machine, the router issues no approve warrant, and the condition says what to change.
|
||||
|
||||
*Revised on 2026-10-09, after the review of the first build.* **Only a positive, recent measure is a pass;
|
||||
the absence of a condition never is.** The first build let the router approve while no `agent-root`
|
||||
condition was open — so before the first self-check, after a probe error, with the conditions view lost, or
|
||||
with the router not knowing its own machine, it approved. Now:
|
||||
|
||||
- **The controller's `root-free` verb** answers, for each machine asked, whether no agent can become root
|
||||
there without a person, judged at the moment it is asked. It is a pass only when the machine names an
|
||||
account its agents run as, the machine's node-engine judged that account unable to become root (ADR 0266:
|
||||
the node-engine reads it as root, on every look, and the controller takes a verdict at most fifteen
|
||||
minutes old), and the login shell's `execute` is not served there (ADR 0268). Anything else — no agent
|
||||
account named, so agents run as the operator; a verdict missing, stale or unhealthy; a store or the bus not
|
||||
answering — is not a pass, and says why.
|
||||
- **The measure is the node-engine's, not the measured account's.** The first build asked the sudo module,
|
||||
served by the machine's runtime — whose account is the one an agent's calls run as — so the account
|
||||
being measured answered its own measure. The node-engine runs as root, which an agent that cannot become
|
||||
root cannot be. The machine's runtime account is counted as able to become root, always: the mesh's
|
||||
acting tools give it passwordless sudo.
|
||||
- **The router asks `root-free`** for its own machine and the answering channel's, when it shows an
|
||||
approve option and again when an approve answer arrives; an error, a missing machine, or a router that does
|
||||
not know its own machine is not a pass. **The controller asks it too when it composes**, and strips
|
||||
`verified-sender` from a kind without a pass; the router's live question covers a pass that ends between
|
||||
compositions. The `root-not-free` condition is raised on the same judgement, so the condition and the
|
||||
verb never disagree.
|
||||
- *Added after the confirmation review of 2026-10-09.* **Only the serving controller answers `root-free`.**
|
||||
Composed from the controller's own manifest, the module principal of the machine running the controller
|
||||
and that machine's runtime were granted the controller seat's verbs too, and that runtime's credential is
|
||||
one an agent there may hold. The controller's own seat is now served by the controller's principal alone,
|
||||
in grants and in memberships, and a grant test composes every principal and finds exactly one that may
|
||||
answer `root-free`. **A machine waiting for its node-engine's first setuid search is not root-free**
|
||||
(ADR 0266 keeps the self-check quiet for that bound after a restart; `root-not-free` keeps the same quiet,
|
||||
and the verb still answers no). The condition has its own key, apart from ADR 0266's
|
||||
`agent-can-become-root`, which a second probe wrote under the same key with other words. **The router asks
|
||||
`root-free` before it takes its work**, never while holding it — a controller slow to answer held every
|
||||
ask, answer and link behind it — and judges an answer with the word it was given then; inside the work it
|
||||
reads only a word under half a minute old, and an ask that failed drops the last word.
|
||||
|
||||
**3. What remains, said plainly.**
|
||||
|
||||
- **Root on the router's or the channel's machine defeats all of this.** The self-check raises an urgent
|
||||
condition while an agent can become root there without a person; on the control node today it can (the
|
||||
operator's account has passwordless sudo, measured on 2026-10-08). **The operator chose, on 2026-10-08,
|
||||
that agents on the control node run under an account of their own without sudo.**
|
||||
- **An agent reaches root through the mesh's own tools too.** Any machine's tool runner may call any tool
|
||||
on any machine, and the login shell seat's `execute` runs a command as the account the runner runs as —
|
||||
on the control node, the operator's account, with passwordless sudo, which the mesh's acting tools rely
|
||||
on (measured on 2026-10-08). So an agent on any machine can run a root command on the control node
|
||||
without touching its own account. The self-check therefore counts both ways in: an account an agent
|
||||
runs as that can become root without a person, **and** a tool an agent may call that runs a command
|
||||
given to it as an account that can. It fails until both are closed; the second is decided separately
|
||||
(the login shell's reach, and what an agent's calls are granted).
|
||||
- **Since 2026-10-09 the second way is closed on the control node**:
|
||||
[ADR 0268](0268-the-login-shells-execute-is-withheld-on-the-control-node-until-a-call-to-it-needs-a-persons-approval.md)
|
||||
withholds `execute` there through its holder's setting. The self-check counts the login shell only where
|
||||
`execute` is served, failing closed: served while the holder's setting for that machine is `serve`, or
|
||||
while the bus hears the verb answered there, or while the bus cannot be asked. The first way closes with
|
||||
ADR 0266 (hq pull request 225, not yet merged): agents on the control node run under an account of their
|
||||
own, without sudo.
|
||||
- **What cannot be measured is not a pass** (added on 2026-10-09). A machine where a trusted party runs and
|
||||
the node-engine's verdict on its agent account is missing, stale or not given raises the same urgent
|
||||
condition, saying it was not measured, and `root-free` answers no for it.
|
||||
- **A known gap, accepted by the operator on 2026-10-09: the operator's own Telegram client on a machine
|
||||
where agents run as the operator** (issue 344, hq pull request 244, not yet merged). The operator's Telegram may also run on such a
|
||||
machine — their laptop, with Telegram Desktop. There an agent can read the client's data (found readable by
|
||||
everyone) and drive the X display, and tap Approve as the operator. The router cannot tell such a tap from
|
||||
one on the phone. Closing it needs agents under an account of their own on that machine, without access to
|
||||
the operator's X display, and the client's data readable by the operator alone; none of it is built or
|
||||
measured now, and approvals are not held back for it. Until then, approve from the phone.
|
||||
- **A known gap, accepted on 2026-10-09: the controller's credential can rewrite the router's state**
|
||||
(issue 344, hq pull request 244, not yet merged). The controller composes every account and grant on the bus, so a grant withheld from
|
||||
its own credential — no write to the router's identities or asks — would be withheld by the very principal
|
||||
that writes the grants, and undone by its next composition. Moving the router's buckets where the
|
||||
controller cannot write therefore protects nothing against a controller that is not trusted, and is not
|
||||
done; the controller is trusted with every record, and root on the control node is the mesh.
|
||||
- **The controller holds the whole JetStream API** today. Narrowing it to the streams and buckets it owns is
|
||||
planned work of its own; until then the controller is trusted with every record.
|
||||
- **On the X11 desk an agent can read a code as it is shown or typed** (ADR 0234's accepted risk). The hour
|
||||
after a link, the replaced account's approval and the announcements bound what a stolen code can do.
|
||||
|
||||
### 9. What is superseded in ADR 0234
|
||||
|
||||
- **§8 "an ask whose answer performs an action is held by the controller", the controller's verbs
|
||||
`authorise request`, `authorise answer` and `authorisations`, and the controller performing:** the
|
||||
router holds the ask and issues a warrant; the asker performs.
|
||||
- **§3 "the controller's list of the operator's identities"** and §8 "the factors' secrets are the
|
||||
controller's own": the identities and, when built, the TOTP factor and its recovery codes are the
|
||||
router's own state, with ADR 0234's rules on showing them once, at a terminal, never on a channel.
|
||||
- Everything else in ADR 0234 stands: the tiers and proofs, the kinded benches, the content rule, the
|
||||
desk never authorising, long polling, every tier completable on the away channel.
|
||||
|
||||
### 10. The first channels and the order of work
|
||||
|
||||
1. **The controller:** the kinded bench with `kind` on a claim; the `channel` and `intake` seats; the
|
||||
caller-named and per-kind grants; `proof` subjects; the asker's read of its own prefix.
|
||||
2. **The tool runner (mesh-tools)** carries a bundle's seat traffic: publishing an accept or an event
|
||||
with its own name or kind, taking a held seat's work, asking and answering a proof, a compare-and-set
|
||||
on its state, and serving one module alone when it runs on that module's own account.
|
||||
3. **The shared library (mesh-sdk):** an asker's client (ask, cancel, read its state, take its warrants)
|
||||
and a channel's (take show, edit and send for a kind; publish a choice or a link; send a proof).
|
||||
4. **The messenger** becomes the router of this record: asks, tokens, identities, judging, warrants,
|
||||
expiry, linking. Its Telegram client and its desk adapter leave it.
|
||||
5. **`telegram`**, a new module claiming kind `telegram` on both benches: long polling, buttons carrying
|
||||
only a token, `/start` linking, the client's defects of to-be 46 §12 fixed in the move (D1, D4 to D8,
|
||||
D10; D2 and D3 are the router's, D9 the watcher's), its token as its own secret. **`desk-channel`**, a new module claiming kind `desktop`.
|
||||
6. **The controller as an asker**: conditions that need the operator are asked; it acts on warrants and
|
||||
records them; the words "from the mesh MCP server" give way to the router's own line, below.
|
||||
7. **The vault** gains a verb by which the operator gives a module's own secret through a hidden prompt
|
||||
on the desk, so a bot's token never passes through an agent's session. *As built (2026-10-09):* the
|
||||
controller's `give`, through its `secret accept` path. It refuses `broker` and any secret the mesh makes
|
||||
itself; the prompt is written by the desk's launcher from names alone (module, secret, machine), never
|
||||
from words a caller chose, and says the controller asks; the bus lets only the controller publish the
|
||||
prompt's verb; no secret value is accepted through a verb otherwise; and every value given raises the
|
||||
urgent condition `secret-given` on every channel until the operator silences it.
|
||||
**A trusted party's secret is never given at a desk** (the confirmation review of 2026-10-09, finding
|
||||
N1-give). The desk prompt's answer comes back over the desk machine's bus, on the subject that machine's
|
||||
runtime answers, and on a desk machine agents run as the operator, who holds that runtime's credential:
|
||||
an agent called `give` for the bot token and answered the prompt first with a token of its own, sealed to
|
||||
the call's key, and the channel the operator's answers are proven on became the agent's. Now the secret of
|
||||
a module running as an account of its own (the router, Telegram, the desk channel) is refused at the desk
|
||||
before anybody is asked, naming the line to type at the controller's terminal instead —
|
||||
`nox secret accept <node> telegram telegram-token` (`nox` is `mesh-cli`'s alias; on the control-node,
|
||||
from the operator's own account, it is the controller's terminal, ADR 0272). There it is announced
|
||||
on every channel, the one being replaced among them (it still runs on its old token until the next push),
|
||||
**before** it is kept, and not kept when that announcement fails. A grant test holds that only the desk
|
||||
machine's launcher, carried by that machine's runtime, may answer its prompt; for any other module's
|
||||
secret that remains the accepted risk of a desk where agents run as the operator.
|
||||
|
||||
**Where an ask can be answered is said by the router**, from the channels that qualify: "Answer on
|
||||
Telegram, or from the mesh MCP server." The asker's words never name a channel.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **A channel is a module and nothing else changes when one is added.** Signal or mail is a module
|
||||
claiming a kind and declaring capabilities.
|
||||
- **Any module may ask a person**, through one seat, and acts on the answer with its own grants. ADR
|
||||
0258's desk Silence becomes an ask of the controller's, and the messenger loses its grant to
|
||||
`conditions silence`.
|
||||
- **The controller loses three verbs it never built** and gains an asker's client.
|
||||
- **The router's state grows**: asks, tokens and identities, each a bucket of its own.
|
||||
- **Harder:** an asker must keep which warrants it acted on; the shared library carries it.
|
||||
- **Harder:** the controller composes two new grant rules, and an asking module must use the seat in its
|
||||
manifest to be granted anything.
|
||||
- **The bot token moves** from the messenger's own secret to the `telegram` module's, given at the desk
|
||||
(`give`, §10 step 7). **Nothing the channel or the router trusts is a setting**
|
||||
([issue 342](../04-ISSUES/342-a-password-is-taken-as-a-setting-and-outranks-the-vaults/00-report.md): a
|
||||
password, token or key is not a setting). The operator's Telegram account is the router's identity list,
|
||||
written by linking alone; no chat id is configured anywhere. The two settings these modules keep — the
|
||||
router's time zone and the desk channel's machines — are the terminal's, as every mergeable file's are
|
||||
since [issue 340](../04-ISSUES/340-an-agent-can-plant-a-hook-in-every-agent-session-on-the-mesh/00-report.md).
|
||||
- **Accepted risk:** a linked Telegram account, if taken, can approve (announced, reversible), never
|
||||
destroy. Linking before a TOTP factor exists carries the X11 risk in §7.
|
||||
|
||||
## How it is checked
|
||||
|
||||
| Rule | Checked by |
|
||||
|---|---|
|
||||
| An asker publishes asks only under its own name, and hears only its own warrants | controller grant tests: a module using `operator-channel` is granted `accept.ask.<itself>` and `event.decided.<itself>`, no wildcard; the lab's proof `asks` publishing under another name on a real bus is refused by the server |
|
||||
| Only the router publishes a warrant | controller grant test: `event.decided.*` is in no principal's publish list but the holder's |
|
||||
| A channel publishes only its own kind's answers, links and proofs | controller grant tests over a kinded claim; a second claim of a kind is refused |
|
||||
| A code is never on a stream | controller test: no stream's subjects cover `mesh.seat.*.proof.>` |
|
||||
| A token is used once, expires, and is bound to its ask, option and kind | router tests: the same token twice; a token past its expiry; a token from another kind; a token of a closed ask |
|
||||
| An answer from an identity not linked is refused and said | router test |
|
||||
| A level a channel's declared capabilities do not meet is refused | router tests: an approve option answered from `desktop`; capabilities taken from the controller's record, a channel's own claim ignored |
|
||||
| A second answer to one ask loses | router test: two answers raced; one warrant |
|
||||
| An authorising ask never defaults | router test: expiry publishes `expired`, never an option |
|
||||
| An ask no ready channel can carry is refused at once, in words | router test |
|
||||
| A condition that needs the operator and could not be asked (no router, or the ask refused) is a condition of its own, `asks-undelivered` | controller test (`TestAnAskThatCannotBeDeliveredIsSaid`) |
|
||||
| Linking needs the code shown on the desk, and is said on every channel | router tests: a wrong code, a late code, a second link of a kind |
|
||||
| A new link approves only after one hour; it acknowledges at once | router tests: an approve answer at 59 minutes refused and said, at 60 accepted; an acknowledge at once accepted |
|
||||
| No runner's user, and no principal running as the operator's account, may publish a warrant, write the router's state, or speak for a verified-sender kind | controller composition test over every such grant; registration refuses a manifest asking for it |
|
||||
| `verified-sender` reaches the router only from a holder with its own account on a machine no agent can become root on | controller test stripping the capability; self-check probe raising the condition |
|
||||
| Unlinking and a relink within a day are approve asks answered on the linked account; the notice reaches it first | router tests, with a fake channel that honours the addressee |
|
||||
| No bus variable reaches a bundle | tool runner test over a bundle's environment |
|
||||
| A module of its own account is served by a runtime of its own, as that account, on its own bus credential, and never by the machine's | controller composition test (`runs-as`); registration refuses root, an account the module does not make, secrets it does not own, no broker; the tool runner test that such a runtime serves that module alone |
|
||||
| While an agent can become root without a person where the router or a verified channel runs, it is said and nothing proven there approves | controller probe test (fails with agents as the operator's account, and with the login shell running as an account with passwordless sudo; passes once neither holds); the sudo module's `sudo_escalation` test; router test refusing approval |
|
||||
| A tool call never publishes an intake event or a proof | tool runner test: only a bundle's own channel reaches the seat publish, for its own kind |
|
||||
| The asker acts once per warrant and records the warrant | shared library test; controller test reading the hand-act after a warrant |
|
||||
| A warrant authorises only the act its option bound, for the ask as it was shown | shared library tests (`TestAWarrantAuthorisesOnlyTheActItsOptionBound`); controller test (`TestAWarrantPerformsOnlyTheActItsOptionBound`); router test (the warrant names the ask's digest) |
|
||||
| Only the serving controller answers `root-free`; a machine waiting for its first setuid search is not root-free | controller grant test `TestOnlyTheServingControllerMayAnswerRootFree` (one answerer); `TestAMachineWaitingForItsFirstSetuidSearchIsNotRootFree`; `TestRootNotFreeIsQuietWhileTheFirstSearchRunsAndKeyedApartFromDA` |
|
||||
| A desk unlink in the first hour bars nothing; root lifts any bar (N1-router) | router tests `TestAnUnlinkFromTheDeskInTheFirstHourBarsNothing`, `TestBreakGlassLiftsABarOnRelinking` |
|
||||
| A trusted party's secret is never given at a desk; only the desk machine's launcher answers its prompt (N1-give) | controller tests `TestATrustedPartysSecretIsNeverTakenAtADesk`, `TestOnlyTheDeskMachinesLauncherMayAnswerItsPrompt` |
|
||||
| At the controller's terminal, a trusted party's secret is announced on every channel before it is kept, and not kept when the announcement fails; a machine the mesh does not know is refused before anybody types (N1-give) | controller tests `TestATrustedPartysSecretGivenAtTheTerminalIsAnnouncedBeforeItIsKept` (fails when the announcement before keeping is removed), `TestAGiveNamingAMachineTheMeshDoesNotKnowAsksNobody` |
|
||||
| A silence never takes an approval back (M1) | controller test `TestAnAcknowledgementNeverSharesAnAskWithAnApproval`, with the condition silenced as the controller silences it |
|
||||
| The router never waits on the controller while holding its work | router test `TestTheRootFreeWordIsNotAskedWhileTheWorkIsHeld` |
|
||||
| The self-check counts the login shell only where `execute` is served, and what it cannot measure fails | controller probe tests (`TestTheLoginShellCountsOnlyWhereExecuteIsServed`, the unmeasured case) |
|
||||
| The whole flow, before anyone uses it | the lab's proof `asks` (mesh-lab): a real bus with the controller's composed grants, the router and the Telegram channel in runtimes of their own accounts, a fake Telegram API — approve acted once, decline, expiry, a replayed tap, a tap from another account, a forged warrant, an ask under another name, an ask from a module that may not ask |
|
||||
| The controller holds no channel code | review; the controller's module imports no channel package |
|
||||
| End to end | live acceptance after rollout (mesh-lab `asks/live-acceptance.sh`): a **rehearsal** (`rehearse`), started at the controller's terminal alone, approved on the phone, acted on once and performing nothing; the hand-act reads `warrant rehearsal rehearsal=approve`, by "the operator, as telegram identity …", via "telegram (telegram), user id verified", proofs P1 |
|
||||
| An answer from another account is refused and the operator is told which account tried | router test (`TestEveryAnswerThatIsNotTheOperatorsIsRefusedAndSaid`); the lab's proof |
|
||||
| A message carrying an ask's answers is never folded into a digest | router test (`TestAnAsksMessageWithOneAnswerIsNotFoldedIntoADigest`); the lab's proof, where it was found |
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0234](0234-the-mesh-holds-a-conversation-with-its-operator.md) (the conversation, the tiers and
|
||||
proofs; §3 and §8 in part superseded here)
|
||||
- [ADR 0258](0258-a-desk-click-performs-only-an-acknowledgement-and-an-answer-is-a-one-time-ticket.md)
|
||||
(the desk performs only an acknowledgement; one-time answer tokens)
|
||||
- [ADR 0253](0253-a-condition-says-itself-to-the-operator-in-plain-words-beside-its-summary.md) (a
|
||||
condition's actions and plain words)
|
||||
- [ADR 0246](0246-a-seats-new-verb-is-promised-before-it-is-required.md) (a new seat's verbs are required
|
||||
from the start)
|
||||
- [ADR 0255](0255-the-bar-takes-blocks-as-data-from-the-module-that-knows-the-hardware-placed-where-the-machine-has-it.md)
|
||||
(a generic boundary in front of pluggable modules)
|
||||
- [To-be 46](../03-DESIGN/01-to-be/46-the-conversation-with-the-operator.md) §3, §4, §10, §11 and §13
|
||||
- [Issue 320](../04-ISSUES/320-the-operator-could-not-read-the-meshs-notifications/00-report.md),
|
||||
[issue 324](../04-ISSUES/324-a-desk-click-could-answer-in-the-operators-name/00-report.md)
|
||||
+9
@@ -153,6 +153,15 @@ The table of ADR 0239 decision 2 gains these rows:
|
||||
never say it reached every machine. mesh-delivery itself, on the control node alone, is such a module: a
|
||||
held merge of it stays held for a person, as before this record. Reading "every machine running it" from
|
||||
the first machines when they are all the machines would close it, and is left for a later change.
|
||||
|
||||
> **Progressive insight — 2026-10-09.** *The gap was wider than one machine.* A build the mesh held back for
|
||||
> a person and sent on the operator's word travels by a backlog walk, whose record gives one step per
|
||||
> machine and no "every machine running it" or "the rest running it"; and that walk's last record keeps only
|
||||
> the machines it is done on, which mesh-delivery read for nothing. So a held delivery whose module reached
|
||||
> every machine that way was never covered either
|
||||
> ([issue 351](../04-ISSUES/351-a-held-delivery-whose-module-a-backlog-walk-carried-is-never-covered/00-report.md)).
|
||||
> Since its fix, a finished backlog walk passes the steps it carried, and a passed step on every machine the
|
||||
> held delivery's plan sends the module to counts as sent everywhere. The one-machine case stays as stated.
|
||||
- **A group member waits longer, rather than being held too soon.** A composed check that never answers leaves
|
||||
the member in `published`. Its bound of 30 minutes lists it as stalled. That is the right owner for a check
|
||||
that does not answer.
|
||||
|
||||
@@ -4,6 +4,7 @@ status: designed
|
||||
code: []
|
||||
updated: 2026-10-08
|
||||
decisions:
|
||||
- 02-DECISIONS/0259-an-ask-is-a-message-on-the-bus-any-channel-answers-it-and-the-asker-acts-on-a-warrant.md
|
||||
- 02-DECISIONS/0253-a-condition-says-itself-to-the-operator-in-plain-words-beside-its-summary.md
|
||||
- 02-DECISIONS/0234-the-mesh-holds-a-conversation-with-its-operator.md
|
||||
- 02-DECISIONS/0227-the-core-holds-nine-rules-each-checked-and-is-built-to-them-in-six-phases.md
|
||||
@@ -13,8 +14,8 @@ decisions:
|
||||
|
||||
**The mesh, its modules and its agents tell the operator things and ask them things; the operator
|
||||
answers, or writes first. Each exchange travels on a channel whose declared capabilities can carry it,
|
||||
chosen by where the operator is working. An answer that performs an action is checked and performed by
|
||||
the controller, never by the channel or the asker** ([ADR 0234](../../02-DECISIONS/0234-the-mesh-holds-a-conversation-with-its-operator.md),
|
||||
chosen by where the operator is working. An answer that performs an action is judged by the router, which
|
||||
issues a warrant; the asker performs it with its own grants, never the channel or the router** ([ADR 0234](../../02-DECISIONS/0234-the-mesh-holds-a-conversation-with-its-operator.md),
|
||||
from [research 028](../../01-RESEARCH/028-the-meshs-output-channel/00-overview.md)).
|
||||
|
||||
This design grows the minimal output channel of [to-be 45](45-a-core-that-cannot-fail-silently.md) §5.
|
||||
@@ -28,14 +29,13 @@ That form stays the first step; §13 below is the order in which it becomes this
|
||||
controller's conditions ──┐ messages and asks ┌──► channel/telegram
|
||||
modules (notify) ────────┼──► required capability → work context ──┼──► channel/desktop
|
||||
agents (ask) ────────┤ → severity → escalation └──► channel/<next kind>
|
||||
controller (authorise │ presence (current state only)
|
||||
any module (ask) ─────────┤ presence (current state only)
|
||||
request) ────────────────┘ ▲ ▲
|
||||
│ │ answer (ordinary asks) intake bench (in)
|
||||
seen ───┘ └────────────────────────────── intake/telegram
|
||||
intake/desktop
|
||||
CONTROLLER ◄──── authorise answer (authorising asks, intake holders only) ─┘
|
||||
│ checks capabilities, identity, code, digest; performs; records the hand-act
|
||||
└──► ask-answered
|
||||
ASKER (any module) ◄── warrant on decided.<asker> ── ROUTER judges: token, identity,
|
||||
│ acts with its own grants; records the warrant capabilities, level; one warrant per ask
|
||||
|
||||
watcher's watcher (outside the seats, own bot, not on the control node) ──► Telegram directly
|
||||
self-check and watcher ──► outside dead-man service
|
||||
@@ -91,14 +91,16 @@ bot) is held by one module claiming both seats under one kind.
|
||||
| `kind` | the holder's kind |
|
||||
| `what` | `message` (written first), `choice` (a button or reaction), `reply` (written in an ask's thread), `mail`, `call` (a webhook), `seen` (activity, for the work context) |
|
||||
| `sender` | the identity on that service, and whether the service authenticated it |
|
||||
| `trusted` | whether the sender is on the **controller's** list of the operator's identities. A holder never sets it: a raw intake event is untrusted by construction, and only the router stamps it, from the controller's answer (§4) |
|
||||
| `trusted` | whether the sender is on the **router's** list of the operator's identities. A holder never sets it: a raw intake event is untrusted by construction, and only the router stamps it, from the controller's answer (§4) |
|
||||
| `conversation` | an opaque handle; sending on it reaches the same chat, room or thread |
|
||||
| `in-reply-to` | the ask or message it answers, if any |
|
||||
| `payload` | the text, or the option's token. **Never a code** (§10) |
|
||||
| `at` | when |
|
||||
|
||||
**Answers go to the ask's owner by request and reply**: the router's `answer` for an ordinary ask, the
|
||||
controller's `authorise answer` for an authorising one; both refuse an answer that is not trusted.
|
||||
**Answers arrive only as intake events of the holder's own kind** (*amended 2026-10-08, ADR 0259*): a
|
||||
`choice` carries the router's token and the sender as the service authenticated it, and the router judges
|
||||
it (§10). A typed code goes by request and reply on the seat's `proof` subject, never in an event. There
|
||||
is no `answer` tool: any tool runner may call any tool, so a tool call proves nothing about its caller.
|
||||
**Everything else is an event on the seat**, taken by the router, which re-emits it stamped (§4): a
|
||||
trusted `message` as an **operator message**, addressed (§5); anything untrusted as **input**, for
|
||||
consumers that accept untrusted input (a later mail rule takes `mail`). The router takes `seen` for the
|
||||
@@ -109,14 +111,13 @@ grants it (the open gap of [to-be 32](32-what-a-module-declares.md) §1).
|
||||
|
||||
## 4. Who counts as the operator
|
||||
|
||||
- **Only a sender on the controller's list of the operator's identities is the operator.** The list
|
||||
is kept per intake kind in the controller's state. While no factor exists, an identity is enrolled
|
||||
- **Only a sender on the router's list of the operator's identities is the operator**
|
||||
(*amended 2026-10-08, ADR 0259:* the router's, not the controller's). The list is kept per intake kind in
|
||||
the router's state, and changed only by linking and unlinking, never by a setting. While no factor exists, an identity is enrolled
|
||||
only at a terminal; once one does, adding or removing an identity is a destroy ask (§10). Telegram's linking verb
|
||||
(§11) produces such an ask.
|
||||
- **The controller serves an identity check** (kind and sender in, trusted or not out). The router keeps
|
||||
the answers as current state, refreshed when the controller announces a change to the list, and
|
||||
stamps every envelope it re-emits with `trusted`. **No holder's word is taken for it**, and the
|
||||
controller checks again itself before authorising (§10).
|
||||
- **The router checks every sender against its own list** and stamps every envelope it re-emits with
|
||||
`trusted`. **No holder's word is taken for it.**
|
||||
- **An untrusted envelope** may start work for a consumer that declares it accepts untrusted input. It
|
||||
can never answer an ask, authorise, or be delivered to an agent as the operator's words. An untrusted
|
||||
sender writing to the bot is told nothing beyond a fixed line, and the attempt is a warning condition
|
||||
@@ -208,10 +209,10 @@ The event **`ask-answered`**, with the ask's id and any conversation handle. An
|
||||
|
||||
### The router's verbs and events
|
||||
|
||||
`ask`, `ask cancel`, `asks`, `answer` (intake holders only), on the `operator-channel` seat beside its
|
||||
existing `notify`, `open` and `history`. Events `ask-opened`, `ask-answered`, `ask-closed`. The router
|
||||
owns every ask that authorises nothing; an authorising ask is owned by the controller (§10) and carried
|
||||
by the router like any other.
|
||||
*Amended 2026-10-08, ADR 0259:* an ask and its cancel are accepted on the `operator-channel` seat under
|
||||
the asker's name; the answer, a warrant or the ask's end, is the seat's event `decided.<asker>`; the tool
|
||||
`asks` lists them, beside the existing `notify`, `open` and `history`. The router owns every ask,
|
||||
authorising or not (§10).
|
||||
|
||||
## 7. The capability vocabulary, `channel-capabilities/1`
|
||||
|
||||
@@ -234,9 +235,9 @@ withdrawn from routing and reported until it passes. A new word is a new version
|
||||
| | `threads` | an answer is tied to what it answers | a reply to A carries A in `in-reply-to` |
|
||||
| | `operator-first` | the operator can write unprompted | a simulated message yields a `message` envelope |
|
||||
| trusting | `verified-sender` | the answer came from the operator's own account, through a holder no agent shares | a choice from an identity not on the list is dropped and reported; placement checked |
|
||||
| | `exact-render` | the ask is shown as the controller rendered it | rendered text equals the controller's, byte for byte |
|
||||
| | `code-factor` | a typed code reaches the controller by request and reply, never judged by the holder | a code never appears in an event; it is deleted from the conversation where the service allows |
|
||||
| | `key-factor` | a security key's assertion over the controller's challenge reaches the controller | the challenge is the controller's; the signature is checked by the controller |
|
||||
| | `exact-render` | the ask is shown as the router rendered it | rendered text equals the router's, byte for byte |
|
||||
| | `code-factor` | a typed code reaches the router by request and reply, never judged by the holder | a code never appears in an event; it is deleted from the conversation where the service allows |
|
||||
| | `key-factor` | a security key's assertion over the router's challenge reaches the router | the challenge is the router's; the signature is checked by the router |
|
||||
|
||||
What the first holders declare:
|
||||
|
||||
@@ -298,7 +299,7 @@ is not exempted.
|
||||
- **Allowed in a message's words:** roles, words, and the mesh's own machine names.
|
||||
- **Refused:** domains, addresses, paths, and anything shaped like a secret — in words, labels and
|
||||
references alike.
|
||||
- **A refusal is said to the sender** — the `notify`, `ask` or `authorise request` call is refused naming
|
||||
- **A refusal is said to the sender** — the `notify` call or the ask is refused naming
|
||||
the offending part, so the asker can rephrase or move the specific into a reference — and raises
|
||||
`channel-refused`. The offending part is never sent to a channel.
|
||||
|
||||
@@ -316,17 +317,61 @@ is not exempted.
|
||||
|
||||
## 10. Asks that authorise
|
||||
|
||||
### The verb table and the controller's verbs
|
||||
*Amended 2026-10-08, [ADR 0259](../../02-DECISIONS/0259-an-ask-is-a-message-on-the-bus-any-channel-answers-it-and-the-asker-acts-on-a-warrant.md):* **the router holds every ask, authorising or not, and
|
||||
judges each answer; the asker performs the chosen option on a warrant.** The controller keeps no channel,
|
||||
no identity and no factor: it asks and acts like any other module. What follows in this section reads
|
||||
with these replacements.
|
||||
|
||||
- **The controller's verb definition gains `authorises`:** the tier, and the arguments that make up
|
||||
### Asks, warrants and answers on the bus
|
||||
|
||||
`<asker>` is the asking module's name, `<kind>` a channel's kind. Each subject's last token names its
|
||||
publisher, and the bus grants it to that publisher alone.
|
||||
|
||||
| What | Subject or bucket | Kept in | Publisher | Reader |
|
||||
|---|---|---|---|---|
|
||||
| an ask | `mesh.seat.operator-channel.accept.ask.<asker>` | the seat's work queue, 7 days | the asker, its own name | the router |
|
||||
| a cancel | `mesh.seat.operator-channel.accept.cancel.<asker>` | the same | the asker, its own name | the router |
|
||||
| asks and their state | the router's bucket `asks`, key `<asker>.<id>` | its state; closed records 30 days | the router | the router; each asker its own prefix |
|
||||
| answer tickets | in each ask's record: one per option and kind, used once | its state | the router | the router |
|
||||
| the operator's identities | the router's bucket `identities`, key `<kind>.<identity>` | its state | the router | the router |
|
||||
| a warrant, or an ask's end | `mesh.seat.operator-channel.event.decided.<asker>`, message id `<asker>.<id>` | EVENTS, deduplicated | the router | the asker, its own name |
|
||||
| show, edit, say | `mesh.seat.channel.accept.<show\|edit\|send>.<kind>` | the seat's work queue, one worker per kind | the router | that kind's holder |
|
||||
| a choice, a link asked, a failed message, standing | `mesh.seat.intake.event.<choice\|link\|failed\|standing>.<kind>` | EVENTS | that kind's holder | the router |
|
||||
| which kinds exist, and what each promises | the router's membership (`seat-traffic.kinds`) | ASSIGNMENTS | the controller, from the claims | the router |
|
||||
| a code | `mesh.seat.intake.proof.code.<kind>` | nothing: core request and reply | that kind's holder | the router answers |
|
||||
|
||||
**An ask** carries an id the asker chose; the question in plain words; at most four options, each a label,
|
||||
what it does in plain words, its assurance level (`acknowledge`, `approve`, `destroy`) and an id; who may
|
||||
answer (`operator`); an expiry; what happens on expiry, in words; and optionally what it is about (a
|
||||
condition's key), so a newer ask about it replaces the older. It is refused at once, on its asker's
|
||||
`decided` subject, when its words break a rule, when no ready channel can carry one of its options' levels,
|
||||
or when the asker already has three open.
|
||||
|
||||
**A warrant** carries the ask's id and asker, the option, `by` ("the operator, via telegram, user id
|
||||
verified"), the channel module, the proofs, the level met and when. No secret. An ask that ends without one
|
||||
ends `expired`, `cancelled`, `replaced` or `refused`, on the same subject.
|
||||
|
||||
**The router judges a `choice` of kind K with token T from sender S**, refusing at the first failure and
|
||||
saying it on K's channel: T is known, unused, unexpired and issued for K; the ask is open and T's option is
|
||||
its own; S is linked for K; K's capabilities, from the controller's record of its claim, meet the option's
|
||||
level; for `approve`, S was linked at least an hour ago; then the ask is closed by compare-and-set and the
|
||||
warrant published. Every copy is edited to the outcome.
|
||||
|
||||
**The asker** acts once per ask id, only on a warrant from its own subject, with its own grants, and its
|
||||
record of the act names the warrant (`by`, `via`, `ask`, `proofs`). The hand-act log counts it as a person's
|
||||
decision.
|
||||
|
||||
**The verbs the controller was to serve** (`authorise request`, `authorise answer`,
|
||||
`authorisations`) are not built: an ask on the seat replaces the first, an intake `choice` the second, and
|
||||
the router's tool `asks` the third. The tiers, the proofs, the direct-call rules and the factors stand, and
|
||||
the factors are the router's state.
|
||||
|
||||
### The verb table
|
||||
|
||||
- **Each asker's verb definition gains `authorises`:** the tier, and the arguments that make up
|
||||
the exact state a person must see (for `retire approve`, the set of consumers).
|
||||
- **`authorise request`** — callable by anyone (an agent, the router on a condition's behalf). Carries
|
||||
the verb and its exact arguments, why, and optionally a conversation handle. The controller renders
|
||||
the ask, stores it with an opaque id, its expiry and a digest of the state shown, and emits
|
||||
`ask-opened` with itself as owner. **Nothing is performed.**
|
||||
- **`authorise answer`** — granted to intake holders only. Carries the ask's id, the option, the
|
||||
sender's identity, and the code or key assertion where the tier needs them.
|
||||
- **`authorisations`** — open and recent authorising asks.
|
||||
- **An authorising ask** is published by its asker on the seat (above); the router renders it and stores
|
||||
it with its expiry. **Nothing is performed** until the asker holds a warrant.
|
||||
|
||||
### What may be authorised, and its tier
|
||||
|
||||
@@ -343,7 +388,7 @@ is not exempted.
|
||||
|
||||
- **P1, a verified sender:** a Telegram tap from the operator's own account, through a holder placed
|
||||
where no agent runs as the operator.
|
||||
- **P2, a TOTP code** from the operator's authenticator app, verified by the controller. Valid for the
|
||||
- **P2, a TOTP code** from the operator's authenticator app, verified by the router. Valid for the
|
||||
current or previous 30-second step; accepted once. **This is the proof everything above acknowledge
|
||||
can rest on.**
|
||||
- **P3, a security key's touch bound to the ask** (the challenge is a hash of the ask's id and the
|
||||
@@ -372,26 +417,27 @@ is not exempted.
|
||||
5. **No agent authorises.** An agent asks. Agents' grants hold no authorising verb.
|
||||
6. **Only a trusted sender authorises** (§4); an untrusted answer is refused before any other check.
|
||||
|
||||
### The controller's checks, in order, refusing at the first failure
|
||||
### The router's checks, in order, refusing at the first failure
|
||||
|
||||
1. The ask is open and unexpired.
|
||||
2. The caller holds an intake kind — by the controller's own seat records, never the request's claim.
|
||||
3. That kind's declared capabilities, from the controller's records, satisfy the tier, and the proofs
|
||||
2. The answer came on an intake kind's own subject — which the bus grants that kind's holder alone.
|
||||
3. That kind's declared capabilities, from the controller's record of its claim, satisfy the tier, and the proofs
|
||||
present are enough.
|
||||
4. A P1 answer's sender is on the controller's list of the operator's identities for that kind.
|
||||
4. A P1 answer's sender is on the router's list of the operator's identities for that kind, and for
|
||||
approve was linked at least an hour ago.
|
||||
5. A code is valid for the current or previous step, and unused.
|
||||
6. A key assertion verifies against the enrolled credential, over this ask's challenge, with user
|
||||
presence.
|
||||
7. The state now has the digest it had when shown; otherwise the ask is void and a new one is
|
||||
requested.
|
||||
7. The asker checks that the state still is what it asked about before it acts; otherwise it refuses
|
||||
the warrant and asks again.
|
||||
|
||||
Then it performs as itself, records the hand-act, closes the ask with a compare-and-set so a second
|
||||
answer loses, and emits `ask-answered`. Every copy is edited to the outcome and its buttons removed.
|
||||
Then it closes the ask with a compare-and-set so a second answer loses, and publishes the warrant; the
|
||||
asker performs as itself and records the hand-act. Every copy is edited to the outcome and its buttons removed.
|
||||
|
||||
### Direct calls
|
||||
|
||||
`retire approve|reject`, `cleanup delete`, and `pin` while `binding-kept` names it refuse any caller
|
||||
except through `authorise answer`, or **break-glass** at the mesh MCP server with a code — recorded and
|
||||
except on a warrant, or **break-glass** at the mesh MCP server with a code — recorded and
|
||||
announced on every channel as break-glass. `retire approve` takes **`expect`**, the set it approves,
|
||||
and refuses if the set now differs. `conditions silence` stays callable; a silence an agent sets is
|
||||
said on the away channel, with its why.
|
||||
@@ -403,10 +449,10 @@ id) and **`proofs`** (which of P1, P2, P3). `by` reads "the operator, as <kind>
|
||||
|
||||
### The factors
|
||||
|
||||
The operator's identities, the TOTP seed, its recovery codes and any enrolled key are the controller's
|
||||
own state. The seed is made by the mesh and shown once, as a URI, to a plain terminal — never through a
|
||||
The operator's identities, the TOTP seed, its recovery codes and any enrolled key are the router's
|
||||
own state (ADR 0259). The seed is made by the mesh and shown once, as a URI, to a plain terminal — never through a
|
||||
channel or an event. Re-enrolling a factor is a destroy ask, except by recovery below. A code travels
|
||||
only by request and reply from an intake holder to the controller, and is deleted from the conversation
|
||||
only by request and reply from an intake holder to the router, and is deleted from the conversation
|
||||
where the service allows.
|
||||
|
||||
### Recovering the factor
|
||||
@@ -414,33 +460,43 @@ where the service allows.
|
||||
A lost phone must not lock the mesh's only person out for good, and re-enrolling is a destroy ask that
|
||||
needs the lost factor's code. So:
|
||||
|
||||
- **`factor enrol`**, at a terminal: the controller makes the seed and **ten one-time recovery codes**,
|
||||
- **`factor enrol`**, at a terminal: the router makes the seed and **ten one-time recovery codes**,
|
||||
shows them once together, and stores the codes **only as hashes**.
|
||||
- **A recovery code counts as one P2 proof, once.** It may stand in for the app's code in any ask.
|
||||
- **`factor recover`**, at a terminal, given a recovery code: a new seed and ten new codes replacing the
|
||||
remaining ones. Recorded as a hand-act, and **announced loudly on every channel**.
|
||||
- **The count left** is shown by `factor status` and in `authorisations`; the self-check warns when
|
||||
- **The count left** is shown by `factor status` and in the router's `asks`; the self-check warns when
|
||||
three or fewer remain.
|
||||
- **The last resort:** root on the control node — at it, or by its SSH key — runs `factor enrol
|
||||
--break-glass` there. It talks to the controller locally, never over the bus; the same verb over the
|
||||
--break-glass` there. It talks to the router locally, never over the bus; the same verb over the
|
||||
bus is refused. Recorded and announced as break-glass.
|
||||
- **No tier requiring P2 is enabled until recovery codes exist.** The controller refuses the setting.
|
||||
- **No tier requiring P2 is enabled until recovery codes exist.** The router refuses the setting.
|
||||
|
||||
## 11. The first holders
|
||||
|
||||
### Telegram — kind `telegram`, both seats, the away channel
|
||||
|
||||
- **Its own module**, holding the bot token and chat id as its own secrets; the router keeps no
|
||||
Telegram client.
|
||||
- *Revised after the security review of 2026-10-08 (ADR 0259 §8, not yet accepted):* the router and every
|
||||
channel declaring `verified-sender` run as their own process and service account with a bus account of
|
||||
their own, never carried by a tool runner; the controller passes `verified-sender` only for a holder so
|
||||
placed, on a machine where no agent's account can become root without a person.
|
||||
- **Its own module**, `telegram`, holding the bot token as its own secret; the router keeps no Telegram
|
||||
client. *Amended 2026-10-08, ADR 0259:* **it runs on the control node under its own account `telegram`**,
|
||||
never root or the operator's: a runtime of its own serves it as that account, on a bus account of its own
|
||||
(`runs-as`), and its state directory, token and bus credential belong to it. If an agent holds root there,
|
||||
none of this holds; the self-check raises a condition while an agent can become root without a person, by
|
||||
its own account or through a tool that runs its command as one that can.
|
||||
- **Long polling**, never a webhook; the update offset kept in its own state.
|
||||
- **Buttons** carry only an opaque `a1:<ask>:<option>` (Telegram allows 64 bytes); everything else is
|
||||
in the router or the controller. A tap is dropped and reported unless both the user and the chat are
|
||||
- **Buttons** carry only the router's one-time token (Telegram allows 64 bytes); everything else is in
|
||||
the router. A tap is dropped and reported unless both the user and the chat are
|
||||
the operator's; otherwise it is acknowledged at once (the phone shows a spinner until then), handed
|
||||
on, and the message edited to the outcome.
|
||||
- **A destroy ask** answers the tap with a forced-reply prompt for the code; the holder reads it,
|
||||
deletes it from the chat, and hands it to the controller by request and reply.
|
||||
- **A linking verb** with a one-time deep-link code replaces reading the chat id by hand. Linking an
|
||||
identity adds it to the controller's list, so after the first it is a destroy ask (§4).
|
||||
deletes it from the chat, and hands it to the router by request and reply.
|
||||
- **Linking** (*amended 2026-10-08, ADR 0259*): the operator sends `/start`; the router shows a six-digit
|
||||
code on the desk for ten minutes; the operator sends it to the bot, and the router links the account.
|
||||
One identity per kind. **A new link acknowledges at once and approves only after one hour**, and every
|
||||
link and unlink is said on every channel. Once a TOTP factor exists, linking is a destroy ask (§4).
|
||||
- **A sender not on the list** is answered with one fixed line and handed on only as untrusted input.
|
||||
- **Placement:** where no agent runs as the operator. Its `verified-sender` depends on it, and the
|
||||
self-check reads the placement.
|
||||
@@ -456,11 +512,12 @@ needs the lost factor's code. So:
|
||||
*Amended 2026-10-08, [ADR 0258](../../02-DECISIONS/0258-a-desk-click-performs-only-an-acknowledgement-and-an-answer-is-a-one-time-ticket.md):*
|
||||
only an acknowledgement (Details, silencing) is offered so: any process of the account can choose an
|
||||
action, so a click is never taken for the operator's word. Every other answer waits for phase 5.
|
||||
- The router's desktop adapter, holding `channel/desktop` and `intake/desktop`, turns that into a
|
||||
`choice` envelope.
|
||||
- *Amended 2026-10-08, ADR 0259:* a module of its own, **`desk-channel`**, holds `channel/desktop` and
|
||||
`intake/desktop`, sends through each machine's `node-notifier` and turns dunst's `action-chosen` into a
|
||||
`choice` envelope. The desk declares no `verified-sender`, so it answers `acknowledge` options only.
|
||||
- **For `text`, `number`, `date` and codes**, the notification's single action opens the
|
||||
`node-launcher` seat's prompt; what is typed returns as a `reply` — or, for a code, straight to the
|
||||
controller by request and reply.
|
||||
router by request and reply.
|
||||
- **The screen-lock holder emits lock and idle changes** for the work context.
|
||||
- It carries every ordinary ask with no account anywhere, and an authorising one only with a code (or
|
||||
an enrolled key's touch).
|
||||
@@ -500,8 +557,32 @@ Each phase ends at its own *done when*. Owning repositories from [`repos.md`](..
|
||||
| **2 — The desk's actions** | `node-notifier.send` gains actions; the dunst holder emits the chosen one; the launcher's prompt returns typed text; the screen-lock holder emits lock and idle changes | mesh-catalog | a notification with two actions returns the chosen token as an event; locking emits an event |
|
||||
| **3 — The seats, and the router** | the kinded bench in the seat set; `kind` and `capabilities` on a claim; `channel-capabilities/1` and its contract tests; publishing on a seat's event subjects and its grant; `channel` and `intake` seats; the output seat's holder becomes the router and holds `channel/desktop` and `intake/desktop`; presence as current state; the operator's identity list and identity check in the controller, the router's `trusted` stamping and untrusted input (§4); the content rule allowing machine names, references and `detail`, refusals naming the offending part (§9) | mesh-controller (seat set, claim fields, registration refusals, grants, the identity list and check); mesh-sdk and mesh-tools (seat-event publishing in the shared library and the tool runner); mesh-catalog (the seats' definitions, the router) | the catalogue refuses an unknown capability and a second holder of one kind; a message is routed to the desk by capability and context; an envelope from an identity not on the list is re-emitted untrusted; a path is refused to its sender by name, and travels as a reference the desk opens |
|
||||
| **4 — Asks, and operator messages** | `ask`, `ask cancel`, `asks`, `answer`; the events; kinds, life, limits, batching, history; escalation; addressing, the register of agents with kept messages, the responder; the untrusted-input rule in every agent module's instructions (§5) | mesh-catalog | the router tests of ADR 0234 pass; live: an agent's question answered at the desk, and with the desk locked, on the phone; `@mesh status` answered; a message to a stopped agent kept and said |
|
||||
| **5 — Authorise, with TOTP** | `authorises` in the verb table; `authorise request`, `authorise answer`, `authorisations`; the seven checks; direct calls refused, break-glass; `retire approve` takes `expect`; the hand-act fields; the TOTP seed as the controller's state, enrolled at a terminal with ten recovery codes; `factor recover`, `factor status`, the local-only break-glass enrolment; no P2 tier before recovery codes exist; optional key enrolment; the self-check probe for the away channel; agents' grants lose the authorising verbs | mesh-controller; mesh-catalog (agents' grants, the desk's code prompt) | one controller test per refusal passes; at the desk, approve needs a code; a recovery drill re-enrols with a recovery code, and is announced |
|
||||
| **6 — Telegram live** | the `telegram` module holding both seats, with buttons, replies, forced-reply codes and the linking verb, placed where no agent runs as the operator; the operator's two bots configured; the watcher assigned off the control node; the dead-man ping | mesh-catalog | the self-check says the away channel carries every tier; live drills: an approve and a destroy on a test condition answered on the phone, with the hand-acts read |
|
||||
| **5a — Asks on the bus** (ADR 0259) | the kinded bench, caller-named verbs, `proof` subjects and the records users read (controller); a bundle's seat traffic, compare-and-set state and its own account (tool runner); the asker's and channel's types (shared library); the router's asks, tokens, identities, judging, warrants, expiry and linking with its hour (messenger); `telegram` and `desk-channel`; the controller asking for its conditions' actions and acting on warrants | mesh-controller, mesh-tools, mesh-sdk, mesh-catalog | a held delivery released from the phone; the hand-act reads "the operator, via telegram, user id verified"; a desk Silence goes through an ask |
|
||||
| **5 — Authorise, with TOTP** | `authorises` in the verb table; the router's checks for the code; direct calls refused, break-glass; `retire approve` takes `expect`; the hand-act fields; the TOTP seed as the router's state, enrolled at a terminal with ten recovery codes; `factor recover`, `factor status`, the local-only break-glass enrolment; no P2 tier before recovery codes exist; optional key enrolment; the self-check probe for the away channel; agents' grants lose the authorising verbs | mesh-controller; mesh-catalog (agents' grants, the desk's code prompt) | one controller test per refusal passes; at the desk, approve needs a code; a recovery drill re-enrols with a recovery code, and is announced |
|
||||
| **6 — Telegram live** | the `telegram` module (built in 5a) with replies and forced-reply codes, under its own account on the control node; the operator's two bots configured; the watcher assigned off the control node; the dead-man ping | mesh-catalog | the self-check says the away channel carries every tier; rehearsals: an approve and a destroy answered on the phone, with the hand-acts read |
|
||||
|
||||
**Switching phase 5a on: the operator's steps, in order** (amended 2026-10-09, ADR 0259). Each line is typed with
|
||||
`nox` (`mesh-cli`) on the control-node, from the operator's own account, which is the controller's terminal
|
||||
(ADR 0272):
|
||||
|
||||
1. The pull requests merge in their order, each after its checks pass on the merge result; nothing is assigned
|
||||
by the merges.
|
||||
2. The agent account (ADR 0266) runs on the control-node and the login shell's `execute` is withheld there
|
||||
(ADR 0268): until both hold, the self-check's `agent-root` or `root-not-free` stays open and Telegram only
|
||||
acknowledges.
|
||||
3. `messenger`, `telegram` and `desk-channel` are assigned to the control-node (`desktop-machines` set on
|
||||
`desk-channel` in the same step), and the control-node is pushed.
|
||||
4. A bot is made with Telegram's BotFather, and its token is given at the controller's terminal:
|
||||
`nox secret accept <control-node> telegram telegram-token`. It is announced on every channel before it is
|
||||
kept. Then the control-node is pushed again, so the channel holds it.
|
||||
5. The operator sends `/start` to the bot from their own account, in a private chat, and sends back the six digits
|
||||
the desk shows. Nobody configures a chat id: the router's identity list is written by linking alone.
|
||||
6. An hour later, `live-acceptance.sh` (mesh-lab `asks/`) runs a rehearsal: a question on the phone, approved
|
||||
there, read back from the hand-act log.
|
||||
|
||||
Until `mesh-cli` carries standard input (ADR 0272, a later step), step 4's line cannot read the token from the
|
||||
terminal; it is given with `--from` a file the controller's own account reads and nobody else does, removed
|
||||
right after.
|
||||
|
||||
**Not touched:** the node engine (`mesh-host` repository). Presence comes from the screen-lock seat's
|
||||
holder, not the engine; if a later measure shows the engine must relay it, that is a change here.
|
||||
|
||||
+53
@@ -0,0 +1,53 @@
|
||||
---
|
||||
status: located
|
||||
opened: 2026-10-09
|
||||
located-in: [mesh-catalog modules/dnsmasq, mesh-controller cmd/mesh-controller (gate.go, machine_network.go, probes.go), mesh-controller internal/conditions, mesh-catalog modules/docker, mesh-catalog modules/netcheck]
|
||||
fixed-by:
|
||||
amended-design:
|
||||
---
|
||||
|
||||
# 348. A resolver that stopped answering ran on unnoticed, and builds sent after it were blamed for it
|
||||
|
||||
## Symptom
|
||||
|
||||
On 2026-10-09, from 10:57:57 to 11:18:12 UTC, the mesh's resolver on the control node refused every
|
||||
question asked on its private address. Its unit (`dnsmasq.service`, module `dnsmasq`, seat
|
||||
`mesh-dns-resolver`) stayed active and running the whole time, and its journal said nothing. Every
|
||||
question to that address got "connection refused". Loopback still answered.
|
||||
|
||||
What it cost:
|
||||
|
||||
- The certificate authority (`step-ca`, on the machine's own network stack) failed its health check from 10:59:37 UTC.
|
||||
- The route proxy logged "lookup <machine>.internal on <private address>:53: … connection refused" for
|
||||
every proxied request to a machine name, for twenty minutes.
|
||||
- The control node's own network check failed (`machine.<control node>.network`, raised 10:58:45 UTC).
|
||||
- The controller (build 5438d7ff) and the node-engine (build 8f763ee9), sent to their first machines at
|
||||
10:59:34 UTC, failed their gate at 11:10:29 UTC with "raised since it was sent:
|
||||
machine.<control node>.network" and were put back. The fault began before they were sent.
|
||||
- The second resolver of the mesh, on the home server, did not take over the route proxy's lookups.
|
||||
- D2, the controller's probe that every resolver answers, raised the resolver only at 11:06:00 UTC, eight
|
||||
minutes after it stopped answering.
|
||||
- A person restarted the unit at 11:18:12 UTC by hand, and that fixed it. Nothing in the mesh would have.
|
||||
|
||||
Five seconds before it stopped answering, at 10:57:52 UTC, a merge check on the control node removed its
|
||||
throwaway containers, and the runtime tore down their bridge network and its virtual links.
|
||||
|
||||
## Why it is an issue
|
||||
|
||||
Three rules of the mesh did not hold.
|
||||
|
||||
- **Failure must be loud** (ADR 0240). A module is judged healthy when its unit runs. Nothing asked the
|
||||
resolver a question. A resolver that answers nothing on the address every machine and container uses
|
||||
read as healthy for twenty minutes.
|
||||
- **A send is judged on what it caused** (ADR 0236, issue 281). The gate failed two builds for a fault that
|
||||
began before they were sent.
|
||||
- **Nothing restarts on health; a healer that does is its own record** (ADR 0240 rule 6, ADR 0231). No
|
||||
healer and no module repairs a resolver that stops answering, so the only repair was a person's.
|
||||
|
||||
The operator's acceptance for the fix: if the mesh's name service stops answering on a node, the mesh
|
||||
notices within a minute or two, restarts it by itself, and tells the operator only if that did not help.
|
||||
A build is never put back for a problem that started before it was sent.
|
||||
|
||||
The diagnosis is in [`01-diagnosis.md`](01-diagnosis.md).
|
||||
|
||||
The fixes are in review: novox/mesh-controller pull request 179 and novox/mesh-catalog pull request 161.
|
||||
+187
@@ -0,0 +1,187 @@
|
||||
# 348. Diagnosis
|
||||
|
||||
## 2026-10-09
|
||||
|
||||
**The record.** A search of the records for "bind-dynamic", "raised since it was sent" and "dnsmasq"
|
||||
found issue 110 (why the resolver is told an address and not an interface), issue 281 (a machine's
|
||||
condition read as a module's), issue 277 (D2 asks again before it says anything) and research 032's
|
||||
evidence that the runtime's event history is about a minute long. None of them is about a resolver that
|
||||
stops answering while it runs.
|
||||
|
||||
### Timeline (UTC)
|
||||
|
||||
| time | what was seen | where |
|
||||
|---|---|---|
|
||||
| 10:57:52 | a merge check's throwaway containers removed, their bridge network and virtual links torn down | the runtime, on the control node |
|
||||
| 10:57:57 | the route proxy's first "lookup … on <private address>:53: … connection refused" | the route proxy's log |
|
||||
| 10:58:36 | `machine.<home server>.network` raised: one of its two resolvers does not answer | conditions |
|
||||
| 10:58:45 | `machine.<control node>.network` raised; the home server's cleared, held under it | conditions |
|
||||
| 10:59:34 | the controller 5438d7ff and the node-engine 8f763ee9 sent to their first machines | the walks |
|
||||
| 10:59:37 | `module.step-ca.<control node>.unhealthy` raised | conditions |
|
||||
| 10:59:42 | `machine.<control node>.network` **cleared**, its newest evidence (10:59:37) still "connection refused" | conditions |
|
||||
| 11:00:23 | `machine.<control node>.network` reopened, raised time 11:00:23 | conditions |
|
||||
| 11:06:00 | `seat.mesh-dns-resolver.<control node>.wrong` raised by D2 | conditions |
|
||||
| 11:10:29 | both builds failed their gate: "<control node> as a whole: raised since it was sent: machine.<control node>.network" — put back | conditions |
|
||||
| 11:18:12 | `dnsmasq.service` restarted by hand; every question answered again | the operator |
|
||||
| 11:21:46 | `machine.<each>.core-behind` raised: "… and no plan is rolling them out", while a release walked the node-engine | conditions |
|
||||
|
||||
### What is measured, and what is guessed
|
||||
|
||||
**Measured: the resolver ran and did not answer.** The unit stayed active; the journal said nothing; a
|
||||
restart fixed it. Every question to the private address on port 53 got "connection refused" — the
|
||||
kernel's answer when no socket is bound there. Loopback answered throughout. So dnsmasq had closed, or
|
||||
lost, its socket on the private address and kept the one on loopback.
|
||||
|
||||
**Guessed, then tested: the bridge teardown closed it.** The configuration says `bind-dynamic` and two
|
||||
`listen-address` lines. With `bind-dynamic`, dnsmasq binds one socket per address and re-reads the
|
||||
machine's addresses on every netlink change, closing what it finds gone. The teardown five seconds before
|
||||
is a burst of exactly those changes. The reproduction is in the section below: the mechanism is measured,
|
||||
the twenty-minute loss is not.
|
||||
|
||||
**Measured: why the gate blamed the builds.** The gate counts a condition against a send when it was
|
||||
raised after the send (`judgeHealth`, `aboutTheMachine` in the controller's `gate.go`). The comparison
|
||||
was right; the raised time was not the fault's. The control node's network condition cleared at 10:59:42
|
||||
although its newest evidence still said "connection refused", and reopened at 11:00:23 with a new raised
|
||||
time — after the send. The node-engine sent at 10:59:34 restarted, and a node-engine just started judges
|
||||
each part of the network again from nothing: its first statement said the names part `unknown`, "one look
|
||||
failed; a second decides" (`mesh-host` `internal/network`, `Look`). The controller clears every network
|
||||
condition a statement no longer says unhealthy (`judgeNetworks`), so the restarted engine's first,
|
||||
undecided statement cleared it, and its second look raised it anew. A reopening within ten minutes is the
|
||||
same fault by the conditions' own rule (`ReopenWithin`), but a reopening took the new raised time.
|
||||
|
||||
**Measured: why D2 took eight minutes.** The self-check runs every five minutes. D2 holds a resolver that
|
||||
answered none of its questions for the next run (`Confirm`, issue 277), because on a loaded machine a
|
||||
question can be lost. Its first run after 10:57:57 held it; the next, at 11:06:00, raised it. Every one
|
||||
of those questions had been refused, which no load explains.
|
||||
|
||||
**Measured: why the second walk of the node-engine waited on its own first machine.** A release keeps no
|
||||
record per module — its modules are in its tier only — and D10 read only the records to know what a plan
|
||||
is rolling out. So while a release walked the node-engine at 11:21, D10 raised `core-behind` on every
|
||||
machine, "and no plan is rolling them out", and the gate on the home server, its first machine, waited
|
||||
on it (issue 281's settle window) until D10's next run five minutes later found the new engine there.
|
||||
The walk then passed. It did not fail, but it waited five minutes for a condition its own send caused.
|
||||
|
||||
**Not explained: why the second resolver did not save the lookups.** Every machine's resolver file lists
|
||||
both resolvers (`options timeout:1 attempts:2`), and the control node's own file lists its private address
|
||||
first and the home server's second (read with `node-uplink.resolvers`). The control node's node-engine
|
||||
reached the second resolver: its condition said one of two resolvers did not answer. The route proxy runs
|
||||
on the machine's own network stack and resolves with Go's own resolver, which tries every listed server and, when all
|
||||
fail, names the **last** one it tried (Go `net/dnsclient_unix.go`, `tryOneName`). Its error named the
|
||||
private address of the control node, the first one. I think the route proxy's copy of the resolver file
|
||||
lists only the control node's own resolver, because Go would otherwise have asked the second and named it.
|
||||
That copy is the runtime's, made when the container was created (08:04 UTC that day), and no tool could
|
||||
read it: it is root's, and the docker module had no way to show it. Next step: the new `docker_resolv_conf`
|
||||
tool (below) reads it, on the next look after it is delivered. Whether the certificate authority's failure
|
||||
has the same cause is not known.
|
||||
|
||||
**Measured: why `docker_events` returned nothing for 180 minutes.** The runtime keeps about its last 256
|
||||
events in memory, and the containers' own health checks fill them within minutes (research 032). Asked at
|
||||
11:24 UTC, the tool returned 254 events from the last ten minutes only. The teardown at 10:57:52 had already
|
||||
been pushed out when the question was asked. The tool answered "0 events" as though nothing had happened.
|
||||
|
||||
### The bind behaviour
|
||||
|
||||
Reproduced on the laptop only, in a user and network namespace (never on a machine of the mesh): a dummy
|
||||
link holding the private address, the module's configuration with `bind-dynamic` and both
|
||||
`listen-address` lines, dnsmasq 2.93 in the foreground, a client asking the machine's name on both
|
||||
addresses every 0.1 s, and a bridge with 4 to 80 virtual link pairs brought up and deleted, one by one or
|
||||
all at once.
|
||||
|
||||
**Measured.** Every few teardowns dnsmasq closed both listeners ("stopped listening on … port 53", at
|
||||
debug level only, so never in a normal journal). A build with two lines of logging added showed each
|
||||
closing follows a failed reading of the machine's addresses: "iface_allowed: index N gone (No such
|
||||
device)", N a virtual link being deleted — 59 failed readings and 53 closings in 120 teardowns of 8 links
|
||||
at once; 107 and 104 in 20 teardowns of 80. In every trial the next address change re-bound them within
|
||||
the same burst: two seconds after a teardown the private address was unbound 0 times of 180, and the
|
||||
client was never refused.
|
||||
|
||||
**Read in the source** (dnsmasq 2.93, `src/`): `iface_allowed` returns 0 when a link's index no longer
|
||||
resolves (`network.c:255-259`); that fails the whole enumeration (`netlink.c:225`, `256-258`, `297-301`);
|
||||
a failed IPv6 pass skips the IPv4 pass, so no IPv4 address is marked found (`network.c:832-840`); under
|
||||
`bind-dynamic` alone, every listener not found is closed (`network.c:849-875`); and one is bound again only
|
||||
by a later netlink address event followed by a clean reading (`network.c:1177-1190`, `netlink.c:406-411`).
|
||||
|
||||
**Guessed.** That on the control node the closing came with no later address event to re-bind it — the
|
||||
netlink socket overflowing in the burst and dropping the events after it (it is read without blocking,
|
||||
and what overflowed is lost), or no address changing on the machine for twenty minutes. Why loopback kept
|
||||
its listener there is not known; a reading that failed at a link ordered after loopback and before the
|
||||
private network's would close only the private address.
|
||||
|
||||
**Tested alternatives.** `bind-interfaces` closed nothing under the same churn, kept answering after the
|
||||
private address and its link were deleted and made again, and answered a query from another namespace
|
||||
through a bridge (issue 110's case). It refuses to start when the address is not there yet. `interface=`
|
||||
lines with `bind-dynamic` closed listeners as before (14 in one run). So the configuration now says
|
||||
`bind-interfaces`, and a drop-in makes the unit wait up to 80 seconds for the private address before it
|
||||
starts and retry every ten seconds, rather than leave it failed. The watch stays: it catches whatever else
|
||||
stops the resolver answering.
|
||||
|
||||
## The fixes
|
||||
|
||||
One pull request per repository, each naming this issue.
|
||||
|
||||
**mesh-catalog — the resolver binds once.** `bind-interfaces` in place of `bind-dynamic`, and the unit
|
||||
waits for the private address (above).
|
||||
|
||||
**mesh-catalog — the resolver asks itself, and restarts itself.** The `dnsmasq` module is now Go (its
|
||||
TypeScript tools were never served on any machine). Its new long-running process, the watch, runs as root
|
||||
and every 15 seconds asks the machine's own name — from the roster — on every address the configuration
|
||||
listens on, over UDP and TCP, and checks the answer is that machine's address. After two failing looks in
|
||||
a row it restarts the unit — only one that runs: a unit stopped or disabled on purpose, still starting
|
||||
(waiting at boot for the private address) or failed is the service manager's or a person's, and is held
|
||||
and said — without waiting for the start, at most once every two minutes and three times an hour (a
|
||||
budget it reads back when it restarts itself), and the looks after it
|
||||
say whether the restart helped. Its tool `dnsmasq_health` is the watch's declared check (ADR 0240): it asks
|
||||
again and reads what the watch did, so a restart that helped leaves the module healthy and the operator
|
||||
hears nothing, and one that did not becomes `module.dnsmasq.<machine>.unhealthy`, saying "restarted at …,
|
||||
which did not help". The unit itself is judged by its unit (`health: unit`). The bundle emits `restarted`
|
||||
and, once the looks decide it, `restart-judged` (helped or not) on the bus beside the names it answers,
|
||||
each remembered as said only once emitted. The repair is the module's own, as the identity provider repairs
|
||||
its administrator (ADR 0224 §5), because a resolver that stops answering can cut its machine off from the
|
||||
bus by name, and then nothing outside the machine reaches it. **This departs from ADR 0240 rule 6**
|
||||
("nothing restarts on health"), which left a restart on health to a healer's own record under ADR 0231.
|
||||
That record is not written; whether this module-local repair stands, or becomes a row of the healer
|
||||
registry, is the operator's decision.
|
||||
|
||||
**mesh-controller — a fault is judged by whether it was there at the send.**
|
||||
|
||||
- A condition reopened within `ReopenWithin` keeps when the fault was first raised (`first`) and its
|
||||
gaps, each from a clearing to the reopening after it, across a controller restart too (from the
|
||||
history). The gate asks `OpenAt(send)`: the fault began at or before the send and the send fell in none
|
||||
of its gaps. So a fault there at the send that flapped after it is not the send's, and a machine that
|
||||
had recovered before the send and broke after it fails the gate, as it must (review of the controller's
|
||||
pull request: a silent pass is worse than a false blame).
|
||||
- An undecided part of a network statement — `unknown` or `starting`, as a node-engine just started says
|
||||
— holds the open network conditions whose newest evidence names that part, and only those. A statement
|
||||
undecided as a whole holds every part. The next statement decides.
|
||||
- D10 counts what a release's tier names as rolling out, so a walked node-engine is not `core-behind` on
|
||||
its own first machine.
|
||||
- D2 is unchanged: a resolver that answered nothing, refused or silent, is held for the next run, because
|
||||
a few hundred milliseconds of refusals is a resolver restarting as well as one that stopped. D2 says it
|
||||
five to ten minutes after; the node-engine's own network check is the fast one (48 seconds on the day).
|
||||
|
||||
**mesh-catalog — the tools that were missing.**
|
||||
|
||||
- `netcheck_dns` asks one named resolver directly (`server`, `tcp`), and `netcheck_listening` lists what
|
||||
listens on the machine (`ss -lunt`). netcheck is assigned on the home server and the laptop only;
|
||||
assigning it on the control node and the workstation is the operator's.
|
||||
- `docker_events` reads the runtime's whole buffer of events: the window is complete when the buffer is
|
||||
not full (it then holds every event since the runtime started) or reaches past the window's start, and
|
||||
otherwise says where the history begins, instead of answering an empty list.
|
||||
- `docker_resolv_conf` reads a container's resolver file as the container sees it.
|
||||
|
||||
## How it is checked
|
||||
|
||||
| Rule | Check |
|
||||
|---|---|
|
||||
| A fault there at the send fails no gate; one recovered at the send and back after it does | controller `TestReplay348` (the day's statements: raised, an undecided restart, raised again; the gate on a send between), `TestAFaultThatFlappedAfterTheSendIsNotTheSendsAndOneThatRecoveredBeforeItIs`, conditions `TestAReopeningKeepsWhenTheFaultBegan`, `TestAFaultKeepsItsNewestGapsAndForgetsWhatWasBefore` |
|
||||
| An undecided part holds only what names it | `TestReplay348`, `TestAnUndecidedPartHoldsOnlyWhatNamesIt` |
|
||||
| A release's modules are rolling | `TestAReleaseRollsOutWhatItsTierNames` |
|
||||
| The resolver is asked a real question, restarted, and said when it did not help | dnsmasq `TestALookFindsAResolverThatStoppedAnsweringOnOneAddress`, `TestTheWatchRestartsOnASecondFailingLookWithinItsBudget`, `TestTheWatchRestartsAResolverThatStoppedAnsweringAndHealthSaysWhetherItHelped`, `TestTheWatchRestartsOnlyAUnitThatRuns`, `TestTheWatchLoopKeepsItsBudgetAcrossItsOwnRestart`, `TestHealthSaysWhenTheWatchIsNotLookingOrTheUnitIsNotRunning`, `TestTheAnnouncerSaysWhatChangedAndRemembersOnlyWhatWasSaid` |
|
||||
| Every changed condition is tested | the controller's and the modules' changes were mutated one at a time (28 in the controller, 31 in the modules); each failed a test |
|
||||
| live | after delivery: `dnsmasq_health` on both resolvers healthy; a restart by the watch said as `restarted` on the bus |
|
||||
|
||||
A replay in mesh-lab's register is still owed before this core issue resolves (ADR 0237).
|
||||
|
||||
The controller's own test of the resolver's manifest (`TestTheResolverForwardsToFixedUpstreamsAndNeverReadsResolvConf`)
|
||||
reads the catalogue beside it, so for now it takes either binding; once the catalogue's main binds once,
|
||||
it should refuse `bind-dynamic` (a follow-up in the controller).
|
||||
+64
@@ -0,0 +1,64 @@
|
||||
---
|
||||
status: located
|
||||
opened: 2026-10-09
|
||||
located-in: [mesh-controller cmd/mesh-controller (release_plan.go supersededBy, upgrades.go), mesh-controller internal/inventory (release_plan)]
|
||||
fixed-by:
|
||||
amended-design:
|
||||
---
|
||||
|
||||
# 349. A merge acted on late superseded the merge after it, and built the older commit
|
||||
|
||||
## Symptom
|
||||
|
||||
On 2026-10-09 the plans of the catalogue superseded each other out of the branch's order, twice
|
||||
(`mesh-controller.plans`):
|
||||
|
||||
- the plan at 1833045c was "superseded at tier 0 by" the plan at a783ac94, a merge made before it;
|
||||
- the plan at a082615b — the merge of a security fix to the forge's module — was "superseded at tier 0 by"
|
||||
the plan at 8ff8197a, the merge before it, with "gitea planned there again". The forge's module was then
|
||||
to be built from the commit without the fix. A third merge (9e4267bf) came a minute later and superseded
|
||||
that plan in turn, so the fix was built from 9e4267bf. Had nothing come after, it would not have been.
|
||||
|
||||
## Why it is an issue
|
||||
|
||||
A newer plan of a branch supersedes the older open ones and builds what they had not
|
||||
([ADR 0218](../../02-DECISIONS/0218-a-plan-sends-grants-before-code-rolls-out-one-machine-first-and-a-newer-merge-takes-over-an-older-plan.md),
|
||||
issue 254). "Newer" was read from when each plan was made, never from the commits. The catch-up acts on a
|
||||
merge the bus did not hand over (issue 266) after the merges that came after it, so its plan was made last
|
||||
and won. The mesh built an older commit than the branch held, and said nothing wrong: each plan said
|
||||
"superseded", as designed.
|
||||
|
||||
## Diagnosis
|
||||
|
||||
The order of merges into one branch is the forge's: each merge is made on the branch as the one before
|
||||
left it, so a later merge's commit contains every earlier one's, and the forge's merge time
|
||||
(`merged_at`, which every announcement carries) is the branch's order. A plan kept none of it.
|
||||
`supersededBy` compared `Created`.
|
||||
|
||||
## Fix
|
||||
|
||||
In mesh-controller, in the same pull request as issue 348 (PR 179):
|
||||
|
||||
- A plan keeps when its merge was made, to the nanosecond (`merged_at`, migration 0086; null for a release
|
||||
and for a plan kept before it).
|
||||
- `supersededBy` orders two plans of a branch by their merge times where both are known, and by when they
|
||||
were made only where one is not.
|
||||
- A merge older than the newest planned merge of its branch — that plan in any state, open or already done
|
||||
— is planned at that merge's commit, which contains it, with that merge's time. A module the newer merge
|
||||
already looked at then reads as history and is not built again; what only the older merge moved, and its
|
||||
dependents, are built from the newest commit. Comparing with open plans alone left a gap the review
|
||||
found: a late merge arriving after the newer plan was done rebuilt its dependents at the older commit.
|
||||
That gap is closed.
|
||||
- Not covered: a newer merge that moved nothing the mesh builds made no plan, so it is not known here; a
|
||||
late older merge after it is then planned at its own commit, as before. Its modules are those the newer
|
||||
merge did not change, so what it builds from the older commit is what the newer one holds too, unless a
|
||||
dependent's build reads shared code the newer merge changed.
|
||||
|
||||
## How it is checked
|
||||
|
||||
`TestTwoMergesActedOnInReverseOrderBuildTheNewerCommit` replays the day's case: the later merge acted on
|
||||
first, the earlier one second; one plan is left open, at the later commit, building what both moved.
|
||||
`TestALateMergeAfterTheNewerPlanEndedBuildsTheNewerCommit` acts on the late merge after the newer plan is
|
||||
done. `TestTheBranchOrderIsTheMerges` holds the order and the lookup, to the nanosecond. The first and the
|
||||
last fail on the commit before the fix, and every condition of the change was mutated and caught.
|
||||
A replay in mesh-lab's register is still owed before this core issue resolves (ADR 0237).
|
||||
@@ -1,35 +0,0 @@
|
||||
---
|
||||
status: open
|
||||
opened: 2026-10-09
|
||||
located-in: []
|
||||
fixed-by:
|
||||
amended-design:
|
||||
---
|
||||
|
||||
# 350. An agent waits for a check by polling
|
||||
|
||||
## Symptom
|
||||
|
||||
On 2026-10-09 the agents working on the mesh spent about 3.5 of 23 agent-hours in waiting loops: after a push
|
||||
they asked the delivery service again and again whether the merge check had answered. A check takes five to ten
|
||||
minutes, part of it waiting for a free build agent, so an agent either asks every few seconds or sleeps and
|
||||
answers late. Both cost time, and the asking adds load on the controller and the bus.
|
||||
|
||||
Measured from the agents' own records that day: of about 8.3 hours spent in tools, 3.5 hours were waits and
|
||||
polls, 1.6 hours local runs of `merge-check.sh` and 1.4 hours local test suites. The mesh's own tools answered
|
||||
fast: 232 reads of a check's verdict took 3 minutes in total.
|
||||
|
||||
## Why it is an issue
|
||||
|
||||
The mesh already says what happens as it happens: a build announces itself on the bus (ADR 0157), and a
|
||||
delivery moves through states the delivery service keeps. Nothing lets a caller ask once and be answered when
|
||||
the verdict exists. So every caller builds its own loop, each with its own interval and its own idea of when
|
||||
to give up, and an agent that gives up early reports a check as missing when it is only queued.
|
||||
|
||||
## What would close it
|
||||
|
||||
A verb of the delivery service that waits: given a repository and a pull request, or a commit, it answers when
|
||||
both required checks have a verdict, or when a bound the caller gives runs out, saying which. It answers from
|
||||
the delivery's own state changes rather than by polling the forge, and a caller that times out is told where
|
||||
the check stands (queued, building, judging). Checked by a test that asks before the check exists, sees no
|
||||
answer until the verdict is set, and then receives it once.
|
||||
+66
@@ -0,0 +1,66 @@
|
||||
---
|
||||
status: located
|
||||
opened: 2026-10-09
|
||||
located-in: [mesh-catalog modules/mesh-delivery (holder.go, creditBacklog and carries)]
|
||||
fixed-by: mesh-catalog #163
|
||||
amended-design:
|
||||
---
|
||||
|
||||
# 351. A held delivery whose module a backlog walk carried to every machine is never covered
|
||||
|
||||
## Symptom
|
||||
|
||||
On 2026-10-09 the catalogue's delivery of mesh-catalog #125 (the lighting module's account group,
|
||||
[issue 316](../316-a-group-member-held-before-its-composed-check-answered-stayed-held-after-it-passed/00-report.md))
|
||||
was still `held` 28 hours after its merge, raising `delivery.<id>.stalled` to the operator every hour.
|
||||
mesh-delivery had run the build of [ADR 0270](../../02-DECISIONS/0270-a-held-delivery-that-moves-nothing-or-that-a-later-delivered-merge-contains-ends-by-itself-and-a-hold-whose-reason-is-gone-lifts.md)
|
||||
(mesh-catalog #150) since 07:34 UTC that day, and its table (`mesh-delivery.table`) carried the `covered`
|
||||
row. Seven other held deliveries ended `covered` in its first tick at 07:31 UTC, as the record promised.
|
||||
This one did not, and the holder's journal held no line about it: the forge was never asked whether a
|
||||
later merge contains it.
|
||||
|
||||
What the records say, in UTC (`mesh-delivery.show`, `mesh-controller.plans`, `hand-acts`, `module list`):
|
||||
|
||||
| time | what happened |
|
||||
|---|---|
|
||||
| 10-08 10:09 | #125 merged as a6385479 and held for its group's composed check (issue 316); its walk waited for its word |
|
||||
| 10-08 11:06 | three later catalogue merges took its walk over in turn ([issue 317](../317-a-held-deliverys-module-was-walked-by-a-later-merges-plan/00-report.md)); the last of them, #123's at 86077780, built the module and sent it to the laptop |
|
||||
| 10-08 11:21 | that gate failed: the module's unit was unhealthy on the laptop; the walk ended `failed`, then superseded |
|
||||
| 10-08 14:27 | #127 merged as bf542128; its walk built and delivered its own three modules, `done` at 14:34; the module was built from bf542128 too, and its moves were held back for a person ([issue 318](../318-one-modules-wait-for-a-new-login-held-every-other-modules-walk/00-report.md)) |
|
||||
| 10-08 14:43 | the operator released the held moves (`upgrade release-backlog`); a backlog walk sent the module to the laptop at 14:43 and the desktop at 14:46, judging each, and was `done` on both at 14:48 |
|
||||
| 10-09 14:40 | `module list`: the module built from bf542128, on the laptop and the desktop; `status` lists it as not behind its source; #127's delivery shows the module on both machines as `judging — carried by` that backlog walk, as it had for 24 hours |
|
||||
|
||||
No merge into the catalogue's trunk since #125 touched the module's source, so no later merge will ever
|
||||
move it again ([ADR 0267](../../02-DECISIONS/0267-an-artifact-is-rebuilt-only-when-its-build-source-changes-and-a-module-moves-only-along-build-dependencies-on-artifacts.md)):
|
||||
left as it is, the delivery stays held for ever.
|
||||
|
||||
## Why it is an issue
|
||||
|
||||
ADR 0270 covers a held merge when a later delivered merge's walk "sent the module to every machine running
|
||||
it, as its machine steps say". The change that runs on both machines is #125's, carried by #127's merge
|
||||
commit, which the forge says contains a6385479. The decision's question — does this change go on? — was
|
||||
answered by the operator's own release of the build. Yet the record cannot end:
|
||||
|
||||
- **A backlog walk's end is read by nothing.** While it judges, the controller's record of it names what
|
||||
its gate carried, and mesh-delivery credits the delivery that published each carried build one step per
|
||||
machine. Its last record keeps no gate, only the machines it is done on, and mesh-delivery reads nothing
|
||||
from that: the steps stay `judging` for ever.
|
||||
- **`covered` reads only a walk's own aggregate steps.** A walk's own record says "every machine running
|
||||
it" or "the rest running it"; a step per machine, which is all a backlog walk gives, never counts as the
|
||||
module sent everywhere.
|
||||
|
||||
ADR 0270 stated one such gap, for a module that runs on one machine only. This is a wider one: any module
|
||||
whose build reached its machines on the operator's word, after the mesh held it back, rather than by the
|
||||
walk of the merge that built it. A held delivery in that case has no end but a person's wrong act
|
||||
([issue 319](../319-a-held-delivery-a-later-commit-superseded-has-no-end-but-release/00-report.md)), and
|
||||
the condition it raises every hour is one nobody can clear correctly.
|
||||
|
||||
## What was ruled out
|
||||
|
||||
- A forge that could not answer: the holder's journal has no "not yet covered" line for it, and
|
||||
`gitea_contains` answers `contains: true` for a6385479 in both 86077780 and bf542128.
|
||||
- A coverer forgotten: a final delivery is kept 30 days, and #127's is read back whole, with its steps.
|
||||
- The walk that failed: #123's delivery is `failed` with the module's step `failed` on the laptop; it is
|
||||
rightly no coverer.
|
||||
- A person's `release` of #125 (the verb): it would walk a6385479, an older commit than the one the module
|
||||
was built from, and is what ADR 0270 ruled out.
|
||||
+73
@@ -0,0 +1,73 @@
|
||||
# 351 — diagnosis
|
||||
|
||||
**2026-10-09.** Read from mesh-delivery's code on the catalogue's main (`modules/mesh-delivery/cmd/mesh-delivery`,
|
||||
at the build of mesh-catalog #150 and #160), `mesh-delivery.show` of #125, #123, #124 and #127,
|
||||
`mesh-controller.delivery-walks` for #127's walk and for the backlog walk, `plans`, `hand-acts`, `module list`,
|
||||
the forge's `gitea_contains`, and the holder's journal on the control node.
|
||||
|
||||
**Where a backlog walk's steps come from.** `walkMoved` (holder.go) hands a walk whose record carries a
|
||||
`release` to `creditBacklog`, which reads the gate of its current record and, through `creditCarried`, gives
|
||||
the delivery whose merge commit each carried build was made from one step per machine: `judging` while the
|
||||
gate judges, `passed` or `failed` on its verdict. The controller's last record of a finished backlog walk
|
||||
holds no gate and no modules: `state: done`, `release.done: [the laptop, the desktop]`, `note: released on …`.
|
||||
`creditBacklog` reads nothing from `done`, so the two `judging` steps it credited to #127's delivery at 14:43
|
||||
and 14:46 UTC were never moved on. (The controller's own record of #127's walk never named the module: the
|
||||
module was not in that walk's tiers, and the backlog walk is a walk of its own.)
|
||||
|
||||
**Why `carries` refuses #127.** `coverOf` asks, for each module the held delivery's plan moves, whether a
|
||||
later delivery `carries` it. `carries` reads the candidate's steps for the module and sets "everywhere" only
|
||||
on a `sent` step whose machine is the walk's own aggregate, "every machine running it" or "the rest running
|
||||
it". #127's steps for the module are two per-machine steps, `judging`, so `carries` is false, no candidate is
|
||||
found, and the forge is never asked — which is why the journal has no line.
|
||||
|
||||
**What "done on a machine" means for a backlog walk.** Read from the controller's records, not its code: a
|
||||
backlog walk that failed a gate ends `FAILED at tier … failed its gate on <machine>` and puts the builds back
|
||||
(seen the same day for the mesh's own parts); one that ends `done` lists every machine in its order as done,
|
||||
with "skipped: none reported". So a machine in `done` is one the gate passed on. That reading is a fact of
|
||||
the records as they are; if the controller ever lists a machine as done without a gate's pass, this diagnosis
|
||||
is wrong there.
|
||||
|
||||
**Ruled out.** A forge that could not answer, a coverer forgotten, the failed walk, and the `release` verb —
|
||||
see the report.
|
||||
|
||||
**Located** in mesh-delivery: `creditBacklog`, which reads a finished backlog walk's `done` machines for
|
||||
nothing, and `carries`, which counts no per-machine step.
|
||||
|
||||
## The fix
|
||||
|
||||
In mesh-catalog #163, one pull request, in `modules/mesh-delivery`:
|
||||
|
||||
- **A finished backlog walk passes the steps it carried.** `creditReleased`: every step credited to that
|
||||
walk (`carried by <its id>`), on a machine it is done on, still `sent` or `judging`, moves to `passed`. A
|
||||
step it failed on, and a step another walk carried, are left as they are.
|
||||
- **`carries` reads per-machine steps against the held delivery's plan.** A module is sent everywhere when
|
||||
the walk's own aggregate step says so, as before, or when a `passed` step stands on every machine the held
|
||||
delivery's plan sends the module to (`machines[].receives`). A plan that names no machines, a machine still
|
||||
judging, one failed or one never sent leave the delivery held.
|
||||
- **A walk that carried a step still judging is asked for by its id.** The controller lists back only its
|
||||
open walks and the last fifty ended at a start; the backlog walk of this case, ended the day before, was
|
||||
no longer among them when read on 2026-10-09, so its end would never have been heard. The holder asks for
|
||||
such a walk by its id (`delivery-walks` with `plan`), at a start at once and then once an hour, and takes
|
||||
its record as said.
|
||||
|
||||
Nothing else in ADR 0270 changes: the later merge must still be delivered on its own passing check, by its own
|
||||
walk of its merge commit on the same trunk, and the forge must still say that commit contains the held one's.
|
||||
The record's consequence that named the one-machine gap gains a progressive insight naming this one.
|
||||
|
||||
**How it is checked**, in `covered_test.go`:
|
||||
|
||||
- `TestAModuleAReleaseCarriedToEveryMachineCovers`: passed on every machine the plan sends it to is covered;
|
||||
still judging on one, failed on one, never sent to one, or a plan naming no machines stays held.
|
||||
- `TestAFinishedReleaseMovesTheStepsItCarriedToPassed`: done on both passes both; done on one passes that one
|
||||
and leaves the other judging; a failed step stays failed; a step another walk carried is not touched.
|
||||
- `TestAHeldDeliveryWhoseModuleAReleaseCarriedIsCoveredOnceTheReleaseIsDone`: the day's case whole — held
|
||||
while the backlog walk judges, covered once it is done and read by its id, naming the later merge and the
|
||||
module.
|
||||
|
||||
All three fail on the catalogue's main before the change and pass after it.
|
||||
|
||||
**After the fix is delivered.** At the holder's first ticks with the new build, #127's two judging steps are
|
||||
asked for by the backlog walk's id and pass; #127 then carries the module for #125; the forge is asked once
|
||||
whether bf542128 contains a6385479, which it does; and #125 ends `covered` by #127. Its group then reads
|
||||
delivered, and the stalled condition clears at the self-check's next reading. Whether that happened is the
|
||||
acceptance of this issue, read with `mesh-delivery.show` of #125.
|
||||
+46
@@ -0,0 +1,46 @@
|
||||
---
|
||||
status: located
|
||||
opened: 2026-10-09
|
||||
located-in: [mesh-controller cmd/mesh-controller (gate.go, release.go, release_plan.go, acting.go, push.go), mesh-controller internal/inventory, mesh-controller internal/catalogue]
|
||||
fixed-by:
|
||||
amended-design:
|
||||
---
|
||||
|
||||
# 352. A gate read another send's report, and a put-back chose a controller older than its store
|
||||
|
||||
## Symptom
|
||||
|
||||
On 2026-10-09, local times on the control node (its controller's and node-engine's journals, and `plans`):
|
||||
|
||||
- 16:19 a release sent the control node three core builds — the controller (2913c54c), the node-engine
|
||||
(d8ff1540) and the tool runner (83c19941). 16:21:43 the control node applied them; 16:21:53 its
|
||||
node-engine said each was healthy. The release's gate did not pass in the nine minutes that followed.
|
||||
- 16:30:56 a merge of the controller (e6b00e2e) made a plan; 16:31:37 that plan sent the control node the
|
||||
newer controller — a newer send to the same machine, while the release's gate was judging there;
|
||||
16:31:44 the serving controller was replaced by the newer one.
|
||||
- 16:32:20 the new process judged the release's gate **failed**: "not healthy within 10m0s of its apply:
|
||||
the control node has not reported on what it was sent" — and put all three builds back, on every
|
||||
machine, in one send. The control node had reported them, ten minutes earlier.
|
||||
- 16:32:24 the put-back controller (ef26d4cb) started and said it was behind its own row: it carried fewer
|
||||
migrations than the store had applied, and served what it could.
|
||||
- 16:35:59 that older controller judged the newer plan's gate **passed** on the control node ("healthy 3
|
||||
times over 4m17s") — from the health of the build it was itself, not of the build the plan had sent,
|
||||
which the put-back had removed from the machine. The plan walked on, believing the controller delivered.
|
||||
|
||||
## Why it is an issue
|
||||
|
||||
Four rules did not hold.
|
||||
|
||||
- **A gate judges its own send** (ADR 0236). The gate held the machine's report against the declaration
|
||||
sent *last*, which another plan's send had just replaced; the report on the gate's own send read as
|
||||
none. The report was never lost; it was compared with the wrong send.
|
||||
- **A gate judges the build it sent.** The controller's gate read the lease holder's readiness and the
|
||||
machine's report, and neither said *which* controller build was running. A put-back build passed the
|
||||
gate of the build it had replaced.
|
||||
- **A put-back puts back something that can run the mesh** (ADR 0236: "put back by something other than
|
||||
itself"). A controller older than the store's schema starts behind its own records. Nothing refused it.
|
||||
- **One walk of a module on a machine at a time** (issue 281, ADR 0236). A release's walk was invisible to
|
||||
the merge plan's send: a release keeps no module records, and the check for a walk in progress read
|
||||
only those.
|
||||
|
||||
The diagnosis and the fix are in [`01-diagnosis.md`](01-diagnosis.md).
|
||||
+86
@@ -0,0 +1,86 @@
|
||||
# 352. Diagnosis
|
||||
|
||||
## 2026-10-09
|
||||
|
||||
**The record.** A search of the records for "has not reported on what it was sent" and "older than the
|
||||
store" found nothing. Issue 335 (a passed gate failed by later reports) and issue 281 (a machine's condition
|
||||
read as a module's) are the nearest; neither is this.
|
||||
|
||||
### What is measured
|
||||
|
||||
- **D1, the report compared with the wrong send.** The gate reads each machine's last report beside the
|
||||
declaration the machine was *last sent* (`LastReports`: `Current` is "the report names the declaration
|
||||
in the node's `sent` column"). The release's send wrote `sent`; the control node reported on it, applied;
|
||||
the merge plan's send at 16:31:37 overwrote `sent`; from then the same report read as not current, and
|
||||
the gate's wait ended at its bound with "has not reported on what it was sent". The controller restart
|
||||
at 16:31:44 is in the timeline but is not the cause: the plan and the gate are kept in the store, and a
|
||||
fresh process read the same `sent` and the same report. Nothing was lost across it.
|
||||
- **D2, the build judged is not the build running.** `judgeHealth` for the controller asks: the lease is
|
||||
held since after the send, and the holder says ready. The holder's `Build` is the link-time version
|
||||
stamp, which the mesh's own builds do not set ("development build"), so it names no build. After the
|
||||
put-back, the registered controller build, the lease holder and the machine's last send all were the
|
||||
put-back build, and the newer plan's gate — comparing none of them with the build it sent — passed on
|
||||
the put-back build's health.
|
||||
- **D3, a put-back to a controller behind the store.** A failed gate restores the module's registered build
|
||||
to the one the first machine ran before (`PreviousBuild`) and sends it. Nothing asked whether that
|
||||
build can read the store as the failed build left it. The controller says so itself when it starts
|
||||
("this control plane is behind the mesh-controller row"), and serves anyway (ADR 0185).
|
||||
- **The merge plan's send crossed the release's gate.** `walkedBy` finds a walk in progress by a plan's
|
||||
module records (`p.Modules[module].First`). A release (the walk of the builds waiting for a gate) keeps its walk in its gate's carried moves,
|
||||
not in module records, so the release's open gate on the control node held nothing back. And a plan's
|
||||
own modules were never checked against another walk at all: the check ran over the moves a machine
|
||||
had waiting, and skipped the plan's own.
|
||||
|
||||
### What is guessed
|
||||
|
||||
- Why the release's gate had not passed in the nine minutes before the merge is not established. The
|
||||
node-engine said all three healthy at 16:21:53; the gate needs three passes forty seconds apart after a
|
||||
two-minute settle, and the controller's own readiness (its self-check run once, `status` within ten
|
||||
seconds). I think the controller's readiness came late — it was the build being judged — but I did not
|
||||
read its lease's health at the time.
|
||||
|
||||
## The fix
|
||||
|
||||
In mesh-controller, pull request 185.
|
||||
|
||||
- **A gate keeps what it sent, and reads the report against that.** Each gate keeps, per machine, the
|
||||
declaration its send carried — its digest and its sequence (`PlanGate.Sent`, read from the node's
|
||||
record right after the send). A report is on that send when it names that declaration, or one the
|
||||
machine was sequenced after it. The gate's judging and the merge plan's first-machine wait both read
|
||||
this; a gate kept before sends were kept reads the last send, as before. Kept with the plan in the
|
||||
store, so a controller restarted mid-judging reads the same.
|
||||
- **A gate judges only the build the machine was last sent.** Each gate judges what the machine was
|
||||
last sent of the module (`sent_builds`) against the commit it judges. Another build there means the
|
||||
judging is **superseded**: a new verdict, decided at once, with no verdict on the build, nothing marked,
|
||||
nothing put back; a release so superseded ends, and what still waits is released again; a merge plan
|
||||
so superseded ends, said, and the newer plan carries on.
|
||||
- **The controller is never put back to a build older than the store's schema.** A controller is told
|
||||
which build it is by its declaration (`MESH_CONTROLLER_VERSION`, the bundle's version, `${version}`
|
||||
composed by the catalogue — now also inside a process's env), and records when it serves the highest
|
||||
migration it carries, by that version (`controller_schema`, migration 0087). A failed controller gate
|
||||
reads the record of the build it would put back: one that reaches less than the store has applied, or
|
||||
never recorded what it reaches, is not put back; the current build is kept, and the condition
|
||||
(`rollback-failed`, urgent) says why and that a person decides. The lease holder's `Build` is that
|
||||
version too.
|
||||
- **A walk of a module on a machine holds every other send of another build of it there.** `walkedBy`
|
||||
reads a release's open gate as its walk, and a plan's own modules are held by it as any other move is:
|
||||
a merge plan whose first machine is under a release's gate for the same module waits, said as
|
||||
"waiting for … which release-… is walking". The same build to the same machine is not held: it is
|
||||
already there.
|
||||
|
||||
Not changed: whether a release's walk and a merge plan's walk of the core should be one walk (the walk
|
||||
carrying over) is left to a design; serialising them is what this fixes.
|
||||
|
||||
## How it is checked
|
||||
|
||||
| Rule | Check |
|
||||
|---|---|
|
||||
| A newer send to the judged machine does not unreport the gate's | `TestReplay352` (the backlog fixture: the release sends, the machine reports, another send lands unreported, the gate still passes) — fails on the commit before the fix with "has not reported on what it was sent" |
|
||||
| A send that moves the module supersedes the judging, puts nothing back | `TestReplay352`, `TestAPlansJudgingIsSupersededByASendThatMovesItsModule` |
|
||||
| A report is held against the gate's own send, by digest or sequence | `TestAReportIsHeldAgainstTheGatesOwnSend`, inventory `TestASendAndAReportAreKnownByTheirDeclaration` |
|
||||
| Only the build last sent is judged | `TestAGateJudgesOnlyTheBuildTheMachineWasLastSent` |
|
||||
| A walk holds other sends of the module there, a release's too | `TestAWalkWaitsForAJudgingOfTheSameModuleOnThatMachine`, `TestAPlansFirstSendWaitsForAReleaseJudgingTheModuleThere` |
|
||||
| No put-back to a controller behind the store, or one unproved | `TestTheControllerIsNotPutBackToABuildOlderThanTheStoresSchema`, inventory `TestASchemaReachIsKeptByBuild`, catalogue `TestAProcessEnvCanNameTheBuildsOwnVersion`, `TestTheHolderNamesTheBuildTheDeclarationToldIt` |
|
||||
| Every condition of the change is tested | 31 mutations, each failing a test |
|
||||
|
||||
A replay in mesh-lab's register is still owed before this core issue resolves (ADR 0237).
|
||||
@@ -0,0 +1,67 @@
|
||||
---
|
||||
status: open
|
||||
opened: 2026-10-09
|
||||
located-in: []
|
||||
fixed-by:
|
||||
amended-design:
|
||||
---
|
||||
|
||||
# 353. An agent cannot put a commit on the forge
|
||||
|
||||
## Symptom
|
||||
|
||||
On 2026-10-09, in an agent session on the operator's laptop, local times:
|
||||
|
||||
- An agent finished a change to this repository's connectivity design on a branch, committed it in a
|
||||
worktree, and ran `merge-check.sh`; all four checks passed.
|
||||
- `git push` was not possible: this repository's clone on the laptop has its push URL set to the literal
|
||||
`DISABLED`. The laptop's clones of the mesh's code repositories push normally. Who set it, when, and
|
||||
why is not recorded anywhere this session could find.
|
||||
- The forge's tools have no verb that pushes a branch or makes a commit. `gitea_api` refused
|
||||
`POST /repos/{owner}/{repo}/diffpatch`, the forge's endpoint that applies a patch to a branch: *not a
|
||||
write gitea_api makes* ([ADR 0266](../../02-DECISIONS/0266-on-the-control-node-agents-run-under-an-account-of-their-own-that-cannot-become-root-without-a-person.md)).
|
||||
- The one file write `gitea_api` allows is the contents endpoint, which takes the **whole file**,
|
||||
base64-encoded. The changed design document is about 90 KB, so the call would carry about 121 KB of
|
||||
base64 that the agent must reproduce character for character. That is not a workable path.
|
||||
|
||||
The change waits for the operator to push it by hand. This report waits for the same push.
|
||||
|
||||
## Why it is an issue
|
||||
|
||||
- **The mesh's own rule for a change cannot be followed by an agent.** Every change goes through a branch,
|
||||
a pull request and a person's approval. An agent off the control node can make the branch locally and
|
||||
open the pull request through `gitea_create_pull_request`, but it cannot put the commits between the two
|
||||
on the forge.
|
||||
- **Where no tool covers a need, none is worked around** ([ADR 0245](../../02-DECISIONS/0245-a-verb-says-what-it-replaces-and-the-agent-is-guarded-from-working-round-the-mesh.md)).
|
||||
The only ways round are pushing to an explicit URL past the disabled one, or retyping a file as base64.
|
||||
The first undoes a setting someone made on purpose. The second risks a commit that silently differs
|
||||
from what was checked.
|
||||
- **The control node already has an answer the laptop does not.** On the control node the agent account
|
||||
holds a forge key of its own, so its pushes are its own (ADR 0266). An agent on any other machine has
|
||||
no equivalent.
|
||||
|
||||
## Other gaps in the forge's tools
|
||||
|
||||
Found by listing the forge module's 37 tools on 2026-10-09. Only the first one blocks work. The others
|
||||
can be done through `gitea_api`, which costs a raw call and a large answer each time.
|
||||
|
||||
| Gap | Today | Checked |
|
||||
|---|---|---|
|
||||
| **No push, commit or patch on a branch** | blocked, as above | yes |
|
||||
| **No commit history or compare** | `GET …/commits` through `gitea_api`: about 100 lines of JSON for one commit, its author and committer printed in full each time. `gitea_contains` answers only whether one commit contains another. | yes |
|
||||
| **No reading a pull request's reviews or line comments** | `gitea_list_comments` returns the conversation only. Reviews and comments on lines need `GET …/pulls/{n}/reviews` through `gitea_api`. An agent asked to address review feedback does not see it unless it knows to ask there. No *approve* tool is right, since approval is a person's. | yes |
|
||||
| **No creating a branch** | `POST …/branches` through `gitea_api` | no |
|
||||
| **No editing an issue** — title, body, labels | only close and reopen exist | no |
|
||||
| **No listing a directory** | `gitea_get_file` reads one file | no |
|
||||
| **No code search across repositories** | none | no |
|
||||
|
||||
*Checked* says whether this session called the raw endpoint and saw it answer. The rows marked *no* are
|
||||
read from the tool list and the forge's API, not tried.
|
||||
|
||||
## Open
|
||||
|
||||
- Who disabled the push URL on the laptop's clone of this repository, and whether it was meant to keep
|
||||
agents from pushing here or was a step in something else.
|
||||
- Whether an agent off the control node should push with a forge key of its own, as on the control node,
|
||||
or whether the forge module should take a commit as a patch and write it under the agent's account.
|
||||
Either needs a decision record.
|
||||
Reference in New Issue
Block a user