novox/hq
6
0
Fork 0
Code Issues Pull Requests 5 Releases Wiki Activity

Merge pull request 'Issue 356: the operator's command ran a stale engine, and a hand-over had no channel; ADR 0272 gains the operator's rule and words.py the daemons' lines' (#256) from issues/356-the-operators-command-ran-a-stale-engine-and-a-hand-over-had-no-channel into main

Browse Source
This commit was merged in pull request #256.
This commit is contained in:
mesh-admin
2026-10-09 17:57:18 +00:00
parent b962e87cae 2cb779d06f
commit 0e7af9988a
13 changed files with 372 additions and 29 deletions
Download Patch File Download Diff File Expand all files Collapse all files
00-META/checks/README.md
+10 -1
View File
@@ -4,8 +4,9 @@
python3 00-META/checks/records.py structure: links, citations, supersession, topics
python3 00-META/checks/index.py every record's topic is in the reading order, and no list of records is stored
python3 00-META/checks/index.py --print the records, by topic, in reading order (generated, never written)
python3 00-META/checks/words.py the glossary's retired words are not used, and no word is defined twice
python3 00-META/checks/words.py the glossary's retired words are not used, no daemon's line is written, and no word is defined twice
python3 00-META/checks/words.py --list tools the words the catalogue's copy must list
python3 00-META/checks/words.py --list lines the daemons' lines the catalogue's copy must list
```
Non-zero exit on any problem, so it can be a gate rather than a report.
@@ -88,6 +89,14 @@ never checked. It also fails when a glossary head word heads two entries or is a
a graduated research effort kept as written. With `MESH_CATALOG_DIR` set it compares the catalogue's copy
of the tools' retired words (`retired-words`) with the glossary.
It also fails on a **daemon's line** ([ADR 0272](../../02-DECISIONS/0272-the-operator-reaches-the-controller-with-mesh-cli-on-every-node-and-a-node-is-the-terminal-only-where-no-agent-runs-as-its-operator.md),
its insight of 2026-10-09): the operator types one command, `nox`, and a document names a `nox` line, never the
controller's or the node-engine's binary followed by one of its commands. The lines are the check's own list
(`DAEMON_LINES`); a command that is a line only with an argument after it is matched as a line is written. Looked
for in the same documents, from 2026-10-09 for research and issues, in the document's own words — code blocks and
spans included, quotations and block quotes taken out. With `MESH_CATALOG_DIR` set it compares the catalogue's copy
(`retired-lines`) with the list.
It failed on something real before it passed: 581 uses of retired words in 63 documents on its first run, besides the glossary itself and research 034 —
"the host" for the node-engine in 30 designs, "control plane" in 10, the glossary's own entry for the tool runner —
and two glossary contradictions (the "console", the deprecated broker's seat), fixed in the change that
00-META/checks/words.py
+114 -13
View File
@@ -28,9 +28,23 @@ What is enforced:
copy the catalogue's own check keeps (mesh-catalog `retired-words`). With
MESH_CATALOG_DIR set to a checkout of the catalogue it also fails when that copy differs.
lines no daemon's line in a checked document (ADR 0272, its insight of 2026-10-09). The operator
types one command, `nox` (`mesh-cli`); the node-engine (`nox-mesh-host`) and the
controller (`mesh-controller`) are daemons, and a person never types their names — so a
document names a `nox` line, never `mesh-controller push` or `mesh-host hand-over`. The
lines are DAEMON_LINES below: each daemon's commands, and which of them are a line only
with an argument after them ("the mesh-controller module" is prose; `mesh-controller
module check …` is a line). A line is looked for in the document's own words — code
blocks and code spans included, since that is where a line to type is written — with
quotations, block quotes, link targets and comments taken out. `--list lines` prints the
list, which the catalogue's own check keeps a copy of (mesh-catalog `retired-lines`); with
MESH_CATALOG_DIR set it also fails when that copy differs. The node-engine's install-time
subcommands (`nox-mesh-host enrol`, `key`, `reconcile`, …) are not lines here: they are
typed at a machine no mesh reaches yet, and the engine's usage says so.
Checked documents: 00-META/, 03-DESIGN/ (both layers), AGENTS.md and README.md, always; a research
effort initiated, or an issue opened, on or after FROM. Never 02-DECISIONS/: a decision record keeps the
words it was written with.
effort initiated, or an issue opened, on or after FROM (on or after LINES_FROM for the lines). Never
02-DECISIONS/: a decision record keeps the words it was written with.
A document the check fails on that cannot be reworded in the change that found it is named, with a date
and a reason, in words-allowed.md beside this file. An entry past its date fails like the word itself, and
@@ -39,6 +53,7 @@ graduated research effort, which is a record of what was said, like a decision r
python3 00-META/checks/words.py
python3 00-META/checks/words.py --list tools
python3 00-META/checks/words.py --list lines
"""
import datetime
@@ -55,6 +70,25 @@ FROM = "2026-10-07"
ALWAYS = ("00-META/", "03-DESIGN/", "AGENTS.md", "README.md")
# The daemons' lines (ADR 0272, its insight of 2026-10-09), and the day that rule began. "alone" is a
# command that is a line by itself; "with an argument" one that is a line only when written as one.
LINES_FROM = "2026-10-09"
DAEMON_LINES = {
"mesh-controller": {
"alone": ("adopt assign broker cancel cleanup collect conditions converge doctor durations hand-act "
"hand-acts healers identity kill licence migrate operator pause pin plan prepare push rebuild "
"rehearse replay resume retire rollout rotate secret settings status token unassign unpin "
"upgrade").split(),
"with an argument": ("artifacts ask builds bus collection data delivery images mirrors module node "
"overlay plans queue seat seats take version").split(),
},
"mesh-host": {
"alone": "apply bundle enrol enroll hand-over help inventory key overlay owned profile reconcile run version".split(),
"with an argument": [],
},
"nox-mesh-host": {"alone": ["hand-over"], "with an argument": []},
}
NOT_LINE = re.compile(r"^\s*(?:[-*]\s+)?\*Not:\*(.*)$", re.M)
STRUCK = re.compile(r"~~([^~]+)~~(?:\s*\((hq|tools)\))?")
IDENT_LINE = re.compile(r"^\s*(?:[-*]\s+)?\*Identifier until renamed:\*(.*)$", re.M)
@@ -134,8 +168,8 @@ def prose(text, glossary_file=False):
return text
def research_and_issues():
"""The research efforts and issue reports dated on or after FROM, file by file."""
def research_and_issues(since=FROM):
"""The research efforts and issue reports dated on or after `since`, file by file."""
out = []
for folder, key, first in (("01-RESEARCH", "initiated", "00-overview.md"), ("04-ISSUES", "opened", "00-report.md")):
base = os.path.join(ROOT, folder)
@@ -144,7 +178,7 @@ def research_and_issues():
if not os.path.isfile(head):
continue
date = frontmatter_field(read(rel(head)), key)
if not date or date < FROM:
if not date or date < since:
continue
for name in sorted(os.listdir(os.path.join(base, effort))):
if name.endswith(".md"):
@@ -152,7 +186,7 @@ def research_and_issues():
return out
def checked_documents():
def checked_documents(since=FROM):
docs = []
for entry in ALWAYS:
path = os.path.join(ROOT, entry)
@@ -164,7 +198,7 @@ def checked_documents():
for name in sorted(files):
if name.endswith(".md"):
docs.append(rel(os.path.join(base, name)))
return docs + research_and_issues()
return docs + research_and_issues(since)
def allowed():
@@ -223,6 +257,60 @@ def check_retired(retired, identifiers):
return problems
def lines_list():
"""The daemons' lines as the catalogue's copy lists them: `<daemon> <command>`, ` …` when an argument must follow."""
out = []
for daemon, kinds in DAEMON_LINES.items():
out += ["%s %s" % (daemon, c) for c in kinds["alone"]]
out += ["%s %s …" % (daemon, c) for c in kinds["with an argument"]]
return sorted(out)
def line_pattern(daemon, command, with_argument):
"""The line whole: the daemon, white space, the command. One that is a line only with an argument is matched
as a line is written — at the start of a line or a code span, a prompt allowed, an argument after it."""
d, c = re.escape(daemon), re.escape(command)
if with_argument:
return re.compile(r"(?m)(?:^|`)[ \t]*(?:\$[ \t]*)?(" + d + r"\s+" + c + r"\s+[^\s`])")
return re.compile(r"(?<![\w-])" + d + r"\s+" + c + r"(?![\w-])")
def own_words(text):
"""The text with what is not the document's own words blanked: as prose(), but a code block and a code span
are kept, since a line to type is written in one."""
if text.startswith("---\n"):
end = text.find("\n---", 4)
if end != -1:
text = re.sub(r"[^\n]", " ", text[: end + 4]) + text[end + 4:]
for step in (
r"(?s)<!--.*?-->", # comments
r"(?m)^\s*>.*$", # block quotes
r"\]\([^)\s]*\)", # link targets
r"~~[^~\n]+~~", # struck through
r"\"[^\"\n]*\"", # "quoted"
r"\u201c[^\u201d]*\u201d", # curly double quotes
r"\u2018[^\u2019\n]*\u2019", # curly single quotes
):
text = re.sub(step, blank, text)
return text
def check_lines():
problems = []
patterns = []
for daemon, kinds in DAEMON_LINES.items():
patterns += [(daemon, c, line_pattern(daemon, c, False)) for c in kinds["alone"]]
patterns += [(daemon, c, line_pattern(daemon, c, True)) for c in kinds["with an argument"]]
for doc in checked_documents(LINES_FROM):
text = own_words(read(doc))
for daemon, command, rx in patterns:
for m in rx.finditer(text):
line = text.count("\n", 0, m.start()) + 1
problems.append("%s:%d: `%s %s` is a daemon's line — the operator types `nox …` (ADR 0272)"
% (doc, line, daemon, command))
return problems
def check_unique(retired, heads):
problems = []
seen = {}
@@ -253,29 +341,42 @@ def check_copy(retired):
except OSError as err:
return ["the catalogue's copy of the retired words cannot be read: %s" % err]
want = tools_list(retired)
problems = []
if sorted(copy, key=str.lower) != want:
return ["the catalogue's retired-words differs from the glossary: it should list %s" % ", ".join(want)]
return []
problems.append("the catalogue's retired-words differs from the glossary: it should list %s" % ", ".join(want))
path = os.path.join(catalogue, "retired-lines")
try:
with open(path, encoding="utf-8") as handle:
copy = [l.strip() for l in handle if l.strip() and not l.startswith("#")]
except OSError as err:
return problems + ["the catalogue's copy of the daemons' lines cannot be read: %s" % err]
if sorted(copy) != lines_list():
problems.append("the catalogue's retired-lines differs from this check's list: regenerate it with "
"`python3 00-META/checks/words.py --list lines`")
return problems
def main(argv):
retired, identifiers, heads = glossary()
if argv[1:2] == ["--list"]:
scope = argv[2] if len(argv) > 2 else "tools"
words = tools_list(retired) if scope == "tools" else sorted({w for w, s in retired if s in ("all", scope)})
if scope == "lines":
words = lines_list()
else:
words = tools_list(retired) if scope == "tools" else sorted({w for w, s in retired if s in ("all", scope)})
print("\n".join(words))
return 0
if not retired:
print("words: the glossary names no retired word on a *Not:* line — the check would pass on anything")
return 1
problems = check_unique(retired, heads) + check_retired(retired, identifiers) + check_copy(retired)
problems = check_unique(retired, heads) + check_retired(retired, identifiers) + check_lines() + check_copy(retired)
for p in problems:
print(p)
if problems:
print("words: %d problem(s)" % len(problems))
return 1
print("words: %d retired words and %d identifiers, none in running prose; %d head words, each once"
% (len(retired), len(identifiers), len(heads)))
print("words: %d retired words and %d identifiers, none in running prose; %d daemons' lines, none written; "
"%d head words, each once" % (len(retired), len(identifiers), len(lines_list()), len(heads)))
return 0
00-META/glossary.md
+5 -1
View File
@@ -468,7 +468,11 @@ asking, never the act. **Decided by** ADR 0152, 0175, 0234, 0259, 0272.
([ADR 0272](../02-DECISIONS/0272-the-operator-reaches-the-controller-with-mesh-cli-on-every-node-and-a-node-is-the-terminal-only-where-no-agent-runs-as-its-operator.md)).
- **nox** — `mesh-cli`'s alias, in interactive zsh only. A script, `sudo` and a document meant to work everywhere
say `mesh-cli`. The operator also calls the mesh itself *nox mesh*; on this page and in the records *nox* is the
alias alone.
alias alone. It is the one command the operator types: every line the mesh addresses to the operator — a
condition's words, a verb's answer, a README, a design — names a `nox` line, never a daemon's (the controller's
or the node-engine's binary followed by a command), which `words.py` and the catalogue's check refuse
([ADR 0272](../02-DECISIONS/0272-the-operator-reaches-the-controller-with-mesh-cli-on-every-node-and-a-node-is-the-terminal-only-where-no-agent-runs-as-its-operator.md),
its insight of 2026-10-09).
- **channel / intake** — the two kinded benches the mesh talks to its operator through: `channel`
sends, `intake` turns what arrives into one envelope. A holder of either declares **capabilities** from
the fixed vocabulary `channel-capabilities/1`. Not *notifier* (that is the desktop's node seat, one
02-DECISIONS/0272-the-operator-reaches-the-controller-with-mesh-cli-on-every-node-and-a-node-is-the-terminal-only-where-no-agent-runs-as-its-operator.md
+21
View File
@@ -210,6 +210,27 @@ fails if one changes alone: if the operator's account lost its passwordless `sud
> 3. **An ordinary line is reads and settings.** §4 is exact as it stands: an ordinary line is the generic verb's
> reading forms, or a `settings` line composed as the settings verb composes it — nothing else runs.
> **Progressive insight — 2026-10-09, the operator's rule.** The operator types **one** command, `mesh-cli`
> (`nox`). `nox-mesh-host` (the node-engine) and `mesh-controller` (the controller) are daemons, and a person
> never types their names. The consequence above — a record that says to type `mesh-controller <command>` means
> `mesh-cli <command>` — is from this day a rule on every line the mesh addresses to the operator: a condition's
> words, a verb's answer, a module's README, a design document name a `nox` line, never a daemon's. Found on
> 2026-10-09 at 18:01 by the operator typing the node-engine's `hand-over` of
> [issue 339](../04-ISSUES/339-a-verb-can-place-a-directory-at-the-machines-own/00-report.md), which a condition
> had told them to run "at the machine": the command on the machine's `PATH` was a copy of an engine from before
> that subcommand existed, and the act had no channel
> ([issue 356](../04-ISSUES/356-the-operators-command-ran-a-stale-engine-and-a-hand-over-had-no-channel/00-report.md)).
> Now `nox node hand-over <node> <path>` on the control-node is that act, terminal-only, carried by the controller
> to the node's engine over the bus. **How it is checked:** hq's `words.py` refuses a daemon's line —
> `mesh-controller` or `mesh-host` followed by one of its commands, and `nox-mesh-host hand-over` — in the
> documents it checks from this day, quotations excepted, code blocks and spans included; the catalogue's
> `checks/words` refuses the same lines in every string a module's tools can show and every manifest description,
> from its copy `retired-lines`, which `words.py --list lines` prints and compares. **What is not a line here:** the
> node-engine's install-time subcommands (`nox-mesh-host key`, `enrol`, `overlay take`, `reconcile`, `apply`,
> `profile`, `inventory`, `owned`, `bundle`, `version`) are typed as root at a machine no mesh reaches yet, or at
> one no mesh can reach any more; the engine's usage says so. The daemons' binaries keep their names. The decision
> stands.
### 5. What comes back
**Every line is a call**, run, recorded and answered as a seat's verb is: the bus lets a request be answered for a
03-DESIGN/01-to-be/09-the-node-lifecycle.md
+6 -5
View File
@@ -223,8 +223,9 @@ The same path, with the mesh built in the middle of it.
# 1 — raise the foundation and the controller from the carried bundle
nox-mesh-host reconcile
# 2 — the controller now exists, and issues the first token
mesh-controller token issue
# 2 — the controller now exists, and issues the first token (nox, the operator's command,
# at the controller's terminal: ADR 0272)
nox token issue
# 3 — the machine joins the mesh it just raised
nox-mesh-host enrol --token <token>
@@ -618,8 +619,8 @@ rather than left to be worked out.
**A token is issued *for* a node record**, and that is where a re-enrolment is decided.
```
mesh-controller token issue --node workstation # this machine is that node again
mesh-controller token issue --new # a machine the mesh has not seen
nox token issue --node workstation # this machine is that node again
nox token issue --new # a machine the mesh has not seen
```
The node-engine does not need to know which it is. It presents a token and receives an identity; what
@@ -699,7 +700,7 @@ keeps cataloguing.
### Where the enrolment token comes from
**`mesh-controller token issue` prints it once**, to the person running it. Single-use, and it
**`nox token issue` prints it once**, to the person running it at the controller's terminal. Single-use, and it
expires whether used or not ([ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)).
It is carried by hand — read off a screen, pasted into a terminal. That is the design rather than
03-DESIGN/01-to-be/37-the-operators-machine.md
+1 -1
View File
@@ -113,7 +113,7 @@ resolver is asked by the second graphical module; the display server itself is g
## 6. The order of the build
1. **The operator account on every node** — `mesh-controller node` with the login name; empty on
1. **The operator account on every node** — `nox node account <node> <login>` at the controller's terminal; empty on
all four today. Nothing home-scoped composes before it.
2. **The tool runner** — `mesh-host` supervises it; mesh-tools serves bundles from memberships
and reloads; mesh-controller composes the bundle into the declaration and the memberships to one
03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md
+3 -1
View File
@@ -72,6 +72,7 @@ Every kind of state the core keeps, and the one component that writes it. Anyone
| State | Writer | Kept in | Others |
|---|---|---|---|
| a machine's declaration | controller (lease holder) | the bus, last per subject | read |
| a hand-over asked of a machine ([issue 356](../../04-ISSUES/356-the-operators-command-ran-a-stale-engine-and-a-hand-over-had-no-channel/00-report.md)) | controller, at its terminal | the machine, beside its state | the engine verifies the mesh's signature and records it, or refuses |
| a machine's applied state and its report | the node-engine's apply queue (§6) | the machine; the report on the bus | the reconcile and a delivery *enqueue*, never apply |
| the controller lease | the controller instance holding it | key-value `mesh-controller_lease` | a candidate waits |
| plans and their tiers | controller (lease holder), compare-and-set on the plan's revision | the controller's store | read through `plans` |
@@ -96,7 +97,8 @@ this table marks it shared. Changing a writer is a change to this table, through
**How it is enforced** ([ADR 0229](../../02-DECISIONS/0229-the-cores-order-is-a-lease-the-store-remembers-and-an-epoch-a-machine-is-sent-once-it-reads-one.md)):
the table is compiled into the controller, each row the bus carries with the subjects a write of it
publishes to — a declaration `mesh.node.*.declare`, a report `mesh.control.*.report` (its own machine's
publishes to — a declaration `mesh.node.*.declare`, a hand-over asked `mesh.node.*.ask.hand-over` (signed as
well, because a responder can reach a subject through a reply, which no grant refuses), a report `mesh.control.*.report` (its own machine's
only), the controller's buckets, stream creation, update and deletion and durable consumer creation, a
build's outcome, a merge announced, a provider's standing — and its writer among the principals. A
grant that overlaps another writer's subject is refused at composition, naming the state and its
04-ISSUES/343-the-controllers-terminal-has-no-command/00-report.md
+2 -2
View File
@@ -23,8 +23,8 @@ The same day, read through the mesh's tools: the controller runs on the control-
**The records name a command that does not exist.** Issue 339, issue 340, ADR 0266 (hq pull request 225, not yet
merged) and the agent module's README say that certain changes are made "at the controller's terminal" alone, and
some give the line to type, such as `mesh-controller settings set claude-code '<json>'` and
`mesh-controller node agent-account <node> <account>`. None of them says how the operator reaches that terminal, and
some give the line to type, such as "`mesh-controller settings set claude-code '<json>'`" and
"`mesh-controller node agent-account <node> <account>`". None of them says how the operator reaches that terminal, and
nothing in the mesh provides it:
- The controller is a mesh process. The node-engine unpacks its bundle under the mesh's own directory and writes a
04-ISSUES/343-the-controllers-terminal-has-no-command/01-diagnosis.md
+2 -2
View File
@@ -58,12 +58,12 @@ Four repositories, in the order they merge:
and runs the line through `runVerb`'s path, without `MESH_VERB` for the terminal and with
`MESH_VERB=mesh-cli` otherwise, after the `command` verb's refusals.
4. **mesh-catalog:** the module `mesh-cli` (its program, its archive into `/usr/local/bin`, its zsh alias `nox`),
assigned to every node; and the agent module's README, whose `mesh-controller settings set …` becomes
assigned to every node; and the agent module's README, whose "`mesh-controller settings set …`" becomes
`mesh-cli settings set …`.
Each code change is a branch and a pull request, merged by the operator after ADR 0272 is accepted. The records
that tell the operator to type `mesh-controller <command>` at the terminal mean `mesh-cli <command>` on the
control-node once phase 1 is live; ADR 0266's `mesh-controller node agent-account` line is changed on its own
control-node once phase 1 is live; ADR 0266's "`mesh-controller node agent-account`" line is changed on its own
branch, which is not yet merged.
## Later
04-ISSUES/354-a-decided-rename-was-never-scheduled-and-the-operator-believed-it-live/00-report.md
+2 -1
View File
@@ -29,7 +29,8 @@ On 2026-10-09 the operator took the rename as long implemented and live. It is n
`node-tools` stand among its assigned modules, and `node-tools.runtime process node-tools.service` among
what it runs.
- **The conditions.** An open condition on the control-node (messenger not healthy) says that
`mesh-host hand-over` at the machine hands a directory to the mesh; every hand-act record of the day
"`mesh-host hand-over` at the machine" hands a directory to the mesh (its words, quoted; the operator's line is
`nox node hand-over`, issue 356); every hand-act record of the day
names its caller as `<the laptop>.node-tools`.
- **The record.** `records_search` for "until the code rename" finds ADR 0244's own table row and nothing
else; for "rename mesh-host", nothing. No issue, design or record tracks the rename.
04-ISSUES/354-a-decided-rename-was-never-scheduled-and-the-operator-believed-it-live/01-diagnosis.md
+2 -2
View File
@@ -29,8 +29,8 @@ its words (ADR 0244 §5).
The state directory `/var/lib/mesh-host`: the node's state, the foundation lock, and the launcher's
files (known-good, rolled-back, trial, the start counter). The commands `cmd/mesh-host` (its flag set is
named `mesh-host`) and `cmd/mesh-bootstrap`. The placement guard's `engineModule` (`mesh-host`) and
`engineTrees` (both directories). The words a report says, which conditions quote: `mesh-host hand-over` at
the machine. The Go module path, which the controller's builder check vendors. The
`engineTrees` (both directories). The words a report says, which conditions quote:
"`mesh-host hand-over` at the machine". The Go module path, which the controller's builder check vendors. The
report names a socket under `/run/nox-mesh-host`; I did not find one in the checkout at `b0d03be`, so
the diagnosis in that repository confirms it or strikes it.
- **The tool runner**, in the `mesh-tools` repository under `node-tools/`: the module `node-tools`, the
04-ISSUES/356-the-operators-command-ran-a-stale-engine-and-a-hand-over-had-no-channel/00-report.md
+119
View File
@@ -0,0 +1,119 @@
---
status: located
opened: 2026-10-09
located-in: [mesh-host (cmd/mesh-host, internal/upgrade, internal/link), mesh-controller (cmd/mesh-controller/nodes.go, internal/broker, internal/link), mesh-catalog (checks/words)]
fixed-by:
amended-design:
---
# 356. The operator's command on a node ran a stale engine, and a hand-over had no channel
**Severity: high.** A command on every node answered for a build that was not running, and the one act a
condition asked of the operator could not be done through the mesh.
## Symptom
On 2026-10-09 at 18:01 (local) the operator, on the control-node, did what a module's condition told them to:
the messenger module was waiting for a person, its data directory used as found
([issue 339](../339-a-verb-can-place-a-directory-at-the-machines-own/00-report.md)), and the condition said to
run the node-engine's `hand-over` "at the machine", as root. They typed it, with the directory's path, and the
node-engine answered **"unexpected argument"**.
Measured on that machine, right after:
- The command on `PATH` is the engine placed by hand at installation (`/usr/bin/nox-mesh-host`, the launcher's
fallback). Its `help` lists `profile`, `inventory`, `apply`, `reconcile`, `bundle`, `overlay`, `owned` and
`version`, and no `hand-over`: it is a build from before that subcommand existed.
- The engine running the machine is a delivered one, under the engine's own versions directory, started by its
launcher ([ADR 0141](../../02-DECISIONS/0141-the-host-delivers-its-own-successor.md)). It has `hand-over`.
Nothing had ever written the delivered build over the one placed by hand: the launcher runs the fallback only
while nothing is delivered, and the engine's retirement of old versions never touches it.
- Meanwhile the messenger's process crash-looped: its data directory was owned by an account of the machine
(uid 1000, mode 700) and left so, as issue 339 decides, while the module now runs as an account of its own.
So two faults, one visible through the other:
1. **The command the operator types on a node is not the engine that runs it.** After every delivery of the
engine, the file on `PATH` falls one build further behind, and answers — with `help`, `version`, `owned`,
or a refusal — for a build the machine is not running.
2. **The hand-over of issue 339 was a terminal act at the machine with no channel.** The condition told the
operator to go to the machine, as root, and type the daemon's name. [ADR 0272](../../02-DECISIONS/0272-the-operator-reaches-the-controller-with-mesh-cli-on-every-node-and-a-node-is-the-terminal-only-where-no-agent-runs-as-its-operator.md)
had given the operator one command, `nox`, on every node, and the operator's rule of the same day says a
person never types a daemon's name; the way out of issue 339 was written before either.
## Why it is an issue
- **A tool that answers for a build that is not running is the lie this mesh exists to refuse.** The mesh's own
words: evidence over assertion, failure must be loud. The file on `PATH` asserted a version, listed commands,
and refused one that exists, with a successful-looking refusal.
- **An act the mesh asks of its operator must have a channel the mesh provides.** ADR 0272 made "at the
controller's terminal" a real place. "At the machine, as root" is the opposite: `ssh` and `sudo`, which the
operator's rules forbid for a mesh machine, and which no agent may do either.
- **The texts the mesh addresses to its operator named the daemons' lines** — the condition, the engine's
statement, the apply's log, READMEs and design documents — and nothing checked them.
## Impact
- On every node, from its first delivered engine on, `nox-mesh-host …` typed at the machine ran a stale build.
On the four nodes today that is a build from before `hand-over`, the tunnel take's `--tunnel`, and the
judgements of ADR 0266.
- The messenger module stayed down for as long as its directory stayed as found, and the only way out the mesh
named could not be typed.
## The acceptance test (the operator's view)
- After a delivery of the node-engine proves itself on a node, `nox-mesh-host version` typed at that machine
answers the build the launcher is running, and its `help` lists what that build serves.
- `nox node hand-over <node> <path>` on the control-node, from the operator's own account, hands the directory
the module's condition names to the mesh; the engine on that node says so in the answer; the condition clears
at the node's next apply, which gives the directory its declared owner and mode. The same line through any
verb, or from another node's `nox`, is refused and nothing is recorded.
- No condition, verb answer, README or design document tells the operator to type the node-engine's or the
controller's binary; `merge-check.sh` here and in the catalogue fail on one that does.
## How it is checked
- **The engine placed by hand becomes the proved build** (`mesh-host` `internal/upgrade`, `cmd/mesh-host`): on the
evidence known-good is written on — the mesh has taken a report under this build — the engine copies itself
over the fallback when the bytes differ; a copy, renamed into place, never a link. Tests: a stale fallback
becomes the proved engine's bytes with mode 0755 and the running file is untouched; one already a copy is
left as it is (same inode, no temporary file left); an engine running from the fallback itself renews nothing;
a missing fallback is placed; a link at the fallback's path is refused and its target untouched; an
unreadable running engine is said, not copied as nothing; the fallback's path is the launcher's variable or
its default. Each fails with its check taken out.
- **The hand-over is the controller's terminal's, carried to the engine** (mesh-controller
`cmd/mesh-controller/nodes.go`, `internal/link`, `internal/broker`; `mesh-host` `internal/link`,
`cmd/mesh-host`): `node hand-over <node> <path>` is a `node` subcommand that is not a read, so the controller's
terminal-only rule refuses it through every verb and through `mesh-cli` outside the terminal — tested
through `terminalOnly`, the generic command verb's composition and the ordinary mesh-cli line. The line's
node and path are judged before anything is asked (absolute, clean, no `..`, no doubled or trailing
separator; a node the mesh knows). The ask is a request on the machine's own subject. **The subject
proves nothing about who asked**: only the controller is granted a publish there (a row of the writers table,
held at composition), but the bus lets any principal allowed to answer reply to a message it received on the
reply subject that message named, so a tool server running as the operator's account could deliver a body there
(found by the review of 2026-10-09). So the controller **signs the ask** with the mesh's key — the node, the
path, who asked, an expiry a minute ahead and a fresh nonce, over a fixed context so it is never a declaration's
signature — and the engine verifies it with the key it verifies declarations with before it reads anything out
of it, then refuses an ask for another node, expired, good for too long, without a nonce of its own, or taken
before. Tests on the engine's side refuse each of those (unsigned, another key, signed as a declaration, the
bare ask, another node, expired, no expiry, too long, no or a short nonce, a replay) with the handler given
nothing; the controller's side holds the signature, the context, the nonce and the expiry. The node's grant gains
the subject and the right to answer what it was asked (the golden composition shows exactly that). The engine
then judges every value before it records anything — who asked (one short line), the path (absolute, clean,
one its state records as used as found, on disk a directory and not a link, reached through no link) — and a
refusal records nothing. **At the apply, the link is refused again at the moment of use**: a handed-over
directory must still resolve to itself, and is changed only through one descriptor opened from `/` with no
link followed on any component; a test swaps its parent, its grandparent and itself for links to a victim
tree after the hand-over, and the victim keeps its mode. The request's and the answer's field names are held by a test on each
side.
- **The words**: the engine's statement, the controller's wait and condition name
`nox node hand-over <node> <directory>` on the control-node, and never the engine's binary; the condition's
words are plain (a path is in the evidence, never in the summary). hq's `words.py` and the catalogue's
`checks/words` refuse a daemon's line (ADR 0272, its insight of 2026-10-09); each has a test of its pattern,
and `words.py` compares the catalogue's copy.
- **Live, at acceptance:** after the controller and then the engine are delivered, the four nodes' journals say
"renewed the engine placed by hand … to this build", and `nox-mesh-host version` at a machine answers the
delivered build; `nox node hand-over <control-node> <the messenger's data directory>` answers that it is handed
to the mesh, the messenger's condition clears at the next apply, and the module runs.
The diagnosis is in [01-diagnosis.md](01-diagnosis.md).
04-ISSUES/356-the-operators-command-ran-a-stale-engine-and-a-hand-over-had-no-channel/01-diagnosis.md
+85
View File
@@ -0,0 +1,85 @@
# Diagnosis — issue 356
## 2026-10-09, 18:01 — what was measured
On the control-node, the operator typed the node-engine's `hand-over` with the messenger's data directory and was
answered "unexpected argument". `help` from the same command listed no `hand-over`. The launcher's journal named
the engine it runs: a delivered build under the engine's versions directory. The messenger's unit crash-looped;
its data directory was used as found, owned by a machine account (uid 1000) with mode 700, while the module had
moved to an account of its own.
## The causes, from the code
**1. Nothing ever wrote a delivered engine over the one placed by hand.** The launcher (`mesh-host`
`packaging/nox-mesh-host-launch`, delivered as the engine's module file) picks the newest delivered version that
was not rolled back, and falls back to `/usr/bin/nox-mesh-host` — the file a person copied at installation
([ADR 0141](../../02-DECISIONS/0141-the-host-delivers-its-own-successor.md)) — only while nothing is delivered.
The engine's own upgrade code (`internal/upgrade`) delivers under the versions directory, proves a build by
writing `known-good`, and retires versions older than the running one's predecessor; it never reads or writes the
fallback. The fallback is also the one `nox-mesh-host` on any `PATH`, so from the first delivery on, what a person
typed was a build older than what ran. Nothing said so: the engine's `version` answers from its own path, and the
stale file answers its own.
**2. The hand-over was a local subcommand only.** Issue 339's way out, `hand-over <dir>` (`mesh-host`
`cmd/mesh-host/found.go`), reads the node's state and writes `handed-over.json` beside it; the next apply reads
that file. It had no bus side: the engine hears the mesh on its declaration subject and on
`mesh.node.<node>.ask.report` only, and the controller had no `node hand-over`. The engine's statement, the
controller's wait text (`module_health.go`) and the condition's "needs" line all told the operator to run it "at
the machine", as root.
**3. No check held the texts to ADR 0272.** `words.py` reads the glossary's retired words; a daemon's command
line is not a word.
## Ruled out
- A delivery that failed half way (the versions directory holding an older build): the launcher's journal named
the delivered build, and `known-good` named it.
- The launcher running the fallback: it does so only while nothing is delivered, and three versions were.
## The fix
**Which fix for the stale command.** Two were weighed. (a) `/usr/bin/nox-mesh-host` becomes a small program that
runs the launcher's current version. Rejected: the launcher runs that very file as the engine while nothing is
delivered, so a shim there would run itself; and the choice of version (newest delivered, never one rolled back,
a rollback's pin) is the launcher's policy, which a shim would copy. (b) The proved engine writes itself over the
file when the bytes differ: a copy renamed into place, never a link, on the same evidence `known-good` is written
on. Chosen. The launcher's fallback is then the newest proved build, which is also the best one to fall back on,
and the command on the machine is the engine that runs it.
**The hand-over's channel.** `nox node hand-over <node> <path>` on the control-node. The controller's `node`
subcommands that are not reads are terminal-only already (ADR 0266's allow list), so the line is refused through
every verb and through `mesh-cli` from any other node without a new rule. The controller judges the line, asks
the node's engine on `mesh.node.<node>.ask.hand-over` — a request on core NATS — and prints the engine's answer,
failing on a refusal. Only the controller is granted a publish on that subject (a writers-table row), but that
does not say who the engine hears: the bus lets a principal allowed to answer reply to a message it received on
any reply subject the message named, and the tool runner, which runs as the operator's account and so as every
agent, is allowed to answer (the review of 2026-10-09 proved it on a permissioned server). So the ask is signed
with the mesh's key (node, path, who asked, a minute's expiry, a nonce; over a fixed context, so the signature is
never a declaration's), and the engine verifies it before reading anything, refusing another node's, an expired
one and a replayed nonce. The engine judges every value and records the hand-over, or refuses and records
nothing. **The apply checks again**: a component of the path may have become a link after the hand-over was
recorded, so a handed-over directory must still resolve to itself and is changed only through a descriptor opened
from `/` that followed no link. The engine's local `hand-over` stays, for a
machine no mesh can reach; its usage says the operator's line.
**The words.** Every text addressed to the operator names the `nox` line. hq's `words.py` gains the daemons'
lines (ADR 0272's insight of 2026-10-09 says the rule and the check); the catalogue's `checks/words` refuses the
same lines in module strings and manifest descriptions, from its copy `retired-lines`.
The pull requests: `mesh-host` #74 (the fallback renewed, the engine's side of the ask, its words),
mesh-controller #187 (the `node hand-over` line, the node's grant, its words),
mesh-catalog #165 (the catalogue's check and its copy of the lines), and this one (the record, `words.py`, the
documents' lines).
**Order of delivery.** The controller first: the node's bus grant (the subject, and the right to answer) is in
the controller's composition of the user list, and an engine delivered before it would have its subscription
refused and say so. Then the engine. The catalogue's check and this record in any order.
## What remains
- A mesh-lab replay of the stale command (an engine delivered twice, the fallback compared) and of the hand-over
(asked, refused through a verb, recorded through the terminal) is owed before this issue resolves: a core
issue resolves with `replay:` ([ADR 0237](../../02-DECISIONS/0237-a-change-is-judged-against-the-mesh-that-runs-before-it-merges-on-the-build-seat.md)).
- The engine's other local subcommands (`key`, `enrol`, `overlay take`, `reconcile`, `apply`, `profile`,
`inventory`, `owned`, `bundle`, `version`) stay install-time and rescue commands, typed as root at a machine no
mesh reaches; the engine's usage and README say so. A `nox` line for any of them is a later decision.