Issue 152: a node the mesh could not read withdrew its names from every machine #191

2026-09-29T21:26:45Z

jschoubben commented

2026-09-29 21:26:45 +00:00

Records the live fault behind tonight's churn on the control node, and marks it resolved by the mesh-controller branch of the same name.

Also reconciles a numbering collision: ace opened its two records as 147 and 148, which this repository already uses. They become 150 and 151, as the second 127 became 149.

151 stays open — this removes the false reasons a roster changes, not the fact that a real change still replaces every container.

Stacked on decisions/settle-the-proposed and ace's branch, so its own diff is the last two commits.

Checks: records, index and cycle all pass.

Records the live fault behind tonight's churn on the control node, and marks it resolved by the mesh-controller branch of the same name. Also reconciles a numbering collision: ace opened its two records as 147 and 148, which this repository already uses. They become 150 and 151, as the second 127 became 149. 151 stays open — this removes the false reasons a roster changes, not the fact that a real change still replaces every container. Stacked on `decisions/settle-the-proposed` and ace's branch, so its own diff is the last two commits. Checks: records, index and cycle all pass.

jschoubben added 15 commits 2026-09-29 21:26:45 +00:00

Issue 146: a first node now enrols, and is enrolled twice f6ed3545b7

Two more faults behind the three already fixed. The account a token is the
password of was never recorded, and the comment above the issuing code said
it was; issuing now records it. Placing the composed list at genesis is the
other half — the control plane says what it composed and whoever raises the
machine writes it beside the bus, because no declaration can reach a machine
that has not enrolled.

With that a first node enrols. It is then enrolled twice from one attempt,
each minting a credential, and it keeps the answer to the first while the mesh
keeps the second. The trail for that one stops at a duplicate that survived
message-id deduplication.

Issue 146: what made it slow, and what was changed so it is not e9b1010bc0

Issue 146: the double enrolment was two consumers sharing a delivery subject eef54917ec

Not about enrolment. A push consumer delivers onto an ordinary subject and
everything subscribed to it gets a copy; the controller's two consumers were
both named after it, so both were given the same subject and the one process
acted on every message twice. Enrolment is where it drew blood because a second
enrolment mints a second credential.

Issue 147: the operator's tools still dial the bus that was removed 0dd00e88b6

Every tool call fails with 'AMQP not connected', on every node including the
local one, because the tool surface still opens an AMQP connection and that
transport was deleted at the cut-over. The mesh reports healthy throughout —
what broke is the thing standing outside asking it questions, so nothing the
mesh checks is about it.

Issue 147: the tool surface is not the mesh's, it is the predecessor's 3c0f7082e6

Diagnosed from the configuration: the tool server is HAL's brain, a local
process on the workstation with the predecessor's broker URL in the assistant's
own config. No manifest, no assignment, no seat, no account. Nothing regressed
— the mesh removed a transport this program still dials, and the program was
never part of the mesh. The mesh has a tool model and nothing publishes an
operator-facing surface onto it.

Grooming: five issues were fixed and never closed, and one is not 14be8576f8

088, 089, 120, 128 and 130 each name a commit that is on main and cites them —
the forge's address following a moved port, a route naming its endpoint, a
provisioner asking the backend what is there, the hosts file written into a
marked block, and undeclaring giving a unit back the state it was found in.
Each says it was closed by reading commits rather than by a run, so nobody
reads a green that was never measured.

129 stays located on purpose: ca-trust is merged and no machine holds it, so
the symptom it opened on is still true everywhere.

Grooming: issue 006's knowledge base is the predecessor's, and is gone 9a1dc4665c

The record catches README claiming this repository is indexed into a knowledge
base. Nothing indexed it, and since the cut-over there is nothing to index it
into — the surface that answered is on the transport the mesh removed (issue
147). Noted where the record is, so the next reader does not go looking for a
search that cannot exist.

Merge the 146/147 records, which 006's note links to d199de40db

Three proposed records were already built; two are still yours to call 4eb16f1028

0037 (where a module lives), 0113 (the vault makes every secret) and 0115 (one
assignment of a module per node) described arrangements the mesh has — a
catalogue of other people's software plus a table filled by module add, a vault
answering a secret requirement six modules make, and a rule the assignment
table's primary key already enforces. Each is accepted against what was built,
and says so in its own words.

0037's other half is not built: a manifest outside this catalogue has no check,
which is issue 148.

0068 (the lab takes requests) and 0114 (a shared credential rotates over two
credentials) stay proposed. Neither is built, and both are decisions rather
than records of something that happened.

Two records were numbered 127; the second becomes 149, and is resolved 96bdffa9bc

A number identifies a record, and two were given 127 on 2026-09-27. The one
three documents and three source files cite by number keeps it; the other
becomes 149, says so in its own heading, and its two inbound references are
repointed.

It is also resolved: an empty declaration is sent carrying owns_nothing rather
than skipped, and the host refuses an empty body that does not carry it, so
emptiness cannot be read as a truncated declaration.

Issue 114 is answered: the controller is a process, by ADR 0142 2b5119ecd2

It asked a question rather than reporting a defect, and the question was taken
five days later — the mesh's own components are binaries on the machine, and
third-party software stays a container because an image is the right way to
carry somebody else's build. The delivery of them is issue 142.

Issues 147 and 148: a route before its module is taken; a new name recreates every container 3bd6f34de3

Both found migrating the first module on ace (searxng), adopted with route-adapter:
147 — assign is not a no-op for a routed module: its route reaches the predecessor's
proxy while its container is held, and the name serves a dead backend until take.
148 — every push that moved the mesh's names replaced every container on novox, twice,
the control plane's own store included.

Merge remote-tracking branch 'origin/issue/147-148-found-migrating-ace' into issue/the-roster-flicker-recreates-every-container 741625e725

Issue 152: a node whose plan will not compose withdraws its names from every machine 72eaf52867

ace numbered its two records 147 and 148, which this repository already
uses; they become 150 and 151, as 127 became 149.

The new record is why the control node could not stop applying: the
roster alternates between two values because routeNamesInTheMesh
swallows a per-node plan failure, and the roster is part of every
container's identity. The loop closes through the control plane's own
store, which each pass replaces.

Issue 152 is fixed: a lookup failure is no longer an absence 3c2b4fc6b6

The three gatherers pass over a node whose set does not compose, and
raise anything else. 151 stays open: this removes the false reasons a
roster changes, not the fact that a real change still replaces every
container.

jschoubben added 1 commit 2026-09-29 21:27:53 +00:00

Issue 152: the loop is metastable, and it cleared at 23:11 18f37c25b2

It ran 22:46-23:11, five full replacements, and stopped when a pass
happened to read the store during a window it was up. The record said
nothing about it stops; that was wrong. Exiting by luck is the finding,
not a mitigation.

jschoubben merged commit 0d64677c70 into main

2026-09-29 21:37:17 +00:00

jschoubben deleted branch fix/152-a-lookup-failure-is-not-an-absence

2026-09-29 21:37:17 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: novox/hq#191