novox/hq
3
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

self-hosting, provisioning and delivery efforts, and the dotfiles origin

Browse Source 
The identity provider is settled as not-substrate: the mesh does not
require one, tier 2 authenticates natively, and it is a hosted service
like any other. Four substrate services, not five. The tier test's second
step gains the verb that matters — can the control plane START without it,
not function fully without it.

That verb answers the forge and the registries. They are not substrate and
they are not duplicated: the control plane starts and manages nodes
without a forge, it just cannot change itself. One gitea module, tier 4,
and the mesh's own instance is distinguished by what it is bound to rather
than by being a different module — the same answer as postgres, from the
same test. It also buys a property worth having: if the forge dies the
mesh keeps running.

Delivery needing them is not an upward dependency, resolved the way the
constitution already says to: tier 2 declares requirements, tier 4
provides implementations, the binding is data. The mechanism is
provisioning, and the new idea is that the control plane is itself a
consumer.

Self-hosting therefore becomes a state the mesh REACHES, not a
precondition. A first node comes up from pinned external artifacts and
re-binds to internal providers once they exist. Today's mesh assumes the
second state from the first moment, which is why the first-node path needs
a script that papers over an impossibility and is the least-exercised code
in the system. Made explicit, the transition is also reversible.

Research 007 and 008 opened for the two areas flagged as important and
complex, scoped from the weaknesses the as-is layer already documents
rather than started blank.

And the origin: this began as a dotfiles repository. The first two days
adopt dotfiles, add per-node overrides, and introduce service symlinking
with an ignore file. The flat one-directory-per-tool catalogue, linking
over copying, adoption of already-configured machines, per-node overrides
and the desktop modules are all inherited rather than chosen for a mesh.
That is the single most useful fact for anyone changing the catalogue, it
strengthens ADR 0018 — the case for links was never made for a mesh — and
it explains research 005's silent fifty: dotfiles-era entries for one tool
never shared a domain because they never had one.
This commit is contained in:
jschoubben
2026-08-23 20:55:14 +02:00
parent 7a20358113
commit daf3e17c32
5 changed files with 249 additions and 8 deletions
Download Patch File Download Diff File Expand all files Collapse all files
01-RESEARCH/006-mesh-from-scratch/00-overview.md
+11
View File
@@ -40,6 +40,17 @@ look unsolvable from inside that frame:
Designing from nothing is a way to find out which parts of the current shape are requirements
and which are residue.
**And there is more residue than expected, from a knowable source.** This began as a
**dotfiles repository** — the first two days of history adopt dotfiles, add per-node dotfile
overrides, and introduce service symlinking with an ignore file. The flat one-directory-per-tool
catalogue, linking rather than copying, adoption of already-configured machines, per-node
overrides, and the desktop modules are all inherited from that, not chosen for a mesh. Recorded
in [`03-DESIGN/00-as-is/10-module-catalogue.md`](../../03-DESIGN/00-as-is/10-module-catalogue.md).
That makes this effort's question sharper than "what would we do differently": much of what
looks like design is a generalisation of *place files on my machines*, never revisited because
it was never stated as an assumption.
## The requirements this is designed against
Stated by the operator, recorded here so the skeleton can be checked against them rather than
01-RESEARCH/006-mesh-from-scratch/code-skeleton.md
+98 -8
View File
@@ -15,7 +15,9 @@ A decision procedure, so placement is answerable rather than argued. Ask in orde
wins:
1. **Does it apply state on a machine?** → tier 0, inside the host.
2. **Can the control plane exist without it?** If *no* → tier 1, substrate.
2. **Can the control plane *start* without it?** If *no* → tier 1, substrate. Note the verb:
*start*, not *function fully*. A capability the control plane loses without something is not
the same as a thing it cannot come up without — see "becoming self-hosting" below.
3. **Does it decide what should be true across nodes?** → tier 2, a control-plane context.
4. **Is it a way to talk to tier 2, holding no logic of its own?** → tier 3, a surface.
5. **Otherwise** → tier 4, a workload.
@@ -50,13 +52,102 @@ inverse — substrate reaching into the catalogue for a module — would not be,
the naive "twice" answer would have created.
The same reasoning places the rest of the substrate: the bus, the object store, the image
registry. Each fails step 2, each lives once, each is pinned.
registry. Each fails step 2, each lives once, each is pinned. That is the whole substrate: four
services, and the deliberate absence of a fifth.
**One genuine boundary case, flagged rather than decided.** The identity provider passes step 2
only if the control plane delegates authentication rather than doing it natively. If tier 2
authenticates callers itself, the identity provider drops to tier 4 and the substrate has four
services instead of five. The test does not answer this; it turns it into a question with a
clear shape, which is what a test is for.
**The identity provider was raised as a boundary case and is settled: it is not substrate.**
The mesh does not require one — tier 2 authenticates its own callers natively, and an identity
provider is a service the mesh hosts like any other. The substrate is four services, not five.
The test earned its keep here by turning a vague unease into one answerable question — *does the
control plane delegate authentication?* — rather than a debate about how important identity
feels.
## Becoming self-hosting — the forge and the registries
Self-improvement means the mesh hosts the things it improves itself with: a forge, an image
registry, a package registry. The obvious worry is that these duplicate — a `hal-mesh-gitea`
for the mesh and a `gitea` for everyone else. **They do not, and the reason is worth stating
carefully, because it is the same reason the bootstrap keeps failing today.**
### They are not substrate
Run the test with the sharpened verb. *Can the control plane start without a forge?* **Yes.** It
comes up, holds inventory, answers questions and manages nodes with the modules it already has.
What it cannot do is **change itself**. That is a capability, not a precondition.
So: forge, image registry and package registry are **tier 4 workloads**. One module each, in the
catalogue, exactly like the media server. There is no mesh-specific copy.
This is not a technicality. It buys a property worth having: **if the forge dies, the mesh keeps
running.** Nodes stay managed, services stay up, only self-modification stops. Putting the forge
in the substrate would make losing it fatal, for no gain.
### But delivery needs them — is that not an upward dependency?
It would be, stated naively, and that would break the one rule the whole skeleton rests on.
It is resolved the way the constitution already says to resolve it — **depend on abstractions,
not on concrete dependencies**. Tier 2's delivery context does not require *the forge module*.
It declares requirements:
| Delivery requires | Satisfied by |
|---|---|
| a source of record for module code | whichever module provides it |
| somewhere to publish images | whichever module provides it |
| somewhere to publish packages | whichever module provides it |
| somewhere to put build artifacts | the substrate's object store |
Tier 2 defines the requirement; tier 4 provides the implementation; the binding is data. The
dependency points **downward from the provider to the interface**, which is legal, and the
control plane never names a concrete module.
The mechanism for this already exists and is the mesh's most valuable one: **provisioning**. A
module declares what it provides; a consumer declares what it requires; the mesh binds them.
The only new idea is that **the control plane is itself a consumer** — it has requirements, and
they are satisfied the same way a workload's are.
That generalisation is significant enough to need its own study, and is not settled here.
### Self-hosting is a state the mesh reaches, not a precondition
This is the part today's mesh gets wrong, and it explains a recurring class of pain.
A first node comes up from **pinned external artifacts** — upstream images, by digest, carried
in the bundle. It has to: the mesh's own registry does not exist yet, and cannot. The mesh at
this point is running and manages nodes, and is not yet self-hosting.
Self-hosting is then **reached**: the forge is installed as an ordinary workload, the mesh's own
source moves into it, the registries come up, and delivery's requirements are re-bound from
external providers to internal ones. From that point the mesh builds and deploys itself.
Stated as a lifecycle:
```
pinned external artifacts ─► mesh runs, manages nodes
│
│ forge + registries installed as workloads
│ delivery's requirements re-bound
▼
mesh builds and deploys itself
```
**Today's mesh assumes the second state from the first moment.** Its source, its packages and
its images are all expected to be self-hosted before there is anything to host them — which is
why raising a first node needs a script that exists solely to paper over the impossibility, and
why that script is the least-exercised path in the system.
Making the transition explicit has a second benefit: it is reversible. A mesh whose forge is
broken can re-bind delivery to external providers and keep improving itself while it repairs
the forge. Today that escape hatch does not exist, because the dependency is not expressed
anywhere it could be changed.
### So, concretely
One `gitea` module. One image-registry module. One package-registry module. Each a tier-4
workload. The mesh's own instances are distinguished from any other instance **by what they are
bound to, not by being different modules** — precisely the same answer as postgres, arrived at
by the same test.
## The five fates of a module
@@ -145,7 +236,6 @@ hal-substrate/ TIER 1
bus/<broker>/
objects/<object-store>/
images/<registry>/
identity/<idp>/ boundary case — see above
hal-mesh/ TIER 2
record/ the event log contexts integrate through
01-RESEARCH/007-provisioning-as-the-core/00-overview.md
+51
View File
@@ -0,0 +1,51 @@
---
status: active
initiated: 2026-08-23
touches:
- 03-DESIGN/00-as-is/03-provisioning.md
- 02-DECISIONS/0005-capabilities-are-provisioned-on-declaration.md
- 01-RESEARCH/006-mesh-from-scratch/code-skeleton.md
became: []
---
# 007 — Provisioning as the mesh's core mechanism
## What is being investigated
Provisioning is the mechanism the whole mesh rests on: a module declares what it needs, and the
mesh makes it exist, generates the credential, records the grant, and puts the values where the
module will read them. [ADR 0005](../../02-DECISIONS/0005-capabilities-are-provisioned-on-declaration.md)
calls it the mesh's core concern rather than its plumbing.
[Research 006](../006-mesh-from-scratch/code-skeleton.md) then asks it to carry **more**: the
control plane becomes a consumer with its own requirements — a source of record, an image
registry, a package registry — satisfied by the same mechanism. That generalisation is only
safe if the mechanism is sound, and the as-is record says it is not, in named ways.
## Why now
Four weaknesses are already documented in
[`03-DESIGN/00-as-is/03-provisioning.md`](../../03-DESIGN/00-as-is/03-provisioning.md), each
observed rather than theorised:
- **Rotation has no fan-out.** A shared credential can be rotated without telling the peers
holding the old one. This has locked the mesh out of its own broker.
- **A grant is not a check.** The record says a resource was provisioned. Nothing verifies it
still exists, still has that credential, or is reachable from where the consumer runs.
- **A frozen password outlives its generation.** A generated secret written once diverges from a
persistent data directory initialised earlier, and presents as an authentication error.
- **No requirements is indistinguishable from provisioning that did not run.** A module that
declares nothing skips the stage, which is correct, and looks identical to failure.
Generalising a mechanism with these properties to the control plane's own dependencies would
make each of them fatal rather than annoying.
## The questions
| Question | Why it matters |
|---|---|
| What does a **grant** mean, exactly — a record that a resource was created, or a claim about the world that is continuously reconciled? | The difference between the current model and one where "provisioned" is checkable. Almost every weakness above is a symptom of the first answer. |
| How is a credential **rotated** with fan-out to every holder? | The mechanism grants easily and regrants not at all. This is the most damaging gap and it has taken the mesh down. |
| Can the **control plane** hold requirements, and what satisfies them before anything is installed? | The generalisation research 006 needs. Ties directly to the self-hosting transition. |
| What happens when a requirement **cannot** be satisfied — no provider, provider on an unreachable node, provider not yet installed? | Today this is silent or a stall. It should be a stated, visible state. |
| Does a requirement belong to a **module** or to one of its **parts**? | Research 006 splits `feature` into artifact and part. A part-scoped requirement means a database is not provisioned where the part that needs it is not installed. |
01-RESEARCH/008-delivery-coordinator/00-overview.md
+53
View File
@@ -0,0 +1,53 @@
---
status: active
initiated: 2026-08-23
touches:
- 03-DESIGN/00-as-is/04-delivery.md
- 02-DECISIONS/0014-build-publish-and-deploy-are-three-silos.md
- 02-DECISIONS/0013-an-artifact-is-build-output.md
- 01-RESEARCH/006-mesh-from-scratch/code-skeleton.md
became: []
---
# 008 — The coordinator: a change checked in becomes a deployed state
## What is being investigated
The mesh's own continuous delivery: a change is committed, and the mesh ends up in the state
that change describes — across every node the change touches, with a verdict that says whether
it worked.
The coordinator is what orchestrates that, and it is the mesh's most consequential machinery:
everything reaches every node through it.
## Why now
The as-is record ([`03-DESIGN/00-as-is/04-delivery.md`](../../03-DESIGN/00-as-is/04-delivery.md))
names problems that are structural rather than incidental:
- **A green pipeline proves transport, not effect.** The stages report that a message was
dispatched and accepted, which is not the same as the thing running, correct, or present.
This is the mesh's single most consistent failure shape.
- **Detection is the most fragile input.** A merge that creates no pipeline, with nothing saying
so, is the characteristic bad outcome — and it has happened for reasons unrelated to the
change.
- **The fan-out point is asymmetric.** The build node has already passed two silos when work
fans out, and code that knew only about the first parked it forever while every other node
deployed cleanly.
- **There is no end-to-end coverage.** The harness has not built since 2026-06-04
([`04-ISSUES/005`](../../04-ISSUES/005-pipeline-test-harness-unbuildable/00-report.md)).
Research 006 adds a requirement the current design does not have: the coordinator must work
**before the mesh is self-hosting**, when source and artifacts come from outside, and keep
working across the transition to self-hosted providers.
## The questions
| Question | Why it matters |
|---|---|
| What is a **deployed state**, and how does the mesh know it is in one? | Everything follows from this. If a stage reports transport, "deployed" is a claim nobody checked. A desired-state model with reconciliation gives a different answer from a job-completion model. |
| Does the coordinator dispatch **stages**, or converge nodes on a **declaration**? | The current model is a state machine over stages. The alternative is that a node is told what should be true and reports what is. The second makes drift visible; the first cannot see it. |
| How does a change **become** a pipeline, reliably? | Detection has failed for reasons unrelated to the change, silently. |
| What produces a **verdict**, and what is it a verdict about? | Ties to the lab ([ADR 0016](../../02-DECISIONS/0016-a-lab-node-is-a-virtual-machine.md)) and to a module carrying its own assertions. |
| How does delivery work **before self-hosting**, and across the transition? | From research 006: source and artifacts start external and are re-bound to internal providers. The coordinator has to be indifferent to which. |
| Does the **three-silo** split survive the artifact/part split? | [ADR 0014](../../02-DECISIONS/0014-build-publish-and-deploy-are-three-silos.md) is cardinality-driven, and research 006 renames the thing the cardinality is about. |