novox/hq
3
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

research 006: the code skeleton, and where postgres lands

Browse Source 
A tier test as a decision procedure — five ordered questions, first match
wins — so placement is answerable rather than argued.

Postgres was the test case and the naive answer is wrong. Not twice, once:
the control plane cannot exist without a relational store, so it is tier 1
and lives in hal-substrate/store/postgres. What differs between the mesh's
own database and a project's is not the module but how that instance is
brought up — pinned bundle applied by the host, versus the ordinary
delivery and provisioning path. Tier is a property of the module; the
bundle is a property of the mesh's own instance. The naive answer would
also have made substrate reach up into the catalogue, which the dependency
rule forbids.

Working the test across the catalogue surfaces a third fate that neither
of research 005's options covers, and it is the most common one: absorbed
into the host, ceasing to be a module at all. That explains 005's one
positive measurement rather than confirming it — the reachability cluster
is not four modules that should be one domain module, it is four facets of
one thing the host should own, expressed as modules because a module was
the only unit available. Under this skeleton the overlay and firewall
modules stop existing. It also partly answers the silent fifty: several
are host concerns, so silence was the right signal and grouping was the
wrong inference.

Flags rather than settles: the identity provider is a genuine boundary
case (four substrate services or five), and absorbing six concerns into a
binary whose argument is that it has no dependencies is the skeleton's
biggest unproven claim.
This commit is contained in:
jschoubben
2026-08-23 20:40:32 +02:00
parent 00b8398d07
commit 7a20358113
3 changed files with 188 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
01-RESEARCH/005-domain-grouping/00-overview.md
+1
View File
@@ -58,3 +58,4 @@ open questions below.
| Whether provider modules group at all, and if so what a consumer's requirement names instead of a module. | The provisioning reference is load-bearing; getting it wrong is expensive. Opinion and evidence in the analysis; not yet decided. |
| Whether applications group into domains now and leave the monorepo later as a unit, or leave first. | Decided in principle — group first, then split — but the migration order has real cost either way. |
| What to do with the ~50 modules that co-change with nothing. | The evidence gives no grouping signal for them at all. That may mean they are correctly sized already. |
| Whether "group or leave" is even the right pair of options. | [Research 006](../006-mesh-from-scratch/code-skeleton.md) finds a third fate — **absorbed into the node host**, ceasing to be a module at all — and argues it is the correct answer for the reachability cluster this effort measured. If so, the cluster this effort found is evidence for absorption rather than for grouping. |
01-RESEARCH/006-mesh-from-scratch/00-overview.md
+5 -2
View File
@@ -17,7 +17,9 @@ What the mesh would look like if it were laid out today, with the requirements k
of the accumulated shape — expressed as a **skeleton**: repositories at the root, modules
inside them, and whatever turns out to be the right leaf unit below that.
The deliverable is [`skeleton.md`](skeleton.md).
The deliverables are [`skeleton.md`](skeleton.md) — tiers, repositories and the four design
moves — and [`code-skeleton.md`](code-skeleton.md) — the tier test, what a module looks like on
disk, and where today's catalogue lands.
## Why
@@ -76,5 +78,6 @@ the catalogue where modules genuinely change together under one intent. The skel
| Does the record — the event log contexts integrate through — belong to the substrate or the control plane? | It is infrastructure by shape and domain by content. Placing it wrong reintroduces a circularity. |
| One repository per tier, or per context? | Already open from ADR 0015 as "catalogue destination — one repository or many". The skeleton assumes per tier and does not settle it. |
| Does an unprivileged node earn a place in the inventory, or only a presence? | Decides whether "node" means one thing or two. |
| Does absorbing overlay, filtering, packages, supervision and the container runtime make the host too large? | It is the skeleton's biggest unproven claim. A binary whose whole argument is that it has no dependencies now carries six concerns. |
| Four substrate services or five? | The identity provider passes the tier test only if the control plane delegates authentication rather than doing it natively. |
| Does `feature` survive? | The skeleton splits it in two and argues the conflation is what makes the delivery pipeline hard to reason about. Unproven. |
| Is `tier` the right word? | It is this document's coinage, not established vocabulary, and it did not land on first reading. `boot order` and `ring` are the alternatives. The concept is settled; the word is not. |
01-RESEARCH/006-mesh-from-scratch/code-skeleton.md
+182
View File
@@ -0,0 +1,182 @@
---
effort: 006-mesh-from-scratch
updated: 2026-08-23
---
# The code skeleton
[`skeleton.md`](skeleton.md) laid out tiers and repositories. This is the level below: what a
module looks like on disk, and — the question that forced a correction — where a given piece of
today's catalogue actually lands.
## The tier test
A decision procedure, so placement is answerable rather than argued. Ask in order; first match
wins:
1. **Does it apply state on a machine?** → tier 0, inside the host.
2. **Can the control plane exist without it?** If *no* → tier 1, substrate.
3. **Does it decide what should be true across nodes?** → tier 2, a control-plane context.
4. **Is it a way to talk to tier 2, holding no logic of its own?** → tier 3, a surface.
5. **Otherwise** → tier 4, a workload.
## Worked example — where postgres ends up
The obvious answer is "twice": once as the mesh's own database in the substrate, once as a
hosted database in the catalogue. That answer is wrong, and seeing why fixes something.
Run the test. *Can the control plane exist without a relational store?* No. **Postgres is
tier 1.** It lives once:
```
hal-substrate/store/postgres/
```
There is no second copy in the catalogue, because the thing that differs between the mesh's own
database and a project's database is **not the module**. It is how that instance is brought up:
| | The mesh's own instance | A project's database |
|---|---|---|
| Brought up by | the host, from the pinned bundle, with no control plane present | the ordinary delivery and provisioning path |
| Declared in | `hal-substrate/bundle.yml` | the consuming module's `requires:` |
| Exists because | the control plane cannot start without it | something asked for it |
Same module, two roles. **Tier is a property of the module** — what must exist before what — and
**the bundle is a property of the mesh's own instance.**
This is also legal under the dependency rule, which is worth checking rather than assuming: a
tier-4 workload that requires a database depends on tier 1, which points *downward*. The
inverse — substrate reaching into the catalogue for a module — would not be, and is the shape
the naive "twice" answer would have created.
The same reasoning places the rest of the substrate: the bus, the object store, the image
registry. Each fails step 2, each lives once, each is pinned.
**One genuine boundary case, flagged rather than decided.** The identity provider passes step 2
only if the control plane delegates authentication rather than doing it natively. If tier 2
authenticates callers itself, the identity provider drops to tier 4 and the substrate has four
services instead of five. The test does not answer this; it turns it into a question with a
clear shape, which is what a test is for.
## The five fates of a module
Research 005 asked whether a catalogue module should be **grouped into a domain** or **leave
the repository**. Working the tier test across the catalogue surfaces a third answer that
neither option covers, and it is the most common one.
| Fate | Means | Examples from today |
|---|---|---|
| **Absorbed into the host** | It is not a module at all. It is part of what "managing a machine" means, and belongs in tier 0. | overlay membership, packet filtering, package management, service supervision, container runtime, filesystem management |
| **Substrate** | The control plane cannot exist without it. Pinned, host-applied. | relational store, bus, object store, image registry |
| **Control-plane context** | It decides something across nodes. | connectivity policy, inventory, delivery, provisioning, observability |
| **Workload module** | The mesh hosts it. Grouped per [ADR 0017](../../02-DECISIONS/0017-modules-outside-the-core-are-grouped-by-domain.md). | media library, desktop session, collaboration tooling |
| **Leaves the repository** | A standalone application, per [ADR 0010](../../02-DECISIONS/0010-applications-live-in-their-own-repository.md). | the applications identified in research 005 |
**The first fate is the finding.** Research 005 measured the reachability cluster — proxy,
resolver, firewall, overlay — as the only place in the catalogue where modules genuinely change
together under one intent. The skeleton explains *why*: they are not four modules that ought to
be one domain module. They are four facets of one thing the host should own, currently
expressed as modules because a module was the only unit available.
Under this skeleton the overlay module and the firewall module **stop existing**. The host holds
membership and applies filtering; tier 2 decides the policy; the swappable backends stay
modules. That is a different and better answer than grouping them, and it was not visible from
inside the current frame.
It also partly answers research 005's other open question — the fifty modules that co-change
with nothing. Several are host concerns rather than domains: package management, container
runtime, filesystem tooling. Silence was the right signal after all; the wrong conclusion was
that grouping was the only available fix.
## What a module looks like on disk
Per [`skeleton.md`](skeleton.md) Move 4, a module declares **parts** — independently selectable
pieces of desired state — and produces **artifacts** — things built once per version.
```
<module>/
module.yml identity, what it provides, what it requires,
which host profiles it can land on
parts/
service/ desired state: container, volumes, exposure
provisioner/ how it grants its resource to consumers
migrations/ its own persistent state
tools/ capabilities it contributes
artifacts/
<name>/ source for something built and published
```
A module with no artifacts consumes an upstream image and builds nothing. A module with no
parts is not a module.
Worked through for the substrate's relational store:
```
hal-substrate/store/postgres/
module.yml provides: database · profiles: [managed]
parts/
service/ the container, its volume, its network exposure
provisioner/ grants a database and role to a consumer
migrations/ none — it holds no state of its own
artifacts/ none — upstream image, pinned by digest in bundle.yml
```
## The tree, at file level
```
hal-host/ TIER 0
cmd/host/
internal/
apply/ reconcile declared state
inventory/ what this machine is and can do
link/ outbound connection to the control plane
overlay/ membership: address, keys, tunnel
filter/ packet filtering from tier-2 policy
packages/ package management
services/ supervision
containers/ container runtime
store/ embedded local state
profile/ managed · user · edge
substrate.lock pinned tier-1 descriptor
hal-substrate/ TIER 1
bundle.yml the pinned set, by digest
store/postgres/
bus/<broker>/
objects/<object-store>/
images/<registry>/
identity/<idp>/ boundary case — see above
hal-mesh/ TIER 2
record/ the event log contexts integrate through
inventory/ nodes · modules · assignments · versions
config/ settings · secrets · derivation
connectivity/ addresses · resolution · exposure · filtering · certificates
provisioning/ grants between modules
delivery/ source → artifact → node
observability/ health · logs · metrics
identity/ agents · humans · services · authorisation
work/ tasks · workflows · runs
knowledge/ memory · documents · retrieval
api/ the one interface surfaces speak to
hal-surfaces/ TIER 3
tools/ web/ cli/
hal-catalog/ TIER 4
<domain>/<module>/ layout as above
hal-lab/ hal-sdk/ hal-hq/
```
## What this does not settle
- **The identity boundary case.** Four substrate services or five.
- **Where the record lives.** Still the open question from
[`00-overview.md`](00-overview.md), and the tier test does not resolve it: the record is
needed by tier 2 and is *of* tier 2, which is exactly the shape that produces a circularity.
- **Whether absorbing into the host makes the host too large.** Six internal concerns is already
a lot for a binary whose whole argument is that it has no dependencies. The counter-argument
is that each is small and none can be optional — but this is the skeleton's biggest unproven
claim, and it should be tested by writing the host's interface before anything else.
- **The migration.** Nothing here says how today becomes this.