novox/hq
3
0
Fork 0
Code Issues 5 Pull Requests Releases Wiki Activity

HQ — the mesh's own documentation

Browse Source 
What the mesh is, what it is becoming, and why. Implementation lives in the
code repositories; the reasoning lives here.

  00-GENESIS   mission, engineering context, effect, and the rules that hold
  01-RESEARCH  investigations, before they harden into design
  02-DESIGN    the authoritative specification
  adr          numbered decisions — what was chosen, and what was rejected
  DECISIONS.md the ledger: every decision, in the order it was taken

Written for a reader who is not its author and has no access to the mesh it
describes. Addresses use the documentation ranges of RFC 5737 and RFC 1918;
nodes are named by role.

Single initial commit by intent. The prior history came from a private
repository and carried operational detail — a routable address identified as a
VPN hub, real domain names, a hosting provider — which sanitising a tip commit
would not have removed from the log.
This commit is contained in:
jschoubben
2026-08-22 22:01:32 +02:00
commit cf9357e8e9
22 changed files with 2376 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
01-RESEARCH/003-service-supervision/analysis.md
+225
View File
@@ -0,0 +1,225 @@
# Who supervises a service — the cost of leaving systemd
Measured 2026-08-22. Every claim is a file location or a count.
---
## 1. What systemd actually does for HAL
**14 modules** ship a `systemd/` directory. The units divide cleanly into three kinds:
| kind | count | what it is |
|---|---|---|
| long-running daemons | 9 | `hal/brain`, `hal/meshware`, `hal/coordinator`, `hal/cortex`, `hal/env-sync`, `hal/file-share`, `hal/thoughts`, `noxflow/runtime`, `desktop_notifications` — all `Restart=always`, `RestartSec=10` |
| scheduled one-shots | 6 timers | docker-prune, docker-registry maintenance, claude-code sessions + usage, health, mailu cert-sync |
| the template | 1 | `hal-module@.service` — the per-module Docker lifecycle |
Only `mailu` is system-scope. Everything else is a user unit under
`~/.config/systemd/user`, auto-detected from directory presence — the module author names
the file and that filename *is* the unit name
(`modules/hal/sdk/src/feature-handlers/systemd.ts:31-42`).
The template is the piece `002` tripped over
(`modules/hal/meshware/systemd/hal-module@.service`):
```
ExecStart=docker compose --project-directory /services/%i --project-name %i up --remove-orphans --pull always
ExecStop=docker compose --project-directory /services/%i --project-name %i down
Restart=on-failure
StartLimitIntervalSec=120
StartLimitBurst=5
```
Systemd's roles, then: restart-on-failure, start at login, env-file loading, ordering,
per-module lifecycle, timers, and — via journald — **the only log store the 9 Node daemons
have** (`modules/hal/sdk/src/tools/log-tail.ts:50-53`).
---
## 2. The question splits in two, and the halves disagree
### For `docker compose` stacks, systemd is mostly redundant
**44 of 44** module `docker-compose.yml` files declare a container-level `restart:` policy —
`always` or `unless-stopped`, none missing. Docker's own daemon already restarts crashed
containers, independently of systemd.
`hal-module@.service` does not supervise the containers. It supervises the **`docker compose
up` foreground process**. It is a second layer on top of a restart policy that already
works. What it genuinely adds is narrower than it looks:
- one uniform verb for every module type — `systemctl --user start hal-module@X` rather than
remembering each module's compose invocation, relied on across a dozen call sites in
`meshware.ts:124-154`, `dev-env.ts:142-163`, `installer-core.ts`, `infra.ts:166`
- recovery when the `docker compose up --pull always` process itself dies — a failed image
pull, not a crashed container
- rate-limited restart (`StartLimitBurst=5`) so a broken stack does not spin
That is real, but it is a convenience layer, not a safety layer. **This half could go at
moderate cost.**
### For the 9 Node daemons, systemd is load-bearing
There is no alternative supervisor anywhere in the repo. No PM2, no forever, no nodemon, no
watchdog loop — all checked, zero hits. `Restart=always` is the only thing standing between
a crashed daemon and a dead node.
**This half is the actual question.**
---
## 3. The hard part is fate-sharing
The difficulty is not systemd. It is that **a supervisor must not share fate with what it
supervises**, and the codebase already has a scar from exactly this.
meshware cannot restart itself mid-request: killing the process before it ACKs the AMQP
message loses the message. So it defers its own restart by two seconds after closing the
connection (`modules/hal/meshware/daemon/src/cerebellum.ts:815-828`) — a commented
workaround for a problem that only exists because the thing being restarted is the thing
doing the restarting.
Any mesh-native supervisor inherits this recursively. Something has to be the outermost
always-alive process, and if it is written by the mesh, the mesh must supervise it, and so
on. The recursion only terminates at a process the mesh does not own. Today that is
systemd. **A "more mesh" supervisor that is itself a mesh process is not a smaller problem;
it is the same problem with a new name.**
This is the strongest argument for the status quo, and it is worth stating plainly before
looking at alternatives.
---
## 4. The option neither of us named
There is a third answer that terminates the recursion in something that is not systemd and
not written by us: **run HAL's own daemons as containers.**
Docker is already the outermost supervisor for 44 of 44 module stacks. It does not share
fate with the mesh. It already has restart policies, backoff, and a log store that
`log_tail` already speaks (`log-tail.ts:43-47` reads `docker logs` for Docker modules
today). Extending it from "the things the mesh runs" to "the mesh itself" is not new
machinery — it is applying machinery the mesh already trusts to one more case.
What that buys, beyond supervision:
- **Phase 0 stops being a translation.** A containerised node becomes the same shape as a
production node, rather than a local approximation with a systemd-shaped hole in it. The
divergence that `002` open question 3 worries about largely disappears.
- **`/services/` and `~/.config/hal/` stop being special.** Mounts, not host paths.
- **The GENESIS "dogfood everything" value gets easier**, not harder: the mesh's own
components would ship and run exactly like everything else it carries.
What it costs, honestly:
- **journald → docker logs** for the 9 daemons. `log_tail` already handles both, but
`systemd_journal` and the health checks that read unit state
(`modules/hal/mesh/health.sh:34-40`, `modules/hal/health/hal-health.sh:335-395`) would
need a container-aware path.
- **Boot start** becomes Docker's `restart: always` plus the Docker daemon being enabled at
boot — which is still one systemd unit, but the OS's own, not ours.
- **A container needs the host to be reachable** for anything that touches the node itself.
Some of these daemons exist precisely to write host files.
---
## 5. Why it cannot be all-or-nothing — and ADR 0001 already says so
Some of what runs under systemd today **cannot** be containerised, and the reason is
already in the domain model. ADR 0001:
> a non-human agent acts through a spawned session — a human agent acts through a shell or
> desktop
`hal/brain` has two modes (`modules/hal/brain/daemon/src/brainstem.ts:7-13`): a daemon mode
that is an AMQP relay, and a **cortex mode that is an MCP server over stdio for an
interactive Claude Code session**. The second is a human agent's modality. It runs in the
human's shell, on the human's node, against the human's `~/.claude`. Containerising it is
not a hard engineering problem, it is a category error.
The same holds for `desktop_notifications` and everything `hal/desktop-environment` touches.
So the line is not "systemd or not". It is:
| | belongs where |
|---|---|
| mesh daemons — meshware, coordinator, env-sync, thoughts, file-share, brain **in daemon mode** | supervisable by Docker; candidates to containerise |
| human-modality surfaces — brain **in cortex mode**, desktop notifications, desktop environment | on the host, by definition |
| module stacks | already Docker; systemd layer is the redundant part |
This split is not a compromise between the options. It is what the domain model implies,
and it is a decent sign that the model is doing work.
---
## 6. Options, with costs
| | option | cost | what it buys |
|---|---|---|---|
| **A** | **Keep systemd; systemd-in-container for Phase 0** | privileged containers, heavy images, slow iteration; local mesh keeps a shape production does not have | nothing changes in production; smallest change to the mesh |
| **B** | **Drop the `hal-module@` layer only** — let Docker's restart policies supervise stacks directly | reimplement uniform start/stop across ~12 call sites; lose rate-limited restart and pull-failure recovery | removes the redundant layer; does **not** solve Phase 0 on its own, since the daemons still need supervising |
| **C** | **Containerise the mesh daemons; Docker supervises** | container-aware `systemd_journal`/health; host access for daemons that write host files; the human-modality surfaces stay on the host regardless | terminates the fate-sharing recursion without writing a supervisor; makes local and production the same shape; Phase 0 becomes much less of a special case |
| **D** | **Write a mesh-native supervisor** | the fate-sharing recursion (§3), plus matching systemd's maturity — backoff, resource limits, clean SIGTERM (`noxflow/runtime` already depends on `TimeoutStopSec=60`) — on machines that are somebody's daily driver | most "mesh"; least justified by the evidence |
**D is the option the phrasing "more hal mesh approach" points at, and the evidence argues
against it.** Supervision is not a domain concern the mesh is better placed to solve than
the OS; the mesh's distinguishing feature is brokering capabilities, not restarting
processes. C gets the benefit D is reaching for — the mesh not depending on host-specific
init — without the recursion.
**C and B compose.** C is the one that pays for Phase 0.
Worth noting what is *not* in this table: the dozens of `systemctl` call sites that
configure the **host OS's own** units — NetworkManager, resolved, oomd, zram, docker.service,
fail2ban, sshd, zfs, across `modules/asusd`, `g14-power`, `wireguard`, `sshd`, `zfs` and
others. Those are not HAL supervising itself; they are HAL configuring an Arch box. They
are out of scope for every option above and do not go away under any of them.
---
## 7. An incidental finding
Documentation describes an automatic node rescue: `hal-rescue.sh:20` states it is "triggered
automatically by `hal-health.timer` when hal-meshware is failed".
**It is not.** `hal-health.sh` contains no call to `hal-rescue.sh` (checked, zero matches),
and **no unit in the repository declares `OnFailure=`** (checked, zero matches). The only
real triggers are the manual `rescue_node` tool
(`modules/hal/sdk/src/tools/deploy-rescue.ts:77-92`) and running `install.d/rescue.sh` by
hand.
This is worth recording for two reasons. It weakens any argument that systemd-adjacent
self-healing is already wired — it is not. And it is another instance of the pattern this
whole refactor is about: **a documented mechanism that does not exist, believed because it
was written down.** `00-GENESIS/how-we-build.md` calls this out as a rule; here it is again,
found by grep.
---
## 8. Open question
**Which supervision model does the mesh adopt?** A, B, C, D or a combination.
**This no longer gates Phase 0.** When this was written, the local mesh was assumed to be
built from application containers, which forced the question — there is no natural way to
run an init system inside one. The decision of 2026-08-22 to build development nodes as
**system containers** (see [`02-DESIGN/01-end-to-end-testing.md`](../../02-DESIGN/01-end-to-end-testing.md))
removes that pressure entirely: a system container runs a real init, so the existing model
works unmodified and the lab needs no answer here to exist.
What remains is the question on its own merits, which is worth keeping open because the
evidence above still holds: option B removes a layer that 44 of 44 module stacks have already
made redundant, and option C would let the mesh stop depending on host-specific init. Neither
is urgent. Both are now cheap to *try*, because there is somewhere to try them.
Option D — a mesh-written supervisor — remains the one the evidence argues against, for the
fate-sharing reason in §3.
## References
- [`002-local-mesh`](../002-local-mesh/analysis.md) — the effort this came out of
- [`adr/0001`](../../adr/0001-mesh-brokers-nodes-host-agents-think.md) — agent modality, which
decides what cannot leave the host
- `modules/hal/meshware/daemon/src/cerebellum.ts:815-828` — the self-restart workaround
- `modules/hal/meshware/systemd/hal-module@.service` — the per-module Docker lifecycle
- `modules/hal/sdk/src/feature-handlers/systemd.ts` — detection, install, start
01-RESEARCH/003-service-supervision/status.md
+47
View File
@@ -0,0 +1,47 @@
# 003 — Who supervises a service
- **Status:** ONGOING — evidence gathered, options costed, decision open.
**No longer blocks Phase 0** (see below).
- **Initiated by:** jochen, 2026-08-22, in response to
[`002-local-mesh`](../002-local-mesh/analysis.md) open question 1
- **Areas touched:** every module shipping a `systemd/` directory (14), the
`hal-module@` template, `hal/sdk` feature handlers, `dev_up`, `log_tail` /
`systemd_journal`, the bootstrap scripts.
## The question
`002` asked how a containerised node runs a module service, given that a module service is
defined today as a systemd unit running `docker compose` against `/services/`. The response
was the better question:
> If it's possible to run systemd inside a container, that's the way to go I think. However,
> what would the cost be to step away from systemd to run our services and set it up in a
> different way? More hal mesh approach.
This effort answers the cost half. It does not choose.
## Summary of findings
- **The question splits in two**, and the halves have opposite answers. Supervising
`docker compose` stacks through systemd is largely **redundant** — 44 of 44 module compose
files already declare a restart policy, so Docker is already the supervisor. Supervising
HAL's **9 long-running Node daemons** is not redundant: `Restart=always` is currently the
only thing between a crash and a dead node.
- **The hard part is fate-sharing, not systemd.** meshware already cannot restart itself and
carries a documented workaround for it. Any mesh-native supervisor inherits that problem
recursively unless it sits outside the mesh's own process tree — at which point it is an
OS-level supervisor again, just reinvented.
- **There is a third option neither of us named**, and it is the one that also solves Phase 0:
run HAL's own daemons as containers, making Docker the supervisor for everything. Local and
production then have the same shape rather than a translation layer between them.
- **It cannot be all-or-nothing**, and ADR 0001 already says why: a human agent acts through a
shell and a desktop. Those parts are on the host by definition.
- One incidental finding: the automatic node rescue that documentation describes **does not
exist**. No unit declares `OnFailure=`, and nothing calls `hal-rescue.sh` on a timer.
Detail and costs in [`analysis.md`](analysis.md).
## Decision needed
Which supervision model the mesh adopts, recorded in ADR 0002 before Phase 0 builds
anything. The options and their costs are in `analysis.md` under "Options".