jschoubben
3249b9a0cc
The predecessor serves the registry under a public name, behind htpasswd basic auth, with a twenty-gigabyte body limit for layer pushes. The mesh's registry has no name, no lock and no limit — by design inside the mesh, where the private network is the boundary and every node pulls without an account (hq ADR 0082). Taking the name over must not change that. A route on `distribution` itself would: contributing a route is requiring one, and the store is raised at genesis on a node with no proxy. So the public door is `distribution-gate`, a second registry process on the same volume, behind the registry's own htpasswd (the predecessor's realm, the predecessor's file, carried in with `secret accept`), with the route and its limit. It requires the store's storage as a node-scoped provision, so it can only land beside the store. The store's own door is untouched — no auth, no htpasswd — which is what keeps the builder's pushes and every node's pulls working. Both processes read the predecessor's configuration where it changed behaviour: delete enabled, which tag retention depends on; no per-process descriptor cache, which two processes over one store cannot share; the CORS headers for the retired interface dropped. route-adapter writes the limit as the predecessor's own buffering middleware, named after the router, only when asked for — and skips a route whose limit it cannot read rather than carrying what the module said not to. hq ADR 0082/0104, the registry hand-over.
182 lines
9.7 KiB
Markdown
182 lines
9.7 KiB
Markdown
# route-adapter — `route`, answered by writing into the predecessor's proxy
|
|
|
|
**This is migration scaffolding.** It exists so that a node being adopted can migrate one web
|
|
module at a time, and it is deleted when that migration ends. novox/hq
|
|
[ADR 0104](https://git.novox.be/novox/hq), which decides it, and
|
|
[issue 093](https://git.novox.be/novox/hq), which found the fault.
|
|
|
|
## What it is
|
|
|
|
Every module reachable by name requires a **route**. The mesh has one provider of it — its own
|
|
proxy, [`route-proxy`](../route-proxy/README.md) — and that proxy binds the two public ports on the
|
|
machine's own network. On a node that is adopted (novox/hq ADR 0100) the predecessor's proxy holds
|
|
those ports and serves every public name there, so the two cannot run at once. That is a circle: no
|
|
web module can be taken until the mesh's proxy runs, the mesh's proxy cannot run until the
|
|
predecessor's stops, and when it stops every name the predecessor served goes dark.
|
|
|
|
This module answers the same provision **without binding anything**. It provides `route`, receives
|
|
exactly the contributions file the proxy receives, and turns each contribution into one route file
|
|
in the directory the predecessor's file provider reads. The predecessor keeps serving every name it
|
|
already serves and keeps renewing its certificates; a name whose module has migrated is served by
|
|
the same proxy, pointing at the mesh's container instead of the predecessor's.
|
|
|
|
| field | value | why |
|
|
|---|---|---|
|
|
| `provides` | `route` (scope `mesh`) | exactly what `route-proxy` provides, so the controller resolves `route` from this unchanged — a module that publishes through it cannot tell which one answered |
|
|
| `serves.route` | `{}` | a route hands back a name, not a credential |
|
|
| `receives.route` | `…/routes/mesh.json` | the same contributions file, in the same shape |
|
|
| `listens` | *nothing* | **the point.** It opens no port, so it stands beside the predecessor rather than replacing it |
|
|
| `requires` | *nothing* | in particular not `acme-ca`: the predecessor holds the certificates and this must not ask for one |
|
|
|
|
Assign **either** this or `route-proxy` to a node, never both — two providers of one mesh-scoped
|
|
provision is an ambiguity the resolver is right to refuse.
|
|
|
|
## What it writes
|
|
|
|
One file per contribution, named `mesh-<name>.yml`, in the predecessor's dynamic directory. For a
|
|
contribution naming `git.example` on port 2999, on this machine:
|
|
|
|
```yaml
|
|
# written by the novox mesh route-adapter (novox/hq ADR 0104)
|
|
# gitea contributed this route. It is removed when that contribution goes.
|
|
http:
|
|
routers:
|
|
mesh-git-example:
|
|
entryPoints: [websecure]
|
|
rule: Host(`git.example`)
|
|
service: mesh-git-example
|
|
tls:
|
|
certResolver: le
|
|
domains:
|
|
- main: git.example
|
|
services:
|
|
mesh-git-example:
|
|
loadBalancer:
|
|
servers:
|
|
- url: http://host.docker.internal:2999
|
|
```
|
|
|
|
A contribution may also carry **`max-request-body`**, the largest request body in bytes the proxy
|
|
may carry to it — the registry's public name needs twenty gigabytes for a layer push (novox/hq
|
|
ADR 0082, the registry hand-over). It is written as the predecessor's own `buffering` middleware,
|
|
named after the router, and only when asked for:
|
|
|
|
```yaml
|
|
routers:
|
|
mesh-registry-api-example:
|
|
# …
|
|
middlewares: [mesh-registry-api-example-body]
|
|
middlewares:
|
|
mesh-registry-api-example-body:
|
|
buffering:
|
|
maxRequestBodyBytes: 21474836480
|
|
```
|
|
|
|
- **The certificate resolver is the predecessor's own**, by its own name. The predecessor already
|
|
holds a certificate for every public name it serves, so naming its resolver means a migrated name
|
|
is served from the certificate that exists. A resolver of the mesh's would ask a public authority
|
|
for one in the same window the module is cut over — the risk ADR 0104 exists to remove.
|
|
- **The port is the contributor's**, straight out of the contribution: the mesh assigned the
|
|
machine-side number (novox/hq ADR 0038) and carries it there. Nothing here guesses it.
|
|
- **A limit it cannot honour is a route it does not write.** A `max-request-body` that is not a
|
|
whole positive number of bytes is skipped and named, like a port that is not one — written
|
|
without the limit, the predecessor would carry exactly what the module said not to carry.
|
|
- **The address is where the mesh says that machine is.** Empty means this one, reached from inside
|
|
the predecessor's container at `host.docker.internal` — not loopback, which from inside that
|
|
container is the container. A contributor on another node carries its overlay address and the
|
|
predecessor is sent straight there.
|
|
|
|
**It never touches a file it did not write.** Two tests, not one: the name must match `mesh-*.yml`,
|
|
*and* the file must start with the marker line above. A file somebody else happened to call
|
|
`mesh-something` is not this module's — it is left alone, and the route that wanted that name is
|
|
reported as unserved rather than written over. The directory belongs to the predecessor; the mesh
|
|
is a guest in it.
|
|
|
|
It removes a file it wrote when that contribution goes — including when nothing is contributed at
|
|
all, which is what unassigning the contributing module has to mean.
|
|
|
|
## How it runs again
|
|
|
|
It is a **step**, not a daemon: one `run-once` container that reads two files, reconciles the
|
|
directory and exits. It names both files under `restart-on`, so the host runs it again the moment
|
|
either changes — the mechanism already in the catalogue for *a fact arrived, do it again*
|
|
(novox/hq ADR 0099).
|
|
|
|
| `restart-on` | what changed |
|
|
|---|---|
|
|
| `received-route` | a module contributing a route arrived or left, or its port moved |
|
|
| `config` | this node's settings for the adapter changed |
|
|
|
|
A watcher of its own would be a second way to notice the same event, and it would keep running
|
|
after the module was unassigned.
|
|
|
|
## The node setting
|
|
|
|
Where the predecessor keeps the directory its file provider reads is a fact about **one machine**,
|
|
so it is a setting laid over the module's default rather than a constant in the catalogue —
|
|
novox/hq ADR 0100's `ports` is the precedent, and cloudflare-dns's `config.json` is the mechanism:
|
|
a `merge: "json"` file the mesh composes from the module's defaults and the node's layer, mounted
|
|
into the container.
|
|
|
|
| key | default | meaning |
|
|
|---|---|---|
|
|
| `dynamic` | `/services/traefik/dynamic` | the directory the predecessor's file provider reads, on this machine |
|
|
| `entrypoint` | `websecure` | the entry point it serves public HTTPS on |
|
|
| `certificate-resolver` | `le` | the predecessor's resolver, by its own name |
|
|
| `machine` | `host.docker.internal` | how the predecessor's proxy reaches this machine |
|
|
|
|
```
|
|
mesh-controller settings set route-adapter settings.json --node <node>
|
|
```
|
|
|
|
**One thing does not follow the setting yet, and it fails loudly rather than quietly.** The
|
|
container has to have that directory bind-mounted, and a bind mount's host side is written in the
|
|
manifest — the control plane can settle a file's content and a container's published ports, but not
|
|
a volume. So the mount is the default path. A node whose predecessor keeps its directory elsewhere
|
|
needs the manifest's mount changed with the setting; until then the module refuses on its first
|
|
pass and says exactly that, rather than writing route files somewhere nothing reads and reporting
|
|
success. The general fix — a per-node path reaching a container's volumes, as `ports` reaches its
|
|
published ports — is a decision for novox/hq, not for a module that is scheduled for deletion.
|
|
|
|
## Using it on an adopted node
|
|
|
|
1. The node is **adopted** and the predecessor's control is stopped (novox/hq ADR 0100). This module
|
|
is refused on a converged node: it writes into a directory the predecessor owns, and that is only
|
|
safe while nothing else is writing there.
|
|
2. `assign <node> route-adapter`, and set `dynamic` for the node if the predecessor keeps its
|
|
directory anywhere but the default.
|
|
3. The predecessor's proxy must be able to reach the machine — `host.docker.internal` resolving to
|
|
the host gateway, as it already does for every service of the predecessor's that runs beside it.
|
|
4. Assign and then **`take <node> <module>`** for each web module, one at a time, when its data has
|
|
moved. On the take, the module's container starts on the machine port the mesh gave it and the
|
|
adapter writes its route file; **remove the predecessor's own route file for that name** in the
|
|
same step, or two routers claim one `Host` rule and which one answers is not something to rely
|
|
on.
|
|
5. At the end, `converge <node>`: the predecessor's proxy retires, `route-proxy` takes the ports and
|
|
already knows every route, because by then every name belongs to a module that contributes it.
|
|
6. **Then delete this module.** It has a stated end; something has to delete it, and that something
|
|
is the operator.
|
|
|
|
## How it ships
|
|
|
|
The tool runtime carrying this module's compiled code, as
|
|
[cloudflare-dns](../cloudflare-dns/Dockerfile) and [mosquitto](../mosquitto/Dockerfile) do — built
|
|
from this directory and nothing else. It connects to no broker: reconciling files on the machine it
|
|
runs on is an offline operation, and `mesh-tools run` gives it exactly that.
|
|
|
|
| env | default | meaning |
|
|
|---|---|---|
|
|
| `MESH_RECEIVES` | `/var/lib/route-adapter/routes/mesh.json` | the contributions file (`receives.route`) |
|
|
| `MESH_ROUTE_ADAPTER_CONFIG` | `/run/config/config.json` | the settled settings file |
|
|
|
|
## Tests
|
|
|
|
```
|
|
cd modules/route-adapter && npm test
|
|
```
|
|
|
|
They hold it to what ADR 0104 says holds it: one file per contribution, a file removed when its
|
|
contribution goes, every file it did not write left alone — and the three facts a route file has to
|
|
get right: the port the contributor publishes, the address of the machine it is on, and the body
|
|
limit it asked for, written as the predecessor's middleware or not at all.
|