jochen
4dde21f0ab
The bundle raised the store as mesh-store, on upstream postgres, in a docker volume; the postgres module keeps it as postgres, on its pgvector image, in /var/lib/mesh-store owned 999:70, reading its superuser password from a file. The store step refused, and genesis stopped at step 17 of 24. The bundle now raises the store with the module's name, image, data directory and password file, and every step of the bundle and the installer reaches it by that name. The refusal to raise a store over data it cannot open looks for a cluster in the store's directory, no longer for the old volume.
265 lines
19 KiB
Plaintext
265 lines
19 KiB
Plaintext
// foundation-first-node.lock — what a machine must be before a mesh exists: the one genesis bundle
|
|
// (novox/hq ADR 0331 §2), on the bus the mesh runs on (ADR 0106, ADR 0131, design 25).
|
|
//
|
|
// The whole bootstrap (novox/hq 03-DESIGN/01-to-be/07-the-foundation.md): a container runtime, a
|
|
// filter, a store, a database per context, those contexts' schemas, the bus with its certificate
|
|
// and its first user list, and the control plane running on top of them.
|
|
//
|
|
// **The mesh writes its own user list, and at genesis there is no mesh yet to write it.** So this
|
|
// carries the first one — the controller's own account — and the control plane's address names the
|
|
// same account. Both say `bootstrap`, and both are placeholders: the installer makes the account's
|
|
// password, puts it in the address and its hash in the list (mesh-host internal/bootstrap
|
|
// rootsecrets.go), and refuses a bundle whose bus connection it does not recognise. Until the first
|
|
// node has enrolled the installer places the list the controller composes; from the bus's adoption
|
|
// as the `nats` module on, the module receives it like any module (ADR 0331 §1, §5).
|
|
//
|
|
// The accounts file is its own file beside the server's configuration, because the server's own
|
|
// settings belong to whoever raises it and the users belong to the mesh (design 25 §4). Both live in
|
|
// one directory, of necessity: an include path is resolved relative to the including file's own
|
|
// directory, so a server given an absolute one looks for it underneath that directory and refuses to
|
|
// start.
|
|
//
|
|
// No `verify` on the TLS block, deliberately — that setting makes the server demand a *client*
|
|
// certificate, and nothing in the mesh presents one: a host pins this server's exact certificate and
|
|
// authenticates with a password (ADR 0004, design 25 §4). The control plane pins it too, through
|
|
// MESH_BROKER_CERTIFICATE, and dials the bus on the machine's own loopback.
|
|
//
|
|
// PINNED BY DIGEST, and the digest is not decoration: a tag can be made to point at a different
|
|
// image, and this file is applied on a machine with no mesh to ask about anything (ADR 0006). The
|
|
// registry prefix is the lab's; the digests are upstream's.
|
|
//
|
|
// **The store is raised where the `postgres` module keeps it** (novox/hq ADR 0338): the container
|
|
// `postgres`, on the module's pgvector image, its data in the directory /var/lib/mesh-store owned by
|
|
// the image's postgres user, its superuser password read from the module's own-secret file — so the
|
|
// store step adopts it as it runs. Pinned upstream by the module's own digest, not through the lab's
|
|
// registry prefix, because the two must be the same reference (mesh-host internal/bootstrap holds
|
|
// them to each other against the module's manifest).
|
|
|
|
{
|
|
"declaration": 1,
|
|
"resources": [
|
|
{
|
|
"id": "container-runtime",
|
|
"type": "package",
|
|
"package": "docker"
|
|
},
|
|
{
|
|
"id": "container-runtime-running",
|
|
"type": "service",
|
|
"unit": "docker.service",
|
|
"state": "running",
|
|
"boot": "enabled"
|
|
},
|
|
// **A filter before anything listens** (novox/hq issue 054, ADR 0088). The store and the
|
|
// broker are adopted as modules later and so bind to every interface from the moment they
|
|
// start; the packet filter that governs who may reach them is a module too, installed a
|
|
// dozen steps later. Between the two, a control-node facing the network had its store and
|
|
// its bus open to anyone who could reach the machine. So the foundation carries a filter of
|
|
// its own — the same table the filter module will replace wholesale once it can derive one:
|
|
// drop by default, keep loopback, replies, ssh and the mesh's own ports (the bus a node
|
|
// enrols over, the registry a node pulls from), and let the container runtime's own
|
|
// networks through the forward chain so containers keep working. A published container port
|
|
// is forwarded, never input (issue 047), which is why the forward chain is where the store's
|
|
// and broker's ports are refused from outside — and a container on this machine dialling a
|
|
// port this machine publishes reaches it through the runtime's proxy, which IS input, which
|
|
// is why the bus and the registry are opened in both chains, exactly as the derived ruleset
|
|
// does.
|
|
{
|
|
"id": "base-filter-package",
|
|
"type": "package",
|
|
"package": "nftables"
|
|
},
|
|
{
|
|
"id": "base-filter",
|
|
"type": "file",
|
|
"path": "/etc/nftables.conf",
|
|
"mode": "0644",
|
|
"content": "#!/usr/sbin/nft -f\n# the foundation's own filter, until the mesh derives one (novox/hq issue 054)\ntable inet mesh {}\ndelete table inet mesh\n\ntable inet mesh {\n\tchain input {\n\t\ttype filter hook input priority filter; policy drop;\n\t\tct state established,related accept\n\t\tct state invalid drop\n\t\tiif lo accept\n\t\ticmp type echo-request accept\n\t\ticmpv6 type { echo-request, nd-neighbor-solicit, nd-neighbor-advert, nd-router-advert } accept\n\t\t# ssh, from anywhere — never closed\n\t\ttcp dport 22 accept\n\t\t# the mesh's own, from anywhere: the bus a node enrols over and a container on this machine reaches through the proxy, the registry a node pulls from\n\t\ttcp dport 5671 accept\n\t\ttcp dport 5000 accept\n\t}\n\tchain output {\n\t\ttype filter hook output priority filter; policy accept;\n\t}\n\tchain forward {\n\t\ttype filter hook forward priority filter; policy drop;\n\t\tct state established,related accept\n\t\tct state invalid drop\n\t\t# the container runtime's bridge networks, and the networks its compose files are given\n\t\tip saddr 172.16.0.0/12 accept\n\t\tip saddr 192.168.128.0/17 accept\n\t\t# the mesh's own: the bus a node enrols over, the registry a node pulls from\n\t\tct original proto-dst 5671 accept\n\t\tct original proto-dst 5000 accept\n\t}\n}\n"
|
|
},
|
|
{
|
|
"id": "base-filter-loaded",
|
|
"type": "service",
|
|
"unit": "nftables.service",
|
|
"state": "running",
|
|
"boot": "enabled",
|
|
"restart-on": ["base-filter"]
|
|
},
|
|
// The store's data is a directory owned by the image's postgres user (999), made before the
|
|
// container starts: a directory the runtime made would be root's, and the database could not
|
|
// write it. Measured on the module's image 2026-10-13: it initialises a new store on a directory
|
|
// made so (novox/hq ADR 0338 §5).
|
|
{
|
|
"id": "store-data",
|
|
"type": "directory",
|
|
"path": "/var/lib/mesh-store",
|
|
"mode": "0700",
|
|
"owner": "999:70"
|
|
},
|
|
{
|
|
"id": "store",
|
|
"type": "container",
|
|
"name": "postgres",
|
|
"image": "pgvector/pgvector@sha256:cf134a767f474095eeba57e0117be8e568e011a63f33fbf252f14c9b760f8e6f",
|
|
"env": {
|
|
"POSTGRES_PASSWORD": "bootstrap",
|
|
"PGDATA": "/var/lib/postgresql/data/pgdata"
|
|
},
|
|
"ports": ["5432:5432"],
|
|
"volumes": ["/var/lib/mesh-store:/var/lib/postgresql/data"]
|
|
},
|
|
// Over TCP, not the socket. While the store initialises it runs a temporary server on the
|
|
// socket ONLY, then stops it and starts the real one — so a socket check passes, the action
|
|
// exits happy, and the verify a moment later lands in the gap and fails. The action and its
|
|
// verify must ask the same question, or the action can succeed into a state verify rejects.
|
|
{
|
|
"id": "store-ready",
|
|
"type": "action",
|
|
"in": "postgres",
|
|
"command": ["sh", "-c", "for i in $(seq 1 180); do pg_isready -h 127.0.0.1 -U postgres >/dev/null 2>&1 && exit 0; sleep 1; done; echo 'the store did not answer within 180s; its own last words follow'; pg_isready -h 127.0.0.1 -U postgres; tail -n 20 /var/lib/postgresql/data/log/*.log 2>/dev/null; exit 1"],
|
|
"verify": ["pg_isready", "-h", "127.0.0.1", "-U", "postgres"]
|
|
},
|
|
{
|
|
"id": "inventory-database",
|
|
"type": "action",
|
|
"in": "postgres",
|
|
"command": ["sh", "-c", "psql -U postgres -c 'CREATE DATABASE inventory'"],
|
|
"verify": ["sh", "-c", "psql -U postgres -lqt | cut -d'|' -f1 | grep -qw inventory"]
|
|
},
|
|
{
|
|
"id": "identity-database",
|
|
"type": "action",
|
|
"in": "postgres",
|
|
"command": ["sh", "-c", "psql -U postgres -c 'CREATE DATABASE identity'"],
|
|
"verify": ["sh", "-c", "psql -U postgres -lqt | cut -d'|' -f1 | grep -qw identity"]
|
|
},
|
|
// Each context owns its own database (novox/hq ADR 0008). A third one is a third database,
|
|
// created the same way and named the same way — which is the whole of adding a context to the
|
|
// bootstrap, and is why the count is not something the foundation has an opinion about.
|
|
{
|
|
"id": "licences-database",
|
|
"type": "action",
|
|
"in": "postgres",
|
|
"command": ["sh", "-c", "psql -U postgres -c 'CREATE DATABASE licences'"],
|
|
"verify": ["sh", "-c", "psql -U postgres -lqt | cut -d'|' -f1 | grep -qw licences"]
|
|
},
|
|
{
|
|
"id": "context-schemas",
|
|
"type": "action",
|
|
"command": ["docker", "run", "--rm", "--network", "container:postgres",
|
|
"-e", "MESH_STORE_INVENTORY=postgres://postgres:bootstrap@127.0.0.1:5432/inventory?sslmode=disable",
|
|
"-e", "MESH_STORE_IDENTITY=postgres://postgres:bootstrap@127.0.0.1:5432/identity?sslmode=disable",
|
|
"-e", "MESH_STORE_LICENCES=postgres://postgres:bootstrap@127.0.0.1:5432/licences?sslmode=disable",
|
|
"192.0.2.250:5000/mesh-controller@sha256:c67db38439ff0aee242b467486765467bb95801f52175fc5727cc4e437338ace",
|
|
"migrate"],
|
|
"verify": ["sh", "-c", "docker exec postgres psql -U postgres -d inventory -tAc \"select to_regclass('public.node')\" | grep -qx node && docker exec postgres psql -U postgres -d identity -tAc \"select to_regclass('public.signing_key')\" | grep -qx signing_key && docker exec postgres psql -U postgres -d licences -tAc \"select to_regclass('public.licence')\" | grep -qx licence"]
|
|
},
|
|
// **The bus is raised where the `nats` module keeps it** (novox/hq ADR 0333 §1): the container
|
|
// `mesh-broker-nats`, its configuration and user list in /var/lib/nats-module/conf, its streams in
|
|
// /var/lib/mesh-broker-nats and its certificate in /var/lib/mesh-broker-tls — the names and paths the
|
|
// catalogue's module declares. So adopting it as that module at the broker step keeps it: the
|
|
// module's built image takes the place of the upstream one below, once, on the same streams, users
|
|
// and certificate, and nothing else moves. mesh-host's internal/bootstrap holds the two to each other
|
|
// against the module's own manifest (busLayout), at the broker step and in its tests.
|
|
//
|
|
// Directories on the machine, not volumes the runtime names: the module mounts these paths, and the
|
|
// control plane, once it runs as the mesh's process, reads the certificate at its path on the machine.
|
|
// The certificate's directory is readable by everyone because the certificate is public and the control
|
|
// plane runs as nobody; its key is written 0600 by the program that makes it.
|
|
{
|
|
"id": "bus-tls-dir",
|
|
"type": "directory",
|
|
"path": "/var/lib/mesh-broker-tls",
|
|
"mode": "0755"
|
|
},
|
|
{
|
|
"id": "bus-data-dir",
|
|
"type": "directory",
|
|
"path": "/var/lib/mesh-broker-nats",
|
|
"mode": "0700"
|
|
},
|
|
{
|
|
"id": "bus-certificate",
|
|
"type": "action",
|
|
// **The mesh makes its own** (novox/hq 04-ISSUES/146). This ran `openssl` inside the
|
|
// broker's image while the broker was one that carried it; the bus that replaced it has a
|
|
// shell and no openssl, and no other image the bundle names has one either. So the program
|
|
// that needs the certificate writes it — already on this machine, since the schema step ran
|
|
// it, and asking nothing of the image it writes into. Self-signed on purpose: a host pins
|
|
// this server's exact certificate (novox/hq ADR 0004), and at this moment there is no mesh
|
|
// to ask an authority of.
|
|
// `--user 0:0` because the directory is root's and this image runs as nobody, which is right
|
|
// for the long-running control plane and wrong for a one-shot writing into a fresh directory.
|
|
"command": ["docker", "run", "--rm", "--user", "0:0", "-v", "/var/lib/mesh-broker-tls:/tls",
|
|
"192.0.2.250:5000/mesh-controller@sha256:c67db38439ff0aee242b467486765467bb95801f52175fc5727cc4e437338ace",
|
|
"broker", "certificate", "--into", "/tls"],
|
|
"verify": ["docker", "run", "--rm", "--user", "0:0", "-v", "/var/lib/mesh-broker-tls:/tls",
|
|
"192.0.2.250:5000/mesh-controller@sha256:c67db38439ff0aee242b467486765467bb95801f52175fc5727cc4e437338ace",
|
|
"broker", "certificate", "--check", "--into", "/tls"]
|
|
},
|
|
// **Seeds, written only where nothing is** (create-once). The bus's configuration and user list are
|
|
// the bundle's until the bus is adopted, and the module's from then on (ADR 0333 §4): an installer
|
|
// run again after the adoption applies this bundle again, and an ordinary file would put the
|
|
// bundle's one user back over every user the mesh composed, and leave the node unable to reach its
|
|
// bus. Until the adoption the installer places the composed list over the seed (enrol.go).
|
|
{
|
|
"id": "bus-conf-dir",
|
|
"type": "directory",
|
|
"path": "/var/lib/nats-module/conf",
|
|
"mode": "0700"
|
|
},
|
|
{
|
|
"id": "bus-conf",
|
|
"type": "file",
|
|
"path": "/var/lib/nats-module/conf/nats.conf",
|
|
"mode": "0644",
|
|
"create-once": true,
|
|
"content": "port: 4222\nhttp: 0.0.0.0:8222\n\ntls {\n cert_file: \"/tls/tls.crt\"\n key_file: \"/tls/tls.key\"\n}\n\njetstream {\n store_dir: \"/data\"\n}\n\ninclude accounts.conf\n"
|
|
},
|
|
{
|
|
"id": "bus-accounts",
|
|
"type": "file",
|
|
"path": "/var/lib/nats-module/conf/accounts.conf",
|
|
"mode": "0600",
|
|
"create-once": true,
|
|
"content": "// The first user list, carried by the installer because at genesis there is no mesh to\n// compose one. A bootstrap credential, rotated with the store's and replaced by the\n// controller's own composition from its first start onward.\naccounts {\n MESH {\n jetstream: enabled\n users = [\n { user: \"controller\", password: \"$2a$10$AHqJgOifIVbU41KmATiMhuXFs8xa7Wl2HuN4UVBCXdN2jIQzjqApy\", permissions: {\n publish: { allow: [\"$JS.ACK.CONTROL.controller.>\", \"$JS.ACK.EVENTS.controller.>\", \"$JS.ACK.DEAD_LETTER_NOTICES.controller.>\", \"$JS.API.>\", \"$KV.mesh-controller_asked.>\", \"$KV.mesh-controller_calls.>\", \"$KV.mesh-controller_hand-acts.>\", \"$KV.mesh-controller_conditions.>\", \"$KV.mesh-controller_condition-history.>\", \"$KV.mesh-controller_lease.>\", \"$KV.SEAT_MESH_BUILD_MACHINE_cancelled.>\", \"$KV.SEAT_NODE_BUILD_AGENT_cancelled.>\", \"_INBOX.enrol.>\", \"mesh.assignment.>\", \"mesh.mod.*.tool.>\", \"mesh.node.>\", \"mesh.seat.mesh-build-machine.accept.>\", \"mesh.seat.node-build-agent.accept.>\", \"mesh.seat.mesh-build-machine.tool.>\", \"mesh.seat.node-build-agent.tool.>\", \"mesh.seat.mesh-controller.event.applied\", \"mesh.seat.mesh-controller.event.built-before\", \"mesh.seat.mesh-controller.event.refused\", \"mesh.seat.mesh-controller.event.rolled-back\", \"mesh.seat.mesh-controller.event.checked\", \"mesh.seat.mesh-controller.event.plan-moved\", \"mesh.seat.mesh-controller.event.condition-raised\", \"mesh.seat.mesh-controller.event.condition-changed\", \"mesh.seat.mesh-controller.event.condition-cleared\", \"mesh.seat.mesh-controller.event.doctor-heartbeat\", \"mesh.seat.mesh-controller.event.secret-replaced\", \"mesh.seat.mesh-controller.event.healer-acted\", \"$SRV.INFO\", \"mesh.seat.node-intrusion-prevention.tool.banned.*\", \"mesh.seat.node-backup.tool.backed-up.*\", \"mesh.seat.node-backup.tool.now.*\", \"mesh.seat.mesh-delivery.tool.stalled\", \"mesh.seat.mesh-delivery.tool.close\", \"mesh.seat.node-launcher.tool.secret.*\", \"mesh.seat.mesh-delivery.tool.release\", \"mesh.seat.mesh-delivery.tool.stop\", \"mesh.seat.mesh-delivery.tool.times\", \"mesh.seat.mesh-delivery.tool.groups\", \"mesh.seat.node-service-manager.tool.restart.*\", \"mesh.seat.mesh-controller.tool.plans\", \"mesh.events.dead.>\", \"mesh.again.>\"] }\n subscribe: { allow: [\"$JS.API.>\", \"$JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES.>\", \"$JS.EVENT.ADVISORY.CONSUMER.DELETED.>\", \"_DELIVER.controller\", \"_DELIVER.controller.>\", \"_INBOX.controller.>\", \"mesh.control.>\", \"mesh.mod.*.event.provisioner.failing\", \"mesh.mod.*.event.provisioner.recovered\", \"mesh.mod.*.event.provisioner.retirement\", \"mesh.mod.gitea.event.pull.merged\", \"mesh.mod.gitea.event.pull.updated\", \"mesh.mod.mesh-catalog.event.catching-up\", \"mesh.mod.mesh-catalog.event.upgraded\", \"mesh.seat.mesh-build-machine.event.built\", \"mesh.seat.node-build-agent.event.built\", \"mesh.seat.operator-channel.event.decided.mesh-controller\", \"mesh.seat.mesh-controller.tool.>\", \"$SRV.PING\", \"$SRV.INFO\", \"$SRV.PING.mesh-controller\", \"$SRV.PING.mesh-controller.>\", \"$SRV.INFO.mesh-controller\", \"$SRV.INFO.mesh-controller.>\", \"$SRV.STATS\", \"$SRV.STATS.mesh-controller\", \"$SRV.STATS.mesh-controller.>\"] }\n allow_responses: { max: 1, ttl: \"1m\" }\n } }\n ]\n }\n}\n"
|
|
},
|
|
// The upstream server, until the broker step adopts it: the `nats` module's own image, built by the
|
|
// mesh, takes its place there once, on these same directories (ADR 0333 §3). The published port is the
|
|
// node's bus port, 5671 unless given, which every enrolment token names; the installer gives the
|
|
// module the setting that maps its declared 4222 to it (ADR 0333 §2).
|
|
{
|
|
"id": "broker",
|
|
"type": "container",
|
|
"name": "mesh-broker-nats",
|
|
"image": "192.0.2.250:5000/nats@sha256:b83efabe3e7def1e0a4a31ec6e078999bb17c80363f881df35edc70fcb6bb927",
|
|
"ports": ["5671:4222", "127.0.0.1:8222:8222"],
|
|
"volumes": ["/var/lib/mesh-broker-nats:/data", "/var/lib/nats-module/conf:/etc/nats:ro", "/var/lib/mesh-broker-tls:/tls:ro"],
|
|
"args": ["-c", "/etc/nats/nats.conf", "-js"]
|
|
},
|
|
{
|
|
"id": "broker-ready",
|
|
"type": "action",
|
|
"command": ["sh", "-c", "for i in $(seq 1 60); do docker run --rm --network host --entrypoint sh 192.0.2.250:5000/nats@sha256:b83efabe3e7def1e0a4a31ec6e078999bb17c80363f881df35edc70fcb6bb927 -c 'nc -z 127.0.0.1 5671' >/dev/null 2>&1 && exit 0; sleep 1; done; exit 1"],
|
|
"verify": ["sh", "-c", "docker run --rm --network host --entrypoint sh 192.0.2.250:5000/nats@sha256:b83efabe3e7def1e0a4a31ec6e078999bb17c80363f881df35edc70fcb6bb927 -c 'nc -z 127.0.0.1 5671'"]
|
|
},
|
|
{
|
|
"id": "control-plane",
|
|
"type": "container",
|
|
"name": "mesh-controller",
|
|
"image": "192.0.2.250:5000/mesh-controller@sha256:c67db38439ff0aee242b467486765467bb95801f52175fc5727cc4e437338ace",
|
|
"network": "host",
|
|
"args": ["serve"],
|
|
"volumes": ["/var/lib/mesh-broker-tls:/broker-tls:ro"],
|
|
"env": {
|
|
"MESH_STORE_INVENTORY": "postgres://postgres:bootstrap@127.0.0.1:5432/inventory?sslmode=disable",
|
|
"MESH_STORE_IDENTITY": "postgres://postgres:bootstrap@127.0.0.1:5432/identity?sslmode=disable",
|
|
"MESH_STORE_LICENCES": "postgres://postgres:bootstrap@127.0.0.1:5432/licences?sslmode=disable",
|
|
"MESH_BUS_NATS": "nats://controller:bootstrap@127.0.0.1:5671",
|
|
"MESH_BROKER_ADDRESS": "192.0.2.10:5671",
|
|
"MESH_BROKER_CERTIFICATE": "/broker-tls/tls.crt"
|
|
}
|
|
}
|
|
|
|
]
|
|
}
|