jochen
bc2a448512
The bundle raised the bus as mesh-broker on docker volumes; the catalogue's nats module keeps it as mesh-broker-nats on directories of the machine, so the broker step refused rather than raise a second bus, and genesis stopped at step 18 of 24. The bundle now raises mesh-broker-nats with its configuration and user list in /var/lib/nats-module/conf, its streams in /var/lib/mesh-broker-nats and its certificate in /var/lib/mesh-broker-tls, published on the node's bus port. The broker step holds that layout to the module's manifest field by field before registering anything, says the one restart onto the module's image, and waits until the bus runs that image and its monitor answers. The installer's bridge writes the composed users at the module's bus-users path and signals mesh-broker-nats. The bundle's bus configuration and user list are seeds, so a re-run after the adoption does not put the bundle's one user back over the module's.
253 lines
18 KiB
Plaintext
253 lines
18 KiB
Plaintext
// foundation-first-node.lock — what a machine must be before a mesh exists: the one genesis bundle
|
|
// (novox/hq ADR 0331 §2), on the bus the mesh runs on (ADR 0106, ADR 0131, design 25).
|
|
//
|
|
// The whole bootstrap (novox/hq 03-DESIGN/01-to-be/07-the-foundation.md): a container runtime, a
|
|
// filter, a store, a database per context, those contexts' schemas, the bus with its certificate
|
|
// and its first user list, and the control plane running on top of them.
|
|
//
|
|
// **The mesh writes its own user list, and at genesis there is no mesh yet to write it.** So this
|
|
// carries the first one — the controller's own account — and the control plane's address names the
|
|
// same account. Both say `bootstrap`, and both are placeholders: the installer makes the account's
|
|
// password, puts it in the address and its hash in the list (mesh-host internal/bootstrap
|
|
// rootsecrets.go), and refuses a bundle whose bus connection it does not recognise. Until the first
|
|
// node has enrolled the installer places the list the controller composes; from the bus's adoption
|
|
// as the `nats` module on, the module receives it like any module (ADR 0331 §1, §5).
|
|
//
|
|
// The accounts file is its own file beside the server's configuration, because the server's own
|
|
// settings belong to whoever raises it and the users belong to the mesh (design 25 §4). Both live in
|
|
// one directory, of necessity: an include path is resolved relative to the including file's own
|
|
// directory, so a server given an absolute one looks for it underneath that directory and refuses to
|
|
// start.
|
|
//
|
|
// No `verify` on the TLS block, deliberately — that setting makes the server demand a *client*
|
|
// certificate, and nothing in the mesh presents one: a host pins this server's exact certificate and
|
|
// authenticates with a password (ADR 0004, design 25 §4). The control plane pins it too, through
|
|
// MESH_BROKER_CERTIFICATE, and dials the bus on the machine's own loopback.
|
|
//
|
|
// PINNED BY DIGEST, and the digest is not decoration: a tag can be made to point at a different
|
|
// image, and this file is applied on a machine with no mesh to ask about anything (ADR 0006). The
|
|
// registry prefix is the lab's; the digests are upstream's.
|
|
//
|
|
// The store's data is a NAMED VOLUME, not a directory on the machine. A directory the host
|
|
// creates is owned by root, and the database runs as somebody else inside the container — so it
|
|
// could not write, and the container crash-looped. A named volume lets the image set up its own
|
|
// ownership, and outlives the container, which is what you want for the thing holding the mesh's
|
|
// state.
|
|
|
|
{
|
|
"declaration": 1,
|
|
"resources": [
|
|
{
|
|
"id": "container-runtime",
|
|
"type": "package",
|
|
"package": "docker"
|
|
},
|
|
{
|
|
"id": "container-runtime-running",
|
|
"type": "service",
|
|
"unit": "docker.service",
|
|
"state": "running",
|
|
"boot": "enabled"
|
|
},
|
|
// **A filter before anything listens** (novox/hq issue 054, ADR 0088). The store and the
|
|
// broker are adopted as modules later and so bind to every interface from the moment they
|
|
// start; the packet filter that governs who may reach them is a module too, installed a
|
|
// dozen steps later. Between the two, a control-node facing the network had its store and
|
|
// its bus open to anyone who could reach the machine. So the foundation carries a filter of
|
|
// its own — the same table the filter module will replace wholesale once it can derive one:
|
|
// drop by default, keep loopback, replies, ssh and the mesh's own ports (the bus a node
|
|
// enrols over, the registry a node pulls from), and let the container runtime's own
|
|
// networks through the forward chain so containers keep working. A published container port
|
|
// is forwarded, never input (issue 047), which is why the forward chain is where the store's
|
|
// and broker's ports are refused from outside — and a container on this machine dialling a
|
|
// port this machine publishes reaches it through the runtime's proxy, which IS input, which
|
|
// is why the bus and the registry are opened in both chains, exactly as the derived ruleset
|
|
// does.
|
|
{
|
|
"id": "base-filter-package",
|
|
"type": "package",
|
|
"package": "nftables"
|
|
},
|
|
{
|
|
"id": "base-filter",
|
|
"type": "file",
|
|
"path": "/etc/nftables.conf",
|
|
"mode": "0644",
|
|
"content": "#!/usr/sbin/nft -f\n# the foundation's own filter, until the mesh derives one (novox/hq issue 054)\ntable inet mesh {}\ndelete table inet mesh\n\ntable inet mesh {\n\tchain input {\n\t\ttype filter hook input priority filter; policy drop;\n\t\tct state established,related accept\n\t\tct state invalid drop\n\t\tiif lo accept\n\t\ticmp type echo-request accept\n\t\ticmpv6 type { echo-request, nd-neighbor-solicit, nd-neighbor-advert, nd-router-advert } accept\n\t\t# ssh, from anywhere — never closed\n\t\ttcp dport 22 accept\n\t\t# the mesh's own, from anywhere: the bus a node enrols over and a container on this machine reaches through the proxy, the registry a node pulls from\n\t\ttcp dport 5671 accept\n\t\ttcp dport 5000 accept\n\t}\n\tchain output {\n\t\ttype filter hook output priority filter; policy accept;\n\t}\n\tchain forward {\n\t\ttype filter hook forward priority filter; policy drop;\n\t\tct state established,related accept\n\t\tct state invalid drop\n\t\t# the container runtime's bridge networks, and the networks its compose files are given\n\t\tip saddr 172.16.0.0/12 accept\n\t\tip saddr 192.168.128.0/17 accept\n\t\t# the mesh's own: the bus a node enrols over, the registry a node pulls from\n\t\tct original proto-dst 5671 accept\n\t\tct original proto-dst 5000 accept\n\t}\n}\n"
|
|
},
|
|
{
|
|
"id": "base-filter-loaded",
|
|
"type": "service",
|
|
"unit": "nftables.service",
|
|
"state": "running",
|
|
"boot": "enabled",
|
|
"restart-on": ["base-filter"]
|
|
},
|
|
{
|
|
"id": "store",
|
|
"type": "container",
|
|
"name": "mesh-store",
|
|
"image": "192.0.2.250:5000/postgres@sha256:7abf537131b66ed5af448d90653abf1679b0c7e9a1f07efdd4c3108a401b259a",
|
|
"env": {
|
|
"POSTGRES_PASSWORD": "bootstrap",
|
|
"PGDATA": "/var/lib/postgresql/data/pgdata"
|
|
},
|
|
"ports": ["5432:5432"],
|
|
"volumes": ["mesh-store-data:/var/lib/postgresql/data"]
|
|
},
|
|
// Over TCP, not the socket. While the store initialises it runs a temporary server on the
|
|
// socket ONLY, then stops it and starts the real one — so a socket check passes, the action
|
|
// exits happy, and the verify a moment later lands in the gap and fails. The action and its
|
|
// verify must ask the same question, or the action can succeed into a state verify rejects.
|
|
{
|
|
"id": "store-ready",
|
|
"type": "action",
|
|
"in": "mesh-store",
|
|
"command": ["sh", "-c", "for i in $(seq 1 180); do pg_isready -h 127.0.0.1 -U postgres >/dev/null 2>&1 && exit 0; sleep 1; done; echo 'the store did not answer within 180s; its own last words follow'; pg_isready -h 127.0.0.1 -U postgres; tail -n 20 /var/lib/postgresql/data/log/*.log 2>/dev/null; exit 1"],
|
|
"verify": ["pg_isready", "-h", "127.0.0.1", "-U", "postgres"]
|
|
},
|
|
{
|
|
"id": "inventory-database",
|
|
"type": "action",
|
|
"in": "mesh-store",
|
|
"command": ["sh", "-c", "psql -U postgres -c 'CREATE DATABASE inventory'"],
|
|
"verify": ["sh", "-c", "psql -U postgres -lqt | cut -d'|' -f1 | grep -qw inventory"]
|
|
},
|
|
{
|
|
"id": "identity-database",
|
|
"type": "action",
|
|
"in": "mesh-store",
|
|
"command": ["sh", "-c", "psql -U postgres -c 'CREATE DATABASE identity'"],
|
|
"verify": ["sh", "-c", "psql -U postgres -lqt | cut -d'|' -f1 | grep -qw identity"]
|
|
},
|
|
// Each context owns its own database (novox/hq ADR 0008). A third one is a third database,
|
|
// created the same way and named the same way — which is the whole of adding a context to the
|
|
// bootstrap, and is why the count is not something the foundation has an opinion about.
|
|
{
|
|
"id": "licences-database",
|
|
"type": "action",
|
|
"in": "mesh-store",
|
|
"command": ["sh", "-c", "psql -U postgres -c 'CREATE DATABASE licences'"],
|
|
"verify": ["sh", "-c", "psql -U postgres -lqt | cut -d'|' -f1 | grep -qw licences"]
|
|
},
|
|
{
|
|
"id": "context-schemas",
|
|
"type": "action",
|
|
"command": ["docker", "run", "--rm", "--network", "container:mesh-store",
|
|
"-e", "MESH_STORE_INVENTORY=postgres://postgres:bootstrap@127.0.0.1:5432/inventory?sslmode=disable",
|
|
"-e", "MESH_STORE_IDENTITY=postgres://postgres:bootstrap@127.0.0.1:5432/identity?sslmode=disable",
|
|
"-e", "MESH_STORE_LICENCES=postgres://postgres:bootstrap@127.0.0.1:5432/licences?sslmode=disable",
|
|
"192.0.2.250:5000/mesh-controller@sha256:c67db38439ff0aee242b467486765467bb95801f52175fc5727cc4e437338ace",
|
|
"migrate"],
|
|
"verify": ["sh", "-c", "docker exec mesh-store psql -U postgres -d inventory -tAc \"select to_regclass('public.node')\" | grep -qx node && docker exec mesh-store psql -U postgres -d identity -tAc \"select to_regclass('public.signing_key')\" | grep -qx signing_key && docker exec mesh-store psql -U postgres -d licences -tAc \"select to_regclass('public.licence')\" | grep -qx licence"]
|
|
},
|
|
// **The bus is raised where the `nats` module keeps it** (novox/hq ADR 0333 §1): the container
|
|
// `mesh-broker-nats`, its configuration and user list in /var/lib/nats-module/conf, its streams in
|
|
// /var/lib/mesh-broker-nats and its certificate in /var/lib/mesh-broker-tls — the names and paths the
|
|
// catalogue's module declares. So adopting it as that module at the broker step keeps it: the
|
|
// module's built image takes the place of the upstream one below, once, on the same streams, users
|
|
// and certificate, and nothing else moves. mesh-host's internal/bootstrap holds the two to each other
|
|
// against the module's own manifest (busLayout), at the broker step and in its tests.
|
|
//
|
|
// Directories on the machine, not volumes the runtime names: the module mounts these paths, and the
|
|
// control plane, once it runs as the mesh's process, reads the certificate at its path on the machine.
|
|
// The certificate's directory is readable by everyone because the certificate is public and the control
|
|
// plane runs as nobody; its key is written 0600 by the program that makes it.
|
|
{
|
|
"id": "bus-tls-dir",
|
|
"type": "directory",
|
|
"path": "/var/lib/mesh-broker-tls",
|
|
"mode": "0755"
|
|
},
|
|
{
|
|
"id": "bus-data-dir",
|
|
"type": "directory",
|
|
"path": "/var/lib/mesh-broker-nats",
|
|
"mode": "0700"
|
|
},
|
|
{
|
|
"id": "bus-certificate",
|
|
"type": "action",
|
|
// **The mesh makes its own** (novox/hq 04-ISSUES/146). This ran `openssl` inside the
|
|
// broker's image while the broker was one that carried it; the bus that replaced it has a
|
|
// shell and no openssl, and no other image the bundle names has one either. So the program
|
|
// that needs the certificate writes it — already on this machine, since the schema step ran
|
|
// it, and asking nothing of the image it writes into. Self-signed on purpose: a host pins
|
|
// this server's exact certificate (novox/hq ADR 0004), and at this moment there is no mesh
|
|
// to ask an authority of.
|
|
// `--user 0:0` because the directory is root's and this image runs as nobody, which is right
|
|
// for the long-running control plane and wrong for a one-shot writing into a fresh directory.
|
|
"command": ["docker", "run", "--rm", "--user", "0:0", "-v", "/var/lib/mesh-broker-tls:/tls",
|
|
"192.0.2.250:5000/mesh-controller@sha256:c67db38439ff0aee242b467486765467bb95801f52175fc5727cc4e437338ace",
|
|
"broker", "certificate", "--into", "/tls"],
|
|
"verify": ["docker", "run", "--rm", "--user", "0:0", "-v", "/var/lib/mesh-broker-tls:/tls",
|
|
"192.0.2.250:5000/mesh-controller@sha256:c67db38439ff0aee242b467486765467bb95801f52175fc5727cc4e437338ace",
|
|
"broker", "certificate", "--check", "--into", "/tls"]
|
|
},
|
|
// **Seeds, written only where nothing is** (create-once). The bus's configuration and user list are
|
|
// the bundle's until the bus is adopted, and the module's from then on (ADR 0333 §4): an installer
|
|
// run again after the adoption applies this bundle again, and an ordinary file would put the
|
|
// bundle's one user back over every user the mesh composed, and leave the node unable to reach its
|
|
// bus. Until the adoption the installer places the composed list over the seed (enrol.go).
|
|
{
|
|
"id": "bus-conf-dir",
|
|
"type": "directory",
|
|
"path": "/var/lib/nats-module/conf",
|
|
"mode": "0700"
|
|
},
|
|
{
|
|
"id": "bus-conf",
|
|
"type": "file",
|
|
"path": "/var/lib/nats-module/conf/nats.conf",
|
|
"mode": "0644",
|
|
"create-once": true,
|
|
"content": "port: 4222\nhttp: 0.0.0.0:8222\n\ntls {\n cert_file: \"/tls/tls.crt\"\n key_file: \"/tls/tls.key\"\n}\n\njetstream {\n store_dir: \"/data\"\n}\n\ninclude accounts.conf\n"
|
|
},
|
|
{
|
|
"id": "bus-accounts",
|
|
"type": "file",
|
|
"path": "/var/lib/nats-module/conf/accounts.conf",
|
|
"mode": "0600",
|
|
"create-once": true,
|
|
"content": "// The first user list, carried by the installer because at genesis there is no mesh to\n// compose one. A bootstrap credential, rotated with the store's and replaced by the\n// controller's own composition from its first start onward.\naccounts {\n MESH {\n jetstream: enabled\n users = [\n { user: \"controller\", password: \"$2a$10$AHqJgOifIVbU41KmATiMhuXFs8xa7Wl2HuN4UVBCXdN2jIQzjqApy\", permissions: {\n publish: { allow: [\"$JS.ACK.CONTROL.controller.>\", \"$JS.ACK.EVENTS.controller.>\", \"$JS.ACK.DEAD_LETTER_NOTICES.controller.>\", \"$JS.API.>\", \"$KV.mesh-controller_asked.>\", \"$KV.mesh-controller_calls.>\", \"$KV.mesh-controller_hand-acts.>\", \"$KV.mesh-controller_conditions.>\", \"$KV.mesh-controller_condition-history.>\", \"$KV.mesh-controller_lease.>\", \"$KV.SEAT_MESH_BUILD_MACHINE_cancelled.>\", \"$KV.SEAT_NODE_BUILD_AGENT_cancelled.>\", \"_INBOX.enrol.>\", \"mesh.assignment.>\", \"mesh.mod.*.tool.>\", \"mesh.node.>\", \"mesh.seat.mesh-build-machine.accept.>\", \"mesh.seat.node-build-agent.accept.>\", \"mesh.seat.mesh-build-machine.tool.>\", \"mesh.seat.node-build-agent.tool.>\", \"mesh.seat.mesh-controller.event.applied\", \"mesh.seat.mesh-controller.event.built-before\", \"mesh.seat.mesh-controller.event.refused\", \"mesh.seat.mesh-controller.event.rolled-back\", \"mesh.seat.mesh-controller.event.checked\", \"mesh.seat.mesh-controller.event.plan-moved\", \"mesh.seat.mesh-controller.event.condition-raised\", \"mesh.seat.mesh-controller.event.condition-changed\", \"mesh.seat.mesh-controller.event.condition-cleared\", \"mesh.seat.mesh-controller.event.doctor-heartbeat\", \"mesh.seat.mesh-controller.event.secret-replaced\", \"mesh.seat.mesh-controller.event.healer-acted\", \"$SRV.INFO\", \"mesh.seat.node-intrusion-prevention.tool.banned.*\", \"mesh.seat.node-backup.tool.backed-up.*\", \"mesh.seat.node-backup.tool.now.*\", \"mesh.seat.mesh-delivery.tool.stalled\", \"mesh.seat.mesh-delivery.tool.close\", \"mesh.seat.node-launcher.tool.secret.*\", \"mesh.seat.mesh-delivery.tool.release\", \"mesh.seat.mesh-delivery.tool.stop\", \"mesh.seat.mesh-delivery.tool.times\", \"mesh.seat.mesh-delivery.tool.groups\", \"mesh.seat.node-service-manager.tool.restart.*\", \"mesh.seat.mesh-controller.tool.plans\", \"mesh.events.dead.>\", \"mesh.again.>\"] }\n subscribe: { allow: [\"$JS.API.>\", \"$JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES.>\", \"$JS.EVENT.ADVISORY.CONSUMER.DELETED.>\", \"_DELIVER.controller\", \"_DELIVER.controller.>\", \"_INBOX.controller.>\", \"mesh.control.>\", \"mesh.mod.*.event.provisioner.failing\", \"mesh.mod.*.event.provisioner.recovered\", \"mesh.mod.*.event.provisioner.retirement\", \"mesh.mod.gitea.event.pull.merged\", \"mesh.mod.gitea.event.pull.updated\", \"mesh.mod.mesh-catalog.event.catching-up\", \"mesh.mod.mesh-catalog.event.upgraded\", \"mesh.seat.mesh-build-machine.event.built\", \"mesh.seat.node-build-agent.event.built\", \"mesh.seat.operator-channel.event.decided.mesh-controller\", \"mesh.seat.mesh-controller.tool.>\", \"$SRV.PING\", \"$SRV.INFO\", \"$SRV.PING.mesh-controller\", \"$SRV.PING.mesh-controller.>\", \"$SRV.INFO.mesh-controller\", \"$SRV.INFO.mesh-controller.>\", \"$SRV.STATS\", \"$SRV.STATS.mesh-controller\", \"$SRV.STATS.mesh-controller.>\"] }\n allow_responses: { max: 1, ttl: \"1m\" }\n } }\n ]\n }\n}\n"
|
|
},
|
|
// The upstream server, until the broker step adopts it: the `nats` module's own image, built by the
|
|
// mesh, takes its place there once, on these same directories (ADR 0333 §3). The published port is the
|
|
// node's bus port, 5671 unless given, which every enrolment token names; the installer gives the
|
|
// module the setting that maps its declared 4222 to it (ADR 0333 §2).
|
|
{
|
|
"id": "broker",
|
|
"type": "container",
|
|
"name": "mesh-broker-nats",
|
|
"image": "192.0.2.250:5000/nats@sha256:b83efabe3e7def1e0a4a31ec6e078999bb17c80363f881df35edc70fcb6bb927",
|
|
"ports": ["5671:4222", "127.0.0.1:8222:8222"],
|
|
"volumes": ["/var/lib/mesh-broker-nats:/data", "/var/lib/nats-module/conf:/etc/nats:ro", "/var/lib/mesh-broker-tls:/tls:ro"],
|
|
"args": ["-c", "/etc/nats/nats.conf", "-js"]
|
|
},
|
|
{
|
|
"id": "broker-ready",
|
|
"type": "action",
|
|
"command": ["sh", "-c", "for i in $(seq 1 60); do docker run --rm --network host --entrypoint sh 192.0.2.250:5000/nats@sha256:b83efabe3e7def1e0a4a31ec6e078999bb17c80363f881df35edc70fcb6bb927 -c 'nc -z 127.0.0.1 5671' >/dev/null 2>&1 && exit 0; sleep 1; done; exit 1"],
|
|
"verify": ["sh", "-c", "docker run --rm --network host --entrypoint sh 192.0.2.250:5000/nats@sha256:b83efabe3e7def1e0a4a31ec6e078999bb17c80363f881df35edc70fcb6bb927 -c 'nc -z 127.0.0.1 5671'"]
|
|
},
|
|
{
|
|
"id": "control-plane",
|
|
"type": "container",
|
|
"name": "mesh-controller",
|
|
"image": "192.0.2.250:5000/mesh-controller@sha256:c67db38439ff0aee242b467486765467bb95801f52175fc5727cc4e437338ace",
|
|
"network": "host",
|
|
"args": ["serve"],
|
|
"volumes": ["/var/lib/mesh-broker-tls:/broker-tls:ro"],
|
|
"env": {
|
|
"MESH_STORE_INVENTORY": "postgres://postgres:bootstrap@127.0.0.1:5432/inventory?sslmode=disable",
|
|
"MESH_STORE_IDENTITY": "postgres://postgres:bootstrap@127.0.0.1:5432/identity?sslmode=disable",
|
|
"MESH_STORE_LICENCES": "postgres://postgres:bootstrap@127.0.0.1:5432/licences?sslmode=disable",
|
|
"MESH_BUS_NATS": "nats://controller:bootstrap@127.0.0.1:5671",
|
|
"MESH_BROKER_ADDRESS": "192.0.2.10:5671",
|
|
"MESH_BROKER_CERTIFICATE": "/broker-tls/tls.crt"
|
|
}
|
|
}
|
|
|
|
]
|
|
}
|