jschoubben
f4bcb320fe
A module's contributes was map[string]map[string]any — one JSON object key
per requirement, structurally exactly one contribution to "route" ever.
minio needs two public hostnames (the S3 API and the console), which is
two different contributions to route from one module, and nothing let it
say so.
This is the same shape of problem ADR 0094 solved for secrets (a module
needing several values from one provider that gives one per pair):
contributes now accepts either the ordinary {label, port} object, or an
object of local names to several such objects. Detected per requirement
key by what's inside, since (unlike secrets' string-vs-object split) both
shapes are JSON objects: an ordinary contribution's fields are scalars, the
several-instance shape is local-name -> object. Confirmed against every
module.json in mesh-catalog before relying on that split.
Both route-proxy and the migration-era route-adapter already key generated
routers off the composed hostname (Values["name"]), not the module name,
so two contributions with the same From reach them as two independent
routes with no changes needed on the receiving side.
1473 lines
65 KiB
Go
1473 lines
65 KiB
Go
package catalogue
|
|
|
|
// Turning a resolution into the declaration a node is sent.
|
|
//
|
|
// Separate from resolving because they answer different questions. Resolving asks *what should
|
|
// this machine run*; this asks *what does that look like as resources*, and the second is where
|
|
// settings are applied, generators are called, contributions are collected and credentials are
|
|
// placed. Both lived in one file until it was doing four jobs at once — which is the shape the
|
|
// system this replaces failed in, one import at a time.
|
|
|
|
import (
|
|
"encoding/json"
|
|
"fmt"
|
|
"sort"
|
|
"strconv"
|
|
"strings"
|
|
"time"
|
|
)
|
|
|
|
// SettingsBy is the layers that apply to each module, keyed by module name.
|
|
type SettingsBy map[string][]Layer
|
|
|
|
// Generator works out a module's resources for one node, where they cannot be written in advance.
|
|
type Generator interface {
|
|
// Resources for this node. Absent means the node is not part of whatever this generates,
|
|
// which is an ordinary answer rather than a failure — a machine assigned the module before it
|
|
// has an address on the network is in exactly that state.
|
|
Resources(node string) ([]map[string]any, bool, error)
|
|
}
|
|
|
|
// OpensPorts is a generator that also says what its resources accept connections on.
|
|
//
|
|
// **Separate from Generator, because most generators have nothing to say here** and requiring an
|
|
// empty method of each would be a cost paid everywhere for one caller.
|
|
//
|
|
// It exists because a static field cannot express this. A hub accepts connections from every node
|
|
// at other sites; a machine that is not a hub dials out and needs nothing open — and they are the
|
|
// same module. `listens` in a manifest is one answer for every machine that runs it, so the
|
|
// machine that most needs filtering, the one facing the public internet, was the one whose rule
|
|
// set would have closed its own overlay.
|
|
type OpensPorts interface {
|
|
// Listens is what this node accepts on because of what was computed for it. Nothing is the
|
|
// ordinary answer: most machines running a computed module open no port at all.
|
|
Listens(node string) ([]Listening, error)
|
|
}
|
|
|
|
// Grant is one consumer's credential, on the machine that must create it.
|
|
type Grant struct {
|
|
// Provision is what was required.
|
|
Provision string
|
|
// Consumer is the node that will use it.
|
|
Consumer string
|
|
// From is the module on that machine which asked.
|
|
//
|
|
// **Part of who this credential is for, not a label** (novox/hq 04-ISSUES/022). Together with
|
|
// Consumer it names one consumer; the node alone does not, because a machine routinely runs
|
|
// several modules wanting the same thing. It is also what the provider names the role or the
|
|
// bucket or the client after, so withdrawing one consumer does not take another's away.
|
|
From string
|
|
// Values are what that module contributed — the name it wants, and anything else the
|
|
// provision's own vocabulary defines.
|
|
Values map[string]any
|
|
// Local is the name the credential goes by inside the consumer where it keeps several for one
|
|
// provision (ADR 0094); empty for the ordinary one. The provider sees it as a holder of its own.
|
|
Local string
|
|
// Slug is the consumer module's identity slug, if it declared one — carried on the grant so the
|
|
// provider side derives the same login the consumer does, even across nodes where the consumer's
|
|
// manifest is not in view (novox/hq ADR 0049). Empty means "use the module name".
|
|
Slug string
|
|
// At is where the consuming machine is on the private network, empty if it is not on one.
|
|
//
|
|
// Passed in with the grant because it is a fact about another machine, and resolution answers
|
|
// questions about one. A provider that must reach back to its consumer — a reverse proxy is
|
|
// the whole reason this exists — otherwise has to know how the mesh names machines.
|
|
At string
|
|
// Sealed is the credential, closed to the providing node.
|
|
Sealed string
|
|
}
|
|
|
|
// Rendering is everything needed to turn a resolution into the declaration a node is sent.
|
|
type Rendering struct {
|
|
// Certificate is what the mesh issued for this machine's internal name, and the mesh's own
|
|
// certificate. Both public — the key they belong to never left the machine.
|
|
Certificate string
|
|
Authority string
|
|
|
|
// Needed is each module's own secrets, sealed to this node, keyed by module and then by the
|
|
// name the module gave it.
|
|
Needed map[string]map[string]string
|
|
|
|
// Mesh is every node's address on the private network, which is what a rule saying "from the
|
|
// mesh" resolves to. Passed in for the same reason grants are: who else is on the network is
|
|
// a fact about the mesh, and resolution answers questions about one machine.
|
|
Mesh []string
|
|
// Suffix is what a machine's internal name ends in, as the control plane composed Names —
|
|
// `internal` unless the operator chose another — so a fact writing those names does not
|
|
// compose it a second time.
|
|
Suffix string
|
|
|
|
// Kept is every operator-sealed secret in the mesh, for a module that `keeps` them. Nil when
|
|
// nothing on this node keeps them, or the mesh has no operator key.
|
|
Kept *KeptExport
|
|
|
|
// Foundation is the ports the mesh itself needs reachable on every machine, which no module
|
|
// declares because the foundation is not a module (novox/hq 04-ISSUES/051 and 052). The broker
|
|
// is the one that matters: a machine dials it to enrol, and a firewall derived only from
|
|
// modules closes it.
|
|
Foundation []int
|
|
|
|
// Names is every machine's internal name and its address, for containers to be given.
|
|
//
|
|
// **A container does not inherit the machine's names**, so every internal name the mesh wrote
|
|
// is invisible to what the machine runs. Given here rather than looked up on the machine,
|
|
// because which machines exist is a fact about the mesh.
|
|
Names map[string]string
|
|
|
|
// Machines is only the machines, by the same internal name — the subset of Names that is a
|
|
// node of this mesh rather than a name it was told to serve. Both matter and they are not the
|
|
// same set: a container's hosts wants every name, so a routed name resolves to the proxy that
|
|
// serves it, while a resolver told the mesh's suffix is authoritative for it answers from what
|
|
// it is given and forwards nothing — so a routed name written there is a name nobody asks for,
|
|
// standing beside the machines and looking as real as they do.
|
|
Machines map[string]string
|
|
|
|
Settings SettingsBy
|
|
Generators map[string]Generator
|
|
// Grants are the credentials this node must create, for the provisions it offers. Passed in
|
|
// rather than resolved, because who consumes a node is a fact about the rest of the mesh and
|
|
// resolution answers questions about one machine.
|
|
Grants []Grant
|
|
|
|
// Ports is where this machine puts what each module needs reachable, by module and by the
|
|
// port the software itself uses (novox/hq ADR 0038).
|
|
//
|
|
// **The one place the number now lives.** A module used to write it three times — for the rule
|
|
// set, for what a consumer is told, and for what the runtime publishes — and nothing checked
|
|
// that the three agreed. They are all derived from this.
|
|
Ports map[string]map[int]int
|
|
|
|
// Adopted says the node is adopted (novox/hq ADR 0100): the firewall found on it stays in
|
|
// force, so no module that loads a filter is declared there, and what the mesh needs
|
|
// reachable is declared as openings, with its own ports guarded by a table that only refuses.
|
|
Adopted bool
|
|
|
|
// Given is the machine ports this node was given for its modules' ports, by module and by the
|
|
// port the software uses (novox/hq ADR 0100) — the foundation's ports, as genesis chose them.
|
|
// They win over anything the mesh would assign and over a manifest's own long-form mapping.
|
|
Given map[string]map[int]int
|
|
|
|
// Taken is the modules taken on this adopted node (novox/hq ADR 0100). The guard is derived
|
|
// from these only (ADR 0103): a port of a module assigned but not taken may still be the
|
|
// predecessor's.
|
|
Taken map[string]bool
|
|
|
|
// Seats is where this machine put each mesh-scoped seat's holder, by seat and by the port the
|
|
// holder's software uses (novox/hq 04-ISSUES/102) — read from the node's settings and
|
|
// assignments for whichever module claims the seat, whether or not it is in this node's set.
|
|
// What ${seat:…} answers with; see seat_into.go for why the answer may be absent.
|
|
Seats map[string]map[int]int
|
|
|
|
// ArtifactStore is the mesh's artifact store as this network reaches it (host:port) — the
|
|
// node holding it and the port that node put it on — or empty when the mesh has none on its
|
|
// network yet. Composed into every image and archive the mesh built, at this moment and never
|
|
// stored (novox/hq 04-ISSUES/102).
|
|
ArtifactStore string
|
|
|
|
// Built is every `<module>/<artifact>` the mesh has built. What tells a reference recorded
|
|
// with an address — before references were kept without one — from an image a module runs
|
|
// straight from a public registry.
|
|
Built map[string]bool
|
|
}
|
|
|
|
// machinePort is where a module's port lives on this machine, or the port itself when the mesh has
|
|
// not been asked. Unassigned is not an error here: a module with no `listens` never needed one,
|
|
// and a caller composing a declaration without a store still gets something coherent.
|
|
func (r Rendering) machinePort(module string, wanted int) int {
|
|
if at, given := r.Given[module][wanted]; given {
|
|
return at
|
|
}
|
|
if at, known := r.Ports[module][wanted]; known {
|
|
return at
|
|
}
|
|
return wanted
|
|
}
|
|
|
|
// Declaration is everything the resolved modules put on the node, with settings applied.
|
|
//
|
|
// Resource identities are prefixed with the module they came from. Two modules may reasonably
|
|
// both call something "config", and without this the second would silently replace the first —
|
|
// the node applying one of them and reporting success.
|
|
func (r Resolution) Declaration(with Rendering) ([]map[string]any, error) {
|
|
composed, err := r.Compose(with)
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
return composed.Resources, nil
|
|
}
|
|
|
|
// Composed is a declaration's resources and which module each came from.
|
|
//
|
|
// Owner is kept beside the resources because a resource id cannot be split back into its module:
|
|
// a module's name may itself contain a dot. What the mesh adds of its own — an opening, the guard —
|
|
// has no owner.
|
|
type Composed struct {
|
|
Resources []map[string]any
|
|
Owner map[string]string
|
|
}
|
|
|
|
// Compose is Declaration with the owner of every resource said.
|
|
func (r Resolution) Compose(with Rendering) (Composed, error) {
|
|
owner := map[string]string{}
|
|
resources, err := r.compose(with, owner)
|
|
if err != nil {
|
|
return Composed{}, err
|
|
}
|
|
return Composed{Resources: resources, Owner: owner}, nil
|
|
}
|
|
|
|
func (r Resolution) compose(with Rendering, owner map[string]string) ([]map[string]any, error) {
|
|
// Where each provision's credentials land, so a contribution can name the file rather than
|
|
// carry a value the mesh does not have.
|
|
directories := map[string]string{}
|
|
for _, m := range r.Modules {
|
|
for provision, where := range m.Grants {
|
|
directories[provision] = where
|
|
}
|
|
}
|
|
given, err := r.contributions(with.Settings, with.Grants, directories)
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
|
|
// A workload on THIS machine contributes the port it declared, and the machine may have
|
|
// published it somewhere else (novox/hq ADR 0066). The mirror of the redirect below: 038 fixed
|
|
// what a consumer is TOLD about a provider, and this is what a workload TELLS the provider about
|
|
// itself — gitea declaring 3000, published as 20000:3000, and the co-located proxy dialling 3000,
|
|
// where nothing listens, for every request.
|
|
//
|
|
// **The contributing module's assignment, never the provider's.** The port belongs to the
|
|
// workload; using the map of whatever answers the requirement would move a route to wherever the
|
|
// proxy happens to be published. A contribution carried here from another machine is left exactly
|
|
// as it is — its port is that machine's to assign, and this map knows nothing about it.
|
|
for provision := range given {
|
|
for i := range given[provision] {
|
|
said := &given[provision][i]
|
|
if said.Node != "" && said.Node != r.Node {
|
|
continue
|
|
}
|
|
said.Values = atMachinePort(said.Values, said.From, with.Ports)
|
|
}
|
|
}
|
|
|
|
// A provider answered on this same machine never passed through the walk that works out what a
|
|
// provider on ANOTHER machine serves (novox/hq 04-ISSUES/038, ADR 0066). That walk does two
|
|
// things — it derives the served facts with the provider node's port assignments, and it settles
|
|
// them with that node's settings layers — and the same-node paths (resolve.go's servedHere, and
|
|
// here() below) did neither, because both run while resolving, before either is known.
|
|
//
|
|
// The port was the first half to be noticed (038). The second half is worse and quieter: a served
|
|
// value the OPERATOR supplied — a certificate authority's root, which is the one kind of value a
|
|
// manifest cannot carry, because it differs on every mesh — arrived as the manifest's empty
|
|
// default. A proxy then wrote an empty CA bundle, fell back to the system trust store, could not
|
|
// verify the internal authority, and stopped issuing with nothing saying why.
|
|
//
|
|
// So it is re-derived here, in full, exactly as the cross-node path derives it. **After the
|
|
// closure is known**, which also removes an order dependence: the resolver built these needs
|
|
// mid-walk from whichever modules had been chosen by then, so what a co-located binding carried
|
|
// depended on the order somebody happened to assign things in.
|
|
for i := range r.Needs {
|
|
if r.Needs[i].ByRecord || r.Needs[i].From != r.Node {
|
|
continue
|
|
}
|
|
serves, answered, err := r.servedOnThisMachine(r.Needs[i].Name, with)
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
if answered {
|
|
r.Needs[i].Serves = serves
|
|
}
|
|
}
|
|
|
|
// Once, from every module's listens -- not per module. A module receiving only its own ports
|
|
// would write a rule set that closed every other module on the machine. Each module's per-node
|
|
// exposure settings override its listens' source first (novox/hq ADR 0046).
|
|
rules, err := r.Rules(with)
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
filtering := AsNftables(rules, with.Mesh, r.PublicDomain != "", with.Foundation)
|
|
|
|
var out []map[string]any
|
|
for _, m := range r.Modules {
|
|
if with.Adopted && m.Filtering != nil {
|
|
// Nothing of a module that loads a filter, on an adopted node: its table would drop
|
|
// by default and hold accepts, and the found firewall stays in force. Every resource,
|
|
// not only the rule set — its service must not run, and a node returned to adopted
|
|
// stops it by the ordinary removal of what is no longer declared.
|
|
continue
|
|
}
|
|
resources := m.Resources
|
|
|
|
// What the mesh computes for this module goes FIRST, before the module's own resources.
|
|
//
|
|
// **Order is stated, not derived — the host does not sort** (novox/hq ADR 0005), so
|
|
// whatever the mesh writes down is the order a machine applies. A module's service or
|
|
// container routinely depends on one of these files; nothing here ever depends on a
|
|
// module's resources, because none of it is computed from them.
|
|
//
|
|
// Appended, this was wrong in a way that only showed on the first apply and then healed:
|
|
// the service started before its certificate or its rule set existed, failed, and the next
|
|
// reconcile fixed it. A fault that repairs itself on the second attempt is worse than one
|
|
// that does not, because what gets remembered is that it works.
|
|
var first []map[string]any
|
|
if f := m.Filtering; f != nil {
|
|
first = append(first, map[string]any{
|
|
"id": FilteringID(), "type": "file", "path": f.Into,
|
|
"content": filtering, "mode": "0600",
|
|
})
|
|
}
|
|
if c := m.Certificate; c != nil {
|
|
if with.Certificate == "" {
|
|
// Asked for and not issued. Refused rather than skipped: a module that serves TLS
|
|
// with no certificate does not start, and the reason is somewhere else entirely.
|
|
return nil, fmt.Errorf(
|
|
"%s wants a certificate for this machine and none was issued", m.Module)
|
|
}
|
|
first = append(first, map[string]any{
|
|
"id": CertificateID(), "type": "file", "path": c.Into,
|
|
// Public. It travels in the open like any other file, because it is a statement
|
|
// about a key rather than the key.
|
|
"content": with.Certificate, "mode": "0644",
|
|
})
|
|
if c.Authority != "" {
|
|
first = append(first, map[string]any{
|
|
"id": AuthorityID(), "type": "file", "path": c.Authority,
|
|
"content": with.Authority, "mode": "0644",
|
|
})
|
|
}
|
|
}
|
|
for _, name := range sortedKeys(m.OwnSecrets) {
|
|
sealed := with.Needed[m.Module][name]
|
|
if sealed == "" {
|
|
// Declared and not made. Refused rather than skipped: a module whose own
|
|
// credential is silently absent starts, fails to authenticate, and the reason is
|
|
// three layers away from the machine reporting it.
|
|
return nil, fmt.Errorf(
|
|
"%s needs a secret called %q and none was made for it", m.Module, name)
|
|
}
|
|
first = append(first, ownedBy(m.SecretsOwner, map[string]any{
|
|
"id": NeedID(name), "type": "file", "path": m.OwnSecrets[name], "sealed": sealed,
|
|
}))
|
|
}
|
|
// Operator-owned paths this module is granted use of (novox/hq ADR 0051). Written before
|
|
// the module's own resources, and so before the container that mounts them: the host must
|
|
// find each present — refusing clearly if the operator has not provided it — before it
|
|
// starts anything that depends on it. The mesh creates, chowns and reconciles none of it;
|
|
// an `access` resource says only *this path must exist, and this module reaches it*.
|
|
for _, a := range m.Accesses {
|
|
first = append(first, map[string]any{
|
|
"id": AccessID(a.Path), "type": "access", "path": a.Path, "mode": a.At(),
|
|
})
|
|
}
|
|
for _, to := range m.SecretRequirements() {
|
|
for _, file := range m.SecretFiles(to) {
|
|
var found *Needed
|
|
for i, n := range r.Needs {
|
|
// **This module's need, not the provision's** (novox/hq 04-ISSUES/022). Matching
|
|
// on the name alone, every consumer of a provision took whichever credential
|
|
// happened to be last in the list — so on a node with two of them, one module
|
|
// would be given the other's password and fail to authenticate with a valid
|
|
// credential belonging to somebody else. And this file's local name, where the
|
|
// module keeps several (ADR 0094).
|
|
if n.Name == to && n.For == m.Module && n.Local == file.Local {
|
|
found = &r.Needs[i]
|
|
}
|
|
}
|
|
if found != nil && found.ByRecord && found.Sealed == "" && !found.Manager {
|
|
// Answered by a record whose key has not been supplied since this consumer was
|
|
// put on it. **Refused, not skipped.** The mesh discarded the plaintext when the
|
|
// key was accepted and cannot seal another, so a machine that resolved cleanly
|
|
// would receive no file at all and fail at whatever tried to read it — which is
|
|
// the outcome ADR 0024 exists to avoid, arrived at politely.
|
|
//
|
|
// The manager holder is the one exception (novox/hq ADR 0050): an empty refresh token
|
|
// is a licence whose manager has not adopted one yet, a real waiting state rather than
|
|
// a lost key. It falls through to the skip below — its bound facts (carrying the
|
|
// manager's public key) are still delivered, which is what adoption needs to seal the
|
|
// first refresh token.
|
|
return nil, fmt.Errorf(
|
|
"%s on this machine uses the licence %q and no key has been sealed to it. "+
|
|
"The mesh cannot make one; supply it again with `licence key %s`",
|
|
m.Module, found.From, found.From)
|
|
}
|
|
if found == nil || found.Sealed == "" {
|
|
// Answered on this machine, or answered by a node the mesh could not seal to.
|
|
// Nothing to write either way, and writing an empty credential file would be
|
|
// worse than none: something would read it and fail authenticating.
|
|
continue
|
|
}
|
|
first = append(first, ownedBy(m.SecretsOwner, map[string]any{
|
|
"id": SecretID(SecretLocal(to, file.Local)), "type": "file", "path": file.Path,
|
|
"sealed": found.Sealed,
|
|
}))
|
|
}
|
|
}
|
|
for _, to := range sortedKeys(m.Grants) {
|
|
for _, g := range with.Grants {
|
|
if g.Provision != to {
|
|
continue
|
|
}
|
|
if g.From == "" {
|
|
// Nothing on that machine asks for this any more. Skipped here rather than
|
|
// where grants are gathered, so the rule holds whoever gathers them.
|
|
//
|
|
// **This is how a credential is withdrawn.** The provisioner removes what
|
|
// nobody asks for, and it can only do that if the mesh stops asking — a
|
|
// consumer that was unassigned would otherwise keep a working login for ever,
|
|
// and nothing would say so.
|
|
continue
|
|
}
|
|
first = append(first, map[string]any{
|
|
// One file per holder — the consumer's module with its local name after it
|
|
// where it keeps several (ADR 0094); the lab found two files with one id.
|
|
"id": GrantID(to, g.Consumer+"."+holderAs(g.From, g.Local)),
|
|
"type": "file",
|
|
"path": grantPath(m.Grants[to], g.Consumer, holderAs(g.From, g.Local)),
|
|
"sealed": g.Sealed,
|
|
})
|
|
}
|
|
}
|
|
for _, to := range sortedKeys(m.Binds) {
|
|
var found *Needed
|
|
for i, n := range r.Needs {
|
|
if n.Name == to && n.For == m.Module {
|
|
found = &r.Needs[i]
|
|
}
|
|
}
|
|
if found == nil {
|
|
// Answered on this same machine rather than from elsewhere in the mesh.
|
|
//
|
|
// **Still written.** It used to be skipped, reasoning that "it is on this node"
|
|
// is a fact nobody needs — and that is true of the *location* and false of
|
|
// everything beside it. A binding also carries what the provider said a consumer
|
|
// must know, which is the port; a consumer cannot invent that, and got an absent
|
|
// file with no explanation. A build machine sharing a node with the registry it
|
|
// pushes to sat in a loop saying it could not read its own binding.
|
|
here, err := here(r, to, with)
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
if here == nil {
|
|
// Nothing in this node's set offers it either, so there is genuinely nothing
|
|
// to say. Resolution has already refused anything unanswerable, so this is a
|
|
// requirement met by the module itself.
|
|
continue
|
|
}
|
|
found = here
|
|
}
|
|
file, err := boundFile(*found, m.Binds[to], ConsumerIdentity(r.Node, IdentitySource(m.Slug, m.Module)))
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
first = append(first, file)
|
|
}
|
|
for _, to := range sortedKeys(m.Receives) {
|
|
file, err := receivedFile(to, m.Receives[to], given[to])
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
first = append(first, file)
|
|
}
|
|
if m.Keeps != "" && with.Kept != nil {
|
|
file, err := keptFile(m.Keeps, with.Kept)
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
first = append(first, file)
|
|
}
|
|
if m.Computed != "" {
|
|
generator, known := with.Generators[m.Computed]
|
|
if !known {
|
|
return nil, fmt.Errorf(
|
|
"%s says its resources are computed by %q, and this control plane has no %q",
|
|
m.Module, m.Computed, m.Computed)
|
|
}
|
|
generated, part, err := generator.Resources(r.Node)
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
if !part {
|
|
// Assigned, and not yet part of what this generates. Nothing to put on the
|
|
// machine, which is different from an error: a node given the network module
|
|
// before it has an address is in exactly that state, briefly.
|
|
continue
|
|
}
|
|
resources = generated
|
|
}
|
|
|
|
// Now, and not before: a module whose resources are computed replaces them wholesale, and
|
|
// merging earlier would throw away the files it still needs.
|
|
resources = append(append([]map[string]any{}, first...), resources...)
|
|
|
|
// Every container is given the mesh's names. Not a choice a module makes: a module that
|
|
// listed them would go stale the day a machine joins, and one that did not would be a
|
|
// module whose containers cannot reach anything by name.
|
|
//
|
|
// A container that was given names of its own keeps them and gets the mesh's beside them:
|
|
// the mesh does not know what else a workload needs to reach, and taking something away
|
|
// to add something is not what "also" means.
|
|
if len(with.Names) > 0 {
|
|
resources = withMeshNames(resources, with.Names)
|
|
}
|
|
|
|
// What this module may name from inside one of its own files. Gathered once per module
|
|
// rather than per file, because it is a fact about the module.
|
|
sealed, err := sealedFor(m, r.Needs, with)
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
// And what its bindings say, for the half of a connection that is not secret.
|
|
known := knownFor(m, r.Needs, r.Node)
|
|
// And the machine underneath, which no binding of its own can tell it.
|
|
thisMachine := machineFacts(r, with.Names)
|
|
|
|
// Which of this module's files carry a secret, for the rule that a container may not read
|
|
// one of them as its environment without saying so (ADR 0086, issue 041).
|
|
secretFiles := secretFilesOf(resources)
|
|
|
|
for _, unsettled := range resources {
|
|
resource, err := ApplySettings(unsettled, with.Settings[m.Module])
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
copied := map[string]any{}
|
|
for k, v := range resource {
|
|
copied[k] = v
|
|
}
|
|
if err := refuseSecretsInEnvironment(copied, secretFiles, m.Module); err != nil {
|
|
return nil, err
|
|
}
|
|
// Said in the catalogue, not on the machine: the host parses strictly and knows no
|
|
// such field, and the reason is for a reader of the manifest.
|
|
delete(copied, SecretsInEnvironment)
|
|
// **After settings, and that is the whole reason it is here.** A module's file
|
|
// content is where a setting lands, so a placeholder may only exist once the setting
|
|
// has been put in — filling secrets first would look at content that is not yet what
|
|
// the machine receives.
|
|
if err := intoFile(copied, sealed, m.Module); err != nil {
|
|
return nil, err
|
|
}
|
|
// **Substituted here, not on the machine.** A bound value is not secret — the mesh
|
|
// holds it in the clear — so there is nothing for the host to be the only witness of,
|
|
// and sending it already filled in means the host learns no new field.
|
|
if err := boundInto(copied, known, m.Module); err != nil {
|
|
return nil, err
|
|
}
|
|
// And what the module could not have written: the machine it turned out to be
|
|
// assigned to. Beside the bound values because it is the same kind of fact — the
|
|
// mesh's own, held in the clear — and because a module that must name itself to
|
|
// something else has no binding to learn it from (novox/hq ADR 0066).
|
|
// Which port this machine gave it, for a module that binds directly rather than
|
|
// through a runtime that can remap (ADR 0038). Applied before the machine's facts so
|
|
// a refusal names the port rather than whatever came after it.
|
|
if err := portInto(copied, m.Module, m.Listens, with); err != nil {
|
|
return nil, err
|
|
}
|
|
// And where this machine put the foundation's servers, for the one module that
|
|
// reaches them by seat rather than by binding (novox/hq 04-ISSUES/102).
|
|
if err := seatInto(copied, m.Module, with); err != nil {
|
|
return nil, err
|
|
}
|
|
if err := machineInto(copied, thisMachine, m.Module); err != nil {
|
|
return nil, err
|
|
}
|
|
if err := pinned(copied, m.Module); err != nil {
|
|
return nil, err
|
|
}
|
|
// And the other half of the same question: an image nobody has published yet is named
|
|
// by an artifact rather than by a placeholder digest, and is just as unrunnable.
|
|
if err := built(copied, m.Module); err != nil {
|
|
return nil, err
|
|
}
|
|
// What the mesh built is kept by digest and path; the store's address is this
|
|
// network's now, composed here and never recorded (novox/hq 04-ISSUES/102).
|
|
if err := artifactsInto(copied, m.Module, with); err != nil {
|
|
return nil, err
|
|
}
|
|
publishedOn(copied, m.Module, with)
|
|
copied["id"] = m.Module + "." + fmt.Sprint(resource["id"])
|
|
// A service saying what it reflects names resources within its own module, so those
|
|
// are prefixed too or they would point at nothing.
|
|
//
|
|
// **Unless it already names one.** A module may reflect a file another module put on
|
|
// the machine — the case this exists for is a resolver restarting when the mesh
|
|
// rewrites the names, which are computed by the mesh and belong to its module, not to
|
|
// the daemon's. Written `<module>.<id>`, and a dot is what marks it as already
|
|
// answered: prefixing it again would point at nothing, silently, and the daemon would
|
|
// serve the old names for ever while everything reported success.
|
|
if renamed := reflectsRenamed(m.Module, resource["restart-on"]); renamed != nil {
|
|
copied["restart-on"] = renamed
|
|
}
|
|
// And what it is reloaded on, by the same rule (novox/hq ADR 0102): an id left
|
|
// unprefixed matches nothing, and the service is never reloaded.
|
|
if renamed := reflectsRenamed(m.Module, resource["reload-on"]); renamed != nil {
|
|
copied["reload-on"] = renamed
|
|
}
|
|
owner[fmt.Sprint(copied["id"])] = m.Module
|
|
out = append(out, copied)
|
|
}
|
|
|
|
// **What only the mesh knows, where this module asked for it.** The graph is the control
|
|
// plane's; making a name resolve is the module's software. Emitted as ordinary files under
|
|
// this module's name, so they are applied, reported and removed exactly as anything else
|
|
// it declares.
|
|
given, err := FactsInto(m, r, with.Names, with.Machines, with.Suffix)
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
for _, fact := range given {
|
|
fact["id"] = m.Module + "." + fmt.Sprint(fact["id"])
|
|
owner[fmt.Sprint(fact["id"])] = m.Module
|
|
out = append(out, fact)
|
|
}
|
|
}
|
|
if with.Adopted {
|
|
// First, before anything a module declares: what the mesh needs reachable, then its guard.
|
|
// The order a machine applies is the order written here.
|
|
ours := Openings(rules, with.Foundation, Published(out))
|
|
ours = append(ours, GuardResources(r.guarded(out, owner, rules, with))...)
|
|
out = append(ours, out...)
|
|
}
|
|
return out, nil
|
|
}
|
|
|
|
// guarded is what the mesh's guard refuses on an adopted node (novox/hq ADR 0103): derived, and
|
|
// for taken modules only.
|
|
//
|
|
// For each taken module, every machine port its containers publish that the filter would admit
|
|
// from the private network only — a published port is forwarded, not received, so a found
|
|
// firewall filtering only what it receives never sees it — together with the ports the module's
|
|
// manifest guards explicitly (the store's port, the broker's management port), wherever the
|
|
// machine put them. A port of a module assigned but not taken is not guarded: it may still be the
|
|
// predecessor's, serving the predecessor's other machines. The foundation's ports are admitted
|
|
// from everywhere and are never guarded.
|
|
func (r Resolution) guarded(out []map[string]any, owner map[string]string, rules []Rule,
|
|
with Rendering) []int {
|
|
meshOnly := map[int]bool{}
|
|
fromEverywhere := map[int]bool{}
|
|
for _, rule := range rules {
|
|
if rule.Protocol != "tcp" {
|
|
continue
|
|
}
|
|
switch rule.From {
|
|
case FromMesh:
|
|
meshOnly[rule.Port] = true
|
|
case FromEverywhere:
|
|
fromEverywhere[rule.Port] = true
|
|
}
|
|
}
|
|
for _, port := range with.Foundation {
|
|
delete(meshOnly, port)
|
|
fromEverywhere[port] = true
|
|
}
|
|
seen := map[int]bool{}
|
|
var ports []int
|
|
guard := func(at int) {
|
|
if !seen[at] {
|
|
seen[at] = true
|
|
ports = append(ports, at)
|
|
}
|
|
}
|
|
for _, m := range r.Modules {
|
|
if !with.Taken[m.Module] {
|
|
continue
|
|
}
|
|
var mine []map[string]any
|
|
for _, resource := range out {
|
|
if owner[fmt.Sprint(resource["id"])] == m.Module {
|
|
mine = append(mine, resource)
|
|
}
|
|
}
|
|
for outer := range Published(mine)["tcp"] {
|
|
if meshOnly[outer] {
|
|
guard(outer)
|
|
}
|
|
}
|
|
for _, want := range m.Guards {
|
|
at := with.machinePort(m.Module, want)
|
|
for _, resource := range mine {
|
|
if fmt.Sprint(resource["type"]) != "container" {
|
|
continue
|
|
}
|
|
listed, _ := resource["ports"].([]any)
|
|
for _, entry := range listed {
|
|
outer, inner, _, ok := mapping(fmt.Sprint(entry))
|
|
if ok && inner == want {
|
|
at = outer
|
|
}
|
|
}
|
|
}
|
|
// Not what this node is told to open to everyone: a per-node exposure setting that
|
|
// widens a guarded port is the operator saying so, and declaring an opening for it
|
|
// and a guard dropping it would be one statement refusing the other.
|
|
if !fromEverywhere[at] {
|
|
guard(at)
|
|
}
|
|
}
|
|
}
|
|
sort.Ints(ports)
|
|
return ports
|
|
}
|
|
|
|
// mapping reads a container's port mapping — `[address:]outer:inner[/protocol]`, the address
|
|
// possibly an IPv6 one in brackets — indexing from the end, so an address's own colons never
|
|
// shift the ports. Not ok for a short form or anything that is not a mapping.
|
|
func mapping(written string) (outer, inner int, address string, ok bool) {
|
|
written = strings.TrimSpace(written)
|
|
if cut := strings.LastIndex(written, "/"); cut >= 0 {
|
|
written = written[:cut]
|
|
}
|
|
parts := strings.Split(written, ":")
|
|
if len(parts) < 2 {
|
|
return 0, 0, "", false
|
|
}
|
|
inner, err := strconv.Atoi(parts[len(parts)-1])
|
|
if err != nil {
|
|
return 0, 0, "", false
|
|
}
|
|
outer, err = strconv.Atoi(parts[len(parts)-2])
|
|
if err != nil {
|
|
return 0, 0, "", false
|
|
}
|
|
return outer, inner, strings.Join(parts[:len(parts)-2], ":"), true
|
|
}
|
|
|
|
// Rules is the rule set this node's filter is derived from: every module's listens, what was
|
|
// computed for this machine, and each module's per-node exposure. The same answer whether the node
|
|
// is adopted or converged — the one loads it as a filter, the other declares it as openings.
|
|
func (r Resolution) Rules(with Rendering) ([]Rule, error) {
|
|
exposure := map[string]map[int]string{}
|
|
for _, m := range r.Modules {
|
|
e, err := Exposure(m, with.Settings[m.Module])
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
if e != nil {
|
|
exposure[m.Module] = e
|
|
}
|
|
}
|
|
return r.Filtering(with.Generators, with.Ports, exposure)
|
|
}
|
|
|
|
// Contribution is one module telling the answer to a requirement what it needs from it.
|
|
type Contribution struct {
|
|
// From is the module that said it, so the provider and a person reading the file can tell
|
|
// which route belongs to what.
|
|
From string `json:"from"`
|
|
// Node is the machine it said it from, empty when that is this one.
|
|
//
|
|
// A provision answered from anywhere in the mesh has consumers on other machines, and the
|
|
// provider has to know who they are — a database told to create a password and not who for
|
|
// cannot do anything with it. Contributions were node-local until this, which meant the one
|
|
// case that most needed them was the one they did not reach.
|
|
Node string `json:"node,omitempty"`
|
|
// At is where that machine is on the private network, empty when it is not on one or when it
|
|
// is this machine.
|
|
//
|
|
// The mesh knows it and a provider should not have to derive it. A reverse proxy is told
|
|
// *send traffic to this consumer* and has to open a connection — so without this every
|
|
// provider that reaches back to a consumer would have to know how the mesh names machines,
|
|
// which is a convention leaking into every module that implements a provision.
|
|
At string `json:"at,omitempty"`
|
|
// As is who this consumer is: what the provider should call the login it creates.
|
|
//
|
|
// **Said by the mesh rather than invented by the provisioner** (novox/hq 04-ISSUES/023). It
|
|
// used to be neither — the provisioner made a name, and the consumer, which has to present it
|
|
// to authenticate, had no way to learn it. One derivation reaches both ends, so they agree by
|
|
// construction.
|
|
// Omitted when there is none. A contribution that is not a credential grant — a module
|
|
// offering something to another on its own machine — has nobody to be identified to, and a
|
|
// field that is always present and usually empty teaches a reader to ignore it.
|
|
As string `json:"as,omitempty"`
|
|
// Secret is the file on this machine holding that consumer's credential, sealed to it.
|
|
//
|
|
// Named rather than carried, for the same reason the private network's key is: the mesh
|
|
// discarded the value and could not put it here if it wanted to. What is here is where to
|
|
// find it.
|
|
Secret string `json:"secret,omitempty"`
|
|
// Values are the module's own, with settings applied. What the keys mean is agreed by the
|
|
// requirement's name — everything providing `reverse-proxy` understands the same shape, which
|
|
// is what makes swapping one for another cost nothing.
|
|
Values map[string]any `json:"values"`
|
|
}
|
|
|
|
// grantPath is where one consumer's sealed credential lands on the providing machine.
|
|
//
|
|
// Suffixed, so the directory can also hold whatever the module writing it keeps there and so a
|
|
// node named like something else in that directory cannot collide with it.
|
|
// One file per consumer, and a consumer is a module on a machine (novox/hq 04-ISSUES/022).
|
|
//
|
|
// Named after both. Named after the machine alone, two modules on one node wrote to one path: the
|
|
// second overwrote the first, and the provisioner — reading a directory — saw one consumer where
|
|
// there were two.
|
|
// reflectsRenamed is a resource's restart-on list under the module's prefix, or nil when it has
|
|
// none. It reads both the shape JSON parsing produces ([]any) and the shape code composing
|
|
// resources natively produces ([]string): a reference that was skipped because its list arrived
|
|
// in the other shape would point at nothing — silently, with the service never restarting and
|
|
// every check passing, which is how a runtime kept serving without the registry trust its
|
|
// daemon file already carried.
|
|
func reflectsRenamed(module string, reflects any) []any {
|
|
var names []string
|
|
switch v := reflects.(type) {
|
|
case []any:
|
|
for _, id := range v {
|
|
names = append(names, fmt.Sprint(id))
|
|
}
|
|
case []string:
|
|
names = v
|
|
}
|
|
if names == nil {
|
|
return nil
|
|
}
|
|
var renamed []any
|
|
for _, named := range names {
|
|
if strings.Contains(named, ".") {
|
|
renamed = append(renamed, named)
|
|
continue
|
|
}
|
|
renamed = append(renamed, module+"."+named)
|
|
}
|
|
return renamed
|
|
}
|
|
|
|
func grantPath(directory, consumer, module string) string {
|
|
return strings.TrimRight(directory, "/") + "/" + consumer + "." + module + ".secret"
|
|
}
|
|
|
|
// holderAs is a consumer's name at the provider with a local name after it, where it keeps several
|
|
// credentials for one provision (ADR 0094); the name alone otherwise.
|
|
func holderAs(as, local string) string {
|
|
if local == "" {
|
|
return as
|
|
}
|
|
return as + "_" + local
|
|
}
|
|
|
|
// contributions collects what every module in this set contributes, by requirement.
|
|
//
|
|
// Ordered by contributing module, because the result becomes a file on a machine and a file whose
|
|
// lines move about is a file that looks changed when nothing changed.
|
|
func (r Resolution) contributions(settings SettingsBy, grants []Grant,
|
|
directories map[string]string) (map[string][]Contribution, error) {
|
|
out := map[string][]Contribution{}
|
|
modules := append([]Manifest{}, r.Modules...)
|
|
sort.Slice(modules, func(i, j int) bool { return modules[i].Module < modules[j].Module })
|
|
|
|
// What consumers on other machines asked for. Merged in with this machine's own, because from
|
|
// the provider's side they are the same thing — somebody wanting something — and a provider
|
|
// that had to read two lists would be a provider that reads one of them.
|
|
sorted := append([]Grant{}, grants...)
|
|
sort.Slice(sorted, func(i, j int) bool {
|
|
if sorted[i].Provision != sorted[j].Provision {
|
|
return sorted[i].Provision < sorted[j].Provision
|
|
}
|
|
if sorted[i].Consumer != sorted[j].Consumer {
|
|
return sorted[i].Consumer < sorted[j].Consumer
|
|
}
|
|
if sorted[i].From != sorted[j].From {
|
|
return sorted[i].From < sorted[j].From
|
|
}
|
|
return sorted[i].Local < sorted[j].Local
|
|
})
|
|
// A consumer already carried by the grants loop, keyed (provision, module). When provider and
|
|
// consumer are co-located, `grantsFor` enumerates the same-node consumer too, so without this the
|
|
// module would be emitted a second time by the m.Contributes loop below — once full (with the
|
|
// grant's secret/as) and once partial — which is the duplicate seen in a co-located mesh.json.
|
|
granted := map[string]map[string]bool{}
|
|
for _, g := range sorted {
|
|
if g.From == "" {
|
|
// As above: nothing on that machine asks for this any more, so the provider is not
|
|
// told about it and withdraws the login on its next pass.
|
|
continue
|
|
}
|
|
out[g.Provision] = append(out[g.Provision], Contribution{
|
|
From: g.From, Node: g.Consumer, At: g.At, Values: g.Values,
|
|
// One holder per local name: the identity the consumer is known by, and the local name
|
|
// after it where the module keeps several (ADR 0094). Not a login any backend checks —
|
|
// a secret is not a login — so the identity limit does not apply to the suffix.
|
|
As: holderAs(ConsumerIdentity(g.Consumer, IdentitySource(g.Slug, g.From)), g.Local),
|
|
Secret: grantPath(directories[g.Provision], g.Consumer, holderAs(g.From, g.Local)),
|
|
})
|
|
if granted[g.Provision] == nil {
|
|
granted[g.Provision] = map[string]bool{}
|
|
}
|
|
granted[g.Provision][g.From] = true
|
|
}
|
|
for _, m := range modules {
|
|
for _, to := range sortedKeys(m.Contributes) {
|
|
// The grants loop already emitted this module's contribution to this provision, with its
|
|
// minted secret — emitting the partial copy again would duplicate it. A contribution with
|
|
// no grant (a reverse-proxy route names a host, not a credential) is not in `granted`, so
|
|
// it still reaches the provider from here.
|
|
if granted[to][m.Module] {
|
|
continue
|
|
}
|
|
// Settings reach a contribution the same way they reach a file. A route's hostname is
|
|
// exactly the kind of thing that differs between one mesh and the next, and a module
|
|
// that could not have it set would have to be edited to be reused.
|
|
values, err := settle(m.Contributes[to], settings[m.Module], nil,
|
|
m.Module+" contributing to "+to)
|
|
if err != nil {
|
|
return nil, fmt.Errorf("%s contributing to %s: %w", m.Module, to, err)
|
|
}
|
|
composeName(values, r.PublicDomain)
|
|
out[to] = append(out[to], Contribution{From: m.Module, Values: values})
|
|
}
|
|
// Several contributions to one requirement (ADR 0094's sibling for `contributes`): an
|
|
// object store's data API and its console are two different public names from one module,
|
|
// not one. Never in `granted` — a route names a host, not a credential — so every local
|
|
// name always reaches the provider from here.
|
|
for _, to := range sortedKeys(m.ContributesMany) {
|
|
for _, local := range sortedKeys(m.ContributesMany[to]) {
|
|
values, err := settle(m.ContributesMany[to][local], settings[m.Module], nil,
|
|
m.Module+" contributing "+local+" to "+to)
|
|
if err != nil {
|
|
return nil, fmt.Errorf("%s contributing %s to %s: %w", m.Module, local, to, err)
|
|
}
|
|
composeName(values, r.PublicDomain)
|
|
out[to] = append(out[to], Contribution{From: m.Module, Values: values})
|
|
}
|
|
}
|
|
}
|
|
return out, nil
|
|
}
|
|
|
|
// composeName joins a contribution's label with a node's public domain, in place (novox/hq ADR
|
|
// 0056).
|
|
//
|
|
// **The whole of what the mesh does with a route's name: join two given strings.** A contribution
|
|
// carries a `label` — the subdomain its operator chose — and the node carries its public domain;
|
|
// the granted name is `<label>.<public-domain>` and the mesh interprets neither half. It runs on
|
|
// any contribution carrying a label, not only a route's, because the mesh does not know what a
|
|
// provision means — a name it can compose from parts it was given is the point, whatever the
|
|
// provision is called.
|
|
//
|
|
// **Additive, so an unmigrated catalogue still works.** A contribution that already carries a full
|
|
// `name` and no `label` is left exactly as it is: the catalogue can migrate module by module while
|
|
// the running mesh keeps serving the full names it has. And a labelled contribution on a node with
|
|
// no public domain composes nothing — there is nothing to join it to — which reads downstream as a
|
|
// route that named no host, the same as it would have before this existed.
|
|
func composeName(values map[string]any, publicDomain string) {
|
|
if values == nil || publicDomain == "" {
|
|
return
|
|
}
|
|
if _, already := values["name"]; already {
|
|
// A full name was given rather than a label. Left as-is: this is the legacy shape, and the
|
|
// point of the label is to not have to write the full name — a contribution that wrote both
|
|
// has said what it wants and the mesh does not second-guess it.
|
|
return
|
|
}
|
|
label, ok := values["label"].(string)
|
|
if !ok || strings.TrimSpace(label) == "" {
|
|
return
|
|
}
|
|
if strings.TrimSpace(label) == "@" {
|
|
// The apex: a module served at the bare public domain, no subdomain — the zone-file
|
|
// convention `@`. Composes to the domain itself, so a node's own site is a label like any
|
|
// other rather than the one route that must still carry a full name.
|
|
values["name"] = publicDomain
|
|
return
|
|
}
|
|
values["name"] = strings.TrimSpace(label) + "." + publicDomain
|
|
}
|
|
|
|
// receivedFile is the file a provider is given its consumers' contributions in.
|
|
func receivedFile(requirement, path string, given []Contribution) (map[string]any, error) {
|
|
if given == nil {
|
|
// Nobody contributed. The file is still written, empty, rather than left absent: a
|
|
// provider that finds no file cannot tell "nothing asked for me" from "the mesh never
|
|
// wrote it", and the two want completely different responses.
|
|
given = []Contribution{}
|
|
}
|
|
// The note goes *inside* the document, not above it. The first version wrote a `//` header
|
|
// and produced a file that says "do not edit" to a person and fails to parse for the program
|
|
// meant to read it — which is the whole audience.
|
|
body, err := json.MarshalIndent(map[string]any{
|
|
"contributions": 1,
|
|
"requirement": requirement,
|
|
"generated": "by the mesh — do not edit; replaced whenever a module contributing to " +
|
|
requirement + " arrives or leaves",
|
|
"given": given,
|
|
}, "", " ")
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
return map[string]any{
|
|
"id": ReceivedID(requirement), "type": "file", "path": path, "mode": "0644",
|
|
"content": string(body) + "\n",
|
|
}, nil
|
|
}
|
|
|
|
// Kept is one secret as the operator can recover it: where it belongs, and the value sealed to the
|
|
// operator's key. Never a node's blob, and never a value.
|
|
//
|
|
// Two kinds, addressed the same way — by the node and module that hold it and the name they know
|
|
// it by. An `own` secret is one a module has for itself; a `pair` secret is a credential the
|
|
// module was granted for a provision it requires (`name` is the provision), and Provider says which
|
|
// node grants it.
|
|
type Kept struct {
|
|
Node string `json:"node"`
|
|
Module string `json:"module"`
|
|
Name string `json:"name"`
|
|
// Kind is `own` or `pair`.
|
|
Kind string `json:"kind"`
|
|
// Provider is the node granting a pair credential; empty for an own secret.
|
|
Provider string `json:"provider,omitempty"`
|
|
// Origin is `made` or `accepted` — whether the mesh minted it or a person supplied it.
|
|
Origin string `json:"origin"`
|
|
// Sealed is the value, sealed to the operator key named in Key.
|
|
Sealed string `json:"sealed"`
|
|
Key string `json:"key"`
|
|
MadeAt time.Time `json:"made-at"`
|
|
// Local is the credential's name inside the consumer where it holds several (ADR 0094).
|
|
Local string `json:"local,omitempty"`
|
|
}
|
|
|
|
// KeptExport is what a person keeps beside the operator key, and what a vault keeps on its disk:
|
|
// every operator-sealed copy, and the honest list of what has none.
|
|
type KeptExport struct {
|
|
Export int `json:"export"`
|
|
OperatorKey string `json:"operator-key"`
|
|
Fingerprint string `json:"fingerprint"`
|
|
// Kept is sealed to OperatorKey. EarlierKey is sealed to a key the mesh has since replaced —
|
|
// recoverable with that key, if the person still has it, and with nothing else. Unrecoverable
|
|
// has no operator copy at all.
|
|
Kept []Kept `json:"kept"`
|
|
EarlierKey []Kept `json:"sealed-to-earlier-key,omitempty"`
|
|
Unrecoverable []Kept `json:"unrecoverable,omitempty"`
|
|
}
|
|
|
|
// KeptID is the resource that carries the export to a module that keeps it.
|
|
func KeptID() string { return "kept" }
|
|
|
|
// keptFile is the export, written where the module said. Ciphertext throughout — see Keeps.
|
|
func keptFile(dir string, kept *KeptExport) (map[string]any, error) {
|
|
body, err := json.MarshalIndent(kept, "", " ")
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
return map[string]any{
|
|
"id": KeptID(), "type": "file", "path": strings.TrimRight(dir, "/") + "/export.json",
|
|
"mode": "0600", "content": string(body) + "\n",
|
|
}, nil
|
|
}
|
|
|
|
// ownedBy gives a secret file the owner the module named, when it named one (Manifest.SecretsOwner).
|
|
func ownedBy(owner string, file map[string]any) map[string]any {
|
|
if owner != "" {
|
|
file["owner"] = owner
|
|
}
|
|
return file
|
|
}
|
|
|
|
// sortedKeys is map iteration made repeatable, which everything written to a machine needs.
|
|
func sortedKeys[V any](m map[string]V) []string {
|
|
out := make([]string, 0, len(m))
|
|
for k := range m {
|
|
out = append(out, k)
|
|
}
|
|
sort.Strings(out)
|
|
return out
|
|
}
|
|
|
|
// boundFile is what a module is told about something it requires from another machine.
|
|
//
|
|
// Where it is and what the providing module said about using it. **No credential**, and the file
|
|
// says so rather than leaving a reader to wonder whether one was meant to be there — a missing
|
|
// field looks like a bug, and a stated absence looks like a boundary.
|
|
func boundFile(n Needed, path, as string) (map[string]any, error) {
|
|
// A record has no machine and no address. Saying so is the difference between a reader
|
|
// concluding "somewhere with no address" and concluding the mesh failed to fill something in.
|
|
where := any(n.At)
|
|
if n.ByRecord {
|
|
where = "a record in this mesh, not a machine"
|
|
}
|
|
body, err := json.MarshalIndent(map[string]any{
|
|
"binding": 1,
|
|
"provision": n.Name,
|
|
"from": n.From,
|
|
"at": where,
|
|
// Who this module is at the other end. **The half that was missing** (novox/hq
|
|
// 04-ISSUES/023): a consumer was told the address, the port and where its password is,
|
|
// and not the name it must present — which the provisioner had invented.
|
|
"as": as,
|
|
"serves": n.Serves,
|
|
// **Where the credential is, not what it is.** It stopped being true that the mesh
|
|
// cannot issue one when 021 was fixed, and a comment asserting a fact about the mesh that
|
|
// has become false is worse than none — somebody reads it and stops looking.
|
|
"generated": "by the mesh — do not edit; replaced whenever this changes. " +
|
|
"The credential is not here: it is sealed, in the file this module's manifest " +
|
|
"names under `secrets`",
|
|
}, "", " ")
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
return map[string]any{
|
|
"id": BoundID(n.Name), "type": "file", "path": path, "mode": "0644",
|
|
"content": string(body) + "\n",
|
|
}, nil
|
|
}
|
|
|
|
// ContributionsFrom is what one module on this node asked of one requirement, settled.
|
|
//
|
|
// Exported because a provider's grants are assembled from its consumers' resolutions, one machine
|
|
// at a time, and the alternative was for the control plane to reimplement settling.
|
|
//
|
|
// **One module, not one machine** (novox/hq 04-ISSUES/022). This used to take a requirement alone
|
|
// and refuse whenever two modules wanted it — correctly, given what it had: the credential was
|
|
// keyed by node, so the two would have shared one, and sharing is worse than refusing. But the
|
|
// arrangement refused is the ordinary one. A node running eight services against one database is
|
|
// not an edge case; it is what a machine looks like. Now each consumer has its own credential and
|
|
// there is nothing left to refuse.
|
|
func (r Resolution) ContributionsFrom(requirement, module string, settings SettingsBy) (
|
|
map[string]any, bool, error) {
|
|
all, err := r.contributions(settings, nil, nil)
|
|
if err != nil {
|
|
return nil, false, err
|
|
}
|
|
for _, g := range all[requirement] {
|
|
if g.From == module {
|
|
return g.Values, true, nil
|
|
}
|
|
}
|
|
// It contributes no payload — but a require-only consumer of a parameterless provision (one whose
|
|
// `serves` names no consumer-supplied key: `redis-cache`, `amqp`) still ASKS for it and must be
|
|
// granted a credential. Keying "asks" on contributions alone marked those grants withdrawn
|
|
// (From=""), so the provider never created the account and the consumer authenticated nowhere. A
|
|
// module asks iff it still requires the provision, whether or not it hands anything up with it.
|
|
for _, m := range r.Modules {
|
|
if m.Module != module {
|
|
continue
|
|
}
|
|
for _, req := range m.Requires {
|
|
if req == requirement {
|
|
return map[string]any{}, true, nil
|
|
}
|
|
}
|
|
}
|
|
// Nothing on that machine asks for this any more. Said as "not found" rather than as an
|
|
// error: it is how a credential is withdrawn, and the provider removes what nobody asks for.
|
|
return nil, false, nil
|
|
}
|
|
|
|
// here is the module on this same machine that answers a requirement, as a binding.
|
|
//
|
|
// **Only when the provider said something a consumer must know.** That is the line: the original
|
|
// reasoning — a file saying "it is on this node" is a fact nobody needs — is right about the
|
|
// location and wrong about everything beside it. A shell is answered here and there is nothing to
|
|
// say about it. A registry is answered here and the consumer still cannot guess the port.
|
|
//
|
|
// The address is this machine's own name on the private network when it has one, and loopback
|
|
// when it does not — a machine not on the network still reaches itself, and naming it by a name
|
|
// nothing resolves would be worse than naming it by an address that always works.
|
|
func here(r Resolution, requirement string, with Rendering) (*Needed, error) {
|
|
serves, answered, err := r.servedOnThisMachine(requirement, with)
|
|
if err != nil || !answered {
|
|
return nil, err
|
|
}
|
|
at := r.At
|
|
if at == "" {
|
|
at = "127.0.0.1"
|
|
}
|
|
return &Needed{Name: requirement, From: r.Node, At: at, Serves: serves}, nil
|
|
}
|
|
|
|
// servedOnThisMachine is what a provider in this node's own set says a consumer must know —
|
|
// derived exactly as the mesh derives it for a provider on any OTHER machine.
|
|
//
|
|
// **The one derivation, so the two arrangements cannot disagree.** For a provider elsewhere the
|
|
// control plane walks that node, reads its manifest with that machine's port assignments, and
|
|
// settles the result with that node's settings layers before offering it (cmd/mesh-controller plan.go,
|
|
// theRestOfTheMesh). A provider on the consumer's own machine never passes through that walk, so
|
|
// every step of it has to be repeated here — and each step that was not repeated was a promise the
|
|
// co-located arrangement quietly broke: first the port (novox/hq 04-ISSUES/038), then the settled
|
|
// values (ADR 0066, an internal CA's root arriving empty).
|
|
//
|
|
// Here rather than in the resolver, because here is the first moment both halves exist: resolving
|
|
// runs before a machine's ports are assigned and knows nothing of settings.
|
|
//
|
|
// The first module in the resolved order that says it serves the provision answers, which is the
|
|
// same choice here() has always made and is stable — providersFirst orders the slice, and nothing
|
|
// downstream depends on a map's iteration.
|
|
func (r Resolution) servedOnThisMachine(provision string, with Rendering) (map[string]any, bool, error) {
|
|
for _, m := range r.Modules {
|
|
if _, said := m.Serves[provision]; !said {
|
|
continue
|
|
}
|
|
serves := ServedOn(m, provision, with.Ports[m.Module])
|
|
if len(serves) == 0 {
|
|
// It named the provision and said nothing about it, which is the same as not answering:
|
|
// there is no fact for a consumer to be given. Another module in the set may still have
|
|
// one, so keep looking rather than concluding from the first.
|
|
continue
|
|
}
|
|
settled, err := Settle(serves, with.Settings[m.Module])
|
|
if err != nil {
|
|
return nil, false, fmt.Errorf("%s serving %s: %w", m.Module, provision, err)
|
|
}
|
|
return settled, true, nil
|
|
}
|
|
return nil, false, nil
|
|
}
|
|
|
|
// withMeshNames gives every container in a set the mesh's names.
|
|
//
|
|
// Copied rather than edited in place: these maps come from a module's manifest, and mutating one
|
|
// would change what the catalogue holds for every other machine running that module.
|
|
//
|
|
// A host-network container gets the names too. It was once skipped, on the belief that it "shares
|
|
// the machine's hosts file already" — but it does not: `docker run --network host` still gives the
|
|
// container its own /etc/hosts (localhost and its own id only), so every `<node>.internal` name the
|
|
// mesh wrote for the machine is invisible inside it, and a client that dials one gets EAI_AGAIN. The
|
|
// remedy is the same `--add-host` every other container gets — the runtime accepts it with
|
|
// `--network host` (verified), and without it a host-network consumer cannot reach a provider by the
|
|
// `.internal` address the mesh hands it as `${bound:...:at}`.
|
|
func withMeshNames(resources []map[string]any, names map[string]string) []map[string]any {
|
|
out := make([]map[string]any, 0, len(resources))
|
|
for _, r := range resources {
|
|
if r["type"] != "container" {
|
|
out = append(out, r)
|
|
continue
|
|
}
|
|
|
|
copied := map[string]any{}
|
|
for k, v := range r {
|
|
copied[k] = v
|
|
}
|
|
var given []any
|
|
if already, ok := copied["hosts"].([]any); ok {
|
|
given = append(given, already...)
|
|
}
|
|
for _, name := range sortedKeys(names) {
|
|
given = append(given, name+":"+names[name])
|
|
}
|
|
copied["hosts"] = given
|
|
out = append(out, copied)
|
|
}
|
|
return out
|
|
}
|
|
|
|
// pinned refuses an image that is not really pinned, on its way to a machine.
|
|
//
|
|
// **Here and not at parse** (novox/hq 04-ISSUES/025). A manifest in a repository names artifacts
|
|
// and the manifest the mesh holds names digests — they are deliberately not the same document, so
|
|
// a file awaiting a pin is legitimate exactly as the bundle's is. What must never happen is a
|
|
// placeholder reaching a machine, and this is the last moment before one does.
|
|
//
|
|
// The host checks only the *shape* of a reference, and cannot do more: verifying a digest exists
|
|
// means reaching a registry, which is the one thing a host must never have to do. So sixty-four
|
|
// zeros satisfies every gate in the system and stops on the machine at `docker pull` — which is
|
|
// how eighteen of them shipped across five modules that parse, resolve and compose cleanly.
|
|
func pinned(resource map[string]any, module string) error {
|
|
image, ok := resource["image"].(string)
|
|
if !ok {
|
|
return nil
|
|
}
|
|
_, digest, found := strings.Cut(image, "@")
|
|
if !found || strings.Trim(strings.TrimPrefix(digest, "sha256:"), "0") != "" {
|
|
return nil
|
|
}
|
|
return fmt.Errorf(
|
|
"%s would send %v to a machine pinned to a placeholder digest, which is never a real "+
|
|
"image — it would be fetched and fail there. Resolve the tag to a digest first",
|
|
module, resource["id"])
|
|
}
|
|
|
|
// built refuses a container that still names an artifact nobody has made.
|
|
//
|
|
// **Said here, by the thing that knows what "artifact" means.** A container naming an artifact is
|
|
// a module saying "the mesh builds this"; the field is resolved into an image when a build
|
|
// publishes one, and until then there is nothing to run. A host receiving it refuses the whole
|
|
// declaration — correctly, since its language has no such field — but what it can say is that a
|
|
// container does not use "artifact", which tells a reader the manifest is malformed. It is not:
|
|
// it is unbuilt, which is a different problem with a different fix, and only the mesh is in a
|
|
// position to tell them apart (novox/hq ADR 0073).
|
|
func built(resource map[string]any, module string) error {
|
|
if fmt.Sprint(resource["type"]) != "container" {
|
|
return nil
|
|
}
|
|
artifact, ok := resource["artifact"].(string)
|
|
if !ok || artifact == "" {
|
|
return nil
|
|
}
|
|
return fmt.Errorf(
|
|
"%s has not been built for this mesh: %v is its %q artifact, and no build has published "+
|
|
"one. Build it — `build <repository> --path <path>` — and the module will name what "+
|
|
"came out instead",
|
|
module, resource["id"], artifact)
|
|
}
|
|
|
|
// publishedOn puts the machine's own port on the outside of a container's mapping.
|
|
//
|
|
// **A module writes the port the software uses; the mesh says where the machine puts it**
|
|
// (novox/hq ADR 0038). So `"5432"` means *publish what the software calls 5432*, and this fills in
|
|
// the half only the mesh can know.
|
|
//
|
|
// A mapping written the long way — `"5433:5432"` — is left exactly as it is. Some things genuinely
|
|
// must be pinned down by hand, and quietly overruling somebody who wrote both halves would be
|
|
// worse than not offering the short form at all.
|
|
func publishedOn(resource map[string]any, module string, with Rendering) {
|
|
if fmt.Sprint(resource["type"]) != "container" {
|
|
return
|
|
}
|
|
listed, ok := resource["ports"].([]any)
|
|
if !ok {
|
|
return
|
|
}
|
|
out := make([]any, 0, len(listed))
|
|
for _, entry := range listed {
|
|
written := fmt.Sprint(entry)
|
|
if strings.Contains(written, ":") {
|
|
// Written the long way, and left alone — unless this node was given a machine port for
|
|
// it (novox/hq ADR 0100): the foundation's ports are the node's, and a manifest's
|
|
// number is only the default. The outer port only; an address and the software's
|
|
// port stay as written.
|
|
out = append(out, givenOuter(written, with.Given[module]))
|
|
continue
|
|
}
|
|
wanted, err := strconv.Atoi(strings.TrimSpace(written))
|
|
if err != nil {
|
|
// Not a port at all. Passed through, so the host refuses it with its own words rather
|
|
// than this quietly dropping something somebody meant.
|
|
out = append(out, written)
|
|
continue
|
|
}
|
|
out = append(out, fmt.Sprintf("%d:%d", with.machinePort(module, wanted), wanted))
|
|
}
|
|
resource["ports"] = out
|
|
}
|
|
|
|
// givenOuter rewrites the machine side of a long-form mapping to the port this node was given for
|
|
// it, when it was given one.
|
|
//
|
|
// **Under either of the mapping's names.** A node gives a port by the number the module names it
|
|
// by, and a module publishing `"2222:22"` may say it listens on 22 or on 2222 — GivenPorts accepts
|
|
// both and answers to both, so looking the machine side up first and the container's port second
|
|
// finds the same number either way. Read only by the container's port, this moved nothing for the
|
|
// module that declares the machine side, and the setting was refused before it got here.
|
|
func givenOuter(written string, given map[int]int) string {
|
|
if len(given) == 0 {
|
|
return written
|
|
}
|
|
mapping, protocol := written, ""
|
|
if cut := strings.LastIndex(written, "/"); cut >= 0 {
|
|
mapping, protocol = written[:cut], written[cut:]
|
|
}
|
|
parts := strings.Split(mapping, ":")
|
|
if len(parts) < 2 {
|
|
return written
|
|
}
|
|
inner, err := strconv.Atoi(strings.TrimSpace(parts[len(parts)-1]))
|
|
if err != nil {
|
|
return written
|
|
}
|
|
at, ok := given[inner]
|
|
if outer, err := strconv.Atoi(strings.TrimSpace(parts[len(parts)-2])); err == nil {
|
|
if machine, named := given[outer]; named {
|
|
at, ok = machine, true
|
|
}
|
|
}
|
|
if !ok {
|
|
return written
|
|
}
|
|
parts[len(parts)-2] = strconv.Itoa(at)
|
|
return strings.Join(parts, ":") + protocol
|
|
}
|
|
|
|
// ServedOn is what a provider tells a consumer, with the port that machine actually uses.
|
|
//
|
|
// **The module writes the port once, in `listens`** (novox/hq ADR 0038). It used to write it three
|
|
// times — for the rule set, for what a consumer is told, and for what the runtime publishes — and
|
|
// nothing checked that the three agreed. This is what fills the second in.
|
|
//
|
|
// The assignment wins when there is one, and the declared port stands in when there is not: a
|
|
// caller composing without a store still gets something coherent, and a mesh that has assigned one
|
|
// tells the truth about where it put it.
|
|
//
|
|
// Only when the module offers exactly one port. A module offering several has not said which
|
|
// belongs to which provision, and guessing would give a consumer a port that answers something
|
|
// else — so it keeps whatever the manifest said, which may be nothing.
|
|
func ServedOn(m Manifest, provision string, ports map[int]int) map[string]any {
|
|
serves := m.Serves[provision]
|
|
|
|
out := make(map[string]any, len(serves)+1)
|
|
for k, v := range serves {
|
|
out[k] = v
|
|
}
|
|
if _, said := out["port"]; said {
|
|
// Written by hand. Redirected to wherever the machine put it, and otherwise left alone.
|
|
if number, ok := asPort(out["port"]); ok {
|
|
if at, known := ports[number]; known {
|
|
out["port"] = at
|
|
}
|
|
}
|
|
return out
|
|
}
|
|
if len(m.Listens) != 1 {
|
|
return out
|
|
}
|
|
wanted := m.Listens[0].Port
|
|
if at, known := ports[wanted]; known {
|
|
out["port"] = at
|
|
} else {
|
|
out["port"] = wanted
|
|
}
|
|
return out
|
|
}
|
|
|
|
func asPort(v any) (int, bool) {
|
|
switch n := v.(type) {
|
|
case int:
|
|
return n, true
|
|
case float64:
|
|
return int(n), true
|
|
}
|
|
return 0, false
|
|
}
|
|
|
|
// atMachinePort redirects a `port` written by a module on this machine to where this machine
|
|
// actually published it (novox/hq 04-ISSUES/038, ADR 0066).
|
|
//
|
|
// **A module writes the port its software uses; only the mesh knows where the machine put it.** A
|
|
// bare `ports` mapping is assigned a host port when the declaration is composed, which is after
|
|
// everything a module wrote has been read — so any number carried out of a manifest is the declared
|
|
// one until it passes through here. Both directions need it: what a co-located consumer is TOLD
|
|
// about a provider, and what a co-located workload TELLS a provider about itself.
|
|
//
|
|
// Named by the module the port belongs to, which is not always the one being talked about: a route
|
|
// contribution's port is the contributing workload's, never the proxy's.
|
|
//
|
|
// A fresh map when it changes anything, so the manifest-derived value handed back by the resolver —
|
|
// or by a grant gathered elsewhere — is never mutated under a caller that still holds it. Keyed by
|
|
// the *declared* port, so applying it a second time is a no-op: once redirected the value is the
|
|
// machine port, which is not itself a key.
|
|
func atMachinePort(serves map[string]any, module string, ports map[string]map[int]int) map[string]any {
|
|
number, ok := asPort(serves["port"])
|
|
if !ok {
|
|
return serves
|
|
}
|
|
at, known := ports[module][number]
|
|
if !known || at == number {
|
|
return serves
|
|
}
|
|
copied := make(map[string]any, len(serves))
|
|
for k, v := range serves {
|
|
copied[k] = v
|
|
}
|
|
copied["port"] = at
|
|
return copied
|
|
}
|
|
|
|
// AtPublishedPort redirects a contribution's port to where the CONSUMER's own machine published it.
|
|
//
|
|
// The same redirection as [atMachinePort], for the one caller that cannot reach it. A grant is
|
|
// assembled on the providing machine out of a consumer that lives on another one, so that
|
|
// consumer's assignments are fetched there and handed in, rather than looked for here where they
|
|
// are not. Without it a proxy told to reach a workload on another machine is told the port the
|
|
// workload's *software* uses, and dials a number that machine never published — the same fault as
|
|
// novox/hq 04-ISSUES/038, one node over.
|
|
func AtPublishedPort(values map[string]any, module string, published map[int]int) map[string]any {
|
|
return atMachinePort(values, module, map[string]map[int]int{module: published})
|
|
}
|