jschoubben
8fa5443862
ContributionsFrom settled to whichever of a module's several contributions to one requirement sorted first, arbitrarily — the grant minted for it then carried that contribution's label and port under a credential the OTHER contribution's consumer never sees, and collided with that same contribution's own entry from contributions() besides. Confirmed live: minio's two route contributions (files-api, files) produced three entries in route-adapter's received file — files-api twice, once credentialed and once not, files not credentialed at all. Every single-contribution module (gitea, keycloak, umami) already mints an unused credential for `route` too — route never needs one, by its own documentation — but with exactly one contribution to match there was nothing to collide with, so it never surfaced. Where a module contributes more than once, there is no single value to settle on. The module still asks, still gets its one credential — a pair credential is not a place for a label or a port anyway — and each named contribution reaches the provider on its own, unchanged. No cleanup needed for the secret already minted live for minio+route: the sealed blob is a random pair credential unrelated to Values, which is recomputed fresh on every plan/push regardless.
1490 lines
66 KiB
Go
1490 lines
66 KiB
Go
package catalogue
|
|
|
|
// Turning a resolution into the declaration a node is sent.
|
|
//
|
|
// Separate from resolving because they answer different questions. Resolving asks *what should
|
|
// this machine run*; this asks *what does that look like as resources*, and the second is where
|
|
// settings are applied, generators are called, contributions are collected and credentials are
|
|
// placed. Both lived in one file until it was doing four jobs at once — which is the shape the
|
|
// system this replaces failed in, one import at a time.
|
|
|
|
import (
|
|
"encoding/json"
|
|
"fmt"
|
|
"sort"
|
|
"strconv"
|
|
"strings"
|
|
"time"
|
|
)
|
|
|
|
// SettingsBy is the layers that apply to each module, keyed by module name.
|
|
type SettingsBy map[string][]Layer
|
|
|
|
// Generator works out a module's resources for one node, where they cannot be written in advance.
|
|
type Generator interface {
|
|
// Resources for this node. Absent means the node is not part of whatever this generates,
|
|
// which is an ordinary answer rather than a failure — a machine assigned the module before it
|
|
// has an address on the network is in exactly that state.
|
|
Resources(node string) ([]map[string]any, bool, error)
|
|
}
|
|
|
|
// OpensPorts is a generator that also says what its resources accept connections on.
|
|
//
|
|
// **Separate from Generator, because most generators have nothing to say here** and requiring an
|
|
// empty method of each would be a cost paid everywhere for one caller.
|
|
//
|
|
// It exists because a static field cannot express this. A hub accepts connections from every node
|
|
// at other sites; a machine that is not a hub dials out and needs nothing open — and they are the
|
|
// same module. `listens` in a manifest is one answer for every machine that runs it, so the
|
|
// machine that most needs filtering, the one facing the public internet, was the one whose rule
|
|
// set would have closed its own overlay.
|
|
type OpensPorts interface {
|
|
// Listens is what this node accepts on because of what was computed for it. Nothing is the
|
|
// ordinary answer: most machines running a computed module open no port at all.
|
|
Listens(node string) ([]Listening, error)
|
|
}
|
|
|
|
// Grant is one consumer's credential, on the machine that must create it.
|
|
type Grant struct {
|
|
// Provision is what was required.
|
|
Provision string
|
|
// Consumer is the node that will use it.
|
|
Consumer string
|
|
// From is the module on that machine which asked.
|
|
//
|
|
// **Part of who this credential is for, not a label** (novox/hq 04-ISSUES/022). Together with
|
|
// Consumer it names one consumer; the node alone does not, because a machine routinely runs
|
|
// several modules wanting the same thing. It is also what the provider names the role or the
|
|
// bucket or the client after, so withdrawing one consumer does not take another's away.
|
|
From string
|
|
// Values are what that module contributed — the name it wants, and anything else the
|
|
// provision's own vocabulary defines.
|
|
Values map[string]any
|
|
// Local is the name the credential goes by inside the consumer where it keeps several for one
|
|
// provision (ADR 0094); empty for the ordinary one. The provider sees it as a holder of its own.
|
|
Local string
|
|
// Slug is the consumer module's identity slug, if it declared one — carried on the grant so the
|
|
// provider side derives the same login the consumer does, even across nodes where the consumer's
|
|
// manifest is not in view (novox/hq ADR 0049). Empty means "use the module name".
|
|
Slug string
|
|
// At is where the consuming machine is on the private network, empty if it is not on one.
|
|
//
|
|
// Passed in with the grant because it is a fact about another machine, and resolution answers
|
|
// questions about one. A provider that must reach back to its consumer — a reverse proxy is
|
|
// the whole reason this exists — otherwise has to know how the mesh names machines.
|
|
At string
|
|
// Sealed is the credential, closed to the providing node.
|
|
Sealed string
|
|
}
|
|
|
|
// Rendering is everything needed to turn a resolution into the declaration a node is sent.
|
|
type Rendering struct {
|
|
// Certificate is what the mesh issued for this machine's internal name, and the mesh's own
|
|
// certificate. Both public — the key they belong to never left the machine.
|
|
Certificate string
|
|
Authority string
|
|
|
|
// Needed is each module's own secrets, sealed to this node, keyed by module and then by the
|
|
// name the module gave it.
|
|
Needed map[string]map[string]string
|
|
|
|
// Mesh is every node's address on the private network, which is what a rule saying "from the
|
|
// mesh" resolves to. Passed in for the same reason grants are: who else is on the network is
|
|
// a fact about the mesh, and resolution answers questions about one machine.
|
|
Mesh []string
|
|
// Suffix is what a machine's internal name ends in, as the control plane composed Names —
|
|
// `internal` unless the operator chose another — so a fact writing those names does not
|
|
// compose it a second time.
|
|
Suffix string
|
|
|
|
// Kept is every operator-sealed secret in the mesh, for a module that `keeps` them. Nil when
|
|
// nothing on this node keeps them, or the mesh has no operator key.
|
|
Kept *KeptExport
|
|
|
|
// Foundation is the ports the mesh itself needs reachable on every machine, which no module
|
|
// declares because the foundation is not a module (novox/hq 04-ISSUES/051 and 052). The broker
|
|
// is the one that matters: a machine dials it to enrol, and a firewall derived only from
|
|
// modules closes it.
|
|
Foundation []int
|
|
|
|
// Names is every machine's internal name and its address, for containers to be given.
|
|
//
|
|
// **A container does not inherit the machine's names**, so every internal name the mesh wrote
|
|
// is invisible to what the machine runs. Given here rather than looked up on the machine,
|
|
// because which machines exist is a fact about the mesh.
|
|
Names map[string]string
|
|
|
|
// Machines is only the machines, by the same internal name — the subset of Names that is a
|
|
// node of this mesh rather than a name it was told to serve. Both matter and they are not the
|
|
// same set: a container's hosts wants every name, so a routed name resolves to the proxy that
|
|
// serves it, while a resolver told the mesh's suffix is authoritative for it answers from what
|
|
// it is given and forwards nothing — so a routed name written there is a name nobody asks for,
|
|
// standing beside the machines and looking as real as they do.
|
|
Machines map[string]string
|
|
|
|
Settings SettingsBy
|
|
Generators map[string]Generator
|
|
// Grants are the credentials this node must create, for the provisions it offers. Passed in
|
|
// rather than resolved, because who consumes a node is a fact about the rest of the mesh and
|
|
// resolution answers questions about one machine.
|
|
Grants []Grant
|
|
|
|
// Ports is where this machine puts what each module needs reachable, by module and by the
|
|
// port the software itself uses (novox/hq ADR 0038).
|
|
//
|
|
// **The one place the number now lives.** A module used to write it three times — for the rule
|
|
// set, for what a consumer is told, and for what the runtime publishes — and nothing checked
|
|
// that the three agreed. They are all derived from this.
|
|
Ports map[string]map[int]int
|
|
|
|
// Adopted says the node is adopted (novox/hq ADR 0100): the firewall found on it stays in
|
|
// force, so no module that loads a filter is declared there, and what the mesh needs
|
|
// reachable is declared as openings, with its own ports guarded by a table that only refuses.
|
|
Adopted bool
|
|
|
|
// Given is the machine ports this node was given for its modules' ports, by module and by the
|
|
// port the software uses (novox/hq ADR 0100) — the foundation's ports, as genesis chose them.
|
|
// They win over anything the mesh would assign and over a manifest's own long-form mapping.
|
|
Given map[string]map[int]int
|
|
|
|
// Taken is the modules taken on this adopted node (novox/hq ADR 0100). The guard is derived
|
|
// from these only (ADR 0103): a port of a module assigned but not taken may still be the
|
|
// predecessor's.
|
|
Taken map[string]bool
|
|
|
|
// Seats is where this machine put each mesh-scoped seat's holder, by seat and by the port the
|
|
// holder's software uses (novox/hq 04-ISSUES/102) — read from the node's settings and
|
|
// assignments for whichever module claims the seat, whether or not it is in this node's set.
|
|
// What ${seat:…} answers with; see seat_into.go for why the answer may be absent.
|
|
Seats map[string]map[int]int
|
|
|
|
// ArtifactStore is the mesh's artifact store as this network reaches it (host:port) — the
|
|
// node holding it and the port that node put it on — or empty when the mesh has none on its
|
|
// network yet. Composed into every image and archive the mesh built, at this moment and never
|
|
// stored (novox/hq 04-ISSUES/102).
|
|
ArtifactStore string
|
|
|
|
// Built is every `<module>/<artifact>` the mesh has built. What tells a reference recorded
|
|
// with an address — before references were kept without one — from an image a module runs
|
|
// straight from a public registry.
|
|
Built map[string]bool
|
|
}
|
|
|
|
// machinePort is where a module's port lives on this machine, or the port itself when the mesh has
|
|
// not been asked. Unassigned is not an error here: a module with no `listens` never needed one,
|
|
// and a caller composing a declaration without a store still gets something coherent.
|
|
func (r Rendering) machinePort(module string, wanted int) int {
|
|
if at, given := r.Given[module][wanted]; given {
|
|
return at
|
|
}
|
|
if at, known := r.Ports[module][wanted]; known {
|
|
return at
|
|
}
|
|
return wanted
|
|
}
|
|
|
|
// Declaration is everything the resolved modules put on the node, with settings applied.
|
|
//
|
|
// Resource identities are prefixed with the module they came from. Two modules may reasonably
|
|
// both call something "config", and without this the second would silently replace the first —
|
|
// the node applying one of them and reporting success.
|
|
func (r Resolution) Declaration(with Rendering) ([]map[string]any, error) {
|
|
composed, err := r.Compose(with)
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
return composed.Resources, nil
|
|
}
|
|
|
|
// Composed is a declaration's resources and which module each came from.
|
|
//
|
|
// Owner is kept beside the resources because a resource id cannot be split back into its module:
|
|
// a module's name may itself contain a dot. What the mesh adds of its own — an opening, the guard —
|
|
// has no owner.
|
|
type Composed struct {
|
|
Resources []map[string]any
|
|
Owner map[string]string
|
|
}
|
|
|
|
// Compose is Declaration with the owner of every resource said.
|
|
func (r Resolution) Compose(with Rendering) (Composed, error) {
|
|
owner := map[string]string{}
|
|
resources, err := r.compose(with, owner)
|
|
if err != nil {
|
|
return Composed{}, err
|
|
}
|
|
return Composed{Resources: resources, Owner: owner}, nil
|
|
}
|
|
|
|
func (r Resolution) compose(with Rendering, owner map[string]string) ([]map[string]any, error) {
|
|
// Where each provision's credentials land, so a contribution can name the file rather than
|
|
// carry a value the mesh does not have.
|
|
directories := map[string]string{}
|
|
for _, m := range r.Modules {
|
|
for provision, where := range m.Grants {
|
|
directories[provision] = where
|
|
}
|
|
}
|
|
given, err := r.contributions(with.Settings, with.Grants, directories)
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
|
|
// A workload on THIS machine contributes the port it declared, and the machine may have
|
|
// published it somewhere else (novox/hq ADR 0066). The mirror of the redirect below: 038 fixed
|
|
// what a consumer is TOLD about a provider, and this is what a workload TELLS the provider about
|
|
// itself — gitea declaring 3000, published as 20000:3000, and the co-located proxy dialling 3000,
|
|
// where nothing listens, for every request.
|
|
//
|
|
// **The contributing module's assignment, never the provider's.** The port belongs to the
|
|
// workload; using the map of whatever answers the requirement would move a route to wherever the
|
|
// proxy happens to be published. A contribution carried here from another machine is left exactly
|
|
// as it is — its port is that machine's to assign, and this map knows nothing about it.
|
|
for provision := range given {
|
|
for i := range given[provision] {
|
|
said := &given[provision][i]
|
|
if said.Node != "" && said.Node != r.Node {
|
|
continue
|
|
}
|
|
said.Values = atMachinePort(said.Values, said.From, with.Ports)
|
|
}
|
|
}
|
|
|
|
// A provider answered on this same machine never passed through the walk that works out what a
|
|
// provider on ANOTHER machine serves (novox/hq 04-ISSUES/038, ADR 0066). That walk does two
|
|
// things — it derives the served facts with the provider node's port assignments, and it settles
|
|
// them with that node's settings layers — and the same-node paths (resolve.go's servedHere, and
|
|
// here() below) did neither, because both run while resolving, before either is known.
|
|
//
|
|
// The port was the first half to be noticed (038). The second half is worse and quieter: a served
|
|
// value the OPERATOR supplied — a certificate authority's root, which is the one kind of value a
|
|
// manifest cannot carry, because it differs on every mesh — arrived as the manifest's empty
|
|
// default. A proxy then wrote an empty CA bundle, fell back to the system trust store, could not
|
|
// verify the internal authority, and stopped issuing with nothing saying why.
|
|
//
|
|
// So it is re-derived here, in full, exactly as the cross-node path derives it. **After the
|
|
// closure is known**, which also removes an order dependence: the resolver built these needs
|
|
// mid-walk from whichever modules had been chosen by then, so what a co-located binding carried
|
|
// depended on the order somebody happened to assign things in.
|
|
for i := range r.Needs {
|
|
if r.Needs[i].ByRecord || r.Needs[i].From != r.Node {
|
|
continue
|
|
}
|
|
serves, answered, err := r.servedOnThisMachine(r.Needs[i].Name, with)
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
if answered {
|
|
r.Needs[i].Serves = serves
|
|
}
|
|
}
|
|
|
|
// Once, from every module's listens -- not per module. A module receiving only its own ports
|
|
// would write a rule set that closed every other module on the machine. Each module's per-node
|
|
// exposure settings override its listens' source first (novox/hq ADR 0046).
|
|
rules, err := r.Rules(with)
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
filtering := AsNftables(rules, with.Mesh, r.PublicDomain != "", with.Foundation)
|
|
|
|
var out []map[string]any
|
|
for _, m := range r.Modules {
|
|
if with.Adopted && m.Filtering != nil {
|
|
// Nothing of a module that loads a filter, on an adopted node: its table would drop
|
|
// by default and hold accepts, and the found firewall stays in force. Every resource,
|
|
// not only the rule set — its service must not run, and a node returned to adopted
|
|
// stops it by the ordinary removal of what is no longer declared.
|
|
continue
|
|
}
|
|
resources := m.Resources
|
|
|
|
// What the mesh computes for this module goes FIRST, before the module's own resources.
|
|
//
|
|
// **Order is stated, not derived — the host does not sort** (novox/hq ADR 0005), so
|
|
// whatever the mesh writes down is the order a machine applies. A module's service or
|
|
// container routinely depends on one of these files; nothing here ever depends on a
|
|
// module's resources, because none of it is computed from them.
|
|
//
|
|
// Appended, this was wrong in a way that only showed on the first apply and then healed:
|
|
// the service started before its certificate or its rule set existed, failed, and the next
|
|
// reconcile fixed it. A fault that repairs itself on the second attempt is worse than one
|
|
// that does not, because what gets remembered is that it works.
|
|
var first []map[string]any
|
|
if f := m.Filtering; f != nil {
|
|
first = append(first, map[string]any{
|
|
"id": FilteringID(), "type": "file", "path": f.Into,
|
|
"content": filtering, "mode": "0600",
|
|
})
|
|
}
|
|
if c := m.Certificate; c != nil {
|
|
if with.Certificate == "" {
|
|
// Asked for and not issued. Refused rather than skipped: a module that serves TLS
|
|
// with no certificate does not start, and the reason is somewhere else entirely.
|
|
return nil, fmt.Errorf(
|
|
"%s wants a certificate for this machine and none was issued", m.Module)
|
|
}
|
|
first = append(first, map[string]any{
|
|
"id": CertificateID(), "type": "file", "path": c.Into,
|
|
// Public. It travels in the open like any other file, because it is a statement
|
|
// about a key rather than the key.
|
|
"content": with.Certificate, "mode": "0644",
|
|
})
|
|
if c.Authority != "" {
|
|
first = append(first, map[string]any{
|
|
"id": AuthorityID(), "type": "file", "path": c.Authority,
|
|
"content": with.Authority, "mode": "0644",
|
|
})
|
|
}
|
|
}
|
|
for _, name := range sortedKeys(m.OwnSecrets) {
|
|
sealed := with.Needed[m.Module][name]
|
|
if sealed == "" {
|
|
// Declared and not made. Refused rather than skipped: a module whose own
|
|
// credential is silently absent starts, fails to authenticate, and the reason is
|
|
// three layers away from the machine reporting it.
|
|
return nil, fmt.Errorf(
|
|
"%s needs a secret called %q and none was made for it", m.Module, name)
|
|
}
|
|
first = append(first, ownedBy(m.SecretsOwner, map[string]any{
|
|
"id": NeedID(name), "type": "file", "path": m.OwnSecrets[name], "sealed": sealed,
|
|
}))
|
|
}
|
|
// Operator-owned paths this module is granted use of (novox/hq ADR 0051). Written before
|
|
// the module's own resources, and so before the container that mounts them: the host must
|
|
// find each present — refusing clearly if the operator has not provided it — before it
|
|
// starts anything that depends on it. The mesh creates, chowns and reconciles none of it;
|
|
// an `access` resource says only *this path must exist, and this module reaches it*.
|
|
for _, a := range m.Accesses {
|
|
first = append(first, map[string]any{
|
|
"id": AccessID(a.Path), "type": "access", "path": a.Path, "mode": a.At(),
|
|
})
|
|
}
|
|
for _, to := range m.SecretRequirements() {
|
|
for _, file := range m.SecretFiles(to) {
|
|
var found *Needed
|
|
for i, n := range r.Needs {
|
|
// **This module's need, not the provision's** (novox/hq 04-ISSUES/022). Matching
|
|
// on the name alone, every consumer of a provision took whichever credential
|
|
// happened to be last in the list — so on a node with two of them, one module
|
|
// would be given the other's password and fail to authenticate with a valid
|
|
// credential belonging to somebody else. And this file's local name, where the
|
|
// module keeps several (ADR 0094).
|
|
if n.Name == to && n.For == m.Module && n.Local == file.Local {
|
|
found = &r.Needs[i]
|
|
}
|
|
}
|
|
if found != nil && found.ByRecord && found.Sealed == "" && !found.Manager {
|
|
// Answered by a record whose key has not been supplied since this consumer was
|
|
// put on it. **Refused, not skipped.** The mesh discarded the plaintext when the
|
|
// key was accepted and cannot seal another, so a machine that resolved cleanly
|
|
// would receive no file at all and fail at whatever tried to read it — which is
|
|
// the outcome ADR 0024 exists to avoid, arrived at politely.
|
|
//
|
|
// The manager holder is the one exception (novox/hq ADR 0050): an empty refresh token
|
|
// is a licence whose manager has not adopted one yet, a real waiting state rather than
|
|
// a lost key. It falls through to the skip below — its bound facts (carrying the
|
|
// manager's public key) are still delivered, which is what adoption needs to seal the
|
|
// first refresh token.
|
|
return nil, fmt.Errorf(
|
|
"%s on this machine uses the licence %q and no key has been sealed to it. "+
|
|
"The mesh cannot make one; supply it again with `licence key %s`",
|
|
m.Module, found.From, found.From)
|
|
}
|
|
if found == nil || found.Sealed == "" {
|
|
// Answered on this machine, or answered by a node the mesh could not seal to.
|
|
// Nothing to write either way, and writing an empty credential file would be
|
|
// worse than none: something would read it and fail authenticating.
|
|
continue
|
|
}
|
|
first = append(first, ownedBy(m.SecretsOwner, map[string]any{
|
|
"id": SecretID(SecretLocal(to, file.Local)), "type": "file", "path": file.Path,
|
|
"sealed": found.Sealed,
|
|
}))
|
|
}
|
|
}
|
|
for _, to := range sortedKeys(m.Grants) {
|
|
for _, g := range with.Grants {
|
|
if g.Provision != to {
|
|
continue
|
|
}
|
|
if g.From == "" {
|
|
// Nothing on that machine asks for this any more. Skipped here rather than
|
|
// where grants are gathered, so the rule holds whoever gathers them.
|
|
//
|
|
// **This is how a credential is withdrawn.** The provisioner removes what
|
|
// nobody asks for, and it can only do that if the mesh stops asking — a
|
|
// consumer that was unassigned would otherwise keep a working login for ever,
|
|
// and nothing would say so.
|
|
continue
|
|
}
|
|
first = append(first, map[string]any{
|
|
// One file per holder — the consumer's module with its local name after it
|
|
// where it keeps several (ADR 0094); the lab found two files with one id.
|
|
"id": GrantID(to, g.Consumer+"."+holderAs(g.From, g.Local)),
|
|
"type": "file",
|
|
"path": grantPath(m.Grants[to], g.Consumer, holderAs(g.From, g.Local)),
|
|
"sealed": g.Sealed,
|
|
})
|
|
}
|
|
}
|
|
for _, to := range sortedKeys(m.Binds) {
|
|
var found *Needed
|
|
for i, n := range r.Needs {
|
|
if n.Name == to && n.For == m.Module {
|
|
found = &r.Needs[i]
|
|
}
|
|
}
|
|
if found == nil {
|
|
// Answered on this same machine rather than from elsewhere in the mesh.
|
|
//
|
|
// **Still written.** It used to be skipped, reasoning that "it is on this node"
|
|
// is a fact nobody needs — and that is true of the *location* and false of
|
|
// everything beside it. A binding also carries what the provider said a consumer
|
|
// must know, which is the port; a consumer cannot invent that, and got an absent
|
|
// file with no explanation. A build machine sharing a node with the registry it
|
|
// pushes to sat in a loop saying it could not read its own binding.
|
|
here, err := here(r, to, with)
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
if here == nil {
|
|
// Nothing in this node's set offers it either, so there is genuinely nothing
|
|
// to say. Resolution has already refused anything unanswerable, so this is a
|
|
// requirement met by the module itself.
|
|
continue
|
|
}
|
|
found = here
|
|
}
|
|
file, err := boundFile(*found, m.Binds[to], ConsumerIdentity(r.Node, IdentitySource(m.Slug, m.Module)))
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
first = append(first, file)
|
|
}
|
|
for _, to := range sortedKeys(m.Receives) {
|
|
file, err := receivedFile(to, m.Receives[to], given[to])
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
first = append(first, file)
|
|
}
|
|
if m.Keeps != "" && with.Kept != nil {
|
|
file, err := keptFile(m.Keeps, with.Kept)
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
first = append(first, file)
|
|
}
|
|
if m.Computed != "" {
|
|
generator, known := with.Generators[m.Computed]
|
|
if !known {
|
|
return nil, fmt.Errorf(
|
|
"%s says its resources are computed by %q, and this control plane has no %q",
|
|
m.Module, m.Computed, m.Computed)
|
|
}
|
|
generated, part, err := generator.Resources(r.Node)
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
if !part {
|
|
// Assigned, and not yet part of what this generates. Nothing to put on the
|
|
// machine, which is different from an error: a node given the network module
|
|
// before it has an address is in exactly that state, briefly.
|
|
continue
|
|
}
|
|
resources = generated
|
|
}
|
|
|
|
// Now, and not before: a module whose resources are computed replaces them wholesale, and
|
|
// merging earlier would throw away the files it still needs.
|
|
resources = append(append([]map[string]any{}, first...), resources...)
|
|
|
|
// Every container is given the mesh's names. Not a choice a module makes: a module that
|
|
// listed them would go stale the day a machine joins, and one that did not would be a
|
|
// module whose containers cannot reach anything by name.
|
|
//
|
|
// A container that was given names of its own keeps them and gets the mesh's beside them:
|
|
// the mesh does not know what else a workload needs to reach, and taking something away
|
|
// to add something is not what "also" means.
|
|
if len(with.Names) > 0 {
|
|
resources = withMeshNames(resources, with.Names)
|
|
}
|
|
|
|
// What this module may name from inside one of its own files. Gathered once per module
|
|
// rather than per file, because it is a fact about the module.
|
|
sealed, err := sealedFor(m, r.Needs, with)
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
// And what its bindings say, for the half of a connection that is not secret.
|
|
known := knownFor(m, r.Needs, r.Node)
|
|
// And the machine underneath, which no binding of its own can tell it.
|
|
thisMachine := machineFacts(r, with.Names)
|
|
|
|
// Which of this module's files carry a secret, for the rule that a container may not read
|
|
// one of them as its environment without saying so (ADR 0086, issue 041).
|
|
secretFiles := secretFilesOf(resources)
|
|
|
|
for _, unsettled := range resources {
|
|
resource, err := ApplySettings(unsettled, with.Settings[m.Module])
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
copied := map[string]any{}
|
|
for k, v := range resource {
|
|
copied[k] = v
|
|
}
|
|
if err := refuseSecretsInEnvironment(copied, secretFiles, m.Module); err != nil {
|
|
return nil, err
|
|
}
|
|
// Said in the catalogue, not on the machine: the host parses strictly and knows no
|
|
// such field, and the reason is for a reader of the manifest.
|
|
delete(copied, SecretsInEnvironment)
|
|
// **After settings, and that is the whole reason it is here.** A module's file
|
|
// content is where a setting lands, so a placeholder may only exist once the setting
|
|
// has been put in — filling secrets first would look at content that is not yet what
|
|
// the machine receives.
|
|
if err := intoFile(copied, sealed, m.Module); err != nil {
|
|
return nil, err
|
|
}
|
|
// **Substituted here, not on the machine.** A bound value is not secret — the mesh
|
|
// holds it in the clear — so there is nothing for the host to be the only witness of,
|
|
// and sending it already filled in means the host learns no new field.
|
|
if err := boundInto(copied, known, m.Module); err != nil {
|
|
return nil, err
|
|
}
|
|
// And what the module could not have written: the machine it turned out to be
|
|
// assigned to. Beside the bound values because it is the same kind of fact — the
|
|
// mesh's own, held in the clear — and because a module that must name itself to
|
|
// something else has no binding to learn it from (novox/hq ADR 0066).
|
|
// Which port this machine gave it, for a module that binds directly rather than
|
|
// through a runtime that can remap (ADR 0038). Applied before the machine's facts so
|
|
// a refusal names the port rather than whatever came after it.
|
|
if err := portInto(copied, m.Module, m.Listens, with); err != nil {
|
|
return nil, err
|
|
}
|
|
// And where this machine put the foundation's servers, for the one module that
|
|
// reaches them by seat rather than by binding (novox/hq 04-ISSUES/102).
|
|
if err := seatInto(copied, m.Module, with); err != nil {
|
|
return nil, err
|
|
}
|
|
if err := machineInto(copied, thisMachine, m.Module); err != nil {
|
|
return nil, err
|
|
}
|
|
if err := pinned(copied, m.Module); err != nil {
|
|
return nil, err
|
|
}
|
|
// And the other half of the same question: an image nobody has published yet is named
|
|
// by an artifact rather than by a placeholder digest, and is just as unrunnable.
|
|
if err := built(copied, m.Module); err != nil {
|
|
return nil, err
|
|
}
|
|
// What the mesh built is kept by digest and path; the store's address is this
|
|
// network's now, composed here and never recorded (novox/hq 04-ISSUES/102).
|
|
if err := artifactsInto(copied, m.Module, with); err != nil {
|
|
return nil, err
|
|
}
|
|
publishedOn(copied, m.Module, with)
|
|
copied["id"] = m.Module + "." + fmt.Sprint(resource["id"])
|
|
// A service saying what it reflects names resources within its own module, so those
|
|
// are prefixed too or they would point at nothing.
|
|
//
|
|
// **Unless it already names one.** A module may reflect a file another module put on
|
|
// the machine — the case this exists for is a resolver restarting when the mesh
|
|
// rewrites the names, which are computed by the mesh and belong to its module, not to
|
|
// the daemon's. Written `<module>.<id>`, and a dot is what marks it as already
|
|
// answered: prefixing it again would point at nothing, silently, and the daemon would
|
|
// serve the old names for ever while everything reported success.
|
|
if renamed := reflectsRenamed(m.Module, resource["restart-on"]); renamed != nil {
|
|
copied["restart-on"] = renamed
|
|
}
|
|
// And what it is reloaded on, by the same rule (novox/hq ADR 0102): an id left
|
|
// unprefixed matches nothing, and the service is never reloaded.
|
|
if renamed := reflectsRenamed(m.Module, resource["reload-on"]); renamed != nil {
|
|
copied["reload-on"] = renamed
|
|
}
|
|
owner[fmt.Sprint(copied["id"])] = m.Module
|
|
out = append(out, copied)
|
|
}
|
|
|
|
// **What only the mesh knows, where this module asked for it.** The graph is the control
|
|
// plane's; making a name resolve is the module's software. Emitted as ordinary files under
|
|
// this module's name, so they are applied, reported and removed exactly as anything else
|
|
// it declares.
|
|
given, err := FactsInto(m, r, with.Names, with.Machines, with.Suffix)
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
for _, fact := range given {
|
|
fact["id"] = m.Module + "." + fmt.Sprint(fact["id"])
|
|
owner[fmt.Sprint(fact["id"])] = m.Module
|
|
out = append(out, fact)
|
|
}
|
|
}
|
|
if with.Adopted {
|
|
// First, before anything a module declares: what the mesh needs reachable, then its guard.
|
|
// The order a machine applies is the order written here.
|
|
ours := Openings(rules, with.Foundation, Published(out))
|
|
ours = append(ours, GuardResources(r.guarded(out, owner, rules, with))...)
|
|
out = append(ours, out...)
|
|
}
|
|
return out, nil
|
|
}
|
|
|
|
// guarded is what the mesh's guard refuses on an adopted node (novox/hq ADR 0103): derived, and
|
|
// for taken modules only.
|
|
//
|
|
// For each taken module, every machine port its containers publish that the filter would admit
|
|
// from the private network only — a published port is forwarded, not received, so a found
|
|
// firewall filtering only what it receives never sees it — together with the ports the module's
|
|
// manifest guards explicitly (the store's port, the broker's management port), wherever the
|
|
// machine put them. A port of a module assigned but not taken is not guarded: it may still be the
|
|
// predecessor's, serving the predecessor's other machines. The foundation's ports are admitted
|
|
// from everywhere and are never guarded.
|
|
func (r Resolution) guarded(out []map[string]any, owner map[string]string, rules []Rule,
|
|
with Rendering) []int {
|
|
meshOnly := map[int]bool{}
|
|
fromEverywhere := map[int]bool{}
|
|
for _, rule := range rules {
|
|
if rule.Protocol != "tcp" {
|
|
continue
|
|
}
|
|
switch rule.From {
|
|
case FromMesh:
|
|
meshOnly[rule.Port] = true
|
|
case FromEverywhere:
|
|
fromEverywhere[rule.Port] = true
|
|
}
|
|
}
|
|
for _, port := range with.Foundation {
|
|
delete(meshOnly, port)
|
|
fromEverywhere[port] = true
|
|
}
|
|
seen := map[int]bool{}
|
|
var ports []int
|
|
guard := func(at int) {
|
|
if !seen[at] {
|
|
seen[at] = true
|
|
ports = append(ports, at)
|
|
}
|
|
}
|
|
for _, m := range r.Modules {
|
|
if !with.Taken[m.Module] {
|
|
continue
|
|
}
|
|
var mine []map[string]any
|
|
for _, resource := range out {
|
|
if owner[fmt.Sprint(resource["id"])] == m.Module {
|
|
mine = append(mine, resource)
|
|
}
|
|
}
|
|
for outer := range Published(mine)["tcp"] {
|
|
if meshOnly[outer] {
|
|
guard(outer)
|
|
}
|
|
}
|
|
for _, want := range m.Guards {
|
|
at := with.machinePort(m.Module, want)
|
|
for _, resource := range mine {
|
|
if fmt.Sprint(resource["type"]) != "container" {
|
|
continue
|
|
}
|
|
listed, _ := resource["ports"].([]any)
|
|
for _, entry := range listed {
|
|
outer, inner, _, ok := mapping(fmt.Sprint(entry))
|
|
if ok && inner == want {
|
|
at = outer
|
|
}
|
|
}
|
|
}
|
|
// Not what this node is told to open to everyone: a per-node exposure setting that
|
|
// widens a guarded port is the operator saying so, and declaring an opening for it
|
|
// and a guard dropping it would be one statement refusing the other.
|
|
if !fromEverywhere[at] {
|
|
guard(at)
|
|
}
|
|
}
|
|
}
|
|
sort.Ints(ports)
|
|
return ports
|
|
}
|
|
|
|
// mapping reads a container's port mapping — `[address:]outer:inner[/protocol]`, the address
|
|
// possibly an IPv6 one in brackets — indexing from the end, so an address's own colons never
|
|
// shift the ports. Not ok for a short form or anything that is not a mapping.
|
|
func mapping(written string) (outer, inner int, address string, ok bool) {
|
|
written = strings.TrimSpace(written)
|
|
if cut := strings.LastIndex(written, "/"); cut >= 0 {
|
|
written = written[:cut]
|
|
}
|
|
parts := strings.Split(written, ":")
|
|
if len(parts) < 2 {
|
|
return 0, 0, "", false
|
|
}
|
|
inner, err := strconv.Atoi(parts[len(parts)-1])
|
|
if err != nil {
|
|
return 0, 0, "", false
|
|
}
|
|
outer, err = strconv.Atoi(parts[len(parts)-2])
|
|
if err != nil {
|
|
return 0, 0, "", false
|
|
}
|
|
return outer, inner, strings.Join(parts[:len(parts)-2], ":"), true
|
|
}
|
|
|
|
// Rules is the rule set this node's filter is derived from: every module's listens, what was
|
|
// computed for this machine, and each module's per-node exposure. The same answer whether the node
|
|
// is adopted or converged — the one loads it as a filter, the other declares it as openings.
|
|
func (r Resolution) Rules(with Rendering) ([]Rule, error) {
|
|
exposure := map[string]map[int]string{}
|
|
for _, m := range r.Modules {
|
|
e, err := Exposure(m, with.Settings[m.Module])
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
if e != nil {
|
|
exposure[m.Module] = e
|
|
}
|
|
}
|
|
return r.Filtering(with.Generators, with.Ports, exposure)
|
|
}
|
|
|
|
// Contribution is one module telling the answer to a requirement what it needs from it.
|
|
type Contribution struct {
|
|
// From is the module that said it, so the provider and a person reading the file can tell
|
|
// which route belongs to what.
|
|
From string `json:"from"`
|
|
// Node is the machine it said it from, empty when that is this one.
|
|
//
|
|
// A provision answered from anywhere in the mesh has consumers on other machines, and the
|
|
// provider has to know who they are — a database told to create a password and not who for
|
|
// cannot do anything with it. Contributions were node-local until this, which meant the one
|
|
// case that most needed them was the one they did not reach.
|
|
Node string `json:"node,omitempty"`
|
|
// At is where that machine is on the private network, empty when it is not on one or when it
|
|
// is this machine.
|
|
//
|
|
// The mesh knows it and a provider should not have to derive it. A reverse proxy is told
|
|
// *send traffic to this consumer* and has to open a connection — so without this every
|
|
// provider that reaches back to a consumer would have to know how the mesh names machines,
|
|
// which is a convention leaking into every module that implements a provision.
|
|
At string `json:"at,omitempty"`
|
|
// As is who this consumer is: what the provider should call the login it creates.
|
|
//
|
|
// **Said by the mesh rather than invented by the provisioner** (novox/hq 04-ISSUES/023). It
|
|
// used to be neither — the provisioner made a name, and the consumer, which has to present it
|
|
// to authenticate, had no way to learn it. One derivation reaches both ends, so they agree by
|
|
// construction.
|
|
// Omitted when there is none. A contribution that is not a credential grant — a module
|
|
// offering something to another on its own machine — has nobody to be identified to, and a
|
|
// field that is always present and usually empty teaches a reader to ignore it.
|
|
As string `json:"as,omitempty"`
|
|
// Secret is the file on this machine holding that consumer's credential, sealed to it.
|
|
//
|
|
// Named rather than carried, for the same reason the private network's key is: the mesh
|
|
// discarded the value and could not put it here if it wanted to. What is here is where to
|
|
// find it.
|
|
Secret string `json:"secret,omitempty"`
|
|
// Values are the module's own, with settings applied. What the keys mean is agreed by the
|
|
// requirement's name — everything providing `reverse-proxy` understands the same shape, which
|
|
// is what makes swapping one for another cost nothing.
|
|
Values map[string]any `json:"values"`
|
|
}
|
|
|
|
// grantPath is where one consumer's sealed credential lands on the providing machine.
|
|
//
|
|
// Suffixed, so the directory can also hold whatever the module writing it keeps there and so a
|
|
// node named like something else in that directory cannot collide with it.
|
|
// One file per consumer, and a consumer is a module on a machine (novox/hq 04-ISSUES/022).
|
|
//
|
|
// Named after both. Named after the machine alone, two modules on one node wrote to one path: the
|
|
// second overwrote the first, and the provisioner — reading a directory — saw one consumer where
|
|
// there were two.
|
|
// reflectsRenamed is a resource's restart-on list under the module's prefix, or nil when it has
|
|
// none. It reads both the shape JSON parsing produces ([]any) and the shape code composing
|
|
// resources natively produces ([]string): a reference that was skipped because its list arrived
|
|
// in the other shape would point at nothing — silently, with the service never restarting and
|
|
// every check passing, which is how a runtime kept serving without the registry trust its
|
|
// daemon file already carried.
|
|
func reflectsRenamed(module string, reflects any) []any {
|
|
var names []string
|
|
switch v := reflects.(type) {
|
|
case []any:
|
|
for _, id := range v {
|
|
names = append(names, fmt.Sprint(id))
|
|
}
|
|
case []string:
|
|
names = v
|
|
}
|
|
if names == nil {
|
|
return nil
|
|
}
|
|
var renamed []any
|
|
for _, named := range names {
|
|
if strings.Contains(named, ".") {
|
|
renamed = append(renamed, named)
|
|
continue
|
|
}
|
|
renamed = append(renamed, module+"."+named)
|
|
}
|
|
return renamed
|
|
}
|
|
|
|
func grantPath(directory, consumer, module string) string {
|
|
return strings.TrimRight(directory, "/") + "/" + consumer + "." + module + ".secret"
|
|
}
|
|
|
|
// holderAs is a consumer's name at the provider with a local name after it, where it keeps several
|
|
// credentials for one provision (ADR 0094); the name alone otherwise.
|
|
func holderAs(as, local string) string {
|
|
if local == "" {
|
|
return as
|
|
}
|
|
return as + "_" + local
|
|
}
|
|
|
|
// contributions collects what every module in this set contributes, by requirement.
|
|
//
|
|
// Ordered by contributing module, because the result becomes a file on a machine and a file whose
|
|
// lines move about is a file that looks changed when nothing changed.
|
|
func (r Resolution) contributions(settings SettingsBy, grants []Grant,
|
|
directories map[string]string) (map[string][]Contribution, error) {
|
|
out := map[string][]Contribution{}
|
|
modules := append([]Manifest{}, r.Modules...)
|
|
sort.Slice(modules, func(i, j int) bool { return modules[i].Module < modules[j].Module })
|
|
|
|
// What consumers on other machines asked for. Merged in with this machine's own, because from
|
|
// the provider's side they are the same thing — somebody wanting something — and a provider
|
|
// that had to read two lists would be a provider that reads one of them.
|
|
sorted := append([]Grant{}, grants...)
|
|
sort.Slice(sorted, func(i, j int) bool {
|
|
if sorted[i].Provision != sorted[j].Provision {
|
|
return sorted[i].Provision < sorted[j].Provision
|
|
}
|
|
if sorted[i].Consumer != sorted[j].Consumer {
|
|
return sorted[i].Consumer < sorted[j].Consumer
|
|
}
|
|
if sorted[i].From != sorted[j].From {
|
|
return sorted[i].From < sorted[j].From
|
|
}
|
|
return sorted[i].Local < sorted[j].Local
|
|
})
|
|
// A consumer already carried by the grants loop, keyed (provision, module). When provider and
|
|
// consumer are co-located, `grantsFor` enumerates the same-node consumer too, so without this the
|
|
// module would be emitted a second time by the m.Contributes loop below — once full (with the
|
|
// grant's secret/as) and once partial — which is the duplicate seen in a co-located mesh.json.
|
|
granted := map[string]map[string]bool{}
|
|
for _, g := range sorted {
|
|
if g.From == "" {
|
|
// As above: nothing on that machine asks for this any more, so the provider is not
|
|
// told about it and withdraws the login on its next pass.
|
|
continue
|
|
}
|
|
out[g.Provision] = append(out[g.Provision], Contribution{
|
|
From: g.From, Node: g.Consumer, At: g.At, Values: g.Values,
|
|
// One holder per local name: the identity the consumer is known by, and the local name
|
|
// after it where the module keeps several (ADR 0094). Not a login any backend checks —
|
|
// a secret is not a login — so the identity limit does not apply to the suffix.
|
|
As: holderAs(ConsumerIdentity(g.Consumer, IdentitySource(g.Slug, g.From)), g.Local),
|
|
Secret: grantPath(directories[g.Provision], g.Consumer, holderAs(g.From, g.Local)),
|
|
})
|
|
if granted[g.Provision] == nil {
|
|
granted[g.Provision] = map[string]bool{}
|
|
}
|
|
granted[g.Provision][g.From] = true
|
|
}
|
|
for _, m := range modules {
|
|
for _, to := range sortedKeys(m.Contributes) {
|
|
// The grants loop already emitted this module's contribution to this provision, with its
|
|
// minted secret — emitting the partial copy again would duplicate it. A contribution with
|
|
// no grant (a reverse-proxy route names a host, not a credential) is not in `granted`, so
|
|
// it still reaches the provider from here.
|
|
if granted[to][m.Module] {
|
|
continue
|
|
}
|
|
// Settings reach a contribution the same way they reach a file. A route's hostname is
|
|
// exactly the kind of thing that differs between one mesh and the next, and a module
|
|
// that could not have it set would have to be edited to be reused.
|
|
values, err := settle(m.Contributes[to], settings[m.Module], nil,
|
|
m.Module+" contributing to "+to)
|
|
if err != nil {
|
|
return nil, fmt.Errorf("%s contributing to %s: %w", m.Module, to, err)
|
|
}
|
|
composeName(values, r.PublicDomain)
|
|
out[to] = append(out[to], Contribution{From: m.Module, Values: values})
|
|
}
|
|
// Several contributions to one requirement (ADR 0094's sibling for `contributes`): an
|
|
// object store's data API and its console are two different public names from one module,
|
|
// not one. Never in `granted` — a route names a host, not a credential — so every local
|
|
// name always reaches the provider from here.
|
|
for _, to := range sortedKeys(m.ContributesMany) {
|
|
for _, local := range sortedKeys(m.ContributesMany[to]) {
|
|
values, err := settle(m.ContributesMany[to][local], settings[m.Module], nil,
|
|
m.Module+" contributing "+local+" to "+to)
|
|
if err != nil {
|
|
return nil, fmt.Errorf("%s contributing %s to %s: %w", m.Module, local, to, err)
|
|
}
|
|
composeName(values, r.PublicDomain)
|
|
out[to] = append(out[to], Contribution{From: m.Module, Values: values})
|
|
}
|
|
}
|
|
}
|
|
return out, nil
|
|
}
|
|
|
|
// composeName joins a contribution's label with a node's public domain, in place (novox/hq ADR
|
|
// 0056).
|
|
//
|
|
// **The whole of what the mesh does with a route's name: join two given strings.** A contribution
|
|
// carries a `label` — the subdomain its operator chose — and the node carries its public domain;
|
|
// the granted name is `<label>.<public-domain>` and the mesh interprets neither half. It runs on
|
|
// any contribution carrying a label, not only a route's, because the mesh does not know what a
|
|
// provision means — a name it can compose from parts it was given is the point, whatever the
|
|
// provision is called.
|
|
//
|
|
// **Additive, so an unmigrated catalogue still works.** A contribution that already carries a full
|
|
// `name` and no `label` is left exactly as it is: the catalogue can migrate module by module while
|
|
// the running mesh keeps serving the full names it has. And a labelled contribution on a node with
|
|
// no public domain composes nothing — there is nothing to join it to — which reads downstream as a
|
|
// route that named no host, the same as it would have before this existed.
|
|
func composeName(values map[string]any, publicDomain string) {
|
|
if values == nil || publicDomain == "" {
|
|
return
|
|
}
|
|
if _, already := values["name"]; already {
|
|
// A full name was given rather than a label. Left as-is: this is the legacy shape, and the
|
|
// point of the label is to not have to write the full name — a contribution that wrote both
|
|
// has said what it wants and the mesh does not second-guess it.
|
|
return
|
|
}
|
|
label, ok := values["label"].(string)
|
|
if !ok || strings.TrimSpace(label) == "" {
|
|
return
|
|
}
|
|
if strings.TrimSpace(label) == "@" {
|
|
// The apex: a module served at the bare public domain, no subdomain — the zone-file
|
|
// convention `@`. Composes to the domain itself, so a node's own site is a label like any
|
|
// other rather than the one route that must still carry a full name.
|
|
values["name"] = publicDomain
|
|
return
|
|
}
|
|
values["name"] = strings.TrimSpace(label) + "." + publicDomain
|
|
}
|
|
|
|
// receivedFile is the file a provider is given its consumers' contributions in.
|
|
func receivedFile(requirement, path string, given []Contribution) (map[string]any, error) {
|
|
if given == nil {
|
|
// Nobody contributed. The file is still written, empty, rather than left absent: a
|
|
// provider that finds no file cannot tell "nothing asked for me" from "the mesh never
|
|
// wrote it", and the two want completely different responses.
|
|
given = []Contribution{}
|
|
}
|
|
// The note goes *inside* the document, not above it. The first version wrote a `//` header
|
|
// and produced a file that says "do not edit" to a person and fails to parse for the program
|
|
// meant to read it — which is the whole audience.
|
|
body, err := json.MarshalIndent(map[string]any{
|
|
"contributions": 1,
|
|
"requirement": requirement,
|
|
"generated": "by the mesh — do not edit; replaced whenever a module contributing to " +
|
|
requirement + " arrives or leaves",
|
|
"given": given,
|
|
}, "", " ")
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
return map[string]any{
|
|
"id": ReceivedID(requirement), "type": "file", "path": path, "mode": "0644",
|
|
"content": string(body) + "\n",
|
|
}, nil
|
|
}
|
|
|
|
// Kept is one secret as the operator can recover it: where it belongs, and the value sealed to the
|
|
// operator's key. Never a node's blob, and never a value.
|
|
//
|
|
// Two kinds, addressed the same way — by the node and module that hold it and the name they know
|
|
// it by. An `own` secret is one a module has for itself; a `pair` secret is a credential the
|
|
// module was granted for a provision it requires (`name` is the provision), and Provider says which
|
|
// node grants it.
|
|
type Kept struct {
|
|
Node string `json:"node"`
|
|
Module string `json:"module"`
|
|
Name string `json:"name"`
|
|
// Kind is `own` or `pair`.
|
|
Kind string `json:"kind"`
|
|
// Provider is the node granting a pair credential; empty for an own secret.
|
|
Provider string `json:"provider,omitempty"`
|
|
// Origin is `made` or `accepted` — whether the mesh minted it or a person supplied it.
|
|
Origin string `json:"origin"`
|
|
// Sealed is the value, sealed to the operator key named in Key.
|
|
Sealed string `json:"sealed"`
|
|
Key string `json:"key"`
|
|
MadeAt time.Time `json:"made-at"`
|
|
// Local is the credential's name inside the consumer where it holds several (ADR 0094).
|
|
Local string `json:"local,omitempty"`
|
|
}
|
|
|
|
// KeptExport is what a person keeps beside the operator key, and what a vault keeps on its disk:
|
|
// every operator-sealed copy, and the honest list of what has none.
|
|
type KeptExport struct {
|
|
Export int `json:"export"`
|
|
OperatorKey string `json:"operator-key"`
|
|
Fingerprint string `json:"fingerprint"`
|
|
// Kept is sealed to OperatorKey. EarlierKey is sealed to a key the mesh has since replaced —
|
|
// recoverable with that key, if the person still has it, and with nothing else. Unrecoverable
|
|
// has no operator copy at all.
|
|
Kept []Kept `json:"kept"`
|
|
EarlierKey []Kept `json:"sealed-to-earlier-key,omitempty"`
|
|
Unrecoverable []Kept `json:"unrecoverable,omitempty"`
|
|
}
|
|
|
|
// KeptID is the resource that carries the export to a module that keeps it.
|
|
func KeptID() string { return "kept" }
|
|
|
|
// keptFile is the export, written where the module said. Ciphertext throughout — see Keeps.
|
|
func keptFile(dir string, kept *KeptExport) (map[string]any, error) {
|
|
body, err := json.MarshalIndent(kept, "", " ")
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
return map[string]any{
|
|
"id": KeptID(), "type": "file", "path": strings.TrimRight(dir, "/") + "/export.json",
|
|
"mode": "0600", "content": string(body) + "\n",
|
|
}, nil
|
|
}
|
|
|
|
// ownedBy gives a secret file the owner the module named, when it named one (Manifest.SecretsOwner).
|
|
func ownedBy(owner string, file map[string]any) map[string]any {
|
|
if owner != "" {
|
|
file["owner"] = owner
|
|
}
|
|
return file
|
|
}
|
|
|
|
// sortedKeys is map iteration made repeatable, which everything written to a machine needs.
|
|
func sortedKeys[V any](m map[string]V) []string {
|
|
out := make([]string, 0, len(m))
|
|
for k := range m {
|
|
out = append(out, k)
|
|
}
|
|
sort.Strings(out)
|
|
return out
|
|
}
|
|
|
|
// boundFile is what a module is told about something it requires from another machine.
|
|
//
|
|
// Where it is and what the providing module said about using it. **No credential**, and the file
|
|
// says so rather than leaving a reader to wonder whether one was meant to be there — a missing
|
|
// field looks like a bug, and a stated absence looks like a boundary.
|
|
func boundFile(n Needed, path, as string) (map[string]any, error) {
|
|
// A record has no machine and no address. Saying so is the difference between a reader
|
|
// concluding "somewhere with no address" and concluding the mesh failed to fill something in.
|
|
where := any(n.At)
|
|
if n.ByRecord {
|
|
where = "a record in this mesh, not a machine"
|
|
}
|
|
body, err := json.MarshalIndent(map[string]any{
|
|
"binding": 1,
|
|
"provision": n.Name,
|
|
"from": n.From,
|
|
"at": where,
|
|
// Who this module is at the other end. **The half that was missing** (novox/hq
|
|
// 04-ISSUES/023): a consumer was told the address, the port and where its password is,
|
|
// and not the name it must present — which the provisioner had invented.
|
|
"as": as,
|
|
"serves": n.Serves,
|
|
// **Where the credential is, not what it is.** It stopped being true that the mesh
|
|
// cannot issue one when 021 was fixed, and a comment asserting a fact about the mesh that
|
|
// has become false is worse than none — somebody reads it and stops looking.
|
|
"generated": "by the mesh — do not edit; replaced whenever this changes. " +
|
|
"The credential is not here: it is sealed, in the file this module's manifest " +
|
|
"names under `secrets`",
|
|
}, "", " ")
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
return map[string]any{
|
|
"id": BoundID(n.Name), "type": "file", "path": path, "mode": "0644",
|
|
"content": string(body) + "\n",
|
|
}, nil
|
|
}
|
|
|
|
// ContributionsFrom is what one module on this node asked of one requirement, settled.
|
|
//
|
|
// Exported because a provider's grants are assembled from its consumers' resolutions, one machine
|
|
// at a time, and the alternative was for the control plane to reimplement settling.
|
|
//
|
|
// **One module, not one machine** (novox/hq 04-ISSUES/022). This used to take a requirement alone
|
|
// and refuse whenever two modules wanted it — correctly, given what it had: the credential was
|
|
// keyed by node, so the two would have shared one, and sharing is worse than refusing. But the
|
|
// arrangement refused is the ordinary one. A node running eight services against one database is
|
|
// not an edge case; it is what a machine looks like. Now each consumer has its own credential and
|
|
// there is nothing left to refuse.
|
|
//
|
|
// **One credential, even where a module contributes several times.** A module may answer one
|
|
// requirement more than once (ADR 0094's sibling for `contributes`) — an object store's data API
|
|
// and its console are two different names, not one. There is still only one `Needed` for it, one
|
|
// credential minted, one grant to settle: a pair credential is not a place to put a label or a
|
|
// port. So where several of this module's contributions reach the same requirement, none of them
|
|
// is "the" value — settling to the first, arbitrarily, would hand the grant one contribution's
|
|
// values under a credential the OTHER contribution's consumer never sees, and would collide with
|
|
// that contribution's own entry from contributions() besides. Empty values, still granted: the
|
|
// module asked, gets its credential, and each named contribution reaches the provider on its own.
|
|
func (r Resolution) ContributionsFrom(requirement, module string, settings SettingsBy) (
|
|
map[string]any, bool, error) {
|
|
all, err := r.contributions(settings, nil, nil)
|
|
if err != nil {
|
|
return nil, false, err
|
|
}
|
|
var mine []map[string]any
|
|
for _, g := range all[requirement] {
|
|
if g.From == module {
|
|
mine = append(mine, g.Values)
|
|
}
|
|
}
|
|
if len(mine) == 1 {
|
|
return mine[0], true, nil
|
|
}
|
|
if len(mine) > 1 {
|
|
return map[string]any{}, true, nil
|
|
}
|
|
// It contributes no payload — but a require-only consumer of a parameterless provision (one whose
|
|
// `serves` names no consumer-supplied key: `redis-cache`, `amqp`) still ASKS for it and must be
|
|
// granted a credential. Keying "asks" on contributions alone marked those grants withdrawn
|
|
// (From=""), so the provider never created the account and the consumer authenticated nowhere. A
|
|
// module asks iff it still requires the provision, whether or not it hands anything up with it.
|
|
for _, m := range r.Modules {
|
|
if m.Module != module {
|
|
continue
|
|
}
|
|
for _, req := range m.Requires {
|
|
if req == requirement {
|
|
return map[string]any{}, true, nil
|
|
}
|
|
}
|
|
}
|
|
// Nothing on that machine asks for this any more. Said as "not found" rather than as an
|
|
// error: it is how a credential is withdrawn, and the provider removes what nobody asks for.
|
|
return nil, false, nil
|
|
}
|
|
|
|
// here is the module on this same machine that answers a requirement, as a binding.
|
|
//
|
|
// **Only when the provider said something a consumer must know.** That is the line: the original
|
|
// reasoning — a file saying "it is on this node" is a fact nobody needs — is right about the
|
|
// location and wrong about everything beside it. A shell is answered here and there is nothing to
|
|
// say about it. A registry is answered here and the consumer still cannot guess the port.
|
|
//
|
|
// The address is this machine's own name on the private network when it has one, and loopback
|
|
// when it does not — a machine not on the network still reaches itself, and naming it by a name
|
|
// nothing resolves would be worse than naming it by an address that always works.
|
|
func here(r Resolution, requirement string, with Rendering) (*Needed, error) {
|
|
serves, answered, err := r.servedOnThisMachine(requirement, with)
|
|
if err != nil || !answered {
|
|
return nil, err
|
|
}
|
|
at := r.At
|
|
if at == "" {
|
|
at = "127.0.0.1"
|
|
}
|
|
return &Needed{Name: requirement, From: r.Node, At: at, Serves: serves}, nil
|
|
}
|
|
|
|
// servedOnThisMachine is what a provider in this node's own set says a consumer must know —
|
|
// derived exactly as the mesh derives it for a provider on any OTHER machine.
|
|
//
|
|
// **The one derivation, so the two arrangements cannot disagree.** For a provider elsewhere the
|
|
// control plane walks that node, reads its manifest with that machine's port assignments, and
|
|
// settles the result with that node's settings layers before offering it (cmd/mesh-controller plan.go,
|
|
// theRestOfTheMesh). A provider on the consumer's own machine never passes through that walk, so
|
|
// every step of it has to be repeated here — and each step that was not repeated was a promise the
|
|
// co-located arrangement quietly broke: first the port (novox/hq 04-ISSUES/038), then the settled
|
|
// values (ADR 0066, an internal CA's root arriving empty).
|
|
//
|
|
// Here rather than in the resolver, because here is the first moment both halves exist: resolving
|
|
// runs before a machine's ports are assigned and knows nothing of settings.
|
|
//
|
|
// The first module in the resolved order that says it serves the provision answers, which is the
|
|
// same choice here() has always made and is stable — providersFirst orders the slice, and nothing
|
|
// downstream depends on a map's iteration.
|
|
func (r Resolution) servedOnThisMachine(provision string, with Rendering) (map[string]any, bool, error) {
|
|
for _, m := range r.Modules {
|
|
if _, said := m.Serves[provision]; !said {
|
|
continue
|
|
}
|
|
serves := ServedOn(m, provision, with.Ports[m.Module])
|
|
if len(serves) == 0 {
|
|
// It named the provision and said nothing about it, which is the same as not answering:
|
|
// there is no fact for a consumer to be given. Another module in the set may still have
|
|
// one, so keep looking rather than concluding from the first.
|
|
continue
|
|
}
|
|
settled, err := Settle(serves, with.Settings[m.Module])
|
|
if err != nil {
|
|
return nil, false, fmt.Errorf("%s serving %s: %w", m.Module, provision, err)
|
|
}
|
|
return settled, true, nil
|
|
}
|
|
return nil, false, nil
|
|
}
|
|
|
|
// withMeshNames gives every container in a set the mesh's names.
|
|
//
|
|
// Copied rather than edited in place: these maps come from a module's manifest, and mutating one
|
|
// would change what the catalogue holds for every other machine running that module.
|
|
//
|
|
// A host-network container gets the names too. It was once skipped, on the belief that it "shares
|
|
// the machine's hosts file already" — but it does not: `docker run --network host` still gives the
|
|
// container its own /etc/hosts (localhost and its own id only), so every `<node>.internal` name the
|
|
// mesh wrote for the machine is invisible inside it, and a client that dials one gets EAI_AGAIN. The
|
|
// remedy is the same `--add-host` every other container gets — the runtime accepts it with
|
|
// `--network host` (verified), and without it a host-network consumer cannot reach a provider by the
|
|
// `.internal` address the mesh hands it as `${bound:...:at}`.
|
|
func withMeshNames(resources []map[string]any, names map[string]string) []map[string]any {
|
|
out := make([]map[string]any, 0, len(resources))
|
|
for _, r := range resources {
|
|
if r["type"] != "container" {
|
|
out = append(out, r)
|
|
continue
|
|
}
|
|
|
|
copied := map[string]any{}
|
|
for k, v := range r {
|
|
copied[k] = v
|
|
}
|
|
var given []any
|
|
if already, ok := copied["hosts"].([]any); ok {
|
|
given = append(given, already...)
|
|
}
|
|
for _, name := range sortedKeys(names) {
|
|
given = append(given, name+":"+names[name])
|
|
}
|
|
copied["hosts"] = given
|
|
out = append(out, copied)
|
|
}
|
|
return out
|
|
}
|
|
|
|
// pinned refuses an image that is not really pinned, on its way to a machine.
|
|
//
|
|
// **Here and not at parse** (novox/hq 04-ISSUES/025). A manifest in a repository names artifacts
|
|
// and the manifest the mesh holds names digests — they are deliberately not the same document, so
|
|
// a file awaiting a pin is legitimate exactly as the bundle's is. What must never happen is a
|
|
// placeholder reaching a machine, and this is the last moment before one does.
|
|
//
|
|
// The host checks only the *shape* of a reference, and cannot do more: verifying a digest exists
|
|
// means reaching a registry, which is the one thing a host must never have to do. So sixty-four
|
|
// zeros satisfies every gate in the system and stops on the machine at `docker pull` — which is
|
|
// how eighteen of them shipped across five modules that parse, resolve and compose cleanly.
|
|
func pinned(resource map[string]any, module string) error {
|
|
image, ok := resource["image"].(string)
|
|
if !ok {
|
|
return nil
|
|
}
|
|
_, digest, found := strings.Cut(image, "@")
|
|
if !found || strings.Trim(strings.TrimPrefix(digest, "sha256:"), "0") != "" {
|
|
return nil
|
|
}
|
|
return fmt.Errorf(
|
|
"%s would send %v to a machine pinned to a placeholder digest, which is never a real "+
|
|
"image — it would be fetched and fail there. Resolve the tag to a digest first",
|
|
module, resource["id"])
|
|
}
|
|
|
|
// built refuses a container that still names an artifact nobody has made.
|
|
//
|
|
// **Said here, by the thing that knows what "artifact" means.** A container naming an artifact is
|
|
// a module saying "the mesh builds this"; the field is resolved into an image when a build
|
|
// publishes one, and until then there is nothing to run. A host receiving it refuses the whole
|
|
// declaration — correctly, since its language has no such field — but what it can say is that a
|
|
// container does not use "artifact", which tells a reader the manifest is malformed. It is not:
|
|
// it is unbuilt, which is a different problem with a different fix, and only the mesh is in a
|
|
// position to tell them apart (novox/hq ADR 0073).
|
|
func built(resource map[string]any, module string) error {
|
|
if fmt.Sprint(resource["type"]) != "container" {
|
|
return nil
|
|
}
|
|
artifact, ok := resource["artifact"].(string)
|
|
if !ok || artifact == "" {
|
|
return nil
|
|
}
|
|
return fmt.Errorf(
|
|
"%s has not been built for this mesh: %v is its %q artifact, and no build has published "+
|
|
"one. Build it — `build <repository> --path <path>` — and the module will name what "+
|
|
"came out instead",
|
|
module, resource["id"], artifact)
|
|
}
|
|
|
|
// publishedOn puts the machine's own port on the outside of a container's mapping.
|
|
//
|
|
// **A module writes the port the software uses; the mesh says where the machine puts it**
|
|
// (novox/hq ADR 0038). So `"5432"` means *publish what the software calls 5432*, and this fills in
|
|
// the half only the mesh can know.
|
|
//
|
|
// A mapping written the long way — `"5433:5432"` — is left exactly as it is. Some things genuinely
|
|
// must be pinned down by hand, and quietly overruling somebody who wrote both halves would be
|
|
// worse than not offering the short form at all.
|
|
func publishedOn(resource map[string]any, module string, with Rendering) {
|
|
if fmt.Sprint(resource["type"]) != "container" {
|
|
return
|
|
}
|
|
listed, ok := resource["ports"].([]any)
|
|
if !ok {
|
|
return
|
|
}
|
|
out := make([]any, 0, len(listed))
|
|
for _, entry := range listed {
|
|
written := fmt.Sprint(entry)
|
|
if strings.Contains(written, ":") {
|
|
// Written the long way, and left alone — unless this node was given a machine port for
|
|
// it (novox/hq ADR 0100): the foundation's ports are the node's, and a manifest's
|
|
// number is only the default. The outer port only; an address and the software's
|
|
// port stay as written.
|
|
out = append(out, givenOuter(written, with.Given[module]))
|
|
continue
|
|
}
|
|
wanted, err := strconv.Atoi(strings.TrimSpace(written))
|
|
if err != nil {
|
|
// Not a port at all. Passed through, so the host refuses it with its own words rather
|
|
// than this quietly dropping something somebody meant.
|
|
out = append(out, written)
|
|
continue
|
|
}
|
|
out = append(out, fmt.Sprintf("%d:%d", with.machinePort(module, wanted), wanted))
|
|
}
|
|
resource["ports"] = out
|
|
}
|
|
|
|
// givenOuter rewrites the machine side of a long-form mapping to the port this node was given for
|
|
// it, when it was given one.
|
|
//
|
|
// **Under either of the mapping's names.** A node gives a port by the number the module names it
|
|
// by, and a module publishing `"2222:22"` may say it listens on 22 or on 2222 — GivenPorts accepts
|
|
// both and answers to both, so looking the machine side up first and the container's port second
|
|
// finds the same number either way. Read only by the container's port, this moved nothing for the
|
|
// module that declares the machine side, and the setting was refused before it got here.
|
|
func givenOuter(written string, given map[int]int) string {
|
|
if len(given) == 0 {
|
|
return written
|
|
}
|
|
mapping, protocol := written, ""
|
|
if cut := strings.LastIndex(written, "/"); cut >= 0 {
|
|
mapping, protocol = written[:cut], written[cut:]
|
|
}
|
|
parts := strings.Split(mapping, ":")
|
|
if len(parts) < 2 {
|
|
return written
|
|
}
|
|
inner, err := strconv.Atoi(strings.TrimSpace(parts[len(parts)-1]))
|
|
if err != nil {
|
|
return written
|
|
}
|
|
at, ok := given[inner]
|
|
if outer, err := strconv.Atoi(strings.TrimSpace(parts[len(parts)-2])); err == nil {
|
|
if machine, named := given[outer]; named {
|
|
at, ok = machine, true
|
|
}
|
|
}
|
|
if !ok {
|
|
return written
|
|
}
|
|
parts[len(parts)-2] = strconv.Itoa(at)
|
|
return strings.Join(parts, ":") + protocol
|
|
}
|
|
|
|
// ServedOn is what a provider tells a consumer, with the port that machine actually uses.
|
|
//
|
|
// **The module writes the port once, in `listens`** (novox/hq ADR 0038). It used to write it three
|
|
// times — for the rule set, for what a consumer is told, and for what the runtime publishes — and
|
|
// nothing checked that the three agreed. This is what fills the second in.
|
|
//
|
|
// The assignment wins when there is one, and the declared port stands in when there is not: a
|
|
// caller composing without a store still gets something coherent, and a mesh that has assigned one
|
|
// tells the truth about where it put it.
|
|
//
|
|
// Only when the module offers exactly one port. A module offering several has not said which
|
|
// belongs to which provision, and guessing would give a consumer a port that answers something
|
|
// else — so it keeps whatever the manifest said, which may be nothing.
|
|
func ServedOn(m Manifest, provision string, ports map[int]int) map[string]any {
|
|
serves := m.Serves[provision]
|
|
|
|
out := make(map[string]any, len(serves)+1)
|
|
for k, v := range serves {
|
|
out[k] = v
|
|
}
|
|
if _, said := out["port"]; said {
|
|
// Written by hand. Redirected to wherever the machine put it, and otherwise left alone.
|
|
if number, ok := asPort(out["port"]); ok {
|
|
if at, known := ports[number]; known {
|
|
out["port"] = at
|
|
}
|
|
}
|
|
return out
|
|
}
|
|
if len(m.Listens) != 1 {
|
|
return out
|
|
}
|
|
wanted := m.Listens[0].Port
|
|
if at, known := ports[wanted]; known {
|
|
out["port"] = at
|
|
} else {
|
|
out["port"] = wanted
|
|
}
|
|
return out
|
|
}
|
|
|
|
func asPort(v any) (int, bool) {
|
|
switch n := v.(type) {
|
|
case int:
|
|
return n, true
|
|
case float64:
|
|
return int(n), true
|
|
}
|
|
return 0, false
|
|
}
|
|
|
|
// atMachinePort redirects a `port` written by a module on this machine to where this machine
|
|
// actually published it (novox/hq 04-ISSUES/038, ADR 0066).
|
|
//
|
|
// **A module writes the port its software uses; only the mesh knows where the machine put it.** A
|
|
// bare `ports` mapping is assigned a host port when the declaration is composed, which is after
|
|
// everything a module wrote has been read — so any number carried out of a manifest is the declared
|
|
// one until it passes through here. Both directions need it: what a co-located consumer is TOLD
|
|
// about a provider, and what a co-located workload TELLS a provider about itself.
|
|
//
|
|
// Named by the module the port belongs to, which is not always the one being talked about: a route
|
|
// contribution's port is the contributing workload's, never the proxy's.
|
|
//
|
|
// A fresh map when it changes anything, so the manifest-derived value handed back by the resolver —
|
|
// or by a grant gathered elsewhere — is never mutated under a caller that still holds it. Keyed by
|
|
// the *declared* port, so applying it a second time is a no-op: once redirected the value is the
|
|
// machine port, which is not itself a key.
|
|
func atMachinePort(serves map[string]any, module string, ports map[string]map[int]int) map[string]any {
|
|
number, ok := asPort(serves["port"])
|
|
if !ok {
|
|
return serves
|
|
}
|
|
at, known := ports[module][number]
|
|
if !known || at == number {
|
|
return serves
|
|
}
|
|
copied := make(map[string]any, len(serves))
|
|
for k, v := range serves {
|
|
copied[k] = v
|
|
}
|
|
copied["port"] = at
|
|
return copied
|
|
}
|
|
|
|
// AtPublishedPort redirects a contribution's port to where the CONSUMER's own machine published it.
|
|
//
|
|
// The same redirection as [atMachinePort], for the one caller that cannot reach it. A grant is
|
|
// assembled on the providing machine out of a consumer that lives on another one, so that
|
|
// consumer's assignments are fetched there and handed in, rather than looked for here where they
|
|
// are not. Without it a proxy told to reach a workload on another machine is told the port the
|
|
// workload's *software* uses, and dials a number that machine never published — the same fault as
|
|
// novox/hq 04-ISSUES/038, one node over.
|
|
func AtPublishedPort(values map[string]any, module string, published map[int]int) map[string]any {
|
|
return atMachinePort(values, module, map[string]map[int]int{module: published})
|
|
}
|