novox/hq
3
0
Fork 0
Code Issues 4 Pull Requests Releases Wiki Activity

A second route-proxy cannot be assigned: two modules on novox provide acme-ca, and pin takes a node #258

New Issue
Open
opened 2026-10-01 14:35:35 +00:00 by mesh-admin · 5 comments
mesh-admin commented 2026-10-01 14:35:35 +00:00
Contributor
Copy Link
Copy Source

Seen from ace on 2026-10-01 while retiring HAL's traefik (ace's last HAL unit) in favour of route-proxy.

assign ace route-proxy answers:

mesh-controller: these assignments cannot be applied:
  - 2 nodes provide "acme-ca", wanted by route-proxy — say which with `pin ace acme-ca <node>`: novox, novox

Both providers are on novox: public-acme (provides acme-ca, scope mesh) and step-ca (provides acme-ca and internal-acme-ca, both scope mesh). pin <node> <provision> <from-node> names a node, so it cannot tell them apart — the message itself lists "novox, novox". novox's own route-proxy resolved to public-acme (its env says ACME_DIRECTORY=https://acme-v02.api.letsencrypt.org…), presumably by the same-node rule; a route-proxy on any other node has no such rule and is stuck.

Two ways out, either is fine from ace's side:

  1. step-ca stops offering acme-ca (it is the internal authority; internal-acme-ca is its name) — then the name has one provider mesh-wide; or
  2. pin learns to name a module (pin ace acme-ca novox/public-acme), and the controller's message stops suggesting a form that cannot work.

Until then ace keeps HAL's traefik as its ingress (and, with it, ~/.hal, since every /services/<m>/docker-compose.yml is a link into it). Everything else of HAL on ace is retired as of today. The store is back to route-adapter on ace; nothing was pushed.

Seen from ace on 2026-10-01 while retiring HAL's traefik (ace's last HAL unit) in favour of `route-proxy`. `assign ace route-proxy` answers: ``` mesh-controller: these assignments cannot be applied: - 2 nodes provide "acme-ca", wanted by route-proxy — say which with `pin ace acme-ca <node>`: novox, novox ``` Both providers are on novox: `public-acme` (provides `acme-ca`, scope mesh) and `step-ca` (provides `acme-ca` **and** `internal-acme-ca`, both scope mesh). `pin <node> <provision> <from-node>` names a node, so it cannot tell them apart — the message itself lists "novox, novox". novox's own route-proxy resolved to public-acme (its env says `ACME_DIRECTORY=https://acme-v02.api.letsencrypt.org…`), presumably by the same-node rule; a route-proxy on any other node has no such rule and is stuck. Two ways out, either is fine from ace's side: 1. `step-ca` stops offering `acme-ca` (it is the internal authority; `internal-acme-ca` is its name) — then the name has one provider mesh-wide; or 2. `pin` learns to name a module (`pin ace acme-ca novox/public-acme`), and the controller's message stops suggesting a form that cannot work. Until then ace keeps HAL's traefik as its ingress (and, with it, `~/.hal`, since every `/services/<m>/docker-compose.yml` is a link into it). Everything else of HAL on ace is retired as of today. The store is back to route-adapter on ace; nothing was pushed.
mesh-admin commented 2026-10-01 14:44:49 +00:00
Author
Contributor
Copy Link
Copy Source

Diagnosis, from the code and the records (ace, 2026-10-01):

  • provision_pin (inventory migration 0008, ADR 0084) stores node → provision → from-node; the comment in the migration says in as many words "Not a module". Design 23 (choosing a provider) and HolderAmong's comment say the opposite since: a provider is a (node, module) pair, "two modules on one node could both provide a provision". The pin predates that and was never widened.
  • With a node-only pin and two providers on that node, resolve.go does for … { if w.Node == chosenNode { chosen = &where[i] } } — the last one wins, silently. So pin ace acme-ca novox would not fail; it would hand ace's public names to whichever of public-acme / step-ca sorts last. Exactly the "wrong answer delivered quietly" design 23 refuses.
  • novox's own route-proxy never hits this: both providers are in its own set, so the co-location branch answers (and happened to give Let's Encrypt).
  • step-ca offering acme-ca is deliberate (ADR 0056, lab bed whole-mesh-novox: "route-proxy requires acme-ca, and step-ca provides that with a local authority"), so dropping it from step-ca is not free.

Three ways out, smallest first:

  1. Pin names the provider (completes ADR 0084 to design 23): pin ace acme-ca novox/public-acme; a nullable module column on provision_pin; the resolver matches node+module when given and refuses a node-only pin that matches two providers on that node instead of taking the last; the refusal lists node/module pairs. Plus a pin/unpin tool on the console, which has none.
  2. A seat delivers acme-ca: let a module-declared seat carry delivers (today only the mesh's own seats in seats.go can), public-acme claims "the public certificate authority" — then every route-proxy on every node resolves to the holder, no pins. Lab unchanged (no holder → co-location → step-ca). Larger: manifest field, registration check that the holder provides what it delivers.
  3. step-ca stops offering acme-ca — breaks the lab bed unless it gets a local acme-ca module.

ace is waiting on this for its last HAL unit (traefik). Happy to do 1 or 2 on a branch if you say which.

Diagnosis, from the code and the records (ace, 2026-10-01): - `provision_pin` (inventory migration 0008, ADR 0084) stores **node → provision → from-node**; the comment in the migration says in as many words "Not a module". Design 23 (choosing a provider) and `HolderAmong`'s comment say the opposite since: a provider is a **(node, module)** pair, "two modules on one node could both provide a provision". The pin predates that and was never widened. - With a node-only pin and two providers on that node, `resolve.go` does `for … { if w.Node == chosenNode { chosen = &where[i] } }` — **the last one wins, silently**. So `pin ace acme-ca novox` would not fail; it would hand ace's public names to whichever of public-acme / step-ca sorts last. Exactly the "wrong answer delivered quietly" design 23 refuses. - novox's own route-proxy never hits this: both providers are in its own set, so the co-location branch answers (and happened to give Let's Encrypt). - step-ca offering `acme-ca` is deliberate (ADR 0056, lab bed whole-mesh-novox: "route-proxy requires acme-ca, and step-ca provides that with a local authority"), so dropping it from step-ca is not free. Three ways out, smallest first: 1. **Pin names the provider** (completes ADR 0084 to design 23): `pin ace acme-ca novox/public-acme`; a nullable `module` column on `provision_pin`; the resolver matches node+module when given and **refuses** a node-only pin that matches two providers on that node instead of taking the last; the refusal lists `node/module` pairs. Plus a `pin`/`unpin` tool on the console, which has none. 2. **A seat delivers `acme-ca`**: let a module-declared seat carry `delivers` (today only the mesh's own seats in seats.go can), public-acme claims "the public certificate authority" — then every route-proxy on every node resolves to the holder, no pins. Lab unchanged (no holder → co-location → step-ca). Larger: manifest field, registration check that the holder provides what it delivers. 3. step-ca stops offering `acme-ca` — breaks the lab bed unless it gets a local acme-ca module. ace is waiting on this for its last HAL unit (traefik). Happy to do 1 or 2 on a branch if you say which.
mesh-admin commented 2026-10-01 15:02:21 +00:00
Author
Contributor
Copy Link
Copy Source

Resolved for ace without a controller change, by design 23's first rule (co-location): public-acme is a facts-only module (no container; it names Let's Encrypt), so assigning it on ace makes ace's route-proxy resolve acme-ca on its own node, and internal-acme-ca has one provider (step-ca on novox). The plan resolves and hands route-proxy ACME_DIRECTORY=…letsencrypt… and INTERNAL_ACME_DIRECTORY=…novox.internal:9000…. Every node that runs a route-proxy should carry public-acme — worth a line in route-proxy's README.

What remains of this issue is the trap, not the blocker: a node-only pin that matches two providers on that node silently takes the last one (resolve.go, the chosen loop). The scan of both catalogues finds two names with several providers: acme-ca (public-acme, step-ca) and route (route-adapter, route-proxy). Suggested hardening, small: refuse such a pin with both node/module pairs named, and let a pin name node/module. I can send that PR if wanted.

Resolved for ace without a controller change, by design 23's first rule (co-location): `public-acme` is a facts-only module (no container; it names Let's Encrypt), so **assigning it on ace** makes ace's route-proxy resolve `acme-ca` on its own node, and `internal-acme-ca` has one provider (step-ca on novox). The plan resolves and hands route-proxy `ACME_DIRECTORY=…letsencrypt…` and `INTERNAL_ACME_DIRECTORY=…novox.internal:9000…`. Every node that runs a route-proxy should carry public-acme — worth a line in route-proxy's README. What remains of this issue is the trap, not the blocker: a node-only `pin` that matches two providers on that node silently takes the last one (resolve.go, the `chosen` loop). The scan of both catalogues finds two names with several providers: `acme-ca` (public-acme, step-ca) and `route` (route-adapter, route-proxy). Suggested hardening, small: refuse such a pin with both `node/module` pairs named, and let a pin name `node/module`. I can send that PR if wanted.
mesh-admin commented 2026-10-01 15:24:05 +00:00
Author
Contributor
Copy Link
Copy Source

The controller change is up: mesh-controller PR #195 (feat/pin-names-the-provider), with hq PR #261 for the note on ADR 0084.

The rule the operator set, in one line: a pin names the module providing the provision and the node it runs on — both, always; a node alone makes no sense. So pin <node> <provision> <from-node> <module>, a pin/unpin tool on the console, refusal (never a pick) wherever two providers could answer — across machines and beside the consumer alike — and the records already made are completed by migration where the node they name answers once.

Rollout consequence for novox: its own route-proxy sits beside both public-acme and step-ca, which the resolver used to settle by a map walk (random per plan). After the rollout novox's plan is refused until pin novox acme-ca novox public-acme is said once. Nothing else in either catalogue has two providers of one bound provision on one node (scan: only acme-ca, and route on different nodes).

Not merged by ace: the rollout is novox's to time.

The controller change is up: mesh-controller PR #195 (`feat/pin-names-the-provider`), with hq PR #261 for the note on ADR 0084. The rule the operator set, in one line: *a pin names the module providing the provision and the node it runs on — both, always; a node alone makes no sense.* So `pin <node> <provision> <from-node> <module>`, a `pin`/`unpin` tool on the console, refusal (never a pick) wherever two providers could answer — across machines and beside the consumer alike — and the records already made are completed by migration where the node they name answers once. **Rollout consequence for novox:** its own route-proxy sits beside both public-acme and step-ca, which the resolver used to settle by a map walk (random per plan). After the rollout novox's plan is refused until `pin novox acme-ca novox public-acme` is said once. Nothing else in either catalogue has two providers of one bound provision on one node (scan: only `acme-ca`, and `route` on different nodes). Not merged by ace: the rollout is novox's to time.
mesh-admin commented 2026-10-01 15:26:18 +00:00
Author
Contributor
Copy Link
Copy Source

Merged on review: mesh-controller #195 and hq #261 are on main. Not rolled out from ace. When the controller is next built and pushed to novox, say once:

pin novox acme-ca novox public-acme

(or, through the console, pin {"node":"novox","provision":"acme-ca","from":"novox","module":"public-acme"}) — until then novox's plan is refused with exactly that line in it. This issue closes with that rollout.

Merged on review: mesh-controller #195 and hq #261 are on main. Not rolled out from ace. When the controller is next built and pushed to novox, say once: ``` pin novox acme-ca novox public-acme ``` (or, through the console, `pin {"node":"novox","provision":"acme-ca","from":"novox","module":"public-acme"}`) — until then novox's plan is refused with exactly that line in it. This issue closes with that rollout.
mesh-admin commented 2026-10-01 15:34:25 +00:00
Author
Contributor
Copy Link
Copy Source

Regression from #195, live since the 15:26 rollout, hotfixed in mesh-controller #196 (merged): the new refusal of two providers beside a consumer also fired in the resolver's first pass, so novox vanished from every other node's world — ace's plan read "nothing in this mesh provides secret / oidc-client". The hotfix skips it in the first pass (as the sibling case already did) and refuses only in the second. Needs a controller build + push to novox; novox's own plan still wants pin novox acme-ca novox public-acme once.

**Regression from #195, live since the 15:26 rollout, hotfixed in mesh-controller #196 (merged):** the new refusal of two providers beside a consumer also fired in the resolver's first pass, so novox vanished from every other node's world — ace's plan read "nothing in this mesh provides secret / oidc-client". The hotfix skips it in the first pass (as the sibling case already did) and refuses only in the second. Needs a controller build + push to novox; novox's own plan still wants `pin novox acme-ca novox public-acme` once.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
No items
No labels
Milestone
No items
No Milestone
Assignees
Clear assignees
jschoubben miel.truyen
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: novox/hq#258