Merge pull request 'ADR 0264 and issue 330: a message a consumer gives up on is kept until a person delivers it again or drops it' (#223) from fix/330-a-message-given-up-on-is-kept into main
This commit was merged in pull request #223.
This commit is contained in:
+150
@@ -0,0 +1,150 @@
|
||||
---
|
||||
topic: the mesh
|
||||
status: accepted
|
||||
date: 2026-10-08
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
---
|
||||
|
||||
# 264. A message a consumer gives up on is kept until a person delivers it again or drops it
|
||||
|
||||
## Context
|
||||
|
||||
[Design 25](../03-DESIGN/01-to-be/25-the-bus-on-nats.md) promises it three times: an event is
|
||||
"dead-lettered when it cannot be" delivered (§1); "after `max-deliver` attempts the advisory feeds
|
||||
`mesh.events.dead` (its own small stream)" (§3); and "an event whose consumer keeps failing dead-letters
|
||||
after `max-deliver`" is how it is checked (§10). [ADR 0042](0042-the-shape-of-an-event-on-the-wire.md)
|
||||
decided the dead-letter destination on the broker the mesh had before, and
|
||||
[ADR 0106](0106-the-bus-is-nats.md) names dead-lettering in how the bus is checked.
|
||||
|
||||
None of it exists ([issue 330](../04-ISSUES/330-the-dead-letter-stream-design-25-promises-does-not-exist/00-report.md)).
|
||||
A durable consumer on the bus hands a message over at most five times. When it gives up, the server
|
||||
publishes a maximum-deliveries advisory naming the stream, the consumer and the message's sequence. The
|
||||
controller heard it and raised a `max-deliveries` condition, which cleared an hour after the advisories
|
||||
stopped. The message stayed only in its source stream: EVENTS drops an event after a week. On 2026-10-06 a
|
||||
media server's consumer gave up on several downloads that way; nothing records which, and they cannot be
|
||||
delivered again.
|
||||
|
||||
Three things the mesh had decide the shape:
|
||||
|
||||
- **The server says it, and says it once.** An advisory is a core message. A controller restarting, which
|
||||
happens with every delivery of the controller, would miss one said meanwhile, and the message with it.
|
||||
- **A message on EVENTS has many consumers.** Publishing it again on its own subject hands it to every
|
||||
module that consumes it, not only to the one that gave up. A merge would be built twice; a download taken
|
||||
twice.
|
||||
- **A module's runtime reads an event's key from its subject**, from the token before `.event.` and what
|
||||
follows it, in both runtimes (the TypeScript binding and the Go one). It does not read the subject's
|
||||
first tokens.
|
||||
|
||||
## Considered Options
|
||||
|
||||
### Where the server's word is kept
|
||||
|
||||
1. **The controller's subscription to the advisory, as it was.** Lost while no controller listens.
|
||||
Rejected.
|
||||
2. **A stream that captures the advisory subject itself**, which the controller takes from. The server
|
||||
writes it whether or not anyone listens. Chosen.
|
||||
|
||||
### Where the message is kept
|
||||
|
||||
1. **Leave it in its source stream and keep a pointer.** The source drops it after a week; a pointer to
|
||||
nothing is the fault this issue describes. Rejected.
|
||||
2. **A copy in a stream of its own**, fetched by its sequence as soon as the notice is taken, while the
|
||||
source still holds it. Chosen.
|
||||
|
||||
### How it is delivered again to its consumer alone
|
||||
|
||||
1. **Publish it again on its own subject.** Every consumer of the subject gets it again. Rejected.
|
||||
2. **Make the consumer start again before the message** (delete and re-create it). It would hand over
|
||||
everything after the message as well. Rejected.
|
||||
3. **A subject only that consumer filters.** Every consumer on EVENTS also filters
|
||||
`mesh.again.<consumer>.>`; a message delivered again is published on `mesh.again.<consumer>.` and its
|
||||
own subject without `mesh.`, so the runtime reads the same key from it. Chosen. It needs no change to
|
||||
any runtime, and the controller's own consumer reads it back to the subject it was published on.
|
||||
|
||||
## Decision
|
||||
|
||||
1. **Two streams beside EVENTS, declared by the controller like the others:**
|
||||
- **DEAD_LETTER_NOTICES** captures `$JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES.>`, as a work queue kept
|
||||
for a week (the longest a source keeps what a notice is about). The controller's consumer `controller`
|
||||
on it has no maximum of deliveries: a notice it could not act on is offered again a minute later.
|
||||
- **DEAD_LETTERS** holds the kept messages under `mesh.events.dead.<stream>.<consumer>`. It keeps them
|
||||
with no age and up to 256 MiB, and when full it refuses the next copy rather than dropping one it
|
||||
holds. Its de-duplication window is a day.
|
||||
2. **The serving controller takes each notice**, fetches the message it names from its source by sequence,
|
||||
and keeps a copy in DEAD_LETTERS with the message's own headers and body, and with its stream, consumer,
|
||||
subject, sequence, deliveries, the time it was given up and the time it was first published. The
|
||||
server's own headers (`Nats-…`) are not copied, because they describe the first publish and would refuse
|
||||
or de-duplicate the copy. The copy's id is `<stream>.<consumer>.<sequence>`, so one message is kept once
|
||||
however often its notice is taken. A message its source no longer holds is kept as a record that says
|
||||
so. A copy that cannot be made is said in the controller's log and as the consumer's condition, and the
|
||||
notice is taken again.
|
||||
3. **Every consumer on EVENTS filters its own `mesh.again.<consumer>.>`**, and EVENTS takes `mesh.again.>`.
|
||||
The controller's consumer reads a message there as the subject it was first published on.
|
||||
4. **The controller's verb `dead-letters`** lists what DEAD_LETTERS holds, newest first, or for one
|
||||
consumer; shows one whole by its id; **delivers one again** to the consumer that gave it up; or **drops
|
||||
one**. Delivering again and dropping each require `why`, are recorded in the hand-act log first, and are
|
||||
a person's decision, not a repair (S15 does not count them). Only an event is delivered again, on the
|
||||
consumer's again subject, and only when that consumer is on the bus and filters its again subject: a
|
||||
dead letter is never let go as delivered while nothing would receive it. An ask a seat's worker gave up
|
||||
on, a message from any other stream, and a record without its message can only be dropped, and the verb
|
||||
says why (asks: [issue 334](../04-ISSUES/334-an-ask-a-seats-worker-gave-up-on-can-only-be-dropped/00-report.md)).
|
||||
Delivering again carries the message's id, so asking twice delivers once. It is served by the serving
|
||||
controller on its own connection, not by a fresh process.
|
||||
5. **The `max-deliveries` condition is open while DEAD_LETTERS holds a message for that consumer**, one
|
||||
condition per consumer, and clears when its messages are delivered again or dropped, never because the
|
||||
server stopped saying it. Its words say who gave up on how many messages, that what they asked for was
|
||||
not done, that the mesh keeps them, and what the operator does: "Needs you: deliver them again or drop
|
||||
them, from the mesh MCP server." A copy that could not be made is its own condition (token `not-kept`),
|
||||
kept for an hour after the last failure, as the advisories were.
|
||||
6. **The writers table gains a row** (to-be 45 §1): a message a consumer gave up on is written by the
|
||||
controller alone, on `mesh.events.dead.>` and `mesh.again.>`, and read, delivered again or dropped
|
||||
through its verb.
|
||||
|
||||
## Consequences
|
||||
|
||||
- Design 25 §1, §3 and §10 hold. The stream §3 names `mesh.events.dead` is DEAD_LETTERS; its subjects carry
|
||||
the stream and consumer.
|
||||
- The controller may publish on `mesh.events.dead.>` and `mesh.again.>`, and acknowledge on
|
||||
DEAD_LETTER_NOTICES. It already reached the stream API. It gets no grant over the seats' queues.
|
||||
- Taking the notices never stops the controller serving: when they cannot be taken, that is said as a
|
||||
condition (`not-kept`) and tried again every minute.
|
||||
- Every module's consumer changes once, when its filters gain its again subject. Its position is kept.
|
||||
- A message given up on is now visible and costs space until somebody acts. A consumer that gives up on
|
||||
many fills DEAD_LETTERS; full, further copies are refused and said, which is the loud failure the mesh
|
||||
prefers to dropping what it holds.
|
||||
- A module handler sees an event delivered again as the same key. It can tell it was delivered again from
|
||||
the `Mesh-Delivered-Again` header, and need not.
|
||||
- What a desk click may do does not change ([ADR 0258](0258-a-desk-click-performs-only-an-acknowledgement-and-an-answer-is-a-one-time-ticket.md)):
|
||||
delivering again and dropping take a person's why, so the notification offers no button for them.
|
||||
|
||||
## How it is checked
|
||||
|
||||
- mesh-controller `internal/link/replay330_test.go`, **R330** in mesh-lab's replays register: a module's
|
||||
consumer as the controller makes it never acknowledges an event; once it gives up, the event is kept with
|
||||
its consumer, stream, subject, deliveries and the time it was given up. It fails on the commit before the
|
||||
fix and passes on it.
|
||||
- mesh-controller `internal/link/deadletters_test.go`: delivered again, an event reaches the consumer that
|
||||
gave it up and not another module that handled it; one is refused, and kept, when its consumer is gone or
|
||||
does not filter its again subject, and when it is not an event; the list's total is the held count however
|
||||
few it lists; notices that cannot be taken are said while the controller serves on; it is no longer kept, and a second delivery finds
|
||||
nothing; a dropped one is gone; a notice whose message is gone is kept as a record that cannot be
|
||||
delivered again; a notice taken twice is kept once; the controller acts on its own event delivered again.
|
||||
- mesh-controller `cmd/mesh-controller/deadletters_test.go`: the verb lists, refuses an act without why or
|
||||
with an argument it would pass over before anything is done, drops with the act recorded as a person's
|
||||
decision; the condition is open per consumer while messages are held, in plain words that say where to
|
||||
act, and is not raised when none are.
|
||||
- mesh-controller `internal/broker`: the streams' retention, the consumer's again filter, the writers table
|
||||
against to-be 45 §1, and the composed grants against their golden file.
|
||||
- mesh-tools `node-tools/internal/bus/bus_test.go`: an event delivered again has the key it had.
|
||||
- **Live**, once delivered: the self-check (D7) finds both streams as declared; `nats_streams` lists them;
|
||||
the next consumer that gives up shows in `mesh-controller.dead-letters`.
|
||||
|
||||
## References
|
||||
|
||||
- [Issue 330](../04-ISSUES/330-the-dead-letter-stream-design-25-promises-does-not-exist/00-report.md)
|
||||
- [Design 25](../03-DESIGN/01-to-be/25-the-bus-on-nats.md) §1, §3, §10;
|
||||
[to-be 45](../03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md) §1 and S9
|
||||
- [ADR 0042](0042-the-shape-of-an-event-on-the-wire.md), [ADR 0106](0106-the-bus-is-nats.md),
|
||||
[ADR 0253](0253-a-condition-says-itself-to-the-operator-in-plain-words-beside-its-summary.md)
|
||||
- [Research 036](../01-RESEARCH/036-everything-over-nats/00-overview.md) (G7)
|
||||
@@ -8,8 +8,9 @@ code:
|
||||
- mesh-catalog modules/nats (to be written)
|
||||
- mesh-sdk src (the protocol's NATS binding, step 3)
|
||||
- mesh-tools node-tools/internal/bus (a module's state, ADR 0201)
|
||||
updated: 2026-10-06
|
||||
updated: 2026-10-08
|
||||
decisions:
|
||||
- 02-DECISIONS/0264-a-message-a-consumer-gives-up-on-is-kept-until-a-person-delivers-it-again-or-drops-it.md
|
||||
- 02-DECISIONS/0235-the-bus-is-backed-up-by-its-own-snapshot-of-each-stream.md
|
||||
- 02-DECISIONS/0201-a-module-keeps-its-current-state-in-key-value-buckets-it-declares-and-reaches-through-the-runtime.md
|
||||
- 02-DECISIONS/0167-a-membership-carries-what-its-module-receives-and-who-the-mesh-is.md
|
||||
@@ -167,7 +168,9 @@ Core NATS is at-most-once. Everything the mesh must not lose lives in a JetStrea
|
||||
|---|---|---|---|
|
||||
| CONTROL | `mesh.control.>` except `alive` (a build's outcome moved to its seat, ADR 0121) | work queue, one consumer (the controller), explicit ack | the store-window guarantee ([ADR 0083](../../02-DECISIONS/0083-one-push-leaves-the-mesh-consistent.md)): the controller `nak`s with a delay while its store is away and the message is redelivered; nothing is dropped |
|
||||
| NODES | `mesh.node.>` | last per subject | one declaration per node, always the newest |
|
||||
| EVENTS | `mesh.mod.*.event.>` and `mesh.seat.*.event.>` | limits (age, size), durable consumer per subscribing module | a subscriber that was down catches up; after `max-deliver` attempts the advisory feeds `mesh.events.dead` (its own small stream). *2026-10-01:* a build's whole log is here too, as the build seat's `log.<build id>` events ([ADR 0157](../../02-DECISIONS/0157-a-build-says-what-it-does-on-the-bus-as-it-happens.md)) — one subject per build, a week of retention, read back by `builds --log <id>` with a consumer that is gone when the reading is done |
|
||||
| EVENTS | `mesh.mod.*.event.>` and `mesh.seat.*.event.>`; and `mesh.again.>` (ADR 0264) | limits (age, size), durable consumer per subscribing module | a subscriber that was down catches up; after `max-deliver` attempts the advisory feeds `mesh.events.dead` (its own small stream). *2026-10-08, [ADR 0264](../../02-DECISIONS/0264-a-message-a-consumer-gives-up-on-is-kept-until-a-person-delivers-it-again-or-drops-it.md):* that stream is DEAD_LETTERS, below; each consumer also filters `mesh.again.<consumer>.>`, where a message it gave up on is delivered again to it alone. *2026-10-01:* a build's whole log is here too, as the build seat's `log.<build id>` events ([ADR 0157](../../02-DECISIONS/0157-a-build-says-what-it-does-on-the-bus-as-it-happens.md)) — one subject per build, a week of retention, read back by `builds --log <id>` with a consumer that is gone when the reading is done |
|
||||
| DEAD_LETTER_NOTICES | the server's maximum-deliveries advisory | work queue, a week; one consumer (the controller), no `max-deliver` | the server's word that a consumer gave up on a message, kept whether or not a controller is listening, until the controller has copied the message (ADR 0264) |
|
||||
| DEAD_LETTERS | `mesh.events.dead.<stream>.<consumer>` | limits: no age, 256 MiB, refuses new when full; a day's de-duplication | every message a consumer gave up on, with its consumer, subject, deliveries and when, until a person delivers it again or drops it with why (ADR 0264); a `max-deliveries` condition is open per consumer while it holds any |
|
||||
| `KV_<module>_<name>` | `$KV.<module>_<name>.>` | the newest value per key — as many past values as the owner declared — no age unless the owner declared one; a value at most 256 KiB, a bucket at most 64 MiB | a module's state (ADR 0201): one per name in a manifest's `state`, created from the catalogue on every raise, so it exists before its owner runs anywhere; kept when the module is unassigned, because what it holds is data |
|
||||
|
||||
Tool calls and heartbeats stay on core NATS: a lost heartbeat is the next heartbeat; a lost tool
|
||||
|
||||
@@ -5,6 +5,7 @@ code: [mesh-controller, mesh-host, mesh-tools, mesh-catalog, mesh-sdk, mesh-lab]
|
||||
updated: 2026-10-08
|
||||
decisions:
|
||||
- 02-DECISIONS/0265-a-verb-that-reads-is-answered-on-the-serving-controllers-own-connection-and-every-connection-says-what-it-is.md
|
||||
- 02-DECISIONS/0264-a-message-a-consumer-gives-up-on-is-kept-until-a-person-delivers-it-again-or-drops-it.md
|
||||
- 02-DECISIONS/0261-an-assignment-made-while-its-modules-build-runs-is-kept-pending-and-the-controller-makes-it.md
|
||||
- 02-DECISIONS/0254-a-wait-for-a-persons-new-login-is-not-a-gates-failure-and-each-module-of-a-send-keeps-its-own-pass.md
|
||||
- 02-DECISIONS/0253-a-condition-says-itself-to-the-operator-in-plain-words-beside-its-summary.md
|
||||
@@ -87,6 +88,7 @@ Every kind of state the core keeps, and the one component that writes it. Anyone
|
||||
| the operator-channel's open messages | the seat's holder | its own key-value state | — |
|
||||
| a machine's declared data: what it measured, what is retired, what was deleted (ADR 0233) | controller, from what each `node-backup` holder answers | the controller's store, `data_item` and `data_reading` | the holder measures and says; it records nothing the controller keeps |
|
||||
| the facts snapshot | controller | the artifact store, `facts/latest` | the build seat reads |
|
||||
| a message a consumer gave up on (ADR 0264) | controller, from the server's maximum-deliveries notice | the bus, the stream `DEAD_LETTERS` | read, delivered again or dropped through the controller's `dead-letters` verb, with why |
|
||||
|
||||
A bus subject two components may publish on is refused when the controller composes grants, unless
|
||||
this table marks it shared. Changing a writer is a change to this table, through a decision.
|
||||
@@ -223,7 +225,7 @@ signal and asserts its condition. `doctor signals` shows, for every row, the age
|
||||
| S6 | a build asked → its outcome | build seat | each ask | max(20 min, 3 × the p90 of measured builds), 1 h while nothing is measured, *provisional* (the build seat declares no timeout) | `ask-lost` | warning | — |
|
||||
| S7 | a call running → finished | controller | each call | the verb's bound: push, rotate and command 30 min; assign and unassign 15 min; `doctor` 3 min; others 10 min | `call-hung` | warning | — |
|
||||
| S8 | a provider's failing word repeated | provider | every 15 min while failing (ADR 0224) | 30 min | `provider-silent` | warning | — |
|
||||
| S9 | bus advisories: slow consumer, maximum deliveries, permission violation, consumer deleted | bus server's system subjects | any | any occurrence | `slow-consumer`, `max-deliveries`, `refused`, `consumer-lost` — each naming the call, consumer or module in the mesh's words | warning | H3 for `consumer-lost` |
|
||||
| S9 | bus advisories: slow consumer, maximum deliveries, permission violation, consumer deleted | bus server's system subjects | any | any occurrence; a message given up on for as long as `DEAD_LETTERS` holds it (ADR 0264) | `slow-consumer`, `max-deliveries`, `refused`, `consumer-lost` — each naming the call, consumer or module in the mesh's words | warning | H3 for `consumer-lost` |
|
||||
| S10 | the self-check's heartbeat | controller's `doctor` | every run | 2 × its interval, watched **from the second machine** (§5) | `self-check-silent` | urgent | — |
|
||||
| S11 | tool runner heartbeat | tool runner | its interval | 3 × interval, *provisional* | `tools-silent` | warning | — |
|
||||
| S12 | the controller lease renewed | controller | every 5 s | 15 s; a holder that lost the lease or was found expired, and a lease bucket found raised again from nothing, said for an hour; a controller serving without the lease, while it does | `lease-lost` | urgent | — |
|
||||
|
||||
+27
-2
@@ -2,8 +2,9 @@
|
||||
status: located
|
||||
opened: 2026-10-08
|
||||
located-in: [mesh-controller internal/broker (streams.go, the streams the controller declares; derived.go, each event consumer's maximum), mesh-controller internal/link (advisories.go, the maximum-deliveries advisory)]
|
||||
fixed-by:
|
||||
amended-design:
|
||||
fixed-by: mesh-controller#159, mesh-tools#24, mesh-lab#70, mesh-host#60
|
||||
replay: R330
|
||||
amended-design: 03-DESIGN/01-to-be/25-the-bus-on-nats.md
|
||||
---
|
||||
|
||||
# 330. The dead-letter stream design 25 promises does not exist
|
||||
@@ -65,3 +66,27 @@ The controller declares a small dead-letter stream beside EVENTS. On each maximu
|
||||
copies the given-up message (the advisory names its stream and sequence) into that stream with the
|
||||
consumer's name and when. A verb lists what it holds, delivers one again or drops it, as a hand act. If
|
||||
the operator decides against a stream, design 25 §1, §3 and §10 are amended to say what happens instead.
|
||||
|
||||
## Resolution
|
||||
|
||||
Decided in [ADR 0264](../../02-DECISIONS/0264-a-message-a-consumer-gives-up-on-is-kept-until-a-person-delivers-it-again-or-drops-it.md),
|
||||
which [design 25](../../03-DESIGN/01-to-be/25-the-bus-on-nats.md) §3 and
|
||||
[to-be 45](../../03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md) §1 and S9 now name:
|
||||
|
||||
- **DEAD_LETTER_NOTICES** captures the server's maximum-deliveries advisory, so a notice said while no
|
||||
controller listens waits for the next one.
|
||||
- **DEAD_LETTERS** keeps a copy of each message given up on, fetched by its sequence while the source still
|
||||
holds it, with its consumer, stream, subject, deliveries and when it was given up. It keeps them with no
|
||||
age until a person acts.
|
||||
- **The controller's `dead-letters` verb** lists them, shows one whole, delivers one again to the consumer
|
||||
that gave it up and to nobody else, or drops one. An ask a seat's worker gave up on can only be dropped
|
||||
for now ([issue 334](../334-an-ask-a-seats-worker-gave-up-on-can-only-be-dropped/00-report.md)).
|
||||
Delivering and dropping say why and are recorded in the hand-act log.
|
||||
- **The `max-deliveries` condition** is open per consumer while DEAD_LETTERS holds its messages, and says
|
||||
where to act: "deliver them again or drop them, from the mesh MCP server."
|
||||
- The comments that said a failing event dead-letters (the controller's streams and derived consumers, the
|
||||
tool runner's NATS binding) say what happens now.
|
||||
|
||||
**Replay R330** (mesh-controller `internal/link/replay330_test.go`, registered in mesh-lab) fails on the
|
||||
commit before the fix and passes on it.
|
||||
|
||||
|
||||
@@ -0,0 +1,49 @@
|
||||
---
|
||||
status: open
|
||||
opened: 2026-10-08
|
||||
located-in: [mesh-controller internal/link (deadletters.go, AgainTo), mesh-controller internal/broker (nats.go, the controller's grants)]
|
||||
fixed-by:
|
||||
amended-design:
|
||||
---
|
||||
|
||||
# 334. An ask a seat's worker gave up on can only be dropped
|
||||
|
||||
## Symptom
|
||||
|
||||
[ADR 0264](../../02-DECISIONS/0264-a-message-a-consumer-gives-up-on-is-kept-until-a-person-delivers-it-again-or-drops-it.md)
|
||||
keeps every message a durable consumer gives up on in DEAD_LETTERS. The controller's `dead-letters` verb
|
||||
delivers an **event** again to the one consumer that gave it up. An **ask** to a seat that its worker gave up
|
||||
on (a seat's work queue, `SEAT_<seat>`, consumer `SEAT_<seat>_worker`, five deliveries) is kept and said
|
||||
the same way, but the verb refuses to deliver it again: it can only be dropped, and its sender asked to say
|
||||
it again.
|
||||
|
||||
Found in the review of the fix for issue 330 (2026-10-08). The first version published the ask again on its
|
||||
own subject. The review refused it for two reasons:
|
||||
|
||||
- **The grant.** It needed the controller to publish on every seat's `accept` subjects
|
||||
(`mesh.seat.*.accept.>`), wider than the seats it asks today, and wider than least authority allows.
|
||||
- **The original stays.** On a work-queue stream, a message its consumer gave up on is never acknowledged.
|
||||
It stays in the queue until the stream's age drops it, so publishing a copy leaves two.
|
||||
|
||||
## Why it is an issue
|
||||
|
||||
An ask given up on is said and kept, but its answer is "drop it and have the sender ask again". The sender
|
||||
may be a person or a module that does not know its ask was given up. The mesh holds the message and cannot
|
||||
act on it.
|
||||
|
||||
## How it is checked
|
||||
|
||||
When fixed: a test in mesh-controller where a seat's worker gives up on an ask, the verb delivers it again to
|
||||
that seat's worker, the original is gone from the work queue, and the controller's grant names only the
|
||||
seats the mesh declares.
|
||||
|
||||
## Core or not
|
||||
|
||||
**Core.** It is located in the controller, so it resolves with a replay in mesh-lab's replays register or a
|
||||
`replay-none:` saying why.
|
||||
|
||||
## Suggested fix
|
||||
|
||||
Grant the controller each declared seat's `accept` subjects from the catalogue (as its build seats are
|
||||
granted now), deliver the ask again on its own subject, and delete the original from the work queue by its
|
||||
`Mesh-Dead-Sequence` once the copy is published.
|
||||
Reference in New Issue
Block a user