Carmine message queue v2 #278

ptaoussanis · 2022-12-12T15:28:10Z

No description provided.

Introduces a significant rewrite of major parts of Carmine's message queue. The underlying architecture remains unchanged. The API remains MOSTLY unchanged: a small number of users may be affected, please see the changes below. CHANGES - [BREAKING] `enqueue` return value has changed: It used to return <mid> or {:carmine.mq/error <message-status>}. It now always returns a map with possible keys: [mid action error]. Please see docstring for more info. - Improved error messages and logging output. NEW - Added admin utils: `queue-names`, `clear-all-queues`. - Added `enqueue` option: `:lock-ms` to support per-message lock times [#223]. - Added `enqueue` option: `:can-update?` to support message updating. - Handler fn data now includes `:age-ms` to support integration with Tufte or other profiling tools. - `queue-status` util now includes a `:by-mid` {<mid> <message-status>} hash. - Worker object's string/pprint representation is now more useful. - Worker object can now be dereffed to get useful state and stats. - Improved docstrings and mq architecture documentation. - General improvements to implementation, debuggability, and tests.

Introduces a significant rewrite of major parts of Carmine's message queue. The underlying architecture remains unchanged. The API remains MOSTLY unchanged: a small number of users may be affected, please see the changes below. CHANGES - [BREAKING] `enqueue` return value has changed: It used to return <mid> or {:carmine.mq/error <message-status>}. It now always returns a map with possible keys: [mid action error]. Please see docstring for more info. - Improved error messages and logging output. NEW - Added admin utils: `queue-names`, `clear-all-queues`. - Added `enqueue` option: `:lock-ms` to support per-message lock times [#223]. - Added `enqueue` option: `:can-update?` to support message updating. - Handler fn data now includes `:age-ms` to support integration with Tufte or other profiling tools. - `queue-status` util now includes a `:by-mid` {<mid> <message-status>} hash. - Worker object's string/pprint representation is now more useful. - Worker object can now be dereffed to get useful state and stats. - Worker threads are now automatically desynchronized to reduce contention. - Improved docstrings and mq architecture documentation. - General improvements to implementation, debuggability, and tests.

devurandom · 2023-07-10T05:52:51Z

Do you have an ETA for Carmine v3.3, i.e. when this feature will be released?

ptaoussanis · 2023-07-10T07:28:50Z

@devurandom Hi Dennis,

Was actually planning to have it out already last week - but got sidetracked by some other open-source work.

Should be ready within the next 1-2 weeks (so ~mid July).

I'm curious - is there some particular reason you're waiting on this?

devurandom · 2023-07-10T08:09:14Z

Thanks!

is there some particular reason you're waiting on this?

Yes. As I understood from #223, this will bring per-task timeouts, which has proven to be necessary for our system.

ptaoussanis · 2023-07-10T08:26:52Z

Yes. As I understood from #223, this will bring per-task timeouts

Correct, that will be included 👍 Sorry about the delays!
Will try get a pre-release out by the end of this week.

This merge introduces a major rewrite of Carmine's message queue system and includes **POTENTIAL BREAKING CHANGES**. - If you do NOT use Carmine's message queue, you can safely ignore this. - If you DO use Carmine's message queue, please carefully read the details below. 1. `enqueue` return value has changed. This change is relevant to you iff you use the return value of `enqueue` calls (most users do not). Check your `enqueue` call sites to be sure. The fn previously returned `<mid>` (message id) on success, or `{:carmine.mq/error <message-status>}` on error. The fn now always returns a map with possible keys: [success? mid action error]. See the updated `enqueue` docstring for details. 2. `queue-status` return value has changed. This change is relevant to you iff you use the `queue-status` util. The fn previously returned a detailed map of all queue content in O(queue-size). The fn now returns a small {:keys [nwaiting nlocked nbackoff ntotal]} map in O(1). If you want the detailed map of all queue content in O(queue-size), use the new `queue-content` util. 3. The definition of "queue-size" has changed. The old definition: total size of queue. The new definition: total size of queue, LESS mids that may be locked or in backoff. I.e. the new definition now better represents the number of messages awaiting processing. Most users won't be affected by this change since the new definition better corresponds to how most users actually understood the term. 4. `clear-queues` has been deprecated. This utility is now called `queues-clear!!` to better match the rest of the API. - Significantly improved latency (esp. worst-case latency) of handling new messages. Workers will now always prioritise handling of newly queued messages when available, and otherwise fall back to maintaining the mid circle. - Decouple threads for handling and queue maintenance. Thread counts can now be individually customized. - Worker end-of-queue backoff sleeps are now interrupted by new messages. Sleeping workers will awaken automatically when new messages arrive. - Prioritize requeues (treat as "ready"). Requeues no longer need to wait for a queue cycle to be reprocessed. - Smart worker throttling. The `:throttle-ms` worker option can now be a function of the current queue size, enabling dynamic worker throttling. The default `:throttle-ms` value is now `:auto`, which uses such a function. See the updated `worker` docstring for details. - Worker threads are now automatically desynchronized to reduce contention. - Added `enqueue` option: `:lock-ms` to support per-message lock times [#223]. - Added `enqueue` option: `:can-update?` to support message updating. - Handler fn data now includes `:worker`, `:queue-size`. - Handler fn data now includes `:age-ms`. This enables easy integration with Tufte or other profiling tools. - Added utils: `queue-size`, `queue-names`, `queues-clear!!`, `queues-clear-all!!!`. - Worker object's string/pprint representation is now more useful. - Worker object can now be dereffed to get useful state and stats. - Workers can now be dereffed to get various diagnostic info. In particular, the new `:stats` key contains detailed statistics on queue size, queueing time, handling time, etc. - Workers can now be invoked as fns to execute common actions. Actions include: `:start`, `:stop`, `:queue-size`, `:queue-status`. - Various improvements to docstrings, error messages, and logging output - Improved message queue state diagram - General improvements to implementation, observability, and tests.

This merge introduces a major rewrite of Carmine's message queue system and includes **POTENTIAL BREAKING CHANGES**. - If you do NOT use Carmine's message queue, you can safely ignore this. - If you DO use Carmine's message queue, please carefully read the details below. === POTENTIAL BREAKING CHANGES === 1. `enqueue` return value has changed This change is relevant to you iff you use the return value of `enqueue` calls (most users do not). Check your `enqueue` call sites to be sure. The fn previously returned `<mid>` (message id) on success, or `{:carmine.mq/error <message-status>}` on error. The fn now always returns a map with possible keys: [success? mid action error]. See the updated `enqueue` docstring for details. 2. `queue-status` return value has changed This change is relevant to you iff you use the `queue-status` util. The fn previously returned a detailed map of all queue content in O(queue-size). The fn now returns a small {:keys [nwaiting nlocked nbackoff ntotal]} map in O(1). If you want the detailed map of all queue content in O(queue-size), use the new `queue-content` util. 3. The definition of "queue-size" has changed The old definition: total size of queue. The new definition: total size of queue, LESS mids that may be locked or in backoff. I.e. the new definition now better represents the number of messages awaiting processing. Most users won't be affected by this change since the new definition better corresponds to how most users actually understood the term. 4. `clear-queues` has been deprecated This utility is now called `queues-clear!!` to better match the rest of the API. === Performance improvements === - Significantly improved latency (esp. worst-case latency) of handling new messages. Workers will now always prioritise handling of newly queued messages when available, and otherwise fall back to maintaining the mid circle. - Decouple threads for handling and queue maintenance. Thread counts can now be individually customized. - Worker end-of-queue backoff sleeps are now interrupted by new messages. Sleeping workers will awaken automatically when new messages arrive. - Prioritize requeues (treat as "ready"). Requeues no longer need to wait for a queue cycle to be reprocessed. - Smart worker throttling. The `:throttle-ms` worker option can now be a function of the current queue size, enabling dynamic worker throttling. The default `:throttle-ms` value is now `:auto`, which uses such a function. See the updated `worker` docstring for details. - Worker threads are now automatically desynchronized to reduce contention. === New stuff === - Added `enqueue` option: `:lock-ms` to support per-message lock times [#223]. - Added `enqueue` option: `:can-update?` to support message updating. - Handler fn data now includes `:worker`, `:queue-size`. - Handler fn data now includes `:age-ms`. This enables easy integration with Tufte or other profiling tools. - Added utils: `queue-size`, `queue-names`, `queues-clear!!`, `queues-clear-all!!!`. - Worker object's string/pprint representation is now more useful. - Worker object can now be dereffed to get useful state and stats. - Workers can now be dereffed to get various diagnostic info. In particular, the new `:stats` key contains detailed statistics on queue size, queueing time, handling time, etc. - Workers can now be invoked as fns to execute common actions. Actions include: `:start`, `:stop`, `:queue-size`, `:queue-status`. - Various improvements to docstrings, error messages, and logging output - Improved message queue state diagram - General improvements to implementation, observability, and tests.

ptaoussanis · 2023-07-17T17:20:51Z

Closing, will be addressed in forthcoming release.
Should be up tomorrow, working on final testing and release notes now.

Thanks again for all the patience on this!

This merge introduces a major rewrite of Carmine's message queue system and includes **POTENTIAL BREAKING CHANGES**. - If you do NOT use Carmine's message queue, you can safely ignore this. - If you DO use Carmine's message queue, please carefully read the details below. === POTENTIAL BREAKING CHANGES === 1. `enqueue` return value has changed This change is relevant to you iff you use the return value of `enqueue` calls (most users do not). Check your `enqueue` call sites to be sure. The fn previously returned `<mid>` (message id) on success, or `{:carmine.mq/error <message-status>}` on error. The fn now always returns a map with possible keys: [success? mid action error]. See the updated `enqueue` docstring for details. 2. `queue-status` return value has changed This change is relevant to you iff you use the `queue-status` util. The fn previously returned a detailed map of all queue content in O(queue-size). The fn now returns a small {:keys [nwaiting nlocked nbackoff ntotal]} map in O(1). If you want the detailed map of all queue content in O(queue-size), use the new `queue-content` util. 3. The definition of "queue-size" has changed The old definition: total size of queue. The new definition: total size of queue, LESS mids that may be locked or in backoff. I.e. the new definition now better represents the number of messages awaiting processing. Most users won't be affected by this change since the new definition better corresponds to how most users actually understood the term. 4. `clear-queues` has been deprecated This utility is now called `queues-clear!!` to better match the rest of the API. === Performance improvements === - Significantly improved latency (esp. worst-case latency) of handling new messages. Workers will now always prioritise handling of newly queued messages when available, and otherwise fall back to maintaining the mid circle. - Decouple threads for handling and queue maintenance. Thread counts can now be individually customized. - Worker end-of-queue backoff sleeps are now interrupted by new messages. Sleeping workers will awaken automatically when new messages arrive. - Prioritize requeues (treat as "ready"). Requeues no longer need to wait for a queue cycle to be reprocessed. - Smart worker throttling. The `:throttle-ms` worker option can now be a function of the current queue size, enabling dynamic worker throttling. The default `:throttle-ms` value is now `:auto`, which uses such a function. See the updated `worker` docstring for details. - Worker threads are now automatically desynchronized to reduce contention. === New stuff === - Added `enqueue` option: `:lock-ms` to support per-message lock times [#223]. - Added `enqueue` option: `:can-update?` to support message updating. - Handler fn data now includes `:worker`, `:queue-size`. - Handler fn data now includes `:age-ms`. This enables easy integration with Tufte or other profiling tools. - Added utils: `queue-size`, `queue-names`, `queues-clear!!`, `queues-clear-all!!!`. - Worker object's string/pprint representation is now more useful. - Worker object can now be dereffed to get useful state and stats. - Workers can now be dereffed to get various diagnostic info. In particular, the new `:stats` key contains detailed statistics on queue size, queueing time, handling time, etc. - Workers can now be invoked as fns to execute common actions. Actions include: `:start`, `:stop`, `:queue-size`, `:queue-status`. - Various improvements to docstrings, error messages, and logging output - Improved message queue state diagram - General improvements to implementation, observability, and tests.

ptaoussanis · 2023-07-18T09:56:55Z

v3.3.0-RC1 is now on Clojars, release notes here.

Given the amount of code touched, please tread carefully and confirm everything's working as expected before putting into production 🙏

ptaoussanis added the enhancement label Dec 12, 2022

ptaoussanis self-assigned this Dec 12, 2022

ptaoussanis mentioned this issue Dec 12, 2022

Task level lock-ms instead of a global lock period #223

Closed

ptaoussanis added a commit that referenced this issue Dec 12, 2022

[mq] [nop] [#278] Update message queue diagram

2baf355

ptaoussanis added a commit that referenced this issue Dec 12, 2022

[mq] [nop] [#278] Update message queue diagram

8a246e2

ptaoussanis added a commit that referenced this issue Dec 17, 2022

[mq] [nop] [#278] Update message queue diagram

8782dd1

ptaoussanis added a commit that referenced this issue Dec 20, 2022

[mq] [nop] [#278] Update message queue diagram

f0f1ce6

ptaoussanis added a commit that referenced this issue Dec 21, 2022

[mq] [nop] [#278] Update message queue diagram

7bf9f88

ptaoussanis added a commit that referenced this issue Mar 2, 2023

[nop] [mq] [#278] Update message queue diagram

58e34c4

ptaoussanis added a commit that referenced this issue Mar 2, 2023

[nop] [mq] [#278] Update message queue diagram

41011eb

ptaoussanis added a commit that referenced this issue Mar 9, 2023

[nop] [mq] [#278] Update message queue diagram

a529d83

ptaoussanis added a commit that referenced this issue Mar 18, 2023

[nop] [mq] [#278] Update message queue diagram

4cafd09

ptaoussanis added a commit that referenced this issue Apr 24, 2023

[nop] [mq] [#278] Update message queue diagram

2d8641b

ptaoussanis added a commit that referenced this issue May 26, 2023

[nop] [mq] [#278] Update message queue diagram

6f54be8

ptaoussanis closed this as completed Jul 17, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Carmine message queue v2 #278

Carmine message queue v2 #278

ptaoussanis commented Dec 12, 2022

devurandom commented Jul 10, 2023

ptaoussanis commented Jul 10, 2023

devurandom commented Jul 10, 2023

ptaoussanis commented Jul 10, 2023

ptaoussanis commented Jul 17, 2023

ptaoussanis commented Jul 18, 2023

Carmine message queue v2 #278

Carmine message queue v2 #278

Comments

ptaoussanis commented Dec 12, 2022

devurandom commented Jul 10, 2023

ptaoussanis commented Jul 10, 2023

devurandom commented Jul 10, 2023

ptaoussanis commented Jul 10, 2023

ptaoussanis commented Jul 17, 2023

ptaoussanis commented Jul 18, 2023