[FEATURE]: Enabling distributed locking in Dynomite

[charybr]

Enabling distributed locking in Dynomite

We need to enable distributed locking to address the problem described in https://conductor.netflix.com/technicaldetails.html.
The race condition between sweeper decide and task update threads can happen even in single instance under high-load (but chances are more in case of multiple instances) . There was a discussion regarding - Workflow decider not properly scheduling next tasks under high-load (#1664).
So it sounds like we need to enable locking for consistency irrespective of single or multiple conductor instance.

Problem
To enable locking currently Conductor supports redis server (single, cluster or sentinel) or zookeeper. But Dyno client (Netflix/dyno#271) has code changes to implement Redlock. For some reason, this is not integrated in Conductor.

Describe Preferred Solution

Since Dynomite is the default persistence and queuing mechanism in Conductor, there should be an option to use Dynomite for locking as well. If agreed, I can create a PR for this.

Describe Alternatives

Alternative is using either Redis (single, cluster or sentinel) or Zookeeper. In scenarios where already Dynomite is used, this becomes an overhead to have separate Redis or Zookeeper just for locking.

[charybr]

In the past, there was an attempt to implement Redlock in Dynomite itself. Please refer
#759
#1137

There was a ask for this same feature request in below comment
#1137 (comment)

[github-actions]

This issue is stale, because it has been open for 45 days with no activity. Remove the stale label or comment, or this will be closed in 7 days.

[charybr]

Will soon create a PR

[charybr]

still working the code changes

[github-actions]

This issue is stale, because it has been open for 45 days with no activity. Remove the stale label or comment, or this will be closed in 7 days.

[github-actions]

This issue was closed, because it has been stalled for 7 days with no activity.