Concurrency control question

[shouichi]

Hello. Thank you for the great library! I have a question about how concurrency control works.

Let's say we want to limit the concurrent API call to the 3rd party service to 1. In other words, we want to perform jobs serially.

Using good_job_control_concurrency_with(perform_limit: 1) seems the right choice but jobs that failed to obtain an advisory lock are marked as failed and are retried later. This is not desirable because it does exponential backoff (am I right?). Here we just want to perform them serially.

Another way is to use a dedicated queue and a process but we really don't want to run a dedicated process if we can. Am I missing something?

Thank you again.

[bensheldon]

@shouichi You've discovered how the Concurrency Control plugin works: raise and retry with incremental backoff. GoodJob doesn't have a different mechanism for managing concurrency, other than through a dedicated queue and process. It doesn't necessarily have to be a dedicated process if you write some dynamic configuration, but you would have to manage that in you ops process; or do both (hold down your true concurrency, and use the concurrency plugin as backstop)

I'm always open to alternative implementations if you have ideas.

[shouichi]

Thank you for the response!

It doesn't necessarily have to be a dedicated process if you write some dynamic configuration

Do you mean the --queue argument? If I understand it correctly, it would be something like good_job --queues="serial:1;-serial:5".

I'm always open to alternative implementations if you have ideas.

The rate limit feature can be implemented as sidekiq does. Although rate limiting seems to be a different domain.

https://www.mikeperham.com/2015/08/06/sidekiq-enterprise/

[bensheldon]

If the operation cannot be executed due to the rate limit, it will raise an error by default. If this error is raised within a Sidekiq job, Sidekiq will catch the error and reschedule the job to execute in the near future.

That is exactly like GoodJob. Great minds do think alike 😊

Thanks for sharing that link. The way Sidekiq leader election works is also really similar to what I've been considering too.

For dynamic configuration, I mean that you can have something like:

# config/application.rb

config.good_job.queues = ENV['HEROKU_DYNO_NAME'] == 'worker-0' ? "single:1;-single:5" : "-single:6"

[shouichi]

That is exactly like GoodJob. Great minds do think alike

🆒

So it seems like if a lock contention occurs, it should be treated as a misconfiguration that leads to an error (at least by default).

For dynamic configuration, I mean that you can have something like:

Thank you for the clarification. I understand.

It might be a good idea to add a section on how to deal with the typical rate limiting in the README.

Thank you for the help! You and good_job helped us a lot 👍

[bensheldon]

So it seems like if a lock contention occurs, it should be treated as a misconfiguration that leads to an error

I don't think that's quite the right characterization. I think that if lock contention occurs, raise-and-retry is the mechanism by which the concurrency extension operates. An exception is not ideal for control flow, but raising an exception is the only way to "cancel" a job from within an ActiveJob job.

It sounds like I should explain what is happening in the readme so that it's not surprising.

Thank you for asking the question and discussing!

[bensheldon]

@shouichi I made a PR, could you please review? #695

[shouichi]

What I meant was that if we wanted to handle a queue serially but go the concurrent exception, it should be a misconfiguration (e.g., booting two processes/threads for the queue).

Reviewed the PR, thank you!