Scaling over 1-million jobs/day

[andychongyz]

I'm currently using Sidekiq Pro and considering switching to Good Job for free ACID behavior.
But I'm concerned because Good Job seems geared towards apps that enqueue 1 million jobs per day.

Can it handle scaling beyond that? Can I scale by using a separate DB, scaling DB resources, or sharding?

[bensheldon]

Good question. I don't generally recommend developers migrate from Sidekiq Pro to GoodJob, rather the other way around when they outgrow GoodJob (a good problem to have!). That said, there are applications using GoodJob that are pushing 20M jobs/day, though the ones I'm aware of are fairly specialized applications with sizable databases and infrastructure behind them. There's no science behind the 1M jobs/day number, that was more just the level of experience I have that I feel confident in supporting as an open source maintainer.

Can it handle scaling beyond that? Can I scale by using a separate DB, scaling DB resources, or sharding?

It can do all of those things, though for the purpose of your consideration, all of those things result in the loss of transactional integrity with your application database.

Have you looked at Solid Queue? I think its design is more oriented towards performance; GoodJob strikes a balance between performance, flexibility/features, and budget-consciousness (Solid Queue has lots of potential).

[andychongyz]

Thanks for the response. I'll stick with Sidekiq Pro for now. I read about Solid Queue, but it's missing some features like CRON and there's no performance comparison with Sidekiq.