Non unique schedule redis lock key

[marcowindt]

We ran into the issue that Redis lock key for job scheduler is non-deterministic across nodes when using Jason encode, since the key is based on the MD5 hash of the serialized job. Right now the order of the keys in json object has to be the same which is not always the case, resulting in different keys for the same job.

Using a custom serializer fixes this. We now use Jason encode with ordered keys option.

Leaving a comment in case someone else runs into this in the future.

[ananthakumaran]

Thanks for reporting. If your fix is generic, we can do the same in ExqScheduler itself. I will look into this weekend

[barttenbrinke]

Ran into this too, this might be a good thing :)

[mkarnebeek]

Exq Scheduler pushes jobs into the queue at intervals specified by the schedule configuration. It is designed to run on more than one machine for redundancy without causing duplicate jobs to be scheduled.

One of the first lines from the readme and the reason for using this library.

I'd say this is the main feature of this library and needs to be addressed. Have you had a chance to take a look at this?

[ananthakumaran]

Using a custom serializer fixes this. We now use Jason encode with ordered keys option.

@marcowindt I couldn't find any option named ordered_keys or similar, did you use Jason.OrderedObject?

@mkarnebeek I have the fix ready #44, if you could test it, it would be great.

[mkarnebeek]

Thank you for the very fast fix! I will test this today and let you know

[marcowindt]

@ananthakumaran Yes, we use it as follows:

input
|> Jason.decode!(objects: :ordered_objects)

[barttenbrinke]

Nice!

[mkarnebeek]

Deployed to various stages and ended on production yesterday. This morning it looked good. So ✅! Thank you very much for the fast response!