[BUG] Monitor jobs do not reschedule when flipping the SWEEPER_ENABLED setting

[adityaj1107]

Issue by qreshi
Thursday Jan 07, 2021 at 02:23 GMT
Originally opened as opendistro-for-elasticsearch/alerting#333

---

When the SWEEPER_ENABLED setting is set to false, all Alerting jobs are descheduled as expected. However, once this setting is set back to true, the expectation is that existing enabled Monitors would be rescheduled but this is currently not the case.

The issue seems to be occurring in the sweep() logic. This is indirectly invoked from the enable() method when the SWEEPER_ENABLED setting is set to true. At this time, any jobs owned by the shard in question are skipped if the newVersion being passed in is less than or equal to the currentVersion. So when the sweeper is re-enabled, all the jobs that went through no change are skipped over and never rescheduled.

Here is a snippet of the code in question:
https://github.com/opendistro-for-elasticsearch/alerting/blob/8daf9e64ccfd4219441002a821a2bb48bbeeaaf9/core/src/main/kotlin/com/amazon/opendistroforelasticsearch/alerting/core/JobSweeper.kt#L348-L361

The solution would involve accounting for the case where the job version is unchanged only when coming from a scenario where the jobs are previously descheduled and coming from the enable() code path, so that the logic where the jobs are scheduled can occur. Not accounting for the aforementioned scenario could lead to existing jobs being repeatedly descheduled and rescheduled during the routine background runs, which should be avoided.

[adityaj1107]

Comment by pkriete
Thursday Feb 11, 2021 at 00:00 GMT

---

We would love to see this fixed soon. We ran into this issue on AWS ES after toggling opendistro.scheduled_jobs.enabled to false during an outage. Setting it back to true results in a completely opaque failure mode where everything looks like it's running but nothing happens. We have customer facing alerts, the ability to turn them off when they would otherwise be incorrect is critical for us.

For anyone else that might come across this before a patch goes out, bumping the version number on the alerting documents will restart your alerts:

POST .opendistro-alerting-config/_update_by_query