Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix scheduler/runner sync-up #1511

Open
3 tasks
mfeit-internet2 opened this issue Jan 29, 2025 · 0 comments
Open
3 tasks

Fix scheduler/runner sync-up #1511

mfeit-internet2 opened this issue Jan 29, 2025 · 0 comments
Assignees

Comments

@mfeit-internet2
Copy link
Member

On some systems, the runner's query for work to do doesn't consistently return all of the rows it should, causing runs to be missed.

This query has been the same for years and the problem seems to be worse on more-recent versions of PostgreSQL, notably 15 on Debian 12 systems. It should work because the notifications sent to the runner happen after the transaction that produced them is committed.

Make the following improvements:

  • Remove notifications that aren't used (run_new being one)
  • Have the database send a run_ready notification with a row ID as the payload.
  • Change the runner to pick up those notifications and act on them individually if the start time is within the next iteration time
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Ready
Development

No branches or pull requests

1 participant