From 03dc9dec25b13e77996a070eb9eb038c35419ca8 Mon Sep 17 00:00:00 2001
From: Florian Braun <5863788+FloThinksPi@users.noreply.github.com>
Date: Tue, 27 Jun 2023 15:50:44 +0200
Subject: [PATCH] Fix performance issues with delayed_jobs by extending index

We found out that when having a lot of delayed_jobs rows in the table,
postgresql query planner did a sequential scan of the table.
This was due to the query of cc_workers to fetch new jobs being:

```
SELECT *
FROM "delayed_jobs"
WHERE (((("run_at" <= '2023-06-27 07:18:28.061781+0000')
AND ("locked_at" IS NULL))
(.....)
ORDER BY "priority" ASC, "run_at" ASC
LIMIT 1 FOR Update;
```

As an order by is used on `priority` the index over the columns
`queue, locked_at, locked_by, failed_at, run_at` is not used.

This is more severe as the query above gets more and more expensive
by increasing row count and is queried a lot. Every worker every few
seconds does this select. This quickly can become a load issue for the
database.

This change adds the `priority` column to the already existing
index `delayed_jobs_reserve`. Improves query times as well as
database load significantly in our tests.
---
 ...627110500_add_priority_by_to_delayed_jobs_index.rb | 11 +++++++++++
 1 file changed, 11 insertions(+)
 create mode 100644 db/migrations/20230627110500_add_priority_by_to_delayed_jobs_index.rb

diff --git a/db/migrations/20230627110500_add_priority_by_to_delayed_jobs_index.rb b/db/migrations/20230627110500_add_priority_by_to_delayed_jobs_index.rb
new file mode 100644
index 00000000000..2ae140f2bf3
--- /dev/null
+++ b/db/migrations/20230627110500_add_priority_by_to_delayed_jobs_index.rb
@@ -0,0 +1,11 @@
+Sequel.migration do
+  up do
+    drop_index :delayed_jobs, nil, name: :delayed_jobs_reserve
+    add_index :delayed_jobs, [:queue, :locked_at, :locked_by, :failed_at, :run_at, :priority], name: :delayed_jobs_reserve
+  end
+
+  down do
+    drop_index :delayed_jobs, nil, name: :delayed_jobs_reserve
+    add_index :delayed_jobs, [:queue, :locked_at, :locked_by, :failed_at, :run_at], name: :delayed_jobs_reserve
+  end
+end