-
Notifications
You must be signed in to change notification settings - Fork 361
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix migration deadlock on busy systems #3424
Fix migration deadlock on busy systems #3424
Conversation
force no transaction in down too force no transaction, so we don't have sub-transactions
5f6d2e1
to
9db7847
Compare
Hey @fearoffish, thanks for submitting this PR. You are right the current change to fix the locking issue was not enough(we also noticed it today on more systems with higher load). |
Also woth mentioning this has happened before and the solution they found was splitting each table into an own migration file: 00321c7 |
@philippthun has tested the behaviour of
This is not so well documented in Sequel, or i just did not find it. This in return means when migrating multiple tables not only put the contents of each table in an own Will add this to the docs i prepared as one can do so many things wrong in Sequel migrations as the framework does a lot of magic and psql and mysql behave quite differently. |
As found out while testing the behaviour of Sequel in #3424 we must call no_transaction for migration that define own tranaction blocks.
Disable automatic transactions, follow up of #3424
In #3417 there was a change to fix deadlocks created by table joins. However, on our busy system at gov.uk a deadlock still occurred and our deployment attempts in production failed repeatedly.
We fixed this and successfully deployed by forcing a
no_transaction
outside theup
anddown
but not inside the index migration. I'm not entirely sure why this worked, but our assumption was that theup
created a transaction and then we end up with sub-transactions for each migration.Either way, this is the only way we could deploy on a busy system.
I'm open to a discussion on why if anyone can explain, and maybe tell me why this was the wrong solution.
I have reviewed the contributing guide
I have viewed, signed, and submitted the Contributor License Agreement
I have made this pull request to the
main
branchI have run all the unit tests using
bundle exec rake
I have run CF Acceptance Tests