Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix migration deadlock on busy systems #3424

Merged

Conversation

fearoffish
Copy link

@fearoffish fearoffish commented Sep 5, 2023

In #3417 there was a change to fix deadlocks created by table joins. However, on our busy system at gov.uk a deadlock still occurred and our deployment attempts in production failed repeatedly.

We fixed this and successfully deployed by forcing a no_transaction outside the up and down but not inside the index migration. I'm not entirely sure why this worked, but our assumption was that the up created a transaction and then we end up with sub-transactions for each migration.

Either way, this is the only way we could deploy on a busy system.

I'm open to a discussion on why if anyone can explain, and maybe tell me why this was the wrong solution.

  • I have reviewed the contributing guide

  • I have viewed, signed, and submitted the Contributor License Agreement

  • I have made this pull request to the main branch

  • I have run all the unit tests using bundle exec rake

  • I have run CF Acceptance Tests

force no transaction in down too

force no transaction, so we don't have sub-transactions
@fearoffish fearoffish force-pushed the no_transaction_in_index_migration branch from 5f6d2e1 to 9db7847 Compare September 5, 2023 09:55
@FloThinksPi
Copy link
Member

Hey @fearoffish, thanks for submitting this PR. You are right the current change to fix the locking issue was not enough(we also noticed it today on more systems with higher load).
Another constraint worth adding to #3407
For this migration it would be fine to use no transaction at all. However sometimes you just dont want sequel to do its "implicit transaction magic" yet still have manually defined transactions functioning. We`ll need to test this further how sequel behaves here and add it to the documentation and also patch this in later migration as they may suffer from the same issue.

@FloThinksPi
Copy link
Member

Also woth mentioning this has happened before and the solution they found was splitting each table into an own migration file: 00321c7

@FloThinksPi
Copy link
Member

FloThinksPi commented Sep 7, 2023

@philippthun has tested the behaviour of no_transaction as well as the transaction do blocks inside Sequel migration by looking at the RAW SQL Logs of CC. He found out Sequel behaves as following:

  1. The transaction do block inside a migration block (up/down/change do) has no effect at all by default. In the logged SQL statements there is only one ocurence of BEGIN for the whole migration. If removing the transaction do block still only one ocurence of BEGIN can be seen.
  2. When adding no_transaction before the migration blocks and additionally removing the transaction do block there are no BEGIN statements logged at all.
  3. When adding no_transaction before the migration blocks and having the transaction do block still in place, 24 BEGIN statements are logged. As wanted now every loop run over the tables opens an own transaction and closes before starting with the next table.

This is not so well documented in Sequel, or i just did not find it. This in return means when migrating multiple tables not only put the contents of each table in an own transaction do block, now one also is absolutely required to add no_transaction before the migration blocks.

Will add this to the docs i prepared as one can do so many things wrong in Sequel migrations as the framework does a lot of magic and psql and mysql behave quite differently.
Also i will open another PR to introduce this property also for other migrations that suffer from the same problem.

@FloThinksPi FloThinksPi merged commit 829181a into cloudfoundry:main Sep 7, 2023
@FloThinksPi FloThinksPi self-requested a review September 7, 2023 11:30
FloThinksPi added a commit that referenced this pull request Sep 7, 2023
As found out while testing the behaviour of Sequel in #3424 we must
call no_transaction for migration that define own tranaction blocks.
philippthun added a commit that referenced this pull request Sep 8, 2023
Disable automatic transactions, follow up of #3424
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants