Fix occasional "database is locked" errors while loading sample data

[elia]

Summary

SQLite doesn't support full-concurrency and often fails with "SQLite3::BusyException: database is locked" when ActiveJob uses the default threaded :async adapter. ActiveJob is used by ActiveStorage to analyze uploaded images for metadata.

A side effect of this is that SQLite timeouts are no longed triggered and the sample data loads sequentially in way less time (e.g. during sandbox generation).

From https://stackoverflow.com/a/5767295:

SQLite locks the entire database during a write operation (i.e. when a write is happening on any table, no other write, to any table anywhere can happen at the same time). Some databases provide concurrent writes via table-level locks, or sometimes row-level locks. To contrast this to SQLite's implementation, a table-level lock basically means that when you're writing data to a given table, no other thread can write to any record in that table, at the same time (however, writes to other tables can occur simultaneously, in some circumstances). Similarly, row-level locks take it even further, and allow only the necessary rows involved to be locked, allowing concurrent writes to the same table to happen from multiple threads. The idea here is to minimize the amount of data you need to lock for a write operation, which effectively increases the amount of concurrent writes possible across the database, and depending on your implementation/how you use your database, this can significantly increase throughput.

Now, back to your question...

The fact that SQLite is threadsafe doesn't mean that multiple threads can concurrently write to it - it means that it has a way of handling access from multiple threads - which is to (a) allow timeouts/retries, and (b) to return a useful error (SQLITE:Busy) when a lock is currently held on the database. That is, threadsafe means nothing more than, "Multiple threads can access this data in a way that won't result in data corruption due to simultaneous access."

Basically, somewhere in the code, one thread is trying to do its update before another thread has released its lock on the database. This is a common hurdle with SQLite, because the authors/documentation will tell you that SQLite can handle concurrency like a champ. The reality is that what SQLite considers "concurrency support" amounts to trying to be very fast so that locks on the database are only held for a very short time, and therefore locks on the database are released before timeouts are hit. In a lot of cases, this works just fine and never gets in your way. However, having very short-lived locks is not the same as actually allowing concurrent writes from multiple threads.

Think of it like the way that iOS does multitasking (at least as of iOS 5, when I'm writing this) - really what it's doing is putting other apps on pause, and coming back to them. This has the effect that (a) battery life is much better due to lower CPU utilization, and (b) you don't have to start an app from scratch every time you launch it. This is great, but the actual word "multitasking" as used in iOS doesn't technically mean the same thing as "multitasking" in other environments (even Mac OS X).

SQLite is the same way. Do they have "concurrency" support? Well sort of, but the way they define the word "concurrency" isn't the way the rest of the DB world defines "concurrency".

No one is really wrong, but in cases like these, it adds to implementation confusion.

Checklist

Check out our PR guidelines for more details.

The following are mandatory for all PRs:

• I have written a thorough PR description.
• I have kept my commits small and atomic.
• I have used clear, explanatory commit messages.

The following are not always needed (cross them out if they are not):

• I have added automated tests to cover my changes.
• I have attached screenshots to demo visual changes.
• I have opened a PR to update the guides.
• I have updated the readme to account for my changes.

[waiting-for-dev]

Thanks, @elia!