improve snapshot generation under load #1380
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Grist uses the SQLite API to make backups. This PR reorganizes the code so that the SQLite connection used by ActiveDoc (the main class representing an open document) is also the one used for backups. Otherwise, if the document is under continuous modification (e.g. because of some automation), attempting to perform the backup may not terminate since SQLite may keep restarting the backup under the hood. With this change, it is now also worth beginning the backup even if the document is busy, so the PR also does that. In testing, when a document is under heavy load, the last step call in the SQLite backup API may be relatively slow, perhaps as all changes that accumulated are flushed also to the backup. But since a since SQLite connection is being used now, there should not be any new SQLite-level busy timeouts.
paulfitz
force-pushed
the
paulfitz/backup-tweak
branch
from
5077bc9 to
64e56d8
Compare
|
(rebased to pick up the test fix on main) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Context
Grist uses the SQLite API to make backups. This PR reorganizes the code so that the SQLite connection used by ActiveDoc (the main class representing an open document) is also the one used for backups. Otherwise, if the document is under continuous modification (e.g. because of some automation), attempting to perform the backup may not terminate since SQLite may keep restarting the backup under the hood.
With this change, it is now also worth beginning the backup even if the document is busy, so the PR also does that.
In testing, when a document is under heavy load, the last step call in the SQLite backup API may be relatively slow, perhaps as all changes that accumulated are flushed also to the backup. But since a since SQLite connection is being used now, there should not be any new SQLite-level busy timeouts.
Has this been tested?