Migrate to sqlite db backend #644
Open
+40,370
−23,532
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tmpfs
force-pushed
the
sqlite-db
branch
2 times, most recently
from
4cf84ef to
79d7c4a
Compare
Update doc comments to reflect new language.
Make truncate() a private implementation detail.
So it's clearer if we are just streaming records or also the decoded events.
We can get the CommitHash from the EventRecord.
Lookup account and folder row ids from the database based on the AccountId and VaultId identifiers.
In order to do strict verification of account root hashes we need to depend upon the sos-client-storage library which creates a circular dependency with sos-database.
Pass the authenticated user identity into the client storage.
Ensures that the imported account events exactly match the original account events.
Prevent duplicate calls as the storage constructor now automatically loads the folder event logs.
So we can check if the server is running with db storage or not.
So that compiling the server does not depend upon sos_account or sos_client_storage.
A hard conflict is being encountered which is triggering a sign out which causes the test to fail with a no search index error.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Database Importer
All the main data is being imported but we also need to support:
Database backup/archive
Housekeeping
sos_protocoldependency should not depend onsos_accountas it bloats the dependencies forsos_serverServer
Local Accounts
Once client storage for db is ready check all the folders are being created correctly:
Account Upgrade
File Blobs
These are currently being imported into the database however large blobs will slow the db read and writes down so we need to consider whether to:
Decided to go with option 1) and keep all file blobs external. -- this means we will still need a backup archive that supports compressing the database and files for all accounts.
Columns
Backend
Update the backend writers to support both file system and database backends:
tools db upgradeNote we would like to use the
STRICTtable keyword to prevent type coercion but it doesn't work with theDATETIMEcolumn type.Date time handling
If we rely on sqlite to insert and generate date and times then they are in a non-ISO 8601 format (space-delimited instead of
T) however in some instances we must insert the date/time (eg: event rows) which we use RFC3339. If we don't rely on SQLite to generate or update date and time and instead rely only on the Rust code then we can always use RFC3339 date/times for consistency.We would then also be able to remove
UtcDateTime::parse_utc_iso8601.Initial work on this completed in 9cec5be
Pragma Hardening
https://www.sqlite.org/pragma.html#pragma_synchronous
MacOS only - https://www.sqlite.org/pragma.html#pragma_fullfsync