Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

added Litestream Replication #4554

Merged
merged 3 commits into from
Aug 26, 2021
Merged

added Litestream Replication #4554

merged 3 commits into from
Aug 26, 2021

Conversation

Alex-CodeLab
Copy link
Contributor

Litestream.io enables easy SQLite replication for continuous backups

@Alex-CodeLab
Copy link
Contributor Author

Litestream requires periodic but short write locks on the database in order to sync changes. SQLite will return an error by default if your application tries to obtain a write lock at the same time. To prevent this, your application will need to set the busy_timeout pragma when it initializes the database connection.

How does c-lightning handle write-locks now?

@rustyrussell
Copy link
Contributor

I think this will break c-lightning: we do not expect db operations to fail at all. We would definitely want to block in this case.

@Alex-CodeLab
Copy link
Contributor Author

Alex-CodeLab commented May 24, 2021 •
edited
Loading

I think this will break c-lightning: we do not expect db operations to fail at all. We would definitely want to block in this case.

I noticed the same issue was discussed here already #4207 (comment)
Setting PRAGMA busy_timeout is a reasonable solution.

-- edit --
seems the quote above about periodic locks was outdated

Originally, Litestream used the write locks to demarcate the transactions in the WAL but that was fixed in v0.3.4.
Now, the only time Litestream requires a write lock is during checkpointing which occurs after ~40MB of page writes.

@rustyrussell rustyrussell force-pushed the master branch 2 times, most recently from 7334f2a to 1d7dae9 Compare June 28, 2021 21:37
@rustyrussell rustyrussell added this to the v0.10.1 milestone Jun 29, 2021
@rustyrussell
Copy link
Contributor

OK, I (trivially) rebased on master, and added commits so that we set the busy timeout. I also remove the copy operation in our tests, which should actually exercise this a little.

@rustyrussell
Copy link
Contributor

Trivial rebase.

@cdecker cdecker modified the milestones: v0.10.1, v0.10.2 Jul 30, 2021
Alex-CodeLab and others added 3 commits August 26, 2021 09:13
@Alex-CodeLab @rustyrussell
added Litestream Replication
da4ed69 
Litestream enables easy SQLite replication
@rustyrussell
wallet: set a timeout if the db is busy.
72557e8 
This is recommended for litestream, which allows for easy async backup,
and harmless otherwise.

Signed-off-by: Rusty Russell <[email protected]>
Changelog-Changed: db: we now set a busy timeout to safely allow others to access sqlite3 db (e.g. litestream)
@rustyrussell
pytest: don't copy db to query it.
858156a 
Should be safe to do in-place.

Signed-off-by: Rusty Russell <[email protected]>
@rustyrussell
Copy link
Contributor

YA trivial rebase.

rustyrussell
rustyrussell approved these changes Aug 26, 2021
View reviewed changes
Copy link
Contributor

@rustyrussell rustyrussell left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ack 858156a

@rustyrussell rustyrussell merged commit 8f782b0 into ElementsProject:master Aug 26, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@rustyrussell rustyrussell rustyrussell approved these changes

@cdecker cdecker Awaiting requested review from cdecker

@ZmnSCPxj ZmnSCPxj Awaiting requested review from ZmnSCPxj

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
v0.10.2
Development

Successfully merging this pull request may close these issues.
3 participants
@Alex-CodeLab @rustyrussell @cdecker