feat: add support for dumping the database to stream

[link2xt]

Serialization is ready, dump <filename> in repl produces a dump in bencode format.
Deserialization is ready too, in REPL read <filename> reads the dump.

Serializer holds a read transaction, so it should be possible to run serialization twice and get the same result, e.g. for hashing and actual streaming, without having to store the dump anywhere.

The blobs are currently not dumped into the same stream. Dumping the blobs into the stream has the advantage that the dump will not contain $BLOBDIR filenames at all, so there are no assumptions about whether the blobs are stored inside the database or in the filesystem. We can rethink it if we decide to store the blobs into the database.

Note that in bencode dictionary keys are stored in lexicographical order. This makes it very easy to parse structures stored as dictionaries, because you either know the next key if the key is required or only need one token of lookahead (peek() operation on the token stream) if they key is optional.

[dignifiedquire]

These format calls are pretty expensive, definitely would try to avoid them.

[link2xt]

My 208M database serializes to 149M bencode dump, so even without compression replacing the database with a dump is an improvement. Additional 50M are likely all the indices that can be recreated from the data.

[Hocuri]

I'm a bit worried that people will use this function for writing without calling write_lock(). For now, I couldn't come up with a better idea to make this safer than a doc comment - maybe a doc comment is fine / the best we can do.

[Hocuri]

This probably needs to call self.write_lock().

[Hocuri]

Since you're holding the Transaction over an await point, the future returned by the function will not be Send.

JsonRPC requires all futures to be Send, so this:

diff --git a/deltachat-jsonrpc/src/api/mod.rs b/deltachat-jsonrpc/src/api/mod.rs
index b71718299..88f19e504 100644
--- a/deltachat-jsonrpc/src/api/mod.rs
+++ b/deltachat-jsonrpc/src/api/mod.rs
@@ -146,6 +146,16 @@ async fn inner_get_backup_qr(&self, account_id: u32) -> Result<Qr> {
 
 #[rpc(all_positional, ts_outdir = "typescript/generated")]
 impl CommandApi {
+    async fn t(&self) {
+        let mut w = Vec::new();
+        self.get_context(0)
+            .await
+            .unwrap()
+            .sql
+            .serialize(&mut w)
+            .await;
+    }
+
     /// Test function.
     async fn sleep(&self, delay: f64) {
         tokio::time::sleep(std::time::Duration::from_secs_f64(delay)).await
diff --git a/src/context.rs b/src/context.rs
index 627e63c78..b2bf48129 100644
--- a/src/context.rs
+++ b/src/context.rs
@@ -188,7 +188,7 @@ fn deref(&self) -> &Self::Target {
 pub struct InnerContext {
     /// Blob directory path
     pub(crate) blobdir: PathBuf,
-    pub(crate) sql: Sql,
+    pub sql: Sql,
     pub(crate) smeared_timestamp: SmearedTimestamp,
     /// The global "ongoing" process state.
     ///

will give you a vey long and unreadable error.

https://docs.rs/tokio/latest/tokio/task/struct.LocalSet.html#use-with-run_until looks like it might make it possible to run a non-send function from JsonRPC, but I didn't research further for now. Maybe you know more about this, anyway.

[Hocuri]

So, we'll have to update serializing and deserializing everytime we change the db scheme, and (de)serializing only works between the same core version. We should make sure that everyone is OKish with this trade-off before merging.