feat(storage): use a connection pool

[kkovaacs]

Instead of re-opening the Sqlite database for each request Storage now contains a connection pool.

We're using r2d2 with a blocking API because this was the smallest change (the rusqlite API is blocking, too, and these calls are already called from spawn_blocking() tasks). This also means that when we've reached the maximum number of connections (10 by default) getting a connection from the pool will block. This is probably better behavior than running an arbitrary number of SQL queries in parallel (which we were doing) but it still means that we're using one thread from tokio's blocking thread pool for waiting for a new connection to be available.

The load-test tool shows significantly improved performance.

• Before:
  

• After:
  

[koivunej]

Would rather see this implemented via TLS and better split threads, but I guess this is an ok crutch to get forwards.

[koivunej]

On a later PR it'd be good to align the defaults and tokio configuration we use:

• r2d2 defaults to 10 connections per docs
• I would expect that larger hosts (for ex 16 cores) will run into that with tokio's blocking thread by default limiting at 512 threads

Finding good limits does become difficult as we have multiple purposes to the thread pool currently (compressions at least). I don't know what would be a good number.

EDIT: later noticed that the max latency numbers shot up as well, though those might be sensitive to other things as well.

[kkovaacs]

I've also made some experiments with changing the pool size (I have 12 cores) and did not really seem to make a difference in my case. (We'd probably need to test on beefier HW if we have to optimize for that.)

I explicitly left all these at the default settings because I couldn't come up with other values that were clearly better.

[kkovaacs]

Re-run load tests on my laptop.

Results for main:

Results for this PR:

Report files: reports.zip