Re-try zstd compression in sled, after the data format has stabilised

[teor2345]

Sled has a zstd compression feature, should we use it?

The compression level is configurable. Since Zebra is CPU-bound, let's use a low compression level. A bunch of Zcash data is pseudo-random hashes or nonces, so we'll get the most benefit from compressing blocks of zeroes.

Pros:

• lower disk space usage
• faster loading from disk, if disk latency is slower than compression CPU usage

Cons:

• extra CPU usage
• an extra dependency
• using non-default features of sled, which may have less testing

TODO:

• stabilise the data format
• rebase PR feature: Use zstd compression with sled #790 on main
• if secp256k1 is on 0.17.3 or later, remove the Cargo.toml override
• test compressed and uncompressed sizes on:
  • mainnet
  • testnet

[hdevalence]

I think this is a reasonable choice to make, and it's low-cost if we decide to change our mind later (we already don't commit to a stable on-disk format).

[teor2345]

I'll go make a PR and do some testing.

Note: I don't know if sled can transparently change from zstd to non-zstd databases, it's not documented. But if we drop the zstd feature, it definitely won't be able to.

[teor2345]

I opened PR #790, using zstd compression level 1.

As of 29 July, I had a fully synced testnet data directory which took up 12 GB, at around block 1021029. I'll do some testing and let you know what the compression ratio is.

[teor2345]

Early test results: you can't change the compression of an existing sled database.

[teor2345]

As of 29 July, I had a fully synced testnet data directory which took up 12 GB, at around block 1021029. I'll do some testing and let you know what the compression ratio is.

16 GB at around block 1021473.

I'm not sure what sled compression is doing, but it's definitely not making our state smaller. (Or our state sizes are extremely variable for each run.)

I'm going to re-test with and without compression to make sure.

[hdevalence]

Did we already settle on the data layout we want to use? Since the compression is downstream of that, it might make sense to defer this decision until later.

[teor2345]

I think the data layout will depend on what we decide to store on disk, which depends (in part) on the parallel verification RFC #763.

[teor2345]

(I've confirmed that compressed data sizes are bigger right now, I'll defer this ticket until after the first alpha.)

[mpguerra]

we should close this one if we decide to go for rocksdb in the end

[teor2345]

We don't think compression is ever going to work well for sled, because we store lots of small keys.

RocksDB does compression at a block (database page) level, so it might work much better.

[teor2345]

I opened #1326 for RocksDB's compaction and compression.