Flush cache of KVStore and DeltaTree in background

[flowbehappy]

Enhancement

Currently, the write throughput of TiFlash is limited. The worse case: most of the transactions only update a few rows randomly amount many regions. In this case, most of the flush only flush very few rows, including KVStore persist and DeltaTree flush cache. Check KVStore::tryFlushRegionData for details.

The flush requests are mostly from CompactLog in raft logs. Currently, we use flash.compact_log_min_period to control the execution frequency of CompactLog. But there are still too many flush and cause serious write performance issue, especially when there are many regions and the total CompactLogs are very frequent. In my tests, with over 300GB of data, a TiFlash node can only support ~2.5MiB/s write throughput with the worst write pattern.

I think the ultimate solution is doing tryFlushRegionData by background threads. So foreground apply raft logs process completely ignores CompactLog commands. And the background flush threads notify raft store after they finished the flush jobs, to do the following tasks, including removing raft logs.

This optimization is expected to increase the max write throughput of TiFlash to at least 300% in the worse case. I.e. from 2.5 MiB/s to 7.5MiB/s.

[flowbehappy]

The suitable flush ops of a TiFlash node should be less than 100 ops. Otherwise, there is the risk of write delay.

[JinheLin]

Duplicated with #7564.