Delay bottommost level single file compactions #11701
Conversation
cbi42
force-pushed
the
delay-bottommost-simpler
branch
2 times, most recently
from
09016b8 to
d329e9b
Compare
|
@cbi42 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
|
@cbi42 has updated the pull request. You must reimport the pull request before landing. |
|
@cbi42 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
|
I guess we want to add this to the list of unreleased features. |
cbi42
force-pushed
the
delay-bottommost-simpler
branch
from
16ac331 to
6abb2cd
Compare
Added a note under "new_features". |
|
@cbi42 has updated the pull request. You must reimport the pull request before landing. |
|
@cbi42 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
cbi42
force-pushed
the
delay-bottommost-simpler
branch
from
6abb2cd to
d8d5fa9
Compare
|
@cbi42 has updated the pull request. You must reimport the pull request before landing. |
|
@cbi42 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
For leveled compaction, RocksDB has a special kind of compaction with reason "kBottommmostFiles" that compacts bottommost level files to clear data held by snapshots (more detail in #3009). Such compactions can happen soon after a relevant snapshot is released. For some use cases, a bottommost file may contain only a small amount of keys that can be cleared, so compacting such a file has a high write amp. In addition, these bottommost files may be compacted in compactions with reason other than "kBottommmostFiles" if we wait for some time (so that enough data is ingested to trigger such a compaction). This PR introduces an option
bottommost_file_compaction_delayto specify the delay of these bottommost level single file compactions.VersionStorageInfo::ComputeBottommostFilesMarkedForCompaction()where we only add a file tobottommost_files_marked_for_compaction_if it oldest_snapshot is larger than its non-zero largest_seqno and the file is old enough. Note that if a file is not old enough but its largest_seqno is less than oldest_snapshot, we exclude it from the calculation ofbottommost_files_mark_threshold_. This makes the change simpler, but such a file's eligibility for compaction will only be checked the next timeComputeBottommostFilesMarkedForCompaction()is called. This happens when a new Version is created (compaction, flush, SetOptions()...), a new enough snapshot is released (VersionStorageInfo::UpdateOldestSnapshot()) or when a compaction is picked and compaction score has to be re-calculated.Test plan: