filter deletion in compaction filter #344

tabokie · 2023-08-01T12:09:37Z

And delay the buffer initialization of writable file to first actual write.

Signed-off-by: tabokie <[email protected]>

overvenus · 2023-08-04T09:41:55Z

file/writable_file_writer.h

@@ -195,7 +195,8 @@ class WritableFileWriter {
    TEST_SYNC_POINT_CALLBACK("WritableFileWriter::WritableFileWriter:0",
                             reinterpret_cast<void*>(max_buffer_size_));
    buf_.Alignment(writable_file_->GetRequiredBufferAlignment());
-    buf_.AllocateNewBuffer(std::min((size_t)65536, max_buffer_size_));
+    // Removed to reduce WAL writer memory usage.
+    // buf_.AllocateNewBuffer(std::min((size_t)65536, max_buffer_size_));


Does it affect raftstore v1？

Maybe we can keep the code and use a smaller max_buffer_size in raftstore v2.

The allocation is delayed to actual write, which I think is good for both v1 and v2. Because the initialization is also on critical path, moving it elsewhere amortizes the overhead.

I see. How about skipping alloc is max_buffer_size_ == 0?

Suggested change

// buf_.AllocateNewBuffer(std::min((size_t)65536, max_buffer_size_));

if max_buffer_size_ != 0 {

buf_.AllocateNewBuffer(std::min((size_t)65536, max_buffer_size_));

}

/cc tikv/tikv#15275

Oh, I forgot this flag works for both WAL and SST. So we can't simply change the config.

Signed-off-by: tabokie <[email protected]>

overvenus

Rest LGTM

overvenus · 2023-08-07T05:47:55Z

file/writable_file_writer.h

@@ -195,7 +195,8 @@ class WritableFileWriter {
    TEST_SYNC_POINT_CALLBACK("WritableFileWriter::WritableFileWriter:0",
                             reinterpret_cast<void*>(max_buffer_size_));
    buf_.Alignment(writable_file_->GetRequiredBufferAlignment());
-    buf_.AllocateNewBuffer(std::min((size_t)65536, max_buffer_size_));
+    // Removed to reduce WAL writer memory usage.
+    // buf_.AllocateNewBuffer(std::min((size_t)65536, max_buffer_size_));


I see. How about skipping alloc is max_buffer_size_ == 0?

Suggested change

// buf_.AllocateNewBuffer(std::min((size_t)65536, max_buffer_size_));

if max_buffer_size_ != 0 {

buf_.AllocateNewBuffer(std::min((size_t)65536, max_buffer_size_));

}

overvenus · 2023-08-07T05:51:08Z

include/rocksdb/compaction_filter.h

@@ -212,6 +217,21 @@ class CompactionFilter : public Customizable {
                    skip_until);
  }

+  // This interface is reserved for TiKV's region range filter. Only this
+  // interface can accept `value_type=kTypeDeletion`.
+  virtual Decision UnsafeFilter(int level, const Slice& key,


Is is possible to add a test case to show its usage?

Signed-off-by: tabokie <[email protected]>

This reverts commit ba85ffa. Signed-off-by: tabokie <[email protected]>

Signed-off-by: tabokie <[email protected]>

And delay the buffer initialization of writable file to first actual write. --------- Signed-off-by: tabokie <[email protected]>

compaction_filter: add bottommost_level into context (tikv#160) Signed-off-by: qupeng <[email protected]> Signed-off-by: tabokie <[email protected]> add range for compaction filter context (tikv#192) * add range for compaction filter context Signed-off-by: qupeng <[email protected]> Signed-off-by: tabokie <[email protected]> allow no_io for VersionSet::GetTableProperties (tikv#211) * allow no_io for VersionSet::GetTableProperties Signed-off-by: qupeng <[email protected]> Signed-off-by: tabokie <[email protected]> expose seqno from compaction filter and iterator (tikv#215) This PR supports to access `seqno` for every key/value pairs in compaction filter or iterator. It's helpful to enhance GC in compaction filter in TiKV. Signed-off-by: qupeng <[email protected]> Signed-off-by: tabokie <[email protected]> allow to query DB stall status (tikv#226) This PR adds a new property is-write-stalled to query whether the column family is in write stall or not. In TiKV there is a compaction filter used for GC, in which DB::write is called. So if we can query whether the DB instance is stalled or not, we can skip to create more compaction filter instances to save some resources. Signed-off-by: qupeng <[email protected]> Signed-off-by: tabokie <[email protected]> Fix compatibilty issue with Titan Signed-off-by: v01dstar <[email protected]> filter deletion in compaction filter (tikv#344) And delay the buffer initialization of writable file to first actual write. --------- Signed-off-by: tabokie <[email protected]> Adjustments for compaptibilty with 8.10.facebook Signed-off-by: v01dstar <[email protected]> Adjust tikv related changes with upstream Signed-off-by: v01dstar <[email protected]>

And delay the buffer initialization of writable file to first actual write. --------- Signed-off-by: tabokie <[email protected]>

And delay the buffer initialization of writable file to first actual write. --------- Signed-off-by: tabokie <[email protected]> Signed-off-by: Yang Zhang <[email protected]>

tabokie requested review from overvenus and tonyxuqqi August 1, 2023 12:09

tabokie mentioned this pull request Aug 1, 2023

raftstore-v2: fix error handling during merge trim check tikv/tikv#15243

Merged

filter deletion in compaction filter

66dcaa3

Signed-off-by: tabokie <[email protected]>

tabokie force-pushed the fix-compact-range branch from 5632fe8 to 66dcaa3 Compare August 1, 2023 12:11

tabokie added 2 commits August 2, 2023 03:28

fix build

289617b

Signed-off-by: tabokie <[email protected]>

remove writer buffer initial allocation

02828a8

Signed-off-by: tabokie <[email protected]>

tonyxuqqi approved these changes Aug 4, 2023

View reviewed changes

overvenus reviewed Aug 4, 2023

View reviewed changes

fix test

26defc0

Signed-off-by: tabokie <[email protected]>

overvenus reviewed Aug 7, 2023

View reviewed changes

address comment

ba85ffa

Signed-off-by: tabokie <[email protected]>

overvenus approved these changes Aug 7, 2023

View reviewed changes

tabokie added 3 commits August 7, 2023 10:00

add test

a71c6d2

Signed-off-by: tabokie <[email protected]>

Revert "address comment"

0edc5e6

This reverts commit ba85ffa. Signed-off-by: tabokie <[email protected]>

fix build

b39844b

Signed-off-by: tabokie <[email protected]>

tabokie merged commit de47e8e into tikv:6.29.tikv Aug 7, 2023

tabokie deleted the fix-compact-range branch August 7, 2023 10:03

ti-chi-bot mentioned this pull request Aug 10, 2023

raftstore-v2: fix error handling during merge trim check (#15243) tikv/tikv#15308

Closed

overvenus mentioned this pull request Aug 15, 2023

[Dynamic Regions] RocksDB WritableFileWriter takes too many memory during DBImpl::RecoverLogFiles tikv/tikv#15275

Closed

Connor1996 mentioned this pull request Nov 28, 2023

Change compaction filter interface tikv/titan#290

Merged

v01dstar pushed a commit to v01dstar/rocksdb that referenced this pull request Feb 21, 2024

filter deletion in compaction filter (tikv#344)

af73f3c

And delay the buffer initialization of writable file to first actual write. --------- Signed-off-by: tabokie <[email protected]>

v01dstar pushed a commit to v01dstar/rocksdb that referenced this pull request May 29, 2024

filter deletion in compaction filter (tikv#344)

bb9cb78

And delay the buffer initialization of writable file to first actual write. --------- Signed-off-by: tabokie <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

filter deletion in compaction filter #344

filter deletion in compaction filter #344

tabokie commented Aug 1, 2023 •

edited

Loading

overvenus Aug 4, 2023

overvenus Aug 4, 2023

tabokie Aug 4, 2023

overvenus Aug 7, 2023

overvenus Aug 7, 2023

tabokie Aug 7, 2023 •

edited

Loading

overvenus left a comment

overvenus Aug 7, 2023

overvenus Aug 7, 2023

filter deletion in compaction filter #344

filter deletion in compaction filter #344

Conversation

tabokie commented Aug 1, 2023 • edited Loading

overvenus Aug 4, 2023

Choose a reason for hiding this comment

overvenus Aug 4, 2023

Choose a reason for hiding this comment

tabokie Aug 4, 2023

Choose a reason for hiding this comment

overvenus Aug 7, 2023

Choose a reason for hiding this comment

overvenus Aug 7, 2023

Choose a reason for hiding this comment

tabokie Aug 7, 2023 • edited Loading

Choose a reason for hiding this comment

overvenus left a comment

Choose a reason for hiding this comment

overvenus Aug 7, 2023

Choose a reason for hiding this comment

overvenus Aug 7, 2023

Choose a reason for hiding this comment

tabokie commented Aug 1, 2023 •

edited

Loading

tabokie Aug 7, 2023 •

edited

Loading