Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimize blobstore gc when current stat total size is 0 #5015

Merged
merged 9 commits into from
May 27, 2022
Merged

Optimize blobstore gc when current stat total size is 0 #5015

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
0b6139b
Optimize blobstore gc when stat totalsize is 0
jiaqizho May 27, 2022
2a1d161
Update dbms/src/Storages/Page/V3/BlobStore.cpp
jiaqizho May 27, 2022
51401f1
Update dbms/src/Storages/Page/V3/BlobStore.cpp
jiaqizho May 27, 2022
581d09f
Update dbms/src/Storages/Page/V3/BlobStore.h
jiaqizho May 27, 2022
325519a
Update dbms/src/Storages/Page/V3/BlobStore.cpp
jiaqizho May 27, 2022
d8fcb78
change assert to exception
jiaqizho May 27, 2022
4c2abbb
Merge branch 'master' into fix-blob-total-size-0
ti-chi-bot May 27, 2022
2146879
Update dbms/src/Storages/Page/V3/BlobStore.cpp
flowbehappy May 27, 2022
2b55f53
Update dbms/src/Storages/Page/V3/BlobStore.cpp
jiaqizho May 27, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 17 additions & 2 deletions dbms/src/Storages/Page/V3/BlobStore.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -914,7 +914,22 @@ std::vector<BlobFileId> BlobStore::getGCStats()
// Avoid divide by zero
if (right_margin == 0)
{
LOG_FMT_TRACE(log, "Current blob is empty [blob_id={}, total size(all invalid)={}].", stat->id, stat->sm_total_size);
if (unlikely(stat->sm_valid_rate != 0))
{
throw Exception(fmt::format("Current blob is empty, but valid rate is not 0. [blob_id={}][valid_size={}][valid_rate={}]",
stat->id,
stat->sm_valid_size,
stat->sm_valid_rate));
}

LOG_FMT_TRACE(log, "Current blob is empty [blob_id={}, total size(all invalid)={}] [valid_rate={}].", stat->id, stat->sm_total_size, stat->sm_valid_rate);

// If current blob empty, the size of in disk blob may not empty
// So we need truncate current blob, and let it be reused.
auto blobfile = getBlobFile(stat->id);
LOG_FMT_TRACE(log, "Truncate empty blob file [blob_id={}] to 0.", stat->id);
blobfile->truncate(right_margin);
flowbehappy marked this conversation as resolved.
Show resolved Hide resolved
blobstore_gc_info.appendToTruncatedBlob(stat->id, stat->sm_valid_rate);
continue;
}

Expand Down Expand Up @@ -1468,7 +1483,7 @@ void BlobStore::BlobStats::BlobStat::recalculateSpaceMap()
const auto & [total_size, valid_size] = smap->getSizes();
sm_total_size = total_size;
sm_valid_size = valid_size;
sm_valid_rate = valid_size * 1.0 / total_size;
sm_valid_rate = total_size == 0 ? 0.0 : valid_size * 1.0 / total_size;
recalculateCapacity();
}

Expand Down
2 changes: 1 addition & 1 deletion dbms/src/Storages/Page/V3/BlobStore.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -107,7 +107,7 @@ class BlobStore : private Allocator<false>
UInt64 sm_max_caps = 0;
UInt64 sm_total_size = 0;
UInt64 sm_valid_size = 0;
double sm_valid_rate = 1.0;
double sm_valid_rate = 0.0;

public:
BlobStat(BlobFileId id_, SpaceMap::SpaceMapType sm_type, UInt64 sm_max_caps_, BlobStatType type_ = BlobStatType::NORMAL)
Expand Down