Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: run memory decommit after snapshot load/save #3828

Merged
merged 6 commits into from
Oct 6, 2024
Merged

chore: run memory decommit after snapshot load/save #3828

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
736131b
chore: run memory decommit after snapshot load/save
kostasrim Sep 30, 2024
02e1b76
small fixes
kostasrim Sep 30, 2024
0fcaad2
add kAll
kostasrim Sep 30, 2024
63276aa
comments
kostasrim Oct 1, 2024
0765ffe
refactor tests
kostasrim Oct 1, 2024
1fa33b6
fix: destroy shard flows on replica stop
kostasrim Oct 2, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 2 additions & 4 deletions src/server/memory_cmd.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -144,10 +144,8 @@ void MemoryCmd::Run(CmdArgList args) {
}

if (sub_cmd == "DECOMMIT") {
shard_set->pool()->AwaitBrief([](unsigned, auto* pb) {
ServerState::tlocal()->DecommitMemory(ServerState::kDataHeap | ServerState::kBackingHeap |
ServerState::kGlibcmalloc);
});
shard_set->pool()->AwaitBrief(
[](unsigned, auto* pb) { ServerState::tlocal()->DecommitMemory(ServerState::kAll); });
return cntx_->SendSimpleString("OK");
}

Expand Down
8 changes: 8 additions & 0 deletions src/server/rdb_load.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1991,6 +1991,14 @@ RdbLoader::~RdbLoader() {
break;
delete item;
}

// Decommit local memory.
// We create an RdbLoader for each thread, so each one will Decommit for itself after
// full sync ends (since we explicitly reset the RdbLoader).
auto* tlocal = ServerState::tlocal();
if (tlocal) {
tlocal->DecommitMemory(ServerState::kAll);
}
}

error_code RdbLoader::Load(io::Source* src) {
Expand Down
6 changes: 6 additions & 0 deletions src/server/rdb_save.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1406,6 +1406,12 @@ RdbSaver::RdbSaver(::io::Sink* sink, SaveMode save_mode, bool align_writes) {
}

RdbSaver::~RdbSaver() {
// Decommit local memory.
// We create an RdbSaver for each thread, so each one will Decommit for itself.
auto* tlocal = ServerState::tlocal();
if (tlocal) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

when is this null?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tests 😄

For example LoadEmpty in rdb_test.cc, does not run RdbLoad::Load in a shard thread so it crashes.

I do not know if a similar unit test exist for rdb_save but I assumed this is the case and I added this here as well to avoid crashes like that

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that odd because rdb_save calls service init. please check this again why it is not set

Copy link
Contributor Author

@kostasrim kostasrim Oct 1, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see why this is odd. The service is created but we just don't run the rdb loader on a proactor thread

ServerState::tlocal()->DecommitMemory(ServerState::kAll);
}
}

void RdbSaver::StartSnapshotInShard(bool stream_journal, const Cancellation* cll,
Expand Down
7 changes: 6 additions & 1 deletion src/server/server_state.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -282,7 +282,12 @@ class ServerState { // public struct - to allow initialization.
// Decommits 3 possible heaps according to the flags.
// For decommit_glibcmalloc the heap is global for the process, for others it's specific only
// for this thread.
enum { kDataHeap = 1, kBackingHeap = 2, kGlibcmalloc = 4 };
enum {
kDataHeap = 1,
kBackingHeap = 2,
kGlibcmalloc = 4,
kAll = kDataHeap | kBackingHeap | kGlibcmalloc
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this is an anonymous enum, I'd use a more verbose name. Maybe kAllHeaps or kAllMemory?
Alternatively we can name this enum (preferably use enum class) so that it will not be confusing.
Anyway, this is a nit so feel free to ignore.

};
void DecommitMemory(uint8_t flags);

// Exec descriptor frequency count for this thread.
Expand Down
Loading