Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(server): dont use channel for replication / save df #4041

Merged
merged 7 commits into from
Nov 5, 2024

Conversation

adiholden
Copy link
Collaborator

fixes #4040

src/server/rdb_save.h Outdated Show resolved Hide resolved
src/server/rdb_save.h Outdated Show resolved Hide resolved
src/server/rdb_save.cc Show resolved Hide resolved
src/server/rdb_save.cc Outdated Show resolved Hide resolved
src/server/rdb_save.cc Outdated Show resolved Hide resolved
src/server/rdb_save.cc Show resolved Hide resolved
}

void RdbSaver::Impl::StartIncrementalSnapshotting(Context* cntx, EngineShard* shard,
LSN start_lsn) {
auto& db_slice = namespaces.GetDefaultNamespace().GetDbSlice(shard->shard_id());
auto& s = GetSnapshot(shard);
s = std::make_unique<SliceSnapshot>(&db_slice, &channel_, compression_mode_);
auto on_finalize_cb = std::bind(&RdbSaver::Impl::FinalizeSnapshotWriting, this);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's possible to do like this. not critical but when you pass two lambdas, maybe it's a sign that using an virtual interface is a better pattern here, and make RdbSaver::Impl derive from it.

src/server/snapshot.h Outdated Show resolved Hide resolved
src/server/rdb_save.cc Outdated Show resolved Hide resolved
src/server/rdb_save.cc Outdated Show resolved Hide resolved
src/server/rdb_save.cc Outdated Show resolved Hide resolved
@adiholden adiholden requested a review from romange November 4, 2024 11:07
void SaveStagesController::SaveCb(unsigned index) {
if (auto& snapshot = snapshots_[index].first; snapshot && snapshot->HasStarted())
void SaveStagesController::SaveBody(unsigned index) {
CHECK(!use_dfs_format_ || index == shard_set->size());
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this CHECK is weird index == shard_set->size()

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we use SaveBody for rdb save and for summary file in df save.
We can do several refactoring here for more readable code but I want this pr to be not so big

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added a comment for now

@@ -85,7 +76,7 @@ class SliceSnapshot {

// Waits for a regular, non journal snapshot to finish.
// Called only for non-replication, backups usecases.
void Join() {
void WaitSnapshoting() {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WaitSnapshoting -> WaitSnapshotting
please check if you have more shoting cases

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Copy link
Collaborator

@romange romange left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

I think it will be more readable if we deduplicate some of the RdbSerializer functions so that we won't have two sets of functions - one for replication and one for snapshotting.

@adiholden
Copy link
Collaborator Author

LGTM

I think it will be more readable if we deduplicate some of the RdbSerializer functions so that we won't have two sets of functions - one for replication and one for snapshotting.

I will do addittional refactoring changes in a separate PR ok?

@romange
Copy link
Collaborator

romange commented Nov 5, 2024

Yeah sure, it was a general comment 👌

romange
romange previously approved these changes Nov 5, 2024
adiholden and others added 7 commits November 5, 2024 13:48
Signed-off-by: adi_holden <[email protected]>
Signed-off-by: adi_holden <[email protected]>
Co-authored-by: Roman Gershman <[email protected]>
Signed-off-by: adiholden <[email protected]>
Co-authored-by: Roman Gershman <[email protected]>
Signed-off-by: adiholden <[email protected]>
Co-authored-by: Roman Gershman <[email protected]>
Signed-off-by: adiholden <[email protected]>
Signed-off-by: adi_holden <[email protected]>
@adiholden adiholden force-pushed the snapshot_write_to_sink branch from 549ea37 to 5c14022 Compare November 5, 2024 12:16
@adiholden adiholden requested a review from romange November 5, 2024 12:57
@adiholden adiholden merged commit ae3faf5 into main Nov 5, 2024
13 checks passed
@adiholden adiholden deleted the snapshot_write_to_sink branch November 5, 2024 14:50
kostasrim pushed a commit that referenced this pull request Nov 8, 2024
* feat server: dont use channel for replication / save df

Signed-off-by: adi_holden <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Pass io::Sink to snapshot in case we perform DFS/Replication snapshotting, and give up on the channel.
2 participants