Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

add progress logs during snapshot loading #395

Merged
merged 1 commit into from
Jul 24, 2024
Merged

add progress logs during snapshot loading #395

merged 1 commit into from
Jul 24, 2024

Conversation

spoonincode
Copy link
Member

Adds progress logs during snapshot loading; can look something like

info  2024-07-24T03:30:21.700 nodeos    controller.cpp:1831           startup              ] Starting initialization from snapshot and no block log, this may take a significant amount of time
info  2024-07-24T03:30:26.008 nodeos    snapshot.hpp:418              progress             ] Snapshot initialization 2% complete
info  2024-07-24T03:30:31.001 nodeos    snapshot.hpp:418              progress             ] Snapshot initialization 4% complete
info  2024-07-24T03:30:36.003 nodeos    snapshot.hpp:418              progress             ] Snapshot initialization 17% complete
info  2024-07-24T03:30:41.008 nodeos    snapshot.hpp:418              progress             ] Snapshot initialization 28% complete
...
info  2024-07-24T03:31:31.029 nodeos    snapshot.hpp:418              progress             ] Snapshot initialization 90% complete
info  2024-07-24T03:31:36.009 nodeos    snapshot.hpp:418              progress             ] Snapshot initialization 92% complete
info  2024-07-24T03:31:41.019 nodeos    snapshot.hpp:418              progress             ] Snapshot initialization 93% complete
info  2024-07-24T03:31:46.035 nodeos    snapshot.hpp:418              progress             ] Snapshot initialization 96% complete
info  2024-07-24T03:31:51.017 nodeos    snapshot.hpp:418              progress             ] Snapshot initialization 99% complete
info  2024-07-24T03:31:53.283 nodeos    controller.cpp:1838           startup              ] Snapshot loaded, lib: 310859547

Resolves AntelopeIO/leap#1167

spoonincode
spoonincode commented Jul 24, 2024
View reviewed changes
unittests/snapshot_tests.cpp
@@ -61,6 +61,7 @@ class snapshotted_tester : public base_tester {
controller::config copied_config = (copy_files_from_config == copy_config_files)
? copy_config_and_files(config, ordinal) : copy_config(config, ordinal);

BOOST_CHECK_GT(snapshot->total_row_count(), 0);
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is good enough to make sure all three impls of total_row_count() are sane (and indeed I found my rapidjson impl was incorrect initially). I don't think we need to actually make sure the exact count is accurate and precise.

spoonincode
spoonincode commented Jul 24, 2024
View reviewed changes
libraries/chain/include/eosio/chain/snapshot.hpp
size_t count = 0;
const size_t total = 0;
time_t last_print = time(NULL);
};
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a little tempting to move this completely inside of snapshot_reader so that maybe the onus to call progress() remains entirely inside of the snapshot_reaader & friends (which would probably help ensuring there isn't missed calls). But snapshot_reader's interface technically allows you to read the same section multiple times (even though nobody does that..) so, eh.. not a perfect match.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I could see a desire for a user to provide their own version of snapshot_loaded_row_counter say for diff logging purposes. Keeping it external would make that easier. I don't think it worth making it a template or derivable type at this point until that is needed.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea buuuutt.. this all probably needs to be refactored again when making loading parallel so that might not be possible. I didn't mention that this may be somewhat temporary since I'm not committing to parallel loading just yet..

heifner
heifner approved these changes Jul 24, 2024
View reviewed changes
libraries/chain/include/eosio/chain/snapshot.hpp
size_t count = 0;
const size_t total = 0;
time_t last_print = time(NULL);
};
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I could see a desire for a user to provide their own version of snapshot_loaded_row_counter say for diff logging purposes. Keeping it external would make that easier. I don't think it worth making it a template or derivable type at this point until that is needed.

@spoonincode spoonincode merged commit f344e2b into main Jul 24, 2024
36 checks passed
@spoonincode spoonincode deleted the snap_load_progress_logs branch July 24, 2024 15:24
@ericpassmore
Copy link
Contributor

Note:start
group: LOGGING
category: INTERNALS
summary: Log progress during snapshot loading.
Note: end

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@greg7mdp greg7mdp greg7mdp approved these changes

@heifner heifner heifner approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

When nodeos is starting up from a snapshot print more logs
4 participants
@spoonincode @ericpassmore @greg7mdp @heifner