Made imdb download (data_imdb) function atomic #14225
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
alamb
approved these changes
Jan 23, 2025
There was a problem hiding this comment.
Thank you @Spaarsh
I tried this out locally and found that now the conversion code was not triggered.
I pushed a fix to your branch locally (moving where the fi was) as this PR has been hanging out for a while.
:~/Software/datafusion/benchmarks$ ./bench.sh data imdb
***************************
DataFusion Benchmark Runner and Data Generator
COMMAND: data
BENCHMARK: imdb
DATA_DIR: /Users/andrewlamb/Software/datafusion/benchmarks/data
CARGO_COMMAND: cargo run --release
PREFER_HASH_JOIN: true
***************************
Looking for imdb.tgz... found
Checking size... OK 1263193115
andrewlamb@Andrews-MacBook-Pro-2:~/Software/datafusion/benchmarks$ echo "foo" > data/imdb/imdb.tgz
andrewlamb@Andrews-MacBook-Pro-2:~/Software/datafusion/benchmarks$ ./bench.sh data imdb
***************************
DataFusion Benchmark Runner and Data Generator
COMMAND: data
BENCHMARK: imdb
DATA_DIR: /Users/andrewlamb/Software/datafusion/benchmarks/data
CARGO_COMMAND: cargo run --release
PREFER_HASH_JOIN: true
***************************
Looking for imdb.tgz... found
Checking size... MISMATCH
Size less than expected: 4 found, 1263193115 required
Downloading IMDB dataset...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
Oh thanks! Sorry for the oversight. I will be more careful from now on. Thanks again! You've been extremely patient!! |
No worries -- I am sorry we have had so many back and forths. I think this is looking nice now and thank you again |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #14128
Rationale for this change
Due to non-atomic downloads, the user would need to manually remove files/folders created by the script in order to not encounter the "file already exists" error.
What changes are included in this PR?
The changes primarily focus on adding size checks before initiating a download and after the completion of a download. If the size is mismatched, the partially downloaded file is then removed.
Are these changes tested?
Yes, these changes have been tested. The following are the results:
For the
benchmarks/bench.sh data imdbcommand:Are there any user-facing changes?
Only that users shall now see if the imdb.tgz has already been installed, its status (partial/full) and if there is a size mismatch. The user shall also be informed to re-run the download command in case of partial download despite no user-side interruptions.