Download only new hash files #242
Conversation
Checks the specified directory for previously-downloaded hash files and only downloads new ones, conserving virusshare's bandwidth.
check for existing hashfile downloads
|
Hey @ktneely, can you base your PR on the development branch? That would be cool. Thanks for your contribution. |
3c7
added
category:enhancement
|
@3c7 I've rebased the PR on |
| do | ||
| echo $u | ||
| wget https://virusshare.com/$u | ||
| if [ -e $1/$u ]; then |
There was a problem hiding this comment.
As you've disabled commits for members, I cannot change that line. You have to delete $1/ as the scripts cds into the directory before.
There was a problem hiding this comment.
The script does cd into $1/, but when I'm checking the file names, the update performs two cut commands, lopping off the directory assigned to $1. A little convoluted, but it makes the comparison easier.
There was a problem hiding this comment.
That's not what I wanted to say. You cutsomething off, that's right but it only affects the filename, after that you still have that commands:
cd $1if [ -e $1/$u ]
That leads to "being in" directory $1 and checks if there's an directory $1 with file $u in it. Or I'm completely blind and oversee something. :D
There was a problem hiding this comment.
prior to the cd, the script cuts the full path twice, making u only the file name as pulled from the virusshare website, which it then compares to the filenames in directory $1.
See, e.g. | cut -d\" -f2 | cut -d\/ -f2 in the update as opposed to only cut -d\" -f2 in the original.
The first cuts down to e.g. "hashes/VirusShare_00049.md5", the second lops off the directory on the hosting server and leaves only e.g. "VirusShare_00049.md5", which is then compared to the filename in $1.
Modified the download script so it only downloads newly added hash files from virusshare, in order to conserve bandwidth for both sides.