Installation: use a lower wget timeout

[lbeltrame]

wget's read-timeout option is by default set at 900 seconds, meaning that it will stop and retry if no data have been received after that amount of time.
For spotty connections (like mine, who could have guessed ;) this is actually a problem, because it means that once the connection stops receiving data, one would have to wait 15 minutes before retrying. As this can occur multiple times in a day, this means that installations can take days (3 and counting on the one I'm currently using).

A lower timeout (probabyl around 300 seconds) would probably be better.

This snippet shows the issue

--2016-07-22 11:33:02--  (try: 3)  https://s3.amazonaws.com/gemini-annotations/ExAC.r0.3.sites.vep.tidy.vcf.gz
Connecting to s3.amazonaws.com (s3.amazonaws.com)|54.231.40.26|:443... connected.
HTTP request sent, awaiting response... 206 Partial Content
Length: 3239649414 (3.0G), 3165936782 (2.9G) remaining [application/x-gzip]
Saving to: \u2018ExAC.r0.3.sites.vep.tidy.vcf.gz\u2019

 7% [+===>                                                                              ] 231,288,383  930KB/s   in 5m 39s 

2016-07-22 11:53:42 (454 KB/s) - Read error at byte 231288383/3239649414 (Success). Retrying.

--2016-07-22 11:53:45--  (try: 4)  https://s3.amazonaws.com/gemini-annotations/ExAC.r0.3.sites.vep.tidy.vcf.gz
Connecting to s3.amazonaws.com (s3.amazonaws.com)|54.231.40.26|:443... connected.
HTTP request sent, awaiting response... 206 Partial Content
Length: 3239649414 (3.0G), 3008361031 (2.8G) remaining [application/x-gzip]
Saving to: \u2018ExAC.r0.3.sites.vep.tidy.vcf.gz\u2019

 7% [+++++                                                                              ] 232,976,517 18.7KB/s   in 4m 25s 

2016-07-22 12:13:10 (6.22 KB/s) - Read error at byte 232976517/3239649414 (Success). Retrying.

As you can see, the second try had immediately network issues, but the very large timeout meant it sat there doing nothing.

For reference: https://askubuntu.com/questions/72663/how-to-make-wget-retry-download-if-speed-goes-below-certain-threshold

[chapmanb]

Luca -- thanks for digging into the download problem and offering the suggestion for the fix. Do you have a sense of if particular files are especially problematic? There are a lot of wget downloads in the pipeline so adding this to all of them would be an undertaking, plus ones like you post above actually happen in other software like gemini. If there are specific pain points, we could definitely try to address those. Would that help?

[lbeltrame]

Mainly larger files that come from S3 (perhaps due to the fact that my pipe "over the pond" is not particularly fast) or via FTP: the example showed GEMINI but I had similar issues with the "variation" datatarget files and the larger genome files.

[lpantano]

Hi @lbeltrame

is this still an issue?

[lbeltrame]

It isn't, at the moment (but mostly because my institution has upgraded the network). Closing.