Add download retries to deal better with connection hiccups during build

[ghost]

This adds download retries to deal better with connection hiccups during build. I can't count how many times I got either a minor wifi hiccup, or actually one of the servers just didn't wanna cooperate with a download and the entire build fell apart. This is really annoying to happen, and so while this retry mechanism may not cover all types of downloads in p4a, at least it will add retrying for a good chunk of them!

[hackalog]

Do you ever see hiccups like this on python 2: kivy/kivy-ios#322? I get it constantly on MacOS, and proposed a more radical fix there: kivy/kivy-ios#332

[ghost]

No I'm using Python 3 and this is for legitimate internet hiccups (connection closed/timed out/...) - see also me only catching OSError, not an AttributeError. It definitely won't fix whatever you are seeing - also as a side note, your "fix" catching all errors looks somewhat radical indeed, I'm not involved with kivy-ios but it looks like it would be preferable to find a better solution

[AndreMiras]

I'm not sure this should sit in p4a code base. It's a transport layer level issue, right?
Plus it adds complexity to the code base.

[ghost]

@AndreMiras well where else would you put it? Unless urlretrieve accepts retry mechanism patches, I don't see a better place

It's a transport layer level issue, right?

Whatever the issue is, the impact is very real. I have builds aborting all the time even with my relatively stable home connection, and since a build from scratch takes 20 minutes this has a real productivity impact - and the retry code is relatively simple. IMHO it's worth it

[hackalog]

This very same code (and to some degree, issue) exists in kivy-ios as well.
(and you'll see, I've incorporated this fix into a fix I was proposing over there: kivy/kivy-ios#332)

[hackalog]

@AndreMiras I agree that it belongs somewhere else, but for a different reason.

A lot of the code in p4a was blatantly copy-and-pasted to make the original kivy-ios. (e.g. Recipe, Arch, Target). I So there's an argument to be made that both plumbing functions like download (or download-with-retry) and much of the Recipe infrastructure should be shared library code that both p4a and kivy-ios can build on top of.

Here's a radical proposal: move this kind of code (Arch, Target, Recipe, download helpers) to somewhere else and retain just the architecture-specific parts in p4a and kivy-ios.

My first thought would be put it in buildozer, since that has to interact closely with p4a and kivy anyway

[ghost]

buildozer uses p4a and not the other way round, e.g. I often use p4a without buildozer installed. So that doesn't make too much sense.

It would be nice to unite some of kivy-ios and p4a at some point, but it's really kind of beyond the scope of what I tried to do here 😂

[hackalog]

@Jonast Yes. Buildozer is a common front-end for invoking p4a and kivy-ios (and other platforms). I'm suggesting that it could just as easily be a common back-end, too. It's that, or move all the shared code to a 3rd library that all 3 draw from.

Anyway, as you point out, this is all beyond what this patch attempts to address. I think the patch is good, and I've suggested we do the same over on the kivy-ios side.

[ghost]

@AndreMiras should I move the download code from recipe.py as a whole in a separate file as part of this pull request? or is your vote to not include it at all for now due to the added complexity? or should we go ahead with it as it is?

I'm pretty open for other options, just let me know where we're heading (and I'm not dying if this isn't included in the near future, but it'd be nice for me to have for sure)

[AndreMiras]

Well it's approved now. So up to you. Do you have rights to merge it now that's approved?

[ghost]

@AndreMiras right now I can just approve/disapprove things (and with that kind of block them due my new role), since apparently there is an additional restriction for merging: it says "The base branch restricts merging to authorized users."

Which btw is okay with me really, I'm happy to help to actually merge approved things if you want me to, but I don't need to be able to do that. Entirely up to you / the team!