Improve retry logic for downloading base image - Implement a --pull-retry flag

[Crapshit]

Actual behavior
We encounter the below problem in 1 out of of 20/30 pipeline jobs.

error building image: error building stage: failed to get filesystem from image: read tcp source-ip:source-port->destination-ip:destination-port: read: connection reset by peer

Kaniko (pipeline job) is trying to download a Docker base image located in our Microsoft Azure Container Registry and here the job fails sometimes to get the filesystem (layer) from the base image.

I cannot reopen #301 or #1267, so I open a new issue. Sorry and thank you for your support.

Expected behavior
Kaniko should have a retry flag for pulling the base image mentioned in the FROM statement like for the push operation.
We had the hope that the retry flag that was introduced in version 1.5.1 will fix our issue, but the issue still exist even with the "--push-retry" flag in Kaniko version 1.5.2.

To Reproduce
Steps to reproduce the behavior:

1. Create a textfile / Dockerfile inside of a running Kaniko docker container:

/workspace # cat textfile.txt
hello

/workspace # cat Dockerfile
FROM my-azure-cr/my-repository/my-image:base
COPY textfile.txt .

2. Execute below command with "--push-retry" flag and sometimes the issue occurs

/kaniko/executor --push-retry 5 --context /workspace --dockerfile /workspace/Dockerfile --destination my-azure-cr/my-repository/my-image:debug

Additional Information

• Dockerfile:

FROM my-azure-cr/my-repository/my-image:base
COPY textfile.txt .

• Build Context
  textfile:

hello

• Kaniko Image (fully qualified with digest):

gcr.io/kaniko-project/executor debug 7def25a8de00 2 weeks ago 117MB

• We also saw that this issue happens with Docker Build command, but Docker tries per default 5 times to pull the base image with all the layser:

time="2021-03-24T14:08:36.657464548Z" level=info msg="Download failed, retrying (1/5): read tcp source-ip:source-port->destination-ip:destination-port: read: connection reset by peer"

So even the issue happens with Docker, Docker retries it 5 times and always succeed in the retries. It never failed completely.

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing
Please check if the build works in docker but not in kaniko	^--- Please check last additional information
Please check if this error is seen when you use --cache flag
Please check if your dockerfile is a multistage dockerfile