packer: cache docker images

[v1v]

Motivation/summary

I initially was tempted to add a retry/sleep command to wrap the packaging stage but it looks like it might cause some other issues, such as the workspace being reused (maybe go doesn't like it) and a bit hard to maintain.

This is a bit more elegant, based on the packer-cache approach that we use for other projects.
It will cache the required docker images for the packaging stage in the CI since those docker images are pulled from the docker registry everytime a build is triggered by the master branch. It will help to speed up the builds and avoid any kind of environmental issues with 500 errors.

[2020-04-06T12:00:22.040Z] ec813748fdd2: Waiting
[2020-04-06T12:00:22.040Z] docker: Error response from daemon: received unexpected HTTP status: 500 Internal Server Error.
[2020-04-06T12:00:22.040Z] See 'docker run --help'.
[2020-04-06T12:00:22.040Z] >> golangCrossBuild: Building for darwin/amd64
[2020-04-06T12:00:22.040Z] docker: Error response from daemon: received unexpected HTTP status: 500 Internal Server Error.
[2020-04-06T12:00:22.040Z] See 'docker run --help'.
[2020-04-06T12:00:22.040Z] >> golangCrossBuild: Building for linux/386
[2020-04-06T12:00:22.040Z] Unable to find image 'docker.elastic.co/beats-dev/golang-crossbuild:1.13.9-darwin' locally

Just to clarify we do have something in place:

• https://github.com/elastic/apm-integration-testing/blob/dbe5f30317b7f2e7dc4af70cc729ab9d49382421/.ci/packer_cache.sh#L10-L16

But it's not dynamic based on the go-version.

How to test these changes

sh -x .ci/packer_cache.sh 
+ source /usr/local/bin/bash_standard_lib.sh
++ readonly CACHE_DIR=/Users/vmartinez/.git-references
++ CACHE_DIR=/Users/vmartinez/.git-references
++ declare -a cloned_git_repos
+ source ./script/common.bash
++++ dirname ./script/common.bash
+++ cd ./script
+++ pwd
++ _sdir=/Users/vmartinez/work/src/github.com/elastic/apm-server/script
+ get_go_version
++ cat /Users/vmartinez/work/src/github.com/elastic/apm-server/script/../.go-version
+ GO_VERSION=1.13.9
+ '[' -z 1.13.9 ']'
+ DOCKER_IMAGES='
docker.elastic.co/beats-dev/golang-crossbuild:1.13.9-arm
docker.elastic.co/beats-dev/golang-crossbuild:1.13.9-darwin
docker.elastic.co/beats-dev/golang-crossbuild:1.13.9-main
docker.elastic.co/beats-dev/golang-crossbuild:1.13.9-main-debian7
docker.elastic.co/beats-dev/golang-crossbuild:1.13.9-main-debian8
docker.elastic.co/beats-dev/golang-crossbuild:1.13.9-mips
docker.elastic.co/beats-dev/golang-crossbuild:1.13.9-ppc
docker.elastic.co/beats-dev/golang-crossbuild:1.13.9-s390x
golang:1.13.9
'
+ for image in '${DOCKER_IMAGES}'
+ retry 2 docker pull docker.elastic.co/beats-dev/golang-crossbuild:1.13.9-arm
+ local retries=2
+ shift
+ local count=0
+ docker pull docker.elastic.co/beats-dev/golang-crossbuild:1.13.9-arm
1.13.9-arm: Pulling from beats-dev/golang-crossbuild
c0c53f743a40: Already exists 
...

Related issues

Further details

There is an entry for this project in the infra side ansible/playbooks/group_vars/apm-ci/jenkins.yml:

git_reference_repos:
...
  - [email protected]:elastic/apm-server.git