[Bug]: ReaperImage pull doesn't use RegistryCred for ContainerRequest

[cyberbeast]

Testcontainers version

0.16.0

Using the latest Testcontainers version?

Yes

Host OS

Macos

Host arch

x86

Go version

1.19

Docker version

Client:
 Cloud integration: v1.0.29
 Version:           20.10.20
 API version:       1.41
 Go version:        go1.18.7
 Git commit:        9fdeb9c
 Built:             Tue Oct 18 18:20:35 2022
 OS/Arch:           darwin/amd64
 Context:           default
 Experimental:      true

Server: Docker Desktop 4.13.1 (90346)
 Engine:
  Version:          20.10.20
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.7
  Git commit:       03df974
  Built:            Tue Oct 18 18:18:35 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.8
  GitCommit:        9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Docker info

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc., v0.9.1)
  compose: Docker Compose (Docker Inc., v2.12.1)
  dev: Docker Dev Environments (Docker Inc., v0.0.3)
  extension: Manages Docker extensions (Docker Inc., v0.2.13)
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc., 0.6.0)
  scan: Docker Scan (Docker Inc., v0.21.0)

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 17
 Server Version: 20.10.20
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6
 runc version: v1.1.4-0-g5fd4c4d
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 5.15.49-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 7.772GiB
 Name: docker-desktop
 ID: UT6I:Z3MH:BVO7:4GHT:YRHP:MGEJ:734U:DELR:HXWX:AEIS:XLYK:VATP
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5000
  127.0.0.0/8
 Live Restore Enabled: false

What happened?

I am using a private repository hosted on AWS ECR. I have the reaper image available in this private repository. I use the aws-sdk-go-v2 to obtain the RegistryCred which I pass to my testcontainers.ContainerRequest{} struct that I use to spin up a test container.

testcontainers.GenericContainerRequest{
	ContainerRequest: testcontainers.ContainerRequest{
		RegistryCred: credentials,
		Image:        <IMAGE>,
		ReaperImage: <IMAGE>,
	},
	Started: true,
}

If I specify the reaper image in the ReaperImage field, I get the following error when starting up

2022/11/29 11:21:41 Failed to pull image: Error response from daemon: Head "<OMITTED>ryuk:0.3.4": no basic auth credentials, will retry

implying that the RegistryCred is not being used for pulling the reaper image.

I've verified that my RegistryCred is okay because if I specify the same image ...ryuk:0.3.4 in the Image field, testcontainers is able to pull the image correctly using the appropriate credentials, thus suggesting that the RegistryCred is being used for Image pulls but not for ReaperImage pulls.

Relevant log output

No response

Additional information

No response

[cyberbeast]

@mdelapenya
Reopening this issue as the observation is still happening.

Looks like the reaper image pull process doesn't return an error if it can't pull the image from a container registry.

Disabling it, and attempting to pulling a valid docker image with invalid credentials results in an error

panic: failed to start container: Error response from daemon: pull access denied for <OMITTED>/rabbitmq, repository does not exist or may require 'docker login': denied: Your Authorization Token is invalid.: failed to create container

There is no panic/error returned on an unsuccessful ReaperImage pull. That's just a side effect of the bigger problem which is that the reaper image pull doesn't use the provided RegistryCred value.

[mdelapenya]

HI @cyberbeast I have to admit that the public API exposing the reaper image at the container request level is something we must deprecate. At we explain in #549, this should be done just once, per test execution.

In the meantime, we have to deal with the current implementation and indeed, it's possible to define a custom reaper image per container.

I'm currently working on a fix, which adds a functional option to pass the registry credential to the NewReaper function. Ideally, as as discussed on Slack and in the #discussion, everything configurable should go as a functional opt.

[cyberbeast]

Hi @mdelapenya, thanks for taking a look at this. Appreciate the input on this. Yes, I think that the reaper feature could utilize a lifecycle that looks similar to the process of launching regular containers. That being said though, additionally I think we could also add better error handling around what happens if the reaper is unable to start up. From a user's point of view, maybe a user might want to continue running their tests (and the test-container spins up) or maybe a user wants execution to panic and/or halt if the reaper can't start up. Currently, if the reaper is unable to spin up (in my case it's because the location of my ryuk container image is hosted on a private repository, which testcontainers doesn't pull with the RegistryCred I specify in my ContainerRequest, but I could imagine other failure scenarios), it should provide a mechanism for the caller to handle that failure explicitly (at the very least a panic, or at most a callback function that registers at the time of creating the container request)

[mdelapenya]

Indeed. At the same time the reaper abstraction is an opinionated way in the Testcontainers ecosystem to remove resources created by them. In our docs we advice against disabling it i.e., and at the same time we should include a warning on replacing our reaper.

From that, I'd like to ask you about the motivation of having a custom reaper in a private registry. Is it doing different tasks than the public one?

[cyberbeast]

@mdelapenya So in my use case I am not using a custom reaper. I am using the standard testcontainer/ryuk:0.3.4 image only, just hosted on a private registry rather than the public one (hosted on DockerHub). DockerHub introduced rate limits and our CI system can easily hit that limit. Since our org already has a private repository we clone public images there.

rate limit is set to 100 pulls per 6 hours per IP address
Source