Running Testcontainers with Rancher Desktop and Apple Silicon fails

[therealjsie]

Actual Behavior

Running a test using Testcontainers on a MacBook Pro M1 leads to exceptions and a failed test with Rancher Desktop. Using the same setup with Docker Desktop does not result in a failed test.

Steps to Reproduce

1. Use a 2021 MacBook Pro with M1 Chip.
2. Check out the repo: https://github.com/therealjsie/rdtc-test
3. Run mvn verify with Rancher Desktop (Kubernetes disabled, using dockerd runtime)
4. Should result in a failed test.

Result

Failing logs with Rancher Desktop: repo-postgres-simple-test-macos-m1.log

Expected Behavior

Successful logs with Docker Desktop: repo-postgres-simple-test-macos-m1-docker-desktop.log

Additional Information

No response

Rancher Desktop Version

1.4.1

Rancher Desktop K8s Version

disabled

Which container runtime are you using?

moby (docker cli)

What operating system are you using?

macOS

Operating System / Build Version

macOs Monterey 12.4

What CPU architecture are you using?

arm64 (Apple Silicon)

Linux only: what package format did you use to install Rancher Desktop?

N/A

Windows User Only

No response

[INeedMySpace]

I have the similar issues with flaky tests running on my M1 MacBook Pro (macOS Monterey 12.5) / Rancher Desktop 1.5.0 / testcontainers 1.17.3. Running well with Docker Desktop.

[ZombieSocrates]

Same as the prior commenter, I believe I experienced these issues with tests running on an M1 MacBook Pro (macOS Monterey 12.3)/ Rancher Desktop 1.5.0/ and two separate versions of testcontainers, 1.15.3 and 1.17.1.

As soon as I installed Docker Desktop, I could run the tests I needed to ( via Gradle, if that's important) on both of the above testcontainers.

[edeandrea]

Same here - quarkusio/quarkus#25682

[overheadhunter]

I don't think this issue is related to the CPU architecture. I'm experiencing the same problem (well the linked Quarkus Issue) on an Intel Mac.

[adroste]

Observation:
If I disable RYUK with export TESTCONTAINERS_RYUK_DISABLED="true" it works like 3 out of 4 times on my M1 MBP.
Still, not a solution.

[resilva87]

Things have worked out for me when I enabled the option Allow Rancher Desktop to acquire administrative credentials (sudo access) in Preferences > Application > Behavior menu

[adroste]

@resilva87 Isn't that the default mode? (It is for me)

[resilva87]

@adroste For me at least it was not enabled. I'm running Rancher Desktop v1.6.1

[apara]

I am running Rancher Desktop v1.6.2 and still have "Connection Refused" errors when talking to TestContainers embedded Postgres. Docker Desktop seems to work fine, but errors persist with Rancher Desktop. The test mentioned here re-creates the problem well.

Here is the specific error I am seeing in my tests:

java.sql.SQLException: org.postgresql.util.PSQLException: Connection to localhost:49312 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
Caused by: org.postgresql.util.PSQLException: Connection to localhost:49312 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
Caused by: java.net.ConnectException: Connection refused

Oddly, when I manually test the connection while images are running, I am able to connect from Command Prompt.

[mobiuscog]

Maybe this helps some others, but looking at a similar problem, I found this to work with the provided example under Colima which exhibited the same problem, adding in a wait strategy to ensure the external port was mapped:

                .withPassword("PWD")
                .waitingFor(Wait.forListeningPort()))

[NagarajuKoppu]

Hi, I have similar problem with "MAC-Rancher Desktop-Test Container" combination. The intermittent exceptions of "Connection Failures" are the biggest challenge. Below changes would solve your problems.

1. Rancher-desktop would remove "localhost" from hosts configuration in MAC. So, we need to use "127.0.0.1" in the container URL's
2. Make use of "Startables" utility from test containers rather waiting for listener ports before stating the container

private static Network network = Network.newNetwork();
private static final PostgreSQLContainer postgreSQLContainer;


   static {
       postgreSQLContainer = new PostgreSQLContainer<>(DockerImageName.parse("postgres:11.1"))
               .withNetwork(network)
               .withDatabaseName("test_container_postgres")
               .withUsername("test")
               .withPassword("test");
       postgreSQLContainer.start();
   }

   @DynamicPropertySource
   public static void setDynamicProps(DynamicPropertyRegistry registry) {
       Startables.deepStart(postgreSQLContainer).join();
       registry.add("spring.r2dbc.url", () -> "r2dbc:postgresql://127.0.0.1:" + postgreSQLContainer.getFirstMappedPort() + "/" + postgreSQLContainer.getDatabaseName());
       registry.add("spring.r2dbc.username", postgreSQLContainer::getUsername);
       registry.add("spring.r2dbc.password", postgreSQLContainer::getPassword);
}

[vaibsaraswat]

I am facing the same issue.

[INFO ] 2023-05-03 18:59:05 [main ] [/] [] 0:440 - Container testcontainers/sshd:1.1.0 is starting: 2e7992a0f868d11f0f947ed4a863db7a5c1b5b4be2b8503036cadc413fd33296
[ERROR] 2023-05-03 19:00:06 [main ] [/] [] 0:540 - Could not start container
org.testcontainers.containers.ContainerLaunchException: Timed out waiting for container port to open (localhost ports: [49154] should be listening)

Rancher Desktop v1.8.1
MacOS 13.3.1 M1
Test container v1.18.0

Do we have a solution or a workaround?

[jandubois]

The mvn verify command works for me on Intel, but fails on M1 because ports don't seem to be forwarded correctly. Not sure yet why. I could work around it by providing the port of the VM explicit:

export TESTCONTAINERS_HOST_OVERRIDE=$(rdctl shell ip a show rd0 | awk '/inet / {sub("/.*",""); print $2}')

Output:

...
[INFO] --- jar:3.3.0:jar (default-jar) @ testcontainers-java-repro ---
[WARNING] JAR will be empty - no content was marked for inclusion!
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  3.565 s
[INFO] Finished at: 2023-09-13T17:07:54-07:00
[INFO] ------------------------------------------------------------------------

This requires that you run Rancher Desktop with admin access enabled, so it will allocate a routable IP:

[jandubois]

What I suspect is that the M1 machines are too fast. It takes up to 3 seconds for the port to be forwarded to the host, and if the app checks to connect to the port on localhost before the forwarding is in place, it will fail.

My Intel machine is slower, so the forward may be already created by the time the connection is attempted.

BTW, if you are using VZ emulation, then the interface name is vznat instead of rd0 (requires Rancher Desktop 1.10):

export TESTCONTAINERS_HOST_OVERRIDE=$(rdctl shell ip a show vznat | awk '/inet / {sub("/.*",""); print $2}')

[jandubois]

If you are using VZ emulation (requires macOS 13.3+ Ventura), then you can also get it to work without administrative access by exporting 2 more settings:

export DOCKER_HOST=unix://$HOME/.rd/docker.sock
export TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE=/var/run/docker.sock
export TESTCONTAINERS_HOST_OVERRIDE=$(rdctl shell ip a show vznat | awk '/inet / {sub("/.*",""); print $2}')

This is possible because vzNAT networking doesn't require admin access to allocate a routable IP address. It is automatically enabled with VZ emulation, but only in the just release Rancher Desktop 1.10.0.

[jandubois]

I doubt it has anything to do with the macOS or Rancher Desktop versions, but is some local system issue, as this is all managed by vmnet.

I would try to reboot the machine, just to see if that fixes it, given that it has worked on the same machine before. While generally unsatisfying, it would at least maybe unblock you.

[pdrosos]

Hi @jandubois, @udf2457

Unfortunately restart didn't help, it was the first I tried. I am with Rancher Desktop 1.10.0 (latest), MacBook Pro M1, MacOS Sonoma. It's strange indeed why this suddenly stopped working and there's no IPv4 any more, I have no idea why.

[jandubois]

I've found 2 people on Slack that have experienced something similar, and it is still unknown why this is happening. One person claimed it happend with 13.4.0, was fixed in 13.4.1, but happened again with 13.5.0, but the other person seemed to have experienced it with 13.4.1 as well.

Now @pdrosos reports it happening in 15.?? as well, so it seems like an ongoing problem.

I don't know if this can be worked around with the upcoming user-v2 networking support in Lima, but we won't be able to support this until 1.12 late in the year, as the work on 1.11 has to wrap up pretty soon.

[udf2457]

@pdrosos

Ah, Sonoma. I'm still on Ventura (13.6).

So we really need some more feedback from Sonoma users I guess, just to rule that possibility out.

[Fonzerelli13]

Has the same issue with Rancher Desktop and Test Containers.
My config is: MacBook Pro with M1 chip, MacOS Sonoma. Rancher Desktop 1.10.0 Administrative access allowed, QEMU Virtual machine type.

With such settings I am getting the same error "Connection refused". But interesting thing here is that if I stop in debug in place where it tries to connect to the database, wait for several seconds and then proceed the test then it works fine.
With VZ emulation have the same issues as @pdrosos regarding IPV6 adddress.

[pdrosos]

Hi @Fonzerelli13

I had to add additional Wait.forListeningPort() strategy to my PostgreSQL testcontainer in order to work on my MacBook Pro M1. This was related to another Lima specific problem with a delay in port forwarding: abiosoft/colima#71 (comment)

private static final PostgreSQLContainer<?> POSTGRESQL_CONTAINER = new PostgreSQLContainer<>(DockerImageName.parse("postgres:15.4"))
      .withUsername(USERNAME)
      .withPassword(PASSWORD)
      .withDatabaseName(SCHEMA)
      .waitingFor(new WaitAllStrategy()
          .withStrategy(Wait.forListeningPort())
          .withStrategy(Wait.forLogMessage(".*database system is ready to accept connections.*\\s", 2)
              .withStartupTimeout(Duration.ofSeconds(60))
          )
      );

[Fonzerelli13]

@pdrosos thank you for your advice. At the moment looks that Test Containers works correctly after adding permanent enviroment variables into .zprofile file and rebooting my mac.

[stefanscheidt]

@jandubois Any updates regarding rdctl shell ip a show rd0 only returning inet6 addresses?

On my M1 mac with macOS 14.1.1 (23B81) and Rancher Desktop 1.11.0 I only see this:

❯ rdctl shell ip a show rd0
4: rd0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:55:55:96:30:ad brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5055:55ff:fe96:30ad/64 scope link
       valid_lft forever preferred_lft forever

[stefanscheidt]

@jandubois It seems to depend on the network the host machine is in. This morning I was sitting in the office connected to the WLAN there and had the no-inet-issue mentioned just above. Now I'm at home in my own WLAN and see this:

❯ rdctl shell ip a show rd0
4: rd0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:55:55:96:30:ad brd ff:ff:ff:ff:ff:ff
    inet 192.168.178.109/24 brd 192.168.178.255 scope global rd0
       valid_lft forever preferred_lft forever
    inet6 fe80::5055:55ff:fe96:30ad/64 scope link
       valid_lft forever preferred_lft forever

Thus export TESTCONTAINERS_HOST_OVERRIDE=$(rdctl shell ip a show rd0 | awk '/inet / {sub("/.*",""); print $2}') works again ...

[bsideup]

I understand that this comment might be a little bit unethical to make given that this is an issue in Rancher Desktop's repo, but I wanted to point out that (at least as a temporary solution), Testcontainers Desktop now comes with an embedded runtime that has a built-in solution for this (detecting the IP while the VZ backend is used).

P.S. we (Testcontainers) love Rancher Desktop and even mention it specifically as one of the available local runtime options in Testcontainers Desktop! ❤️ I was just following this thread and did not feel good about all these Testcontainers users who are having issues 🙌

[stefanscheidt]

@jandubois I can reproduce my problem: When being at home, the Rancher Desktop VM can obtain an IPv4 address, when connected with the company WIFI, it can't - maybe some certificate issues, I don't know - I'm more or less a network noob.

Is there a way to not use a "bridged setup" (which I assume RD does) but a "NAT setup" for the VM and still have a reachable IP address for Testcontainers?

[stefanscheidt]

And another update: I tried to use VZ emulation without administrative rights in my home office network to avoid the bridged network. But then no inet shows up, only inet6:

$ rdctl shell ip a show vznat
3: vznat: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:55:55:1a:a7:50 brd ff:ff:ff:ff:ff:ff
    inet6 fddf:917c:6f8d:8527:5055:55ff:fe1a:a750/64 scope global dynamic flags 100
       valid_lft 2591994sec preferred_lft 604794sec
    inet6 fe80::5055:55ff:fe1a:a750/64 scope link
       valid_lft forever preferred_lft forever

Thus Testcontainers will not work, because we can not set TESTCONTAINERS_HOST_OVERRIDE as shown here

export TESTCONTAINERS_HOST_OVERRIDE=$(rdctl shell ip a show vznat | awk '/inet / {sub("/.*",""); print $2}')

[sairaghavak]

I tried this approach #2609 (comment)

It worked without giving sudo access to rancher.

Again, the issue popped up and the fix was not permanent.

Later, I tried the wait strategy as suggested here: #2609 (comment)

It worked in my case with and without corporate VPN and verified in build pipeline as well, it works! Thanks.

MacOs: Sonoma 14.3.1
Rancher Desktop Version: 1.12.3

[kari-jumo]

Running
rdctl shell ip a show vznat
I get
ip: can't find device 'vznat'

Rancher Desktop 1.13.1 for aarch64 on M3

[slayton]

@kari-jumo I ran into the same thing until I disabled run as administrator and ensured Kubernetes was disabled.

[mwisnicki]

Still does not work.

Rancher 1.15.0 in default configuration with Kubernetes disabled.

Following https://docs.rancherdesktop.io/how-to-guides/using-testcontainers/

git clone https://github.com/testcontainers/testcontainers-java-repro
cd testcontainers-java-repro
./mvnw verify

First failure:

 Tests in error: 
 demonstration(org.testcontainers.repro.ReproExampleTest): Could not find a valid Docker environment. Please see logs and 
 check configuration

Website forgot to mention DOCKER_HOST but ok

export DOCKER_HOST=unix://$HOME/.rd/docker.sock

Still fails:

. org.testcontainers.containers.ContainerLaunchException: Container startup failed for image testcontainers/ryuk:0.6.0
 Caused by: org.rnorth.ducttape.RetryCountExceededException: Retry limit hit with exception
. Caused by: org.testcontainers.containers.ContainerLaunchException: Could not create/start container
. Caused by: com.github.dockerjava.api.exception.NotFoundException: Status 404: {"message":"No such container: 60a9216f94ea7800b824fd5419aac9f1d8c7c0765ab1248f2d0d82afdb65d72a"}

export TESTCONTAINERS_RYUK_DISABLED=true

BUILD SUCCESS

[mwisnicki]

Got it to work with VZ instead of QEMU, same as #2609 (comment)

Also no IPv4 on vznat while on home wifi (but with Tailwind vpn)

[onderilkesever]

Running rdctl shell ip a show vznat I get ip: can't find device 'vznat'

Rancher Desktop 1.13.1 for aarch64 on M3

Followed the recommendations on this issue, but I also still get this.. Any other suggestion here?

[arunmuthiyarkath]

Running rdctl shell ip a show vznat I get ip: can't find device 'vznat'

Same issue here. And recommendations did not not work. Any expert opinions here ?
Version: Version: 1.17.0

[onc]

Same issue for me. Running rancher desktop version 1.17.1.

[michaeldisaro]

Same problem here with rancher 1.17.1 and macos sequoia. It seems that rancher is not enabling vz.

[Fonzerelli13]

@onderilkesever @arunmuthiyarkath @onc @michaeldisaro for me this command worked:
export TESTCONTAINERS_HOST_OVERRIDE=$(ifconfig en0 | awk '/inet / {print $2}')
so, the problem may be in command to show interface address and interface name itself. In my case ifconfig as command ant en as interface name worked. You can just type ifconfig in termial to list all interfaces and fined one with ip v4 address, in my case it was en0.

[udf2457]

Whilst I have no comment on @Fonzerelli13's general answer, I would comment that maybe instead of:

just type ifconfig in termial to list all interfaces and fined one with ip v4 address

You might find it easier to do:

export TESTCONTAINERS_HOST_OVERRIDE=$(ifconfig $(route -n get 0.0.0.0 | grep -F interface | cut -d: -f2) | awk '/inet / {print $2}')

This will print the IP address associated with the default route (i.e. the one your internet traffic is going over).

Disclaimer: Read up on all commands before running them, run at your own risk, yada yada yada 😉

[michaeldisaro]

I solved my problem.
I have my company mac behind an MDM.

I noticed that without administrative privileges any time the GUI crashes it seems that any change made to preferences is not saved.

I just tried this:

1. cleaned my .zshrc from all the previously exported env vars such as:
   TESTCONTAINERS_HOST_OVERRIDE TESTCONTAINERS_RYUK_DISABLED DOCKER_HOST_OVERRIDE etc.
2. asked admin status to my MDM
3. launched command rdctl factory-reset
4. launched command "/Applications/Rancher Desktop.app/Contents/Resources/resources/darwin/bin/rdctl" start --experimental.virtual-machine.type vz --application.admin-access

Rancher started fine with enabled kubernetes, and I was able to disable kubernetes and select virtiofs and enable rosetta options later from the GUI.

Now intellij can correctly connect to docker host and run tests on a testcontainersofficial/ryuk container.

I have a MacBookPro M3 Pro with Sequoia 15.3 and Rancher 1.17.1.

[onc]

@udf2457 thank you. this solved my problems for me.

[tylervangorder-8451]

@udf2457 Yep, that worked for us as well. THANK YOU

[gong]

I have a MacBookPro M4 Pro with Sequoia 15.2 and Rancher 1.18.0.
I meet error Container startup failed for image testcontainers/ryuk
I add below config to .zshrc

export DOCKER_HOST=unix://$HOME/.rd/docker.sock
export TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE=/var/run/docker.sock
export TESTCONTAINERS_HOST_OVERRIDE=$(ifconfig $(route -n get 0.0.0.0 | grep -F interface | cut -d: -f2) | awk '/inet / {print $2}')

It could address Container startup failed for image testcontainers/ryuk. But I will meet below problem when I use mysql image.

tc.mysql:5.7 - Log output from the failed container: exec /usr/local/bin/docker-entrypoint.sh: exec format error

I change Rancher config virtual machine to vz, my issue is addressed.