Network Timeouts

[michaelst]

When running a docker build network requests made inside the Dockerfile are blocked. It pulls the FROM image just fine. I'm not sure what info would help debug this so just let me know.

[mumoshu]

@michaelst Hey! Could you share your workflow yaml file that contains the job running docker build? Probably that would help us reproduce/investigate the issue.

[michaelst]

Here is the workflow file, it works up to the docker build step on mix local.hex --force with this error

Could not install Hex because Mix could not download metadata at https://repo.hex.pm/installs/hex-1.x.csv.

I've been doing some more testing around that and it actually works to curl that url. This flow works on regular GitHub Actions. From looking at tickets with similar issues it has been a DNS issue. Is the dind image doing something weird with DNS?

name: Deploy
on:
  push:
    branches:
      - master

jobs:
  backend:
    name: Backend
    runs-on: [self-hosted, Linux, ARM64]

    defaults:
      run:
        working-directory: backend

    env:
      KUBE_CONFIG: ${{ secrets.KUBE_CONFIG }}

    steps:
      - uses: actions/checkout@main
          
      - name: GCP Authentication
        run: |
          echo ${{ secrets.GCP_SA_KEY }} | base64 -d > google_credentials.json
          gcloud auth activate-service-account --key-file google_credentials.json --quiet
          gcloud auth configure-docker us.gcr.io --quiet

      - name: Build
        run: |    
          docker build -t us.gcr.io/cloud-57/spendable-api:${GITHUB_SHA} -f .devops/Dockerfile .
          docker push us.gcr.io/cloud-57/spendable-api:${GITHUB_SHA}

Here is my Dockerfile

FROM elixir:1.11-slim AS build

ENV MIX_ENV=prod \
    LANG=C.UTF-8

RUN mkdir /app
WORKDIR /app

RUN mix local.hex --force && \
    mix local.rebar --force

COPY mix.exs mix.lock ./
COPY config/config.exs config/prod.exs ./config/
RUN mix do deps.get, deps.compile

COPY lib lib
COPY priv priv
RUN mix compile

COPY config/releases.exs ./config/
RUN mix release

FROM ubuntu:20.04 AS release

ENV LANG=C.UTF-8

RUN apt-get update \
  && apt-get install -y --no-install-recommends openssl \
  && useradd --create-home -u 1000 app \
  && rm -rf /var/lib/apt/lists/*

USER app
WORKDIR /home/app

COPY --from=build --chown=app:app /app/_build/prod/rel/spendable ./
COPY --chown=app:app .devops/entrypoint.sh .
CMD ["./entrypoint.sh"]

[mumoshu]

@michaelst Thanks! Does curl https://repo.hex.pm/installs/hex-1.x.csv work on one of your runner pods (not controller pods) work? Probably you could try kubectl exec -it into a pod and manually run the curl command. If it fails, I think enabling verbose logs on curl gives more information. If it's a DNS issue, you should be able to reproduce it without mix.

[michaelst]

So the curl did work when inside of the Dockerfile, which container in the runner pod should I do that, it doesn't work in the docker container as it doesn't have curl installed and limited shell access.

[michaelst]

I'm also getting random timeouts on apt-get update

Get:5 http://security.debian.org/debian-security buster/updates InRelease [65.4 kB]
46
Err:5 http://security.debian.org/debian-security buster/updates InRelease
47
  Connection failed [IP: 151.101.66.132 80]
48
Fetched 7919 kB in 60s (132 kB/s)
49
Reading package lists...
50
W: Failed to fetch http://security.debian.org/debian-security/dists/buster/updates/InRelease  Connection failed [IP: 151.101.66.132 80]
51
W: Some index files failed to download. They have been ignored, or old ones used instead.

[mumoshu]

@michaelst I think this kind of issue can happen when your network requires smaller MTU size. If you think it worth trying, would you mind trying #385 and try setting it to lower to adapt your environment? The feature is available in the canary tag of our controller image and the latest chart.

[michaelst]

Looks like that is my problem as eth0 has an MTU of 1450, however, how do I use the canary flag, I don't see that on docker and the latest docker image is 7 days ago, I am using the 0.9.0 version chart.

[mumoshu]

@michaelst Thanks for confirming. Re docker image tag, it's about actions-runner-controller rather than actions-runner. The former is for the controller and the latter is for runners created and managed by the runner. MTU is configured by the controller at the time of each runner is created, so you should update the controller image, rather than runner's.

[michaelst]

I'm getting this error with the canary image installed trying to setup the RunnerDeployment

failed to create resource: Internal error occurred: failed calling webhook "mutate.runnerdeployment.actions.summerwind.dev": Post "https://github-runner-controller-actions-runner-controller-webhook.github-runner.svc:443/mutate-actions-summerwind-dev-v1alpha1-runnerdeployment?timeout=30s": context deadline exceeded

[mumoshu]

@michaelst Would you mind removing mutatingwebhookconfiguration and validatingwebhookconfiguration temporarily to see if it resolves the issue? If not, kubectl -n $NS logs deploy/actions-runner-controller would possibly show some useful error messages.

[michaelst]

ok deleting those allowed the pods to deploy, however, docker0 still has 1500

docker0   Link encap:Ethernet  HWaddr 02:42:20:D5:06:0F  
          inet addr:172.17.0.1  Bcast:172.17.255.255  Mask:255.255.0.0
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

and I do see this configured on the pod now

- name: docker
      image: 'docker:dind'
      env:
        - name: DOCKER_TLS_CERTDIR
          value: /certs
        - name: DOCKERD_ROOTLESS_ROOTLESSKIT_MTU
          value: '1450'

[mumoshu]

@michaelst Ah, seems like we've accidentally committed some wrong code there. The env var name must be just MTU to work.

46ca924#diff-f0d17d25b8858cbb2b4871bde93cb452e3a7f84ef75a00e9f7026949610915fbR37

cc/ @bkimbrough88

[michaelst]

Ok I manually set the mtu on the runner pods and was able to get mix local.hex --force to run, however, it now fails on mix local.rebar --force. I tried an MTU of 1450 and 1000.

Step 5/21 : RUN mix local.hex --force &&     mix local.rebar --force
39
 ---> Running in 2a570d2d6e9e
40
* creating /root/.mix/archives/hex-0.21.1
41
** (Mix) request timed out after 60000ms
42

43
Could not install Rebar because Mix could not download metadata at https://repo.hex.pm/installs/rebar-1.x.csv.

[michaelst]

looks like that may be due to this showing up when the docker build starts and using an MTU of 1500

veth45544c0 Link encap:Ethernet  HWaddr DA:30:4B:12:8B:0C  
          inet6 addr: fe80::d830:4bff:fe12:8b0c/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:749 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2707 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:52661 (51.4 KiB)  TX bytes:801118 (782.3 KiB)

[mumoshu]

@michaelst Thanks for testing/additional info!

I thought it might be due to that additional docker network being created by some process still receiving MTU of 1500 somehow (if so that's mix's problem), but apparently not after reading moby/moby#34981.

Perhaps giving --mtu 1450 to dockerd will automagically set MTU of 1450 to those devices used by docker builds?

To do so, you'll probably need to create your own dind image with something like the below:

FROM docker:dind

ENTRYPOINT ["dockerd", "--mtu", "1450"]

The custom dind image can be passed to actions-runner-controller via the --docker-image flag.
https://github.com/summerwind/actions-runner-controller/blob/728829be7b29b1258cca4ea951f967d8f2f2f4d4/main.go#L81

You may prefer trying our "dind within runner container" option: https://github.com/summerwind/actions-runner-controller#runner-with-dind. That one doesn't require you to set --docker-image flag and the docker image you need to tweak is https://github.com/summerwind/actions-runner-controller/blob/master/runner/dindrunner.Dockerfile#L114. In that case dockerd is managed by supervisord so you can probably locate the dockerd startup config in some standard location

[michaelst]

this is what ended up working for me

FROM docker:dind

CMD ["--mtu", "1450"]

[mumoshu]

@michaelst Thanks a lot for sharing! I'll definitely add the ability to set optional --mtu flag in the controller so that you won't need to fork dind image in the future

[xiwenc]

I think I have the same problem. My builds fail with network timeouts. I'm deploying the actions-runner-controller with helm. How do I proceed to override the MTU? I read through this thread but couldn't follow it completely.

[mumoshu]

@xiwenc Unfortunately you need to build a custom dind image from a Dockerfile like

FROM docker:dind

CMD ["--mtu", "1450"]

and specify it via the --docker-image flag of the controller.

[xiwenc]

@mumoshu Thanks for summarizing it.

For future reference, helm users can override the docker image with for instance:

helm upgrade -n actions-runner-system actions-runner-controller actions-runner-controller/actions-runner-controller --set image.dindSidecarRepositoryAndTag=cinaq/docker-dind-mtu

[mumoshu]

@xiwenc Thanks for sharing!

[mumoshu]

#421 should make the dockerMTU setting just work. Thanks for your support @michaelst and @xiwenc!

[mumoshu]

#421 has been merged and dockerMTU: 1450 will be usable starting v0.18.1 with recent versions of runner and dind runner images.

[acrogenesis]

Hi @mumoshu I'm having this same problem using the latest version. I'm not sure where I'm supposed to add dockerMTU: 1450, I'm using the provided helm chart

[mumoshu]

Hey @acrogenesis 👋 Please see https://github.com/actions/actions-runner-controller/blob/master/TROUBLESHOOTING.md#outgoing-network-action-hangs-indefinitely if you have not done so yet!

[acrogenesis]

Thank you that helped me a lot!