-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
MTU change doesn't cascade to containers #848
Comments
@FalconerTC DockerMTU is propagated to Probably it isn't working as expected, or there's a different setting for another use-case? 🤔 |
I don't. I'm mostly learning about docker MTU as part of investigating this. I'm currently running with
And can confirm those values are being set
But eth0 still doesn't seem to get it
Not sure how cap_add |
@FalconerTC We have a ticket open with Google and they seem to be working to rollback a change that might have caused issues.
They previously mentioned
|
Oh wow. Thanks for the info here @shadiramadan . This has been driving me crazy all week |
@FalconerTC The fix might be out or it is the workaround... but I finally got a build to work after adding:
To my runner deployment. |
Only for it to fail after another run.... nvm lol |
If that GKE-specific workaround didn't work, try this workaround: And also read this for more context: Perhaps BTW could you share your workflow definition? I thought another folk contributed DockerMTU and the way we currently configure dockerd. I believe it works in certain cases and not work in another cases. I'm currently wondering if this might fail only when you are using service containers in your workflow. |
As to the MTU issue: It seems that GHA creating it's own Docker network is causing the config to be ignored. Github actions runs the following for
I ran the same my self without specifying the network and confirmed the MTU gets set then
edit: The workflow contains the following:
The current behavior is that the |
I experienced the same issue yesterday and end up with recreating whole VPC with GKE cluster in order to align MTU to 1500.. |
I'm successfully using a workaround, based on the runner issue posted above. My modified docker wrapper script is
Waiting to hear back from a Google ticket for more info on what really went wrong here. Given the limitations of |
kubernetes/test-infra#23741 (comment) Current ETA from Google on a fix is Thursday |
@FalconerTC after the workaround: are you seeing annotations like these in your workflows? (Check your post actions)
|
I'm not, no. Haven't seen any issues with it today. Here's output from a Stop Containers task on the workflow I linked earlier
|
I want to confirm the workaround also works for me. |
GKE has resolved the ticket on their side and there's a usable workaround here for any other MTU issues. Closing this. |
@callum-tait-pbx I think this is worth being covered in our documentation as the MTU issue itself is not GKE specific |
I'm not too keen on building my own custom docker image, can't we include the work-around from #848 (comment) in the official image? There are at least 6 different issues reported on this, https://github.com/actions-runner-controller/actions-runner-controller/issues?q=label%3A%22docker+mtu+issue%22+is%3Aclosed |
I'm not very keen to do that either, as I consider this as something that needs to be fixed on GitHub and actions/runner side, isn't it? 🤔 |
@mumoshu You're absolutely right, my mistake. |
Describe the bug
I run actions-runner-controller on GKE, which uses a default MTU of 1460. I've been investigating out bound connection (
npm install
,apk add
, etc) that have been freezing periodically. I saw the following (#385) and setFor my runners, and confirmed this change
But I continue to see inconsistent outbound activity on workflows using
container
(https://docs.github.com/en/actions/learn-github-actions/workflow-syntax-for-github-actions#jobsjob_idcontainer). It seems like MTU changes aren't applying to these containersIs there a way to use customMTU for workflows using
container
?Checks
The text was updated successfully, but these errors were encountered: