[calico] don't enable ipip encapsulation by default and use vxlan in CI #8434
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
k8s-ci-robot
added
kind/bug
|
At my job we switched to vxlan crosssubnet by default a long time ago because IPIP offload is broken with many driver / firmware combination. I think RHEL started to enable IPIP offload by default (if supported) in 8.3 or 8.4 and I found at least 3 different broken cases. Nobody uses IPIP except Calico I think, so it's not tested, when vxlan is used by OpenStack and many cloud. Also CrossSubnet because why encapsulate when not needed. |
|
Thanks for the extra information @champtar ! it seems to put one more nail on the |
k8s-ci-robot
added
the
do-not-merge/work-in-progress
|
I opened an issue on calico project side to track this: projectcalico/calico#5449 |
k8s-ci-robot
removed
the
do-not-merge/work-in-progress
|
The eBPF test case is actually manual so I wrote the above conclusion too soon. Keeping this back in draft. |
cristicalin
force-pushed
the
calico_encapsulation
branch
from
d1f4444 to
23ddc07
Compare
k8s-ci-robot
added
do-not-merge/work-in-progress
cristicalin
force-pushed
the
calico_encapsulation
branch
from
23ddc07 to
21c57c2
Compare
k8s-ci-robot
added
size/M
cristicalin
force-pushed
the
calico_encapsulation
branch
7 times, most recently
from
00f8e3a to
6a2a48f
Compare
…issues during netchecker check * service proxy mode still fails connectivity tests so keeping it manual mode
cristicalin
force-pushed
the
calico_encapsulation
branch
from
1d4a1c1 to
839e174
Compare
|
Sorry for late response and thanks for updating. /lgtm |
k8s-ci-robot
added
the
lgtm
floryut
added
kind/network
|
The reason IPIP was not working in Calico was due to an internal change to IPIP code in kernel. We identified the change, and fixed our code to respect the new behaviour. The fix (projectcalico/calico#5846) will be available in Calico v3.23.0. |
sakuraiyuta
pushed a commit
to sakuraiyuta/kubespray
that referenced
this pull request
Apr 16, 2022
…CI (kubernetes-sigs#8434) * [calico] make vxlan encapsulation the default * don't enable ipip encapsulation by default * set calico_network_backend by default to vxlan * update sample inventory and documentation * [CI] pin default calico parameters for upgrade tests to ensure proper upgrade * [CI] improve netchecker connectivity testing * [CI] show logs for tests * [calico] tweak task name * [CI] Don't run the provisioner from vagrant since we run it in testcases_run.sh * [CI] move kube-router tests to vagrant to avoid network connectivity issues during netchecker check * service proxy mode still fails connectivity tests so keeping it manual mode * [kube-router] account for containerd use-case
sathieu
added a commit
to sathieu/kubespray
that referenced
this pull request
Apr 26, 2022
sathieu
added a commit
to sathieu/kubespray
that referenced
this pull request
Apr 26, 2022
k8s-ci-robot
pushed a commit
that referenced
this pull request
Apr 27, 2022
Closed
LuckySB
pushed a commit
to southbridgeio/kubespray
that referenced
this pull request
Jun 29, 2023
…CI (kubernetes-sigs#8434) * [calico] make vxlan encapsulation the default * don't enable ipip encapsulation by default * set calico_network_backend by default to vxlan * update sample inventory and documentation * [CI] pin default calico parameters for upgrade tests to ensure proper upgrade * [CI] improve netchecker connectivity testing * [CI] show logs for tests * [calico] tweak task name * [CI] Don't run the provisioner from vagrant since we run it in testcases_run.sh * [CI] move kube-router tests to vagrant to avoid network connectivity issues during netchecker check * service proxy mode still fails connectivity tests so keeping it manual mode * [kube-router] account for containerd use-case
LuckySB
pushed a commit
to southbridgeio/kubespray
that referenced
this pull request
Jun 30, 2023
LuckySB
pushed a commit
to southbridgeio/kubespray
that referenced
this pull request
Oct 23, 2023
…CI (kubernetes-sigs#8434) * [calico] make vxlan encapsulation the default * don't enable ipip encapsulation by default * set calico_network_backend by default to vxlan * update sample inventory and documentation * [CI] pin default calico parameters for upgrade tests to ensure proper upgrade * [CI] improve netchecker connectivity testing * [CI] show logs for tests * [calico] tweak task name * [CI] Don't run the provisioner from vagrant since we run it in testcases_run.sh * [CI] move kube-router tests to vagrant to avoid network connectivity issues during netchecker check * service proxy mode still fails connectivity tests so keeping it manual mode * [kube-router] account for containerd use-case
LuckySB
pushed a commit
to southbridgeio/kubespray
that referenced
this pull request
Oct 23, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What type of PR is this?
/kind bug
/kind failing-test
What this PR does / why we need it:
While debugging calico ebpf failures in CI for AlmaLinux 8 we discovered an issue with Alma/Centos 8.5 and
ipipencapsulation preventing pod-to-pod communication. Upon further investigation and discussions with the Calico team, their recomandation is to usevxlanencapsulation on newer kernels instead ofipipdue to performance reasons even though both should work in bothiptablesandebpfmode.For now this PR changes the default to
ipip: falsedisablingipipencapsulation and we continue to investigate the reason for theipipfailure.During the development of this fix I discovered that the kubespray CI needs encapsulation for multi-node deployments and I had to enable
vxlanfor calico based CI tests.Additionally to properly troubleshoot the CI failures I had to modify the network connectivity test and actually make it fail when
netcheckeragents could not connect to thenetchecker-server. This new change exposed connectivity issues with thekube-routerCNI which were worked around by moving these tests tovagrantinstead ofpacket. There is still a know failure in runningkube-routerinservice proxymode which I was not able to fully address, given that vagrant jobs in general are allowed to tail and this test is left on manual this will not block future CI jobs.Which issue(s) this PR fixes:
Fixes #8456
Special notes for your reviewer:
Given previous experience with changing defaults such as when changing the
container_managertocontainerd, I'm somewhat conflicted on the actual change and an alternative would be to just change the settings in the CI but, for now the change as is seems like the best course of action to me but I welcome input here.Does this PR introduce a user-facing change?: