Calico: Allow operators to choose which encapsulation mode to use

[seh]

Introduce a new "encapsulationMode" field in Calico's portion of the Cluster specification to allow switching between the the IP-in-IP and VXLAN encapsulation protocols. For now, we accept the values "ipip" and "vxlan," and forgo a possible "none" value that would disable encapsulation altogether (at least for the default Calico IP pool).

Augment the default-populating procedure for Calico to take this field into account when deciding both which networking backend to use and whether to use IP-in-IP or VXLAN encapsulation for the default IP pool. Note that these values supplied for the "CALICO_IPV4POOL_IPIP" and "CALICO_IPV4POOL_VXLAN" environment variables in the "calico-node" DaemonSet pod spec only matter for creating the "default" IPPool pool object when no such objects already exist.

Generalize the documentation for the "crossSubnet" field to cover environments more broad than just AWS, as Calico can employ this selective encapsulation in any environment in which it can detect boundaries between subnets.

Fixes #10168.

[k8s-ci-robot]

Hi @seh. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[hakman]

I wonder if we really need Backend added. What do you think?

[seh]

Should we still allow IP-in-IP traffic in the security rules even if the Calico backend is VXLAN?

[rifelpet]

is migrating from one backend to another a supported operation for existing clusters? If it is, then modifying security group rule definitions could mean that kops update cluster revokes a security group rule while the backend that depended on that access is still in use by a portion of the cluster.

[seh]

Is migrating from one backend to another a supported operation for existing clusters?

Yes, it is, though at present kOps would not orchestrate this all on its own. The reason is that we'd wind up changing the requisite field in the "calico-node" ConfigMap, but we wouldn't automatically restart all the pods that need to consume that updated field value in order for the change to actually take effect. After running kops update cluster and waiting for the updated Calico manifests to be applied, an operator would have to run the following command:

kubectl --namespace=kube-system rollout restart calico-node

I am not sure if it's necessary to restart the pods managed by the "calico-kube-controllers" Deployment. I will ask the Calico maintainers about that.

[seh]

Please hold pending discussion with the Calico maintainers.

[hakman]

Thanks @seh. This looks mostly good. Will take another look tomorrow and give it a try.

[seh]

This looks mostly good.

Thank you for the prompt review. I'll follow up with corrections tomorrow morning (EST).

[hakman]

This looks 👍 . Thanks @seh !
/ok-to-test
/lgtm

[hakman]

/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: hakman, seh

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [hakman]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[hakman]

/retest