calico picks virtual-ip (i.e /128) address configured on eth0 for BGP peering and calico-node pod crashes

[hanamantagoudvk]

We have a v6 only k8s cluster (1.21.1) with calico CNI (3.19.1). Our cluster has 3 master nodes for HA purpose (master-0, master-1,master-2). HA-proxy /keepalived is being used to load-balance the requests being sent to api-server . HA proxy uses VIP as front-end for api-servers.

Is there a way to skip this address (i.e /128 or /32) address ?

Our configuration is shown below :

PodCIDR: fd00:eccd:81:ffff::/64
Internal network CIDR (Node cidr): fd00:eccd:81:1111::/64

Nodes:
master-0-ejlnoox-ibd-stack Ready control-plane,master 4h15m v1.21.1 fd00:eccd:81:1111::3 SUSE Linux Enterprise S4
master-1-ejlnoox-ibd-stack Ready control-plane,master 4h15m v1.21.1 fd00:eccd:81:1111::e SUSE Linux Enterprise S4
master-2-ejlnoox-ibd-stack Ready control-plane,master 4h15m v1.21.1 fd00:eccd:81:1111::4 SUSE Linux Enterprise S4
worker-pool1-ilq0mgem-ejlnoox-ibd-stack Ready worker 4h9m v1.21.1 fd00:eccd:81:1111::2 SUSE Linux Enterprise S4

VIP address: fd00:eccd:81:1111::123/128

Master 0 Configurations:

eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2090 qdisc pfifo_fast state UP group default qlen 1000
link/ether fa:16:3e:8d:d4:4a brd ff:ff:ff:ff:ff:ff
inet 10.0.16.12/21 brd 10.0.23.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fd00:eccd:81:1111::123/128 scope global nodad deprecated
valid_lft forever preferred_lft 0sec
inet6 fd00:eccd:81:1111::3/64 scope global
valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:fe8d:d44a/64 scope link
valid_lft forever preferred_lft forever

2021-08-10 06:23:13.886 [INFO][81] monitor-addresses/startup.go 782: Using autodetected IPv6 address fd00:eccd:19:1111::9/128 on matching interface eth0
2021-08-10 06:23:13.886 [INFO][81] monitor-addresses/startup.go 589: Node IPv6 changed, will check for conflicts
2021-08-10 06:23:13.892 [WARNING][81] monitor-addresses/startup.go 1134: Calico node 'master-0-eccd-ci-os-19-jenkins' is already using the IPv6 address fd00:eccd:19:1111::9.
2021-08-10 06:23:13.892 [WARNING][81] monitor-addresses/startup.go 1120: IPv6 address has changed. This could happen if there are multiple nodes with the same name. node="master-2-eccd-ci-os-19-jenkins" original="fd00:eccd:19:1111::c" updated="fd00:eccd:19:1111::9"
2021-08-10 06:23:13.892 [INFO][81] monitor-addresses/startup.go 351: Clearing out-of-date IPv6 address from this node IP="fd00:eccd:19:1111::9/128"
2021-08-10 06:23:13.899 [WARNING][81] monitor-addresses/startup.go 363: Failed to clear node addresses error=update conflict: Node(master-2-eccd-ci-os-19-jenkins)
2021-08-10 06:23:13.899 [WARNING][81] monitor-addresses/startup.go 1340: Terminating

[caseydavenport]

@hanamantagoudvk have you looked at this page at all? https://docs.projectcalico.org/networking/ip-autodetection

This might help configure selection of the correct address.

[hanamantagoudvk]

@caseydavenport i forgot to mention above. We are using auto detection method (i.e interface=eth0). But in this eth0 interface VIP sits along with node address. So calico picks the VIP instead of node address. So i want to propose one thing here, there must be a way to skip ip addr along with auto detection method. I see that skip-interface method is there, but skip ip is not present.

Even @fasaxc is ok with this proposal. I can work this fix.

[hanamantagoudvk]

@caseydavenport @fasaxc : Currently calico/node code doesn't guarantee/ensure that nodeAddress it picks under an eth interface remains same every time. Deterministic way of selecting same ip every time is our requirement. Isn't it possible to pick the ip which is present in k8s node object ?

[caseydavenport]

Isn't it possible to pick the ip which is present in k8s node object ?

This is an existing feature request, but not implemented yet. If you are using manifests (not operator) you can use the Kubernetes downward API to configure the IP env var with the node IP.

We are using auto detection method (i.e interface=eth0)

Would autodetection mode cidr=A,B,C work for your case?

[hanamantagoudvk]

This is an existing feature request, but not implemented yet. If you are using manifests (not operator) you can use the Kubernetes downward API to configure the IP env var with the node IP.

Yes we do use manifest for the daemonset. But manifest file is generic file , how can we specify ip address of a given node in IP env var ?

[hanamantagoudvk]

@caseydavenport : Downward API seems to be having certain limitations. We want set the IP or IP6 values like shown below.
But downward API currently supports only status.podIP . With dual-stack feature coming into picture now , we should enhance downward API to support status.podIPs

      env:
        - name: IP
  	  valueFrom:
            fieldRef:
              fieldPath: status.podIPs[0].ip
        - name: IP6
  	  valueFrom:
            fieldRef:
              fieldPath: status.podIPs[1].ip	

[hanamantagoudvk]

@caseydavenport : When i asked couple of folks in community , they say not possible to access indexed values like above. Hence downward API can't be used by us. Is it ok to introduce skip-ip field ?

[caseydavenport]

Would autodetection mode cidr=A,B,C work for your case?

I think my question above still stands.

Isn't it possible to pick the ip which is present in k8s node object ?

This is my preferred option, if all of the above doesn't work for you. We would need to implement it.

e.g., something like this

IP_AUTODETECT_METHOD=KubernetesInternalIP
IP6_AUTODETECT_METHOD=KubernetesInternalIPv6

[hanamantagoudvk]

Would autodetection mode cidr=A,B,C work for your case?

Let me explain why can't we use cidr=A,B,C auto detection method.

Our k8s cluster deployment is done on VMs (worker nodes) which are managed by openstack.
worker node IPs belong to say cidr A. The VIP configured on eth0 (port where cidr A IP is assigned) should also belong to same cidr A for security purpose. In openstack , security rules are tightly coupled with port concept. By default it only allows IP addresses assigned by opestack itself. If we configure any additional IPs such as VIP , we need to add them to security groups (using allowed-address-pair config) to let the traffic destined for VIP. If we try to configure VIP from say cidr B , openstack doesn't allow it. Hence VIP should also belong to same cidr A.