KEDA error establishing rabbitmq connection in EKS cluster with Calico networking

[visla-xugeng]

Hi
I installed EKS cluster with Calico networking(xvlan). In this env, I have rabbitmq installed through helm chart. A KEDA setup through helm chart as well. Before Calico network, I was using aws vpc cni network by default. The KEDA scaling worked well. After switched to the Calico network (overlay), I got below errors:

btw: I already updated the KEDA to use "ClusterFirstWithHostNet", since I am using overlay network.

I got below error. As a result, the KEDA cannot scaling more consumers(pods) to process messages.

E0222 03:51:12.400833       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"error when getting scalers error getting scaler for trigger #0: error establishing rabbitmq connection: dial tcp: lookup rabbitmq.default.svc.cluster.local on 10.64.0.2:53: no such host"}

NAME       TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)         AGE
kube-dns   ClusterIP   172.20.0.10   <none>        53/UDP,53/TCP   4h18m

kubectl logs --follow -n kube-system --selector 'k8s-app=kube-dns'
[INFO] 192.168.206.133:47082 - 16294 "A IN rabbitmq.default.svc.cluster.local.us-west-2.compute.internal. udp 79 false 512" NXDOMAIN qr,rd,ra 192 0.002322684s
[INFO] 192.168.206.133:50325 - 13295 "A IN rabbitmq.default.svc.cluster.local. udp 52 false 512" NOERROR qr,aa,rd 102 0.000217494s
[INFO] 192.168.206.133:54154 - 16701 "A IN rabbitmq.default.svc.cluster.local.svc.cluster.local. udp 70 false 512" NXDOMAIN qr,aa,rd 163 0.000200126s
[INFO] 192.168.206.133:59313 - 30695 "AAAA IN rabbitmq.default.svc.cluster.local.keda.svc.cluster.local. udp 75 false 512" NXDOMAIN qr,aa,rd 168 0.000163252s
[INFO] 192.168.206.133:50486 - 25237 "A IN rabbitmq.default.svc.cluster.local.keda.svc.cluster.local. udp 75 false 512" NXDOMAIN qr,aa,rd 168 0.000179662s
[INFO] 192.168.206.133:39588 - 12812 "AAAA IN rabbitmq.default.svc.cluster.local.svc.cluster.local. udp 70 false 512" NXDOMAIN qr,aa,rd 163 0.000321733s
[INFO] 192.168.206.133:37375 - 53129 "AAAA IN rabbitmq.default.svc.cluster.local.cluster.local. udp 66 false 512" NXDOMAIN qr,aa,rd 159 0.000156612s
[INFO] 192.168.206.133:47697 - 37961 "A IN rabbitmq.default.svc.cluster.local.cluster.local. udp 66 false 512" NXDOMAIN qr,aa,rd 159 0.000164842s
[INFO] 192.168.206.133:36139 - 35017 "AAAA IN rabbitmq.default.svc.cluster.local.us-west-2.compute.internal. udp 79 false 512" NXDOMAIN qr,rd,ra 79 0.000721258s
[INFO] 192.168.206.133:35654 - 52322 "AAAA IN rabbitmq.default.svc.cluster.local. udp 52 false 512" NOERROR qr,aa,rd 145 0.0008279s
[INFO] 192.168.206.133:50002 - 12758 "A IN rabbitmq.default.svc.cluster.local. udp 52 false 512" NOERROR qr,aa,rd 102 0.000239663s
[INFO] 192.168.206.133:60410 - 22924 "A IN rabbitmq.default.svc.cluster.local.us-west-2.compute.internal. udp 79 false 512" NXDOMAIN qr,aa,rd,ra 192 0.000127624s
[INFO] 192.168.206.133:46533 - 44750 "A IN rabbitmq.default.svc.cluster.local.keda.svc.cluster.local. udp 75 false 512" NXDOMAIN qr,aa,rd 168 0.000227403s
[INFO] 192.168.206.133:43526 - 35349 "AAAA IN rabbitmq.default.svc.cluster.local.keda.svc.cluster.local. udp 75 false 512" NXDOMAIN qr,aa,rd 168 0.00022409s
[INFO] 192.168.206.133:56380 - 29165 "A IN rabbitmq.default.svc.cluster.local.svc.cluster.local. udp 70 false 512" NXDOMAIN qr,aa,rd 163 0.000213602s
[INFO] 192.168.206.133:54311 - 57911 "AAAA IN rabbitmq.default.svc.cluster.local.cluster.local. udp 66 false 512" NXDOMAIN qr,aa,rd 159 0.000151722s
[INFO] 192.168.206.133:54528 - 1582 "A IN rabbitmq.default.svc.cluster.local.cluster.local. udp 66 false 512" NXDOMAIN qr,aa,rd 159 0.000304043s
[INFO] 192.168.206.133:38030 - 25405 "AAAA IN rabbitmq.default.svc.cluster.local.svc.cluster.local. udp 70 false 512" NXDOMAIN qr,aa,rd 163 0.000210289s
[INFO] 192.168.206.133:45711 - 25891 "AAAA IN rabbitmq.default.svc.cluster.local.us-west-2.compute.internal. udp 79 false 512" NXDOMAIN qr,rd,ra 79 0.000494616s
[INFO] 192.168.206.133:54905 - 60151 "A IN rabbitmq.default.svc.cluster.local. udp 52 false 512" NOERROR qr,aa,rd 102 0.000232012s
[INFO] 192.168.206.133:43114 - 51847 "A IN rabbitmq.default.svc.cluster.local.us-west-2.compute.internal. udp 79 false 512" NXDOMAIN qr,aa,rd,ra 192 0.00021606s
[INFO] 192.168.206.133:40021 - 3288 "AAAA IN rabbitmq.default.svc.cluster.local. udp 52 false 512" NOERROR qr,aa,rd 145 0.00027382s

any idea will be appreciated.

[tomkerkhove]

As you point out, this seems to be related to Calico but unfortunately I don't have experience with that. Are you sure that everything is accessible from KEDA?

Once we manage to solve that we should document it on https://keda.sh/docs/latest/troubleshooting/

[visla-xugeng]

@tomkerkhove Thanks for your response.
1: When I use aws vpc cni network, the RabbitMQ, KEDA all works well.
2: After switch to the calico network, the KEDA cannot reach out my RabbitMQ server since the DNS resolving failed(mentioned above). KEDA cannot get the length of Queue, so it cannot scale the consumer to more pods.

I checked the apiservice of KEDA, it looks good.
v1beta1.external.metrics.k8s.io keda/keda-operator-metrics-apiserver True 135m

Since the calico network is using, the node and pod is using different networks. The node is using VPC: 10.10.0.0/16, pod is using the default network: 192.168.0.0/16. The DNS setting on the node is: nameserver 10.10.0.2. This DNS cannot resolve the DNS names of pods.

My question is how can we fix this DNS issue?

Thanks,

[tomkerkhove]

Unfortunately I'm not a Calico expert so I won't be able to help. Maybe somebody in our slack can help you.

[visla-xugeng]

Unfortunately I'm not a Calico expert so I won't be able to help. Maybe somebody in our slack can help you.

Hi Tomkerkhove,

Do you think you can forward my question to your slack community? Thanks,