504 TIMEOUT WITH SECURITY GROUPS

[richardmatthewsdev]

Hey,

Describe the bug

There appears to be a difference in the way that security groups are applied between v2.3.0 and v2.3.1. The way that I am using the security groups is that we have extra security groups applied to the ALB to restrict direct access to it so that it can only be accessed via our CDN. The way that we do this is with the alb.ingress.kubernetes.io/security-groups: annotation on the ingress resource.

When we upgraded the aws load balancer controller our websites that use that load balancer all started timing out and would only work again once I removed the security group annotation.

I found that I was able to reproduce this issue by copying the exact security groups that were generated by the controller into the annotation as well, so it is not an issue with my security groups specifically.

Also it would be great if the annotation was something like additional security groups, rather than all security groups so I didn't have to include the required automatically generated security group in the annotation as well.

Steps to reproduce

Deploy an ingress-nginx controller with service type NodePort.
Deploy an ingress resource which sends traffic to the ingress nginx controller service.
Deploy an application that is accessed via this loadbalancer
Visit the EC2 dashboard and copy the security group ids off of the generated loadbalancer.
Add this annotation alb.ingress.kubernetes.io/security-groups: with the generated security group ids to your ingress resource.
Visit your application and see that it gets a 504 timeout.

My existing set up works fine again once I reverted the version of aws loadbalancer controller back to v2.3.0.

Expected outcome
Traffic to flow normally between the ALB and our applications.

Environment

• AWS Load Balancer controller version v.2.3.1
• Kubernetes version 1.21
• Using EKS (yes/no), if so version? Yes, 1.21

Thanks for your help!

[kishorj]

@reinkaos, when you specify the alb.ingress.kubernetes.io/security-groups annotation, controller does not configure the ingress rules on your EC2/ENI security group, you will need to add the necessary rules manually. However, starting v2.3.0 release, we provide an additional annotation alb.ingress.kubernetes.io/manage-backend-security-group-rules, if set to "true", controller will configure the necessary SG rules for ingress access.

With auto-generated security groups, controller always manages the backend security group rules. SG will be auto-generated only if you don't specify the alb.ingress.kubernetes.io/security-groups annotation. The auto-generated security groups should not be specified in alb.ingress.kubernetes.io/security-groups.

[kishorj]

I will update the live docs for alb.ingress.kubernetes.io/manage-backend-security-group-rules. #2486

[richardmatthewsdev]

Hey @kishorj,

Thank you for your reply and letting me know about that. I can confirm with the manage-backend-security-group-rules annotation my set up is working now.

Thanks for updating the docs as well!