Use AWS ELB Proxy Protocol

[julianvmodesto]

Currently, when the CLOUD_PROVIDER is set to aws, a new Service is created for the Ingress of type LoadBalancer by default.

The behavior I desire is for the LoadBalancer Service to use the "service.beta.kubernetes.io/aws-load-balancer-proxy-protocol": "*" annotation to enable the Proxy Protocol on the AWS ELB, so that the proper headers like X-Real-IP and X-Forwarded-For are set correctly.

Here are my deployments:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: voyager-operator
  namespace: default
  labels:
    run: voyager-operator
spec:
  replicas: 1
  selector:
    matchLabels:
      run: voyager-operator
  template:
    metadata:
      labels:
        run: voyager-operator
    spec:
      containers:
      - name: voyager-operator
        image: appscode/voyager:1.5.4
        args:
        - --cloud-provider=$(CLOUD_PROVIDER)
        - --cluster-name=$(CLUSTER_NAME)
        - --v=3
        ports:
        - containerPort: 1234
          name: zero
          protocol: TCP
        env:
        - name: CLOUD_PROVIDER
          value: aws
        - name: CLUSTER_NAME
          value: mcclusterface

---

kind: Service
metadata:
  name: voyager-operator
  namespace: default
  labels:
    run: voyager-operator
spec:
  ports:
  - name: zero
    port: 1234
    targetPort: zero
  selector:
    run: voyager-operator

[julianvmodesto]

Seems like this is simply adding the annotation when CLOUD_PROVIDER = 'aws' into the Service template?

Or allow for custom annotations for the Voyager Service within the Ingress.

      serviceAnnotations:
      - "service.beta.kubernetes.io/aws-load-balancer-proxy-protocol": "*"
      - "prometheus.io/scrape": "true"
      - "prometheus.io/port": "8090"

[tamalsaha]

@julianvmodesto, I have filed a new issue to track the work for custom annotation #103 .

One question I have, are you using CoreOS' prometheus operator to setup your exporter? We are considering running a side car container with the HAproxy to expose its metrics. I would like to hear how you see this work.

[tamalsaha]

The full suite of annotations available in 1.5.x are:

https://github.com/kubernetes/kubernetes/blob/release-1.5/pkg/cloudprovider/providers/aws/aws.go#L79

[tamalsaha]

As @sadlil mentioned in #103 (comment), users will be able to provide custom annotations as they see fit with 1.5.5 release.

We are going to cut a release early next week.

[julianvmodesto]

@tamalsaha appreciate your help here, love your project and looking forward to 1.5.5.

Re: prometheus, the sidecar for haproxy stats looks like a pretty good solution to me.

[julianvmodesto]

@tamalsaha I added the annotation below to a new Ingress, and the annotation exists for the new Ingress!

    ingress.appscode.com/annotations.service: '{"service.beta.kubernetes.io/aws-load-balancer-proxy-protocol": "*"}'

Two issues, though:

• updating an existing Ingress with the above annotation doesn't cause an update to the service by the controller
• actually enabling the AWS Proxy Protocol seems to cause HAProxy 400s. I tested by removing and re-adding the service.beta.kubernetes.io/aws-load-balancer-proxy-protocol annotation from the Voyager service

[tamalsaha]

We are going to look into that and get back to you.

[julianvmodesto]

Ack, sorry, regarding the second issue I have w/ 400s, I think I just need to figure out how to configure HAProxy to accept the Proxy Protocol.

[julianvmodesto]

How would I be able to attach accept-proxy to the bind below? I think this is needed for accepting the Proxy Protocol from the AWS ELB.

frontend http-frontend
    bind *:80 accept-proxy

I manually edited the configmap with the above, and this solved the issue with the 400s.

Maybe there should be a way to add modifiers to the bind directive?

[tamalsaha]

You can't add options to the bind https://github.com/appscode/voyager/blob/master/pkg/controller/ingress/template/template.go#L140..

We are going to push a new update this fixed this week. @sadlil , I found some more docs here: https://jve.linuxwall.info/ressources/taf/haproxy-aws/

[julianvmodesto]

Great, thanks in advance Tamal!

[julianvmodesto]

Created #115.

[tamalsaha]

@julianvmodesto , just to clarify one thing, with 1.5.6, you only need to add
ingress.appscode.com/keep-source-ip: true annotation to ingress. Voyager operator will automatically add "service.beta.kubernetes.io/aws-load-balancer-proxy-protocol": "*" annotation to service and enable PROXY protocol in HAPRoxy for "aws" provider.