nginx ingress not working with grpc services

[gameofdatas]

I am trying to get nginx ingress running as a Reverse proxy for my gRPC service.

my deployment.yml file looks like

          apiVersion: apps/v1
          kind: Deployment
          metadata:
            name: service-app
            labels:
              k8s-app: service-app
            namespace: service
          spec:
            replicas: 1
            selector:
              matchLabels:
                k8s-app: service-app
            template:
              metadata:
                labels:
                  k8s-app: service-app
              spec:
                containers:
                - name: service
                  image: "{{ ecr_repo }}/service:{{ git_id['stdout'] }}"
                  env:
                    - name: PORT
                      value: ":50051"
                  ports:
                  - containerPort: 50051
                    name: grpc

my svc.yml file looks like

         apiVersion: v1
         kind: Service
         metadata:
           name: service
           namespace: service
           labels:
             k8s-app: service-app
         spec:
           ports:
                 - port: 50051
                   targetPort: 50051
                   name: grpc
           selector:
             k8s-app: service-app

my ingress.yml file looks like

         apiVersion: networking.k8s.io/v1beta1
         kind: Ingress
         metadata:
             annotations:
                 kubernetes.io/ingress.class: "nginx"
                 nginx.ingress.kubernetes.io/ssl-redirect: "true"
                 nginx.ingress.kubernetes.io/backend-protocol: "GRPC"
             name: service
             namespace: service
         spec:
             rules:
               - host: 
                 http:
                   paths:
                     - path:
                       backend:
                         serviceName: service
                         servicePort: grpc

When I do get ingress I get

NAME              HOSTS   ADDRESS       PORTS   AGE
service   *       10.0.213.16   80      9m44s

When I hit the alb arn with port 80, then I get 400 error similar to this

"PRI * HTTP/2.0" 400 157 "-" "-" 0 0.040 [] [] - - - - 9098e278e6c5ee7e1a2a4ecf02c6c2a1

Can any mistakes be pointed in the yml files of ingress deployment. It will be very helpful to get this thing running.

[aledbf]

nginx.ingress.kubernetes.io/ssl-redirect: "true"

You don't have a TLS section, without that GRPC will not work. #4095 (comment)

[aledbf]

@Rahul7794 do you have more services? Why are you not defining a host in the ingress rule?

[gameofdatas]

@aledbf I have updated the Ingress YAML file. I have added the TLS section and still, I am facing the same issue.

my ingress yml file looks like:

          apiVersion: networking.k8s.io/v1beta1
          kind: Ingress
          metadata:
              annotations:
                  kubernetes.io/ingress.class: "nginx"
                  nginx.ingress.kubernetes.io/ssl-redirect: "true"
                  nginx.ingress.kubernetes.io/backend-protocol: "GRPC"
              name: service
              namespace: service
          spec:
              rules:
                - host: sample.grpc.com
                  http:
                    paths:
                      - path:
                        backend:
                          serviceName: service
                          servicePort: grpc
              tls:
                # This secret must exist beforehand
                # The cert must also contain the subj-name fortune-teller.stack.build
                # https://github.com/kubernetes/ingress-nginx/blob/master/docs/examples/PREREQUISITES.md#tls-certificates
                - secretName: tls-sercet
                  hosts:
                    - sample.grpc.com

I have a grpc client written in go, and when I hit the server with the alb arn on port 443 it still return 400
[14/Jul/2020:00:43:39 +0000] "PRI * HTTP/2.0" 400 157 "-" "-" 0 0.031 [] [] - - - - 1704fb9e97219681ec53b95887bf6c95

I tried port forwarding and port forwarding was working fine. But not the ingress.

When I describe ingress, I get following result

Name:             service
Namespace:    service
Address:          10.0.213.16
Default backend:  default-http-backend:80 (10.0.214.250:8080)
TLS:
  tls-sercet terminates sample.grpc.com
Rules:
  Host             Path  Backends
  ----             ----  --------
  sample.grpc.com  
                      service:grpc ()
Annotations:
  kubernetes.io/ingress.class:                   nginx
  nginx.ingress.kubernetes.io/backend-protocol:  GRPC
  nginx.ingress.kubernetes.io/ssl-redirect:      true
Events:
  Type    Reason  Age   From                      Message
  ----    ------  ----  ----                      -------
  Normal  CREATE  13m   nginx-ingress-controller  Ingress service/service
  Normal  UPDATE  12m   nginx-ingress-controller  Ingress service/service

[aledbf]

@Rahul7794 I cannot reproduce the issue

Create a cluster using kind: https://kind.sigs.k8s.io/docs/user/ingress/#create-cluster
Install ingress-nginx: https://kind.sigs.k8s.io/docs/user/ingress/#ingress-nginx

# change the IP address
export MY_IP=127.0.0.1

# create an SSL certificate
openssl req -x509 -newkey rsa:4096 -sha256 -days 3650 -nodes \
  -keyout xip.key -out xip.crt -subj "/CN=$MY_IP.xip.io" \
  -addext "subjectAltName=DNS:$MY_IP.xip.io"

kubectl create secret tls xip-tls --cert=xip.crt --key=xip.key

echo "
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: grpcbin
  annotations:
    nginx.ingress.kubernetes.io/backend-protocol: GRPC
spec:
  tls:
    - hosts:
      - $MY_IP.xip.io
      secretName: xip-tls
  rules:
  - host: $MY_IP.xip.io
    http:
      paths:
      - path: /
        backend:
          serviceName: grpcbin
          servicePort: 9000

---
kind: Pod
apiVersion: v1
metadata:
  name: grpcbin
  labels:
    app: grpcbin
spec:
  containers:
  - name: grpcbin
    image: moul/grpcbin

---
kind: Service
apiVersion: v1
metadata:
  name: grpcbin
spec:
  selector:
    app: grpcbin
  ports:
  - name: plain
    port: 9000
  - name: secured
    port: 9001

" | kubectl apply -f -

# https://github.com/fullstorydev/grpcurl
grpcurl -v -insecure $MY_IP.xip.io:443 hello.HelloService/SayHello

Resolved method descriptor:
{
  "name": "SayHello",
  "inputType": ".hello.HelloRequest",
  "outputType": ".hello.HelloResponse",
  "options": {
    
  }
}

Request metadata to send:
(empty)

Response headers received:
content-type: application/grpc
strict-transport-security: max-age=15724800; includeSubDomains
server: nginx/1.19.1
date: Tue, 14 Jul 2020 02:05:03 GMT


Response contents:
{
  "reply": "hello noname"
}

Response trailers received:
(empty)
Sent 0 requests and received 1 response

[gameofdatas]

@aledbf the grpc curl is done from the cluster right? grpcurl -v -insecure $MY_IP.xip.io:443 hello.HelloService/SayHello

[aledbf]

@aledbf the grpc curl is done from the cluster right? grpcurl -v -insecure $MY_IP.xip.io:443 hello.HelloService/SayHello

No, my machine (kind cluster is exposing ports 80 and 443 with hostPort)

[gameofdatas]

I am getting Failed to dial target host "127.0.0.1.xip.io:443": dial tcp 127.0.0.1:443: connect: connection refused

[aledbf]

I am getting Failed to dial target host "127.0.0.1.xip.io:443": dial tcp 127.0.0.1:443: connect: connection refused

Please check the kind links

[gameofdatas]

@aledbf I tried the https://kind.sigs.k8s.io/docs/user/ingress/#create-cluster but no success. I tried with the fortune teller example and I was not able to run that too.

Ingress definition :

          apiVersion: networking.k8s.io/v1beta1
          kind: Ingress
          metadata:
            annotations:
              nginx.ingress.kubernetes.io/backend-protocol: "GRPC"
            name: fortune-ingress
            namespace: default
          spec:
              rules:
                - host: 127.0.0.1.xip.io
                  http:
                    paths:
                      - path: /
                        backend:
                          serviceName: fortune-teller-service
                          servicePort: grpc
              tls:
                # This secret must exist beforehand
                # The cert must also contain the subj-name fortune-teller.stack.build
                # https://github.com/kubernetes/ingress-nginx/blob/master/docs/examples/PREREQUISITES.md#tls-certificates
                - secretName: scret-tls-1
                  hosts:
                    - 127.0.0.1.xip.io

but still ending up getting Failed to dial target host "127.0.0.1.xip.io:443": context deadline exceeded
when I run grpcurl -insecure $MY_IP.xip.io:443 build.stack.fortune.FortuneTeller/Predict

and even I don't get any logs of incoming request on my ingress controller

[samsends]

@Rahul7794 Hey, we are having an identical issue on AWS. From the testing we have done, it seems like an AWS specific nginx deployment may be the problem (https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.34.1/deploy/static/provider/aws/deploy.yaml).

Are you using AWS?

Have you made any progress?

[gameofdatas]

@sbsends hey, Yes I am using AWS, and I haven't made any progress on the issue, I cannot just make it work.
Meanwhile, I just moved on to use Envoy with headless service in Kubernetes and it serves my purpose. For info on using Envoy you can refer here

[gameofdatas]

@sbsends you can also refer here for Envoy setting if you are looking to move ahead with other Ingresses.

[a75c6]

May want to set a specific path value for the grpc service that aligns with the proto file. Currently you have an empty path value or path: /.

Examples:

- path: /build.stack.fortune.FortuneTeller
- path: /hello.HelloService

This allows the ingress to route the grpc requests specifically to the service.

In this case the ingress class annotations are not complete whereas here they are present.

May also need to add nginx.ingress.kubernetes.io/backend-protocol: "GRPCS" as indicated here.

[waelghaith123]

Your gRPC working with a secure channel ?!
What is Nginx Ingress LoadBalancer type ?!

[thatInfrastructureGuy]

Client side workaround which worked for me: #4622 (comment).

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[phuongTP1998]

I am getting the same issue

[originswift-sys]

Me too. GRPC Web works though. A real pain