bug: error in gPRC protocol as reverse proxy

[Rustixir]

Current Behavior

hi, I wanted use apisix, but after I created route and set gPRC protocol for communicate with upstream,
I test that route with postman, apisix raise error:

2024/01/17 12:30:03 [error] 50#50: *158580 upstream sent too large http2 frame: 4740180 while reading response header from upstream, client: 172.10.0.1, server: _, request: "POST /ping HTTP/1.0", upstream: "grpc://172.10.0.3:9081", host: "aaa.com"

after I searched about that , I found some article about is nginx problem and with increasing ( proxy_buffer_size ) be fix, but this solution isn't work for me.

in one article said about increase buffer size,
I add this into config file and mount to " /usr/local/apisix/conf/config.yaml "

my config was :

nginx_config:
  http_configuration_snippet: |
    proxy_buffer_size          512k;
    proxy_buffers              4 256k;
    proxy_busy_buffers_size    512k;
    http2_max_field_size       16k;
    http2_max_header_size      32k;

Steps to Reproduce

apisix docker image ~> apache/apisix:3.7.0-debian

this is my route config ( apisix-dashboard format ) :

{
  "uri": "/ping",
  "name": "/vouchers/otp",
  "methods": [
    "POST"
  ],
  "plugins": {
    "grpc-transcode": {
      "_meta": {
        "disable": true
      },
      "method": "/scms_api.VoucherService/Otp",
      "proto_id": "voucher",
      "service": "scms"
    }
  },
  "upstream": {
    "nodes": [
      {
        "host": "scms",
        "port": 9081,
        "weight": 1
      }
    ],
    "timeout": {
      "connect": 6,
      "send": 6,
      "read": 6
    },
    "type": "roundrobin",
    "scheme": "grpc",
    "pass_host": "pass",
    "keepalive_pool": {
      "idle_timeout": 60,
      "requests": 1000,
      "size": 320
    }
  },
  "status": 1
}

[sheharyaar]

@Rustixir , can you share the response or request that triggered this error.

[shreemaan-abhishek]

what was the size of your response?

[Rustixir]

after done configuration, I implemented route and just wanted test my first endpoint
but this error is occured for all gPRC request .

the client request for "/ping" path with POST method
and body is { "mobile": "" }

And below is our proto model

service VoucherService {
      rpc Otp (VoucherOtpReq) returns (VoucherOtpRes);
}

message VoucherOtpReq{
  // @inject_tag: json:"mobile"
  string Mobile = 1;
}

message VoucherOtpRes{
  // @inject_tag: json:"life_time"
  int64 LifeTime = 1;
  // @inject_tag: json:"length"
  int32 Length = 2;
}

everything to be seems normal

[hanqingwu]

@Rustixir
maybe this can help you ?

4740180 is ascii-encoded HTT so it appears it's getting stumbled on HTTP/1 reply when reading from upstream connection socket.

kubernetes/ingress-nginx#4095