bug: response-rewrite filters.regex does not apply when content-encoding: gzip

[JoonaHa]

Current Behavior

I have an APISIX route that uses proxy-rewrite to proxy users to specific endpoint. The endpoint response is a JSON payload with content headers Content-Type: application/json and content-encoding: gzip.

When I add response-rewrite plugin with filters.regex APISIX drops the content-encoding: gzip header. When I add the header back manually using response-rewrite's headers.add functionality, the filters.regex replacement is not applied.

Here is my route configuration with placeholder urls:

{
    "id":"testroute",
    "methods":[
        "GET"
    ],
    "plugins":{
        "proxy-rewrite":{
            "uri":"/specific/path",
            "host":"example.org"
        },
        "response-rewrite":{
            "headers":{
                "add":[
                    "content-encoding: gzip"
                ]
            },
            "filters":[
                {
                    "regex":"conformsTo",
                    "scope":"global",
                    "replace":"replacedvalue"
                }
            ],
            "vars":[
                [
                    "status",
                    "==",
                    200
                ]
            ]
        }
    },
    "upstream":{
        "type":"roundrobin",
        "scheme":"https",
        "nodes":{
            "example.org:443":1
        }
    },
    "uri":"/testroute"
}

Expected Behavior

Upstream's content-encoding: gzip header is passed normally trough APISIX and response-rewrite's filters.regex is applied to the response JSON.

Error Logs

No response

Steps to Reproduce

1. Have an endpoint which responds with headers: Content-Type: application/json and content-encoding: gzip
2. Make a APISIX route with proxy-rewrite and response-rewrite
3. See if response-rewrite is applied with correct encoding headers.

Environment

APISIX Docker image apache/apisix:3.7.0-debian

[yuweizzz]

In my options, when upstream return content-encoding means the body has been compressed, so the filters.regex may not work it the compressed body, manually using response-rewrite's headers.add should not work.

[shreemaan-abhishek]

this isn't a bug, but gzip encoded data isn't supported for rewrite.

[JoonaHa]

Thank you for checking. I would have presumed that Apisix would handle different kind of content-encoding settings under the hood, so this feature would be much appreciated.

[yuweizzz]

I want to try it, assign this issue to me?

[JoonaHa]

For a workaround I used proxy-rewrite's headers to set Accept-Encoding to an empty value so the upstream server doesn't respond with an compressed payload.

{
   "id":"testroute",
   "methods":[
      "GET"
   ],
   "plugins":{
      "proxy-rewrite":{
         "uri":"/specific/path",
         "host":"example.org",
         "headers":{
            "Accept-Encoding": ""
         }
      },
      "response-rewrite":{
         "filters":[
            {
               "regex":"conformsTo",
               "scope":"global",
               "replace":"replacedvalue"
            }
         ],
         "vars":[
            [
               "status",
               "==",
               200
            ]
         ]
      }
   },
   "upstream":{
      "type":"roundrobin",
      "scheme":"https",
      "nodes":{
         "example.org:443":1
      }
   },
   "uri":"/testroute"
}

[yuweizzz]

use "Accept-Encoding": "" tell upstream don't compress the payload, it need to work with proxy-rewrite, the another way is disable compression on upstream, it doesn't need a new header.

[yuweizzz]

Do you think we still need to support encoded data in response-rewrite? @shreemaan-abhishek @JoonaHa

[JoonaHa]

@yuweizzz By default APISIX seems to accepts gzip encoded data so the ideal situation would be for response-rewrite to also support different encoding methods.
Or at least mention the limitations of response-rewrite in the documentation.

[monkeyDluffy6017]

@JoonaHa please help to review #10637