Add GZip Request Compression

[johnlcox]

The way I'm using retrofit/okhttp most of the requests are POSTs with JSON bodies that aren't huge (2KB - 30KB), but happen fairly frequently. The server APIs support gzip compressed requests and so I spent some time trying to find a quick way to use gzip compression with retrofit/okhttp and it looks like it wouldn't be possible without modifications to okhttp itself.

Is there any possibility of this being added or accepted as a pull request?

[swankjesse]

Yeah, though things get complicated when you layer the content-encoding over the transfer-encoding. Plus retries and supporting both HTTP and SPDY.

Any chance of converting your server to SPDY? I think SPDY guarantees that the serverside supports gzip, which will allow us to avoid a configuration option.

[johnlcox]

Our servers are IIS which I don't think support SPDY, so we wouldn't be able to switch to that.

We aren't in a hurry to add GZIP compression to our requests. It would be a nice to have eventually. I was just investigating how easy it would be to do, and it appears it would likely be a good amount of work.

[Turbo87]

I would appreciate that option too for larger uploads...

[swankjesse]

Yeah, seems like a good win. I'm trying to think of a way we can expose this in the API. Presumably some servers won't be able to handle it so we can't give this behavior to everyone.

[swankjesse]

For the time being your best option is to do it manually: compress the content and add Content-Encoding: gzip.

Note that gzip probably won't help for uploads whose payload is already compressed. For example, images and video.

[cowtowncoder]

One suggestion I have is to make sure that whoever tries to do compression uses simple heuristics to not compress, if a well-known header of gzip/deflate is seen. I use that on another project (also check for signature of LZF compression).

[JakeWharton]

You now have the option to do this via an interceptor. See: https://github.com/square/okhttp/wiki/Interceptors#rewriting-requests

[balazsgerlei]

Hi!

I tried to use the GzipRequestInterceptor Jake linked in the last comment, but the webserver I'm trying to reach returns the response "411 Length Required".
I noticed that the GzipRequestInterceptor sets the compressed RequestBody's contentLength to -1. This must be the problem, and I can't find a way to determine the length of the compressed body. How can I do that?

[JakeWharton]

You need to buffer the entire body by writing it to a Buffer. Then you can
get its size to use when writing.

On Sat, Apr 18, 2015, 1:03 PM Gerlot [email protected] wrote:

Hi!

I tried to use the GzipRequestInterceptor Jake linked in the last comment,
but the webserver I'm trying to reach returns the response "411 Length
Required".
I noticed that the GzipRequestInterceptor sets the compressed
RequestBody's contentLength to -1. This must be the problem, and I can't
find a way to determine the length of the compressed body. How can I do
that?

—
Reply to this email directly or view it on GitHub
#350 (comment).

[balazsgerlei]

Yeah, I've just figured it out with the help of a partly related conversation:
#1349

Maybe you should add this solution to the linked page about interceptors. It took me 1.5 hour to figure it out.
Btw I did it like this (maybe it will be useful for others):

private RequestBody requestBodyWithContentLength(final RequestBody requestBody) throws IOException {
        final Buffer buffer = new Buffer();
        try {
            requestBody.writeTo(buffer);
        } catch (IOException e) {
            throw new IOException("Unable to copy RequestBody");
        }
        return new RequestBody() {
            @Override
            public MediaType contentType() {
                return requestBody.contentType();
            }

            @Override
            public long contentLength() {
                return buffer.size();
            }

            @Override
            public void writeTo(BufferedSink sink) throws IOException {
                sink.write(ByteString.read(buffer.inputStream(), (int) buffer.size()));
            }
        };
    }

And I pass the whole thing to the Request like this:

Request compressedRequest = originalRequest.newBuilder()
                .header("Content-Encoding", "gzip")
                .method(originalRequest.method(), requestBodyWithContentLength(gzip(originalRequest.body())))
                .build();

[JakeWharton]

sink.write(ByteString.read(buffer.inputStream(), (int) buffer.size()));

This is wildly inefficient.

At the very least do:

sink.writeAll(buffer);

Or a bit more correctly:

ByteString body = buffer.snapshot();

// .. RequestBody code
sink.write(body);

[balazsgerlei]

Ok, thanks for the correction!

[felipecsl]

Here's the full gzip interceptor with content length, to whom it may concern:

class GzipRequestInterceptor implements Interceptor {
  @Override public Response intercept(Chain chain) throws IOException {
    Request originalRequest = chain.request();
    if (originalRequest.body() == null || originalRequest.header("Content-Encoding") != null) {
      return chain.proceed(originalRequest);
    }

    Request compressedRequest = originalRequest.newBuilder()
        .header("Content-Encoding", "gzip")
        .method(originalRequest.method(), forceContentLength(gzip(originalRequest.body())))
        .build();
    return chain.proceed(compressedRequest);
  }

  /** https://github.com/square/okhttp/issues/350 */
  private RequestBody forceContentLength(final RequestBody requestBody) throws IOException {
    final Buffer buffer = new Buffer();
    requestBody.writeTo(buffer);
    return new RequestBody() {
      @Override
      public MediaType contentType() {
        return requestBody.contentType();
      }

      @Override
      public long contentLength() {
        return buffer.size();
      }

      @Override
      public void writeTo(BufferedSink sink) throws IOException {
        sink.write(buffer.snapshot());
      }
    };
  }

  private RequestBody gzip(final RequestBody body) {
    return new RequestBody() {
      @Override public MediaType contentType() {
        return body.contentType();
      }

      @Override public long contentLength() {
        return -1; // We don't know the compressed length in advance!
      }

      @Override public void writeTo(BufferedSink sink) throws IOException {
        BufferedSink gzipSink = Okio.buffer(new GzipSink(sink));
        body.writeTo(gzipSink);
        gzipSink.close();
      }
    };
  }
}

[kamoljan]

If this issue is still open, +1

[galex]

Looking for this as well.

@felipecsl 's solution does not work on Okhttp 2.5.0 (last one), as I get a TimeoutException. I think the content Length and the size of the gzipped body do not match, so the server is waiting for more.

[felipecsl]

when you gzip you dont know the contentLength in advance (see it returns -1) above

[minliang1219]

If I want to compress my request body, and the receiver returns compressed data service, then my request header need to add '' content-encoding "and" accept-encoding "?

[swankjesse]

This is implemented by a straightforward interceptor. I can’t see a reasonable safe way to turn it on by default otherwise! No other action to take here.

[balazsgerlei]

Is the interceptor that @felipecsl wrote still works with Retrofit2 and OkHttp3?

[felipecsl]

@balazsgerlei it does

[laoxiao79]

Here is my method:

    public class GzipRequestInterceptor implements Interceptor {
        @Override
        public Response intercept(Chain chain) throws IOException {
            Request originalRequest = chain.request();
            if(originalRequest.body() == null){
                Request newRequest = originalRequest.newBuilder()
                        .header("Content-Encoding", "gzip")
                        .build();
                Log.i(TAG, "GzipRequestInterceptor　newRequest.toString():" + newRequest.toString());
                return chain.proceed(newRequest);
            }

            if (originalRequest.header("Content-Encoding") != null) {
                return chain.proceed(originalRequest);
            }
            Request compressedRequest = originalRequest.newBuilder()
                    .header("Content-Encoding", "gzip")
                    .method(originalRequest.method(), forceContentLength(gzip(originalRequest.body())))
                    .build();
            return chain.proceed(compressedRequest);
        }

        /**
         * https://github.com/square/okhttp/issues/350
         */
        private RequestBody forceContentLength(final RequestBody requestBody) throws IOException {
            final Buffer buffer = new Buffer();
            requestBody.writeTo(buffer);
            return new RequestBody() {
                @Override
                public MediaType contentType() {
                    return requestBody.contentType();
                }

                @Override
                public long contentLength() {
                    return buffer.size();
                }

                @Override
                public void writeTo(BufferedSink sink) throws IOException {
                    sink.write(buffer.snapshot());
                }
            };
        }

        private RequestBody gzip(final RequestBody body) {
            return new RequestBody() {
                @Override
                public MediaType contentType() {
                    return body.contentType();
                }

                @Override
                public long contentLength() {
                    return -1; // We don't know the compressed length in advance!
                }

                @Override
                public void writeTo(BufferedSink sink) throws IOException {
                    BufferedSink gzipSink = Okio.buffer(new GzipSink(sink));
                    body.writeTo(gzipSink);
                    gzipSink.close();
                }
            };
        }
    }

[laoxiao79]

or:

 builder.addInterceptor(new Interceptor() {
            @Override
            public Response intercept(Chain chain) throws IOException {
                Log.i(TAG,"GzipRequestInterceptor　chain.request().toString():"+chain.request().toString());
                Request request = chain.request()
                        .newBuilder()
                        .header("Content-Encoding", "gzip")
                        .build();
                Log.i(TAG,"GzipRequestInterceptor　request.toString():"+request.toString());
                return chain.proceed(request);
            }
        });

[karussell]

This is implemented by a straightforward interceptor. I can’t see a reasonable safe way to turn it on by default otherwise! No other action to take here.

@swankjesse would it make sense to put the GzipRequestInterceptor in okhttp? I've copied this now to 3 applications ;) ... or is this already available somewhere and I overlooked it?

[swankjesse]

@karussell how do you decide which requests to gzip?

[karussell]

Good question :). If I add this interceptor I would probably assume all requests are gzipped. Or one could configure the interceptor to gzip only under special conditions like "too big".

[swankjesse]

Gotcha. And you know your webservers will accept Gzip. The problem with making it on-by-default is that not all servers do unfortunately.

[karussell]

Yes, the webserver has to support this. If not, it will likely respond with an explicit error of the unsupported Content-Encoding.

If the GzipInterceptor has the code as above we can do requests without gzipping even with the GzipInterceptor added by setting the Content-Encoding header to identity I think
(leading to != null)

[ashraf-atef]

@felipecsl I tried your solution but unfortunately the response body is omitted from the log of http logging interceptor but I tried to log the request body by myself but the compression text was so weird.

Original request body: {"data":{"lng":"-122.0840926","lat":"37.4219524"},"device_id":"XXXXX"}
Compressed request body: ���������������9
� �����ڈ�h.�\� �$�x���L�0��D��x��������=���ኤӤ��r�Z*w�{D�A��J�ՒNX?��r�Q������

So why do you think this is happening?