[filebeat] Gzip does not work with different content type for s3 input

[nathankodilla]

With filebeat 7.5.0 gzip no longer works if the Content-Type is set to text/plain; charset=utf-8 and Content-Encoding of gzip. This worked on 7.4.2. However, we have a separate bucket that has Content-Type set to application/x-gzip for the files and that works in 7.5.0.

Logs:

ERROR [s3] s3/input.go:259 handleS3Objects failed: newS3BucketReader failed: Failed to decompress gzipped file s3file.txt.gz: gzip: invalid header
panic: send on closed channel

goroutine 121 [running]:

github.com/elastic/beats/x-pack/filebeat/input/s3.(*s3Input).processMessage(0xc0000db340, 0x34b5c20, 0xc000484190, 0x0, 0xc0009d0250, 0xc0009d02d0, 0x0, 0x0, 0xc0009d0300, 0xc0009d0330, ...)

/go/src/github.com/elastic/beats/x-pack/filebeat/input/s3/input.go:260 +0x2f6

created by github.com/elastic/beats/x-pack/filebeat/input/s3.(*s3Input).processor

/go/src/github.com/elastic/beats/x-pack/filebeat/input/s3/input.go:240 +0x19c

Originally opened here:
https://discuss.elastic.co/t/filebeats-7-5-0-issue-with-s3-input-with-gzip-files/210800

[kaiyan-sheng]

Hi @nathankodilla, could you help me reproduce this problem please? What does your log file in S3 bucket looks like? Does the file name ends with .gz? Thanks!!

[nathankodilla]

@kaiyan-sheng The filename ends in .txt.gz and the following metadata:

Inside the file is json formatted logs on individual lines.

[kaiyan-sheng]

I'm able to reproduce this issue: for a .gz file with Content-Type: application/x-gzip, the s3 input reads the file just fine. But when the file metadata property changed to Content-Type: text/plain, s3 input fails with: 2019-12-23T11:12:01.255-0700 ERROR [s3] s3/input.go:259 handleS3Objects failed: newS3BucketReader failed: Failed to decompress gzipped file test-log-3.txt.gz: unexpected EOF.