[Filebeat] S3 input issue for cloudtrail: createEventsFromS3Info failed

[lliknart]

I have the following error when aws module is enable

• createEventsFromS3Info failed
• gzip.NewReader failed: gzip: invalid header

According to the release notes for beats 7.6.0, it should be fixed

• related PR: [Filebeat] Check content type when reading s3 files #15252
• other related issue: [filebeat] Gzip does not work with different content type for s3 input #15225

filebeat[8549]: 2020-02-17T16:22:37.354Z        WARN        [s3]        s3/input.go:277        Processing message failed, updating visibility timeout
filebeat[8549]: 2020-02-17T16:22:37.355Z        ERROR        [s3]        s3/input.go:434        gzip.NewReader failed: gzip: invalid header
filebeat[8549]: 2020-02-17T16:22:37.356Z        ERROR        [s3]        s3/input.go:386        createEventsFromS3Info failed for AWSLogs/xxxxxxxxxxxx/CloudTrail/us-east-1/2020/02/15/xxxxxxxxxxxx_CloudTrail_us-east-1_20200215T0900Z_iJSOIOJP6ErPw7vU.json.gz: gzip.NewReader failed: gzip: invalid header
filebeat[8549]: 2020-02-17T16:22:37.356Z        WARN        [s3]        s3/input.go:277        Processing message failed, updating visibility timeout
filebeat[8549]: 2020-02-17T16:22:37.356Z        INFO        [s3]        s3/input.go:282        Message visibility timeout updated to 300
filebeat[8549]: 2020-02-17T16:22:37.357Z        INFO        [s3]        s3/input.go:282        Message visibility timeout updated to 300
filebeat[8549]: 2020-02-17T16:22:37.358Z        INFO        [s3]        s3/input.go:282        Message visibility timeout updated to 300
filebeat[8549]: 2020-02-17T16:22:37.360Z        INFO        [s3]        s3/input.go:282        Message visibility timeout updated to 300
filebeat[8549]: 2020-02-17T16:22:37.365Z        ERROR        [s3]        s3/input.go:434        gzip.NewReader failed: gzip: invalid header
filebeat[8549]: 2020-02-17T16:22:37.365Z        ERROR        [s3]        s3/input.go:386        createEventsFromS3Info failed for AWSLogs/xxxxxxxxxxxx/CloudTrail/eu-west-1/2020/02/15/xxxxxxxxxxxx_CloudTrail_eu-west-1_20200215T1015Z_C9pEYy8aTobIJdDL.json.gz: gzip.NewReader failed: gzip: invalid header
filebeat[8549]: 2020-02-17T16:22:37.365Z        WARN        [s3]        s3/input.go:277        Processing message failed, updating visibility timeout
filebeat[8549]: 2020-02-17T16:22:37.370Z        INFO        [s3]        s3/input.go:282        Message visibility timeout updated to 300
filebeat[8549]: 2020-02-17T16:22:37.390Z        ERROR        [s3]        s3/input.go:434        gzip.NewReader failed: gzip: invalid header
filebeat[8549]: 2020-02-17T16:22:37.390Z        ERROR        [s3]        s3/input.go:386        createEventsFromS3Info failed for AWSLogs/xxxxxxxxxxxx/CloudTrail/ca-central-1/2020/02/15/xxxxxxxxxxxx_CloudTrail_ca-central-1_20200215T1845Z_OY8MOAzSebm6DKum.json.gz: gzip.NewReader failed: gzip: invalid header

For confirmed bugs, please report:

• Version: 7.6.0
  filebeat version 7.6.0 (amd64), libbeat 7.6.0 [6a23e8f8f30f5001ba344e4e54d8d9cb82cb107c built 2020-02-05 23:06:45 +0000 UTC]
• Operating System: Ubuntu
• Steps to Reproduce:
  Hereafter the configuration file used

filebeat.yml

filebeat.config.modules.path: /etc/filebeat/modules.d/*.yml

##==================== Output Logstash settings =====================
output.logstash:
  hosts: ["localhost:5044"]

module aws.yml

  cloudtrail:
    enabled: true

    # AWS SQS queue url
    var.queue_url: https://sqs.region.amazonaws.com/xxxxxxxxxxxx/queue_name

[kaiyan-sheng]

Thanks for posting this @lliknart ! Could you check what is the Metadata under your CloudTrail logs in S3? For example like this:

[kaiyan-sheng]

@lliknart Please ignore my previous comment. I was able to reproduce this bug by using the Metadata shown above. I just created a PR to fix this. Thanks!

[nc-andersenb]

I am also impacted by this bug. Is there any way to implement this as a patch while waiting for the PR to be merged and released? Or must I just wait?

[kaiyan-sheng]

Sorry this PR is merged several hours too late for 7.6.1...
@nc-andersenb Could you build Filebeat locally and use it till next release?

[kaiyan-sheng]

Still failing, reopen the issue.

[nithyagomathi]

Hi,

I'm having the same issue for Cloudtrail logs from the S3 bucket. But the metadata of cloud trail looks just like you mentioned in the above screenshot. And I'm using Filebeat version 7.6.1

Below is the error:

2020-04-01T17:42:02.454+0530 ERROR [s3] s3/input.go:447 gzip.NewReader failed: gzip: invalid header
2020-04-01T17:42:02.454+0530 ERROR [s3] s3/input.go:386 createEventsFromS3Info failed

[kaiyan-sheng]

@nithyagomathi Hi! Are you using Cloudtrail fileset or s3 input separately? If you can post your config here, that would be great! Or if you prefer in the discuss forum(https://discuss.elastic.co/c/beats/filebeat), that would be even better 😄 Thanks!!

[nithyagomathi]

Hi @kaiyan-sheng

This is my config:

> filebeat.inputs:
> - type: s3
>   queue_url: https://sqs.us-west-2.amazonaws.com/XXXXXXXXXX/sqs-name
>   visibility_timeout: 300s
>   credential_profile_name: default
> cloud.id: "cloudid"
> cloud.auth: "elastic:{password}"

And I also created a topic in the forum ([https://discuss.elastic.co/t/filebeat-reading-logs-from-s3/226065])