Check when response content type is nil #28457

kaiyan-sheng · 2021-10-14T23:04:47Z

What does this PR do?

Filebeat failed to process any SQS message with aws-s3 input and here is the error in Filebeat log:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x10f339af4]

goroutine 42 [running]:
github.com/elastic/beats/v7/x-pack/filebeat/input/awss3.(*s3ObjectProcessor).download(0xc0007e22c0)
        /Users/kaiyansheng/go/src/github.com/elastic/beats/x-pack/filebeat/input/awss3/s3_objects.go:181 +0xf4
github.com/elastic/beats/v7/x-pack/filebeat/input/awss3.(*s3ObjectProcessor).ProcessS3Object(0xc0007e22c0)
        /Users/kaiyansheng/go/src/github.com/elastic/beats/x-pack/filebeat/input/awss3/s3_objects.go:136 +0x165
github.com/elastic/beats/v7/x-pack/filebeat/input/awss3.(*sqsS3EventProcessor).processS3Events(0xc0005f4190, {0x110a71950, 0xc000522100}, 0xc0003359e0, {0xc000537400, 0x203000})
        /Users/kaiyansheng/go/src/github.com/elastic/beats/x-pack/filebeat/input/awss3/sqs_s3_event.go:245 +0x49f
github.com/elastic/beats/v7/x-pack/filebeat/input/awss3.(*sqsS3EventProcessor).ProcessSQS(0xc0005f4190, {0x110a71950, 0xc000522100}, 0xc0007dc440)
        /Users/kaiyansheng/go/src/github.com/elastic/beats/x-pack/filebeat/input/awss3/sqs_s3_event.go:111 +0x246
github.com/elastic/beats/v7/x-pack/filebeat/input/awss3.(*sqsReader).Receive.func1({{}, 0xc000687da0, 0xc000b2c0f0, 0xc000b2c120, 0x0, 0x0, 0xc000b2c150, 0xc000b2c180}, {0xc052496a37d38cb8, 0x287b04758, ...})
        /Users/kaiyansheng/go/src/github.com/elastic/beats/x-pack/filebeat/input/awss3/sqs.go:90 +0x1b8
created by github.com/elastic/beats/v7/x-pack/filebeat/input/awss3.(*sqsReader).Receive
        /Users/kaiyansheng/go/src/github.com/elastic/beats/x-pack/filebeat/input/awss3/sqs.go:82 +0x595

This PR is to check if GetObject API call response content type exists.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

mergify · 2021-10-14T23:04:55Z

This pull request does not have a backport label. Could you fix it @kaiyan-sheng? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-v./d./d./d is the label to automatically backport to the 7./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

elasticmachine · 2021-10-14T23:06:53Z

Pinging @elastic/integrations (Team:Integrations)

elasticmachine · 2021-10-14T23:09:13Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2021-10-15T16:35:18.611+0000
Duration: 104 min 14 sec
Commit: 6cf48ff

Test stats 🧪

Test	Results
Failed	0
Passed	7717
Skipped	1201
Total	8918

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.

andrewkroh

LGTM

aspacca · 2021-10-15T07:38:45Z

LGTM, can you add a test case on s3_objects_test.go? @kaiyan-sheng
see the one I've already added for resp = nil at https://github.com/elastic/beats/blob/master/x-pack/filebeat/input/awss3/s3_objects_test.go#L152-L171

aspacca · 2021-10-15T16:50:46Z

x-pack/filebeat/input/awss3/s3_objects_test.go

 		},
 	}
+
+	if contentType != "" {


(cherry picked from commit ccc2206)

(cherry picked from commit ccc2206) Co-authored-by: kaiyan-sheng <[email protected]>

Check when response content type is nil

e5b1ba7

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Oct 14, 2021

mergify bot added the backport-skip Skip notification from the automated backport with mergify label Oct 14, 2021

add changelog

308bc12

kaiyan-sheng added backport-v7.15.0 Automated backport with mergify backport-v7.16.0 Automated backport with mergify Team:Integrations Label for the Integrations team labels Oct 14, 2021

kaiyan-sheng self-assigned this Oct 14, 2021

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Oct 14, 2021

kaiyan-sheng requested a review from aspacca October 14, 2021 23:06

mergify bot removed the backport-skip Skip notification from the automated backport with mergify label Oct 14, 2021

andrewkroh approved these changes Oct 15, 2021

View reviewed changes

aspacca approved these changes Oct 15, 2021

View reviewed changes

add unit test

6cf48ff

aspacca reviewed Oct 15, 2021

View reviewed changes

x-pack/filebeat/input/awss3/s3_objects_test.go

},

}

if contentType != "" {

Copy link

aspacca Oct 15, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

kaiyan-sheng merged commit ccc2206 into elastic:master Oct 15, 2021

kaiyan-sheng deleted the fix_aws_s3 branch October 15, 2021 18:49

mergify bot pushed a commit that referenced this pull request Oct 15, 2021

Check when response content type is nil (#28457)

ba190ec

(cherry picked from commit ccc2206)

mergify bot mentioned this pull request Oct 15, 2021

[7.x](backport #28457) Check when response content type is nil #28477

Merged

mergify bot pushed a commit that referenced this pull request Oct 15, 2021

Check when response content type is nil (#28457)

b4d3527

(cherry picked from commit ccc2206)

mergify bot mentioned this pull request Oct 15, 2021

[7.15](backport #28457) Check when response content type is nil #28478

Merged

kaiyan-sheng added a commit that referenced this pull request Oct 15, 2021

Check when response content type is nil (#28457) (#28477)

38b3d0d

(cherry picked from commit ccc2206) Co-authored-by: kaiyan-sheng <[email protected]>

kaiyan-sheng added a commit that referenced this pull request Oct 15, 2021

Check when response content type is nil (#28457) (#28478)

bbc3b7a

(cherry picked from commit ccc2206) Co-authored-by: kaiyan-sheng <[email protected]>

Icedroid pushed a commit to Icedroid/beats that referenced this pull request Nov 1, 2021

Check when response content type is nil (elastic#28457)

c299de3

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Check when response content type is nil #28457

Check when response content type is nil #28457

kaiyan-sheng commented Oct 14, 2021 •

edited

Loading

mergify bot commented Oct 14, 2021

elasticmachine commented Oct 14, 2021

elasticmachine commented Oct 14, 2021 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

andrewkroh left a comment

aspacca commented Oct 15, 2021

aspacca Oct 15, 2021

Check when response content type is nil #28457

Check when response content type is nil #28457

Conversation

kaiyan-sheng commented Oct 14, 2021 • edited Loading

What does this PR do?

Checklist

mergify bot commented Oct 14, 2021

elasticmachine commented Oct 14, 2021

elasticmachine commented Oct 14, 2021 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

🤖 GitHub comments

andrewkroh left a comment

Choose a reason for hiding this comment

aspacca commented Oct 15, 2021

aspacca Oct 15, 2021

Choose a reason for hiding this comment

kaiyan-sheng commented Oct 14, 2021 •

edited

Loading

elasticmachine commented Oct 14, 2021 •

edited by jenkins-beats-ci bot

Loading