[Filebeat][CometD] Add CometD input for Salesforce connector #31492
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…om/kush-elastic/beats into salesforce_cometd_input
|
Hey @cmacknz, this is the new PR for CometD input. E2E test are still in pending state. All other checks are passing successfully. |
|
/test |
|
If I check out this PR locally and run the tests it is still flaky for me. For flaky tests you need to run them more then once, I usually put them in a shell while loop with a go test flag that will disable test caching like ~/go/src/github.com/elastic/beats pr/kush-elastic/31492-1:salesforce_cometd_input ?1 ···· 7s 11:49:18 AM
❯ while go test -count 1 ./x-pack/filebeat/input/cometd/...; do :; done
ok github.com/elastic/beats/v7/x-pack/filebeat/input/cometd 5.426s
ok github.com/elastic/beats/v7/x-pack/filebeat/input/cometd 5.436s
--- FAIL: TestMultiInput (5.00s)
input_test.go:339:
Error Trace: input_test.go:339
Error: Received unexpected error:
not able to get events.
Test: TestMultiInput
FAIL
FAIL github.com/elastic/beats/v7/x-pack/filebeat/input/cometd 5.434s
FAIL |
|
Also odd, in a Jenkins run where the build is green I can see this test actually failed. Looking at the console text for https://beats-ci.elastic.co/job/Beats/job/beats/job/PR-31492/5/ I can see the following: [2022-05-03T10:24:36.970Z] === Failed
[2022-05-03T10:24:36.970Z] === FAIL: x-pack/filebeat/input/cometd TestMultiInput (5.00s)
[2022-05-03T10:24:36.970Z] input_test.go:339:
[2022-05-03T10:24:36.970Z] Error Trace: input_test.go:339
[2022-05-03T10:24:36.970Z] Error: Received unexpected error:
[2022-05-03T10:24:36.970Z] not able to get events.
[2022-05-03T10:24:36.970Z] Test: TestMultiInput
[2022-05-03T10:24:36.970Z]
[2022-05-03T10:24:36.970Z] DONE 832 tests, 7 skipped, 1 failure in 190.543s
[2022-05-03T10:24:36.970Z] Error: go test returned a non-zero value: exit status 1
[2022-05-03T10:24:36.978Z] [Pipeline] }
[2022-05-03T10:24:36.979Z] ERROR: script returned exit code 1
[2022-05-03T10:24:36.979Z] RetryingI think Jenkins might have some built in retries here |
|
Thanks for the details, I looked into those tests. there were some problems with immediate starting of all the tests. and cause of server issues. I have updated those. let's see if that works for Jenkins or not. Here's what you suggested. |
|
Thanks, I left it in the background for a while and managed to get it to fail: |
Thanks, @cmacknz! Do you have anything on top of your head which can help us resolve this? I am also trying some things out. |
|
No, but I'll look closer at the code. I have had tests like this that require 10+ minutes of continuous running to see failures, so it could be worse :) |
|
Hey @cmacknz, I was expecting empty response from server after few events. after receiving those I should not send more events in response based on condition. so what was missing is WDYT? |
|
/test |
|
Hey @cmacknz, The E2E tests are again failing. Still I am re-triggering CI with |
|
/package |
|
/test |
cmacknz
reviewed
May 5, 2022
cmacknz
reviewed
May 5, 2022
cmacknz
approved these changes
May 5, 2022
|
Hey @cmacknz, |
|
I synced with @kush-elastic regarding the status of E2E tests. We agreed to merge based on the same justification as here. |
chrisberkhout
pushed a commit
that referenced
this pull request
Jun 1, 2023
* Add authentication mechanism for cometd input for Salesforce connector
* update based on comments: clear empty line and add error.Errorf with context
* Add data collection mechanism for Salesforce cometd input
* Update the approach of the channel
* Add asciidoc for cometd input
* Resolve review comments
* Update cometd input to import forked dependency
* Temorarily remove changelog entry
* Run mage check and update
* Update go.mod and go.sum
* Update go.mod and NOTICE.txt
* Make changes with respect to ownership transfer
* Update go.mod and go.sum
* Add two unit test cases and nit
* Add unit test cases and integration tests
* Address review comments
* Resolve some linting issues
* Resolve some linting issues
* Add unit test for input
* Add integration test
* Lint issues
* Lint
* Lint
* update bayeux v1.0.3
* Add NOTICE.txt
* Add negative test cases
* requested changes
* Resolve CI lint errors
* Update go.sum
* Update NOTICE.txt
* Update unit tests
* Update unit test name
* Update tests
* Remove test.out
* Use time.After to eliminate possible flaky test
* Add buffer of 1 in message channel
* Update tests based on bayeux client status watcher changes
* incorporate bayeux changes
* Update unit tests
* resolve unit tests
* additional tests with multiple inputs, added channel_name filed in comman cometd fields
* nit: Event struct -> event struct
* test case improvement
* update looger Error -> Errorw to print formatted errors
* update docs with reference links
* Add CHANGELOG entry and update docs
* Use mapstr.M instead of common.MapStr
* Changes based on Replace common.Config and friends with config.C
* resolve make check error
* resolve comman config errors
* update unit tests
* update multiple input unit test
* Update TestMultiInput unit test
* Update minor nit to re-trigger CI
* Resolve flanky tests
* Resolve linting issues
* Resolve tests -> send empty response {}
* nit: inputType -> expectedHTTPEventCount
* nit: wg -> waitForEventCollection
Co-authored-by: yug-crest <[email protected]>
Co-authored-by: yug-elastic <[email protected]>
Co-authored-by: Craig MacKenzie <[email protected]>
Co-authored-by: yug-elastic <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Real-time logs from the Salesforce Streaming API can be collected using the cometD input. For example, live Login & Logout related logs.
Right now with this PR, when the cometd input is enabled, you can start seeing the authentication log messages that are retrieved after successful authentication using the credentials provided as a part of the configuration. You will also be able to see the real-time data being collected from the Salesforce Steaming API.
For configuration:
Why is it important?
Checklist
CHANGELOG.next.asciidocorCHANGELOG-developer.next.asciidoc.