fix chain call issue in HTTP JSON input

[kush-elastic]

What does this PR do?

In HTTP JSON input of filebeat, chain calls were being executed and every response was collected first then they were published. So, it was using a lot of memory. Therefore filebeat was exiting without publishing any event.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

• Relates [Filebeat][httpjson] Fix httpjson to not store responses in memory and publish as soon as received in Chain Calls #34310
• Relates https://github.com/elastic/sdh-beats/issues/2720

Use cases

This PR contains changes to execute the publishing and collection of events concurrently. So, events will not be stored in memory while being collected and published later. Instead, events collection, processing, and publishing will be done simultaneously.

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @kush-elastic? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2023-07-10T06:48:40.745+0000

• Duration: 76 min 42 sec

Test stats 🧪

Test	Results
Failed	0
Passed	3055
Skipped	178
Total	3233

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b httjson-chaincall-fix upstream/httjson-chaincall-fix
git merge upstream/main
git push upstream httjson-chaincall-fix

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b httjson-chaincall-fix upstream/httjson-chaincall-fix
git merge upstream/main
git push upstream httjson-chaincall-fix

[efd6]

The invariant docs need to be in place, the diff is suggestion to consider.

[kush-elastic]

@kush-elastic can you resolve the conflicts so we can we go ahead and merge this ?

I have resolved the merge conflicts. let's wait for CI to pass.

[efd6]

LGTM after final nits.

[ShourieG]

@kush-elastic after having gone through the code and having some internal discussions, this is certainly not a trivial change as it changes the fundamentals of chaining and how it functions. Since httpjson is used in most of our integrations we need to be extra careful in implementing major overhauls, so I kindly ask that the following be done :-

1. Keep the old logic as a fallback that can be easily toggled on/off if the need arises, this is extremely important as it will help us avoid a lot of hassle and user outrage if something breaks-down at scale. This fallback can be removed later after the current code is stable for a while. This change is no-trivial and will possibly require a re-work of the logic. The fallback toggle could be done as part of the config as then it would be very easy to toggle it on/off.

2. Maintain an official documentation detailing the changes to functionality in the current flow and add flow diagrams if necessary.

3. Add more meaningful tests that uses data at scale/large amounts along with tests for the fallback mechanism also.

With these changes, I feel this PR would be ready to be merged.

[jamiehynds]

@kush-elastic is this non-trivial change purely related to Salesforce or are there other integrations that require this new approach to chained calls? Since this PR was originally raised, we've added a new CEL input, which may be a better fit for this use case, and avoid making fundamental changes to httpjson just for the Salesforce integration.

While we don't have a perfect approach to switching inputs in an existing integration, we do have some options which we can walk through. We are currently switching inputs on our Office 365 integration.

cc @SubhrataK

[narph]

@kush-elastic , @SubhrataK , just a heads up here, we have planned efforts to improve and simplify the httpjson input #35432. After some discussion we've decided to hold off on the changes from this PR until the refactoring is complete and reassess the functionality after.

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b httjson-chaincall-fix upstream/httjson-chaincall-fix
git merge upstream/main
git push upstream httjson-chaincall-fix

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b httjson-chaincall-fix upstream/httjson-chaincall-fix
git merge upstream/main
git push upstream httjson-chaincall-fix

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @kush-elastic? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[ShourieG]

Closing this issue as this was handled in the following meta issue : #35432