-
Notifications
You must be signed in to change notification settings - Fork 4.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix recovering from invalid output configuration under Elastic-Agent #36016
Fix recovering from invalid output configuration under Elastic-Agent #36016
Conversation
This pull request does not have a backport label.
To fixup this pull request, you need to add the backport labels for the needed
|
Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane) |
This pull request is now in conflicts. Could you fix it? 🙏
|
This commit fixes two problems in the `ManagerV2`: - 1. If the first output configuration was invalid, it would never be "saved", so if a new output configuration was received, the ManagerV2 would try to reload it even with `stopOnOutputReload` enabled. 2. When `stopOnOutputReload` was set the output reload was skipped, but the ManagerV2 would still proceed with reloading inputs. This created a race condition where some inputs/runners would receive their stop signal before even start, effectively locking the shutdown process.
This commit add tests to ensure Beats can recover from an invalid output configuration when running under Elastic-Agent. It also refactors some of the code used to mock the Elastic-Agent and aggregates some shared code into a single package.
24e4cbe
to
e3367ed
Compare
a3abbbe
to
8c8bb9a
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Tested locally, individually reverting the two fixes here (exiting reload when a restart is detected, unconditionally saving the output configuration) both cause the new test to fail.
…36016) * Fix recovering from invalid output configuration under Elastic-Agent This commit fixes two problems in the `ManagerV2`: - 1. If the first output configuration was invalid, it would never be "saved", so if a new output configuration was received, the ManagerV2 would try to reload it even with `stopOnOutputReload` enabled. 2. When `stopOnOutputReload` was set the output reload was skipped, but the ManagerV2 would still proceed with reloading inputs. This created a race condition where some inputs/runners would receive their stop signal before even start, effectively locking the shutdown process. * Add and refactor tests This commit add tests to ensure Beats can recover from an invalid output configuration when running under Elastic-Agent. It also refactors some of the code used to mock the Elastic-Agent and aggregates some shared code into a single package. * PR improvements * Add changelog * PR improvements (cherry picked from commit 15a9ec7) # Conflicts: # libbeat/tests/integration/framework.go
…36016) (#36022) * Fix recovering from invalid output configuration under Elastic-Agent This commit fixes two problems in the `ManagerV2`: - 1. If the first output configuration was invalid, it would never be "saved", so if a new output configuration was received, the ManagerV2 would try to reload it even with `stopOnOutputReload` enabled. 2. When `stopOnOutputReload` was set the output reload was skipped, but the ManagerV2 would still proceed with reloading inputs. This created a race condition where some inputs/runners would receive their stop signal before even start, effectively locking the shutdown process. * Add and refactor tests This commit add tests to ensure Beats can recover from an invalid output configuration when running under Elastic-Agent. It also refactors some of the code used to mock the Elastic-Agent and aggregates some shared code into a single package. * PR improvements * Add changelog * PR improvements (cherry picked from commit 15a9ec7) # Conflicts: # libbeat/tests/integration/framework.go Co-authored-by: Tiago Queiroz <[email protected]> Co-authored-by: Craig MacKenzie <[email protected]>
…lastic#36016) * Fix recovering from invalid output configuration under Elastic-Agent This commit fixes two problems in the `ManagerV2`: - 1. If the first output configuration was invalid, it would never be "saved", so if a new output configuration was received, the ManagerV2 would try to reload it even with `stopOnOutputReload` enabled. 2. When `stopOnOutputReload` was set the output reload was skipped, but the ManagerV2 would still proceed with reloading inputs. This created a race condition where some inputs/runners would receive their stop signal before even start, effectively locking the shutdown process. * Add and refactor tests This commit add tests to ensure Beats can recover from an invalid output configuration when running under Elastic-Agent. It also refactors some of the code used to mock the Elastic-Agent and aggregates some shared code into a single package. * PR improvements * Add changelog * PR improvements
What does this PR do?
This PR fixes two problems in the
ManagerV2
:If the first output configuration was invalid, it would never be
"saved", so if a new output configuration was received, the ManagerV2
would try to reload it even with
stopOnOutputReload
enabled.When
stopOnOutputReload
was set the output reload was skipped,but the ManagerV2 would still proceed with reloading inputs. This
created a race condition where some inputs/runners would receive their
stop signal before even start, effectively locking the shutdown
process.
It also refactors some of the code used to mock the Elastic-Agent and
aggregates some shared code into a single package.
Why is it important?
Checklist
- [ ] I have made corresponding changes to the documentation- [ ] I have made corresponding change to the default configuration filesCHANGELOG.next.asciidoc
orCHANGELOG-developer.next.asciidoc
.## Author's ChecklistHow to test this PR locally
Run the Go integration tests. Optionally you can also run the single test added by this PR:
Related issues
## Use cases## Screenshots## Logs