Agent must restart Beats when output is changed

[urso]

In libbeat the unit tests for updating the output dynamically have been disabled due to flakiness. It seemed like events get lost when updating the output. Inputs that rely on end-to-end ACK (especially in Filebeat) might not receive all ACKs for published events, or the bookeeping might bet out of sync if the outputs loose events. At worst this can lead to a deadlock (local to single inputs) in Filebeat.

As the API to reload the output in libbeat can't be assumed to be stable we need the Agent to restart the Beat if the output is reconfigured.

[elasticmachine]

Pinging @elastic/agent (Team:Agent)

[ph]

Ok, I didn't know there were disabled? This mean this issue is there for quite some time, IIRC we made changes to fix a mem leak for the output reloading.

@urso Not sure we can easily restart on output changes, do you think the bug is fixable or we need to do major refactoring to make it work?

[urso]

do you think the bug is fixable or we need to do major refactoring to make it work?

We (@ycombinator and me) did spend quite some time on it in the past, but still didn't find the cause. So we have had to disabled the tests. I don't think there is a quick and easy solution in libbeat as we are looking for a race condition that only shows every now and then in CI (I didn't manage to reproduce it locally).

[ph]

@blakerouse or @michalpristas Is what @urso is proposing doable?

[blakerouse]

@ph It is do able. Being that its a libbeat issue it should affect all sub-processes (not Endpoint) of Agent so we can just change the code to restart the entire process.

[ph]

I've added it to 7.14, if we can do it before the better.

@ferullo Small question concerning endpoint if an output settings changes like the API key does endpoint 's ES' output will pick up that change?

[ferullo]

yeah, the change would take effect immediately without an endpoint restart being required.

[ph]

@michalpristas from our email thread, we should investigate if we can only a partial reload if the credentials changes.

[michalpristas]

linking this as it may be related #23596

[ruflin]

@michalpristas Can this be closed thanks to #24907 ?