[Elastic Agent] Agent doesn't write logs to updated Elasticsearch host

[jen-huang]

Summary

I noticed that when the ES host is updated in the agent yaml output, the change does not appear to be picked up by the agent for where to send its agent logs. Other data streams (like system ones) do appear to pick up the change and sends the data to the new location correctly.

outputs:
  default:
    type: elasticsearch
    hosts:
      - 'http://localhost:9201'

Steps

1. Start Elasticsearch on non-default port, e.g. 9201
2. Start Kibana with --elasticsearch.hosts=http://localhost:9201 flag to point to ES
3. Enroll agent into Fleet with Default policy
4. Observe that no data (no agent logs or any other data streams) is ingested because Fleet defaults incorrectly to port 9200 for ES
5. Update Elasticsearch URL in Fleet settings flyout to the correct URL, http://localhost:9201
6. Wait a bit for Default policy to update with new ES output and for enrolled agent to pick up the change
7. Observe that system data streams start coming in, but Elastic Agent logs do not

If I uninstall that agent and enroll it again, the new agent does send its logs to the right ES, so it seems to only be a problem when the ES host is changed while the agent is already running.

[elasticmachine]

Pinging @elastic/agent (Team:Agent)

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[EricDavisX]

while this is a pain, I think it is ok to fix it in 7.13 cycle - @ph you agree?

[michalpristas]

tested with master cloud kibana and local agent (darwin) and could not reproduce
@jh which version you used?

code wise it should get updated as well

1. deployed kibana and changed ES output to point to non existing localhost
2. agent enrolled
3. get _cat/indices show nothing expected
4. logs tab in agent detail shows nothing as expected
5. updated ES output to point to cloud
6. get _cat/indices includes .ds-logs-elastic_agent.metricbeat-default and .ds-metrics-elastic_agent.elastic_agent-default
7. logs visible in UI

[urso]

The issue reported here sounds to be related to: #24538

It might be quite difficult to reproduce it. The next time this happens please check connection attempts with netstat/wireshark. The potential issue described in #24538 can lead to a deadlock in Filebeat, such that Filebeat eventually will stop sending any bulk requests.

Restarting the agent should normally resolve the issue. If restarting does not work we might have a problem that the updated configuration is not properly propagated. @jh Did you manage to reproduce it?

[EricDavisX]

I will call out the larger scale deployment need here: will an admin of a 100k Agent system know this is going on with a particular Agent, do we have way to provide observability on it? And if/when identified, do we have documented remedy of restarting the Agent in our troubleshooting docs? The formal troubleshooting docs are currently this link: https://www.elastic.co/guide/en/fleet/master/fleet-troubleshooting.html

[kvch]

I have tried to break it with various port changes to ES ports, everything worked as expected.