Skip to content
This repository has been archived by the owner on Sep 21, 2023. It is now read-only.

Allow configuring the Elasticsearch output's flush and worker parameters #149

Closed
cmacknz opened this issue Oct 31, 2022 · 2 comments
Closed
Assignees
Labels
8.7-candidate Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team

Comments

@cmacknz
Copy link
Member

cmacknz commented Oct 31, 2022

After switching to the go-elasticsearch client (#14) we need to allow the user to configure at least the following parameters:

  1. FlushBytes, which per Implement more efficient output tuning parameters to manage throughput #28 we should name maximum_batch_bytes in our configuration.
  2. FlushInterval, which per Implement more efficient output tuning parameters to manage throughput #28 we should name output_queue_flush_timeout in our configuration.
  3. NumWorkers, which we should likely map to the existing worker parameter for backwards compatibility reasons as long as we are convinced it works the same way to increase or decrease request concurrency. The BulkIndexer has a NumWorkers parameter that controls the number of concurrent requests that can be made at once, but it is not clear if it also bounds the number of connections that can be made to Elasticsearch. We should look at the underlying Elastictransport metrics to confirm the number of connections is what we expect.

The outcome of this issue should be an updated shipper configuration file that can be parsed and used to configure the go-elasticsearch BulkIndexer appropriately. The concurrency model of the go-elasticsearch BulkIndexer should also be clarified as part of this work, to confirm whether we can move forward with a single worker configuration or if we need to separate the worker configuration from the number of connections.

@cmacknz cmacknz changed the title Add support for bulk_max_size configuration to the Elasticsearch output Allow configuring the maximum size of each batch written by the Elasticsearch output Oct 31, 2022
@cmacknz cmacknz changed the title Allow configuring the maximum size of each batch written by the Elasticsearch output Allow configuring the Elasticsearch output's flush parameters Oct 31, 2022
@cmacknz cmacknz changed the title Allow configuring the Elasticsearch output's flush parameters Allow configuring the Elasticsearch output's flush and worker parameters Oct 31, 2022
@cmacknz cmacknz added the Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team label Oct 31, 2022
@cmacknz
Copy link
Member Author

cmacknz commented Nov 8, 2022

I provided a preliminary specification for some of these parameters to the Fleet UI team to unblock UI developmenet: elastic/kibana#141508 (comment)

If we decide to implement these configurations in a different way we will need to follow up and ensure those changes are reflected in Fleet.

@jlind23
Copy link
Contributor

jlind23 commented Jan 23, 2023

Closing thanks to #227

@jlind23 jlind23 closed this as completed Jan 23, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
8.7-candidate Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team
Projects
None yet
Development

No branches or pull requests

3 participants