You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository has been archived by the owner on Sep 21, 2023. It is now read-only.
NumWorkers, which we should likely map to the existing worker parameter for backwards compatibility reasons as long as we are convinced it works the same way to increase or decrease request concurrency. The BulkIndexer has a NumWorkers parameter that controls the number of concurrent requests that can be made at once, but it is not clear if it also bounds the number of connections that can be made to Elasticsearch. We should look at the underlying Elastictransport metrics to confirm the number of connections is what we expect.
The outcome of this issue should be an updated shipper configuration file that can be parsed and used to configure the go-elasticsearch BulkIndexer appropriately. The concurrency model of the go-elasticsearch BulkIndexer should also be clarified as part of this work, to confirm whether we can move forward with a single worker configuration or if we need to separate the worker configuration from the number of connections.
The text was updated successfully, but these errors were encountered:
cmacknz
changed the title
Add support for bulk_max_size configuration to the Elasticsearch output
Allow configuring the maximum size of each batch written by the Elasticsearch output
Oct 31, 2022
cmacknz
changed the title
Allow configuring the maximum size of each batch written by the Elasticsearch output
Allow configuring the Elasticsearch output's flush parameters
Oct 31, 2022
cmacknz
changed the title
Allow configuring the Elasticsearch output's flush parameters
Allow configuring the Elasticsearch output's flush and worker parameters
Oct 31, 2022
After switching to the go-elasticsearch client (#14) we need to allow the user to configure at least the following parameters:
FlushBytes
, which per Implement more efficient output tuning parameters to manage throughput #28 we should namemaximum_batch_bytes
in our configuration.FlushInterval
, which per Implement more efficient output tuning parameters to manage throughput #28 we should nameoutput_queue_flush_timeout
in our configuration.NumWorkers
, which we should likely map to the existing worker parameter for backwards compatibility reasons as long as we are convinced it works the same way to increase or decrease request concurrency. The BulkIndexer has a NumWorkers parameter that controls the number of concurrent requests that can be made at once, but it is not clear if it also bounds the number of connections that can be made to Elasticsearch. We should look at the underlying Elastictransport metrics to confirm the number of connections is what we expect.The outcome of this issue should be an updated shipper configuration file that can be parsed and used to configure the go-elasticsearch BulkIndexer appropriately. The concurrency model of the go-elasticsearch BulkIndexer should also be clarified as part of this work, to confirm whether we can move forward with a single
worker
configuration or if we need to separate the worker configuration from the number of connections.The text was updated successfully, but these errors were encountered: