Skip to content
This repository has been archived by the owner on Sep 21, 2023. It is now read-only.

Use the go-elasticsearch client for the Elasticsearch output #14

Closed
cmacknz opened this issue Mar 18, 2022 · 1 comment · Fixed by #187
Closed

Use the go-elasticsearch client for the Elasticsearch output #14

cmacknz opened this issue Mar 18, 2022 · 1 comment · Fixed by #187
Assignees
Labels
estimation:Month Task that represents a month of work. Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team

Comments

@cmacknz
Copy link
Member

cmacknz commented Mar 18, 2022

The existing shipper Elasticsearch output uses the legacy Beats Elasticsearch client, https://github.com/elastic/beats/tree/main/libbeat/esleg.

The shipper should switch to the official go-elasticsearch client and use its BulkIndexer for writing events to Elasticsearch. In particular, the BulkIndexer will give us the desired set of tuning parameters described in #28 out of the box via the FlushBytes and FlushInterval parameters.

The APM server team has already done this migration successfully, with the caveat that they forked the BulkIndexer into a ModelIndexer specialized to the APM data types.

The code for the APM output is at https://github.com/elastic/apm-server/tree/02740f761049d6c04a283ef4708cdeb51c997b74/internal/model/modelindexer. The main entrypoint is ProcessBatch. The initial sequence of PRs are here for reference:

Exposing the new go-elasticsearch configuration will be handled in follow up issues. The scope of this issue is to switch the existing MVP Elasticsearch output implementation from #137 to use the go-elasticsearch client and its BulkIndexer.

@jlind23 jlind23 added the Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team label Mar 21, 2022
@faec faec added the estimation:Week Task that represents a week of work. label Jul 14, 2022
@cmacknz cmacknz added the v8.6.0 label Sep 14, 2022
@cmacknz cmacknz changed the title Use the official go-elasticsearch client for the Elasticsearch output Evaluate using the official go-elasticsearch client Oct 31, 2022
@cmacknz cmacknz changed the title Evaluate using the official go-elasticsearch client Use the go-elasticsearch client for the Elasticsearch output Oct 31, 2022
@faec faec added estimation:Month Task that represents a month of work. and removed estimation:Week Task that represents a week of work. labels Nov 7, 2022
@cmacknz
Copy link
Member Author

cmacknz commented Nov 8, 2022

Corrected the link to the APM modelindexer code to https://github.com/elastic/apm-server/tree/02740f761049d6c04a283ef4708cdeb51c997b74/internal/model/modelindexer in the description.

https://github.com/elastic/apm-server/blob/6e7546304e4d2f0829120e3dfcf7c60c054d5e1a/dev_docs/ARCHITECTURE.md#modelindexer-architecture contains a brief description of their architecture. I don't know that it will directly map to what we eventually do, but it will help navigate the code if we need to explore it.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
estimation:Month Task that represents a month of work. Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants