This repository has been archived by the owner on Sep 21, 2023. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 17
Use the go-elasticsearch client for the Elasticsearch output #14
Labels
estimation:Month
Task that represents a month of work.
Team:Elastic-Agent-Data-Plane
Label for the Agent Data Plane team
Comments
This was referenced Mar 18, 2022
jlind23
added
the
Team:Elastic-Agent-Data-Plane
Label for the Agent Data Plane team
label
Mar 21, 2022
29 tasks
cmacknz
changed the title
Use the official go-elasticsearch client for the Elasticsearch output
Evaluate using the official go-elasticsearch client
Oct 31, 2022
cmacknz
changed the title
Evaluate using the official go-elasticsearch client
Use the go-elasticsearch client for the Elasticsearch output
Oct 31, 2022
faec
added
estimation:Month
Task that represents a month of work.
and removed
estimation:Week
Task that represents a week of work.
labels
Nov 7, 2022
Corrected the link to the APM modelindexer code to https://github.com/elastic/apm-server/tree/02740f761049d6c04a283ef4708cdeb51c997b74/internal/model/modelindexer in the description. https://github.com/elastic/apm-server/blob/6e7546304e4d2f0829120e3dfcf7c60c054d5e1a/dev_docs/ARCHITECTURE.md#modelindexer-architecture contains a brief description of their architecture. I don't know that it will directly map to what we eventually do, but it will help navigate the code if we need to explore it. |
6 tasks
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Labels
estimation:Month
Task that represents a month of work.
Team:Elastic-Agent-Data-Plane
Label for the Agent Data Plane team
The existing shipper Elasticsearch output uses the legacy Beats Elasticsearch client, https://github.com/elastic/beats/tree/main/libbeat/esleg.
The shipper should switch to the official go-elasticsearch client and use its BulkIndexer for writing events to Elasticsearch. In particular, the BulkIndexer will give us the desired set of tuning parameters described in #28 out of the box via the
FlushBytes
andFlushInterval
parameters.The APM server team has already done this migration successfully, with the caveat that they forked the
BulkIndexer
into aModelIndexer
specialized to the APM data types.The code for the APM output is at https://github.com/elastic/apm-server/tree/02740f761049d6c04a283ef4708cdeb51c997b74/internal/model/modelindexer. The main entrypoint is ProcessBatch. The initial sequence of PRs are here for reference:
Exposing the new go-elasticsearch configuration will be handled in follow up issues. The scope of this issue is to switch the existing MVP Elasticsearch output implementation from #137 to use the go-elasticsearch client and its BulkIndexer.
The text was updated successfully, but these errors were encountered: