Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Gracefully handle less recent metric data prior to data stream creation #99343

Open
martijnvg opened this issue Sep 8, 2023 · 2 comments
Open

Comments

@martijnvg
Copy link
Member

Tsdb data streams are designed to accept live metric data. So when a tsdb data stream gets created by default it will accept documents with @timestamp field that have a value from $now-2h to $now+2h. This seems to work well in most cases.

However when metric data is queued up this may not be ideal. This can lead to a large volume of metrics not being indexed because it was older than 2 hours. The 4 hour time window is configurable, but changing that isn't ideal. For example it can lead to very large first backing index, because the older data also gets indexed into the same backing index.

Note that this is only an issue at the time a tsdb data stream is created. After it has been created queued metrics can be indexed without an issue.

This issue is a place holder to keep track of how less recent data prior to the creation of a tsdb data stream can be indexed correctly (and not being dropped).

Note that the issue (#91340) for handling historic metric data is somewhat related to this issue, but it is a different use case, that likely needs a different solution.

@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-analytics-geo (Team:Analytics)

@elasticsearchmachine elasticsearchmachine added the Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) label Sep 8, 2023
@wchaparro wchaparro removed the Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) label Mar 20, 2024
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-storage-engine (Team:StorageEngine)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants