Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add statistics about rate and reason of dropped and degraded documents #95536

Open
felixbarny opened this issue Apr 25, 2023 · 2 comments
Open

Add statistics about rate and reason of dropped and degraded documents #95536

felixbarny opened this issue Apr 25, 2023 · 2 comments
Assignees
@jbaiera
Labels
:Data Management/Stats Statistics tracking and retrieval APIs >enhancement Team:Data Management Meta label for data/management team

Comments

@felixbarny
Copy link
Member

Today, we don't know how many documents ES drops. Having more metrics and logs about the number of successful, degraded (a doc with an _ignored field), and rejected documents could help to prioritize improvements in specific areas. We'd also want to drill into detailed information about the data stream and index (not the backing index) as well as the reason of the failure.
@elasticsearchmachine elasticsearchmachine added the needs:triage Requires assignment of a team area label label Apr 25, 2023
@felixbarny felixbarny added :Data Management/Stats Statistics tracking and retrieval APIs and removed needs:triage Requires assignment of a team area label labels Apr 25, 2023
@elasticsearchmachine elasticsearchmachine added the Team:Data Management Meta label for data/management team label Apr 25, 2023
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-data-management (Team:Data Management)

@felixbarny
Copy link
Member Author

Update: this will be implemented on top of the failure store:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jbaiera jbaiera

Labels
:Data Management/Stats Statistics tracking and retrieval APIs >enhancement Team:Data Management Meta label for data/management team
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@dakrone @jbaiera @felixbarny @elasticsearchmachine