Failed document handler

[felixbarny]

Instead of dropping documents that have failed ingestion due to an exception during pipeline execution or indexing, it should be possible to store the failed document.

High-level options where to store the failed documents:

1. In a dedicated data stream
2. Within the same data stream, in dedicated failure backing indices
3. Within the same index, only storing the _source and with an indicator that these reflect failed documents

Tasks

Preview Give feedback

1. Basic functionality of data streams that can have failure store backing indices with a hard-coded shard count of one Add ability to create a data stream failure store #99134
2. Send failures during ingest pipeline execution to the failure store
3. Send failures during shard-level failures to the store (mapping issues etc.)
4. Flag for search API to be able to search items in the data store
5. Lazily create the failure store
6. Turn the failure store on by default
7. Testing edge cases: verify CCR, modify APIs, snapshot/restore, etc.
8. Evaluate scalability and make sure this plays nicely with shard auto-scaling

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[lachlann562]

This would be extremely helpful, we recently discovered we were losing a huge number of messages because of inconsistency in data type and we couldn't find any easy way to identify these messages or recover the data associated with them. The abscence of log records is a huge risk when you are reliant on ELK as the "source of truth".

[ruflin]

@lachlann562 Glad to hear this will be useful for you. To also expose the failure in the UI as soon as the feature lands, we are working on a dataset quality page. Some related issue can be found here: elastic/kibana#184572