Bulk create multiple ingest pipelines

[ycombinator]

In the same vein as #35495, @ruflin wondered if Elastisearch could support creating multiple ingest pipelines at once, in bulk, in an all-or-nothing fashion. Filebeat may soon support the loading of multiple ingest pipelines, so this would be a useful feature to have in Elasticsearch.

[elasticmachine]

Pinging @elastic/es-core-features

[ruflin]

I think this is especially valuable in the context of the pipeline processors where multiple pipelines are needed: https://www.elastic.co/guide/en/elasticsearch/reference/master/pipeline-processor.html

[dakrone]

This has been open for quite a while, and we haven't made much progress on this due to focus in other areas. For now I'm going to close this as something we aren't planning on implementing. We can re-open it later if needed.

[ruflin]

@amitkanfer @kpollich For awareness. This feature would also have been useful for Fleet that sometimes installs quite a few ingest pipelines. It could speed up and simplify the package installation process.

[amitkanfer]

@kpollich how useful this would be for fleet? i mean... what's the heaviest integration this will be useful for, and how much time will it save during installation?
also - do we have an estimation for the complexity / effort of implementing this in ES?

[kpollich]

@kpollich how useful this would be for fleet? i mean... what's the heaviest integration this will be useful for, and how much time will it save during installation?
also - do we have an estimation for the complexity / effort of implementing this in ES?

Heaviest integrations are probably things like AWS with many, many data streams available. I ran a quick installation of the AWS integration with every single input + data streams enabled and here's what I see in our APM traces:

Total time spent installing is ~4.5s.

Of that, ~1.3s is spent installing ingest pipelines.

Today, we queue up an individual ES request per ingest pipeline and execute those requests concurrently, e.g.

https://github.com/elastic/kibana/blob/05e3db182c07ea3862b81767bcb95dfd8b032744/x-pack/plugins/fleet/server/services/epm/elasticsearch/ingest_pipeline/install.ts#L198-L216

It's hard to measure what difference a bulk API being available for this in ES would make without actually having the API available to benchmark, so I'm not sure we can answer that at this time. At a minimum we'd go from having to make many concurrent requests to Elasticsearch in this case to making a single request, which will be noticeable in situations where a cluster is under heavy load during package installation.