bad indexing performance in elasticsearch 8.2.0

[AlexanderOtt85]

Elasticsearch Version

8.2.0

Installed Plugins

No response

Java Version

bundled

OS Version

docker-image = docker.elastic.co/elasticsearch/elasticsearch:8.2.0

Problem Description

We are trying to migrate from Elasticsearch 6.7.1 to Elasticsearch 8.2.0.

With the same amount of documents per bulk request (100 docs per request), we always get a socket timeout in Elasticsearch 8.2.0. Everything works fine in Elasticsearch 6.7.1.

I noticed that indexing the same document in Elasticsearch 8.2.0 takes 44152ms. In Elasticsearch 6.7.1, indexing takes 50ms.

Steps to Reproduce

elasticsearch_8.2.0.txt
elasticsearch_6.7.1.txt

Logs (if relevant)

No response

[ywelsch]

I've run the steps and was able to reproduce the issue. The document takes multiple seconds to index on 8.x while it takes fractions of a second to index on 6.x. There looks to be a regression here, hot_threads have not been very helpful pinpointing it though.

I've minimized the example that shows the issue (only the "du": { "bc": [ ... ] part of the document seems to be relevant, and adding more items to that array looks to lead to blow-up of indexing time). I suspect it has to do something with these weird copy_to clauses.

Can you provide more context on why there is so much cross-copying?

[elasticmachine]

Pinging @elastic/es-search (Team:Search)

[AlexanderOtt85]

Regarding to our much cross-copying, we have alot of different searchfields on different layers of our search tree in our application. Therefore we do much cross-copying. Up until Elasticsearch 6.7.1, this always worked without any problems.

What may also be interesting in this context is the index size for the index request which i provided previously

• In Elasticsearch 6.7.1 the size is about 137.7kb
• In Elasticsearch 8.2.0 the size is about 18.5mb