[BUG] Opensearch 1.1.0 ShardBulkDocs Metrics is always empty

[pmarjou22]

What is the bug?
ShardBulkDocs Metrics is always empty when calling performance Analyzer API. Other metrics are working.

How can one reproduce the bug?
Steps to reproduce the behavior:

1. docker compose enclosed
   docker-compose.txt

2. Follow step to activate performance analyzer : https://opensearch.org/docs/latest/opensearch/install/docker/#optional-set-up-performance-analyzer

3. Create test index and POST documents in this index PUT test (see enclosed)
   postdata.txt

4. while pushing documents, call from your browser : http://localhost:9600/_plugins/_performanceanalyzer/metrics?metrics=ShardBulkDocs,ShardEvents&agg=sum,sum
   you 'll get only ShardEvents numbers and no data on ShardBulkDocs

   {"CkXsbxtMTDisNx2ahHqrIw": {"timestamp": 1635425175000, "data": {"fields":[{"name":"ShardBulkDocs","type":"DOUBLE"},{"name":"ShardEvents","type":"DOUBLE"}],"records":[[0.0,27.0]]}}}

What is the expected behavior?
On previous Open Distro 1.13.1 this call was working

What is your host/environment?

• Opensearch 1.1.0 Docker

[pmarjou22]

I investigated and found out that there are no more metrics from operation : "shardbulk" available from the API

[pmarjou22]

we have upgraded to 1.2.0, the bug is still here

[systemosyn]

I face a similar issue somehow with have an opensearch cluster of two nodes, (1 hot and 1 warm), and the shardbulkdocs that we received were always equals to 0, receiving only the shardfetch and sharquery operation (there was no shardbulk operation). when I look into the eventLog files. I see an issue which is, only the shabulk started event are written, i was not able to see a single finished event of a shardbulk operation:

1667494070000.txt
1667494060000.txt
1667494065000.txt

As an ugly quick fix I manage to patch my rca agent, by removing the WHERE clause that filter events when they are not finished. I did it in org.opensearch.performanceanalyzer.reader.ShardRequestMetricsSnapshot.fetchLatency() from performance-analyzer-rca replacing lines 177 to 182 by this:

But this does not solve the root cause at all.

@rguo-aws @jotok @sruti1312 @kaushalmahi12 , Do you have any suggestion to help me to investigate this issue please? Don't hesitate if you need more information.

Thank you very much for your help.
Best regards,