[Backport 2.x] Optimize read write lock constructs during translog upload to remote store

[opensearch-trigger-bot]

Backport b5cc002 from #9636.

[github-actions]

Compatibility status:

Checks if related components are compatible with change d091853

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/neural-search.git]

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.repositories.azure.AzureBlobContainerRetriesTests.testWriteBlobWithRetries

• URL: https://build.ci.opensearch.org/job/gradle-check/25797/
• CommitID: d091853
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[codecov]

Codecov Report

Merging #10102 (d091853) into 2.x (9d3ee61) will decrease coverage by 0.01%.
Report is 1 commits behind head on 2.x.
The diff coverage is 72.34%.

@@             Coverage Diff              @@
##                2.x   #10102      +/-   ##
============================================
- Coverage     70.85%   70.84%   -0.01%     
+ Complexity    58295    58226      -69     
============================================
  Files          4811     4811              
  Lines        275183   275209      +26     
  Branches      40442    40447       +5     
============================================
+ Hits         194976   194977       +1     
+ Misses        63573    63508      -65     
- Partials      16634    16724      +90     

Files Changed	Coverage Δ
...org/opensearch/index/fielddata/IndexFieldData.java	74.07% <0.00%> (-1.40%)	⬇️
...g/transfer/TranslogCheckpointTransferSnapshot.java	87.01% <ø> (+1.11%)	⬆️
...ain/java/org/opensearch/index/IndexSortConfig.java	73.17% <33.33%> (-1.83%)	⬇️
...rg/opensearch/index/translog/RemoteFsTranslog.java	73.36% <72.41%> (-0.17%)	⬇️
...nsearch/index/fielddata/IndexNumericFieldData.java	85.07% <80.00%> (-1.14%)	⬇️
...c/main/java/org/opensearch/index/IndexService.java	75.59% <100.00%> (+0.32%)	⬆️
.../main/java/org/opensearch/index/IndexSettings.java	87.19% <100.00%> (+0.06%)	⬆️
...dex/translog/transfer/TranslogTransferManager.java	79.84% <100.00%> (+0.07%)	⬆️

... and 473 files with indirect coverage changes