[Backport 2.x] Introducing ZStd compression codec plugin (#9658) #9832

andrross · 2023-09-06T17:06:04Z

Backports 76f1b52 from #9658 to 2.x

github-actions · 2023-09-06T17:10:13Z

Compatibility status:

Checks if related components are compatible with change 4096817

Incompatible components

Skipped components

Compatible components

github-actions · 2023-09-06T17:18:27Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/24701/
CommitID: 4096817
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-09-06T18:28:48Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/24724/
CommitID: 13639a8
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

* introducing zstd compression codec plugin Signed-off-by: Sarthak Aggarwal <[email protected]> * Moving zstd compression codec as a plugin Signed-off-by: Sarthak Aggarwal <[email protected]> * introducing zstd compression codec plugin Signed-off-by: Sarthak Aggarwal <[email protected]> * Adding checks to EngineConfig and fixing tests Signed-off-by: Prabhakar Sithanandam <[email protected]> * incorporating review comments Signed-off-by: Sarthak Aggarwal <[email protected]> * fixing tests Signed-off-by: Sarthak Aggarwal <[email protected]> * introducing zstd compression codec plugin Signed-off-by: Sarthak Aggarwal <[email protected]> * addressing review comments Signed-off-by: Sarthak Aggarwal <[email protected]> * nit fixes Signed-off-by: Sarthak Aggarwal <[email protected]> * implementing codec aliases Signed-off-by: Sarthak Aggarwal <[email protected]> * addressing review comments Signed-off-by: Sarthak Aggarwal <[email protected]> * review comments Signed-off-by: Sarthak Aggarwal <[email protected]> * moving codec aliases to custom codec Signed-off-by: Sarthak Aggarwal <[email protected]> * adding zstd default codec for backward compatibility Signed-off-by: Sarthak Aggarwal <[email protected]> * renaming to deprecated codec Signed-off-by: Sarthak Aggarwal <[email protected]> * incorporating review comments Signed-off-by: Sarthak Aggarwal <[email protected]> * nit fixes Signed-off-by: Sarthak Aggarwal <[email protected]> --------- Signed-off-by: Sarthak Aggarwal <[email protected]> Signed-off-by: Prabhakar Sithanandam <[email protected]> Signed-off-by: Andrew Ross <[email protected]> Co-authored-by: Prabhakar Sithanandam <[email protected]> Co-authored-by: Andrew Ross <[email protected]> (cherry picked from commit 76f1b52) Signed-off-by: Andrew Ross <[email protected]>

github-actions · 2023-09-06T18:54:26Z

Compatibility status:

Checks if related components are compatible with change 13639a8

Incompatible components

Incompatible components: [https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/cross-cluster-replication.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/performance-analyzer-rca.git]

github-actions · 2023-09-06T19:02:55Z

Compatibility status:

Checks if related components are compatible with change a1d85ea

Incompatible components

Incompatible components: [https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/cross-cluster-replication.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/performance-analyzer-rca.git]

github-actions · 2023-09-06T19:20:39Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      2 org.opensearch.index.shard.RemoteStoreRefreshListenerTests.classMethod
      1 org.opensearch.index.shard.RemoteStoreRefreshListenerTests.testReplicaPromotion

URL: https://build.ci.opensearch.org/job/gradle-check/24728/
CommitID: a1d85ea
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

codecov · 2023-09-06T19:22:52Z

Codecov Report

Merging #9832 (a1d85ea) into 2.x (0d4263e) will decrease coverage by 0.74%.
Report is 1 commits behind head on 2.x.
The diff coverage is 87.30%.

@@             Coverage Diff              @@
##                2.x    #9832      +/-   ##
============================================
- Coverage     71.32%   70.59%   -0.74%     
+ Complexity    58565    57945     -620     
============================================
  Files          4815     4820       +5     
  Lines        274976   275019      +43     
  Branches      40412    40422      +10     
============================================
- Hits         196135   194151    -1984     
- Misses        62415    64223    +1808     
- Partials      16426    16645     +219

Files Changed	Coverage Δ
...customcodecs/Lucene95CustomStoredFieldsFormat.java	`74.07% <ø> (ø)`
.../index/codec/customcodecs/ZstdCompressionMode.java	`86.66% <ø> (ø)`
.../codec/customcodecs/ZstdNoDictCompressionMode.java	`79.45% <ø> (ø)`
.../java/org/opensearch/index/codec/CodecAliases.java	`0.00% <0.00%> (ø)`
.../java/org/opensearch/index/codec/CodecService.java	`63.15% <ø> (+5.15%)`	⬆️
.../index/codec/customcodecs/ZstdDeprecatedCodec.java	`50.00% <50.00%> (ø)`
...java/org/opensearch/index/engine/EngineConfig.java	`94.61% <89.47%> (-1.45%)`	⬇️
...h/index/codec/customcodecs/CustomCodecService.java	`93.33% <93.33%> (ø)`
...ch/index/codec/customcodecs/CustomCodecPlugin.java	`100.00% <100.00%> (ø)`
.../codec/customcodecs/CustomCodecServiceFactory.java	`100.00% <100.00%> (ø)`
... and 3 more

... and 517 files with indirect coverage changes

andrross · 2023-09-06T20:00:30Z

@reta This is ready to go. Thanks!

* introducing zstd compression codec plugin * Moving zstd compression codec as a plugin * introducing zstd compression codec plugin * Adding checks to EngineConfig and fixing tests * incorporating review comments * fixing tests * introducing zstd compression codec plugin * addressing review comments * nit fixes * implementing codec aliases * addressing review comments * review comments * moving codec aliases to custom codec * adding zstd default codec for backward compatibility * renaming to deprecated codec * incorporating review comments * nit fixes --------- (cherry picked from commit 76f1b52) Signed-off-by: Sarthak Aggarwal <[email protected]> Signed-off-by: Prabhakar Sithanandam <[email protected]> Signed-off-by: Andrew Ross <[email protected]> Co-authored-by: Sarthak Aggarwal <[email protected]> Co-authored-by: Prabhakar Sithanandam <[email protected]> (cherry picked from commit 90310c3) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* introducing zstd compression codec plugin * Moving zstd compression codec as a plugin * introducing zstd compression codec plugin * Adding checks to EngineConfig and fixing tests * incorporating review comments * fixing tests * introducing zstd compression codec plugin * addressing review comments * nit fixes * implementing codec aliases * addressing review comments * review comments * moving codec aliases to custom codec * adding zstd default codec for backward compatibility * renaming to deprecated codec * incorporating review comments * nit fixes --------- (cherry picked from commit 76f1b52) (cherry picked from commit 90310c3) Signed-off-by: Sarthak Aggarwal <[email protected]> Signed-off-by: Prabhakar Sithanandam <[email protected]> Signed-off-by: Andrew Ross <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Sarthak Aggarwal <[email protected]> Co-authored-by: Prabhakar Sithanandam <[email protected]>

…pensearch-project#9832) (opensearch-project#9844) * introducing zstd compression codec plugin * Moving zstd compression codec as a plugin * introducing zstd compression codec plugin * Adding checks to EngineConfig and fixing tests * incorporating review comments * fixing tests * introducing zstd compression codec plugin * addressing review comments * nit fixes * implementing codec aliases * addressing review comments * review comments * moving codec aliases to custom codec * adding zstd default codec for backward compatibility * renaming to deprecated codec * incorporating review comments * nit fixes --------- (cherry picked from commit 76f1b52) (cherry picked from commit 90310c3) Signed-off-by: Sarthak Aggarwal <[email protected]> Signed-off-by: Prabhakar Sithanandam <[email protected]> Signed-off-by: Andrew Ross <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Sarthak Aggarwal <[email protected]> Co-authored-by: Prabhakar Sithanandam <[email protected]>

andrross force-pushed the backport/backport-9658-to-2.x branch from 4096817 to 13639a8 Compare September 6, 2023 18:14

andrross force-pushed the backport/backport-9658-to-2.x branch from 13639a8 to a1d85ea Compare September 6, 2023 18:35

reta approved these changes Sep 6, 2023

View reviewed changes

reta merged commit 90310c3 into opensearch-project:2.x Sep 6, 2023

andrross deleted the backport/backport-9658-to-2.x branch September 6, 2023 20:06

reta added the backport 2.10 Backport to 2.10 branch label Sep 6, 2023

opensearch-trigger-bot bot mentioned this pull request Sep 6, 2023

[Backport 2.10] [Backport 2.x] Introducing ZStd compression codec plugin (#9658) #9844

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Backport 2.x] Introducing ZStd compression codec plugin (#9658) #9832

[Backport 2.x] Introducing ZStd compression codec plugin (#9658) #9832

andrross commented Sep 6, 2023

github-actions bot commented Sep 6, 2023

github-actions bot commented Sep 6, 2023

github-actions bot commented Sep 6, 2023

github-actions bot commented Sep 6, 2023

github-actions bot commented Sep 6, 2023

github-actions bot commented Sep 6, 2023

codecov bot commented Sep 6, 2023

andrross commented Sep 6, 2023

[Backport 2.x] Introducing ZStd compression codec plugin (#9658) #9832

[Backport 2.x] Introducing ZStd compression codec plugin (#9658) #9832

Conversation

andrross commented Sep 6, 2023

github-actions bot commented Sep 6, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Sep 6, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 6, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 6, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Sep 6, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Sep 6, 2023

Gradle Check (Jenkins) Run Completed with:

codecov bot commented Sep 6, 2023

Codecov Report

andrross commented Sep 6, 2023